[Index state tracking] End-to-end design

Design how to keep track of the state of indexed namespaces / projects.

We will only index namespaces where the top-level namespace has a Duo Pro or Duo Enterprise license. An individual project can disable AI features and this should also be respected.

The Zoekt indexing pipeline implements an efficient state tracking system that can be used as an example.

We'd likely need 2 tables:

1. top-level indexed namespace. Partitioned table
2. repository state (tied to namespace)

We also need to handle orphaned data.

The outcome of this issue is a designed schema that can be implemented in [Index state tracking] Create tables for tracki... (#536221 - closed)