raft: Add replica placement support to Gitaly node manager.
In the recent implementation of WAL, each storage on a Gitaly server hosts its partitions. All partitions are created and managed locally by storage.Storage. A partition is accessed from other layers (gRPC server middleware for example) by its assigned local partition ID. To keep the backward compatibility with the prior identity system based on relative path, the partition ID is translated from there if the caller doesn't know the allocated partition ID. This system works fairly well for non-cluster local WAL.
Unfortunately, this system is not sufficient for Raft. From the design, a storage hosts replicas of partitions from different storages (may or may not include the ones it created). A replica's global identity is a tuple of (Partition ID, Node ID, Replica Storage name) (more about Node ID in the doc). We need to restructure the directory tree to support replica placement. The existing node manager and storage manager also need to be modified so that we can start the Transaction Manager (and corresponding Raft Manager) of a replicated partition.
Some notable requirements:
- The new structure must be compatible with both Raft and non-Raft servers. Backward compatibility allows us to turn Raft on/off freely during the rolling-out phase. We should avoid on-disk data migration.
- The list of partitions is persisted in KV DB.
- The relative Path of a repository could be considered to be unique globally. Although it is not ideal, we do store repositories in GitLab Rails' DB. This system won't go anywhere soon.
- Be aware of the object pool! All members of an object pool must co-locate in the same partition and sequentially on the same storage.
- Node ID is a part of the replica ID. The same storage receives a new node ID after re-joining. It is used to differentiate the data of two points in time. In theory, a storage needs to remove all data of the Rat group after leaving. Even if the storage missed that event (being offline for example), the data will be cleaned eventually after it resumes. So, we might consider skipping node ID.
For reference, the following chart is the current tree structure of a WAL-based storage.
storage-a
├── database
│ ├── 000005.vlog
│ ├── 000006.sst
│ ├── 000006.vlog
│ ├── 000007.sst
│ ├── DISCARD
│ ├── KEYREGISTRY
│ └── MANIFEST
├── partitions
│ ├── 19
│ │ └── 58
│ │ └── 9
│ │ └── wal
│ │ ├── 0000000000001
│ │ │ ├── MANIFEST
│ │ │ └── RAFT
│ │ ├── 0000000000002
│ │ │ └── MANIFEST
│ │ ├── 0000000000003
│ │ │ ├── 1
│ │ │ ├── 2
│ │ │ ├── 3
│ │ │ └── MANIFEST
│ │ └── 0000000000004
│ │ ├── 1
│ │ └── MANIFEST
│ ├── 2c
│ │ └── 62
│ │ └── 8
│ │ └── wal
├── @hashed
│ ├── 19
│ │ └── 58
│ │ ├── 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.git
│ │ │ ├── FETCH_HEAD
│ │ │ ├── HEAD
│ │ │ ├── config
│ │ │ ├── gitaly-language.stats
│ │ │ ├── objects
│ │ │ │ ├── info
│ │ │ │ └── pack
│ │ │ │ ├── pack-0872114703e629ffa5cbb0106a8e190306df1034.idx
│ │ │ │ ├── pack-0872114703e629ffa5cbb0106a8e190306df1034.pack
│ │ │ │ └── pack-0872114703e629ffa5cbb0106a8e190306df1034.rev
│ │ │ ├── packed-refs
│ │ │ └── refs
│ │ │ ├── heads
│ │ │ ├── keep-around
│ │ │ │ └── d2e7e613a597bec6b16c946e42c91950dcc1a215
│ │ │ └── tags
│ │ └── 19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7.wiki.git
│ │ ├── HEAD
│ │ ├── config
│ │ ├── objects
│ │ │ ├── info
│ │ │ └── pack
│ │ └── refs
│ │ ├── heads
│ │ └── tags
│ ├── 2c
│ │ └── 62
│ │ ├── 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.git
│ │ │ ├── FETCH_HEAD
│ │ │ ├── HEAD
│ │ │ ├── config
│ │ │ ├── gitaly-language.stats
│ │ │ ├── objects
│ │ │ │ ├── 17
│ │ │ │ │ └── 00b9d85c2cc1d208141878a51b00e53611bc71
│ │ │ │ ├── 3f
│ │ │ │ │ └── 3e8b1b3a7520d2b3a099ddfb271a74415d6acd
│ │ │ │ ├── 99
│ │ │ │ │ └── d082b9a2160e280b591645f0fdec679e9fa839
│ │ │ │ ├── ec
│ │ │ │ │ └── 8d2d44d888dcd85d187c8e8d59907cdea294f3
│ │ │ │ ├── info
│ │ │ │ └── pack
│ │ │ │ ├── pack-f8c57da7f8b7a6162f01cd4be10cd3318428e0ea.idx
│ │ │ │ ├── pack-f8c57da7f8b7a6162f01cd4be10cd3318428e0ea.pack
│ │ │ │ └── pack-f8c57da7f8b7a6162f01cd4be10cd3318428e0ea.rev
│ │ │ ├── packed-refs
│ │ │ └── refs
│ │ │ ├── heads
│ │ │ ├── keep-around
│ │ │ │ ├── 312d4811a44b169c7d597919b682932752a26b0a
│ │ │ │ ├── 33ab95374ebe87d910e89ef07010ae4751d1f15c
│ │ │ │ ├── 588440f66559714280628a4f9799f0c4eb880a4a
│ │ │ │ ├── 5b2fef7d28717b53baa097197e5a9c7a46609c8b
│ │ │ │ ├── 99d082b9a2160e280b591645f0fdec679e9fa839
│ │ │ │ ├── e40ecd9356ad0a882fb80170314b8a220930f4a2
│ │ │ │ ├── e4d6fbfcde06c7ca8f844df1015e4d4eb129ee66
│ │ │ │ └── fd9e76b9136bdd9fe217061b497745792fe5a5ee
│ │ │ ├── merge-requests
│ │ │ │ ├── 1
│ │ │ │ │ └── head
│ │ │ │ └── 2
│ │ │ │ ├── head
│ │ │ │ └── merge
│ │ │ ├── pipelines
│ │ │ └── tags
│ │ └── 2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3.wiki.git
│ │ ├── HEAD
│ │ ├── config
│ │ ├── objects
│ │ │ ├── info
│ │ │ └── pack
│ │ └── refs
│ │ ├── heads
│ │ └── tags
├── staging
└── tmp