raft: Add membership management to Raft manager
#6303 (closed) lets Raft handle the full life cycle of a log entry. Essentially, Gitaly spawns a single-node cluster per partition in which the current node is the only member. This cluster hasn't had any cluster features, such as replication or even membership management. This issue adds membership management to raftmgr.Manager, including the following operations:
- Add a new node
- Remove a node
- Update a node
- Add a learner node. The learning node is a replica that is involved in replication but is not involved in election.
When a new node is added, the leader issues a ConfigChange log entry ([conf change types]) (https://github.com/etcd-io/raft/blob/61efb81db45c7e4eb9fba8fd13a05a63b7746df4/raftpb/raft.proto#L153). This entry follows the normal proposal steps with existing members in the Raft group. After being committed, the leader, existing members, and the new member apply that log entry for the first time. The same flow applies to the remaining operations.
Membership records are persisted into the routing table, powered by the badger KV database: #6628 (closed).
The new member's node should pick up this signal from the transportation layer and boot a Raft manager on their side. This will be covered by #6304. This issue should focus on the core logic inside the Raft manager. The networking layer is bypassed. The logic should
This issue should pay attention to observability aspects such as logs, metrics, timelines of the events, etc.
Prerequisites:
Next: