WIP: CleanupOrphanedRPC
fixes: #1808 (closed)
In order to deprecate ListDirectories, we need an RPC that allows Gitaly to clean up orphaned repositories. The only place ListDirectories is used is a rake task that cleans up orphaned repositories.
If we don't want to expose the directories to the rails application, one way is for the rails application to send Gitaly all valid repositories it knows about, and Gitaly can delete everything on disk that is not included in that set known set.
CleanupOrphanedRepos will be part of the Server service. We can have two implementations, one on Praefect and one in Gitaly
Praefect implementation
- use the datastore as a source of truth for what repositories are in praefect. Compare the input list of valid repositories, take the difference. Those are the repositories we want to delete
- delete these off of the primaries by calling
RemoveRepository(we still need a mechanism for propagating deletes, but that won't be part of this MR) - before we have the datastore in sql, we will need to read the storage directories and seed the datastore with the repositories on disk on praefect startup. Otherwise, when praefect starts up the datastore will not know about any repositories that are already on disk. It will only know about new repositories that get added after praefect starts up.
- With this approach however, we should never accidentally delete more than we need to. Even if the datastore is somehow empty, we won't delete any repositories.
Gitaly Implementation
- read the repositories on disk, and compare it with the input list of valid repositories
- delete repositories that are not on the list of valid repositories.
With both Praefect and Gitaly implementations of this RPC, we can modify the (rails code)[https://gitlab.com/gitlab-org/gitlab-foss/blob/master/lib/tasks/gitlab/cleanup.rake#L7] to call CleanupOrphanedRepos