diff --git a/doc/administration/gitaly/praefect/_index.md b/doc/administration/gitaly/praefect/_index.md index 806fb39dc69874e93d7d1a04879eab564e780b92..0a18c6850010dbb3c8929b35ff17fc975fb14c4e 100644 --- a/doc/administration/gitaly/praefect/_index.md +++ b/doc/administration/gitaly/praefect/_index.md @@ -128,6 +128,93 @@ variable replication factor is tracked in [this issue](https://gitlab.com/groups As with standard Gitaly storages, virtual storages can be sharded. +### Multiple virtual storages + +You can configure multiple virtual storages in a Gitaly Cluster (Praefect) deployment. This allows you to: + +- Organize repositories into separate clusters with different performance characteristics. +- Apply different replication factors to different groups of repositories. +- Scale different parts of your infrastructure independently. + +Virtual storages are configured in `gitlab_rails['repositories_storages']` on the GitLab server. Each entry in +this hash represents a distinct virtual storage. The Praefect configuration defines which Gitaly nodes serve each +virtual storage. Repositories in different virtual storages are completely independent and are not replicated +between virtual storages. + +For example, you might configure: + +- `storage-1`: A virtual storage for critical production repositories with a replication factor of 3. +- `storage-2`: A virtual storage for less critical repositories with a replication factor of 2. + +Each virtual storage requires its own set of Gitaly nodes. + +```mermaid +%%{init: { "fontFamily": "GitLab Sans" }}%% +graph TD + accTitle: Multiple virtual storages + accDescr: Example of multiple virtual storages, one with a replication factor of three and one with a replication factor of two. + + GitLab[[GitLab server]] + Storage1[(storage-1
Praefect cluster)] + Storage2[(storage-2
Praefect cluster)] + + GitLab --> Storage1 + GitLab --> Storage2 + + Storage1 --> G1[Gitaly node 1] + Storage1 --> G2[Gitaly node 2] + Storage1 --> G3[Gitaly node 3] + + Storage2 --> G4[Gitaly node 4] + Storage2 --> G5[Gitaly node 5] +``` + +For configuration instructions, see [Configure multiple virtual storages](configure.md#configure-multiple-virtual-storages). + +### Mixed configuration + +You can configure GitLab to use a combination of: + +- Standalone Gitaly instances (direct Gitaly storage). +- Gitaly Cluster (Praefect) virtual storages. + +You can use mixed configuration when: + +- Migrating from standalone Gitaly to Gitaly Cluster (Praefect) incrementally. +- Some repositories require high availability while others do not. +- You want to optimize costs by using Gitaly Cluster (Praefect) only for critical repositories. + +In a mixed configuration, each storage is configured independently in GitLab: + +- Standalone Gitaly storages connect directly to Gitaly nodes. +- Gitaly Cluster (Praefect) storages connect to the Praefect load balancer. + +GitLab treats all configured storages equally, regardless of whether they are standalone or clustered. When creating +a new repository, GitLab selects a storage based on the configured storage weights and available capacity. + +```mermaid +%%{init: { "fontFamily": "GitLab Sans" }}%% +graph TD + accTitle: Mixed configuration + accDescr: Example result of mixed configuration, with a Gitaly Cluster (Praefect) and standalone Gitaly configured together. + + GitLab[[GitLab server]] + Praefect[(Praefect cluster)] + Standalone1[(Standalone gitaly)] + + GitLab -->|cluster storage| Praefect + GitLab -->|default storage| Standalone1 + + Praefect --> G1[Gitaly node 1] + Praefect --> G2[Gitaly node 2] + Praefect --> G3[Gitaly node 3] +``` + +For more information, see: + +- [Mixed Configuration](../configure_gitaly.md#mixed-configuration) for configuration examples. +- [Use TCP for existing GitLab instances](configure.md#use-tcp-for-existing-gitlab-instances) for migration guidance. + ## Storage layout {{< alert type="warning" >}} diff --git a/doc/administration/gitaly/praefect/configure.md b/doc/administration/gitaly/praefect/configure.md index e21054a87121631f1a59b4f63ebdf2ee2e268a34..27c627b5f4cab3b632089dacf5949fb3415967eb 100644 --- a/doc/administration/gitaly/praefect/configure.md +++ b/doc/administration/gitaly/praefect/configure.md @@ -1517,6 +1517,174 @@ gitlab_rails['repositories_storages'] = { See [Mixed Configuration](../configure_gitaly.md#mixed-configuration) for further information on running multiple Gitaly storages. +#### Configure multiple virtual storages + +You can configure multiple virtual storages to organize repositories into separate Gitaly Cluster (Praefect) clusters. +Each virtual storage operates independently with its own set of Gitaly nodes and replication settings. + +To configure multiple virtual storages: + +1. On each Praefect node, edit `/etc/gitlab/gitlab.rb` to add multiple entries in the `virtual_storage` array: + + ```ruby + praefect['configuration'] = { + # ... + virtual_storage: [ + { + name: 'storage-1', + default_replication_factor: 3, + node: [ + { + storage: 'gitaly-1', + address: 'tcp://GITALY_HOST_1:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-2', + address: 'tcp://GITALY_HOST_2:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-3', + address: 'tcp://GITALY_HOST_3:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + } + ] + }, + { + name: 'storage-2', + default_replication_factor: 2, + node: [ + { + storage: 'gitaly-4', + address: 'tcp://GITALY_HOST_4:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-5', + address: 'tcp://GITALY_HOST_5:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-6', + address: 'tcp://GITALY_HOST_6:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + } + ] + } + ] + } + ``` + +1. Save the changes and [reconfigure Praefect](../../restart_gitlab.md#reconfigure-a-linux-package-installation): + + ```shell + gitlab-ctl reconfigure + ``` + +1. On the GitLab server, edit `/etc/gitlab/gitlab.rb` to configure both virtual storages: + + ```ruby + gitlab_rails['repositories_storages'] = { + "storage-1" => { + "gitaly_address" => "tcp://PRAEFECT_1_LOADBALANCER_HOST:2305", + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + }, + "storage-2" => { + "gitaly_address" => "tcp://PRAEFECT_2_LOADBALANCER_HOST:2305", + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } + } + ``` + +1. Save the changes and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation): + + ```shell + gitlab-ctl reconfigure + ``` + +1. Verify the configuration: + + ```shell + gitlab-rake gitlab:gitaly:check + ``` + +After configuration, you can: + +- Assign storage weights to control which storage is used for new repositories. See + [Repository storage weights](../../repository_storage_paths.md#configure-where-new-repositories-are-stored). +- Move existing repositories between storages. See [Move repositories](../../operations/moving_repositories.md). + +#### Configure mixed standalone and cluster storages + +You can configure GitLab to use both standalone Gitaly instances and Gitaly Cluster (Praefect) virtual storages +simultaneously. You might do this during migration or when only some repositories require high availability. + +To configure a mixed setup: + +1. Ensure your standalone Gitaly instance is configured to listen on TCP. On the standalone Gitaly node, + edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['configuration'] = { + # ... + listen_addr: '0.0.0.0:8075' + } + ``` + +1. Configure authentication for the standalone Gitaly instance: + + ```ruby + gitaly['configuration'] = { + # ... + auth: { + token: 'GITALY_AUTH_TOKEN', + }, + } + ``` + +1. Save and [reconfigure](../../restart_gitlab.md#reconfigure-a-linux-package-installation): + + ```shell + gitlab-ctl reconfigure + ``` + +1. On the GitLab server, edit `/etc/gitlab/gitlab.rb` to configure both standalone and cluster storages: + + ```ruby + gitlab_rails['repositories_storages'] = { + 'default' => { + 'gitaly_address' => 'tcp://STANDALONE_GITALY_HOST:8075', + 'gitaly_token' => 'GITALY_AUTH_TOKEN' + }, + 'cluster' => { + 'gitaly_address' => 'tcp://PRAEFECT_LOADBALANCER_HOST:2305', + 'gitaly_token' => 'PRAEFECT_EXTERNAL_TOKEN' + } + } + ``` + +1. Save and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation): + + ```shell + gitlab-ctl reconfigure + ``` + +1. Verify both storages are accessible: + + ```shell + gitlab-rake gitlab:gitaly:check + ``` + +In this configuration: + +- The `default` storage connects directly to a standalone Gitaly node. +- The `cluster` storage connects to Gitaly Cluster (Praefect) through the load balancer. +- GitLab treats both storages equally and can store repositories on either storage. +- You can [configure storage weights](../../repository_storage_paths.md#configure-where-new-repositories-are-stored) to prefer one storage over another for new repositories. + +For more information, see [Mixed Configuration](../configure_gitaly.md#mixed-configuration). + ### Grafana Grafana is included with GitLab, and can be used to monitor your Praefect