From d0c3b421ab17b6dcbe4a6469bd343c8c9a95fe9f Mon Sep 17 00:00:00 2001 From: John Cai Date: Tue, 21 Oct 2025 16:06:49 -0400 Subject: [PATCH 1/2] docs: Add valid configurations for Gitaly Cluster Having multiple virtual storages and a mixed configuration of virtual storages and Gitaly standalone are both supported configurations. Let's call these out explicitly in the docs. --- doc/administration/gitaly/praefect/_index.md | 95 ++++++++++ .../gitaly/praefect/configure.md | 168 ++++++++++++++++++ 2 files changed, 263 insertions(+) diff --git a/doc/administration/gitaly/praefect/_index.md b/doc/administration/gitaly/praefect/_index.md index 806fb39dc69874..855546cd3bfd22 100644 --- a/doc/administration/gitaly/praefect/_index.md +++ b/doc/administration/gitaly/praefect/_index.md @@ -128,6 +128,101 @@ variable replication factor is tracked in [this issue](https://gitlab.com/groups As with standard Gitaly storages, virtual storages can be sharded. +### Multiple virtual storages + +You can configure multiple virtual storages in a Gitaly Cluster (Praefect) deployment. This allows you to: + +- Organize repositories into separate clusters with different performance characteristics. +- Apply different replication factors to different groups of repositories. +- Scale different parts of your infrastructure independently. + +Virtual storages are configured in `gitlab_rails['repositories_storages']` on the GitLab server. Each entry in +this hash represents a distinct virtual storage. The Praefect configuration defines which Gitaly nodes serve each +virtual storage. Repositories in different virtual storages are completely independent and are not replicated +between virtual storages. + +For example, you might configure: + +- `storage-1`: A virtual storage for critical production repositories with a replication factor of 3. +- `storage-2`: A virtual storage for less critical repositories with a replication factor of 2. + +Each virtual storage requires its own set of Gitaly nodes. + +```mermaid +graph TD + GitLab[GitLab Server] + Storage1[storage-1
Praefect Cluster] + Storage2[storage-2
Praefect Cluster] + + GitLab --> Storage1 + GitLab --> Storage2 + + Storage1 --> G1[Gitaly Node 1] + Storage1 --> G2[Gitaly Node 2] + Storage1 --> G3[Gitaly Node 3] + + Storage2 --> G4[Gitaly Node 4] + Storage2 --> G5[Gitaly Node 5] + + style GitLab fill:#e8eef5,stroke:#5b6c8f,stroke-width:2px + style Storage1 fill:#fff4e6,stroke:#d9a44a,stroke-width:2px + style Storage2 fill:#fff4e6,stroke:#d9a44a,stroke-width:2px + style G1 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px + style G2 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px + style G3 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px + style G4 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px + style G5 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px +``` + +For configuration instructions, see [Configure multiple virtual storages](configure.md#configure-multiple-virtual-storages). + +### Mixed configuration + +You can configure GitLab to use a combination of: + +- Standalone Gitaly instances (direct Gitaly storage). +- Gitaly Cluster (Praefect) virtual storages. + +This mixed configuration is useful when: + +- Migrating from standalone Gitaly to Gitaly Cluster (Praefect) incrementally. +- Some repositories require high availability while others do not. +- You want to optimize costs by using Gitaly Cluster (Praefect) only for critical repositories. + +In a mixed configuration, each storage is configured independently in GitLab: + +- Standalone Gitaly storages connect directly to Gitaly nodes. +- Gitaly Cluster (Praefect) storages connect to the Praefect load balancer. + +GitLab treats all configured storages equally, regardless of whether they are standalone or clustered. When creating +a new repository, GitLab selects a storage based on the configured storage weights and available capacity. + +```mermaid +graph TD + GitLab[GitLab Server] + Praefect[Praefect Cluster] + Standalone1[Standalone Gitaly] + + GitLab -->|cluster storage| Praefect + GitLab -->|default storage| Standalone1 + + Praefect --> G1[Gitaly Node 1] + Praefect --> G2[Gitaly Node 2] + Praefect --> G3[Gitaly Node 3] + + style GitLab fill:#e8eef5,stroke:#5b6c8f,stroke-width:2px + style Praefect fill:#fff4e6,stroke:#d9a44a,stroke-width:2px + style Standalone1 fill:#fce8e8,stroke:#c67171,stroke-width:2px + style G1 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px + style G2 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px + style G3 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px +``` + +For more information, see: + +- [Mixed Configuration](../configure_gitaly.md#mixed-configuration) for configuration examples. +- [Use TCP for existing GitLab instances](configure.md#use-tcp-for-existing-gitlab-instances) for migration guidance. + ## Storage layout {{< alert type="warning" >}} diff --git a/doc/administration/gitaly/praefect/configure.md b/doc/administration/gitaly/praefect/configure.md index e21054a8712163..509aebd8180975 100644 --- a/doc/administration/gitaly/praefect/configure.md +++ b/doc/administration/gitaly/praefect/configure.md @@ -1517,6 +1517,174 @@ gitlab_rails['repositories_storages'] = { See [Mixed Configuration](../configure_gitaly.md#mixed-configuration) for further information on running multiple Gitaly storages. +#### Configure multiple virtual storages + +You can configure multiple virtual storages to organize repositories into separate Gitaly Cluster (Praefect) clusters. +Each virtual storage operates independently with its own set of Gitaly nodes and replication settings. + +To configure multiple virtual storages: + +1. On each Praefect node, edit `/etc/gitlab/gitlab.rb` to add multiple entries in the `virtual_storage` array: + + ```ruby + praefect['configuration'] = { + # ... + virtual_storage: [ + { + name: 'storage-1', + default_replication_factor: 3, + node: [ + { + storage: 'gitaly-1', + address: 'tcp://GITALY_HOST_1:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-2', + address: 'tcp://GITALY_HOST_2:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-3', + address: 'tcp://GITALY_HOST_3:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + } + ] + }, + { + name: 'storage-2', + default_replication_factor: 2, + node: [ + { + storage: 'gitaly-4', + address: 'tcp://GITALY_HOST_4:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-5', + address: 'tcp://GITALY_HOST_5:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + }, + { + storage: 'gitaly-6', + address: 'tcp://GITALY_HOST_6:8075', + token: 'PRAEFECT_INTERNAL_TOKEN' + } + ] + } + ] + } + ``` + +1. Save the changes and [reconfigure Praefect](../../restart_gitlab.md#reconfigure-a-linux-package-installation): + + ```shell + gitlab-ctl reconfigure + ``` + +1. On the GitLab server, edit `/etc/gitlab/gitlab.rb` to configure both virtual storages: + + ```ruby + gitlab_rails['repositories_storages'] = { + "storage-1" => { + "gitaly_address" => "tcp://PRAEFECT_1_LOADBALANCER_HOST:2305", + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + }, + "storage-2" => { + "gitaly_address" => "tcp://PRAEFECT_2_LOADBALANCER_HOST:2305", + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } + } + ``` + +1. Save the changes and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation): + + ```shell + gitlab-ctl reconfigure + ``` + +1. Verify the configuration: + + ```shell + gitlab-rake gitlab:gitaly:check + ``` + +After configuration, you can: + +- Assign storage weights to control which storage is used for new repositories. See + [Repository storage weights](../../repository_storage_paths.md#configure-where-new-repositories-are-stored). +- Move existing repositories between storages. See [Move repositories](../../operations/moving_repositories.md). + +#### Configure mixed standalone and cluster storages + +You can configure GitLab to use both standalone Gitaly instances and Gitaly Cluster (Praefect) virtual storages +simultaneously. This is useful during migration or when only some repositories require high availability. + +To configure a mixed setup: + +1. Ensure your standalone Gitaly instance is configured to listen on TCP. On the standalone Gitaly node, + edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['configuration'] = { + # ... + listen_addr: '0.0.0.0:8075' + } + ``` + +1. Configure authentication for the standalone Gitaly instance: + + ```ruby + gitaly['configuration'] = { + # ... + auth: { + token: 'GITALY_AUTH_TOKEN', + }, + } + ``` + +1. Save and [reconfigure](../../restart_gitlab.md#reconfigure-a-linux-package-installation): + + ```shell + gitlab-ctl reconfigure + ``` + +1. On the GitLab server, edit `/etc/gitlab/gitlab.rb` to configure both standalone and cluster storages: + + ```ruby + gitlab_rails['repositories_storages'] = { + 'default' => { + 'gitaly_address' => 'tcp://STANDALONE_GITALY_HOST:8075', + 'gitaly_token' => 'GITALY_AUTH_TOKEN' + }, + 'cluster' => { + 'gitaly_address' => 'tcp://PRAEFECT_LOADBALANCER_HOST:2305', + 'gitaly_token' => 'PRAEFECT_EXTERNAL_TOKEN' + } + } + ``` + +1. Save and [reconfigure GitLab](../../restart_gitlab.md#reconfigure-a-linux-package-installation): + + ```shell + gitlab-ctl reconfigure + ``` + +1. Verify both storages are accessible: + + ```shell + gitlab-rake gitlab:gitaly:check + ``` + +In this configuration: + +- The `default` storage connects directly to a standalone Gitaly node. +- The `cluster` storage connects to Gitaly Cluster (Praefect) through the load balancer. +- GitLab treats both storages equally and can store repositories on either storage. +- You can [configure storage weights](../../repository_storage_paths.md#configure-where-new-repositories-are-stored) to prefer one storage over another for new repositories. + +For more information, see [Mixed Configuration](../configure_gitaly.md#mixed-configuration). + ### Grafana Grafana is included with GitLab, and can be used to monitor your Praefect -- GitLab From 768ecb3f18c86f5161d6a960fc3cbd67d7705f4b Mon Sep 17 00:00:00 2001 From: John Cai Date: Wed, 22 Oct 2025 11:23:55 -0400 Subject: [PATCH 2/2] Apply 2 suggestion(s) to 1 file(s) Co-authored-by: Evan Read --- doc/administration/gitaly/praefect/_index.md | 54 +++++++++----------- 1 file changed, 23 insertions(+), 31 deletions(-) diff --git a/doc/administration/gitaly/praefect/_index.md b/doc/administration/gitaly/praefect/_index.md index 855546cd3bfd22..a143a2b0bb2cbf 100644 --- a/doc/administration/gitaly/praefect/_index.md +++ b/doc/administration/gitaly/praefect/_index.md @@ -149,29 +149,24 @@ For example, you might configure: Each virtual storage requires its own set of Gitaly nodes. ```mermaid +%%{init: { "fontFamily": "GitLab Sans" }}%% graph TD - GitLab[GitLab Server] - Storage1[storage-1
Praefect Cluster] - Storage2[storage-2
Praefect Cluster] + accTitle: Multiple virtual storages + accDescr: Example of multiple virtual storages, one with a replication factor of three and one with a replication factor of two. + + GitLab[[GitLab server]] + Storage1[(storage-1
Praefect cluster)] + Storage2[(storage-2
Praefect cluster)] GitLab --> Storage1 GitLab --> Storage2 - Storage1 --> G1[Gitaly Node 1] - Storage1 --> G2[Gitaly Node 2] - Storage1 --> G3[Gitaly Node 3] - - Storage2 --> G4[Gitaly Node 4] - Storage2 --> G5[Gitaly Node 5] - - style GitLab fill:#e8eef5,stroke:#5b6c8f,stroke-width:2px - style Storage1 fill:#fff4e6,stroke:#d9a44a,stroke-width:2px - style Storage2 fill:#fff4e6,stroke:#d9a44a,stroke-width:2px - style G1 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px - style G2 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px - style G3 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px - style G4 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px - style G5 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px + Storage1 --> G1[Gitaly node 1] + Storage1 --> G2[Gitaly node 2] + Storage1 --> G3[Gitaly node 3] + + Storage2 --> G4[Gitaly node 4] + Storage2 --> G5[Gitaly node 5] ``` For configuration instructions, see [Configure multiple virtual storages](configure.md#configure-multiple-virtual-storages). @@ -198,24 +193,21 @@ GitLab treats all configured storages equally, regardless of whether they are st a new repository, GitLab selects a storage based on the configured storage weights and available capacity. ```mermaid +%%{init: { "fontFamily": "GitLab Sans" }}%% graph TD - GitLab[GitLab Server] - Praefect[Praefect Cluster] - Standalone1[Standalone Gitaly] + accTitle: Mixed configuration + accDescr: Example result of mixed configuration, with a Gitaly Cluster (Praefect) and standalone Gitaly configured together. + + GitLab[[GitLab server]] + Praefect[(Praefect cluster)] + Standalone1[(Standalone gitaly)] GitLab -->|cluster storage| Praefect GitLab -->|default storage| Standalone1 - Praefect --> G1[Gitaly Node 1] - Praefect --> G2[Gitaly Node 2] - Praefect --> G3[Gitaly Node 3] - - style GitLab fill:#e8eef5,stroke:#5b6c8f,stroke-width:2px - style Praefect fill:#fff4e6,stroke:#d9a44a,stroke-width:2px - style Standalone1 fill:#fce8e8,stroke:#c67171,stroke-width:2px - style G1 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px - style G2 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px - style G3 fill:#e6f3ec,stroke:#6ba37d,stroke-width:2px + Praefect --> G1[Gitaly node 1] + Praefect --> G2[Gitaly node 2] + Praefect --> G3[Gitaly node 3] ``` For more information, see: -- GitLab