[Feature flag] Enable fetch errors for FetchInternalRemote

What

Enable the :gitaly_fetch_internal_remote_errors feature flag to make the FetchInternalRemote RPC return errors instead of a boolean on fetch errors. This should allow us to better diagnose fetch failures in ReplicateRepository.

Owners

• Team: Gitaly
• Most appropriate slack channel to reach out to: #g_create_gitaly
• Best individual to reach out to: @proglottis

Expectations

What release does this feature occur in first?

Introduced in !3391 (merged)

What are we expecting to happen?

This RPC should only be used internally to gitaly. Better errors when ReplicateRepository fails on fetch.

What might happen if this goes wrong?

We expect more errors from FetchInternalRemote but a higher error rate in ReplicateRepository might indicate a fault.

Set GITALY_DISABLE_FETCH_INTERNAL_REMOTE_ERRORS to 1 and restart the service to disable the feature.

What can we monitor to detect problems with this?

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

• gitlab-org/gitlab / gitlab-org/gitaly projects
• gitlab-org/gitlab-com groups
• ...

Roll Out Steps

• Read the documentation of feature flags
• Add featureflagstaging to this issue (howto)
• Is the required code deployed? (howto)
• Do we need to create a change management issue? (howto)
• Enable on staging (howto)
• Test on staging (howto)
• Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
• Announce on this issue an estimated time this will be enabled on GitLab.com
• Add featureflagproduction to this issue
• Enable on GitLab.com by running chatops command in #production (howto)
• Cross post chatops slack command to #support_gitlab-com and in your team channel
• Verify the feature flag is being used by checking Prometheus metric gitaly_feature_flag_checks_total
• Announce on the issue that the flag has been enabled
• Did you set the feature to both 100% and true (howto)
• Submit a MR to have the feature OnByDefault: true and add changelog entry (howto)
• Have that MR merged
• Possibly wait for at least one deployment cycle (howto)
• Submit an MR to remove the pre-feature code from the codebase and add changelog entry (howto)
• Have that MR merged
• Remove the feature flag via chatops (howto)
• Close this issue