Migrate Code Suggestions to CloudConnector::Tokens

What does this MR do and why?

We are retiring AvailableServices for Cloud Connector. Going forward, all client code must work with Unit Primitives instead.

This means we need to stop passing service names around or use them to obtain Cloud Connector access tokens. In this MR, we replace the use of the code_suggestions service by breaking it up into the complete_code and generate_code UPs instead, both of which combine into the Code Suggestions feature.

Implementation notes

• For this to work, we need to distinguish between AI features and Unit Primitives. AI features are often aggregates of multiple smaller units (unit primitives). For example, we maintain the code_suggestions AI feature, but it consists of 2 UPs (complete and generate code). AI features are owned by AI stage groups and relates to AI configuration. Unit Primitive management is owned by groupprovision and relates to cloud entitlements/permissions.
• Not all changes were easy to feature flag. However, a new FF code_suggestions_new_tokens_path is introduced in the Tokens factory, such that until the FF is rolled out, we continue to use AvailableServices#access_token internally. This is to de-risk the rollout.
• Amazon Q and Self-hosted models (SHM) continue to require special treatment/workarounds. This MR does not mean to solve this gap.

References

Migrate Code Suggestions to CloudConnector::Tokens (#559256 - closed)

Screenshots or screen recordings

Refactor, not user-facing.

How to set up and validate locally

1. Set CLOUD_CONNECTOR_SELF_SIGN_TOKENS=1 and set up GitLab Duo

2. In a Rails console enable the following feature flag

   Feature.enable(:code_suggestions_new_tokens_path)

3. Send CS request, e.g.:

   curl -v -H'content-type: application/json' -H"Authorization: Bearer $(op read 'op://Employee/API access tokens/local')" -d"{\"current_file\":{\"file_name\":\"test.rb\",\"content_above_cursor\":\"def hello_world\",\"content_below_cursor\":\"\"},\"prompt_version\":1,\"model_name\":\"codestral-2501\",\"model_provider\":\"fireworks_ai\"}" gitlab.local:3000/api/v4/code_suggestions/completions

   It should 200 OK

4. Additionally or alternatively, go to Admin > Gitlab Duo and ensure the health-check is green

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #559256 (closed)