[go: up one dir, main page]

Skip to content

Adds GCS bucket connector for SQL traffic replay

What does this MR do and why?

Adds GCP bucket config for SQL traffic replay. Adds the initial configuration to store SQL data for traffic replay.

How it works

I didn't find an easy way to test it locally. It was easy to just set up a new Google Cloud testing account.

  1. I configured a Google Cloud testing account. In this account, I created a new Cloud Storage bucket called database-traffic.
  2. I created a new IAM/Service Accounts to use to access the bucket. This account has two permissions: Storage Bucket Viewer (Beta) and Storage Object Admin - One to access the bucket and the other to store objects in it.
  3. I downloaded the service account key under the temp folder. The key looks like this:

tmp/gcskey.json

{
  "type": "service_account",
  "project_id": "project-id",
  "private_key_id": <private-key-id>,
  "private_key": "private-key",
  "client_email": "mac-local-testing@project-id.iam.gserviceaccount.com",
  "client_id": "client_id",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/cert-url",
  "universe_domain": "googleapis.com"
}
  1. I configured config/gitlab.yaml to:
  database_traffic_capture:
    config:
      storage:
        connector:
          provider: Google
          project_id: 'project-id'
          credentials: 'tmp/gcskey.json'
          bucket: 'database-traffic'
  1. In rails c
filename = 'test_file.json'
data = {id: 1, sql: 'SELECT 1 FROM users LIMIT 1'}.to_json

Gitlab::Database::Capture::Storage.upload(filename, data)

The result is a Google Storage Object:

=> #<Google::Cloud::Storage::File:0x0000000324b11d78
 @gapi=
  #<Google::Apis::StorageV1::Object:0x000000031cc3b008
   @bucket="database-traffic",
   @content_type="application/json",
   @crc32c="<>",
   @etag="<>",
   @generation=<>,
   @id="database-traffic/test_file.json/<>",
   @kind="storage#object",
   @md5_hash="<hash>",
   @media_link="https://storage.googleapis.com/download/storage/v1/b/database-traffic/o/test_file.json?generation=<>&alt=media",
   @metadata={"original_filename"=>"test_file.json", "encoded"=>"true"},
   @metageneration=1,
   @name="test_file.json",
   @self_link="https://www.googleapis.com/storage/v1/b/database-traffic/o/test_file.json",
   @size=44,
   @storage_class="STANDARD",
   @time_created=Fri, 05 Sep 2025 12:59:47 +0000,
   @time_finalized=Fri, 05 Sep 2025 12:59:47 +0000,
   @time_storage_class_updated=Fri, 05 Sep 2025 12:59:47 +0000,
   @updated=Fri, 05 Sep 2025 12:59:47 +0000>,
 @service=Google::Cloud::Storage::Service(<project-id>),
 @user_project=nil>
  1. The file was created in the bucket:

image

Related to #562377 (closed)

Merge request reports

Loading