Adds GCS bucket connector for SQL traffic replay
What does this MR do and why?
Adds GCP bucket config for SQL traffic replay. Adds the initial configuration to store SQL data for traffic replay.
How it works
I didn't find an easy way to test it locally. It was easy to just set up a new Google Cloud testing account.
- I configured a Google Cloud testing account. In this account, I created a new Cloud Storage bucket called
database-traffic
. - I created a new IAM/Service Accounts to use to access the bucket. This account has two permissions:
Storage Bucket Viewer (Beta)
andStorage Object Admin
- One to access the bucket and the other to store objects in it. - I downloaded the service account key under the temp folder. The key looks like this:
tmp/gcskey.json
{
"type": "service_account",
"project_id": "project-id",
"private_key_id": <private-key-id>,
"private_key": "private-key",
"client_email": "mac-local-testing@project-id.iam.gserviceaccount.com",
"client_id": "client_id",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/cert-url",
"universe_domain": "googleapis.com"
}
- I configured
config/gitlab.yaml
to:
database_traffic_capture:
config:
storage:
connector:
provider: Google
project_id: 'project-id'
credentials: 'tmp/gcskey.json'
bucket: 'database-traffic'
- In
rails c
filename = 'test_file.json'
data = {id: 1, sql: 'SELECT 1 FROM users LIMIT 1'}.to_json
Gitlab::Database::Capture::Storage.upload(filename, data)
The result is a Google Storage Object:
=> #<Google::Cloud::Storage::File:0x0000000324b11d78
@gapi=
#<Google::Apis::StorageV1::Object:0x000000031cc3b008
@bucket="database-traffic",
@content_type="application/json",
@crc32c="<>",
@etag="<>",
@generation=<>,
@id="database-traffic/test_file.json/<>",
@kind="storage#object",
@md5_hash="<hash>",
@media_link="https://storage.googleapis.com/download/storage/v1/b/database-traffic/o/test_file.json?generation=<>&alt=media",
@metadata={"original_filename"=>"test_file.json", "encoded"=>"true"},
@metageneration=1,
@name="test_file.json",
@self_link="https://www.googleapis.com/storage/v1/b/database-traffic/o/test_file.json",
@size=44,
@storage_class="STANDARD",
@time_created=Fri, 05 Sep 2025 12:59:47 +0000,
@time_finalized=Fri, 05 Sep 2025 12:59:47 +0000,
@time_storage_class_updated=Fri, 05 Sep 2025 12:59:47 +0000,
@updated=Fri, 05 Sep 2025 12:59:47 +0000>,
@service=Google::Cloud::Storage::Service(<project-id>),
@user_project=nil>
- The file was created in the bucket:
Related to #562377 (closed)
Edited by Leonardo da Rosa - OOO until 2025-10-31