Unable to upload the artifacts to S3 bucket with Object lock, retention policy settings enabled.

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Close this issue

Summary

When configuring the consolidated object storage with object lock(retention period) enabled S3 bucket, we're getting a 400 error while uploading the artifacts.

Error: Content-MD5 HTTP header is required for Put Object requests with Object Lock parameters

The artifacts are uploading successfully when the object lock, retention period are disabled on S3

CI job logs:

Uploading artifacts...
++ /usr/bin/gitlab-runner artifacts-uploader --url http://example.mylabserver.com/ --token [MASKED] --id 57 --path file.txt --artifact-format zip --artifact-type archive
Runtime platform                                    arch=amd64 os=linux pid=55998 revision=c1edb478 version=14.0.1
file.txt: found 1 matching files and directories   
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=57 responseStatus=400 Bad Request status=400 token=QSSQVvjx
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=57 responseStatus=400 Bad Request status=400 token=QSSQVvjx
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=57 responseStatus=400 Bad Request status=400 token=QSSQVvjx
FATAL: invalid argument                            
ERROR: Job failed: exit status 1

As per the docs on consolidated object storage configuration - When consolidated object storage is used, direct upload is enabled automatically. It looks like the temporary files are stored in the s3 bucket and the s3 bucket's object lock(retention policy) conflicts the removal of temp files.

Steps to reproduce

Create an S3 bucket and enable the object lock feature
Configure the GitLab instance with consolidated object storage configuration settings
Run a job with artifacts

samplejob:
   variables:
    CI_DEBUG_TRACE: "true"
   script:
    - touch file.txt
   artifacts:
    paths:
      - file.txt

What is the current bug behavior?

Unable to upload the artifacts to the S3 bucket and the job fails.

Uploading artifacts...
++ /usr/bin/gitlab-runner artifacts-uploader --url http://example.mylabserver.com/ --token [MASKED] --id 57 --path file.txt --artifact-format zip --artifact-type archive
Runtime platform                                    arch=amd64 os=linux pid=55998 revision=c1edb478 version=14.0.1
file.txt: found 1 matching files and directories   
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=57 responseStatus=400 Bad Request status=400 token=QSSQVvjx
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=57 responseStatus=400 Bad Request status=400 token=QSSQVvjx
WARNING: Retrying...                                context=artifacts-uploader error=invalid argument
WARNING: Uploading artifacts as "archive" to coordinator... failed  id=57 responseStatus=400 Bad Request status=400 token=QSSQVvjx
FATAL: invalid argument                            
ERROR: Job failed: exit status 1

What is the expected correct behavior?

Relevant logs and/or screenshots

workhorse logs:

root@:/var/log/gitlab/gitlab-workhorse# cat current | grep 01FAA1CG92PEFWEJWGDD7FJ4DZ
{"client_mode":"s3","copied_bytes":201,"correlation_id":"01FAA1CG92PEFWEJWGDD7FJ4DZ","is_local":false,"is_multipart":true,"is_remote":true,"level":"info","msg":"saved file","remote_id":"1625982124-8323-0002-1920-835e1f4f4c607bb57903bc547c4f1ce0","remote_temp_object":"tmp/uploads/1625982124-8323-0002-1920-835e1f4f4c607bb57903bc547c4f1ce0","temp_file_prefix":"artifacts.zip","time":"2021-07-11T05:42:05Z"}
{"client_mode":"local","copied_bytes":132,"correlation_id":"01FAA1CG92PEFWEJWGDD7FJ4DZ","is_local":true,"is_multipart":false,"is_remote":false,"level":"info","local_temp_path":"/tmp","msg":"saved file","remote_id":"","temp_file_prefix":"metadata.gz","time":"2021-07-11T05:42:06Z"}
{"content_type":"application/json","correlation_id":"01FAA1CG92PEFWEJWGDD7FJ4DZ","duration_ms":9426,"host":".mylabserver.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"36.144.116.162:0","remote_ip":"35.154.116.162","route":"^/api/v4/jobs/[0-9]+/artifacts\\z","status":400,"system":"http","time":"2021-07-11T05:42:13Z","ttfb_ms":9426,"uri":"/api/v4/jobs/66/artifacts?artifact_format=zip\u0026artifact_type=archive","user_agent":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; linux/amd64)","written_bytes":1492}

api_json.log:

"time":"2021-07-11T05:42:13.747Z","severity":"INFO","duration_s":7.21091,"db_duration_s":0.07827,"view_duration_s":7.13264,"status":400,"method":"POST","path":"/api/v4/jobs/66/artifacts","params":[{"key":"artifact_format","value":"zip"},{"key":"artifact_type","value":"archive"},{"key":"file.name","value":"artifacts.zip"},{"key":"file.remote_id","value":"1625982124-8323-0002-1920-835e1f4f4c607bb57903bc547c4f1ce0"},{"key":"file.size","value":"201"},{"key":"file.sha1","value":"d267717b047590716cb197eae4e619297fc8a298"},{"key":"file.sha256","value":"2d12eeb8aa926d17309a86971b3203704c3849de1be6cf81f9145cb0d7fc8ddf"},{"key":"file.path","value":""},{"key":"file.remote_url","value":"https://s3.us-east-2.amazonaws.com/tmp/uploads/1625982124-8323-0002-1920-835e1f4f4c607bb57903bc547c4f1ce0"},{"key":"file.sha512","value":"00d33e231e1c764d19648f09354895402f39e3d0383969c2ret554ywserg630ca4c11be6d06ggg8a9a1cb5e1a33a4b75d3a71160bd7c514d08871db0176a9818fad09c5a3e6488f"},{"key":"file.md5","value":"4f46c92fb6errd2519138439deae98cd8d3"},{"key":"file.gitlab-workhorse-upload","value":"eyJhbGciOiJIUzI1NiIsInR5c..."},{"key":"metadata.remote_url","value":""},{"key":"metadata.sha512","value":"8bc79f64a53a2f711c8ab7958b86bfbc5b2998432344c91b4fg8940da6112013bb2123da46487abed751206c70f8633a588127a338c5abfec56332e2a1f233f5f"},{"key":"metadata.md5","value":"d500ba61792d117b5af033669547be58"},{"key":"metadata.sha1","value":"8b8a9bb6205090b32927acebe1f7776b71c88c5d"},{"key":"metadata.gitlab-workhorse-upload","value":"eyJhbGciOiJIUz..."},{"key":"metadata.remote_id","value":""},{"key":"metadata.size","value":"132"},{"key":"metadata.name","value":"metadata.gz"},{"key":"metadata.path","value":"/tmp/metadata.gz122745314"},{"key":"metadata.sha256","value":"c53de3a64639a83ce170bawrgtgcfa9gerg06bafd43f8agtee320616514c55fda54da5c6822"},{"key":"file","value":null},{"key":"metadata","value":null}],"host":".mylabserver.com","remote_ip":"35.154.116.162, 35.154.116.162","ua":"gitlab-runner 14.0.1 (refs/pipelines/326100216; go1.13.8; linux/amd64)","route":"/api/:version/jobs/:id/artifacts","api_error":["{\"message\":\"Expected(200) \\u003c=\\u003e Actual(400 Bad Request)\\nexcon.error.response\\n  :body          =\\u003e \\\"\\u003c?xml version=\\\\\\\"1.0\\\\\\\" encoding=\\\\\\\"UTF-8\\\\\\\"?\\u003e\\\\n\\u003cError\\u003e\\u003cCode\\u003eInvalidRequest\\u003c/Code\\u003e\\u003cMessage\\u003eContent-MD5 HTTP header is required for Put Object requests with Object Lock parameters\\u003c/Message\\u003e\\u003cRequestId\\u003e6Q9XKNH08XSPT40J\\u003c/RequestId\\u003e\\u003cHostId\\u003elGt1/yM+TO0ZGYuvtin+AMfDGt1+jjkwRgf0uwA6xfk0BTHm6HUr/DD+MAdTzbhqThRBxIt5FOs=\\u003c/HostId\\u003e\\u003c/Error\\u003e\\\"\\n  :cookies       =\\u003e [\\n  ]\\n  :headers       =\\u003e {\\n    \\\"Connection\\\"       =\\u003e \\\"close\\\"\\n    \\\"Content-Type\\\"     =\\u003e \\\"application/xml\\\"\\n    \\\"Date\\\"             =\\u003e \\\"Sun, 11 Jul 2021 05:42:13 GMT\\\"\\n    \\\"Server\\\"           =\\u003e \\\"AmazonS3\\\"\\n    \\\"x-amz-id-2\\\"       =\\u003e \\\"lGt1/yM+TO0ZGYuvtin+AMfDGt1+jjkwRgf0uwA6xfkggg0BTHm6HUr/DD+MAdTzbhqTggghRBxIt5FOs=\\\"\\n    \\\"x-amz-request-id\\\" =\\u003e \\\"6Q9XKNH08XSPT40J\\\"\\n  }\\n  :host          =\\u003e \\\"bucket.s3.us-east-2.amazonaws.com\\\"\\n  :local_address =\\u003e \\\"172.31.32.78\\\"\\n  :local_port    =\\u003e 47324\\n  :path          =\\u003e \\\"/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/2021_07_11/66/219/metadata.gz\\\"\\n  :port          =\\u003e 443\\n  :reason_phrase =\\u003e \\\"Bad Request\\\"\\n  :remote_ip     =\\u003e \\\"42.239.136.34\\\"\\n  :status        =\\u003e 400\\n  :status_line   =\\u003e \\\"HTTP/1.1 400 Bad Request\\\\r\\\\n\\\"\\n\"}"],"queue_duration_s":0.022256,"redis_calls":1,"redis_duration_s":0.003066,"redis_read_bytes":2,"redis_write_bytes":112,"redis_cache_calls":1,"redis_cache_duration_s":0.003066,"redis_cache_read_bytes":2,"redis_cache_write_bytes":112,"db_count":16,"db_write_count":5,"db_cached_count":0,"external_http_count":5,"external_http_duration_s":4.172495332999915,"cpu_s":0.55809,"mem_objects":129922,"mem_bytes":50145752,"mem_mallocs":130991,"mem_total_bytes":55342632,"correlation_id":"01FAA1CG92PEFWEJWGDD7FJ4DZ","meta.user":"root","meta.project":"root/test","meta.root_namespace":"root","meta.subscription_plan":"default","meta.caller_id":"POST /api/:version/jobs/:id/artifacts","meta.remote_ip":"45.144.416.142","meta.feature_category":"continuous_integration","meta.client_id":"user/1","content_length":5491}

Output of checks

Results of GitLab environment info

System information
System:         Ubuntu 20.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.3
Redis Version:  6.0.12
Git Version:    2.31.1
Sidekiq Version:5.2.9
Go Version:     unknown

**GitLab information**: 

Version:        13.12.4-ee
Revision:       19b68f276ee
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.6
URL:            
HTTP Clone URL: 
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        13.18.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Possible fixes

Edited Aug 04, 2025 by 🤖 GitLab Bot 🤖