Gitlab exporter/postgres errors after Docker upgrade to 18.1.0 re ci_pipelines and ci_job_artifacts
Summary
After upgrading self-hosted Docker from 17.11.4 to 18.1.0 the error logs for gitlab_exporter and postgres are full of broken/not found relations. I never experienced such issues before, and I am using the dockerized version since 14 or 15.
Steps to reproduce
Normal docker upgrade process
What is the current bug behavior?
After running 18.1.0, you get the following error messages in postgres log over and over: ==> /var/log/gitlab/postgresql/current <== 2025-06-23_14:53:06.78458 ERROR: relation "ci_job_artifacts" does not exist at character 135 2025-06-23_14:53:06.78459 STATEMENT: SELECT 2025-06-23_14:53:06.78459 COUNT(*) as count 2025-06-23_14:53:06.78460 FROM p_ci_builds 2025-06-23_14:53:06.78460 JOIN ci_build_trace_chunks 2025-06-23_14:53:06.78460 ON ci_build_trace_chunks.build_id = p_ci_builds.id 2025-06-23_14:53:06.78461 LEFT JOIN ci_job_artifacts 2025-06-23_14:53:06.78461 ON ci_job_artifacts.job_id = p_ci_builds.id 2025-06-23_14:53:06.78461 AND ci_job_artifacts.file_type = 3 2025-06-23_14:53:06.78462 WHERE p_ci_builds.type = 'Ci::Build' 2025-06-23_14:53:06.78462 AND p_ci_builds.status IN ('success', 'failed', 'canceled') 2025-06-23_14:53:06.78463 AND p_ci_builds.finished_at < '2025-06-22 14:53:06' 2025-06-23_14:53:06.78463 AND ci_job_artifacts.job_id IS NULL
==> /var/log/gitlab/postgresql/current <== 2025-06-23_14:53:06.78425 ERROR: relation "ci_pipelines" does not exist at character 312 2025-06-23_14:53:06.78427 STATEMENT: SELECT 2025-06-23_14:53:06.78427 p_ci_builds.runner_id, 2025-06-23_14:53:06.78428 ci_runners.runner_type, 2025-06-23_14:53:06.78428 projects.namespace_id, 2025-06-23_14:53:06.78429 ci_pipelines.pipeline_schedule_id, 2025-06-23_14:53:06.78429 p_ci_builds.trigger_request_id, 2025-06-23_14:53:06.78430 COUNT(*) AS count 2025-06-23_14:53:06.78431 FROM p_ci_builds 2025-06-23_14:53:06.78431 JOIN ci_runners 2025-06-23_14:53:06.78432 ON ci_runners.id = p_ci_builds.runner_id 2025-06-23_14:53:06.78432 JOIN projects 2025-06-23_14:53:06.78433 ON projects.id = p_ci_builds.project_id 2025-06-23_14:53:06.78433 JOIN ci_pipelines 2025-06-23_14:53:06.78434 ON ci_pipelines.id = p_ci_builds.commit_id 2025-06-23_14:53:06.78434 WHERE p_ci_builds.type = 'Ci::Build' 2025-06-23_14:53:06.78434 AND p_ci_builds.status = 'running' 2025-06-23_14:53:06.78435 -- The created_at filter has been introduced for performance reasons only 2025-06-23_14:53:06.78435 AND p_ci_builds.created_at > NOW() - INTERVAL '7 days' 2025-06-23_14:53:06.78436 AND projects.pending_delete = 'f' 2025-06-23_14:53:06.78436 GROUP BY 2025-06-23_14:53:06.78437 p_ci_builds.runner_id, 2025-06-23_14:53:06.78437 ci_runners.runner_type, 2025-06-23_14:53:06.78437 projects.namespace_id, 2025-06-23_14:53:06.78438 ci_pipelines.pipeline_schedule_id, 2025-06-23_14:53:06.78438 p_ci_builds.trigger_request_id
and in gitlab_exporter log file: ==> /var/log/gitlab/gitlab-exporter/current <== 2025-06-23_14:53:06.78461 E, [2025-06-23T14:53:06.784384 #342 (closed)] ERROR -- : Error connecting to the database: ERROR: relation "ci_pipelines" does not exist 2025-06-23_14:53:06.78464 LINE 13: JOIN ci_pipelines 2025-06-23_14:53:06.78464 ^ 2025-06-23_14:53:06.78465 2025-06-23_14:53:06.78487 E, [2025-06-23T14:53:06.784719 #342 (closed)] ERROR -- : Error connecting to the database: ERROR: relation "ci_job_artifacts" does not exist 2025-06-23_14:53:06.78488 LINE 6: LEFT JOIN ci_job_artifacts 2025-06-23_14:53:06.78489 ^ 2025-06-23_14:53:06.78489
What is the expected correct behavior?
No errors in the logfiles, for the moment these are only mitigable by disabling the exporter completely in gitlab.rb. To me it is not clear if there went something wrong at the migration level, or if the exporter is not correctly updated/has a bug. Everything else re the installation seems to work from what I could see
Relevant logs and/or screenshots
see above
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 3.2.5 Gem Version: 3.6.9 Bundler Version:2.6.9 Rake Version: 13.0.6 Redis Version: 7.2.9 Sidekiq Version:7.3.9 Go Version: unknown GitLab information Version: 18.1.0 Revision: 6229547bbed Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 16.8 URL: https://gitlab.domain.com HTTP Clone URL: https://gitlab.domain.com/some-group/some-project.git SSH Clone URL: git@gitlab.domain.com:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.42.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 18.1.0 - default Git Version: 2.49.0.gl2
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.42.0 ? ... OK (14.42.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking glinc@gitlab.domain.com yes Mailroom enabled? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes Tables are truncated? ... skipped
Possible fixes
I do not know, sadly