[go: up one dir, main page]
Skip to content

gitlab-exporter(15.6.0) /metrics endpoint returns empty reply while health probes pass in chart 9.2.1 (App Version: 18.2.1)

Summary

After upgrading the GitLab Helm chart from version 8.11.7(17.11.6) to 9.2.1(18.2.1), Prometheus scrapes for the gitlab-exporter component are failing.

The core issue is that requests to the /metrics endpoint consistently result in an "Empty reply from server". This behavior occurs despite the pod's liveness and readiness probes passing successfully. No errors are visible in the container logs during the failed requests to /metrics. When debugging with kubectl port-forward, the connection to the pod breaks after a single request consistently. No errors are visible in the container logs during these failures.

Steps to reproduce

  • Deploy a GitLab instance using chart version 8.11.7 and confirm gitlab-exporter is scraped successfully.

  • Upgrade the Helm release to chart version 9.2.1 .

  • Observe that the gitlab-exporter target in Prometheus now has a status of DOWN.

  • Use kubectl port-forward to connect to the gitlab-exporter pod.

  • Attempt to access the metrics endpoint using curl http://localhost:9168/metrics. The command will fail with an "Empty reply from server" error. Also the port-forward session will terminate

curl -vvv http://localhost:9168/metrics
* Host localhost:9168 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9168...
* Connected to localhost (::1) port 9168
> GET / HTTP/1.1
> Host: localhost:9168
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

Configuration used

gitlab-exporter:
      metrics:
        enabled: true
        port: 9168
        path: /metrics
        serviceMonitor:
          enabled: true
          additionalLabels:
            prometheus: abc
            owner: abc

Current behavior

  • Prometheus scrapes for gitlab-exporter fail resulting in an up metric value of 0.

  • A direct curl request to the /metrics endpoint receives an empty reply.

  • The pod's liveness and readiness probes consistently pass.

  • No corresponding error messages are generated in the container logs when the /metrics endpoint is accessed.

Expected behavior

  • The /metrics endpoint should return a 200 OK status with a valid Prometheus metrics payload.

  • The gitlab-exporter target should have a status of UP in Prometheus.

Versions

  • Chart: 9.2.1
  • Platform:
    • Cloud: GKE
    • Self-hosted: (OpenShift | Minikube | Rancher RKE | ?)
  • Kubernetes: (kubectl version)
    • Client: v1.31.1
    • Server: v1.32.4-gke.1767000
  • Helm: (helm version)
    • Client:v3.15.1
    • Server:v3.15.1

Relevant logs

gitlab-test1-gitlab-exporter-7b896b6d76-smq2d configure '/init-config/redis/./redis-password' -> '/init-secrets/redis/./redis-password'
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d configure '/init-config/postgres/./psql-password-ci' -> '/init-secrets/postgres/./psql-password-ci'
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d configure '/init-config/postgres/./psql-password-main' -> '/init-secrets/postgres/./psql-password-main'
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d certificates Begin parsing .tpl templates from /etc
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d certificates Updating certificates in /etc/ssl/certs...
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d certificates 142 added, 0 removed; done.
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d certificates Running hooks in /etc/ca-certificates/update.d...
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d certificates done.
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d gitlab-exporter Begin parsing .erb templates from /var/opt/gitlab-exporter/templates
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d gitlab-exporter Writing /etc/gitlab-exporter/gitlab-exporter.yml
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d gitlab-exporter Begin parsing .tpl templates from /var/opt/gitlab-exporter/templates
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d gitlab-exporter Copying other config files found in /var/opt/gitlab-exporter/templates to /etc/gitlab-exporter
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d gitlab-exporter [2025-08-04 07:12:31] INFO  WEBrick 1.9.1
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d gitlab-exporter [2025-08-04 07:12:31] INFO  ruby 3.3.9 (2025-07-24) [x86_64-linux]
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d gitlab-exporter == Sinatra (v2.2.3) has taken the stage on 9168 for development with backup from WEBrick
gitlab-test1-gitlab-exporter-7b896b6d76-smq2d gitlab-exporter [2025-08-04 07:12:31] INFO  WEBrick::HTTPServer#start: pid=11 port=9168
Stream closed EOF for gitlab-test1/gitlab-test1-gitlab-exporter-7b896b6d76-smq2d (configure)
Stream closed EOF for gitlab-test1/gitlab-test1-gitlab-exporter-7b896b6d76-smq2d (certificates)