Cannot install Helm Chart, Problem accessing main database (gitlab). Confirm username, password, and permissions.
Summary
When I try to install GitLab with Helm (fresh install for me), the migration/webservice/sidekiq/gitlab-runner pods are crashing. In many of these, I can see logs that let thinking of psql connection issue but I tried multiple password, different declarations in values.yml and nothing work. I tried to connect with a custom pod that run postgresql:17 and manual connection is working.
What can I do to fix that?
Thanks in advance!
Steps to reproduce
kubectl apply -f gitlab/namespace.yml
kubectl apply -f gitlab/database.yml -f gitlab/s3.yml -f gitlab/secret.yml
helm repo add gitlab https://charts.gitlab.io/
helm repo update
helm upgrade --install -n gitlab --create-namespace gitlab gitlab/gitlab -f gitlab/values.ymlConfiguration used
namespace.yml
apiVersion: v1
kind: Namespace
metadata:
name: gitlabdatabase.yml (I'm using CNPG database)
apiVersion: postgresql.cnpg.io/v1
kind: Database
metadata:
name: database-gitlab
namespace: cnpg-database
spec:
name: gitlab
owner: gitlab
cluster:
name: database-cluster
extensions:
- name: btree_gist
- name: pg_trgm
- name: plpgsql
- name: amcheck
ensure: presents3.yml: (I'm using rook-ceph RGW)
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
name: gitlab
namespace: rook-ceph
spec:
store: ceph-objectstore
clusterNamespace: rook-ceph
keys:
- accessKeyRef:
name: gitlab-bucket-credentials
key: s3-access-key
secretKeyRef:
name: gitlab-bucket-credentials
key: s3-secret-key
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: git-lfs
namespace: rook-ceph
spec:
bucketName: git-lfs
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-artifacts
namespace: rook-ceph
spec:
bucketName: gitlab-artifacts
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-uploads
namespace: rook-ceph
spec:
bucketName: gitlab-uploads
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-packages
namespace: rook-ceph
spec:
bucketName: gitlab-packages
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-mr-diffs
namespace: rook-ceph
spec:
bucketName: gitlab-mr-diffs
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-terraform-state
namespace: rook-ceph
spec:
bucketName: gitlab-terraform-state
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-ci-secure-files
namespace: rook-ceph
spec:
bucketName: gitlab-ci-secure-files
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-dependency-proxy
namespace: rook-ceph
spec:
bucketName: gitlab-dependency-proxy
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-backups
namespace: rook-ceph
spec:
bucketName: gitlab-backups
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"
---
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: gitlab-tmp
namespace: rook-ceph
spec:
bucketName: gitlab-tmp
storageClassName: ceph-bucket
additionalConfig:
bucketOwner: "gitlab"secret.yml: (these passwords are generated only for this issue but followed the same rules for characters)
apiVersion: v1
kind: Secret
metadata:
name: gitlab-bucket-credentials
namespace: rook-ceph
type: Opaque
stringData:
s3-access-key: "E1QJQ%6aBRylSoSnwn3xalODBF3U%u3KwTq6LW8a4!15N$7ZpM2fgaQhgrQhFhYL"
s3-secret-key: "H6gmGjGWchRs$nACY2tDau5nHLuam1kGPv&v8SA%vps4swEgNF1WLG9miZ3M6u3h"
---
apiVersion: v1
kind: Secret
metadata:
name: gitlab-credentials
namespace: gitlab
type: Opaque
stringData:
postgresql-password: ewp6uHqI5KitSqia5DAMptC8Faam9N8NIzW8DTmVAhefa335gj2qYi52VKj55i4U
redis-password: "MM3l9efRW2urXS^sl&gg5G2!3d&#EvmY0X9^NS*GyrB^pu0r#rxFSGq9uF%zHcP#"
gitaly-authtoken: "pGfrAp3WH9bM6umyCSozEU3SrgK0AapUPESGYr1hXy0i0xMe6T8s3TY1UjoMr4eB"
s3-access-key: "E1QJQ%6aBRylSoSnwn3xalODBF3U%u3KwTq6LW8a4!15N$7ZpM2fgaQhgrQhFhYL"
s3-secret-key: "H6gmGjGWchRs$nACY2tDau5nHLuam1kGPv&v8SA%vps4swEgNF1WLG9miZ3M6u3h"
s3-connection-block: |
provider: AWS
region: eu-west-1
aws_access_key_id: E1QJQ%6aBRylSoSnwn3xalODBF3U%u3KwTq6LW8a4!15N$7ZpM2fgaQhgrQhFhYL
aws_secret_access_key: H6gmGjGWchRs$nACY2tDau5nHLuam1kGPv&v8SA%vps4swEgNF1WLG9miZ3M6u3h
endpoint: "http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc.cluster.local"
oidc-block: |
name: "openid_connect"
label: "Authentik"
args:
name: "openid_connect"
scope: ["openid", "profile", "email"]
response_type: "code"
issuer: "https://auth.my.domain/application/o/gitlab/"
client_auth_method: "query"
discovery: true
uid_field: "preferred_username"
send_scope_to_token_endpoint: "true"
pkce: true
client_options:
identifier: "pm1hb1ALlkCZKgRVXmsSDKo5cNcb1WiUXI7rpl0T"
secret: "Z5JAdi5NxTCMhQXaPBYKILXP1Tp36w5azFGOLgPpPQ5kz7fMkev5wsZs3ZPgM3Ilf4YqEcAnzpiTCgU4tLq3TtJBKAlopYpWPmvx9vBL7awufMiFI8lYz3u2ENekcI13"
redirect_uri: "https://git.my.domain/users/auth/openid_connect/callback"
shell_auth_token: "vALol0Vn2l73YZuzxUIn4dGUGbVdbx4UsEvgseAidAqK5aHZ61MO9OKFlarCpOvA"
ssh_host_rsa_key: |
-----BEGIN OPENSSH PRIVATE KEY-----
<self generated private key>
-----END OPENSSH PRIVATE KEY-----
ssh_host_rsa_key.pub: |
ssh-rsa <rsapublickey> git.my.domain
ssh_host_ed25519_key: |
-----BEGIN OPENSSH PRIVATE KEY-----
<self generated private key>
-----END OPENSSH PRIVATE KEY-----
ssh_host_ed25519_key.pub: |
ssh-ed25519 <ed25519publickey> git.my.domain
ssh_host_ecdsa_key: |
-----BEGIN OPENSSH PRIVATE KEY-----
<self generated private key>
-----END OPENSSH PRIVATE KEY-----
ssh_host_ecdsa_key.pub: |
ecdsa-sha2-nistp521 <ecdsapublickey> git.my.domain
registry-auth.key: |
-----BEGIN PRIVATE KEY-----
<self generated private key>
-----END PRIVATE KEY-----
registry-auth.crt: |
-----BEGIN CERTIFICATE-----
<self generated certificate>
-----END CERTIFICATE-----
registry_http_secret: "XRDj2u9nNsEBO7X8sqXrwC0eio1jWx0yJQPbHxw8fWhqMHSNep8sUFw2Udy2GITq"
registry_notification_secret: "[WArPn6qS80HoG4ZhIlwq6xabVozNfRl2]"
smtp_password: "73utiNzyrga4#mHlrpyl#Irumh9eXHKQ8q&5M$BAOyhotW95kZ&CroohRTsi9l*w"values.yml: (little bit obfuscated for privacy)
global:
edition: ee
gitlabVersion: "18.4.1"
## https://docs.gitlab.com/charts/charts/globals#configure-host-settings
hosts:
domain: <mydomain>
https: true
gitlab:
name: git.<mydomain>
minio: {}
registry:
name: registry.<mydomain>
tls: {}
smartcard: {}
kas: {}
pages:
name: pages.<mydomain>
openbao: {}
workspaces:
name: workspaces.<mydomain>
## https://docs.gitlab.com/charts/charts/globals#configure-ingress-settings
ingress:
configureCertmanager: false
useNewIngressForCerts: false
provider: traefik
class: traefik
annotations:
cert-manager.io/cluster-issuer: letsencrypt-production
traefik.ingress.kubernetes.io/router.tls: "true"
traefik.ingress.kubernetes.io/router.entrypoints: websecure
enabled: true
tls:
enabled: true
secretName: gitlab-my-domain-tls
path: /
pathType: Prefix
monitoring:
enabled: true
gitlab:
## Enterprise license for this GitLab installation
## Secret created according to https://docs.gitlab.com/charts/installation/secrets#initial-enterprise-license
## If allowing shared-secrets generation, this is OPTIONAL.
license: {}
# secret: RELEASE-gitlab-license
# key: license
## https://docs.gitlab.com/charts/charts/globals#configure-postgresql-settings
psql:
password:
useSecret: true
secret: gitlab-credentials
key: postgresql-password
host: database-cluster-rw.cnpg-database.svc.cluster.local
port: 5432
username: gitlab
database: gitlab
preparedStatements: false
databaseTasks: true
load_balancing:
hosts:
- database-cluster-ro.cnpg-database.svc.cluster.local
## https://docs.gitlab.com/charts/charts/globals#configure-redis-settings
redis:
auth:
enabled: true
secret: gitlab-credentials
key: redis-password
host: redis-master.redis.svc.cluster.local
port: 6379
database: 1
## https://docs.gitlab.com/charts/charts/globals#configure-gitaly-settings
gitaly:
enabled: true
authToken:
secret: gitlab-credentials
key: gitaly-authtoken
internal:
names: [default]
external: []
service:
name: gitaly
type: ClusterIP
externalPort: 8075
internalPort: 8075
tls:
externalPort: 8076
internalPort: 8076
tls:
enabled: false
praefect:
enabled: false
## https://docs.gitlab.com/charts/charts/globals#configure-minio-settings
minio:
enabled: false
## https://docs.gitlab.com/charts/charts/globals#configure-appconfig-settings
## Rails based portions of this chart share many settings
appConfig:
## https://docs.gitlab.com/charts/charts/globals#general-application-settings
enableUsagePing: true
enableSeatLink: false
enableImpersonation: true
applicationSettingsCacheSeconds: 60
usernameChangingEnabled: false
defaultTheme: 9
defaultColorMode: 3
defaultSyntaxHighlightingTheme: 1
defaultProjectsFeatures:
issues: true
mergeRequests: true
wiki: true
snippets: true
builds: true
containerRegistry: true
## https://docs.gitlab.com/charts/charts/globals#content-security-policy
contentSecurityPolicy:
enabled: true
report_only: false
## https://docs.gitlab.com/charts/charts/globals#lfs-artifacts-uploads-packages-external-mr-diffs-and-dependency-proxy
object_store:
enabled: true
proxy_download: true
storage_options:
server_side_encryption: AES256
connection:
secret: gitlab-credentials
key: s3-connection-block
lfs:
enabled: true
proxy_download: true
bucket: git-lfs
artifacts:
enabled: true
proxy_download: true
bucket: gitlab-artifacts
uploads:
enabled: true
proxy_download: true
bucket: gitlab-uploads
packages:
enabled: true
proxy_download: true
bucket: gitlab-packages
externalDiffs:
enabled: true
when:
proxy_download: true
bucket: gitlab-mr-diffs
terraformState:
enabled: true
bucket: gitlab-terraform-state
ciSecureFiles:
enabled: true
bucket: gitlab-ci-secure-files
dependencyProxy:
enabled: true
proxy_download: true
bucket: gitlab-dependency-proxy
backups:
bucket: gitlab-backups
tmpBucket: gitlab-tmp
## https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration
## https://docs.gitlab.com/charts/charts/gitlab/mailroom/index.html#incoming-email
incomingEmail:
enabled: false
## https://docs.gitlab.com/charts/charts/gitlab/mailroom/index.html#service-desk-email
serviceDeskEmail:
enabled: false
## https://docs.gitlab.com/charts/charts/globals#ldap
ldap:
preventSignin: false
servers: {}
duoAuth:
enabled: false
## https://docs.gitlab.com/charts/charts/globals#kas-settings
gitlab_kas: {}
workspaces: {}
## Configure GitLab Cells. Cells is a GitLab internal-use experiment.
## https://docs.gitlab.com/administration/cells/
cell:
enabled: false
## https://docs.gitlab.com/charts/charts/globals#suggested-reviewers-settings
suggested_reviewers: {}
## https://docs.gitlab.com/charts/charts/globals#omniauth
omniauth:
enabled: true
autoSignInWithProvider: openid_connect
syncProfileFromProvider: [openid_connect]
syncProfileAttributes: [email]
allowSingleSignOn: [openid_connect]
blockAutoCreatedUsers: false
autoLinkLdapUser: false
autoLinkSamlUser: false
autoLinkUser: [openid_connect]
externalProviders: []
allowBypassTwoFactor: []
providers:
- secret: gitlab-credentials
key: oidc-block
## https://docs.gitlab.com/charts/charts/globals#kerberos
kerberos:
enabled: false
## https://docs.gitlab.com/charts/charts/globals#configure-appconfig-settings
sentry:
enabled: false
gitlab_docs:
enabled: false
oidcProvider:
openidIdTokenExpireInSeconds: 120
smartcard:
enabled: false
sidekiq:
routingRules: []
# Config that only applies to the defaults on initial install
initialDefaults: {}
## End of global.appConfig
oauth:
gitlab-pages: {}
## https://docs.gitlab.com/charts/advanced/geo/
geo:
enabled: false
## https://docs.gitlab.com/charts/charts/gitlab/kas/
kas:
enabled: true
service:
apiExternalPort: 8153 # port for connections from the GitLab backend
tls:
enabled: false
verify: true
workspaces:
## If workspaces is enabled, you must provide a hostname under global.hosts
enabled: true
## https://docs.gitlab.com/charts/charts/gitlab/spamcheck/
spamcheck:
enabled: true
## https://docs.gitlab.com/charts/charts/globals#configure-gitlab-shell
shell:
authToken:
secret: gitlab-credentials
key: shell_auth_token
hostKeys:
secret: gitlab-credentials
## https://docs.gitlab.com/charts/charts/globals#tcp-proxy-protocol
tcp:
proxyProtocol: false
traefik:
entrypoint: "gitlab-shell"
## Rails application secrets
## Secret created according to https://docs.gitlab.com/charts/installation/secrets#gitlab-rails-secret
## If allowing shared-secrets generation, this is OPTIONAL.
railsSecrets: {}
## Rails generic setting, applicable to all Rails-based containers
rails:
bootsnap: # Enable / disable Shopify/Bootsnap cache
enabled: true
sessionStore:
sessionCookieTokenPrefix: ""
## https://docs.gitlab.com/charts/charts/globals#configure-registry-settings
registry:
bucket: registry
certificate:
secret: gitlab-credentials
key: registry-auth.crt
httpSecret:
secret: gitlab-credentials
key: registry_http_secret
notificationSecret:
secret: gitlab-credentials
key: registry_notification_secret
tls:
enabled: true
secretName: gitlab-registry-my-domain-tls
# Settings utilized by other services referencing registry:
enabled: true
pages:
enabled: true
accessControl: true
customDomainMode: https
artifactsServer: true
objectStore:
enabled: true
bucket: gitlab-pages
namespaceInPath: false
## GitLab Runner
## Secret created according to https://docs.gitlab.com/charts/installation/secrets#gitlab-runner-secret
## If allowing shared-secrets generation, this is OPTIONAL.
runner:
registrationToken: {}
## https://docs.gitlab.com/charts/charts/globals#outgoing-email
## Outgoing email server settings
smtp:
enabled: true
address: mail.<my-provider>
port: 587
user_name: "no-reply@my-domain"
## https://docs.gitlab.com/charts/installation/secrets#smtp-password
password:
secret: "gitlab-credentials"
key: smtp_password
authentication: "login"
starttls_auto: true
openssl_verify_mode: "peer"
open_timeout: 30
read_timeout: 60
pool: false
## https://docs.gitlab.com/charts/charts/globals#outgoing-email
## Email persona used in email sent by GitLab
email:
from: "GitLab <no-reply@my-domain>"
display_name: GitLab
reply_to: "no-reply@my-domain"
subject_suffix: ""
smime:
enabled: false
secretName: ""
keyName: "tls.key"
certName: "tls.crt"
## Timezone for containers.
time_zone: Europe/Paris
## https://docs.gitlab.com/charts/charts/globals#configure-workhorse-settings
## Global settings related to Workhorse
workhorse:
serviceName: webservice-default
tls:
enabled: false
## https://docs.gitlab.com/charts/charts/globals#configure-webservice
webservice:
workerTimeout: 60
## End of global
upgradeCheck:
enabled: true
image: {}
securityContext:
# in alpine/debian/busybox based images, this is `nobody:nogroup`
runAsUser: 65534
fsGroup: 65534
seccompProfile:
type: "RuntimeDefault"
## Allow to overwrite the specific security context under which the container is running.
containerSecurityContext:
runAsUser: 65534
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop: [ "ALL" ]
tolerations: []
annotations: {}
configMapAnnotations: {}
resources:
requests:
cpu: 50m
priorityClassName: ""
## Installation of certmananger.
## This value replaces `certmanager.install` to allow certmanager schema validation to pass.
## See dependencies in Chart.yaml for current version
installCertmanager: false
## Configuration of jetstack/cert-manager
certmanager:
installCRDs: false
nameOverride: certmanager
## https://docs.gitlab.com/charts/charts/nginx/
## https://docs.gitlab.com/charts/architecture/decisions#nginx-ingress
## Installation & configuration of charts/ingress-nginx:
nginx-ingress: &nginx-ingress
enabled: false
# Ingress controller to handle requests forwarded from other Geo sites.
# Configuration differences compared to the main nginx ingress:
# - Pass X-Forwarded-For headers as is
# - Use a different IngressClass name
nginx-ingress-geo:
<<: *nginx-ingress
enabled: false
haproxy:
install: false
## Installation & configuration of stable/prometheus
## See dependencies in Chart.yaml for current version
prometheus:
install: false
## Configuration of Redis
## https://docs.gitlab.com/charts/architecture/decisions#redis
## https://docs.gitlab.com/charts/installation/deployment.html#redis
redis:
install: false
## Installation & configuration of stable/postgresql
## See dependencies in Chart.yaml for current version
postgresql:
install: false
## Installation & configuration charts/registry
## https://docs.gitlab.com/charts/architecture/decisions#registry
## https://docs.gitlab.com/charts/charts/registry/
# registry:
# enabled: false
## Automatic shared secret generation
## https://docs.gitlab.com/charts/installation/secrets
## https://docs.gitlab.com/charts/charts/shared-secrets.html
shared-secrets:
enabled: true
rbac:
create: true
selfsign:
image:
# pullPolicy: IfNotPresent
# pullSecrets: []
repository: registry.gitlab.com/gitlab-org/build/cng/cfssl-self-sign
# Default tag is `master`, overridable by `global.gitlabVersion`.
# tag: master
keyAlgorithm: "rsa"
keySize: "4096"
expiry: "3650d"
caSubject: "GitLab Helm Chart"
env: production
serviceAccount:
enabled: true
create: true
name: # Specify a pre-existing ServiceAccount name
resources:
requests:
cpu: 50m
securityContext:
# in debian/alpine based images, this is `nobody:nogroup`
runAsUser: 65534
fsGroup: 65534
seccompProfile:
type: "RuntimeDefault"
containerSecurityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop: [ "ALL" ]
tolerations: []
podLabels: {}
annotations: {}
## Installation & configuration of gitlab/gitlab-runner
## See dependencies in Chart.yaml for current version
gitlab-runner:
install: true
rbac:
create: true
runners:
locked: false
# Set secret to an arbitrary value because the runner chart renders the gitlab-runner.secret template only if it is not empty.
# The parent/GitLab chart overrides the template to render the actual secret name.
secret: "nonempty"
config: |
[[runners]]
[runners.kubernetes]
image = "ubuntu:22.04"
{{- if .Values.global.minio.enabled }}
[runners.cache]
Type = "s3"
Path = "gitlab-runner"
Shared = true
[runners.cache.s3]
ServerAddress = {{ include "gitlab-runner.cache-tpl.s3ServerAddress" . }}
BucketName = "runner-cache"
BucketLocation = "us-east-1"
Insecure = false
{{ end }}
podAnnotations:
gitlab.com/prometheus_scrape: "true"
gitlab.com/prometheus_port: 9252
podSecurityContext:
seccompProfile:
type: "RuntimeDefault"
traefik:
install: false
## Settings for individual sub-charts under GitLab
## Note: Many of these settings are configurable via globals
gitlab:
## https://docs.gitlab.com/charts/charts/gitlab/toolbox
toolbox:
replicas: 1
antiAffinityLabels:
matchLabels:
app: gitaly
backups:
objectStorage:
config:
secret: gitlab-credentials
key: s3-connection-block
## https://docs.gitlab.com/charts/charts/gitlab/migrations
# migrations:
# enabled: false
## https://docs.gitlab.com/charts/charts/gitlab/webservice
webservice:
ingress:
tls:
secretName: gitlab-webservice-my-domain-tls
# enabled: false
## https://docs.gitlab.com/charts/charts/gitlab/sidekiq
# sidekiq:
# enabled: false
## https://docs.gitlab.com/charts/charts/gitlab/gitaly
# gitaly:
## https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell
# gitlab-shell:
# enabled: false
## https://docs.gitlab.com/charts/charts/gitlab/gitlab-pages
# gitlab-pages:
## https://docs.gitlab.com/charts/charts/gitlab/kas
kas:
ingress:
tls:
secretName: gitlab-kas-my-domain-tls
## https://docs.gitlab.com/charts/charts/gitlab/praefect
# praefect:
## Installation & configuration of gitlab/gitlab-zoekt
gitlab-zoekt:
install: false
## Installation & configuration of OpenBao
openbao:
install: falseCurrent behavior
gitlab gitlab-gitaly-0 1/1 Running 0 22m
gitlab gitlab-gitlab-exporter-66f5bbc67-pchj5 1/1 Running 0 22m
gitlab gitlab-gitlab-pages-c4ff6dc68-97flk 1/1 Running 0 22m
gitlab gitlab-gitlab-runner-6b4c4dbffc-4gx8b 0/1 CrashLoopBackOff 6 (83s ago) 22m
gitlab gitlab-gitlab-shell-6745cbbd6b-ql7kc 1/1 Running 0 22m
gitlab gitlab-gitlab-shell-6745cbbd6b-qq9qb 1/1 Running 0 22m
gitlab gitlab-kas-6556dd55b8-dl5tw 0/1 Running 0 22m
gitlab gitlab-kas-6556dd55b8-nw5wl 0/1 Running 0 22m
gitlab gitlab-registry-65d957985f-n8k5c 1/1 Running 0 22m
gitlab gitlab-registry-65d957985f-rp9nl 1/1 Running 0 22m
gitlab gitlab-sidekiq-all-in-1-v2-5db7c9cb48-k64bm 0/1 Init:CrashLoopBackOff 6 (58s ago) 22m
gitlab gitlab-spamcheck-86f4f7769b-wndw6 1/1 Running 0 22m
gitlab gitlab-toolbox-5fc5699c4-p5wj7 1/1 Running 0 22m
gitlab gitlab-webservice-default-c687c7674-dhx27 0/2 Init:CrashLoopBackOff 6 (102s ago) 22m
gitlab gitlab-webservice-default-c687c7674-xp7jz 0/2 Init:Error 6 (5m1s ago) 22mExpected behavior
gitlab gitlab-gitaly-0 1/1 Running 0 22m
gitlab gitlab-gitlab-exporter-66f5bbc67-pchj5 1/1 Running 0 22m
gitlab gitlab-gitlab-pages-c4ff6dc68-97flk 1/1 Running 0 22m
gitlab gitlab-gitlab-runner-6b4c4dbffc-4gx8b 1/1 Running 6 (83s ago) 22m
gitlab gitlab-gitlab-shell-6745cbbd6b-ql7kc 1/1 Running 0 22m
gitlab gitlab-gitlab-shell-6745cbbd6b-qq9qb 1/1 Running 0 22m
gitlab gitlab-kas-6556dd55b8-dl5tw 1/1 Running 0 22m
gitlab gitlab-kas-6556dd55b8-nw5wl 1/1 Running 0 22m
gitlab gitlab-registry-65d957985f-n8k5c 1/1 Running 0 22m
gitlab gitlab-registry-65d957985f-rp9nl 1/1 Running 0 22m
gitlab gitlab-sidekiq-all-in-1-v2-5db7c9cb48-k64bm 1/1 Running 6 (58s ago) 22m
gitlab gitlab-spamcheck-86f4f7769b-wndw6 1/1 Running 0 22m
gitlab gitlab-toolbox-5fc5699c4-p5wj7 1/1 Running 0 22m
gitlab gitlab-webservice-default-c687c7674-dhx27 2/2 Running 6 (102s ago) 22m
gitlab gitlab-webservice-default-c687c7674-xp7jz 2/2 Running 6 (5m1s ago) 22mVersions
-
Chart:
$ helm show chart gitlab/gitlab apiVersion: v2 appVersion: v18.4.2 dependencies: - name: gitlab repository: "" version: '*.*.*' - name: certmanager-issuer repository: "" version: '*.*.*' - name: minio repository: "" version: '*.*.*' - name: registry repository: "" version: '*.*.*' - alias: certmanager condition: installCertmanager name: cert-manager repository: https://charts.jetstack.io/ version: v1.17.4 - condition: prometheus.install name: prometheus repository: https://prometheus-community.github.io/helm-charts version: 27.23.0 - condition: postgresql.install name: postgresql repository: https://charts.bitnami.com/bitnami version: 13.4.4 - condition: gitlab-runner.install name: gitlab-runner repository: https://charts.gitlab.io/ version: 0.80.1 - condition: redis.install name: redis repository: https://charts.bitnami.com/bitnami version: 18.19.4 - condition: nginx-ingress.enabled name: nginx-ingress repository: "" version: '*.*.*' - alias: nginx-ingress-geo condition: nginx-ingress-geo.enabled name: nginx-ingress repository: "" version: '*.*.*' - condition: gitlab-zoekt.install name: gitlab-zoekt repository: https://charts.gitlab.io/ version: 3.4.0 - condition: openbao.install name: openbao repository: https://charts.gitlab.io/ version: 0.5.1 - condition: traefik.install name: traefik repository: https://helm.traefik.io/traefik version: 10.19.4 - alias: haproxy condition: haproxy.install name: kubernetes-ingress repository: https://haproxytech.github.io/helm-charts version: 1.39.4 description: GitLab is the most comprehensive AI-powered DevSecOps Platform. home: https://about.gitlab.com/ icon: https://gitlab.com/gitlab-com/gitlab-artwork/raw/master/logo/logo-square.png keywords: - gitlab maintainers: - email: support@gitlab.com name: GitLab Inc. name: gitlab sources: - https://gitlab.com/gitlab-org/charts/gitlab version: 9.4.2 -
Platform:
- Self-hosted: NixOS 25.05 K8s 3-nodes cluster
-
Kubernetes: (
kubectl version)- Client: v1.32.0
- Server: v1.33.0
-
Helm: (
helm version)- Client: v3.17.1
Relevant logs
from migrations pod and some dependencies init container:
Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Attempting to run '/bin/bash -c set -e;
/scripts/wait-for-deps;
/scripts/db-migrate;
' as a main process
Checking: resque.yml, cable.yml
[ClickHouse] INFO: ClickHouse is not configured. Skipping migration checks.
+ SUCCESS connecting to 'redis://redis-master.redis.svc.cluster.local:6379/1' from resque.yml, through redis-master.redis.svc.cluster.local
+ SUCCESS connecting to 'redis://redis-master.redis.svc.cluster.local:6379/1' from cable.yml, through redis-master.redis.svc.cluster.local
Checking: main, ci
WARNING: Problem accessing ci database (gitlab). Confirm username, password, and permissions.
Database Schema - ci (gitlab)
NOTICE: Database has not been initialized yet.
WARNING: schema version check bypassed by BYPASS_SCHEMA_VERSION='true'
WARNING: Problem accessing main database (gitlab). Confirm username, password, and permissions.
Database Schema - main (gitlab)
NOTICE: Database has not been initialized yet.
WARNING: schema version check bypassed by BYPASS_SCHEMA_VERSION='true'
NOTICE: There are 4857 pending migrations.
NOTICE: There are 4857 pending migrations.
Checking database migrations are up-to-date
Performing migrations (this will initialized if needed)
Skipping Topology Service health check due to the cell being disabled
Running db:schema:load:main rake task
rake aborted!
TypeError: Invalid type for configuration. Expected Symbol, String, or Hash. Got nil
/srv/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.1.5.2/lib/active_record/database_configurations.rb:183:in `resolve'
/srv/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/connection_handler.rb:324:in `resolve_pool_config'
/srv/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/connection_handler.rb:134:in `establish_connection'
/srv/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.1.5.2/lib/active_record/tasks/database_tasks.rb:535:in `with_temporary_pool'
/srv/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.1.5.2/lib/active_record/tasks/database_tasks.rb:519:in `with_temporary_connection'
/srv/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.1.5.2/lib/active_record/tasks/database_tasks.rb:510:in `with_temporary_connection_for_each'
/srv/gitlab/vendor/bundle/ruby/3.2.0/gems/activerecord-7.1.5.2/lib/active_record/railties/databases.rake:588:in `block (5 levels) in <main>'
/srv/gitlab/lib/tasks/gitlab/db.rake:188:in `configure_database'
/srv/gitlab/lib/tasks/gitlab/db.rake:163:in `block in configure_pg_databases'
/srv/gitlab/lib/tasks/gitlab/db.rake:160:in `each'
/srv/gitlab/lib/tasks/gitlab/db.rake:160:in `configure_pg_databases'
/srv/gitlab/lib/tasks/gitlab/db.rake:102:in `block (3 levels) in <main>'
Tasks: TOP => db:schema:load:main => db:test:purge:main
(See full trace by running task with --trace)