Gitlab 5.10.1-bb.0 Helm Chart Timed Out

I have deployed bigbang release 1.34.0 with flux 0.29.5. Gitlab(14.10.1) is still failing to deploy. We get Gitlab helm chart timed out error. Gitlab webservice and sidekiq pods are stuck on init0:2 and eventually goes into crashloopbackoff.

@micah.nagel @jasonkrause @ryan.j.garcia I'd appreciate your help with this. Thanks

changed the description

changed the description

Hello @sipvw5 please refer to our Production recommendations for Gitlab, you will need to increase the HR timeout settings for Gitlab: https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/master/docs/production.md#flux-settings

In your above screenshot the -migrations- pod has been running for almost 10 minutes and the default timeout for an HR is 10 minutes so it thinks it has failed. Once you increase the timeout the upgrade will succeed.

@ryan.j.garcia I have increased the time out to 30m/45m and tried deploying again. I am seeing the same behavior? The dependencies container still goes in to crashloopbackoff due to Error checking main: connection to server at "gitlab-postgresql.gitlab.svc" (10.43.203.33), port 5432 failed: fe_sendauth: error sending password authentication Checking: main

Hi @sipvw5 - we are seeing this issue as well in our nightly CI (does not have to do with the timeout). It strongly resembles an issue we had in the past - https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab/-/issues/123 - which was caused by password encryption type when on FIPS enabled nodes. We will be looking into it to resolve our issue with CI and update as we have more info.

Confirmed this is the same issue. In the past this was the fix.

For Big Bang what we had done was:

addons:
  gitlab:
    values:
      postgresql:
        # Required for FIPS since Gitlab 14.7.X
        postgresqlExtendedConf:
          passwordEncryption: "scram-sha-256"

Unfortunately this no longer seems to work...I would note that we always encourage using an external DB instead of the in cluster one - in which case you could set the encryption by editing the DB parameters (i.e. AWS DB config).

We're still digging into the correct way to set values for the in cluster DB.

assigned to @micah.nagel

assigned to @ryan.j.garcia

added kindbug label

changed milestone to %1.36.0

changed iteration to Big Bang Iterations May 31, 2022 - Jun 13, 2022

added statusdoing label

added gitlab label

Will most likely need to chop and change portions of the sub-chart packaged with Gitlab, since the registry1 postgresql image is coded to check for files/configs in /var/lib/postgresql instead of /bitnami/postgresql : https://github.com/bitnami/charts/blob/master/bitnami/postgresql/templates/primary/statefulset.yaml#L447

is this going to be addressed as part of bigbang release 1.34.x?

Since 1.34.0 is already out we will not be including any changes in that. We did document this as a requirement in BB 1.29.0 upgrade notices for FIPS nodes here. I'll update you on any changes we require for the values when using the in cluster DB - at most it may require you to pin to a new gitlab tag such as:

addons:
  gitlab:
    git:
      tag: "x.y.z-bb.0"

As well as adding some values to override the postgres setup.

mentioned in merge request !1708 (merged)

@sipvw5 setting this value should get your pods spinning up:

addons:
  gitlab:
    values:
      postgresql:
        postgresqlInitdbArgs: "-A scram-sha-256"
        postgresqlConfiguration: null
        pgHbaConfiguration: null

Essentially this ensures that during the initialization of the database the scram-sha-256 auth encryption method is used for the gitlab user password. These values are also set by default if you pin to the newer Gitlab 5.10.1-bb.1 tag.

Note that if you have an already existing database you will need to handle it slightly differently:

• Log into your postgres DB
• Execute the following, replacing <password> with your gitlab user password for postgres

  SET password_encryption  = 'scram-sha-256'; 
  ALTER USER "gitlab" with password '<password>';

That worked. Thank you guys for the quick fix. Appreciate it

closed with merge request !1708 (merged)

mentioned in commit 7b1d3e53

removed statusdoing label