UNCLASSIFIED - NO CUI

Skip to content

Istio Injection Support

Bug

Description

I work on the Blackpearl product (specifically Lighthouse) and we use the BB helm chart for deploying gitlab as part of our software package. We pair it with Istio injection with strict mTLS. The current helm chart doesn't support istio injection as written and has regressed further from supporting it with the latest tags. We are maintaining our own version of the helm chart in order to properly support istio injection and would love to be able to push our fix upstream here.

We are currently having to maintain our own version of this chart with updates to support Istio injection even though we do set the global.istio.injection value in the values file to "enabled".

As the chart is currently written, the dependencies init container for the Sidekiq and Webservices pod goes into a crash loop. It runs a check on Postgres and Redis to confirm it's configured right, but that fails because the init container is trying to hit the istio proxy which isn't running yet. I'm currently fixing it in those two deployment files by setting the runAsUser to 1337 so Istio injection ignores that init container's network calls and updating an environment variable for the ruby script to work as expected.

Steps to replicate:

  1. Configure istio on the cluster and ensure the gitlab namespace has the istio-injection: "enabled" label on it
  2. Install the helm chart with the values file override ensuring istio injection is set to enabled
  3. Observe sidekiq and webservice pods stuck in the init phases with dependencies container throwing errors with no route to host for the postgres and redis services
---
global:
  gitlabBase:
    image:
      repository: registry1.dso.mil/ironbank/redhat/ubi/ubi9
  istio:
    enabled: false
    injection: enabled
  edtion: ee
  appConfig:
    omniauth:
      enabled: "###ZARF_VAR_OMNIAUTH_ENABLE###"
      allowSingleSignOn: ["###ZARF_VAR_SSO_VALUE###"]
      blockAutoCreatedUsers: false
      providers:
        - secret: gitlab-saml
  deployment:
    annotations:
      holdApplicationUntilProxyStarts: "true"
  hosts:
    domain: gitlab.###ZARF_VAR_DEPLOY_URL###
    gitlab:
      name: gitlab.###ZARF_VAR_DEPLOY_URL###
    registry:
      name: registry.###ZARF_VAR_DEPLOY_URL###
  certificates:
    customCAs:
      - secret: lighthouse-ca
  minio:
    enabled: true
    email:
      display_name: Lighthouse
    from: help@lighthouse.blackpearl.us
  gitlab:
    license:
      secret: gitlab-license
      key: license
  smtp:
    enabled:  ###ZARF_VAR_SMTP_ENABLE###
    address:  ###ZARF_VAR_SMTP_ADDRESS###
    port:  ###ZARF_VAR_SMTP_PORT###
    user_name:  ###ZARF_VAR_SMTP_USER_NAME###
    password:
      secret:  ###ZARF_VAR_SMTP_PASSWORD_SECRET###
      key:  ###ZARF_VAR_SMTP_PASSWORD_KEY##
    authentication:  ###ZARF_VAR_SMTP_AUTHENTICATION###
    starttls_auto:  ###ZARF_VAR_SMTP_START_TLS_AUTO###
    openssl_verify_mode:  ###ZARF_VAR_SMTP_OPENSSL_VERIFY_MODE###
    open_timeout:  ###ZARF_VAR_SMTP_OPEN_TIMEOUT###
    read_timeout:  ###ZARF_VAR_SMTP_READ_TIMEOUT###
    pool:  ###ZARF_VAR_SMTP_VAR_POOL###


  lfs:
    enabled: true
    proxy_download: true
    bucket: git-lfs
    connection: {}
  artifacts:
    enabled: true
    proxy_download: true
    bucket: gitlab-artifacts
  uploads:
    enabled: true
    proxy_download: true
  packages:
    enabled: true
    proxy_download: true
  backups:
    bucket: gitlab-backups
    tmpBucket: tmp
  psql:
    host: gitlab-postgresql.gitlab.svc.cluster.local
    username: gitlab
    password:
      secret: gitlab-db-config
      key: APP_DB_PASSWORD
minio:
  jobAnnotations:
    sidecar.istio.io/inject: "false"
  podAnnotations:
    proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
  init:
    resources:
      limits:
        memory: 500Mi
  resources:
    limits:
      memory: 1Gi
  minioMc:
    image: registry1.dso.mil/ironbank/opensource/minio/mc
    tag: RELEASE.2024-01-11T05-49-32Z
postgresql:
  install: false
gitlab:
  gitaly:
    annotations:
      proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
  gitlab-shell:
    enabled: false
  toolbox:
    annotations:
      proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
    init:
      containerSecurityContext:
        runAsUser: 65534
  minio:
    init:
      containerSecurityContext:
        runAsUser: 65534
  migrations:
    init:
      containerSecurityContext:
        runAsUser: 65534
    annotations:
      sidecar.istio.io/inject: "false"
  webservice:
    annotations:
      proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
  sidekiq:
    annotations:
      proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
gitlab-runner:
  install: false
registry:
  annotations:
    proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
  init:
    containerSecurityContext:
      runAsUser: 65534
# Jobs annotations
upgradeCheck:
  annotations:
    sidecar.istio.io/inject: "false"
shared-secrets:
  annotations:
    sidecar.istio.io/inject: "false"

Additional Regression

It used to be, when you set global.istio.injection to "enabled", all of the jobs would get the required sidecar.istio.io/inject: "false" annotation for the jobs to complete. That was removed from the helm chart in the last month or so resulting in all jobs to fail to complete because the sidecar proxy never stops. This can be fixed via the values file, but it would be nice if it was added back in when enabling injection so we don't have to do that additionally in the values file.

Proposed Fix

This is the code I swap in to the Sidekiq and Webservices deployment to fix it in our helm chart. It's adding the runAsUser: 1337 for the dependencies init container security context. In addition, the CONFIG_DIRECTORY environment variable needs to be set to /tmp so the container user has the necessary permissions to do the file changes it does.

        - name: dependencies
          image: {{ include "webservice.image" $ }}
          {{- include "gitlab.image.pullPolicy" $imageCfg | indent 10 }}
          {{- if eq $.Values.global.istio.injection "enabled" }}
          securityContext:
            capabilities:
              drop:
              - ALL
            runAsUser: 1337
          {{- else }}
          {{- include "gitlab.init.containerSecurityContext" $ | indent 10 }}
          {{- end }}
          args:
            - /scripts/wait-for-deps
          env:
            - name: GITALY_FEATURE_DEFAULT_ON
              value: "1"
            - name: CONFIG_TEMPLATE_DIRECTORY
              value: '/var/opt/gitlab/templates'
            {{- if eq $.Values.global.istio.injection "enabled" }}
            - name: CONFIG_DIRECTORY
              value: '/tmp'
            {{ else }}
            - name: CONFIG_DIRECTORY
              value: '/srv/gitlab/config'
            {{- end }}

BigBang Version

We are currently running version 8.1.2-bb.3 deploying the helm chart via the oci:// url. We use Zarf to package and deploy it into airgapped environments.

UNCLASSIFIED - NO CUI