Promtail in bigbang 1.35.0 doesn't support extraClientConfigs

Overview

Updating my bigbang installation from 1.34.0 -> 1.35.0 introduced a breaking change to the recommended values configuration for GEL in the Promtail chart.

Example values:

promtail:
  enabled: true
  values:
    config:
      lokiAddress: http://logging-loki-gel-gateway.logging.svc.cluster.local/loki/api/v1/push
      snippets:
        extraClientConfigs: |
          basic_auth:
            username: cloudops-prd
            password: <password>
          tenant_id: cloudops-prd
          external_labels:
            agency: 2f-cloudops
            environment: production

Observed behavior in BB `1.34.0` (Promtail chart `3.11.0-bb.1`):

Promtail starts as expected, injecting the extraClientConfig (namely, auth information) into the logging-promtail secret and subsequent app config.

Observed behavior in BB `1.35.0` (Promtail chart `4.2.0-bb.0`):

Promtail crashes on startup with: Unable to parse config: /etc/promtail/promtail.yaml: yaml: line 6: did not find expected '-' indicator

Temporary resolution:

Rolled back Promtail in all our environments by pinning to 3.11.0-bb.1:

promtail:
  git:
    tag: "3.11.0-bb.1"

Root cause (probably?)

It appears this was introduced upstream, and included in the BB chart here. The issue may be resolved in upstream version 5.0.0.

Proposed Solution

Short term: Perhaps a note in the known issues for BB 1.35.0
Long(er) term: Update the BB chart to include the MR listed above

Admin message