UNCLASSIFIED - NO CUI

fix(metrics): preserve metrics user password across renders

Summary

metricsUserSecret.yaml regenerates TWISTLOCK_METRICS_PASSWORD with randAlphaNum on every Helm render, and only the init Job reconciles the new value into the Twistlock bigbang-metrics-sa user account. twistlock-init.sh runs set -e and dispatches metrics after license, DNS, SSO, users, and defenders, so any earlier failure leaves the Secret holding a password the user account has never seen. Prometheus then 401s on every scrape.

Two changes in one MR:

  1. Add a lookup fallback in metricsUserSecret.yaml so the password is generated once on first install and preserved on every subsequent render. Stops the drift and stops gratuitous password rotation on chart bumps. Matches the pattern used by other BB packages for persistent chart-managed secrets.
  2. Move the metrics dispatch in twistlock-init.sh to run directly after license. Metrics only needs the auth token and license. It doesn't need DNS, SSO, users, or defender rollout. Narrows the remaining failure window if the password ever does need to be rewritten.

No changes to values, Secret name, Secret keys, the ServiceMonitor, or twistlock-metrics.sh itself. Chart bumped 0.25.0-bb.0 -> 0.25.0-bb.2 (reserving -bb.1 for the in-flight console.extraEnv MR so the two don't collide on the same fork-patch).

Drive-by: stripped stray trailing tabs in the Groups block of twistlock-init.sh while touching the file.

Relevant logs/screenshots

New helm-unittest suite (chart/tests/metrics_secret_test.yaml):

$ helm unittest chart -f 'tests/metrics_secret_test.yaml'

### Chart [ twistlock ] chart

 PASS  test metrics user secret persistence  chart/tests/metrics_secret_test.yaml

Charts:      1 passed, 1 total
Test Suites: 1 passed, 1 total
Tests:       6 passed, 6 total

Covers: conditional render gates (monitoring.enabled, init.enabled), Secret name and kind, TWISTLOCK_METRICS_USER pinned to bigbang-metrics-sa, password is non-empty and base64 charset on fresh render, app.kubernetes.io/component: monitoring label present.

Shellcheck clean on the reordered init script:

$ bash -n chart/scripts/twistlock-init.sh && shellcheck chart/scripts/twistlock-init.sh

Rendered output still quotes the password correctly:

$ helm template t chart/ --set init.enabled=true --set monitoring.enabled=true \
  --show-only templates/init/metrics/metricsUserSecret.yaml | tail -3
stringData:
  TWISTLOCK_METRICS_USER: "bigbang-metrics-sa"
  TWISTLOCK_METRICS_PASSWORD: "dFZzdmw3ckhyNklUc2lyVXU4bDdWUDloTjJZSUlyVnI="

Idempotency on upgrade (run against a dev cluster with the chart installed, identical --set flags on both invocations so the only variable is the lookup path):

PW1=$(kubectl get secret -n twistlock twistlock-console-metrics-auth \
  -o jsonpath='{.data.TWISTLOCK_METRICS_PASSWORD}' | base64 -d)
helm upgrade t chart/ --set init.enabled=true --set monitoring.enabled=true
PW2=$(kubectl get secret -n twistlock twistlock-console-metrics-auth \
  -o jsonpath='{.data.TWISTLOCK_METRICS_PASSWORD}' | base64 -d)
[ "$PW1" = "$PW2" ] && echo PASS || echo FAIL
# PASS

Recovery (delete Secret, confirm regeneration + init Job resyncs the user):

kubectl delete secret -n twistlock twistlock-console-metrics-auth
helm upgrade t chart/ --set init.enabled=true --set monitoring.enabled=true
kubectl logs -n twistlock -l job-name=twistlock-init-<timestamp> \
  | grep "metrics service account user 'bigbang-metrics-sa'"
# Ensuring default metrics service account user 'bigbang-metrics-sa' ... OK

Linked Issue

issue

Upgrade Notices

N/A

Edited by Christopher O'Connell

Merge request reports

Loading
Loading