Add an init container to the authservice deployment to verify redis connectivity, avoiding initial errors and CrashLoopBackOff
General MR
Summary
Add "wait-for-redis" init container to prevent authservice CrashLoopBackOff during startup
This change resolves startup failures where the authservice container enters CrashLoopBackOff when attempting to connect to Redis before it's ready. An init container now verifies Redis connectivity before allowing the authservice to start.
Additional changes:
- Remove unused redis.image.tag variable - image configuration should be passed to the redis-bb chart instead
- bump bb-common dependency to bb-common-0.8.2
Relevant logs/screenshots
Existing Authservice experience...
#Delete the authservice pods...
kubectl delete pods -n authservice --all
#After about 5 seconds, pods go into error.
kubectl get pods -n authservice
NAME READY STATUS RESTARTS AGE
authservice-6bb7c7b8bf-8885v 1/2 Error 0 10s
authservice-6bb7c7b8bf-t4279 1/2 Error 0 10s
authservice-authservice-redis-bb-master-0 1/2 Running 0 8s
kubectl logs -n authservice authservice-6bb7c7b8bf-8885v -f
#connection reset by peer occurs, because it tries to connect right away to a redis cluster that is not running.
...
time="2025/10/03 15:53:33" level=info msg="initializing redis session store" scope="session" chain="alertmanager" redis-url="redis://authservice-authservice-redis-bb-master:6379"
time="2025/10/03 15:53:33" level=debug msg="pre-run-exit" scope="default" name="OIDC session store factory" item="(5/7)"
time="2025/10/03 15:53:33" level=error msg="unexpected exit" scope="default" error="pre-run OIDC session store factory: read: connection reset by peer"
Unexpected exit: pre-run OIDC session store factory: read: connection reset by peer
...
#After about 10-12 seconds, you then get a CrashLoopBackoff...
kubectl get pods -n authservice
NAME READY STATUS RESTARTS AGE
authservice-6bb7c7b8bf-8885v 1/2 CrashLoopBackOff 1 (3s ago) 12s
...
Upgrade and test using this branch...
helm upgrade -i bigbang ./bigbang/chart/ \
--debug \
-n bigbang \
--create-namespace \
--set registryCredentials.username="$REGISTRY_USERNAME" \
--set registryCredentials.password="$REGISTRY_PASSWORD" \
--values ./bigbang/tests/test-values.yaml \
--values ./bigbang/chart/ingress-certs.yaml \
--values authservice/docs/dev-overrides/minimal.yaml \
--values authservice/docs/dev-overrides/authservice-testing-local-keycloak.yaml \
--values kiali/docs/dev-overrides/kiali-testing-local-keycloak.yaml \
--values <(cat << YAML
addons:
authservice:
enabled: true
git:
branch: 193-authservice-probes
tag: null
YAML
)
#Open up multiple terminals
#Wait for helmrelease to be ready, then....
kubectl delete pods -n authservice --all
#On terminal one, notice that authservice never goes into CrashLoopBackOff
watch "kubectl get pods -n authservice"
#On terminal two, check that the init container will keep trying every 2 seconds until redis is available.
kubectl logs -n authservice <authservicepod> wait-for-redis -f
#Verify that init container will reach a failure state after 2 minutes, and then it will try to restart the init container around the two minute mark....
kubectl scale sts -n authservice --all --replicas=0
kubectl delete pods -n authservice --all
watch "kubectl get pods -n authservice"
Every 2.0s: kubectl get pods -n authservice ironman: Fri Oct 3 12:19:31 2025
NAME READY STATUS RESTARTS AGE
authservice-5bc644b4bf-fk29z 1/2 Init:3/4 1 (48s ago) 2m56s
authservice-5bc644b4bf-twzx9 1/2 Init:3/4 1 (52s ago) 2m56s
Linked Issue
Upgrade Notices
N/A
Closes #193 (closed)
Edited by Jesse Borden