UNCLASSIFIED - NO CUI

Skip to content

Add an init container to the authservice deployment to verify redis connectivity, avoiding initial errors and CrashLoopBackOff

General MR

Summary

Add "wait-for-redis" init container to prevent authservice CrashLoopBackOff during startup

This change resolves startup failures where the authservice container enters CrashLoopBackOff when attempting to connect to Redis before it's ready. An init container now verifies Redis connectivity before allowing the authservice to start.

Additional changes:

  • Remove unused redis.image.tag variable - image configuration should be passed to the redis-bb chart instead
  • bump bb-common dependency to bb-common-0.8.2

Relevant logs/screenshots

Existing Authservice experience...

#Delete the authservice pods...
kubectl delete pods -n authservice --all

#After about 5 seconds, pods go into error.
kubectl get pods -n authservice
NAME                                        READY   STATUS    RESTARTS   AGE
authservice-6bb7c7b8bf-8885v                1/2     Error     0          10s
authservice-6bb7c7b8bf-t4279                1/2     Error     0          10s
authservice-authservice-redis-bb-master-0   1/2     Running   0          8s

kubectl logs -n authservice authservice-6bb7c7b8bf-8885v -f

#connection reset by peer occurs, because it tries to connect right away to a redis cluster that is not running.
...
time="2025/10/03 15:53:33" level=info msg="initializing redis session store" scope="session" chain="alertmanager" redis-url="redis://authservice-authservice-redis-bb-master:6379"
time="2025/10/03 15:53:33" level=debug msg="pre-run-exit" scope="default" name="OIDC session store factory" item="(5/7)"
time="2025/10/03 15:53:33" level=error msg="unexpected exit" scope="default" error="pre-run OIDC session store factory: read: connection reset by peer"
Unexpected exit: pre-run OIDC session store factory: read: connection reset by peer
...

#After about 10-12 seconds, you then get a CrashLoopBackoff...
kubectl get pods -n authservice
NAME                                        READY   STATUS             RESTARTS     AGE
authservice-6bb7c7b8bf-8885v                1/2     CrashLoopBackOff   1 (3s ago)   12s
...

Upgrade and test using this branch...

helm upgrade -i bigbang ./bigbang/chart/ \
    --debug \
    -n bigbang \
    --create-namespace \
    --set registryCredentials.username="$REGISTRY_USERNAME" \
    --set registryCredentials.password="$REGISTRY_PASSWORD" \
    --values ./bigbang/tests/test-values.yaml \
    --values ./bigbang/chart/ingress-certs.yaml \
    --values authservice/docs/dev-overrides/minimal.yaml \
    --values authservice/docs/dev-overrides/authservice-testing-local-keycloak.yaml \
    --values kiali/docs/dev-overrides/kiali-testing-local-keycloak.yaml \
    --values <(cat << YAML
addons:
  authservice:
    enabled: true
    git:
      branch: 193-authservice-probes
      tag: null
YAML
)
#Open up multiple terminals
#Wait for helmrelease to be ready, then....
kubectl delete pods -n authservice --all

#On terminal one, notice that authservice never goes into CrashLoopBackOff
watch "kubectl get pods -n authservice"

#On terminal two, check that the init container will keep trying every 2 seconds until redis is available.
kubectl logs -n authservice <authservicepod> wait-for-redis -f

#Verify that init container will reach a failure state after 2 minutes, and then it will try to restart the init container around the two minute mark....
kubectl scale sts -n authservice --all --replicas=0
kubectl delete pods -n authservice --all
watch "kubectl get pods -n authservice"
Every 2.0s: kubectl get pods -n authservice                                                                               ironman: Fri Oct  3 12:19:31 2025

NAME                           READY   STATUS     RESTARTS      AGE
authservice-5bc644b4bf-fk29z   1/2     Init:3/4   1 (48s ago)   2m56s
authservice-5bc644b4bf-twzx9   1/2     Init:3/4   1 (52s ago)   2m56s

Linked Issue

issue

Upgrade Notices

N/A

Closes #193 (closed)

Edited by Jesse Borden

Merge request reports

Loading