General MR

Summary

Updated logic to allow Keycloak to work on either gateway
Added new dod certificate chain (required for Keycloak to work properly on TLS terminated gateway)
Updated Keycloak template to allow merging on extraEnvs key so our logic survives (but can still be overridden)
Updated k3d-dev script to include a new parameter to allow dev cluster to use it on the TLS terminated gateway
Added helm unit tests to validate templating works as expected

Relevant logs/screenshots

Added additional test to package: big-bang/product/packages/keycloak!374 (merged)

Used the following override as the last override:

addons:  
  keycloak:
    ingress:
      key: "" # Clear this out to make sure the existing logic works as expected
      cert: "" # Clear this out to make sure the existing logic works as expected
    values:
      upstream:
        extraVolumes: |-
          - name: keycloak-conf
            emptyDir: {}
          - name: import-data
            emptyDir: {}
          - name: plugin
            emptyDir: {}
        extraVolumeMounts: |-
          - name: keycloak-conf
            mountPath: /opt/keycloak/conf/custom-registration-config.yaml
            subPath: custom-registration-config.yaml
          - name: keycloak-conf
            mountPath: /opt/keycloak/conf/quarkus.properties
            subPath: quarkus.properties
          - name: import-data
            mountPath: /opt/keycloak/data/import/realm.json
            subPath: realm.json
          - name: plugin
            mountPath: /opt/keycloak/providers/p1-keycloak-plugin.jar
            subPath: p1-keycloak-plugin.jar

Before Migration:

kubectl get gateway public-ingressgateway -n istio-gateway -o yaml

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  annotations:
    meta.helm.sh/release-name: public-ingressgateway
    meta.helm.sh/release-namespace: istio-gateway
  creationTimestamp: "2026-09-23T22:40:15Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: public-ingressgateway
    helm.toolkit.fluxcd.io/namespace: bigbang
  name: public-ingressgateway
  namespace: istio-gateway
  resourceVersion: "7362"
  uid: 4652da69-5895-4e9a-9798-9da6f261f53a
spec:
  selector:
    app: public-ingressgateway
  servers:
  - hosts:
    - '*.dev.bigbang.mil'
    port:
      name: http
      number: 8080
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - '*.dev.bigbang.mil'
    port:
      name: https
      number: 8443
      protocol: HTTPS
    tls:
      credentialName: public-cert
      mode: SIMPLE

kubectl get vs -n keycloak

NAME       GATEWAYS                                       HOSTS                          AGE
keycloak   ["istio-gateway/passthrough-ingressgateway"]   ["keycloak.dev.bigbang.mil"]   26m

kubectl get netpol -n keycloak | grep gateway

allow-ingress-to-keycloak-8443-from-ns-istio-gateway-pod-passthrough-ingressgateway   app.kubernetes.io/name=keycloak   26m

kubectl get ap -n keycloak | grep gateway

keycloak-passthrough-ingressgateway-authz-policy                                                                 ALLOW    26m

After Migration:

NOTE: Make sure to update any local DNS entries including any inside k8s cluster and do a rollout restart on coredns pod

kubectl get gateway public-ingressgateway -n istio-gateway -o yaml

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  annotations:
    meta.helm.sh/release-name: public-ingressgateway
    meta.helm.sh/release-namespace: istio-gateway
  creationTimestamp: "2026-09-23T22:40:15Z"
  generation: 2
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: public-ingressgateway
    helm.toolkit.fluxcd.io/namespace: bigbang
  name: public-ingressgateway
  namespace: istio-gateway
  resourceVersion: "51507"
  uid: 4652da69-5895-4e9a-9798-9da6f261f53a
spec:
  selector:
    app: public-ingressgateway
  servers:
  - hosts:
    - '*.dev.bigbang.mil'
    port:
      name: http
      number: 8080
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - '*.dev.bigbang.mil'
    port:
      name: https
      number: 8443
      protocol: HTTPS
    tls:
      credentialName: public-cert
      mode: SIMPLE
  - hosts:
    - keycloak.dev.bigbang.mil
    port:
      name: https-keycloak
      number: 8443
      protocol: HTTPS
    tls:
      credentialName: public-cert
      mode: OPTIONAL_MUTUAL

kubectl get vs -n keycloak

NAME       GATEWAYS                                  HOSTS                          AGE
keycloak   ["istio-gateway/public-ingressgateway"]   ["keycloak.dev.bigbang.mil"]   41m

kubectl get netpol -n keycloak | grep gateway

allow-ingress-to-keycloak-8080-from-ns-istio-gateway-pod-public-ingressgateway   app.kubernetes.io/name=keycloak   3m51s

kubectl get ap -n keycloak | grep gateway

keycloak-public-ingressgateway-authz-policy                                                                      ALLOW    3m56s

Verified login functionality to Keycloak (tied my CAC user to existing Cypress user), Grafana, and Prometheus prior to the change.

Validated above login functionality continued to work after migration and also tested fresh logins to Kiali, Alert-Manager, and Kyverno-Reporter.

Verified no other changes required outside of updating values and DNS.

Linked Issue

issue

Upgrade Notices

Keycloak can now use either a TLS-terminated gateway or a passthrough gateway. We recommend migrating to TLS termination at Big Bang’s public gateway because this will become the default behavior in Big Bang 4.0. If the public and passthrough gateways use different addresses, update the DNS record for keycloak.<domain> to point to the public gateway.

To use Big Bang’s public gateway, clear any explicit gateway selection and remove the Keycloak certificate and key:

addons:
  keycloak:
    ingress:
      gateway: ""
      cert: ""
      key: ""

When both cert and key are empty, TLS terminates at the public gateway and traffic is forwarded to Keycloak over HTTP. Supplying both values continues to configure Keycloak for TLS passthrough.

When Keycloak uses the built-in public gateway, Big Bang automatically adds an OPTIONAL_MUTUAL server for keycloak.<domain>. Big Bang also creates the corresponding <credentialName>-cacert Secret containing the bundled DoD CA chain. Automatic CA creation is limited to an OPTIONAL_MUTUAL server for keycloak.<domain> on the built-in public gateway. Big Bang does not create CA Secrets for other public-gateway hosts, MUTUAL servers, passthrough gateways, or custom gateways. Check for conflicts if the CA Secret for the Keycloak public-gateway credential, normally public-cert-cacert, is already managed separately.

If Keycloak uses a custom TLS-terminating gateway, add an OPTIONAL_MUTUAL server for the Keycloak hostname under that gateway:

istioGateway:
  values:
    gateways:
      <gateway-name>:
        gateway:
          servers:
            - hosts:
                - "keycloak.<domain>"
              port:
                name: https-keycloak
                number: 8443
                protocol: HTTPS
              tls:
                credentialName: <gateway-credential-name>
                mode: OPTIONAL_MUTUAL

User-provided Keycloak extraEnv entries are now merged with Big Bang’s defaults. Entries with the same environment-variable name override the Big Bang entry. When moving to TLS termination, remove any user-provided KC_HTTPS_CERTIFICATE_FILE and KC_HTTPS_CERTIFICATE_KEY_FILE entries.

The extraEnvFrom, extraVolumeMounts, and extraVolumes values remain YAML strings and are still replaced in full when supplied by the user. Remove the tlscert and tlskey volumes and their corresponding mounts from any user-provided configuration. Retain unrelated entries such as custom configuration volumes and mounts.

As an example, this is being used with Keycloak using the passthrough gateway:

        extraVolumes: |-
          - name: keycloak-conf
            emptyDir: {}
          - name: tlscert
            secret:
              secretName: {{ include "keycloak.fullname" . }}-tlscert
          - name: tlskey
            secret:
              secretName: {{ include "keycloak.fullname" . }}-tlskey
        extraVolumeMounts: |-
          - name: tlscert
            mountPath: /etc/x509/https/tls.crt
            subPath: tls.crt
            readOnly: true
          - name: tlskey
            mountPath: /etc/x509/https/tls.key
            subPath: tls.key
            readOnly: true
          - name: tlscert
            mountPath: /opt/keycloak/conf/tls.crt
            subPath: tls.crt
            readOnly: true
          - name: tlskey
            mountPath: /opt/keycloak/conf/tls.key
            subPath: tls.key
            readOnly: true
          - name: keycloak-conf
            mountPath: /opt/keycloak/conf/custom-registration-config.yaml
            subPath: custom-registration-config.yaml

While the following would be used when using the TLS terminated gateway:

        extraVolumes: |-
          - name: keycloak-conf
            emptyDir: {}
        extraVolumeMounts: |-
          - name: keycloak-conf
            mountPath: /opt/keycloak/conf/custom-registration-config.yaml
            subPath: custom-registration-config.yaml

Edited Sep 28, 2026 by Jimmy Bourque

Admin message

Updated Keycloak templating to allow usage on either gateway