AutoInit job fails with out-of-the-box configuration

changed title from AurtoInit job fails with out-of-the-box configuration to AutoInit job fails with out-of-the-box configuration

mentioned in merge request !95 (merged)

Hi @sam.james would you mind pasting in some sanitized logs and overrides and environment information? VAULT_SERVICE_HOST is a var set internally within the pod environment and since adding it we haven't seen any issues and has actually improved the reliability of the job

Sure! Give me a bit to re-create things and gather some info for you.

Steps I took to reproduce:

Create a local k8s deployment:

colima start vault-test -c 8 -m 16 -k

Deploy Flux

kubectl create ns flux-system 
kubectl create secret -n flux-system docker-registry private-registry --docker-server="registry1.dso.mil" --docker-username="Samuel_James" --docker-password=<redacted>
kustomize build "https://repo1.dso.mil/big-bang/bigbang.git/base/flux?ref=2.2.0" | kubectl apply -f -

Deploy BigBang

helm upgrade --install bigbang oci://registry1.dso.mil/bigbang/bigbang --version 2.2.0 \
--values https://repo1.dso.mil/platform-one/big-bang/bigbang/-/raw/master/chart/ingress-certs.yaml \
--values ib_creds.yaml \
--values test_values.yaml \
--namespace=bigbang --create-namespace

The contents of ib_creds.yaml are just the registryCredentials block.

The contents of test_values.yaml are as follows:

kiali:
  enabled: false
kyverno:
  enabled: false
kyvernoPolicies:
  enabled: false
kyvernoReporter:
  enabled: false
promtail:
  enabled: false
loki:
  enabled: false
neuvector:
  enabled: false
tempo:
  enabled: false
monitoring:
  enabled: false
addons:
  vault:
    enabled: true

The init job's init container will be stuck in the Init:0/1 status. Its logs are as follows:

 % kubectl logs -n vault vault-vault-job-init-ht89r vault-init-job
---=== Waiting For Vault Server ===---
---=== Waiting For Vault Server ===---
---=== Waiting For Vault Server ===---
---=== Waiting For Vault Server ===---
---=== Waiting For Vault Server ===---

Here's a dump of the container's ENV:

% kubectl exec -it -n vault vault-vault-job-init-ht89r -c vault-init-job -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=vault-vault-job-init-ht89r
TERM=xterm
KUBERNETES_SERVICE_HOST=10.43.0.1
VAULT_VAULT_UI_PORT=tcp://10.43.235.149:8200
VAULT_VAULT_UI_PORT_8200_TCP=tcp://10.43.235.149:8200
VAULT_VAULT_UI_PORT_8200_TCP_ADDR=10.43.235.149
VAULT_VAULT_PORT_8201_TCP_PROTO=tcp
VAULT_VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_PROTO=tcp
VAULT_VAULT_UI_SERVICE_HOST=10.43.235.149
VAULT_VAULT_UI_PORT_8200_TCP_PORT=8200
VAULT_VAULT_AGENT_INJECTOR_SVC_SERVICE_PORT_HTTPS=443
VAULT_VAULT_AGENT_INJECTOR_SVC_PORT=tcp://10.43.120.97:443
VAULT_VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_ADDR=10.43.120.97
VAULT_VAULT_PORT_8201_TCP_ADDR=10.43.105.158
KUBERNETES_PORT_443_TCP_PROTO=tcp
VAULT_VAULT_UI_SERVICE_PORT=8200
VAULT_VAULT_PORT_8200_TCP=tcp://10.43.105.158:8200
KUBERNETES_PORT_443_TCP_PORT=443
VAULT_VAULT_AGENT_INJECTOR_SVC_SERVICE_PORT=443
VAULT_VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_PORT=443
VAULT_VAULT_SERVICE_PORT_HTTP=8200
VAULT_VAULT_PORT_8200_TCP_PROTO=tcp
KUBERNETES_PORT=tcp://10.43.0.1:443
VAULT_VAULT_SERVICE_PORT=8200
VAULT_VAULT_PORT=tcp://10.43.105.158:8200
VAULT_VAULT_PORT_8200_TCP_ADDR=10.43.105.158
VAULT_VAULT_PORT_8201_TCP=tcp://10.43.105.158:8201
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
VAULT_VAULT_AGENT_INJECTOR_SVC_SERVICE_HOST=10.43.120.97
VAULT_VAULT_SERVICE_HOST=10.43.105.158
VAULT_VAULT_SERVICE_PORT_HTTPS_INTERNAL=8201
VAULT_VAULT_PORT_8200_TCP_PORT=8200
VAULT_VAULT_PORT_8201_TCP_PORT=8201
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
VAULT_VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP=tcp://10.43.120.97:443
VAULT_VAULT_UI_SERVICE_PORT_HTTP=8200
VAULT_VAULT_UI_PORT_8200_TCP_PROTO=tcp
container=oci
HOME=/

The overlay values are empty:

% kubectl get secrets -n bigbang bigbang-vault-values -o go-template='{{ index .data "overlays" | base64decode }}'
{}

Looks like the variable name should be VAULT_VAULT_SERVICE_HOST then? I can update my MR.

Hmm should probably be {{ .Release.Name }}_SERVICE_HOST . All of our pipelines and environments are just a single vault- prefix

Looks like we can use {{ upper .Release.Name }} to replace VAULT on that export line

Actually @sam.james we can use {{ include "vault.name" . | upper }} which will be vault-vault and looks like that's the name used in our release cluster as well while release name is still only vault

Should be all set in !95 (merged)

mentioned in commit c2f82fe2

closed with merge request !95 (merged)

mentioned in merge request big-bang/bigbang!2820 (merged)

mentioned in commit big-bang/bigbang@70b96e11

AutoInit job fails with out-of-the-box configuration

Designs

Child items ...

Activity