SKIP UPGRADE vault update to 0.31.0-bb.4
Package Merge Request
SKIP UPGRADE since we know baseline is broken
Package Changes
https://repo1.dso.mil/big-bang/product/packages/vault/-/blob/0.31.0-bb.4/CHANGELOG.md
Package MR
big-bang/product/packages/vault!271 (merged)
big-bang/product/packages/vault!272 (merged)
big-bang/product/packages/vault!276 (merged)
For Issue
Closes big-bang/product/packages/vault#176 (closed)
Closes big-bang/product/packages/vault#175 (closed)
Closes big-bang/product/packages/vault#178 (closed)
Upgrade Notices
If you have used Big Bang's provided Vault autoInit for your Vault installtion -- please read on.
Prior versions of the auto init script provided by Big Bang passed in the token_reviewer_jwt and kubernetes_ca_cert flags when configuring Kubernetes authentication in Vault. As Kubernetes has moved to short-lived tokens, the jwt token used during the initual autoInit will change when the Vault pod is restarted. This can prevent other pods e.g. Promtheus from being able to connect to Vault when the pods are bounced. You may see the vault-agent-init container fail with these types of errors:
Error making API request...Code: 403. Errors:...Code: 403. Errors:
This update no longer passes in token_reviewer_jwt and kubernetes_ca_cert during the initial Vault installation when the autoInit logic runs. Not passing in those two flags configures Vault to dynamically re-read the CA and token files. ( See Vault documentation for more details https://developer.hashicorp.com/vault/docs/auth/kubernetes#kubernetes-1-21 )
If you are upgrading Vault, and you see a pod like prometheus-monitoring-monitoring-kube-prometheus-0 failing to start when restarted due to the vault-agent-init container not connecting to vault, you can update the Vault kubernetes config using one of two methods:
Web Interface Method
- Navigate to your domains vault config -- e.g
https://vault.dev.bigbang.mil/ui/vault/settings/auth/configure/kubernetes/configuration - Verify
Disable use of local CA and service account JWTis not checked - Click the Save button ( even if the previous value was not checked )
- delete the pod and verify it starts up correctly
Command Line Method
- Retrieve your vault login token
- $ kubectl get secrets -n vault vault-token -o json | jq -r '.data["key"]' | base64 -d
- login to the vault container
- $ kubectl exec -it -n vault vault-vault-0 -- bash
- inside the vault container, login to vault by passing in your token from the previous step
- bash-5.1$ vault login
- inside the vault container, update the kubernetes auth
- bash-5.1$ vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" issuer="https://kubernetes.default.svc.cluster.local"
- Success! Data written to: auth/kubernetes/config
- bash-5.1$ exit
- delete any vault-injected pods and verify they start back up correctly
Reach out to the Big Bang team member if you run into issues.