UNCLASSIFIED - NO CUI

Support Short Term SA Tokens

Bug

Description

Our current auto-init function no longer works due to k8s service account using short-term tokens.

If the vault pod is bounced, the registered k8s token that was installed on initial create is no longer valid, and the prometheus pod is not able to connect to vault.

We can fix this per Vault documentation by removing token_reviewer_jwt and kubernetes_ca_cert from our auth init commands. See https://developer.hashicorp.com/vault/docs/auth/kubernetes#kubernetes-1-21

This behavior can be demonstrated by:

  • bounce the vault-vault-0 pod. A new service token is created under /run/secrets/kubernetes.io/serviceaccount/token
  • bounce the prometheus-monitoring-monitoring-kube-prometheus-0 -- the vault-agent-init pod fails to connect, since vault's running service token has now changed from the initial vault write auth/kubernetes/config ... command
Edited by Michael Martin