Token Permissions Prevent Prometheus

Not sure if this issue belongs in prometheus or here, but adding it here for now.

Permissions to /vault/secrets/token prevent prometheus from reading the file and monitoring vault:

Prometheus pod runs as:

uid=1000 gid=2000 groups=2000

Permissions are:

$ ls -la /vault/secrets/token
-rw-r-----. 1 100 1000 95 Dec  5 17:33 /vault/secrets/token

the vault-agent-init init container fires off to create the file and set permissions.

Error seen here: https://prometheus.dogfood.bigbang.mil/targets?search=&scrapePool=vault

Temp work-around was to shell into the vault-agent init container and chmod 644 on the /vault/secrets/token file

We might need to set the group permissions /vault/secrets/token to 2000 as one fix, or find another option so prometheus uid:gid of 1000:2000 can read the file