check for existing secret before creating a new password
Currently, the password for the metrics user is autogenerated in the Bigbang chart.
This is fine for the initial installation, but when an existing release is updated, the password stored in the secret(s) is also updated.
We observed a scenario in which the neuvector-prometheus
pod entered a fail loop because it was authenticating with incorrect credentials. At the point we discovered it, we were also blocked from logging into the Neuvector console with the metrics
user due to too many invalid login attempts.
We believe there are two possible error cases:
- The password was changed in the deployment, but the prometheus pod was not restarted, causing it to attempt (and fail) to login with the old credentials
- The password was changed in the secrets, but NOT the actual deployment -- and the metrics pod remained healthy until a "natural" error caused a restart, at which point it attempted (and failed) to login with the (incorrect) new credentials
Suggested resolution:
- Use a 'lookup' to determine if the secret already exists before autogenerating a new password
- Add a checksum annotation to the metrics pod