Another Prometheus mem increase

changed milestone to %1.15.0

added statusreview label

I think this will violate our no big containers rule - https://repo1.dso.mil/platform-one/big-bang/apps/core/policy/-/blob/main/chart/values.yaml#L304

If needed we can obviously add the exception when we change it to enforcing just good to keep in mind.

Ryan sent me this from ironbank POPS team:

prometheus-k8s-0 551m 7246Mi

So in production this will definitely use more than what we have given it. An exception will have to be added for it

https://www.robustperception.io/why-does-prometheus-use-so-much-ram

https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion

yup 100%

with the value pasted above that was collected from a production cluster, should we be upping limits to something like 8-9Gi since if it wants to use 7246Mi it will get OOM killed.

approved this merge request

approved this merge request

unapproved this merge request

While we're tearing apart this MR, can we scale back the CI usage so we're not killing dogfood/runners with this?

https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring/-/blob/main/tests/test-values.yaml

adding the values to test now

set it to 1Gi for the package pipeline

Think you can edit your comment and get away with it? My email tells the truth. I'll never approve this MR!!!!!!!

oh you caught that did ya?

added 1 commit

• b82ca1da - Update test-values.yaml

Compare with previous version

added 1 commit

• 95d1c2f4 - Update test-values.yaml

Compare with previous version

added 1 commit

• fa8f7628 - Update test-values.yaml

Compare with previous version

approved this merge request

merged

mentioned in commit b8172902