`mimir` is failing smoke tests in BB all-packages pipelines

Issue

When an all-packages pipeline is run in BB umbrella, mimir is failing to run its smoke tests.

Example MR with failing all-packages pipeline: big-bang/bigbang!5723

Relevant logs:

Error: 1 error occurred:
	* job mimir-mimir-smoke-test failed: BackoffLimitExceeded'
++ sed -n '/Error:/,$p'
+ log_errors='Error: 1 error occurred:
	* job mimir-mimir-smoke-test failed: BackoffLimitExceeded'
+ [[ '' != \N\o\n\e ]]
+ [[ -n Error: 1 error occurred:
	* job mimir-mimir-smoke-test failed: BackoffLimitExceeded ]]

Acceptance Criteria

• mimir's smoke tests run and succeed in BB umbrella all-packages pipelines.

added kindci mimir priority5 teamObservability labels

set weight to 2

marked this issue as related to #61 (closed)

mentioned in merge request big-bang/bigbang!5718 (merged)

marked this issue as blocking kyverno-policies#150 (closed)

added statusready-to-work label

changed iteration to Big Bang Iterations Feb 4, 2025 - Feb 17, 2025

assigned to @peter.sigur

added statusdoing label and removed statusready-to-work label

mentioned in issue #61 (closed)

removed iteration Big Bang Iterations Feb 4, 2025 - Feb 17, 2025

changed iteration to Big Bang Iterations Feb 18, 2025 - Mar 3, 2025

Was able to get (most of) an all-packages cluster running locally to test and recreate the issue.

Seeing following errors in Mimir smoke-test pod before it hits backofflimitexceeded:

Posting the "success" logs for comparison - Successful when not installing all-packages

Smoke test containter args - Note HTTP over 80

│         - -target=continuous-test                                                                                                                                                                                                                         │
│         - -activity-tracker.filepath=                                                                                                                                                                                                                     │
│         - -tests.smoke-test                                                                                                                                                                                                                               │
│         - -tests.write-endpoint=http://mimir-mimir-gateway.mimir.svc:80                                                                                                                                                                                   │
│         - -tests.read-endpoint=http://mimir-mimir-gateway.mimir.svc:80/prometheus                                                                                                                                                                         │
│         - -tests.tenant-id=                                                                                                                                                                                                                               │
│         - -tests.write-read-series-test.num-series=1000                                                                                                                                                                                                   │
│         - -tests.write-read-series-test.max-query-age=48h                                                                                                                                                                                                 │
│         - -server.http-listen-port=8080           

Looking like this should be fixed by increasing or removing the max_global_series_per_user limit: big-bang/bigbang!5793 (merged)

(credit to @bjacksonfv for the find and fix)

Nice. Great find @bjacksonfv.

added statusblocked label and removed statusdoing label

removed iteration Big Bang Iterations Feb 18, 2025 - Mar 3, 2025

changed iteration to Big Bang Iterations Mar 4, 2025 - Mar 17, 2025

Closing: fixed by changes made in #61 (closed)

closed

removed statusblocked label