Configure Prometheus to write to Mimir when Mimir is enabled

As a developer, I want to connect Grafana Mimir with our Prometheus data sources to centralize metrics collection and storage:

Acceptance Criteria:

• Configure Mimir to pull metrics from Prometheus and Metrics Server
• Verify that Mimir is storing metrics in a format compatible with Prometheus queries
• Test basic query operations to ensure data accuracy and availability

Note: Requires big-bang/apps/sandbox/mimir#27 to validate metrics with Grafana dashboard

added to epic big-bang&414 (closed)

added teamObservability label

added triage-kind label

added triage-priority label

changed the description

set weight to 3

added statusready-to-work label

removed triage-priority label

removed triage-kind label

marked big-bang/apps/sandbox/mimir#26 as a duplicate of this issue

marked this issue as related to big-bang/apps/sandbox/mimir#26

added triage-kind label

added triage-priority label

assigned to @bjacksonfv

changed iteration to Big Bang Iterations Nov 12, 2024 - Nov 25, 2024

Completed proof of concept - Prometheus is successfully pushing logs to Mimir

➜  k logs monitoring-mimir-nginx-6fb8c5cdb-tq5vz -n monitoring | tail
10.42.2.11 - - [15/Nov/2024:14:26:19 +0000]  200 "POST /api/v1/push HTTP/1.1" 0 "-" "Prometheus/2.54.1" "-"
10.42.2.11 - - [15/Nov/2024:14:26:20 +0000]  200 "POST /api/v1/push HTTP/1.1" 0 "-" "Prometheus/2.54.1" "-"
10.42.2.11 - - [15/Nov/2024:14:26:21 +0000]  200 "POST /api/v1/push HTTP/1.1" 0 "-" "Prometheus/2.54.1" "-"
10.42.2.11 - - [15/Nov/2024:14:26:26 +0000]  200 "POST /api/v1/push HTTP/1.1" 0 "-" "Prometheus/2.54.1" "-"

added statusdoing label and removed statusready-to-work label

Running into issues with Prometheus -> Mimir communications when istio gets enabled; continuing to troubleshoot.

removed iteration Big Bang Iterations Nov 12, 2024 - Nov 25, 2024

changed iteration to Big Bang Iterations Nov 26, 2024 - Dec 9, 2024