MinIO serviceMonitor/dashboard not working

Looking in dogfood currently the serviceMonitor is not being deployed for MinIO and the dashboard is empty.

There are a couple problems at play here:

• Metrics are not being scraped (no service monitor)
• Dashboard may be outdated (and should really be deployed as part of the MinIO package rather than the monitoring package)

With regard to metrics not being scraped, there are missing values from the tenant to set up metrics via the operator:

tenants:
  metrics:
    enabled: true
  prometheusOperator: true

Additionally, the operator has changed how the metrics scraping works - now a scraping config is deployed, which requires the operator to be deployed with knowledge of the namespace for prometheus. As a result you need to add an ENV to the operator pod.

We may want to evaluate whether we should just disable the metrics provisioning from the operator and handle deployment of the servicemonitor on our own.

I have a draft MR that has some of the base changes to get MinIO to be scraped again (using the new operator way) - https://repo1.dso.mil/platform-one/big-bang/bigbang/-/merge_requests/1473 . This does not seem to resolve the dashboard issues but could be used as a starting point to explore the problem and a potential solution to the scraping issue.

AC:

• Metrics being scraped automatically when minio and monitoring are enabled
• Grafana dashboard being populated with data for Minio when both are enabled
• Grafana dashboard removed from monitoring package and added to Minio package