[P1BIGROCKS-2752] Create global PodMonitor / ServiceMonitor to scrape metrics using labels
| Team Lead | Members |
| --- | --- |
| Alfredo Diaz | |
[P1BIGROCKS-2752](https://jira.il2.dso.mil/browse/P1BIGROCKS-2752)
# Problem Statement
Most packages in Big Bang are dependent on the Monitoring package when deployed because they have implemented a `PodMonitor` or `ServiceMonitor` resource and need the CRDs created in advance. This can cause delays in deployment as monitoring must install before them. In addition, each package is free to implement their own version of a helm template for the ServiceMonitor.
# Proposal
Use `additionalScrapeConfigs` in the operator to scrape services with label configurations for `scrape`, `port`, `path` and `scheme` like [this example](https://github.com/prometheus-operator/prometheus-operator/issues/1547#issuecomment-446711073). Note that the developers [recommend using labels over annotations](https://github.com/prometheus-operator/prometheus-operator/issues/1547#issuecomment-401092041), so the example should be updated.
# Benefits
- Removes dependency for most packages on monitoring, allowing it to deploy sooner / in parallel in the flux cycle
- Simple way for mission apps to add metrics scraping to their application without deploying additional resources
- Reduces maintenance of Big Bang configuring different implementations of monitors in the packages. (Offset by maintaining different ways of adding labels to services/pods)
- Still allows use of `*Monitor` resources if labeling is a more powerful configuration is needed
# Limitations
- Cannot handle multiple scrape ports in same service/pod
- Cannot handle different authentication schemes (e.g. some http and some https) in same service/pod
- Cannot handle subsets of targets in same service/pod
> I don't see any of our packages (so far) that need a more complex configuration than just setting the `port` and `path`.
epic