Draft: Resolve "Thanos ServiceMonitors showing down"

General MR

Summary

Fix the multiple prometheus replica serviceMonitor down issue as shown described in bigbang/monitoring!162

Relevant logs/screenshots

Linked Issue

#40 (closed)

Upgrade Notices

N/A

added kindbug priority3 teamObservability + 1 deleted label

assigned to @jimmyungerman

Going to need to create logic in the monitoring chart to create a peerAuthentication bypass rule for the Thanos port if it's activated.

closed

MR not needed. All changes for this fix happen in bigbang/monitoring!162

added thanos label