Resolve "The AlertmanagerClusterDown alert fires, even though Alertmanager is healthy"

General MR

Summary

The customer wrote the following with a theory. The AlertmanagerClusterDown alert fires, even though Alertmanager is healthy. This happens because Prometheus cannot access the Alertmanager reloader-web port, due to lack of an AuthorizationPolicy port entry.

Initial thought was all that had to be added was a authorizationpolicy. However, upon further investigation in order to allow for prometheus to be able to access the necessary port 8080 via a GET involved a network policy, authorizationpolicy and peerauthentication.

Relevant logs/screenshots

The following shows the error when one of the 3 policies/kinds is missing

After the fix with all 3 configured

Relates #146 (closed)