The AlertmanagerClusterDown alert fires, even though Alertmanager is healthy

Bug

The AlertmanagerClusterDown alert fires, even though Alertmanager is healthy. This happens because Prometheus cannot access the Alertmanager reloader-web port, due to lack of an AuthorizationPolicy port entry.

Description

With the addition of the reloader-web port to the AlertManager ServiceMonitor configuration, the AlertmanagerClusterDown alert fires, even when there is a healthy Alertmanger instance. This happens because there are now two service ports to monitor, so the up metric is only reporting 1/2 "alertmanagers" healthy.

The metric scraping failure happens because the Alertmanager port (9093) has an AuthorizationPolicy which allows for Prometheus scraping, however, it does not have an entry to allow scraping of the reloader-web port (8080).

The reloader-web port (8080) should be added to the Alertmanager AuthorizationPolicy to allow for scraping both service ports.

BigBang Version

What version of BigBang were you running? 2.14.0