Prometheus scrapes completed pods

Currently Prometheus will attempt to scrape all endpoints regardless of the pod status. This surfaced with Anchore, which runs several jobs with istio sidecars. Prometheus will attempt to scrape metrics from the sidecars even after the jobs have completed. This can be seen in the Prometheus targets list when deploying istio + anchore + sso or an upgrade to trigger jobs.

A brief investigation yielded this PR which seems to indicate Prometheus has knowledge of the pods' status/phase - https://github.com/prometheus/prometheus/pull/4824

What we want to do is somehow exclude resources from the targets when they are in a non-Running phase. There may be a way to do this via config in the helm chart for monitoring, or possibly a setting in each servicemonitor? Don't be afraid to reach out/post an issue upstream with the Prometheus community to find out what options we might have.

There could be a number of outcomes from this.

AC (if feature is available to do this):

• Prometheus/ServiceMonitors updated to only scrape Running pods

AC (if feature is not available):

• Upstream issue posted as feature request to be able to only scrape Running pods, new BB issue opened to track that one