Upgrading from Big Bang 3.19.0 to 3.20.0 can break Grafana datasource access to Prometheus when monitoring.sso.enabled=true, even if Istio hardened settings are not explicitly enabled.

After upgrade, Grafana dashboards fail to query Prometheus and Grafana logs show:

status=403 body="RBAC: access denied"

This is consistent with an Istio AuthorizationPolicy denial

This is a regression introduced by the monitoring bb-common migration in 3.20.0.

In 3.19.0:

monitoring rendered an explicit monitoring-prometheus-authz-policy that policy allowed requests from the monitoring namespace to Prometheus this covered Grafana -> Prometheus traffic

In 3.20.0:

the old explicit Prometheus authz policy was removed monitoring now relies on bb-common authz generation but bb-common authz generation is only enabled when hardened-derived authz is enabled at the same time, when monitoring.sso.enabled=true, Prometheus still gets the protect: keycloak label and is selected by shared-monitoring-authz-policy

That shared policy allows Prometheus and Alertmanager service accounts, but does not allow Grafana. So the upgrade leaves an active SSO-related authz policy in effect while removing the old same-namespace allow that previously let Grafana query Prometheus.

Impact

Clusters using:

monitoring.sso.enabled=true

authservice for Prometheus / Alertmanager

Grafana querying in-cluster Prometheus

may see Grafana datasource failures immediately after upgrading from 3.19.0 to 3.20.0.

Workaround

A manual AuthorizationPolicy allowing Grafana / monitoring namespace traffic to Prometheus restores functionality

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-grafana-to-prometheus
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: prometheus
  action: ALLOW
  rules:
  - from:
    - source:
        namespaces:
        - monitoring