Bug: Grafana dashboards may not be compatible with Thanos

Bug

Description

A bug has been identified impacting Big Bang packages with Grafana dashboards with Prometheus datasources when Thanos is enabled ("datasource": "Prometheus").

Expected Behavior

When Thanos is enabled in a deployment, it supplants the Prometheus datasource in Grafana with a Thanos datasource. Because both of these datasources are of type prometheus, they should be interchangeable. Dashboards built from Prometheus datasources should automatically pull from the Thanos datasource.

Observed Behavior

When Thanos is enabled, dashboards where a datasource is defined as "datasource": "Prometheus" are unable to read from the Thanos datasource and display errors. This can be temporarily circumvented by renaming the datasource to Prometheus.

To Reproduce

• Deploy this package as recommended and additionally enable monitoring, grafana, and thanos.
• In your browser, navigate to Grafana > Dashboards and search for the dashboards defined in this package's /chart/dashboards
• Panels will display error icons and no data. If clicked, the error message will indicated that the dashboard cannot find the Prometheus datasource.

Existing Solutions

Every dashboard is slightly different, but the solutions utilized in these packages may also apply to this one:

loki#92 (closed)

istio-controlplane#147 (closed)

Acceptance Criteria

• Dashboards display correctly with only grafana and monitoring enabled
• Dashboards display correctly with grafana, monitoring, and thanos enabled
• Dashboards display correctly when adding thanos to an existing deployment where grafana and monitoring are already enabled