Bug: Grafana dashboards may not be compatible with Thanos
Bug
Description
A bug has been identified impacting Big Bang packages with Grafana dashboards with Prometheus datasources when Thanos is enabled ("datasource": "Prometheus"
).
Expected Behavior
When Thanos is enabled in a deployment, it supplants the Prometheus
datasource in Grafana with a Thanos
datasource. Because both of these datasources are of type prometheus
, they should be interchangeable. Dashboards built from Prometheus
datasources should automatically pull from the Thanos
datasource.
Observed Behavior
When Thanos is enabled, dashboards where a datasource is defined as "datasource": "Prometheus"
are unable to read from the Thanos
datasource and display errors. This can be temporarily circumvented by renaming the datasource to Prometheus
.
To Reproduce
- Deploy this package as recommended and additionally enable
monitoring
,grafana
, andthanos
. - In your browser, navigate to
Grafana > Dashboards
and search for the dashboards defined in this package's/chart/dashboards
- Panels will display error icons and no data. If clicked, the error message will indicated that the dashboard cannot find the Prometheus datasource.
Existing Solutions
Every dashboard is slightly different, but the solutions utilized in these packages may also apply to this one:
istio-controlplane#147 (closed)
Acceptance Criteria
-
Dashboards display correctly with only grafana
andmonitoring
enabled -
Dashboards display correctly with grafana
,monitoring
, andthanos
enabled -
Dashboards display correctly when adding thanos
to an existing deployment wheregrafana
andmonitoring
are already enabled