thanos unable to connect to prometheus sidecar
Our thanos Query is able to connect to the storegateway in the thanos namespace, but not the sidecars within the monitoring namespace's prometheus pods. We have turned on thanosService as well as dnsDiscovery as the bitnami chart instructs, but we are still getting a connection issue. We initially thought it was a clusterRole issue where the rbac.create = true would create the permissions for query to reach out, however this doesn't generate a cluster role (looking at the templates, it looks like it will only create a cluster role if pspEnabled is also turned on, but that is deprecated so it is a moot avenue.) Have you heard similar stories? Do you have any potential remedies and/ or suggestions to get the query pod to successfully connect to the thanos-sidecars? Appreciate your time all! If there is a better channel for this question, please let me know and I can move it that way.
level=warn component=endpointset msg="update of endpoint failed" err="getting metadata: rpc error: code = Unavailable desc = error reading from │
│ server: EOF"