PLG logging stack not showing logs in BB 2.12.0

Bug

BigBang Version

What version of BigBang were you running? 2.12.0

Description

As of Big Bang 2.12.0 the PLG logging stack with Loki strategy: scalable and s3 compatible storage is no longer displaying logs in Grafana. This problem can be reproduced in AWS Govcloud with RKE2 and with remote k3d dev environment. The Loki reader logs are

kubectl logs -n logging logging-loki-read-7854d6558c-9p6tr
level=error ts=2023-10-11T13:11:36.669737417Z caller=retry.go:73 org_id=fake traceID=69c56015ad042f77 msg="error processing request" try=4 query="{namespace=\"monitoring\", pod=~\".*\"}" err="rpc error: code = Code(500) desc = index gateway get ring: too many unhealthy instances in the ring\n"
ts=2023-10-11T13:11:36.669801418Z caller=spanlogger.go:86 middleware=QueryShard.astMapperware org_id=fake traceID=69c56015ad042f77 org_id=fake traceID=69c56015ad042f77 level=warn msg="failed mapping AST" err="rpc error: code = Code(500) desc = index gateway get ring: too many unhealthy instances in the ring\n" query="{namespace=\"monitoring\", pod=~\".*\"} |~ \"\""
level=warn ts=2023-10-11T13:11:36.669866499Z caller=logging.go:123 traceID=69c56015ad042f77 orgID=fake msg="GET /loki/api/v1/query_range?direction=backward&end=1697029896134000000&limit=1000&query=%7Bnamespace%3D%22monitoring%22%2C+pod%3D~%22.%2A%22%7D+%7C~+%22%22&start=1697029596133000000&step=200ms (500) 105.259407ms Response: \"index gateway get ring: too many unhealthy instances in the ring\\n\" ws: false; Accept-Encoding: gzip; User-Agent: Grafana/10.0.3; X-Dashboard-Uid: liz0yRCZz; X-Datasource-Uid: P8E80F9AEF21F6940; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/logging/sa/logging-loki;Hash=fbb9bb8fb6a9f5eee9c384863ccf26663a3b727884acfee61ab8708f35d0242d;Subject=\"\";URI=spiffe://cluster.local/ns/monitoring/sa/monitoring-grafana; X-Forwarded-Proto: http; X-Grafana-Org-Id: 1; X-Panel-Id: 2; X-Request-Id: 77ebe5a3-6f9d-4be9-a599-c1a5769a4578; "
level=error ts=2023-10-11T13:11:36.670500237Z caller=retry.go:73 org_id=fake traceID=71a194ba8d7c37e2 msg="error processing request" try=4 query="{namespace=\"monitoring\", pod=~\".*\"}" err="rpc error: code = Code(500) desc = index gateway get ring: too many unhealthy instances in the ring\n"
ts=2023-10-11T13:11:36.670570448Z caller=spanlogger.go:86 middleware=QueryShard.astMapperware org_id=fake traceID=71a194ba8d7c37e2 org_id=fake traceID=71a194ba8d7c37e2 level=warn msg="failed mapping AST" err="rpc error: code = Code(500) desc = index gateway get ring: too many unhealthy instances in the ring\n" query="sum(count_over_time({namespace=\"monitoring\", pod=~\".*\"} |~ \"\"[200ms]))"
level=warn ts=2023-10-11T13:11:36.67064943Z caller=logging.go:123 traceID=71a194ba8d7c37e2 orgID=fake msg="GET /loki/api/v1/query_range?direction=backward&end=1697029896333000000&limit=1000&query=sum%28count_over_time%28%7Bnamespace%3D%22monitoring%22%2C+pod%3D~%22.%2A%22%7D+%7C~+%22%22%5B200ms%5D%29%29&start=1697029596333000000&step=200ms (500) 103.98043ms Response: \"index gateway get ring: too many unhealthy instances in the ring\\n\" ws: false; Accept-Encoding: gzip; User-Agent: Grafana/10.0.3; X-Dashboard-Uid: liz0yRCZz; X-Datasource-Uid: P8E80F9AEF21F6940; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/logging/sa/logging-loki;Hash=fbb9bb8fb6a9f5eee9c384863ccf26663a3b727884acfee61ab8708f35d0242d;Subject=\"\";URI=spiffe://cluster.local/ns/monitoring/sa/monitoring-grafana; X-Forwarded-Proto: http; X-Grafana-Org-Id: 1; X-Panel-Id: 6; X-Request-Id: 2223e99b-7729-448a-ba9a-9aba34e8e2fc; "

How to reproduce

1. git clone bigbang repo

git clone https://repo1.dso.mil/big-bang/bigbang.git

2. Checkout tag 2.12.0

cd bigbang
git checkout 2.12.0

3. Launch remote k3d dev environment with any appropriate environment variables set.

./docs/assets/scripts/developer/k3d-dev.sh -a

4. Install flux to k8s cluster

./scripts/install_flux.sh -u username -p password

5. Deploy bigbang using this values override file.
   logging-monitoring-values.yaml
   Provide your own registry-values.yaml.

helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../overrides/logging-monitoring-values.yaml -f ../overrides/registry-values.yaml -f ./chart/ingress-certs.yaml

6. Login to https://grafana.bigbang.dev with default credentials and navigate to
   Home > Dashboards > Loki Dashboard quick search
   There are no logs are displayed.
   

Additional comments

Logs display in Grafana if Big Bang release tag 2.11.1 is used. There is either a bug or some new undocumented configuration that is needed to make 2.12.0 work.