UNCLASSIFIED - NO CUI

Skip to content

Big Bang 2.13.1 Loki AWS S3 issue

Bug

Description

After deploying BB v2.13.1 on EKS using AWS S3 for Loki storage, when trying to view logs through Grafana using the Loki datasource, no logs are shown but instead get errors (in the logs and UI).

BigBang Version

What version of BigBang were you running?

v2.13.1

Using this values file for Loki:

loki:
  strategy: "scalable"
  objectStorage:
    endpoint: https://s3.amazonaws.com
    region: "<<aws-region>>"
    type: s3
    bucketNames:
      chunks: "<<loki-s3-bucket>>"
      ruler: "<<loki-s3-bucket>>"
      admin: "<<loki-s3-bucket>>"
  values:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: "<<loki-s3-role-arm>>"
    loki:
      storage:
        s3:
          endpoint: ""

Errors in Grafana UI

Issues seen

grafana-loki-dash-error.png

grafana-explore-loki-error.png

Errors in Loki logs

logging-loki-write pods:

level=warn ts=2023-10-27T20:39:47.602383829Z caller=logging.go:123 traceID=7e89a72a7d5e9030 orgID=fake msg="POST /loki/api/v1/push (500) 6.182627ms Response: \"at least 1 live replicas required, could only find 0 - unhealthy instances: 192.168.35.215:9095\\n\" ws: false; Content-Length: 213781; Content-Type: application/x-protobuf; User-Agent: promtail/; X-B3-Parentspanid: 8d6135623cf2ce56; X-B3-Sampled: 1; X-B3-Spanid: 289273db659a04cb; X-B3-Traceid: 305964370f5a09d68d6135623cf2ce56; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/logging/sa/logging-loki;Hash=82ae79ad4f8c757f38e9f811fa0d28545a5f59ec33bd00f77a38b31c84309517;Subject=\"\";URI=spiffe://cluster.local/ns/promtail/sa/promtail-promtail; X-Forwarded-Proto: http; X-Request-Id: 88c45832-bee5-9032-a80c-66404e24cf48; "

logging-loki-read pods:

level=error ts=2023-10-27T20:16:51.059074597Z caller=retry.go:73 org_id=fake traceID=46634fe5db594df3 msg="error processing request" try=4 query="{namespace=\"kube-system\", pod=~\".*\"}" err="rpc error: code = Code(500) desc = too many unhealthy instances in the ring\n"
ts=2023-10-27T20:16:51.059125247Z caller=spanlogger.go:86 middleware=QueryShard.astMapperware org_id=fake traceID=46634fe5db594df3 org_id=fake traceID=46634fe5db594df3 level=warn msg="failed mapping AST" err="rpc error: code = Code(500) desc = too many unhealthy instances in the ring\n" query="{namespace=\"kube-system\", pod=~\".*\"} |~ \"\""
level=error ts=2023-10-27T20:16:51.099816445Z caller=retry.go:73 org_id=fake traceID=46634fe5db594df3 msg="error processing request" try=4 query="{namespace=\"kube-system\", pod=~\".*\"}" err="rpc error: code = Code(500) desc = too many unhealthy instances in the ring\n"
ts=2023-10-27T20:16:51.099871859Z caller=spanlogger.go:86 middleware=QueryShard.astMapperware org_id=fake traceID=46634fe5db594df3 org_id=fake traceID=46634fe5db594df3 level=warn msg="failed mapping AST" err="rpc error: code = Code(500) desc = too many unhealthy instances in the ring\n" query="{namespace=\"kube-system\", pod=~\".*\"} |~ \"\""

UNCLASSIFIED - NO CUI