Loki not writing to S3 when configured for scalable
We have loki configured for scalable, but we are not seeing anything writing to s3. We see no errors in the write pods or the backend pods. Fluentbit says it is is flushing backlog chunks. when trying to get labels in grafana, it finds no labels. All three s3 buckets show empty.
When we switch loki to monolith, it works fine.
I checked the permissions on the nodes and we are able to do aws s3 cp commands and rm commands, so the pods should have all capabilities.
here is our deployments and versions:
[root@ip-192-168-89-50 ~]# flux -n bigbang get hr NAME REVISION SUSPENDED READY MESSAGE bigbang 2.13.1 False True Release reconciliation succeeded fluentbit 0.37.0-bb.2 False True Release reconciliation succeeded grafana 6.58.9-bb.4 False True Release reconciliation succeeded istio 1.19.0-bb.2 False True Release reconciliation succeeded istio-operator 1.19.0-bb.1 False True Release reconciliation succeeded loki 5.23.1-bb.1 False True Release reconciliation succeeded metrics-server 3.10.0-bb.2 False True Release reconciliation succeeded monitoring 51.1.0-bb.2 False True Release reconciliation succeeded
Could you point us into a direction to help troubleshoot this?