Default configuration is broken due to replication_factor mismatch.

In the default configuration (using single binary), promtail is unable to ship logs to loki.

The error in the loki logs is:

level=warn ts=2024-01-18T19:50:26.673848511Z caller=logging.go:123 traceID=41ec0e0a0144475d orgID=fake msg="POST /loki/api/v1/push (500) 3.209161ms Response: \"at least 2 live replicas required, could only find 1\\n\" ws: false; Content-Length: 183217; Content-Type: application/x-protobuf; User-Agent: promtail/; X-B3-Parentspanid: 6f245018048a41bb; X-B3-Sampled: 1; X-B3

After reading this issue on GitHub, I realized this is caused by a replica count mismatch between the StatefulSet and the loki config. In its default configuration, the StatefulSet has replicas: 1, while the loki config expects three replicas with replication_factor: 3.

I fixed the issue by setting the loki config's replication_factor: 1 with the following yaml override:

loki:
  values:
    loki:
      commonConfig:
        # needed if there is only one replica (the default)
        replication_factor: 1

It seems like the two settings should match so the defaults aren't broken. Likely, the replication_factor should be automatically set to 1 when using single binary.

We are using 5.31.0-bb.10 - I'm unsure of when this bug was introduced.