Troubleshoot Loki 5.21.0 v2.9.0 Package

Last week 5.21.0-bb.0/1 of Loki was merged in. Tested fine locally but was never passing in CI.

Testing early this week now I never see any of the loki pods stay up healthy (not crashing) and logs are showing :

read: connection reset by peer talking to memberlist :7946

level=error ts=2023-09-19T17:29:21.484981718Z caller=gateway_client.go:348 msg="client do failed for instance 127.0.0.1:9095" err="rpc error: code = Canceled desc = context canceled"

level=warn ts=2023-09-19T17:27:12.341282358Z caller=scheduler_processor.go:98 msg="error contacting scheduler" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.42.2.27:9095: connect: connection refused\"" addr=10.42.2.27:9095

Troubleshoot and see if any package or BigBang level config or syntax changes are required to get loki happy and healthy again.

Review Upstream documentation https://grafana.com/docs/loki/latest/configure/ and 2.9.0 release notes https://github.com/grafana/loki/blob/release-2.9.x/CHANGELOG.md#290-2023-09-06