Resolve "Too many labels added for Loki 3.0"
General MR
Summary
We need to update fluentbit to drop certain labels because loki v3.0.0 has a new default limit of 15 labels. The implementation closely tracks promtail!79 (merged) and going forward the two implementations should probably be kept in sync.
Relevant logs/screenshots
Erroneous log messages look like this within loki:
loki level=error ts=2024-05-02T18:52:25.887436901Z caller=manager.go:49 component=distributor path=write msg="write operation failed" details="entry for stream '{app=\"prometheus\", app_kubernetes_io_instance=\"monitoring-monitoring-kube-prometheus\", app_kubernetes_io_managed_by=\"prometheus-operator\", app_kubernetes_io_name=\"prometheus\", app_kubernetes_io_version=\"2.51.1\", apps_kubernetes_io_pod_index=\"0\", container=\"istio-proxy\", controller_revision_hash=\"prometheus-monitoring-monitoring-kube-prometheus-6f4f7c789b\", job=\"fluentbit\", namespace=\"monitoring\", node_name=\"k3d-k3s-default-agent-1\", operator_prometheus_io_name=\"monitoring-monitoring-kube-prometheus\", operator_prometheus_io_shard=\"0\", pod=\"prometheus-monitoring-monitoring-kube-prometheus-0\", prometheus=\"monitoring-monitoring-kube-prometheus\", statefulset_kubernetes_io_pod_name=\"prometheus-monitoring-monitoring-kube-prometheus-0\"}' has 16 label names; limit 15" org_id=fake
Linked Issue
Upgrade Notices
N/A
Closes #104 (closed)