Too many labels added for Loki 3.0

Bug

Description

Loki 3.0 requires that log streams have a maximum of 15 labels, but the current logstreams for Loki contain up to 17.

BigBang Version

bigbang 2.25.0

changed milestone to %2.27.0

added kindbug priority2 promtail statusready-to-work teamObservability labels

set weight to 2

changed iteration to Big Bang Iterations Apr 30, 2024 - May 13, 2024

mentioned in issue loki#102 (closed)

mentioned in merge request loki!166 (merged)

created branch 48-too-many-labels-added-for-loki-3-0 to address this issue

mentioned in merge request !79 (merged)

Right now, these are the 17 labels that promtail adds to every log coming from Loki:

These labels are solely used for querying the logs, and don't actually change anything with the underlying resources (e.g. removing a label in promtail only removes it when sending the log data to loki, it doesn't remove it from the actual loki deployment object).

Knowing this, I think our best course of action is to add a labeldrop job to our promtail scrapes for the loki streams and drop all of the istio labels from it.

@chris.oconnell @michaelmartin tagging anchors for their feedback.

adding this block to our scrapeConfig drops all istio labels from the loki logs successfully after testing.

- action: labeldrop
regex: "service_istio.+|security_istio_io_tlsMode"

This should probably be added at the umbrella level where we can conditionally set it if the Loki version is at least 3.0.

I left a comment on loki relating to this here

assigned to @jimmyungerman

added statusdoing label and removed statusready-to-work label

changed weight to 3 from 2

mentioned in merge request big-bang/bigbang!4267 (merged)

closed with merge request big-bang/bigbang!4267 (merged)

mentioned in commit big-bang/bigbang@85203560

marked this issue as related to fluentbit#104 (closed)

marked this issue as blocking loki#102 (closed)