Possibly remove k8s label relabel scrapeconfig action?

added teamObservability label

added community-contribution label

added triage-kind label

added triage-priority label

added statusready-to-work label

changed iteration to Big Bang Iterations Jun 11, 2024 - Jun 24, 2024

set weight to 3

assigned to @jimmyungerman

added kindenhancement priority4 labels

removed triage-priority label

removed triage-kind label

This could be a good issue to set a defined set of "BigBang Labels" we want to add as right now we're just adding every kubernetes label imaginable which is resulting in a lot of noise.

Test scrape config provided by @michaelabixler:

        scrapeConfigs: |
          # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
          - job_name: kubernetes-pods
            pipeline_stages:
              - cri: {}
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              - source_labels:
                  - __meta_kubernetes_pod_node_name
                target_label: host
              - action: replace
                replacement: $1
                separator: /
                source_labels:
                  - __meta_kubernetes_namespace
                  - __meta_kubernetes_pod_name
                target_label: job
              - action: replace
                source_labels:
                  - __meta_kubernetes_namespace
                target_label: namespace
              - replacement: /var/log/pods/*$1/*.log
                separator: /
                source_labels:
                  - __meta_kubernetes_pod_uid
                  - __meta_kubernetes_pod_container_name
                target_label: __path__
          - job_name: systemd-journal
            journal:
              json: true
              max_age: 6h
              path: /var/log/journal
            relabel_configs:
            - source_labels:
              - __journal_systemd_unit
              target_label: systemd_unit
            - source_labels:
              - __journal_hostname
              target_label: systemd_hostname
            - source_labels:
              - __journal_syslog_identifier
              target_label: syslog_identifier

removed statusready-to-work label

added promtail statusdoing labels

added statusready-to-work label and removed statusdoing label

added statusdoing label and removed statusready-to-work label

removed iteration Big Bang Iterations Jun 11, 2024 - Jun 24, 2024

changed iteration to Big Bang Iterations Jun 25, 2024 - Jul 8, 2024

changed milestone to %2.31.0

created branch 52-possibly-remove-k8s-label-relabel-scrapeconfig-action to address this issue

mentioned in merge request !83 (merged)

Github Comment Mirrored Here

After reading the Best Practices doc, it appears we want the absolute least amount of labels as possible so we don't have to chunk our queries.

To do this, I'm formally proposing the following default labels config:

        pipelineStages:
          - cri: {}
          - labeldrop:
            - filename
        common:
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_node_name
            target_label: node_name
          - action: replace
            source_labels:
              - __meta_kubernetes_namespace
            target_label: namespace
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_name
            target_label: pod
          - action: replace
            source_labels:
              - __meta_kubernetes_pod_container_name
            target_label: container
          - action: replace
            replacement: /var/log/pods/*$1/*.log
            separator: /
            source_labels:
              - __meta_kubernetes_pod_uid
              - __meta_kubernetes_pod_container_name
            target_label: __path__

        scrapeConfigs: |
          - job_name: kubernetes-pods
            pipeline_stages:
              {{- toYaml .Values.config.snippets.pipelineStages | nindent 4 }}
            kubernetes_sd_configs:
              - role: pod
            relabel_configs:
              {{- if .Values.config.snippets.addScrapeJobLabel }}
              - replacement: kubernetes-pods
                target_label: scrape_job
              {{- end }}
              {{- toYaml .Values.config.snippets.common | nindent 4 }}
              {{- with .Values.config.snippets.extraRelabelConfigs }}
              {{- toYaml . | nindent 4 }}
              {{- end }}