Investigate Promtail capturing system logs and viewing them in Grafana

Older grafana documentation references these configuration options for promtail. See if we can get this support working in the latest chart version and have them display in Grafana.

https://grafana.com/docs/loki/v2.4.x/installation/helm/#run-promtail-with-syslog-support

• Promtail successfully sending system logs to Loki/Grafana

added fluentbit kindenhancement loki priority6 promtail teamXForce labels

set weight to 3

added 1 deleted label

removed 1 deleted label

added 1 deleted label

removed 1 deleted label

added 1 deleted label

removed 1 deleted label

added 1 deleted label

removed 1 deleted label

added 1 deleted label

removed 1 deleted label

added 1 deleted label

removed 1 deleted label

changed iteration to Big Bang Iterations Jan 10, 2023 - Jan 23, 2023

assigned to @ryan.j.garcia

added statusdoing label

Tested with the following config:

promtail:
  values:
    config:
      snippets:
        extraScrapeConfigs: |
          - job_name: systemd-journal
            journal:
              labels:
                cluster: dogfood
                job: default/systemd-journal
              path: /var/log/messages
            relabel_configs:
            - source_labels:
              - __journal__systemd_unit
              target_label: systemd_unit
            - source_labels:
              - __journal__hostname
              target_label: nodename
            - source_labels:
              - __journal_syslog_identifier
              target_label: syslog_identifier
    extraVolumes:
      - name: journal
        hostPath:
          path: /var/log/messages
      - name: runjournal
        hostPath:
          path: /var/run/journal
      - name: machine-id
        hostPath:
          path: /etc/machine-id
    extraVolumeMounts:
      - name: journal
        mountPath: /var/log/messages
        readOnly: true
      - name: runjournal
        mountPath: /var/run/journal
        readOnly: true
      - name: machine-id
        mountPath: /etc/machine-id
        readOnly: true

Our RKE2 cluster nodes don't have a /var/log/journal and mounting /var/log/messages leads to:

error="failed to make journal target manager: creating journal reader: failed to open journal in directory \"/var/log/messages\": not a directory"

removed iteration Big Bang Iterations Jan 10, 2023 - Jan 23, 2023

changed iteration to Big Bang Iterations Jan 24, 2023 - Feb 6, 2023

mentioned in issue #1414 (closed)

set weight to 2

Look like this config works for RKE2 clusters, but nodename/hostname nor service name is able to be pulled out into a searchable field reliably

promtail:
  values:
    config:
      snippets:
        extraScrapeConfigs: |
          - job_name: systemd-journal
            static_configs:
              - targets: [localhost]
                labels:
                  job: varlogs
                  __path__: /var/log/*
            relabel_configs:
            - source_labels:
              - __journal_systemd_unit
              target_label: systemd_unit
            - source_labels:
              - __journal_hostname
              target_label: nodename
            - source_labels:
              - __journal_syslog_identifier
              target_label: syslog_identifier
    extraVolumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: journal
        hostPath:
          path: /var/run/journal
      - name: machine-id
        hostPath:
          path: /etc/machine-id
    extraVolumeMounts:
      - name: varlog
        mountPath: /var/log
        readOnly: true
      - name: journal
        mountPath: /var/run/journal
        readOnly: true
      - name: machine-id
        mountPath: /etc/machine-id
        readOnly: true

marked the checklist item Promtail successfully sending system logs to Loki/Grafana as completed

With the config pasted in the comment below, /var/log/* (messages|lastlog|secure) are successfully being fed in and searchable by filename & hostname:

promtail:
  values:
    extraEnv:
      - name: NODE_HOSTNAME
        valueFrom:
          fieldRef:
            fieldPath: spec.nodeName
    extraArgs:
      - -config.expand-env=true
    config:
      snippets:
        extraScrapeConfigs: |
          - job_name: systemd-messages
            static_configs:
              - targets: [localhost]
                labels:
                  job: varlogs
                  host: "${NODE_HOSTNAME}"
                  __path__: /var/log/*
            relabel_configs:
            - source_labels:
              - __journal_systemd_unit
              target_label: systemd_unit
            - source_labels:
              - __journal_hostname
              target_label: nodename
            - source_labels:
              - __journal_syslog_identifier
              target_label: syslog_identifier
    extraVolumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: machine-id
        hostPath:
          path: /etc/machine-id
    extraVolumeMounts:
      - name: varlog
        mountPath: /var/log
        readOnly: true
      - name: machine-id
        mountPath: /etc/machine-id
        readOnly: true

Updating description to track flb configuration in separate issue

changed the description

added statusreview label and removed statusdoing label

mentioned in merge request !2478 (merged)

closed with merge request !2478 (merged)

mentioned in commit 53b07b36

@ryan.j.garcia I know this is a year old but could use some help your last config I added but do I need to do anything else can you point me to a resource to use for both systemd and syslog set up in big bang promtail.