Elasticsearch K8s Labels bug fix

General MR

Summary

Change in logic in fluentbit template to allow elasticsearch to consume kuberenetes labels for searchability with KQL within es/kibana. This will allow users in elasticsearch who are using fluentbit to be able to search by labels attached pods.

Relevant logs/screenshots

With only fluentbit and elasticsearch installed(note being searched by app.kuberenetes.io/name:

fluent-bit.conf with only fluentbit and elasticsearch enabled:

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    # -- Setting up storage buffer on filesystem and slightly upping backlog mem_limit value.
    storage.path /var/log/flb-storage/
    storage.sync normal
    storage.backlog.mem_limit 15M
    Health_Check On

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    # -- Excluding fluentbit logs from sending to ECK, along with gatekeeper-audit logs which are shipped by clusterAuditor.
    Exclude_Path /var/log/containers/*fluent*.log
    Parser containerd
    Tag kube.*
    Mem_Buf_Limit 50MB
    Skip_Long_Lines On
    storage.type filesystem

[INPUT]
    Name systemd
    Tag host.*
    Systemd_Filter _SYSTEMD_UNIT=kubelet.service
    Read_From_Tail On
    storage.type filesystem

[FILTER]
    Name kubernetes
    Match kube.*
    Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log On
    Merge_Log_Key log_processed
    K8S-Logging.Parser On
    K8S-Logging.Exclude Off
    Buffer_Size 1M
[FILTER]
    Alias         lua.add
    Name          lua
    Match         kube.*
    script        /fluent-bit/scripts/add_labels.lua
    call          add_labels

[OUTPUT]
    Name                     es
    Match                    kube.*
    Host                     logging-ek-es-http.logging
    HTTP_User                elastic
    HTTP_Passwd              ${FLUENT_ELASTICSEARCH_PASSWORD}
    Logstash_Format          On
    Suppress_Type_Name       On
    Retry_Limit              False
    Replace_Dots             On
    tls                      On
    tls.verify               On
    tls.ca_file              /etc/elasticsearch/certs/ca.crt
    storage.total_limit_size 10G
[OUTPUT]
    Name                     es
    Match                    host.*
    Host                     logging-ek-es-http.logging
    HTTP_User                elastic
    HTTP_Passwd              ${FLUENT_ELASTICSEARCH_PASSWORD}
    Logstash_Format          On
    Suppress_Type_Name       On
    Logstash_Prefix          node
    Retry_Limit              False
    tls                      On
    tls.verify               On
    tls.ca_file              /etc/elasticsearch/certs/ca.crt
    storage.total_limit_size 10G

With fluentbit, loki and elasticsearch installed:

fluent-bit.conf with loki installed as well:

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level info
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server On
    HTTP_Listen 0.0.0.0
    HTTP_Port 2020
    # -- Setting up storage buffer on filesystem and slightly upping backlog mem_limit value.
    storage.path /var/log/flb-storage/
    storage.sync normal
    storage.backlog.mem_limit 15M
    Health_Check On

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    # -- Excluding fluentbit logs from sending to ECK, along with gatekeeper-audit logs which are shipped by clusterAuditor.
    Exclude_Path /var/log/containers/*fluent*.log
    Parser containerd
    Tag kube.*
    Mem_Buf_Limit 50MB
    Skip_Long_Lines On
    storage.type filesystem

[INPUT]
    Name systemd
    Tag host.*
    Systemd_Filter _SYSTEMD_UNIT=kubelet.service
    Read_From_Tail On
    storage.type filesystem

[FILTER]
    Name kubernetes
    Match kube.*
    Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log On
    Merge_Log_Key log_processed
    K8S-Logging.Parser On
    K8S-Logging.Exclude Off
    Buffer_Size 1M
[FILTER]
    Alias         lua.add
    Name          lua
    Match         kube.*
    script        /fluent-bit/scripts/add_labels.lua
    call          add_labels
[FILTER]
    Name          lua
    Match         kube.*
    script        /fluent-bit/scripts/remove_labels.lua
    call          remove_labels

[OUTPUT]
    Name                     es
    Match                    kube.*
    Host                     logging-ek-es-http.logging
    HTTP_User                elastic
    HTTP_Passwd              ${FLUENT_ELASTICSEARCH_PASSWORD}
    Logstash_Format          On
    Suppress_Type_Name       On
    Retry_Limit              False
    Replace_Dots             On
    tls                      On
    tls.verify               On
    tls.ca_file              /etc/elasticsearch/certs/ca.crt
    storage.total_limit_size 10G
[OUTPUT]
    Name                     es
    Match                    host.*
    Host                     logging-ek-es-http.logging
    HTTP_User                elastic
    HTTP_Passwd              ${FLUENT_ELASTICSEARCH_PASSWORD}
    Logstash_Format          On
    Suppress_Type_Name       On
    Logstash_Prefix          node
    Retry_Limit              False
    tls                      On
    tls.verify               On
    tls.ca_file              /etc/elasticsearch/certs/ca.crt
    storage.total_limit_size 10G
[OUTPUT]
    name                   loki
    match                  kube.*
    labels                 job=fluentbit, container=$kubernetes['container_name'], pod=$kubernetes['pod_name'], namespace=$kubernetes['namespace_name'], node_name=$kubernetes['host']
    host                   logging-loki.logging
    port                   3100
    auto_kubernetes_labels on
    Retry_Limit            False
    tls                    Off
    storage.total_limit_size 10G
[OUTPUT]
    name                   loki
    match                  host.*
    labels                 job=fluentbit, container=$kubernetes['container_name'], pod=$kubernetes['pod_name'], namespace=$kubernetes['namespace_name'], node_name=$kubernetes['host']
    host                   logging-loki.logging
    port                   3100
    auto_kubernetes_labels on
    Retry_Limit            False
    tls                    Off
    storage.total_limit_size 10G

NOTE: remove_labels job only installed with Loki, because Loki is only able to accept logs with a set amount of labels.

Linked Issue

Closes (big-bang/product/packages/fluentbit#213 (closed))

Upgrade Notices

N/A