Fluent-Bit Log Transmission Stability

Bug

Description

Once starting up Fluent-bit pods send in log data just fine, but transmission stops after several minutes until the pod(s) is restarted. Log ingestion catches up until the issue starts all over again.

BigBang Version

1.X

Notes

Periodic events in the log file showing that input processing has paused because of a buffer limit being reached: [ warn] [input] tail.0 paused (mem buf overlimit)

When DEBUG Log_Level is enabled we see issues with the Merge_Log setting for the Kubernetes FILTER: [filter:kubernetes:kubernetes.0] could not merge JSON log as requested

Also seeing several "dismissed" debug notifications for log files: [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/gitlab-webservice-default-___.log

Fluent-bit docs recommend for best reliability in production to use both memory and storage buffering https://docs.fluentbit.io/manual/v/1.6/administration/buffering-and-storage