Fluent-Bit Log Transmission Stability
Bug
Description
Once starting up Fluent-bit pods send in log data just fine, but transmission stops after several minutes until the pod(s) is restarted. Log ingestion catches up until the issue starts all over again.
BigBang Version
1.X
Notes
Periodic events in the log file showing that input processing has paused because of a buffer limit being reached:
[ warn] [input] tail.0 paused (mem buf overlimit)
When DEBUG Log_Level is enabled we see issues with the Merge_Log setting for the Kubernetes FILTER:
[filter:kubernetes:kubernetes.0] could not merge JSON log as requested
Also seeing several "dismissed" debug notifications for log files:
[debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/gitlab-webservice-default-___.log
Fluent-bit docs recommend for best reliability in production to use both memory and storage buffering https://docs.fluentbit.io/manual/v/1.6/administration/buffering-and-storage