Add optional additional output for fluentbit for P1 value streams enterprise security logging
Feature Request
Why
As decided by Jedi Order and Cyber leadership, the P1 logging pipeline will receive logs from P1 Value Stream fluentbits into a fluentd instance deployed in all P1 VS prod clusters. This depends on adding an optional output from fluentbit to forward logs to the vs fluentd instance. Project details can be found here.
This is to request Big Bang add an output to the BB Core Fluentbit configs to allow initial ingest to the vs fluentd instances.
Proposed Solution
Please add the following lines to the [OUTPUT] portion of the fluentbit
# # output from fluentbit to the local cluster FluentD as part of P1 Logging Pipeline
# # optional to comment out the other outputs from Fluentbit as the local cluster FluentD
# # forwards logs to Elasticsearch with dynamic indexes
# [OUTPUT]
# Name forward
# Match *
# # -- Points to local FluentD as part of P1 Logging Pipeline, for use by P1 Value Streams only
# # -- Contact P1 SecOps Team for further coordination
# Host fluentd.{{ .Values.domain }}
# Port 24224
# Shared_Key XXXXXXXXXXXX # Would be better as an environment variable
# # create shared key string and use to update FluentD config
# tls off
# tls.verify off
# Retry_Limit False
This does not change fluentbit's behavior, it just adds another output. This could be used to change fluentbit's behavior if decided as the logging pipeline does also have an option of outputting to Elasticsearch with indexes and values which are searchable (unlike the current setup from fluenbit to Elasticsearch) though that's an option.
Additionally, this will only be used by P1 value streams on their prod clusters.