Minimize config required for Fluentd output from Fluentbit
Currently in fluentbit we have set up the default values to output to the elasticsearch in BB cluster: https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit/-/blob/main/chart/values.yaml#L302
The issue arises when someone wants to add a fluentd output for these logs. They need to copy all the existing config as well as writing up new output sections following the upstream guidance - https://docs.fluentbit.io/manual/pipeline/outputs/forward .
We would like to make this easier for the majority of customers so that they can provide a "minimal config" and the BB chart will setup the proper [OUTPUT]
block for them.
As a baseline/rough sketch of what we're looking for...a customer should be able to supply values like the below:
additionalOutputs:
fluentd:
host:
port:
additionalConfig:
key: value
And then with modifications to the chart we take these values and build the required [OUTPUT]
block(s) into the fluentbit config file so that all logs are shipped to the supplied destination. NOTE: We want the basic config values here to be the absolute minimum required to configure a fluentd output - there may be additional ones discovered during testing. We should add anything that is required under additionalOutputs.fluentd
but beyond that... if a customer has a more advanced usecase, additionalConfig
would allow them to specify anything listed in https://docs.fluentbit.io/manual/pipeline/outputs/forward#configuration-parameters .
Finally, we will want to make sure networkPolicies are modified to allow egress to the additional output when someone specifies one.
AC:
-
Provide a clean/simple interface for customers to be able to specify a minimal config for a fluentd output -
Update networkPolicies to allow egress to external fluentd when one is provided