Minimize config required for Elastic output from Fluentbit
Currently in fluentbit we have set up the default values to output to the elasticsearch in BB cluster: https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit/-/blob/main/chart/values.yaml#L302
The issue arises when someone wants to add an additional Elastic output for these logs. They need to copy all the existing config as well as writing up new output sections following the upstream guidance - https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch .
We would like to make this easier for the majority of customers so that they can provide a "minimal config" and the BB chart will setup the proper [OUTPUT]
block for them.
As a baseline/rough sketch of what we're looking for...a customer should be able to supply values like the below:
additionalOutputs:
elasticsearch:
host:
user:
password:
port:
additionalConfig:
key: value
And then with modifications to the chart we take these values and build the required [OUTPUT]
block(s) into the fluentbit config file so that all logs are shipped to the supplied destination. NOTE: We want the basic config values here to be the absolute minimum required to configure an elastic output - host/user/password are certainly required but there may be additional ones discovered during testing. We should add anything that is required under additionalOutputs.elasticsearch
but beyond that... if a customer has a more advanced usecase, additionalConfig
would allow them to specify anything listed in https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#configuration-parameters .
In addition we should provide an option to disable the default elastic output, such as:
additionalOutputs:
disableDefault: true
Finally, we will want to make sure networkPolicies are modified to allow egress to the additional output when someone specifies one.
AC:
-
Provide a clean/simple interface for customers to be able to specify a minimal config for an additional elastic output -
Provide a value for customers to be able to toggle off the default output to elastic (this toggle should only work if an additionalOutput
has been supplied) -
Update networkPolicies to allow egress to external elastic when one is provided