Minimize config required for Loki output from Fluentbit
Currently in fluentbit we have set up the default values to output to the elasticsearch in BB cluster: https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit/-/blob/main/chart/values.yaml#L302
The issue arises when someone wants to add a Loki output for these logs. They need to copy all the existing config as well as writing up new output sections following the upstream guidance - https://docs.fluentbit.io/manual/pipeline/outputs/loki .
We would like to make this easier for the majority of customers so that they can provide a "minimal config" and the BB chart will setup the proper [OUTPUT]
block for them.
As a baseline/rough sketch of what we're looking for...a customer should be able to supply values like the below:
additionalOutputs:
loki:
host:
port:
user:
password:
additionalConfig:
key: value
And then with modifications to the chart we take these values and build the required [OUTPUT]
block(s) into the fluentbit config file so that all logs are shipped to the supplied destination. NOTE: We want the basic config values here to be the absolute minimum required to configure a loki output - host/port/user/password are certainly required but there may be additional ones discovered during testing. We should add anything that is required under additionalOutputs.loki
but beyond that... if a customer has a more advanced usecase, additionalConfig
would allow them to specify anything listed in https://docs.fluentbit.io/manual/pipeline/outputs/loki#configuration-parameters .
Finally, we will want to make sure networkPolicies are modified to allow egress to the additional output when someone specifies one.
AC:
-
Provide a clean/simple interface for customers to be able to specify a minimal config for a loki output -
Update networkPolicies to allow egress to external loki when one is provided