Minimize config required for S3 output from Fluentbit
Currently in fluentbit we have set up the default values to output to the elasticsearch in BB cluster: https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit/-/blob/main/chart/values.yaml#L302
The issue arises when someone wants to add a S3 output for these logs. They need to copy all the existing config as well as writing up new output sections following the upstream guidance - https://docs.fluentbit.io/manual/pipeline/outputs/s3 .
We would like to make this easier for the majority of customers so that they can provide a "minimal config" and the BB chart will setup the proper [OUTPUT]
block for them.
As a baseline/rough sketch of what we're looking for...a customer should be able to supply values like the below:
additionalOutputs:
s3:
region:
bucket:
endpoint:
accessKey:
secretKey:
additionalConfig:
key: value
And then with modifications to the chart we take these values and build the required [OUTPUT]
block(s) into the fluentbit config file so that all logs are shipped to the supplied destination. AWS credentials will need to be handled slightly differently, they should not be added to the output config. NOTE: We want the basic config values here to be the absolute minimum required to configure a s3 output - there may be additional ones discovered during testing. We should add anything that is required under additionalOutputs.s3
but beyond that... if a customer has a more advanced usecase, additionalConfig
would allow them to specify anything listed in https://docs.fluentbit.io/manual/pipeline/outputs/s3#configuration-parameters .
Finally, we will want to make sure networkPolicies are modified to allow egress to the additional output when someone specifies one.
AC:
-
Provide a clean/simple interface for customers to be able to specify a minimal config for a s3 output -
Update networkPolicies to allow egress to external s3 when one is provided