UNCLASSIFIED - NO CUI

Minimize config required for Fluentd output from Fluentbit

Currently in fluentbit we have set up the default values to output to the elasticsearch in BB cluster: https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit/-/blob/main/chart/values.yaml#L302

The issue arises when someone wants to add a fluentd output for these logs. They need to copy all the existing config as well as writing up new output sections following the upstream guidance - https://docs.fluentbit.io/manual/pipeline/outputs/forward .

We would like to make this easier for the majority of customers so that they can provide a "minimal config" and the BB chart will setup the proper [OUTPUT] block for them.

As a baseline/rough sketch of what we're looking for...a customer should be able to supply values like the below:

additionalOutputs:
  fluentd:
    host:
    port:
    additionalConfig:
      key: value

And then with modifications to the chart we take these values and build the required [OUTPUT] block(s) into the fluentbit config file so that all logs are shipped to the supplied destination. NOTE: We want the basic config values here to be the absolute minimum required to configure a fluentd output - there may be additional ones discovered during testing. We should add anything that is required under additionalOutputs.fluentd but beyond that... if a customer has a more advanced usecase, additionalConfig would allow them to specify anything listed in https://docs.fluentbit.io/manual/pipeline/outputs/forward#configuration-parameters .

Finally, we will want to make sure networkPolicies are modified to allow egress to the additional output when someone specifies one.

AC:

  • Provide a clean/simple interface for customers to be able to specify a minimal config for a fluentd output
  • Update networkPolicies to allow egress to external fluentd when one is provided
Edited by Micah Nagel