UNCLASSIFIED - NO CUI

Minimize config required for Loki output from Fluentbit

Currently in fluentbit we have set up the default values to output to the elasticsearch in BB cluster: https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit/-/blob/main/chart/values.yaml#L302

The issue arises when someone wants to add a Loki output for these logs. They need to copy all the existing config as well as writing up new output sections following the upstream guidance - https://docs.fluentbit.io/manual/pipeline/outputs/loki .

We would like to make this easier for the majority of customers so that they can provide a "minimal config" and the BB chart will setup the proper [OUTPUT] block for them.

As a baseline/rough sketch of what we're looking for...a customer should be able to supply values like the below:

additionalOutputs:
  loki:
    host: 
    port:
    user: 
    password:
    additionalConfig:
      key: value

And then with modifications to the chart we take these values and build the required [OUTPUT] block(s) into the fluentbit config file so that all logs are shipped to the supplied destination. NOTE: We want the basic config values here to be the absolute minimum required to configure a loki output - host/port/user/password are certainly required but there may be additional ones discovered during testing. We should add anything that is required under additionalOutputs.loki but beyond that... if a customer has a more advanced usecase, additionalConfig would allow them to specify anything listed in https://docs.fluentbit.io/manual/pipeline/outputs/loki#configuration-parameters .

Finally, we will want to make sure networkPolicies are modified to allow egress to the additional output when someone specifies one.

AC:

  • Provide a clean/simple interface for customers to be able to specify a minimal config for a loki output
  • Update networkPolicies to allow egress to external loki when one is provided
Edited by Micah Nagel