SPIKE: Explore decoupling addons options

In order to properly scope/plan out the work involved for this epic we need a spike to explore technical solutions to the challenge.

The challenge that has been identified is maintaining the BB "translation" logic / values passdown to properly configure addons, but having it live in the package rather than umbrella. At a basic level that means moving pieces of this directory somewhere inside the package repo and ensuring it is deployed / used in the same way.

Initial thoughts for consideration:

• Mimic the setup for a 3rd party/sandbox package, have these templates live in an adjoining directory
• Potentially leverage Flux Kustomizations/HelmRelease to deploy those (this would be something like a mini-umbrella)
• This route would align with the architecture of the customer template repo

The best case solution would allow someone to effectively just swap addons to packages in their values, have everything deploy seamlessly, but the underlying architecture would be different.

Output of this spike should be a POC (or multiple) showing how someone could deploy an addon package from umbrella, with no specific package knowledge living in Umbrella.

Proposed Solutions

Kustomize PoC - Docs

Deploys a Big Bang chart by creation a kustomization that creates a helm release

Pros:

• Uses packages values functionality to deploy a big bang "Addon"
• No additional helm release resources required
• May be able to deploy with a single gitrepo/OCI artifact

Cons:

• Adds an additional layer (kustomization) to the flux reconcile loop
• requires mapping helm values to kustomize variables back to helm values
• All supported packages will require additional work to support functionality
• Kustomize can only do simple variable replacement. Some of the packages have complex secret templates that need conditional logic and advanced methods like regex.

Nested HelmRelease PoC (demo values)

Pros:

• Sandbox and third-party packages will integrate with Big Bang seamlessly the same way supported package do.
• Supported packages can upgrade to BB 2.0 without any change to the package chart. No breaking changes to BB 1.x
• Sandbox and third-party packages will use the same CI pipeline as supported packages
• Packages deployment will behave the same way they do now as an "addon"

Cons:

• Nested HelmRelease adds an extra layer of complexity. But the extra complexity is mostly hidden from end-users.
• Nested HelmRelease would requrie two OCI artifacts per package unless we change how the OCI artifact is packaged and change from HelmRepository to OCIRepository.

Behavior of nested helm release (not necessarily a pro or a con):

• Nested helm releases behave as if they were a single helm release. Errors from second helm release show in the first helm release.
• Nested helm releases have a single set of merged values. Helm values for the first helm release cannot be retrieved. The second helm release has the full set of merged values.

Keeping Addons

Keep the current state of "Addons" as-is while still adding the wrapper functionality for mission apps. Does not include decoupling packages.
Pros:

• We've identified that the above solutions are temporary band-aids towards more mature logic
• We can reduce 2.0 adoption churn by maintaining current structure - providing more time to focus on preferred solutions.
• We can release 2.0 sooner and begin working on preferred solution
• Less support required for 2.0 as it will not break consumer deployments (besides modifications to defaults)

Cons:

• Does not de-couple the packages - meaning we are still responsible for testing/supporting addons
• Big Bang would still need a new release every time a non-core package release happens. This does not get us to where we need to be for BB 2.0
• Sandbox and third-party packages would still have no practical way to integrate with a Big Bang deployment.

Other Options:

Operators/Controllers - Out of Scope

Likely one of the long term solutions for where Big Bang should head and has been discussed / presented at T3 previously.

• Reduce Big Bang down to the core "Capabilities"
  • Service Mesh / Monitoring / Container runtime scanner / logging / SSO / etc
• Create an operator/controller that understands available capability implementations (IE service mesh capability implemented by Istio)
  • Allows for ability to switch out implementations of a capability without having to modify the API
• Implement a CRD for defining how an "Addon" or Mission App integrates with Capabilities.

Other?