#385 : Support kustomizer scripts to dynamically populate kustomize files

General MR

Summary

This MR allows for pre-deployment modification or generation of kustomization objects in bigbang packages which deploy with kustomize. The new logic is:

• Is kustomize: true in the values.yaml? If not, deploy with helm.
• Is there a kustomizer file in the root of the project? If so, execute it.
• kubectl apply -k kustomization/

This is necessary for some projects (External Secrets Operator being the example again here) which need to be deployed from one set of source files in multiple scenarios. Consider the two flows this chart may be deployed through, and the objects in them:

(package pipeline) 
    -> kubectl apply -k 
    -> kustomize dir 
    -> (objects needing dynamic data)

(umbrella pipeline) 
    -> helm upgrade bigbang 
    -> bigbang package chart 
    -> kustomization 
    -> kustomize dir 
    -> (objects needing dynamic data)

The dynamic data may be anything, but in this case for External Secrets Operator, it is the name of the source branch we are deploying from. The source branch needs to be patched into a GitRepository object in the kustomization directory - but ONLY if the package is being deployed through the package pipeline kustomize workflow. In the umbrella deployment case, that gitrepo kustomization object actually must be absent - because the umbrella chart creates a gitrepo for us, which means our secondary git repo is not needed in that case. So the default state of the repository in gitlab should be like this:

chart                   <-- contains the helm chart that will be used by the helm release
kustomization           <-- contains kustomizations which will also be used by the 
  |                             umbrella chart and package pipeline
  |-kustomization.yaml  <-- contains list of kustomization steps
  |-crds.yaml           <-- contains list of custom resource definitions to deploy from the git repo
  |-helmrelease.yaml    <-- contains a helmrelease that points back to the chart in the git repo

When we initially added support for deploying kustomizations to the pipeline, the ESO repo contained the gitrepository in the kustomization directory, with a reference to the current source branch inside of it. So you had the sourceref in 3 places: the gitrepo, the helmrelease, and the crds. The workflow would now look like this:

1. Create a branch
2. Modify the CRD, gitrepo and helmrelease kustomizations to point to your branch
3. Create your MR and run your pipelines
4. Get approval
5. Merge - but you can't merge to master, because the kustomizations have your branch as the sourceref, not master. You have to modify your MR after approving it and before merging it. This is bad.

So now, in order to avoid having people touch their MRs after approval but before merge, we have a need to modify a variable in 3 files before applying the kustomization. I initially explored 2 options here:

1. Kustomize overlays. This won't work because, while we COULD point the kustomization resource from the package chart to a directory in the repository containing the overlays, we would still need a way to dynamically overwrite the contents of those overlays, which leads us right back to the same problem we have right now.
2. Variable substitution syntax in the kustomization files replaced at runtime from environment variables. Environment variable substitution works in the package pipeline case (assuming we only ever need to substitute a known subset of variables provided by the pipeline); but it won't work in the umbrella chart application because the umbrella chart does not perform such substitution (and even if it did, how would it know which ones to provide?)

This is why I landed on the use of a kustomizer script: ultimately, it is a punt to the package maintainer to do the needful in the case of a local package deployment outside of the umbrella chart to prepare the kustomized chart for deployment. This allows us to store the chart in the repository in a state that is ready for deployment with the umbrella chart with only the customizations provided through values.yaml; it also allows us to add the necessary additional objects and patch existing objects to make it deploy correctly outside of the umbrella chart.

Here is the kustomizer being used by the External Secrets Operator, for reference. It is simple but verbose, trying to provide a full account of its actions upon completion.

There are definitely risks with this approach.

• We are allowing for arbitrary code execution inside of our pipelines. A kustomizer can do anything the pod it's running on can do. Kustomizer scripts should be discouraged and heavily scrutinized (including this one - I'm happy to have someone provide an alternative that makes this unnecessary).
• We are allowing builds that are not practically idempotent. Assuming that the same build environment can be supplied to the same revision of the repository at a later date, and assuming the kustomizer only makes decisions based on those inputs, we will get the same outcome. However it will be quite possible for the kustomizer to behave in ways that cannot be reproduced later.

In a normal enterprise environment I would probably argue in favor of this approach even given those two risks. In the Platform One environment I expect more pushback. However, I (in the time I had yesterday to try and solve this problem) did not see a better way to go about it.

Relevant logs/screenshots

Here is a successful package validation using the kustomize flag and a kustomizer script

+++ cat chart/values.yaml
+++ yq .kustomize
++ '[' true == true ']'
++ '[' -f ./kustomizer ']'
++ echo 'Executing external-secrets/kustomizer to update kustomizations before application'
Executing external-secrets/kustomizer to update kustomizations before application
++ chmod +x ./kustomizer
++ ./kustomizer
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: external-secrets-kustomization
  namespace: external-secrets
  labels:
    app.kubernetes.io/name: external-secrets
spec:
  interval: 5m
  url: https://repo1.dso.mil/big-bang/apps/sandbox/external-secrets.git
  ref:
    branch: "akesterson-external-secrets-13_DEVELOPMENT_MAINTENANCE.md"
diff --git a/kustomization/crds.yaml b/kustomization/crds.yaml
index 7440592..454e11e 100644
--- a/kustomization/crds.yaml
+++ b/kustomization/crds.yaml
@@ -10,4 +10,4 @@ spec:
   prune: true
   sourceRef:
     kind: GitRepository
-    name: external-secrets
+    name: external-secrets-kustomization
diff --git a/kustomization/deployment.yaml b/kustomization/deployment.yaml
index b199c17..4485560 100644
--- a/kustomization/deployment.yaml
+++ b/kustomization/deployment.yaml
@@ -14,7 +14,7 @@ spec:
       reconcileStrategy: ChartVersion
       sourceRef:
         kind: GitRepository
-        name: external-secrets
+        name: external-secrets-kustomization
         namespace: external-secrets
   values:
     installCRDs: false
diff --git a/kustomization/kustomization.yaml b/kustomization/kustomization.yaml
index 356e170..075e9a8 100644
--- a/kustomization/kustomization.yaml
+++ b/kustomization/kustomization.yaml
@@ -1,5 +1,6 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - gitrepo.yaml
   - crds.yaml
   - deployment.yaml
++ echo 'Kustomize installing external-secrets/kustomization into external-secrets namespace'
Kustomize installing external-secrets/kustomization into external-secrets namespace
++ kubectl apply -k kustomization
Warning: v2beta1 HelmRelease is deprecated, upgrade to v2beta2
helmrelease.helm.toolkit.fluxcd.io/external-secrets created
Warning: v1beta2 Kustomization is deprecated, upgrade to v1
kustomization.kustomize.toolkit.fluxcd.io/external-secrets-crds created
gitrepository.source.toolkit.fluxcd.io/external-secrets-kustomization created

Here is a successful build that does not use the kustomize flag

+++ cat chart/values.yaml
+++ yq .kustomize
++ '[' null == true ']'
+++ ls -1 tests/test-values.yaml
+++ wc -l
++ '[' 1 -gt 0 ']'
++ echo 'Helm installing jira/chart into jira namespace using jira/tests/test-values.yaml for values'
Helm installing jira/chart into jira namespace using jira/tests/test-values.yaml for values
++ helm upgrade -i --wait --timeout 600s jira chart -n jira -f tests/test-values.yaml --set istio.enabled=false
Release "jira" does not exist. Installing it now.
NAME: jira
LAST DEPLOYED: Wed May 15 19:21:41 2024
NAMESPACE: jira
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing Jira.

Linked Issue

385

Upgrade Notices

N/A