add developer documentation

docs/developer/develop-package.md

+# Package Development
+
+Package is the term we use for an application that has been prepared to be deployed with the BigBang helm chart. BigBang Packages are wrappers around Helm charts. All of the pertinent information should be included in the chart/values.yaml for configuration of the Package. These values are then available to be overridden in the BigBang chat values.yaml file. The goal of these Packages is to take something that might be very complex and simplify it for consumers of BigBang. Rational and safe defaults should be used where possible while also allowing for overriding values when fine-grained control is needed. As much as possible test after each step so that the errors don't pile up, "code a little, test a little". Here are the steps:
+
+1. Create a repository under the appropriate group ( example: Security Tools, Developer Tools, Collaboration Tools) in [Repo1](https://repo1.dso.mil/platform-one/big-bang/apps).
+
+1. Create a "main" branch that will serve as the master branch.
+
+1. There are two ways to start a new Package.  
+   A. If there is no upstream helm chart we create a helm chart from scratch. Here is a T3 video that demonstrates creating a new helm chart. Create a directory called "chart" in your repo, change to the chart directory, and scaffold a new chart in the chart directory
+
+   ```bash
+   # scaffold new helm chart
+   mkdir chart
+   cd chart
+   helm create name-of-your-application
+   ```
+
+   B. If there is an existing upstream chart we will use it and modify it. Essentially we create a "fork" of the upstream code. Use kpt to import the helm chart code into your repository. Note that kpt is not used to keep the Package code in sync with the upstream chart. It is a one time pull just to document where the upstream chart code came from. Kpt will generate a Kptfile that has the details. Do not manually create the "chart" directory.  The kpt command will create it. Here is an example from when Gitlab Package was created. It is a good idea to push a commit "initial upstream chart with no changes" so you can refer back to the original code while you are developing.
+
+   ```bash
+   kpt pkg get https://gitlab.com/gitlab-org/charts/gitlab.git@v4.8.0 chart
+   ```
+
+1. Run a helm dependency update that will download any external sub-chart dependencies. Commit any *.tgz files that are downloaded into the "charts" directory. The reason for doing this is that BigBang Packages must be able to be installed in an air-gap without any internet connectivity.
+
+   ```bash
+   helm dependency update
+   ```
+
+1. Edit the Chart.yaml and set the chart ```version:``` number to be compliant with the charter versioning which is {UpstreamChartVersion}-bb.{BigBangVersion}. Note that the chart version is not the same thing as the application version. If this is a patch to an existing Package chart then increment the {BigBangVersion}. Here is an example from Gitlab Runner.
+
+   ```yaml
+   apiVersion: v1
+   name: gitlab-runner
+   version: 0.19.2-bb.3
+   appVersion: 13.2.2
+   description: GitLab Runner
+   ```
+
+1. Some upstream helm charts have a file called "requirements.yaml" that contains internet links to external dependencies. Point all external links to the local file system. Also delete the dependency lock file that was generated during the previous step. Again, BigBang Packages must be able to be installed in an air-gap without any internet connectivity. It will look like this example from Gitlab.
+
+   ```yaml
+   dependencies:
+   - name: gitlab
+     version: '*.*.*'
+     repository: file://./charts/gitlab
+   - name: cert-manager
+     version: 0.10.1
+     repository: file://./charts/cert-manager-v0.10.1.tgz
+     condition: certmanager.install
+     alias: certmanager
+   ```
+
+1. In the values.yaml replace public upstream images with IronBank hardened images. The image version should be compatible with the chart version. Here is a command to identify the images that need to be changed.
+
+   ```bash
+   # list images
+   helm template <releasename> ./chart -n <namespace> -f chart/values.yaml | grep image:
+   ```
+
+   Add the "imagePullSecrets" tag if not already there.  You can still test without the private-registry secret existing if your k8s cluster is configured with the pull credentials. Here is an example from Gitlab Package.
+
+   ```yaml
+   registry:
+     enabled: true
+     host: "registry.bigbang.dev"
+     image:
+       repository: registry1.dso.mil/ironbank/gitlab/gitlab/gitlab-container-registry
+       tag: 13.7.2
+       pullSecrets:
+       - name: private-registry
+   ```
+
+1. Add a VirtualService if your application has a back-end API or a front-end GUI. Create the VirtualService in the sub-directory  "chart/templates/bigbang/VirtualService.yaml". You will need to manually create the "bigbang" directory. It is convenient to copy VirtualService code from one of the other Packages and then modify it. You should be able to load the application in your browser if all the configuration is correct.
+
+1. Add a continuous integration(CI) pipeline to the Package. Copy the ./gitlab-ci.yml from another Package and add it to your Package.
+
+   ```yaml
+   include:
+     - project: 'platform-one/big-bang/pipeline-templates/pipeline-templates'
+       ref: master 
+       file: '/templates/package-tests.yml'
+   ```
+
+1. If the Package requires dependencies to be installed before it can function properly add a ./tests/dependencies.yaml file to specify these. A great example of where this is required is packages that require an operator or a database.
+
+   ```yaml
+   dependencyname:
+     git: "https://repo1.dso.mil/platform-one/big-bang/apps/DEPENDENCY.git" # Required: Git clone link for the dependency
+     namespace: "namespace" # Optional: Specify the namespace to install the dependency chart, this will default to the "dependencyname" for the yaml block
+     branch: "master" # Optional: Specify a specific branch/tag to install the dependency from
+   ```
+
+1. Add CI pipeline test to the Package. A Package should be able to be deployed by itself, independently from the BigBang chart. The Package pipeline takes advantage of this to run a Package pipeline test. Create a test directory and a test yaml file at "tests/test-values.yml".  Set any values that are necessary for this test to pass.  The pipeline automatically creates an image pull secret "private-registry-mil".  All you need to do is reverence that secret in your test values. You can view the pipeline status from the Repo1 console. Keep iterating on your Package code and the test code until the pipeline passes. Refer to the test-values.yml from other Packages to get started. The repo structure must match what the CI pipeline code expects.
+
+   If your pipeline deploys a customresource (i.e. istio-controlplane deploys an "istiooperator" resource) you should add a wait script to make sure these resources are running properly before Cypress runs. Review this [README](https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates#using-the-infrastructure-in-your-package-ci-gitlab-pipeline) for how to set this up. The pipeline will run configuration tests, install your helm chart, and run cypress tests. If there is no UI or external facing API that could be tested using Cypress do not include those files and the pipeline will automatically skip over cypress testing.
+
+   ```yaml
+   |-- .gitlab-ci.yml
+   |-- chart
+   |   |-- Chart.yml
+   |   |-- charts
+   |   |-- templates
+   |   `-- values.yml
+   `-- tests
+       |-- cypress
+       |   `-- integration
+       |       `-- health.spec.js
+       |-- cypress.json
+       |-- main-test-gateway.yml
+       `-- test-values.yml
+   ```
+
+1. Documentation for the Package should be included. A "docs" directory would include all detailed documentation. Reference other that Packages for examples.
+
+1. Add the following markdown files to complete the Package. Reference other that Packages for examples of how to create them.
+
+   ```bash  
+   CHANGELOG.md      <  standard history of changes made  
+   CODEOWNERS        <  list of the code maintainers. Minimum of two people from separate organizations  
+   CONTRIBUTING.md   <  instructions for how to contribute to the project  
+   README.md         <  introduction and high level information  
+   ```
+
+1. The Package structure should look like this when you are finished
+![package-structure](uploads/ab22cc8f1d4295b84ddf9fc80a8fc4bc/package-structure.png)
+
+1. Merging code should require approval from a minimum of 2 codeowners. To setup merge requests to work properly with CODEOWNERS approval change these settings in your project:  
+Under Settings → General → Merge Request Approvals, change "Any eligible user" "Approvals required" to 1. Also ensure that "Require new approvals when new commits are added to an MR" is checked.  
+Under Settings → Repository → Protected Branches, add the main branch with "Developers + Maintainers" allowed to merge, "No one" allowed to push, and "Codeowner approval required" turned on.  
+Under Settings → Repository → Default Branch, ensure that main is selected.  
+
+1. Development Testing Cycle: Test your Package chart by deploying with helm. Test frequently so you don't pile up multiple layers of errors. The goal is for Packages to be deployable independently of the bigbang chart. Most upstream helm charts come with internal services like a database that can be toggled on or off. If available use them for testing and CI pipelines. In some cases this is not an option. You can manually deploy required in-cluster services in order to complete your development testing.  
+   Here is an example of an in-cluster postgres database
+
+   ```bash
+   helm repo add bitnami https://charts.bitnami.com/bitnami
+   helm install postgres bitnami/postgresql -n postgres --create-namespace --set postgresqlPostgresPassword=postgres --set postgresqlPassword=postgres
+   # test it
+   kubectl run postgresql-postgresql-client --rm --tty -i --restart='Never' --namespace default --image bitnami/postgresql --env="PGPASSWORD=postgres" --command -- psql --host postgres-postgresql-headless.postgres.svc.cluster.local -U postgres -d postgres -p 5432
+   # postgres commands
+    \l             < list tables
+    \du            < list users
+    \q             < quit
+   ```
+
+   Here is an example of an in-cluster object storage service using MinIO (api compatible with AWS S3 storage)
+
+   ```bash
+   helm repo add minio https://helm.min.io/
+   helm install minio minio/minio --set accessKey=myaccesskey --set secretKey=mysecretkey -n minio --create-namespace
+   # test and configure it
+   kubectl run minio-mc-client --rm --tty -i --restart='Never' --namespace default --image minio/mc --command -- bash
+   # MinIo commands
+   mc alias set minio http://minio.minio.svc.cluster.local:9000 myaccesskey mysecretkey      < set a connection alias
+   mc mb minio/myBucket          < make a bucket
+   mc ls minio                   < list the buckets
+   ```
+
+   Here are the dev test steps you can iterate:
+
+   ```bash
+   # test that the helm chart templates successfully and examine the output to insure expected results
+   helm template <releasename> ./chart -n <namespace> -f chart/values.yaml
+   # deploy with helm
+   helm upgrade -i <releasename> ./chart -n <namespace> --create-namespace -f chart/values.yaml
+   # conduct testing
+   # tear down
+   helm delete <releasename> -n <namespace>
+   # manually delete the namespace to insure that everything is gone
+   kubectl delete ns <namespace>
+   ```
+
+1. Wait to create a git tag release until integration testing with BigBang chart is completed.  You will very likely discover more Package changes that are needed during BigBang integration. When you are confident that the Package code is complete, squash commits and rebase your development branch with the "main" branch.
+
+   ```bash
+   git reset $(git merge-base origin/main $(git rev-parse --abbrev-ref HEAD))
+   git add -A
+   git commit -m "feat: example conventional commit"
+   git push --force
+   ```
+
+1. Then, create a merge request to branch "main"
+
+1. After the merge create a git tag following the charter convention of {UpstreamChartVersion}-bb.{BigBangVersion}. The tag should exactly match the chart version in the Chart.yaml.
+example:    1.2.3-bb.0

docs/developer/developer-documentation.md

+# Developer Documentation
+
+## Charter
+
+The [BigBang Charter](https://repo1.dso.mil/platform-one/big-bang/bigbang/-/tree/master/charter) is required reading for BigBang developers. Study all the documents carefully before you start developing.  The Charter lays out the policies, requirements, and responsibilities for the BigBang product and the supported Packages (applications). At a high level, the BigBang product is a helm chart that wraps the deployment of DevSecOps applications (Packages). The goal of BigBang is to hide the complexity of deploying and integrating the supported Packages. Customers should be able to easily deploy and configure a DevSecOps environment. BigBang is intended to deploy on any OCI/CNCF compliant Kubernetes cluster.
+
+## Communications
+
+Join Mattermost channels to ask questions and communicate with the team. The team also has a daily stand up at 8:30 am MST. The link for the stand up is usually found in the channel headers. Here is the list of relevant Mattermost channels for BigBang development:  
+[public Big Bang](https://chat.il2.dso.mil/platform-one/channels/team---big-bang)  
+[private Big Bang - by invitation only](https://chat.il2.dso.mil/platform-one/channels/team---bigbang)  
+[Big Bang add-ons](https://chat.il2.dso.mil/platform-one/channels/substream---bb---add-ons)  
+[Big Bang packages](https://chat.il2.dso.mil/platform-one/channels/substream---bb---packages)
+[Big Bang pipelines](https://chat.il2.dso.mil/platform-one/channels/substream---bb---umbrella-pipelines)  
+[Big Bang package pipelines](https://chat.il2.dso.mil/platform-one/channels/substream---bb---package-pipelines)  
+
+## Big Bang Framework
+
+Big Bang is a helm chart of helm charts. Big Bang uses [Flux 2](https://fluxcd.io/) to deploy [Helm](https://helm.sh/) charts. The Helm charts that are deployed by BigBang are called Packages. The [Package repositories](https://repo1.dso.mil/platform-one/big-bang/apps) are organized into groups such as core, security tools, developer tools, collaboration tools, etc.
+
+## Set up a development environment
+
+[Development Environment](./development-environment.md)
+
+## Package Development
+
+[Develop a Big Bang Package](./develop-package.md)
+
+## Add Package to Big Bang
+
+[Integrate Package with Big Bang](./package-integration.md)

docs/developer/package-integration.md

+# Integrate a Package with BigBang helm chart
+1. Make a branch from the BigBang chart repository master branch. You can automatically create a branch from the Repo1 Gitlab issue. Or, in some cases you might manually create the branch. You should name the branch with your issue number. If your issue number is 9999 then your branch name can be "9999-my-description". It is best practice to make branch names short and simple. 
+
+1. Create a directory for your package at chart/templates/<your-package-name>
+
+1. Inside this folder will be 3 helm template files. You can copy one of the other package folders and tweak the code for your package. Gitlab is a good example to reference because it is one of the more complicated Packages. Note that the Istio VirtualService comes from the Package and is not created in the BigBang chart. The purpose of these helm template files is to create an easy-to-use spec for deploying supported applications. Reasonable and safe defaults are provided and any needed secrets are auto-created. We accept the trade off of easy deployment for complicated template code. More details are in the following steps. 
+   ```
+   gitrepository.yaml  # Flux GitRepository. Is configured by BigBang chart values.
+   helmrelease.yaml    # Flux HelmRelease. Implements all the BigBang customizations of the base Package.
+   namespace.yaml      # Contains the namespace and any needed secrets
+   ```
+1. More details about helmrelease.yaml:  Code reasonable and safe defaults but prioritize any user defined passthrough values wherever this makes sense. Avoid duplicating tags that are provided in the upstream chart values. Instead code reasonable defaults in the helmrelease.yaml template. The following is an example from Gitlab that handles SSO config. The code uses Package chart passthrough values if the user has entered them but otherwise defaults to the BigBang chart values or the HelmRelease default values. Notice that the secret is not handled this way. The assumption is that if the user has enabled the BigBang SSO feature the secret will be auto created. In this case the user should not be overriding the secret. If the user wants to create their own secret they should not be enabling the BigBang SSO feature.  
+
+   Note that helm does not handle any missing parent tags in the yaml tree. The 'if' statement and 'default' method throw 'nil' errors when parent tags are missing. The work-around is to inspect each level of the tree and assign an empty 'dict' if the value does not exist. Then you will be able to use 'hasKey' in your 'if' statements as shown below in this example from Gitlab. Having described all this, you should understand that coding conditional values is optional. The passthrough values will take priority regardless. But the overridden values will not show up in the deployed flux HelmRelease object if you don't code the conditional values. The value overrides will be obscured in the Package values secret. The only way to confirm that the overrides have been applied is to use "helm get values <releasename> -n bigbang" command on the deployed helm release. When the passthrough values show up in the HelmRelease object the Package configuration is much easier to see and verify. Use your own judgement on when to code conditional values.
+
+   ```
+   global: 
+     appConfig:
+       {{- if .Values.addons.gitlab.sso.enabled }}
+       omniauth:
+         enabled: true
+         {{- $global := .Values.addons.gitlab.values.global | default dict }}
+         {{- $appConfig := $global.appConfig | default dict }}
+         {{- $omniauth := $appConfig.omniauth | default dict }}
+         {{- if hasKey $omniauth "allowSingleSignOn" }}
+         allowSingleSignOn: {{ .Values.addons.gitlab.values.global.appConfig.omniauth.allowSingleSignOn }}
+         {{- else }}
+         allowSingleSignOn: ['openid_connect']
+         {{- end }}
+         {{- if hasKey $omniauth "blockAutoCreatedUsers" }}
+         blockAutoCreatedUsers: {{ .Values.addons.gitlab.values.global.appConfig.omniauth.blockAutoCreatedUsers }}
+         {{- else }}
+         blockAutoCreatedUsers: false
+         {{- end }}
+         providers:
+             - secret: gitlab-sso-provider
+               key: gitlab-sso.json
+   ```
+1. More details about namespace.yaml: This template is where the code for secrets go. Typically you will see secrets for imagePullSecret, sso, and database. These secrets are a BigBang chart enhancement.  They are created conditionally based on what the user enables in the config.
+
+1. Edit the chart/templates/values.yaml.  Add your Package to the list of Packages.  Just copy one of the others and change the name. This supports adding chart values from a secret. Pay attention to whether this is a core Package or an add-on package, the toYaml values are different for add-ons. This template allows a Package to add chart values that need to be encrypted in a secret.
+
+1. Edit the chart/values.yaml.  Add your Package to the bottom of the core section if a core package or addons section if an add-on. You can copy from one of the other packages and modify appropriately.  Some possible tags underneath your package are [ enabled, git, sso, database, objectstorage ].  Avoid duplicating value tags from the upstream chart in the BigBang chart.  The goal is not to cover every edge case. Instead code reasonable defaults in the helmrelease template and allow customer to override values in addons.<packageName>.values
+   ```
+   addons:
+     mypackage:
+       enabled: false     # default to false
+       git:
+         repo: https://repo1.dsop.io/platform-one/big-bang/apps/developer-tools/mypackage.git
+         path: "./chart"
+         tag: "1.2.3-bb.0"
+       sso:
+         enabled: false   # default to false
+         client_id: ""
+       database:
+         host: ""
+         port: ""
+         username: ""
+         database: ""
+         password: {} # unencoded stringData
+       objectstorage:
+         type: s3   # supported types are "s3" or "minio"
+         endpoint: "" # ignored if type is "s3". used only for minio. example " http://minio.minio.svc.cluster.local:9000"
+         host: s3.amazonaws.com  # used for gitlab backup storage
+         region: us-west-1
+         accessKey: ""
+         accessSecret: ""
+         bucketPrefix: ""  # optional. example: dev-
+       values: {}
+   ```
+
+1. Edit tests/ci/k3d/values.yaml. These are the settings that the CI pipeline uses to run a deployment test.  Set your Package to be enabled and add any other necessary values.  When you commit your code the pipeline will run. You can view the pipeline in the Repo1 Gitlab console. Fix any errors in the pipeline output. The pipeline automatically runs a "smoke" test. It deploys bigbang on a k3d cluster using the test values file.
+
+1. Add your packages name to the ORDERED_HELMRELEASES list in scripts/deploy/02_wait_for_helmreleases.sh.
+
+1. Verify your Package works when deployed through bigbang. While testing you should use a git branch instead of tag, (i.e. replace tag: "1.2.3-bb.0" with branch: "main").  After you have tested BigBang integration, tag the commit in your Package following the convention of {UpstreamChartVersion}-bb.{BigBangVersion} – example 1.2.3-bb.0.
+
+1. Make sure to change the chart/values.yaml file to point to the tag rather than your branch (i.e. tag: "1.2.3-bb.0" in place of branch: "bb-9999").
+
+1. When you are done developing the BigBang chart features for your Package make a merge request. Keep your merge request in "Draft" status until all issues are resolved. The merge request will start a pipeline. Fix any errors.  Address any issues raised in the merge request comments.
+
+1. The final step before removing "draft" status from your merge request is to squash commits and rebase your development branch with the master branch
+   ```
+   git reset $(git merge-base origin/master $(git rev-parse --abbrev-ref HEAD))
+   git add -A
+   git commit -m "feat: example conventional commit"
+   git push --force
+   ```
+
+
+# BigBang Development and Testing Cycle
+There are two ways to test BigBang, imperative or GitOps with Flux.  Your initial development can start with imperative testing.  But you should finish with GitOps to make sure that your code works with Flux.
+
+1. **Imperative:** you can manually deploy bigbang with helm command line. With this method you can test local code changes without committing to a repository. Here are the steps that you can iterate with "code a little, test a little".  From the root of your local bigbang repo:
+   ```
+   # deploy with helm while pointing to your test values files
+   # bigbang packages should create any needed secrets from the chart values
+   # if you have the values file encrypted with sops, temporarily decrypt it
+   helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../customers/template/dev/configmap.yaml -f ./chart/ingress-certs.yaml -f ../customers/template/dev/registry-values.enc.yaml
+ 
+   # conduct testing
+   # if you make code changes you can run another helm upgrade to pick up the new changes
+   helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../customers/template/dev/configmap.yaml -f ./chart/ingress-certs.yaml -f ../customers/template/dev/registry-values.enc.yaml
+ 
+   # tear down
+   helm delete bigbang -n bigbang
+   # helm delete will not delete the bigbang namespace
+   kubectl delete ns bigbang
+   # istio namespace will be stuck in "finalizing". So run the script to delete it.
+   hack/remove-ns-finalizer.sh istio-system
+   ```
+
+2. **GitOps with Flux:**  You can deploy your development code the same way a customer would deploy using GitOps. You must commit any code changes to your development banches because this is how GitOps works. There is a [customer template repository](https://repo1.dso.mil/platform-one/big-bang/customers/template) that has an example template for how to deploy using BigBang. You can create a branch from one of the other developer's branch or start clean from the master branch. Make the necessary modifications as explained in the README.md. The setup information is not repeated here. This is a public repo so DO NOT commit unencrypted secrets. Before committing code it is a good idea to manually run "helm template" and a "helm install" with dry run.  This will reveal many errors before you make a commit. Here are the steps you can iterate:
+   ```
+   # verify chart code before committing
+   helm template bigbang ./chart -n bigbang -f ../customers/template/dev/configmap.yaml --debug
+   helm install bigbang ./chart -n bigbang -f ../customers/template/dev/configmap.yaml --dry-run
+   # commit and push your code
+   # deploy your bigbang template
+   kubectl apply -f dev/bigbang.yaml
+   # monitor rollout
+   watch kubectl get pod,helmrelease -A
+   # conduct testing
+   # tear down
+   kubectl delete -f dev/bigbang.yaml
+   # istio namespace will be stuck in "finalizing". So run the script to delete it. You will need 'jq' installed
+   hack/remove-ns-finalizer.sh istio-system
+ 
+   # if you have pushed code changes before the tear down, occasionally the bigbang deployments are not terminated
+   # because Flux has not had enough time to reconcile the helmreleases
+   # re-deploy bigbang
+   kubectl apply -f dev/bigbang.yaml
+   # run the sync script.
+   hack/sync.sh
+   # tear down
+   kubectl delete -f dev/bigbang.yaml
+   hack/remove-ns-finalizer.sh istio-system
+   ```