Merge branch 'package-pipeline-docs' into 'master'

Docs - Developer: Secret Values, Package Pipelines

See merge request platform-one/big-bang/bigbang!373

docs/developer/develop-package.md

@@ -74,27 +74,9 @@ Package is the term we use for an application that has been prepared to be deplo
 
 1. Add a VirtualService if your application has a back-end API or a front-end GUI. Create the VirtualService in the sub-directory  "chart/templates/bigbang/VirtualService.yaml". You will need to manually create the "bigbang" directory. It is convenient to copy VirtualService code from one of the other Packages and then modify it. You should be able to load the application in your browser if all the configuration is correct.
 
-1. Add a continuous integration(CI) pipeline to the Package. Copy the ./gitlab-ci.yml from another Package and add it to your Package.
+1. Add a continuous integration (CI) pipeline to the Package. The [pipeline documentation](https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates#using-the-infrastructure-in-your-package-ci-gitlab-pipeline) provides a great rundown of how to reference the package pipeline template and handle more complicated situations like dependencies.
 
-   ```yaml
-   include:
-     - project: 'platform-one/big-bang/pipeline-templates/pipeline-templates'
-       ref: master 
-       file: '/templates/package-tests.yml'
-   ```
-
-1. If the Package requires dependencies to be installed before it can function properly add a ./tests/dependencies.yaml file to specify these. A great example of where this is required is packages that require an operator or a database.
-
-   ```yaml
-   dependencyname:
-     git: "https://repo1.dso.mil/platform-one/big-bang/apps/DEPENDENCY.git" # Required: Git clone link for the dependency
-     namespace: "namespace" # Optional: Specify the namespace to install the dependency chart, this will default to the "dependencyname" for the yaml block
-     branch: "master" # Optional: Specify a specific branch/tag to install the dependency from
-   ```
-
-1. Add CI pipeline test to the Package. A Package should be able to be deployed by itself, independently from the BigBang chart. The Package pipeline takes advantage of this to run a Package pipeline test. Create a test directory and a test yaml file at "tests/test-values.yml".  Set any values that are necessary for this test to pass.  The pipeline automatically creates an image pull secret "private-registry-mil".  All you need to do is reverence that secret in your test values. You can view the pipeline status from the Repo1 console. Keep iterating on your Package code and the test code until the pipeline passes. Refer to the test-values.yml from other Packages to get started. The repo structure must match what the CI pipeline code expects.
-
-   If your pipeline deploys a customresource (i.e. istio-controlplane deploys an "istiooperator" resource) you should add a wait script to make sure these resources are running properly before Cypress runs. Review this [README](https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates#using-the-infrastructure-in-your-package-ci-gitlab-pipeline) for how to set this up. The pipeline will run configuration tests, install your helm chart, and run cypress tests. If there is no UI or external facing API that could be tested using Cypress do not include those files and the pipeline will automatically skip over cypress testing.
+1. Add CI pipeline test values to the Package. A Package should be able to be deployed by itself, independently from the BigBang chart. The Package pipeline takes advantage of this to run a Package pipeline test. Create a tests directory and a test yaml file at "tests/test-values.yaml".  Set any values that are necessary for this test to pass.  The pipeline automatically creates an image pull secret "private-registry-mil".  All you need to do is reference that secret in your test values. You can view the pipeline status from the Repo1 console. Keep iterating on your Package code and the test code until the pipeline passes. Refer to the test-values.yaml from other Packages to get started. The repo structure must match what the CI pipeline code expects.
 
    ```
    |-- .gitlab-ci.yml

docs/developer/package-integration.md

@@ -6,16 +6,20 @@
 1. Inside this folder will be 3 helm template files. You can copy one of the other package folders and tweak the code for your package. Gitlab is a good example to reference because it is one of the more complicated Packages. Note that the Istio VirtualService comes from the Package and is not created in the BigBang chart. The purpose of these helm template files is to create an easy-to-use spec for deploying supported applications. Reasonable and safe defaults are provided and any needed secrets are auto-created. We accept the trade off of easy deployment for complicated template code. More details are in the following steps. 
    ```
    gitrepository.yaml  # Flux GitRepository. Is configured by BigBang chart values.
-   helmrelease.yaml    # Flux HelmRelease. Implements all the BigBang customizations of the base Package.
+   helmrelease.yaml    # Flux HelmRelease. Is configured by BigBang chart values.
    namespace.yaml      # Contains the namespace and any needed secrets
+   values.yaml         # Implements all the BigBang customizations of the package and passthrough for values.
    ```
-1. More details about helmrelease.yaml:  Code reasonable and safe defaults but prioritize any user defined passthrough values wherever this makes sense. Avoid duplicating tags that are provided in the upstream chart values. Instead code reasonable defaults in the helmrelease.yaml template. The following is an example from Gitlab that handles SSO config. The code uses Package chart passthrough values if the user has entered them but otherwise defaults to the BigBang chart values or the HelmRelease default values. Notice that the secret is not handled this way. The assumption is that if the user has enabled the BigBang SSO feature the secret will be auto created. In this case the user should not be overriding the secret. If the user wants to create their own secret they should not be enabling the BigBang SSO feature.  
+1. More details about values.yaml:  Code reasonable and safe defaults but prioritize any user defined passthrough values wherever this makes sense. Avoid duplicating tags that are provided in the upstream chart values. Instead code reasonable defaults in the values.yaml template. The following is an example from Gitlab that handles SSO config. The code uses Package chart passthrough values if the user has entered them but otherwise defaults to the BigBang chart values or the Helm default values. Notice that the secret is not handled this way. The assumption is that if the user has enabled the BigBang SSO feature the secret will be auto created. In this case the user should not be overriding the secret. If the user wants to create their own secret they should not be enabling the BigBang SSO feature.  
 
    Note that helm does not handle any missing parent tags in the yaml tree. The 'if' statement and 'default' method throw 'nil' errors when parent tags are missing. The work-around is to inspect each level of the tree and assign an empty 'dict' if the value does not exist. Then you will be able to use 'hasKey' in your 'if' statements as shown below in this example from Gitlab. Having described all this, you should understand that coding conditional values is optional. The passthrough values will take priority regardless. But the overridden values will not show up in the deployed flux HelmRelease object if you don't code the conditional values. The value overrides will be obscured in the Package values secret. The only way to confirm that the overrides have been applied is to use "helm get values <releasename> -n bigbang" command on the deployed helm release. When the passthrough values show up in the HelmRelease object the Package configuration is much easier to see and verify. Use your own judgement on when to code conditional values.
 
    ```yaml
    global: 
+     {{- if or .Values.addons.gitlab.sso.enabled .Values.addons.gitlab.objectStorage.endpoint }}
      appConfig:
+     {{- end }}
+
        {{- if .Values.addons.gitlab.sso.enabled }}
        omniauth:
          enabled: true
@@ -32,9 +36,12 @@
          {{- else }}
          blockAutoCreatedUsers: false
          {{- end }}
+
          providers:
-             - secret: gitlab-sso-provider
-               key: gitlab-sso.json
+           - secret: gitlab-sso-provider
+             key: gitlab-sso.json
+     {{- end }}
+
    ```
 1. More details about namespace.yaml: This template is where the code for secrets go. Typically you will see secrets for imagePullSecret, sso, and database. These secrets are a BigBang chart enhancement.  They are created conditionally based on what the user enables in the config.
 
@@ -69,7 +76,7 @@
        values: {}
    ```
 
-1. Edit tests/ci/k3d/values.yaml. These are the settings that the CI pipeline uses to run a deployment test.  Set your Package to be enabled and add any other necessary values.  When you commit your code the pipeline will run. You can view the pipeline in the Repo1 Gitlab console. Fix any errors in the pipeline output. The pipeline automatically runs a "smoke" test. It deploys bigbang on a k3d cluster using the test values file.
+1. Edit tests/ci/k3d/values.yaml. These are the settings that the CI pipeline uses to run a deployment test.  Set your Package to be enabled and add any other necessary values. Where possible reduce the number of replicas to a minumum to reduce straing on the CI infrastructure. When you commit your code the pipeline will run. You can view the pipeline in the Repo1 Gitlab console. Fix any errors in the pipeline output. The pipeline automatically runs a "smoke" test. It deploys bigbang on a k3d cluster using the test values file.
 
 1. Add your packages name to the ORDERED_HELMRELEASES list in scripts/deploy/02_wait_for_helmreleases.sh.
 
@@ -77,17 +84,7 @@
 
 1. Make sure to change the chart/values.yaml file to point to the tag rather than your branch (i.e. tag: "1.2.3-bb.0" in place of branch: "bb-9999").
 
-1. When you are done developing the BigBang chart features for your Package make a merge request. Keep your merge request in "Draft" status until all issues are resolved. The merge request will start a pipeline. Fix any errors.  Address any issues raised in the merge request comments.
-
-1. The final step before removing "draft" status from your merge request is to squash commits and rebase your development branch with the master branch
-   ```bash
-   git rebase origin/master
-   git reset $(git merge-base origin/master $(git rev-parse --abbrev-ref HEAD))
-   git add -A
-   git commit -m "feat: example conventional commit"
-   git push --force
-   ```
-
+1. When you are done developing the BigBang chart features for your Package make a merge request in "Draft" status and add a label corresponding to your package name (must match the name in `values.yaml`). Also add any labels for dependencies of the package that are NOT core apps. The merge request will start a pipeline and use the labels to determine which addons to deploy. Fix any errors that appear in the pipeline. When the pipeline has pass and the MR is ready take it out of "Draft" and add the `status::review` label. Address any issues raised in the merge request comments.
 
 # BigBang Development and Testing Cycle
 There are two ways to test BigBang, imperative or GitOps with Flux.  Your initial development can start with imperative testing.  But you should finish with GitOps to make sure that your code works with Flux.