Merge branch '687-docs-document-creation-of-overrides' into 'master'

Resolve "Docs: Document creation of overrides"

Closes #687

See merge request platform-one/big-bang/bigbang!781

docs/developer/develop-package.md

@@ -60,7 +60,7 @@ Package is the term we use for an application that has been prepared to be deplo
 
 1. Add a VirtualService if your application has a back-end API or a front-end GUI. Create the VirtualService in the sub-directory  "chart/templates/bigbang/VirtualService.yaml". You will need to manually create the "bigbang" directory. It is convenient to copy VirtualService code from one of the other Packages and then modify it. You should be able to load the application in your browser if all the configuration is correct.
 
-1. Add NetworkPolices templates in the sub-directory "chart/templates/bigbang/networkpolicies/*.yaml". The intent is to lock down all ingress and egress traffic except for what is required for the application to function properly. Start with a deny-all policy and then add additionl policies to open traffic as needed. Refer to the other Packages code for examples.
+1. Add NetworkPolices templates in the sub-directory "chart/templates/bigbang/networkpolicies/*.yaml". The intent is to lock down all ingress and egress traffic except for what is required for the application to function properly. Start with a deny-all policy and then add additionl policies to open traffic as needed. Refer to the other Packages code for examples. The [Gitlab package](https://repo1.dso.mil/platform-one/big-bang/apps/developer-tools/gitlab/-/tree/main/chart/templates/bigbang/networkpolicies) is a good/complete example.
 
 1. Add a continuous integration (CI) pipeline to the Package. A Package should be able to be deployed by itself, independently from the BigBang chart. The Package pipeline takes advantage of this to run a Package pipeline test. The package testing is done with a helm test library. Reference the [pipeline documentation](https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates#using-the-infrastructure-in-your-package-ci-gitlab-pipeline) for how to create a pipeline and also [detailed instructions](https://repo1.dso.mil/platform-one/big-bang/apps/library-charts/gluon/-/blob/master/docs/bb-tests.md) in the gluon library. Instructions are not repeated here.
 
@@ -74,9 +74,34 @@ Package is the term we use for an application that has been prepared to be deplo
    CONTRIBUTING.md   <  instructions for how to contribute to the project  
    README.md         <  introduction and high level information  
    ```
+1. Create a top-level tests directory and inside put a test-values.yaml file that includes any special values overrides that are needed for CI pipeline testing. Refer to other packages for examples. But this is specific to what is needed for your package.
+   ```
+   mkdir tests
+   touch test-values.yaml
+   ```
 
-1. The Package structure should look like this when you are finished
-![package-structure](uploads/ab22cc8f1d4295b84ddf9fc80a8fc4bc/package-structure.png)
+1. At a high level, a Package structure should look like this when you are finished  
+   ```text
+   ├── chart/
+      └── templates/
+         └── bigbang/
+            ├── networkpolicies/
+               ├── egress-*.yaml
+               └── ingress-*.yaml
+            └── virtualservice.yaml
+      ├── tests/
+         ├── cypress/
+         └── scripts/
+   ├── docs/
+      ├── documentation-file-1.md
+      └── documentation-file-2.md
+   ├── tests/
+      └── test-values.yaml
+   ├── CHANGELOG.md
+   ├── CODEOWNERS
+   ├── CONTRIBUTING.md
+   └── README.md
+   ```
 
 1. Merging code should require approval from a minimum of 2 codeowners. To setup merge requests to work properly with CODEOWNERS approval change these settings in your project:  
 Under Settings → General → Merge Request Approvals, change "Any eligible user" "Approvals required" to 1. Also ensure that "Require new approvals when new commits are added to an MR" is checked.  
@@ -110,13 +135,19 @@ Under Settings → Repository → Default Branch, ensure that main is selected.
    mc ls minio                   < list the buckets
    ```
 
+   Create a local directory on your workstation where you store your helm values override files. Don't make test changes in the Package values.yaml because they could accidentally be committed. The most convenient location is in a sibling directory next to the Package repo. Here is an example directory structure:
+   ```text
+   ├── PackageRepo/
+   └── overrides/
+         └── override-values.yaml
+   ```
    Here are the dev test steps you can iterate:
 
    ```shell
    # Test that the helm chart templates successfully and examine the output to insure expected results
-   helm template <releasename> ./chart -n <namespace> -f chart/values.yaml
+   helm template <releasename> ./chart -n <namespace> -f ../overrides/override-values.yaml
    # Deploy with helm
-   helm upgrade -i <releasename> ./chart -n <namespace> --create-namespace -f chart/values.yaml
+   helm upgrade -i <releasename> ./chart -n <namespace> --create-namespace -f ../overrides/override-values.yaml
    # Conduct testing
    # Tear down
    helm delete <releasename> -n <namespace>

docs/developer/development-environment.md

@@ -204,7 +204,7 @@ cd ./bigbang
 ./scripts/install_flux.sh -u your-user-name -p your-pull-secret
 ```
 
-**Note:** When deploying to k3d, istio-system should be added from `excludedNamespaces` under the `allowedDockerRegistries` violations for gatekeeper. This can be done by modifying `chart/values.yaml` file or passing an override file with the values set as seen below. This is for development purposes only: production should not allow containers in the `istio-system` namespace to be pulled from outside of Registry1. 
+**Note1:** When deploying to k3d, istio-system should be added from `excludedNamespaces` under the `allowedDockerRegistries` violations for gatekeeper. This can be done by modifying `chart/values.yaml` file or passing an override file with the values set as seen below. This is for development purposes only: production should not allow containers in the `istio-system` namespace to be pulled from outside of Registry1. 
 
 ```yaml
 gatekeeper:
@@ -212,9 +212,23 @@ gatekeeper:
     violations:
       allowedDockerRegistries:
         match:
-          excludedNamespaces: 
-            - istio-system # allows creation for loadbalancer pods for various ports and various vendor loadbalancers
+          excludedNamespaces:
+            # Allows load balancer images for k3d from public repo
+            - istio-system
+      hostNetworking:
+        match:
+          excludedNamespaces:
+            # Allows load balancer containers to map ports for k3d
+            - istio-system
 ```
+**Note2:** The information in this note is simply to give you awareness in advance. You should create local directory on your workstation where you store your helm values override files. Development changes made in the code for testing could accidentally be committed. That is why you should create a separate local directory to hold your override values for testing. The location can be anywhere on your workstation but it is most convenient to place them in a sibling directory next to the BigBang repos. Below is an example directory structure. The directory names are fake (for example only). Other documents will give more specific detail as needed.
+  ```text
+  ├── BigBangCodeRepo/
+  └── overrides/
+      ├── override-values-1.yaml
+      ├── override-values-2.yaml
+      └── override-values-3.yaml
+  ```
 
 ## Addendum
 
@@ -312,7 +326,7 @@ docker network inspect k3d-k3s-default | jq .[0].IPAM.Config[0]
 ```shell
 kubectl create -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml
 kubectl create -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml
-cat <<EOF | > metallb-config.yaml
+cat << EOF > metallb-config.yaml
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -381,7 +395,21 @@ sudo vim /etc/hosts
     ```
   - With these DNS settings in place you will now be able to reach the external *.bigbang.dev URLs from this EC2 instance.
 
-  - To reach outside the EC2 instance use either SSH or SSHUTTLE commands to specify a local port for Dynamic application-level port forwarding (ssh -D) and utilize Firefox's built in SOCKS proxy configuration to route DNS and web traffic through the application-level port forward from the SSH command.
+  - To reach outside the EC2 instance use either SSH or SSHUTTLE commands to specify a local port for Dynamic application-level port forwarding (ssh -D). Example
+      ```shell
+      sshuttle --dns -vr ubuntu@$EC2_PRIVATE_IP 172.31.0.0/16 --ssh-cmd 'ssh -i ~/.ssh/your.pem -D 127.0.0.1:12345'
+      ```
+   - and utilize Firefox's built in SOCKS proxy configuration to route DNS and web traffic through the application-level port forward from the SSH command.
+      1. Open Firefox browser
+      1. Click on hamburger menu in upper right corner and select ```Settings```
+      1. At the bottom of ```Settings``` page in the ```Network Settings``` section select ```Settings```
+      1. Select ```Manual proxy configuration``` and the following values
+          ```
+          SOCKS Host:  localhost
+          Port:  12345
+          ```
+          and select SOCKS v5
+      1. Select ```Proxy DNS when using SOCKS v5```
 
 ### Amazon Linux 2
 

docs/developer/package-integration.md

@@ -88,9 +88,42 @@
 
 1. Add your packages name to the ORDERED_HELMRELEASES list in scripts/deploy/02_wait_for_helmreleases.sh.
 
-1. Verify your Package works when deployed through bigbang. While testing you should use a git branch instead of tag, (i.e. replace tag: "1.2.3-bb.0" with branch: "main").  After you have tested BigBang integration, tag the commit in your Package following the convention of {UpstreamChartVersion}-bb.{BigBangVersion} – example 1.2.3-bb.0.
-
-1. Make sure to change the chart/values.yaml file to point to the tag rather than your branch (i.e. tag: "1.2.3-bb.0" in place of branch: "bb-9999").
+1. Create an overrrides directory as a sibling directory next to the bigbang code directory. Put your override yaml files in this directory. The reason we do this is to avoid modifying the bigbang values.yaml that is under source control. You could accidentally commit it with your secrets. Avoid that mistake and create a local overrides directory. One option is to copy the tests/ci/k3d/values.yaml to make the override-values.yaml and make modifications. The file structure is like this:
+    ```text
+    ├── bigbang/
+    └── overrides/
+        ├── override-values.yaml
+        ├── registry-values.yaml
+        └── any-other-values.yaml
+    ```
+    Make the registry-values yaml like this:
+    ```yaml
+    registryCredentials:
+    - registry: registry1.dso.mil
+      username: your-name
+      password: your-pull-token
+      email: xxx@xxx.xxx
+    ```
+    You will use these files as arguments in your helm commands.
+
+1. Verify your Package works when deployed through bigbang. See the instructions below in the ```BigBang Development and Testing Cycle``` for the manual ```imperative``` way to deploy with helm upgrade commands. While testing you should use your package git branch instead of a tag. If you don't null the tag your branch will not get deployed. example:
+    ```yaml
+    addons:
+      app1:
+        git:
+          tag: null
+          branch: "999-your-dev-branch-name"
+    ```
+
+1. After you have tested BigBang integration complete a Package MR and contact the codeowners to create a release tag. Package release tags follow the naming convention of {UpstreamChartVersion}-bb.{BigBangVersion} – example 1.2.3-bb.0.
+
+1. Make sure to change the chart/values.yaml file to point to the new release tag rather than your dev branch (i.e. tag: "1.2.3-bb.0" in place of branch: "999-your-dev-branch-name"). example:
+    ```yaml
+    addons:
+      app1:
+        git:
+          tag: "1.2.3-bb.0"
+    ```
 
 1. When you are done developing the BigBang chart features for your Package make a merge request in "Draft" status and add a label corresponding to your package name (must match the name in `values.yaml`). Also add any labels for dependencies of the package that are NOT core apps. The merge request will start a pipeline and use the labels to determine which addons to deploy. Fix any errors that appear in the pipeline. When the pipeline has pass and the MR is ready take it out of "Draft" and add the `status::review` label. Address any issues raised in the merge request comments.
 
@@ -100,7 +133,7 @@ There are two ways to test BigBang, imperative or GitOps with Flux.  Your initia
 
 ### Imperative
 
-You can manually deploy bigbang with helm command line. With this method you can test local code changes without committing to a repository. Here are the steps that you can iterate with "code a little, test a little".  From the root of your local bigbang repo:
+You can manually deploy bigbang with helm command line. With this method you can test local code changes without committing to a repository. Here are the steps that you can iterate with "code a little, test a little". You should have previously created the ../overrides directory as described in step #10 above. From the root of your local bigbang repo:
 
 ```shell
 # Deploy with helm while pointing to your override values files. 
@@ -123,7 +156,7 @@ hack/remove-ns-finalizer.sh istio-system
 
 ### GitOps with Flux
 
-You can deploy your development code the same way a customer would deploy using GitOps. You must commit any code changes to your development branches because this is how GitOps works. There is a [customer template repository](https://repo1.dso.mil/platform-one/big-bang/customers/template) that has an example template for how to deploy using BigBang. You can create a branch from one of the other developer's branch or start clean from the master branch. Make the necessary modifications as explained in the README.md. The setup information is not repeated here. This is a public repo so DO NOT commit unencrypted secrets. Before committing code it is a good idea to manually run `helm template` and a `helm install` with dry run.  This will reveal many errors before you make a commit. Here are the steps you can iterate:
+Using GitOps for development is NOT recommended. Your development iteration cycle time will be slowed down waiting for flux reconciliation. This is not an efficient use of your time. These instructions are included here for informational purposes. You can deploy your development code the same way a customer would deploy using GitOps. You must commit any code changes to your development branches because this is how GitOps works. There is a [customer template repository](https://repo1.dso.mil/platform-one/big-bang/customers/template) that has an example template for how to deploy using BigBang. You must fork or copy this repo to your own private repo. Make the necessary modifications as explained in the README.md. The setup information is not repeated here. Before committing code it is a good idea to manually run `helm template` and a `helm install` with dry run. This will reveal many errors before you make a commit. Here are the steps you can iterate:
   
 ```shell
 # Verify chart code before committing