Merge branch 'master' into velero-restore

chart/templates/gitlab/secret-objectstore.yaml

@@ -20,8 +20,12 @@ stringData:
       {{- end }}
     registry: |-
       s3:
+        {{- $global := .Values.addons.gitlab.values.global | default dict }}
+        {{- $registryConfig := $global.registry | default dict }}
         {{- if .Values.addons.gitlab.objectStorage.bucketPrefix }}
         bucket: {{ .Values.addons.gitlab.objectStorage.bucketPrefix }}-gitlab-registry
+        {{- else if hasKey $registryConfig "bucket" }}
+        bucket: {{ .Values.addons.gitlab.values.global.registry.bucket }}
         {{- else }}
         bucket: gitlab-registry
         {{- end }}

chart/templates/logging/fluentbit/values.yaml

@@ -14,6 +14,9 @@ imagePullSecrets:
 networkPolicies:
   enabled: {{ .Values.networkPolicies.enabled }}
 
+istio:
+  enabled: {{ .Values.istio.enabled }}
+
 {{- if .Values.monitoring.enabled }}
 serviceMonitor:
   enabled: true

chart/templates/velero/values.yaml

@@ -11,6 +11,9 @@ istio:
 monitoring:
   enabled: {{ .Values.monitoring.enabled }}
 
+networkPolicies:
+  enabled: {{ .Values.networkPolicies.enabled }}
+
 imagePullSecrets:
 - name: private-registry
 

chart/values.yaml

@@ -301,7 +301,7 @@ fluentbit:
   git:
     repo: https://repo1.dso.mil/platform-one/big-bang/apps/core/fluentbit.git
     path: "./chart"
-    tag: "0.15.8-bb.3"
+    tag: "0.15.8-bb.4"
 
   # -- Flux reconciliation overrides specifically for the Fluent-Bit Package
   flux: {}
@@ -436,7 +436,7 @@ addons:
     git:
       repo: https://repo1.dso.mil/platform-one/big-bang/apps/core/authservice.git
       path: "./chart"
-      tag: "0.4.0-bb.5"
+      tag: "0.4.0-bb.6"
 
     # -- Flux reconciliation overrides specifically for the Authservice Package
     flux: {}

docs/developer/develop-package.md

@@ -38,20 +38,6 @@ Package is the term we use for an application that has been prepared to be deplo
    description: GitLab Runner
    ```
 
-1. Some upstream helm charts have a file called "requirements.yaml" that contains internet links to external dependencies. Point all external links to the local file system. Also delete the dependency lock file that was generated during the previous step. Again, BigBang Packages must be able to be installed in an air-gap without any internet connectivity. It will look like this example from Gitlab.
-
-   ```yaml
-   dependencies:
-   - name: gitlab
-     version: '*.*.*'
-     repository: file://./charts/gitlab
-   - name: cert-manager
-     version: 0.10.1
-     repository: file://./charts/cert-manager-v0.10.1.tgz
-     condition: certmanager.install
-     alias: certmanager
-   ```
-
 1. In the values.yaml replace public upstream images with IronBank hardened images. The image version should be compatible with the chart version. Here is a command to identify the images that need to be changed.
 
    ```shell
@@ -74,25 +60,10 @@ Package is the term we use for an application that has been prepared to be deplo
 
 1. Add a VirtualService if your application has a back-end API or a front-end GUI. Create the VirtualService in the sub-directory  "chart/templates/bigbang/VirtualService.yaml". You will need to manually create the "bigbang" directory. It is convenient to copy VirtualService code from one of the other Packages and then modify it. You should be able to load the application in your browser if all the configuration is correct.
 
-1. Add a continuous integration (CI) pipeline to the Package. The [pipeline documentation](https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates#using-the-infrastructure-in-your-package-ci-gitlab-pipeline) provides a great rundown of how to reference the package pipeline template and handle more complicated situations like dependencies.
+1. Add NetworkPolices templates in the sub-directory "chart/templates/bigbang/networkpolicies/*.yaml". The intent is to lock down all ingress and egress traffic except for what is required for the application to function properly. Start with a deny-all policy and then add additionl policies to open traffic as needed. Refer to the other Packages code for examples.
 
-1. Add CI pipeline test values to the Package. A Package should be able to be deployed by itself, independently from the BigBang chart. The Package pipeline takes advantage of this to run a Package pipeline test. Create a tests directory and a test yaml file at "tests/test-values.yaml".  Set any values that are necessary for this test to pass.  The pipeline automatically creates an image pull secret "private-registry-mil".  All you need to do is reference that secret in your test values. You can view the pipeline status from the Repo1 console. Keep iterating on your Package code and the test code until the pipeline passes. Refer to the test-values.yaml from other Packages to get started. The repo structure must match what the CI pipeline code expects.
+1. Add a continuous integration (CI) pipeline to the Package. A Package should be able to be deployed by itself, independently from the BigBang chart. The Package pipeline takes advantage of this to run a Package pipeline test. The package testing is done with a helm test library. Reference the [pipeline documentation](https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates#using-the-infrastructure-in-your-package-ci-gitlab-pipeline) for how to create a pipeline and also [detailed instructions](https://repo1.dso.mil/platform-one/big-bang/apps/library-charts/gluon/-/blob/master/docs/bb-tests.md) in the gluon library. Instructions are not repeated here.
 
-   ```yaml
-   |-- .gitlab-ci.yml
-   |-- chart
-   |   |-- Chart.yml
-   |   |-- charts
-   |   |-- templates
-   |   `-- values.yml
-   `-- tests
-       |-- cypress
-       |   `-- integration
-       |       `-- health.spec.js
-       |-- cypress.json
-       |-- main-test-gateway.yml
-       `-- test-values.yml
-   ```
 
 1. Documentation for the Package should be included. A "docs" directory would include all detailed documentation. Reference other that Packages for examples.
 

docs/developer/developer-documentation.md

@@ -9,10 +9,6 @@ The [BigBang Charter](https://repo1.dso.mil/platform-one/big-bang/bigbang/-/tre
 Join Mattermost channels to ask questions and communicate with the team. The team also has a daily stand up at 8:30 am MST. The link for the stand up is usually found in the channel headers. Here is the list of relevant Mattermost channels for BigBang development:  
 [public Big Bang](https://chat.il2.dso.mil/platform-one/channels/team---big-bang)  
 [private Big Bang - by invitation only](https://chat.il2.dso.mil/platform-one/channels/team---bigbang)  
-[Big Bang add-ons](https://chat.il2.dso.mil/platform-one/channels/substream---bb---add-ons)  
-[Big Bang packages](https://chat.il2.dso.mil/platform-one/channels/substream---bb---packages)
-[Big Bang pipelines](https://chat.il2.dso.mil/platform-one/channels/substream---bb---umbrella-pipelines)  
-[Big Bang package pipelines](https://chat.il2.dso.mil/platform-one/channels/substream---bb---package-pipelines)  
 
 ## Big Bang Framework
 

docs/developer/development-environment.md

@@ -4,7 +4,7 @@ BigBang developers use [k3d](https://k3d.io/), a lightweight wrapper to run [k3s
 
 It is not recommend to run k3d with BigBang on your local computer. BigBang can be quite resource-intensive and it requires a huge download bandwidth for the images. It is best to use a remote k3d cluster running on an AWS EC2 instance. If you do insist on running k3d locally you should disable certain packages before deploying. You can do this in the values.yaml file by setting the package deploy to false. One of the packages that is most resource-intensive is the logging package. And you should create a local image registry cache to minimize the amount of image downloading. A script that shows how to create a local image cache is in the [BigBang Quick Start](https://repo1.dso.mil/platform-one/quick-start/big-bang/-/blob/master/init.sh)
 
-There are 2 methods to create a remote k3d cluster. Manually or with IaC/CaC code. The manual steps are in this page. Here is the automated [IaC/CaC](https://repo1.dso.mil/platform-one/big-bang/terraform-modules/k3d-dev-env/-/tree/dev) code and instructions. You can use whichever one you prefer. It would be a good idea to get a live demonstration by someone who already knows how to do it. You can also watch the [first half of this T3](https://confluence.il2.dso.mil/download/attachments/10161790/T3%20Eric%20and%20Zack.mp4) showing a Big Bang deployment or start this T3 around 17:45 to get a better handle on how BigBang works. We strive to make the documentation as good as possible but it is hard to keep it up-to-date and there are still pitfalls and gotchas.
+There are 2 methods to create a remote k3d cluster. Manually or with IaC/CaC code. For new bigbang developers the manual way is recommended so that you understand how it works. The manual steps are in this page. Here is the automated [IaC/CaC](https://repo1.dso.mil/platform-one/big-bang/terraform-modules/k3d-dev-env/-/tree/dev) code and instructions. This code has not been maintained and might not work. It would be a good idea to get a live demonstration by someone who already knows how to do it. You can also watch the [first half of this T3](https://confluence.il2.dso.mil/download/attachments/10161790/T3%20Eric%20and%20Zack.mp4) showing a Big Bang deployment or start this T3 around 17:45 to get a better handle on how BigBang works. We strive to make the documentation as good as possible but it is hard to keep it up-to-date and there are still pitfalls and gotchas.
 
 ## Prerequisites
 
@@ -32,16 +32,16 @@ Create an Ubuntu EC2 instance using the AWS console with the following attribute
 - User Data (as Text):
 
 ```shell
-    MIME-Version: 1.0
-    Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
+MIME-Version: 1.0
+Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
     
-    --==MYBOUNDARY==
-    Content-Type: text/x-shellscript; charset="us-ascii"
+--==MYBOUNDARY==
+Content-Type: text/x-shellscript; charset="us-ascii"
 
-    #!/bin/bash
-    # Set the vm.max_map_count to 262144. 
-    # Required for Elastic to run correctly without OOM errors.
-    sysctl -w vm.max_map_count=262144
+#!/bin/bash
+# Set the vm.max_map_count to 262144. 
+# Required for Elastic to run correctly without OOM errors.
+sysctl -w vm.max_map_count=262144
 ```
 
 - 50 Gigs of disk space
@@ -107,7 +107,8 @@ k3d cluster create \
     --k3s-server-arg "--disable=metrics-server" \
     --k3s-server-arg "--tls-san=$EC2_PUBLIC_IP" \
     --port 80:80@loadbalancer \
-    --port 443:443@loadbalancer
+    --port 443:443@loadbalancer \
+    --api-port 6443
 ```
 
 **_Optionally_** you can set your image pull secret on the cluster so that you don't have to put your credentials in the code or in the command line in later steps
@@ -140,7 +141,8 @@ k3d cluster create \
     --k3s-server-arg "--disable=metrics-server" \
     --k3s-server-arg "--tls-san=$EC2_PUBLIC_IP" \
     --port 80:80@loadbalancer \
-    --port 443:443@loadbalancer
+    --port 443:443@loadbalancer \
+    --api-port 6443
 ```
 
 Here is an explanation of what we are doing with this command:
@@ -154,6 +156,7 @@ Here is an explanation of what we are doing with this command:
 - `--port 443:443@loadbalancer` Exposes the cluster on the host on port 443
 - `--volume ~/.k3d/p1-registries.yaml:/etc/rancher/k3s/registries.yaml` volume mount image pull secret config for k3d cluster.
 - `--volume /etc/machine-id:/etc/machine-id` volume mount so k3d nodes have a file at /etc/machine-id for fluentbit DaemonSet.
+- `--api-port 6443` port that your k8s api will use. 6443 is the standard default port for k8s api
 
 **STEP 3:**  
 Test the cluster from your local workstation. Copy the contents of the k3d kubeconfig from the EC2 instance to your local workstation. Do it manually with copy and paste.
@@ -201,7 +204,8 @@ k3d cluster create \
     --k3s-server-arg "--disable=metrics-server" \
     --k3s-server-arg "--tls-san=$EC2_PUBLIC_IP" \
     --port 80:80@loadbalancer \
-    --port 443:443@loadbalancer
+    --port 443:443@loadbalancer \
+    --api-port 6443
 ```
 
 Then on your workstation edit the kubeconfig with the EC2 private ip. In a separate terminal window start a tunnel session with sshuttle using the EC2 public IP.
@@ -308,16 +312,16 @@ aws ec2 authorize-security-group-ingress \
 # Create userdata.txt
 # https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/
 cat << EOF > userdata.txt
-    MIME-Version: 1.0
-    Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
+MIME-Version: 1.0
+Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
 
-    --==MYBOUNDARY==
-    Content-Type: text/x-shellscript; charset="us-ascii"
+--==MYBOUNDARY==
+Content-Type: text/x-shellscript; charset="us-ascii"
 
-    #!/bin/bash
-    # Set the vm.max_map_count to 262144.
-    # Required for Elastic to run correctly without OOM errors.
-    sysctl -w vm.max_map_count=262144
+#!/bin/bash
+# Set the vm.max_map_count to 262144.
+# Required for Elastic to run correctly without OOM errors.
+sysctl -w vm.max_map_count=262144
 EOF
 
 # Create new instance

docs/developer/package-integration.md

@@ -4,12 +4,13 @@
 
 1. Create a directory for your package at `chart/templates/<your-package-name>`
 
-1. Inside this folder will be 3 helm template files. You can copy one of the other package folders and tweak the code for your package. Gitlab is a good example to reference because it is one of the more complicated Packages. Note that the Istio VirtualService comes from the Package and is not created in the BigBang chart. The purpose of these helm template files is to create an easy-to-use spec for deploying supported applications. Reasonable and safe defaults are provided and any needed secrets are auto-created. We accept the trade off of easy deployment for complicated template code. More details are in the following steps.
+1. Inside this folder will be various helm template files. The rule is one document per yaml file. You can copy one of the other package folders and tweak the code for your package. Gitlab is a good example to reference because it is one of the more complicated Packages. Note that the Istio VirtualService comes from the Package and is not created in the BigBang chart. The purpose of these helm template files is to create an easy-to-use spec for deploying supported applications. Reasonable and safe defaults are provided and any needed secrets are auto-created. We accept the trade off of easy deployment for complicated template code. More details are in the following steps.
 
    ```shell
    gitrepository.yaml  # Flux GitRepository. Is configured by BigBang chart values.
    helmrelease.yaml    # Flux HelmRelease. Is configured by BigBang chart values.
    namespace.yaml      # Contains the namespace and any needed secrets
+   secret-*.yaml       # various template files that create any needed k8s secrets
    values.yaml         # Implements all the BigBang customizations of the package and passthrough for values.
    ```
 
@@ -47,7 +48,7 @@
 
    ```
 
-1. More details about namespace.yaml: This template is where the code for secrets go. Typically you will see secrets for imagePullSecret, sso, and database. These secrets are a BigBang chart enhancement.  They are created conditionally based on what the user enables in the config.
+1. More details about secret-*.yaml: The secret template is where the code for secrets go. Typically you will see secrets for imagePullSecret, sso, database, and possibly object storage. These secrets are a BigBang chart enhancement. They are created conditionally based on what the user enables in the config.
 
 1. Edit the chart/templates/values.yaml.  Add your Package to the list of Packages.  Just copy one of the others and change the name. This supports adding chart values from a secret. Pay attention to whether this is a core Package or an add-on package, the toYaml values are different for add-ons. This template allows a Package to add chart values that need to be encrypted in a secret.
 
@@ -98,14 +99,15 @@ There are two ways to test BigBang, imperative or GitOps with Flux.  Your initia
 1. **Imperative:** you can manually deploy bigbang with helm command line. With this method you can test local code changes without committing to a repository. Here are the steps that you can iterate with "code a little, test a little".  From the root of your local bigbang repo:
 
    ```shell
-   # Deploy with helm while pointing to your test values files
+   # Deploy with helm while pointing to your override values files. 
+   # In this example the files are placed on your workstation at ../overrides/*
    # Bigbang packages should create any needed secrets from the chart values
-   # Ff you have the values file encrypted with sops, temporarily decrypt it
-   helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../customers/template/dev/configmap.yaml -f ./chart/ingress-certs.yaml -f ../customers/template/dev/registry-values.enc.yaml
+   # If you have the values file encrypted with sops, temporarily decrypt it
+   helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../overrides/override-values.yaml -f ../overrides/registry-values.yaml -f ./chart/ingress-certs.yaml
  
    # Conduct testing
    # If you make code changes you can run another helm upgrade to pick up the new changes
-   helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../customers/template/dev/configmap.yaml -f ./chart/ingress-certs.yaml -f ../customers/template/dev/registry-values.enc.yaml
+   helm upgrade -i bigbang ./chart -n bigbang --create-namespace -f ../overrides/override-values.yaml -f ../overrides/registry-values.yaml -f ./chart/ingress-certs.yaml
  
    # Tear down
    helm delete bigbang -n bigbang