Merge branch 'docs-extended' into 'master'

Docs extended

Large amount of Big Bang documentation in /docs folder

See merge request platform-one/big-bang/umbrella!93

docs/0_release_notes.md

+# Big Bang Release Notes
+
+Table of Contents
+
+- [Big Bang Release Notes](#big-bang-release-notes)
+  - [Version 0.0.2](#version-002)
+    - [What's New](#whats-new)
+    - [Bug Fixes](#bug-fixes)
+
+---
+
+## Version 0.0.2
+
+Date: December 8th, 2020
+
+### What's New
+
+- Initial release of Big Bang
+
+### Bug Fixes
+
+- None
+
+---

docs/1_overview.md

+# Big Bang Overview
+
+Table of Contents
+
+- [Big Bang Overview](#big-bang-overview)
+  - [Key Concepts](#key-concepts)
+    - [Big Bang](#big-bang)
+    - [Big Bang Umbrella](#big-bang-umbrella)
+    - [Flux v2](#flux-v2)
+    - [GitOps](#gitops)
+    - [SOPS](#sops)
+    - [Kustomize](#kustomize)
+    - [Helm](#helm)
+  - [Packages](#packages)
+  - [Architecture](#architecture)
+    - [Configuration](#configuration)
+    - [Deployment](#deployment)
+    - [Diagram](#diagram)
+
+---
+
+## Key Concepts
+
+### Big Bang
+
+Big Bang is a declarative, continuous delivery tool for core DoD hardened and approved [packages](#packages) into a Kubernetes cluster.  Big Bang follows a [GitOps](#gitops) approach to configuration management, using [Flux v2](#flux-v2) to reconcile Git with the cluster.  Environments (e.g. dev, prod) and packages (e.g. istio) can be fully configured to suit the deployment needs.
+
+### Big Bang Umbrella
+
+The umbrella is a sub-component of Big Bang that controls the deployment of all the packages.  This is deployed via a Helm chart.
+
+### Flux v2
+
+[Flux v2](https://toolkit.fluxcd.io/) is the tool used by Big Bang to reconcile sources of configuration in Git with actual code deployment.  Flux includes resources to reconcile Git repositories, [Kustomizations](https://kustomize.io/) to Kubernetes manifests, and [Helm Chart](https://helm.sh/docs/topics/charts/) releases.  Flux automatically monitors changes in these items and reconciles with the Kubernetes cluster.
+
+### GitOps
+
+[GitOps](https://www.weave.works/technologies/gitops/) is a way to declaratively manage a Kubernetes cluster using a single source of truth in Git.  All Kubernetes configuration is stored in Git.  And, a tool (e.g. [Flux](#flux-v2)) is used to automatically monitor and reconcile the configuration with the running Kubernetes cluster.  This allows all changes to the cluster to be version controlled and pipelined.
+
+### SOPS
+
+[SOPS](https://github.com/mozilla/sops) is a way to encrypt values in YAML, JSON, ENV, INI, or BINARY files so that sensitive data can be securely shared.  Big Bang uses SOPS to encrypt pull secrets, certificates, and other secrets so that it can be stored into the Git repository securely.  Big Bang (via Flux) uses the private key, stored in a secure area (e.g. KMS) to decrypt the secrets and deploy them into the cluster for use.
+
+### Kustomize
+
+[Kustomize](https://kustomize.io/) is a tool that assists with configuration management for Kubernetes manifest files.  It uses overlays and patching to customize a base set of manifests for deployment.  Big Bang uses Kustomize to setup the environment (e.g. dev, prod) configuration.
+
+### Helm
+
+[Helm](https://helm.sh/) is a tool that assists with configuration management for Kubernetes manifest files.  It uses charts, which includes templates and values, to create a set of manifests for deployment. Big Bang uses Helm charts to deploy all of its packages.
+
+---
+
+## Packages
+
+The following packages are included in a default deployment of Big Bang:
+
+|Name|Description|Helm Chart Repo|
+|--|--|--|
+|[Istio](https://istio.io/)|[Service Mesh](https://www.redhat.com/en/topics/microservices/what-is-a-service-mesh)|[Link](https://repo1.dsop.io/platform-one/big-bang/apps/core/servicemesh.git)|
+|Cluster Auditor| |[Link](https://repo1.dsop.io/platform-one/big-bang/apps/core/cluster-auditor.git)|
+|[Open Policy Agent Gatekeeper](https://github.com/open-policy-agent/gatekeeper)| Policy Management | [Link](https://repo1.dsop.io/platform-one/big-bang/apps/core/policy.git)|
+|[Elastic Cloud on Kubernetes (ECK)](https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html)| Logging; Incl. Elasticsearch & Kibana |[Link](https://repo1.dsop.io/platform-one/big-bang/apps/core/elasticsearch-kibana.git)|
+|[Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/)| Monitoring |[Link](https://repo1.dsop.io/platform-one/big-bang/apps/core/monitoring.git)|
+|[Twistlock](https://www.paloaltonetworks.com/prisma/cloud)| Security Scanning |[Link](https://repo1.dsop.io/platform-one/big-bang/apps/security-tools/twistlock.git)|
+
+In addition, the following packages can be added onto the default deployment:
+
+|Name|Description|Helm Chart Repo|
+|--|--|--|
+|[Argo CD](https://argoproj.github.io/argo-cd/)| Continuous Delivery |[Link](https://repo1.dsop.io/platform-one/big-bang/apps/core/argocd.git)|
+|Auth Service|Single Sign On|[Link](https://repo1.dsop.io/platform-one/big-bang/apps/sandbox/authservice.git)
+
+---
+
+## Architecture
+
+The diagram below shows a typcial deployment of Big Bang into a Kubernetes cluster.  The items in blue require user interaction.  The remaining items are automated by Big Bang.
+
+### Configuration
+
+1. The user must [setup an encryption key pair](./3_encryption.md) for SOPS and store the private key securely (e.g. KMS).  This shoud **NOT** be stored in Git.
+1. The user should then [configure Big Bang](./4_configuration.md) values and secrets for the targeted Kubernetes cluster.
+1. All secrets should be encrypted with SOPS to protect them.
+1. Once all of the configuration has been completed, it must be pushed to a Git repository.
+
+### Deployment
+
+1. With everything in Git, the user can [deploy BigBang](./5_deployment.md) using a Kubernetes manifest.
+1. The manifest holds two Flux resources, one pointing to the Git repository holding the custom environment, and one telling Flux to run Kustomize on a targeted folder within the repo.
+   1. The repository is reconciled first, pulling the files from Git.
+   1. Next, Kustomize is run on the environment configuration
+      1. The Kustomize files use Big Bang's Git repo as a base before applying overlays and patches for the configuration.
+      1. Flux uses SOPS to decrypt any secrets before deploying the manifests
+      1. After completing the Kustomization process, Flux deploys two ConfigMaps, two Secrets, and flux resources for Big Bang Umbrella
+1. Big Bang's flux resources include a Git repository holding the umbrella Helm chart and a Helm Release resource that tells Flux how to deploy the Helm chart.
+   1. The repository is reconciled first, pulling the Helm chart from Git.
+   1. The Helm Release will check for the Helm chart and the Secrets / ConfigMaps deployed before performing a Helm install
+1. Once the Helm release deploys the Helm chart for Big Bang umbrella, each package that is enabled will have a Flux Git Repository and Helm Release resource deployed.
+1. All of the package Git repos containing Helm charts will be pulled so that Flux can reconcile dependencies.
+1. Each package's Helm Release has dependencies built in.  Flux will reconcile these dependencies and deploy the Helm chart for the package once all of the dependencies are ready.
+1. Once all of the packages are ready, Big Bang will monitor Git periodically for changes and reconcile using the methods above.
+
+### Diagram
+
+```mermaid
+graph TD
+    style SetupSOPS fill:#00758f
+    style CustomSecrets fill:#00758f
+    style CustomVals fill:#00758f
+    style EncryptSecrets fill:#00758f
+    style PushToGit fill:#00758f
+    style DeployBB fill:#00758f
+
+    SetupSOPS(Setup SOPS keys) --> EncryptSecrets(Encrypt secrets)
+    SetupSOPS --> CustomVals(Customize values)
+    CustomSecrets(Customize secrets) --> EncryptSecrets
+    CustomVals --> PushToGit(Push customization to Git)
+    EncryptSecrets --> PushToGit
+    PushToGit --> DeployBB(Deploy BigBang Manifest)
+
+    DeployBB --> KustResEnv[[Deploy Environment Kustomization Resource]]
+    KustResEnv --> HelmResUmb[[Deploy Umbrella Helm Release Resource]]
+    DeployBB --> GitResEnv[[Deploy Environment Git Repository Resource]]
+    KustResEnv --> GitResUmb[[Deploy Umbrella Git Repository Resource]]
+    KustResEnv --> SOPS
+
+    GitResEnv --> PullEnv[[Pull environment]]
+    PullEnv --> SOPS[[SOPS Decrypt secrets]]
+    SOPS --> DeployVals[[Deploy ConfigMap and Secrets]]
+
+    GitResUmb --> PullUmb[[Pull Umbrella Helm Chart]]
+    PullUmb --> DeployUmb[[Deploy Umbrella w/ Helm Chart]]
+    HelmResUmb --> DeployUmb
+    DeployVals --> DeployUmb
+
+    DeployUmb --> PackGit[[Deploy Package Git Repository Resources]]
+    PackGit --> PackPull[[Pull Package Helm Charts]]
+    PackPull --> PackDep[[Resolve Package Dependencies]]
+    PackDep --> PackReady{Package Ready?}
+    PackReady --Yes--> PackDeploy[[Deploy Package w/ Helm Chart]]
+    PackReady --No--> PackDep
+```

docs/2_getting_started.md

+# Big Bang Getting Started
+
+Table of Contents
+
+- [Big Bang Getting Started](#big-bang-getting-started)
+  - [System Requirements](#system-requirements)
+  - [Flux Installation](#flux-installation)
+  - [Configuration Template](#configuration-template)
+    - [Overview](#overview)
+    - [Environments (Multi-cluster)](#environments-multi-cluster)
+  - [Next Steps](#next-steps)
+
+## System Requirements
+
+- Admin tools
+  - [Docker](https://docs.docker.com/engine/install/)
+  - [Flux CLI](https://toolkit.fluxcd.io/get-started/#install-the-flux-cli):     `curl -s https://toolkit.fluxcd.io/install.sh | sudo bash`
+  - [Git](https://git-scm.com/download/)
+  - [Helm](https://helm.sh/docs/intro/install/)
+  - [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
+  - [Kustomize](https://kubectl.docs.kubernetes.io/installation/kustomize/)
+  - [SOPS](https://github.com/mozilla/sops/releases)
+
+- Kubernetes cluster
+  > CPU, Memory, and Disk space vary depending on what is enabled.  If everything is enabled, the following is the minimum viable setup:
+  - vCores - 8
+  - Memory - 32GB
+  - Disk Space - 20GB
+
+## Flux Installation
+
+[Flux v2](https://toolkit.fluxcd.io/) must be installed into the Kubernetes cluster before deploying Big Bang.  There are three options for doing this:
+
+1. (Recommended) Deploy officially through [Iron Bank](registry1.dsop.io)
+
+    ```bash
+    # The script will do the following:
+    #   Check flux prerequisites
+    #   Interactively login to Iron Bank and store credentials in Secret
+    #   Install flux into Kubernetes cluser using Iron Bank repo
+    #   Remove Iron Bank credentials from cluster
+    hack/flux-install.sh
+    ```
+
+1. Deploy unofficially through [Big Bang's Repo](https://repo1.dsop.io/platform-one/big-bang/apps/sandbox/fluxv2/container_registry)
+
+   ```bash
+   flux install --registry registry.dsop.io/platform-one/big-bang/apps/sandbox/fluxv2
+   ```
+
+1. Deploy for development through [DockerHub](https://hub.docker.com/search?q=fluxcd)
+
+   ```bash
+   flux install
+   ```
+
+## Configuration Template
+
+A [template for configuring multi-cluster environments](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang/-/tree/master/bigbang) is provided to assist with getting Big Bang setup correctly.  You should make a copy of the `./bigbang` folder from the [customer Big Bang repository](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang) and place it into a Git repository under your control.
+
+### Overview
+
+The template is setup to allow you to customize the Big Bang deployment for your environment.  The template contains support for two cluster environments, `dev` and `prod`, but can be easily extended to more.  At a minimum, the following must be configured in the template for a properly working deployment:
+
+- Big Bang version - allows you to control when to upgrade
+- Environment Git repository - where your copy of the configuration template is located
+- Hostname - the base domain to use for your packages
+- Reference to SOPS private key - See the [encryption help](3_encryption.md) for more information.
+- Iron Bank pull credentials
+
+The [Configuration Template help](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang/-/blob/master/bigbang/README.md) contains details on how to setup these items.
+
+If there is additional configuration you want, refer to the [configuration help](4_configuration.md) for details.
+
+### Environments (Multi-cluster)
+
+In the template, there are two folders used for each cluster environment, `base` and a named folder (e.g. `dev` or `prod`).  `base` is used as a shared folder between all the environments and a `<env>` folder is specific to an environment.  Overrides proceed as follows, with `<env>` having the highest precedence.
+
+```mermaid
+graph LR
+  pkg[Package values]-->bb[Big Bang values]-->base[`base` values]-->named[`<env>` values]
+```
+
+## Next Steps
+
+Follow the steps in the [encryption](3_encryption.md) and [configuration](4_configuration.md) documentation to setup the minimum viable configuration for deployment.

docs/3_encryption.md

+# Big Bang Encryption
+
+Table of Contents
+
+- [Big Bang Encryption](#big-bang-encryption)
+  - [SOPS](#sops)
+  - [Create Encryption Keys](#create-encryption-keys)
+    - [Samples](#samples)
+  - [Configure SOPS](#configure-sops)
+  - [Deploy Private Key](#deploy-private-key)
+    - [GPG](#gpg)
+    - [AWS KMS](#aws-kms)
+    - [GCP KMS](#gcp-kms)
+    - [Azure KeyVault](#azure-keyvault)
+    - [Hashicorp Vault](#hashicorp-vault)
+  - [Configure Big Bang](#configure-big-bang)
+
+Big Bang follows a [GitOps](https://www.weave.works/technologies/gitops/) approach to managing the Big Bang Kubernetes cluster configuration.  Using GitOps, we must securely store secrets in Git using encryption.  The private key, which is stored in key storage, is used by the continous deployment tool to decrypt and deploy the secrets for use in the cluster.
+
+## SOPS
+
+[Secrets Operations (SOPS)](https://github.com/mozilla/sops) is used to securely encrypt values in YAML, JSON, ENV, INI and BINARY formats. Secrets, such as pull credentials or certificates, should be encrypted with SOPS prior to committing into a Git repository.
+> The private key used in SOPS should **NEVER** be stored in Git along side the encrypted secrets.
+
+SOPS supports the ability to [add multiple keys](https://dev.to/stack-labs/manage-your-secrets-in-git-with-sops-common-operations-118g) to the same file so multiple key pairs can use the same secret.  This is useful for environments which may have different keys, but use the same secrets.  For each key used, SOPS writes the public key, used to encrypt, and an encrypted copy of the data to the file.  Decryption requires use of one of the private keys used.  After editing, the embedded public keys are used to re-encrypt the file for all key pairs.
+
+## Create Encryption Keys
+
+To setup Big Bang with SOPS, a key pair must be created.  The private key is used for decryption and must be securely stored but accessible to the cluster.  The public key is used for encryption.  Follow the appropriate instructions below to create your key pair.
+
+| Key Management | Key Pair Instructions | Notes |
+|--|--|--|
+| [GNU Privacy Guard (GPG)*](https://gnupg.org/) | `gpg --full-generate-key` | Use `key type` = `RSA and RSA`, `keysize` = `4096`, `expiration` = `0` |
+| [Amazon Web Services (AWS) Key Management Service (KMS)](https://aws.amazon.com/kms/) | [Link](https://github.com/mozilla/sops#2usage) | [Advanced setup help](https://github.com/mozilla/sops#26kms-aws-profiles) (e.g. roles, profiles, contexts)
+| [Google Cloud Platform (GCP) Key Management Service (KMS)](https://cloud.google.com/security-key-management) | [Link](https://github.com/mozilla/sops#encrypting-using-gcp-kms) |
+| [Hashicorp Vault](https://www.vaultproject.io/) | [Link](https://github.com/mozilla/sops#23encrypting-using-azure-key-vault) |
+
+> *GPG is not recommended for production use because the private key can be misplaced or comprimised too easily
+
+### Samples
+
+If you plan to utilize Big Bang provided samples, either in the template or in this repository, setup the following:
+
+1. Install [gpg](https://gnupg.org/download/)
+1. Import the [Big Bang development key](../hack/bigbang-dev.asc)
+   > Do **NOT** use this key for any deployment.  It is only for demonstration purposes.
+
+   ```bash
+   gpg --import <private key>
+   ```
+
+1. Validate by decrypting and opening a sample file for editing
+
+   ```bash
+   sops <filename.enc.yaml>
+   ```
+
+## Configure SOPS
+
+SOPS uses `.sops.yaml` as a configuration file for which keys to use for newly created files.  Once a file is created, the key fingerprints are stored in the file and must be re-keyed to use any changes to `.sops.yaml`.
+
+1. Follow the [SOPS instructions](https://github.com/mozilla/sops#210using-sopsyaml-conf-to-select-kmspgp-for-new-files) to configure `.sops.yaml` based on the encryption method you used.  Multiple keys of the same type can be added using the block scalar yaml construct, `>-`, and separating them by a comman and newline.
+   > If you are using the Big Bang sample files, make sure to remove the development Big Bang key.
+
+1. Add the following regex to only encrypt data in the yaml files
+
+    ```yaml
+    creation_rules:
+    - encrypted_regex: '^(data|stringData)$'
+    ```
+
+1. Save `.sops.yaml` in the root of folder of your configuration
+1. If you have existing secrets, use the following to re-key them with the configuration in `.sops.yaml`
+
+   ```bash
+   # You must have the old private key to rekey the file
+   sops updatekeys <encrypted file>
+   ```
+
+## Deploy Private Key
+
+> This must be completed before deploying Big Bang or else deploying Secrets will fail.
+
+### GPG
+
+1. Deploy your SOPS private key to a secret named `sops-gpg` in the cluster
+
+   ```bash
+   gpg --export-secret-keys --armor <new key fingerprint> | kubectl create secret generic sops-gpg -n bigbang --from-file=bigbangkey=/dev/stdin
+   ```
+
+### AWS KMS
+
+TBD - [This article](https://blog.doit-intl.com/injecting-secrets-from-aws-gcp-or-vault-into-a-kubernetes-pod-d5a0e84ba892) may help to automate secret consumption in Kubernetes.
+
+### GCP KMS
+
+TBD - [This article](https://blog.doit-intl.com/injecting-secrets-from-aws-gcp-or-vault-into-a-kubernetes-pod-d5a0e84ba892) may help to automate secret consumption in Kubernetes.
+
+### Azure KeyVault
+
+TBD - [This article](https://blog.doit-intl.com/injecting-secrets-from-aws-gcp-or-vault-into-a-kubernetes-pod-d5a0e84ba892) may help to automate secret consumption in Kubernetes.
+
+### Hashicorp Vault
+
+TBD - [This article](https://blog.doit-intl.com/injecting-secrets-from-aws-gcp-or-vault-into-a-kubernetes-pod-d5a0e84ba892) may help to automate secret consumption in Kubernetes.
+
+## Configure Big Bang
+
+Big Bang needs to know how to retrieve the private key so it can deploy the encrypted secrets from Git.  Decryption configuration is placed in the top-level manifest (e.g. `dev.yaml`, `prod.yaml`) from the [Big Bang template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang).  By default, the `Kustomization` resource uses a Secret named `sops-gpg` for the private key as shown here:
+
+```yaml
+apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
+kind: Kustomization
+metadata:
+  name: environment
+spec:
+  decryption:
+    provider: sops
+    secretRef:
+      name: sops-gpg
+```
+
+TBD - Instructions on how to update for AWS, GCP, Vault

docs/4_configuration.md

+# Big Bang Configuration
+
+Table of Contents
+
+- [Big Bang Configuration](#big-bang-configuration)
+  - [Overview](#overview)
+  - [Pre-configuration](#pre-configuration)
+  - [Minimum Viable Configuration](#minimum-viable-configuration)
+  - [Big Bang Globals](#big-bang-globals)
+    - [`hostname`](#hostname)
+    - [`registryCredentials`](#registrycredentials)
+    - [`flux`](#flux)
+    - [Package](#package)
+  - [Flux Resources](#flux-resources)
+  - [Big Bang Version](#big-bang-version)
+  - [Environment Location](#environment-location)
+  - [Registry Pull Credentials](#registry-pull-credentials)
+  - [Package settings](#package-settings)
+
+## Overview
+
+Configuration of Big Bang is achieved by overriding default values set in the package or Big Bang using the [environment template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang).  The template has a 4 potential locations for setting values: `base/secrets.enc.yaml`, `base/configmap.yaml`, `<env>/secrets.enc.yaml`, and `<env>/configmap.yaml`.  Overrides proceed as follows, with `<env>/configmap.yaml` having the highest precedence.
+
+```mermaid
+graph TD
+  pkg[Package values]
+  -->bb[Big Bang values]
+  -->base-s[`base/secrets.enc.yaml` values]
+  -->base-c[`base/configmap.yaml` values]
+  -->env-s[`<env>/secrets.enc.yaml` values]
+  -->env-c[`<env>/configmap.yaml` values]
+```
+
+In all four cases, Big Bang reads a single key named `values.yaml` that contains the data to override.  See the [Big Bang environment template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang) for examples on how to use these files to override values.
+
+## Pre-configuration
+
+Before configuring Big Bang, it is expected that you have already setup:
+
+- A Kubernetes cluster
+- A [SOPS key pair](3_encryption.md)
+- A Git repository to hold your configuration
+  - Pull credentials for the Git repository (if not public)
+- An Iron Bank robot account (for production)
+- Certificates specific to your environment (if needed)
+
+## Minimum Viable Configuration
+
+At a minimum, the following items must be configured for a default Big Bang deployment:
+
+- [Big Bang version](#big-bang-version)
+- [Environment Git repository](#environment-location)
+- [Hostname](#hostname)
+- [SOPS private key reference](3_encryption.md).
+- [Registry pull credentials](#registry-pull-credentials)
+
+The Big Bang [Environment Template](https://repo1.dsop.io/platform-one/big-bang/customers/bigbang) has placeholders for all of the above.
+
+## Big Bang Globals
+
+### `hostname`
+
+Hostname is used to override the domain of deployed packages.  This allows you to go to the DNS name of a server using the domain.  For example, if the domain is `bigbang.dev`, Kiali can be reached at `kiali.bigbang.dev`.
+
+| Key | Description | Type | Default |
+|--|--|--|--|
+| `hostname` | Domain to use for deployed servers | Domain Name | `bigbang.dev`
+
+### `registryCredentials`
+
+Registry credentials are used to pull images for Big Bang.  By default, it points to Iron Bank, but can be modified to use a private registry.  These credentials are passed down to all relevant namespaces as an image pull secret.
+
+| Key | Description | Type | Default |
+|--|--|--|--|
+| `registry` | Container registry location | Domain Name | `registry1.dsop.io`
+| `username`* | Container registry username | String | "" |
+| `password`* | User's password | String | "" |
+| `email` | User's email | Email | "" |
+> *Credentials should be SOPS encrypted
+
+### `flux`
+
+Flux settings are used to setup the default continous deployment configuration for Big Bang packages.
+
+| Key | Description | Type | Default |
+|--|--|--|--|
+| `interval` | Polling interval to check for Git or Helm chart updates | ##m##s | 2m |
+| `install.retries` | The number of retries that should be attempted on Helm chart installation failures before bailing. | int | 3 |
+| `upgrade.retries` | The number of retries that should be attempted on Helm chart upgrade failures before bailing. | int | 3 |
+| `rollback.timeout` | The time to wait for any individual Kubernetes operation (like Jobs for hooks) during the performance of a Helm rollback action. | ##m##s | 5m |
+| `rollback.cleanupOnFail` | Allows deletion of new resources created during the Helm rollback action when it fails. | Boolean | `true` |
+
+### Package
+
+Each package (e.g. `istio`, `clusterAuditor`) has configuration to control how Big Bang deploys the package
+
+| Key | Description | Type | Default |
+|--|--|--|--|
+| `enabled` | Determines if the package will get deployed or skipped | Boolean | `true` (unless its an `addon`) |
+| `git.repo` | Location of the Git repo holding the package deployment resources | URL | `https://repo1.dsop.io/platform-one/big-bang/apps/...`
+| `git.branch` | Branch to use for package deployment resources | string | `chart-release` or `release-vx.x.x` |
+| `git.commit` | SHA of specific commit to use in Git for package deployment resources | SHA | null |
+| `git.tag` | Git tag to use for package deployment resources | string | null |
+| `values` | Package specific values to configure | List of key/values pairs | {} |
+
+## Flux Resources
+
+Big Bang deploys four flux resources that can be customized:
+
+| Resource | Controls | Location |
+|--|--|--|
+| GitRepository | Environment | Top-level manifest (e.g. `dev.yaml`, `prod.yaml`)
+| Kustomization | Environment | Top-level manifest (e.g. `dev.yaml`, `prod.yaml`)
+| GitRepostiory | Big Bang Umbrella | [Link](../base/gitrepository.yaml) |
+| HelmRelease | Big Bang Umbrealla | [Link](../base/helmrelease.yaml) |
+
+In addition, each package contains its own GitRepository and HelmRelease resource that can be customized.  Look in the [Umbrella templates](../chart/templates) for the these resources.
+
+Settings for eny of these resources can be overridden by [patching](https://kubectl.docs.kubernetes.io/references/kustomize/patches/) the resource in your environment's kustomization files.  Use Flux's documentation for [GitRepository](https://toolkit.fluxcd.io/components/source/gitrepositories/), [HelmRelease](https://toolkit.fluxcd.io/components/helm/helmreleases/), and [Kustomization](https://toolkit.fluxcd.io/components/kustomize/kustomization/) to find settings for these resources.
+
+## Big Bang Version
+
+In your `kustomization.yaml` under your environment, here is an example of how to override the version of Big Bang you will use.  You can also use a tag or branch if desired.
+
+```yaml
+bases:
+- https://repo1.dsop.io/platform-one/big-bang/umbrella.git/base/?ref=v1.2.*
+patchesStrategicMerge:
+- |-
+  apiVersion: source.toolkit.fluxcd.io/v1beta1
+  kind: GitRepository
+  metadata:
+    name: umbrella-repo
+  spec:
+    ref:
+      $patch: replace
+      semver: "1.2.x"
+```
+
+> Note: You must put the version in two places, once for Kustomize to pull the right configuration base and two for Git Repository to pull the right Helm Chart.
+
+## Environment Location
+
+In your top-level `<env>.yaml` Kubernetes manifest, you would place configuration for the location of your environment.  Here is an example:
+
+```yaml
+apiVersion: source.toolkit.fluxcd.io/v1beta1
+kind: GitRepository
+metadata:
+  name: environment-repo
+  namespace: bigbang
+spec:
+  interval: 1m
+  url: https://repo1.dsop.io/platform-one/big-bang/customers/bigbang.git
+  ref:
+    branch: master
+---
+apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
+kind: Kustomization
+metadata:
+  name: environment
+  namespace: bigbang
+spec:
+  interval: 1m
+  sourceRef:
+    kind: GitRepository
+    name: environment-repo
+  path: ./bigbang/dev
+  prune: true
+  decryption:
+    provider: sops
+    secretRef:
+      name: sops-gpg
+```
+
+## Registry Pull Credentials
+
+If you have pull credentials for your docker registry, add them to `secrets.enc.yaml`.  Here is an example:
+
+> The name of the Secret must be `secrets` or `secrets-common` for Big Bang to read values from it.
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+    name: secrets-common
+stringData:
+    values.yaml: |-
+        registryCredentials:
+          username: iron-bank-user
+          password: iron-bank-password
+```
+
+You will also need to update your `kustomization.yaml` to merge with the existing secret:
+
+```yaml
+namespace: bigbang
+patchesStrategicMerge:
+- secrets.enc.yaml
+```
+
+## Package settings
+
+Besides the [global settings](#big-bang-globals), package settings are defined by the individual packet's helm charts.   You can find these by reviewing the `git.registry` setting for the package in Big Bang's [default values.yaml](../chart/values.yaml)
+
+To modify a non-sensitive package setting, add it to your `configmap.yaml`.  For sensitive information, follow the pattern for setting [registry pull credentials](#registry-pull-credentials).  Here we disable `twistlock` and set `gatekeeper`'s replicas to `1`:
+
+```yaml
+twistlock:
+  enabled: false
+
+gatekeeper:
+  values:
+    replicas: 1
+```
+
+You will also need to merge this file with the existing configmaps in `kustomization.yaml`.
+
+> The name of the ConfigMap must be `configmap` or `configmap-common` for Big Bang to read values from it.
+
+```yaml
+namespace: bigbang
+configMapGenerator:
+  - name: configmap
+    behavior: merge
+    files:
+      - values.yaml=configmap.yaml
+```

docs/5_deployment.md

+# Big Bang Deployment
+
+Table of Contents
+
+- [Big Bang Deployment](#big-bang-deployment)
+  - [GitOps](#gitops)
+  - [Installation](#installation)
+  - [Upgrade](#upgrade)
+  - [Monitor](#monitor)
+
+## GitOps
+
+Big Bang follows a [GitOps](https://www.weave.works/blog/what-is-gitops-really) approach to deployment.  All configuration changes will be pulled and reconciled with what is stored in the Git repository.  The only exception to this is the initial manifests (e.g. `dev.yaml`) which points to the Git repository and path.
+
+## Installation
+
+1. Before pushing changes to Git, validate all configuration is syntatically correct.
+
+   ```bash
+   # If everything is successful, YAML should be output
+   kustomize build ./dev
+   ```
+
+1. If you have not already done so, push configuration changes to Git
+
+   ```bash
+   git push
+   ```
+
+1. Validate the Kuberbernetes context is correct
+
+   ```bash
+   # This should match the environment you intend to deploy
+   kubectl config current-context
+   ```
+
+1. Deploy the Big Bang manifest to the cluster
+
+   ```bash
+   kubectl apply -f dev.yaml
+   ```
+
+1. [Monitor the deployment](#monitor)
+
+## Upgrade
+
+All changes to the Big Bang cluster should be made through Git.  After changes are pushed, Big Bang will automatically reconcile the difference with the cluster. 
+
+> It may take Big Bang up to 10 minutes to recognize your changes and start to deploy them.  This is based on the `interval` value set for polling.  You can force Big Bang to immediately check for changes by running the [sync.sh](../hack/sync.sh) script.
+
+Changes to values can be tested in each environment using the named folders to override values and/or point to specific repo branches or tags.  After testing, the changes can be placed into the `./base` folder if the change is shared between all environments.
+
+## Monitor
+
+The following commands will help you monitor the progress of the Big Bang deployment.  Review the [flowchart](./1_overview.md#Diagram), if needed, to understand the progression.  Use the [Troubleshooting Guide](./b_troubleshooting.md) if you have failures.
+
+1. Verify Flux is running
+
+   ```bash
+   kubectl get deploy -n flux-system
+
+   # All resources should be in the 'Ready' state
+    NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
+    source-controller         1/1     1            1           106s
+    kustomize-controller      1/1     1            1           106s
+    notification-controller   1/1     1            1           105s
+    helm-controller           1/1     1            1           106s
+   ```
+
+1. Verify the environment was pulled from the Git repo
+
+   ```bash
+   kubectl get gitrepository -A
+
+   # `environment-repo`: STATUS should be True
+    NAMESPACE   NAME               URL                                                                     READY   STATUS                                                                      AGE
+    bigbang     environment-repo   https://repo1.dsop.io/platform-one/big-bang/customers/bigbang.git       True    Fetched revision: master/185e252f4452d897531ab0314adc7a189562be31     2m7s
+   ```
+
+1. Verify the environment Kustomization properly worked
+
+   ```bash
+   kubectl get kustomizations -A
+
+   # `environment`: READY should be True
+    NAMESPACE   NAME          READY   STATUS                                                                    AGE
+    bigbang     environment   True    Applied revision: master/185e252f4452d897531ab0314adc7a189562be31   6m41s
+   ```
+
+1. Verify the ConfigMaps were deployed
+
+   ```bash
+   kubectl get configmap -l kustomize.toolkit.fluxcd.io/namespace -A
+
+   # 'configmap' and 'configmap-common' should exist
+    NAMESPACE   NAME                          DATA   AGE
+    bigbang     configmap-cch6942dk9          1      19m
+    bigbang     configmap-common-d2tgb27f56   1      19m
+   ```
+
+1. Verify the Secrets were deployed
+
+   ```bash
+   kubectl get secrets -l kustomize.toolkit.fluxcd.io/namespace -A
+
+   # 'secrets' and 'secrets-common' should exist
+    NAMESPACE   NAME                        TYPE     DATA   AGE
+    bigbang     secrets-common-kc5t8dbdfh   Opaque   1      18m
+    bigbang     secrets-mhddkt46bd          Opaque   1      18m
+   ```
+
+1. Verify the Big Bang Helm Chart was pulled
+
+   ```bash
+   kubectl get gitrepositories -A
+
+   # 'umbrella-repo' READY should be True
+    NAME            URL                                                        READY   STATUS                                                                      AGE
+    umbrella-repo   https://repo1.dsop.io/platform-one/big-bang/umbrella.git   True    Fetched revision: master/8a4a1ddd0c9edf316f5362680cf2921baf0c3451   25m
+   ```
+
+1. Verify the Big Bang Helm Chart was deployed
+
+   ```bash
+   kubectl get hr -A
+
+   # 'umbrella-helm' READY should be True
+    NAMESPACE   NAME             READY   STATUS                             AGE
+    bigbang     umbrella-helm    True    Release reconciliation succeeded   28m
+   ```
+
+1. Verify Big Bang package Helm charts are pulled
+
+   ```bash
+   kubectl get gitrepository -A
+
+   # The Git repository holding the Helm charts for each package can be seen in the URL column.
+   # The STATUS column shows the branch and tag of the revision being used.
+    NAMESPACE     NAME              URL                                                                             READY   STATUS                                                                      AGE
+    bigbang       umbrella-chart    https://repo1.dsop.io/platform-one/big-bang/apps/sandbox/umbrella               True    Fetched revision: master/3a44686520152e576a8c2c6f264876efff497c4b           8m25s
+    bigbang       logging           https://repo1.dsop.io/platform-one/big-bang/apps/core/logging.git               True    Fetched revision: release-v0.2.x/9cfe1e14c12098464ee89eb877614f781cd78fb7   8m23s
+    bigbang       certmanager       https://repo1.dsop.io/platform-one/big-bang/apps/sandbox/cert-manager.git       True    Fetched revision: release-v1.0.x/1247135baf145dcfad4a4a02ef679c48fb76d9fb   8m23s
+    bigbang       istio             https://repo1.dsop.io/platform-one/big-bang/apps/core/servicemesh.git           True    Fetched revision: chart-release/2b02a51b7950ce21bac26403fa25d09e7e3f86c3    8m23s
+    bigbang       twistlock         https://repo1.dsop.io/platform-one/big-bang/apps/security-tools/twistlock.git   True    Fetched revision: chart-release/faf038197291915713e0f213a4e35991e72f73f6    8m23s
+    bigbang       gatekeeper        https://repo1.dsop.io/platform-one/big-bang/apps/core/policy.git                True    Fetched revision: chart-release/1a5f32c8e7f672c3b5937b604e5f38eaa08ce246    8m23s
+    bigbang       monitoring        https://repo1.dsop.io/platform-one/big-bang/apps/core/monitoring.git            True    Fetched revision: release-v0.2.x/ca60bedcc106b95beb0bf9ccdc6e0e759e6fd6bf   8m23s
+    bigbang       cluster-auditor   https://repo1.dsop.io/platform-one/big-bang/apps/core/cluster-auditor.git       True    Fetched revision: chart-release/598c35670db0cbdb3a48063b2d558965afe73185    8m23s
+   ```
+
+1. Verify the packages get deployed
+
+   ```bash
+   # Use watch since it take a long time to deploy
+   watch kubectl get hr,deployments,po -A
+
+   # Flux will not attempt to deploy a package until its dependencies are ready
+   # All Helm Release resources and Pods
+    Every 2.0s: kubectl get hr,deployments,po -A                                                                            localhost: Mon Nov  9 10:14:56 2020
+
+    NAMESPACE     NAME                                                          READY    STATUS                                                 AGE
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/bigbang                    True     Release reconciliation succeeded                       64s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/gatekeeper                 True     Release reconciliation succeeded                       62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/eck-operator               False    dependency 'bigbang/gatekeeper' is not ready           62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/istio-operator             Unknown  Reconciliation in progress                             62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/istio                      False    dependency 'bigbang/istio-operator' is not ready       62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/efk                        False    dependency 'bigbang/eck-operator' is not ready         62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/logging-operator           False    dependency 'bigbang/gatekeeper' is not ready           62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/twistlock                  False    dependency 'bigbang/gatekeeper' is not ready           62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/cluster-auditor-policies   False    dependency 'bigbang/gatekeeper' is not ready           62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/cluster-auditor            False    dependency 'bigbang/gatekeeper' is not ready           62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/certmanager                True     Release reconciliation succeeded                       62s
+    bigbang       helmrelease.helm.toolkit.fluxcd.io/monitoring                 False    dependency 'bigbang/gatekeeper' is not ready           62s
+
+    NAMESPACE           NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
+    kube-system         deployment.apps/local-path-provisioner          1/1     1            1           4m48s
+    kube-system         deployment.apps/coredns                         1/1     1            1           4m48s
+    flux-system         deployment.apps/helm-controller                 1/1     1            1           4m6s
+    flux-system         deployment.apps/notification-controller         1/1     1            1           4m6s
+    flux-system         deployment.apps/source-controller               1/1     1            1           4m7s
+    flux-system         deployment.apps/kustomize-controller            1/1     1            1           4m7s
+    gatekeeper-system   deployment.apps/gatekeeper-controller-manager   1/1     1            1           2m8s
+    gatekeeper-system   deployment.apps/gatekeeper-audit                1/1     1            1           2m8s
+    istio-operator      deployment.apps/istio-operator                  0/1     1            0           8s
+
+    NAMESPACE           NAME                                                 READY   STATUS              RESTARTS   AGEkube-system         pod/local-path-provisioner-6d59f47c7-s6rln           1/1     Running             0          4m36s
+    kube-system         pod/coredns-7944c66d8d-flk4p                         1/1     Running             0          4m36s
+    flux-system         pod/helm-controller-578cdbcd8b-tjzs7                 1/1     Running             0          4m6s
+    flux-system         pod/notification-controller-7c59d85f77-92ckv         1/1     Running             0          4m6s
+    flux-system         pod/source-controller-7d6f889df9-f888j               1/1     Running             0          4m7s
+    flux-system         pod/kustomize-controller-5cfb78859c-n85xn            1/1     Running             0          4m6s
+    gatekeeper-system   pod/gatekeeper-controller-manager-5b9cf6c85d-cqd8t   1/1     Running             0          2m8s
+    gatekeeper-system   pod/gatekeeper-audit-7db49c54d5-pwzwh                1/1     Running             0          2m8s
+    istio-operator      pod/istio-operator-79f966cfc-rjhhc                   0/1     ContainerCreating   0          8s
+   ```
+
+1. Wait until all Helm Releases, Deployments, and Pods are Ready.  Be patient, this can take 15-30 minutes.
+
+> The Git repositories are monitored periodically (default is 10m) for changes.  If a change is detected, the configuration will be reconciled using Flux.  The monitoring techniques above can be used to monitor the reconciliation.

docs/a_faqs.md

+# Appendix A - Big Bang FAQs

docs/b_troubleshooting.md

+# Appendix B - Big Bang Troubleshooting
+
+- [Appendix B - Big Bang Troubleshooting](#appendix-b---big-bang-troubleshooting)
+  - [Iron Bank authentication](#iron-bank-authentication)
+  - [Flux install](#flux-install)
+  - [Git Repository](#git-repository)
+  - [ConfigMap or Secrets](#configmap-or-secrets)
+  - [Helm Release](#helm-release)
+  - [Kustomization](#kustomization)
+  - [Packages](#packages)
+
+Big Bang can take a long time to run.  After making changes, it could take 10-15 minutes to take effect.  Use the [sync.sh](../hack/sync.sh) script to speed this up.
+
+Big Bang is configured to retry failed package installations and upgrades.  Before concluding you have a failure, make sure you allow Big Bang to attempt to resolve dependencies and retry.
+
+## Iron Bank authentication
+
+| Symptom | Cause | Resolution |
+|--|--|--|--|
+| Despite entering correct credentials, get `anauthorized: authentication required` from Iron Bank | Using a non-robot account with an expired token | Login with the non-robot account manually at `registry1.dsop.io`, then retry.  For production, contact the Iron Bank team to obtain a robot account and update pull credentials to use it in your environment |
+
+## Flux install
+
+Helpful debugging commands:
+
+```bash
+# Get the status
+kubectl get pods -n flux-system
+
+# Get the logs
+kubectl get events -n flux-system
+```
+
+| Symptom | Cause | Resolution |
+|--|--|--|--|
+| Install script timed and pods are still pulling the image | Slow connection to docker registry | Adjust `--timeout` value in `flux install` to wait longer |
+| Pod status is `ImagePullBackOff` or `ErrImagePull` | Bad registry, version, or credentials | Fix the `--registry`, `--version`, or `--image-pull secret` options or use the `hack/flux-install.sh` script for pulling from Iron Bank |
+
+## Git Repository
+
+Helpful debugging commands:
+
+```bash
+# Get the status
+kubectl get gitrepositories -A
+
+# Get the logs
+kubectl get events --field-selector involvedObject.kind=GitRepository -A
+```
+
+| Symptom | Cause | Resolution |
+|--|--|--|--|
+| `unable to clone ... error: authentication required` | Pull credentials for Git invalid or not provided | Add credentials to a `Secret` and reference it in `GitRepository.spec.secretRef.name`.  If possible, encrypt the secret and include it in the Kustomization deployment for your environment.
+| `auth secret error: Secret ... not found` | `GitRepository` is trying to use credentials but cannot find the `Secret` | Make sure the secret exists and is in the same namespace as the `GitRepository` resource.   If possible, encrypt the secret and include it in the Kustomization deployment for your environment.
+| `unable to clone ... error: repository not found` | Invalid Git url | Fix url for Git repository and redeploy |
+| `unable to clone ... error: couldn't find remote ref` | Invalid branch or tag | Fix branch or tag for Git repository and redeploy |
+
+## ConfigMap or Secrets
+
+| Symptom | Cause | Resolution |
+|--|--|--|--|
+|`ConfigMap` or `Secret` does not exist| GitRepository or Kustomization failed.  Namespace was incorrect. | Use [GitRepository](#git-repository) and [Kustomization](#kustomization) sections to troubleshoot.  Use `kubectl get secrets,configmaps -A` to verify resource was not in the wrong Namespace. |
+
+## Helm Release
+
+Helpful debugging commands:
+
+```bash
+# Get the status
+kubectl get hr -A
+
+# Get the logs
+kubectl get events --field-selector involvedObject.kind=HelmRelease -A
+```
+
+| Symptom | Cause | Resolution |
+|--|--|--|--|
+| `Reconcilliation in Progress` | This is normal and indicates flux is currently applying updates | Wait |
+| `dependency ... is not ready` | This is normal and indicates flux is currently waiting on another resource to complete | Wait |
+| `Error: YAML parse error on ...` | Syntax error in helm chart | Use `helm template` to narrow down the problem.  Fix it and commit to Git |
+| `Helm install failed: failed to create resource ... unable to create new content in namespace because it is being terminated` | This seems to happen when a re-deploy of Big Bang occurs to early after a Big Bang delete. |  Try to remove the namespace using `kubectl get ns <stuck namespace> -o json | jq '.spec.finalizers = []' | kubectl replace --raw "/api/v1/namespaces/$NS/finalize" -f`.  If this does not work, a cluster restart may be necessary. |
+| `Error: failed to download ...` | Path to Helm chart is incorrect | Find the HelmRelease configuration and update `spec.path` to the correct path of the helm chart |
+
+## Kustomization
+
+Helpful debugging commands:
+
+```bash
+# Get the status
+kubectl get kustomizations -A
+
+# Get the logs
+kubectl get events --field-selector involvedObject.kind=Kustomization -A
+```
+
+| Symptom | Cause | Resolution |
+|--|--|--|--|
+| `kustomization path not found` | `spec.path` in Kustomization resource in is incorrect | Fix `spec.path` and redeploy |
+| `Source not found` | `spec.sourceRef` in Kustomization resource is incorrect | Fix `spec.sourceRef` to point to repository resource and redeploy |
+| `decryption secret error: Secret ... not found` | SOPS private key secret is missing or misconfigured | Check `decryption` settings in the Kustomization resource to make sure `secretRef` is pointing to the correct secret.  Make sure the `Secret` holding the private key is deployed in the cluster. |
+| `kustomize build failed: json: unknown field` | There is a syntax error with the kustomization files. | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem.  Fix the error and push to Git. |
+| `evalsymlink failure ... no such file or directory` | A reference to a file in `kustomization.yaml` is incorrect | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem.  Fix the error and push to Git. |
+| `Error: accumulating resources ...` | A reference to a base is incorrect | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem.Review the `bases:` section for correct paths to find the error.  Fix the error and push to Git. |
+| `Error fetchingref: fatal: couldn't find remote ref ...` | The branch, tag, or sha used for a remote base is incorrect | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem.  It is likely the remote reference to the Big Bang umbrella in the `base` folder.  Review the `bases:` section for correct paths to find the error.  Fix the error and push to Git. |
+| `Error: merging from generator ...` | Kustomize is trying to merge with a resource that is non-existant.  This is usually due to naming the merging `ConfigMap` or `Secret` incorrectly compared to a base `ConfigMap` or `Secret`. | Use `kustomize build` on the `<env>` folder or `base` folder to narrow down the problem.  Look for the keyword `merge` in the `kustomization.yaml` files and verify the `name` is correctly set. |
+
+## Packages
+
+Helpful debugging commands:
+
+```bash
+# Get the status
+kubectl get deployments,po -n <namespace of package>
+
+# Get the logs
+kubectl get events --field-selector involvedObject.kind=Deployment -n <namespace of package>
+kubectl get events --field-selector involvedObject.kind=Pod -n <namespace of package>
+```

docs/developer_setup.md → docs/c_development.md

+# Appendix C - Big Bang Development
+
 ## So you want to develop on Big Bang Umbrella?
 
 Included here is a setup that will allow you to checkout and begin development using your workstation and a minimal EC2 instance in AWS.
@@ -14,7 +16,7 @@ Included here is a setup that will allow you to checkout and begin development u
 
 This section will cover the creation of an environment manually. This is a good place to start because it creates an understanding of everything that the automated method does for you.
 
-Step 1: Create an Ubuntu 20.04 xlarge EC2 instance with the following attributes:  
+Step 1: Create an Ubuntu 20.04 xlarge EC2 instance with the following attributes:
         (see addendum for using Amazon Linux2 - but it really does not matter)
 
 + 50 Gigs of disk space
@@ -22,7 +24,7 @@ Step 1: Create an Ubuntu 20.04 xlarge EC2 instance with the following attributes
 + A security group that allows all TCP traffic from your IP address.
 + The following in the User Data
 
-```
+```bash
 MIME-Version: 1.0
 Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="
 
@@ -39,7 +41,7 @@ Step 2: SSH into your new EC2 instance and configure it with the following:
 
 + Install Docker CE
 
-```
+```bash
 #Remove any old Docker items
 sudo apt remove docker docker-engine docker.io containerd runc
 
@@ -47,7 +49,7 @@ sudo apt remove docker docker-engine docker.io containerd runc
 sudo apt update
 sudo apt install apt-transport-https ca-certificates curl     gnupg-agent software-properties-common
 
-#Add the Docker repository, we are installing from Docker and not the 
+#Add the Docker repository, we are installing from Docker and not the
 #Ubuntu APT repo.
 curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
 
@@ -63,13 +65,13 @@ sudo apt install docker-ce docker-ce-cli containerd.io
 #to run docker commands
 sudo usermod -aG docker $USER
 
-** It is important at this point that you log out and back in to 
+** It is important at this point that you log out and back in to
 ** have the user group changes take effect.
 ```
 
 + Install K3D on the EC2 instance
 
-```
+```bash
 wget -q -O - https://raw.githubusercontent.com/rancher/k3d/main/install.sh | bash
 
 k3d //to check the install
@@ -77,17 +79,17 @@ k3d //to check the install
 
 + We can now spin up our dev cluster on the EC2 instance using K3D
 
-```
+```bash
 k3d cluster create -s 1 -a 3  --k3s-server-arg "--disable=traefik" --k3s-server-arg "--disable=metrics-server" --k3s-server-arg "--tls-san=<your-public-ec2-ip>"  -p 80:80@loadbalancer -p 443:443@loadbalancer
 ```
 
 + Optionally you can set your image pull secret on the cluster so that you don't have to put your credentials in the code or in the command line in later steps
 
-```
-# create the directory for the k3s registry config.  
+```bash
+# create the directory for the k3s registry config.
 mkdir ~/.k3d/
 
-# create the config file. Use your registry1 credentials. Copy your user name and token secret from your Harbor profile.  
+# create the config file. Use your registry1 credentials. Copy your user name and token secret from your Harbor profile.
 
 cat << EOF > ~/.k3d/p1-registries.yaml
 configs:
@@ -102,38 +104,38 @@ k3d cluster create --servers 1 --agents 3 -v ~/.k3d/p1-registries.yaml:/etc/ranc
 
 Here is a break down of what we are doing with this command.
 
-(-s 1) Creating 1 master/server
+`-s 1` Creating 1 master/server
 
-(-a 3) Creating 3 agent nodes
+`-a 3` Creating 3 agent nodes
 
-(--k3s-server-arg "--disable=traefik") Disable the default Traefik Ingress
+`--k3s-server-arg "--disable=traefik"` Disable the default Traefik Ingress
 
-(--k3s-server-arg "--disable=metrics-server") Disable default metrics
+`--k3s-server-arg "--disable=metrics-server"` Disable default metrics
 
-(--k3s-server-arg "--tls-san=<your public ec2 ip>") This adds the public IP to the kubeapi certificate so that you can access it remotely.
+`--k3s-server-arg "--tls-san=<your public ec2 ip>"` This adds the public IP to the kubeapi certificate so that you can access it remotely.
 
-(-p 80:80@loadbalancer) Exposes the cluster on the host on port 80
+`-p 80:80@loadbalancer` Exposes the cluster on the host on port 80
 
-(-p 443:443@loadbalancer) Exposes the cluster on the host on port 443
+`-p 443:443@loadbalancer` Exposes the cluster on the host on port 443
 
-optional:  
-(-v ~/.k3d/p1-registries.yaml:/etc/rancher/k3s/registries.yaml) volume mount image pull secret config for k3d cluster  
-(--api-port 0.0.0.0:38787) Chooses a port for the API server instead of being assigned a random one. You can set this to any port number that you want.
+optional:
+`-v ~/.k3d/p1-registries.yaml:/etc/rancher/k3s/registries.yaml` volume mount image pull secret config for k3d cluster
+`--api-port 0.0.0.0:38787` Chooses a port for the API server instead of being assigned a random one. You can set this to any port number that you want.
 
 + Once your cluster is up, you can bring over the kubeconfig from the EC2 instance to your workstation.
 
-```
+```bash
 cat ~/.kube/config
 ```
 
 + Move to your workstation and install Big Bang Umbrella on the cluster
 
-```
+```bash
 # Test to see if you can connect to your cluster
 
 kubectl get nodes
 
-# From the base of the project 
+# From the base of the project
 
 flux install
 
@@ -143,7 +145,7 @@ kubectl apply -f examples/complete/envs/dev/source-secrets.yaml
 
 # Helm install BigBang umbrella
 
-# Method 1 - go for it. (Note: You don't need to set registryCredentials if you configured registry pull secret on the cluster in previous steps)  
+# Method 1 - go for it. (Note: You don't need to set registryCredentials if you configured registry pull secret on the cluster in previous steps)
 
 yq r examples/complete/envs/dev/patch-bigbang.yaml 'spec.values' | helm upgrade -i bigbang chart -n bigbang --create-namespace --set registryCredentials.username='<your user>' --set registryCredentials.password=<your cli key> -f -
 
@@ -152,14 +154,14 @@ yq r examples/complete/envs/dev/patch-bigbang.yaml 'spec.values' | helm upgrade
 yq r examples/complete/envs/dev/patch-bigbang.yaml 'spec.values' > my-values.yaml
 
 # Modify my-values.yaml
-# Install using your new values. You could also modify the values in place. (Note: You don't need to set registryCredentials if you configured registry pull secret on the cluster in previous steps)  
+# Install using your new values. You could also modify the values in place. (Note: You don't need to set registryCredentials if you configured registry pull secret on the cluster in previous steps)
 
 helm upgrade -i bigbang chart -n bigbang --create-namespace --set registryCredentials.username='<your user>' --set registryCredentials.password=<your cli key> -f my-values.yaml
 ```
 
 + You can now modify your local /etc/hosts files (Or whatever the Windows people call it these days)
 
-```
+```bash
 160.1.38.137     kibana.bigbang.dev
 160.1.38.137     kiali.bigbang.dev
 160.1.38.137    prometheus.bigbang.dev
@@ -168,22 +170,24 @@ helm upgrade -i bigbang chart -n bigbang --create-namespace --set registryCreden
 
 + You can watch your install take place with
 
-```
+```bash
 watch kubectl get po,gitrepository,kustomizations,helmreleases -A
 ```
 
 As of this time, Twistlock is the last thing to be installed. Once you see Twistlock sync and everything else is up and healty you are fully installed.
 
 ### Addendum for Amazon Linux 2
+
 Here are the configuration steps if you want to use a Fedora based instance. All other steps are similar to Ubuntu.
-```
+
+```bash
 # update system
 sudo yum update -y
 # install and start docker
 sudo yum install docker -y
 sudo usermod -aG docker $USER
-sudo systemctl enable docker.service 
-sudo systemctl start docker.service 
+sudo systemctl enable docker.service
+sudo systemctl start docker.service
 # fix docker config for ulimit nofile.
 # this is a bug in the AMI that will eventually be fixed
 sudo sed -i 's/^OPTIONS=.*/OPTIONS=\"--default-ulimit nofile=65535:65535\"/' /etc/sysconfig/docker