Merge branch 'documentation/non-aws-deployment-with-AWS-KMS' into 'master'

Documentation for non-AWS deployments which rely on AWS KMS

See merge request platform-one/big-bang/bigbang!472

.gitlab-ci/jobs/ci-cluster/.gitlab-ci.yml

 .k8s-util:
-  image: registry.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/k8s-ci:v1.20.4-bb.2
+  image: registry.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/k8s-ci:v1.20.4-bb.3
 
 .dind-runner:
   tags:

base/flux/kustomization.yaml

+# start with a default flux deployment
+resources:
+- gotk-components.yaml
+
+# update flux components to use ironbank images
+images:
+- name: registry1.dso.mil/ironbank/fluxcd/helm-controller:v0.10.0
+  newName: registry1.dso.mil/ironbank/fluxcd/helm-controller
+  newTag: v0.9.0
+- name: registry1.dso.mil/ironbank/fluxcd/kustomize-controller:v0.12.0
+  newName: registry1.dso.mil/ironbank/fluxcd/kustomize-controller
+  newTag: v0.9.3
+- name: registry1.dso.mil/ironbank/fluxcd/notification-controller:v0.13.0
+  newName: registry1.dso.mil/ironbank/fluxcd/notification-controller
+  newTag: v0.12.0
+- name: registry1.dso.mil/ironbank/fluxcd/source-controller:v0.12.1
+  newName: registry1.dso.mil/ironbank/fluxcd/source-controller
+  newTag: v0.9.1
+
+patches:
+  - target:
+      kind: Deployment
+    patch: |-
+      apiVersion: apps/v1
+      kind: Deployment
+      metadata:
+        name: whatever
+      spec:
+        template:
+          spec:
+            imagePullSecrets:
+            - name: private-registry

docs/configuration.md

@@ -116,7 +116,19 @@ Big Bang deploys four flux resources that can be customized:
 
 In addition, each package contains its own GitRepository and HelmRelease resource that can be customized.  Look in the [Helm chart templates](../chart/templates) for the these resources.
 
-Settings for eny of these resources can be overridden by [patching](https://kubectl.docs.kubernetes.io/references/kustomize/patches/) the resource in your environment's kustomization files.  Use Flux's documentation for [GitRepository](https://toolkit.fluxcd.io/components/source/gitrepositories/), [HelmRelease](https://toolkit.fluxcd.io/components/helm/helmreleases/), and [Kustomization](https://toolkit.fluxcd.io/components/kustomize/kustomization/) to find settings for these resources.
+Settings for eny of these resources can be overridden by [patching](https://kubectl.docs.kubernetes.io/references/kustomize/patches/) the resource in your environment's kustomization files.  Use Flux's documentation for [GitRepository](https://toolkit.fluxcd.io/components/source/gitrepositories/), [HelmRelease](https://toolkit.fluxcd.io/components/helm/helmreleases/), and [Kustomization](https://toolkit.fluxcd.io/components/kustomize/kustomization/) to find settings for these resources. The following are examples of commonly reqeusted custom patches covered in the [bigbang template repo]https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/main#flux-components):
+
+- Updating flux-system component resource usage
+  - [Example `kustomization.yaml`](https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/main#adjust-resource-allocation-for-a-flux-system-component)
+  - This patch could be used to adjust the resources requested by the `flux-system/helm-controller` resource. A similar patch could be used to adjust the resources required by the other flux components.
+  > NOTE: If flux is under-resourced, occasionally requests can fail in a manner that looks like a network connectivity issue (use with caution)
+- Adding environment variables to flux-system components
+  - [Example `kustomization.yaml`](https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/main#adjust-resource-allocation-for-a-flux-system-component)
+  - This patch could be used to add AWS credential environment variables into the `flux-system/kustomize-controller` resource to enable SOPS decryption using a KMS key from outside of AWS.
+- Changing the image name / version
+  - [Example `kustomization.yaml`](https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/main#updating-a-flux-system-component-image-tag)
+  - This patch could be used to update the tag of the flux-system component image to be deployed.
+> NOTE: Multiple patches could be applied within a single kustomization.yaml
 
 ## Big Bang Version
 

docs/encryption.md

@@ -14,6 +14,8 @@ Table of Contents
     - [Azure KeyVault](#azure-keyvault)
     - [HashiCorp Vault](#hashicorp-vault)
   - [Configure Big Bang](#configure-big-bang)
+    - [GPG](#gpg-1)
+    - [AWS KMS](#aws-kms-1)
 
 Big Bang follows a [GitOps](https://www.weave.works/technologies/gitops/) approach to managing the Big Bang Kubernetes cluster configuration.  Using GitOps, we must securely store secrets in Git using encryption.  The private key, which is stored in key storage, is used by the continuous deployment tool to decrypt and deploy the secrets for use in the cluster.
 
@@ -91,7 +93,21 @@ SOPS uses `.sops.yaml` as a configuration file for which keys to use for newly c
 
 ### AWS KMS
 
-TBD - [This article](https://blog.doit-intl.com/injecting-secrets-from-aws-gcp-or-vault-into-a-kubernetes-pod-d5a0e84ba892) may help to automate secret consumption in Kubernetes.
+1. Configure your KMS key(s) in your `.sops.yaml` by adding the target key's ARN to the `kms` field within each creation rule.
+    ```yaml
+    creation_rules:
+    - encrypted_regex: '^(data|stringData)$'
+      path_regex: ./dev/.*
+      kms: '<kms_key_arn>'
+    ```
+2. Ensure your cluster (specifically the `flux-system/flux-controller`) has access to the specified key.
+   1. For AWS deployments, this can be managed via IAM roles as described in the [SOPS documentation](https://github.com/mozilla/sops#28assuming-roles-and-using-kms-in-various-aws-accounts).
+   2. For non-AWS deployments
+      1. Create an AWS user with appropriate permissions as described in the [SOPS documentation](https://github.com/mozilla/sops#28assuming-roles-and-using-kms-in-various-aws-accounts).
+      2. Create a secret named `sops-aws-creds` in the cluster using the access creds from the target user:
+          ```bash
+          k create secret generic -n flux-system sops-aws-creds --from-literal=access_key_id=<key_id> --from-literal=access_key_secret=<key>
+          ```
 
 ### GCP KMS
 
@@ -107,8 +123,10 @@ TBD - [This article](https://blog.doit-intl.com/injecting-secrets-from-aws-gcp-o
 
 ## Configure Big Bang
 
-Big Bang needs to know how to retrieve the private key so it can deploy the encrypted secrets from Git.  Decryption configuration is placed in the top-level manifest (e.g. `dev.yaml`, `prod.yaml`) from the [Big Bang template](https://repo1.dso.mil/platform-one/big-bang/customers/template).  By default, the `Kustomization` resource uses a Secret named `sops-gpg` for the private key as shown here:
+Big Bang needs to know how to retrieve the private key so it can deploy the encrypted secrets from Git.  Decryption configuration is placed in the top-level manifest (e.g. `dev.yaml`, `prod.yaml`) from the [Big Bang template](https://repo1.dso.mil/platform-one/big-bang/customers/template). 
 
+### GPG
+By default, the `Kustomization` resource uses a Secret named `sops-gpg` for the private key as shown here:
 ```yaml
 apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
 kind: Kustomization
@@ -121,4 +139,51 @@ spec:
       name: sops-gpg
 ```
 
-TBD - Instructions on how to update for AWS, GCP, Vault
+### AWS KMS
+Configure the `Kustomization` resource to use sops for decryption:
+```yaml
+apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
+kind: Kustomization
+metadata:
+  name: environment
+spec:
+  decryption:
+    provider: sops
+```
+> Note, we are not providing the `secretRef` field, which is specific to GPG
+
+If Big Bang is deployed within AWS, KMS key access can be handled via IAM roles and permissions on the cluster resources themselves. 
+However, if the deployment is in a different environment from the KMS keys, AWS credentials may need to be provided via a secret as follows. 
+
+Configure the flux-system `kustomize-controller` component with AWS credential environment variables using `kustomize`. Specific instructions for doing this may vary by deployment and environment but [an example](https://repo1.dso.mil/platform-one/big-bang/customers/template/-/tree/master#flux-components) is covered in the bigbang template repo. Broadly speaking, adding environment variables to the `kustomize-controller` component can be accomplished by adding a patch to the `flux/kustomization.yaml` for the target deployment or environment. An example of such a `kustomization.yaml` is shown below:
+```yaml
+bases:
+- ../../base/flux
+
+patchesStrategicMerge:
+- |-
+  apiVersion: apps/v1
+  kind: Deployment
+  metadata:
+    name: kustomize-controller
+    namespace: flux-system
+  spec:
+    template:
+      spec:
+        containers:
+        - name: manager
+          env:
+          - name: AWS_ACCESS_KEY_ID
+            valueFrom:
+              secretKeyRef:
+                name: sops-aws-creds
+                key: access_key_id
+          - name: AWS_SECRET_ACCESS_KEY
+            valueFrom:
+              secretKeyRef:
+                name: sops-aws-creds
+                key: access_key_secret
+```
+> Values should come from the `sops-aws-creds` secret created in [AWS KMS](#aws-kms) above
+
+TBD - Instructions on how to update for GCP, Vault

scripts/install_flux.sh

@@ -9,7 +9,7 @@ set -e
 #
 
 REGISTRY_URL=registry1.dso.mil
-FLUX_MANIFEST=scripts/deploy/flux.yaml
+FLUX_KUSTOMIZATION=base/flux
 FLUX_SECRET=private-registry
 WAIT_TIMEOUT=120
 
@@ -118,8 +118,8 @@ kubectl create secret docker-registry "$FLUX_SECRET" -n flux-system \
   --docker-email="$REGISTRY_EMAIL" \
   --dry-run=client -o yaml | kubectl apply -n flux-system -f -
 
-echo "Installing flux from manifest"
-kubectl apply -f "$FLUX_MANIFEST" 
+echo "Installing flux from kustomization"
+kustomize build "$FLUX_KUSTOMIZATION" | kubectl apply -f -
 
 #
 # verify flux