C-provision-gpg-encryption-keypair-and-configure-gitops-encrypted-secrets.md

Provision GPG encryption keypair and configure gitops encrypted secrets

Background Context:
In the previous lab we did the equivalent of forking the Big Bang customer template repo
https://repo1.dso.mil/platform-one/big-bang/customers/template
Into a ssh VM based git repo that we have full access to.

The template repo is missing a few key items:

• Proper gitops encryption configuration for storing encrypted secrest in git
• Pull secrets for pulling images from the Iron Bank harbor registry

This Lab will walk you through creating a GPG encryption keypair and related config needed for deploying BigBang using generic gitops encryption.

Step 1: Generate a GPG keypair

To make sure your pull secrets are not comprimized when uploaded to Git, you must generate your own encryption key:

Remember to verify your $NAME environment variable is set on your laptop

# [user@Laptop:~]
echo $NAME
# chrism
# (We configured this as part of lab guide 1E, just verifying it's still good)

# [user@Laptop:~]
# Generate a GPG encryption/decription key pair
# Copy and paste the following as a multi line command, including the EOF
# NOTE: For if any's running a really old version of linux using gpg 2.0.x, 
# try updating it to 2.2.x ; otherwise, substitute '--gen-key' for '--full-generate-key'
# 
# Notice: 
# By default, gpg 2.3.x generated keys, will generate errors when 
# consumed by clients running gpg 2.0.x - 2.2.x 
# 
# Workaround: 
# When the following gpg key gen command can be copy pasted and ran against
# Most Linux Distros (which use gpg 2.2.x) or Mac / RHEL 9 (which uses gpg 2.3.x)
# The resulting generated key will work correctly for gpg 2.0.x - 2.3.x clients
# (Technically the --rfc4880 & --digest-algo flags are only needed for gpg 2.3.x;
# however, they don't hurt when added to gpg 2.2.x, so it's recommended to 
# always use them for the sake of consistency) 
# 
export KEY_NAME="$NAME"
export KEY_COMMENT="For use in $NAME's Big Bang dev environment"
gpg --batch --full-generate-key --rfc4880 --digest-algo sha512 --cert-digest-algo sha512 <<EOF
    %no-protection
    # %no-protection: means the private key won't be password protected 
    # (no password is a fluxcd requirement, it might also be true for argo & sops)
    Key-Type: RSA
    Key-Length: 4096
    Subkey-Type: RSA
    Subkey-Length: 4096
    Expire-Date: 0
    Name-Real: ${KEY_NAME}
    Name-Comment: ${KEY_COMMENT}
EOF

# If the above is taking "too long", your system might not be generating enough entropy
# use ctrl + c to abort the current task and return to an interactive terminal
# 
# install the 'rng-tools' package and run the following in another terminal:
#
# sudo rngd -r /dev/urandom
#
# you can kill the rngd command when gpg finishes creating your key

gpg --list-keys --fingerprint #This version of the command syntax works for multiple versions of gpg
# You'll see something similiar to this
# pub   rsa4096 2022-02-28 [SCEA]
#       CEB7 F6F3 F629 F774 6083  60DE 1984 3848 0DB3 E1E9
# uid           [ultimate] chrism (For use in chrism's Big Bang dev environment)
# sub   rsa4096 2022-02-28 [SEA]


# Let's Learn some Bash Fu
gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1
# grep "$KEY_COMMENT" -B 1 -----> Find $KEY_COMMENT and the 1 line before it

gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT"
# grep -v "$KEY_COMMENT"   -----> Find line with $KEY_COMMENT, the -v inverts the selection to only include the line before it
# What you see is a messy version of a GPG (GNU Privacy Guard) / PGP (Pretty Good Privacy)
# Key Pair's "finger print"

gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ' 
# tr -d ' '       -----> Translate delete spaces (cleans up the messyness) 

# Lets store it in a variable, then verify the contents of the variable
export fp=$(gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ')

## Following added to improve consistency in the event of edge cases around gpg 2.0.x
export fp=$(gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ' | tr -d '-' | tr -d ' ' | tr -d 'Keyfingerprint=')

echo $fp #fp for "fingerprint"
# CEB7F6F3F629F774608360DE198438480DB3E1E9
# ^-- should look something like this


# We will cover how you can safely share this key with a hypothetical team mate, as part of this lab.
# A consquence of this is that while we won't cover it, it will be possible for other cohort attendees 
# to decode your key. 
# To be extra safe security wise we'll update the key pair which was originally created to never expire, 
# to expire in 7 days, this is to ensure no one will accidently use the lab key for any real world purposes.  

# Here we reduce this down to 7 days for learning/demo purposes
gpg --quick-set-expire ${fp} 7d

# for users of gpg versions prior to 2.1.17, the --quick-set-expire option will not be available
# You will have to manually change the expiration:
# gpg --edit ${fp}
# gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
# This is free software: you are free to change and redistribute it.
# There is NO WARRANTY, to the extent permitted by law.
# 
# Secret key is available.
# 
# pub  4096R/F41870CB  created: 2022-02-28  expires: 2022-03-08  usage: SCEA
#                      trust: unknown       validity: unknown
# sub  4096R/84DB66A2  created: 2022-02-28  expires: never       usage: SEA
# [ unknown] (1). reperalta (For use in reperalta's Big Bang dev environment)
# 
# gpg> expire
# Changing expiration time for the primary key.
# Please specify how long the key should be valid.
#          0 = key does not expire
#       <n>  = key expires in n days
#       <n>w = key expires in n weeks
#       <n>m = key expires in n months
#       <n>y = key expires in n years
# Key is valid for? (0) 7d
# Key expires at Tues 08 Mar 2022 02:14:25 PM MDT
# Is this correct? (y/N) y
# 
# pub  4096R/F41870CB  created: 2022-02-28  expires: 2022-03-08  usage: SCEA
#                      trust: unknown       validity: unknown
# sub  4096R/84DB66A2  created: 2022-02-28  expires: never       usage: SEA
# [ unknown] (1). reperalta (For use in reperalta's Big Bang dev environment)
# 
# gpg> save
# 

gpg --list-keys --fingerprint
# pub   rsa4096 2022-02-28 [SCEA] [expires: 2022-03-08]   <---FEEDBACK THAT EXPIRATION HAS BEEN SET
#       CEB7 F6F3 F629 F774 6083  60DE 1984 3848 0DB3 E1E9
# uid           [ultimate] chrism (For use in chrism's Big Bang dev environment)
# sub   rsa4096 2022-02-28 [SEA]

Step 2: Edit SOPS CLI's config file to leverage your generated gpg keypair

[user@Laptop:~]
cd ~/Desktop/bootstrap
head .sops.yaml
# Inside your instance of the customer template repo
# You'll find a mock config file with fake placeholder values that look like this:

# creation_rules:
# - encrypted_regex: '^(data|stringData)$'
#   # Base is shared, you can add keys from each environment that uses base here
#   pgp: FALSE_KEY_HERE
# #    ,ANOTHER_FALSE_KEY_HERE

We will use "sed" AKA "stream editor" CLI utility
to do the CLI equivalent of a find and replace
to find the fake placeholder value, and replace it with a real value
to properly setup a config file that will be used by the SOPS cli utility to encrypt values. 

[user@Laptop:~/Desktop/bootstrap]
## command syntax for Linux Users
sed -i "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml

## command syntax for Mac Users
sed -i "" "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml

head .sops.yaml
# Notice the file was updated with the fingerprint of the GPG key pair you generated

# creation_rules:
# - encrypted_regex: '^(data|stringData)$'
#   # Base is shared, you can add keys from each environment that uses base here
#   pgp: 8CEC83AF96E716A4C7669B037943F697A24A4CCC                   <---- SHOULD NOW LOOK SIMILAR TO THIS
# #    ,ANOTHER_FALSE_KEY_HERE
# ...

Step 3: Create Iron Bank Harbor Image Pull Credentials Secret

You will need pull credentials for Iron Bank to retrieve images for Big Bang.

Secrets can be specific to an environment if they are located in that environment's folder (e.g. prod, dev). Or, they can be shared between environments if located in the base directory.

# [user@Laptop:~/Desktop/bootstrap]
# Create a new encrypted secret to contain your pull credentials
cd ~/Desktop/bootstrap/base
EDITOR="subl --wait" sops ~/Desktop/bootstrap/base/secrets.enc.
# The following also works (assuming you've configured VS Code to be launchable via CLI)
# EDITOR="code --wait" sops ~/Desktop/bootstrap/base/secrets.enc.

Copy paste the following contents to the newly created sops secret. Put your Iron Bank user/PAT where it states replace-with-your-iron-bank-user and replace-with-your-iron-bank-personal-access-token. (Remember these can be found on https://registry1.dso.mil, logging in with OIDC provider, and clicking on your user profile in the top right.)

The name of the secret must be common-bb if the secret is in the base folder or environment-bb if the secret is in the dev or prod folder. The environment-bb values take precedence over the common-bb values.

apiVersion: v1
kind: Secret
metadata:
  name: common-bb
stringData:
  values.yaml: |-
    registryCredentials:
      - registry: registry1.dso.mil
        username: replace-with-your-iron-bank-user
        password: replace-with-your-iron-bank-personal-access-token

Step 4: Update the Iron Bank Harbor Image Pull Secret with your credentials

• Make sure you don't forget to update the credentials

# [user@Laptop:~/Desktop/bootstrap]
EDITOR="subl --wait" sops ~/Desktop/bootstrap/base/secrets.enc.yaml

• The following is a copy pasteable command to help validate that you correctly edited the file

sops --decrypt ~/Desktop/bootstrap/base/secrets.enc.yaml | grep "replace" ; echo $? | grep 0 && for i in {1..10}; do echo "You still need to update the username and password, rerun the edit command"; done || echo "Good you successfully updated your credentials"

• The following is a compy pasteable set of commands to test your credentials

export REGISTRY1_USER=$(sops --decrypt ~/Desktop/bootstrap/base/secrets.enc.yaml | grep username | tr -d " " | cut -d : -f 2)
export REGISTRY1_TOKEN=$(sops --decrypt ~/Desktop/bootstrap/base/secrets.enc.yaml | grep password | tr -d " " | cut -d : -f 2)

echo $REGISTRY1_TOKEN | docker login registry1.dso.mil --username $REGISTRY1_USER --password-stdin | grep "Login Succeeded" ; echo $? | grep 0 && echo "You edited the file correctly and your registry1 credentials work" || echo "Something looks wrong, double check your registry1 credentials"

# Save encrypted secrets into Git
# Configuration changes must be stored in Git to take affect
git status
git add secrets.enc.yaml ../.sops.yaml
git status
git commit -m "feat(gitops)!: added encrypted credentials"
git push
git status

Your private key to decrypt these secrets is stored in your GPG key ring.
WARNING If you export this key and commit it to a Git Repo, anyone would be able to decrypt secrets in your git repo that were encrypted using the key. That being said... It's common to work on a team, and you'd need to be able to safely share the decryption key with a colleague, or have some DR(Disaster Recovery) where you don't lose your decryption key if your laptop dies.
We will use CSP KMS as a key wrapping key to pull this off. CSP KMS uses identity based decryption vs Key based Decryption, you can read more here: https://oteemo.com/hashicorp-vault-is-overhyped-and-mozilla-sops-with-kms-and-git-is-massively-underrated/

Step 5: Export your generated gpg keypair, encrypt it with CSP KMS, and store it in the git repo for others to use.

[user@Laptop:~]
# Temporarily export your generated gpg key to your home directory
gpg --armor --export-secret-keys $fp > ~/$NAME-bigbang-gpg-keypair.dev
# (the --armor flag specifies a cleartext encoding of output file vs binary encoding of output file)


# Make a folder in the repo where a kms encrypted version of the keypair can be stored for sharing with your team
mkdir -p ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets
cd ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets


# Encrypt the ~/$NAME-bigbang-gpg-keypair.dev and store it in the git repo
# Note we're referencing a kms key existing in the bb onboarding aws account.
export AWS_PROFILE=bb-onboarding; export AWS_DEFAULT_PROFILE=bb-onboarding
sops --encrypt --kms arn:aws-us-gov:kms:us-gov-west-1:110518024095:key/b6bf63f0-dc65-49b4-acb9-528308195fd6 ~/$NAME-bigbang-gpg-keypair.dev > ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets/$NAME-bigbang-gpg-keypair.encrypted.dev

cat ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets/$NAME-bigbang-gpg-keypair.encrypted.dev | grep '"data"'
# IMPORTANT
# READ
# CAREFULLY
echo -e " IMPORTANT \n READ \n CAREFULLY \n Verify that \"data\": contains a really long value/doesn't have empty double quotes"

########################################################################
# IMPORTANT NOTE:
# kms encryption is superior to asymmetric and symmetric encryption
# because, it allows you to introduce RBAC and give specific identities
# rights to decrypt data, that was encrypted using kms.
#
# Example: you could make a dev kms key and a prod kms key
# give devs rights to the dev key and ops rights to both keys
# Then devs could decrypt only data encrypted with the dev kms key
#
# That being said we're in a lab environment where all attendees are
# sharing the same kms key. So be aware that anything you encrypt with
# kms key can be accessed by the other lab attendees. They can also use
# the shared lab ssh keypair to access your private git repo as well.
#
# Further on in the lab you'll be told to encrypt your personal
# IronBank Access credentials. Don't sweat it as everyone has the same
# read only access, at the end of the residency rotate your read only
# IronBank Credentials.
########################################################################

# Let's cat the file to verify it's encrypted
cat ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets/$NAME-bigbang-gpg-keypair.encrypted.dev
# Verify that it was encrypted with kms key, and that you see pgp: null (not encrypted with pgp)


# Delete your local cleartext copy of the keypair that was in your home directory
rm ~/$NAME-bigbang-gpg-keypair.dev


# Push the encrypted key pair to the repo
git add .
git status
# You should see $NAME-bigbang-gpg-keypair.encrypted.dev listed as a change to be committed
git commit -m "feat(gitops): adding KMS encrypted gpg keypair for dev environment to repo"
git push

Step 6: Dry run through how a team mate would get access to the gpg keypair

[user@Laptop:~]
echo $fp #Let's verify we have our fingerprint var set
# If you need to repopulate $fp, use the following (copy pasted from above)
export fp=$(gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ' | tr -d '-' | tr -d ' ' | tr -d 'Keyfingerprint=')
echo $fp

# The following will have human input prompts, press y for yes to get through the prompts
gpg --delete-secret-keys $fp
gpg --delete-keys $fp

gpg --list-secret-keys --fingerprint 
# Your keypair is gone / you've successfully simulated being in the position of a team mate who needs to get onboarded

# Assume they've already cloned the git repo
cd ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets
export AWS_PROFILE=bb-onboarding; export AWS_DEFAULT_PROFILE=bb-onboarding
sops --decrypt $NAME-bigbang-gpg-keypair.encrypted.dev | gpg --import -
# gpg: key 7E03CC83EA23AF3E: "chrism (for use in chrism's big bang dev environment) <chrism@bigbang.dev>" not changed
# gpg: key 7E03CC83EA23AF3E: secret key imported
# gpg: Total number processed: 1
# gpg:              unchanged: 1
# gpg:       secret keys read: 1
# gpg:  secret keys unchanged: 1

# (NOTE: I'm not sure where 7E03CC83EA23AF3E comes from)
# (ignore it and trust the identifer given by the 'gpg --list-secret-keys --fingerprint' command)
# Re-Look at your available keypairs
gpg --list-secret-keys --fingerprint
# /Users/chrism/.gnupg/pubring.kbx
# ---------------------------------
# sec   rsa4096 2022-02-28 [SCEA] [expires: 2022-03-08]
#       CEB7 F6F3 F629 F774 6083  60DE 1984 3848 0DB3 E1E9
# uid           [ultimate] chrism (For use in chrism's Big Bang dev environment)
# ssb   rsa4096 2022-02-28 [SEA]

Step 7: Create BigBang namespace and turn the gpg keypair into a Kubernetes Secret that the Flux GitOps Operator can use to decrypt encrypted secrets in the git repo

# [user@Laptop:~]
kubectl create namespace bigbang 
export KEY_COMMENT="For use in $NAME's Big Bang dev environment"
export fp=$(gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ' | tr -d '-' | tr -d ' ' | tr -d 'Keyfingerprint=')
echo $fp #quick check to verify this looks right, similiar to 8CEC83AF96E716A4C7669B037943F697A24A4CCC
gpg --export-secret-key --armor ${fp} | kubectl create secret generic sops-gpg -n bigbang --dry-run=client -o yaml --from-file=bigbangkey.asc=/dev/stdin | kubectl apply -f -

Step 8: Verify the sops-gpg secret has been successfully created

kubectl get secret -n=bigbang
# NAME                  TYPE                                  DATA   AGE
# default-token-mhm8t   kubernetes.io/service-account-token   3      58s
# sops-gpg              Opaque                                1      18s

Install Default Storage Class