# Provision GPG encryption keypair and configure gitops encrypted secrets

Background Context:     
In the previous lab we did the equivalent of forking the Big Bang customer template repo    
https://repo1.dso.mil/platform-one/big-bang/customers/template     
Into a ssh VM based git repo that we have full access to.

The template repo is missing a few key items:
* Proper gitops encryption configuration for storing encrypted secrest in git
* Pull secrets for pulling images from the Iron Bank harbor registry

This Lab will walk you through creating a GPG encryption keypair and related config needed for deploying BigBang using generic gitops encryption.

## Step 1: Generate a GPG keypair
To make sure your pull secrets are not comprimized when uploaded to Git, you must generate your own encryption key:    
> Remember to verify your $NAME environment variable is set on your laptop
```bash
# [user@Laptop:~]
echo $NAME
# chrism
# (We configured this as part of lab guide 1E, just verifying it's still good)
```

```shell
# [user@Laptop:~]
# Generate a GPG encryption/decription key pair
# Copy and paste the following as a multi line command, including the EOF
# NOTE: For if any's running a really old version of linux using gpg 2.0.x, 
# try updating it to 2.2.x ; otherwise, substitute '--gen-key' for '--full-generate-key'
# 
# Notice: 
# By default, gpg 2.3.x generated keys, will generate errors when 
# consumed by clients running gpg 2.0.x - 2.2.x 
# 
# Workaround: 
# When the following gpg key gen command can be copy pasted and ran against
# Most Linux Distros (which use gpg 2.2.x) or Mac / RHEL 9 (which uses gpg 2.3.x)
# The resulting generated key will work correctly for gpg 2.0.x - 2.3.x clients
# (Technically the --rfc4880 & --digest-algo flags are only needed for gpg 2.3.x;
# however, they don't hurt when added to gpg 2.2.x, so it's recommended to 
# always use them for the sake of consistency) 
# 
export KEY_NAME="$NAME"
export KEY_COMMENT="For use in $NAME's Big Bang dev environment"
gpg --batch --full-generate-key --rfc4880 --digest-algo sha512 --cert-digest-algo sha512 <<EOF
    %no-protection
    # %no-protection: means the private key won't be password protected 
    # (no password is a fluxcd requirement, it might also be true for argo & sops)
    Key-Type: RSA
    Key-Length: 4096
    Subkey-Type: RSA
    Subkey-Length: 4096
    Expire-Date: 0
    Name-Real: ${KEY_NAME}
    Name-Comment: ${KEY_COMMENT}
EOF

# If the above is taking "too long", your system might not be generating enough entropy
# use ctrl + c to abort the current task and return to an interactive terminal
# 
# install the 'rng-tools' package and run the following in another terminal:
#
# sudo rngd -r /dev/urandom
#
# you can kill the rngd command when gpg finishes creating your key

gpg --list-keys --fingerprint #This version of the command syntax works for multiple versions of gpg
# You'll see something similiar to this
# pub   rsa4096 2022-02-28 [SCEA]
#       CEB7 F6F3 F629 F774 6083  60DE 1984 3848 0DB3 E1E9
# uid           [ultimate] chrism (For use in chrism's Big Bang dev environment)
# sub   rsa4096 2022-02-28 [SEA]


# Let's Learn some Bash Fu
gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1
# grep "$KEY_COMMENT" -B 1 -----> Find $KEY_COMMENT and the 1 line before it

gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT"
# grep -v "$KEY_COMMENT"   -----> Find line with $KEY_COMMENT, the -v inverts the selection to only include the line before it
# What you see is a messy version of a GPG (GNU Privacy Guard) / PGP (Pretty Good Privacy)
# Key Pair's "finger print"

gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ' 
# tr -d ' '       -----> Translate delete spaces (cleans up the messyness) 

# Lets store it in a variable, then verify the contents of the variable
export fp=$(gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ')

## Following added to improve consistency in the event of edge cases around gpg 2.0.x
export fp=$(gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ' | tr -d '-' | tr -d ' ' | tr -d 'Keyfingerprint=')

echo $fp #fp for "fingerprint"
# CEB7F6F3F629F774608360DE198438480DB3E1E9
# ^-- should look something like this


# We will cover how you can safely share this key with a hypothetical team mate, as part of this lab.
# A consquence of this is that while we won't cover it, it will be possible for other cohort attendees 
# to decode your key. 
# To be extra safe security wise we'll update the key pair which was originally created to never expire, 
# to expire in 7 days, this is to ensure no one will accidently use the lab key for any real world purposes.  

# Here we reduce this down to 7 days for learning/demo purposes
gpg --quick-set-expire ${fp} 7d

# for users of gpg versions prior to 2.1.17, the --quick-set-expire option will not be available
# You will have to manually change the expiration:
# gpg --edit ${fp}
# gpg (GnuPG) 2.0.22; Copyright (C) 2013 Free Software Foundation, Inc.
# This is free software: you are free to change and redistribute it.
# There is NO WARRANTY, to the extent permitted by law.
# 
# Secret key is available.
# 
# pub  4096R/F41870CB  created: 2022-02-28  expires: 2022-03-08  usage: SCEA
#                      trust: unknown       validity: unknown
# sub  4096R/84DB66A2  created: 2022-02-28  expires: never       usage: SEA
# [ unknown] (1). reperalta (For use in reperalta's Big Bang dev environment)
# 
# gpg> expire
# Changing expiration time for the primary key.
# Please specify how long the key should be valid.
#          0 = key does not expire
#       <n>  = key expires in n days
#       <n>w = key expires in n weeks
#       <n>m = key expires in n months
#       <n>y = key expires in n years
# Key is valid for? (0) 7d
# Key expires at Tues 08 Mar 2022 02:14:25 PM MDT
# Is this correct? (y/N) y
# 
# pub  4096R/F41870CB  created: 2022-02-28  expires: 2022-03-08  usage: SCEA
#                      trust: unknown       validity: unknown
# sub  4096R/84DB66A2  created: 2022-02-28  expires: never       usage: SEA
# [ unknown] (1). reperalta (For use in reperalta's Big Bang dev environment)
# 
# gpg> save
# 

gpg --list-keys --fingerprint
# pub   rsa4096 2022-02-28 [SCEA] [expires: 2022-03-08]   <---FEEDBACK THAT EXPIRATION HAS BEEN SET
#       CEB7 F6F3 F629 F774 6083  60DE 1984 3848 0DB3 E1E9
# uid           [ultimate] chrism (For use in chrism's Big Bang dev environment)
# sub   rsa4096 2022-02-28 [SEA]
```


## Step 2: Edit SOPS CLI's config file to leverage your generated gpg keypair
```shell
[user@Laptop:~]
cd ~/Desktop/bootstrap
head .sops.yaml
# Inside your instance of the customer template repo
# You'll find a mock config file with fake placeholder values that look like this:

# creation_rules:
# - encrypted_regex: '^(data|stringData)$'
#   # Base is shared, you can add keys from each environment that uses base here
#   pgp: FALSE_KEY_HERE
# #    ,ANOTHER_FALSE_KEY_HERE
```

```text
We will use "sed" AKA "stream editor" CLI utility
to do the CLI equivalent of a find and replace
to find the fake placeholder value, and replace it with a real value
to properly setup a config file that will be used by the SOPS cli utility to encrypt values. 
```

```bash
[user@Laptop:~/Desktop/bootstrap]
## command syntax for Linux Users
sed -i "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml

## command syntax for Mac Users
sed -i "" "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml

head .sops.yaml
# Notice the file was updated with the fingerprint of the GPG key pair you generated

# creation_rules:
# - encrypted_regex: '^(data|stringData)$'
#   # Base is shared, you can add keys from each environment that uses base here
#   pgp: 8CEC83AF96E716A4C7669B037943F697A24A4CCC                   <---- SHOULD NOW LOOK SIMILAR TO THIS
# #    ,ANOTHER_FALSE_KEY_HERE
# ...
```

## Step 3: Create Iron Bank Harbor Image Pull Credentials Secret

You will need pull credentials for Iron Bank to retrieve images for Big Bang.

> Secrets can be specific to an environment if they are located in that environment's folder (e.g. `prod`, `dev`).  Or, they can be shared between environments if located in the `base` directory.

``` shell
# [user@Laptop:~/Desktop/bootstrap]
# Create a new encrypted secret to contain your pull credentials
cd ~/Desktop/bootstrap/base
EDITOR="subl --wait" sops ~/Desktop/bootstrap/base/secrets.enc.
# The following also works (assuming you've configured VS Code to be launchable via CLI)
# EDITOR="code --wait" sops ~/Desktop/bootstrap/base/secrets.enc.
```

Copy paste the following contents to the newly created sops secret.  Put your Iron Bank user/PAT where it states `replace-with-your-iron-bank-user` and `replace-with-your-iron-bank-personal-access-token`. (Remember these can be found on https://registry1.dso.mil, logging in with OIDC provider, and clicking on your user profile in the top right.)

> The name of the secret must be `common-bb` if the secret is in the `base` folder or `environment-bb` if the secret is in the `dev` or `prod` folder.  The `environment-bb` values take precedence over the `common-bb` values.

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: common-bb
stringData:
  values.yaml: |-
    registryCredentials:
      - registry: registry1.dso.mil
        username: replace-with-your-iron-bank-user
        password: replace-with-your-iron-bank-personal-access-token
```

## Step 4: Update the Iron Bank Harbor Image Pull Secret with your credentials
* Make sure you don't forget to update the credentials
```shell
# [user@Laptop:~/Desktop/bootstrap]
EDITOR="subl --wait" sops ~/Desktop/bootstrap/base/secrets.enc.yaml
```

* The following is a copy pasteable command to help validate that you correctly edited the file
```shell
sops --decrypt ~/Desktop/bootstrap/base/secrets.enc.yaml | grep "replace" ; echo $? | grep 0 && for i in {1..10}; do echo "You still need to update the username and password, rerun the edit command"; done || echo "Good you successfully updated your credentials"
```

* The following is a compy pasteable set of commands to test your credentials
```shell
export REGISTRY1_USER=$(sops --decrypt ~/Desktop/bootstrap/base/secrets.enc.yaml | grep username | tr -d " " | cut -d : -f 2)
export REGISTRY1_TOKEN=$(sops --decrypt ~/Desktop/bootstrap/base/secrets.enc.yaml | grep password | tr -d " " | cut -d : -f 2)

echo $REGISTRY1_TOKEN | docker login registry1.dso.mil --username $REGISTRY1_USER --password-stdin | grep "Login Succeeded" ; echo $? | grep 0 && echo "You edited the file correctly and your registry1 credentials work" || echo "Something looks wrong, double check your registry1 credentials"
```

```shell
# Save encrypted secrets into Git
# Configuration changes must be stored in Git to take affect
git status
git add secrets.enc.yaml ../.sops.yaml
git status
git commit -m "feat(gitops)!: added encrypted credentials"
git push
git status
```

> Your private key to decrypt these secrets is stored in your GPG key ring.       
> **WARNING** If you export this key and commit it to a Git Repo, anyone would be able to decrypt secrets in your git repo that were encrypted using the key. 
> That being said... It's common to work on a team, and you'd need to be able to safely share the decryption key with a colleague, or have some DR(Disaster Recovery) where you don't lose your decryption key if your laptop dies.  
> We will use CSP KMS as a key wrapping key to pull this off. CSP KMS uses identity based decryption vs Key based Decryption, you can read more here: https://oteemo.com/hashicorp-vault-is-overhyped-and-mozilla-sops-with-kms-and-git-is-massively-underrated/


### Step 5: Export your generated gpg keypair, encrypt it with CSP KMS, and store it in the git repo for others to use.
```bash
[user@Laptop:~]
# Temporarily export your generated gpg key to your home directory
gpg --armor --export-secret-keys $fp > ~/$NAME-bigbang-gpg-keypair.dev
# (the --armor flag specifies a cleartext encoding of output file vs binary encoding of output file)


# Make a folder in the repo where a kms encrypted version of the keypair can be stored for sharing with your team
mkdir -p ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets
cd ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets


# Encrypt the ~/$NAME-bigbang-gpg-keypair.dev and store it in the git repo
# Note we're referencing a kms key existing in the bb onboarding aws account.
export AWS_PROFILE=bb-onboarding; export AWS_DEFAULT_PROFILE=bb-onboarding
sops --encrypt --kms arn:aws-us-gov:kms:us-gov-west-1:110518024095:key/b6bf63f0-dc65-49b4-acb9-528308195fd6 ~/$NAME-bigbang-gpg-keypair.dev > ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets/$NAME-bigbang-gpg-keypair.encrypted.dev

cat ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets/$NAME-bigbang-gpg-keypair.encrypted.dev | grep '"data"'
# IMPORTANT
# READ
# CAREFULLY
echo -e " IMPORTANT \n READ \n CAREFULLY \n Verify that \"data\": contains a really long value/doesn't have empty double quotes"

########################################################################
# IMPORTANT NOTE:
# kms encryption is superior to asymmetric and symmetric encryption
# because, it allows you to introduce RBAC and give specific identities
# rights to decrypt data, that was encrypted using kms.
#
# Example: you could make a dev kms key and a prod kms key
# give devs rights to the dev key and ops rights to both keys
# Then devs could decrypt only data encrypted with the dev kms key
#
# That being said we're in a lab environment where all attendees are
# sharing the same kms key. So be aware that anything you encrypt with
# kms key can be accessed by the other lab attendees. They can also use
# the shared lab ssh keypair to access your private git repo as well.
#
# Further on in the lab you'll be told to encrypt your personal
# IronBank Access credentials. Don't sweat it as everyone has the same
# read only access, at the end of the residency rotate your read only
# IronBank Credentials.
########################################################################

# Let's cat the file to verify it's encrypted
cat ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets/$NAME-bigbang-gpg-keypair.encrypted.dev
# Verify that it was encrypted with kms key, and that you see pgp: null (not encrypted with pgp)


# Delete your local cleartext copy of the keypair that was in your home directory
rm ~/$NAME-bigbang-gpg-keypair.dev


# Push the encrypted key pair to the repo
git add .
git status
# You should see $NAME-bigbang-gpg-keypair.encrypted.dev listed as a change to be committed
git commit -m "feat(gitops): adding KMS encrypted gpg keypair for dev environment to repo"
git push
```


### Step 6: Dry run through how a team mate would get access to the gpg keypair
```bash
[user@Laptop:~]
echo $fp #Let's verify we have our fingerprint var set
# If you need to repopulate $fp, use the following (copy pasted from above)
export fp=$(gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ' | tr -d '-' | tr -d ' ' | tr -d 'Keyfingerprint=')
echo $fp

# The following will have human input prompts, press y for yes to get through the prompts
gpg --delete-secret-keys $fp
gpg --delete-keys $fp

gpg --list-secret-keys --fingerprint 
# Your keypair is gone / you've successfully simulated being in the position of a team mate who needs to get onboarded

# Assume they've already cloned the git repo
cd ~/Desktop/bootstrap/pre-git-ops-encrypted-secrets
export AWS_PROFILE=bb-onboarding; export AWS_DEFAULT_PROFILE=bb-onboarding
sops --decrypt $NAME-bigbang-gpg-keypair.encrypted.dev | gpg --import -
# gpg: key 7E03CC83EA23AF3E: "chrism (for use in chrism's big bang dev environment) <chrism@bigbang.dev>" not changed
# gpg: key 7E03CC83EA23AF3E: secret key imported
# gpg: Total number processed: 1
# gpg:              unchanged: 1
# gpg:       secret keys read: 1
# gpg:  secret keys unchanged: 1

# (NOTE: I'm not sure where 7E03CC83EA23AF3E comes from)
# (ignore it and trust the identifer given by the 'gpg --list-secret-keys --fingerprint' command)
# Re-Look at your available keypairs
gpg --list-secret-keys --fingerprint
# /Users/chrism/.gnupg/pubring.kbx
# ---------------------------------
# sec   rsa4096 2022-02-28 [SCEA] [expires: 2022-03-08]
#       CEB7 F6F3 F629 F774 6083  60DE 1984 3848 0DB3 E1E9
# uid           [ultimate] chrism (For use in chrism's Big Bang dev environment)
# ssb   rsa4096 2022-02-28 [SEA]
```

### Step 7: Create BigBang namespace and turn the gpg keypair into a Kubernetes Secret that the Flux GitOps Operator can use to decrypt encrypted secrets in the git repo

```shell
# [user@Laptop:~]
kubectl create namespace bigbang 
export KEY_COMMENT="For use in $NAME's Big Bang dev environment"
export fp=$(gpg --list-keys --fingerprint | grep "$KEY_COMMENT" -B 1 | grep -v "$KEY_COMMENT" | tr -d ' ' | tr -d '-' | tr -d ' ' | tr -d 'Keyfingerprint=')
echo $fp #quick check to verify this looks right, similiar to 8CEC83AF96E716A4C7669B037943F697A24A4CCC
gpg --export-secret-key --armor ${fp} | kubectl create secret generic sops-gpg -n bigbang --dry-run=client -o yaml --from-file=bigbangkey.asc=/dev/stdin | kubectl apply -f -
```

### Step 8: Verify the sops-gpg secret has been successfully created
```shell
kubectl get secret -n=bigbang
# NAME                  TYPE                                  DATA   AGE
# default-token-mhm8t   kubernetes.io/service-account-token   3      58s
# sops-gpg              Opaque                                1      18s
```

[Install Default Storage Class](D-install-default-storage-class.md)