README.md

# k3d-dev-env

Deploy a multi-node k8s cluster on an AWS EC2 instance using terraform, ansible, and k3d; primarily for deploying the bigbang baseline.

# Requirements

## Supported Systems

Other systems may be used, but documentation is written for the following:
* MacOS

## Install Packages

Install the required packages (MacOS):

```
brew install awscli terraform ansible kubectl
```

## AWS Configuration

**Note**: You will need to obtain access to a supported AWS account and region to use this project.

All required AWS resources are stored in the BigBang AWS account, and shared to several other supported accounts and regions.

Using unlisted AWS accounts or regions is currently not supported.

Only the following AWS account IDs are supported:
* `927962728993`
* `792771988927`

Only the following AWS regions are supported:
* `us-gov-west-1`

Configure your AWS credentials and region:
* Make sure this region equals the `aws_region` variable.

```
aws configure
```

## SSH Key

Make sure to generate an SSH key:
* If you use something other than `<home_full_path>/.ssh/id_rsa`, make sure to update the `private_key_path` and the `public_key_path` variables.

```
ssh-keygen
```

## Variable File Configuration

Set up (and optionally modify) your `terraform.tfvars.json` file:

```
cp terraform.tfvars.json.example terraform.tfvars.json
```

# Initialization

Check out the latest stable release into a unique local dev branch you own:

```
release=$(git describe --match "release-v*" --abbrev=0 --tags master)
git fetch --tags origin
git checkout tags/$release -b <your_local_dev_branch_name>
git reset --hard $release
```

If you already have a unique local dev branch, bring in the latest release changes:

```
git fetch --all
release=$(git describe --match "release-v*" --abbrev=0 --tags master)
git checkout <your_local_dev_branch_name>
git merge $release
```

Initialize the terraform module, this only needs to be done once:

```
terraform init
```

# Variables / Customization

All custom variables must be defined in the `terraform.tfvars.json` file.

This file must exist, and is the only tfvars file type allowed.

**No other method of passing variables is supported.**

This is due to limitations with passing terraform variables to ansible.

# Running

Steps that will be done after application:
* A security group will be created that only allows your IP ingress.
* Your SSH key will be created and linked as a new AWS keypair.
* A spot EC2 instance is spawned from the `k3d-dev-env` AMI.
* A k3d cluster will be created according to set requirements.
* (Optional) Your local kubeconfig file will be overwritten.
* Output information will be populated for debugging use.

Plan the terraform module:

```
terraform plan -out=.tfplan
```

Apply the terraform module:

```
terraform apply .tfplan
```

# Verifying

Check to see if your cluster is up and running:

```
kubectl get nodes -o wide
```

# Proxying

Several methods of ingress proxying can be used, here is one example:

```
sudo ssh -i $(terraform output private_key_path) -L 443:127.0.0.1:443 $(terraform output ami_user)@$(terraform output instance_ip)
```

# Re-Deploying

If you want to re-deploy easily, you can taint and re-apply the EC2 instance:

```
terraform taint aws_spot_instance_request.k3d
terraform apply --auto-approve
```

# Destroying

Please make sure to destroy your resources when not in use:

```
terraform destroy --auto-approve
```

# Other Notes

* The VPC / security group is tied to your local client public IP address, only this IP address will have access to the created infrastructure. If you need to open this up, do so using the AWS web management console or the AWS command line tool.
* Each AWS resource is tagged with a name and an owner, this allows you to more easily search for the infrastructure in the AWS console for debugging; the owner tag will be the top-level username retrieved from the configured AWS credentials.
* Check to make sure you're not connected to a VPN, and that your local firewall filter is not causing issues. Both of these may conflict with some of the AWS resources, and are hard to debug. If possible, disconnect from your VPN and disable your firewall filter.

# Cheat Sheet

Additionally, you can also generate an `/etc/hosts` entry with all virtual service gateways in your cluster:

```
echo "$(terraform output instance_ip) $(kubectl get vs -A -o json | jq -r '[.items[] | .spec.hosts[]] | join (" ")')"
```

The output should be similar to this:

```
111.1.111.111 jaeger.bigbang.dev kiali.bigbang.dev grafana.bigbang.dev prometheus.bigbang.dev alertmanager.bigbang.dev twistlock.bigbang.dev
```

If you use other clusters, leverage the `KUBECONFIG` environment variable along with the `kubeconfig_path` terraform variable.

```
export KUBECONFIG="<home_full_path>/.kube/config:<home_full_path>/.kube/k3d-dev-env
```

# Notable Features

## Airgap Support

Test airgap capability is available by using the following variables:

* `k3d_upload_images` - Optionally load images from tar into k3d cluster
* `k3d_images_tarball` - Local path to the images tarball (tar.gz) to load into k3d
* `aws_airgap` - Optionally disallow any egress except to the client machine

Combining these variables allows you to test an airgap deployment of an application using a previously generated tarball of container images.

To generate this tarball, leverage the `hack/bundle_images.sh` script.

Also provided are `hack/k8s_dashboard_images.txt` and `hack/k8s_dashboard.yaml`, these files can be used to test an example deployment of the Kubernetes Dashboard in an airgapped enviornment.

An example tarball generation using this image list would be:

```
./hack/bundle_images.sh --image-list hack/k8s_dashboard_images.txt --images hack/images.tar.gz
```

With the additional variable settings in `terraform.tfvars.json` of:

```
{
    ...
    "k3d_upload_images": true,
    "k3d_images_tarball": "<project_full_path>/hack/images.tar.gz",
    "aws_airgap": true,
    ...
}
```

All images provided in the tarball are loaded before cluster creation into each node, and all baseline images required by k3d / k3s are already loaded into the cluster (regardless of airgap variable settings), so there is no need to specify any k3d / k3s cluster specific container images.

# Contributing

Additional documentation can be found in several README files throughout the repository.

These files are meant to assist baseline development for this module and its components.

If you would like to request a feature or report a bug, please [open a GitLab issue](https://repo1.dso.mil/platform-one/big-bang/terraform-modules/k3d-dev-env/-/issues).

If you would like to submit a merge request, refer to the [contributor documentation](./CONTRIBUTING.md) for more information.