diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..d473179276f26628c5984dab352d9fd72e4313f7 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +**/.terra* \ No newline at end of file diff --git a/terraform/README.md b/terraform/README.md new file mode 100644 index 0000000000000000000000000000000000000000..40278d9f78ae7c83c24345c61e4b7e32f886e52b --- /dev/null +++ b/terraform/README.md @@ -0,0 +1,318 @@ +# Big Bang Infrastructure as Code (IaC) + +#### _This is a mirror of a government repo hosted on [Repo1](https://repo1.dso.mil/) by [DoD Platform One](http://p1.dso.mil/). Please direct all code changes, issues and comments to https://repo1.dso.mil/platform-one/big-bang/customers/template_ + +The terraform/terragrunt code in this directory will setup the infrastructure for a Big Bang deployment in Amazon Web Services (AWS). It starts from scratch with a new VPC and finishes by deploying a multi-node [RKE2 Cluster](https://docs.rke2.io/). The cluster can then be used to deploy Big Bang. + +> This code is intended to be a starting point / example for users to get their infrastructure setup quickly. It is up to the users to futher customize and secure the infrastructure for the intended use. + +## Layout + +The following directory tree shows the layout of the the configuration files in this repository. Users should be able to customize the most common items by adjusting values in the `.yaml` files. Additional regions and/or environment directories can be created to maintain multiple deployments without changing the main terraform code. + +```text +terraform +└── main # Shared terraform code +└── us-gov-west-1 # Terragrunt code for a specific AWS region + ├── region.yaml # Regional configuration + └── prod # Teragrunt code for a specific environment (e.g. prod, stage, dev) + └── env.yaml # Environment specific configuration +``` + +## Prerequisites + +- An AWS cloud account with admin privileges +- [Terraform](https://www.terraform.io/downloads.html) +- [Terragrunt](https://terragrunt.gruntwork.io/docs/getting-started/install/) +- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) +- [Kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) + +## Quickstart + +- Review the configuration + - Review [region.yaml](./us-gov-west-1/region.yaml). Update your deployment region if necessary. + - Review [env.yaml](./us-gov-west-1/prod/env.yaml). At a minimum, update `name` to identify your deployment. + +- Validate your configuration + + ```bash + cd ./terraform/us-gov-west-1/prod + terragrunt run-all validate + # Successful output: Success! The configuration is valid. + ``` + +- Run the deployment + + ```bash + # Initialize + terragrunt run-all init + + # Pre-check + terragrunt run-all plan + + # Deploy + terragrunt run-all apply + ``` + +- Connect to cluster + + ```bash + # Setup your cluster name (same as `name` in `env.yaml`) + export CNAME="bigbang-dev" + + # Get Bastion Security Group ID + export BSG=`aws ec2 describe-instances --filters "Name=tag:Name,Values=$CNAME-bastion" --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text` + + # Get your public IP address + export MYIP=`curl -s http://checkip.amazonaws.com/` + + # Add SSH ingress for your IP to the bastion security group + aws ec2 authorize-security-group-ingress --group-id $BSG --protocol tcp --port 22 --cidr $MYIP/32 + + # Get Bastion public IP address + export BIP=`aws ec2 describe-instances --filters "Name=tag:Name,Values=$CNAME-bastion" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text` + + # Use sshuttle to tunnel traffic through bastion public IP + # You can add the '-D' option to sshuttle to run this as a daemon. + # Otherwise, you will need another terminal to continue. + sshuttle --dns -vr ec2-user@$BIP 10.0.0.0/8 --ssh-cmd "ssh -i ~/.ssh/$CNAME.pem" + + # Validate connectivity + kubectl get no + ``` + + > If you get an error with `sshuttle` where Python is not installed, you can manually install it by using the following: + > + > ```shell + > ssh -i ~/.ssh/$CNAME.pem ec2-user@$BIP + > sudo yum update -y + > sudo yum install -y python3 + > exit + > ``` + +The infrastructure is now setup. You still need to configure the [storage class](#storage-class) and [node ports](#node-ports) in the Kubernetes cluster for Big Bang. + +## Big Bang Deployment + +Prior to deploying Big Bang, you should setup the following in the Kubernetes cluster created by the [Quickstart](#quickstart). + +### Storage Class + +Big Bang must have a default storage class. The following will install a storage class for [AWS EBS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html). + +> Without a default storage class, some Big Bang components, like Elasticsearch, Jaeger, or Twistlock, will never reach the running state. + +```bash +kubectl apply -f ./terraform/storageclass/ebs-gp2-storage-class.yaml +``` + +If you have an alternative storage class, you can run the following to replace the EBS GP2 one provided. + +```bash +kubectl patch storageclass ebs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' +kubectl apply -f +# For example... +# Local-path: https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml +# Longhorn: https://raw.githubusercontent.com/longhorn/longhorn/v1.1.0/deploy/longhorn.yaml +kubectl patch storageclass -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' +``` + +### Node Ports + +In order for the external load balancer to map to the RKE2 agents, Istio's Ingress Gateways must be configured to listen and route Node Ports. The following configuration in Big Bang's values.yaml will setup Node Ports to match the [Quickstart](#quickstart) configuration. + +```yaml +# Big Bang's values.yaml +istio: + values: + ingressGateway: + # Use Node Ports instead of creating a load balancer + type: NodePort + ports: + - name: status # Istio's default health check port + port: 15021 + targetPort: 15021 + nodePort: 32021 # Port configured in terraform ELB + - name: http2 + port: 80 + targetPort: 8080 + nodePort: 30080 # Port configured in terraform ELB + - name: https + port: 443 + targetPort: 8443 + nodePort: 30443 # Port configured in terraform ELB + - name: sni # Istio's SNI Routing port + port: 15443 + targetPort: 15443 + nodePort: 32443 # Port configured in terraform ELB +``` + +> The node port values can be customized using the `node_port_*` inputs to the [elb terraform](./modules/elb). + +### Post Deployment + +After Big Bang is deployed, you will need to [setup DNS entries to point to the Elastic Load Balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/using-domain-names-with-elb.html?icmpid=docs_elb_console). You can also connect without DNS using the [debug steps below](#debug) + +## Infrastructure + +### Manifest + +Once the terraform has run, you will have the following resources deployed: + +- [Virtual Private Cloud (VPC)](https://aws.amazon.com/vpc/?vpc-blogs.sort-by=item.additionalFields.createdDate&vpc-blogs.sort-order=desc) +- [Internet Gateway](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Internet_Gateway.html) +- Public subnets + - One for each [availability zone](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) + - VPC CIDR traffic routed locally + - Connected to Internet +- Private subnets + - One for each [availability zone](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) + - VPC CIDR traffic routed locally + - Other traffic routed to [NAT Gateway](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) +- NAT Gateway + - [Elastic IP](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html) assigned for internet access + - Prevents internet ingress to private subnet + - Allows internet egress from private subnet +- [RKE2](https://docs.rke2.io/) Kubernetes Cluster + - RKE2 Control Plane + - RKE2 Servers + - Autoscaled node pool + - Anti-affinity + - RKE2 Agents (Generic) + - Autoscaled node pool + - Anti-affinity + - Security Groups + - Egress not restricted + - Internal cluster ingress allowed + - Control Plane traffic limited to port 6443 and 9345 to servers + - SSH keys created and stored on all nodes. Private key is stored locally in `~/.ssh` +- Elastic Load Balancer + - Internet facing + - Security group allows ingress on ports 80 and 443 only + - Balances loads to RKE2 agents +- [CoreDNS](https://coredns.io/) +- [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) +- Bastion + - Autoscale group insures one bastion is available + - Security group allows SSH on Whitelist IP + - Python installed to allow sshuttle to function +- S3 Storage Bucket + - RKE2 Kubeconfig for accessing cluster + - RKE2 access token for adding additional nodes + +### Diagram + +```mermaid +flowchart TD; + +internet((Internet)) --- igw + +subgraph VPC + igw(Internet Gateway) ---|HTTP / HTTPS Only| elb + + igw(Internet Gateway) ---|SSH\nWhitelist IPs| jump + igw(Internet Gateway) ---|Egress Only| nat1 & nat2 & nat3 + + elb(Elastic Load Balancer) ---|Node Ports| a1 & a2 & a3 + + jump -.- bscale(Autoscale: Bastion) + + subgraph za [Zone A] + nat1 --- zapriv + jump --- zapriv + subgraph zapub [Public Subnet] + nat1(NAT Gateway) + jump(Bastion Server) + end + subgraph zapriv [Private Subnet] + cp(RKE2 Control Plane\nLoad Balancer) + s1(RKE2 Server Node) + a1(RKE2 Agent Node) + end + end + + subgraph zb [Zone B] + nat2 --- zbpriv + subgraph zbpub [Public Subnet] + nat2(NAT Gateway) + end + subgraph zbpriv [Private Subnet] + s2(RKE2 Server Node) + a2(RKE2 Agent Node) + end + end + + subgraph zc [Zone C] + nat3 --- zcpriv + subgraph zcpub [Public Subnet] + nat3(NAT Gateway) + end + subgraph zcpriv [Private Subnet] + s3(RKE2 Server Node) + a3(RKE2 Agent Node) + end + end + + s1 & s2 & s3 -.- sscale(Autoscale: Server Node Pool) + a1 & a2 & a3 -.- ascale(Autoscale: Agent Node Pool) + +end + +subgraph store [S3 Bucket] + subgraph RKE2 + yaml(RKE2 Kubeconfig) + token(RKE2 Access Token) + end +end +``` + +## Debug + +After Big Bang deployment, if you wish to access your deployed web applications that are not exposed publically, add an entry into your /etc/hosts to point the host name to the elastic load balancer. + +> This bypasses load balancing since you are using the resolved IP address of one of the connected nodes in the pool + +```bash +# Setup cluster name from env.yaml +export CName="bigbang-dev" + +# Get VPC info +export VPCId=`aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CName" --query 'Vpcs[*].VpcId' --output text` + +# Get load balancer in VPC that does not contain cluster name +# Istio in Big Bang creates a load balancer +export LBDNS=`aws elb describe-load-balancers --query "LoadBalancerDescriptions[? VPCId == '$VPCId' && "'!'"contains(DNSName, 'internal')].DNSName" --output text` + +# Retrieve IP address of load balancer for /etc/hosts +export ELBIP=`dig $LBDNS +short | head -1` + +# Now add the hostname of the web appliation into /etc/hosts (or `C:\Windows\System32\drivers\etc\hosts` on Windows) +# You may need to log out and back into for hosts to take effect +printf "\nAdd the following line to /etc/hosts to alias Big Bang core products:\n${ELBIP} twistlock.bigbang.dev kibana.bigbang.dev prometheus.bigbang.dev grafana.bigbang.dev tracing.bigbang.dev kiali.bigbang.dev alertmanager.bigbang.dev\n\n" +``` + +## Terraform Destroy + +If you need to teardown the infrastructure, you need to follow this procedure to insure success: + +```shell +# Uninstall Big Bang (sshuttle must be running) +helm delete -n bigbang bigbang + +# Stop sshuttle +pkill sshuttle + +# Destroy +terragrunt run-all destroy +``` + +## Optional Terraform + +Depending on your needs, you may want to deploy additional infrastructure, such as Key Stores, S3 Buckets, or Databases, that can be used with your deployment. In the [options](./options) directory, you will find terraform / terragrunt snippits that can assist you in deploying these items. + +> These examples may required updates to be compatible with the [Quickstart](#quickstart) + +## Additional Resources + +- [Rancher Kubernetes Engine Government (RKE2) Docs](https://docs.rke2.io/) +- [RKE2 AWS Terraform Docs](https://github.com/rancherfederal/rke2-aws-tf) diff --git a/terraform/modules/bastion/dependencies/install_python.sh b/terraform/modules/bastion/dependencies/install_python.sh new file mode 100644 index 0000000000000000000000000000000000000000..790a347474df0c92be857f3feecd1b361731b175 --- /dev/null +++ b/terraform/modules/bastion/dependencies/install_python.sh @@ -0,0 +1,6 @@ +#!/bin/bash +# WARNING: This script will be executed as root on initial boot of the instance + +# Python is required for sshuttle to work +dnf update -y +dnf install -y python3 \ No newline at end of file diff --git a/terraform/modules/bastion/main.tf b/terraform/modules/bastion/main.tf new file mode 100644 index 0000000000000000000000000000000000000000..6a0cd66372414c64e8c067c8f6bff9c84566814f --- /dev/null +++ b/terraform/modules/bastion/main.tf @@ -0,0 +1,59 @@ +# Creates a bastion (aka jump box) instance in the VPC/subnet specified +# - Uses an auto-scale group to provide re-deployment if an availability zone is unavailable +# - No ingress is allowed by default. Users should add their IPs to the security group for access +# - Python is installed on the bastion to allow `sshuttle` to function + +# Security group for bastion +resource "aws_security_group" "bastion_sg" { + name_prefix = "${var.name}-bastion-" + description = "${var.name} bastion" + vpc_id = "${var.vpc_id}" + + # Allow all egress + egress { + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + } + + tags = var.tags +} + +# Bastion Launch Template +resource "aws_launch_template" "bastion" { + + name_prefix = "${var.name}-bastion-" + description = "Bastion launch template for ${var.name} cluster" + image_id = "${var.ami}" + instance_type = "${var.instance_type}" + key_name = "${var.key_name}" + + network_interfaces { + associate_public_ip_address = true + security_groups = ["${aws_security_group.bastion_sg.id}"] + } + + update_default_version = true + user_data = filebase64("${path.module}/dependencies/install_python.sh") + + tag_specifications { + resource_type = "instance" + tags = merge({"Name" = "${var.name}-bastion"}, var.tags) + } +} + +# Bastion Auto-Scaling Group +resource "aws_autoscaling_group" "bastion" { + name_prefix = "${var.name}-bastion-" + max_size = 2 + min_size = 1 + desired_capacity = 1 + + vpc_zone_identifier = var.subnet_ids + + launch_template { + id = aws_launch_template.bastion.id + version = "$Latest" + } +} \ No newline at end of file diff --git a/terraform/modules/bastion/variables.tf b/terraform/modules/bastion/variables.tf new file mode 100644 index 0000000000000000000000000000000000000000..683bd4e620f58e267c61c2733d821b2a6f1cce16 --- /dev/null +++ b/terraform/modules/bastion/variables.tf @@ -0,0 +1,38 @@ +variable "name" { + description = "The project name to prepend to resources" + type = string + default = "bigbang-dev" +} + +variable "vpc_id" { + description = "The VPC where the bastion should be deployed" + type = string +} + +variable "subnet_ids" { + description = "List of subnet ids where the bastion is allowed" + type = list(string) +} + +variable "ami" { + description = "The image to use for the bastion" + type = string + default = "ami-017e342d9500ef3b2" # RKE2 RHEL8 STIG (even though we don't need RHEL8, it is hardened) +} + +variable "instance_type" { + description = "The AWS EC2 instance type for the bastion" + type = string + default = "t2.micro" +} + +variable "key_name" { + description = "The key pair name to install on the bastion" + type = string + default = "" +} +variable "tags" { + description = "The tags to apply to resources" + type = map(string) + default = {} +} \ No newline at end of file diff --git a/terraform/modules/elb/main.tf b/terraform/modules/elb/main.tf new file mode 100644 index 0000000000000000000000000000000000000000..c9d515d433aa7d9fc92ca3e870a17f83b56d8e57 --- /dev/null +++ b/terraform/modules/elb/main.tf @@ -0,0 +1,116 @@ +# Creates an Elastic Load Balancer in the VPC/subnet specified +# - Allows ingress traffic on ports 80 and 443 only +# - Supports Istio health checking and SNI in the cluster +# - Maps to node ports in cluster +# - Security group created for other entities to use for ingress from the ELB +# - Attaching a pool to the load balancer is done outside of this Terraform + +# Security group for load balancer +resource "aws_security_group" "elb" { + name_prefix = "${var.name}-elb-" + description = "${var.name} Elastic Load Balancer" + vpc_id = "${var.vpc_id}" + + # Allow all HTTP traffic + ingress { + description = "HTTP Traffic" + from_port = 80 + to_port = 80 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + + # Allow all HTTPS traffic + ingress { + description = "HTTPS Traffic" + from_port = 443 + to_port = 443 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + + # Allow all egress + egress { + description = "All traffic out" + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + } + + tags = var.tags +} + +# Security group for server pool to allow traffic from load balancer +resource "aws_security_group" "elb_pool" { + name_prefix = "${var.name}-elb-pool-" + description = "${var.name} Traffic to Elastic Load Balancer server pool" + vpc_id = "${var.vpc_id}" + + # Allow all traffic from load balancer + ingress { + description = "Allow Load Balancer Traffic" + from_port = 0 + to_port = 0 + protocol = "-1" + security_groups = [aws_security_group.elb.id] + } + + tags = var.tags +} + +# Create Elastic Load Balancer +module "elb" { + source = "terraform-aws-modules/elb/aws" + version = "~> 3.0" + name = "${var.name}-elb" + subnets = var.subnet_ids + security_groups = [aws_security_group.elb.id] + internal = false + + # Port: Description + # 80: HTTP for applications + # 443: HTTPS for applications + # 15021: Istio Health Checks + # 15443: Istio SNI Routing in multi-cluster environment + listener = [ + { + instance_port = var.node_port_http + instance_protocol = "TCP" + lb_port = 80 + lb_protocol = "tcp" + }, + { + instance_port = var.node_port_https + instance_protocol = "TCP" + lb_port = 443 + lb_protocol = "tcp" + }, + { + instance_port = var.node_port_health_checks + instance_protocol = "TCP" + lb_port = 15021 + lb_protocol = "tcp" + }, + { + instance_port = var.node_port_sni + instance_protocol = "TCP" + lb_port = 15443 + lb_protocol = "tcp" + }, + ] + + health_check = { + target = "TCP:${var.node_port_health_checks}" + interval = 10 + healthy_threshold = 2 + unhealthy_threshold = 6 + timeout = 5 + } + + access_logs = {} + + tags = merge({ + "kubernetes.io/cluster/${var.name}" = "shared" + }, var.tags) +} \ No newline at end of file diff --git a/terraform/modules/elb/outputs.tf b/terraform/modules/elb/outputs.tf new file mode 100644 index 0000000000000000000000000000000000000000..089e6667e7c77c8a3ee9223b7cb59a6399b8c87c --- /dev/null +++ b/terraform/modules/elb/outputs.tf @@ -0,0 +1,9 @@ +output "elb_id" { + description = "The Elastic Load Balancer (ELB) ID" + value = module.elb.elb_id +} + +output "pool_sg_id" { + description = "The ID of the security group used as an inbound rule for load balancer's back-end application instances" + value = aws_security_group.elb_pool.id +} \ No newline at end of file diff --git a/terraform/modules/elb/variables.tf b/terraform/modules/elb/variables.tf new file mode 100644 index 0000000000000000000000000000000000000000..848b9cbcaf878059767252aa7567ecc7e29bd285 --- /dev/null +++ b/terraform/modules/elb/variables.tf @@ -0,0 +1,45 @@ +variable "name" { + description = "The name to apply to the external load balancer resources" + type = string + default = "bigbang-dev" +} + +variable "vpc_id" { + description = "The VPC where the load balancer should be deployed" + type = string +} + +variable "subnet_ids" { + description = "The subnet ids to load balance" + type = list(string) +} + +variable "node_port_http" { + description = "The node port to use for HTTP traffic" + type = string + default = "30080" +} + +variable "node_port_https" { + description = "The node port to use for HTTPS traffic" + type = string + default = "30443" +} + +variable "node_port_health_checks" { + description = "The node port to use for Istio health check traffic" + type = string + default = "32021" +} + +variable "node_port_sni" { + description = "The node port to use for Istio SNI traffic" + type = string + default = "32443" +} + +variable "tags" { + description = "The tags to apply to resources" + type = map(string) + default = {} +} \ No newline at end of file diff --git a/terraform/modules/k8s/main.tf b/terraform/modules/k8s/main.tf new file mode 100644 index 0000000000000000000000000000000000000000..1924376a510a69fab674ccbc7da2d1ce403e667b --- /dev/null +++ b/terraform/modules/k8s/main.tf @@ -0,0 +1,28 @@ +# After the cluster is setup, this script will retrieve the Kubeconfig +# file from S3 storage and merge in the local ~/.kube/config + +# Retrieves kubeconfig +resource "null_resource" "kubeconfig" { + triggers = { + kubeconfig_path = var.kubeconfig_path + } + provisioner "local-exec" { + interpreter = ["bash", "-c"] + command = <<-EOF + # Get kubeconfig from storage + aws s3 cp ${var.kubeconfig_path} ~/.kube/new + + # Merge new config into existing + export KUBECONFIGBAK=$KUBECONFIG + export KUBECONFIG=~/.kube/new:~/.kube/config + # Do not redirect to ~/.kube/config or you may truncate the results + kubectl config view --flatten > ~/.kube/merged + mv -f ~/.kube/merged ~/.kube/config + + # Cleanup + rm -f ~/.kube/new + export KUBECONFIG=$KUBECONFIGBAK + unset KUBECONFIGBAK + EOF + } +} \ No newline at end of file diff --git a/terraform/modules/k8s/variables.tf b/terraform/modules/k8s/variables.tf new file mode 100644 index 0000000000000000000000000000000000000000..a7c10bf00f65a9b0a94548454f02688696f1d196 --- /dev/null +++ b/terraform/modules/k8s/variables.tf @@ -0,0 +1,4 @@ +variable "kubeconfig_path" { + description = "Remote path to kubeconfig" + type = string +} \ No newline at end of file diff --git a/terraform/modules/pool/main.tf b/terraform/modules/pool/main.tf new file mode 100644 index 0000000000000000000000000000000000000000..7e656c85b0f5203f4a01b664258267a70338cb43 --- /dev/null +++ b/terraform/modules/pool/main.tf @@ -0,0 +1,6 @@ +# Connects an Elastic Load Balancer to a pool of servers + +resource "aws_autoscaling_attachment" "pool" { + elb = var.elb_id + autoscaling_group_name = var.pool_asg_id +} diff --git a/terraform/modules/pool/variables.tf b/terraform/modules/pool/variables.tf new file mode 100644 index 0000000000000000000000000000000000000000..1adae132d833c66a6e3ed12889c012131f9219da --- /dev/null +++ b/terraform/modules/pool/variables.tf @@ -0,0 +1,9 @@ +variable "elb_id" { + description = "The load balancer ID to attach the pool" + type = string +} + +variable "pool_asg_id" { + description = "The autoscale group IDs that make up the pool to attach to the load balancer" + type = string +} \ No newline at end of file diff --git a/terraform/modules/ssh/main.tf b/terraform/modules/ssh/main.tf new file mode 100644 index 0000000000000000000000000000000000000000..562d3c02e6013deca3c4d8d7029f5cb05a887cd8 --- /dev/null +++ b/terraform/modules/ssh/main.tf @@ -0,0 +1,22 @@ +# Creates a SSH key pair that can be used for access to infrastructure +# servers. The private key is stored on the local computer only in the +# path specified by the variables + +# Create SSH key pair +resource "tls_private_key" "ssh" { + algorithm = "RSA" + rsa_bits = 4096 +} + +# Store private key locally +resource "local_file" "pem" { + filename = pathexpand("${var.private_key_path}/${var.name}.pem") + content = tls_private_key.ssh.private_key_pem + file_permission = "0600" +} + +# +resource "aws_key_pair" "ssh" { + key_name = "${var.name}" + public_key = tls_private_key.ssh.public_key_openssh +} \ No newline at end of file diff --git a/terraform/modules/ssh/outputs.tf b/terraform/modules/ssh/outputs.tf new file mode 100644 index 0000000000000000000000000000000000000000..bce9f0ca9d10697c53ee25c09152949b2e69efd6 --- /dev/null +++ b/terraform/modules/ssh/outputs.tf @@ -0,0 +1,9 @@ +output "key_name" { + description = "The name of the AWS SSH key pair" + value = aws_key_pair.ssh.key_name +} + +output "public_key" { + description = "The public SSH key" + value = tls_private_key.ssh.public_key_openssh +} \ No newline at end of file diff --git a/terraform/modules/ssh/variables.tf b/terraform/modules/ssh/variables.tf new file mode 100644 index 0000000000000000000000000000000000000000..93e095f0a6fd3b2526bcf40faa73df6a1fced132 --- /dev/null +++ b/terraform/modules/ssh/variables.tf @@ -0,0 +1,11 @@ +variable "private_key_path" { + description = "Local path to store private key for SSH" + type = string + default = "~/.ssh" +} + +variable "name" { + description = "Name of the SSH keypair to create" + type = string + default = "bigbang" +} \ No newline at end of file diff --git a/terraform/modules/vpc/data.tf b/terraform/modules/vpc/data.tf new file mode 100644 index 0000000000000000000000000000000000000000..56bdab41afc73827112d4e2bbb7e7de8598d7190 --- /dev/null +++ b/terraform/modules/vpc/data.tf @@ -0,0 +1,7 @@ +data "aws_availability_zones" "available" { + state = "available" + filter { + name = "group-name" + values = [var.aws_region] + } +} \ No newline at end of file diff --git a/terraform/modules/vpc/main.tf b/terraform/modules/vpc/main.tf new file mode 100644 index 0000000000000000000000000000000000000000..0cfc799282bad1508de57943237b51b873e7b878 --- /dev/null +++ b/terraform/modules/vpc/main.tf @@ -0,0 +1,66 @@ +# Creates an AWS Virtual Private Cloud (VPC) with the following: +# - Public subnets in each availability zone +# - Private subnets in each availability zone +# - NAT gateways in each public subnet +# - All traffic from private subnets routed through NAT Gateway +# - No ingress allowed from internet on NAT Gateways +# - Elastic IPs assigned to each NAT Gateway +# - Internal / external load balancer tags assigned to subnets + +locals { + # Number of availability zones determines number of CIDRs we need + num_azs = length(data.aws_availability_zones.available.names) + + # Size of the CIDR range, this is added to the VPC CIDR bits + # For example if the VPC CIDR is 10.0.0.0/16 and the CIDR size is 8, the CIDR will be 10.0.xx.0/24 + cidr_size = 8 + + # Step of CIDR range. How much space to leave between CIDR sets (public, private, intra) + cidr_step = max(10, local.num_azs) + + # Based on VPC CIDR, create subnet ranges + cidr_index = range(local.num_azs) + public_subnet_cidrs = [ for i in local.cidr_index : cidrsubnet(var.vpc_cidr, local.cidr_size, i) ] + private_subnet_cidrs = [ for i in local.cidr_index : cidrsubnet(var.vpc_cidr, local.cidr_size, i + local.cidr_step) ] +} + +# https://github.com/terraform-aws-modules/terraform-aws-vpc +module "vpc" { + source = "terraform-aws-modules/vpc/aws" + + name = var.name + cidr = var.vpc_cidr + + azs = data.aws_availability_zones.available.names + public_subnets = local.public_subnet_cidrs + private_subnets = local.private_subnet_cidrs + + # If you have resources in multiple Availability Zones and they share one NAT gateway, + # and if the NAT gateway’s Availability Zone is down, resources in the other Availability + # Zones lose internet access. To create an Availability Zone-independent architecture, + # create a NAT gateway in each Availability Zone. + enable_nat_gateway = true + single_nat_gateway = false + one_nat_gateway_per_az = true + + enable_dns_hostnames = true + enable_dns_support = true + + # Create EIPs for NAT gateways + reuse_nat_ips = false + + # Add in required tags for proper AWS CCM integration + public_subnet_tags = merge({ + "kubernetes.io/cluster/${var.name}" = "shared" + "kubernetes.io/role/elb" = "1" + }, var.tags) + + private_subnet_tags = merge({ + "kubernetes.io/cluster/${var.name}" = "shared" + "kubernetes.io/role/internal-elb" = "1" + }, var.tags) + + tags = merge({ + "kubernetes.io/cluster/${var.name}" = "shared" + }, var.tags) +} \ No newline at end of file diff --git a/terraform/modules/vpc/outputs.tf b/terraform/modules/vpc/outputs.tf new file mode 100644 index 0000000000000000000000000000000000000000..adc3008b721a1311b726503f79861cd4ea13c871 --- /dev/null +++ b/terraform/modules/vpc/outputs.tf @@ -0,0 +1,14 @@ +output "vpc_id" { + description = "The Virtual Private Cloud (VPC) ID" + value = module.vpc.vpc_id +} + +output "private_subnet_ids" { + description = "The list of private subnet IDs in the VPC" + value = module.vpc.private_subnets +} + +output "public_subnet_ids" { + description = "Thge list of public subnet IDs in the VPC" + value = module.vpc.public_subnets +} \ No newline at end of file diff --git a/terraform/modules/vpc/variables.tf b/terraform/modules/vpc/variables.tf new file mode 100644 index 0000000000000000000000000000000000000000..c7eea97aa095d3d076fa9d5e3558013d5caf65d7 --- /dev/null +++ b/terraform/modules/vpc/variables.tf @@ -0,0 +1,22 @@ +variable "name" { + description = "The name to apply to the VPC and Subnets" + type = string + default = "bigbang-dev" +} + +variable "vpc_cidr" { + description = "The CIDR block for the VPC in the format xx.xx.xx.xx/xx" + type = string +} + +variable "aws_region" { + description = "The AWS region to deploy resources" + type = string + default = "us-gov-west-1" +} + +variable "tags" { + description = "The tags to apply to resources" + type = map(string) + default = {} +} \ No newline at end of file diff --git a/terraform/options/database/mysql/terragrunt.hcl b/terraform/options/database/mysql/terragrunt.hcl new file mode 100644 index 0000000000000000000000000000000000000000..5da12af0274b9acf6ab2899d37323a2185b96ae8 --- /dev/null +++ b/terraform/options/database/mysql/terragrunt.hcl @@ -0,0 +1,45 @@ +locals { + env = merge( + yamldecode(file(find_in_parent_folders("region.yaml"))), + yamldecode(file(find_in_parent_folders("env.yaml"))) + ) +} + +terraform { + source = "git::https://github.com/terraform-aws-modules/terraform-aws-rds-aurora.git?ref=v3.5.0" +} + +include { + path = find_in_parent_folders() +} + +dependency "server" { + config_path = "../server" + mock_outputs = { + cluster_sg = "mock_cluster_sg" + } +} + +inputs = { + name = "${local.env.locals.name}-mysql" + + vpc_id = local.env.locals.vpc_id + subnets = local.env.locals.subnets + + engine = "aurora-mysql" + engine_version = "5.7.12" + + replica_count = 1 + + allowed_security_groups = [dependency.server.cluster_sg] +// allowed_cidr_blocks = ["10.20.0.0/20"] + instance_type = "db.t2.small" + storage_encrypted = true + apply_immediately = true + + username = "bigbang" + skip_final_snapshot = true + ca_cert_identifier = "rds-ca-2017" + + tags = merge({}, local.env.locals.tags) +} diff --git a/terraform/options/database/postgres/terragrunt.hcl b/terraform/options/database/postgres/terragrunt.hcl new file mode 100644 index 0000000000000000000000000000000000000000..b22afa0ddc0aa8ca83debd7804f5bee4ec587db0 --- /dev/null +++ b/terraform/options/database/postgres/terragrunt.hcl @@ -0,0 +1,45 @@ +locals { + env = merge( + yamldecode(file(find_in_parent_folders("region.yaml"))), + yamldecode(file(find_in_parent_folders("env.yaml"))) + ) +} + +terraform { + source = "git::https://github.com/terraform-aws-modules/terraform-aws-rds-aurora.git?ref=v3.5.0" +} + +include { + path = find_in_parent_folders() +} + +dependency "server" { + config_path = "../server" + mock_outputs = { + cluster_sg = "mock_cluster_sg" + } +} + +inputs = { + name = "${local.env.locals.name}-postgres" + + vpc_id = local.env.locals.vpc_id + subnets = local.env.locals.subnets + + engine = "aurora-postgresql" + engine_version = "12.4" + + replica_count = 1 + + allowed_security_groups = [dependency.server.outputs.cluster_sg] +// allowed_cidr_blocks = ["10.20.0.0/20"] + instance_type = "db.t3.medium" + storage_encrypted = true + apply_immediately = true + + username = "bigbang" + skip_final_snapshot = true + ca_cert_identifier = "rds-ca-2017" + + tags = merge({}, local.env.locals.tags) +} diff --git a/terraform/options/kms/main.tf b/terraform/options/kms/main.tf new file mode 100644 index 0000000000000000000000000000000000000000..af47a4c8de3eca2e99d34583f06955280bdc03e4 --- /dev/null +++ b/terraform/options/kms/main.tf @@ -0,0 +1,20 @@ +resource "aws_kms_key" "this" { + description = "${var.name} key" + enable_key_rotation = true + key_usage = "ENCRYPT_DECRYPT" + + tags = merge({}, var.tags) +} + +resource "aws_kms_grant" "grants" { + count = length(var.principal_grants) + + grantee_principal = var.principal_grants[count.index] + key_id = aws_kms_key.this.key_id + operations = ["Decrypt"] +} + +resource "aws_kms_alias" "this" { + name = "alias/${var.name}" + target_key_id = aws_kms_key.this.key_id +} diff --git a/terraform/options/kms/outputs.tf b/terraform/options/kms/outputs.tf new file mode 100644 index 0000000000000000000000000000000000000000..109d4d71f45d238568f692629551075ca38384db --- /dev/null +++ b/terraform/options/kms/outputs.tf @@ -0,0 +1,11 @@ +output "alias" { + value = aws_kms_alias.this.name +} + +output "id" { + value = aws_kms_key.this.key_id +} + +output "arn" { + value = aws_kms_key.this.arn +} \ No newline at end of file diff --git a/terraform/options/kms/variables.tf b/terraform/options/kms/variables.tf new file mode 100644 index 0000000000000000000000000000000000000000..91f5346362744e0058893386619e8575d5151ce1 --- /dev/null +++ b/terraform/options/kms/variables.tf @@ -0,0 +1,12 @@ +variable "name" {} + +variable "principal_grants" { + type = list(string) + description = "principals to grant Decrypt to" + default = [] +} + +variable "tags" { + type = map(string) + default = {} +} \ No newline at end of file diff --git a/terraform/options/s3/gitlab/main.tf b/terraform/options/s3/gitlab/main.tf new file mode 100644 index 0000000000000000000000000000000000000000..8aac792e16f49d46f26189e99118aa8e7faf24cb --- /dev/null +++ b/terraform/options/s3/gitlab/main.tf @@ -0,0 +1,211 @@ +module "registry_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-registry" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "lfs_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-lfs" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "artifacts_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-artifacts" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "uploads_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-uploads" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "tfstate_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-tfstate" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "packages_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-packages" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "pseudonymizer_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-pseudonymizer" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "backups_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-backups" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "backups-tmp_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-backups-tmp" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "dependency_proxy_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-dependency-proxy" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "mr_diffs_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-mr-diffs" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +module "runner_cache_bucket" { + source = "terraform-aws-modules/s3-bucket/aws" + bucket = "${var.name}-gitlab-runner-cache" + acl = "private" + force_destroy = var.bucket_force_destroy + + versioning = { + enabled = false + } + + tags = merge({}, var.tags) +} + +resource "aws_iam_user" "this" { + name = "${var.name}-gitlab-objectstore" + path = "/" + + tags = merge({}, var.tags) +} + +resource "aws_iam_access_key" "this" { + user = aws_iam_user.this.name +} + +resource "aws_iam_user_policy" "all_access" { + name = "${var.name}-gitlab-objecstore-all" + user = aws_iam_user.this.name + + policy = < /etc/sysctl.d/vm-max_map_count.conf + + # Increase file descriptor and process limits for packages (e.g. SonarQube) + sysctl -w fs.file-max=131072 + ulimit -n 131072 + ulimit -u 8192 + + # Preload kernel modules required by istio-init, required for selinux enforcing instances using istio-init + modprobe xt_REDIRECT + modprobe xt_owner + modprobe xt_statistic + + # Persist modules after reboots + printf "xt_REDIRECT\nxt_owner\nxt_statistic\n" | sudo tee -a /etc/modules + +bastion: + # RHEL 8.3 RKE2 v1.20.5+rke2r1 STIG. We don't need RKE2, but take advantage of this being hardened. + image: "ami-017e342d9500ef3b2" + + # Number of replicas + replicas: 1 + + # EC2 Instance type + type: "t2.micro" + +tags: + Environment: "prod" \ No newline at end of file diff --git a/terraform/us-gov-west-1/prod/main/terragrunt.hcl b/terraform/us-gov-west-1/prod/main/terragrunt.hcl new file mode 100644 index 0000000000000000000000000000000000000000..dbcdbe5b933049ac54f19a0350de1d58128bb59c --- /dev/null +++ b/terraform/us-gov-west-1/prod/main/terragrunt.hcl @@ -0,0 +1,20 @@ +# This file performs post-cluster actions, like downloading the kubeconfig + +terraform { + source = "${path_relative_from_include()}//modules/k8s" +} + +include { + path = find_in_parent_folders() +} + +dependency "server" { + config_path = "../server" + mock_outputs = { + kubeconfig_path = "kubeconfig_mock_path" + } +} + +inputs = { + kubeconfig_path = dependency.server.outputs.kubeconfig_path +} \ No newline at end of file diff --git a/terraform/us-gov-west-1/prod/pool/terragrunt.hcl b/terraform/us-gov-west-1/prod/pool/terragrunt.hcl new file mode 100644 index 0000000000000000000000000000000000000000..46c6a03b0fb459eccb49b9590771f501de24f8fb --- /dev/null +++ b/terraform/us-gov-west-1/prod/pool/terragrunt.hcl @@ -0,0 +1,34 @@ +locals { + env = merge( + yamldecode(file(find_in_parent_folders("region.yaml"))), + yamldecode(file(find_in_parent_folders("env.yaml"))) + ) +} + +terraform { + source = "${path_relative_from_include()}//modules/pool" +} + +include { + path = find_in_parent_folders() +} + +dependency "elb" { + config_path = "../elb" + mock_outputs = { + elb_id = "mock_elb_id" + } +} + +dependency "agent" { + config_path = "../agent" + mock_outputs = { + nodepool_id = "mock_nodepool_id" + } +} + +inputs = { + elb_id = dependency.elb.outputs.elb_id + pool_asg_id = dependency.agent.outputs.nodepool_id + tags = merge(local.env.region_tags, local.env.tags, {}) +} \ No newline at end of file diff --git a/terraform/us-gov-west-1/prod/server/terragrunt.hcl b/terraform/us-gov-west-1/prod/server/terragrunt.hcl new file mode 100644 index 0000000000000000000000000000000000000000..176e007235f2d76f56e4d56f0663199365f62128 --- /dev/null +++ b/terraform/us-gov-west-1/prod/server/terragrunt.hcl @@ -0,0 +1,62 @@ +# This file sets up the RKE2 cluster servers in AWS using an autoscale group + +locals { + env = merge( + yamldecode(file(find_in_parent_folders("region.yaml"))), + yamldecode(file(find_in_parent_folders("env.yaml"))) + ) +} + +terraform { + source = "git::https://repo1.dsop.io/platform-one/distros/rancher-federal/rke2/rke2-aws-terraform.git//?ref=v1.1.8" +} + +include { + path = find_in_parent_folders() +} + +dependency "vpc" { + config_path = "../vpc" + mock_outputs = { + vpc_id = "vpc-mock" + private_subnet_ids = ["mock_priv_subnet1", "mock_priv_subnet2", "mock_priv_subnet3"] + } +} + +dependency "ssh" { + config_path = "../ssh" + mock_outputs = { + public_key = "mock_public_key" + } +} + +inputs = { + cluster_name = local.env.name + vpc_id = dependency.vpc.outputs.vpc_id + subnets = dependency.vpc.outputs.private_subnet_ids + ami = local.env.cluster.server.image + servers = local.env.cluster.server.replicas + instance_type = local.env.cluster.server.type + download = false + enable_ccm = true + + block_device_mappings = { + size = local.env.cluster.server.storage.size + encrypted = local.env.cluster.server.storage.encrypted + type = local.env.cluster.server.storage.type + } + + ssh_authorized_keys = [dependency.ssh.outputs.public_key] + + pre_userdata = local.env.cluster.init_script + + tags = merge(local.env.region_tags, local.env.tags, {}) + + # Big Bang uses Istio instead of NGINX + # https://docs.rke2.io/advanced/#disabling-server-charts/ + rke2_config = <