Improve k3d cluster management, especially for multiple clusters

Package Merge Request

Package Changes

This MR adds the following functionality to the k3d-dev script:

• Manage multiple cluster instance(s) based on a Project tag (which defaults to default)
• Update the security groups of an existing cluster instance to reflect your current IP
• Give the ability to report on the your cluster instances, their Project tag and their state

$ ./k3d-dev.sh -h
# ... snip 
 -t   Set the project tag on the instance
 -r   Report on all instances owned by your user
 -u   Update security group for instances

Managing Multiple Cluster Instances

The script now adds a Project tag to the created EC2 instances, and includes the Project name in the name of the Security Group, SSH Key and ~/.kube/config file. The default value is default, so the script can still be called the same way it always was with no issue.

You can control this Project value by passing -t <TAG_NAME> to provide a different value. This allows me to have more than one k3d cluster running at the same time. If I want to run one cluster to perform some testing on external-secrets operator (ESO) and one on vault, I can run these commands:

$ ./k3d-dev.sh -t vault
$ ./k3d-dev.sh -t external-secrets

... and get two completely different instances, SSH Keys, Security groups, and kube configs:

$ aws ec2 describe-instances \
    --filters "Name=tag:Name,Values=Andrew.Kesterson-dev" "Name=instance-state-name,Values=running" \
    --query 'Reservations[*].Instances[*].[InstanceId,State.Name,PublicIpAddress,SecurityGroups[0].GroupId,Tags[?Key==`Project`].Value | [0]]' \
    --output text
i-0407f487682a033b2     running 15.205.147.83   sg-0b6baea2e55994a8a    external-secrets
i-0572c8fde40535757     running 160.1.16.174    sg-0a9bd4ecaa44e06ff    vault

$ ls -l ~/.ssh/Andrew.Kesterson-dev-*pem
-rw-------@ 1 andrewkesterson2  staff  1675 Oct 23 12:29 /Users/andrewkesterson2/.ssh/Andrew.Kesterson-dev-external-secrets.pem
-rw-------@ 1 andrewkesterson2  staff  1675 Oct 23 12:17 /Users/andrewkesterson2/.ssh/Andrew.Kesterson-dev-vault.pem

$ ls -l ~/.kube/*config*
-rw-------@ 1 andrewkesterson2  staff  3025 Oct 23 15:50 /Users/andrewkesterson2/.kube/Andrew.Kesterson-dev-external-secrets-config
-rw-------@ 1 andrewkesterson2  staff  3032 Oct 23 16:24 /Users/andrewkesterson2/.kube/Andrew.Kesterson-dev-vault-config

And we can switch between them by setting a different KUBECONFIG environment variable.

Updating security groups for a new IP

If you work on the road, you may experience your IP changing, and locking you out of your instance. You could open up the AWS UI and change it there, or through the CLI, but now you can just do this:

$ ./k3d-dev.sh -u -t vault                                                                                
AWS User Name: Andrew.Kesterson                                                                                                                                          
-t option passed to use additional tags on instance                                                                                                                      
Retrieving ID for security group Andrew.Kesterson-dev-vault ...done                                                                                                      
Checking if 174.130.2.225 is authorized in security group ...missing                                                                                                     
Adding 174.130.2.225 to security group Andrew.Kesterson-dev-vault ...                                                                                                    
{                                                                                                                                                                        
    "Return": true,                                                                                                                                                      
    "SecurityGroupRules": [                                                                                                                                              
        {                                                                                                                                                                
            "SecurityGroupRuleId": "sgr-0434de2efdeafdf81",                                                                                                              
            "GroupId": "sg-0a9bd4ecaa44e06ff",                                                                                                                           
            "GroupOwnerId": "141078740716",                                                                                                                              
            "IsEgress": false,                                                                                                                                           
            "IpProtocol": "-1",                                                                                                                                          
            "FromPort": -1,                                                                                                                                              
            "ToPort": -1,                                                                                                                                                
            "CidrIpv4": "174.130.2.225/32"                                                                                                                               
        }                                                                                                                                                                
    ]                                                                                                                                                                    
}                                                                                                                                                                        
done 

The same -t <PROJECT_NAME> syntax is supported here. If you don't provide -t <TAG_NAME> the default is assumed.

Reporting on the status of your k3d instances

There is a new -r command that will print a report of your running k3d instances. It does not honor -t <TAG_NAME>, it reports on all instances.

$ ./k3d-dev.sh -r
AWS User Name: Andrew.Kesterson
i-0407f487682a033b2     running 15.205.147.83   sg-0b6baea2e55994a8a    external-secrets
i-0572c8fde40535757     running 160.1.16.174    sg-0a9bd4ecaa44e06ff    vault

The output includes the instance ID, its state, public IP, security group ID, and the Project tag (-t) value.

Untested / Known to Not Work

The -t <TAG_NAME> functionality has been applied to Elastic IP creation/checking, but I don't use anything that exercises elastic IP addresses, so I haven't checked it. It probably works?

The -u flag does not remove old IP address entries from the security group, it only adds new ones.

Package MR

N/A

For Issue

N/A

Upgrade Notices

N/A

changed milestone to %2.39.0

added statusreview label

assigned to @akesterson

requested review from @michaelmartin, @chris.oconnell, @andrewshoell, and @troymobley

added 1 commit

• 0b43aaf4 - Add functionality to k3d-dev.sh to manage multiple clusters, report the state...

Compare with previous version

marked this merge request as ready

changed the description

added 1 commit

• 1edac0bf - Add functionality to k3d-dev.sh to manage multiple clusters, report the state...

Compare with previous version

changed the description

requested review from @cbowmanclare

added 1 commit

• 7dc99e76 - clean up help arguments a bit

Compare with previous version

looks good and works for me! nice work! I added a few, small changes to the help usage print out.

approved this merge request

resolved all threads

changed milestone to %2.40.0