General MR

Summary

This fixes the issue where a metallb load balancer cannot be deployed. Before this fix, you would get one or more errors related to SCP or metallb timeouts. With this fix, you get a working metallb.

This also fixes an issue in the "Big Bang Cluster Management menu" the prevented the "Recreate the instance from scratch" option from working correctly
Corrects an incorrect (or at least suboptimal) temporary directory setting (~/aws/ versus a better mktemp -d)
Abandons a pattern wherein metallb configuration is constructed by "heredocs within heredocs" (templates within templates), which may have contributed to failure, and certainly contributes to complexity
Fixes a bug wherein an scp and a mkdir command fight over a directory. The scp command now takes care of the directory creation through recursive copying, the mkdir command is removed.
Fixes a bug when recreating the K3D cluster on an existing EC2 instance with MetalLB and/or a secondary public IP
Increasing the metallb timeout from 120s to 300s (I sometimes observed long delays, but it was rare)
Fixed a bug in the printing of help messages that prevented correct instructions for -a being printed

Relevant logs/screenshots

./docs/assets/scripts/developer/k3d-dev.sh -t harbor -a -b
-t option passed to use additional tags on instance
-a option passed to create secondary public IP (-p and -m flags are skipped if set)
-b option passed for big k3d cluster using M5 instance
AWS User Name: Andrew.Kesterson
Checking for existing cluster for Andrew.Kesterson.
Will use large m5a.4xlarge spot instance

# ... SNIP ...
Allocating a new/another secondary elastic IP...
Associating Secondary IP 56.136.179.202 address to instance i-0d1f857d4e62bfc02...eipassoc-079e6fde10d309431
Secondary public IP is 56.136.179.202

# ... SNIP ...
Installing MetalLB...                                                                                                                                 
kustomization.yaml                                                                                                  100% 1149     9.6KB/s   00:00     
Building MetalLB configuration for -a mode.                                                                                                           
primaryProxy.yaml                                                                                                   100%   73     0.7KB/s   00:00     
secondaryProxy.yaml                                                                                                 100%   73     0.7KB/s   00:00     
metallb-config.yaml                                                                                                 100% 1131     8.5KB/s   00:00    
+ kubectl create namespace metallb-system
namespace/metallb-system created
+ kubectl create secret docker-registry registry1 --docker-server=registry1.dso.mil --docker-username=AndrewKesterson --docker-password=dVxPlVyCLl7pcxE7OVI9FE17dQziZk -n metallb-system
secret/registry1 created
+ kubectl apply -k /tmp/metallb
Warning: resource namespaces/metallb-system is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/metallb-system configured
customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
customresourcedefinition.apiextensions.k8s.io/servicel2statuses.metallb.io created
serviceaccount/controller created
serviceaccount/speaker created
role.rbac.authorization.k8s.io/controller created
role.rbac.authorization.k8s.io/pod-lister created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/controller created
rolebinding.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created 
configmap/metallb-excludel2 created
secret/metallb-webhook-cert created
service/metallb-webhook-service created
deployment.apps/controller created
daemonset.apps/speaker created
validatingwebhookconfiguration.admissionregistration.k8s.io/metallb-webhook-configuration created
+ echo 'Waiting for MetalLB controller...'
Waiting for MetalLB controller...
+ kubectl wait --for=condition=available --timeout 300s -n metallb-system deployment controller
deployment.apps/controller condition met
+ echo 'MetalLB is installed'                                              
MetalLB is installed
+ docker run -d --name=primaryProxy --network=k3d-network -p 172.31.1.89:443:443 -v /home/ubuntu/primaryProxy.yaml:/etc/confd/values.yaml ghcr.io/k3d-
io/k3d-proxy:5.7.3                                                                                                                                    4c497f72752af4605ed3d442611f7aacfd1fba74d4e014e0b9ab3fd9a32f0fc9                                                                                      + docker run -d --name=secondaryProxy --network=k3d-network -p 172.31.3.6:443:443 -v /home/ubuntu/secondaryProxy.yaml:/etc/confd/values.yaml ghcr.io/k
3d-io/k3d-proxy:5.7.3       
5acec4fbd90f3b847de0619b09f97d7520ce95193d6faf868955bf987fbc563c
+ kubectl create -f /home/ubuntu/metallb-config.yaml
ipaddresspool.metallb.io/primary created                                                                                                              
ipaddresspool.metallb.io/secondary created                                                                                                            
l2advertisement.metallb.io/primary created                                                                                                            
l2advertisement.metallb.io/secondary created

Checking the pods on the host indicate it's healthy

$ kubectl get pods -n metallb-system
NAME                          READY   STATUS    RESTARTS   AGE
controller-5c7486f596-pzhv6   1/1     Running   0          2m11s
speaker-99c5q                 1/1     Running   0          2m11s
speaker-j46p8                 1/1     Running   0          2m11s
speaker-l56tj                 1/1     Running   0          2m11s
speaker-trdcv                 1/1     Running   0          2m11s

Tested by:

configuring /etc/hosts on the k3d instance to route services through the metallb IP
Disabling all other entries for the services in question
Spinning up a SOCKS5 proxy to my k3d instance
Accessing grafana, prometheus, and sonarqube through the socks proxy
Observing grafana, prometheus and sonarqube loading properly
Deploying vault configured to use the public ingress gateway via the primary public IP
Configuring /etc/hosts on laptop to point vault.dev.bigbang.mil to the public IP of the instance
Accessing vault using the public IP via hosts dns
Observing vault loading properly

I wasn't able to get keycloak to work, but I think that's due to something in my keycloak configuration, rather than the metallb configuration. Metallb and the public ingress gateway appears to be working as designed.

I did not observe the public ingress gateway publishing on the secondary public IP. Not sure if that's a configuration issue or what.

Linked Issue

Closes #2492 (closed)

Upgrade Notices

N/A

Edited Feb 04, 2025 by Andrew Kesterson

Admin message