Resolve "dynamic inclusion of addons in ci"

Summary

• ci: change k3s as a sidecar to k3d using (tls enabled) dockerd as a sidecar using ci created bridge networks with properly tuned mtu
• ci: add logic to enable addons based on MR labels (name must be case exact match to the helm value block)
• infra: bump ci-optimized rke2 node pool to c5d.12xlarge, resize to { min: 1, desired: 1, max: 5 }
• infra: introduce cluster-autoscaler to ci-optimized rke2 node pool and actually test it, new nodes join in ~2m but to be safe, set ci timeout to 5m
• infra: use instance storage for image ci image layers (reduces i/o bottlenecks by ~80%)
• dogfood cluster: embed dind service into the dogfood gitlab ci deployment with fixes for /etc/machine-id, dind tls, and runtime dockerd config (see here)
• dogfood cluster: deploy gitlab ci with default dns (instead of the default ClusterFirst) dns to properly resolve dind-k3d dns

Closes #258 (closed)

.gitlab-ci.yml

@@ -101,17 +101,20 @@ pre vars:
 clean install:
   stage: smoke tests
   extends:
-    - .k3s-ci
+    - .k3d-ci
   variables:
     CLUSTER_NAME: "clean-${CI_COMMIT_SHORT_SHA}"
   rules:
+    - if: '$CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_BRANCH == "master"'
+      when: never
     - *chart_changes
   script:
     - *deploy_bigbang
     - *test_bigbang
 
     # Fetch list of all images ran
-    - curl -sfL k3s:8081/images | jq -r '.images[].repoTags[0]' > images.txt
+    - cid=$(docker ps -aqf "name=k3d-${CI_JOB_ID}-server-0")
+    - docker exec $cid crictl images -o json | jq -r '.images[].repoTags[0] | select(. != null)' | tee images.txt
   artifacts:
     paths:
       - images.txt
@@ -122,7 +125,7 @@ upgrade:
   dependencies:
     - pre vars
   extends:
-    - .k3s-ci
+    - .k3d-ci
   rules:
     # skip job when MR title starts with 'Breaking Change'
     - if: '$CI_MERGE_REQUEST_TITLE =~ /^Breaking Change/'