Investigate and update CI wait timeouts
When running a BB MR pipeline with many or all addons installed, the pipeline will occassionally time out on the kubectl wait
commands (typically the HR wait).
This tends to happen with gitlab/gitlab runner (sometimes other packages) and only when run with the majority of other addons enabled. This will be a challenging issue to debug as it can be hard to replicate and understand what is really going on.
One suggested way to approach the challenge is to spin up an ec2 instance, enable all packages (except keycloak) with the k3d values and run the deploy scripts in order the same way the pipeline does. This will likely not be a perfect test as the CI setup is slightly different (k3d run inside of a pod, in an RKE2 cluster, on an EC2 instance) but it should help to get a baseline of the time it takes for all package HRs to go to a healthy ready state.
AC:
-
Script updated to better handle waiting on packages (likely increased timeout), verifiable with multiple runs of the MR pipeline with all-packages enabled