[P1BIGROCKS-2219](https://jira.il2.dso.mil/browse/P1BIGROCKS-2219) The current BigBang Pipelines need cleaning up. - [x] Dogfood cluster improvements &117 - [x] Break this into what can be done now, vs what should be done after the CI process improvements have been implemented - [x] CI Process improvements for MRs - Spike - [x] CI Process improvements for MRs - Implementation - [x] Document CI process - [x] On each MR - [x] On each commit to master - [x] Reproducing locally to help developers - [x] CI Cleanup - [x] remove unused files in the CI job - [x] assess duplicate code blocks/etc. Current issues with CI - [x] Cleanup of dead code in .gitlab-ci.yml and `.gitlab-ci` folder. Provided consistency on where things are located and whether they're in this repo or in the library - [x] Wait for downstream objects in operator HelmCharts (not just CRD being created) like Mattermost (wait for healthy deployment) ( https://kuttl.dev/ maybe eventually? ) - [x] Run Helm tests inside the BigBang level (currently package specific currently) - [x] diagram/workflow of all steps in CI jobs (all of them) with timing information about how long each step takes - [x] When CI breaks, we don't get enough log to debug WHY. Logs for pods that fail, HelmRelease object status, etc could/should be included - [x] Reduce PVC sizes for pipeline runs (k3d) - [x] RKE2 cluster fails to come up on random nightly jobs. Implement retry logic, or get someone who's from rancher to fix the terraform - [x] As we use "set -e" on most of our bash scripts which causes them to exit if any command within them fails, it is often not clear where/why the script is exiting.