Improve RKE2 Pipeline Cleanup
Currently we have cleanup stages that are:
- bb down: helm uninstall bb - https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/-/blob/master/pipelines/bigbang.yaml#L413
- cluster/network down: terraform destroy - https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/-/blob/master/clusters/rke2/templates.yaml#L32 , https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/-/blob/master/infrastructure/aws/templates.yaml#L30
The problem is that the terraform destroy doesn't always clean up all resources. This leaves a lot of orphans in our AWS account - which is a problem if running multiple pipelines in a row. We should find a better way to handle cleanup so that there are no orphan resources left. Also should ensure that this cleanup works well even if a stage of the pipeline (like bb up) is cancelled.
Edited by Micah Nagel