Resolve "Add retry for RKE2 cluster creation/destruction"

Summary

Testing adding retry logic to our nightly infra CI pipelines. So if terraform or the runners have an issue, the stage can attempt to retry before marking everything as a failure.

Part of #713 (closed)

Terraform is NOT happy being re-ran if it encounters an issue creating or verifying a resource the first time: yikes, and will require some better terraform logic from someone smarter than I.

This MR includes some per-stage retry logic for all stages of a master nightly pipeline run so that if the runner has an issue, times-out, or fails to schedule it will retry. So still an attempt to make things more resilient!

changed milestone to %1.17.0

added Big Bang Continuous Process Improvement kindci statusdoing test-ciinfra labels

added 1 commit

• 2ee79989 - Gitlab doesn't like 3 maxes

Compare with previous version

added 1 commit

• e783d448 - Testing Nexus with 5G PVC

Compare with previous version

added 1 commit

• 24f0e17a - Removing retry for script_failure on some CI jobs

Compare with previous version

added 1 commit

• e319e843 - Revertubg CI Values Nexus PVC size

Compare with previous version

added 1 commit

• d796d26f - Ensuring all retry values are equal settings

Compare with previous version

changed the description

added statusreview label and removed statusdoing label

approved this merge request

@ryan.j.garcia would it be wise to open a new issue to tackle the stuff you mentioned (getting an actual retry of the TF to work)? Otherwise LGTM.

changed the description

Updated description to just to mention linked issue not mark is as Closes since we still would like to add more terraform settings or ability to re-run entire pipeline if certain stage fails (which doesn't seem possible from my research).

merged

mentioned in commit effb63d7