UNCLASSIFIED - NO CUI

Terminating pod logic is error prone

https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/-/blob/master/library/templates.sh#L275-278

https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/-/blob/master/library/templates.sh#L330-333

Both of these are error prone. Essentially the issue happens when the initial check finds some pods in a terminating state but they terminate before it hits the kubectl wait.

We need to better account for this situation.

Example failing pipeline: https://repo1.dso.mil/platform-one/big-bang/apps/core/cluster-auditor/-/jobs/8081354

Edited by Micah Nagel