[P1BIGROCKS-2618] Add verbose support to pipeline runs
[P1BIGROCKS-2618](https://jira.il2.dso.mil/browse/P1BIGROCKS-2618)
As a developer, I'd like to be able to turn up the verboseness of logs for pipelines that are failing. This allows me to troubleshoot failures without having to exec into the GitLab runner pod and run additional commands. We could trigger this using a "Verbose" label on the MR. Here are the items I'd like to see in the additional logs:
> Because of the large number of logs coming out of this, each log should be contained and collapsed so the developer can easily expand what they want to look into.
- Dump of the current and previous logs for pods
- kubectl describe on resources (this will spit out events as well as the yaml used); Pods and nodes make sense, maybe endpoints too? What other items should we dump?
- Snapshot of memory/cpu use for pods
- Tracing turned on for any shell scripts run, including in gluon for script tests (-x option)
- kubectl cluster-info dump
- Control plane logs (if we have access): https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/#looking-at-logs
The goal of this epic is to produce enough information for developers on pipeline troubleshooting that there is no longer a need to log into the GitLab Runner pods to debug. This would allow us to lock down Big Bang's cluster for better security.
epic