Big Bang Nightly CI Failure (2/22/23)
https://repo1.dso.mil/big-bang/bigbang/-/jobs/19043134
Failure while waiting on pod/deployment health:
error: timed out waiting for the condition on deployments/neuvector-prometheus-exporter-poActivity
assigned to @micah.nagel
Events do not provide any additional insight into the failure. I think we've run into similar failures in some previous runs:
added statusdoing label
added teambigbang label
-
https://repo1.dso.mil/big-bang/bigbang/-/jobs/19084141
Rob ran a debug pipeline to try and reproduce the error. The logs are not super helpful although it seems like the consul agent is resulting in a crashloop on the controllers, which may explain why the exporter can't connect.
mentioned in issue dsop/neuvector/neuvector/controller#21 (closed)
mentioned in merge request !2541 (merged)
-
Temp solve to prevent nightly issues, thanks @rob.ferguson
assigned to @rob.ferguson
-
https://repo1.dso.mil/big-bang/bigbang/-/jobs/19279241
This job has logs from the failure. Discovered in a bit of testing that disabling networkpolicies oddly seems to fix the issue in rke2?
Addressing the missing egress rules to allow controller component access to the API server: big-bang/product/packages/neuvector!23 (merged)
mentioned in merge request !2548 (merged)
-
Kicking off an RKE2 pipeline or 2 here - !2545 (merged)
-
k3d pipeline with 3 replicas: https://repo1.dso.mil/big-bang/bigbang/-/pipelines/1412695
rke2 pipeline with 3 replicas: https://repo1.dso.mil/big-bang/bigbang/-/pipelines/1412696
mentioned in merge request !2545 (merged)
closed with merge request !2545 (merged)
mentioned in commit 4f62da08