Resolve "CI logging improvement: show event log on pipeline failure"

Closes #895 (closed)

changed milestone to %1.20.0

added kindfeature teamci/pipelines labels

added 1 commit

• f1ab9f0f - Move existing trap to to scripts/traps.sh and then source it, to make editing the traps easier.

Compare with previous version

changed the description

added 1 commit

• 864bb424 - Replace exit builtin with function that dumps the k8s event log and then exits.

Compare with previous version

added 1 commit

• cf54d621 - Add exit statement to test

Compare with previous version

added 1 commit

• 79a488df - Cleanup

Compare with previous version

marked this merge request as ready

added 4 commits

• 79a488df...53bac2b1 - 2 commits from branch master
• b03c992d - Replace exit builtin with function that dumps the k8s event log and then exits.
• 85000fdf - Cleanup

Compare with previous version

added statusreview label

I don't think it really makes sense to do this event dump for all namespaces on every script. Some of my thoughts:

• airgap scripts aren't even used in CI
• package/gits.sh script wouldn't benefit from an event log and might even error (not sure we have a cluster in that stage)
• hosts script, I don't see much benefit from events but maybe could be convinced on this one?
• install_flux - this is a customer used script, would prefer to not add to this (or just add flux-system events)
• install_metallb - metallb-system ns events
• deploy_k3d - depending on when this errors you might not even have a cluster, probably shouldn't do any event dump here
• 00_deploy_flux.sh - just flux-system events
• 01_deploy_bigbang.sh - events on bigbang
• 02_wait_for_helmreleases - this is where we really get the benefit....but we should do this in a smarter way. Dump events for bigbang (since thats where the HR is created) + the namespace for the HR (example, we already have the below loop in the script but the comments I added are what I envision for getting package events):

            for hr in $failed_hrs; do
                kubectl describe hr -n bigbang $hr
                # Add something here to get the namespace (might be a different jsonpath
                # hr_namespace=$(kubectl get hr -n bibang $hr -o jsonpath='{.spec.targetNamespace}')
                # kubectl get events -n $hr_namespace
            done

• 01_virtualservices - I don't think we get any benefit from events, we already dump cluster state
• 02_non_ironbank - same, don't see any benefit
• 03_helm_tests - same, tests dump their logs if failed

Hopefully that makes sense. I just think we're going to be dumping a ton of irrelevant information into CI output if we take the approach of doing this universally.

added statusdoing label and removed statusreview label

resolved all threads

added 1 commit

• d364c0af - Add k8s event dumps on pipeline failure

Compare with previous version

added 26 commits

• d364c0af...e6318331 - 25 commits from branch master
• a20fe201 - Add k8s event dumps on pipeline failure

Compare with previous version

added 2 commits

• 05c9da67 - Sso integration improvements
• bfeb8325 - Run event dump at the end of every pipeline in a collapsed section

Compare with previous version

added 1 commit

• b0db4aaa - Colorize event log header

Compare with previous version

resolved all threads

resolved all threads

added 1 commit

• f3fc1e18 - Apply 1 suggestion(s) to 1 file(s)

Compare with previous version

approved this merge request

added statusreview label and removed statusdoing label

merged

mentioned in commit 53a7cb39