CI Runs Not Catching Test Errors
Bug
Description
We've come across two cases where we see the helm tests pass even though there were failures in the tests.
These can probably be traced to the logic in 03_helm_tests.sh.
- The 1st case is where keycloak cypress tests fails when running under umbrella big bang. There might be some sso/tests issues here in play causing the error. However, we should detect the failed cypress test. See attached keycloak-cypress-test.txt .
...
++ sed -n '/Error:/,$p'
+ log_errors=
+ [[ '' != \N\o\n\e ]]
+ [[ -n '' ]]
+ echo '✅ All tests successful for keycloak'
✅ All tests successful for keycloak
...
Finished running all helm tests.
+ '[' 0 -gt 0 ']'
+ echo '✅ All helm tests ran successfully.'
✅ All helm tests ran successfully.- The 2nd case is with gitlab.
gitlab-script-testfails, but the helm tests pass. See gitlab-log.txt
Finished running all helm tests.
✅ All helm tests ran successfully.gitlab pod/gitlab-script-test 0/2 Error 0 7m55s