Run Helm tests inside the BigBang level

Currently, helm tests are only run at the package level. Need to add helm test execution at the bigbang ci in both the nightly and MR pipelines

added to epic &116 (closed)

changed the description

added Big Bang Continuous Process Improvement kindci labels

Draft MR already out there that might help as a starting point. - https://repo1.dso.mil/platform-one/big-bang/bigbang/-/merge_requests/595

There may be a couple pieces to this:

• Solution for helm test pods to be able to resolve *.bigbang.dev to the proper host
  • Newest gluon adds a way to do this via bbtests.istio values and hostAliases, but would need updating to handle new ingress gateway stuff
  • Perhaps a good option would be to do this via "adding entries" to CoreDNS for each VS?
• Adding values to BB k3d values to make sure tests are enabled and deployed with proper testing values
• Adding script to BB to run all helm tests and grab cypress artifacts (this is the piece that should be mostly covered by the above MR)

changed milestone to %1.17.0

changed iteration to Big Bang Iterations Sep 7, 2021 - Sep 20, 2021

set weight to 3

assigned to @micah.nagel

added statusdoing label

Going to start by tackling the DNS portion, attempting to do it via CoreDNS entries.

Turning the above into a list for me:

• Add CoreDNS entries for each VS
• Add values to k3d values to enable/configure
• Run all tests via the script and get artifacts (logs + cypress)
• Allow upgrade tests to fail? All tests to fail? Could use the allowed exit codes here

On the 4th bullet, it'll be necessary to at least allow upgrade testing to fail. Uncertain on clean install testing since ideally all package tests are robust...

CoreDNS portion accomplished (tentatively) - https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/helm-tests/tests/tests/03_helm_tests.sh#L6-34

Test running accomplished (tentatively) - https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/helm-tests/tests/tests/03_helm_tests.sh#L36-81

Values adds in progress, couple things I'll probably have to disable or tweak - https://repo1.dso.mil/platform-one/big-bang/bigbang/-/blob/helm-tests/tests/ci/k3d/values.yaml

Testing with scripts is being weird...the configmaps always turn up empty. Not sure why this is happening since they work in the packages and the same glob technique used for them is working on he cypress configmaps.

Trying to figure out if there is a helmignore or something similar that is causing issues...

Found the culprit for script testing. The flux gitignore has .sh. Attempting to include (exclude?) chart/tests/.sh from the gitignore.

Based on the current state of package tests I don't feel comfortable making them fail as is. Will "allow_failure" with certain exit codes so that tests dont break people's pipelines when they shouldn't. The orange warning should be enough for a maintainer to do a quick scan of the logs and determine what tests failed and whether it was a testing infra issue or an issue with whatever package changed.

mentioned in merge request !863 (merged)

mentioned in merge request !595 (merged)

added priority6 label

added statusreview label and removed statusdoing label

added teamci/pipelines label

closed with merge request !595 (merged)

mentioned in commit a8a66f3e

removed statusreview label