wait jobs causing kyverno policy issues

changed milestone to %2.41.0

added kindbug priority3 statusready-to-work teamStorage & Collaboration labels

set weight to 1

changed iteration to Big Bang Iterations Nov 12, 2024 - Nov 25, 2024

changed title from Disable wait jobs to fix kyverno policy issues to wait jobs causing kyverno policy issues

changed the description

assigned to @samvongsay

added statusdoing label and removed statusready-to-work label

created branch 139-wait-jobs-causing-kyverno-policy-issues to address this issue

mentioned in merge request !221 (merged)

Suggested solution will not work. Reference to error explaination Summary: if statements will undermine the top level requirement define needs.

Seeking other options to address the issue.

The issue here is in the way the gluon wait job template is toggled on/off and executed, and how kiali processes that output. In order to use the gluon wait job, you have to implement a template that conditionally creates some resources. If you disable the wait job, that results in helm template producing a set of empty YAML files. Helm doesn't care about this, neither does flux, they will both deploy the product just fine. However, kiali processes the helm template output as a stream of yaml documents and doesn't know how to handle an empty yaml document. It expects there to be content.

There are two other options I can think of to try to resolve this:

Place a shim in the pipeline between the helm template command, and the kiali command, that looks for and removes empty YAML documents from the stream.
Add some kind of dummy resource to the YAML files which contain the conditionally-generated YAML, so that even if the wait job is disabled, the dummy resource is still there so kiali would not complain. Something like an empty configmap would be sufficient, any valid kubernetes resource would work.

The second option is the simplest but also looks stupid and results in a wasted resource. The first option requires making changes to the pipeline-templates and affects every project, not just ours.

For simplicity's sake, I would try option 2 first. Just add some dummy resource at the top of the file that will always be generated, regardless of whether or not the wait job is enabled, and see if that stops the issue.

removed iteration Big Bang Iterations Nov 12, 2024 - Nov 25, 2024

changed iteration to Big Bang Iterations Nov 26, 2024 - Dec 9, 2024

changed milestone to %2.42.0

changed weight to 4 from 1

removed iteration Big Bang Iterations Nov 26, 2024 - Dec 9, 2024

changed iteration to Big Bang Iterations Dec 10, 2024 - Dec 23, 2024

changed milestone to %2.43.0

removed iteration Big Bang Iterations Dec 10, 2024 - Dec 23, 2024

changed iteration to Big Bang Iterations Dec 24, 2024 - Jan 6, 2025

changed milestone to %2.44.0

removed iteration Big Bang Iterations Dec 24, 2024 - Jan 6, 2025

changed iteration to Big Bang Iterations Jan 7, 2025 - Jan 20, 2025

This issue isn't specific to Kiali, it also appears to be happening on Mattermost if you try and disable the waitJob for Minio:

https://repo1.dso.mil/big-bang/product/packages/mattermost/-/jobs/41452468#L311

It appears the waitJob was also disable on Loki about a month ago which points to a bigger underlying issue. I'm not sure if its due to the way gluon is implementing it or not, but disabling bbtests does not give the same issue; Those required resources are only created when bbtests.enabled is set to true and leaving it off does not generate empty yaml.

Another issue here I think comes down to what exactly the minio wait script is doing. The reason I disabled it in Mattermost was because its hanging in my local environment and causing the pipeline to fail as that hangs as well. Looking at the wait script for minio it appears to be hard coded to look for a tenant in the minio namespace which maybe doesn't make sense for packages that use this as a resource. The minio tenant for mattermost is created in the mattermost namespace and the waitJob hangs because it doesn't have permissions outside of the mattermost namespace.

I'm not sure if that's the same reason it was disabled in Loki, but it appears to have been disabled there due to hanging as well:

loki!286 (dd2ae69e)

This may actually be two issues, one to fix gluon, and the other to make the wait script check for the tenant in the namespace in which its being used.

changed milestone to %2.45.0

I stand corrected, setting bbtests.enabled to False results in the same issue.

It looks like the only reason its never been seen before is because that section is only present in test-values.yaml as its intended to be used as part of the CI/CD process. I would think moving the waitJob section from values.yaml into test-values.yaml on the minio package would resolve the issue. Since that's also technically part of the CI/CD process it probably belongs there in the first place.

If another package that's using minio needs to run the wait script for minio that part can simply be added to that package's test-values.yaml under the minio section.