Zarf Helm Install fails due to image version mismatch

I can fix this manually, however it appears like there is a deeper automation issue. It may also make sense to add tests to our packages that validates that the helm.sh/images annotation reference in Chart.yaml matches what is in the chart, or if there is a way to, add a helper in our packages that directly references the annotation to keep this DRY and avoid version mismatches like this. This will become more important as more people leverage Zarf more and more.

100% agreed that we need automation to catch this. Thanks for calling this one out - we may want/need to do a patch version update to 1.57.1 to fix this in the release.

If this is an issue with automation, there could be other places where this is happening.

It definitely has happened in a case or two already - the Zarf team updated a few spots when they were testing the new Big Bang support. We already grab images run in cluster (see artifact images.txt) so it shouldn't be too difficult to add a validation of those images against what is specified in the Chart.yaml. We should be able to at least confirm that all images in Chart.yaml exist in that list.

I get a 404 on that link, probably an access issue.

I do as well now...the artifacts might've expired - here's a new one https://repo1.dso.mil/big-bang/product/packages/argocd/-/jobs/20466268/artifacts/file/images.txt

Thanks.

I think part of the issue is that the Zarf BB baseline tests probably don't cover addons (like ArgoCD) and out-of-tree helm charts. Has it been discussed referencing the images from Chart.yaml directly, rather than repeating them and checking to see if they match, or relying on Renovate to update it in multiple places?

Obviously the other thing would be a test that checks that everything matches.

Has it been discussed referencing the images from Chart.yaml directly, rather than repeating them and checking to see if they match

Not sure I totally follow this question? We end up having to duplicate them since Chart.yaml annotations can't be used in the templates. So image versions/tags need to be in chart/values.yaml. The benefit of also having them in chart/Chart.yaml is to have a standardized spot with all images listed.

We do ideally have Renovate update both spots - it looks like in this case the renovate config was out of date so we didn't catch it.

The current way that images.txt file is constructed is by checking all images that were pulled in cluster (we use a crictl command to grab those). My suggestion would be that we compare that list against what is in the Chart.yaml. That would give us:

images.txt: The actual list of images pulled when the chart was installed
Chart.yaml: The list of images we claim are needed to install the chart

In most cases those lists should be 1:1 if we accurately made them. There are probably a few edge cases today where images.txt has an extra image based on conditional deployment, but validating that everything in Chart.yaml is in images.txt would be a good first step to preventing most errors with Zarf or other places that the chart annotations are referenced.

Ok, my thought was referencing the chart annotations. I assumed it was something that could be done, but I was wrong (https://github.com/helm/helm/issues/10653).

Ah tracking...yeah that could be doable technically, to some extent even if Helm didn't support it, but that also mean Zarf would need to do the templating of the Chart.yaml as well. Probably not the best path right now, although I love the idea.

Maybe I'll submit an MR to make helm do this!

mentioned in merge request !128 (merged)

This issue is occurring on an open MR by the bot to bump to 2.6.7 as well. !127 (merged)

This looks like an issue with our Renovate config in this case - Renovate bumps up the image versions and should be catching all the spots where that is needed...

(although we still need extra pipeline checks beyond this, having Renovate properly update should make this happen far less often)