cmd/version: add ironbank sha checking
General MR
Summary
Enables image checking for all running containers against published release artifacts.
All releases for packages are published with an artifact called images.txt that lists the containers published by that release. This MR will:
- Figure out which releases are deployed
- Download the
images.txt
file for the given release - Reach out to the registry (ironbank usually) and download the published SHA for each of the images, building a map of the remote image
- Query the running pods in a release
- Compare their deployed SHAs against the upstream, expected SHAs
- Report the comparison
Coverage was reduced on a few things but I've unit-tested most, if not all, of the actual code paths we could see ourselves hitting. Wasting time on 100% coverage for this at this moment does not seem worth it.
Why is this MR so... crazy?
This MR was a true testament to poor planning. It should have been multiple, smaller MRs that tackled various components, since this required implementing a new client, modifying the existing gitlab client, and updating a looooot of version.go
code. Regardless, it's here now.
Relevant logs/screenshots
Checking for SHAs is enabled by default now. But there is a flag --no-shas
to disable the checks.
Checks will return one of three things:
-
All SHAs match
if all the SHAs running match -
Error: SHA for running container "%s" not found in published release artifacts
if the SHAs running in the cluster are not part of the publishedimages.txt
. See the grafana artifact for an example in the screenshot -
Error: SHA mismatch for image "%s". Local: "%s", upstream: "%s"
if the SHA running is published and does NOT match the expected one.
Errors are aggregated in a list and compiled into a multi-line string at the end of execution.
danieldides@BPH-MAC14 ~/P/b/bigbang (master)> bbctl version
bbctl:
version: 0.7.3
bigbang:
version: 2.34.0
danieldides@BPH-MAC14 ~/P/b/bigbang (master)> bbctl version grafana
grafana:
shasMatch: |
Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-operator:v0.75.0" not found in published release artifacts
Error: SHA for running container "registry1.dso.mil/ironbank/opensource/kubernetes/kube-state-metrics:v2.12.0" not found in published release artifacts
Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus/node-exporter:v1.8.1" not found in published release artifacts
Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus/prometheus:v2.53.0" not found in published release artifacts
Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus/alertmanager:v0.27.0" not found in published release artifacts
Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-config-reloader:v0.75.0" not found in published release artifacts
version: 8.4.6-bb.1
danieldides@BPH-MAC14 ~/P/b/bigbang (master)> bbctl version grafana --no-shas
grafana:
version: 8.4.6-bb.1
Linked Issue
closes #168 (closed)
Upgrade Notices
N/A