UNCLASSIFIED - NO CUI

Skip to content

cmd/version: add ironbank sha checking

Daniel Dides requested to merge dd/168-ib-check into main

General MR

Summary

Enables image checking for all running containers against published release artifacts.

All releases for packages are published with an artifact called images.txt that lists the containers published by that release. This MR will:

  1. Figure out which releases are deployed
  2. Download the images.txt file for the given release
  3. Reach out to the registry (ironbank usually) and download the published SHA for each of the images, building a map of the remote image
  4. Query the running pods in a release
  5. Compare their deployed SHAs against the upstream, expected SHAs
  6. Report the comparison

Coverage was reduced on a few things but I've unit-tested most, if not all, of the actual code paths we could see ourselves hitting. Wasting time on 100% coverage for this at this moment does not seem worth it.

Why is this MR so... crazy?

This MR was a true testament to poor planning. It should have been multiple, smaller MRs that tackled various components, since this required implementing a new client, modifying the existing gitlab client, and updating a looooot of version.go code. Regardless, it's here now.

Relevant logs/screenshots

Checking for SHAs is enabled by default now. But there is a flag --no-shas to disable the checks.

Checks will return one of three things:

  • All SHAs match if all the SHAs running match
  • Error: SHA for running container "%s" not found in published release artifacts if the SHAs running in the cluster are not part of the published images.txt. See the grafana artifact for an example in the screenshot
  • Error: SHA mismatch for image "%s". Local: "%s", upstream: "%s" if the SHA running is published and does NOT match the expected one.

Errors are aggregated in a list and compiled into a multi-line string at the end of execution.

danieldides@BPH-MAC14 ~/P/b/bigbang (master)> bbctl version 
bbctl:
  version: 0.7.3
bigbang:
  version: 2.34.0
danieldides@BPH-MAC14 ~/P/b/bigbang (master)> bbctl version grafana
grafana:
  shasMatch: |
    Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-operator:v0.75.0" not found in published release artifacts
    Error: SHA for running container "registry1.dso.mil/ironbank/opensource/kubernetes/kube-state-metrics:v2.12.0" not found in published release artifacts
    Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus/node-exporter:v1.8.1" not found in published release artifacts
    Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus/prometheus:v2.53.0" not found in published release artifacts
    Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus/alertmanager:v0.27.0" not found in published release artifacts
    Error: SHA for running container "registry1.dso.mil/ironbank/opensource/prometheus-operator/prometheus-config-reloader:v0.75.0" not found in published release artifacts
  version: 8.4.6-bb.1
danieldides@BPH-MAC14 ~/P/b/bigbang (master)> bbctl version grafana --no-shas
grafana:
  version: 8.4.6-bb.1

Linked Issue

closes #168 (closed)

Upgrade Notices

N/A

Edited by Daniel Dides

Merge request reports