UNCLASSIFIED - NO CUI

Skip to content

BigBang deployment via customers/template -style kustomization deploys HEAD rather than specified BB tag references

Bug

Description

FluxCD winds-up deploying the HEAD on the Bigbang HelmRelease as well as HEAD from all sub-project HelmReleases.

Repro Context

Using the FluxCD currently described in the customers/template ( https://repo1.dso.mil/platform-one/big-bang/customers/template ), it appears that when BigBang is deployed that the git repo reference provided is disregarded. Instead BigBang and each of the BigBang sub-projects get deployed based upon the HEAD of BigBang and whatever is referenced. This winds-up deploying unreleased BigBang and subprojects instead of the version specified in the base/kustomization.yaml file.

Often this not only doesn't deploy a reproducible/expected BigBang version, but it also has been breaking the BigBang installation due to new gatekeeper policies, expected chart values (e.g. istio.ingress.key, istio.ingress.cert, etc. ).

BigBang Version

1.12.0 (but probably earlier as well)

Notes

  • This behavior seems somewhat new, perhaps starting around July 24, 2021.
  • The observed behavior was application containers that could not be deployed, eventually tracked-back to BigBang modules that were inconsistent with the specified version in the base/kustomization.yaml files. Specifically profiles enforced by gatekeeper that were still in development and not part of any yet tagged release of BigBang. Later it was observed that each new commit to submodules (e.g. "policy" or gatekeeper) was introducing newly broken applications - again because FluxCD was constantly updating to latest instead of sticking with the specified tags.
  • The behavior affected at least several teams that were actively reporting this on the Mattermost channel: https://chat.il2.dso.mil/platform-one/channels/team---big-bang around July 26/27, 2021
  • Some peopled seemed to find workarounds
  • The behavior does not appear to affect helm releases
  • Some odd behavior was observed on July 26/27, 2021 with Gitlab not resolving the Kustomization-style gitrepository references (e.g. //base?ref=XXX ) where the version tag is normally specified. This might be the closest observation to a cause: if the //base?ref=XXX style logic is not resolving properly then it might be drifting to the last commit or HEAD on the master branch. There have been some recurring reports of various Gitlab versions suffering from a problem wherein Gitlab will not resolve tags with the //base?ref=XXX syntax used by the gitrepository definitions that are provided in the BigBang customers/template/base/kustomization.yaml file.

Attempts to Resolve

I have tried:

UNCLASSIFIED - NO CUI