Update Flux in BB

Ya know. Cuz its old

Already ran into the 1st error on this. https://github.com/fluxcd/flux2/releases 1.18.0 release notes mention breaking change

If someone installs the latest version of the flux cli client 1.18.0, it doesn't like IronBank flux's API versions, or maybe it's the version of the APIs in the BB helm chart?/maybe those are what actually need to get updated:

flux reconcile kustomization environment -n=bigbang

✗ no matches for kind "Kustomization" in version "kustomize.toolkit.fluxcd.io/v1beta2"

k get kustomization environment -n=bigbang -o yaml | head | grep apiVersion:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1

https://github.com/fluxcd/flux2/discussions/1916
The above link explains changes about flux's breaking change, it looks like it it has a few nice updates, I'm particularly looking forward to the reliability improvements:

• Detect and report drift between the desired state (git, s3, etc) and cluster state reliably.
• Reconcile empty sources including pruning of all the resources previously applied.
• Mask secrets data in logs, events and alerts.

In the short term we can recommend folks use flux 0.17.2, instructions for pinning are here. https://repo1.dso.mil/platform-one/onboarding/big-bang/engineering-cohort/-/blob/master/lab_guides/01-Preflight-Access-Checks/A-software-check.md#flux

Should this ticket be repurposed to update the api references so the newer version of flux cli can be used? or is this just the IB image reference and what I'm proposing a separate ticket, I suspend since it's a breaking change it'd be the kind of thing where maybe both should be done at the same time.

mentioned in issue #856 (closed)

added teamcore/security label

added priority6 label

changed milestone to %1.19.0

assigned to @jasonkrause

changed iteration to Big Bang Iterations Oct 5, 2021 - Oct 18, 2021

added statusdoing label

assigned to @tawsif95

unassigned @jasonkrause

This ticket involves several risks. Just updating the flux version is not enough. The docker flux API needs to be updated. The IB APIs and other BB APIs need to be changed too from v1 to v2. This adds more complexity and risks to this issue. We need to have a spike/discovery sprint and plan out how to move forward with this ticket. Therefore, I am removing this issue from Milestone 1.19.0.

unassigned @tawsif95

removed milestone %1.19.0

removed iteration Big Bang Iterations Oct 5, 2021 - Oct 18, 2021

removed statusdoing label

assigned to @micah.nagel

Ya know. Cuz its old

https://fluxcd.io/blog/2021/09/server-side-reconciliation-is-coming/

So looking at the upstream docs here's what I'm seeing:

• Everything used in BB (umbrella helm chart) should work fine with the new version
• Customer template may be affected since kustomization has graduated to the new API version - although flux says it is backwards compatible
• Based on Chris' discovery looks like the flux CLI may not be backwards compatible...will investigate this one further

Still need to test all of this, but it's looking a bit less scary than initially thought.

https://github.com/fluxcd/flux2/issues/2009#issuecomment-951984118

The Flux CLI is at present not officially backwards compatible, and the minor version of the binary should target the same minor version range as running on the cluster. In some cases it may still work, but if there have been API version changes, it is kind of expected to break most of the time.

Impacts of this...

• BB Base image needs to be updated: https://repo1.dso.mil/dsop/big-bang/base/-/blob/development/hardening_manifest.yaml#L40
• BB CI should be updated (although its already past 0.18.0): https://repo1.dso.mil/platform-one/big-bang/pipeline-templates/pipeline-templates/-/blob/master/dockerfiles/bb-ci/Dockerfile#L64
• Customers alerted about the new change so that they can update their flux CLI if they aren't using the packaged one + update their Kustomization resources
• Release engineers/Devs will need the new flux cli

We could dodge that by staying on the v1beta1 api version initially...but then customers/devs have to stay on the older flux cli.

Also good to note that people can just avoid updating flux if they want more time for the transition. All our install guides point to installing flux with the script and I'm not aware of other methods floating around...so as long as they don't run the install script to upgrade they should be safe.

Flux says that this is all backwards compatible and the specs have been deprecated, not removed.

Here's the proposal:

• upgrade to flux 0.24.0 per your MR https://repo1.dso.mil/platform-one/big-bang/bigbang/-/merge_requests/1233

• Work on integrating the new flux cli version into BB/Base in IronBank.

• If the CLI version is not in before 0.26, then we test with v1beta1 version of Kustomization.

• If the CLI version does get in before 0.26 then we test with v1beta2 version of Kustomization without changing the actual spec. I suspect we don't actually use any of those Kustomization features, but they claim to be backward compatible.

• Release Notes should include the following regardless

  • Link to https://github.com/fluxcd/flux2/discussions/1916
  • Copy out the table from that thread on HOW to upgrade off the deprecated Kustomization

dsop/big-bang/base!32 (merged)

New IB MR.

mentioned in merge request !1233 (merged)

added statusdoing label

changed milestone to %1.26.0

changed iteration to Big Bang Iterations Jan 11, 2022 - Jan 24, 2022

set weight to 2

added statusreview label and removed statusdoing label

closed with merge request !1233 (merged)

mentioned in commit 0c804d8e

Flux is now updated in BB master. IB MR is chugging along with the pipeline, I am making sure everything is updated and passes the checks.

Remaining steps if IB image gets out before next Monday:

• Update the BB base image in synker file to 1.0.0 (maybe just add 1.0.0 so that 8.4 is still available since it is needed for upgrade jobs and such?)
• Update customer template to v1beta2 for Kustomization

Release regardless as included in the comment from Tom above.

https://repo1.dso.mil/platform-one/big-bang/bigbang/-/merge_requests/1255

for the first checkbox

mentioned in merge request !1255 (merged)

removed statusreview label