Patch Release 3.3.1

0. Release Prep

📌 NOTE: _If you need the manual steps and/or R2D2 steps, They can be found _Here

Set your AWS profile to bb_prod export AWS_PROFILE=bb-prod
Verify the Release cluster is up and running normally
./gogru release-cluster-check
- If there are any issues with the Helm Releases or Pods in the output, investigate and fix them before continuing

1. Release Prep

If using R2-D2 automated release tool, make sure release_type in ~/.r2d2/config.yaml is set to patch
Check last release SHAs
- Verify that the previous release branch commit hash matches the last release tag hash. Investigate with previous release engineer if they do not match.
- R2-D2: Check last release SHAs
Manual Steps
- Go to Branches in Big Bang (product). - Find the latest release branch. It should be in the format release-..x (e.g. release-2.7.x). You will see an eight-character commit hash below it (e.g. 6c746edd) - Go to Tags in a separate tab. Here you will see the latest release tag listed first. Verify that this tag has the same hash you found in the previous step.

Check out the most recent minor release branch, all patch changes will be applied to this branch.

# CLI example, replace `<release-branch>` with the name of the most recent minor release branch
$ cd [path to local bigbang]
$ git checkout <release-branch>
$ git pull

Cherrypick the required patch commit(s) into the corresponding release branch eg: release-2.18.x for 2.18.1+ patch release.
- Reopen expired minor milestone: Anchor or Joe reopen expired milestone and assign the cherry picked MRs the minor milestone.
- Close the expired minor milestone once assigned to related cherry picked BB MRs, package MRs and related issues

Upgrade Big Bang (product) version references

R2-D2: Upgrade Version references

Manual Steps

💡 Tip Make the following changes in a single commit so that they can be cherry picked into master later. All changes should be made in the Big Bang (product) repository

In base/gitrepository.yaml, update ref.tag to your current release version.
Update chart release version chart/Chart.yaml

Add a new changelog entry for the release, ex:

## [2.<minor>.<patch>]

- [!2.<minor>.<patch>](https://repo1.dso.mil/big-bang/bigbang/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&milestone_title=2.<minor>.0); List of merge requests in this release.

<!-- Note: Until R2D2 has been modified to generate release notes using multiple milestones (minor and patch)...
You need to use the .0 patch version for the milestone.. We do not currently use patch milestones. milestone_title=2.<minor>.0 -->

Update /docs/understanding-bigbang/configuration/base-config.md using helm-docs.

# example release 2.<minor>.x
$ cd bigbang
$ git checkout release-2.<minor>.x
$ docker run -v "$(pwd):/helm-docs" -u $(id -u) jnorwood/helm-docs:v1.5.0 -s file -t .gitlab/base_config.md.gotmpl --dry-run > ./docs/understanding-bigbang/configuration/base-config.md

Commit & push changes

$ git commit -am 'version updates for release <release-tag>'
$ git push

Build the amended release notes.
- Return to dogfood-bigbang/docs/release as your current working directory
- Make a copy of the most recent minor release notes
- Increment the patch version in the copied document name and title
- Add an additional block before ## Upgrade Notices with patch changes
```
## Patch-Specific Changes

### Package Name
- [!$MR_NUMBER]($LINK_TO_MR): $MR_TITLE or $SHORT_DESCRIPTION
```
- Update Packages table with new package versions
- Add package changelog entry under package section
- Commit the resulting release notes and push to master in Big Bang (dogfood)
Reach out to ¹ to review your commits to the release branch prior to moving on to the next step

2. Upgrade and Debug Cluster

Verify Flux has reconciled cherrypicked updates
- Packages have fetched new git repository revisions and match the new release
```
$ kubectl get gitrepositories -A
```
- Packages have reconciled
```
$ watch kubectl get gitrepositories,kustomizations,hr,po -A
```
- All pods are in Running or Completed status
```
$ kubectl get pods -A
```

If Flux doesn't reconcile changes after 10 minutes, follow the troubleshooting steps in the minor release process.

3. UI Tests

bbctl

Login to grafana as admin. User name "admin". Retrieve password with cd into/my/release/folder/ and then sops -d deployments/release/release/secrets/environment-bb-secret.enc.yaml | sed 's/\\n/\'$\'\n/g' | grep grafana -A 2
Click on the drop-down on upper-left, choose Dashboards, then search for bbctl. Ensure that all 6 dashboards are present in the search results.
Click into the bbctl-all-logs dashboard and confirm tables tables are populated.
‼️ TROUBLESHOOTING: If any of the tables are missing data, you can run the corresponding command to trigger the job to run immediately instead of waiting for the next CRON schedule trigger.
If there is an error in the bbctl-all-logs table for a command or data still isn't populating after manually running the commands, reach out to the bbctl team and create a bug ticket.
Manually Trigger bbctl CRON Jobs
- Preflight Check Logs
  kubectl create job --from=cronjob/bbctl-bbctl-bigbang-preflight bbctl-bigbang-preflight -n bbctl
- Versions Logs
  kubectl create job --from=cronjob/bbctl-bbctl-bigbang-updater bbctl-bigbang-updater -n bbctl
- Policy Logs
  kubectl create job --from=cronjob/bbctl-bbctl-bigbang-policy bbctl-bigbang-policy -n bbctl
- Status Logs
  kubectl create job --from=cronjob/bbctl-bbctl-bigbang-status bbctl-bigbang-status -n bbctl
- Violations Logs
  kubectl create job --from=cronjob/bbctl-bbctl-bigbang-violations bbctl-bigbang-violations -n bbctl
Click into the other 5 bbctl dashboards and ensure they all show data in the graphs.
- bbctl-preflight-dashboard
- bbctl-version-dashboard
- bbctl-policies-dashboard
- bbctl-status-dashboard
- bbctl-violations-dashboard

GitLab, Gitlab Runner, & Sonarqube

Gitlab

Login to gitlab with SSO
Edit profile and change user avatar. (Avatar may take several minutes to update.)
Go to Security > Access Token and create a personal access token. Select all scopes.
- Record the token and the token name to provide during automated test

Gitlab Runner

R2-D2: Gitlab Test

📌 NOTE: When prompted for the username use the token name and not the name of the user that created the token.

‼️ TROUBLESHOOTING: The pipeline may fail if Sonarqube is getting a major version update. Navigate to 'http://sonarqube.release.bigbang.mil/setup' and upgrade it before re-running the test

Sonarqube

Login to sonarqube with SSO

‼️ TROUBLESHOOTING: You will be unable to login if Sonarqube is getting a major version update. Navigate to 'http://sonarqube.release.bigbang.mil/setup' and upgrade it before proceeding
Verify your Project says PASSED

4. Create Release

Finalize the tag in chart/Chart.yaml (remove -rc.x if present), commit and push this change
Create release candidate tag based on release branch, ie. 2.<minor>.<patch>-rc.0. Tagging will additionally create release artifacts and the pipeline runs north of 1 hour. You will need to request tag permissions from the ¹.
- To do this via the UI (generally preferred): tags page -> new tag, name: 2.<minor>.<patch>-rc.0, create from: release-2.<minor>.x, message: "release candidate", release notes: leave empty
- To do this via git CLI:
```
$ git tag -a 2.<minor>.<patch>-rc.0 -m "release candidate"
# list the tags to make sure you made the correct one
$ git tag -n
# push
$ git push --tags
```
Confirm pipeline passed for Release Candidate tag.
Confirm pipeline passed in bb-docs-compiler for latest RC tag (Gets created/scheduled towards the end of the bigbang release pipeline).

Review all pipeline output looking for failures or warnings. Reach out to the ¹ for a quick review before proceeding.
Create release tag based on release branch. ie. 2.<minor>.<patch>.
- To do this via the UI (generally preferred): tags page -> new tag, name: 2.<minor>.<patch>, create from: release-1.<minor>.x, message: release 2.<minor>.<patch>, release notes: leave empty
- To do this via git CLI:
```
$ git tag -a 2.<minor>.<patch> -m "release 2.<minor>.<patch>"
# list the tags to make sure you made the correct one
$ git tag -n
# push
$ git push --tags
```
Ensure Release Notes are accurate
- Amend Upgrade Notices, based off of the listed notices in the release issue. Also review with ¹ to see if any other major changes should be noted.
- Adjust known issues as needed: If an issue has been resolved it can be removed and if any new issues were discovered they should be added.
- Verify table contains all chart upgrades mentioned in the MR list for both the minor and patch release.
- Verify all internal comments are removed from Release Notes, such as comments directing the release engineer to copy/move things around
Passed release pipeline.

Review all pipeline output looking for failures or warnings. Reach out to the ¹ for a quick review before proceeding. One of the last stages of the pipeline will auto grab the release notes generated above from deployments/bigbang in master. Ensure they populated and are formatted correctly.

Cherry-pick release commit(s) as needed with merge request back to master branch. We do not ever merge release branch back to master. Instead, make a separate branch from master and cherry-pick the release commit(s) into it. Use the resulting MR to close the release issue.

# Navigate to your local clone of the BB repo
cd path/to/my/clone/of/bigbang
# Get the most up to date master
git checkout master
git pull
# Check out new branch for cherrypicking
git checkout -b 2.<minor>-cherrypick

# Repeat the following for whichever commits are required from release
# Typically this is the initial release commit (bumped GitRepo, Chart.yaml, CHANGEGLOG, README) and a final commit that re-ran helm-docs (if needed)
git cherry-pick <commit sha for Nth commit>

git push --set-upstream origin 2.<minor>-cherrypick
# Create an MR using Gitlab, merging this branch back to master, reach out to [^maintainers] to review

Move all OPEN Issues remaining within the current milestone to the next one.
Close Big Bang Milestone in GitLab. (make sure that there are n+2 milestones still, the current one and 1 future milestone, if not contact the maintainers ¹)
Handoff the release to the Maintainers ¹, they will review then celebrate and announce the release in the public MM channel
Reach out to @marknelson to let him know that the release is done and to update the Gitlab Banner

Edited Jul 31, 2025 by Joe Foster