Patch Release 2.21.1

Patch Release Process

Forward

Typically, patch releases are conducted by anchors or more experienced team members. If you are not experienced with the Big Bang minor release process, it is not recommended for you to perform a patch release.
This upgrade process is applicable only to patch Big Bang version upgrades of Big Bang major versions 1 and 2. It is subject to change in future major versions.
Patch releases are added to the existing minor release branch. No new branches should be created as part of this process.
The patch release tag format is as follows: release-<major>.<minor>.<patch> where <patch> is the patch release number. Example: release-2.7.1. The <patch> string is a placeholder for the patch release number in this document.

NOTE: As you work through the release make a list of pain points / unclear steps. Once the release is complete, provide this feedback to the maintainers via MM and/or an MR to update the documentation. This allows us to continuously improve this process for future release engineers.

1. Release Prep

Make sure you have both Big Bang (product) and Big Bang (dogfood) checked out and updated
Ensure that your current working directory is dogfood-bigbang/docs/release
Check last release SHAs
- Verify that the previous release branch commit hash matches the last release tag hash. Investigate with previous release engineer if they do not match.
- Go to Branches in Big Bang (product).
- Find the latest release branch. It should be in the format release-<major_version>.<minor_version>.x (e.g. release-2.7.x). You will see an eight-character commit hash below it (e.g. 6c746edd)
- Go to Tags in a separate tab. Here you will see the latest release tag listed first. Verify that this tag has the same hash you found in the previous step.

Check out the most recent minor release branch, all patch changes will be applied to this branch.

# CLI example, replace `<release-branch>` with the name of the most recent minor release branch
$ cd [path to local bigbang]
$ git checkout <release-branch>
$ git pull

Cherrypick the required patch commit(s) into the corresponding release branch eg: release-2.18.x for 2.18.1+ patch release

Upgrade Big Bang (product) version references

💡 Tip Make the following changes in a single commit so that they can be cherry picked into master later. All changes should be made in the Big Bang (product) repository

In base/gitrepository.yaml, update ref.tag to your current release version.
Update chart release version chart/Chart.yaml

Add a new changelog entry for the release, ex:

## [2.<minor>.<patch>]

- [!2.<minor>.<patch>](https://repo1.dso.mil/big-bang/bigbang/-/merge_requests?scope=all&utf8=%E2%9C%93&state=merged&milestone_title=2.<minor>.<patch>); List of merge requests in this release.

<!-- Note: milestone_title=2.<minor>.<patch> version must match the given patch release version -->

Update /docs/understanding-bigbang/configuration/base-config.md using helm-docs.

# example release 2.<minor>.x
$ cd bigbang
$ git checkout release-2.<minor>.x
$ docker run -v "$(pwd):/helm-docs" -u $(id -u) jnorwood/helm-docs:v1.5.0 -s file -t .gitlab/base_config.md.gotmpl --dry-run > ./docs/understanding-bigbang/configuration/base-config.md

Commit & push changes

$ git commit -am 'version updates for release <release-tag>'
$ git push

Build the amended release notes.
- Return to dogfood-bigbang/docs/release as your current working directory
- Make a copy of the most recent minor release notes
- Increment the patch version in the copied document name and title
- Add an additional block before ## Upgrade Notices with patch changes
```
## Patch-Specific Changes

### Package Name
- [!$MR_NUMBER]($LINK_TO_MR): $MR_TITLE or $SHORT_DESCRIPTION
```
- Update Packages table with new package versions
- Commit the resulting release notes and push to master in Big Bang (dogfood)
Reach out to Anchors @ryan.j.garcia @chris.oconnell @michaelmartin to review your commits to the release branch prior to moving on to the next step

2. Upgrade and Debug Cluster

Connect to the dogfood cluster

📌 NOTE: If you have issues with the AWS CLI commands, adding via the AWS web console is another option. Reach out to core maintainers for assistance.
Verify Flux has reconciled cherrypicked updates
- Packages have fetched new git repository revisions and match the new release
```
$ kubectl get gitrepositories -A
```
- Packages have reconciled
```
$ watch kubectl get gitrepositories,kustomizations,hr,po -A
```
- All pods are in Running or Completed status
```
$ kubectl get pods -A
```

If Flux doesn't reconcile changes after 10 minutes, follow the troubleshooting steps in the minor release process.

3. UI Tests

Perform targeted testing of the patched packaged following the steps in the minor release README.

GitLab Sonarqube Gitlab Runner

Login to gitlab with SSO
Edit profile and change user avatar. If your avatar doesn't change right away, check again later. It can take several minutes or more.
Go to Security > Access Token and create a personal access token. Select all scopes. Record the token. You'll need it for the next two steps.
R2-D2: Gitlab Test
Gitlab Manual Steps
- Create new public group with release name. ie. release-2-<minor>-x
- Create new public project (under the group you just made), also with release name (e.g.release-2.7.x if you're working on release 2.7.0).
- git clone project
- Pick one of the project folders from Sonarqube Samples and copy all the files into your clone from dogfood
  - Go to Security > Access Token and create a personal access token. Select all scopes. Record the token. You'll need it for the next two steps.
  - Git commit and push your changes to the repo. When prompted enter your username and the access token as your password.
- Test a docker image push/pull to/from registry
```
docker pull alpine
docker tag alpine registry.dogfood.bigbang.dev/<GROUPNAMEHERE>/<PROJECTNAMEHERE>/alpine:latest
docker login registry.dogfood.bigbang.dev # Enter your gitlab username and personal access token
docker push registry.dogfood.bigbang.dev/<GROUPNAMEHERE>/<PROJECTNAMEHERE>/alpine:latest
docker image rm registry.dogfood.bigbang.dev/<GROUPNAMEHERE>/<PROJECTNAMEHERE>/alpine:latest
docker pull registry.dogfood.bigbang.dev/<GROUPNAMEHERE>/<PROJECTNAMEHERE>/alpine:latest
```
Sonarqube Manual Steps
- Login to sonarqube with SSO
- Add a project for your release. When prompted for how you want to analyze your repository, choose "Locally."
- Generate a token for the project and copy the token somewhere safe for use later
- When prompted to "Run analysis on your project" choose "Other (for JS, TS, Go, Python, PHP...)". For "What is your OS?" choose Linux. Copy everything that appears under "Running a SonarQube analysis is straighforward. You just need to execute the following commands in your project's folder" and save it somewhere secure for use later.
- After completing the gitlab runner test return to sonar and check that your project now has analysis
  
  ℹ️ NOTE The project token and project key are different values.
Gitlab Runner Manual Steps
- Log back into gitlab and navigate to your project
- Under settings, CI/CD, variables add two vars:
  - SONAR_HOST_URL set equal to https://sonarqube.dogfood.bigbang.dev/
  - SONAR_TOKEN set equal to the token you copied from Sonarqube earlier (make this masked)
- Under settings, CI/CD, deselect "Default to Auto DevOps pipeline" and click Save changes.
- Add a .gitlab-ci.yml file to the root of the project, paste in the contents of sample_ci.yaml, replacing -Dsonar.projectKey=XXXXXXX with what you copied earlier
- Commit, validate the pipeline runs and succeeds (may need to retry if there is a connection error), then return to the last step of the sonar test

ArgoCD

R2-D2: ArgoCD Test

ArgoCD Manual Steps

Logout and login with username admin. The password is in the argocd-initial-admin-secret secret. If that doesn't work attempt a password reset:

kubectl -n argocd get secret argocd-initial-admin-secret -o json |  jq '.data|to_entries|map({key, value:.value|@base64d})|from_entries'

Create application

Click [Create Application], fill in the below

Setting	Value
Application Name	podinfo
Project	default
Sync Policy	Automatic
Sync Policy	check both boxes
Sync Options	check "auto-create namespace"
Repository URL	https://repo1.dso.mil/big-bang/apps/sandbox/podinfo.git
Revision	HEAD
Path	chart
Cluster URL	https://kubernetes.default.svc
Namespace	podinfo

Click [Create] (top of page)
Validate app syncs/becomes healthy

Delete app

Grafana & ArgoCD & Anchore & Mattermost SSO

R2-D2: SSO Grafana, ArgoCD, Anchore and Mattermost Test
SSO Manual Steps
- Login to grafana with SSO
- Login to ArgoCD with SSO
- Login to anchore with SSO
- Login to mattermost

4. Create Release

Finalize the tag in chart/Chart.yaml (remove -rc.x if present), commit and push this change
Create release candidate tag based on release branch, ie. 2.<minor>.<patch>-rc.0. Tagging will additionally create release artifacts and the pipeline runs north of 1 hour. You will need to request tag permissions from the maintainers.
- To do this via the UI (generally preferred): tags page -> new tag, name: 2.<minor>.<patch>-rc.0, create from: release-2.<minor>.x, message: "release candidate", release notes: leave empty
- To do this via git CLI:
```
$ git tag -a 2.<minor>.<patch>-rc.0 -m "release candidate"
# list the tags to make sure you made the correct one
$ git tag -n
# push
$ git push --tags
```
Confirm pipeline passed for Release Candidate tag.
Confirm pipeline passed in bb-docs-compiler for latest RC tag (Gets created/scheduled towards the end of the bigbang release pipeline).

Review all pipeline output looking for failures or warnings. Reach out to the maintainers for a quick review before proceeding.
Create release tag based on release branch. ie. 2.<minor>.<patch>.
- To do this via the UI (generally preferred): tags page -> new tag, name: 2.<minor>.<patch>, create from: release-1.<minor>.x, message: release 2.<minor>.<patch>, release notes: leave empty
- To do this via git CLI:
```
$ git tag -a 2.<minor>.<patch> -m "release 2.<minor>.<patch>"
# list the tags to make sure you made the correct one
$ git tag -n
# push
$ git push --tags
```
Ensure Release Notes are accurate
- Amend Upgrade Notices, based off of the listed notices in the release issue. Also review with maintainers to see if any other major changes should be noted.
- Adjust known issues as needed: If an issue has been resolved it can be removed and if any new issues were discovered they should be added.
- Verify table contains all chart upgrades mentioned in the MR list for both the minor and patch release.
- Verify all internal comments are removed from Release Notes, such as comments directing the release engineer to copy/move things around
Passed release pipeline.

Review all pipeline output looking for failures or warnings. Reach out to the maintainers for a quick review before proceeding. One of the last stages of the pipeline will auto grab the release notes generated above from deployments/bigbang in master. Ensure they populated and are formatted correctly.

Cherry-pick release commit(s) as needed with merge request back to master branch. We do not ever merge release branch back to master. Instead, make a separate branch from master and cherry-pick the release commit(s) into it. Use the resulting MR to close the release issue.

# Navigate to your local clone of the BB repo
cd path/to/my/clone/of/bigbang
# Get the most up to date master
git checkout master
git pull
# Check out new branch for cherrypicking
git checkout -b 2.<minor>-cherrypick

# Repeat the following for whichever commits are required from release
# Typically this is the initial release commit (bumped GitRepo, Chart.yaml, CHANGEGLOG, README) and a final commit that re-ran helm-docs (if needed)
git cherry-pick <commit sha for Nth commit>

git push --set-upstream origin 2.<minor>-cherrypick
# Create an MR using Gitlab, merging this branch back to master, reach out to maintainers to review

Close Big Bang Milestone in GitLab.
Handoff the release to the maintainers, they will review then celebrate and announce the release in the public MM channel

Edited Feb 29, 2024 by Nicole Dupree

Admin message