Release 1.21.0

Release Process

1. Release Prep

Verify that the previous release branch commit hash matches the last release tag. Investigate with previous RE if they do not match
Create release branch with name. Ex: release-1.9.x
Build draft release notes, see release_notes_template.md
Release specific code changes. Make the following changes in a single commit so it can be cherry picked into master later.
- Bump self-reference version in base/gitrepository.yaml
- Update chart release version chart/Chart.yaml
- Bump badge at the top of README.md
- Update /Packages.md with any new Packages
- Update CHANGELOG.md with links to MRs and any upgrade notices/known issues. release-diff update link for release
- Update README.md using helm-docs. Overwrite the existing readme file.
```
# from root dir of your release branch
docker run -v "$(pwd):/helm-docs" -u $(id -u) jnorwood/helm-docs:v1.5.0 -s file -t .gitlab-ci/README.md.gotmpl --dry-run > README.md
```

2. Test and Validate Release Candidate

Deploy release branch on Dogfood cluster

Connect to Cluster

Review Elasticsearch Health and trial License status & follow these steps if expired:

kubectl delete hr ek eck-operator fluentbit cluster-auditor
kubectl delete ns eck-operator logging
flux reconcile kustomization environment -n bigbang
flux suspend hr bigbang -n bigbang
flux resume hr bigbang -n bigbang

Review Mattermost Enterprise trial license status & follow these steps if expired:

To "renew" mattermost enterprise trial license:
Connect to RDS posgres DB using `psql` (get command and auth from Ryan/Micah/Branden)
\c mattermost
select * from "public"."licenses";
delete from "public"."licenses";
\q
kubectl delete mattermost mattermost -n mattermost

Update bigbang/base/kustomization.yaml & bigbang/prod/kustomization.yaml with release branch.

Verify cluster has updated to the new release

Packages have fetched the new revision and match the new release

Packages have reconciled

# check release
watch kubectl get gitrepositories,kustomizations,hr,po -A
# if flux has not updated after 10 minutes.
flux reconcile hr -n bigbang bigbang --with-source
# if it is still not updating, delete the flux source controller 
kubectl get all -n flux-system 
kubectl delete pod/source-controller-xxxxxxxx-xxxxx -n flux-system

Confirm app UIs are loading

Logging

Login to kibana with SSO
Kibana is actively indexing/logging.

Cluster Auditor

Login to kibana with SSO
violations index is present and contains images that aren't from registry1

Monitoring

Login to grafana with SSO
Contains Kubernetes Dashboards and metrics
contains istio dashboards
Login to prometheus
All apps are being scraped, no errors

Kiali

Login to kiali with SSO
Validate graphs and traces are visible under applications/workloads
Validate no errors appear (red notification bell would be visible if there are errors)

GitLab

Login to gitlab with SSO
Edit profile and change user avatar
Create new public group with release name. Example release-1-8-0
Create new public project with release name. Example release-1-8-0
git clone project
Pick one of the project folders from https://github.com/SonarSource/sonar-scanning-examples and copy all the files into your clone from dogfood, then push up
docker push and docker pull image to/from registry

docker pull alpine
docker tag alpine registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest
docker login registry.dogfood.bigbang.dev
docker push registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest

Sonarqube

Login to sonarqube with SSO
Add a project for your release
Generate a token for the project and copy the token somewhere safe for use later
Click other, linux, and copy the projectKey from -Dsonar.projectKey=XXXXXXX for use later
After completing the gitlab runner test return to sonar and check that your project now has analysis

Gitlab Runner

Log back into gitlab and navigate to your project
Under settings, CI/CD, variables add two vars:
- SONAR_HOST_URL set equal to https://sonarqube.dogfood.bigbang.dev/
- SONAR_TOKEN set equal to the token you copied from Sonarqube earlier (make this masked)
Add a .gitlab-ci.yml file to the root of the project, paste in the contents of sample_ci.yaml, replacing -Dsonar.projectKey=XXXXXXX with what you copied earlier
Commit, validate the pipeline runs and succeeds (may need to retry if there is a connection error), then return to the last step of the sonar test

Nexus

Login to Nexus as admin
Validate there are no errors displaying in the UI
Push/pull an image to/from the nexus registry
- docker login containers.dogfood.bigbang.dev with the creds from the encrypted values (or the admin user creds)
- docker tag alpine:latest containers.dogfood.bigbang.dev/alpine:1-20-0 (replace with your release number, pick a different image to tag if you want)
- docker push containers.dogfood.bigbang.dev/alpine:1-20-0
- Pull down the image for the previous release (docker pull containers.dogfood.bigbang.dev/alpine:1-19-0)

Anchore

Login to Anchore with SSO
Log out and log back in as the admin user (this user should have pull creds set up for the registries)
Scan image in dogfood registry, registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest
Scan image in nexus registry, containers.dogfood.bigbang.dev/alpine:1-19-0 (use your release number)
Validate scans complete and Anchore displays data (click the SHA value for each image)

Argocd

Login to argocd with SSO
Logout and login with username admin. The password is in the argocd-initial-admin-secret secret. If that doesn't work attempt a password reset.

Create application

*click* create application
application name: argocd-test
Project: default
Sync Policy: Automatic
Sync Policy: check both boxes
Sync Options: check "auto-create namespace"
Repository URL: https://github.com/argoproj/argocd-example-apps
Revision: HEAD
Path: helm-guestbook
Cluster URL: https://kubernetes.default.svc
Namespace: argocd-test
*click* Create (top of page)

The app will not actually sync because gatekeeper policies prevent the images from docker.io. This is ok.

Delete application

Minio

Log into the minio UI as minio with password minio123
Create bucket
Store file to bucket
Download file from bucket
Delete bucket and files

Mattermost

Login to mattermost with SSO
Update/modify profile picture
Send chats/validate chats from previous releases are around
Under system console -> elastic -> index now and validate success (enterprise feature, if the trial has expired contact @micah.nagel or @BrandenCobb for assist)

Twistlock

Velero

Backup PVCs velero_test.yaml

kubectl apply -f ./velero_test.yaml
# exec into velero_test container
cat /mnt/velero-test/test.log
# take note of log entries and exit exec

velero backup create velero-test-backup-1-8-0 -l app=velero-test
velero backup get
kubectl delete -f ./velero_test.yaml
kubectl get pv | grep velero-test
kubectl delete pv INSERT-PV-ID

Restore PVCs

velero restore create velero-test-restore-1-8-0 --from-backup velero-test-backup-1-8-0
# exec into velero_test container
cat /mnt/velero-test/test.log
# old log entires and new should be in log if backup was done correctly

Cleanup test

kubectl delete -f ./velero_test.yaml
kubectl get pv | grep velero-test
kubectl delete pv INSERT-PV-ID

Keycloak

3. Create Release

Re-run helm docs in case any package tags changed as a result of issues found in testing.
Create release candidate tag based on release branch. Tag EX: 1.8.0-rc.0.
```
Message: release candidate
Release Notes: **Leave Blank**
```
Passed tag pipeline.
Create release tag based on release branch. Tag EX: 1.8.0.
```
Message: release 1.x.x
Release Notes: **Leave Blank**
```
Passed release pipeline.
Add release notes to release.
Cherry-pick release commit(s) as needed with merge request back to master branch
Celebrate and announce release

Draft Release Note

Candidate Release Notes

Please see our documentation page for more information on how to consume and deploy BigBang.

Upgrade Notices

Gitlab Upgrade notice:
This release completes istio sidecar proxy injection. Injection has been added for the jobs. There are no known issues. If you discover an issue please open an issue. The Gitlab istio injection can be disabled if necessary with the following BigBang values

addons:
  gitlab:
    istio:
      injection: disabled

Upgrades from previous releases

If coming from a version pre-1.20, note the additional upgrade notices in any release in between. The BB team doesn't test/guarantee upgrades from anything pre-1.20.

Packages

Package	Type	Package Version	BB Version
Istio Controlplane	Core	`1.11.3`	`1.11.3-bb.1`
Istio Operator	Core	`1.11.3`	`1.11.3-bb.2`
Jaeger	Core	`1.27.0`	`2.26.0-bb.0`
Kiali	Core	`1.42.0`	`1.42.0-bb.0`
Cluster Auditor	Core	`0.3.2`	`0.3.0-bb.7`
OPA Gatekeeper	Core	`3.6.0`	`3.6.0-bb.2`
Elasticsearch Kibana	Core	E: `7.13.4` K: `7.12.0`	`0.1.22-bb.0`
ECK Operator	Core	`1.7.1`	`1.7.1-bb.0`
Fluentbit	Core	`1.8.6`	`0.16.6-bb.1`
Monitoring	Core	G: `7.5.2`, P: `2.25.0`, A: `0.21.0`	`14.0.0-bb.17`
Twistlock	Core	`21.08.520`	`0.0.11-bb.0`
Argocd	Addon	`2.0.1` (w/ p1 plugins)	`3.6.8-bb.10`
Authservice	Addon	`0.4.0`	`0.4.0-bb.18`
MinIO Operator	Addon	`4.2.3`	`4.2.3-bb.2`
MinIO	Addon	`RELEASE.2021-08-31T05-46-54Z`	`4.2.3-bb.6`
Gitlab	Addon	`14.3.1`	`5.3.1-bb.9`
Gitlab Runners	Addon	`14.3.1`	`0.33.1-bb.5`
Nexus	Addon	`3.34.1-01`	`34.1.0-bb.4`
SonarQube	Addon	`8.9` (w/ p1 plugins)	`9.6.3-bb.9`
Anchore	Addon	ENG: `0.10.2`, ENT: `3.1.2`	`1.14.7-bb.1`
HAProxy	Addon	`2.3.2`	`1.1.2-bb.2`
Mattermost Operator	Addon	`1.16.0`	`1.16.0-bb.0`
Mattermost	Addon	`5.39.0`	`0.2.4-bb.0`
Velero	Addon	`1.6.3`	`2.23.6-bb.3`
Keycloak	Addon	`14.0.0`	`11.0.1-bb.9`

Changes in 1.21.0

BigBang

!1057: Pipeline Refactor
!966: Updated Flux Script
!1082: Document ImagePullPolicy Defaults/Overrides

Istio-Operator

!1064: Document ImagePullPolicy

HAProxy

!1107: Add HAProxy Ingress Gateway Values.
!1104: Update HAProxy to remove VirtualService

Elasticsearch-Kibana

!1106: Update Elasticsearch custom resource to use Rolling Update Strategy

AuthService

!1117: RequestAuthentication use jwks value over jwksUri if value is present

Twistlock

!1064: Document ImagePullPolicy

Kiali

!1105: Update Kiali to 1.42.0

Velero

!1102: CI Test Update

Gitab & Gitab Runner

!1068: istio injection for jobs

Mattermost

!1073: Use IronBank init container and disable Ingress by default.

Known Issues

On some k8s distros certain components in the kube-system namespace are unable to be scraped by Prometheus
Prometheus scrapes completed pods, this doesn't affect anything but may see some "Down" targets in Prometheus UI - https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring/-/issues/43
Jaeger operator metrics unable to be scraped by Prometheus - https://repo1.dso.mil/platform-one/big-bang/bigbang/-/issues/893
Mattermost metrics may show 404 in Prometheus targets page

Helpful Links

As always, we welcome and appreciate feedback from our community of users. Please feel free to:

Open issues here
Join our chat
Check out the documentation for guidance on how to get started

Future

Don't see your feature and/or bug fix? Check out our roadmap for estimates on when you can expect things to drop, and as always, feel free to comment or create issues if you have questions, comments, or concerns.

Edited Nov 19, 2021 by Tunde Oladipupo

Admin message