Release 1.9.0

Release Checklist

1. Release Prep

• Verify that the previous release branch commit hash matches the last release tag. Investigate with previous RE if they do not match
• Create release branch with name release-1.x.x
• Build draft release notes, see release_notes_template.md
• Release specific code changes. Make the following changes in a single commit so it can be cherry picked into master later.

  • Bump self-reference version in base/gitrepository.yaml

  • Update chart release version chart/Chart.yaml

  • Update /Packages.md with any new Packages

  • Update CHANGELOG.md with links to MRs and any upgrade notices/known issues

  • Update README.md using helm-docs. Overwrite the existing readme file.

    # from root dir of your release branch
    docker run -v "$(pwd):/helm-docs" -u $(id -u) jnorwood/helm-docs:v1.5.0 -s file -t .gitlab-ci/README.md.gotmpl --dry-run > README.md

2. Test and Validate Release Candidate

Deploy release branch on Dogfood cluster

• Connect to Cluster

• Update bigbang/base/kustomization.yaml & bigbang/prod/kustomization.yaml with release branch.

• Verify cluster has updated to the new release and all package versions match the new chart, flux may take a few minutes to reconcile.

  # check release
  watch kubectl get gitrepositories,kustomizations,hr,po -A
  # if flux has not updated after 10 minutes.
  flux reconcile hr -n bigbang bigbang --with-source
  # if it is still not updating, delete the flux source controller 
  kubectl get all -n flux-system 
  kubectl delete pod/source-controller-xxxxxxxx-xxxxx -n flux-system

Confirm app UIs are loading

• anchore
• argocd
• gitlab
• tracing
• kiali
• kibana
• mattermost
• minio
• alertmanager
• grafana
• prometheus
• sonarqube
• twistlock
• nexus
• TLS/SSL certs are valid

Logging

• Login to kibana with SSO
• Kibana is actively indexing/logging.

Cluster Auditor

• Login to kibana with SSO
• violations index is present and contains images that aren't from registry1

Monitoring

• Login to grafana with SSO
• Contains Kubernetes Dashboards and metrics
• contains istio dashboards

Kiali

• Login to kiali with SSO

Sonarqube

• Login to sonarqube with SSO

GitLab & Runners

• Login to gitlab with SSO

• Create new public group with release name. Example release-1-8-0

• Create new public project with release name. Example release-1-8-0

• git clone and git push to new project

• docker push and docker pull image to registry

  docker pull alpine
  docker tag alpine registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest
  docker login registry.dogfood.bigbang.dev
  docker push registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest

• Edit profile and change user avatar

• Test simple CI pipeline. sample_ci.yaml

Anchore

• Login to anchore with SSO
• Scan image in dogfood registry, registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest
• Scan image in registry1, registry.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana/busybox

Argocd

• Login to argocd with SSO
• Logout and login with admin
• Create application

Minio

• Login to Minio
• Create volume
• Store file to volume
• Download file from volume
• Delete volume and files

Mattermost

• Login to mattermost with SSO
• Elastic integration

Velero

• Backup PVCs

  kubectl apply -f ./velero_test.yaml
  # exec into velero_test container
  cat /mnt/velero-test/test.log
  # take note of log entries and exit exec 

  velero backup create velero-test-backup-1-8-0 -l app=velero-test
  velero backup get
  kubectl delete -f ./velero_test.yaml
  kubectl get pv | grep velero-test
  kubectl delete pv INSERT-PV-ID

• Restore PVCs

  velero restore create velero-test-restore-1-8-0 --from-backup velero-test-backup-1-8-0
  # exec into velero_test container
  cat /mnt/velero-test/test.log
  # old log entires and new should be in log if backup was done correctly

Keycloak

3. Create Release

• Create release candidate tag based on release branch. Tag EX: 1.8.0-rc.0.

  Message: release canidate
  Release Notes: **Leave Blank**

• Passed tag pipeline.
• Create release tag based on release branch. Tag EX: 1.8.0.

  Message: release canidate
  Release Notes: **Leave Blank**

• Passed release pipeline.
• Add release notes to release.
• Cherry-pick release commit(s) as needed with merge request back to master branch
• Celebrate and announce release

Candidate Release Notes

Please see our documentation page for more information on how to consume and deploy BigBang.

Upgrade Notices

• Network polices have been added and are enabled by default for Authservice, ECK Operator, Gitlab, Monitoring, and SonarQube. Network policy enforcement for other packages will be added with future BigBang releases

# -- Global NetworkPolicies settings
networkPolicies:
  # -- Toggle all package NetworkPolicies, can disable specific packages with `package.networkPolicies.enabled`
  enabled: true

Packages

Pull in and replace chart then update for new release

Package	Type	Package Version	BB Version
Istio Operator	Core	1.8.4	1.8.4-bb.1
Istio Controlplane	Core	1.8.4	1.8.4-bb.2
Jaeger	Core	1.22.0	2.19.1-bb.4
Kiali	Core	1.32.0	1.32.0-bb.1
OPA Gatekeeper	Core	3.3.0	3.3.0-bb.2
Monitoring	Core	G: 7.1.3-1, P: 2.22.0, A: 0.21.0	11.0.0-bb.21
ECK Operator	Core	1.4.0	1.4.0-bb.2
Elasticsearch Kibana	Core	E: 7.10.0 K: 7.10.2	0.1.11-bb.0
Fluentbit	Core	1.7.4	0.15.8-bb.1
Cluster Auditor	Core	0.3.2	0.2.0-bb.2
Twistlock	Core	21.04.412	0.0.4-bb.0
Gitlab	Addon	13.10.3	4.10.3-bb.6
Gitlab Runners	Addon	13.9.0	0.26.0-bb.1
Mattermost Operator	Addon	1.14.0	1.14.0-bb.1
Mattermost	Addon	5.34.2	0.1.5-bb.0
MinIO Operator	Addon	2.0.9	2.0.9-bb.3
MinIO	Addon	RELEASE.2020-11-19T23-48-16Z	2.0.9-bb.9
Authservice	Addon	0.4.0	0.4.0-bb.5
Anchore	Addon	ENG: 0.9.3, ENT: 3.0.2	1.12.13-bb.0
SonarQube	Addon	8.7.1 (w/ p1 plugins)	9.2.6-bb.11
Argocd	Addon	1.8.4 (w/ p1 plugins)	2.14.7-bb.5
Velero	Addon	1.5.3	2.14.8-bb.0
Keycloak	Addon	13.0.0	11.0.0-bb.0
Nexus	Addon	3.29.0-02	29.1.0-bb.0

Changes in v1.9.0

• !445: Nexus added to BB
• !488: Authservice support external redis service
• !499: Mattermost Operator optional network policies
• !503: Sonarqube optional network policies
• !504: Gitlab optional network policies
• !510: ECK Operator optional network policies
• !511: Authservice optional network policies
• !513: Monitoring optional network policies
• !514: Cluster Auditor & OPA Gatekeeper constraint-templates and added conditional enforcement

Known Issues

• If the following error is seen on any helm releases scheme "" not supported try updating flux to latest ib images. A simple way to do this is by adding registry credentials to the flux-system namespace and applying the flux.yaml:

kubectl create secret docker-registry private-registry --docker-server=registry1.dso.mil --docker-username=<Your IronBank Username> --docker-password=<Your IronBank Personal Access Token> --docker-email=<Your E-mail Address> -n flux-system
curl https://repo1.dso.mil/platform-one/big-bang/bigbang/-/raw/master/scripts/deploy/flux.yaml | kubectl apply -f -

• #329: Velero unable to restore PersistentVolumes.
• #480: Jaeger fails to install on certain newer versions of kubernetes

Helpful Links

As always, we welcome and appreciate feedback from our community of users. Please feel free to:

• Open issues here
• Join our chat
• Check out the documentation for guidance on how to get started

Future

Don't see your feature and/or bug fix? Check out our roadmap for estimates on when you can expect things to drop, and as always, feel free to comment or create issues if you have questions, comments, or concerns.

changed due date to June 04, 2021

changed milestone to %1.9.0

added priority4 label

changed iteration to Big Bang Iterations Jun 2, 2021 - Jun 14, 2021

set weight to 5

changed the description

added to epic &64 (closed)

removed milestone %1.9.0

changed milestone to %1.9.0

changed the description

marked the checklist item Verify that the previous release branch commit hash matches the last release tag. Investigate with previous RE if they do not match as completed

marked the checklist item Create release branch with name release-1.x.x as completed

marked the checklist item Connect to Cluster as completed

marked the checklist item Update bigbang/base/kustomization.yaml & bigbang/prod/kustomization.yaml with release branch. as completed

changed the description

added statusdoing label

changed the description

marked the checklist item Bump self-reference version in base/gitrepository.yaml as completed

marked the checklist item Update chart release version chart/Chart.yaml as completed