Release 1.32.0

Release Process

1. Release Prep

️ Note
Parts of the release process have been automated by Sweet Release. The process of creating documentation will vary from the instructions below.

• Verify that the previous release branch commit hash matches the last release tag. Investigate with previous RE if they do not match

• Create release branch with name release-1.<x>.x. Example: release-1.30.x

  Important
  The release branch name must end with x.

• Copy markdown from previous release notes. Build new draft release notes in the dogfood repo /docs/release directory. Make a new file release-notes-x-x-x.md. Edit the contents and commit it to the repo for the benefit of the next release engineer.

  The command below will get you the BB Versions for all packages to use in the package table - make sure to run it from the root of the repo while on the release branch:

  yq e '(.*.git.tag | select(. != null) | [{"path":(path | .[-3]), "value":.}], .addons.*.git.tag | select(. != null) | [{"Package":(path | .[-3]), "BB Version":.}])' chart/values.yaml

  For the Package Version you will need to check each package manually. Depending on the package we may be tracking one or more of the image tags or the Chart's appVersion.

• Release specific code changes.

  Tip
  Make the following changes in a single commit so it can be cherry picked into master later.

  • Bump self-reference version in base/gitrepository.yaml

  • Update chart release version chart/Chart.yaml

  • Bump badge at the top of README.md

  • Update /Packages.md with any new Packages

  • Update CHANGELOG.md with links to MRs and any upgrade notices/known issues. release-diff update link for release

  • Update README.md using helm-docs.

    # from root dir of your release branch
    docker run -v "$(pwd):/helm-docs" -u $(id -u) jnorwood/helm-docs:v1.5.0 -s file -t .gitlab/README.md.gotmpl --dry-run > README.md

2. Test and Validate Release Candidate

Upgrade the release branch on Dogfood cluster

WARNING: This cluster is in use by the CI runners. Upgrade only, do not delete and redeploy. Only follow the "Connecting to the API Server" section in the link below.

• Connect to Cluster

• Review Elasticsearch Health and trial License status & follow these steps if expired:

  kubectl delete hr ek eck-operator fluentbit -n bigbang
  kubectl delete ns eck-operator logging
  flux reconcile kustomization environment -n bigbang
  flux suspend hr bigbang -n bigbang
  flux resume hr bigbang -n bigbang

• Review Mattermost Enterprise trial license status & follow these steps if expired:

  To "renew" mattermost enterprise trial license, connect to RDS postgres DB using psql (get command and auth from Ryan/Micah/Branden)

  \c mattermost
  select * from "public"."licenses";
  delete from "public"."licenses";
  \q
  kubectl delete mattermost mattermost -n mattermost

• If Flux has been updated in the latest release, checkout your release branch on the BB repo and run ./scripts/install_flux.sh -s (the -s option will reuse the existing secret so you don't have to provide credentials)

• Update bigbang/base/kustomization.yaml & bigbang/prod/kustomization.yaml with release branch.

• Verify cluster has updated to the new release

  • Packages have fetched the new revision and match the new release

  • Packages have reconciled

    # check release
    watch kubectl get gitrepositories,kustomizations,hr,po -A

    If flux has not updated after ten minutes:

    flux reconcile hr -n bigbang bigbang --with-source

    If flux is still not updating, delete the flux source controller:

    kubectl get all -n flux-system 
    kubectl delete pod/source-controller-xxxxxxxx-xxxxx -n flux-system

    If the helm release shows max retries exhausted, you will need to delete helm release secrets and reconcile in flux as follows:

    $ kubectl get secrets -n bigbang | grep ${HR-NAME}

    sh.helm.release.v1.${HR-NAME}-${HR-NAME}.v1                                       helm.sh/release.v1                    1      18h
    sh.helm.release.v1.${HR-NAME}-${HR-NAME}.v2                                       helm.sh/release.v1                    1      17h
    sh.helm.release.v1.${HR-NAME}-${HR-NAME}.v3                                       helm.sh/release.v1                    1      17m

    # Delete the latest one:
    $ kubectl delete secret -n bigbang sh.helm.release.v1.${HR-NAME}-${HR-NAME}.v3
    
    # Then need to run flux stuff:
    $ flux suspend hr -n bigbang bigbang
    $ flux resume hr -n bigbang bigbang
    $ flux reconcile hr ${HR-NAME} -n bigbang --with-source

Confirm app UIs are loading

Important
When verifying each application UI is loading, also verify the website certificates are valid.

• anchore
• argocd
• gitlab
• tracing
• kiali
• kibana
• mattermost (chat)
• minio
• alertmanager
• grafana
• prometheus
• sonarqube
• twistlock
• nexus
• keycloak

Logging

• Login to kibana with SSO
• Kibana is actively indexing/logging.

Monitoring

• Login to grafana with SSO
• Contains Kubernetes Dashboards and metrics
• contains Istio dashboards
• Login to prometheus
• All apps are being scraped, no errors

Cluster Auditor

• Login to grafana with SSO
• OPA Violations dashboard is present and shows violations in namespaces (check gitlab-runners ns to validate violations over time)

Kiali

• Login to kiali with SSO
• Validate graphs and traces are visible under applications/workloads
• Validate no errors appear

  ️ Note
  Red notification bell would be visible if there are errors.

GitLab

• Login to gitlab with SSO
• Edit profile and change user avatar
• Create new public group with release name. Example release-1-8-0
• Create new public project with release name. Example release-1-8-0
• git clone project
• Pick one of the project folders from https://github.com/SonarSource/sonar-scanning-examples/tree/master/sonarqube-scanner/src and copy all the files into your clone from dogfood, then push up
• docker push and docker pull image to/from registry

docker pull alpine
docker tag alpine registry.dogfood.bigbang.dev/<GROUPNAMEHERE>/<PROJECTNAMEHERE>/alpine:latest
docker login registry.dogfood.bigbang.dev
docker push registry.dogfood.bigbang.dev/<GROUPNAMEHERE>/<PROJECTNAMEHERE>/alpine:latest

Sonarqube

• Login to sonarqube with SSO
• Add a project for your release
• Generate a token for the project and copy the token somewhere safe for use later
• Click other, linux, and copy the projectKey from -Dsonar.projectKey=XXXXXXX for use later
• After completing the gitlab runner test return to sonar and check that your project now has analysis

️ Note
The project token and project key are different values.

Gitlab Runner

• Log back into gitlab and navigate to your project
• Under settings, CI/CD, variables add two vars:
  • SONAR_HOST_URL set equal to https://sonarqube.dogfood.bigbang.dev/
  • SONAR_TOKEN set equal to the token you copied from Sonarqube earlier (make this masked)
• Add a .gitlab-ci.yml file to the root of the project, paste in the contents of sample_ci.yaml, replacing -Dsonar.projectKey=XXXXXXX with what you copied earlier
• Commit, validate the pipeline runs and succeeds (may need to retry if there is a connection error), then return to the last step of the sonar test

Nexus

• Login to Nexus as admin, password is in the nexus-repository-manager-secret secret:

  # looks like the username but not the pw might be stored with a newline, hence the ^ instead of <-
  kubectl get secret nexus-repository-manager-secret -n nexus-repository-manager -o json | jq -r '.data["admin.username"]' | base64 -d ; echo ' ^ admin username'
  kubectl get secret nexus-repository-manager-secret -n nexus-repository-manager -o json | jq -r '.data["admin.password"]' | base64 -d ; echo ' <- admin password'

• Validate there are no errors displaying in the UI

• Push/pull an image to/from the nexus registry

  • docker login containers.dogfood.bigbang.dev with the credentials from the encrypted values (or the admin user credentials)
  • docker tag alpine:latest containers.dogfood.bigbang.dev/alpine:1-20-0 (replace with your release number, pick a different image to tag if you want)
  • docker push containers.dogfood.bigbang.dev/alpine:1-20-0
  • Pull down the image for the previous release (docker pull containers.dogfood.bigbang.dev/alpine:1-19-0)

Anchore

• Login to Anchore with SSO

• Log out and log back in as the admin user - password is in anchore-anchore-engine-admin-pass secret (admin will have pull credentials set up for the registries):

  kubectl get secret anchore-anchore-engine-admin-pass -n anchore -o json | jq -r '.data.ANCHORE_ADMIN_PASSWORD' | base64 -d ; echo ' <- password'

• Scan image in dogfood registry, registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest

• Scan image in nexus registry, containers.dogfood.bigbang.dev/alpine:1-19-0 (use your release number)

• Validate scans complete and Anchore displays data (click the SHA value for each image)

Argocd

• Login to argocd with SSO

• Logout and login with username admin. The password is in the argocd-initial-admin-secret secret. If that doesn't work attempt a password reset:

  kubectl -n argocd get secret argocd-initial-admin-secret -o json |  jq '.data|to_entries|map({key, value:.value|@base64d})|from_entries'

• Create application

  TODO: Test creating application with YAML template.

  apiVersion: argoproj.io/v1alpha1
  kind: Application
  metadata:
  name: podinfo
  spec:
  destination:
      name: ''
      namespace: podinfo
      server: 'https://kubernetes.default.svc'
  source:
      path: chart
      repoURL: 'https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/podinfo.git'
      targetRevision: HEAD
  project: default
  syncPolicy:
      automated:
      prune: true
      selfHeal: true
      syncOptions:
      - CreateNamespace=true

  *click* create application
  application name: podinfo
  Project: default
  Sync Policy: Automatic
  Sync Policy: check both boxes
  Sync Options: check "auto-create namespace"
  Repository URL: https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/podinfo.git
  Revision: HEAD
  Path: chart
  Cluster URL: https://kubernetes.default.svc
  Namespace: podinfo
  *click* Create (top of page)

  The app should sync and come healthy.

• Delete application

Minio

• Log into the Minio UI - access and secret key are in the minio-root-creds-secret secret

    kubectl -n minio get secret minio-root-creds-secret -o json | jq -r '.data.accesskey' | base64 -d ; echo ' <- access key'
    kubectl -n minio get secret minio-root-creds-secret -o json | jq -r '.data.secretkey' | base64 -d ; echo ' <- secret key'

• Create bucket

• Store file to bucket

• Download file from bucket

• Delete bucket and files

Mattermost

• Login to mattermost with SSO

• Update/modify profile picture

• Send chats/validate chats from previous releases are visible.

  Tip
  The ability to see chats in other teams requires Mattermost administrator rights.

Twistlock

• Login to twistlock/prisma cloud with the credentials encrypted in bigbang/prod/environment-bb-secret.enc.yaml

  # from <repo>/bigbang/customers/bigbang project root dir
  sops --decrypt environment-bb-secret.enc.yaml | grep -1 twistlock
  

• Only complete if Twistlock was upgraded

  • Navigate to Manage -> Defenders -> Deploy
  • 3: twistlock-console
  • 12: On Toggle on "Monitor Istio"
  • 14: Off Disable official registry
  • 15: registry1.dso.mil/ironbank/twistlock/defender/defender:latest
  • 16: private-registry
  • 17: On Deploy Defenders with SELinux Policy
  • 17: On Nodes use Container Runtime Interface (CRI), not Docker
  • 17: On Nodes runs inside containerized environment
  • 18b: download the yaml files
  • Apply the yaml in the dogfood cluster, validate the pods go to running

• Under Manage -> Defenders -> Manage, make sure # of defenders online is equal to number of nodes on the cluster

• Under Radars -> Containers, validate pods are shown across all namespaces

Kyverno

• Test secret sync in new namespace

  # create secret in kyverno NS
  kubectl create secret generic \
    -n kyverno kyverno-bbtest-secret \
    --from-literal=username='username' \
    --from-literal=password='password' 
    
  # Create Kyverno Policy
  kubectl apply -f https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/kyverno/-/raw/main/chart/tests/manifests/sync-secrets.yaml
  
  # Check if secret is create in NEW namespace
  kubectl create ns kyverno-test   # wait for 5s for Policy to be ready
  kubectl label ns kyverno-test kubernetes.io/metadata.name=kyverno-bbtest --overwrite=true
  kubectl get secrets kyverno-bbtest-secret -n kyverno-test  # Test passed if found
  
  - [ ] Delete the test resources
  # If successful, delete test resources
  kubectl delete -f https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/kyverno/-/raw/main/chart/tests/manifests/sync-secrets.yaml
  kubectl delete secret kyverno-bbtest-secret -n kyverno
  kubectl delete ns kyverno-test

Velero

• Backup PVCs velero_test.yaml

  kubectl apply -f ./velero_test.yaml
  # exec into velero_test container, check log
  veleropod=`kubectl get pod -n velero-test -o json | jq -r '.items[].metadata.name'`
  kubectl exec $veleropod -n velero-test -- tail /mnt/velero-test/test.log

  Install the velero CLI on your workstation if you don't already have it (for MacOS, run "brew install velero").

  Then set VERSION to the release you are testing and use the CLI to create a test backup:

  VERSION=1-2-3
  velero backup create velero-test-backup-${VERSION} -l app=velero-test
  velero backup get

  Wait a bit, re-run velero backup get, when it shows "Completed" delete the app.

  kubectl delete -f ./velero_test.yaml

   namespace "velero-test" deleted
   persistentvolumeclaim "velero-test" deleted
   deployment.apps "velero-test" deleted

• Restore PVCs

  Now test restoring from backup.

  velero restore create velero-test-restore-${VERSION} --from-backup velero-test-backup-${VERSION}
  # exec into velero_test container
  kubectl exec $veleropod -n velero-test -- cat /mnt/velero-test/test.log
  # Old log entries and new should be in log if backup was done correctly

• Cleanup test

  kubectl delete -f ./velero_test.yaml

Keycloak

• Login to Keycloak admin console. The credentials are in the keycloak-credentials secret:

  kubectl get secret keycloak-credentials -n keycloak -o json | jq -r '.data.adminuser' | base64 -d ; echo " <- admin user"
  kubectl get secret keycloak-credentials -n keycloak -o json | jq -r '.data.password' | base64 -d ; echo " <- password"

3. Create Release

• Re-run helm docs in case any package tags changed as a result of issues found in testing.

• Create release candidate tag based on release branch. Tag EX: 1.8.0-rc.0.

  Message: release candidate
  Release Notes: **Leave Blank**

• Passed tag pipeline.

• Create release tag based on release branch. Tag EX: 1.8.0.

  Message: release 1.x.x
  Release Notes: **Leave Blank**

• Passed release pipeline.

• Add release notes to release.

• Cherry-pick release commit(s) as needed with merge request back to master branch

• Close Big Bang Milestone in GitLab.

• Celebrate and announce release

changed milestone to %1.32.0

added release teamci/pipelines labels

set weight to 3

assigned to @razzle

changed iteration to Big Bang Iterations Apr 19, 2022 - May 2, 2022

added statusdoing label

mentioned in issue #1145 (closed)

added priority4 label

changed due date to April 22, 2022

changed the description

marked the checklist item Verify that the previous release branch commit hash matches the last release tag. Investigate with previous RE if they do not match as completed

marked the checklist item Create release branch with name release-1.<x>.x. Example: release-1.30.x as completed

marked the checklist item Copy markdown from previous release notes. Build new draft release notes in the dogfood repo /docs/release directory. Make a new file release-notes-x-x-x.md. Edit the contents and commit it to the repo for the benefit of the next release engineer. as completed

marked the checklist item Bump self-reference version in base/gitrepository.yaml as completed

marked the checklist item Release specific code changes. as completed

marked the checklist item Update chart release version chart/Chart.yaml as completed

marked the checklist item Bump badge at the top of README.md as completed

marked the checklist item Update /Packages.md with any new Packages as completed

marked the checklist item Update README.md using helm-docs. as completed

marked the checklist item Update CHANGELOG.md with links to MRs and any upgrade notices/known issues. release-diff update link for release as completed

Release specific code changes commit for cherry picking: https://repo1.dso.mil/platform-one/big-bang/bigbang/-/commit/fea1a265cc5b781ad1b4c5d2c3ae47347f5581ec

changed the description

marked the checklist item Update bigbang/base/kustomization.yaml & bigbang/prod/kustomization.yaml with release branch. as completed

marked the checklist item If Flux has been updated in the latest release, checkout your release branch on the BB repo and run ./scripts/install_flux.sh -s (the -s option will reuse the existing secret so you don't have to provide credentials) as completed

marked the checklist item Login to grafana with SSO as completed

marked the checklist item Contains Kubernetes Dashboards and metrics as completed

marked the checklist item contains Istio dashboards as completed

marked the checklist item Login to prometheus as completed

marked the checklist item All apps are being scraped, no errors as completed

marked the checklist item Login to grafana with SSO as completed

marked the checklist item OPA Violations dashboard is present and shows violations in namespaces (check gitlab-runners ns to validate violations over time) as completed

marked the checklist item Login to kiali with SSO as completed

marked the checklist item Validate graphs and traces are visible under applications/workloads as completed

marked the checklist item Validate no errors appear as completed

marked the checklist item Login to gitlab with SSO as completed

marked the checklist item Edit profile and change user avatar as completed

marked the checklist item Create new public group with release name. Example release-1-8-0 as completed

marked the checklist item Create new public project with release name. Example release-1-8-0 as completed

marked the checklist item git clone project as completed

marked the checklist item Pick one of the project folders from https://github.com/SonarSource/sonar-scanning-examples/tree/master/sonarqube-scanner/src and copy all the files into your clone from dogfood, then push up as completed

marked the checklist item docker push and docker pull image to/from registry as completed

marked the checklist item Login to sonarqube with SSO as completed

marked the checklist item Add a project for your release as completed

marked the checklist item Generate a token for the project and copy the token somewhere safe for use later as completed

marked the checklist item Click other, linux, and copy the projectKey from -Dsonar.projectKey=XXXXXXX for use later as completed

marked the checklist item Under settings, CI/CD, variables add two vars: as completed

marked the checklist item Log back into gitlab and navigate to your project as completed

marked the checklist item SONAR_HOST_URL set equal to https://sonarqube.dogfood.bigbang.dev/ as completed

marked the checklist item SONAR_TOKEN set equal to the token you copied from Sonarqube earlier (make this masked) as completed