UNCLASSIFIED - NO CUI

Release 1.32.0

Release Process

1. Release Prep

ℹ️ Note
Parts of the release process have been automated by Sweet Release. The process of creating documentation will vary from the instructions below.

  • Verify that the previous release branch commit hash matches the last release tag. Investigate with previous RE if they do not match

  • Create release branch with name release-1.<x>.x. Example: release-1.30.x

    ❕Important
    The release branch name must end with x.

  • Copy markdown from previous release notes. Build new draft release notes in the dogfood repo /docs/release directory. Make a new file release-notes-x-x-x.md. Edit the contents and commit it to the repo for the benefit of the next release engineer.

    The command below will get you the BB Versions for all packages to use in the package table - make sure to run it from the root of the repo while on the release branch:

    yq e '(.*.git.tag | select(. != null) | [{"path":(path | .[-3]), "value":.}], .addons.*.git.tag | select(. != null) | [{"Package":(path | .[-3]), "BB Version":.}])' chart/values.yaml

    For the Package Version you will need to check each package manually. Depending on the package we may be tracking one or more of the image tags or the Chart's appVersion.

  • Release specific code changes.

    💡 Tip
    Make the following changes in a single commit so it can be cherry picked into master later.

    • Bump self-reference version in base/gitrepository.yaml

    • Update chart release version chart/Chart.yaml

    • Bump badge at the top of README.md

    • Update /Packages.md with any new Packages

    • Update CHANGELOG.md with links to MRs and any upgrade notices/known issues. release-diff update link for release

    • Update README.md using helm-docs.

      # from root dir of your release branch
docker run -v "$(pwd):/helm-docs" -u $(id -u) jnorwood/helm-docs:v1.5.0 -s file -t .gitlab/README.md.gotmpl --dry-run > README.md

2. Test and Validate Release Candidate

Upgrade the release branch on Dogfood cluster

WARNING: This cluster is in use by the CI runners. Upgrade only, do not delete and redeploy. Only follow the "Connecting to the API Server" section in the link below.

  • Connect to Cluster

  • Review Elasticsearch Health and trial License status & follow these steps if expired:

    kubectl delete hr ek eck-operator fluentbit -n bigbang
kubectl delete ns eck-operator logging
flux reconcile kustomization environment -n bigbang
flux suspend hr bigbang -n bigbang
flux resume hr bigbang -n bigbang

  • Review Mattermost Enterprise trial license status & follow these steps if expired:

    To "renew" mattermost enterprise trial license, connect to RDS postgres DB using psql (get command and auth from Ryan/Micah/Branden)

    \c mattermost
select * from "public"."licenses";
delete from "public"."licenses";
\q
kubectl delete mattermost mattermost -n mattermost

  • If Flux has been updated in the latest release, checkout your release branch on the BB repo and run ./scripts/install_flux.sh -s (the -s option will reuse the existing secret so you don't have to provide credentials)

  • Update bigbang/base/kustomization.yaml & bigbang/prod/kustomization.yaml with release branch.

  • Verify cluster has updated to the new release

    • Packages have fetched the new revision and match the new release

    • Packages have reconciled

      # check release
watch kubectl get gitrepositories,kustomizations,hr,po -A

      If flux has not updated after ten minutes:

      flux reconcile hr -n bigbang bigbang --with-source

      If flux is still not updating, delete the flux source controller:

      kubectl get all -n flux-system 
kubectl delete pod/source-controller-xxxxxxxx-xxxxx -n flux-system

      If the helm release shows max retries exhausted, you will need to delete helm release secrets and reconcile in flux as follows:

      $ kubectl get secrets -n bigbang | grep ${HR-NAME}
      sh.helm.release.v1.${HR-NAME}-${HR-NAME}.v1                                       helm.sh/release.v1                    1      18h
sh.helm.release.v1.${HR-NAME}-${HR-NAME}.v2                                       helm.sh/release.v1                    1      17h
sh.helm.release.v1.${HR-NAME}-${HR-NAME}.v3                                       helm.sh/release.v1                    1      17m
      # Delete the latest one:
$ kubectl delete secret -n bigbang sh.helm.release.v1.${HR-NAME}-${HR-NAME}.v3

# Then need to run flux stuff:
$ flux suspend hr -n bigbang bigbang
$ flux resume hr -n bigbang bigbang
$ flux reconcile hr ${HR-NAME} -n bigbang --with-source

Confirm app UIs are loading

❕Important
When verifying each application UI is loading, also verify the website certificates are valid.

Logging

  • Login to kibana with SSO
  • Kibana is actively indexing/logging.

Monitoring

  • Login to grafana with SSO
  • Contains Kubernetes Dashboards and metrics
  • contains Istio dashboards
  • Login to prometheus
  • All apps are being scraped, no errors

Cluster Auditor

  • Login to grafana with SSO
  • OPA Violations dashboard is present and shows violations in namespaces (check gitlab-runners ns to validate violations over time)

Kiali

  • Login to kiali with SSO
  • Validate graphs and traces are visible under applications/workloads
  • Validate no errors appear

    ℹ️ Note
    Red notification bell would be visible if there are errors.

GitLab

docker pull alpine
docker tag alpine registry.dogfood.bigbang.dev/<GROUPNAMEHERE>/<PROJECTNAMEHERE>/alpine:latest
docker login registry.dogfood.bigbang.dev
docker push registry.dogfood.bigbang.dev/<GROUPNAMEHERE>/<PROJECTNAMEHERE>/alpine:latest

Sonarqube

  • Login to sonarqube with SSO
  • Add a project for your release
  • Generate a token for the project and copy the token somewhere safe for use later
  • Click other, linux, and copy the projectKey from -Dsonar.projectKey=XXXXXXX for use later
  • After completing the gitlab runner test return to sonar and check that your project now has analysis

ℹ️ Note
The project token and project key are different values.

Gitlab Runner

  • Log back into gitlab and navigate to your project
  • Under settings, CI/CD, variables add two vars:
    • SONAR_HOST_URL set equal to https://sonarqube.dogfood.bigbang.dev/
    • SONAR_TOKEN set equal to the token you copied from Sonarqube earlier (make this masked)
  • Add a .gitlab-ci.yml file to the root of the project, paste in the contents of sample_ci.yaml, replacing -Dsonar.projectKey=XXXXXXX with what you copied earlier
  • Commit, validate the pipeline runs and succeeds (may need to retry if there is a connection error), then return to the last step of the sonar test

Nexus

  • Login to Nexus as admin, password is in the nexus-repository-manager-secret secret:

    # looks like the username but not the pw might be stored with a newline, hence the ^ instead of <-
kubectl get secret nexus-repository-manager-secret -n nexus-repository-manager -o json | jq -r '.data["admin.username"]' | base64 -d ; echo ' ^ admin username'
kubectl get secret nexus-repository-manager-secret -n nexus-repository-manager -o json | jq -r '.data["admin.password"]' | base64 -d ; echo ' <- admin password'

  • Validate there are no errors displaying in the UI

  • Push/pull an image to/from the nexus registry

    • docker login containers.dogfood.bigbang.dev with the credentials from the encrypted values (or the admin user credentials)
    • docker tag alpine:latest containers.dogfood.bigbang.dev/alpine:1-20-0 (replace with your release number, pick a different image to tag if you want)
    • docker push containers.dogfood.bigbang.dev/alpine:1-20-0
    • Pull down the image for the previous release (docker pull containers.dogfood.bigbang.dev/alpine:1-19-0)

Anchore

  • Login to Anchore with SSO

  • Log out and log back in as the admin user - password is in anchore-anchore-engine-admin-pass secret (admin will have pull credentials set up for the registries):

    kubectl get secret anchore-anchore-engine-admin-pass -n anchore -o json | jq -r '.data.ANCHORE_ADMIN_PASSWORD' | base64 -d ; echo ' <- password'

  • Scan image in dogfood registry, registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest

  • Scan image in nexus registry, containers.dogfood.bigbang.dev/alpine:1-19-0 (use your release number)

  • Validate scans complete and Anchore displays data (click the SHA value for each image)

Argocd

  • Login to argocd with SSO

  • Logout and login with username admin. The password is in the argocd-initial-admin-secret secret. If that doesn't work attempt a password reset:

    kubectl -n argocd get secret argocd-initial-admin-secret -o json |  jq '.data|to_entries|map({key, value:.value|@base64d})|from_entries'

  • Create application

    TODO: Test creating application with YAML template.

    apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: podinfo
spec:
destination:
    name: ''
    namespace: podinfo
    server: 'https://kubernetes.default.svc'
source:
    path: chart
    repoURL: 'https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/podinfo.git'
    targetRevision: HEAD
project: default
syncPolicy:
    automated:
    prune: true
    selfHeal: true
    syncOptions:
    - CreateNamespace=true
    *click* create application
application name: podinfo
Project: default
Sync Policy: Automatic
Sync Policy: check both boxes
Sync Options: check "auto-create namespace"
Repository URL: https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/podinfo.git
Revision: HEAD
Path: chart
Cluster URL: https://kubernetes.default.svc
Namespace: podinfo
*click* Create (top of page)

    The app should sync and come healthy.

  • Delete application

Minio

  • Log into the Minio UI - access and secret key are in the minio-root-creds-secret secret

      kubectl -n minio get secret minio-root-creds-secret -o json | jq -r '.data.accesskey' | base64 -d ; echo ' <- access key'
  kubectl -n minio get secret minio-root-creds-secret -o json | jq -r '.data.secretkey' | base64 -d ; echo ' <- secret key'

  • Create bucket

  • Store file to bucket

  • Download file from bucket

  • Delete bucket and files

Mattermost

  • Login to mattermost with SSO

  • Update/modify profile picture

  • Send chats/validate chats from previous releases are visible.

    💡 Tip
    The ability to see chats in other teams requires Mattermost administrator rights.

Twistlock

  • Login to twistlock/prisma cloud with the credentials encrypted in bigbang/prod/environment-bb-secret.enc.yaml

    # from <repo>/bigbang/customers/bigbang project root dir
sops --decrypt environment-bb-secret.enc.yaml | grep -1 twistlock

  • Only complete if Twistlock was upgraded

    • Navigate to Manage -> Defenders -> Deploy
    • 3: twistlock-console
    • 12: On Toggle on "Monitor Istio"
    • 14: Off Disable official registry
    • 15: registry1.dso.mil/ironbank/twistlock/defender/defender:latest
    • 16: private-registry
    • 17: On Deploy Defenders with SELinux Policy
    • 17: On Nodes use Container Runtime Interface (CRI), not Docker
    • 17: On Nodes runs inside containerized environment
    • 18b: download the yaml files
    • Apply the yaml in the dogfood cluster, validate the pods go to running

  • Under Manage -> Defenders -> Manage, make sure # of defenders online is equal to number of nodes on the cluster

  • Under Radars -> Containers, validate pods are shown across all namespaces

Kyverno

  • Test secret sync in new namespace

    # create secret in kyverno NS
kubectl create secret generic \
  -n kyverno kyverno-bbtest-secret \
  --from-literal=username='username' \
  --from-literal=password='password' 
  
# Create Kyverno Policy
kubectl apply -f https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/kyverno/-/raw/main/chart/tests/manifests/sync-secrets.yaml

# Check if secret is create in NEW namespace
kubectl create ns kyverno-test   # wait for 5s for Policy to be ready
kubectl label ns kyverno-test kubernetes.io/metadata.name=kyverno-bbtest --overwrite=true
kubectl get secrets kyverno-bbtest-secret -n kyverno-test  # Test passed if found

- [ ] Delete the test resources
# If successful, delete test resources
kubectl delete -f https://repo1.dso.mil/platform-one/big-bang/apps/sandbox/kyverno/-/raw/main/chart/tests/manifests/sync-secrets.yaml
kubectl delete secret kyverno-bbtest-secret -n kyverno
kubectl delete ns kyverno-test

Velero

  • Backup PVCs velero_test.yaml

    kubectl apply -f ./velero_test.yaml
# exec into velero_test container, check log
veleropod=`kubectl get pod -n velero-test -o json | jq -r '.items[].metadata.name'`
kubectl exec $veleropod -n velero-test -- tail /mnt/velero-test/test.log

    Install the velero CLI on your workstation if you don't already have it (for MacOS, run "brew install velero").

    Then set VERSION to the release you are testing and use the CLI to create a test backup:

    VERSION=1-2-3
velero backup create velero-test-backup-${VERSION} -l app=velero-test
velero backup get

    Wait a bit, re-run velero backup get, when it shows "Completed" delete the app.

    kubectl delete -f ./velero_test.yaml
     namespace "velero-test" deleted
 persistentvolumeclaim "velero-test" deleted
 deployment.apps "velero-test" deleted

  • Restore PVCs

    Now test restoring from backup.

    velero restore create velero-test-restore-${VERSION} --from-backup velero-test-backup-${VERSION}
# exec into velero_test container
kubectl exec $veleropod -n velero-test -- cat /mnt/velero-test/test.log
# Old log entries and new should be in log if backup was done correctly

  • Cleanup test

    kubectl delete -f ./velero_test.yaml

Keycloak

  • Login to Keycloak admin console. The credentials are in the keycloak-credentials secret:

    kubectl get secret keycloak-credentials -n keycloak -o json | jq -r '.data.adminuser' | base64 -d ; echo " <- admin user"
kubectl get secret keycloak-credentials -n keycloak -o json | jq -r '.data.password' | base64 -d ; echo " <- password"

3. Create Release

  • Re-run helm docs in case any package tags changed as a result of issues found in testing.

  • Create release candidate tag based on release branch. Tag EX: 1.8.0-rc.0.

    Message: release candidate
Release Notes: **Leave Blank**

  • Passed tag pipeline.

  • Create release tag based on release branch. Tag EX: 1.8.0.

    Message: release 1.x.x
Release Notes: **Leave Blank**

  • Passed release pipeline.

  • Add release notes to release.

  • Cherry-pick release commit(s) as needed with merge request back to master branch

  • Close Big Bang Milestone in GitLab.

  • Celebrate and announce release

Edited by razzle

UNCLASSIFIED - NO CUI