UNCLASSIFIED - NO CUI

Skip to content

Release 1.14.0

1. Release Prep

  • Verify that the previous release branch commit hash matches the last release tag. Investigate with previous RE if they do not match
  • Create release branch with name. Ex: release-1.14.x
  • Build draft release notes, see release_notes_template.md
  • Release specific code changes. Make the following changes in a single commit so it can be cherry picked into master later.

    • Bump self-reference version in base/gitrepository.yaml

    • Update chart release version chart/Chart.yaml

    • Bump badge at the top of README.md

    • Update /Packages.md with any new Packages

    • Update CHANGELOG.md with links to MRs and any upgrade notices/known issues. release-diff update link for release

    • Update README.md using helm-docs. Overwrite the existing readme file.

      # from root dir of your release branch
docker run -v "$(pwd):/helm-docs" -u $(id -u) jnorwood/helm-docs:v1.5.0 -s file -t .gitlab-ci/README.md.gotmpl --dry-run > README.md

2. Test and Validate Release Candidate

Deploy release branch on Dogfood cluster

  • Connect to Cluster
  • Update bigbang/base/kustomization.yaml & bigbang/prod/kustomization.yaml with release branch.
  • Verify cluster has updated to the new release

    • Packages have fetched the new revision and match the new release

    • Packages have reconciled

      # check release
watch kubectl get gitrepositories,kustomizations,hr,po -A
# if flux has not updated after 10 minutes.
flux reconcile hr -n bigbang bigbang --with-source
# if it is still not updating, delete the flux source controller 
kubectl get all -n flux-system 
kubectl delete pod/source-controller-xxxxxxxx-xxxxx -n flux-system

Confirm app UIs are loading

Logging

  • Login to kibana with SSO
  • Kibana is actively indexing/logging.

Cluster Auditor

  • Login to kibana with SSO
  • violations index is present and contains images that aren't from registry1

Monitoring

  • Login to grafana with SSO
  • Contains Kubernetes Dashboards and metrics
  • contains istio dashboards
  • Login to prometheus
  • All apps are being scraped, no errors

Kiali

  • Login to kiali with SSO

Sonarqube

GitLab & Runners

  • Login to gitlab with SSO

  • Create new public group with release name. Example release-1-8-0

  • Create new public project with release name. Example release-1-8-0

  • git clone and git push to new project

  • docker push and docker pull image to registry

    docker pull alpine
docker tag alpine registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest
docker login registry.dogfood.bigbang.dev
docker push registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest

  • Edit profile and change user avatar

  • Test simple CI pipeline. sample_ci.yaml

Anchore

  • Login to anchore with SSO
  • Scan image in dogfood registry, registry.dogfood.bigbang.dev/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest

Argocd

  • Login to argocd with SSO
  • Logout and login with admin. password reset
  • Create application 
    *click* create application
application name: argocd-test
Project: default
Sync Policy: Automatic
Sync Policy: check both boxes
Sync Options: check both boxes
Repository URL: https://github.com/argoproj/argocd-example-apps
Revision: HEAD
Path: helm-guestbook
Cluster URL: https://kubernetes.default.svc
Namespace: argocd-test
*click* Create (top of page)
  • Delete application

Minio

  • Create bucket
  • Store file to bucket
  • Download file from bucket
  • Delete bucket and files

Mattermost

  • Login to mattermost with SSO
  • Elastic integration

Velero

  • Backup PVCs velero_test.yaml

    kubectl apply -f ./velero_test.yaml
# exec into velero_test container
cat /mnt/velero-test/test.log
# take note of log entries and exit exec 
    velero backup create velero-test-backup-1-8-0 -l app=velero-test
velero backup get
kubectl delete -f ./velero_test.yaml
kubectl get pv | grep velero-test
kubectl delete pv INSERT-PV-ID

  • Restore PVCs

    velero restore create velero-test-restore-1-8-0 --from-backup velero-test-backup-1-8-0
# exec into velero_test container
cat /mnt/velero-test/test.log
# old log entires and new should be in log if backup was done correctly

Keycloak

3. Create Release

  • Create release candidate tag based on release branch. Tag EX: 1.14.0-rc.0. 
    Message: release candidate
Release Notes: **Leave Blank**
  • Passed tag pipeline.
  • Create release tag based on release branch. Tag EX: 1.14.0. 
    Message: release 1.x.x
Release Notes: **Leave Blank**
  • Passed release pipeline.
  • Add release notes to release.
  • Cherry-pick release commit(s) as needed with merge request back to master branch
  • Celebrate and announce release

DRAFT RELEASE NOTES

Candidate Release Notes

Please see our documentation page for more information on how to consume and deploy BigBang.

Upgrade Notices

  • This release upgrades Elasticsearch to version 7.13.4 and contains a helm hook to run a job which performs steps to perform a Rolling Upgrade. If the autoRollingUpgrade job does not complete successfully or is interfered with it could cause ECK data loss. No intervention is immediately required but please read the documentation first and keep an eye on the pods as they cycle. It is recommended to leave the autoRollingUpgrade value enabled.
    • Before upgrading to these versions you MUST be on Big Bang version 1.11.0 (at least) to make sure your Logging HelmRelease's default timeout value is 20 minutes and your existing ECK cluster is 7.12.0 otherwise you will most likely encounter issues while pods are cycling.
    • If you encounter issues with the ECK stack with this autoRollingUpgrade review this troubleshooting doc to easily resolve some of the most common issues.
    • If your ECK cluster has more than 8 nodes you should make sure the flux timeout value (logging.flux.timeout) for the logging package is set to a larger value eg: 30 minutes (each node takes around 2-2.5 minutes to restart).

Packages

Pull in and replace chart then update for new release

Package Type Package Version BB Version
Istio Operator Core 1.8.4 1.8.4-bb.2
Istio Controlplane Core 1.8.4 1.8.4-bb.6
Jaeger Core 2.23.0 2.23.0-bb.1
Updated: 1.14.0 Kiali Core 1.37.0 1.37.0-bb.0
Updated: 1.14.0 OPA Gatekeeper Core 3.5.1 3.5.1-bb.8
Monitoring Core G: 7.5.2, P: 2.25.0, A: 0.21.0 14.0.0-bb.3
Updated: 1.14.0 ECK Operator Core 1.6.0 1.6.0-bb.2
Updated: 1.14.0 Elasticsearch Kibana Core 7.13.4 0.1.18-bb.0
Updated: 1.14.0 Fluentbit Core 1.8.1 0.16.1-bb.0
Updated: 1.14.0 Cluster Auditor Core 1.16.0 0.3.0-bb.5
Updated: 1.14.0 Twistlock Core 21.04.412 0.0.6-bb.1
Gitlab Addon 13.10.3 4.10.3-bb.14
Gitlab Runners Addon 13.9.0 0.26.0-bb.3
Mattermost Operator Addon 1.14.0 1.14.0-bb.2
Mattermost Addon 5.36.1 0.1.6-bb.8
Updated: 1.14.0 MinIO Operator Addon 2.0.9 2.0.9-bb.3
MinIO Addon RELEASE.2020-11-19T23-48-16Z 2.0.9-bb.13
Updated: 1.14.0 Authservice Addon 0.4.0 0.4.0-bb.10
Anchore Addon ENG: 0.10.0, ENT: 3.1.0 1.13.0-bb.4
SonarQube Addon 8.9 (w/ p1 plugins) 9.2.6-bb.13
Updated: 1.14.0 Argocd Addon 2.0.1 (w/ p1 plugins) 3.6.8-bb.5
Updated: 1.14.0 Velero Addon 1.6.2 2.23.5-bb.0
Updated: 1.14.0 Keycloak Addon 14.0.0 11.0.1-bb.1

Changes in v1.14.0

BigBang

  • !704: Charter updates
  • !702 Add a "readiness probe" to DIND in CI
  • !724 Fixing SOPS MAC mismatch on CI cert
  • !715 patch release 1.13.0 specific changes
  • !718 Merge template fixes
  • !736 update gitignore

Istio

  • !700 monitoring value passed down to Istio Package to make the networkPolicy create and allow metrics scraping
  • !532 documentation update for Istio ingress

Nexus

  • !698 new Nexus version
  • !701 Fix extralabels in Nexus Package

Monitoring

  • !706 Monitoring update for SecurityContext. Monitoring helmrelease does not achieve healthy state.
  • !708 Monitoring for Updated Job SecurityContext
  • !712 allowedHostFilesystem For Prometheus Exporter

Twistlock

  • !707 Twistlock openshift port 5353 egress fix

Authservice

  • !709 authservice support for OpenShift
  • !728 Authservice codeowners update

Kiali

  • !710 Kiali Populate Tracing/Grafana Links
  • !721 Kiali Update to 1.37.0

Minio

  • !713 Bumping minio chart version to consume networkPolicy istio labels

Gatekeeper

  • !714 Update Gatekeeper values
  • !722 OPA Gatekeeper update to reduce memory footprint and bug fixes
  • !730 Use OPA Gatekeeper with kube-system namespace exception

Logging

  • !716 Increment elasticsearch-kibana chart version to allow openshift DNS on udp/5353
  • !732 Logging: set limits = requests
  • !711 update ECK operator
  • !738 upgrade elasticsearch to version 7.13.4

Fluentbit

  • !725 Fluentbit update
  • !731 Update Fluentbit to 1.8.1

Gitlab

  • !719 Set multipart_chunk_size_mb=128 in gitlab S3 backup toml

Keycloak

  • !723 add keycloak image to synker config
  • !691 Keycloak support for Openshift toggle

Valero

  • !727 Update Velero to 1.6.2

Cluster Auditor

  • !733 update cluster auditor

ArgoCD

  • !743 Fix ArgoCD redis endpoint.

Documentation

  • !734 documentation and script update for SOPS

Known Issues

  • #38: Constraint violations are not always properly logged and will not appear in Kibana.
  • #657: Intermittent issue with shards value in monitoring package versions based on 14.0.0.
    To resolve this issue if you encounter it, add the following values to your monitoring package: 
    prometheus:
  prometheusSpec:
    shards: 1

Helpful Links

As always, we welcome and appreciate feedback from our community of users. Please feel free to:

Future

Don't see your feature and/or bug fix? Check out our roadmap for estimates on when you can expect things to drop, and as always, feel free to comment or create issues if you have questions, comments, or concerns.

Edited by kevin.wilder

UNCLASSIFIED - NO CUI