Release 1.19.0

changed due date to October 22, 2021

added priority4 ~2836 labels

set weight to 3

changed iteration to Big Bang Iterations Oct 19, 2021 - Nov 1, 2021

removed priority4 label

Release note for https://repo1.dso.mil/platform-one/big-bang/bigbang/-/merge_requests/947 :

ArgoCD is now istio injected by default. You will need to do a one-time roll of the argo pods to get the new sidecars added. If you run into any issues with istio injection you can disable it and open issues with Big Bang. To disable it:

addons:
  argocd:
    istio:
      injection: disabled

Similar for https://repo1.dso.mil/platform-one/big-bang/bigbang/-/merge_requests/952

assigned to @micah.nagel

added priority4 teamcore/security labels and removed priority8 label

Upgrade notices for Gitlab upgrade:

Must upgrade to postgres >= 12 BEFORE attempting a BB upgrade of gitlab
Rolling upgrade is required, this might be automatic? TBD on this note

Known issues:

On some k8s distros certain components in the kube-system namespace are unable to be scraped by Prometheus
Prometheus scrapes completed pods, this doesn't affect anything but may see some "Down" targets in Prometheus UI - https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring/-/issues/43

Nexus and Jaeger issues were resolved.

added statusdoing label

changed the description

marked the checklist item Verify that the previous release branch commit hash matches the last release tag. Investigate with previous RE if they do not match as completed

marked the checklist item Create release branch with name. Ex: release-1.9.x as completed

Release Notes

Please see our documentation page for more information on how to consume and deploy BigBang.

Upgrade Notices

Gitlab Upgrade

This Big Bang version contains a Gitlab upgrade with security fixes. If using Gitlab it is strongly advised that you update to 1.19.0 as soon as possible (or update your Gitlab tag to 5.3.1-bb.2 if you don't want to update all packages to their 1.19.0 versions).

You MUST upgrade your Gitlab database to a Postgresql version >= 12 prior to attempting this update. Gitlab 14.x is not supported with Postgresql <= 11. Follow upstream recommendations from your DB provisioner for completing this step. Neither Gitlab or Big Bang has preventative measures if you fail to upgrade your database and we cannot guarantee safety or recovery of your data.

An upgrade to 5.0.5-bb.0 is required before upgrading to 5.3.1-bb.2. An upgrade job has been included that will handle this rolling upgrade but if you wish to do this manually you can override addons.gitlab.git.tag with 5.0.5-bb.0, allow it to upgrade, then remove the override when upgrading to BB 1.19.0. If using the automated upgrade job:

You may notice that Gitlab appears to do nothing for a while after the rolling upgrade pod has started. This is expected as Flux needs to time out once before it will reconcile the 5.0.5-bb.0 version. The default Flux timeout is 10 minutes, but check if you have overridden this.
You may experience Gitlab downtime and/or intermittent service availability as pods roll between versions.
You may see some pods go to CrashLoop/Error states temporarily due to the timing of some of Gitlab's jobs (like migrations).
It is recommended that you test this upgrade in a staging (non-prod) environment if you can so that you have a good understanding of what to expect as the upgrade completes and any potential downtime expectations.

New Istio-Injected Packages

This release includes a couple new istio injected packages, specifically Sonarqube and ArgoCD. The Big Bang development team has tested these packages with injection and found no issues but if you run into problems we have provided an override to disable injection for each of the packages (use this workaround, but please also open an issue with the problem you encountered):

addons:
  argocd:
    istio:
      injection: disabled
  sonarqube:
    istio:
      injection: disabled

Istio Gateway Changes

Istio gateways in the BB Istio package now offer the ability to disable the HTTP redirect via a value:

istio:
  gateways:
    myGatewayName:
      autoHttpRedirect:
        enabled: false

In addition, servers/hosts for each gateway resource are now based on the values under your gateway's hosts value, rather than a * wildcard. If you were previously relying on the wildcard make sure to specify the necessary value for your gateway:

istio:
  gateways:
    myGatewayName:
      hosts:
      - "*.{{ .Values.domain }}"

Resources

As a reminder, Big Bang is implementing resource limits across all packages. We have set general limits but depending on your environment you may need to override these. If you run into pods restarting check for OOMKilled and override/increase memory limits. If you notice general slowness on any package check for CPU throttling (Grafana is your friend here) and update accordingly.

Upgrades from previous releases

If coming from a version pre-1.18 note the additional upgrade notices in any release in between. The BB team doesn't test/guarantee upgrades from anything pre-1.18.

Packages

Package	Type	Package Version	BB Version
Istio Controlplane	Core	`1.11.2`	`1.11.2-bb.1`
Istio Operator	Core	`1.11.2`	`1.11.2-bb.0`
Jaeger	Core	`1.24.0`	`2.23.0-bb.4`
Kiali	Core	`1.40.1`	`1.40.1-bb.0`
Cluster Auditor	Core	`0.3.2`	`0.3.0-bb.7`
OPA Gatekeeper	Core	`3.6.0`	`3.6.0-bb.2`
Elasticsearch Kibana	Core	E: `7.13.4` K: `7.12.0`	`0.1.21-bb.2`
ECK Operator	Core	`1.7.1`	`1.7.1-bb.0`
Fluentbit	Core	`1.8.6`	`0.16.6-bb.0`
Monitoring	Core	G: `7.5.2`, P: `2.25.0`, A: `0.21.0`	`14.0.0-bb.11`
Twistlock	Core	`21.04.439`	`0.0.9-bb.1`
Argocd	Addon	`2.0.1` (w/ p1 plugins)	`3.6.8-bb.10`
Authservice	Addon	`0.4.0`	`0.4.0-bb.17`
MinIO Operator	Addon	`4.2.3`	`4.2.3-bb.2`
MinIO	Addon	`RELEASE.2021-08-31T05-46-54Z`	`4.2.3-bb.5`
Gitlab	Addon	`14.3.1`	`5.3.1-bb.2`
Gitlab Runners	Addon	`14.3.1`	`0.33.1-bb.2`
Nexus	Addon	`3.34.1-01`	`34.1.0-bb.2`
SonarQube	Addon	`8.9` (w/ p1 plugins)	`9.6.3-bb.8`
Anchore	Addon	ENG: `0.10.2`, ENT: `3.1.2`	`1.14.7-bb.1`
Mattermost Operator	Addon	`1.15.0`	`1.15.0-bb.0`
Mattermost	Addon	`5.39.0`	`0.2.2-bb.0`
Velero	Addon	`1.6.3`	`2.23.6-bb.2`
Keycloak	Addon	`14.0.0`	`11.0.1-bb.8`

Changes in 1.19.0

Big Bang

!996: Add retries to CI stages

Istio Controlplane

!956: Better Istio/Authservice Conditional
!968: HTTP -> HTTPS redirects and hosts usage

Jaeger

!957: Add auth policy for Kiali -> Jaeger connection

Kiali

!965: Update Kiali to 1.40.1

OPA Gatekeeper

989: Disable imageDigest policy by default

Elasticsearch/Kibana

!992: Lower default CPU limits for Elastic
!975: Helm test/CI job changes

ECK Operator

!972: ECK Operator 1.7.1 Update

Twistlock

!1007: CODEOWNERS + API Version for VirtualService

ArgoCD

!947: Enable Istio injection on ArgoCD namespace

Minio

!1004: Readme update
!1015: Minio network policy fix for monitoring

Gitlab

!990: Gitlab upgrade to 14.3.1
!1002: Gitlab readme + rolling upgrade job change

Gitlab Runners

!1000: Gitlab Runner update to 14.3.1

Nexus

!994: Nexus FIPS fix + update to 3.34.1
!1019: Nexus values fix for release
!1025: Fix extraLabels for Nexus
!1027: Fix caCert secret namespace for Nexus

Sonarqube

!952: Enable Istio injection on Sonarqube namespace
!997: Sonar istio injection changes
!1008: Sonar job disable injection

Anchore

!969: Anchore readOnlyRootFileSystem for dev psql deployment

Velero

!974: Add volumesnapshotclass to Velero for CSI plugin

Keycloak

!959: Helm test/CI job changes

Known Issues

On some k8s distros certain components in the kube-system namespace are unable to be scraped by Prometheus
Prometheus scrapes completed pods, this doesn't affect anything but may see some "Down" targets in Prometheus UI - https://repo1.dso.mil/platform-one/big-bang/apps/core/monitoring/-/issues/43
Istio dashboards in Grafana missing data - https://repo1.dso.mil/platform-one/big-bang/bigbang/-/issues/878
Jaeger operator metrics unable to be scraped by Prometheus - https://repo1.dso.mil/platform-one/big-bang/bigbang/-/issues/893

Helpful Links

As always, we welcome and appreciate feedback from our community of users. Please feel free to:

Open issues here
Join our chat
Check out the documentation for guidance on how to get started

Future

Don't see your feature and/or bug fix? Check out our roadmap for estimates on when you can expect things to drop, and as always, feel free to comment or create issues if you have questions, comments, or concerns.

marked the checklist item Build draft release notes, see release_notes_template.md as completed

marked the checklist item Bump self-reference version in base/gitrepository.yaml as completed

marked the checklist item Update chart release version chart/Chart.yaml as completed

marked the checklist item Bump badge at the top of README.md as completed

marked the checklist item Update /Packages.md with any new Packages as completed

marked the checklist item Update README.md using helm-docs. Overwrite the existing readme file. as completed

marked the checklist item Update CHANGELOG.md with links to MRs and any upgrade notices/known issues. release-diff update link for release as completed

marked the checklist item Release specific code changes. Make the following changes in a single commit so it can be cherry picked into master later. as completed

marked the checklist item Connect to Cluster as completed

marked the checklist item Update bigbang/base/kustomization.yaml & bigbang/prod/kustomization.yaml with release branch. as completed

marked the checklist item Packages have fetched the new revision and match the new release as completed

marked the checklist item Packages have reconciled as completed

marked the checklist item Verify cluster has updated to the new release as completed

marked the checklist item anchore as completed

marked the checklist item argocd as completed

marked the checklist item gitlab as completed

marked the checklist item tracing as completed

marked the checklist item kiali as completed

marked the checklist item kibana as completed

marked the checklist item mattermost as completed

marked the checklist item minio as completed

marked the checklist item alertmanager as completed

marked the checklist item grafana as completed

marked the checklist item prometheus as completed

marked the checklist item sonarqube as completed

marked the checklist item twistlock as completed

marked the checklist item nexus as completed

marked the checklist item TLS/SSL certs are valid as completed

marked the checklist item Login to kibana with SSO as completed

marked the checklist item Kibana is actively indexing/logging. as completed