# Semi-Automated Release Process > ✋ **STOP**: Is this your first release, or has it been a while since your last release? Before proceeding, review the [Getting Started Guide](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/getting-started-guide.md) for instructions on configuring your environment for the release process, expectations of a release engineer, and information about the release process. ## Forward ‼️ This upgrade process is applicable only to **minor** Big Bang version upgrades of Big Bang major version `3`. It is subject to change in future major versions. For patch version upgrades, refer to the [Patch Release README](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/patch-release-readme.md). - The release **branch** format is as follows: `release-<major>.<minor>.0` where `<minor>` is the minor release number. Example: `release-2.7.0`. - The release **tag** format for major version `3` of Big Bang is: `3.<minor>.0`. ❌ **DO NOT** blindly copy and paste. Carefully read and evaluate the commands below and execute them, if applicable. 📌 As you progress, document pain points and unclear or out-of-date steps. Once the release is complete, submit an MR to update this documentation or post feedback in the Release Engineering mattermost channel. ### Useful Links - [Getting Started Guide](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/getting-started-guide.md) - [Manual Release README](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/manual-release-readme.md) - The semi-automated release process relies on a suite of tools that capture most of the process in code. However, tools occasionally fail and processes may occasionally need to be performed manually. - [Patch Release README](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/patch-release-readme.md) ## 1. Release Prep > 📌 **NOTE**: Ensure your Docker engine is running since several steps use docker commands - [x] Pull and build the latest version of `gogru` and update your config ```shell cd your/local/gogru/repository git checkout main git pull go build ./gogru config [init | update] ``` - [x] Initialize the release by creating the release issue, validating configuration, and verifying the release cluster and licenses. > 📌 **NOTE**: Be sure to `aws sso login` or the `log-in-tool` prior to running these commands ```shell ./gogru release-init ``` The release-init command performs the following checks automatically: - Validates your current gogru configuration - Verifies the Release cluster is healthy - Checks and renews ElasticSearch (ECK) licenses - Creates the release issue in the Big Bang Umbrella package - If the ElasticSearch license was updated, it's a good idea to log in **with SSO** and confirm functionality. (Log into [Kibana](https://kibana.release.bigbang.mil)) - It is unlikely, but it may be necessary to [reconfigure role mapping](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release#kibana) if you are unable to log into Kibana with SSO. - If there are any issues with the Helm Releases or Pods in the output, investigate and fix them before continuing - If `fluent-bit` pods are failing, the root cause is often Elasticsearch data pods hitting disk capacity, which triggers a flood-stage write block on logstash indices. Follow the steps below to diagnose and recover. <details> <summary><b>Cleanup ES data pods &amp; restart Fluent Bit</b></summary> **Thresholds to know:** | Watermark | Threshold | Effect | |---|---|---| | `low` | 85% | ES stops allocating new shards to this node | | `high` | 90% | ES starts relocating shards away from this node | | `flood_stage` | 95% | ES sets all indices to `read_only_allow_delete`, blocking Fluent Bit writes | **1. Get the Elasticsearch password** ```shell PASSWORD=$(kubectl get secret logging-ek-es-elastic-user -n logging -o jsonpath='{.data.elastic}' | base64 -d) ``` **2. Check disk usage on all ES data pods** — act before any node hits 95% ```shell for i in 0 1 2 3 4; do echo "=== data-$i ==="; kubectl exec -n logging logging-ek-es-data-$i -- df -h /usr/share/elasticsearch/data 2>/dev/null; done ``` **3. Check for flood-stage blocks on indices** ```shell kubectl exec -n logging logging-ek-es-data-0 -- curl -sk -u "elastic:$PASSWORD" \ "https://localhost:9200/*/_settings/index.blocks.read_only_allow_delete?pretty" | grep -v "null" ``` **4. If a flood-stage block is present — temporarily raise the watermark** ```shell kubectl exec -n logging logging-ek-es-data-0 -- curl -sk -u "elastic:$PASSWORD" \ -X PUT "https://localhost:9200/_cluster/settings" \ -H "Content-Type: application/json" \ -d '{"transient":{"cluster.routing.allocation.disk.watermark.flood_stage":"97%"}}' ``` **5. Remove the read-only block from all logstash indices** ```shell kubectl exec -n logging logging-ek-es-data-0 -- curl -sk -u "elastic:$PASSWORD" \ -X PUT "https://localhost:9200/logstash-*/_settings" \ -H "Content-Type: application/json" \ -d '{"index.blocks.read_only_allow_delete": null}' ``` **6. List logstash indices by size to find candidates to delete** ```shell kubectl exec -n logging logging-ek-es-data-0 -- curl -sk -u "elastic:$PASSWORD" \ "https://localhost:9200/_cat/indices/logstash-*?v&s=index&h=index,health,pri.store.size" ``` **7. Delete old indices** — replace index names with the oldest ones from the previous step ```shell kubectl exec -n logging logging-ek-es-data-0 -- curl -sk -u "elastic:$PASSWORD" \ -X DELETE "https://localhost:9200/logstash-YYYY.MM.DD,logstash-YYYY.MM.DD" ``` **8. Revert the flood-stage watermark back to default** ```shell kubectl exec -n logging logging-ek-es-data-0 -- curl -sk -u "elastic:$PASSWORD" \ -X PUT "https://localhost:9200/_cluster/settings" \ -H "Content-Type: application/json" \ -d '{"transient":{"cluster.routing.allocation.disk.watermark.flood_stage": null}}' ``` **9. Clear stuck Fluent Bit retry chunks and restart** ```shell for pod in $(kubectl get pods -n logging -l app.kubernetes.io/name=fluent-bit -o name); do kubectl exec -n logging $pod -- sh -c 'rm -rf /var/log/flb-storage/*' done kubectl rollout restart daemonset/fluentbit-fluent-bit -n logging kubectl rollout status daemonset/fluentbit-fluent-bit -n logging ``` </details> > 🛑 **STOP**: Before creating the release branch, reach out to the anchors in the Release Engineering mattermost channel and confirm all necessary changes have been merged into the Big Bang master branch. This *should* be complete by 12p EST Tuesday barring extenuating circumstances. - [x] Create the release branch ```shell ./gogru create-rb ``` - [x] Build the release notes and ***carefully read all upgrade notices before proceeding*** - Before running the generate command, take a look at the recent MRs into Big Bang and make sure the milestone is set properly. This will ensure the generation command doesn't miss anything. - Leave the resultant MR in draft. It will be updated later. ```shell ./gogru generate-release-notes --commit ``` ## 2. Upgrade and Debug Cluster > ⚠️ **WARNING**: Upgrade only, **never** delete and redeploy. - [x] Upgrade the release by creating the release branch, updating version references, updating Flux if needed, and upgrading the release cluster. ```shell ./gogru release-upgrade ``` The release-upgrade command performs the following tasks automatically: - Creates release branches for bbsil-release (deployment) and bigbang (product), and verifies the branch hash matches master. - Updates Big Bang product version references with the new release version. - Checks for a Flux upgrade and updates Flux on the cluster if necessary. - Upgrades the release cluster and opens a merge request into bbsil-release. Post-merge actions: - Ping package maintainers in the Release Engineering channel to approve and merge the MR. - Once merged, Flux will pick up the changes and reconcile the Helm releases. This process may take up to an hour depending on release size. - Monitor the progress of the release by checking HelmReleases, GitRepositories, Pods, and Kustomizations: - [x] Verify cluster has updated to the new release and verify that there are no outstanding issues with Pods or Helm Releases ```shell ./gogru release-cluster-check ``` ### Common Upgrade Issues - If `Prometheus` does not start and is logging an error relating to Vault permissions, follow [these instructions](https://repo1.dso.mil/big-bang/product/packages/vault/-/blob/main/docs/PROMETHEUS.md#permission-denied-trying-to-re-authenticate-after-upgrade) to reissue the Vault token. - If a package fails to upgrade, or fails a test, follow the [Package Rejection Process](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/reject-package-readme.md) to rollback the package upgrade. ### Uncommon Upgrade Issues - If the upgrade fails catastrophically and the release needs to be rolled back, follow the [rollback instructions](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/rollback-release.md?ref_type=heads). This should be used only as a last resort to re-test the upgrade process. ## 3. UI Tests 📌 Utilize the **[Default Application Credentials](https://repo1.dso.mil/big-bang/bigbang/-/blob/master/docs/configuration/default-credentials.md)** when accessing services without SSO ‼️ Refer to the **[Manual Test Steps](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/manual-release-readme.md)** to troubleshoot failing tests. ⚠️ Follow the **[Package Rejection Process](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/docs/reject-package-readme.md)** to rollback package upgrades for failing packages. ### SSO Integration - [ ] *R2-D2*: `Native SSO Grafana, ArgoCD, Anchore and Mattermost Test` - [ ] *R2-D2*: `authservice SSO Prometheus and AlertManager Test` **OR** - [x] [ArgoCD](https://argocd.release.bigbang.mil) - [x] [Anchore](https://anchore.release.bigbang.mil) - [x] [Mattermost](https://chat.release.bigbang.mil) - Grafana & Prometheus tested below ### Monitoring ***AlertManager*** - [x] Login to [alertmanager](https://alertmanager.release.bigbang.mil) with SSO - [x] Validate that the watchdog alert at minimum is firing ***Grafana*** - [x] Login to [Grafana](https://grafana.release.bigbang.mil) with SSO - [x] Click on the drop-down menu in the upper-left and choose Dashboards. - [x] Verify `CoreDNS` is listed and contains data - [x] Verify that several Kubernetes and Istio dashboards are listed. - [x] Click on a few dashboards and verify that metrics are populating. ***Prometheus*** - [x] Login to [Prometheus](https://prometheus.release.bigbang.mil) with SSO - [x] Go to Status > Targets and confirm that no unexpected services are marked as unhealthy - **Known Unhealthy Targets** - `serviceMonitor/monitoring/fluentbit-fluent-bit/0` - `serviceMonitor/monitoring/monitoring-monitoring-kube-operator/0` > ‼️ **TROUBLESHOOTING**: If **Vault** shows up as unhealthy, that may indicate it was rebuilt and the permissions on the `vault/secrets/token` file within the prometheus pod need to be updated > <details><summary><b>Update Vault Token Permissions</b></summary> > > - `exec` into the `vault-agent` container on the `prometheus-monitoring-monitoring-kube-prometheus-0` pod > - `chmod` the vault token > > ```shell > chmod 644 /vault/secrets/token > ``` > > </details> ### Logging - [x] Login to [kibana](https://kibana.release.bigbang.mil) with SSO. - [x] Expand the drop-down menu to the right of `Data View` and click `+ Create data view` - [x] In the `Index pattern` field, enter `logstash*` - [x] Verify at least one logstash source dated within the last week appears beneath *All Sources* - [x] Set the `Timestamp` field to `---I don't want to use the time filter---` - [x] Click `Use without saving`. Log data should populate. ‼️ **TROUBLESHOOTING**: For `You do not have permission to access the requested page` errors, run the `eck-license-check` as described in [Release Prep](#1-release-prep), above. ‼️ **NOTE:**: if `eck-license-check` is failing. most likely you need to update the role mappings with the instructions [here](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release#kibana) ‼️ **TROUBLESHOOTING**: if kibana is not showing any indices, delete the `fluentbit` pods and they should regenerate and start sending `logstash*` data to kibana ### Kiali - [x] Login to [kiali](https://kiali.release.bigbang.mil) with SSO - Validate graphs and traces are visible under applications/workloads - [x] Graphs [at the overview](https://kiali.release.bigbang.mil/kiali/console/namespaces/kiali/workloads/kiali?tab=info&duration=60&refresh=60000) - [x] Graphs [for inbound metrics](https://kiali.release.bigbang.mil/kiali/console/namespaces/kiali/workloads/kiali?tab=in_metrics&rangeDuration=600&refresh=60000) - [x] Graphs [for outbound metrics](https://kiali.release.bigbang.mil/kiali/console/namespaces/kiali/workloads/kiali?tab=out_metrics) - [x] Graphs [for Traces](https://kiali.release.bigbang.mil/kiali/console/namespaces/kiali/workloads/kiali?tab=traces) - [x] Validate no errors appear > 📌 **NOTE**: Red notification **bell** would be visible if there are errors. Errors on individual application listings for labels, etc are expected and okay. ### Twistlock - [x] Log in to [Prometheus](https://prometheus.release.bigbang.mil/targets?search=twistlock) - In the Targets drop down choose `serviceMonitor/twistlock/twistlock-console/0` - Click the small "show more" button that appears. You should see a State of UP and no errors. - [x] *R2-D2*: `Twistlock Test` ### Fortify - [x] *R2-D2*: `Fortify Test` ### GitLab, Gitlab Runner, & Sonarqube ***Gitlab*** - [x] Login to [gitlab](https://gitlab.release.bigbang.mil) with SSO - [x] Edit profile and change user avatar. (Avatar may take several minutes to update.) - [x] Go to `Security > Access Token` and create a personal access token. Select all scopes. - Record the token and the token name to provide during automated test ***Gitlab Runner*** - [x] *R2-D2*: `Gitlab Test` > ‼️ **TROUBLESHOOTING**: The pipeline may fail if Sonarqube is getting a major version update. Navigate to '<http://sonarqube.release.bigbang.mil/setup>' and upgrade it before re-running the test ***Sonarqube*** - [x] Login to [sonarqube](https://sonarqube.release.bigbang.mil) with SSO > ‼️ **TROUBLESHOOTING**: You will be unable to login if Sonarqube is getting a major version update. Navigate to '<http://sonarqube.release.bigbang.mil/setup>' and upgrade it before proceeding - [x] Verify your Project says `PASSED` ### Minio - [x] *R2-D2*: `Minio Test` ### Mattermost - [x] *R2-D2*: `Mattermost Test` ### ArgoCD - [x] *R2-D2*: `ArgoCD Test` > 📌 **NOTE**: This test is failing at the moment, and will be resolved with this [ISSUE](https://repo1.dso.mil/big-bang/team/tools/R2-D2/-/work_items/126). review the cypress video and as long as the error was that the podinfo app was just continually `progressing` this is fine to move on. ### Kyverno - [x] *R2-D2*: `Kyverno Test`. ### Velero - [x] *R2-D2*: `Velero Test` - This test depends on the [registry.release.bigbang.mil/velero-test/velero-test/alpine](https://gitlab.release.bigbang.mil/groups/velero-test/-/container_registries) container. If it no longer exists/is accessible, recreate it following the instructions in the [Velero initial image repository configuration](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/manual-release-readme.md#velero). ### Loki - [x] *R2-D2*: `Loki Test` > 📌 **NOTE**: This test is failing at the moment, and will be resolved with this [ISSUE](https://repo1.dso.mil/big-bang/team/tools/R2-D2/-/work_items/125). review the cypress video and if the test fails to find data on the `loki quick search dashboard`, follow the manual instructions below: > <details><summary><b>Loki Dashboard - manual steps</b></summary> > > - Login to [grafana](https://grafana.release.bigbang.mil) as admin. User name "admin". Retrieve password with `sops -d bigbang/prod2/environment-bb-secret.enc.yaml | sed 's/\\n/\'$\'\n/g' | grep grafana -A 2` > - Click on the drop-down on upper-left, choose Dashboards, then `Loki Dashboard quick search`. > - type `istio-system` in the `namespace` field and hit `enter` > - Verify that this dashboard shows data/logs from the past few minutes and hours. </details> ### bbctl - [x] Login to [grafana](https://grafana.release.bigbang.mil) as admin. User name "admin". Retrieve password with `cd into/my/release/folder/` and then `sops -d overlays/release/secrets/release-bb-secret.enc.yaml | sed 's/\\n/\'$\'\n/g' | grep grafana -A 2` - [x] Click on the drop-down on upper-left, choose Dashboards, then search for `bbctl`. Ensure that all 6 dashboards are present in the search results. - [x] Click into the `bbctl-all-logs` dashboard and confirm tables tables are populated. > ‼️ **TROUBLESHOOTING**: If any of the tables are missing data, you can run the corresponding command to trigger the job to run immediately instead of waiting for the next CRON schedule trigger. > If there is an error in the `bbctl-all-logs` table for a command or data still isn't populating after manually running the commands, reach out to the bbctl team and create a bug ticket. > ‼️ **KNOWN ISSUE**: the violations table not populating is a known issue when the output is too large. You should see a red triangle in the top corner saying it timed out. > <details><summary><b>Manually Trigger bbctl CRON Jobs</b></summary> > > - Preflight Check Logs > > ```shell > kubectl create job --from=cronjob/bbctl-bbctl-bigbang-preflight bbctl-bigbang-preflight -n bbctl > ``` > > - Versions Logs > > ```shell > kubectl create job --from=cronjob/bbctl-bbctl-bigbang-updater bbctl-bigbang-updater -n bbctl > ``` > > - Policy Logs > > ```shell > kubectl create job --from=cronjob/bbctl-bbctl-bigbang-policy bbctl-bigbang-policy -n bbctl > ``` > > - Status Logs > > ```shell > kubectl create job --from=cronjob/bbctl-bbctl-bigbang-status bbctl-bigbang-status -n bbctl > ``` > > - Violations Logs > > ```shell > kubectl create job --from=cronjob/bbctl-bbctl-bigbang-violations bbctl-bigbang-violations -n bbctl > ``` > > </details> - [x] Click into the other 5 bbctl dashboards and ensure they all show data in the graphs. - [x] bbctl-preflight-dashboard - [x] bbctl-version-dashboard - [x] bbctl-policies-dashboard - [x] bbctl-status-dashboard - [ ] bbctl-violations-dashboard ### Tempo - [x] *R2-D2*: `Tempo Test` > 📌 **NOTE**: This test is obsolete at the moment as we do not currently have any data tethered to tempo ### Keycloak - [x] *R2-D2*: `Keycloak Test` ### Neuvector - [x] *R2-D2*: `Neuvector Test` > 📌 **NOTE**: This test might fail at the moment, try re-running and if continues to fail proceed with the **[Manual Test Steps](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/blob/main/manual-release-readme.md)** ### Anchore > 🛑 **STOP**: This test is dependent on artifacts created during the [GitLab Runner](#gitlab-gitlab-runner--sonarqube) push. Confirm that test completed successfully before proceeding. - [x] Log into anchore as the admin user - password is in `anchore-enterprise-anchore-enterprise` secret (admin will have pull credentials set up for the registries): ```shell kubectl get secret anchore-enterprise-anchore-enterprise -n anchore -o json | jq -r '.data.ANCHORE_ADMIN_PASSWORD' | base64 -d ``` - [x] Scan image in release registry, `registry.release.bigbang.mil/GROUPNAMEHERE/PROJECTNAMEHERE/alpine:latest` - [x] Go to Images Page, Analyze Tag - Registry: `registry.release.bigbang.mil` - Repository: `GROUPNAME/PROJECTNAME/alpine` - Tag: `latest` - [x] Validate scans complete and Anchore displays data (click the SHA value for each image) ### Mimir - [x] Login to [grafana](https://grafana.release.bigbang.mil) as admin. User name "admin". Retrieve password with `cd into/my/release/folder/` and then `sops -d overlays/release/secrets/release-bb-secret.enc.yaml | sed s/\\n/'$'\n''/g | grep grafana -A 2` (assumes gnu-sed) - [x] Click on the drop-down in the upper-left corner again and choose Data Sources, then choose Mimir. Scroll down and click `Save & Test`. A message should appear that reads "Data source successfully connected." - [x] Click on the drop-down on upper-left, choose Dashboards, then `Prometheus / Remote Write`. Verify that this dashboard shows data/logs from the past few minutes and hours. ### Backstage - [x] Log in to [backstage](https://backstage.release.bigbang.mil/) via SSO - Navigate to `kyverno` component page and validate: - [x] Dashboards iFrame is populated (`ClusterPolicyReport Details`, `PolicyReport Details`, `PolicyReports`) and links redirect to `grafana.release.bigbang.mil` when clicked - [x] All of the header links work (`CI/CD`, `Kubernetes`, `API`, `Dependencies`, `Docs`), though they may not yet be populated with data - Navigate to `monitoring` component page and validate: - [x] Dashboards iFrame is populated (`Alertmanager / Overview`, `Prometheus / Overview`, `Prometheus / Remote Write`) and links redirect to `grafana.release.bigbang.mil` when clicked - [x] Links iFrame is populated (`Monitoring`, `AlertManager`, `Prometheus`) and links redirect to their respective service when clicked - [x] All of the header links work (`CI/CD`, `Kubernetes`, `API`, `Dependencies`, `Docs`), though they may not yet be populated with data - Not yet implemented - [ISSUE](https://repo1.dso.mil/big-bang/team/deployments/bigbang-sil/-/issues/190) - [ ] Verify components added via component.yaml are present ### Headlamp - [x] Log in to [headlamp](https://headlamp.release.bigbang.mil/) with SSO and confirm the landing page loads. 📌 **NOTE**: This currently immediately logs you out of headlamp due to [THIS](https://github.com/kubernetes-sigs/headlamp/issues/2848) upstream issue. - Note, data **will not** populate until [this issue](https://repo1.dso.mil/big-bang/team/deployments/bigbang-sil/-/issues/191) is resolved - [x] Log out and back in, selecting the `Use A Token` option - Generate a temporary auth token: ```shell kubectl create token headlamp-headlamp -n headlamp ``` - [x] Confirm the `Cluster` page is populated with graphs and events - [x] Confirm the Flux plugin is functioning. Select `Flux` in the left sidebar and verify that `Overview`, `Kustomizations`, `HelmReleases`, `Sources`, and `Flux Runtime` are populated with data - [x] Click through a few of the other menu items in the left sidebar and confirm data is populating (`Gateway (beta)` errors are okay) ## 4. Create Release > 🛑 **STOP**: Before finalizing the release, confirm that the latest [bb-docs-compiler pipeline](https://repo1.dso.mil/big-bang/team/tools/bb-docs-compiler/-/pipelines) passed. If not, identify, correct, and cherry-pick a fix into the release branch. This will prevent wasting time re-running pipelines if the docs compiler pipeline fails. ### Prep - [x] Re-generate the version references, helm docs, and release notes to capture any changes implemented during testing. ```shell ./gogru update-version-references ``` ```shell ./gogru generate-release-notes --commit ``` ### Clean up release notes - [x] Check to see if any packages have been added or graduated from BETA. - There is no single resource that lists this information, but you should generally be aware of packages that have been added or moved out of BETA from discussions with other team members during your workdays. You can also ask the Maintainers [^maintainers] if unsure. - [x] Add **Decprecations** (if any) - [x] Add **Upgrade Notices** - Issues encountered during testing often result in upgrade notices - Package MRs might be missed by the bot if the notice includes `## Header Tags` - [x] Review MRs improperly sorted under 'Big Bang MRs' into their specific sections below. - If MR is for Big Bang (docs, CI, etc) and not a specific package, they can remain in place under 'Big Bang MRs'. - *General rule of thumb:* If the git tag changes for a package, put the MR under that package, and if there is no git tag change for a package, put it under BB. - [x] Adjust known issues as needed: - If an issue has been resolved it can be removed and if any new issues were discovered they should be added. - [x] Verify table contains all chart upgrades mentioned in the MR list below. - [x] Verify table contains all application versions. - Compare with previous release notes. If a package was upgraded there will usually be a bump in application version and chart version in the table. - [x] Verify all internal comments are removed from Release Notes, such as comments directing the release engineer to copy/move things around - [x] Any issues encountered during release upgrade testing need to be thoroughly documented in the release notes. If you had to delete a replica set, force restart some pods, delete a PV, please make sure these steps are included in our known issues section of the release notes. - [x] Scroll through every package listed and verify the following 1. it has a link to at least one MR 2. the CHANGELOG entries match the MR listed 3. if more than one MR or CHANGELOG entry for a package exists then verify it makes sense (ie if 2 MRs that there are CHANGELOG entries for both. Conversely, if more than one CHANGELOG entry make sure those entries are tied to the MRs in this release, sometimes changlogs get messed up causing more entries in the release notes than neccessary) - **Publish Release Notes to bbsil-release** - [x] Commit the resulting release notes and push to `<release-branch>` in Big Bang (bbsil-release) - [x] Create an MR for the Codeowners and have it merged before continuing on - [x] Make sure that the Maintainers [^maintainers] are ALSO assigned as reviewers and at least 1 of them reviews/approves before merging ### Create Release Tag - [x] Create release tag based on release branch. ```shell ./gogru create-release-tag ``` ### Confirm All Jobs Pass - [x] Passed release pipeline. - Review all pipeline output looking for failures or warnings. Reach out to the maintainers for a quick review before proceeding. Maintainers [^maintainers] - [x] [Release Notes are present](https://repo1.dso.mil/big-bang/bigbang/-/releases) ### Update Docs - [x] Verify [BB Docs Compiler pipeline](https://repo1.dso.mil/big-bang/team/tools/bb-docs-compiler/-/pipelines) passed. - Review all pipeline output looking for failures or warnings. Reach out to the maintainers for a quick review before proceeding. Maintainers [^maintainers] - [x] once that pipeline complete, Verify that the tag has been mirrored to the [IL2 Big Bang repository](https://code.il2.dso.mil/platform-one/big-bang/docs-bb/bb-docs). > 📌 **NOTE**: You may require access be granted to the IL2 Big Bang repository to view the status. Contact the Maintainers [^maintainers] for assistance. - [x] verify IL2 pipelines, complete successfully [IL2 Big Bang Pipelines](https://code.il2.dso.mil/platform-one/big-bang/docs-bb/bb-docs/-/pipelines) (this will update the https://docs-bigbang.dso.mil/ website) - Ensure that release docs have compiled and appear on [Big Bang Docs](https://docs-bigbang.dso.mil/) once the pipeline is complete. - [x] In [Big Bang Docs](https://docs-bigbang.dso.mil/) select Latest version and go in Packages, make sure that the packages version is updated to match the version specified in the release - [x] Select a couple of packages that were updated in the release and match the version in the [values file](https://repo1.dso.mil/big-bang/bigbang/-/blob/master/chart/values.yaml). - [x] Click around in the docs and make sure docs are properly formatted, pay special attention to lists and tables. ### RELEASE IT - [x] Type up a release announcement for the Maintainers [^maintainers] to post in the Big Bang Value Stream channel. Mention any breaking changes and some of the key updates from this release. Use same format as a prior announcement <https://chat.il2.dso.mil/platform-one/channels/team---big-bang> - [x] Cherry-pick release commit(s) as needed with merge request back to master branch. We do not ever merge release branch back to master. Instead, make a separate branch from master and cherry-pick the release commit(s) into it. Use the resulting MR to close the release issue. ```shell # Navigate to your local clone of the BB repo cd path/to/my/clone/of/bigbang # Get the most up to date master git checkout master git pull # Check out new branch for cherrypicking git checkout -b 2.<minor>-cherrypick # Repeat the following for whichever commits are required from release # Typically this is the initial release commit (bumped GitRepo, Chart.yaml, CHANGELOG, README) and a final commit that re-ran helm-docs (if needed) git cherry-pick <commit sha for Nth commit> git push -u origin 2.<minor>-cherrypick # Create an MR using Gitlab, merging this branch back to master, reach out to maintainers to review ``` - [x] Move all OPEN [Issues](https://repo1.dso.mil/groups/big-bang/-/issues) remaining within the current milestone to the next one. Then close the [Big Bang Milestone](https://repo1.dso.mil/groups/big-bang/-/milestones) in GitLab. (make sure that there are n+2 milestones still, the current one and 1 future milestone, if not contact the maintainers [^maintainers]) ```shell ./gogru closeout-milestone ``` - [x] Manually update the versions in `gitRepo/base/kustomization.yaml` and `helmRepo/base/helmrelease.yaml` in the [Customer Template](https://repo1.dso.mil/big-bang/customers/template) repo, commit create an MR for [^maintainers] to approve. The automation of this task is planned in [this issue](https://repo1.dso.mil/big-bang/team/tools/gogru/-/issues/211) - [x] Handoff the release to the Maintainers [^maintainers], they will review then celebrate and announce the release in the public MM channel - [x] Reach out to @marknelson to let him know that the release is done and to update the Gitlab Banner - 📌 **NOTE**: if release notes need to be edited AFTER a release tag & pipeline have completed. make sure that the following steps are taken - [ ] edit the release manually, [here](https://repo1.dso.mil/big-bang/bigbang/-/releases) (you might need to add yourself to the `Tag Permissions` within the settings to allow the `Edit release` button to appear) - [ ] Create an MR [Here](https://repo1.dso.mil/big-bang/team/deployments/bbsil-release/-/tree/main/docs/release?ref_type=heads) with the edits to the previous release in order to ensure possible `patch releases` get all relevant update notices. [^maintainers]: @jfoster @chris.oconnell @michaelmartin @andrewshoell