mimir update to 5.5.1-bb.6
Package Merge Request
Package Changes
https://repo1.dso.mil/big-bang/product/packages/mimir/-/blob/5.5.1-bb.6/CHANGELOG.md
Package MR
big-bang/product/packages/mimir!49 (merged) and big-bang/product/packages/mimir!54 (merged)
For Issue
Closes big-bang/product/packages/mimir#62 (closed) and big-bang/product/packages/mimir#65 (closed)
Upgrade Notices
- mimir-distributed has deprecated the
nginxservice and related values in favor ofgateway.- All remote_write scraper configurations not included in Big Bang should be updated to point to
mimir-gatewayinstead of nginx.
- All remote_write scraper configurations not included in Big Bang should be updated to point to
Important- The upgrade steps below are applicable only when MinIO is enabled for Mimir. As a reminder, MinIO is only supported for development environments. If your environment is not using Big Bang's MinIO for Mimir, you can ignore the following upgrade notice. If Big Bang's MinIO is enabled for Mimir (minio-tenant.enabled to true), remember to test upgrades and maintain proper backups.
Upgrading from 5.5.1-bb.4 involves changes to Tenant immutable server field that requires 1) the mimir-mimir-minio-tenant tenant to be deleted, 2) remove any finalizers protecting the persistent volumes (pv) that were used by the mimir-mimir-minio-tenant tenant, and 3) delete the persistent volumes and persistent volume claims (pvc) used by the mimir-mimir-minio-tenant tenant to free up the persistent volumes for the new MinIO tenant before upgrading the release. Due to the recreation of the tenant, any historical metric data in the MinIO tenant buckets, pre-upgrade, will be lost as part of the tenant redeployment.
The Mimir upgradeJob.enabled field in values.yaml can deploy a pre-upgrade job that automates the required tenant, pv, and pvc deletion without any additional steps by using a helm pre-upgrade hook, so no additional actions is required. It should be noted, 1) the buckets containing historical metrics will be replaced so historical metrics collected will no longer available and 2) a brief outage is expected during upgrade while the mimir-mimir-minio-tenant tenant is being rolled out.
However, if you would rather manually complete the pre-upgrade steps listed above prior to upgrade, then you would need to set the upgradeJob.enabled=false in the values.yaml and follow steps outlined below. The below command assumes that the Mimir package is deployed in the default Big Bang mimir namespace for Mimir, one should look to confirm the namespace of their Mimir deployment:
1. Deleting the mimir-mimir-minio-tenant Tenant
kubectl delete tenant mimir-mimir-minio-tenant -n mimir2. Remove any Finalizers for the Persistent Volumes that was used by the Tenant
kubectl patch pv <name of the pv used by the Minio Tenant> --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]'3. Delete the Persistent Volumes that was used by the Tenant
kubectl delete pv <name of the pv used by the Minio Tenant>4. Delete the Persistent Volumes Claims that was used by the Tenant
kubectl delete pvc <name of the pvc used by the Minio Tenant> -n mimirOnce the resources have been deleted, you can upgrade the release
Merge request reports
Activity
added botmr mimir statusdoing labels
assigned to @bjacksonfv
requested review from @michaelmartin, @chris.oconnell, and @andrewshoell
Might need to wait on http://repo1.dso.mil/big-bang/bigbang/-/merge_requests/5783?
mentioned in merge request !5783 (closed)
added statusreview label and removed statusdoing label
requested review from @troymobley
Resolved by Michael MartinI'm wondering how much upgrade notice/procedures we really want to include for minio -- since, minio is a dev-only feature--not prod-supported.
Maybe we could add some disclaimer there that these upgrade steps are for dev-only environments and to remember to test upgrades and maintain proper backups. @bjacksonfv @chris.oconnell @andrewshoell
- Last reply by Michael Martin
