SKIP_UPGRADE_CHECK: Created upgrade job for upgrading to 5.5.1-bb.6

General MR

Summary

5.5.1-bb.5 will reduce the MinIO tenant servers from 4 to 1. Due to the immutable field for tenant, end users will need to 1) delete the tenant, 2) delete the finalizers for persistent volume that was claimed, and 3) delete the persistent volume and persistent volume claims. This MR will enable add a pre-upgrade job that will handle the 3 steps above for the user if minio.enabled and upgradeJob.enabled is set to true.

Relevant logs/screenshots

N/A

Linked Issue

issue

Upgrade Notices

Important: If minio-tenant is enabled (minio-tenant.enabled to true) in the Mimir package, upgrading from 5.5.1-bb.4 involves changes to immutable server field that requires 1) the mimir-mimir-minio-tenant tenant to be deleted, 2) remove any finalizers protecting the persistent volumes (pv) that were used by the mimir-mimir-minio-tenant tenant, and 3) delete the persistent volumes and persistent volume claims (pvc) used by the mimir-mimir-minio-tenant tenant to free up the persistent volumes for the new MinIO tenant before upgrading the release. Due to the recreation of the tenant, any historical metric data in the MinIO tenant buckets, pre-upgrade, will be lost as part of the tenant redeployment.

The Mimir upgradeJob.enabled field in values.yaml can deploy a pre-upgrade job that automates the required tenant, pv, and pvc deletion without any additional steps by using a helm pre-upgrade hook, so no additional actions is required. It should be noted, 1) the buckets containing historical metrics will be replaced so historical metrics collected will no longer available and 2) a brief outage is expected during upgrade while the mimir-mimir-minio-tenant tenant is being rolled out.

However, if you would rather manually complete the pre-upgrade steps listed above prior to upgrade, then you would need to set the upgradeJob.enabled=false in the values.yaml and follow steps outlined below. The below command assumes that the Mimir package is deployed in the default Big Bang mimir namespace for Mimir, one should look to confirm the namespace of their Mimir deployment:

1. Deleting the mimir-mimir-minio-tenant Tenant

kubectl delete tenant mimir-mimir-minio-tenant -n mimir

2. Remove any Finalizers for the Persistent Volumes that was used by the Tenant

kubectl patch pv <name of the pv used by the Minio Tenant> --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]'

3. Delete the Persistent Volumes that was used by the Tenant

kubectl delete pv <name of the pv used by the Minio Tenant>

4. Delete the Persistent Volumes Claims that was used by the Tenant

kubectl delete pvc <name of the pvc used by the Minio Tenant> -n mimir

Once the resources have been deleted, you can upgrade the release.

chart/templates/bigbang/upgrade/job.yaml

@@ -47,7 +47,7 @@ spec:
               echo "Patching PV: $pv"
               kubectl patch pv "$pv" --type=json -p '[{"op": "remove", "path": "/metadata/finalizers"}]' || {
                   echo "Failed to remove finalizer from PV: $pv";
-                  continue;
+                  exit 1;
               }
               
               # If patching is successful, add PV to the list for deletion later
@@ -60,7 +60,7 @@ spec:
               echo "Deleting PV: $pv"
               kubectl delete pv "$pv" --wait=true || {
                   echo "Failed to delete PV: $pv";
-                  continue;
+                  exit 1;
               }
           done
 
@@ -74,7 +74,7 @@ spec:
               echo "Deleting PVC: $pvc"
               kubectl delete pvc "$pvc" -n mimir --wait=true || {
                   echo "Failed to delete PVC: $pvc";
-                  continue;
+                  exit 1;
               }
           done