Minio 4.1.2 -> 4.2.3 Upgrade Path

Issue found during 1.17 release testing. It appears that upgrading from the previous minio operator + instance version is hit or miss completing.

Typically what we are seeing is the operator deployment will update and the tenant CR will change but there is no change to the statefulsets or pods for minio.

This needs to be resolved to set up a proper upgrade path. Investigate the problem looking at upstream resources, testing on your cluster, etc. Determine a path forward to make the upgrade consistent and implement the fix(es). Some ideas of what this could look like:

• Validate it is not an IB issue, test with upstream images for operator
• Try a different version of the instance, match the upstream version used by the corresponding operator version
• See if its networkPolicy related (disable all NPs)
• Test upgrading the operator first then the instance (i.e. these would be in separate releases)
• Test upgrading the instance first then the operator (^ ditto ^)

If an upgrade path cannot be determined by end of sprint we should do all of the following:

• Revert the tag bumps in master of BB
• Revert the MRs to the individual packages so that their main branches properly track whats in BB
• Continue to plow forward on the upgrade path?