Mattermost Mino Pool Validate-Arguments Init Container Error

Bug

Description

We are running into an issue with the mattermost-minio-pool-0-* pods failing on startup with the status: Init:CrashLoopBackOff. From the pod logs we can see there is TLS handshake timeout in the validate-arguments init container:

k logs mattermost-minio-pool-0-3 -c validate-arguments
2025/02/25 21:19:38 validator.go:65: Get "https://10.43.0.1:443/apis/minio.min.io/v2/namespaces/mattermost/tenants/mattermost-minio": net/http: TLS handshake timeout
panic: Get "https://10.43.0.1:443/apis/minio.min.io/v2/namespaces/mattermost/tenants/mattermost-minio": net/http: TLS handshake timeout

goroutine 1 [running]:
github.com/minio/operator/sidecar/pkg/validator.Validate({0x7ffe23632b82, 0x10})
github.com/minio/operator/sidecar/pkg/validator/validator.go:66 +0x699
main.startValidator(0xc000629601?)
github.com/minio/operator/sidecar/cmd/sidecar/validate.go:41 +0x2c
github.com/minio/cli.HandleAction({0x1bbad20?, 0x2111d20?}, 0x8?)
github.com/minio/cli@v1.24.2/app.go:492 +0x70
github.com/minio/cli.Command.Run({{0x1ff9fb1, 0x8}, {0x0, 0x0}, {0x3387360, 0x1, 0x1}, {0x20261c4, 0x25}, {0x0, ...}, ...}, ...)
github.com/minio/cli@v1.24.2/command.go:242 +0x935
github.com/minio/cli.(*App).Run(0xc0004fafc0, {0xc00013e0c0, 0x4, 0x4})
github.com/minio/cli@v1.24.2/app.go:260 +0xa7b
main.main()
github.com/minio/operator/sidecar/cmd/sidecar/main.go:129 +0x5e

We identified this error after upgrading the cluster's control plane and data planes RHEL9 image, as well a a BigBang upgrade from 2.44.0 -> 2.45.0 -> 2.45.1 -> 2.26.0

The error has persisted through both a AMI and Mattermost version rollback. From the mattermost-minio-pool-0-* pods we are unable to get any additional logs and can't tell if the pods are failing trying to connect to the Mattermost app, or what is the underlying issue.

BigBang Version

2.46.0