Thanos MinIO Tenant Connection with MinIO Operator
When deploying thanos minio tenant for object storage configuration (can use the following override values below), the thanos minio tenant deployment's istio-proxy sidecar log is outputing an 409 error and the minio-operator logs are showing error syncing 'thanos/thanos-minio': no healthy upstream logs.
thanos-minio-pool-0-0 istio-proxy logs
[2024-08-13T18:26:27.099Z] "PUT /thanos/ HTTP/1.1" 409 - via_upstream - "-" 0 371 1 1 "-" "MinIO (linux; amd64) minio-go/v7.0.68" "8ca8697a-d9b8-913a-aa64-0ba930201222" "minio.thanos.svc.cluster.local" "10.42.0.21:9000" inbound|9000|| 127.0.0.6:59241 10.42.0.21:9000 10.42.2.6:59808 outbound_.80_._.minio.thanos.svc.cluster.local default traceID=20cdb1a763 │
│ [2024-08-13T18:26:32.052Z] "PUT /thanos/ HTTP/1.1" 409 - via_upstream - "-" 0 371 1 1 "-" "MinIO (linux; amd64) minio-go/v7.0.68" "17b10960-a2b2-9557-87ae-ea5f8da0aac7" "minio.thanos.svc.cluster.local" "10.42.0.21:9000" inbound|9000|| 127.0.0.6:59241 10.42.0.21:9000 10.42.2.6:59808 outbound_.80_._.minio.thanos.svc.cluster.local default traceID=d30f53a074minio-operator logs
│ I0813 15:32:01.120079 1 controller.go:80] Starting MinIO Operator │
│ I0813 15:32:01.208427 1 controller.go:145] Watching only namespaces: │
│ I0813 15:32:01.219977 1 main-controller.go:323] Setting up event handlers │
│ I0813 15:32:01.507646 1 main-controller.go:556] Using Kubernetes CSR Version: v1 │
│ I0813 15:32:01.507744 1 main-controller.go:573] Waiting for STS API to start │
│ I0813 15:32:01.507865 1 main-controller.go:409] Starting STS API server │
│ I0813 15:32:01.507932 1 leaderelection.go:250] attempting to acquire leader lease minio-operator/minio-operator-lock... │
│ I0813 15:32:01.608020 1 tls.go:52] Waiting for the sts certificates secret to be issued │
│ I0813 15:32:01.708712 1 leaderelection.go:260] successfully acquired lease minio-operator/minio-operator-lock │
│ I0813 15:32:01.708842 1 main-controller.go:605] minio-operator-b8fcc4c45-mvlmc: I am the leader, applying leader labels on myself │
│ I0813 15:32:01.708947 1 main-controller.go:442] Waiting for Upgrade Server to start │
│ I0813 15:32:01.709004 1 main-controller.go:446] Starting Tenant controller │
│ I0813 15:32:01.709014 1 main-controller.go:449] Waiting for informer caches to sync │
│ I0813 15:32:01.709027 1 main-controller.go:460] Starting workers and Job workers │
│ I0813 15:32:01.709043 1 main-controller.go:496] Console TLS is not enabled │
│ I0813 15:32:01.709045 1 main-controller.go:398] Starting HTTP Upgrade Tenant Image server │
│ I0813 15:32:01.709079 1 main-controller.go:505] STS is enabled, starting API certificate setup │
│ I0813 15:32:01.807296 1 tls.go:125] sts-tls TLS secret not found: secrets "sts-tls" not found │
│ I0813 15:32:01.837832 1 csr.go:181] Start polling for certificate of csr/sts-minio-operator-csr, every 5s, timeout after 20m0s │
│ I0813 15:32:06.843121 1 csr.go:207] Certificate successfully fetched, creating secret with Private key and Certificate │
│ I0813 15:32:06.848055 1 tls.go:128] Waiting for the sts certificates to be issued waiting for sts cert │
│ I0813 15:33:56.569109 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13842", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created │
│ I0813 15:33:56.595202 1 status.go:55] Hit conflict issue, getting latest version of tenant │
│ I0813 15:33:56.635073 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13849", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created │
│ I0813 15:33:56.659164 1 status.go:55] Hit conflict issue, getting latest version of tenant │
│ I0813 15:33:56.706637 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13860", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created │
│ I0813 15:33:56.750564 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13841", FieldPath:""}): type: 'Normal' reason: 'SACreated' Service Account Created │
│ I0813 15:33:56.787686 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13841", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Role Created │
│ I0813 15:33:56.825588 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13841", FieldPath:""}): type: 'Normal' reason: 'BindingCreated' Role Binding Created │
│ I0813 15:33:57.728069 1 status.go:89] Hit conflict issue, getting latest version of tenant │
│ I0813 15:33:57.743889 1 main-controller.go:958] Detected we are updating a legacy tenant deployment │
│ I0813 15:33:57.755128 1 main-controller.go:997] 'thanos/thanos-minio': Deploying pool pool-0 │
│ I0813 15:33:57.822649 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13909", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool pool-0 created │
│ I0813 15:33:58.913522 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13976", FieldPath:""}): type: 'Normal' reason: 'UsersCreated' Users created │
│ I0813 15:33:59.508436 1 status.go:55] Hit conflict issue, getting latest version of tenant │
│ E0813 15:34:00.113841 1 main-controller.go:1500] error syncing 'thanos/thanos-minio': no healthy upstream │
│ I0813 15:34:00.114237 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"13976", FieldPath:""}): type: 'Warning' reason: 'BucketsCreatedFailed' Buckets creation failed: no healthy upstream │
│ I0813 15:34:06.557553 1 minio-services.go:164] Headless Services don't match: service ports don't match │
│ I0813 15:34:06.566605 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"14263", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated │
│ E0813 15:34:06.735314 1 main-controller.go:1500] error syncing 'thanos/thanos-minio': no healthy upstream │
│ I0813 15:34:06.735891 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"14263", FieldPath:""}): type: 'Warning' reason: 'BucketsCreatedFailed' Buckets creation failed: no healthy upstream │
│ I0813 15:34:35.311396 1 status.go:89] Hit conflict issue, getting latest version of tenant │
│ I0813 15:35:06.883867 1 helper.go:782] Successfully created bucket thanos │
│ I0813 15:35:07.002196 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"thanos", Name:"thanos-minio", UID:"b1a2e2ff-979a-471c-9a81-d1c15365ec26", APIVersion:"minio.min.io/v2", ResourceVersion:"15955", FieldPath:""}): type: 'Normal' reason: 'BucketsCreated' Buckets created │
│ I0813 15:35:07.020005 1 status.go:55] Hit conflict issue, getting latest version of tenant │
│ I0813 15:35:07.080444 1 pdb.go:191] PodDisruptionBudget: v1 │
│ I0813 15:35:13.763602 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio-minio-minio-instance", UID:"4e94af38-b853-4a30-b46f-91d18e23956d", APIVersion:"minio.min.io/v2", ResourceVersion:"16190", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' MinIO Service Created │
│ I0813 15:35:13.775718 1 status.go:55] Hit conflict issue, getting latest version of tenant │
│ I0813 15:35:13.865081 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio-minio-minio-instance", UID:"4e94af38-b853-4a30-b46f-91d18e23956d", APIVersion:"minio.min.io/v2", ResourceVersion:"16201", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Console Service Created │
│ I0813 15:35:13.886776 1 status.go:55] Hit conflict issue, getting latest version of tenant │
│ I0813 15:35:14.012495 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio-minio-minio-instance", UID:"4e94af38-b853-4a30-b46f-91d18e23956d", APIVersion:"minio.min.io/v2", ResourceVersion:"16210", FieldPath:""}): type: 'Normal' reason: 'SvcCreated' Headless Service created │
│ I0813 15:35:14.080126 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio-minio-minio-instance", UID:"4e94af38-b853-4a30-b46f-91d18e23956d", APIVersion:"minio.min.io/v2", ResourceVersion:"16188", FieldPath:""}): type: 'Normal' reason: 'SACreated' Service Account Created │
│ I0813 15:35:14.172981 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio-minio-minio-instance", UID:"4e94af38-b853-4a30-b46f-91d18e23956d", APIVersion:"minio.min.io/v2", ResourceVersion:"16188", FieldPath:""}): type: 'Normal' reason: 'RoleCreated' Role Created │
│ I0813 15:35:14.300400 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio-minio-minio-instance", UID:"4e94af38-b853-4a30-b46f-91d18e23956d", APIVersion:"minio.min.io/v2", ResourceVersion:"16188", FieldPath:""}): type: 'Normal' reason: 'BindingCreated' Role Binding Created │
│ I0813 15:35:14.333603 1 status.go:89] Hit conflict issue, getting latest version of tenant │
│ I0813 15:35:14.554716 1 main-controller.go:958] Detected we are updating a legacy tenant deployment │
│ I0813 15:35:15.092711 1 main-controller.go:997] 'minio/minio-minio-minio-instance': Deploying pool pool-0 │
│ I0813 15:35:15.444112 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio-minio-minio-instance", UID:"4e94af38-b853-4a30-b46f-91d18e23956d", APIVersion:"minio.min.io/v2", ResourceVersion:"16237", FieldPath:""}): type: 'Normal' reason: 'PoolCreated' Tenant pool pool-0 created │
│ I0813 15:35:21.568413 1 minio-services.go:164] Headless Services don't match: service ports don't match │
│ I0813 15:35:21.585328 1 event.go:364] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"minio-minio-minio-instance", UID:"4e94af38-b853-4a30-b46f-91d18e23956d", APIVersion:"minio.min.io/v2", ResourceVersion:"16557", FieldPath:""}): type: 'Normal' reason: 'Updated' Headless Service Updated │
│ E0813 15:36:29.036580 1 main-controller.go:1298] [Will try again in 5sec] Update tenant minio-minio-minio-instance statefulset minio-minio-minio-instance-pool-0 error Operation cannot be fulfilled on statefulsets.apps "minio-minio-minio-instance-pool-0": the object has been modified; please apply your changes to the latest version and try again │
│ I0813 15:37:03.134890 1 monitoring.go:170] 'minio/minio-minio-minio-instance' Failed to get storage info: Server not initialized yet, please try again. │
│ I0813 15:37:13.809530 1 status.go:89] Hit conflict issue, getting latest version of tenant │
│ I0813 15:37:15.407015 1 status.go:89] Hit conflict issue, getting latest version of tenant │
│ I0813 15:37:16.673672 1 sync.go:151] created key minio/minio-minio-minio-instance-pool-0 kind policy/v1, Kind=PodDisruptionBudget diff [ObjectMeta.UID: != 3b2c6c23-1567-4619-b6c0-e9d72612f7d8 ObjectMeta.ResourceVersion: != 18844 ObjectMeta.Generation: 0 != 1 ObjectMeta.CreationTimestamp.Time: 0001-01-01 00:00:00 +0000 UTC != 2024-08-13 15:37:16 + │
│ I0813 15:38:02.347633 1 status.go:89] Hit conflict issue, getting latest version of tenant │
│ I0813 18:26:02.456264 1 status.go:89] Hit conflict issue, getting latest version of tenant override-yaml
addons:
thanos:
enabled: true
# sso:
# client_id: ""
git:
tag: null
branch: "67-set-metrics-retention-to-forever"
values:
storegateyway:
enabled: true
compactor:
enabled: true
objstoreConfig: |-
type: s3
config:
bucket: "thanos"
endpoint: minio.thanos.svc.cluster.local:80
access_key: "minio"
secret_key: "minio123"
insecure: true
trace:
enable: true
istio:
enabled: true
hardened:
enabled: true
minio:
enabled: true
secrets:
name: "thanos-objstore-creds"
accessKey: "minio"
secretKey: "minio123" # default key, change this!
tenant:
# -- Buckets to be provisioned to for tenant
buckets:
- name: thanos
# -- Users to to be provisioned to for tenant
users:
- name: minio-user
# -- User credentials to create for above user. Otherwise password is randomly generated.
# This auth is not required to be set or reclaimed for minio use with Loki
defaultUserCredentials:
username: "minio-user"
password: ""
## Specification for MinIO Pool(s) in this Tenant.
pools:
- servers: 1
volumesPerServer: 4
size: 750Mi
securityContext:
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
runAsNonRoot: true
containerSecurityContext:
runAsUser: 1001
runAsGroup: 1001
runAsNonRoot: true
capabilities:
drop:
- ALL
metrics:
enabled: false
port: 9000
memory: 128M
minioOperator:
enabled: true