Test thanos with Minio

Test Thanos with Minio and Amazon S3 bucket.

added to epic &251 (closed)

changed the description

marked this issue as blocking #1651 (closed)

added kindchore priority2 teambigbang thanos labels

changed iteration to Big Bang Iterations Oct 3, 2023 - Oct 16, 2023

set weight to 4

added community-contribution label

added priority5 label and removed community-contribution priority2 labels

changed iteration to Big Bang Iterations Oct 17, 2023 - Oct 30, 2023

assigned to @michaelmartin

added statusdoing label

mentioned in merge request !3113 (merged)

I was able to test Thanos hitting a BB-running minio server and add entries to the Thanos bucket

Essentially, we pass through any objstoreConfig configuration to Thanos and the PrometheusSpec. The objstoreConfig will vary widely depending on the actual object storage used. For a BigBig configuration, something like this will work:

    objstoreConfig: |-
      type: s3
      config:
        bucket: "thanos"
        endpoint: minio.minio.svc.cluster.local:80
        access_key: "minio"
        secret_key: "minio123"
        insecure: false
        trace:
          enable: true
        http_config:
          tls_config:
            key_file: /etc/prom-certs/key.pem
            ca_file: /etc/prom-certs/root-cert.pem
            cert_file: /etc/prom-certs/cert-chain.pem
          insecure_skip_verify: true

In this instance, minio.minio.svc.cluster.local requires STRICT mtls. Because the monitoring.prometheus pod disables auto-istio mtls, we need to manually set up the tls certs here in order for the connection to work. These settings will vary depending on the environment, but the objstoreConfig should be flexible enough and passed straight-through to Thanos and the PrometheusSpec.

marking as completed. Will work any follow on issues related to object stores as they come up.

closed

mentioned in commit 810d8b4f