Unable to disable minio when loki.strategy is set to "scalable" and not setting values for both endpoint and region

Description:

When using the loki subchart and utilizing IRSA and EKS, we needed to remove what would be templated by .Values.loki.objectStorage.endpoint` in our loki config in the child chart. To fit our use case we need the ability to set in this file:

1. loki.strategy to "scalable"
2. not define loki.objectStorage.endpoint
3. define loki.objectStorage.region

Basically, we don't need minio and our loki + irsa config didn't work with endpoint defined.

We have a workaround for this logic currently:

To ensure that minio is disabled, we needed to set loki.objectStorage.endpoint and loki.objectStorage.region, and then also pass in values.loki.storage.s3.endpoint: "" to not break our loki config file

loki:
  enabled: true
  strategy: "scalable"
  objectStorage:
    endpoint: s3.$region.amazonaws.com
    region: $region
    bucketNames:
      chunks: $bucket
      ruler: $bucket
      admin: $bucket
  values:
    serviceAccount:
      annotations: 
        eks.amazonaws.com/role-arn: $iam_role_arn
    loki:
      storage:
        s3:
          endpoint: ""

A different workaround would be to disable MinIO with values (and then you should be able to leave endpoint empty):

loki:
  values:
    minio:
      enabled: false

We can probably get conditionals adjusted so that only one of endpoint/region is required - I think we made similar adjustments for Gitlab and another chart maybe? /cc @ryan.j.garcia

Yeah we have made similar adjustments, and we should be allowing for first class support of IRSA but no matter which way we go there will be another issue in the future I'm sure

It looks like that does work using this as input. Like @micah.nagel suggested, if we just set minio to false it's not that bad.

(after putting in this issue) I threw together some helm templating script that passes the input values through templating BB and uses those an inputs for templating child chart. I don't see any mention of endpoint or minio things in the resulting child chart outputs, works for me ¯\_(ツ)_/¯.

#input

loki:
  enabled: true
  strategy: "scalable"
  objectStorage:
    # endpoint: s3.us-gov-east-1.amazonaws.com
    region: us-gov-east-1
    bucketNames:
      chunks: bucket_blah
      ruler: bucket_blah
      admin: bucket_blah
  values:
    minio:
      enabled: false
    serviceAccount:
      annotations: 
        eks.amazonaws.com/role-arn: whatever-not-real-role
    # loki:
    #   storage:
    #     s3:
    #       endpoint: ""
    istio:
      mtls:
        mode: PERMISSIVE
...

#output "helmrelease" overlay values

common_values:
---


default_values:
---
hostname: 

openshift: false

istio:
  enabled: true

imagePullSecrets:
  - name: private-registry

image:
  pullPolicy: IfNotPresent

networkPolicies:
  enabled: true
  controlPlaneCidr: 0.0.0.0/0

monitoring:
  enabled: true
  serviceMonitor:
    enabled: true
    # conditional passes only for default istio: enabled, mTLS: SCRICT

loki:
  storage:
    type: "s3"
    bucketNames:
      admin: bucket_blah
      chunks: bucket_blah
      ruler: bucket_blah
    s3:
      endpoint: 
      region: us-gov-east-1
      accessKeyId: <no value>
      secretAccessKey: 
  podAnnotations:
    bigbang.dev/istioVersion: 1.17.1
singleBinary:
  replicas: 0
minio:
  enabled: true

overlay_values:
---
istio:
  mtls:
    mode: PERMISSIVE
minio:
  enabled: false
serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: whatever-not-real-role

This workaround does not work when minio is not deployed. There is a dependency injected in the HelmRelease here that prevents the Loki HR from reconciling.

Can we bump this. There is a bug ticket in with Loki here that is still open. Essentially, we CAN'T put endpoint in when NOT using the secret and access key. So if we can't inject the endpoint, and we aren't using minio, then we need to be able to still run this.

We are using a temporary workaround that uses a postrenderer to remove the dependsOn from the loki chart, but it's hardcoded and kind of hacky.

@singleton_jacob the workaround described in the original issue post should still work, specifically this part:

loki:
  objectStorage:
    endpoint: s3.$region.amazonaws.com
...
  values:
    loki:
      storage:
        s3:
          endpoint: ""
...

By setting endpoint at the "big bang level" it removes the dependency on minio from the helm release. Then to workaround the loki issue you linked, you have to "erase" endpoint at the loki layer.

Essentially just not using the loki.objectStorage at all is the only way this works for us without the postrenderer described. That's essentially what you're doing above, but you could literally set loki.objectStorage.endpoint to any value. It doesn't matter and will get overridden. But, if endpoint is set there, then bucketNames has to as well. Or just don't set either of them and skip using the objectStorage all together.

Either way there are workarounds, but it still makes sense to me to implement something that couples the minio flag with the 'dependsOn', because why would you want a dependsOn: minio-operator if you've explicitly set .Values.minio.enabled: false

Would it make sense to update the BB logic to only enable minio related configuration/deployment if loki.objectStorage.endpoint contains minio?

added community-contribution label

@zack_annexstein this issue has been inactive for 30 days and is being labelled as stale. If this issue is still required please take action by removing the stale label and commenting with an update, status, or justification. If this issue is not required please close it or label it as delete-me. If no action is taken this issue will be auto closed in 60 days.

added stale label

added kindbug teamObservability labels and removed stale label

changed iteration to Big Bang Iterations Dec 10, 2024 - Dec 23, 2024

@kscheunemann fysa

added loki needs-labels labels

added statusready-to-work label

set weight to 2

removed iteration Big Bang Iterations Dec 10, 2024 - Dec 23, 2024

changed iteration to Big Bang Iterations Dec 24, 2024 - Jan 6, 2025

assigned to @kscheunemann

added statusdoing label and removed statusready-to-work label

@singleton_jacob @micah.nagel @zack_annexstein Would it make sense to update the BB logic to only enable minio related configuration/deployment if loki.objectStorage.endpoint contains minio?

Yes I think that makes sense

added statusblocked label and removed statusdoing label

removed iteration Big Bang Iterations Dec 24, 2024 - Jan 6, 2025

changed iteration to Big Bang Iterations Jan 7, 2025 - Jan 20, 2025

removed iteration Big Bang Iterations Jan 7, 2025 - Jan 20, 2025

changed iteration to Big Bang Iterations Jan 21, 2025 - Feb 3, 2025

removed statusblocked label

added statusready-to-work label

Here's some logic we need to test and verify: If loki.objectStorage.endpoint contains "local-minio" or some other string. Then we enable dependency on minio-operator, deploy the loki minio target, set the s3 endpoint config to the local kubernetes service for minio.

If loki.objectStorage.endpoint does not contain "local-minio" or some other defined string, then do not set the minio-operator dependency, and do not deploy minio target, just configure loki with the defined s3 settings.

removed iteration Big Bang Iterations Jan 21, 2025 - Feb 3, 2025

changed iteration to Big Bang Iterations Feb 4, 2025 - Feb 17, 2025

added statusdoing label and removed statusready-to-work label

created branch 1473-unable-to-disable-minio-when-loki-strategy-is-set-to-scalable-and-not-setting-values-for-both to address this issue

mentioned in merge request !5733 (merged)

added statusreview label and removed statusdoing label

mentioned in commit f29faf67

closed with merge request !5733 (merged)

removed statusreview label