UNCLASSIFIED - NO CUI

Skip to content

Create Egress Policy for External Objstore

Nicole Dupree requested to merge 43/storegateway-egress into main

General MR

Summary

When deploying thanos with an external object store, such as s3, there is no network policy permitting egress to that store. This adds an allow policy on the storagegateway pods only when the following conditions are met:

  1. Network policies are enabled
  2. Storage gateway is enabled
  3. Thanos minio tenant is disabled

Testing Notes

  1. In a default k3d deployment the controlPlaneCidr is set to allow all, i.e. 0.0.0.0/0 which would also allow storage gateway pods egress. Restrict the kube-api egress policy by setting the controlPlaneCidr to ensure accurate testing.

    networkPolicies:
      controlPlaneCidr: 172.16.0.0/12
  2. Use the dogfood cluster objstore bucket:

    addons:
      thanos:
        enabled: true
        git:
          tag: ""
          branch: "43/storegateway-egress"
        values:
          storegateway:
            enabled: true
          objstoreConfig: [SEE BELOW]

    Set the objstoreConfig to the value used in the dogfood cluster by decrypting environment-bb-secret

    $ sops -d $PATH_TO_LOCAL_DOGFOOD_CLONE/bigbang/prod2/environment-bb-secret.enc.yaml | yq e '.stringData."values.yaml"' | yq .addons.thanos.values

Relevant logs/screenshots

Failing storage gateway pod and logs when deployed with above settings on main branch: Screenshot_2024-04-02_at_1.02.16_PM Screenshot_2024-04-02_at_1.01.57_PM

Storage gateway pod logs when deployed with above settings on 43/storegateway-egress branch: Screenshot_2024-04-02_at_1.08.24_PM

Linked Issue

#43 (closed)

Upgrade Notices

N/A

Edited by Blane Staskiewicz

Merge request reports