Create Egress Policy for External Objstore

General MR

Summary

When deploying thanos with an external object store, such as s3, there is no network policy permitting egress to that store. This adds an allow policy on the storagegateway pods only when the following conditions are met:

1. Network policies are enabled
2. Storage gateway is enabled
3. Thanos minio tenant is disabled

Testing Notes

1. In a default k3d deployment the controlPlaneCidr is set to allow all, i.e. 0.0.0.0/0 which would also allow storage gateway pods egress. Restrict the kube-api egress policy by setting the controlPlaneCidr to ensure accurate testing.

   networkPolicies:
     controlPlaneCidr: 172.16.0.0/12

2. Use the dogfood cluster objstore bucket:

   addons:
     thanos:
       enabled: true
       git:
         tag: ""
         branch: "43/storegateway-egress"
       values:
         storegateway:
           enabled: true
         objstoreConfig: [SEE BELOW]

   Set the objstoreConfig to the value used in the dogfood cluster by decrypting environment-bb-secret

   $ sops -d $PATH_TO_LOCAL_DOGFOOD_CLONE/bigbang/prod2/environment-bb-secret.enc.yaml | yq e '.stringData."values.yaml"' | yq .addons.thanos.values

Relevant logs/screenshots

Failing storage gateway pod and logs when deployed with above settings on main branch:

Storage gateway pod logs when deployed with above settings on 43/storegateway-egress branch:

Linked Issue

#43 (closed)

Upgrade Notices

N/A