Create Egress Policy for External Objstore
General MR
Summary
When deploying thanos with an external object store, such as s3, there is no network policy permitting egress to that store. This adds an allow policy on the storagegateway
pods only when the following conditions are met:
- Network policies are enabled
- Storage gateway is enabled
- Thanos minio tenant is disabled
Testing Notes
-
In a default k3d deployment the
controlPlaneCidr
is set to allow all, i.e.0.0.0.0/0
which would also allow storage gateway pods egress. Restrict the kube-api egress policy by setting thecontrolPlaneCidr
to ensure accurate testing.networkPolicies: controlPlaneCidr: 172.16.0.0/12
-
Use the dogfood cluster objstore bucket:
addons: thanos: enabled: true git: tag: "" branch: "43/storegateway-egress" values: storegateway: enabled: true objstoreConfig: [SEE BELOW]
Set the objstoreConfig to the value used in the dogfood cluster by decrypting
environment-bb-secret
$ sops -d $PATH_TO_LOCAL_DOGFOOD_CLONE/bigbang/prod2/environment-bb-secret.enc.yaml | yq e '.stringData."values.yaml"' | yq .addons.thanos.values
Relevant logs/screenshots
Failing storage gateway pod and logs when deployed with above settings on main
branch:
Storage gateway pod logs when deployed with above settings on 43/storegateway-egress
branch:
Linked Issue
Upgrade Notices
N/A