#9 : Add helm test for external secrets operator
General MR
Summary
This MR adds a helm test that validates external secrets operator is functioning as expected. Some things were NOT included in the scope of this work:
- Testing integration with AWS, GCP or any other cloud provider
- Testing integration with Vault or any other secret provider
The only goal of this MR was to validate that the external secrets operator itself is functioning, and that given an appropriately configured secret store, that it will be able to inject secrets into pods as expected. To prove this, the test mocks up a primitive scenario using kubernetes cluster secrets.
(K8s Cluster Secrets) <----- (External Secret Operator) ------> (Pod)Using this linkage, we can contrive a scenario where a pod requests a value from the External Secrets Operator without knowing anything about the source secret behind ESO, but getting the value we expect
(SECRET testsecret-source)
^-- (ESO Cluster SecretStore)
`--> (SECRET testsecret-target)
`--> (Pod w/ volumeMount)The test requires some associated objects to be setup inside the cluster to facilitate the tests (specifically a role, binding, secretstore, secret and externalsecret). This is in the chart/templates/tests/ directory and is only executed when the values key bbtests.enabled = true. The actual test is executed by the pod in chart/templates/tests/eso_mount.yaml, which mounts the external secret as a volume on the filesystem, and checks the content of the secret against a known value from the config.
The configuration for the test is in the values.yaml file under the bbtests key. This was done to make it easier for us to add additional test configuration in the future if we want to add validation against vault, AWS SSM, or whatever. It could be extracted out and hardcoded into the charts/templates/tests/ files instead if we don't think it's worth having it in the values file for the user to muck with.
Relevant logs/screenshots
$ helm test external-secrets -n external-secrets
NAME: external-secrets
LAST DEPLOYED: Tue May 28 14:37:03 2024
NAMESPACE: external-secrets
STATUS: pending-install
REVISION: 1
TEST SUITE: external-secrets-test-secret-mount
Last Started: Tue May 28 10:37:32 2024
Last Completed: Tue May 28 10:37:47 2024
Phase: Succeeded
NOTES:
external-secrets has been deployed successfully in namespace external-secrets!
In order to begin using ExternalSecrets, you will need to set up a SecretStore
or ClusterSecretStore resource (for example, by creating a 'vault' SecretStore).
More information on the different types of SecretStores and how to configure them
can be found in our Github: https://github.com/external-secrets/external-secretsThe pod runs, mounts the secret, and validates it successfully:
$ kubectl describe pod -n external-secrets external-secrets-test-secret-mount
Name: external-secrets-test-secret-mount
Namespace: external-secrets
Priority: 0
Service Account: default
Node: k3d-k3s-default-agent-2/172.20.0.5
Start Time: Tue, 28 May 2024 10:37:36 -0400
Labels: <none>
Annotations: helm.sh/hook: test
Status: Succeeded
IP: 10.42.2.5
IPs:
IP: 10.42.2.5
Containers:
external-secrets-test-secret-mount:
Container ID: containerd://a53fbffc07e8c58f8c808034105a6d3667e9c650aa3bada0eb555f161c445a3e
Image: busybox
Image ID: docker.io/library/busybox@sha256:5eef5ed34e1e1ff0a4ae850395cbf665c4de6b4b83a32a0bc7bcb998e24e7bbb
Port: <none>
Host Port: <none>
Command:
/bin/grep
Args:
this is a magic value
/tmp/externalsecrets/secretkey
State: Terminated
Reason: Completed
Exit Code: 0
Start False
Restart Count: 0
Environment: <none>
Mounts:
/tmp/externalsecrets from externalsecrets (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5879h (ro)
Conditions:
Type Status
PodReadyToStartContainers False
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
externalsecrets:
Type: Secret (a volume populated by a Secret)
SecretName: testsecret-target
Optional: false
kube-api-access-5879h:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecufor 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 35m default-scheduler Successfully assigned external-secrets/external-secrets-test-secret-mount to k3d-k3s-default-agent-2
Normal Pulling 35m kubelet Pulling image "busybox"
Normal Pulled 35m kubelet Successfully pulled image "busybox" in 1.763s (1.763s including waiting)
Normal Created 35m kubelet Created container external-secrets-test-secret-mount
Normal Started 35m kubelet Started container external-secrets-test-secret-mount... and we can see the external secret is created in the appropriate secret store and that the target the pod is pulling from is only the ESO frontend for the source, which proves that ESO is doing its job
$ kubectl get secretstore -n external-secrets
NAME AGE STATUS CAPABILITIES READY
external-secrets-test-store 38m Valid ReadWrite True
$ kubectl describe externalsecret -n external-secrets testsecret
Name: testsecret
Namespace: external-secrets
Labels: app.kubernetes.io/managed-by=Helm
helm.toolkit.fluxcd.io/name=external-secrets
helm.toolkit.fluxcd.io/namespace=external-secrets
Annotations: meta.helm.sh/release-name: external-secrets
meta.helm.sh/release-namespace: external-secrets
API Version: external-secrets.io/v1beta1
Kind: ExternalSecret
Metadata:
Creation Timestamp: 2024-05-28T14:37:07Z
Generation: 1
Resource Version: 1168
UID: 569d7e49-5cab-4f4a-94da-1b87df937988
Spec:
Data:
Remote Ref:
Conversion Strategy: Default
Decoding Strategy: None
Key: testsecret-source
Metadata Policy: None
Property: secretkey
Secret Key: secretkey
Refresh Interval: 1h
Secret Store Ref:
Kind: SecretStore
Name: external-secrets-test-store
Target:
Creation Policy: Owner
Deletion Policy: Retain
Name: testsecret-target
Status:
Binding:
Name: testsecret-target
Conditions:
Last Transition Time: 2024-05-28T14:37:13Z
Message: Secret was synced
Reason: SecretSynced
Status: True
Type: Ready
Refresh Time: 2024-05-28T14:37:13Z
Synced Resource Version: 1-fc08bb16d79a4d4e355bb0cf71c0aed5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 39m external-secrets Created Secret
$ kubectl get secret -n external-secrets testsecret-source -o yaml
apiVersion: v1
data:
secretkey: dGhpcyBpcyBhIG1hZ2ljIHZhbHVl
kind: Secret
metadata:
annotations:
meta.helm.sh/release-name: external-secrets
meta.helm.sh/release-namespace: external-secrets
creationTimestamp: "2024-05-28T14:37:07Z"
labels:
app.kubernetes.io/managed-by: Helm
helm.toolkit.fluxcd.io/name: external-secrets
helm.toolkit.fluxcd.io/namespace: external-secrets
name: testsecret-source
namespace: external-secrets
resourceVersion: "1025"
uid: 440728cc-c92d-4afc-bed2-a76eddaa5692
type: OpaqueLinked Issue
Closes #9 (closed)
Upgrade Notices
N/A