#18 : Add CRDs that are missing from the upstream chart release version
General MR
Summary
This introduces a second KPT integration to pull the CRDs out of the upstream chart release location, where they cannot be used by helm, to the ./chart/crds/ directory where Helm will deploy them automatically before attempting to deploy the helmrelease. This resolves a known race condition in helm releases that attempt to deploy their own CRDs, which results in an unhealthy deployment.
The maintenance procedure for these nested KPT integrations will be covered by #13 (closed).
Relevant logs/screenshots
When deploying this version of the external-secrets app using the bigbang umbrella chart ...
packages:
external-secrets:
enabled: true
git:
repo: https://repo1.dso.mil/big-bang/apps/sandbox/external-secrets.git
branch: 18_kustomize
path: chart
installCRDs: false... we get the custom resources delivered appropriately:
$ kubectl api-resources | grep -i external
clusterexternalsecrets ces external-secrets.io/v1beta1 false ClusterExternalSecret
clustersecretstores css external-secrets.io/v1beta1 false ClusterSecretStore
externalsecrets es external-secrets.io/v1beta1 true ExternalSecret
pushsecrets external-secrets.io/v1alpha1 true PushSecret
secretstores ss external-secrets.io/v1beta1 true SecretStore
acraccesstokens acraccesstoken generators.external-secrets.io/v1alpha1 true ACRAccessToken
ecrauthorizationtokens ecrauthorizationtoken generators.external-secrets.io/v1alpha1 true ECRAuthorizationToken
fakes fake generators.external-secrets.io/v1alpha1 true Fake
gcraccesstokens gcraccesstoken generators.external-secrets.io/v1alpha1 true GCRAccessToken
githubaccesstokens githubaccesstoken generators.external-secrets.io/v1alpha1 true GithubAccessToken
passwords password generators.external-secrets.io/v1alpha1 true Password
vaultdynamicsecrets vaultdynamicsecret generators.external-secrets.io/v1alpha1 true VaultDynamicSecret
webhooks webhookl generators.external-secrets.io/v1alpha1 true Webhook
... and the pods quickly reach a healthy state:
$ kubectl get pods -n external-secrets
NAME READY STATUS RESTARTS AGE
external-secrets-7885c6d54d-n9v2w 1/1 Running 0 3m4s
external-secrets-webhook-79cc84585f-ddzfj 1/1 Running 0 3m4s
external-secrets-cert-controller-89f775b5d-k524c 1/1 Running 0 3m4s ... The certificate controller properly generates and publishes a certificate:
{"level":"info","ts":1715470401.1115053,"logger":"controllers.webhook-certs-updater","msg":"injecting ca certificate and service names","cacrt":"... SNIP ...","name":"externalsecret-validate"}
{"level":"info","ts":1715470401.1262224,"logger":"controllers.webhook-certs-updater","msg":"updated webhook config","Webhookconfig":{"name":"externalsecret-validate"}}
{"level":"info","ts":1715470401.130197,"logger":"controllers.webhook-certs-updater","msg":"injecting ca certificate and service names","cacrt":"... SNIP ...","name":"secretstore-validate"}
{"level":"info","ts":1715470401.1508305,"logger":"controllers.webhook-certs-updater","msg":"updated webhook config","Webhookconfig":{"name":"secretstore-validate"}}... And the webhook correctly pulls the certificates:
{"level":"info","ts":1715470410.274716,"logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"external-secrets.io/v1alpha1, Kind=ClusterSecretStore"}
{"level":"info","ts":1715470410.2747238,"logger":"setup","msg":"starting manager"}
{"level":"info","ts":1715470410.2748673,"logger":"controller-runtime.metrics","msg":"Starting metrics server"}
{"level":"info","ts":1715470410.2755458,"logger":"controller-runtime.webhook","msg":"Starting webhook server"}
{"level":"info","ts":1715470410.275705,"msg":"starting server","kind":"health probe","addr":"[::]:8081"}
{"level":"info","ts":1715470410.2753787,"logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}
{"level":"info","ts":1715470410.2764072,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1715470410.2765102,"logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":10250}
{"level":"info","ts":1715470410.2768297,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}Now when we attempt to create secretstores or other CRDs with external-secrets, we get new errors:
$ kubectl apply -f secretstore.yaml
Warning: external-secrets.io/v1alpha1 SecretStore is deprecated; use external-secrets.io/v1beta1 SecretStore
secret/webhook-credentials created
Error from server (InternalError): error when creating "secretstore.yaml": Internal error occurred: conversion webhook for external-secrets.io/v1alpha1, Kind=SecretStore failed: Post "https://external-secrets-webhook.external-secrets.svc:443/convert?timeout=30s": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match external-secrets-webhook.external-secrets.svc This error appears to be a separate bug. The certs themselves generated in the cluster for the webhook seem good:
Subject: CN=external-secrets-webhook.external-secrets.svc
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:cc:47:7f:a0:03:75:83:c0:47:42:fb:aa:41:cf:
c1:c8:5b:80:0f:47:33:d6:a8:dc:17:5c:94:da:c7:
82:99:21:16:51:61:31:d7:28:05:3d:e5:fe:3f:69:
a0:5e:c5:20:8d:b1:78:e3:d3:78:69:79:50:82:9b:
73:3b:49:95:cf:07:6c:eb:93:c5:bb:24:cb:2a:09:
4b:34:47:c3:c8:31:97:2b:e3:36:4e:b2:6f:09:9f:
ee:f2:26:68:a5:a6:a8:68:c8:7d:4e:01:e4:9d:c4:
09:db:08:52:1b:3a:7a:2d:09:1c:ad:ef:ff:a1:8d:
62:50:e9:e9:af:4f:7f:e1:fd:5a:23:fc:31:85:3b:
b6:7e:51:7a:4a:b7:48:75:1e:d1:7f:bd:8e:1d:a0:
12:24:6c:b8:f2:af:1f:9a:a3:80:69:9a:9c:22:9d:
9f:c6:43:66:fe:d3:1b:f9:09:e5:7d:16:80:f9:2a:
61:5c:09:60:fc:f1:18:21:fb:7c:6f:39:f2:fd:cc:
cd:18:15:3c:fe:31:06:76:89:72:a2:17:ce:06:82:
55:9a:74:16:95:40:b9:17:f1:ae:21:c4:b7:9c:45:
61:30:03:71:ea:73:83:a2:f4:84:99:0f:64:e4:f8:
30:93:ef:1e:57:78:bc:e8:67:1c:c7:e0:44:67:bb:
60:81
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Authority Key Identifier:
6F:19:E6:27:93:8A:13:9F:F4:AE:49:0C:B5:DD:A7:E4:00:7E:87:5D
X509v3 Subject Alternative Name:
DNS:external-secrets-webhook.external-secrets.svc ... I believe this resolves #18 (closed) sufficiently to unblock other tickets, but we will need another ticket to investigate the TLS issues.
Linked Issue
Upgrade Notices
N/A
Edited by Andrew Kesterson