Kyverno upgrade in BB 2.6.0 causes potential webhook failures
When upgrading our cluster from BB 2.5.0 to 2.6.0 we encountered an issue with the kyverno version update. Kyverno reconciled successfully but when kyverno-policies attempted to reconcile it errored on a webhook response:
failed kyverno-policies-1.1.0-bb.7 1.1.0 Upgrade "kyverno-kyverno-policies" failed: cannot patch "disallow-host-namespaces" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="8043d41d-77ce-4ac1-b8e5-2f8ba05dc37c", got "" && cannot patch "disallow-image-tags" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="2ed06e6e-59e8-426f-aa24-24d96c271a56", got "" && cannot patch "disallow-istio-injection-bypass" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="3a3595c9-ae07-418f-a810-5ee37f188c63", got "" && cannot patch "disallow-namespaces" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="09c3f6ce-4ef4-4762-8757-0f14972d49cb", got "" && cannot patch "disallow-nodeport-services" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="8ccbace0-cc63-4c8e-b0e3-ed5b822c131c", got "" && cannot patch "disallow-privilege-escalation" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="4966cb3c-0aa6-40fc-a789-047a77927b3d", got "" && cannot patch "disallow-privileged-containers" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="c651e6f5-bc55-4273-8ca7-ffc74daa29f9", got "" && cannot patch "disallow-selinux-options" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="c5e80e1a-9137-40c5-9cfc-0c6822a92b00", got "" && cannot patch "require-drop-all-capabilities" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="b219e0ac-fbcb-4891-bdff-898697609706", got "" && cannot patch "require-istio-on-namespaces" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="107b076d-2924-483d-ae38-25dd6afe54cc", got "" && cannot patch "require-non-root-group" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="3b88f1a6-c6af-4096-aa2b-4c8142c2126e", got "" && cannot patch "require-non-root-user" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="01eb85bf-04ab-4109-9c35-ab40446fc8cc", got "" && cannot patch "restrict-apparmor" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="15e45113-7f65-4b2d-a1ac-d642655ead58", got "" && cannot patch "restrict-capabilities" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="925fe325-eacf-43f8-99e8-ea0db6831e4b", got "" && cannot patch "restrict-external-ips" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="89995e1d-d2d2-4c77-ab55-bb90824d4da5", got "" && cannot patch "restrict-external-names" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="1983c82a-74a4-4c5d-86eb-902b306f2c0f", got "" && cannot patch "restrict-host-path-mount-pv" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="f4a0d8ab-9a05-4be1-b6bf-d36fac2e9f74", got "" && cannot patch "restrict-host-path-mount" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="96c63f9d-41ac-4fa2-bc25-5b224d6e7296", got "" && cannot patch "restrict-host-path-write" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="6e238a4c-9f6f-4958-9ef3-5e2db3c16bd6", got "" && cannot patch "restrict-host-ports" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="3e2494b8-225d-4de0-bba1-650be4128a2c", got "" && cannot patch "restrict-image-registries" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="62f41b0e-1594-4668-9fab-a988a241a52c", got "" && cannot patch "restrict-proc-mount" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="38bc0498-8dbb-43fc-bce4-51a2051e643d", got "" && cannot patch "restrict-seccomp" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="173b6c45-9ab4-416d-ac78-380d2fb38172", got "" && cannot patch "restrict-selinux-type" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="0e3e8b99-68a5-4a91-a14e-080233c8b232", got "" && cannot patch "restrict-sysctls" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="ec22bd01-8bec-4871-b3ba-9bc5e4925dfe", got "" && cannot patch "restrict-volume-types" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="d79e5d3c-8193-406c-8bf8-6768e9bfbf38", got ""
Shortened snippet of error:
failed: cannot patch "disallow-host-namespaces" with kind ClusterPolicy: Internal error occurred: failed calling webhook "mutate-policy.kyverno.svc": received invalid webhook response: expected response.uid="8043d41d-77ce-4ac1-b8e5-2f8ba05dc37c", got ""
We saw better results when updating our kyverno values to specify maxUnavailable
but we did still encounter the issue periodically.
kyverno:
values:
updateStrategy:
rollingUpdate:
maxUnavailable: 0
The only other solution that seemed to help was disabling the kyverno-policies HR pre-upgrade and recreating after, but this is far from desired for production environments.
I did also reach out in the kyverno channel on the k8s slack but so far with no answer on the cause - https://kubernetes.slack.com/archives/CLGR9BJU9/p1689612331003819