UNCLASSIFIED - NO CUI

Skip to content

`require-image-signature` policy doesn't work in egress limited/airgapped envs

Bug

When deploying into egress limited or airgapped environments, the kyverno admission controller times out pulling keys from rekor when verifying image signatures.

Description

As configured today, the kyverno admission controller pulls public keys for sigstore's rekor and ctlog from an internet-accessible instance that they maintain. In an environment that does not have the IP address of their instance whitelisted, this will cause verification of image signatures to hang, blocking deployments in some cases.

Despite the require-image-signature policy having a failurePolicy of Ignore(#9 (closed)), this has still blocked deployments of bigbang (big-bang/bigbang#1821 (closed)).

Investigate if either the webhookTimeoutSeconds parameter needs to be adjusted to allow for this type of failure to occur without disruption, of if there is a method of providing these public keys offline.

Logs

kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:14:53.334996400-05:00 I1214 22:14:53.333846       1 imageverifier.go:265] engine.verify "msg"="cache entry not found" "imageRef"="registry1.dso.mil/ironbank/opensource/grafana/loki:2.9.0" "namespace"="" "new.kind"="Pod" "new.name"="test" "new.namespace"="default" "policy"="require-image-signature" "policy.apply"="All" "policy.name"="require-image-signature" "policy.namespace"="" "rule.name"="verify-image" "ruleName"="verify-image"
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:14:53.335081481-05:00 I1214 22:14:53.333916       1 imageverifier.go:321] engine.verify "msg"="verifying image signatures" "attestations"=0 "attestors"=1 "image"="registry1.dso.mil/ironbank/opensource/grafana/loki:2.9.0" "new.kind"="Pod" "new.name"="test" "new.namespace"="default" "policy.apply"="All" "policy.name"="require-image-signature" "policy.namespace"="" "rule.name"="verify-image"
kyverno-admission-controller-5997fbb54d-dsg7z kyverno 2023-12-14T17:15:03.326715062-05:00 I1214 22:15:03.326548       1 validation.go:103] webhooks/resource/validate "msg"="validation failed" "action"="audit" "clusterroles"=["cluster-admin","system:basic-user","system:discovery","system:public-info-viewer"] "failed rules"=["verify-image"] "gvk"={"group":"","version":"v1","kind":"Pod"} "gvr"={"group":"","version":"v1","resource":"pods"} "kind"="Pod" "name"="test" "namespace"="default" "operation"="CREATE" "policy"="require-image-signature" "resource"="default/Pod/test" "resource.gvk"={"Group":"","Version":"v1","Kind":"Pod"} "roles"=null "uid"="ee0ffe2b-0ea0-4893-93f4-e245dddbf82d" "user"={"username":"system:admin","groups":["system:masters","system:authenticated"]}
kyverno-admission-controller-5997fbb54d-dsg7z kyverno 2023-12-14T17:15:03.330508290-05:00 I1214 22:15:03.330351       1 event_broadcaster.go:318] "Event occurred" object="require-image-signature" kind="ClusterPolicy" apiVersion="kyverno.io/v1" type="Warning" reason="PolicyViolation" action="Resource Passed" note="Pod default/test: [verify-image] fail; unverified image registry1.dso.mil/ironbank/opensource/grafana/loki:2.9.0"
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.084226978-05:00 I1214 22:16:58.084076       1 cosign.go:59] cosign "msg"="image verification failed" "error"="failed to load CTLogs public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: failed to download 8.root.json: Get \"https://tuf-repo-cdn.sigstore.dev/8.root.json\": dial tcp 10.255.255.1:443: i/o timeout\nremote status:{\n\t\"mirror\": \"https://tuf-repo-cdn.sigstore.dev\",\n\t\"metadata\": {}\n}"
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.084319939-05:00 I1214 22:16:58.084161       1 imageverifier.go:498] engine.verify "msg"="image attestors verification failed" "errors"=".attestors[0].entries[0].keys: failed to load CTLogs public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: failed to download 8.root.json: Get \"https://tuf-repo-cdn.sigstore.dev/8.root.json\": dial tcp 10.255.255.1:443: i/o timeout\nremote status:{\n\t\"mirror\": \"https://tuf-repo-cdn.sigstore.dev\",\n\t\"metadata\": {}\n}" "new.kind"="Pod" "new.name"="test" "new.namespace"="default" "policy.apply"="All" "policy.name"="require-image-signature" "policy.namespace"="" "requiredCount"=1 "rule.name"="verify-image" "verifiedCount"=0
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.084334730-05:00 E1214 22:16:58.084213       1 imageverifier.go:360] engine.verify "msg"="failed to verify image" "error"=".attestors[0].entries[0].keys: failed to load CTLogs public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: failed to download 8.root.json: Get \"https://tuf-repo-cdn.sigstore.dev/8.root.json\": dial tcp 10.255.255.1:443: i/o timeout\nremote status:{\n\t\"mirror\": \"https://tuf-repo-cdn.sigstore.dev\",\n\t\"metadata\": {}\n}" "new.kind"="Pod" "new.name"="test" "new.namespace"="default" "policy.apply"="All" "policy.name"="require-image-signature" "policy.namespace"="" "rule.name"="verify-image"
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.092085388-05:00 I1214 22:16:58.091902       1 event_broadcaster.go:318] "Event occurred" object="require-image-signature" kind="ClusterPolicy" apiVersion="kyverno.io/v1" type="Warning" reason="PolicyViolation" action="Resource Passed" note=<
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.092128999-05:00 	Pod default/test: [verify-image] fail; failed to verify image registry1.dso.mil/ironbank/opensource/grafana/loki:2.9.0: .attestors[0].entries[0].keys: failed to load CTLogs public keys: updating local metadata and targets: error updating to TUF remote mirror: tuf: failed to download 8.root.json: Get "https://tuf-repo-cdn.sigstore.dev/8.root.json": dial tcp 10.255.255.1:443: i/o timeout
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.092138509-05:00 	remote status:{
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.092146779-05:00 		"mirror": "https://tuf-repo-cdn.sigstore.dev",
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.092153989-05:00 		"metadata": {}
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.092161349-05:00 	}
kyverno-admission-controller-5997fbb54d-lvm4t kyverno 2023-12-14T17:16:58.092168189-05:00  >

Relevant documentation

https://kyverno.io/docs/writing-policies/verify-images/sigstore/#using-custom-rekor-public-key-and-ctlogs-public-key https://github.com/kyverno/kyverno/issues/8691

Edited by Noah Birrer

UNCLASSIFIED - NO CUI