`require-image-signature` Kyverno Policy blocks deployments while in `Audit` mode

Bug

When deploying BigBang, packages after kyverno-policies have intermittently failed due to the require-image-signature policy failing to pull image signatures from IronBank. Currently, the policy is enabled by default.

Description

Logs taken from this pipeline run:

I1101 14:05:22.049899       1 event.go:294] "Event occurred" object="istio-operator/istio-operator-877496f5c" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message=<
    Error creating: admission webhook "mutate.kyverno.svc-fail" denied the request: 
    
    resource Pod/istio-operator/ was blocked due to the following policies 
    
    require-image-signature:
      verify-image: 'failed to verify image registry1.dso.mil/ironbank/opensource/istio/operator:1.19.3:
        .attestors[0].entries[0].keys: Get "https://registry1.dso.mil/v2/": dial tcp 15.205.173.153:443:
        i/o timeout'
 >
E1101 14:05:22.056773       1 replica_set.go:544] sync "istio-operator/istio-operator-877496f5c" failed with admission webhook "mutate.kyverno.svc-fail" denied the request: 

resource Pod/istio-operator/ was blocked due to the following policies 

require-image-signature:
  verify-image: 'failed to verify image registry1.dso.mil/ironbank/opensource/istio/operator:1.19.3:
    .attestors[0].entries[0].keys: Get "https://registry1.dso.mil/v2/": dial tcp 15.205.173.153:443:
    i/o timeout'
E1101 14:05:27.090920       1 replica_set.go:544] sync "istio-operator/istio-operator-877496f5c" failed with admission webhook "mutate.kyverno.svc-fail" denied the request: 

resource Pod/istio-operator/ was blocked due to the following policies 

require-image-signature:
  verify-image: 'failed to verify image registry1.dso.mil/ironbank/opensource/istio/operator:1.19.3:
    .attestors[0].entries[0].keys: Get "https://registry1.dso.mil/v2/": dial tcp 15.205.173.153:443:
    i/o timeout'
I1101 14:05:27.090942       1 event.go:294] "Event occurred" object="istio-operator/istio-operator-877496f5c" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message=<
    Error creating: admission webhook "mutate.kyverno.svc-fail" denied the request: 
    
    resource Pod/istio-operator/ was blocked due to the following policies 
    
    require-image-signature:
      verify-image: 'failed to verify image registry1.dso.mil/ironbank/opensource/istio/operator:1.19.3:
        .attestors[0].entries[0].keys: Get "https://registry1.dso.mil/v2/": dial tcp 15.205.173.153:443:
        i/o timeout'
 >
E1101 14:05:32.124153       1 replica_set.go:544] sync "istio-operator/istio-operator-877496f5c" failed with admission webhook "mutate.kyverno.svc-fail" denied the request: 

resource Pod/istio-operator/ was blocked due to the following policies 

require-image-signature:
  verify-image: 'failed to verify image registry1.dso.mil/ironbank/opensource/istio/operator:1.19.3:
    .attestors[0].entries[0].keys: Get "https://registry1.dso.mil/v2/": dial tcp 15.205.173.153:443:
    i/o timeout'
I1101 14:05:32.124167       1 event.go:294] "Event occurred" object="istio-operator/istio-operator-877496f5c" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message=<
    Error creating: admission webhook "mutate.kyverno.svc-fail" denied the request: 
    
    resource Pod/istio-operator/ was blocked due to the following policies 
    
    require-image-signature:
      verify-image: 'failed to verify image registry1.dso.mil/ironbank/opensource/istio/operator:1.19.3:
        .attestors[0].entries[0].keys: Get "https://registry1.dso.mil/v2/": dial tcp 15.205.173.153:443:
        i/o timeout'
 >
E1101 14:05:38.079983       1 replica_set.go:544] sync "istio-operator/istio-operator-877496f5c" failed with admission webhook "mutate.kyverno.svc-fail" denied the request: 

resource Pod/istio-operator/ was blocked due to the following policies 

require-image-signature:
  verify-image: 'failed to verify image registry1.dso.mil/ironbank/opensource/istio/operator:1.19.3:
    .attestors[0].entries[0].keys: Get "https://registry1.dso.mil/v2/": dial tcp 15.205.173.153:443:
    i/o timeout'
I1101 14:05:38.080046       1 event.go:294] "Event occurred" object="istio-operator/istio-operator-877496f5c" fieldPath="" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message=<
    Error creating: admission webhook "mutate.kyverno.svc-fail" denied the request: 
    
    resource Pod/istio-operator/ was blocked due to the following policies 
    
    require-image-signature:
      verify-image: 'failed to verify image registry1.dso.mil/ironbank/opensource/istio/operator:1.19.3:
        .attestors[0].entries[0].keys: Get "https://registry1.dso.mil/v2/": dial tcp 15.205.173.153:443:
        i/o timeout'
 >

BigBang Version

2.13.1 2.17.0

Workaround

Users can workaround this issue by disabling this policy in their overrides. However, enabling this policy by default (even in Audit mode) appears to be disruptive for some users.

The current workaround is to delete the require-image-signature ClusterPolicy manually.

This issue will occur if the kyverno admission controller in your cluster does not have access to your image registry. Add the following overrides to the kyverno package in your bigbang values:

kyverno:
  values:
    networkPolicies:
      enabled: true
      externalRegistries:
        allowEgress: true

Additional Info