istio-ingressgateway: Envoy proxy is NOT ready: config not received from Pilot

Description

When upgrading from 1.7.3-bb.9 to 1.8.4-bb.1 the new istio-ingressgateway pod doesn't become healthy. I have tried a restart of the deployment as well deleting the deployment and waiting for the istio operator to reconcile it. istiod and the cni pods are up and running. I have also tried restarting istiod after the upgrade, once the new CNI pods were up.

Potentially related GH issue: https://github.com/istio/istio/issues/29291. I have yet to fully grok the back and forth in the comments but it would appear the issue is fixed but only in 1.10 releases (at time of writing).

The error:

2021-04-30T04:37:04.145418Z	info	FLAG: --concurrency="0"
2021-04-30T04:37:04.145521Z	info	FLAG: --domain="istio-system.svc.cluster.local"
2021-04-30T04:37:04.145541Z	info	FLAG: --help="false"
2021-04-30T04:37:04.145555Z	info	FLAG: --log_as_json="false"
2021-04-30T04:37:04.145568Z	info	FLAG: --log_caller=""
2021-04-30T04:37:04.145581Z	info	FLAG: --log_output_level="default:info"
2021-04-30T04:37:04.145594Z	info	FLAG: --log_rotate=""
2021-04-30T04:37:04.145607Z	info	FLAG: --log_rotate_max_age="30"
2021-04-30T04:37:04.145620Z	info	FLAG: --log_rotate_max_backups="1000"
2021-04-30T04:37:04.145635Z	info	FLAG: --log_rotate_max_size="104857600"
2021-04-30T04:37:04.145682Z	info	FLAG: --log_stacktrace_level="default:none"
2021-04-30T04:37:04.145703Z	info	FLAG: --log_target="[stdout]"
2021-04-30T04:37:04.145718Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2021-04-30T04:37:04.145731Z	info	FLAG: --outlierLogPath=""
2021-04-30T04:37:04.145744Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2021-04-30T04:37:04.145756Z	info	FLAG: --proxyLogLevel="warning"
2021-04-30T04:37:04.145769Z	info	FLAG: --serviceCluster="istio-ingressgateway"
2021-04-30T04:37:04.145782Z	info	FLAG: --stsPort="0"
2021-04-30T04:37:04.145795Z	info	FLAG: --templateFile=""
2021-04-30T04:37:04.145811Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2021-04-30T04:37:04.145853Z	info	Version 1.8.4-97e10d79b8b5b32be0f92175586a4e11c466e640-Clean
2021-04-30T04:37:04.146326Z	info	Obtained private IP [10.42.8.27 fe80::4021:2bff:feeb:60c6]
2021-04-30T04:37:04.146688Z	info	Apply mesh config from file accessLogFile: /dev/stdout
defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  proxyMetadata:
    DNS_AGENT: ""
  tracing:
    sampling: 10
    zipkin:
      address: jaeger-collector.jaeger.svc:9411
  zipkinAddress: jaeger-collector.jaeger.svc:9411
enablePrometheusMerge: true
enableTracing: true
rootNamespace: istio-system
trustDomain: cluster.local
2021-04-30T04:37:04.151485Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 0
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  DNS_AGENT: ""
serviceCluster: istio-ingressgateway
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  sampling: 10
  zipkin:
    address: jaeger-collector.jaeger.svc:9411
zipkinAddress: jaeger-collector.jaeger.svc:9411

2021-04-30T04:37:04.151686Z	info	Proxy role: &model.Proxy{RWMutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, Type:"router", IPAddresses:[]string{"10.42.8.27", "fe80::4021:2bff:feeb:60c6"}, ID:"istio-ingressgateway-5446685b56-lmkv9.istio-system", Locality:(*envoy_config_core_v3.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), VerifiedIdentity:(*spiffe.Identity)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), WatchedResources:map[string]*model.WatchedResource(nil)}
2021-04-30T04:37:04.151713Z	info	JWT policy is third-party-jwt
2021-04-30T04:37:04.151844Z	info	PilotSAN []string{"istiod.istio-system.svc"}
2021-04-30T04:37:04.151976Z	info	sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012 Citadel
2021-04-30T04:37:04.152093Z	info	Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2021-04-30T04:37:04.152745Z	info	citadelclient	Citadel client using custom root: istiod.istio-system.svc:15012 -----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

2021-04-30T04:37:04.289194Z	info	Starting gateway SDS
2021-04-30T04:37:04.399343Z	info	sds	SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS"

2021-04-30T04:37:04.399817Z	info	sds	SDS gRPC server for gateway controller starts, listening on "./var/run/ingress_gateway/sds"

2021-04-30T04:37:04.399735Z	info	sds	Start SDS grpc server
2021-04-30T04:37:04.399956Z	info	xdsproxy	Initializing with upstream address istiod.istio-system.svc:15012 and cluster Kubernetes
2021-04-30T04:37:04.400004Z	info	sds	Start SDS grpc server for ingress gateway proxy
2021-04-30T04:37:04.400766Z	info	xdsproxy	adding watcher for certificate var/run/secrets/istio/root-cert.pem
2021-04-30T04:37:04.401293Z	info	Starting proxy agent
2021-04-30T04:37:04.401623Z	info	Opening status port 15020

2021-04-30T04:37:04.402408Z	info	Received new config, creating new Envoy epoch 0
2021-04-30T04:37:04.402548Z	info	Epoch 0 starting
2021-04-30T04:37:04.439905Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.42.8.27~istio-ingressgateway-5446685b56-lmkv9.istio-system~istio-system.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format-prefix-with-location 0 --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error]
2021-04-30T04:37:04.640581Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-30T04:37:04.640988Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-30T04:37:04.642687Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-30T04:37:04.642839Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.http.headermap.lazy_map_min_size
2021-04-30T04:37:04.748802Z	info	xdsproxy	Envoy ADS stream established
2021-04-30T04:37:04.749138Z	info	xdsproxy	connecting to upstream XDS server: istiod.istio-system.svc:15012
2021-04-30T04:37:04.752154Z	warning	envoy main	there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2021-04-30T04:37:04.925277Z	info	sds	resource:ROOTCA new connection
2021-04-30T04:37:04.925611Z	info	sds	Skipping waiting for gateway secret
2021-04-30T04:37:04.925694Z	info	sds	resource:default new connection
2021-04-30T04:37:04.925793Z	info	sds	Skipping waiting for gateway secret
2021-04-30T04:37:05.780696Z	info	cache	Root cert has changed, start rotating root cert for SDS clients
2021-04-30T04:37:05.780841Z	info	cache	GenerateSecret default
2021-04-30T04:37:05.782322Z	info	sds	resource:default pushed key/cert pair to proxy
2021-04-30T04:37:06.327583Z	info	cache	Loaded root cert from certificate ROOTCA
2021-04-30T04:37:06.328506Z	info	sds	resource:ROOTCA pushed root cert to proxy
2021-04-30T04:37:06.528009Z	warning	envoy config	gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_8443: duplicate listener 0.0.0.0_8443 found

2021-04-30T04:37:07.864995Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2021-04-30T04:37:09.855630Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
2021-04-30T04:37:11.855322Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected
...

Environment

rke2: 1.20.6+rke2r1
bigbang: 1.6.1
istio-controlplane helm chart: 1.8.4-bb.1

Istio CNI is manually enabled via:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: enable-cni
  namespace: istio-system
spec:
  profile: empty
  components:
    cni:
      enabled: true
  values:
    cni:
      excludeNamespaces:
      - istio-system
      - kube-system