`app.kubernetes.io/version` label prevents gateways from updating

Bug

Description

Not too long ago we added some kubernetes labels to the gateways as part of #241 (closed). As a result of that change, it will update the labels every time a new version of istio is installed. This is not supported by the operator and generates errors like:

The primary issue is that this prevents the gateway pods from using new container images as the deployment manifest never gets updated.

According to the istio folks, this error is expected behavior as it is a k8s limitation.

Steps To Reproduce

1. Deploy Istio with an ingress gateway
2. On the IstioOperator object, update the app.kubernetes.io/version label for the ingress gateway to simulate what a chart upgrade would do

Expected result:

IstioOperator object successfully reconciles and ingress gateway pods are cycled onto the new deployment spec

Actual result:

IstioOperator errors (see error screenshot above).

Additional Notes

I haven't tested this with the egress gateways, but I imagine the same problem applies.

Workaround

Manually delete any and all deployments for gateways and let the operator recreate them with the new labels.