TLS configuration is invalid with custom TCP ports
Overview
When adding custom TCP ports to a gateway definition (as documented here), helm upgrade fails with the following error:
✗ Helm upgrade failed: cannot patch "private" with kind Gateway: admission webhook "validation.istio.io" denied the request: configuration is invalid: server cannot have TLS settings for non HTTPS/TLS ports
Versions
- Bigbang:
2.5.0
- Istio:
1.17.3-bb.1
- K8s:
v1.24.16-eks-2d98532
Context
We attempted to expose a custom TCP port on our Gateway with the following BB config snippet:
istio:
gateways:
public:
hosts:
- "{{ .Values.domain }}"
- "*.{{ .Values.domain }}"
private:
hosts:
- "*.{{ .Values.domain }}"
ports:
- name: https
number: 8443
protocol: HTTPS
- name: tcp-custom
number: 7687
protocol: TCP
Viewing the bigbang/istio-bigbang-values
secret shows the following (truncated) config:
gateways:
private:
servers:
- hosts:
- '*.bigbang.dev'
port:
name: https
number: 8443
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: private-cert
- hosts:
- '*.bigbang.dev'
port:
name: tcp-custom
number: 7687
protocol: TCP
tls:
mode: SIMPLE
credentialName: private-cert
Reconciling the istio helm release results in the error shown above. It appears the tls
section of the server host entry is added regardless of the port protocol resulting in the invalid configuration.
Temporary Workaround
As a stopgap solution, we were able to override the gateway's servers
via istio.values
:
istio:
values:
gateways:
private:
servers:
- hosts:
- '*.bigbang.dev'
port:
name: https
number: 8443
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: private-cert
- hosts:
- '*.bigbang.dev'
port:
name: tcp-custom
number: 7687
protocol: TCP
# NOTE WE HAVE EXCLUDED THE TLS CONFIG FROM THIS HOST
It is also worth noting that when adding custom ports, 8443
or some other HTTPS port must also be included as shown above. Otherwise the helm upgrade fails similarly with:
Upgrade "istio-system-istio" failed: cannot patch "private" with kind Gateway: admission webhook "validation.istio.io" denied the request: configuration is invalid: server config must contain at least one host
Recommended Solutions
- Wrap the tls configuration block in a conditional based on port protocol
- Adjust the documented example to include (or explain the need for) the HTTPS port along with custom additions