Istio sidecar will not launch without modification on RKE2 cluster (istio-init container iptables errors)

Bug

Description

After launching BigBang on RKE2 clusters, functional pods launching in istio-labeled namespaces fail to start. The istio sidecars fail to fully inject due to a failed istio-init container. The istio-init sidecar logs suggest that iptables is being invoked with a bad command line in istio-init container; however, the actual issue is that the container is not authorized to escalate privs on locked-down RKE2.

Background

I found the issue in our istio sidecar configmap after BigBang loaded. Our app loads as Kustomizations rather than as a helmrelease and I couldn't figure out how to get the helm values inside of the istio sidecar injector configmap to resolve properly for our RKE2 cluster and applications... so I wrote a script to rewrite that configmap on-the-fly, post BigBang installation, replacing the unresolved helm values with "true" to allow the istio sidecar privilege escalation.

I am well aware that the underlying issue might be that I am not using BigBang properly in order to benefit from customer-provided values that should get resolved in the istio sidecar configmap; however, I couldn't figure out how to do it properly and couldn't find anybody using BigBang with RKE2 in same "locked-down" way that we were in order to hope for an example of how to provide the helm values for replacement in the sidecar configmap.

Symptoms

The observed behavior was that after rewriting the istio sidecar injector configmap and waiting for the BigBang "interval" I could re-create pods and the istio sidecar would launch properly. My script just hardcodes the "30s" sleep below as that is our BigBang flux interval (i.e. chart/values.yaml flux.interval)

The functional containers for the pod remain in Pending state due to the istio-init container failing. You can view the istio-init container logs as such:

kubectl logs {PODNAME} -n {NAMESPACE} -c istio-init

Output (truncated):

Environment:
------------
ENVOY_PORT=
INBOUND_CAPTURE_PORT=

...

-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
COMMIT
iptables-restore --noflush /tmp/iptables-rules-1614812817866327204.txt516366226
iptables-restore v1.6.1: unknown option "--to-ports"
Error occurred at line: 7
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
iptables-save
# Generated by iptables-save v1.6.1 on Wed Mar  3 23:06:57 2021
*raw

Workaround

Here is the rewriting script that removes the helm values and reloads the configmap with hard-coded values for "privileged" and "allowPrivilegeEscalation" ... note that this workaround must be applied before any application pods are created with enough time for fluxCD interval to re-apply the new configMap to any newly created objects. If the fix is applied after the pods are created, the istio sidecar injector already has a broken configmap.

#!/usr/bin/env bash
SAVED_ORIGINAL_CM=istio-configmap-original.yaml
SAVED_PATCHED_CM=istio-configmap-patched.yaml
# Pull the configmap from Kube, revise it per the note at the top of this file ... and reload the result back into Kube:
kubectl -n istio-system get configmap istio-sidecar-injector -o yaml | \
    tee $SAVED_ORIGINAL_CM | \
  sed -E 's,^([[:space:]]+privileged:).*,\1 true,' | \
  sed -E 's,^([[:space:]]+allowPrivilegeEscalation:).*,\1 true,' | \
    tee $SAVED_PATCHED_CM | \
  kubectl apply -f -
printf "[INFO] Necessary sleep to allow for istio sidecar injector configmap to take effect\n"
sleep 30s

REPRO Steps

1. Create RKE2 cluster with non-permissive security policy
2. Install fluxCD 3.a Install any pod security policies required for BigBang (e.g. PSPs to allow gatekeeper to start) 3.b Install BigBang
3. Create application namespace
4. Label application namespace for istio auto-injection
5. Try to launch a pod (deployment, replicaSet or statefulSet)
6. Observe that pod never launches but remains in Pending state
7. Obtain logs from pod's istio-init container

BigBang Version

1.0.0, 1.1.0, 1.2.0