Bigbang 2048 enable native sidecars

General MR

Summary

Enabling native sidecars

Relevant logs/screenshots

n/a

Linked Issue

big-bang/bigbang#2048 (closed)

Upgrade Notices

This prefers an upgrade to K8s 1.29 for native sidecars. This gets rid of the istioproxy container and instead builds it into the existing container. This allows jobs to exit gracefully (rather than staying alive forever because istioproxy won't exit), and allows init containers to run because envoy is built into each of them rather than coming up after init containers would run. This requires .Values.values.pilot.env contains {"ENABLE_NATIVE_SIDECARS": true}, so if you are currently passing values there, make sure to add this to them.

You can run this on 1.28 by enabling the SidecarContainers feature gate.

When upgrading, if desired, you can, but don't have to, force the switch to init containers immediately by restarting all of your istio-supporting pods, e.g.

for ns in $(kubectl get ns -l app.kubernetes.io/part-of=bigbang,istio-injection=enabled --no-headers -o custom-columns=":metadata.name"); do
  kubectl rollout restart deployment -n $ns
  kubectl rollout restart statefulset -n $ns
  kubectl rollout restart daemonset -n $ns
done

If you don't do this immediately, they will switch to init containers on their next restart, but this should be backwards compatible with the istioproxy sidecar containers.