Enable Istio mTLS for Thanos

By default, when istio is deployed, it's deployed with a PERMISSIVE mode that allows an istio injected pod to talk to any non-istio pod without mutual TLS.

Istio uses PeerAuthentication to enforce mTLS at the mesh level and can be applied either at the namespace level, which applies to all pods in the namespace, or at a global level when the PeerAuthentication is applied to the istio-system namespace.

1. Create the PeerAuthentication in the {{ .Release.Namespace }} namespace (NOTE: this should be added to the package repo).

   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: default-{package}
     namespace: {{ .Release.Namespace }}
   spec:
     mtls:
       mode: {{ .Values.istio.mtls.mode }}

   If other packages deploy in the same namespace, add a selector so that the STRICT policy only applies to pods from this package.

   Set istio.mtls.mode to STRICT by default in chart/values.yaml. Add a comment above the value that it can be set to PERMISSIVE or STRICT.

   Add conditionals to the PeerAuthentication objects so that they are only created when .Values.istio.enabled is true inside each package chart.

2. Test the functionality of the package

3. When there are issues, add an exception policy (add a conditional on istio.enabled and mtls.mode = STRICT)

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: <package>-<exception description>-exception
  namespace: {{ .Release.Namespace }}
spec:
  selector:
    matchLabels:
      exception: label-here
  mtls:
    mode: {{ .Values.istio.mtls.mode }}
  portLevelMtls:
    "9000": # port number here, in quotes
      mode: PERMISSIVE

4. iterate on 2) and 3)