Setting recommended ingester settings from grafana Github, cleaning up values, adding queryUI templates

Noticing some slight inconsistencies between traceIDs showing up attached to Loki logs and actually being available to view in Tempo.

• Addressing those reliability issues with an NP to allow ingress on tempo:9411 in namespace (to recieve traces from each apps istio-proxy sidecars).

• Enabling zipkin endpoint listener on 0.0.0.0:9411

• Enabling searching via tempoQuery (so it's actually used) which turns out to be JaegerUI itself.

• Adding in values/templates/networkPolicies to facilitate external connections through istio to tempoQuery(Jaeger) UI via https://tracing.bigbang.dev

• Cleaning up comments in values file that we control

Reviewing their documentation pages and recommended configuration on Github, making sure our helm chart makes use of their recommendations.

exporting of traces is a lot better with this configuration, but still seeing magnitudes more of traces in Loki compared to Tempo.

chart/templates/bigbang/networkpolicies/ingress-istio-ingressgateway.yml

+{{- if and .Values.networkPolicies.enabled .Values.istio.enabled }}
+# Allow traffic from the IngressGateway to the tempo query UI
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
+metadata:
+  name: ingress-tempo-istio-igw
+  namespace:  {{ .Release.Namespace }}
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: tempo
+  ingress:
+  - ports:
+    - port: 16686
+      protocol: TCP
+    from:
+    - podSelector:
+        matchLabels:
+          {{- toYaml .Values.networkPolicies.ingressLabels | nindent 10}}
+      namespaceSelector:
+        matchLabels:
+          app.kubernetes.io/name: istio-controlplane
+{{- end }}