Traces do not appear for some services in Kiali

When using Kiali to view traces stored in Tempo, no traces appear for the majority of services.

This appears to be an issue due to a naming mismatch between Istio (and thus Kiali) and Tempo.

Tempo uses the literal service name to query for traces (so for instance monitoring-grafana.monitoring), where as Kiali uses the Istio label service.istio.io/canonical-name: grafana.

This can be observed by navigating to the Kiali UI and loading the traces tab for Grafana. No traces appear and upon request inspection you can it is using <canonical-name.namespace> (grafana.monitoring) to look up traces.

Compare that with what is available by directly querying through Grafana using the service name (monitoring-grafana.monitoring):

This can be fixed by manually adding podLabels to override istio's inferred canonical name with something that aligns with what is stored in tempo:

grafana:
  enabled: true

  values:
    podLabels:
      service.istio.io/canonical-name: monitoring-grafana

A more robust solution would likely only be possible after we shift trace collection to Alloy - see #2520.

Note: Some traces are visible (like Kiali itself) but this is a coincidence because the names end up being the same - kiali.kiali.