Add Missing Auth Policy for Kiali
Currently on a standard deployment of Big Bang with only core packages enabled we are seeing the following behavior in Kiali:
Further investigation into the logs from the istio-proxy on the Kiali pod shows this:
kubectl logs kiali-79fc9565cf-cdc8p -c istio-proxy -n kiali | grep " 403 " | grep "tempo"
[2024-10-11T13:20:00.826Z] "GET /api/services HTTP/1.1" 403 - via_upstream - "-" 0 19 4 2 "-" "Go-http-client/1.1" "3526cf55-3bd6-9314-b5cb-ae23280595e8" "tempo-tempo.tempo.svc.cluster.local:16686" "10.42.0.12:16686" outbound|16686||tempo-tempo.tempo.svc.cluster.local 10.42.1.24:49348 10.43.217.17:16686 10.42.1.24:37526 - default traceID=62ae0e17b110937b11bea9979e359882
[2024-10-11T13:20:16.665Z] "GET /api/services HTTP/1.1" 403 - via_upstream - "-" 0 19 45 45 "-" "Go-http-client/1.1" "b2baa0f8-85d3-9ac8-ac72-787dd1a3cdd2" "tempo-tempo.tempo.svc.cluster.local:16686" "10.42.0.12:16686" outbound|16686||tempo-tempo.tempo.svc.cluster.local 10.42.1.24:49348 10.43.217.17:16686 10.42.1.24:48664 - default traceID=d117ddb9b66b14b693c5faede372febc
[2024-10-11T13:22:00.763Z] "GET /api/services HTTP/1.1" 403 - via_upstream - "-" 0 19 1 1 "-" "Go-http-client/1.1" "be8acbce-f6e1-9235-8bcf-1b826cfcf90f" "tempo-tempo.tempo.svc.cluster.local:16686" "10.42.0.12:16686" outbound|16686||tempo-tempo.tempo.svc.cluster.local 10.42.1.24:49348 10.43.217.17:16686 10.42.1.24:60378 - default traceID=6c9a6506dabd3473054b091c7d6ca2db
[2024-10-11T13:23:00.770Z] "GET /api/services HTTP/1.1" 403 - via_upstream - "-" 0 19 3 2 "-" "Go-http-client/1.1" "138630fd-ab20-95a5-8df2-eff2f4fccb4c" "tempo-tempo.tempo.svc.cluster.local:16686" "10.42.0.12:16686" outbound|16686||tempo-tempo.tempo.svc.cluster.local 10.42.1.24:49348 10.43.217.17:16686 10.42.1.24:36196 - default traceID=60c997a1fd86c4cbd67bd06ddc1c20d3
[2024-10-11T13:24:00.763Z] "GET /api/services HTTP/1.1" 403 - via_upstream - "-" 0 19 1 1 "-" "Go-http-client/1.1" "701ce490-85f1-9688-8537-ede80ca3a931" "tempo-tempo.tempo.svc.cluster.local:16686" "10.42.0.12:16686" outbound|16686||tempo-tempo.tempo.svc.cluster.local 10.42.1.24:49348 10.43.217.17:16686 10.42.1.24:43400 - default traceID=ae375c4a25751aec30b9b68d450572ce
[2024-10-11T13:25:00.699Z] "GET /api/services HTTP/1.1" 403 - via_upstream - "-" 0 19 2 1 "-" "Go-http-client/1.1" "6decec16-db6e-95d5-9e52-be291736d0fa" "tempo-tempo.tempo.svc.cluster.local:16686" "10.42.0.12:16686" outbound|16686||tempo-tempo.tempo.svc.cluster.local 10.42.1.24:49348 10.43.217.17:16686 10.42.1.24:34686 - default traceID=969c8fdfeb235c78e49502043e724f82
Additionally, logs from the istio-proxy on the tempo side revealed the following:
kubectl logs tempo-tempo-0 -c istio-proxy -n tempo | grep "rbac_access_denied"
[2024-10-11T13:32:55.812Z] "GET /api/services HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "fb0e01aa-d65d-9b9f-822a-186bfb9b6bb1" "tempo-tempo.tempo.svc.cluster.local:16686" "-" inbound|16686|| - 10.42.0.12:16686 10.42.1.24:49348 invalid:outbound_.16686_._.tempo-tempo.tempo.svc.cluster.local default traceID=dac91c72b2e7609927bee8a3a6483eb0
[2024-10-11T13:33:55.776Z] "GET /api/services HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "82597e1b-0d8c-93b8-8b7b-2991ac9f9012" "tempo-tempo.tempo.svc.cluster.local:16686" "-" inbound|16686|| - 10.42.0.12:16686 10.42.1.24:49348 invalid:outbound_.16686_._.tempo-tempo.tempo.svc.cluster.local default traceID=4565711c8a672d8fc8fab31a589ddd7b
[2024-10-11T13:35:01.163Z] "GET /api/services HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "0a67a2b4-1b13-92fa-835a-8749b32cf493" "tempo-tempo.tempo.svc.cluster.local:16686" "-" inbound|16686|| - 10.42.0.12:16686 10.42.1.24:49348 invalid:outbound_.16686_._.tempo-tempo.tempo.svc.cluster.local default traceID=fe8f516ec26e0b0799b2fa2e687c41d0
[2024-10-11T13:35:54.830Z] "GET /api/services HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "a7f7b5e0-118b-9a17-9a19-76338bbaffc0" "tempo-tempo.tempo.svc.cluster.local:16686" "-" inbound|16686|| - 10.42.0.12:16686 10.42.1.24:49348 invalid:outbound_.16686_._.tempo-tempo.tempo.svc.cluster.local default traceID=612a550cb227557770944f30829a9307
[2024-10-11T13:35:56.146Z] "GET /api/services HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "c52a84ae-1226-9b94-943c-d451eb4519ee" "tempo-tempo.tempo.svc.cluster.local:16686" "-" inbound|16686|| - 10.42.0.12:16686 10.42.1.24:49348 invalid:outbound_.16686_._.tempo-tempo.tempo.svc.cluster.local default traceID=98bdd58f26c0f89c57056bd77ea2183e
[2024-10-11T13:36:04.643Z] "GET /api/services HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "e0c8df5c-d198-9d51-ba2d-acb57f73b4d1" "tempo-tempo.tempo.svc.cluster.local:16686" "-" inbound|16686|| - 10.42.0.12:16686 10.42.1.24:49348 invalid:outbound_.16686_._.tempo-tempo.tempo.svc.cluster.local default traceID=23b20c34df1b462c99fdea5480d0daa7
[2024-10-11T13:37:07.692Z] "GET /api/services HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "ebbdb39d-e90b-9867-9793-88ee20469f9d" "tempo-tempo.tempo.svc.cluster.local:16686" "-" inbound|16686|| - 10.42.0.12:16686 10.42.1.24:49348 invalid:outbound_.16686_._.tempo-tempo.tempo.svc.cluster.local default traceID=c60af88e994798e4f8085575e7445dd4
Manually adding an authorization policy to allow communication from kiali to tempo resolved the issue so we'll need to get this added.
Edited by Jimmy Bourque