Add Missing Auth Policy for Kiali
Currently on a standard deployment of Big Bang with only core packages enabled we are seeing the following behavior in Kiali:
Further investigation into the logs from the istio-proxy on the Kiali pod shows this:
kubectl logs kiali-79fc9565cf-cdc8p -c istio-proxy -n kiali | grep " 403 " | grep "grafana"
[2024-10-11T13:18:00.681Z] "GET / HTTP/1.1" 403 - via_upstream - "-" 0 19 1 0 "-" "Go-http-client/1.1" "ab217b8d-1a4a-9dbc-b05f-586dca20add0" "monitoring-grafana.monitoring.svc.cluster.local:80" "10.42.2.15:3000" outbound|80||monitoring-grafana.monitoring.svc.cluster.local 10.42.1.24:44086 10.43.219.152:80 10.42.1.24:37464 - default traceID=e5911c3e83147a7ec921d74c0ebb46d4
[2024-10-11T13:19:00.678Z] "GET / HTTP/1.1" 403 - via_upstream - "-" 0 19 1 1 "-" "Go-http-client/1.1" "e48a1812-1eed-968e-a4b3-a555008afdab" "monitoring-grafana.monitoring.svc.cluster.local:80" "10.42.2.15:3000" outbound|80||monitoring-grafana.monitoring.svc.cluster.local 10.42.1.24:44086 10.43.219.152:80 10.42.1.24:60010 - default traceID=341cde863bace5edd385f84be746d953
[2024-10-11T13:20:00.830Z] "GET / HTTP/1.1" 403 - via_upstream - "-" 0 19 4 4 "-" "Go-http-client/1.1" "3ed63493-e560-973a-9bf3-c64853d8252c" "monitoring-grafana.monitoring.svc.cluster.local:80" "10.42.2.15:3000" outbound|80||monitoring-grafana.monitoring.svc.cluster.local 10.42.1.24:44086 10.43.219.152:80 10.42.1.24:57030 - default traceID=63c9b0da745d255137b8cd2533b45084
[2024-10-11T13:20:16.643Z] "GET / HTTP/1.1" 403 - via_upstream - "-" 0 19 1 1 "-" "Go-http-client/1.1" "a55efa78-f75c-900f-b8f9-80e3fc079c12" "monitoring-grafana.monitoring.svc.cluster.local:80" "10.42.2.15:3000" outbound|80||monitoring-grafana.monitoring.svc.cluster.local 10.42.1.24:44086 10.43.219.152:80 10.42.1.24:41298 - default traceID=a659ef705e749305ba7f8f4a699d160a
[2024-10-11T13:22:00.764Z] "GET / HTTP/1.1" 403 - via_upstream - "-" 0 19 2 1 "-" "Go-http-client/1.1" "323d6a11-6fbd-915e-ba6e-42c28940a028" "monitoring-grafana.monitoring.svc.cluster.local:80" "10.42.2.15:3000" outbound|80||monitoring-grafana.monitoring.svc.cluster.local 10.42.1.24:44086 10.43.219.152:80 10.42.1.24:50744 - default traceID=675780948e0df69732b2637f6e11edb7
[2024-10-11T13:23:00.771Z] "GET / HTTP/1.1" 403 - via_upstream - "-" 0 19 2 2 "-" "Go-http-client/1.1" "dbb8a4c6-2d47-9df1-ae8f-87c06fb4399c" "monitoring-grafana.monitoring.svc.cluster.local:80" "10.42.2.15:3000" outbound|80||monitoring-grafana.monitoring.svc.cluster.local 10.42.1.24:44086 10.43.219.152:80 10.42.1.24:60982 - default traceID=60445d6934546ce44a90a81af29cfe4d
Additionally, logs from the istio-proxy on the grafana side revealed the following:
kubectl logs monitoring-grafana-6df894c4dd-hq9bx -c istio-proxy -n monitoring | grep "rbac_access_denied"
[2024-10-11T13:26:16.525Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "fc81fd71-a79b-986e-8840-33616e895b47" "monitoring-grafana.monitoring.svc.cluster.local:80" "-" inbound|3000|| - 10.42.2.15:3000 10.42.1.24:44086 invalid:outbound_.80_._.monitoring-grafana.monitoring.svc.cluster.local default traceID=431886b7b165934b1a9d153c70b47288
[2024-10-11T13:27:16.693Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "91f9971c-8dc7-9cfa-b7ba-cd0758db6b5f" "monitoring-grafana.monitoring.svc.cluster.local:80" "-" inbound|3000|| - 10.42.2.15:3000 10.42.1.24:44086 invalid:outbound_.80_._.monitoring-grafana.monitoring.svc.cluster.local default traceID=eae2d015ae07d941d08801b41bb2be10
[2024-10-11T13:29:00.871Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "6e58ebab-58ec-9d03-98df-152491bc5ce3" "monitoring-grafana.monitoring.svc.cluster.local:80" "-" inbound|3000|| - 10.42.2.15:3000 10.42.1.24:44086 invalid:outbound_.80_._.monitoring-grafana.monitoring.svc.cluster.local default traceID=b1ab58a9072aa6613ff30036293cc846
[2024-10-11T13:29:15.590Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "cc25e3b0-6ce8-99c5-b219-c792f18e0791" "monitoring-grafana.monitoring.svc.cluster.local:80" "-" inbound|3000|| - 10.42.2.15:3000 10.42.1.24:44086 invalid:outbound_.80_._.monitoring-grafana.monitoring.svc.cluster.local default traceID=b9ec3d3f429ff11f3dfdc1a8cbcf7c14
[2024-10-11T13:29:46.869Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "257946c6-4abd-9069-b931-d79490aa5180" "monitoring-grafana.monitoring.svc.cluster.local:80" "-" inbound|3000|| - 10.42.2.15:3000 10.42.1.24:44086 invalid:outbound_.80_._.monitoring-grafana.monitoring.svc.cluster.local default traceID=9e81989641e4fee26f0aebac6ef857cd
[2024-10-11T13:30:55.872Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "b2b2c08b-b699-9a1c-8578-25cabc6fdbad" "monitoring-grafana.monitoring.svc.cluster.local:80" "-" inbound|3000|| - 10.42.2.15:3000 10.42.1.24:44086 invalid:outbound_.80_._.monitoring-grafana.monitoring.svc.cluster.local default traceID=279b3ddcc1cbbc9114ac7de36cffdf5b
[2024-10-11T13:31:55.870Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "Go-http-client/1.1" "ba97a96d-4764-9163-aa9c-c4184ca79114" "monitoring-grafana.monitoring.svc.cluster.local:80" "-" inbound|3000|| - 10.42.2.15:3000 10.42.1.24:44086 invalid:outbound_.80_._.monitoring-grafana.monitoring.svc.cluster.local default traceID=12a85e0da31b849d7bde11e92bb2c412
Manually adding an authorization policy to allow communication from kiali to grafana resolved the issue so we'll need to get this added.
Edited by Jimmy Bourque