Loki Istio Hardening Prevents Jaeger From Starting When ElasticSearch Is Enabled

Commit big-bang/bigbang!3705 (merged) enabled istio hardening for loki. This installs AuthorizationPolicy which prevents jaeger from connecting to the logging-ek-es-http service.

assigned to @andrewshoell

added to epic big-bang&159 (closed)

added jaeger loki monitoring labels

changed title from Loki Istio Hardening Prevent Jaeger From Starting to Loki Istio Hardening Prevents Jaeger From Starting

assigned to @andrewshoell

added kindbug priority2 labels

added statusdoing label

moved from big-bang/bigbang#1923 (moved)

This description is written backwards. If that commit is causing the issues, then it would be because other things need to access jaeger and can't any more. This is because Authorization Policies limit access from other places in to where they are installed, e.g. authorization policies installed in the jaeger namespace would limit/grant access for other processes to call into the jaeger namespace.

Given that I attempted to install with and without the authorization policies and in both cases it worked. I'm not sure how to reproduce this issue or what exactly the issue is. @michaelmartin can you please provide more details?

It seems it's what's causing our nightly all-packages pipelines to fail, jaeger fails to install. It does need to talk to eckOperator and ElasticsearchKibana in the logging namespace placed there alongside loki.

https://repo1.dso.mil/big-bang/bigbang/-/jobs/30815033

more detail was given in chat, so I think we're all on the same page--copied here:

I think the key is we need to also enable elasticsearchKibana -- then a jaeger-es-index-templates-xxxx fires off and tries to hit https://logging-ek-es-http.logging.svc:9200 and gets blocked, so the helm release doesn't install.

I saw there is this which would open things up -- elasticsearch-kibana/.../authorization-policies/allow-jaeger.yaml -- but if loki is hardened and jaeger is not hardened, then the logging namespace is essentially hardened . Seems we might need to fix that logic ?

I think we'll want to document too how we handle the relationships of package hardening -- in this case where multiple packages may be running in the same namespace. e.g. does/should hardening loki also "auto-harden" jaeger/mattermost/etc. in order for the required authorization-policies to get installed into the logging namespace via the elasticsearch-kibana/.../authorization-policies/allow-xxx policies ?

or ... should I be able to independently harden loki, but not harden jaeger ?? A more difficult scenario. Just throwing ideas out there, and then we'll just want to make sure the chosen implementation for now is well understood/documented.

set weight to 1

added teamObservability label

changed iteration to Big Bang Iterations Jan 23, 2024 - Feb 5, 2024

set weight to 2

changed title from Loki Istio Hardening Prevents Jaeger From Starting to Loki Istio Hardening Prevents Jaeger From Starting When ElasticSearch Is Enabled

marked this issue as related to elasticsearch-kibana#93 (closed)

the work was done in this issue.

still need to turn on eck in bb, this should be moved to eck.

closed

moved to elasticsearch-kibana#102 (closed)