elasticsearch kibana gets stuck upgrading from 1.18.0-bb.2 to 1.18.0-bb.3
Traffic to elasticsearch from kibana has an error which indicates the tls certificate issue. See logs from kibana logging-ek-kb pod. Solution should allow for kibana to communicate with elasticsearch with istio injection and without.
The error is caused by upgrading elasticsearch kibana release 1.18.0-bb.2 to 1.18.0-bb.3 the elasticsearch pods get stuck during the upgrade causing a mismatch between the mtls settings between elasticsearch and kibana traffic encryption which causes the elasticsearch health to be in an unknown state. Changing the title to reflect the stuck pending state, currently speculating that it might due to the combination Update Strategy set to Rolling Update and Max Unavailable set to 1.
[2024-09-17T18:36:53.413Z] "- - -" 0 UF,URX - - "TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end" 0 0 4 - "-" "-" "-" "-" "10.42.0.18:9200" outbound|9200||logging-ek-es-http.logging.svc.cluster.local - 10.43.76.101:9200 10.42.2.13:42536 - -