SKIP UPGRADE elasticsearchKibana update to 1.18.0-bb.3

Package Merge Request

Package Changes

https://repo1.dso.mil/big-bang/product/packages/elasticsearch-kibana/-/blob/1.18.0-bb.3/CHANGELOG.md

Package MR

big-bang/product/packages/elasticsearch-kibana!295 (merged)

For Issue

Closes big-bang/product/packages/elasticsearch-kibana#204 (closed)

Upgrade Notices

Elasticsearch Statefulsets Update Strategy are set to OnDelete, so in order for the changes in this package mr to take effect (i.e tunneling certificates/mtls handling through istio-proxy) 1) istio must be enabled and 2) the elasticsearch pods need to be manually deleted (for more information see kubernetes documentation on statefulsets updates strategies).

kubectl instructions

To get a list of pods that will need to be manually restarted run the following

kubectl get pod -n logging -l elasticsearch.k8s.elastic.co/cluster-name=logging-ek

If availability is important than you can manually do a rolling upgrade by deleting the pods one by one while waiting for them to restart kubectl delete pod <name of pods> -n logging

If availability is not important and you want to restart all the pods at once then run the following

kubectl delete pod -n logging -l elasticsearch.k8s.elastic.co/cluster-name=logging-ek

After restarting all the pods and confirming that the pods are back in a ready state, check that elasticsearch cluster health is green by running

kubectl get elasticsearch -n logging

added botmr elasticsearchKibana fluentbit statusdoing labels

assigned to @kliu

changed the description

requested review from @chris.oconnell, @michaelmartin, and @andrewshoell

changed the description

changed milestone to %2.36.0

added debug label

removed statusdoing label

added statusblocked label

Elasticsearch-kibana gets stuck during the upgrade from 1.18.0-bb.2 to 1.18.0-bb.3, and issue has been open to track: big-bang/product/packages/elasticsearch-kibana#207 (closed)

The error is caused by upgrading elasticsearch kibana release 1.18.0-bb.2 to 1.18.0-bb.3 the elasticsearch pods get stuck during the upgrade causing a mismatch between the mtls settings between elasticsearch and kibana traffic encryption which causes the elasticsearch health to be in an unknown state. Changing the title to reflect the stuck pending state, currently speculating that it might due to the combination Update Strategy set to Rolling Update and Max Unavailable set to 1.

changed milestone to %2.37.0

changed the description

mentioned in issue big-bang/product/packages/elasticsearch-kibana#207 (closed)

marked this merge request as ready

removed statusblocked label

added statusreview label

@andrewshoell, @chris.oconnell, @michaelmartin the Elasticsearch Statefulsets Update Strategy are set to OnDelete, so the smoke test for the upgrade will fail(elasticsearch health status is unknown) because it is stuck and requires manual intervention in the form of a delete to upgrade, but the clean install will pass (see same test but pass) because the changes in this package mr to take effect requires the elasticsearch pods need to be manually deleted on existing elasticsearch pods. Is there any procedure that needs to be done/documented before considering this mr for review?

I have updated the Upgrade Notice in this MR to reflect the need for manual intervention in the form of deleting the pods.

marked this merge request as draft