Elastic8 upgrade job
For https://repo1.dso.mil/platform-one/big-bang/apps/core/elasticsearch-kibana/-/issues/53
Two main changes:
- disabled the the operator handling of index template creation (since this does not work for ES8) and overrode the "version" in the Jaeger ES spec to set to 7
- added a job that will hit the Elastic API to create the necessary (ES8 compatible) index templates for Jaeger
Tested 3 scenarios:
- clean install with the BB branch
brettcharrier-master-patch-31618
(ES8 + this jaeger update): Worked cleanly, all functioning. - upgrade from BB branch
master
tobrettcharrier-master-patch-31618
: Worked cleanly, traces persisted. - clean install of BB branch
master
with this Jaeger branch (ES7 + this jaeger update): Worked cleanly, all functioning.
Definitely some caveats here:
- No guarantee when this job might break - we will want to test it on every ES update in case the index template and/or API endpoints change.
- ES index cleaner/rollover I couldn't test due to a mix of network policy issues and cert issues - it seems like these jobs don't even currently work (on ES7 due to BB changes). Might be worth further evaluation or asking customers if they are using these features?
- Unlikely, but customer may be required to override the default priority values to prevent overlap. We do provide helm values to customize this and the job will output a clear error if this is the issue - in my opinion this makes is a minor issue and only warrants a release note callout/documentation note?
I think knowing that we plan to switch to Tempo as the default (and maybe remove Jaeger entirely?) these caveats are acceptable. Job breaking is just something we need to keep in mind. Priority override for customers is something we should notify about, but nothing more should be required. The index cleaner/rollover are the only ones that might need more of a look but I couldn't get these functioning regardless of ES version.
Edited by Micah Nagel