[P1BIGROCKS-1363] High Availability
[P1BIGROCKS-1363](https://jira.il2.dso.mil/browse/P1BIGROCKS-1363)
Big Bang not only needs to be built so each component can be provided in a highly available deployment, but it also needs to help facilitate the creation of Highly Available software in a DSOP configuration.
- [ ] Provide standardized documentation on how to configure each application for High Availability. Include considerations for running across availability zones inside of AWS as well as knowledge about other single points of failure
- [ ] Provide standardized document for other production considerations when working with a package. e.g. How should data be stored?
- [ ] Do all tools support HA? What about Gitlab Gitaly?
Design a Chaos Engineering test that demonstrates how BB packages can survive. How/When should this be run? How do we make sure we haven't lost anything?
- [ ] Leverage a Chaos testing tool to test and ensure each component inside of Big Bang can survive
- [ ] Provide Chaos testing (Gremlin, Kube Chaos, Chaos Kube, Kube Monkey, Chaos Monkey, etc. , etc.) as a Big Bang service
- [ ] Integrate Chaos testing tool into CICD pipelines
epic