Create prometheus/grafana alert for high sidecar memory
One cluster that had a large sidecar configuration file due to importing all service discovery across the cluster saw slow sidecar startup times as this large config was transferred. The team was able to identify this as the issue as the memory in the sidecar was over 40MB-50MB instead of in the 10MB range.
The monitor should warn at the 40-50MB and potentially alert when getting over 100MB. This may also help identify warnings if sidecar performance becomes an issue again.
AC:
- Create
additionalPrometheusRule
values to propagate down to the monitoring package if istio is enabled in BigBang to satisfy the above alerts. Can use these Rules created by the monitoirng package To build the expression