Mimir Prometheus Remote Write Out of Order Errors

After 13-15 minutes the remote write for Prometheus fails and stops with the following error

.08717221Z caller=grpc_logging.go:76 level=warn method=/cortex.Ingester/Push duration=7.752013ms msg=gRPC err="user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and this sample is beyond the out-of-order time window of 10s (err-mimir-sample-timestamp-too-old). The affected sample has timestamp 2024-12-06T20:17:59.898Z and is from series workload:istio_requests_total{connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_canonical_revision=\"2.14.0\", destination_canonical_service=\"mimir\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/monitoring/sa/monitoring-mimir\", destination_service=\"monitoring-mimir-nginx.monitoring.svc.cluster.local\", destination_service_name=\"monitoring-mimir-nginx\", destination_service_namespace=\"monitoring\", destination_workload=\"monitoring-mimir-nginx\", destination_workload_namespace=\"monitoring\", job=\"monitoring/monitoring-monitoring-kube-istio-envoy\", pod_name=\"monitoring-mimir-nginx-6bcd5cbc4f-9rxb2\", prometheus=\"monitoring/monitoring-monitoring-kube-prometheus\", prometheus_replica=\"prometheus-monitoring-monitoring-kube-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"500\", response_flags=\"-\", source_app=\"unknown\", source_canonical_revision=\"latest\", source_canonical_service=\"unknown\", source_cluster=\"unknown\", source_principal=\"spiffe://cluster.local/ns/monitoring/sa/monitoring-monitoring-kube-prometheus\", source_version=\"unknown\", source_workload=\"unknown\", source_workload_namespace=\"monitoring\"} (sampled 1/10)"

Grafana Dashboard for Remote Write Prometheus to Mimir shows writes stops at 13/15