Mimir rejecting OTEL formatted metrics
Certain Kyverno metrics have started to been scraped as OTEL format, which Prometheus "supports", however Mimir does not.
# Kyverno metric example
http_server_duration_bucket{http_method="POST",http_scheme="https",http_status_code="200",net_host_name="kyverno-kyverno-svc.kyverno.svc",net_protocol_name="http",net_protocol_version="1.1"
,otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.60.0",le="30"} 2039
# Mimir distributor log
distributor ts=2025-08-22T12:05:25.258831501Z caller=push.go:251 level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been
partially ingested)" httpCode=400 err="received a series with an invalid label: 'http.method' series: 'kyverno_http_requests_duration_seconds_bucket{container=\"kyverno\", endpoint=\"metri
cs-port\", http.method=\"POST\", http.url=\"/exceptionvalidate?timeout=10s\", instance=\"10.42.2.7:8000\", job=\"kyverno-kyver' (err-mimir-label-invalid)" insight=tr
Potential resolution options include:
-
There is an experimental feature to allow Mimir to ingest UTF-8 characters (such as "." which is OTEL format) which we can try to activate. More information here https://github.com/grafana/mimir/issues/10283
- NOTE: This feature is not yet in a release, looks like it will be added in 2.18.0 - Keep an eye on https://github.com/grafana/mimir/releases/
-
Another option is seeing if we can configure Prometheus to replace all the dots with underscores with the
nameValidationScheme: "Legacy"
configuration https://prometheus.io/docs/guides/utf8/
Edited by Brian Jackson