UNCLASSIFIED - NO CUI

Skip to content

Mimir rejecting OTEL formatted metrics

Certain Kyverno metrics have started to been scraped as OTEL format, which Prometheus "supports", however Mimir does not.

# Kyverno metric example
http_server_duration_bucket{http_method="POST",http_scheme="https",http_status_code="200",net_host_name="kyverno-kyverno-svc.kyverno.svc",net_protocol_name="http",net_protocol_version="1.1"
,otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.60.0",le="30"} 2039
# Mimir distributor log
distributor ts=2025-08-22T12:05:25.258831501Z caller=push.go:251 level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been
 partially ingested)" httpCode=400 err="received a series with an invalid label: 'http.method' series: 'kyverno_http_requests_duration_seconds_bucket{container=\"kyverno\", endpoint=\"metri
cs-port\", http.method=\"POST\", http.url=\"/exceptionvalidate?timeout=10s\", instance=\"10.42.2.7:8000\", job=\"kyverno-kyver' (err-mimir-label-invalid)" insight=tr

Potential resolution options include:

  1. There is an experimental feature to allow Mimir to ingest UTF-8 characters (such as "." which is OTEL format) which we can try to activate. More information here https://github.com/grafana/mimir/issues/10283

  2. Another option is seeing if we can configure Prometheus to replace all the dots with underscores with the nameValidationScheme: "Legacy" configuration https://prometheus.io/docs/guides/utf8/

Edited by Brian Jackson