PROMETHEUS-3.14.0 release notes and known issues

Summary

[CHANGE] API: Deprecate the stats query parameter of /api/v1/query and /api/v1/query_range for values other than true and all . Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 [CHANGE] API: /api/v1/status/config now correctly shows separator: "" and replacement: "" in relabel configs when explicitly set to empty, instead of omitting them. #18653 [CHANGE] Discovery/Hetzner: Drop the __meta_hetzner_datacenter label for hcloud targets, following its removal from the Hetzner Cloud API. #19269 [CHANGE] PromQL: Enable duration expressions by default. The promql-duration-expr feature flag is now a no-op. #19033 [CHANGE] PromQL: Promote first_over_time to stable. It no longer requires the promql-experimental-functions feature flag. #19093 [FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery ( oci_sd_configs ). #18919 [FEATURE] PromQL: Add experimental start_timestamp(instant-vector) function returning the start timestamp of each sample in the given vector. Requires the use-start-timestamps feature flag. #19089 [FEATURE] PromQL: Allow rate() and increase() to use start timestamps as an alternative for rate extrapolation. Hidden behind the use-start-timestamps feature flag. #18619 [FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the histograms-st-encoding feature flag. #18609 [ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g. k8s.pod.name and k8s_pod_name both become k8s_pod_name ), and expose the prometheus_api_otlp_translation_warnings_total counter labelled by category to track such warnings. #18957 [ENHANCEMENT] Promtool: Add --remote-write.path flag to push metrics for backends that use a non-default remote-write endpoint. #19086 [ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903 [ENHANCEMENT] TSDB: Add prometheus_tsdb_head_native_histogram_series and prometheus_tsdb_head_native_histogram_buckets gauges tracking the number of native histogram series and buckets in the head. #19170 [ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions ( step() , range() , min_of() , max_of() ) in range selectors and subqueries. #18625 [ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706 [ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181 [PERF] Speed up regex label matchers matching a set of literal values (e.g. {job=~"foo|bar|baz"} ). #18833 [PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470 [PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143 [PERF] Scrape: Reduce native histogram scrape parsing allocations by ~49%. #19282 [PERF] TSDB: Speed up queries on series with many in-memory chunks. #18300 [BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859 [BUGFIX] Discovery/AWS: Stop promtool check config from making AWS metadata service (IMDS) network calls when the region field is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037 [BUGFIX] Discovery/Docker: Set a request timeout for docker_sd and dockerswarm_sd on unix , npipe , and tcp hosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237 [BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102 [BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778 [BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167 [BUGFIX] P

Improvements and security content

• [CHANGE] API: Deprecate the stats query parameter of /api/v1/query and /api/v1/query_range for values other than true and all . Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 [CHANGE] API: /api/v1/status/config now correctly shows separator: "" and replacement: "" in relabel configs when explicitly set to empty, instead of omitting them. #18653 [CHANGE] Discovery/Hetzner: Drop the __meta_hetzner_datacenter label for hcloud targets, following its removal from the Hetzner Cloud API. #19269 [CHANGE] PromQL: Enable duration expressions by default. The promql-duration-expr feature flag is now a no-op. #19033 [CHANGE] PromQL: Promote first_over_time to stable. It no longer requires the promql-experimental-functions feature flag. #19093 [FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery ( oci_sd_configs ). #18919 [FEATURE] PromQL: Add experimental start_timestamp(instant-vector) function returning the start timestamp of each sample in the given vector. Requires the use-start-timestamps feature flag. #19089 [FEATURE] PromQL: Allow rate() and increase() to use start timestamps as an alternative for rate extrapolation. Hidden behind the use-start-timestamps feature flag. #18619 [FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the histograms-st-encoding feature flag. #18609 [ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g. k8s.pod.name and k8s_pod_name both become k8s_pod_name ), and expose the prometheus_api_otlp_translation_warnings_total counter labelled by category to track such warnings. #18957 [ENHANCEMENT] Promtool: Add --remote-write.path flag to push metrics for backends that use a non-default remote-write endpoint. #19086 [ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903 [ENHANCEMENT] TSDB: Add prometheus_tsdb_head_native_histogram_series and prometheus_tsdb_head_native_histogram_buckets gauges tracking the number of native histogram series and buckets in the head. #19170 [ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions ( step() , range() , min_of() , max_of() ) in range selectors and subqueries. #18625 [ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706 [ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181 [PERF] Speed up regex label matchers matching a set of literal values (e.g. {job=~"foo|bar|baz"} ). #18833 [PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470 [PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143 [PERF] Scrape: Reduce native histogram scrape parsing allocations by ~49%. #19282 [PERF] TSDB: Speed up queries on series with many in-memory chunks. #18300 [BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859 [BUGFIX] Discovery/AWS: Stop promtool check config from making AWS metadata service (IMDS) network calls when the region field is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037 [BUGFIX] Discovery/Docker: Set a request timeout for docker_sd and dockerswarm_sd on unix , npipe , and tcp hosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237 [BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102 [BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778 [BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167 [BUGFIX] P

Affected products and versions

Recommended action

Review the official publisher document before deployment.

Official publisher evidence