Grafana Alloy Updates & Release Notes

Breaking changes due to bugfixes in Prometheus exporters

The prometheus.exporter.blackbox, prometheus.exporter.snmp, and prometheus.exporter.statsd components now use the component ID instead of the hostname as their instance label in their exported metrics. This is a consequence of a bug fix that could lead to missing data when using the exporter with clustering.

If you would like to retain the previous behaviour, you can use discovery.relabel with action = "replace" rule to set the instance label to sys.env("HOSTNAME").

Breaking changes due to major version upgrade of Prometheus

Prometheus dependency had a major version upgrade from v2.55.1 to v3.4.2.

• The . pattern in regular expressions in PromQL matches newline characters now. With this change a regular expressions like .* matches strings that include \n. This applies to matchers in queries and relabel configs in Prometheus and Loki components.

• The enable_http2 in prometheus.remote_write component’s endpoints has been changed to false by default. Previously, in Prometheus v2 the remote write http client would default to use http2. In order to parallelize multiple remote write queues across multiple sockets its preferable to not default to http2. If you prefer to use http2 for remote write you must now set enable_http2 to true in your prometheus.remote_write endpoints configuration section.

• The experimental CLI flag --feature.prometheus.metric-validation-scheme has been deprecated and has no effect. You can configure the metric validation scheme individually for each prometheus.scrape component.

• Log message format has changed for some of the prometheus.* components as part of the upgrade to Prometheus v3.

• The values of the le label of classic histograms and the quantile label of summaries are now normalized upon ingestion. In previous Alloy versions, that used Prometheus v2, the value of these labels depended on the scrape protocol (protobuf vs text format) in some situations. This led to label values changing based on the scrape protocol. E.g. a metric exposed as my_classic_hist{le="1"} would be ingested as my_classic_hist{le="1"} via the text format, but as my_classic_hist{le="1.0"} via protobuf. This changed the identity of the metric and caused problems when querying the metric. In current Alloy release, which uses Prometheus v3, these label values will always be normalized to a float like representation. I.e. the above example will always result in my_classic_hist{le="1.0"} being ingested into Prometheus, no matter via which protocol. The effect of this change is that alerts, recording rules and dashboards that directly reference label values as whole numbers such as le="1" will stop working.

The recommended way to deal with this change is to fix references to integer le and quantile label values, but otherwise do nothing and accept that some queries that span the transition time will produce inaccurate or unexpected results.

See the upstream Prometheus v3 migration guide for more details.

Breaking changes in prometheus.scrape

scrape_native_histograms attribute for prometheus.scrape is now set to false, whereas in previous versions of Alloy it would default to true. This means that it is no longer enough to just configure scrape_protocols to start with PrometheusProto to scrape native histograms - scrape_native_histograms has to be enabled. If scrape_native_histograms is enabled, scrape_protocols will automatically be configured correctly for you to include PrometheusProto. If you configure it explicitly, Alloy will validate that PrometheusProto is in the scrape_protocols list.

In previous versions of Alloy configuring scrape_protocols to start with PrometheusProto was enough to start scraping native histograms because scrape_native_histogram defaulted to true:

Alloy prometheus.scrape "scrape" { scrape_protocols = ["PrometheusProto"] }

Now it has to be enabled and scrape_protocols can be omitted:

Alloy prometheus.scrape "scrape" { scrape_native_histograms = true }

Breaking changes in prometheus.exporter.windows

As the windows_exporter continues to be refactored upstream, there are various breaking changes in metrics.

• windows_process_start_time -> windows_process_start_time_seconds_timestamp in the process collector.
• windows_time_clock_frequency_adjustment_ppb_total -> windows_time_clock_frequency_adjustment_ppb in the time collector.
• windows_net_nic_info -> windows_net_nic_address_info in net collector.
• windows_system_boot_time_timestamp_seconds -> windows_system_boot_time_timestamp in system collector.
• windows_os_physical_memory_free_bytes -> windows_memory_physical_free_bytes from os collector -> memory collector.
• windows_os_process_memory_limit_bytes -> windows_memory_process_memory_limit_bytes from os collector -> memory collector.
• windows_os_processes -> windows_system_processes from os collector -> system collector.
• windows_os_processes_limit -> windows_system_process_limit from os collector -> system collector.
• windows_os_time -> windows_time_current_timestamp_seconds from os collector -> time collector.
• windows_os_timezone -> windows_time_timezone from os collector -> time collector.
• windows_os_users from os collector can be reconstructed by aggregating windows_terminal_services_session_info{state="active"} in terminal_services collector.
• windows_os_virtual_memory_bytes -> windows_memory_commit_limit from os collector -> memory collector.
• windows_os_virtual_memory_free_bytes from os collector can be reconstructed by subtracting windows_memory_committed_bytes from windows_memory_commit_limit in memory collector.
• windows_os_visible_memory_bytes -> windows_memory_physical_total_bytes from os collector -> memory collector.

Deprecated collectors have been removed. Configuration of these collectors will be allowed in Alloy for at least one minor version, but will have no effect.

• logon collector removed, use the terminal_services collector instead.
• cs collector removed, use the os, memory, or cpu collectors instead.

Refer to the release notes for the windows_exporter version v0.31.0 for the breaking change details.

Breaking change: The prometheus.exporter.oracledb component now embeds a different exporter

Breaking change: The prometheus.exporter.oracledb component now embeds a different exporter

The prometheus.exporter.oracledb component now embeds the oracledb_exporter from oracle instead of the deprecated oracledb_exporter from iamseth.

As a result of this change, the following metrics are no longer available by default:

• oracledb_sessions_activity
• oracledb_tablespace_free_bytes

The previously undocumented argument custom_metrics is now expecting a list of paths to custom metrics files.

Breaking change: The enable_context_propagation argument in beyla.ebpf has been replaced with the context_propagation argument.

Breaking change: The enable_context_propagation argument in beyla.ebpf has been replaced with the context_propagation argument.

Set enable_context_propagation to all to get the same behaviour as enable_context_propagation being set to true.

Breaking change: In prometheus.exporter.windows, the service and msmq collectors no longer work with WMI

Breaking change: In prometheus.exporter.windows, the service and msmq collectors no longer work with WMI

The msmq block has been removed. The enable_v2_collector, where_clause, and use_api attributes in the service block are also removed.

Prior to Alloy v1.9.0, the service collector exists in 2 different versions. Version 1 used WMI (Windows Management Instrumentation) to query all services and was able to provide additional information. Version 2 is a more efficient solution by directly connecting to the service manager, but is not able to provide additional information like run_as or start configuration.

In Alloy v1.9.0 the Version 1 collector was removed, hence why some arguments and blocks were removed. In Alloy v1.9.2 those arguments and blocks were re-introduced as a no-op in order to make migrations easier for customers.

Due to this change, the metrics produced by service collector are different in v1.9.0 and above. The msmq collector metrics are unchanged.

Example V2 service metrics:

windows_service_state{display_name="Declared Configuration(DC) service",name="dcsvc",status="continue pending"} 0
windows_service_state{display_name="Declared Configuration(DC) service",name="dcsvc",status="pause pending"} 0
windows_service_state{display_name="Declared Configuration(DC) service",name="dcsvc",status="paused"} 0
windows_service_state{display_name="Declared Configuration(DC) service",name="dcsvc",status="running"} 0
windows_service_state{display_name="Declared Configuration(DC) service",name="dcsvc",status="start pending"} 0
windows_service_state{display_name="Declared Configuration(DC) service",name="dcsvc",status="stop pending"} 0
windows_service_state{display_name="Declared Configuration(DC) service",name="dcsvc",status="stopped"} 1

For more information on V1 and V2 service metrics, see the upstream exporter documentation for version 0.27.3 of the Windows Exporter, which is the version used in Alloy v1.8.3. Alloy v1.9.2 uses version 0.30.7 of the Windows Exporter.

Breaking change: The topics argument in the component loki.source.kafka does not use regex by default anymore

Breaking change: The topics argument in the component loki.source.kafka does not use regex by default anymore

A bug in loki.source.kafka caused the component to treat all topics as regular expressions. For example, setting the topic value to “telemetry” would match any topic containing the substring “telemetry”. With the fix introduced in this version, topic values are now treated as exact matches by default. Regular expression matching is still supported by prefixing a topic with “^”, allowing it to match multiple topics.

Breaking change: Change decision precedence in otelcol.processor.tail_sampling when using and_sub_policy and invert_match

Breaking change: Change decision precedence in otelcol.processor.tail_sampling when using and_sub_policy and invert_match

Alloy v1.5 upgraded to OpenTelemetry Collector v0.104.0, which included a fix to the tail sampling processor:

Previously if the decision from a policy evaluation was NotSampled or InvertNotSampled it would return a NotSampled decision regardless, effectively downgrading the result. This was breaking the documented behaviour that inverted decisions should take precedence over all others.

The “documented behavior” which the above quote is referring to is in the processor documentation :

Each policy will result in a decision, and the processor will evaluate them to make a final decision:

• When there’s an “inverted not sample” decision, the trace is not sampled;
• When there’s a “sample” decision, the trace is sampled;
• When there’s an “inverted sample” decision and no “not sample” decisions, the trace is sampled;
• In all other cases, the trace is NOT sampled

An “inverted” decision is the one made based on the “invert_match” attribute, such as the one from the string, numeric or boolean tag policy.

However, in OpenTelemetry Collector v0.116.0 this fix was reverted :

Reverts #33671, allowing for composite policies to specify inverted clauses in conjunction with other policies. This is a change bringing the previous state into place, breaking users who rely on what was introduced as part of #33671.

Breaking change: Change default value of max_streams in otelcol.processor.deltatocumulative

The default value was changed from 0 to 9223372036854775807 (max int).

Breaking change: Change default value of namespace in otelcol.connector.spanmetrics

The default value was changed from "" to "traces.span.metrics".

Breaking change: The component otelcol.exporter.logging has been removed in favor of otelcol.exporter.debug

Both components are very similar. More information can be found in the announcement issue.

Breaking change: Change default value of revision in import.git

The default value was changed from "HEAD" to "main". Setting the revision to "HEAD", "FETCH_HEAD", "ORIG_HEAD", "MERGE_HEAD" or "CHERRY_PICK_HEAD" is no longer allowed.

Breaking change: Some debug metrics for otelcol components have changed

For example, otelcol.exporter.otlp ’s exporter_sent_spans_ratio_total metric is now otelcol_exporter_sent_spans_total. You may need to change your dashboard and alert settings to reference the new metrics. Refer to each component’s documentation page for more information.

Breaking change: The convert_sum_to_gauge and convert_gauge_to_sum functions in otelcol.processor.transform change context

The convert_sum_to_gauge and convert_gauge_to_sum functions must now be used in the metric context rather than in the datapoint context. This is due to a change upstream.

Breaking change: Renamed metrics in beyla.ebpf

process.cpu.state is renamed to cpu.mode and beyla_build_info is renamed to beyla_internal_build_info.

⚠ BREAKING CHANGES

• otelcol.receiver.otlp, otelcol.receiver.faro, otelcol.receiver.jaeger, otelcol.receiver.zipkin, otelcol.receiver.influxdb, otelcol.receiver.splunkhec, otelcol.receiver.datadog, and otelcol.extension.jaeger_remote_sampling now default to idle_timeout = "1m", read_header_timeout = "1m", and write_timeout = "30s" on their HTTP servers, matching upstream default. These were previously being defaulted to "0s". You can explicitly set them to "0s" to restore the previous behavior.
• otelcol.receiver.kafka and otelcol.exporter.kafka: resolve_canonical_bootstrap_servers_only is now a no-op and deprecated. The upstream migration from migration from sarama to `franz-go dropped the behavior. The argument still parses but no longer resolves bootstrap server addresses.
• otelcol.exporter.splunkhec — batcher block removed (upstream deletion in otel-contrib v0.151). Configs using it will fail to load. Migration: move the equivalent fields under sending_queue.batch (enabled → presence of the block; flush_timeout/sizer/min_size/max_size keep their names).

Features 🌟

• Add alloy.command to override the Alloy Helm container entrypoint (#6509) (fea3056)
• Add k8sleaderelector Extension and k8scluster Receiver to OTel Engine (#6450) (24fc846)
• Add OpAmp Extension to OTel Engine (#6632) (a32e83b)
• Add otelcol.connector.signaltometrics component (#6410) (887ca3f)
• database_observability.mysql: Collect mysql query error info (#6675) (d63d9ff)
• database_observability.postgres: Emit op=error_message Loki entries with query fingerprint (#6180) (377f1d0)
• database_observability.sql_server: Add table_spec to schema_details collector (#6641) (db801a6)
• database_observability.sql_server: Add scaffolding for sql_server schema_details collector (#6624) (0cedb33)
• database_observability.sql_server: Scan all databases in schema_details collector (#6672) (dd1eabe)
• discovery.aws: Add AWS unified service discovery component (#6658) (25cb5f0)
• Expose HTTP server idle/read/read_header/write timeouts in OTel Components (#6636) (7cc28ee)
• Helm Chart: Make service.externalTrafficPolicy avaliable (#4511) (a987e33)
• Inline config support for Alloy Otel Extension (#6442) (53589bd)
• Make alloyengine extension OCB-able (#6395) (b6c5b20)
• metrics: Add native histogram support for self-monitoring metrics (#6385) (6045de1)
• otelcol.connector.spanmetrics: Expose include_collector_instance_id (#6635) (ca46cb4)
• otelcol: Expose small v0.152/v0.153 upstream config field additions (#6664) (f824237)
• prometheus.exporter.windows: Add time collector to the Windows exporter (#6517) (6289603)
• prometheus.remote_write: Expose google_iam, round_robin_dns, and additional sigv4/azuread auth options (#6637) (96dd5e0)
• pyroscope.ebpf: Add no_kernel_version_check argument (#6475) (020d74c)
• pyroscope: Add scope labels to Alloy profiles (#6569) (cb771b6)
• ui: Render pipeline graph via @grafana/alloy-pipeline-graph (#6566) (7772bf8)
• Upgrade Prometheus to v3.12.0 (GA) and OpenTelemetry Collector to v0.153.0 (#6597) (118a5d1)
• Upgrade Prometheus to v3.12.0-rc.0, otel-contrib to v0.151.0, Loki to v3.7.1 (#6240) (b4b6747)
• Upgrade to Beyla 3.22 (#6452) (1cb72f6)

Bug Fixes 🐛

• Back off usage reporting on persistent failure instead of retrying every minute (#6478) (0a1d139)
• Beyla glob parsing (#6525) (beddf1b)
• Bump github.com/prometheus/procfs To Pull XFS Collector Fix (#6630) (976726c)
• Bump moduledeps fixtures to go 1.26.4 (#6521) (9d27acd)
• ci: Only stamp BuildDate for release builds (#6541) (59ed586), closes #4266
• database_observability.mysql: Simplify denylist in-memory map for explain_plans collector (#6473) (18fe267)
• database_observability.postgres: Handle Postgres DSN with unix sockets (#6463) (d065b84)
• database_observability.postgres: Update postgres collect intervals to match mysql (#6639) (d55dee0)
• database_observability/postgres: Route Windows builds to the nocgo fingerprint stub (#6690) (5ec00a4)
• database_observability: Explain-plan conditions now redact (7e8e72d)
• deps: Update high-confidence go dependencies (#6558) (bd61eeb), closes #6547
• Don't emit error log when remotecfg is unused (#6483) (d0f30ea)
• Ensure Alloy Logs Dashboard Accounts for New Native Histograms Options (#6704) (5ab0b06)
• Ensure generate-otel-collector-distro runs "go mod tidy" after all generation steps (#6640) (7b15c04)
• Fix typo in the pull request template (#6676) (a473efe)
• loki.process: Log logfmt decode failures at debug instead of error (#6599) (66990eb)
• loki.process: Prevent data race for metircs (#6519) (2ae1848)
• Make go mod tidy resilient to transient module-proxy failures (#6491) (23e04c9)
• prometheus.exporter.cloudwatch: Stop forcing legacy metric-name validation process-wide (#6616) (fabebf8)
• promethues.exporter.mongodb: Add currentop_slow_time argument (#6539) (dd48729)
• pyroscope.ebpf: Skip empty build ID debug info uploads (#6634) (580d0a1)
• Remove duplicate Time field in windows exporter (#6633) (a97ec91)
• Remove extra bracketing of squid exporter host (#6573) (4259b8f)
• security/web/ui: Update web/ui vite to v7.3.5 [SECURITY] (#6497) (d2bb8c0)
• Skip noop error log at collector registration (#6543) (80968d4)
• Stabilize ServiceMonitor e2e target injection sequencing (#6523) (cddef2d)
• Stabilize Windows CI in Loki tests and static converter (#6524) (7284c0d)
• syntax: Release syntax 0.1.1 (#6549) (c43e289)
• Upgrade homebrew-bump-formula action (#6708) (69cd783)
• web/ui: Bump esbuild to 0.28.1 (#6522) (a2b3981)
• windows_service: Gracefully terminate alloy when service is stopped (#6593) (d3969ba)
• windows: Allow non-admin service account to start Alloy (#6627) (bef2f3d)

Features 🌟

• Add gql subcommand (#6242) (ec09c43)
• database_observability.mysql: Add database_observability_wait_event_seconds_total counter (#6106) (602b0b5)
• database_observability.mysql: Refactor bulk table metadata collection (#6222) (e8d441b)
• database_observability.mysql: Refactor schema_details logging strategy (#6239) (fd51530)
• database_observability.postgres: Add exclude_current_user top-level setting (#6187) (f019b5e)
• database_observability.postgres: Improve monitoring user privileges check (#6177) (96b50d1)
• database_observability.postgres: Make health_check respect user/database exclusions settings (#6144) (b4ee0a7)
• database_observability.postgres: Throttle schema details create_statement logs (#6260) (fcb3d3e)
• database_observability: Add wait_event_v2 op with pre-classified wait_event_type (#6105) (f7a1ebd)
• database_observability: Always extract traceparent in MySQL query_samples collector (#6081) (bf2b436)
• database_observability: Always extract traceparent in Postgres query_samples collector (#6411) (74d595a)
• database_observability: Set query_sample, wait_event timestamps at query start for MySQL (#6291) (a47de00)
• database_observability: Set timestamp query_sample at query start for Postgres (#6393) (62d8ecc)
• database_observability: Update wait_event_v2 to a 6-bucket taxonomy and surface Mysql nested events (#6143) (3df9542)
• faro.receiver: Support gzip-compressed request bodies (#6195) (4fa44d2)
• GraphQL server (#5580) (5a2562f)
• helm: Support externally-managed HPAs via controller.autoscaling.horizontal.externalHPA (#6311) (a1e61c7)
• Integration tests for aws firehose and cloudflare logpull (#6089) (f55f780)
• Integration tests for loki.source.azure_event_hubs (#6113) (8e40165)
• Integration tests for loki.source.gcplog (#6161) (feceb8d)
• loki.process: Add regex field to logfmt and json stages (#4941) (cfbabda)
• loki.rules.kubernetes: Add external_labels support (#6320) (9166a61)
• loki.source.heroku: Drop github.com/heroku/x package (#6064) (3382721)
• Migrate from Docker to Moby (#6167) (1e2bbf9)
• mimir.rules.kubernetes, loki.rules.kubernetes: Add configurable mimir_namespace_separator and loki_namespace_separator arguments to allow using a URL-safe separator (#5961) (7dba4f1)
• otelcol.exporter.prometheus: Config option to prevent stripping service.* attributes (#6434) (c229b3c)
• otelcol.exporter.prometheus: Convert classic histograms to NHCB (#6281) (46f4fb6)
• otelcol: Add Nginx receiver (#6141) (fa8d520)
• prometheus.enrich: Support multi-label matching (#5822) (04a1aba)
• prometheus.relabel: Add opt-in TTL cache mode (#6169) (40632c3)
• telemetry: Add graph connection metrics (#6243) (4f3bbbe)
• telemetry: Add pyroscope_forwarded_entries_total metric (#6244) (65ad64c)
• Update to Beyla 3.9.7 (#6175) (4015bd9)
• Update to Beyla 3.9.8 (#6189) (6fffa30)

Bug Fixes 🐛

• Address Critical CVE's From Scanner (#6232) (e713e7a)
• alloycli: Use filepath.Base for CLI Use name to fix shell completions (#6217) (17cfd01)
• cluster: Fix nodes failing to join the cluster when TLS is enabled (#6437) (efbad6d)
• database_observability.mysql: Exclude system schemas from MySQL health check (#6116) (fcae6f4)
• database_observability.postgres: Count error logs when log_timezone is non-UTC (#6274) (2eaa769)
• database_observability.postgres: Do not retain query text in explain_plans collector (#6461) (4c06a64)
• database_observability: Exclude more explain plan output keywords (#6145) (3ff6c14)
• deb: Restart Alloy only on upgrade in postinst script (#6094) (8c15cb3)
• deps: Pin dependencies (#6376) (79323b5)
• deps: Update dependencies for catchpoint, databricks, and snowflake exporters (#6188) (c1b740c)
• deps: Update module go.opentelemetry.io/obi to v0.8.0 [SECURITY] (#6091) (05c14af)
• documentation: Fix documentation for loki.rules.kubernetes (#6088) (61f2b8a)
• Enforce Singleton For Running Alloy Extension Instances (#5763) (83da6f0)
• faro: Abort on context cancelation (#6104) (bf733a6)
• Fix bug that caused 'logging' block's 'write_to' to be used after config update even if not set (#6264) (cba7243)
• graphql: Update cli flags for consistency (#6441) (7749d4f)
• helm: Honor alloy.configMap.key in templates/configmap.yaml (#6312) (415af2c)
• integration-tests: Forward GO_TAGS to k8s integration tests (#6220) (b00f79c)
• logging: Fix startup deadlock when components log before logging config is evaluated (#6112) (6cdce9e)
• loki.process: Make limit stage shutdown cancelable (#6215) (20717b5)
• loki.process: New action_on_duplicate_timestamp config attribute to fudge identical log timestamps (#5615) (7d56f50)
• loki.process: No longer mutate rules in stage.truncate causing every config update to reload pipeline when this stage is used (#6271) (b52a4d8)
• loki.process: Potential deadlock on update with stage and receiver changes (#6270) (cb22e82)
• loki.process: Wrap NewPipeline error correctly in match stage (#6216) (f3af2dd)
• loki.rules.kubernetes: Add timeout to ruler sync calls (26170d4)
• loki.source.syslog: Fix goroutine leak in UDP listener (#6231) (268b260)
• loki: Clone structured metadata so we can perform fan-out correctly (#6138) (473244d)
• oracledb_exporter: Support CGO_ENABLED=0 cross-compilation (#6168) (63ab53d)
• otelcol.exporter.awss3: Add missing unique_key_func_name attribute (#6184) (9ad2b44)
• prometheus.exporter.oracledb: Fix issue with custom metrics not appearing when more than one instance of prometheus.exporter.oracledb is used (#6228) (57de4f4)
• prometheus.operator.*: Pass ScrapeNativeHistograms to ScrapeOptions (#6356) (a44fced)
• remotewrite: Use blocking send with timeout in test server (#6208) (a279088)
• security: Update module github.com/jackc/pgx/v5 to v5.9.2 [SECURITY] (#6326) (bf3ff2e)
• security: Update x/crypto and x/net for CVEs (#6336) (4c7b93b)
• ui: Reduce UI dependencies (#6349) (85e12ba)
• ui: Update dependency minor versions (#6288) (52a28d2)
• Update go to v1.26.4 (#6418) (51fb7d2)
• validation: Improve type checking of ast.LiteralExpr (#5916) (d0a1177)