NOTE

See the deprecations section for important information about the Loki Helm charts.

Features and enhancements

Key features in Loki 3.7.0 include the following:

• Helm charts: - Effective March 16, 2026, the Grafana Loki Helm chart has been be forked to the new repository grafana-community/helm-charts. The motivation behind this change is to accelerate development and enable direct community maintenance, as outlined in grafana/helm-charts#3955. The Grafana Community now owns maintenance of the Loki Helm Chart, while Grafana Labs continues to maintain the Helm Chart for Grafana Enterprise Logs (GEL). The Loki Helm Charts documentation has been updated to point to the new Grafana-Community repo.
• Docs: The documentation now includes a new, comprehensive Troubleshooting section, documenting the Loki and Logs Drilldown error messages.

NOTE

Note that when appropriate, documentation updates have been backported to previous releases.

Other improvements include the following:

• canary: Support passing arbitrary set of labels to use for the query (#17008) (993b3ae)
• compactor: Add support for storing chunk deletion markers in object storage instead of local disk (#19689) (856c11d)
• distributor: Add gauge to track in-flight bytes (#20091) (23ef8ec)
• distributor: Add resolved policy to blocked and enforced label error (#19826) (48d13d1)
• distributor: Add segmentation keys and resolver (#19927) (c853f2c)
• distributor: Add support for UpdateRates RPC to distributors (#19918) (9018886)
• distributor: Randomly distribute requests to the ingest-limits frontend (#19840) (1605a38)
• distributor: Shuffle shard on tenant rate limit (#19990) (3904c2b)
• distributor: Write to dataobj partitions based on segmentation key (#19946) (3a24f5d)
• helm: nameOverride now passed through helm tpl function. (#19590) (7f56fd2)
• helm: Add ability to toggle grpclb port for query frontend service (#19609) (9c4f022)
• helm: Add startupProbe to distributor (#20073) (5b76589)
• helm: Allow configuration of service trafficDistribution parameter (#19558) (55f95e3)
• helm: Allow set topologySpreadConstraints on singleBinary (#19534) (265601f)
• helm: Make loki-canary readinessProbe configurable via values.yaml (#19328) (7231766)
• helm: Use fsGroupChangePolicy=OnRootMismatch to speed up pod starts (#13942) (c7cec3a)
• index-gateway: Client side index gateway shuffle sharding (#20124) (326c7d1)
• ingest-limits: Add UpdateRates RPC, update rates from the frontend, return no-op in the service (#19894) (e173cf4)
• ingest-limits: Check partition state in parallel (#19884) (b8536aa)
• ingester: Handle state change lock in prepare downscale (#20141) (de092da)
• logcli: Allow custom headers to be passed (#20231) (c524203)
• loki: Add loki health command (#20313) (ef69cfd)
• lokitool: Add regex namespace filtering (#20209) (0c1561d)
• operator: Add option to disable ingress (#19382) (9dc71a6)
• querier: Ability to send query context for limit enforcement (#19900) (1a66d2d)

Key features in Loki 3.7.3 include the following:

• logql: Add case-insensitive regex optimization (#20578) (f6a6a4f)
• loki: Allow disabling cache for query_range requests (#21112) (1242aa5)
• operator: Add option to customize gateway server cert (#20325) (3c8b9e8)
• operator: Update metrics authentication to remove dependency on kube-rbac-proxy (#20853) (2de6e79)

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Promtail was deprecated in Loki 3.0 and has been removed as of Loki 3.7.3 (#21248). The code has been merged into Grafana Alloy. You can find migration documentation and a utility to convert your Promtail configuration to Alloy configuration in the Alloy documentation. Note that this removal does NOT include the lambda-promtail client.
• Deprecated storage options including the deprecation of the BoltDB store.
• Deprecated configuration options.
• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

Upgrade Considerations

For important upgrade guidance, refer to the Upgrade Guide.

• BREAKING CHANGE - engine: Make scheduler aware of total compute capacity (#19876)
• BREAKING CHANGE - engine: Share worker threads across all scheduler connections (#20229)
• BREAKING CHANGE - labels: parsed labels should not override structured metadata (#19991)
• BREAKING CHANGE - operator: Switch default for OpenShift stream labels (#21001)

Bug fixes

3.7.6 (2026-08-05)

• queryrange: Preserve sketch in MergeLabels (#23770) (d48329d)

3.7.5 (2026-08-05)

• ci: Build deb/rpm packages again in the release workflow (#23598) (6f49272)
• ci: Disable docker plugin publishing on release-3.7.x (#23440) (6095f71)
• ingester: Fix flush race in ingester (#23682) (ff152dc)
• security/UNKNOWN/: Update module github.com/klauspost/compress to v1.18.7 (release-3.7.x) (#23610) (941b557)
• security/UNKNOWN/: Update module golang.org/x/text to v0.39.0 (release-3.7.x) (#23418) (2347a9a)
• security/UNKNOWN/: Update module google.golang.org/grpc to v1.82.1 (release-3.7.x) (#23405) (f290af0)
• security/UNKNOWN/cmd/chunks-inspect: Update module github.com/klauspost/compress to v1.18.7 (release-3.7.x) (#23611) (11c4b29)
• security/UNKNOWN/cmd/dataobj-inspect: Update module github.com/klauspost/compress to v1.18.7 (release-3.7.x) (#23612) (1e3108e)
• security/UNKNOWN/cmd/dataobj-inspect: Update module go.opentelemetry.io/otel to v1.42.0 (release-3.7.x) (#23518) (28917d3)
• security/UNKNOWN/cmd/dataobj-inspect: Update module golang.org/x/text to v0.39.0 (release-3.7.x) (#23419) (5d2b9d4)
• security/UNKNOWN/operator: Update module github.com/klauspost/compress to v1.18.7 (release-3.7.x) (#23613) (8d71c59)

3.7.4 (2026-07-22)

• ci: Fix zizmor findings for operator-images in Loki 3.7 (#22820) (abc1186)
• ci: Helm CI warning fix (#22605) (7389428)
• ci: Re-enable docker plugin publishing (#22818) (28f44a8)
• compactor: Fix delete request when using Thanos objstore client with filesystem backend (#22811) (dd6fdb5)
• deps: Update module charm.land/bubbletea/v2 to v2.0.7 (release-3.7.x) (#22768) (e11f935)
• deps: Update module charm.land/lipgloss/v2 to v2.0.4 (release-3.7.x) (#22769) (10213c2)
• deps: Update module github.com/aws/aws-sdk-go-v2/credentials to v1.19.24 (release-3.7.x) (#22776) (3daf198)
• deps: Update module github.com/aws/smithy-go to v1.27.3 (release-3.7.x) (#22795) (1f6a191)
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.270 (release-3.7.x) (#22783) (103ccd8)
• deps: Update module github.com/coder/quartz to v0.3.1 (release-3.7.x) (#22784) (3509e31)
• deps: Update module github.com/IBM/ibm-cos-sdk-go to v1.14.1 (release-3.7.x) (#22785) (d0abe72)
• deps: Update module github.com/klauspost/compress to v1.18.6 (release-3.7.x) (#22797) (327da7c)
• deps: Update module github.com/pierrec/lz4/v4 to v4.1.27 (release-3.7.x) (#22798) (6b3297f)
• deps: Update module github.com/shirou/gopsutil/v4 to v4.26.5 (release-3.7.x) (#22800) (cb7f17d)
• security/CRITICAL/clients/cmd/fluentd/docker: Update dependency fluentd to v1.19.3 (release-3.7.x) (#22693) (6d8f1b5)
• security/HIGH/: Bump Go to 1.26.5 to address CVE-2026-39822 and CVE-2026-42505 (#23393) (b318f28)
• security/HIGH/: Update module github.com/apache/thrift to v0.24.0 (release-3.7.x) (#22767) (36eb5f7)
• security/UNKNOWN/: Update module github.com/containerd/containerd/v2 to v2.0.10 (release-3.7.x) (#22478) (6fa6165)
• security/UNKNOWN/pkg/push: Update module golang.org/x/net to v0.55.0 (release-3.7.x) (#22200) (dd7a124)
• storage: Fix index filename issue with legacy S3 client and chunk_delimiter (#23165) (0eca83c)

3.7.3 (2026-06-24)

• deps: Move to Go 1.26 (#20988) (4fb960c).
• deps: Update go toolchain directive to v1.25.11 (#22192) (d5d7a54).
• deps: Update golang.org/x/crypto to v0.52.0 (#22189) (6383bfe).
• deps: Update golang.org/x/net to v0.55.0 (#22190) (3d93fb8).
• deps: Update golang.org/x/sys to v0.44.0 (#22191) (3379538).
• deps: Update module github.com/containerd/containerd/v2 to v2.0.9 (#22187) (365c7e9).
• deps: Update module go.opentelemetry.io/otel to v1.41.0 (#22188) (dc8f4de).
• distributor: Sanitize User-Agent to valid UTF-8 before using as Prometheus label value (#21162) (ed3cf08).
• engine: Align range-query evaluation timestamps to the step grid (#20904) (28d03ec).
• engine: Fix additional instances of writing/reading on a closed conn (#20982) (1f7cacf).
• ingester: Duplicate lines with different structured metadata (#21039) (60a1567).
• ingester: WAL replay hang when flush doesn’t progress (#21176) (fb58ab8).
• logql: Return error on way-too-long label string (#20937) (9c9660a).
• operator: Fix incorrect image reference in OLM bundle (#20966) (84bc9bc).
• operator: Properly handle the deprecated replicationFactor (#20935) (1b96366).
• operator: Provide AWS region in environment variable for STS mode (#21136) (cb261d9).
• operator: Set ingester minimum available pods based on replication factor (#19517) (efc63e8).
• querier: Prevent goroutine leak in downstreamer on context cancellation (#21062) (2891919).
• query-engine: Fix OR operations being silently dropped (#21204) (20ea556).
• security: Mask all credentials (#20766) (f8d94cb).
• storage: Broken get and replace for S3 client (#21163) (e6abd15).

3.7.2 (2026-05-13)

• deps: CVEs in release 3.7.x (#21771) (bb4c5d8).
• deps: Update module github.com/aws/aws-sdk-go-v2/service/s3 to v1.97.3 (#21457) (7bc9450).
• ruler: Fix ruler panic related to unset validation scheme (backport release-3.7.x) (#21401) (cf65729).
• storage: Attach SHA-256 checksum on PutObject for Object Lock buckets (#21849) (7df13d9).

3.7.1 (2026-03-27)

• deps: Upgrade Go and gRPC versions on 3.7.x (#21282) (2c8fff2).

3.7.0 (2026-03-26)

• cd: Add loki-image to needs (#19870) (a2c4ea6)
• compactor: Compactor file descriptor leak (#20077) (0c3dd8c)
• config: Migrate renovate config (#19436) (97745fe)
• dataobj: Flush into multiple index objects when ErrBuilderFull (#19223) (32dbef9)
• deps: Update dataobj-inspect transitive deps version (#19813) (5b212b7)
• deps: Update module cloud.google.com/go/bigtable to v1.41.0 (main) (#20352) (6102309)
• deps: Update module cloud.google.com/go/pubsub to v2 (main) (#19803) (d47dde3)
• deps: Update module cloud.google.com/go/storage to v1.59.0 (main) (#20407) (5c71db6)
• deps: Update module github.com/alecthomas/chroma/v2 to v2.22.0 (main) (#20409) (c64f044)
• deps: Update module github.com/apache/arrow-go/v18 to v18.5.0 (main) (#20354) (d0861a1)
• deps: Update module github.com/aws/aws-sdk-go-v2 to v1.40.1 (main) (#20137) (e106809)
• deps: Update module github.com/aws/aws-sdk-go-v2/config to v1.32.7 (main) (#20401) (50ce71a)
• deps: Update module github.com/aws/aws-sdk-go-v2/credentials to v1.19.7 (main) (#20402) (f20228d)
• deps: Update module github.com/aws/aws-sdk-go-v2/service/dynamodb to v1.53.6 (main) (#20403) (31a870c)
• deps: Update module github.com/aws/aws-sdk-go-v2/service/s3 to v1.95.1 (main) (#20406) (c7c1411)
• deps: Update module github.com/aws/smithy-go to v1.24.0 (main) (#20117) (b0efa70)
• deps: Update module github.com/axiomhq/hyperloglog to v0.2.6 (main) (#20341) (4469f82)
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.256 (main) (#20342) (69e6254)
• deps: Update module github.com/bits-and-blooms/bloom/v3 to v3.7.1 (main) (#19755) (af47e1f)
• deps: Update module github.com/bmatcuk/doublestar/v4 to v4.9.2 (main) (#20418) (078dc94)
• deps: Update module github.com/coder/quartz to v0.3.0 (main) (#19786) (6f784f9)
• deps: Update module github.com/docker/docker to v28.5.2+incompatible (main) (#19756) (1007ee4)
• deps: Update module github.com/gocql/gocql to v2 (main) (#19794) (898b6d2)
• deps: Update module github.com/google/renameio/v2 to v2.0.2 (main) (#20411) (cc43074)
• deps: Update module github.com/grafana/loki/v3 to v3.6.3 (main) (#20343) (d1ae7a1)
• deps: Update module github.com/grpc-ecosystem/go-grpc-middleware/v2 to v2.3.3 (main) (#19758) (8133da9)
• deps: Update module github.com/hashicorp/consul/api to v1.33.0 (main) (#19788) (e417259)
• deps: Update module github.com/ibm/go-sdk-core/v5 to v5.21.2 (main) (#19988) (d8ab970)
• deps: Update module github.com/ibm/ibm-cos-sdk-go to v1.13.0 (main) (#20364) (52d1d8d)
• deps: Update module github.com/ibm/sarama to v1.46.3 (main) (#19760) (4a19787)
• deps: Update module github.com/influxdata/telegraf to v1.37.0 (main) (#20356) (dc1e0ae)
• deps: Update module github.com/klauspost/compress to v1.18.2 (main) (#20108) (f4f2b2a)
• deps: Update module github.com/leodido/go-syslog/v4 to v4.3.0 (main) (#19416) (036387b)
• deps: Update module github.com/minio/minio-go/v7 to v7.0.98 (main) (#20436) (cf89342)
• deps: Update module github.com/ncw/swift/v2 to v2.0.5 (main) (#19764) (fa5e144)
• deps: Update module github.com/oschwald/geoip2-golang to v2 (main) (#19799) (33eeab6)
• deps: Update module github.com/oschwald/geoip2-golang/v2 to v2.1.0 (main) (#20357) (8853d71)
• deps: Update module github.com/parquet-go/parquet-go to v0.27.0 (main) (#20426) (a283eac)
• deps: Update module github.com/prometheus/alertmanager to v0.30.0 (main) (#20358) (f53a609)
• deps: Update module github.com/prometheus/client_golang to v1.23.2 (main) (#19763) (8317f7e)
• deps: Update module github.com/prometheus/common to v0.67.5 (main) (#20363) (aaacbf4)
• deps: Update module github.com/prometheus/prometheus to v0.309.1 (main) (#20388) (bf79bcf)
• deps: Update module github.com/prometheus/sigv4 to v0.4.0 (main) (#20386) (2f80526)
• deps: Update module github.com/redis/go-redis/v9 to v9.17.2 (main) (#20116) (434a929)
• deps: Update module github.com/schollz/progressbar/v3 to v3.19.0 (main) (#20365) (0b238bc)
• deps: Update module github.com/shirou/gopsutil/v4 to v4.25.12 (main) (#20347) (0740eb8)
• deps: Update module github.com/sirupsen/logrus to v1.9.4 (main) (#20447) (35c8df7)
• deps: Update module github.com/sony/gobreaker/v2 to v2.4.0 (main) (#20366) (090ffd5)
• deps: Update module github.com/tjhop/slog-gokit to v0.1.5 (main) (#19808) (615413e)
• deps: Update module github.com/twmb/franz-go to v1.20.6 (main) (#20348) (5ee4fee)
• deps: Update module github.com/twmb/franz-go/pkg/kadm to v1.17.1 (main) (#19790) (1dad0be)
• deps: Update module github.com/twmb/franz-go/pkg/kmsg to v1.12.0 (main) (#19791) (f28c247)
• deps: Update module github.com/workiva/go-datastructures to v1.1.7 (main) (#19766) (f5e0683)
• deps: Update module github.com/xdg-go/scram to v1.2.0 (main) (#20046) (9e52320)
• deps: Update module go.opentelemetry.io/collector/pdata to v1.49.0 (main) (#20371) (7d759f2)
• deps: Update module go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc to v0.64.0 (main) (#20372) (9da1b1b)
• deps: Update module go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace to v0.64.0 (main) (#20373) (719635a)
• deps: Update module go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp to v0.64.0 (main) (#20374) (f1b3e1b)
• deps: Update module go.opentelemetry.io/otel/sdk to v1.39.0 (main) (#20376) (95b82d6)
• deps: Update module golang.org/x/crypto to v0.44.0 (main) (#19776) (c85c67a)
• deps: Update module golang.org/x/net to v0.47.0 (main) (#19850) (5c422a6)
• deps: Update module golang.org/x/oauth2 to v0.34.0 (main) (#20160) (ebf7b93)
• deps: Update module golang.org/x/sync to v0.19.0 (main) (#20161) (817b9d8)
• deps: Update module golang.org/x/sys to v0.40.0 (main) (#20378) (9538b20)
• deps: Update module golang.org/x/text to v0.33.0 (main) (#20408) (9fd6733)
• deps: Update module golang.org/x/time to v0.14.0 (main) (#19782) (74f68fa)
• deps: Update module google.golang.org/api to v0.257.0 (main) (#20143) (5f1da75)
• deps: Update module google.golang.org/grpc to v1.77.0 (main) (#19945) (f3213bb)
• deps: Update module google.golang.org/protobuf to v1.36.11 (main) (#20349) (a80b52e)
• deps: Update module k8s.io/apimachinery to v0.35.0 (main) (#20381) (731e067)
• distributor: Avoid recalculating the segmentation key hash twice (#19961) (8b78f79)
• distributor: Panic when no healthy instances found (#19998) (1c5dfed)
• distributor: Track discarded entries and bytes when hitting stream limits using the ingest limits service (#20244) (9b0af7c)
• docker: Missing permissions to start docker (#19947) (39d2bea)
• docker: Set WORKDIR to root in loki Dockerfiles (#19941) (13f2b1a)
• helm: Bump Helm deps, publish loki-helm-test w/ release (#19939) (7e4e34e)
• helm: Add startup probe read (#19708) (bce87fb)
• helm: Apply fix from #14126 to example (#20252) (716563a)
• helm: Correct GEL provisioner tenant creation instructions (#20271) (9639e2e)
• helm: Correct GEL version (#19657) (cd8b195)
• helm: Do not mark loki.storage.bucketNames.chunks as required, if an s3 url is provided (#19873) (e9951bb)
• helm: Do not mark loki.storage.bucketNames.chunks as required, if minio is used. (#19871) (eddd4f8)
• helm: Do not mark loki.storage.bucketNames.ruler as required, if rulerConfig.storage.type is local (#19882) (f2f564a)
• helm: Don’t fail for missing bucket name, if local disk is used. (#19675) (ad0a29e)
• helm: Don’t fail for missing bucket name, if minio is enabled. (#19745) (cd0c578)
• helm: Enable volumeAttributesClassName attributes for volumeClaimTemplates (#19719) (06da42a)
• helm: Fix ingester-b volumeAttributesClassName templating (#20188) (d696f18)
• helm: Fix rendering of dnsConfig for backend, read, write, single-binary and table-manager (#20013) (1cdb3c7)
• helm: Respect global registry in sidecar image (#18246) (#19347) (79eae2c)
• helm: Update Chart version in README.md (#19669) (94096b7)
• helm: Update version (#19670) (b90ae22)
• ingest-limits: Reduce FetchMaxBytes to 10MB (#19883) (82cfafd)
• ingest-limits: Revoke partitions if lost (#20030) (1ac5d1f)
• kafka: Deadlock on shutdown (#20384) (272a278)
• logql: Parsed labels should not override structured metadata (#19991) (61f9367)
• lokitool: Update ruler path and enable alternative TLS env variables (#19572) (d1ce5cb)
• nomad: Nomad simple example (#19629) (17aec11)
• operator: Change leader-election parameters (#19707) (86068cf)
• operator: Do not deploy NetworkPolicies automatically on OCP 4.20 (#19680) (8df33ff)
• operator: Return quickstart script to working condition and improve rootless usage (#19960) (397da27)
• parser: Do not cache key conflicts results in intern set (#19984) (0a9b024)
• querier: Support multi-tenant queries in Patterns API (#19809) (f609e27)
• querier: Restrict start/end timestamp to requested range for scheduler (#20086) (448cc74)
• querier: Set Content-Type header for JSON responses in serializeHTTPHandler (#19878) (019d6b4)
• querylimits: Accept request limits over not initialized limits (#19891) (905eac8)
• retry: Do not retry if any of multierrors is a client error (#19887) (9825137)
• ruler: Validate remote write config (#19920) (e916944)
• server: Return status bad request code for interval limit error (#19895) (f21f5d5)
• storage: Do not override S3 region if already specified in configuration chain (#20127) (0046bfb)
• storage: Fix regression in S3 client configuration (#20110) (d3f9532), closes #19908

NOTE

See the deprecations section for important information about the Loki Helm charts.

Features and enhancements

Key features in Loki 3.6.0 include the following:

• Horizontally scalable compactor: The Compactor in Loki is a singleton service that compacts the index and applies retention, including processing delete requests. Because it can only be scaled vertically, it limits how many resources you can add to speed things up when the workload increases in bigger cells. This release adds a horizontally scalable compactor, to speed up processing large delete requests or catching up if there has been a lag in processing for some reason. Note that the horizontally scalable compactor delegates deletion work to the workers. Loki continues to perform compaction and apply retention in the main Compactor. For more information, refer to the documentation.
• OTel: Loki has been refactored to use the OpenTelemetry tracing library instead of the OpenTracing library. (#17859). Note that Loki can still be configured using JAEGER_ env vars and still exports traces in JAEGER_ format.
• meta-monitoring: Moves the meta-monitoring responsibilities from the Grafana meta-monitoring Helm chart to the widely supported Grafana Kubernetes Monitoring Helm chart. This was implemented in two PRs: Meta Monitoring Loki v2 (Part 1) - Telemetry Collection (#16665) (2d0bbfd) and Meta Monitoring Loki v2 (Part 2) - Loki Mixin’s (#16962) (93360f6).
• Helm charts: The Grafana Developer Advocates launched a pilot Community Helm Maintainers group at the end of June made up of Grafana Champions who expressed interest in helping maintain the Loki Helm chart. We have resumed regular releases of the Loki Helm chart and reduced the backlog of open Loki Helm chart pull requests from around 150 to less than 40. For information about recent changes, refer to the Helm chart CHANGELOG.
• UI: We have moved the Loki UI to a Grafana Plugin. The static Operational UI is not going away, and all the APIs that serve it will remain in Loki. The JavaScript that serves the UI, however, has been moved to the Loki Operational UI Grafana plugin (#19390).
• Docs: Published a third Cloud Deployment Guide, this one for Google Cloud Platform (#17503). Added links to three log related Learning paths to the Loki docs (#17956). Revised the LogQL query page (#16286) and added a LogQL video (#18111). Added a caution to the Labels topic and added a new topic with instructions how to modify the default OpenTelemetry resource attributes that Loki converts to labels (#17958). Updated the caching topic (#17420).

NOTE

Note that when appropriate, documentation updates have been backported to previous releases.

Other improvements include the following:

• analytics: Change how analytics proxy works (#19098)
• API: Add endpoint returning tenant configured limits (#17101) (6818c90)
• API: Loki Improvement Document (LID) for query splitting (#16958) (66f0f0b)
• compactor HS: Add a dashboard for monitoring deletion with horizontally scalable compactor (#18588) (10db8ab)
• compactor HS: Add a job runner for processing of deletion jobs (#18058)
• compactor HS: Add job queue and deletion job builder (#17843)
• compactor HS: Add jsonnet for deploying horizontally scalable compactor (#18550) (5e28950)
• compactor HS: Add support for applying storage updates after finishing processing of deletion manifest (#18294) (ba3c670)
• compactor HS: Add support for worker for processing of jobs from the compactor’s job queue (#18165) (d05c4bc)
• compactor HS: Fixes and changes for getting horizontally scalable compactor to work well (#18526) (000f90d)
• compactor HS: Refactor interfaces and build a link between delete requests manager and compactor for HS of compactor (#18326) (3646607)
• compactor HS: Store index updates per processed segment in the manifest to the object storage (#18212) (8007e22)
• compactor HS: Tune compactor worker config and add experimental feature warning to the docs (#18539) (5bf3f74)
• compactor HS: Use streaming RPCs for interacting with job queue over grpc (#18125) (385e317)
• compactor HS: Wire things up and add some metrics (#18421) (bd29320)
• compactor HS: Add a metric to track the number of jobs left to conclude ongoing unit of work (#19506)
• distributor: Add MaxRecvMsgSize config for uncompressed message size limits (#16915) (1d99f4d)
• distributor: Log ingestion policy when log_push_request is set (#18712) (0ec787f)
• distributor: Mark truncated log lines with identifier (#18262) (0ee8e76)
• distributor: Stop pushing to the metadata topic from distributors (#17525) (e0f015c)
• distributor: Remove tenant topic tee (#19082)
• fluentbit: Update fluent/fluent-bit to v4 which adds support for structured metadata to the out_loki plugin (#17200) (f108e1c)
• fluentd: Add ‘compress gzip’ setting to fluentd plugin (#18995)
• helm: Add block_builder config to helm chart (#17451) (a9808a1)
• helm: Add startupProbe to memcached(#19166)
• helm: Add startupProbe to memcached exporter(#19165)
• helm: Add configurable livenessProbe to single binary StatefulSet(#19072)
• helm: Add dnsconfig to all missing resources(#18903)
• helm: Add FOLDER_ANNOTATION logic for sidecar container in loki helm chart, making multi-tenant alerting easier (#13289) (833fcbc)
• helm: Add health probes to admin-api (#18546) (358228a)
• helm: Add health probes to bloom-builder (#18549)
• helm: Add health probes to bloom-gateway(#18551)
• helm: Add health probes to bloom-planner(#18552)
• helm: Add health probes to gateway component(#18545)
• helm: Add health probes to loki in backend(#18548)
• helm: Add location snippet to nginx config (#18105) (5737751)
• helm: Add missing revisionHistoryLimit (#18638) (d0e0d1a)
• helm: Add more specificity to affinity and topology spread constraints(#19061)
• helm: Add namespaceOverride(#18707)
• helm: Add non-default-values.yaml to increase test coverage (#18360) (b0e04f4)
• helm: Add option to use deployment type instead of daemonset in canary (#17364) (a6563cf)
• helm: Add PDB for patternIngester (#17058) (7fad38d)
• helm: Add readiness and liveness support for all memcached components (#18525) (9b54d9f)
• helm: Add readiness probe for memcached (#15609) (6ed1461)
• helm: Add replicas to loki-canary deployment (#19095)
• helm: Add startupProbe to loki-sc-rules (#18547) (121e84e)
• helm: Add support for custom alert rule annotations (#19193)
• helm: Add tpl support for pattern_ingester and ingester_client (#16759) (901b2b0)
• helm: Added trafficDistribution for service-distributor (#17742) (78f154f)
• helm: Adds a flag for tokengen rbac components so that they can be disabled optionally (#15882) (10536b2)
• helm: Allow configuring service types (#18708)
• helm: Allow enabling user namespaces (#18661)
• helm: Allow external memcached setup (#17432) (013aa17)
• helm: Allow extraObjects to be defined as either a list or dictionary.(#13252)
• helm: Allow passing tenant password hash instead of password (#17049) (20b97f7)
• helm: Allow setting labels for ingesters (#18536) (ab426be)
• helm: Enable rules sidecar for the ruler pods (#17231) (2c5ad02)
• helm: Expose all storage configuration (#18509) (db5e021)
• helm: Expose labels on volumeClaimTemplates (#13317) (cdefbbf)
• helm: Expose Loki UI in pure ingress (#18400) (21f3d09)
• helm: Helm issue templates (#18693) (e0ec815)
• helm: Implementing support of l2 chunksCache memcache (#17556) (039e7c1)
• helm: Integrate the enterprise-provisioner into Loki helm chart (#17616)
• helm: Make access modes for persistence on all PVCs and StatefulSets editable (#13474)
• helm: Only delete PVC on statefulset deletion, not scaling (#19097)
• helm: Support bypass all s3/gcs/azure config in object_store (#17597) (d6c2a8e)
• helm: Support separate loki ruler storage configuration (#18510)
• helm: Support templated loki.operational_config (#17045) (078f353)
• helm: Update Grafana Loki version to 3.5.3 (#18619) (907bb61)
• helm: Update Loki version in charts (#18447) (e2770ae)
• helm: Use named ports for memcached services targetports (#18990)
• helm: Delete statefulset if volumeClaimTemplates changes (#19217)
• helm: Added affinity property to the loki-canary DaemonSet (#17720)
• ingester: Add ingester_chunks_flush_requests_total (#18696)
• ingester: Add per policy overridable stream limits (#18994)
• kafka: Support multiple Kafka clients in component (#17757)
• kafka: Adds partition-ingester push latency histogram (#17385) (01cf36d)
• kafka: Optionally enable Kafka histograms for read/write latency (#17089) (e962398)
• kafka: Refactors kafka/client to improve metric collection (#17308) (ab65ac5)
• kafka: Support consuming records from Kafka (#17691)
• limits: Add ability to filter tenant limits endpoint by allowlist (#18521) (73de7b5)
• logcli: Add delete commands to logcli (#18375) (f38950a)
• logcli: Add --log.level parameter to help with diagnosing errors when using logcli. (#19136)
• logs drilldown: Return metric query partial results for drilldown (#18286) (c0c2265)
• logs drilldown: Drilldown config endpoint (#19028)
• logs drilldown: Implement unwrap as a projection (#19409)
• logs drilldown: Persist patterns as aggregated metrics (#17737)
• logs drilldown: Query persisted patterns (#17980)
• loki-canary: Update loki canary to support batched log push (#17558)
• metrics: Adds distributor lag counter to push.go (#18012)
• metrics: Adds User-Agent to distributor push logging (#17764)
• mixins: Allow disabling Thanos dashboard and fix dashboard’s file name (#18749)
• operator: Add alert LokiIngesterFlushFailureRateCritical (#18698)
• operator: Add warning alert for when LokiStack is not getting ready (#19258)
• operator: Allow users to configure virtual-host-style for S3 secrets (#17363) (c4dace5)
• operator: Allow using resource requests as limits (#17484) (0b2bae8)
• operator: Deploy network policies with LokiStack (#19099)
• operator: Extend LokiStack authorization to support OTel Semantics (#16031)
• operator: S3 validation reject endpoints that contain a URL path (#19356)
• operator: Update Loki operand to v3.5.5 (#19187)
• pattern ingester: Pattern ingester pushes detected level as structured metadata (#16924) (66033bb)
• pattern ingester: Implement volume-based pattern filtering (#18838)
• policies: Default ingestion policy mappings merged with per-tenant mappings (#18926)
• thanos: Allow dashes in storage_prefix config (#16934) (c01d9c7)

Grafana Labs and the Loki team are excited to announce the release of Loki 3.5. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Features and enhancements

Key features in Loki 3.5.0 include the following:

• Helm charts: Loki 3.5 includes multiple improvements to the Helm charts, including: Adding support for additional global variables in the Helm chart (#16062) (e5f31b3), adding the extraContainers property to the backend Loki pods (#16205) (528a1c6) [closes #16164], adding tpl() support back to read, write and backend pods (#16664) (af05c38), and allowing provisioner to be namespaced (#16091) (af85ecf).
• storage: Add support for `SQLite’ as a database for storing delete requests (#16437) (5b33e65) and when using SQLite, use the stored completion time to reduce the requests considered for querytime filtering (#16600) (3277321).
• Docs: This release includes a heavily revised meta-monitoring section to support new monitoring requirements for Grafana Enterprise Logs customers.

Other improvements include the following:

• distributor: Add simulated latency (#16733) (7119f6e)
• distributor: Do not enforce labels vs agg metric stream (#16696) (25c0620)
• distributor: Enforce limits in distributors (#16705) (bc2111e)
• distributor: Update log level discovery from json to detect nested fields (#16026) (86dc47b)
• distributor: Use stream labels instead of hash in rate limit reasons (#16738) (fdf8632)
• distributors: Add dry-run mode for checking limits in distributors (#16754) (d6c499b)
• kafka: tenant topics (#15977) (c258419)
• ksonnet: Assert for bad max_query_lookback configuration (#16362) (073c94c)
• ksonnet: Add no_schedule_constraints option to distributors (#16029) (9e9a09d)
• lambda-promtail: Improve relabel configuration parsing and testing (#16100) (2587f34)
• logical planning: Planner playground (#16403) (89f0ed7)
• logql: Add LogQL Benchmark Suite (#16434) (1b57297)
• operator: Add configuration option for dropping OTLP attributes (#15857) (bd1ea23)
• operator: Add support for Swift TLS CA configuration (#15260) (62a72f6)
• operator: Enable time-based stream-sharding (#16390) (1b4f1f5)
• operator: Update Loki operand to v3.4.2 (#16360) (42f87d3)
• storage: Add S3 chunk delimiter configuration to support MinIO running on Windows (#16319) (8429b0f)
• storage: Add a configuration option for custom GCS endpoints (#16419) (f1ebd97)

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Promtail is deprecated, as the code has been merged into Grafana Alloy. You can find migration documentation and a utility to convert your Promtail configuration to Alloy configuration in the Alloy documentation. Note that this deprecation does NOT include the lambda-promtail client.
• Deprecated storage options including the deprecation of the BoltDB store.
• Deprecated configuration options.
• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

CAUTION

Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Upgrade Considerations

For important upgrade guidance, refer to the Upgrade Guide.

• BREAKING CHANGE - ksonnet: Clean-up deprecated configurations (#16007).
• BREAKING CHANGE - operator: Add configuration option for dropping OTLP attributes (#15857).

Bug fixes

[Followed by detailed list of bug fixes and dependency updates for versions 3.5.12 down to 3.5]

Features and enhancements

Grafana Labs and the Loki team are excited to announce the release of Loki 3.2. Explore Logs is also now Generally Available. Upgrade to Loki/GEL 3.2 to get the best possible experience with Explore Logs. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Key features in Loki 3.2.0 include the following:

• API: The Loki APIs have been updated to fail log queries when executed on instant query endpoint (#13421) (ce71f1c), add structured metadata to /detected_fields API (#13604) (ce02cc2), and avoid looking up unnecessary TSDB symbols during Volume API (#13960) (7c1a849).
• Query acceleration with Bloom filters (experimental): The Bloom filters feature introduced in Loki 3.0 remains experimental. Note that the Bloom Compactor component has been removed (#13969) (b75eacc) and replaced by the Bloom Planner and Bloom Builder components (#13997) (be9eb50).
• distributor: Ignore empty streams in distributor if all entries fail validation (#13674) (6c4b062), and limit to block ingestion until configured date (#13958) (b5ac6a0).
• Explore Logs Is now Generally Available (GA). For the best experience, you should be on Grafana 11.2 or later and Loki 3.2. This release includes enhancements to add _extracted suffix to detected fields conflicts (#13993) (ab1caea), collect and serve pre-aggregated bytes and counts (#13020) (467eb1b), and remove cardinality filter (#13652) (4f534d7).
• Helm: This release includes updates to the Helm charts to make gateway container port configurable. (#13294) (05176e4) and to support alibabacloud oss in the Helm chart (#13441) (3ebab6f). It also includes a breaking change to the Helm chart to support distributed mode and 3.0 (#12067).
• ingester: Ingester Stream Limit Improvements: Ingester stream limits now take into account “owned streams” and periodically update when the Ingester ring is changed. Non-owned streams are now also flushed when this update takes place. The stream limit calculation has also been updated for improved accuracy in multi-zone ingester deployments. (#13532) (ec34aaa).
• lambda-promtail: Add S3 log parser support for AWS GuardDuty (#13148) (2d92fff), build lambda with zip file (#13787) (9bf08f7), and ensure messages to Kinesis are usable by refactoring parsing of KinesisEvent to match parsing of CWEvents + code cleanup (#13098) (dbfb19b).
• loki: Add ability to disable AWS S3 dualstack endpoints usage(#13785) (bb257f5), not enforce max-query-bytes-read and max-querier-bytes-read in limited tripperware (#13406) (47f6ea5), and upgrade Prometheus (#13671) (b88583d).
• operator: Add alert for discarded samples (#13512) (5f2a02f), add support for the volume API (#13369) (d451e23), enable leader-election (#13760) (1ba4bff), and update Loki operand to v3.1.0 (#13422) (cf5f52d).
• storage: Convert WalSegmentWriter to io.ReadSeeker (#13340) (19c0509) and instrument failed chunk encoding/decoding (#13684) (5a87ccb).
• docs: Add getting started video for ingesting OTel logs (#13226) (5e560f9).

Other improvements include the following:

• chunks-inspect: Support structured metadata (#11506) (1834065).
• exporter: Include boolean values in limit exporter (#13466) (4220737).
• mempool: Replace sync.Mutex with sync.Once (#13293) (61a9854).
• metrics: Collect duplicate log line metrics (#13084) (40ee766).

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Deprecated storage options including the deprecation of the BoltDB store.
• Deprecated configuration options.
• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

CAUTION

Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Upgrade Considerations

For important upgrade guidance, refer to the Upgrade Guide and the separate Helm Upgrade Guide.

• BREAKING CHANGE - API: Fail log queries when executed on instant query endpoint (#13421).
• BREAKING CHANGE - blooms: Remove bloom compactor component (#13969).
• BREAKING CHANGE - docker: Remove wget from Promtail docker image(#15101).
• BREAKING CHANGE - Helm: Update Helm chart to support distributed mode and 3.0 (#12067).
• BREAKING CHANGE - Helm: Fix how we set imagePullSecrets for enterprise-gateway and admin-api. (#13761) (3be5a45).
• BREAKING CHANGE - jsonnet: Convert read statefulset into deployment for loki-simple-scalable (#13977).

IMPORTANT

Out of an abundance of caution, we advise that users with Loki or Grafana Enterprise Logs (GEL) deployments on AWS upgrade their Helm charts or change the names of their buckets, as outlined in the solutions and mitigations section of this blog post.

Bug fixes

3.2.2 (2024-12-04)

• BREAKING CHANGE - docker: Remove wget from Promtail docker image(#15101).
• docker: Move from base-nossl to static. This PR removes the inclusion of glibc into most of the Docker images created by the Loki build system. (#15203).
• logql: Updated JSONExpressionParser not to unescape extracted values if it is JSON object. (#14499).
• promtail: Switch Promtail base image from Debian to Ubuntu to fix critical security issues (#15195).
• storage: Have GetObject check for canceled context. S3ObjectClient.GetObject incorrectly returned nil, 0, nil when the provided context is already canceled (#14420).

3.2.1 (2024-10-17)

• config: Copy Alibaba and IBM object storage configuration from common (#14315).
• logql: updated JSONExpressionParser not to unescape extracted values if it is JSON object. (backport release-3.2.x) (#14502).
• promtail: Revert build image to Debian Bullseye to fix libc version issue in Promtail (#14386).
• storage: have GetObject check for canceled context (backport release-3.2.x) (#14422).

3.2.0 (2024-09-19)

Extensive list of fixes and improvements related to blooms, ci, deps, detected fields, Helm, index gateways, ingester, LogQL, log results cache, loki, loki-mixin, operator, querier, query engine, retry, ruler, sharding, storage, structured metadata, WAL, and more as detailed in the full changelog.

Grafana Labs is excited to announce the release of Loki 2.9.0 Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Features and enhancements

• Structured metadata: The Structured Metadata feature, which was introduced as experimental in release 2.9.0, is generally available as of release 2.9.4.
• Query Language Improvements: Several improvements to the query language that speed up line parsing and regex matching.
• Remote rule evaluation: Rule evaluation can now be handled by queriers to improve speed.
• Multi-store Index support: Loki now supports reading/writing indexes to multiple object stores which enables the use of different storage buckets across periods for storing index.
• New volume and volume_range endpoints: Two new endpoints, index/volume and index/volume_range, have been added to Loki. They return aggregate volume information from the TSDB index for all streams matching a provided stream selector.
• New Storage Client: Add support for IBM cloud object storage as storage client.
• Block queries by hash: Queries can now be blocked by a query hash.
• logfmt stage improvements: logfmt parser now performs non-strict parsing by default which helps scan semi-structured log lines.

Deprecations

• Legacy index and chunk stores that are not “single store” (such as tsdb, boltdb-shipper) are deprecated. These storage backends are Cassandra (cassandra), DynamoDB (aws, aws-dynamo), BigTable (bigtable, bigtable-hashed), GCP (gcp, gcp-columnkey), and gRPC (grpc). See Configure storage for more information.
• The table-manager target is deprecated, because it is not used by “single store” implementations.
• The -boltdb.shipper.compactor.* CLI flags are deprecated in favor of -compactor.*.
• The -ingester.unordered-writes CLI flag is deprecated and will always default to true in the next major release.
• For the full list of deprecations, see CHANGELOG.md

Bug fixes

For a full list of all changes and fixes, refer to the CHANGELOG.

Grafana Labs is excited to announce the release of Loki 2.8. Here’s a summary of new enhancements and important fixes:

Features and enhancements

• TSDB index no longer experimental After extensive testing of the new TSDB index in our Grafana Cloud Logs offering, we can safely announce that it is no longer experimental and encourage all Loki deployments to use it!
• Query blocker Queries can now be blocked in the Querier/Ruler via a per-tenant runtime configuration.
• New backend target A new, third target was added to Loki’s scalable configuration, which is the default configuration used in the Loki helm chart. This allows Loki to be run as 3 targets (read, write, and backend) and makes the read target stateless and therefore able to be run as a Kubernetes deployment that can be scaled automatically.

For a full list of all changes, look at the CHANGELOG.

Upgrade Considerations

As always, please read the upgrade guide before upgrading Loki.

Bug fixes

2.8.6 (2023-10-17)

• Upgrade go to v1.20.10, golang.org/x/net to v0.17.0 and grpc-go to v1.56.3 to patch CVE-2023-39325 / CVE-2023-44487

2.8.5 (2023-09-14)

• Update Docker base images to mitigate security vulnerability CVE-2022-48174

2.8.2 (2023-05-03)

• Update Go to 1.20.4 to address security vulnerabilities in the previous Go version.
• Promtail: Add new decompression configuration to customize the decompressor behavior.

2.8.1 (2023-04-21)

• Fix bug that was dropping index if period is a zero value.
• Fix redis client to prevent it from incorrectly choosing cluster mode with local address.
• Update go to 1.20.3 to address security vulnerabilities in the previous go version.
• Update alpine image to 3.16.5 to address security vulnerabilities in the previous version.
• Promtail: Fix journald support in amd64 binary build.

For a full list of all changes and fixes, look at the CHANGELOG.

Grafana Labs is excited to announce the release of Loki 2.7. Here’s a summary of new enhancements and important fixes.

Features and enhancements

• New Internal Server and TLS TLS can now be configured everywhere and there is an internal server to allow ring, memberlist, and readiness handlers to continue to work without TLS.
• Better Support for Azure Blob Storage thanks to the ability to use Azure’s Service Principal Credentials.
• Logs can now be pushed from the Loki canary so you don’t have to rely on a scraping service to use the canary.
• Additional label_format fields timestamp and line.
• fifocache has been renamed The in-memory fifocache has been renamed to embedded-cache. Check upgrade guide for more details
• New HTTP endpoint for Ingester shutdown that will also delete the ring token.
• Faster label queries thanks to new parallelization.
• Introducing Stream Sharding an experimental new feature to help deal with very large streams.
• Promtail
  • support for max stream limit
  • config reload endpoint / signal
  • compressed file support
  • lambda-promtail now supports Kinesis data stream events
  • matches for the journal reader
  • basic tracing support

For a full list of all, look at the CHANGELOG.

Upgrade Considerations

As always, please read the upgrade guide before upgrading Loki.

Bug fixes

2.7.5 (2023-03-28)

• Flush buffered logger on exit: this makes sure logs are printed if Loki crashes on startup.

2.7.4 (2023-02-24)

• Fixed different streams for cri tags ending on the same stream.
• Fixed the userdata field (from Windows Event Log) being scraped incorrectly.
• Fixed vector() function producing wrong timestamp.
• Fixed behavior for overlapping chunks with multiple stores.
• Fixed logs results caching causing query-frontend to return logs outside of query window.
• Fixed panics when:
  • /scheduler/ring endpoint is requested with scheduler ring disabled.
  • LogQL clones a specific query.
  • Promtail deals with invalid calls to Details().

2.7.3 (2023-02-01)

• Fixed a bug in compactor that caused divide-by-zero panics when startTime and endTime of a delete request were equal.
• Fixed the output of the --version command that showed an incorrect version information.

2.7.2 (2023-01-25)

• Fixed bug in validation of pattern and regexp parsers where missing or empty parameters for these parsers caused panics.
• Fixed bugs in processing delete requests with line filters:
  • Whole chunks covered by delete requests with a line filter were incorrectly skipped.
  • Chunks partially covered by delete requests with a line filter were not properly processed. The part that was not covered by the delete request was rewritten with a line filter, while the part that was covered by the delete request was completely deleted.
• Fixed bug in multi-tenant querying that caused HTTP 400 responses when multiple tenants where used in X-Scope-OrgID header like so tenant-a|tenant-b.
• Upgraded Go build version and Docker container base images to 1.19.5 to mitigate GO-2022-1144 vulnerability.

2.7.1 (2022-12-09)

• Add single compactor http client for delete and gennumber clients. This fixes a bug caused by the accidental introduction of different HTTP clients for compactor and gennumber operations that resulted in 404s when only the gennumber middlewares were enabled:

  level=error ts=2022-10-18T14:13:41.598649348Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
  ts=2022-10-18T14:13:41.598697295Z caller=spanlogger.go:80 user=application level=error msg="failed loading deletes for user" err="unexpected status code: 404"
  

• Change default deletion mode to filter-only on account of the bug (see below).

Known Issues

• There is a bug with the experimental deletion mode in 2.6.x and 2.7.x that in some cases can cause the wrong chunks to be deleted. As a result we have changed the default deletion mode to filter-only until this bug has been addressed.

Grafana Labs is excited to announce the release of Loki 2.6. Here’s a summary of new enhancements and important fixes.

Features and enhancements

• Query multiple tenants at once. We’ve introduced cross-tenant query federation, which allows you to issue one query to multiple tenants and get a single, consolidated result. This is great for scenarios where you need a global view of logs within your multi-tenant cluster. For more information on how to enable this feature, see Multi-Tenancy.
• Filter out and delete certain log lines from query results. This is particularly useful in cases where users may accidentally write sensitive information to Loki that they do not want exposed. Users craft a LogQL query that selects the specific lines they’re interested in, and then can choose to either filter out those lines from query results, or permanently delete them from Loki’s storage. For more information, see Logs Deletion.
• Improved query performance on instant queries. Loki now splits instant queries with a large time range (for example, sum(rate({app="foo"}[6h]))) into several smaller sub-queries and executes them in parallel. Users don’t need to take any action to enjoy this performance improvement; however, they can adjust the number of sub-queries generated by modifying the split_queries_by_interval configuration parameter, which currently defaults to 30m.
• Support Baidu AI Cloud as a storage backend. Loki users can now use Baidu Object Storage (BOS) as their storage backend. See bos_storage_config for details.

For a full list of all changes, look at the CHANGELOG.

Upgrade Considerations

As always, please read the upgrade guide before upgrading Loki.

Bug fixes

2.6.1

v2.6.1 includes following fixes:

• PR 6658 JordanRushing: Updated the versions of dskit and memberlist to allow configuring cluster labels for memberlist. Cluster labels prevent mixing the members between two consistent hash rings of separate applications that are run on the same Kubernetes cluster.
• PR 6681 MasslessParticle Fixed an HTTP connection leak between the querier and the compactor when the log entry deletion feature is enabled.
• PR 6583 MasslessParticle Fixed noisy error messages when the log entry deletion feature is disabled for a tenant

2.6.0 bug fixes

V2.6.0 fixes numerous bugs. The CHANGELOG has the complete list.

A summary of some of the more important fixes:

• PR 6152 Fixed a scenario where live tailing of logs could cause unbounded ingester memory growth.
• PR 5685 Fixed a bug in Loki’s push request parser that allowed users to send arbitrary non-string data as a log line. We now test that the pushed values are valid strings and return an error if values are not valid strings.
• PR 5799 Fixed incorrect deduplication logic for cases where multiple log entries with the same timestamp exist.
• PR 5888 Fixed a bug in the common configuration where the instance_interface_names setting was getting overwritten by the default ring configuration.

It has been nearly 6 months since Loki 2.4 was released, and we’ve been busy making Loki better than ever. Here’s a summary of new enhancements and important fixes.

Features and enhancements

• Thanks to @bboreham for taking a deep dive on the Go regexp library, and for creating a fork which greatly improves the performance of common regex uses cases in Loki.
• Binary operations are now significantly faster, taking full advantage of Loki’s parallelism.
• A new schema is available, which uses more path prefixes to avoid rate limits on S3.
• That same schema change was also added to the filesystem store, which avoids using one directory to store every chunk.
• A new capability for hedging requests to storage improves performance on highly parallelized queries. Refer to the hedging configuration under the storage_config block for more information.
• Promtail has several new ways to ingest logs:
  • The ability to do service discovery and tailing directly from the Docker daemon.
  • Fetching logs directly from Cloudflare.
  • Receiving logs directly in Graylog Extended Log Format (GELF).
• Promtail permits client-side global rate limiting and pipeline-configurable rate limiting.

For a full list of all changes, look at the CHANGELOG.

Upgrade Considerations

As always, please read the upgrade guide before upgrading Loki.

Changes to the config split_queries_by_interval

The most likely impact many people will see is Loki failing to start because of a change in the YAML configuration for split_queries_by_interval. It was previously possible to define this value in two places.

query_range:
  split_queries_by_interval: 10m

and/or

limits_config:
  split_queries_by_interval: 10m

In 2.5.0 it can only be defined in the limits_config section.

Loki will fail to start if you do not remove the split_queries_by_interval configuration parameter from the query_range section.

Additionally, it has a new default value of 30m rather than 0.

The CLI flag is not changed and remains querier.split-queries-by-interval.

More parallelism by default

We continue to move defaults in Loki to leverage parallelism in all configurations, including the single binary.

In v2.5, all queries will be split and sharded by default. This will likely result in increased memory and CPU usage for Loki processes during queries, if you didn’t previously have these values enabled.

Usage reporting

Loki 2.5 includes code we added to report anonymous usage statistics back to Grafana Labs, an issue was created to outline the intent, and what went into the final implementation can be seen here in the source.

Usage reporting helps provide anonymous information on how people use Loki and what the Loki team should focus on for features and documentation. No private information is collected, and all reports are completely anonymous.

If possible, we ask you to leave the usage reporting feature enabled and help us understand more about Loki! We are also working to figure out how we can share this info with the community so everyone can watch Loki grow.

If you would rather not participate in usage stats reporting, the feature can be disabled in config.

Bug fixes

2.5.0 bug fixes

V2.5.0 fixes numerous bugs. The CHANGELOG has the complete list.

A summary of some of the more important fixes:

• Query correctness:
  • PR 5474 cyriltovena: Disable sharding of count/avg when labels are mutated
  • PR 5444 cyriltovena: Do not insert missing point when sharding
  • PR 5423 cyriltovena: Correctly sets hash value for headblock iterator
  • PR 5289 cyriltovena: Fixes log deduplication when mutating Labels using LogQL
  • PR 5006 sandeepsukhani: Fix splitting of queries when step is larger than split interval
• PR 5413 owen-d: A deadlock in the Azure blob client
• PR 5334 kavirajk: A fix in live tailing that could lead to exploding memory usage
• PR 5144 owen-d: A fix for the ruler when trying to use Basic Auth for remote write
• Query cancellation:
  • PR 5113 kavirajk: Fix cancel issue between query-frontend and query scheduler
  • PR 5080 kavirajk: Handle context cancellation in some of the querier downstream requests
  • PR 5075 cyriltovena: Fixes a possible cancellation issue in the frontend
• PR 4741 sandeepsukhani: An issue where retention didn’t cleanup the index completely

Loki 2.4 focuses on two items:

• Making it easier to get your logs into Loki.
• Making Loki easier to run and operate. Anyone not using Kubernetes or avoiding microservices should pay close attention: this release has a lot for you.

Features and enhancements

• Loki no longer requires logs to be sent in perfect chronological order. Support for out of order logs is one of the most highly requested features for Loki. The strict ordering constraint has been removed.
• Scaling Loki is now easier with a hybrid deployment mode that falls between our single binary and our microservices. The Simple scalable deployment scales Loki with new read and write targets. Where previously you would have needed Kubernetes and the microservices approach to start tapping into Loki’s potential, it’s now possible to do this in a simpler way.
• The new common section results in a 70% smaller Loki configuration. Pair that with updated defaults and Loki comes out of the box with more appropriate defaults and limits. Check out the example local configuration as the new reference for running Loki.
• Recording rules are no longer an experimental feature. We’ve given them a more resilient implementation which leverages the existing write ahead log code in Prometheus.
• The new Promtail Kafka Consumer can easily get your logs out of Kafka and into Loki.
• There are nice LogQL enhancements, thanks to the amazing Loki community. LogQL now has group_left and group_right. And, the label_format and line_format functions now support working with dates and times.
• Another great community contribution allows Promtail to accept ndjson and plaintext log files over HTTP.

All in all, about 260 PR’s went into Loki 2.4, and we thank everyone for helping us make the best Loki yet.

For a full list of all changes, look at the CHANGELOG.

Upgrade Considerations

Please read the upgrade guide before updating Loki.

We made a lot of changes to Loki’s configuration as part of this release. We have tried our best to make sure changes are compatible with existing configurations, however some changes to default limits may impact users who didn’t have values explicitly set for these limits in their configuration files.

v2.4.0 changes to defaults

Here is a list of limit defaults that have changed in v2.4.0:

• ingestion_rate_strategy: new default “global”, old default “local”
• max_global_streams_per_user: new default 5000, old default 0 (no limit)
• max_query_length: new default “721h”, old default “0h” (no limit)
• max_query_parallelism: new default 32, old default 14
• max_streams_per_user: new default 0 (no limit), old default 10000
• reject_old_samples: new default true, old default false
• reject_old_samples_max_age: new default “168h”, old default “336h”
• per_stream_rate_limit: new default 3MB, old default -
• per_stream_rate_limit_burst: new default 15MB, old default -

We have enabled the in-memory FIFO cache by default when no existing external cache is configured. This helps improve Loki’s performance, however it will also allow Loki to consume more memory.

v2.4.2 changes to defaults

Here is a list of limit defaults that have changed in v2.4.2, add in PR 5077:

• parallelise_shardable_queries: new default true, old default false
• split_queries_by_interval: new default 30m, old default 0s
• query_ingesters_within: new default 3h, old default 0s
• max_chunk_age: new default 2h, old default 1h
• max_concurrent: new default 10, old default 20

Bug fixes

2.4.2 bug fixes

2.4.2 fixes these bugs:

• PR 4968 trevorwhitney: Fixes a bug in which querying ingesters wrongly returns a ruler, causing the internal server error code = Unimplemented.
• PR 4875 trevorwhitney: Honor the replication factor specified in the common configuration block when memberlist is the consistent hash ring store.
• PR 4792 AndreZiviani: Corrects the default values of configuration options in the documentation for:
  • scheduler_dns_lookup_period
  • min_ready_duration
  • final_sleep
  • max_transfer_retries
  • chunk_retain_period
  • chunk_target_size
  • batch_size
  • timeout (for Redis requests)

2.4.1 bug fixes

2.4.1 fixes two bugs in the 2.4.0 release:

• PR 4687 owen-d: Eliminate a compactor panic when not using a tenant overrides file.
• PR 4681 slim-bean: Correct the initialization of the read target. Incorrect initialization resulted in query gaps after a chunk was flushed and before the querier downloaded new index tables.

2.4.0 bug fixes

Here are some of the more important fixes in version 2.4.0:

• PR 4598 kavirajk: Fix the IP matcher lexer to differentiate a filter from an identifier.
• PR 4563 cyriltovena: Fixes the Series function to properly handle sharding.
• PR 4518 slim-bean: Fixes a bug in which items are returned to a sync.Pool incorrectly.
• PR 4411 slim-bean: Fixes a frontend bug which caused it to wait for results which would never arrive.
• PR 4238 liguozhong: Fixes a distributor goroutine leak.

Security fixes

2.4.0 security fixes

Version 2.4.0 contains one security-related fix:

• PR 4627 Explicitly define allowed HTTP methods on HTTP endpoints.

A community user noticed that all Loki endpoints would respond to HTTP OPTIONS requests, and the proxy they placed in front of Loki responsible for HTTP authentication would pass OPTIONS requests unauthenticated to Loki, allowing users to make unauthenticated/unauthorized queries to Loki. The fix restricts the type of HTTP methods allowed on each endpoint and disallows OPTIONS requests.

The Loki team is excited to announce the release of Loki 2.3!

It’s been nearly 6 months since 2.2 was released and we have made good use of that time to bring forward several significant improvements and requested features.

2.3 is also the first version of Loki released under the AGPLv3 license. You can read more about our licensing here.

Some parts of the Loki repo will remain Apache-2.0 licensed (mainly clients and some tooling), for more details read LICENSING.md.

Features and enhancements

• Loki now has the ability to apply custom retention based on stream selectors! This will allow much finer control over log retention all of which is now handled by Loki, no longer requiring the use of object store configs for retention.

• Coming along hand in hand with storing logs for longer durations is the ability to delete log streams. The initial implementation lets you submit delete request jobs which will be processed after 24 hours.

• A very exciting new LogQL parser has been introduced: the pattern parser. Much simpler and faster than regexp for log lines that have a little bit of structure to them such as the Common Log Format. This is now Loki’s fastest parser so try it out on any of your log lines!

• Extending on the work of Alerting Rules, Loki now accepts recording rules. This lets you turn your logs into metrics and push them to Prometheus or any Prometheus compatible remote_write endpoint.

• LogQL can understand IP addresses! This enables filtering on IP addresses and subnet ranges.

For those of you running Loki as microservices, the following features will improve performance operations significantly for many operations.

• We created an index gateway which takes on the task of downloading the boltdb-shipper index files allowing you to run your queriers without any local disk requirements, this is really helpful in Kubernetes environments where you can return your queriers from Statefulsets back to Deployments and save a lot of PVC costs and operational headaches.

• Ingester queriers are now shardable, this is a significant performance boost for high volume log streams when querying recent data.

• Instant queries can now be split and sharded making them just as fast as range queries.

A very common feature requested has also been included in 2.3:

• 3919 github-vincent-miszczak: Add ingester.autoforget-unhealthy-timeout opt-in feature

Without revisiting the decisions and discussions around the somewhat controversial behavior of unhealthy ingesters, you can now decided how you would like them to be handled: manually or automatically.

Lastly several useful additions to the LogQL query language have been included:

• More text/template functions are included for label_format and line_format with PR 3515, for more information,see the documentation for template functions.

• Also support for math functions withing label_format and line_format was included with 3434.

• Two additional metric functions with some interesting use cases first_over_time and last_over_time were added in PR 3050. These can be useful for some down sampling approaches where instead of taking an average, max, or min of samples over a range in a metrics query, you can select the first or last log line to use from that range.

Upgrade considerations

The path from 2.2.1 to 2.3.0 should be smooth, as always, read the Upgrade Guide for important upgrade guidance.

One change we consider noteworthy however is:

• 3216 sandeepsukhani: check for stream selectors to have at least one equality matcher.

This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:

{namespace=~".*"}

This query will now be rejected, however there are several ways to modify it for it to succeed:

Add at least one equals label matcher:

{cluster="us-east-1",namespace=~".*"}

Use .+ instead of .*

{namespace=~".+"}

This difference may seem subtle but if we break it down.

. matches any character,

• matches zero or more of the preceding character and

• matches one or more of the preceding character. The .*
  case will match empty values where .+
  will not, this is the important difference.

{namespace=""}

is an invalid request (unless you add another equals label matcher like the example above).

The reasoning for this change has to do with how index lookups work in Loki, if you don’t have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.

Security fixes

List of security fixes for 2.3.x.

2.3.0 security fixes

2.3.0 contains an important security fix:

• 4020 simonswine: Restrict path segments in TenantIDs (CVE-2021-36156 CVE-2021-36157).

NOTE

Exploitation of this vulnerability requires the ability for an attacker to craft and send directly to Loki an X-Scope-OrgID header, end users should not have the ability to create and send this header directly to Loki as it controls access to tenants and is important to control setting of this header for proper tenant isolation and security. We always recommend having a proxy or gateway be responsible for setting the X-Scope-OrgID.

Bug fixes

Lists of bug fixes for 2.3.x.

2.3.0 bug fixes

• An important fix for leaking resources was patched with 3733, when queries were canceled a goroutine was left running which would hold memory resources creating a memory leak.

• 3686 fixes a panic with the frontend when use with downstream URL. Note we recommend using the GRPC Pull Model, better performance and fair scheduling between tenants can be obtained with the GPRC Pull Model.

Features and enhancements

Grafana Labs and the Loki team are excited to announce the release of Loki 3.4. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Key features in Loki 3.4.0 include the following:

• Time sharding ingestion: Add per-tenant time sharding for long out-of-order ingestion. The setting shard_streams.time_sharding_enabled is a per-tenant config that splits incoming log streams by injecting a new time_shard synthetic label in such a way that no stream will ever be longer than 1h (or, rather, max_chunk_age /2). This allows the ingestion of very old logs, because the logs in these new streams will never be “too far behind” by design.
• Thanos object client: Previously Loki and Grafana Enterprise Logs used different Go packages to talk to the underlying object store. Migrating to Thanos means that there is now a single object client across Loki and GEL, and across the other Grafana backend databases: Mimir and Pyroscope. The documentation now includes configuration examples using Thanos-based storage clients.
• API: Add support for Parquet file format as a query response format.
• Structured metadata: Introduces a new feature,based on per-tenant configuration, that allows for extraction of “fields” into structured metadata at ingest time. Fields can either be regular labels, structured metadata keys, or keys from logfmt or json formatted log lines.
• lambda-promtail: Adds support for relabeling log entries in lambda-promtail using Prometheus-style relabel configs. This allows for more flexible label manipulation and filtering of log entries before they are sent to Loki.
• Kafka: Loki 3.4 adds an Apache Kafka development environment to support ongoing development work.
• Helm charts: Loki 3.4 includes multiple improvements to the Helm charts, including: Added support for overrides exporter, Expose topologySpreadConstraints for admin-api pods, Expose topologySpreadConstraints for distributed deployment, Compute ConfigMap/Secret checksum only over .data content, Do not explicitly template managed-by, as it should be set at install time by the installation manager.
• Query acceleration with Bloom filters (experimental): The Bloom filters feature introduced in Loki 3.0 remains experimental. In Loki 3.3, there was a major shift in how Loki leverages bloom filters — pivoting from free-text search to harnessing the power of structured metadata. In Loki 3.4, work continues on the new Bloom Planner.
• Explore logs: The Explore apps suite for metrics, logs, traces, and profiles has a new name: Grafana Drilldown. To learn more about why we made this switch, you can read about it in this blog post. The new name is reflected in the Grafana Cloud user interface and navigation as of 2025-02-20, and it will be part of the Grafana 11.6 release in March.
• Docs: This release includes a heavily revised labels topic and a new Cardinality topic. This release also marks the return of the sizing guidance page, newly updated based on Grafana Labs experience running Loki. Continuing updates to the Query section include a revised logcli topic and new logcli tutorial. There is also two additional new tutorials: Deploying Loki on Azure tutorial and Kubernetes Monitoring Helm tutorial.

NOTE

Note that when appropriate, documentation updates have been backported to previous releases.

Other improvements include the following:

• analytics: Add TLS config to the analytics client.
• approx_topk: Reduce memory usage of HyperLogLog in approx_topk.
• approx_topk: Improve approx_topk performance by reducing allocations.
• blooms: Add task timing and sizing metrics.
• blooms: Apply task timeout in bloom builder.
• blooms: Prefetch bloom blocks as soon as they are built.
• blooms: Compute chunkrefs for series right before sending task to builder.
• blooms: Replace JSON lib for encoding/decoding metas.
• distributor: Add a metric ingestion time SM sanitization.
• docker compose: enable feature toggle.
• docker driver: Add ARM64 build and release of the Docker driver in Drone pipeline.
• lambda-promtail: Add terraform variable for s3 bucket notification filter prefix.
• logcli: Add opt-in support for ProxyFromEnvironment in logcli.
• logcli: Include common labels.
• logql: Allow line_format bypass template execution if possible.
• metrics: Add lines skipped metric to pattern ingesters.
• operator: Add support for managed GCP WorkloadIdentity.
• otel/labels: Add otlp attribute deployment.environment.name to list of default labels.
• querier: Present DF bytes values in queryable format.
• querier: Use context propagation to call the same ingester in GetChunksID as Query.
• ruler: Add Rule Name and Type to Query Tags.
• storage: Add objstore support for Swift using thanos.io/objstore.
• storage: Skip writeback for chunks fetched by queriers older than a duration.
• storage: Support config timeout for AlibabaCloud OSS requests.
• storage: Allow TSDB index creation in memory only.
• structured metadata: Sanitize structured metadata during ingestion in the distributor.

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Promtail is deprecated, as the code has been merged into Grafana Alloy. You can find migration documentation and a utility to convert your Promtail configuration to Alloy configuration in the Alloy documentation. Note that this deprecation does NOT include the lambda-promtail client.
• Deprecated storage options including the deprecation of the BoltDB store.
• Deprecated configuration options.
• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

CAUTION

Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Upgrade Considerations

For important upgrade guidance, refer to the Upgrade Guide.

BREAKING CHANGE - Docker image: Remove wget from Promtail docker image.

Bug fixes

(Details of bug fixes are extensive and include updates to dependencies, fixes in various components such as blooms, distributor, docker, lambda-promtail, logcli, logql, metrics, operator, otel, pattern ingester, promtail, querier, kafka, storage, structured metadata, wal, and more.)

Grafana Labs and the Loki team are excited to announce the release of Loki 3.3. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Features and enhancements

Key features in Loki 3.3.0 include the following:

• Query acceleration with Bloom filters (experimental): The Bloom filters feature introduced in Loki 3.0 remains experimental. There has been a major shift in how Loki leverages bloom filters — pivoting from free-text search to harnessing the power of structured metadata (#14061) (a2fbaa8). This change leverages the growing adoption of OpenTelemetry in observability stacks. The Bloom Compactor component has been removed and replaced by the Bloom Planner and Bloom Builder components (#13997) (bf60455) and blooms has a new block schema (V3) (#14038) (5395daf) which is a breaking change.

• Explore Logs: Starting with Grafana v11.3, the plugin for the Explore Logs app is installed in Grafana by default. This release includes enhancements to improve pattern ingester tracing (#14707) (80aec25), to apply patterns line length limit to json message keys (#14296) (41fafd8), add configuration to set max-allowed-line-length in the pattern ingester (#14070) (0780456) and move metric aggregation to a per-tenant configuration (#14709) (c1fde26).

• logql: We’ve added an approx_topk function to LogQL, which uses probabilistic techniques to give you an approximation of the most frequent values in the input, along with their approximate frequencies. This is great for cases where the equivalent topk query hits the maximum series limit or times out before completion. Its also great in cases where you want a faster response to your query and are fine with an approximation of the true answer. (#14243). For more information, visit LogQL docs.

• helm: Multiple improvements to the Helm charts, including: Updating the chart with Loki version 3.2.0 (#14281) (11b92ee), replacing bloom compactor with bloom planner and builder (#14003) (08e61ca), adding the ability to configure client_max_body_size (#12924) (809a024), adding additional service annotations for components in distributed mode (#14131) (5978f13), adding configurable extraEnvFrom to admin-api and enterprise-gateway (#14533) (5d78a3a), adding kubeVersionOverride for Helm chart (#14434) (0935d77), adding persistence option to memcached on Helm chart (#13619) (ef1df0e), adding tolerations to pattern-ingester statefulset (#13605) (09530c0), allowing setting node attributes to tokengen and provisioner (#14311) (c708ae6), updating Loki Helm chart for restricted environments (#14440) (adc7538), and supporting ruler sidecar in singleBinary mode (#13572) (684baf7).

• mixins: Multiple improvements to the Loki mixins, including: adding a missing cluster label to mixins (#12870) (547ca70), adding support for partition ingester in dashboards (#14498) (70deebf), adding a Loki compaction not successful alert (#14239) (da04f50), allowing overriding of some labels by parameterizing mixin recording/alert rules (#11495) (f1425b6), allowing disabling of bloom dashboards (#14177) (ce2e6d5), allowing hiding of unused rows in loki-operational (#13646) (3aa4f22), and merging read-resources and write-resources dashboards for SSD into one dashboard that also includes the backend-resources(#13471) (45b8719).

• operator: Multiple updates to the Operator, including: Adding support for Loki OTLP limits configuration (#13446) (d02f435), providing default OTLP attribute configuration (#14410) (1b52387), adding a User-guide for OTLP configuration (#14620) (27b4071), updating the Loki operand to v3.1.1 (#14042) (7ae1588), updating the Loki operand to v3.2.1 (#14526) (5e970e5), renaming the Loki API go module (#14568) (976d8ab), introducing 1x.pico size (#14407) (57de81d), migrating the project layout to kubebuilder go/v4 (#14447) (dbb3b6e), declaring feature FIPS support for OpenShift only (#14308) (720c303).

• docs: New videos and tutorials, including a New Cloud deployment guide for Deploying Loki on AWS, a new Fluent Bit tutorial and a Getting Started with OpenTelemetry Collector tutorial. Also there are improvements to the Query documentation, including a new Query Best Practices topic, a reorganized Query Template, and a revised LogQL Simulator topic.

Other improvements include the following:

• api: Added endpoint to return detected field values (#14350) (7983f94).
• blooms: Add disk-backed queue for the bloom-planner (backport k227) (#14927) (1f6828b).
• blooms: Only write key and key=value to blooms (#14686) (3af0004).
• blooms: Do not add empty blooms to offsets (#14577) (51c42e8).
• blooms: Extract task computing into a strategy interface (#13690) (ab5e6ea).
• blooms: New bloom planning using chunk size TSDB stats (#14547) (673ede1).
• blooms: Remove compression of .tar archived bloom blocks (#14159) (cdf084f).
• compactor: Add backoff mechanism to the retention process (#14182) (3136880).
• distributor: Track discarded data by usageTracker (#14081) (c65721e).
• distributor: Add ability to log stream selectors before service name detection (#14154) (d7ff426).
• distributors: Use a pool of worker to push to ingesters. (#14245) (f80d68a).
• fluentd: Support custom http headers (#14299) (e59035e).
• ingester: Implement owned streams calculation using Partition Ring (#14282) (3c36ba9).
• ingester: Implement partition shuffle sharding for ingester (#14304) (1a4436c).
• jsonnet: Allow to name prefix zoned ingesters (#14260) (fac3177).
• labels: Add app_name as a service label (#13660) (f2a16f4).
• logcli: Add gzip compression option (#14598) (4d3f9f5).
• logql: Micro-optimizations for IP filter (#14072) (c5083c7).
• logql: Add query user and query source to “executing query” log lines (#14320) (4d69929).
• promtail: Add structured metadata to the promtail push API (#14153) (66cffcb).
• querier: Move detected field logic to query frontend (#14212) (36ace66).
• ruler: Enable ruler store that uses clients from thanos-io/objstore pkg (#11713) (8bca2e7).
• storage: Add retries for s3 ObjectExists calls (#14062) (73cbbb0).
• storage: Implement IsRetryableErr for S3ObjectClient (#14174) (fc90a63).
• storage: Introduce new ObjectExistsWithSize API to (#14268) (ac422b3).
• storage: AWS backend using thanos.io/objstore (#11221) (b872246).
• storage: Azure backend using thanos.io/objstore (#11315) (5824e3d).
• storage: GCS backend using thanos.io/objstore (#11132) (c059ace).
• structured metadata: Include structured_metadata size while asserting rate limit (#14571) (a962edb).

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Deprecated storage options including the deprecation of the BoltDB store.
• Deprecated configuration options.
• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

CAUTION

Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Upgrade Considerations

For important upgrade guidance, refer to the Upgrade Guide.

• BREAKING CHANGE - blooms: Introduce a new block schema (V3) (#14038).
• BREAKING CHANGE - blooms: Index structured metadata into blooms (#14061).
• BREAKING CHANGE - docker: Remove wget from Promtail docker image(#15101).
• BREAKING CHANGE - operator: Migrate project layout to kubebuilder go/v4 (#14447).
• BREAKING CHANGE - operator: Rename Loki API go module (#14568).
• BREAKING CHANGE - operator: Provide default OTLP attribute configuration (#14410).

Grafana Labs and the Loki team are excited to announce the release of Loki 3.0. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Features and enhancements

NOTE

Note that Loki 3.0 defaults to using the v13 schema. All of the latest features are built against TSDB and the v13 Schema. This version of the schema is compatible with both Loki 2.9.x and Loki 3.0. The main change is to add support for Structured Metadata which is used by the new OTLP native endpoint and is enabled by default.

Key features in Loki 3.0.0 include the following:

• Query acceleration with Bloom filters (experimental): This is designed to speed up filter queries, with best results for queries that are looking for a specific text string like an error message or UUID. For more information, refer to the Query acceleration with Blooms.
• Native OpenTelemetry Support: A simplified ingestion pipeline (Loki Exporter no longer needed) and a more intuitive query experience for OTel logs. For more information, refer to the OTel documentation.
• Helm charts: A major upgrade to the Loki helm chart introduces support for Distributed mode (also known as microservices mode), includes memcached by default, and includes several updates to configurations to improve Loki operations.
• Pattern match filter: LogQL now supports two new pattern match filter operators. You can match any word with just one control character and it is simpler and 10x faster than using regex.
• Caching updates: This release includes multiple updates to caching to improve performance, add new configuration options and support for new features, deprecate features no longer needed, and add automatic background checks.
• Lambda/Promtail: Support dropping labels (#10755) (ec54c72).
• Profiling integration: Added profiling integrations to tracing instrumentation to allow getting a profile for a single request.
• Docs improvements: All the Getting Started topics have been revised, including a new Quickstart to help new users get up and running with Loki faster. The Storage, Configuration Reference, and API documentation have been updated to reflect deprecated and removed code, configuration options, and API endpoints.

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removes a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Deprecated storage options including the deprecation of the BoltDB store.
• Deprecated configuration options.
• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

CAUTION

Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Upgrade Considerations

The path from 2.9 to 3.0 includes several breaking changes. For important upgrade guidance, refer to the Upgrade Guide and the separate Helm Upgrade Guide.

Bug fixes

3.0.1 (2024-08-09)

• deps: Bumped dependencies versions to resolve CVEs (#13833) (e13011d).
• Fixed nil pointer dereference in bloomstore initialization (#12869) (167b468), closes #12270.

3.0.0 (2024-04-08)

• All lifecycler configurations reference a valid IPv6 address and port combination (#11121) (6385b19).
• deps: Update github.com/c2h5oh/datasize digest to 859f65c (main) (#10820) (c66ffd1).
• deps: Update github.com/docker/go-plugins-helpers digest to 6eecb7b (main) (#10826) (fb9c496).
• deps: Update github.com/grafana/gomemcache digest to 6947259 (main) (#10836) (2327789).
• deps: Update github.com/grafana/loki/pkg/push digest to 583aa28 (main) (#10842) (02d9418).
• deps: Update github.com/grafana/loki/pkg/push digest to cfc4f0e (main) (#10946) (d27c4d2).
• deps: Update github.com/grafana/loki/pkg/push digest to e523809 (main) (#11107) (09cb9ae).
• deps: Update github.com/joncrlsn/dque digest to c2ef48c (main) (#10947) (1fe4885).
• deps: Update module google.golang.org/grpc [security] (main) (#11031) (0695424).
• helm: Bump nginx-unprivilege to fix CVE (#10754) (dbf7dd4).
• helm: Sidecar configuration for Backend (#10603) (c29ba97).
• lambda-promtail Fix panic in lambda-promtail due to mishandling of empty DROP_LABELS env var. (#11074).
• loki: Respect query matcher in ingester when getting label values (#10375) (85e2e52).
• loki Generate tsdb_shipper storage_config even if using_boltdb_shipper is false (#11195).
• loki Do not reflect label names in request metrics’ “route” label. (11551).
• loki Fix duplicate logs from docker containers. (#11563).
• loki Ruler: Fixed a panic that can be caused by concurrent read-write access of tenant configs when there are a large amount of rules. (#11601).
• loki Fixed regression adding newlines to HTTP error response bodies which may break client integrations. (#11606).
• loki Log results cache: compose empty response based on the request being served to avoid returning incorrect limit or direction. (#11657).
• loki Fix semantics of label parsing logic of metrics and logs queries. Both only parse the first label if multiple extractions into the same label are requested. (#11587).
• loki Background Cache: Fixes a bug that is causing the background queue size to be incremented twice for each enqueued item. (#11776).
• loki : Parsing: String array elements were not being parsed correctly in JSON processing (#11921).
• promtail: Correctly parse list of drop stage sources from YAML (#10848) (f51ee84).
• promtail: Prevent panic due to duplicate metric registration after reloaded (#10798) (47e2c58).
• promtail : Fix Promtail excludepath not evaluated on newly added files. (#9831).
• tools/lambda-promtail: Do not evaluate empty string for drop_labels (#11074) (94169a0).

Grafana Labs and the Loki team are excited to announce the release of Loki 3.1. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Features and enhancements

Key features in Loki 3.1.0 include the following:

• Query acceleration with Bloom filters (experimental): Continued work on improving the new Bloom filters feature introduced in Loki 3.0. This is designed to speed up filter queries, with best results for queries that are looking for a specific text string like an error message or UUID. For more information, refer to Query acceleration with Blooms.
• Helm: Update Helm chart to support distributed mode and 3.0 (#12067) (79b876b). Add pattern ingester support in SSD mode (#12685) (19bfef4). Enable log volume endpoint by default in Helm (#12690) (e39677f). Allow extraObject items as multiline strings (#12397) (af5be90).Support for PVC Annotations for Non-Distributed Modes (#12023) (efdae3d).
• LogQL: Support negative numbers in LogQL (#13091) (6df81db). Improve performance of first_over_time and last_over_time queries by sharding them (#11605) (f66172e). Improve syntax parser for pattern (#12489) (48dae44).
• Loki: Add ability to disable AWS S3 dual stack endpoints usage (#13795) (464ac73).
• lokitool: Add lokitool to replace cortextool. (#12166) (7b7d3d4). Introduce index audit to lokitool (#13008) (47f0236).
• Explore Logs: Explore Logs, which lets you explore your Loki data without writing LogQL queries, is now available in public preview. If you are a Grafana Cloud user, you can access Explore Logs in the Grafana Cloud main navigation menu. If you are not a Grafana Cloud user, you can install the Explore Logs plugin. For more information, refer to the Explore Logs documentation.
• Docs: Added a video to the Getting Started demo and updated for Grafana Alloy. Added an interactive sandbox to the Loki Quickstart tutorial. Updated the documentation for the SSD and microservices deployment modes using the Helm charts. Documented the new meta-monitoring Helm chart.

Other improvements include the following:

• API: Add step param to Patterns Query API (#12703) (7b8533e).
• API: Expose optional label matcher for label names API (#11982) (8084259).
• cache: Add Cache-Control: no-cache support for Loki instant queries. (#12896) (88e545f).
• canary: Add test to check query results with and without cache. (#13104) (71507a2).
• distributor: Optimize log parsing performance by using unsafe package (#13223) (9f31b25).
• Enable log volume endpoint by default (#12628) (397aa56).
• ingester: Add backoff to flush op (#13140) (9767807).
• ingester: Add ingester_chunks_flush_failures_total (#12925) (cc3694e).
• ingester: add profile tagging to ingester (#13068) (00d3c7a).
• ingester: add recalculateOwnedStreams to check stream ownership if the ring is changed (#13103) (e7689b2).
• ingester: Add profile tagging to ingester (#13068) (00d3c7a).
• ingester: Include the stream we failed to create in the stream limit error message (#12437) (ec81991).
• ingester: New stream count limiter (#13006) (1111595).
• Lambda-Promtail: Parameterise the MaximumEventAgeInSeconds, LogGroupName, and IAMRoleName for Lambda-Promtail CloudFormation template (#12728) (8892dc8).
• loki/main.go: Log which config file path is used on startup (#12985) (7a3338e).
• metamonitoring: Add a version of the mixin dashboards for meta monitoring (#12700) (ec1a057).
• Promtail: Added support to install wget on Promtail docker image to support docker healthcheck (#11711) (ffe684c).
• Promtail: Support of RFC3164 aka BSD Syslog (#12810) (be41525).
• querier: Split gRPC client into two. (#12726) (7b6f057).
• reporting: Report CPU usage (#12970) (87288d3).
• Split detected fields queries (#12491) (6c33809).
• TSDB: Add fast-path to inversePostingsForMatcher (#12679) (402d1d7).

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Deprecated storage options including the deprecation of the BoltDB store.
• Deprecated configuration options.
• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

CAUTION

Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Upgrade Considerations

The path from 2.9 to 3.0 included several breaking changes. For important upgrade guidance, refer to the Upgrade Guide and the separate Helm Upgrade Guide.

• BREAKING CHANGE Update Helm chart to support distributed mode and 3.0 (#12067).

IMPORTANT

Out of an abundance of caution, we advise that users with Loki or Grafana Enterprise Logs (GEL) deployments on AWS upgrade their Helm charts or change the names of their buckets, as outlined in the solutions and mitigations section of this blog post.

Bug fixes

3.1.2 (2024-10-17)

• config: Copy Alibaba and IBM object storage configuration from common (#14316) (7184d45).
• docker-compose: upgrade loki and grafana production image tags to 3.1.1 (#14025) (36fe29e).
• logql: updated JSONExpressionParser not to unescape extracted values if it is JSON object. (backport release-3.1.x) (#14503) (759f9c8).
• promtail: Revert build image to Debian Bullseye to fix libc version issue in Promtail (#14387) (05b6a65).
• storage: have GetObject check for canceled context (backport release-3.1.x) (#14421) (f3d69ff).

3.1.1 (2024-08-08)

• deps: Bumped dependencies versions to resolve CVEs (#13789) (34206cd).

3.1.0 (2024-07-02)

• blooms: Fix duplicate enqueue item problem in bloom download queue when do sync download (#13114) (f98ff7f).
• blooms: Add a missing continue in fuse which may cause incorrect bloom test result (#12650) (0d1ebeb).
• blooms: Allow cluster label override in bloom dashboards (#13012) (987e551).
• blooms: bloomshipper no longer returns empty metas on fetch (#13130) (ad279e5).
• blooms: Clean block directories recursively on startup (#12895) (7b77e31).
• blooms: Correctly return unfiltered chunks for series that are not mapped to any block (#12774) (c36b114).
• blooms: Deduplicate filtered series and chunks (#12791) (3bf2d1f).
• blooms: Disable metas cache on bloom gateway (#12959) (00bdd2f).
• blooms: Do not fail requests when fetching metas from cache fails (#12838) (667076d).
• blooms: Don’t break iterator conventions (#12808) (1665e85).
• blooms: Fix partitionSeriesByDay function (#12900) (738c274).
• blooms: Fix a regression introduced with #12774 (#12776) (ecefb49).
• blooms: Fix findGaps when ownership goes to MaxUInt64 and that is covered by existing meta (#12558) (0ee2a61).
• blooms: Fully deduplicate chunks from FilterChunkRef responses (#12807) (a0f358f).
• blooms: Handle not found metas gracefully (#12853) (37c8822).
• blooms: Reset error on LazyBloomIter.Seek (#12806) (76ba24e).
• blooms: Defer closing blocks iter after checking error from loadWorkForGap (#12934) (cb1f5d9).
• blooms: Fix bloom deleter PR after merge (#13167) (c996349).
• blooms: Fix incorrect sorting of chunks in bloom-filtered response since ChunkRef.Cmp method is used in reverse (#12999) (670cd89).
• blooms: Remove unused parameter causing lint error (#12801) (33e82ec).
• codec not initialized in downstream roundtripper (#12873) (b6049f6).
• Change log level since this is a known case (#13029) (ca030a5).
• Close res body (#12444) (616977a).
• Correct initialization of a few slices (#12674) (0eba448).
• distributor: add detected_level info when the info word appears on log message (#13218) (c9bfa3e).
• docs: Broken link in getting started readme (#12736) (425a2d6).
• docs: Move Promtail configuration to the correct doc (#12737) (1161846).
• docs: Updated Loki OTLP Ingest Configuration (#12648) (ff88f3c).
• Explore logs: Make detected fields work for both json and proto (#12682) (f68d1f7).
• Fix errors reported by the race detector (#13174) (2b19dac), closes #8586.
• Fix for how the loop sync is done (#12941) (5cd850e).
• Fix log level detection (#12651) (6904a65).
• Fix setting of info log level when trying to detect level from log lines (#12635) (0831802).
• lokitool: Fix the lokitool imports (#12673) (6dce988).
• Helm: Fix indentation of query range values in Helm (#12577) (9823f20).
• Helm: Add missing Helm helper loki.hpa.apiVersion (#12755) (3070ea7).
• Helm: Add missing OTLP endpoint to nginx config (#12709) (8096748).
• Helm: Fixes read & backend replicas settings (#12828) (d751134).
• Helm: Set compactor address for distributed mode. (#12748) (521d40a).
• Helm: Fix GEL image tag, bucket name and proxy URLs (#12878) (67ed2f7).
• Helm: Fix query-frontend and ruler targetPort ‘http-metrics’ in Service template (#13024) (1ab9d27).
• Helm: Fix queryScheduler servicemonitor (#12753) (8101e21).
• Helm: Fixed ingress paths mapping (#12932) (5ada92b).
• Helm: Only default bucket names when using MinIO (#12548) (2e32ec5).
• Helm: Removed duplicate bucketNames from documentation and fixed key name deploymentMode (#12641) (0d8ff9e).
• Helm: Ingester zoneAwareReplication (#12659) (9edb0ce).
• Helm: Missing password for Loki-Canary when loki.auth_enabled is true (#12411) (68b23dc).
• indexstats: Do not collect stats from “IndexStats” lookups for other query types (#12978) (1f5291a).
• ingester: Replace channel check with atomic bool in tailer.send() (#12976) (4a5edf1).
• ingester: Not owned stream count (#13030) (4901a5c).
• ingester: Properly return http status codes from ingester to querier for RPC function calls (#13134) (691b174).
• ingester: Track bytes discarded by ingester. (#12981) (88c6711).
• Introduce feature flag for [last|first]_over_time sharding. (#13067) (6e45550).
• Invalidate caches when pipeline wrappers are disabled (#13003) (a772ed7).
• ksonnet: Do not generate RBAC for consul if you are using memberlist (#12688) (2d62fca).
• Lambda-Promtail: Update s3 filename regex to allow finding of log files from AWS GovCloud regions (#12482) (7a81d26).
• Lambda-Promtail: Remove Hardcoded Bucket Name from EventBridge Example CloudFormation Template (#12609) (8c18463).
• LogQL: Fix multiple or filters would get wrong filtertype (#13169) (9981e9e).
• Loki version prefix in Makefile (#12514) (dff72d2).
• Make the TSDB filenames correctly reproducible from the identifier (#12536) (ec888ec).
• metamonitoring: Add missing parentheses in meta monitoring dashboards (#12802) (151d0a5).
• mixins: loki-operational.libsonnet (#12789) (51a841f).
• mixins: Incorrect compactor matcher in loki-deletion dashboard mixin (#12567) (006f88c).
• mixins: Fix compactor matcher in the loki-deletion dashboard (#12790) (a03846b).
• mixin: Mixin generation when cluster label is changed (#12613) (1ba7a30).
• mixin: dashboards $__auto fix (#12707) (91ef72f).
• mixins: Add missing log data source on loki-deletion (#13011) (1948899).
• mixins: Align loki-writes mixins with loki-reads (#13022) (757b776).
• mixins: Remove unnecessary disk panels for SSD read path (#13014) (8d9fb68).
• mixins: Upgrade old plugin for the loki-operational dashboard. (#13016) (d3c9cec).
• nix: Lambda-Promtail vendor hash (#12763) (ae180d6).
• OTLP: Fix parsing of default per tenant OTLP config (#12836) (7cc9a93).
• operator: Add alertmanager client config to ruler template (#13182) (6148c37).
• operator: Bump golang builder to 1.21.9 (#12503) (f680ee0).
• operator: Configure Loki to use virtual-host-style URLs for S3 AWS endpoints (#12469) (0084262).
• operator: Improve API documentation for schema version (#13122) (3a9f50f).
• operator: Use a minimum value for replay memory ceiling (#13066) (4f3ed77).
• Optimize regular initialization (#12926) (a46d14f).
• OTel: Map 500 errors to 503 (#13173) (b31e04e).
• packaging: Require online network in systemd unit file for Loki and Promtail (#12741) (57f78b5).
• Fix panics when ingester response is nil (#12946) (3cc28aa).
• Promtail: Promtail race fixes (#12656) (4e04d07).
• Promtail: Clean up metrics generated from logs after a config reload. (#11882) (39a7181).
• Promtail: Fix bug with Promtail config reloading getting stuck indefinitely (#12795) (4d761ac).
• Promtail: Fix UDP receiver on syslog transport (#10708) (a00f1f1).
• Promtail: Handle docker logs when a log is split in multiple frames (#12374) (c0113db). This changes the current Promtail behavior. Loki has a default max_line_size of 256kb, but With this fix, if the max_line_size config in Promtail has a value, the docker service discovery will now respect that configuration, otherwise it will use the Loki default of 256kb.
• querier: Add retry middleware to the “limited” query roundtripper (#13161) (bb864b3).
• query sharding: Generalize avg -> sum/count sharding using existing binop mapper (#12599) (11e7687).
• regression: reverts grafana/loki #13039 to prevent use-after-free corruptions (#13162) (41c5ee2).
• spans: Corrects early-close for a few spans (#12887) (93aaf29).
• stores: Do not filter out chunks for store when From==Through and From==start (#13117) (d9cc513).
• structured metadata: Add copyString function to symbolizer to avoid retaining memory (#13146) (86b119a).
• Temporarily moving from alloy to alloy dev (#13062) (7ffe0fb).
• Update to build image 0.33.2, fixes bug with Promtail windows DNS resolution (#12732) (759f42d).
• Updated all dockerfiles go1.22 (#12708) (71a8f2c).
• Use to the proper config names in warning messages (#12114) (4a05964).
• workflows: Use an intermediate env variable in GitHub workflow (#12905) (772616c).
• workflows: Don’t run metric collector on forks (#12687) (7253444).