Elasticsearch Updates & Release Notes

Features and enhancements

Machine Learning

• Add EuroBERT and Jina v5 ops to graph validation allowlist #3015
• Better error handling regarding quantiles state documents #2894
• Better handling of invalid JSON state documents #2895
• Better messaging regarding OOM process termination #2841
• Downgrade log severity for a batch of recoverable errors #2889
• Harden pytorch_inference with TorchScript model graph validation #3008 (issue: #2890)
• Improve adherence to memory limits for the bucket gatherer #2848
• Report the actual memory usage of the autodetect process #2846
• Restrict file system access for pytorch models #2851
• Update the PyTorch library to version 2.7.1 #2863

Security

• Update elastic-apm-agent-java8 to 1.55.6 #148271

Fixes

Data streams

• Update failure store redirect logic to exclude backpressure exceptions #148154

ES|QL

• Bugfix - Block Loader Pushdown + Union Types #147940
• Disallow empty lists in named params, only #147748 (issue: #147448)
• ES|QL query approximation: enforce minimum number of sampled source rows #147886

Inference

• Implement RerankingInferenceService for Elastic service #148365
• [Inference API] Fix inference initialization thread exhaustion #147063

Machine Learning

• Fix flaky CIoManagerTest/testFileIoGood test #3017

Vector Search

• [DiskBBQ] Check that precondition should not be overwritten on update #148111 (issue: #148004)

Watcher

• Fix race in TickerScheduleTriggerEngine by checking watcher to node allocation #147678 (issue: #146874)

Original source

Highlights

ES|QL now supports Views: virtual indices whose fields are produced by an ES|QL query. A view is referenced inside a `FROM` clause exactly like a regular index, alongside other indices, views, and wildcards. Complex processing pipelines can be hidden behind a view, exposing a stable set of columns without requiring callers to know the underlying source structure. A single query can combine multiple pre-processed data sources by listing several views in one `FROM` clause, with each view's pipeline running independently. Common transformations such as renames, type conversions, derived fields, and aggregations can be defined once in a view and reused across many queries, dashboards, and alerts. PromQL is now supported as a source command in ES|QL (Tech Preview). Users can now leverage their existing knowledge of PromQL while benefiting from the powerful features and scalability of Elasticsearch. This enhancement expands the versatility of ES|QL and makes it easier for users to integrate with Prometheus data sources.The syntax is illustrated in the following example: ```esql PROMQL index=k8s-downsampled start="2026-02-17T08:00:00Z" end="2026-02-17T09:00:00Z" step=30m avg_bytes=(avg(rate(network.total_bytes_in[30m]))) | SORT avg_bytes DESC, step; ``` ES|QL adds the `METRICS_INFO` command for queries that start with a time series (`TS`) source. It returns one row per distinct metric, with columns such as `metric_name`, `data_stream`, `unit`, `metric_type`, `field_type`, and `dimension_fields`, derived from time series metadata in the index. It unlocks inspecting which metrics exist and how they are typed before you aggregate with `STATS`.For example, list metrics sorted by name: ```esql TS my_data_stream | METRICS_INFO | SORT metric_name ``` Or filter to counters only: ```esql TS my_data_stream | METRICS_INFO | WHERE metric_type == "counter" | SORT metric_name ``` We're introducing a Prometheus-compatible `POST /_prometheus/api/v1/write` REST entrypoint that allows receiving data via Prometheus remote write protocol (Tech Preview). Elasticsearch can now be used as a Prometheus storage backend, consuming data sent in Prometheus native format. ES|QL adds the `TS_INFO` command for time series (`TS`) queries. It returns one row per metric and time series combination. You get the same metadata columns as `METRICS_INFO`, plus a `dimensions` column with a JSON object of dimension keys and values for that series. That unlocks inferring which labels apply to each series when exploring or validating time series data.For example: ```esql TS my_data_stream | TS_INFO | SORT metric_name, dimensions ``` Until Elasticsearch `9.3`, both downsampling methods (`aggregate` and `last_value`) used to store only the last value of a counter in the downsampled document. This works great for the `last_value` method where we optimise for storage efficiency, but it is not ideal for the `aggregate` method where we optimise for accuracy.In Elasticsearch `9.4`, we change the way the (default) `aggregate` sampling method is working. We store the first encountered value for a counter in the downsampled document and then we add auxiliary documents when we detect counter resets. This enables the rate calculation to take the counter resets into account and produce more accurate results. This change is backwards compatible. Time series aggregations in ES|QL are enhanced to support windows smaller than the time bucket. ```esql TS metrics | STATS AVG(RATE(requests, 5m)) BY TBUCKET(10m), host ``` Previously, only window values that were equal or exact multiples of the time bucket were supported. Time series aggregations in ES|QL are enhanced to support windows that are not an exact multiple of the time bucket. ```esql TS metrics | STATS AVG(RATE(requests, 15m)) BY TBUCKET(10m), host ``` Previously, only window values that were exact multiples of the time bucket were supported. This updates our diskbbq algorithm and format. - It now provides 3x or more better search performance on very restrictive filters (prefilters on centroids) - Provides a way to condition non iid vectors (expert API for now) - Gives more bit options (1, 2, 4, and 7 bits!) - More native code improvements for overall performance The `_id` field has a significant storage footprint in metrics applications, as it requires both storing and indexing unique document identifiers that are rarely used for direct lookups. To alleviate this, we are introducing synthetic IDs for indices in time-series mode. Instead of indexing the `_id` field, a Bloom filter is used for fast, lightweight duplicate detection at ingest time. Lookups and operations that previously relied on `_id` are delegated to other indexed fields on the document, such as timestamps, or dimension fields, preserving the same query and retrieval functionality.This offers up to 40% storage improvement for OTLP metrics and reduces the cpu overhead for segment merging due to the lack of an inverted index for `_id` fields. We're introducing a Prometheus-compatible /_prometheus/api/v1/query_range REST endpoint (Tech Preview) that: - Accepts the standard Prometheus range query parameters (query, start, end, step, optional index) - Translates the PromQL expression into an ES|QL PROMQL command and executes it via EsqlQueryAction - Converts the columnar ES|QL response into the Prometheus matrix JSON format and returns it to the caller We're introducing a Prometheus-compatible `GET /_prometheus/api/v1/series` REST entrypoint that accepts Prometheus series selectors and returns matching label sets (Tech Preview). This is typically used for auto-completion in web UIs. The first bytes of a time series id (tsid) include a hash of the metric name(s) for each doc of a time-series index. Counter rate evaluation leverages these bytes to assign tsids to workers inside the ES|QL compute engine. This (a) improves parallelism by dividing work in a granular and uniform fashion, and (b) leads to dense, sequential access patterns per time series that have been optimized to avoid copies between counter value decoding and rate calculations.Rate execution performance thus improves substantially, with up to 5x faster query responses. We're introducing a Prometheus-compatible `GET /_prometheus/api/v1/labels` REST entrypoint for time series discovery and label enumeration and introspection (Tech Preview). Web UIs can use this for label auto-completion. We're introducing a Prometheus-compatible `GET /_prometheus/api/v1/query` REST endpoint that evaluates a PromQL expression at a single point in time and returns vector results (Tech Preview). The instant query endpoint currently runs a short range query under the hood and returns the last sample. In Elasticsearch `9.4` we expand the supportability of `aggregate_metric_double` to include non-native operations in ES|QL, such as `std_dev`, using the average. The average is calculated using the `sum` and `value_count` sub-fields. The average was selected because in most cases it is a more representative signal compared to a single sub-field. Native operations such as `max`, `min`, `sum`, `avg`, and `count` will be supported natively by the respective sub-fields.For example, the following query is now supported where `network.eth0.tx` is a an `aggregate_metric_double`: ```esql FROM k8s-downsampled | STATS max = max(network.eth0.tx), std_dev = STD_DEV(network.eth0.tx) by pod | sort pod ``` Response: ``` max:double | std_dev:double | pod:keyword 1060.0 | 275.6970067 | one 824.0 | 184.1213952 | three 1419.0 | 356.9865993 | two ```

Features and enhancements

Aggregations:

• Bump heap usage limits for INLINE STATS #144679

Analysis:

• Inject circuit breaker into forked SynonymMapBuilder #144800
• Support custom rulesets in analysis-icu/icu-transform plugin #143060

Authentication:

• Add Clone API Key endpoint #142633 (issue: #59304)

Authorization:

• Update View CRUD Actions to be Index Actions #141570
• [Entity Store] Add permissions for Entity Store datastream #145981

CCS:

• CPS and project routing support for templated searches #139446

CRUD:

• Do not mark bulk indexing requests as retried after primary relocations #142157 (issue: #141586)

Codec:

• Add dynamic bloom filter sizing based on document count #141342
• Add panama simd implementation of contains function for BinaryDocValuesContainsTermQuery #143922
• Allow loading BYTE_LENGTH without decompressing Zstd byte ref blocks #141322
• ES819 Binary doc values: compact doc offsets using bit packing #142772
• Enable large blocks for binary doc values by default. This mainly affects fields of type wildcard, ignored source, values hitting ignore above threshold and ignore malformed numbers and dates. #145216
• Fast codePointCount implementation for BytesRef #140388
• Push contains binary doc values query down to es819 codec #143898
• Rewrite *substring* wildcard queries to contains term queries for binary doc values keywords #143433
• Track bloom filter disk usage in IndexDiskUsageAnalyzer #142106
• Upgrade zstd to version 1.5.7 #140530
• Use DirectAccessInput in ZstdDecompressor to avoid intermediate heap copy #145658
• Use max instead of median for merged bloom filter size #143302
• CodePointCount implementation using Panama vectors API #140693 (issue: #140567)

Data streams:

• Add 'logs.otel' and 'logs.ecs' stream types #141564 (issue: #141040)
• Ensure DLM only runs one general loop at a time #143883
• Support Failure Stores in Cross Cluster Search #139316

Distributed:

• Batch index creation #144074
• Batch snapshot update tasks after external change #142091
• Ensure that synthetic _id is usable after restarts/relocations #138678
• Health reports GREEN when provisionally unassigned replica #144773
• Increase the per-index limit for merges to half the CPUs #141389
• Opt-in persistent task reassignment on node shutdown #143306

Downsampling:

• Collect dimensions only once per tsid when downsampling #145089
• Rate calculation for downsampled counters becomes aware of counter resets when the aggregate sampling method is used. #143381 (issue: #136178)
• Use the tdigest type and compression from TDigest in downsampling #143247

ES|QL:

• Add APM telemetry for SET statement #141719
• Add Arrow-native Block & Vector implementations #142981
• Add CCS Remote Views Detection #143384
• Add Connector SPI and gRPC/Arrow Flight module #142667
• Add Google Cloud Storage data source plugin #142563
• Add JSON_EXTRACT ES|QL scalar function #142375
• Add LZ4, Snappy, and Brotli decompression codecs #144688
• Add METRICS_INFO command #141667 (issue: #139296)
• Add MMR command for result diversification #143867
• Add MV_UNION Function #139664
• Add ORC predicate pushdown via SearchArgument #144686
• Add Parquet filter pushdown with bloom filter, statistics, and dictionary row-group skipping #144832
• Add TS_INFO information retrieval command #142721 (issue: #139296)
• Add Views Security Model #141050
• Add Warning for Sort Under Lookup Join #141482 (issue: #141483)
• Add FormatReadContext to consolidate FormatReader API #143928
• Add IntRangeVector for selected groups in aggregation #141205
• Add LongLongSwissHash - specialization for grouping by two long fields #140838
• Add appliesTo to the TRange and TBucket functions #142160
• Add anonymous Azure access via auth=none #144475
• Add anonymous GCS access via auth=none #144476
• Add anonymous S3 access via auth=none #144471
• Add blocks and vectors for more Arrow numeric types #145111
• Add cloud API rate limiting for external sources #144734
• Add column pruning for external datasources #143903
• Add configurable bracket-based multi-value support for CSV reader #143890
• Add coordinator-only caching for external source metadata #145300
• Add data node execution for external sources #143209
• Add dense_vector equality and inequality support in ES|QL #140005 (issue: #139929)
• Add error handling and propagation for external source execution #143333
• Add error policy and configurable options for CSV format reader #143779
• Add extended distribution tests and fault injection for external sources #143420
• Add info into the profile of METRICS_INFO and TS_INFO #145634
• Add limit pushdown for external data sources #143515
• Add local parallelism and partition detection for external sources #143154
• Add logic to fold project tags metadata on data nodes #141935
• Add mapper-size plugin's _size metadata attribute #141427 (issue: #136956)
• Add memory tracking for TS_INFO and METRICS_INFO #143491 (issue: #139296)
• Add parallel execution for Arrow Flight multi-endpoint sources #143345
• Add parameter support in PromQL query durations #139873 (issue: #139508)
• Add pluggable partition detection and virtual columns #143120
• Add positional readBytes API to StorageObject SPI #143703
• Add schema reconciliation for multi-file external sources #145220
• Add split SPI, partition detection, and filter hint extraction #143005
• Add split discovery and distribution for external sources #143114
• Add support for ORC file format #142900
• Add support for dense_vector in COALESCE #142974 (issue: #139928)
• Add support for binary operators with AMD #143996 (issue: #142094)
• Add support for project METADATA #140592
• Add support for top-level arithmetic ops to TS|STATS #140135 (issue: #139570)
• Add syntax support and parsing for SET approximate #139908
• Add telemetry (stack) for query settings #141836
• Add timezone to add and sub operators, and ConfigurationAware planning support #140101
• Add xerial snappy-java to compression-libs #145393
• Added three new simple but useful spatial functions: ST_Dimension, ST_GeometryType, ST_IsEmpty #144703
• Added timezone support to date_format #138517
• Adding ES|QL USER_AGENT command #144384 (issue: #134886)
• Adding ES|QL command REGISTERED_DOMAIN #142680 (issue: #133942)
• Adding ES|QL command URI_PART #140004 (issue: #134885)
• Adding MV_INTERSECTS function #140662
• Adding sparkline aggregate function #141388
• Adds LIMIT BY ESQL command in Tech Preview #145225 (issue: #112918)
• Adds ST_SIMPLIFY geospatial function #136309 (issue: #44747)
• Allow TBUCKET to skip the from/to parameters when Kibana adds a timestamp range filter. Exmaple: TBUCKET(100) #144057
• Allow evaluatable grouping functions (Like BUCKET) in LIMIT BY #146642
• Attribute ES|QL shard search load in Lucene operators #142841
• Avoid caching multiple times in doc-partitioning #142913
• Bridge Connector SPI to ExternalSplit #143331
• Buffer reuse in ParquetStorageObjectAdapter and StorageObject #143700
• Byte-based buffer backpressure for external sources #144218
• CSV schema inference and parsing enhancements #144050
• Case Support for Compound Types #140677
• Converted PackedValuesBlockHash.bytes to BreakingBytesRefBuilder for better memory tracking #140171
• Count aggregation for histograms #141138
• DS: Parquet file handling improvements #145123
• Data sources: Azure plugin #143236
• Data sources: ZSTD, BZIP2 #143228
• Datasources: GZIP #143035
• Document and test Parquet page-index filtering #145571
• ESQL - Add dense_vector field type to SUM function #142129
• ESQL - Improve search performance by adding min competitive aware collection when using multiple shards / threads #142406 (issue: #136267)
• ESQL 137269 some csv tests for lookup join behavior with multivalues #144520
• ESQL mv_difference function #141895
• ESQL: Improve field reference tracking in FORK command #137678 (issue: #137283)
• ESQL: Prune unused regex extract nodes in optimizer #140982 (issue: #132437)
• ESQL: Support intra-row field references in ROW command #140217 (issue: #140119)
• ESQL: enable unmapped_fields="load" in tech preview #145052 (issue: #142369)
• ES|QL - Add parsing, preanalysis and analysis timing information to profile #139540
• ES|QL - Top N queries are parallelized #143133
• ES|QL - dense_vector support for COUNT, PRESENT, ABSENT aggregator functions #139914 (issue: #135688)
• ES|QL CHUNK function multi-valued field support #141240
• ES|QL Improve LOOKUP JOIN on single keyword #144704
• ES|QL Top Snippets multi-valued field support #142117
• ES|QL Views support #134995
• ES|QL TEXT_EMBEDDING function is GA #140555
• ES|QL dense vector functions are GA #140545
• ES|QL approximate analytical queries #131828
• ES|QL command RERANK is GA #141508
• Enable PromQL command in ES|QL #140808
• Enable distributed pipeline breakers for external sources via FragmentExec #143696
• Enable doc-partitioning for more queries #143095
• Extract centroid from doc values for ST_CENTROID_AGG over geo_shape and cartesian_shape #142528 (issue: #142640)
• Fix ORC type support gaps #145074
• Fix Parquet and ORC datasource allocation overhead #143791
• Fix Parquet type support gaps #144059
• Fix review feedback and add test coverage for PR #143703 #143900
• Fix window validation in time-series aggregations when TBUCKET uses a numeric target count #144291
• Format "_query" response dates using the given timezone #139529
• GCS native async I/O via ReadChannel #144733
• Harden distributed external source execution #144277
• Implement EXPLAIN for local data node plans #142748
• Implementing rerank on multi values #140672
• Improve Lookup Join performance with CachedDirectoryReader #139314 (issue: #137268)
• Improve memory usage and tracking by moving union types into ValuesSourceReaderOperator #140384
• Improve ndjson schema inference for date-time #145553
• Introduce "Swiss Table"-based hashing to ES|QL, a SIMD-accelerated hash table resulting in significantly higher throughput on uniform, high-cardinality workloads #145010
• Introduce Geospatial functions ST_Buffer and ST_SimplifyPreserveTopology #145154
• Introduce SwissTable-based hashing for ES|QL STATS #139343
• Introduce adaptive block hash for long/int #141237
• JSON_EXTRACT: zero-copy byte slicing for object, array, and number extraction #143702
• LIMIT BY fixed telemetry and tests #146992
• MMR Command: Grammar and Logical Plan #140684
• Make MV_EXPAND GA #144543
• Make datasources plugins lazy #142815
• Minimize Hadoop dependencies for ORC plugin #146944
• Optimize TopNOperator to avoid resorting when input is already sorted #141094 (issue: #131221)
• Partition rate query using tsid prefixes #144818
• Per-file filter pushdown awareness #145755
• Periodically emit partial aggregation results #141392
• Push STARTS_WITH/LIKE prefix to Parquet and ORC #145640
• Push stats to external source via metadata #143940
• Reapply "Introduce pluggable external datasource framework" #142707
• Reapply "NDJSON datasource" #142855
• Refactor inference operator architecture for multi-value field support #139694
• Register TSV as a separate format with tab delimiter #143906
• Remove Hadoop JARs from Parquet plugin #146780 (issue: #146716)
• Remove hadoop-client-runtime from datasource plugins #146206 (issue: #146203)
• Remove implicit limit appended for each subquery branch #139058
• Remove implicit limit for FORK #145429
• Remove snapshot protection from node reduce late materialization #142834
• Review fixes for datasource framework #142565
• Route external source I/O through esql_worker thread pool #144596
• Schema-aware filter pushdown for DATETIME and DECIMAL #145641
• Shrink description #140089
• Skip files with no projected column overlap in UNION_BY_NAME #145701
• Skip time series field type merge for non-TS agg queries #143262
• Speed up remote Parquet reads #144454
• Stats pushdown past EVAL/RENAME for external sources #144806
• Stream results from topn #140088
• Support arithmetic operations for dense_vectors: scalar version #141060 (issue: #140538)
• Support arithmetic operations for dense_vectors: vector version #140539 (issue: #140537)
• Support of a window that is not an exact multiple of the bucket #143704
• Support shapes in ST_CENTROID_AGG #141657
• Support target bucket count in TBUCKET with explicit from/to date range #142747
• Support window smaller than time bucket #143661
• TRange timezone support #139911
• Type conflict resolution in unmapped-fields load #143693 (issues: #142004, #141912)
• Use avg metric for AMD default metric #141331
• Use less memory in ValuesFromMany #140062
• Validate TOP_SNIPPETS query argument is foldable at verification #142763 (issue: #142462)
• Various fixes to spatial functions (ST_ENVELOPE and ST_NPOINTS) #139618
• [ES|QL|DS] Add circuit breaker to the Parquet datasource #144491
• [ES|QL|DS] Parquet row-group level split parallelism #144018
• [ES|QL|DS] Wire parallel parsing into production for text formats #143997
• ToString/ToDatetime/ToDateNanos converters timezone support #138985
• support DATE_RANGE field type #133309
• Add CHICKEN function to ES|QL #140645

Engine:

• Ensure acquired snapshot commit is always flushed #144067 (issue: #143993)

Indices APIs:

• More actionable PUT /{index}/_settings error #138611

Inference:

• Add FireworksAI chat completion support #142664
• Add FireworksAI inference service for embeddings #137130
• Add embedding task support to ElasticInferenceService #141547
• Add provider validation call to Update Inference Endpoint operation #140003 (issue: #122356)
• Added Reasoning support for Chat Completion in the Inference Plugin #143242
• Added service settings update logic for AI21 provider in the Inference Plugin #142597 (issue: #122356)
• Added service settings update logic for Alibaba Cloud Search provider in the Inference Plugin #142738 (issue: #122356)
• Enable multimodal inputs for all chat completion integrations #144509
• Removed the max_tokens request parameter for Chat Completion with Reasoning in the Inference Plugin #143242
• [Inference API] Add Chat Completion to Amazon Bedrock for the Inference API #139411
• [Inference API] Add custom headers for Azure OpenAI Service #142969
• [Inference API] Add support for embedding task to JinaAI service #140323
• [Inference API] Adding OAuth2 support for Azure OpenAI #143896
• [Inference API] Expose Endpoint Heuristics through Inference API #141393
• [Inference API] Handle preconfigured endpoints with embedding task type #141788
• [Inference API] Parse endpoint metadata from persisted endpoints #143081
• [Inference API] Support multimodal inputs for chat completion #142736
• [Inference API] Update authorized endpoints when their fingerprint or version changed #143567

Infra/Core:

• Add DateFormatter.tryParse() #144474
• Expose byte offsets on XContentParser via getCurrentLocation() #143501 (issue: #142873)

Infra/Plugins:

• [Fleet] Add OpAMP field mappings to fleet-agents #142550
• [Fleet] Add metadata mappings for OpAMP #145824

Infra/Scripting:

• Painless hoist constant collection .contains calls #143311 (issue: #137849)

Ingest Node:

• Update Grok to use the new Matcher#setTimeout #139405
• [INGEST] GrokProcessor: add validate_only option to skip field extraction #145126

Logs:

• Default index.mapping.use_doc_values_skipper to true for logsdb #142851
• Store fallback match only text fields in binary doc values #140189

Machine Learning:

• Add EuroBERT and Jina v5 ops to graph validation allowlist #3015
• Add a suggestion for fixing the ML node allocation error #139520
• Add exponential-backoff retry for AD job opening during system-initiated reassignments #144478
• Add support for nested NDJSON records in TextStructure endpoints #141045 (issue: #127777)
• Better error handling regarding quantiles state documents #2894
• Better handling of invalid JSON state documents #2895
• Better messaging regarding OOM process termination #2841
• Downgrade log severity for a batch of recoverable errors #2889
• Harden pytorch_inference with TorchScript model graph validation #3008 (issue: #2890)
• Improve adherence to memory limits for the bucket gatherer #2848
• Report the actual memory usage of the autodetect process #2846
• Restrict file system access for pytorch models #2851
• Update the PyTorch library to version 2.7.1 #2863

Mapping:

• Add option to enable accurate leaf arrays for flattened fields #145376
• Add passthrough support to flattened field type for mapped sub-fields #145131
• Add properties support to flattened field type #144451
• Aggregate metric double use average #142135
• Improve the supportability of aggregate_metric_double by non-native ES|QL aggregation functions, such as std_dev. #145742
• Remove redundant root doc values from flattened fields if index=false #143907
• Set default semantic_text index type to disk_bbq by using dense_vector defaults #145374
• Store flattened field data in binary doc values #140246
• Update semantic text to use BFLOAT16 by default #144236

Monitoring:

• Add mode and codec fields to Stack Monitoring index template #143673

Packaging:

• Flip cloud-ess-fips default from FIPS 140-2 to FIPS 140-3 #140788

Performance:

• Allow intermediate builds in PR-based benchmarks #142472
• Correctly reference non-main branches in benchmark script #142303
• Relax PR-based benchmarks target branch #142297

PromQL:

• Add Prometheus instant query REST endpoint #145321
• Add Prometheus labels REST endpoint #144952
• Add Prometheus query_range endpoint #144416
• Add Prometheus series REST endpoint #144494
• Implement Prometheus remote write indexing support #141957

Ranking:

• Use VectorScorer to consume AcceptDocs iterator for lazy bulk scoring in VectorScoringUtils #145835 (issue: #145834)

Reindex:

• Add reindex-from-remote blocklist setting #145357
• Disable OCC in update/delete-by-query for seq_no-less indices #143465

Relevance:

• GA chunk_rescorer in text_similarity_reranker #139830

SQL:

• Add project_routing to CLI #138965
• Add support for API key to JDBC and CLI #142021

Search:

• Account for ES|QL Lucene query rewrite in recent search load #141819
• Add semantic_text field type to MMR Result Diversification Retriever #141666
• Add search task watchdog to log hot threads on slow search #142746
• Added return_intermediate_results query param to toggle when partial results are returned for a get async results operation #141073 (issue: #139828)
• CPS handles datastreams #140637
• Expose keep_alive in async task status #144010
• Fail MatchQueryParser if it generates a query with more clauses than allowed by max_clause_count #143233 (issue: #143032)
• Ids Query: Use max result window as upper limit #140515 (issue: #138758)
• Makes scroll CPS compatible #140977
• Making use of sort optimization written from search in search shards #144247 (issue: #143945)
• Only consider the primary sort when determining concurrency #143608
• Optimize script sorts that do not require query scores #139748
• Optimize search shard iterator sort #140747 (issue: #135472)
• PIT context relocation work on main repo #137675
• Prevent creating too many nested boolean clauses while creating the lucene query to avoid query explosion #143220
• Ref-counting SearchHits from InternalTopHits to SearchResponse #142732
• Search/query logging support for _search, ES|QL, EQL, SQL #139920
• Semantic text default inference id setting #143486
• Switch default model for semantic_text to jina-v5 #142980
• Take control of max clause count verification in Lucene searcher #139752
• Update text_similarity_rank_retriever to default to chunking settings optimal for inference ID #137397
• Upgrade Elasticsearch to Apache Lucene 10.4 #141882
• Use IndexOrDocValuesQuery in IpFieldType#termQuery #140735
• Use IndexOrDocValuesQuery in NumberFieldType#termQuery implementations #140734
• CanMatch returns numSkipped per cluster instead of all skipped shards #142170

Searchable Snapshots:

• Add SparseFileTracker.getAbsentBytesWithin #141179
• Split blob-cache freelist using decays #142545
• Trigger cache decay at 5% left on freq 0 #142685

Security:

• Allow deleting multiple views in one request #145816
• Don't allow querying views with DLS or FLS #144903
• Make ServiceAccountToken APIs Available in Serverless #140631
• Upgrade bouncycastle to 1.84 #147197
• Use opaque random session IDs for ESQL compute sessions #142249

Snapshot/Restore:

• Batching of snapshot-delete start updates #141998
• Identify Elasticsearch as user-agent in S3 calls #141881
• Reduce memory usage of TransportGetSnapshotsAction #142468
• Report shard snapshot pauses in shutdown status #144717
• Strengthen MPU-based CAS in S3 repo

Original source

- Elastic Stack: Planned

Features and enhancements

Aggregations

• Bump heap usage limits for INLINE STATS #144679

ES|QL

• Skip time series field type merge for non-TS agg queries #143262

Fixes

EQL

• Fix propagation of filters on join keys for missing events #145813 (issue: #145402)

ES|QL

• Do not discard disjunction conditions when is null/is not null might invalidate them #145941
• Don't use a Literal for constant_keyword fields when used inside full-text functions #145632 (issue: #145570)
• ESQL - Fix performance loading source when vectors are excluded #146223 (issue: #145799)
• Fix handling of values on the time bucket boundaries for ES|QL increase #145794
• Fix nested fields loading under NULLIFY #145741 (issue: #142616)
• Fix rate/increase single-value bucket handling for delta temporality #146518
• Fix starts_with/ends_with with special chars #146348 (issue: #130851)
• Keywords mv count fix #145390

Infra/Core

• Throw a 400 error for malformed parsing input when missing element end #145777

Ingest Node

• Fix pipeline resolution cache for bulk requests #144648
• Fix waiting for enrich policy execution for users without the monitor privilege #145751

Machine Learning

• Omit uncomputed model stats #146186

Mapping

• Fix match only text decoding surrogate pairs #146567 (issue: #146538)

Reindex

• Restore initial thread context during reindex etc #146134

Search

• Add cancellation support to IndicesRequestCache #141708
• Collapse pathological regex quantifier stacking to prevent NFA construction OOM #145452
• Fix terminate_after not honored for aggs when size=0 #146199 (issue: #126665)
• Fix bug parsing "request" parameter in clear cache API, it should clear the request cache only #145726
• Use query circuit breaker for wildcard/regexp determinization #145427 (issue: #145128)

Snapshot/Restore

• Use IllegalArgumentException over RepositoryException for readonly-repository checks #140200

TSDB

• Replace IllegalStateException with IllegalArgumentException for conflicting time series metadata #142370

Vector Search

• Fix NPE in GPU resource pool when CuVSResources creation fails #146632
• Fix NPE when having double nested field with knn query #146933 (issue: #141830)
• [DiskBBQ] Fix offHeap size for empty indices #146347
• [DiskBBQ] Wait for queue saturation in MaxScoreTopKnnCollector #145341

Original source

Features and enhancements

Security

• Skip automaton construction for literal resource strings in application privilege checks #144685

Fixes

CAT APIs

• Fix ArrayIndexOutOfBoundsException in cat nodes for partial load average arrays #144859

CCR

• CCR follower index needs to copy transport version from CCR leader index #145035

ES|QL

• Cancel async query on expiry #143016 (issue: #142662)
• Don't over-alloc when unpacking dimension values #144525
• ESQL: Fix incorrectly optimized fork with nullify unmapped_fields #143030 (issue: #142762)
• ESQL: Fix null comparison type checking #140660 (issue: #140460)
• Fix field caps incorrectly synthesizing object parents under subobjects:false passthrough mappers #144183 (issue: #144179)
• Reduce LuceneOperator.Status memory consumption with large QueryDSL queries #143175 (issue: #143164)

FIPS

• Fix createLDAPCertStore failing in FIPS mode #144453 (issues: #144376, #144377)

Geo

• Fix geo_centroid over geo_shape merging multiple shards #144637 (issue: #144504)

Infra/Core

• Fix system index mapping update for reindexed indices after migration #144782 (issue: #144764)

Machine Learning

• Fix "reset anomaly detection API" crashing when only colliding indices exist #144545 (issue: #144544)

Mapping

• Lazily create matcher in BinaryDvConfirmedQuery #144698

Search

• Fix ArrayIndexOutOfBoundsException in fetch phase with partial results #144385 (issue: #140495)
• Fix circuit breaker leak in percolator query construction #144827
• Fix request cache invalidation to use ES cache helper consistently #144581
• Fix wrong return value in ContextIndexSearcher.totalTermFreq #144333

Snapshot/Restore

• Fix up exception messages in AzureBlobStore #144654

Vector Search

• Fix dense_vector default index options when using BFLOAT16 #145202 (issue: #145204)
• [DiskBBQ] Fix index sorting on flush #144938
• [DiskBBQ] Fix index sorting on flush (2nd attempt) #145076

Original source

Features and enhancements

Authentication

• Improve SAML error handling by adding metadata #137598 (issue: #128179)
• In-response-to in saml successful response #137599 (issue: #128179)

Security

• Skip automaton construction for literal resource strings in application privilege checks #144685

Fixes

CAT APIs

• Fix ArrayIndexOutOfBoundsException in cat nodes for partial load average arrays #144859

CCR

• CCR follower index needs to copy transport version from CCR leader index #145035

ES|QL

• Cancel async query on expiry #143016 (issue: #142662)
• Reduce LuceneOperator.Status memory consumption with large QueryDSL queries #143175 (issue: #143164)

FIPS

• Fix createLDAPCertStore failing in FIPS mode #144453 (issues: #144376, #144377)

Geo

• Fix geo_centroid over geo_shape merging multiple shards #144637 (issue: #144504)

Infra/Core

• Fix system index mapping update for reindexed indices after migration #144782 (issue: #144764)

Machine Learning

• Fix anomaly detection jobs stuck in opening by continously retrying to set the state to opened. #139668
• [Inference API] Support chunking settings for sparse embeddings in custom service #138776

Search

• Fix ArrayIndexOutOfBoundsException in fetch phase with partial results #144385 (issue: #140495)
• Fix circuit breaker leak in percolator query construction #144827
• Fix request cache invalidation to use ES cache helper consistently #144581
• Fix wrong return value in ContextIndexSearcher.totalTermFreq #144333

Snapshot/Restore

• Fix up exception messages in AzureBlobStore #144654

Vector Search

• [DiskBBQ] Fix index sorting on flush #144938
• [DiskBBQ] Fix index sorting on flush (2nd attempt) #145076

Watcher

• Validate Watcher Proxy Allowlist #144759

Original source

Features and enhancements

Authentication

• Bump Kiota Libs #143556

Authorization

• [JupiterOne] Add manage, create_index, read, index, write, delete, permission for third party agent indices kibana_system #140049
• [Sentinel One] Add manage, create_index, read, index, write, delete, permission for third-party agent indices in the Kibana system to support the unified alert data stream. #142648

Monitoring

• Apm-server: update monitor mappings to include new TBS metrics #140700

Search

• Fix nested object float arrays being mapped to dense vectors when they are mapped in dynamic template mappings #143733 (issue: #143732)

Transform

• Skip checkpoint query filter when runtime_mappings are present #142452

Fixes

Aggregations

• Fix ClassCastException when merging TopHits with mixed sort field types #141919 (issue: #141714)

CCS

• Fix: _resolve/index API should be able to accept an empty body #143159

Data streams

• Apm-data: explicit map of timestamp.us to long #143173
• Expand DLM user to allow interaction with .workflows-events #143958

ES|QL

• Account for missing StubRelation due to SurrogateExpressions replacement #142882 (issue: #142219)
• ESQL - enable zero_terms_query option in MATCH function #143668 (issue: #143070)
• Fix KQL/QSTR with unmapped fields in NULLIFY mode #143399 (issues: #142968, #142959)
• Fix incorrect nullify with unmapped fields #142300 (issue: #141870)
• Fix nullify where in stats #144029 (issue: #143991)
• Fix unresolved name pattern #143210
• Promptly clean up CCS exchange sinks #143325
• Skip nullifying aliases for Aggregate groups. #141340
• TS command ignores aliases in BY #143489

Highlighting

• Fix UnsupportedOperationException when using a plain highlighter with a query on a field used for index sorting (index.sort.*). #143680

Machine Learning

• Fix Duplicate ML Model Allocations on Same Node #142872
• Fixed stats API to use correct allocation count for required_native_memory_bytes calculation #143077 (issue: #107831)

Mapping

• Apply the source filter on metadata field mappers when loading from synthetic source #143726 (issue: #143464)

Search

• Add circuit breaker for query construction to prevent OOM from automaton-based queries #142150
• Add timeout support for KNN searches in the DFS phase #142925

Security

• Disable CAE in microsoft-graph-authz plugin #142848 (issue: #142743)
• Fix use-after-free in SearchApplicationIndexService buffer lifecycle #143134

Snapshot/Restore

• Fix cancellation race in CancellableRateLimitedFluxIterator #141974

Transform

• [ML]Fix latest transforms disregarding updates when sort and sync fields are non-monotonic #142856 (issue: #90643)

Vector Search

• Fix GPU merge ClassCastException with wrapped directories #143531

Original source

Features and enhancements

Authentication

• Bump Kiota Libs #143556

Authorization

• [JupiterOne] Add manage, create_index, read, index, write, delete, permission for third party agent indices kibana_system #140049
• [Sentinel One] Add manage, create_index, read, index, write, delete, permission for third-party agent indices in the Kibana system to support the unified alert data stream. #142648

Monitoring

• Apm-server: update monitor mappings to include new TBS metrics #140700

Search

• Fix nested object float arrays being mapped to dense vectors when they are mapped in dynamic template mappings #143733 (issue: #143732)

Transform

• Skip checkpoint query filter when runtime_mappings are present #142452

Fixes

Aggregations

• Fix ClassCastException when merging TopHits with mixed sort field types #141919 (issue: #141714)

Data streams

• Apm-data: explicit map of timestamp.us to long #143173

ES|QL

• ESQL - enable zero_terms_query option in MATCH function #143668 (issue: #143070)
• Promptly clean up CCS exchange sinks #143325

Machine Learning

• Fix Duplicate ML Model Allocations on Same Node #142872
• Fixed stats API to use correct allocation count for required_native_memory_bytes calculation #143077 (issue: #107831)

SQL

• Fix QlIllegalArgumentException with non-foldable date range queries #142386 (issue: #137365)

Search

• Add circuit breaker for query construction to prevent OOM from automaton-based queries #142150
• Add timeout support for KNN searches in the DFS phase #142925

Security

• Disable CAE in microsoft-graph-authz plugin #142848 (issue: #142743)
• Fix use-after-free in SearchApplicationIndexService buffer lifecycle #143134

Snapshot/Restore

• Fix cancellation race in CancellableRateLimitedFluxIterator #141974

Transform

• [ML]Fix latest transforms disregarding updates when sort and sync fields are non-monotonic #142856 (issue: #90643)

Original source

Features and enhancements

Packaging

• Updating bundled jdk version to 25.0.2+10 #142500

Fixes

Cluster Coordination

• Suppress success callback when failing master task #142042

ES|QL

• Avoid a possible NPE by throwing an EIAE instead with more info #141711 (issue: #141267)
• Deep copy BytesRef when creating a constant vector block #141242 (issues: #140615, #140809, #140621)
• Dynamically grow hash in linear counting in HLL #142047 (issue: #41847)
• ESQL fix TO_IP leading_zeros=octal parsing #141776 (issue: #141627)
• Fix IP_PREFIX function leaking data in scratch #141940 (issue: #141628)
• Fix NPE with null field parameter #142328 (issue: #142281)
• Fix bug with multiple spatial aggs filtering in ES|QL #142332 (issue: #142329)

Inference

• [Inference API] Do not write "task" field in Jina embedding request if unsupported #142181
• [Inference API] Fix ChunkingSettings field missing from ModelConfigurations equals method #142238
• [Inference API] Prevent trailing slashes from being included in URLs #141692

Infra/Core

• Reindexing older indices with percolator fields clears migration assistant errors #141539

Infra/Logging

• Fix cluster name in ECS upgrade #141792

Machine Learning

• Add missing job_id filter to Anomaly Detection data deleter #138160
• Reduce locking when persisting ML job statistics #141519 (issue: #140511)

Mapping

• Allow shadowing time series metrics and dimension in non time series indexing #141549 (issue: #140882)

Ranking

• Fixing for NPE when there is no query specified for the standard retriever #142479 (issue: #142336)
• Implement comprehensive top N parameter handling for text similarity reranker #142039

Search

• Ensure Rewriteable.rewriteAndFetch listeners are not executed on transport threads #141904
• Fix Top Hits Incompatible Field Types on Sort Across Indices #142046 (issue: #141906)
• Fix handling empty collapse construct #141973 (issue: #139299)
• Prevent large CancelTasksRequest descriptions by truncating nodes and actions #141815
• Reduce cancellation check interval in CancellableBulkScorer for better responsiveness #141747

Snapshot/Restore

• Ensure paused shard snapshot can be deleted #141408

Transform

• Fix transform producing empty dest index when source query references runtime fields #142450 (issue: #113156)
• Fix transform validation to reject PUT and _start when user lacks remote index permissions #142403 (issue: #95367)

Original source

Features and enhancements

ES|QL

• Fix ILM's 'skipping policy' logging level #141890 (issue: #141876)

Packaging

• Updating bundled jdk version to 25.0.2+10 #142500

Fixes

Allocation

• Don't overwrite decision with NOT_PREFERRED unless its an improvement #141565

Cluster Coordination

• Suppress success callback when failing master task #142042

Data streams

• Allow include_source_on_error param on logs streams #141391 (issue: #141360)

ES|QL

• Avoid a possible NPE by throwing an EIAE instead with more info #141711 (issue: #141267)
• Deep copy BytesRef when creating a constant vector block #141242 (issues: #140615, #140809, #140621)
• Dynamically grow hash in linear counting in HLL #142047 (issue: #41847)
• ESQL fix TO_IP leading_zeros=octal parsing #141776 (issue: #141627)
• Fix IP_PREFIX function leaking data in scratch #141940 (issue: #141628)
• Fix NPE with null field parameter #142328 (issue: #142281)
• Fix TS bug when grouping on aliases #141568
• Fix bug with multiple spatial aggs filtering in ES|QL #142332 (issue: #142329)
• Fix injected attributes's IDs in UnionAll branches #141262
• Remove incorrect inline stats pruning #141056 (issues: #140757, #139359)

Inference

• [Inference API] Do not write "task" field in Jina embedding request if unsupported #142181
• [Inference API] Fix ChunkingSettings field missing from ModelConfigurations equals method #142238
• [Inference API] Prevent trailing slashes from being included in URLs #141692

Infra/Core

• Reindexing older indices with percolator fields clears migration assistant errors #141539

Infra/Logging

• Fix cluster name in ECS upgrade #141792

Machine Learning

• Evict old models from the cache before loading new #140844
• Reduce locking when persisting ML job statistics #141519 (issue: #140511)

Mapping

• Allow shadowing time series metrics and dimension in non time series indexing #141549 (issue: #140882)

Ranking

• Fixing for NPE when there is no query specified for the standard retriever #142479 (issue: #142336)
• Implement comprehensive top N parameter handling for text similarity reranker #142039

SQL

• Fix QlIllegalArgumentException with non-foldable date range queries #142386 (issue: #137365)

Search

• Ensure Rewriteable.rewriteAndFetch listeners are not executed on transport threads #141904
• Fix Top Hits Incompatible Field Types on Sort Across Indices #142046 (issue: #141906)
• Fix handling empty collapse construct #141973 (issue: #139299)
• Prevent large CancelTasksRequest descriptions by truncating nodes and actions #141815
• Reduce cancellation check interval in CancellableBulkScorer for better responsiveness #141747

Security

• Fix built-in roles sync to retry on lock contention instead of silently discarding pending updates #142433

Snapshot/Restore

• Ensure paused shard snapshot can be deleted #141408
• Terminate GCS retries when node is shutting down #142193
• Terminate S3 get blob retries when node is shutting down #142186

Transform

• Fix transform producing empty dest index when source query references runtime fields #142450 (issue: #113156)
• Fix transform validation to reject PUT and _start when user lacks remote index permissions #142403 (issue: #95367)

Vector Search

• [GPU] Handle segments too big for MSAI segment access #141872 (issue: #141746)

Original source

Highlights

Add CEF processor to Ingest node

The cef ingest processor parses a CEF (Common Event Format) message into a structured JSON object. It follows the CEF specification to extract fields such as device vendor, device product, device version, signature ID, name, severity, and extension fields. This allows users to easily analyze and visualize CEF-formatted logs within Elasticsearch and Kibana.

Evaluate time series aggregation functions over sliding windows

Time series aggregations in ES|QL can be optionally calculated over a sliding time window. Using a window larger than the time bucket interval reduces jitter in dashboards, by taking a larger number of data points into account.Windows can be specified in all supported time series aggregations, as a second argument. For instance, the following ES|QL query calculates the average rate of requests per host every minute, using a 10-minute window:TS metrics

| WHERE TRANGE(1h)
| STATS avg(rate(requests, 10m)) BY TBUCKET(1m), hostAcceptable window values are currently limited to multiples of the time bucket value (e.g. BUCKET or TBUCKET), as specified in the BY-clause. If no window is provided, it's implicitly set to the time bucket value, matching existing behavior.

New exponential_histogram field type for native OpenTelemetry exponential histogram support

A new exponential_histogram field type has been added to Elasticsearch, providing native support for OpenTelemetry exponential histograms.Exponential histograms offer an efficient way to collect distributions (e.g. response times) at a high compression ratio while still maintaining good accuracy for percentile estimation. They can be easily collected via OpenTelemetry SDKs, as the only required configuration is the maximum allowed number of buckets. The actual buckets will be chosen automatically based on the observed values.The new exponential_histogram field type is supported in ES|QL. It can be aggregated using the PERCENTILES, AVG, MIN, MAX and SUM functions.

New dense_vector options for storing bfloat16 vectors and utilising on-disk rescoring

New options have been added to the dense_vector field type.The first is support for storing vectors in bfloat16 format. This is a floating-point format that utilises two bytes per value rather than four, halving the storage space required compared to element_type: float. This can be specified with element_type: bfloat16 when creating the index, for all dense_vector indexing types.Float values are automatically rounded to two bytes when writing to disk, so this format can be used with original source vectors at two- or four-byte precision. BFloat16 values are zero-expanded back to four-byte floats when read into memory. Using bfloat16 will cause a loss of precision compared to the original vector values, as well as a small performance hit due to converting between bfloat16 and float when reading and writing vectors; however this may be counterbalanced by a corresponding decrease in I/O, depending on your workload.The second option is to enable on-disk rescoring. When rescoring vectors during kNN searches, the raw vectors are read into memory. When the vector data is larger than the amount of available RAM, this might cause the OS to evict some in-memory pages that then need to be paged back in immediately afterwards. This can cause a significant slowdown in search speed. Enabling on-disk rescoring causes rescoring to use raw vector data on-disk during rescoring, and to not read it into memory first. This can significantly increase search performance in such low-memory situations.Enable on-disk rescoring using the on_disk_rescore: true index option.

Add compression for binary doc values

Add compression for binary doc values using Zstd and blocks with a variable number of values.Block-wise LZ4 compression was previously added to Lucene in LUCENE-9211 and removed in LUCENE-9378 due to query performance issues. This approach stored a constant number of values per block (specifically 32 values). This made it easy to map a given value index (e.g., docId) to the block containing it by doing blockId = docId / 32. Unfortunately, if values are very large, we must still have exactly 32 values per block, and (de)compressing a block could cause very high memory usage. As a result, we had to keep the number of values small, meaning that in the average case, a block was much smaller than ideal.To overcome the issues of blocks with a constant number of values, this PR adds block-wise compression with a variable number of values per block. It stores a minimum of 1 document per block and stops adding values when the size of a block exceeds a threshold or the number of values exceeds a threshold. Like the previous version, it stores an array of addresses for the start of each block. Additionally, it stores a parallel array with the docId at the start of each block. When looking up a given docId, if it is not in the current block, we binary search the array of docId starts to find the blockId containing the value. We then look up the address of the block. After this, decompression works very similarly to the code from LUCENE-9211; the main difference being that Zstd(1) is used instead of LZ4.The introduction of binary doc value compression transparently affects wildcard field types, like URLs that are common in access logs, which will now compress much better.

Enable doc_values skippers

Doc_values skippers add a sparse index to doc_values fields, allowing efficient querying and filtering on a field without having to build a separate BKD or terms index. These are now enabled automatically on any field configured with index=false and doc_values=true if the index setting index.mapping.use_doc_values_skipper is set to true (default false, or true for TSDB indexes).TSDB indexes now default to using skippers in place of indexes for their @timestamp, dimension and _tsid fields, greatly reducing their on-disk footprint. To disable skippers in TSDB indexes, set index.mapping.use_doc_values_skipper to false.

Enable large numeric blocks for TSDB codec

The size of numeric blocks gets increased from 128 to 512, for indices with [index.mode:time-series]. This mostly improves compression for multi-values in fields using sorted-set doc values, such as keyword and ip fields, as run-length encoding covers more data for the same storage footprint.

Features and enhancements

Aggregations:

• Let terms queries rewrite to a filter on constant_keyword fields #139106

Allocation:

• Allocation: add balancer round summary as metrics #136043
• Allocation: add duration and count metrics for write load hotspot #138465
• Iterate directly over contents of RoutingNode #137694
• Shard started reroute high priority #137306
• Simulate shards moved by explicit commands #136066
• Track shardStarted events for simulation in DesiredBalanceComputer #135597

Authentication:

• Additional DEBUG logging on authc failures #137941
• Improve SAML error handling by adding metadata #137598 (issue: #128179)
• In-response-to in saml successful response #137599 (issue: #128179)

Authorization:

• [Axonius] Add manage, create_index, read, index, write, delete, permission for third party agent indices kibana_system #139345
• [Security Solution] Add additional privileges to Kibana System role for .endpoint-scripts-file* indexes #139245
• [Security Solution] Add privileges to Kibana System role for management of internal indexes in support of Elastic Defend features #138993

CCS:

• Add support for project_routing for _search and _async_search #137566
• CPS usage telemetry support #137705
• CPS: Enable flatworld search and project_routing for _msearch #138822
• Do not assume we hear back from all linked projects when validating resolved index expressions for CPS #137916
• MRT should default to true for CPS searches #138105

Cluster Coordination:

• Add ThreadWatchdog to ClusterApplierService #134361
• Report recent tasks updates when master starved #139518

Codec:

• Enable large numeric blocks for TSDB codec in production #139503
• Improved bulk loading for binary doc values #138631
• Integrate stored fields format bloom filter with synthetic _id #138515
• Simple bulk loading of compressed binary doc values #138541

Data streams:

• Adding match_only_text subfield to *.display_name fields in ecs@mappings to be compliant with the latest additions in ECS #136265
• Support choosing the downsampling method in data stream lifecycle #137023

Distributed:

• Disk usage don't include synthetic _id postings #138745
• TransportGetBasicStatusAction runs on local #137567

Downsampling:

• Add new sampling method to the Downsample API #136813
• Move force merge from the downsampling request to the ILM action and allow users to disable it. #135834

EQL:

• Accept project_routing as query parameter #138559
• Enable CPS #137833

ES|QL:

• Add CHUNK function #134320
• Add MV_INTERSECTION Function #139379
• Add TOP_SNIPPETS function to return the best snippets for a field #138940
• Add TRANGE ES|QL function #136441 (issue: #135599)
• Add m alias for minute duration literal #136448 (issue: #135552)
• Add time_zone request param support to KQL and QSTR functions #138695
• Add optional parameters support to KQL function #135895 (issue: #135823)
• Add support for Full Text Functions and Lucene pushable conditions on fields from the Lookup Index for Lookup Join #136104
• Allow single fork branch #136805 (issue: #135825)
• Avoid retrieving unnecessary fields on node-reduce phase #137920 (issue: #134363)
• ESQL - Add planning detailed timing to profile information #138564
• ES|QL - Add TOP_SNIPPETS as tech preview #139272
• ES|QL - Add vector similarity functions #139365
• ES|QL - KNN function options support k and visit_percentage parameters #138372
• ES|QL - Remove vectors from _source when applicable #138013
• ES|QL Update CHUNK to support chunking_settings as optional argument #138123
• ES|QL completion command constant folding #138112 (issue: #136863)
• Enable nullify and fail unmapped resolution in tech-preview #140528
• Enable score function in release builds #136988
• Enable the TEXT_EMBEDDING function in non-snapshot build #136103
• Feature/count by trunc with filter #138765
• Fill in topn values if competitive #135734
• Fix a validation message in TimeSeriesGroupByAll #139882
• Fix slowness in ValuesFromManyReader.estimatedRamBytesUsed #139397
• Further simplify SingleValueMatchQuery #136195
• Fuse MV_MIN and MV_MAX and document process #138029
• GROUP BY ALL #137367
• Group by all optimization #139130
• Implement network_direction function #136133
• Improve performance when a single-valued field filter gets pushed down. (SingleValueMatchQuery) #135714
• Improve value loading for match_only_text mapping #137026
• Inference command: support for CCS #139244 (issue: #136860)
• Introduce a new interface to declare functions depending on the @timestamp attribute #137040 (issue: #136772)
• Introduce support for mapping-unavailable fields (Fork from #139417) #140463
• Late materialization after TopN (Node level) #132757
• Let include_execution_metadata always return data, also in local only #137641
• Locale and timezone argument for date_parse #136548 (issue: #132487)
• Make field fusion generic #137382
• Multiple patterns for grok command #136541 (issue: #132486)
• Optimize geogrid functions to read points from doc-values #138917
• Pull OrderBy followed by InlineJoin on top of it #137648
• Push down COUNT(*) BY DATE_TRUNC #138023
• Push filters past inline stats #137572
• Release CCS support for FORK #139630
• Release CHUNK function as tech preview #138621
• Release decay function #137830
• Release histogram data type #139703
• Run aggregations on aggregate metric double with default metric #138647 (issue: #136297)
• Support extra field (outputField) in TOP function. Values of outputField will be returned instead of values of field #135434 (issue: #128630)
• Support for parameters in LIKE and RLIKE #138051
• Support window function in time-series aggregations #138139
• Timezone support in DATE_TRUNC, BUCKET and TBUCKET #137450
• Use a single array for buffering rate data points #140855
• ESQL][Inference] Introduce usage limits for COMPLETION and RERANK #139074
• BlockSourceReader should always apply source filtering #138438
• DateDiff timezone support #138316

Geo:

• Bumps jts version to 1.20.0 #138351

Health:

• Add settings for health indicator shard_capacity thresholds #136141 (issue: #116697)
• [HealthAPI] Deterministic shard availability key order #138260 (issue: #138043)

ILM:

• Support different downsampling methods through ILM #136951

IdentityProvider:

• Support LimitedRole in idp role resolution #140536

Indices APIs:

• Add convenience API key param to remote reindex #135949
• Add small optimizations to PUT _component_template API #135644
• Improve no-op check in PUT _mapping API #138367

Inference:

• Add max_batch_size setting to EIS dense and sparse service settings #141185
• [Inference] Implementing the completion task type on EIS #137677

Infra/Plugins:

• Add upgrade.rollbacks mapping to .fleet-agents system index #139363

Infra/REST API:

• Cat API: added endpoint for Circuit Breakers #136890

Ingest:

• Logstashbridge: map ES logging to log4j backend #135854

Ingest Node:

• Add CEF processor to Ingest node #122491 (issue: #126201)
• Bump mustache.java to 0.9.14 #138923
• Bump jruby/joni to 2.2.6 #139075

Machine Learning:

• Add "close_job" parameter to the stop datafeed API #138634 (issue: #138010)
• Add Embedding inference task type #138198
• Add Google Model Garden's Meta, Mistral, Hugging Face and Ai21 providers support to Inference Plugin #135701
• Add cached tokens to Unified API response #136412
• Add daily task to manage .ml-state indices #137653
• Add late chunking configuration for JinaAI embedding task settings #137263
• Added Azure OpenAI chat_completion support to the Inference Plugin #136624
• Added Groq chat completion support to the Inference plugin #138251
• Added NVIDIA support to Inference Plugin #132388
• Added OpenShift AI text_embedding, completion, chat_completion and rerank support to the Inference Plugin #136624
• Nightly maintenance for anomaly detection results indices to keep to manageable size. #136065
• Require basic licence for the Elastic Inference Service #137434
• Transition EIS auth polling to persistent task on a single node #136713

Mapping:

• Add index.mapping.nested_parents.limit and raise nested fields limit to 100 #138961
• Add a dedicated field type for T-Digests #139607
• Allow updating inference_id of semantic_text fields #136120
• Default semantic_text fields to use ELSER on EIS when available #134708
• Enable bfloat16 support for semantic text #139347
• Enable new exponential histograms field type #138492
• Improve block loader for source only runtime geo_point fields #135883
• Improve bulk loading of binary doc values #137995
• OTLP: store units in mappings #134709
• Optionally ignore field when indexed field name exceeds length limit #136143 (issue: #135700)
• Remove DOC_VALUES_SKIPPER feature flag #138723
• Remove feature flag to enable binary doc value compression #138524
• Simple bulk loading for binary doc values #137860
• T digest field type docs #140478
• Use binary doc values for pattern_text args column #139466
• Use existing DocumentMapper when creating new MapperService #138489
• AggregateMetricDouble fields should not build BKD indexes #138724

Monitoring:

• Add missing fields to the Logstash Stack Monitoring Template #127053 (issue: #125499)

Network:

• Limit concurrent TLS handshakes #136386

Relevance:

• Add chunk_rescorer usage to output of explain and profile for text_similarity_rank_retriever #137249

SQL:

• Add project_routing option #138718
• Add project routing support to JDBC #138756
• Enable CPS #138803

Search:

• Add time range bucketing attribute to APM took time latency metrics #135549
• Adds retriever for result diversification using MMR #135873
• Allows Cross Project for close PointInTime #138962
• Allows PIT to be cross project #137966
• Allows field caps to be cross project #137530
• Can match phase coordinator duration APM metric #136828
• Can match search shard phase APM metric #136646
• Coordinator phase duration APM metric attributes #137409
• Dfs query phase coordinator metric #136481
• Extend time range bucketing attributes to retrievers #136072
• Fetch search phase coordinator duration APM metric #136547
• Field caps to support project_routing also in the body of the request #138681
• Field caps transport changes to return for each original expression what it was resolved to #136632
• Improve retrying PIT contexts for read-only indices #135231
• Speed up sorts on secondary sort fields #137533
• Use DV rewrites where possible in Keyword queries #137536

Searchable Snapshots:

• Allow fast blob-cache introspection by shard-id #138282

Security:

• Add audit log testing for cert-based cross-cluster authentication #137302
• Add periodic PKC JWK set reloading capability to JWT realm #136996
• Adds certificate identity field to cross-cluster API keys #134604
• Include Secure Setting Names and Keystore Modified Time in Reload API Response #138052 (issue: #112268)
• Send cross cluster api key signature as headers #135674
• Validate certificate identity from cross cluster creds #136299

Snapshot/Restore:

• Add a new setting for s3 API call timeout #138072
• Retry bulk-delete items in GCS #138951 (issue: #138364)
• S3 compareAndExchange using conditional writes #139228
• Use common retry logic for GCS #138553

TSDB:

• Add ES93BloomFilterStoredFieldsFormat for efficient field existence checks #137331
• Add TDigest histogram as metric to time series data streams #139247
• Add support for merges in ES93BloomFilterStoredFieldsFormat #137622
• GROUP BY ALL with the dimensions output #138595
• Late materialization of dimension fields in time-series #135961
• Minimize doc values fetches in TSDBSyntheticIdFieldsProducer #139053
• Use a new synthetic _id format for time-series datastreams #137274
• Use doc values skipper for @timestamp in synthetic _id postings #138568 #138876
• Use doc values skipper for _tsid in synthetic _id postings #138568

Transform:

• Preview index request #137455

Vector Search:

• Add DirectIO bulk rescoring #135380
• Add on_disk_rescore: true option to disk BBQ to rescore vectors on-disk without loading into memory #135778
• Add bfloat16 support to rank_vectors #139463
• Adding base64 indexing for vector values #137072
• Allow semantic_text fields to use optional GPU indexing for HNSW and int8_hnsw #138999
• Enable bfloat16 and on-disk rescoring for dense vectors #138492
• Enable early termination for HNSW by default #130564
• GPU: Restrict GPU indexing to FLOAT element types #139084
• GPU: add support for cosine with cuvs 2025.12 #139821
• Introduce an adaptive HNSW Patience collector #138685
• Remove gpu_vectors_indexing feature flag #139318
• Semantic search CCS support when ccs_minimize_roundtrips=false #138982
• Upgrade cuvs-java to version 25.12 #139747
• Use new bulk scoring dot product for max inner product #139409
• Use the new merge executor for intra-merge parallelism #137853
• [DiskBBQ] Add concurrency on KMeansLocal #139239
• [DiskBBQ] Support for centroid filtering for restrictive filters #137959
• SIMD][ARM] Optimized native bulk dot product scoring for Int7 #138552 (issue: #117096)
• SIMD][x64] Optimized native bulk dot product scoring for Int7 #139069

Fixes

Aggregations:

• Fix SearchContext CB memory accounting #138002
• Use Double.compare to compare doubles in tdigest.Sort #141049

Allocation:

• Allow relocation to NOT_PREFERRED node for evacuating shards #140197
• Always prefer YES over NOT_PREFERRED when allocating unassigned shards #138464
• Fix Decision.Type serialization BWC #140199
• Handle deleted source index in allocation deciders #140699 (issue: #140630)
• Overall Decision for Deciders prioritizes THROTTLE #140237

CCS:

• Set CPS index options only when not using PIT #137728

Codec:

• Binary doc values have stale value offset array if block contains all empty values #139922

Data streams:

• Exempt internal request markers from streams request param restrictions #139386 (issue: #139367)

Downsampling:

• Fix bug when downsampling exponential histograms with last value #139808
• Sort the values of a legacy histogram during downsampling #140771 (issue: #139382)

EQL:

• Fix _index on missing events in CCS #140886
• Fix project_routing #139366

ES|QL:

• Add block loader for the _id field of time-series indices #140102 (issues: #140033, #135689)
• Add support for dot-separated attribute names (e.g. foo.bar) and for parameters (e.g. ??my_param) in FUSE GROUP BY #135901
• Addressing vector similarity concurrency issue with byte vectors #137883 (issue: #137625)
• Aggressively free topn #140126
• Catch-and-rethrow TooComplexToDeterminizeException within ESQL #137024
• Change FUSE KEY BY to receive a list of qualifiedName #139071
• Do not skip a remote cluster base on the query's execution time status #138332
• Do not use Min or Max as Top's surrogate when there is an outputField #138380 (issue: #134083)
• Don't allow MV_EXPAND prior to STATS with TS #136931 (issue: #136928)
• ESQL: Fix INLINE STATS GROUP BY null being incorrectly pruned #140027 (issue: #139887)
• ES|QL - fix ENRICH command when using dense_vector columns #139774 (issue: #137699)
• Fix Page.equals() #136266
• Fix Present/Absent agg functions #140671
• Fix aggregation on null value #139797 (issues: #110257, #137544)
• Fix attribute only in full text function not found #137395 (issue: #137396)
• Fix extent reading when missing #140034
• Fix folding of case() function with date period and time duration #141157
• Fix interpolation for data points at bucket boundaries #139798 (issue: #139732)
• Fix metrics for took between 1 and 10 hours #139257
• Fix wrong pruning of plans with no output columns #133405
• Fixes memory leak in BytesRefLongBlockHash #137050 (issue: #137021)
• Fixing bug when handling 1d literal vectors #136891 (issue: #136364)
• Manage INLINE STATS count(*) on result sets with no columns #137017
• No EsqlIllegalArgumentException for invalid window values #139470
• Prune InlineJoin right aggregations by delegating to the child plan #139357 (issue: #138283)
• Prune columns when using fork #137907 (issue: #136365)
• Pushing down eval expression when it requires data access #136610 (issue: #133462)
• Quantize ST_X, ST_Y and related functions #140963 (issue: #139943)
• Support date trunc in TS #138947
• TS Disallow renaming into timestamp prior to implicit use #137713 (issue: #137655)
• Use DEFAULT_UNSORTABLE topN encoder for the TSID_DATA_TYPE #137706
• Use sub keyword block loader with ignore_above for text fields #140622
• Validate multiple GROK patterns individually #137082
• [ES|QL]: Update Vector Similarity To Support BFLOAT16 #139113

ILM:

• Remove ILM-stopped-check before the cluster update in migrateToDataTiersRouting #140892 (issue: #140885)

Indices APIs:

• Don't fail delete index API if an index is deleted during the request #138015 (issue: #137422)

Inference:

• [Inference API] Include rerank in supported tasks for IBM watsonx integration #140331 (issue: #140328)

Infra/Node Lifecycle:

• Fix readiness edge case on startup #140791 (issue: #136955)

Ingest Node:

• Respect flexible field access pattern in geoip and ip_location processors #138728

Machine Learning:

• Add configurable max_batch_size for GoogleVertexAI embedding service settings #138047
• Add missing job_id filter to Anomaly Detection data deleter #138160
• ChangePointDetector now correctly detects when a constant function returns a different value #128602 (issue: #127517)
• Disallow max_number_of_allocations > 1 for low priority model deployments #140163 (issue: #111227)
• Fix anomaly detection jobs stuck in opening by continously retrying to set the state to opened. #139668
• Preserve deployments with zero allocations during assignment planning #137244 (issue: #137134)
• Skip dataframes when disabled #137220
• Switch TextExpansionQueryBuilder and TextEmbeddingQueryVectorBuilder to return 400 instead of 500 errors #135800
• [Inference API] Add ElasticInferenceServiceDenseTextEmbeddingsServiceSettings to InferenceNamedWriteablesProvider #138484
• [Inference API] Remove worst-case additional 50ms latency for non-rate limited requests #136167
• [Inference API] Support chunking settings for sparse embeddings in custom service #138776

Mapping:

• Don't store keyword multi fields when they trip ignore_above #132962
• Fix index.mapping.use_doc_values_skippers defaults in serverless #139526
• Fixed inconsistency in the isSyntheticSourceEnabled flag #137297
• Provide defaults for index sort settings #135886 (issue: #129062)

Packaging:

• Fix stderr leak in Docker ES process detection #140701

Reindex:

• Disable _delete_by_query and _update_by_query for CCS/stateful #140301

Rollup:

• Fixing _rollup/data performance for a large number of indices #138305

SQL:

• Do not attempt to canonicalize InnerAggregate #136854
• More friendly exceptions for validation errors #137560

Search:

• Delay automaton creation in BinaryDvConfirmedQuery to avoid OOM on queries against WildCard fields #136086
• Ensure integer sorts are rewritten to long sorts for BWC indexes #139293 (issues: #139127, #139128)
• Fix date fields sort formatting with missing values #135899 (issue: #81960)

Security:

• Consistently prevent using exclusion prefix on its own #139337 (issue: #45504)
• Fixed a bug where dash-prefixed expressions were not consistently excluded during index resolution.
  This impacted both specific index names and wildcard patterns (example: -index, -logs-*).
  #138467 (issues: #64752, #83435)

Snapshot/Restore:

• Fix race condition in CancellableRateLimitedFluxIterator #141323
• Ignore abort-on-cleanup failure in S3 repo #138569
• Introduce INDEX_SHARD_COUNT_FORMAT #137210 (issue: #131822)
• Prevent NPE when generating snapshot metrics before initial cluster state is set #136350
• Support weaker consistency model for S3 MPUs #138663
• Suppress Azure SDK error logs #139729

Stats:

• Avoiding creating DataStreamShardStats objects with negative timestamps #139854

Vector Search:

• Auto prefiltering for queries on dense semantic_text fields #138989
• Corrects a seemingly simple bug where we pass numCands instead of k #140839
• Disk bbq license enforcement #139087
• DiskBBQ - missing min competitive similarity check on tail docs #135851
• Ensure we parse on_disk_rescore as its valid to pass false #141158
• Intercept filters to knn queries #138457 (issue: #138410)

Original source