Kyuubi Updates & Release Notes

Highlight

• Fully verified with Java 8, 11, 17, and 21, on Scala 2.12 and 2.13.
• Fully verified with Apache Spark 3.3 to 4.2, support for Spark 3.3 and 3.4 is deprecated.
• Fully verified with Apache Flink 1.17 to 1.20, support for Flink 1.17 to 1.19 is deprecated.
• Introduce Data Agent Engine, with Web UI improvement.
• Introduce Spark YARN connector for reading YARN aggregated logs.
• Remove Chat engine, an experimental feature that never reached production-ready status.
• Remove MySQL frontend protocol, an experimental feature that never got improved.

Server

• [KYUUBI #7387] Replace serverOnly with audience and immutable flags on ConfigEntry
• [KYUUBI #7575] Improve access path validation
• [KYUUBI #7509] Add engine startup time metric
• [KYUUBI #7567] Improve Engine UI proxy 403 response
• [KYUUBI #7542] Validate filenames in Utils
• [KYUUBI #7435] Honor server-level kyuubi.server.info.provider in getInfo
• [KYUUBI #7528] Remove MySQL frontend protocol
• [KYUUBI #7503] Document that kyuubi.session.local.dir.allow.list is only for Spark engine
• [KYUUBI #7488] Fix canceled batch reporting stale RUNNING application state
• [KYUUBI #7443] Fix NullPointerException when SPARK_HOME is misconfigured
• [KYUUBI #7438] Make JSON event logger permission management configurable
• [KYUUBI #7418] Rename kyuubi.engine.security.* to kyuubi.internal.security.* and document them
• [KYUUBI #7430] Remove Chat engine
• [KYUUBI #7407] Free STGroup to avoid OOM Kill
• [KYUUBI #7415] Allow including additional Spark keys as path validation targets
• [KYUUBI #7403] Fix password parsing bug when password contains a colon
• [KYUUBI #7391] Improve session REST API security
• [KYUUBI #7389] Fix parameter binding order in JDBCMetadataStore#transformMetadataState
• [KYUUBI #7374] Fix static zero IV in InternalSecurityAccessor
• [KYUUBI #7348] Separate principal and keytab for ZooKeeper authentication between server and engine
• [KYUUBI #7319] Return 404 for non-existent sessions
• [KYUUBI #7313] Audit pending Kubernetes application status
• [KYUUBI #7290] Prevent engine session leak after Kyuubi session closes
• [KYUUBI #7295] Upgrade playground components to recent versions

Spark Engine, Connectors and Plugins

• [KYUUBI #7551] Optimize InferRebalanceAndSortOrders to skip non-cheap keys and support local sort
• [KYUUBI #7539] Remove rebalance shuffle before writing at AQE time when not worthwhile
• [KYUUBI #7540] Spark Hive Connector supports Spark 4.1
• [KYUUBI #7531] Add advisoryPartitionSizeInBytes config for rebalance
• [KYUUBI #7433] Preserve Hive delegation tokens with non-empty service
• [KYUUBI #7515] Handle Spark variant type in metadata
• [KYUUBI #7495] Avoid overlong Kubernetes pod log directory names
• [KYUUBI #6832] Initial Spark DSv2 YARN connector for reading YARN aggregated logs
• [KYUUBI #6943] Support DPP in HiveScan
• [KYUUBI #7422] Fix FileWriterFactory reusing the same TaskAttemptId across task attempts in Spark Hive Connector
• [KYUUBI #7335] Sort before write in DynamicPartitionDataSingleWriter
• [KYUUBI #7406] Remove redundant table properties when creating tables in Spark Hive Connector
• [KYUUBI #7398] Include SERDEPROPERTIES when creating tables in Spark Hive Connector
• [KYUUBI #7377] Respect catalog-level warehouse when creating databases in Spark Hive Connector
• [KYUUBI #7369] Support PURGE TABLE in Spark Hive Connector
• [KYUUBI #7350] Make TPC-H/DS connectors compatible with Spark 4.2
• [KYUUBI #7304] Fix NumberFormatException when parsing ISO 8601 duration for session idle timeout

Spark AuthZ Plugin

• [KYUUBI #2470] Add Ranger performance tracing to rule authorization
• [KYUUBI #7230] Skip privilege check for DataSourceV2Relation without catalog and identifier
• [KYUUBI #7576] Fix data masking failure on a UNION-ALL view over a masked view
• [KYUUBI #7570] Support Spark 4.2
• [KYUUBI #7568] Eliminate TypeOfPlaceHolder across the whole plan
• [KYUUBI #7552] Support Spark 4.0 and 4.1
• [KYUUBI #7299] Remove redundant concatenation with the list of conditions

Spark Lineage Plugin

• [KYUUBI #7180] Drill down subqueries in the project to resolve lineage relationships

Flink Engine

• [KYUUBI #7310] Support variable substitution in JSON metrics report path

Data Agent Engine

• [KYUUBI #7379] Introduce Data Agent Engine with tool system, agent runtime, OpenAI provider, REST API, SSE streaming, and Web UI

Other Engines

• [KYUUBI #7527] Respect interrupt while waiting for engine launch in JDBC engine
• [KYUUBI #7549] Fix Subject.getSubject() in Kyuubi JDBC driver for JDK 25 compatibility
• [KYUUBI #7305] Route JDBC engine metadata calls through DatabaseMetaData
• [KYUUBI #7317] Fix parsePropertyFromUrl to strip query string from parsed values

Client

• [KYUUBI #7428] Support fetchOrientation parameter in OperationRestApi.getOperationLog

Web UI

• [KYUUBI #7538] Upgrade ESLint to 9 (flat config) and clear remaining dev-dependency advisories
• [KYUUBI #7533] Upgrade web-ui build toolchain and fix dependency security advisories
• [KYUUBI #7504] Add a Batch management page to the Web UI
• [KYUUBI #7483] Restrict Engine UI proxy targets
• [KYUUBI #7386] Replace axios with native fetch in web-ui

Documentation

• [KYUUBI #7513] Remove broken status badges from README
• [KYUUBI #7502] Align Python version and style
• [KYUUBI #7484] Add CLAUDE.md that references AGENTS.md
• [KYUUBI #7445] Add AGENTS.md to guide AI coding agents
• [KYUUBI #7441] Update Apache Paimon references after graduation
• [KYUUBI #7365] Fix documentation links
• [KYUUBI #7344] Bump markdown to 3.8.1
• [KYUUBI #7339] Fix non-consecutive header level increases
• [KYUUBI #7292] Add instructions to archive older releases

Building, Infrastructure and Dependency

• [KYUUBI #7578] Update default Kyuubi version to 1.12.0 in Helm chart
• [KYUUBI #7574] Fix typo in kyuubi-extension-spark-jdbc-dialect pom.xml
• [KYUUBI #7571] Redirect build/mvn fallback message to stderr to avoid corrupting captures
• [KYUUBI #7564] Bump GitHub Actions to latest versions
• [KYUUBI #7561] Cache CI engine archives per URL with a reusable action
• [KYUUBI #7562] Remove Gluten integration test module
• [KYUUBI #7554] Bump Log4j to 2.26.1
• [KYUUBI #7553] Bump Flink to 1.20.5
• [KYUUBI #7548] Bump docker/setup-buildx-action to 4.2.0
• [KYUUBI #7547] Bump docker/build-push-action to 7.3.0
• [KYUUBI #7546] Bump docker/setup-qemu-action to 4.2.0
• [KYUUBI #7545] Bump docker/login-action to 4.4.0
• [KYUUBI #7544] Bump Iceberg to 1.11.0
• [KYUUBI #7543] Bump Spark to 3.5.8, 4.0.3, and 4.1.2
• [KYUUBI #7535] Bump actions/cache to 6.1.0
• [KYUUBI #7536] Bump actions/setup-java to 5.4.0
• [KYUUBI #7537] Bump actions/setup-python to 6.3.0
• [KYUUBI #7526] Bump actions/checkout to 7.0.0
• [KYUUBI #7525] Bump actions/cache to 5.0.5
• [KYUUBI #7524] Bump actions/setup-python to 6.2.0
• [KYUUBI #7523] Bump actions/setup-java to 5.3.0
• [KYUUBI #7517] Bump actions/setup-java to 5.2.0
• [KYUUBI #7508] Bump codecov/codecov-action to 7.0.0
• [KYUUBI #7499] Use ubuntu-slim runner for lightweight workflows
• [KYUUBI #7501] Bump actions/labeler to 6
• [KYUUBI #7500] Bump actions/checkout to 6.0.3
• [KYUUBI #7494] Bump docker/build-push-action to 7.2.0
• [KYUUBI #7491] Bump docker/setup-qemu-action to 4.1.0
• [KYUUBI #7493] Bump docker/login-action to 4.2.0
• [KYUUBI #7492] Bump actions/checkout to 6.0.2
• [KYUUBI #7490] Bump actions/stale to 10.3.0
• [KYUUBI #7485] Run Docs workflow only for documentation changes
• [KYUUBI #7475] Add ASF allowlist check workflow
• [KYUUBI #7480] Bump actions/dependency-review-action to 5
• [KYUUBI #7479] Bump actions/upload-artifact to 7
• [KYUUBI #7478] Bump codecov/codecov-action to 6
• [KYUUBI #7477] Bump actions/setup-java to 5
• [KYUUBI #7481] Bump docker/setup-buildx-action to 4.1.0
• [KYUUBI #7473] Migrate Java tests from JUnit 4 to JUnit 5
• [KYUUBI #7465] Bump docker/build-push-action to 7.1.0
• [KYUUBI #7466] Bump actions/setup-python to 6
• [KYUUBI #7464] Bump actions/stale to 10.2.0
• [KYUUBI #7467] Bump actions/cache to 5
• [KYUUBI #7468] Bump actions/setup-node to 6
• [KYUUBI #7457] Bump Log4j to 2.26.0
• [KYUUBI #7460] Enable Dependabot for GitHub Actions
• [KYUUBI #7461] Set default max-parallel value for all matrices
• [KYUUBI #7446] Upgrade actions/checkout to v6
• [KYUUBI #7440] Update Apache Gluten references after graduation
• [KYUUBI #7444] Upgrade actions/first-interaction to v3
• [KYUUBI #7432] Bump PostgreSQL JDBC driver to 42.7.11
• [KYUUBI #7396] Bump Log4j to 2.25.4
• [KYUUBI #7372] Pin GitHub Actions versions to satisfy ASF policy
• [KYUUBI #7357] Bump Maven to 3.9.14
• [KYUUBI #7356] Bump Hadoop to 3.4.3
• [KYUUBI #7355] Bump Kafka to 3.9.2
• [KYUUBI #7351] Ignore GitHub Copilot and Claude folders in Git
• [KYUUBI #7352] Skip Java 8 tests for Spark engine
• [KYUUBI #7331] Bump Log4j core to 2.25.3
• [KYUUBI #7330] Bump Bouncy Castle bcpkix-jdk18on to 1.79
• [KYUUBI #7327] Bump spark-master profile to 4.2.0-SNAPSHOT
• [KYUUBI #7328] Bump testcontainers-scala to 0.44.1
• [KYUUBI #7318] Allow specifying executionMode when running the TPC-DS benchmark
• [KYUUBI #7303] Bump Spark to 4.1.1
• [KYUUBI #7243] Bump js-yaml to 4.1.1
• [KYUUBI #7291] Update NOTICE files for 2026
• [KYUUBI #7371] Bump brace-expansion to 1.1.13
• [KYUUBI #7342] Bump immutable to 4.3.8
• [KYUUBI #7367] Bump picomatch to 2.3.2
• [KYUUBI #7402] Bump follow-redirects to 1.16.0
• [KYUUBI #7401] Bump pytest to 9.0.3
• [KYUUBI #7411] Bump Bouncy Castle bcprov-jdk18on to 1.84

Credits

Last but not least, this release would not have been possible without the following contributors:

• Akira Ajisaka
• anandnalya
• attilapiros
• Cheng Pan
• Chenliang Lu
• Denis Krivenko
• Eric Liu
• Fei Wang
• GitHub Bot
• Han, Yutong
• Hiroki Egawa
• Hoon Oh
• hutiefang76
• Jiwon Park
• kongjiexin888-source
• maomaodev
• ruanwenjun
• Sunwoo-Shin
• wangzhigang1999
• Xiduo You
• ychris78
• Yuming Wang
• Yutong-1424
• Zhigang Wang

Changelogs

• [KYUUBI #7357] Bump Maven 3.9.14
• [KYUUBI #7355] Bump Kafka 3.9.2
• [KYUUBI #7350] Make TPC-H/DS connector compatible with Spark 4.2
• [KYUUBI #7351] Ignore GitHub Copilot and Claude folders by Git
• [KYUUBI #7339] [DOC] Fix non-consecutive header level increase
• [KYUUBI #7328] Bump testcontainers-scala 0.44.1
• [KYUUBI #7317] Fix parsePropertyFromUrl to remove query string from parsed values
• [KYUUBI #7313] If the k8s application is pending, audit the status as well
• [KYUUBI #7304] Fix NumberFormatException when parsing ISO 8601 duration for session idle timeout
• [KYUUBI #7303] Bump Spark 4.1.1
• [KYUUBI #7299] [AUTHZ] Remove redundant concatenation with the list of conditions
• [KYUUBI #7243] Bump js-yaml from 4.1.0 to 4.1.1
• [KYUUBI #7290] Prevent engine session leak after Kyuubi session closed
• [KYUUBI #7292] Add instruction to archive older releases
• [KYUUBI #7291] Update the NOTICE files for 2026

Credits

Last but not least, this release would not have been possible without the following contributors:

• Akira Ajisaka
• anandnalya
• attilapiros
• Cheng Pan
• dependabot[bot]
• dnskr
• ruanwenjun
• Wang, Fei
• Yuming Wang

Changelogs

• [KYUUBI #7245] Fix arrow batch converter error
• [KYUUBI #7272] Improve build/dependency.sh script
• [KYUUBI #7226] Support to wait the batch recovery appliction submission to throttle the load on the system
• [KYUUBI #7266] Upgrade jersey version to 2.47
• [KYUUBI #7265] Recommend to configure kyuubi.session.local.dir.allowlist for multi-tenant clusters
• [KYUUBI #7261] Bump extra-enforcer-rules 1.11.0
• [KYUUBI #7260] Suppress rat-plugin deprecate warnings
• [KYUUBI #7256] Enable authZ compile support for Spark 4.0 and refactor some test methods
• [KYUUBI #7248] JDBC engine should cancel the statement when receive cancel operation
• [KYUUBI #7251] Use RustFS instead of MinIO in Playground
• [KYUUBI #7241] Transfer engine pool name to lower case
• [KYUUBI #7237] Replace recommonmark with myst
• [KYUUBI #7224] [DOC] Fix “duplicate label usage” warnings
• [KYUUBI #7225] [CI] Skip Python Client unit tests when only documentation changes are made
• [KYUUBI #7222] [DOC] Fix “document isn’t included in any toctree” warnings
• [KYUUBI #7223] [CI] Skip running tests for documentation only changes
• [KYUUBI #7219] Fix endless swagger openapi.json security items
• [KYUUBI #7214] Fix kubernetes container state
• [KYUUBI #7171] Fix empty list engine result when etcd is used as the service registry
• [KYUUBI #7205] Use Apache closer.lua syntax for kyuubi-download to obtain engine archives
• [KYUUBI #7211] [DOC] Remove unsupported ’logo_only’ and ’extra_navbar’ theme options
• [KYUUBI #7202] [AUTHZ] make FilteredShowObjectsExec extends V2CommandExec to avoid running executor side task
• [KYUUBI #7197] Bump Spark 4.0.1
• [KYUUBI #7196] Bump Maven 3.9.11
• [KYUUBI #7194] Speed up K8s integration tests in CI
• [KYUUBI #7153] Share JAAS configuration for Zookeeper client to avoid server OOM
• [KYUUBI #7163] [SPARK] Check whether engine context stopped in engine terminating checker
• [KYUUBI #7165] Remove banned action-shellcheck
• [KYUUBI #7157] Bump form-data from 4.0.0 to 4.0.4 in /kyuubi-server/web-ui
• [KYUUBI #7162] Bump kafka-clients 3.9.1
• [KYUUBI #6928] Bump Spark 4.0.0
• [KYUUBI #7158] Spark engine respects session-level idle timeout threshold
• [KYUUBI #7151] Bump commons-lang3 3.18.0
• [KYUUBI #7139] Fix Spark extension rules to support RebalancePartitions
• [KYUUBI #7135] Fix cannot access /tmp/engine-archives: No such file or directory
• [KYUUBI #7131] Print cached engine archives
• [KYUUBI #7109] Ignore the ? in backticks
• [KYUUBI #7112] Enhance test ‘capture error from spark process builder’ for Spark 4.0
• [KYUUBI #7113] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory
• [KYUUBI #7101] [1.10][FOLLOWUP] Fix code broken
• [KYUUBI #7110] Fix serverOnlyPrefixConfigKeys is iterator issue
• [KYUUBI #7076] Update known_translations
• [KYUUBI #7101] Load the existing pods when initializing kubernetes client to cleanup terminated app pods
• [KYUUBI #7104] Bump Maven 3.9.10
• [KYUUBI #7095] Respect terminated app state when building batch info from metadata
• [KYUUBI #7077] Spark 3.5: Enhance MaxScanStrategy for DSv2
• [KYUUBI #7073] Retry 3 times on deploying to nexus

Credits

Last but not least, this release would not have been possible without the following contributors:

• Wenjun Ruan
• Fei Wang
• GitHub Bot
• moelhoussein
• 1358035421
• Akira Ajisaka
• Lennon Chin
• echo567
• Ziyi Wu
• Denis Krivenko
• cutiechi
• He Zhao
• ruanwenjun
• Cheng Pan

Highlight

• Fully verified with Java 8, 11, 17, and 21, on Scala 2.12 and 2.13
• Fully verified with Apache Spark 3.3 to 4.0, initial support Spark 4.1, support for Spark 3.3 and 3.4 is deprecated.
• Fully verified with Apache Flink 1.17 to 1.20, support for Flink 1.17 to 1.19 is deprecated.

Changelogs

• [KYUUBI #7289] Release requires Java 17 or 21
• [KYUUBI #7288] Bump Maven 3.9.12
• [KYUUBI #7284] Upgrade Flink and Spark to latest patched version
• [KYUUBI #7287] Fix checkstyle workflow
• [KYUUBI #7245] Fix arrow batch converter error
• [KYUUBI #7281] Support to enable asynchronous post event
• [KYUUBI #7277] Add UUID v7 generator
• [KYUUBI #7275] Kyuubi Spark extension supports Spark 4.1
• [KYUUBI #7254] Initial support Spark 4.1
• [KYUUBI #7274] Bump Scala 2.13.17
• [KYUUBI #7273] Bump Netty 4.2.7
• [KYUUBI #7259] Bump Netty 4.1.128 gRPC 1.76.2 Protobuf 3.25.8 Guava 33.4.8-jre
• [KYUUBI #7272] Improve build/dependency.sh script
• [KYUUBI #6884][FOLLOWUP] Support reassign batch with BatchRestApi
• [KYUUBI #6884][FOLLOWUP] Fix internal kyuubi instance ping failure
• [KYUUBI #7226] Support to wait the batch recovery appliction submission to throttle the load on the system
• [KYUUBI #7186] Introduce RuleFunctionAuthorization for persistent function calls authorization
• [KYUUBI #7266] Upgrade jersey version to 2.47
• [KYUUBI #7265] Recommend to configure kyuubi.session.local.dir.allowlist for multi-tenant clusters
• [KYUUBI #7261] Bump extra-enforcer-rules 1.11.0
• [KYUUBI #7260] Suppress rat-plugin deprecate warnings
• [KYUUBI #7257] Bump Jackson 2.20.1 and kubernetes-client 6.14.0
• [KYUUBI #7256] Enable authZ compile support for Spark 4.0 and refactor some test methods
• [KYUUBI #7248] JDBC engine should cancel the statement when receive cancel operation
• [KYUUBI #7251] Use RustFS instead of MinIO in Playground
• [KYUUBI #7250] Fix JDBCMetadataStore Scala 2.13 compile error
• [KYUUBI #7241] Transfer engine pool name to lower case
• [KYUUBI #7237] Replace recommonmark with myst
• [KYUUBI #7238] [DOC] Replace ‘Getting Started with Hive JDBC’ markdown with rst
• [KYUUBI #7224] [DOC] Fix “duplicate label usage” warnings
• [KYUUBI #7225] [CI] Skip Python Client unit tests when only documentation changes are made
• [KYUUBI #7229] Fix countMetadata if the params is empty
• [KYUUBI #7209] Bump mysql image to v8 for authZ plugin test suite on iceberg
• [KYUUBI #7222] [DOC] Fix “document isn’t included in any toctree” warnings
• [KYUUBI #7221] [DOC] Fix nonexisting document ‘client/advanced/features/engine_pool’
• [KYUUBI #7223] [CI] Skip running tests for documentation only changes
• [KYUUBI #7220] Proactive cleanup kubernetes terminated application info
• [KYUUBI #7219] Fix endless swagger openapi.json security items
• [KYUUBI #7217] Support customized session protocol version to support binary type
• [KYUUBI #7215] Make all BatchUtils variables to be final
• [KYUUBI #7214] Fix kubernetes container state
• [KYUUBI #7213] Enable test of Spark lineage plugin for Spark 4.0
• [KYUUBI #7190] Fix Presto SQLAlchemy dialect did not implement get_view_names
• [KYUUBI #7171] Fix empty list engine result when etcd is used as the service registry
• [KYUUBI #7205] Use Apache closer.lua syntax for kyuubi-download to obtain engine archives
• [KYUUBI #7211] [DOC] Remove unsupported ’logo_only’ and ’extra_navbar’ theme options
• [KYUUBI #7172][FOLLOWUP] kyuubi-spark-lineage-shaded artifactId should include scala.binary.version
• [KYUUBI #7212] [DOC] Use unordered list for Appendixes page
• [KYUUBI #7208] Pyhive remove thrift results columns assert
• [KYUUBI #7202] [AUTHZ] make FilteredShowObjectsExec extends V2CommandExec to avoid running executor side task
• [KYUUBI #7203] Bump Iceberg 1.10.0
• [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine
• [KYUUBI #7176] Cleanup metadata with batch size and interval
• [KYUUBI #7198] Remove support for Spark 3.2
• [KYUUBI #7199] Deprecate support for Flink 1.17 and 1.18
• [KYUUBI #7183][LINEAGE] Support collect all input tables by the plan
• [KYUUBI #7197] Bump Spark 4.0.1
• [KYUUBI #7196] Bump Maven 3.9.11
• [KYUUBI #7194] Speed up K8s integration tests in CI
• [KYUUBI #7174] Kyuubi Spark Extention supports Spark 4.0
• [KYUUBI #7175] Enhance the performance for metadata cleanup by leveraging create_time index
• [KYUUBI #7172] Shade kyuubi spark lineage module
• [KYUUBI #7153] Share JAAS configuration for Zookeeper client to avoid server OOM
• [KYUUBI #7126][LINEAGE] Support merge into syntax in row level catalog
• [KYUUBI #7168] Adapt PermanentViewMarker introduced by authz plugin in lineage plugin
• [KYUUBI #7164] Bump Byte Buddy 1.17.6
• [KYUUBI #7163][SPARK] Check whether engine context stopped in engine terminating checker
• [KYUUBI #7165] Remove banned action-shellcheck
• [KYUUBI #7157] Bump form-data from 4.0.0 to 4.0.4 in /kyuubi-server/web-ui
• [KYUUBI #7162] Bump kafka-clients 3.9.1
• [KYUUBI #7138] Respect kyuubi.session.engine.spark.initialize.sql set by client in shared engine mode
• [KYUUBI #7158] Spark engine respects session-level idle timeout threshold
• [KYUUBI #7160] Use single try-with-resources blocks in docs example
• [KYUUBI #7144] Remove direct usage of sun.misc.Signal
• [KYUUBI #7147] Enforce to use Kyuubi’s PlainSASLServer for PLAIN mechanism
• [KYUUBI #7151] Bump commons-lang3 3.18.0
• [KYUUBI #7148] Fix spark.kubernetes.file.upload.path permission
• [KYUUBI #7141] Support to get spark app url with pattern http://{{SPARK_DRIVER_POD_IP}}:{{SPARK_UI_PORT}}
• [KYUUBI #7139] Fix Spark extension rules to support RebalancePartitions
• [KYUUBI #7129] Support PARQUET hive table pushdown filter
• [KYUUBI #7132] Respect kyuubi.session.engine.startup.waitCompletion for wait engine completion
• [KYUUBI #7135] Fix cannot access /tmp/engine-archives: No such file or directory
• [KYUUBI #7131] Print cached engine archives
• [KYUUBI #7118] docs(client): Add comprehensive docs for Engine Pool feature
• [KYUUBI #7122] Support ORC hive table pushdown filter
• [KYUUBI #7121] Improve operation timeout management with configurable executors
• [KYUUBI #6876] Fix hadoopConf for autoCreateFileUploadPath
• [KYUUBI #7109] Ignore the ? in backticks
• [KYUUBI #7112] Enhance test ‘capture error from spark process builder’ for Spark 4.0
• [KYUUBI #7113] Skip Hadoop classpath check if flink-shaded-hadoop jar exists in Flink lib directory
• [KYUUBI #6924] Upgrade Spark Ranger plugin to 2.6.0
• [KYUUBI #7100] [#7099] Ranger Support Check Iceberg Alter Table Command & Change Iceberg Test Use JDBC Catalog
• [KYUUBI #7110] Fix serverOnlyPrefixConfigKeys is iterator issue
• [KYUUBI #7105] [K8S][HELM] Support additional labels for PrometheusRule
• [KYUUBI #7076] Update known_translations
• [KYUUBI #7106] Make response.results.columns optional
• [KYUUBI #6884] [FEATURE] Support to reassign the batches to alternative kyuubi instance in case kyuubi instance lost
• [KYUUBI #7101] Load the existing pods when initializing kubernetes client to cleanup terminated app pods
• [KYUUBI #7103] Bump Delta 4.0.0 and enable Delta tests for Spark 4.0
• [KYUUBI #7104] Bump Maven 3.9.10
• [KYUUBI #7098] Add ability to annotate pods and headless service in Kyuubi helm chart
• [KYUUBI #7072][FOLLOWUP] Fix engine startup permit Grafana panel unit
• [KYUUBI #7095] Respect terminated app state when building batch info from metadata
• [KYUUBI #7094] Add serverOnly flag for metrics config items
• [KYUUBI #7097] [INFRA] Write sorted authors for release contributors
• [KYUUBI #7093] Log the metadata cleanup count
• [KYUUBI #7072] Expose metrics of engine startup permit state
• [KYUUBI #7077] Spark 3.5: Enhance MaxScanStrategy for DSv2
• [KYUUBI #7068] Iceberg ranger support check branch and tag ddl
• [KYUUBI #6870][FOLLOWUP] Correct file name of grafana/README.md
• [KYUUBI #6928] Bump Spark 4.0.0
• [KYUUBI #7073] Retry 3 times on deploying to nexus
• [KYUUBI #7063] Bump Kyuubi Shaded 0.5.0
• [KYUUBI #7065] [#7066] Iceberg Support add partition field check
• [KYUUBI #6920][FOLLOWUP] Spark SQL engine supports Spark 4.0
• [KYUUBI #7062] Bump Delta Lake 3.3.1
• [KYUUBI #7055] Support to filter out server only configs with prefixes
• [KYUUBI #7051] Fix usage without sslTrustStore of JDBC driver
• [KYUUBI #7054] Add server only flag for more server/credentials/frontend/metadata configs
• [KYUUBI #6926] Add SERVER_LOCAL engine share level
• [KYUUBI #7048] Fix KeyError when parsing unknown Hive type_id in schema inspection
• [KYUUBI #7033] Treat YARN/Kubernetes application NOT_FOUND as failed to prevent data quality issue
• [KYUUBI #7028] Persist the kubernetes application terminate state into metastore for app info store fallback
• [KYUUBI #7046] Bump Dropwizard metrics version to 4.2.30
• [KYUUBI #7045] Expose jetty metrics
• [KYUUBI #7044] Bump jetty version to 9.4.57.v20241219
• [KYUUBI #7043] Support to construct the batch info from metadata directly
• [KYUUBI #7034][FOLLOWUP] Decouple the kubernetes pod name and app name
• [KYUUBI #7035] Close the operation by operation manager to prevent operation leak
• [KYUUBI #7041][FOLLOWUP] Fix build for SparkOnKubernetesTestsSuite
• [KYUUBI #7041] Fix NPE when getting metadtamanager in KubernetesApplicationOperation
• [KYUUBI #6828][FOLLOWUP] Fix NPE in KyuubiBaseResultSet::getBigDecimal
• [KYUUBI #7036] [DOCS] Improve docs for kyuubi-extension-spark-jdbc-dialect
• [KYUUBI #7034] [KUBERNETES] Prefer to use pod spark-app-name label as application name than pod name
• [KYUUBI #7025] [KYUUBI #6686][FOLLOWUP] Prefer terminated container app state than terminated pod state
• [KYUUBI #7032] Remove Umbrella/Subtask issue template
• [KYUUBI #7026] Audit the kubernetes pod event type and fix DELETE event process logical
• [KYUUBI #7027] Support to initialize kubernetes clients on kyuubi server startup
• [KYUUBI #7023] Upgrade kubernetes client version to 6.13.5
• [KYUUBI #7022] Update announcement mail template to contain download links
• [KYUUBI #7015] Record the session disconnected info into kyuubi session event
• [KYUUBI #7017] Using mutable JettyServer uri to prevent batch kyuubi instance mismatch
• [KYUUBI #7011] Set kyuubi session engine client after opening engine session successfully
• [KYUUBI #7009] Backport HIVE-26723: Configurable canonical name checking.
• [KYUUBI #7008] Backport HIVE-27817: Disable ssl hostname verification for 127.0.0.1
• [KYUUBI #7005] [DOC] Remove empty page “Getting Started with Jupyter Lap”
• [KYUUBI #7004] Include FastXML Jackson into authZ shaded jar
• [KYUUBI #6979] Support check paimon system producers
• [KYUUBI #7003] Cut out JNA dependencies for authZ plugin
• [KYUUBI #7000] Exclude aws-java-sdk-logs from kyuubi-spark-authz-shaded
• [KYUUBI #6999] Keep JNA in authz-shaded with Scala 2.13
• [KYUUBI #6998] [TEST] Harness SparkProcessBuilderSuite
• [KYUUBI #6997] Get the latest batch app info after submit process terminated to prevent batch ERROR due to engine submit timeout
• [KYUUBI #6990] Add rebalance before InsertIntoHiveDirCommand and InsertIntoDataSourceDirCommand to align with behaviors of hive
• [KYUUBI #6785] Shutdown the executor service in KubernetesApplicationOperation and prevent NPE
• [KYUUBI #6984] Fix ValueError when rendering MapType data
• [KYUUBI #6989] Calculate expected join partitions based on scanned table size
• [KYUUBI #6988] [INFRA] Foward GitHub discussions to ASF mailing list
• [KYUUBI #6941] Test Add new Column for paimon
• [KYUUBI #6951] Test changing column type
• [KYUUBI #6986] [DOC] Fix multiple Pygments lexer name issues
• [KYUUBI #6987] [DOC] Fix Unknown target name issues
• [KYUUBI #6983] Remove support for spark.sql.watchdog.forcedMaxOutputRows
• [KYUUBI #6981] [DOC] Fix nested lists
• [KYUUBI #6964] Fix typos in serveral docs
• [KYUUBI #6973][AUTHZ] Support Paimon DELETE FROM / UPDATE / MERGE INTO commands
• [KYUUBI #6975] Clean up code for Spark 3.5 extension
• [KYUUBI #6977] [DOC] Remove empty note block
• [KYUUBI #6947] Test Dropping Columns
• [KYUUBI #6972] [KYUUBI 6971]Test Table Query&Batch Time Travel
• [KYUUBI #6970] [DOC] Fix “nonexisting document” issues
• [KYUUBI #6969] [DOC] Fix “Title underline too short” issues
• [KYUUBI #6965] Bump Scala 2.13.16
• [KYUUBI #6948] Test Changing Column Comment
• [KYUUBI #6958] Test INSERT TABLE
• [KYUUBI #6960] Bump log4j version to 2.24.3 to fix ConcurrentModificationException in LoggerContext
• [KYUUBI #6936] Test RenameTable command
• [KYUUBI #6949] Test adding column position
• [KYUUBI #6940] Test Unset Table Properties Command
• [KYUUBI #6950] Test changing column position
• [KYUUBI #6939] Bump Spark 3.5.5
• [KYUUBI #6942] Test Rename Column Name for paimon
• [KYUUBI #6908] Connection class ssl context object paramater
• [KYUUBI #6938] [DOC] Refine monitoring docs
• [KYUUBI #6923] Test Create Partitioned Table for Paimon
• [KYUUBI #6932] Test ALTER TBLPROPERTIES for Paimon
• [KYUUBI #6674] Bump Scalafmt to 3.9.x
• [KYUUBI #6921][AUTHZ] Test CTAS for Paimon
• [KYUUBI #6925] Only run Paimon authz tests with Scala 2.12
• [KYUUBI #6920] Spark SQL engine supports Spark 4.0
• [KYUUBI #6918] Cache client ipAddress in kyuubi jdbc connection
• [KYUUBI #6914] Bump Iceberg 1.8.0
• [KYUUBI #6905] PyHive HTTP/HTTPS dialect to use the database name from url
• [KYUUBI #6912][LINEAGE] Properly handle empty attribute set on mergeRelationColumnLineage
• [KYUUBI #6916] Bump Delta 3.3.0 for Spark 3.5
• [KYUUBI #6917] Bump Hudi 1.0.1
• [KYUUBI #6915] Fix ClickHouse integration tests
• [KYUUBI #6901] Default policy for spark
• [KYUUBI #6814] [UI] Bump cross-spawn from 7.0.3 to 7.0.6
• [KYUUBI #6900] [DOCS] Correct spelling errors in ’large_query_results’ part
• [KYUUBI #6891] Fix get existing gauge issue
• [KYUUBI #6875] [K8S][HELM] Add Hadoop configuration files support
• [KYUUBI #6876] Support rolling spark.kubernetes.file.upload.path
• [KYUUBI #6883] Using withOauthTokenProvider instead of withOauthToken to support token refresh
• [KYUUBI #6843] [FOLLOWUP] Fix ‘query-timeout-thread’ thread leak
• [KYUUBI #6843] Fix ‘query-timeout-thread’ thread leak
• [KYUUBI #6870] Add more metrics for grafana dashboard
• [KYUUBI #6866][FOLLOWUP] Prevent register gauge conflicts if both thrift binary SSL and thrift http SSL enabled
• [KYUUBI #6866] Add metrics for SSL keystore expiration time
• [KYUUBI #6867] Refactor the grafana dashboard template
• [KYUUBI #6864][FOLLOWUP] Support to apply instance with existing labels
• [KYUUBI #6864] Support to return prometheus metrics with instance label
• [KYUUBI #6861] Configuration guide of structured logging for Kyuubi server
• [KYUUBI #6862] Spark 3.3: MaxScanStrategy supports DSv2
• [KYUUBI #6859] Exclude log4j12 from hive engine module classpath
• [KYUUBI #6615] Make Jetty sending server version in response configurable
• [KYUUBI #6857] Spark 3.4: MaxScanStrategy supports DSv2
• [KYUUBI #5834] Add Grafana dashboard template
• [KYUUBI #6842] Bump Spark 3.5.4
• [KYUUBI #6848] Fix Gluten CI
• [KYUUBI #6845] Support additional labels for PodMonitor to be discovered by Prometheus
• [KYUUBI #6840] Fix PodMonitor pods selection
• [KYUUBI #6726][FOLLOWUP] Fix compilation on scala-2.13
• [KYUUBI #6846] Enable gluten CI for kyuubi apache repo
• [KYUUBI #6839] Add example for service monitor in Helm chart
• [KYUUBI #6829] Add metrics for batch pending max elapse time
• [KYUUBI #6836] Ship kafka-clients in binary distribution tarball without compression libs
• [KYUUBI #6838] [INFRA] Update archive mailing list address
• [KYUUBI #6837] [INFRA] Disable K8s CI image GHA cache
• [KYUUBI #6815] JDBC Engine supports Oracle
• [KYUUBI #6828] Clean up and improve error message for KyuubiBaseResultSet
• [KYUUBI #6826] Bump log4j from 2.24.1 to 2.24.2
• [KYUUBI #6722] Fix AppState when Engine connection is terminated
• [KYUUBI #6521] [K8S][HELM] Implement new configuration approach
• [KYUUBI #6726] Support trino stage progress
• [KYUUBI #6818] Upgrade mysql jdbc version to 8.4.0
• [KYUUBI #6821] Restore the plain PULL_REQUEST_TEMPLATE
• [KYUUBI #6820] Explicitly disable attach-scaladocs for pure Java modules
• [KYUUBI #6813] [BUILD] Replace java.version 1.8 with 8
• [KYUUBI #6807] [BUILD] Bump maven download plugin from 1.8.1 to 1.11.3
• [KYUUBI #6773] Bump commons-codec from 1.15 to 1.17.1
• [KYUUBI #6765] [CI] speed up the script for freeing disk space by batching removals
• [KYUUBI #6806] [K8S][HELM] Headless service renders ports for enabled protocols only
• [KYUUBI #6804] Bump Iceberg from 1.6.1 to 1.7.0
• [KYUUBI #6774] Bump log4j from 2.20.0 to 2.24.1
• [KYUUBI #6808] [BUILD]Replace systemProperties with systemPropertyVariables in maven-surefire-plugin
• [KYUUBI #6805] [K8S][HELM] Update default Kyuubi version to 1.10.0
• [KYUUBI #6778] [BUILD] allow frontend plugin not to inherits Maven’s http proxy config
• [KYUUBI #6786] Skip repeated checks on convert function in TColumnGenerator
• [KYUUBI #6790] Fix engine cannot exit when gracefully stopped
• [KYUUBI #6787] Improve the compatibility of queryTimeout in more version clients
• Normalize local path in Kyuubi server
• [KYUUBI #7001] Bump nanoid from 3.3.6 to 3.3.11 in /kyuubi-server/web-ui
• [KYUUBI #6967] Bump axios from 1.7.4 to 1.8.2 in /kyuubi-server/web-ui

Credits

Last but not least, this release would not have been possible without the following contributors:

• 1358035421
• Akira Ajisaka
• Alex Wojtowicz
• Bowen Liang
• Cheng Pan
• Chenliang Lu
• cutiechi
• David Yuan
• Denis Krivenko
• echo567
• eric9204
• Fei Wang
• Frank Bertsch
• gabry-lab
• GitHub Bot
• He Zhao
• Igor Khrol
• Jan Willem
• jasonj
• Joao Amaral
• John Zhang
• Kent Yao
• Lennon Chin
• moelhoussein
• naive-zhang
• namanagraw
• Octavian Ciubotaru
• Pat Buxton
• paul8263
• Peiyue Liu
• pionCham
• Reese Feng
• ruanwenjun
• Senmiao Liu
• shuyouZZ
• taylor12805
• Tian Bao
• wangzhigang1999
• Wenjun Ruan
• wenxin-cn
• Xu Guang Lv
• Yaguang Jia
• z1131392774
• Zhen Wang
• zhifanggao
• Ziyi Wu

Highlight

• Support Spark 4.0

Changelogs

• [KYUUBI #7055] Support to filter out server only configs with prefixes
• [KYUUBI #7051] Fix usage without sslTrustStore of JDBC driver
• [KYUUBI #7054] Add server only flag for more server/credentials/frontend/metadata configs
• [KYUUBI #7046] Bump dropwizard metrics version to 4.2.30
• [KYUUBI #7045] Expose jetty metrics
• [KYUUBI #7044] Bump jetty version to 9.4.57.v20241219
• [KYUUBI #7035] Close the operation by operation manager to prevent operation leak
• [KYUUBI #7041] Fix NPE when getting metadtamanager in KubernetesApplicationOperation
• [KYUUBI #6828] Fix NPE in KyuubiBaseResultSet::getBigDecimal
• [KYUUBI #7036] Improve docs for kyuubi-extension-spark-jdbc-dialect
• [KYUUBI #7034] Prefer to use pod
  spark-app-name
  label as application name than pod name
• [KYUUBI #7025] [KYUUBI #6686] Prefer terminated container app state than terminated pod state
• [KYUUBI #7026] Audit the kubernetes pod event type and fix DELETE event process logical
• [KYUUBI #7027] Support to initialize kubernetes clients on kyuubi server startup
• [KYUUBI #7022] Update announcement mail template to contain download links
• [KYUUBI #7015] Record the session disconnected info into kyuubi session event
• [KYUUBI #7017] Using mutable JettyServer uri to prevent batch kyuubi instance mismatch
• [KYUUBI #7011] Set kyuubi session engine client after opening engine session successfully
• [KYUUBI #7009] Backport HIVE-26723: Configurable canonical name checking.
• [KYUUBI #7008] Backport HIVE-27817: Disable ssl hostname verification for 127.0.0.1
• [KYUUBI #7004] Include FastXML Jackson into authZ shaded jar
• [KYUUBI #7003] Cut out JNA dependencies for authZ plugin
• [KYUUBI #7000] Exclude aws-java-sdk-logs from kyuubi-spark-authz-shaded
• [KYUUBI #6998] Harness SparkProcessBuilderSuite
• [KYUUBI #6997] Get the latest batch app info after submit process terminated to prevent batch ERROR due to engine submit timeout
• [KYUUBI #6785] Shutdown the executor service in KubernetesApplicationOperation and prevent NPE
• [KYUUBI #6984] Fix ValueError when rendering MapType data
• [KYUUBI #6965] Bump Scala 2.13.16
• [KYUUBI #6939] KSHC supports Spark 3.5.5
• [KYUUBI #6921] Test CTAS for Paimon
• [KYUUBI #6925] Only run Paimon authz tests with Scala 2.12
• [KYUUBI #6920] Spark SQL engine supports Spark 4.0
• [KYUUBI #6918] Cache client ipAddress in kyuubi jdbc connection
• [KYUUBI #6912] Properly handle empty attribute set on mergeRelationColumnLineage
• [KYUUBI #6917] Bump Hudi 1.0.1
• [KYUUBI #6915] Fix ClickHouse integration tests
• [KYUUBI #6814] Bump cross-spawn from 7.0.3 to 7.0.6
• [KYUUBI #6900] Correct spelling errors in ’large_query_results’ part
• [KYUUBI #6891] Fix get existing gauge issue
• [KYUUBI #6883] Using withOauthTokenProvider instead of withOauthToken to support token refresh
• [KYUUBI #6843] Fix ‘query-timeout-thread’ thread leak
• [KYUUBI #6866] Add metrics for SSL keystore expiration time
• [KYUUBI #6864] Support to return prometheus metrics with instance label
• [KYUUBI #6862] Spark 3.3: MaxScanStrategy supports DSv2
• [KYUUBI #6859] Exclude log4j12 from hive engine module classpath
• [KYUUBI #6857] Spark 3.4: MaxScanStrategy supports DSv2

Credits

Last but not least, this release would not have been possible without the following contributors:

• Igor Khrol
• Fei Wang
• Reese Feng
• Senmiao Liu
• David Yuan
• Xu Guang Lv
• Zhen Wang
• Peiyue Liu
• shuyouZZ
• GitHub Bot
• He Zhao
• Cheng Pan

Highlight

• Support Spark 4.0

Changelogs

• [KYUUBI #7073] Retry 3 times on deploying to nexus
• [KYUUBI #7051] Fix usage without sslTrustStore of JDBC driver
• [KYUUBI #7035] Close the operation by operation manager to prevent operation leak
• [KYUUBI #7041] Fix NPE when getting metadtamanager in KubernetesApplicationOperation
• [KYUUBI #6828] Fix NPE in KyuubiBaseResultSet::getBigDecimal
• [KYUUBI #7011] Set kyuubi session engine client after opening engine session successfully
• [KYUUBI #7008] Backport HIVE-27817: Disable ssl hostname verification for 127.0.0.1
• [KYUUBI #6997] Get the latest batch app info after submit process terminated to prevent batch ERROR due to engine submit timeout
• [KYUUBI #6785] Shutdown the executor service in KubernetesApplicationOperation and prevent NPE
• [KYUUBI #6984] Fix ValueError when rendering MapType data
• [KYUUBI #6939] KSHC supports Spark 3.5.5
• [KYUUBI #6925] Only run Paimon authz tests with Scala 2.12
• [KYUUBI #6920] Spark SQL engine supports Spark 4.0
• [KYUUBI #6912] Properly handle empty attribute set on mergeRelationColumnLineage
• [KYUUBI #6883] Using withOauthTokenProvider instead of withOauthToken to support token refresh
• [KYUUBI #6843] Fix ‘query-timeout-thread’ thread leak
• [KYUUBI #6859] Exclude log4j12 from hive engine module classpath
• [KYUUBI #6838] Update archive mailing list address
• [KYUUBI #6837] Disable K8s CI image GHA cache

Credits

Last but not least, this release would not have been possible without the following contributors:

• Peiyue Liu
• Reese Feng
• Senmiao Liu
• Cheng Pan
• Fei Wang
• Igor Khrol
• Xu Guang Lv

Changelogs

• [KYUUBI #6840] Fix PodMonitor pods selection
• [KYUUBI #6829] Add metrics for batch pending max elapse time
• [KYUUBI #6836] Ship kafka-clients in binary distribution tarball without compression libs
• [KYUUBI #6838] [INFRA] Update archive mailing list address
• [KYUUBI #6837] [INFRA] Disable K8s CI image GHA cache
• [KYUUBI #6828] Clean up and improve error message for KyuubiBaseResultSet
• Revert “[KYUUBI #6639] Port HIVE-27815: Support update numModifiedRows”
• [KYUUBI #6722] Fix AppState when Engine connection is terminated
• [KYUUBI #6820] Explicitly disable attach-scaladocs for pure Java modules
• [KYUUBI #6813] [BUILD] Replace java.version 1.8 with 8
• [KYUUBI #6773] Bump commons-codec from 1.15 to 1.17.1
• [KYUUBI #6808] [BUILD]Replace systemProperties with systemPropertyVariables in maven-surefire-plugin
• [KYUUBI #6805] [K8S][HELM] Update default Kyuubi version to 1.10.0
• [KYUUBI #6778] [BUILD] allow frontend plugin not to inherits Maven’s http proxy config
• [KYUUBI #6786] Skip repeated checks on convert function in TColumnGenerator
• [KYUUBI #6790] Fix engine cannot exit when gracefully stopped

Credits

Last but not least, this release would not have been possible without the following contributors:

• Bowen Liang
• pionCham
• Denis Krivenko
• Fei Wang
• Zhen Wang
• Cheng Pan
• Joao Amaral
• zhifanggao

Changelogs

• [KYUUBI #6828] Clean up and improve error message for KyuubiBaseResultSet
• [KYUUBI #6741] [CI] update GHA actions/upload-artifact from v3 to v4
• [KYUUBI #6722] Fix AppState when Engine connection is terminated
• [KYUUBI #6820] Explicitly disable attach-scaladocs for pure Java modules
• [KYUUBI #6813] [BUILD] Replace java.version 1.8 with 8
• [KYUUBI #6790] Fix engine cannot exit when gracefully stopped
• [KYUUBI #6772] Fix ProcessBuilder to properly handle Java opts as a list
• [KYUUBI #6757] Pass build args to extra spark engine build command
• [KYUUBI #6750] [REST] Using ForbiddenException instead of NotAllowedException
• [KYUUBI #6728] [DOC] update Authz plugin docs of build command with -am option
• [KYUUBI #6688] [SPARK] Avoid trigger execution when getting result schema
• [KYUUBI #6638] Authz shaded should include jsr311-api
• [KYUUBI #6720] K8s pod OOM Killed should be identified as Application failed state
• [KYUUBI #6709] Fix seq and set equals check bug while check auth type in AuthenticationFilter.initAuthHandlers
• [KYUUBI #6708] spark-sql-engine’s classes should not be visible in server module
• [KYUUBI #6699] Bump Spark 4.0.0-preview2
• [KYUUBI #6648] Support to show launch engine log for thrift http mode
• [KYUUBI #6698] Exclude transitive deps of paimon
• [KYUUBI #6680] Format the kyuubi-ctl batch command output
• [KYUUBI #6668] Fix kyuubi batch state abnormal
• [KYUUBI #6661] Improve perf for column-based TRowSet generation
• [KYUUBI #6658] [DOCS] Fixed typo in REST API docs
• [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
• [KYUUBI #6638] Shade jsr311-api in Authz
• [KYUUBI #6635] Fix Hive 2.3.10 integration tests
• [KYUUBI #6583] Support to cancel Spark python operation
• [KYUUBI #6627] Using concurrent collections for session operation handle set
• [KYUUBI #6626] Fix operation never expired issue with periodical listOperations api calls
• [KYUUBI #6620] [K8S][HELM] Update default Kyuubi version to 1.9.2
• [KYUUBI #6615] Make Jetty sending server version in response configurable
• [KYUUBI #6609] Bump Spark 3.5.2
• [KYUUBI #6606] Restore shell compatibility for build/mvn
• [KYUUBI #6606] build/mvn checks project local downloaded binary ahead
• [KYUUBI #6600] Correct usage of maven.scaladoc.skip
• [KYUUBI #6598] Flink engine module supports building with Scala 2.13
• [KYUUBI #6599] Disable Javadoc/Scaladoc by default
• [KYUUBI #6596] Fix typos in architecture page
• [KYUUBI #6591] Support authorization on swagger UI
• [KYUUBI #6592] Add uploadWorkDir into local dir allow list and do not skip check for resource uploaded use case
• [KYUUBI #6588] Get effective USER if env var is unavailable
• [KYUUBI #6571] Bump apache parent pom from 31 to 33
• [KYUUBI #6586] Bump commons-io to 2.16.1
• [KYUUBI #6564] Insert into table check the privilege of table
• [KYUUBI #6549] Correctly handle empty Java options for engines
• [KYUUBI #6579] Show kyuubi batch app details on app state change
• [KYUUBI #6364] Bump scala-maven-plugin from 4.8.0 to 4.9.2
• [KYUUBI #6566] Fix typos in KyuubiConf
• [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor can’t get the ‘database’ attribute if it’s a Paimon plan.
• [KYUUBI #6550] Delete updateStrategy.rollingUpdate.maxUnavailable because it is alpha feature
• [KYUUBI #6543] Bump Kyuubi Shaded 0.4.1

Credits

Last but not least, this release would not have been possible without the following contributors:

• promising-forever
• Cheng Pan
• Bowen Liang
• lifulong
• be-hase
• DenineLu
• Baokun Han
• hh-cn
• LiJie20190102
• Senmiao Liu
• Zhen Wang
• Joao Amaral
• Fei Wang
• Mike Wilson
• Madhukar525722
• liujiayi771
• paul8263
• pionCham

[KYUUBI #6417] Remove scala compile args -Yno-adapted-args
[KYUUBI #6820] Explicitly disable attach-scaladocs for pure Java modules
[KYUUBI #6790] Fix engine cannot exit when gracefully stopped
[KYUUBI #6709] Fix seq and set equals check bug while check auth type in AuthenticationFilter.initAuthHandlers
[KYUUBI #6708] spark-sql-engine’s classes should not be visible in server module
[KYUUBI #6658] Fixed typo in REST API docs
[KYUUBI #6606] Restore shell compatibility for build/mvn
[KYUUBI #6606] build/mvn checks project local downloaded binary ahead
[KYUUBI #6600] Correct usage of maven.scaladoc.skip
[KYUUBI #6591] Support authorization on swagger UI
[KYUUBI #6592] Add uploadWorkDir into local dir allow list and do not skip check for resource uploaded use case
[KYUUBI #6588] Get effective USER if env var is unavailable
[KYUUBI #6564] Insert into table check the privilege of table
[KYUUBI #6549] Correctly handle empty Java options for engines
[KYUUBI #6542] KyuubiBatchService should wait for HTTP server started before picking jobs
[KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
[KYUUBI #6466] Bump minikube and kubernetes version of integration test
[KYUUBI #6463] Release semaphore immediately after startup process exit
[KYUUBI #5957] Flink engine should not load kyuubi-defaults.conf
[KYUUBI #6440] Fix casing in kubectl commands for serviceaccount creation
[KYUUBI #6430] livenessProbe uses absolute path based on KYUUBI_HOME
[KYUUBI #6422] Merge standard error to output
[KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown
[KYUUBI #6394] Fix example SQL in Spark TPC-DS connector docs
[KYUUBI #4847] Exclude the alive probe sessions in terminating checker
[KYUUBI #6172] Support to interrupt the thrift request immediately after marking the engine not alive
[KYUUBI #6373] Avoid NPE in get operation log api
[KYUUBI #6366] Improve log message for YARN getApplicationInfoByTag

Credits

Last but not least, this release would not have been possible without the following contributors:

• Fei Wang
• Zhen Wang
• ic4y
• lifulong
• pionCham
• liujiayi771
• LiJie20190102
• beryllw
• huangzhir
• Fengyu Cao
• lllong33
• Cheng Pan
• Fu Chen
• Qingbo Jiao

Highlight

• Fully verified with Java 8, 11, and 17, on Scala 2.12 and 2.13
• Fully verified with Apache Spark 3.2 to 3.5, initial support for 4.0-preview, support for Spark 3.2 is deprecated and will be removed in the next release
• Fully verified with Apache Flink 1.17 to 1.20
• Support creating batch job with uploading extra resources via REST API
• Support server choosing strategy for HA mode
• Introduce Spark JVM quake plugin
• Periodically cleanup expired temp files and operation logs on server
• Performance improvement for Ranger access requests deduplication in AuthZ plugin

Server

• [KYUUBI #6034] Kyuubi Server HA&ZK get server from serverHosts support more strategy
• [KYUUBI #6756] [REST] Check max file size of uploaded resource and extra resources in batch creation
• [KYUUBI #6750] [REST] Using ForbiddenException instead of NotAllowedException
• [KYUUBI #6731] [REST] Check all required extra resource files uploaded in creating batch request
• [KYUUBI #6720] K8s pod OOM Killed should be identified as Application failed state
• [KYUUBI #6681][FOLLOWUP] Log the kill batch request before closeSession and during closeOperation
• [KYUUBI #6772] Fix ProcessBuilder to properly handle Java opts as a list
• [KYUUBI #6753] Set hadoop fs delegation token renewer to empty
• [KYUUBI #6704] Disable periodic gc if set interval to 0
• [KYUUBI #6680] Format the kyuubi-ctl batch command output
• [KYUUBI #6668] Fix kyuubi batch state abnormal
• [KYUUBI #6681] Log the delete batch request in batch operation log
• [KYUUBI #6669] Track the client ip for internal kyuubi RESTful requests
• [KYUUBI #6645] Size based eviction for server-side temp files cleanup
• [KYUUBI #6661] Improve perf for column-based TRowSet generation
• [KYUUBI #6652] Support to list batches in descending order
• [KYUUBI #6648] Support to show launch engine log for thrift http mode
• [KYUUBI #6646] Fix RESTful API NPE when restarting Kyuubi server
• [KYUUBI #6587] Periodically expire temp files and operation logs on server to avoid memeory leak by Files.deleteOnExit
• [KYUUBI #6335][FOLLOWUP] Using sessionId for sessionUploadFolderPath
• [KYUUBI #6627] Using concurrent collections for session operation handle set
• [KYUUBI #6626] Fix operation never expired issue with periodical listOperations api calls
• [KYUUBI #6615] Make Jetty sending server version in response configurable
• [KYUUBI #6585] Support Multiple Configuration Files in FileSessionConfAdvisor
• [KYUUBI #6592] Add uploadWorkDir into local dir allow list and do not skip check for resource uploaded use case
• [KYUUBI #6588] Get effective USER if env var is unavailable
• [KYUUBI #6335] [REST] Support uploading extra resources in creating batch jobs via REST API
• [KYUUBI #6582] [KYUUBI-6581] Zorder clause syntax does not support special characters
• [KYUUBI #6554] Delete redundant code related to zorder
• [KYUUBI #6551] Allow insert zorder when global sort is false and the plan is Repartition or RepartitionByExpression.
• [KYUUBI #6542] KyuubiBatchService should wait for HTTP server started before picking jobs
• [KYUUBI #6534] Fix checkEngineConnectionAlive may throw NPE
• [KYUUBI #6524] Trino engine supports insecure configuration
• [KYUUBI #6008] RESTful API supports killing engine forcibly
• [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
• [KYUUBI #6500] Remove service loader descriptor for relocated HMS DelegationTokenIdentifier
• [KYUUBI #6499] Rewrite some utility methods in Java
• [KYUUBI #6469] Lazily initialize RecordReaderIterator to avoid driver oom when fetching big result set
• [KYUUBI #6476] Fix incomplete app events deserialization in SHS
• [KYUUBI #6302] Call cancelJobGroup immediately after statement execution finished
• [KYUUBI #4847][FOLLOWUP] Fix engine session never idle issue
• [KYUUBI #6422] Merge standard error to output
• [KYUUBI #6423] Fix Scalatest exclusion tags
• [KYUUBI #6396][FOLLOWUP] Avoid NPE
• [KYUUBI #6415] Manage Jersey in kyuubi-server module
• [KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown
• [KYUUBI #6408] Change to Base64 RFC4648 for handle guid and credential
• [KYUUBI #6400] Fix memory leak when using saveToFile
• [KYUUBI #6376] Improve closeEngine method names and logs due to closeSession
• [KYUUBI #6379] Return engine name and engine url within KyuubiSessionEvent/SessionData
• [KYUUBI #4847][FOLLOWUP] Exclude the alive probe sessions in terminating checker
• [KYUUBI #6172][TASK][EASY] Support to interrupt the thrift request immediately after marking the engine not alive
• [KYUUBI #6373] Avoid NPE in get operation log api
• [KYUUBI #6359] [REST] Return more fields within session/batch data
• [KYUUBI #6366] Improve log message for YARN getApplicationInfoByTag
• [KYUUBI #6244][TEST] Fix test logs upload error
• [KYUUBI #6322] Expose more config items of server internal rest client to users
• [KYUUBI #6049] Support to filter sessions/operations with session type
• [KYUUBI #6320] Fix terminated application pods not deleted issue when kyuubi server restarted
• [KYUUBI #6278] Update DB state when the query fallback to resource manager and the batch app is terminal (#6284)
• [KYUUBI #6291] Avoid NPE in MySQLErrPacket
• [KYUUBI #6216] Support to deny some client ips to make connection
• [KYUUBI #6199] Support to run HiveSQLEngine on kerberized YARN
• [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
• [KYUUBI #6165] Add some HttpClient options for EngineUIProxyServlet

Spark Engine

• [KYUUBI #6688] [SPARK] Avoid trigger execution when getting result schema
• [KYUUBI #6708] spark-sql-engine’s classes should not be visible in server module
• [KYUUBI #6699] Bump Spark 4.0.0-preview2
• [KYUUBI #6698] Exclude transitive deps of paimon
• [KYUUBI #6686] Ignore Spark pod container state if pod is terminated
• [KYUUBI #6653] Persist spark pod name conf into batch metadata
• [KYUUBI #6583] Support to cancel Spark python operation
• [KYUUBI #6609] Bump Spark 3.5.2
• [KYUUBI #6601] Bump Delta Lake 4.0.0rc1 and enable Delta Lake test for Spark 4.0
• [KYUUBI #6574] Skip eagerly execute command in PlanOnly mode of Spark Engine
• [KYUUBI #6549] Correctly handle empty Java options for engines
• [KYUUBI #6579] Show kyuubi batch app details on app state change
• [KYUUBI #6545] Deprecate and remove building support for Spark 3.2
• [KYUUBI #6546] Update incorrect descriptions in Zorder related configurations
• [KYUUBI #6531] Fix SPARK-EngineTab stop/gracefulstop not work
• [KYUUBI #6446] Add tests for Spark saveToFile function
• [KYUUBI #6518] Support extracting URL for Spark 4 on YARN
• [KYUUBI #6516] Fix KyuubiSparkUtil.buildURI
• [KYUUBI #6489] [PYTHON] PyKyuubi get_table_names also supports Spark SQL dialect
• [KYUUBI #6302][FOLLOWUP] Skip spark job group cancellation on incremental collect mode
• [KYUUBI #6453] Make KSHC support Spark 4.0 and enable CI for Spark 4.0
• [KYUUBI #6452] Add cross test for Spark 4.0
• [KYUUBI #6447] Use static regex Pattern instances in JavaUtils.timeStringAs and JavaUtils.byteStringAs
• [KYUUBI #6437] Fix Spark engine query result save to HDFS
• [KYUUBI #6441] Kyuubi Spark TPC-DS/H Connector cross version test
• [KYUUBI #6247] Make KSHC binary compatible with multiple Spark versions
• [KYUUBI #6425] Fix tests in spark engine and kyuubi server modules with Spark 4.0
• [KYUUBI #6424] TPC-H/DS connector support Spark 4.0
• [KYUUBI #6413] SPARK_HOME detection supports Spark 4
• [KYUUBI #6404] Fix HiveResult.toHiveString compatibility for Spark 4.0
• [KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs namespaces
• [KYUUBI #6399] Spark Kyuubi UI supports both javax and jakarta servlet namespaces
• Revert “[KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and above”
• [KYUUBI #6390] Temporarily disable UI Tab for Spark 4.0 and above
• [KYUUBI #6305][FOLLOWUP] Improve package Spark SQL engine both Scala 2.12 and 2.13
• [KYUUBI #6377] Fix isCommand check and set min rows threshold for saveToFile
• [KYUUBI #6370] For saveToFile, checking idx to prevent IndexOutOfBoundsException
• [KYUUBI #6323] Upgrade Spark 3.4.3
• [KYUUBI #6321] Support to get Spark Kubernetes app URL
• [KYUUBI #6315] Spark 3.5: MaxScanStrategy supports DSv2
• [KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13
• [KYUUBI #6290] Add custom exception serialization for SparkOperationEvent
• [KYUUBI #6287] Avoid missing matche
• [KYUUBI #6250][FOLLOWUP] Simplify code after dropping Spark 3.1
• [KYUUBI #6250] Drop support for Spark 3.1
• [KYUUBI #6260] Clean up and improve comments for spark extensions
• [KYUUBI #5767][FOLLOWUP] Fix spark batch conf not convert issue
• [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
• [KYUUBI #6215] Improve DropIgnoreNonexistent rule for Spark 3.5
• [KYUUBI #6211] Check memory offHeap enabled for CustomResourceProfileExec
• [KYUUBI #6207] Support to retrieve Spark UserDefinedType result
• [KYUUBI #6205] Backport SPARK-47300: quoteIfNeeded should quote identifier starts with digits
• [KYUUBI #6206] Update comments for job group id of Spark broadcast jobs
• [KYUUBI #6180] Kyuubi Spark nightly testing should use JDK 17

Spark AuthZ Plugin

• [KYUUBI #6754][AUTHZ] Improve the performance of Ranger access requests deduplication
• [KYUUBI #6638][FOLLOWUP] Authz shaded should include jsr311-api
• [KYUUBI #6709] Fix seq and set equals check bug while check auth type in AuthenticationFilter.initAuthHandlers
• [KYUUBI #6666][AUTHZ]Upgrade Ranger plugin to 2.5.0
• [KYUUBI #5402] Introduce Spark JVM quake plugin
• [KYUUBI #6619][FOLLOWUP] Do not log auth audit log for thrift http cookie auth
• [KYUUBI #6640] [AUTHZ] Adapt Derby 10.16 new JDBC driver package name
• [KYUUBI #6638] Shade jsr311-api in Authz
• [KYUUBI #6619] Support http bearer authentication for ThriftHttp protocol
• [KYUUBI #6623] Support bearer token authentication on swagger UI
• [KYUUBI #6618] Support http bearer token authentication for REST protocol
• [KYUUBI #6564] Insert into table check the privilege of table
• [KYUUBI #6541] [AUTHZ] Fix DataSourceV2RelationTableExtractor can’t get the ‘database’ attribute if it’s a Paimon plan.
• [KYUUBI #6197] Revise dependency management of Spark authZ plugin
• [KYUUBI #6194] AuthZ shaded should include ranger-plugins-cred

Other Spark Plugins

• [KYUUBI #6398] Fix lineage plugin UT for Spark 4.0

Flink Engine

• [KYUUBI #6368] Flink engine supports user impersonation
• [KYUUBI #6598] Flink engine module supports building with Scala 2.13
• [KYUUBI #6557] Support Flink 1.20
• [KYUUBI #6537] Upgrade Flink from 1.19.0 to 1.19.1
• [KYUUBI #5957] Flink engine should not load kyuubi-defaults.conf
• [KYUUBI #6353] Catch exception for closing flink internal session
• [KYUUBI #6367] Flink SQL engine supports RenewDelegationToken
• [KYUUBI #6344] FlinkProcessBuilder prioritizes user configurations
• [KYUUBI #6249] Drop support for Flink 1.16.
• [KYUUBI #6134][FOLLOWUP] Add profile and CI for Flink 1.19

Trino Engine

• [KYUUBI #6696] Fix Trino Status Printer to Prevent Thread Leak
• [KYUUBI #6485] Fix the Presto TABLE NOT FOUND error message that failed to match
• [KYUUBI #6524] Trino engine supports insecure configuration
• [KYUUBI #6387] [TRINO] Trino engine improve operation log

JDBC Engine

• Revert “[KYUUBI #6526] Kyuubi BeeLine wrongly process JDBC URL that contains --”
• [KYUUBI #6526] Kyuubi BeeLine wrongly process JDBC URL that contains --
• [KYUUBI #6339] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf
• [KYUUBI #6396] Add caching for KerberosAuthentication using ticketCache key
• [KYUUBI #6346] Change Kyuubi JDBC Driver default port and make it effective
• [KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0
• [KYUUBI #6316] Support switching jdbc dialect based on session conf
• [KYUUBI #6253] Support running JDBC engine on YARN AM
• [KYUUBI #5374][FOLLOWUP] Fix JDBC ClickHouse TRowSet Generator
• [KYUUBI #6243] Add missing Impala alias for JDBC engine
• [KYUUBI #5374] JDBC Engine supports ClickHouse
• [KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
• [KYUUBI #2782][FOLLOWUP] Add missed UNION_TYPE back
• [KYUUBI #6182] JDBC engine using the overlay conf as session conf

Other Engine

• [KYUUBI #6594] Port HIVE-26633: Make thrift client maxMessageSize configurable
• [KYUUBI #6639] Port HIVE-27815: Support update numModifiedRows
• [KYUUBI #6635] Fix Hive 2.3.10 integration tests
• [KYUUBI #6201] Distinguish the collect mode of different engines

Python client

• [KYUUBI #6281][PY] Enable hive test in python client
• [KYUUBI #6281][PY] Speed up testing with xdist plugin
• [KYUUBI #6281][PY] Initialize github action for python unit testing
• [KYUUBI #6567] Fix compatibility of pyhive with setuptools==72.0.0
• [KYUUBI #6434] Add footnote about pyhive origin

Web UI

• [KYUUBI #6727] replace immutable empty list and map in BatchRequest initialization
• ⬆️ Bump rollup from 3.29.4 to 3.29.5 in /kyuubi-server/web-ui (#6717)
• [KYUUBI #6712] [UI] Bump pnpm to v9 and nodejs to v20 LTS
• [KYUUBI #6660][UI] Improve the display of sql editor and query result
• [KYUUBI #6591] Support authorization on swagger UI
• [KYUUBI #6463] Release semaphore immediately after startup process exit
• ⬆️ Bump braces from 3.0.2 to 3.0.3 in /kyuubi-server/web-ui (#6461)
• [KYUUBI #6428] kyuubi-server/web-ui should not to set a timeout for axios requests
• [KYUUBI #6299] Support disabling Web UI
• [KYUUBI #6079] Web UI support Basic authN
• [KYUUBI #6283] Improve the message of distribution built without enabling web ui
• [KYUUBI #6248] Web SQL Editor supports run SQL snippets
• [KYUUBI #6257] Bump vite from 4.5.2 to 4.5.3 in /kyuubi-server/web-ui
• [KYUUBI #6228] Dropdown Selection and Parameter Parsing Issues in Kyuubi SQL Editor
• [KYUUBI #6188] Bump follow-redirects from 1.15.4 to 1.15.6 in web-ui

Documentation

• [KYUUBI #6752] [DOC] Bump doc build requirements
• [KYUUBI #6728] [DOC] update Authz plugin docs of build command with -am option
• ⬆️ Bump get-func-name in /kyuubi-server/web-ui (#6735)
• [KYUUBI #6734] [DOC] add authentication example in REST API docs
• ⬆️ Bump micromatch from 4.0.5 to 4.0.8 in /kyuubi-server/web-ui (#6729)
• [KYUUBI #6719] [DOC] Fix a couple of typos
• [KYUUBI #6673] [DOC] Fix typos in logging.md
• [KYUUBI #6671] [DOC] Fix typo in ENGINE SHARE LEVEL docs
• [KYUUBI #6658] [DOCS] Fixed typo in REST API docs
• [KYUUBI #6628] [DOCS] Improve docs for GROUP Share Level
• [KYUUBI #6596] Fix typos in architecture page
• [KYUUBI #6566] Fix typos in KyuubiConf
• [KYUUBI #6512] Improve docs for KSHC
• [KYUUBI #6514] Add link to Github Issues on Contributing Documentation page
• [KYUUBI #6507] Correct Kyuubi BeeLine help message
• [KYUUBI #6504] Improve docs for Solution for Large Query Results
• [KYUUBI #6440] Fix casing in kubectl commands for serviceaccount creation
• [KYUUBI #6432] Mention support for Spark 4.0 in docs
• [KYUUBI #6394] Fix example SQL in Spark TPC-DS connector docs
• [KYUUBI #6242] Remove block cleaner docs
• [KYUUBI #6234] Fix invalid code-block syntax in docs
• [KYUUBI #6191] Update docs to mention support of Flink 1.19

Building, Infrastructure and Dependency

• [KYUUBI #6775] Bump Github actions action/cache and actions/dependency-review-action v4
• [KYUUBI #6768] [CI] skip explicitly setting check-latest to false
• [KYUUBI #6762] [BUILD] cleanup dockerfile warnings
• [KYUUBI #6763] [CI] Specify JDK 11 for spark engine on kubernetes IT
• [KYUUBI #6741] [CI] update GHA actions/upload-artifact from v3 to v4
• [KYUUBI #6739] [CI] Bump runner image from Ubuntu 22.04 to 24.04
• [KYUUBI #6687] [BUILD] Use MAVEN_ARGS for passing Maven CLI arguments
• [KYUUBI #6757] Pass build args to extra spark engine build command
• [KYUUBI #6740] [CI] update GHA docker/build-push-action from v5 to v6 to support build summary
• [KYUUBI #6755] Upgrade default flink version to 1.20.0
• [KYUUBI #6742] Bump Jetty from 9.4.54 to 9.4.56
• [KYUUBI #6743] Bump protobuf from 3.25.4 to 3.25.5
• [KYUUBI #6744] Bump vite from 4.5.3 to 4.5.5
• [KYUUBI #6745] Bump testcontainers-scala from 0.41.0 to 0.41.4
• [KYUUBI #6746] Bump commons-lang3 from 3.13.0 to 3.17.0
• [KYUUBI #6747] Bump Guava from 32.1.3 to 33.3.1
• [KYUUBI #6737] Bump vertx-grpc and vertx-core from 4.5.1 to 4.5.3
• ⬆️ Bump semver in /kyuubi-server/web-ui (#6736)
• ⬆️ Bump axios from 1.6.0 to 1.7.4 in /kyuubi-server/web-ui (#6617)
• [KYUUBI #6733] Bump SQLlite from 3.42 to 3.46
• [KYUUBI #6682] Bump super-linter to v7
• [KYUUBI #6677] Bump scala-collection-compat to 2.12.0
• [KYUUBI #6676] Bump Scala to 2.12.19
• [KYUUBI #6672] Cleanup unused Commons Lang 2 dependency
• [KYUUBI #6664] [BUILD] Bump Maven from 3.8.8 to 3.9.9
• [KYUUBI #6659] Bump Iceberg to 1.6.1
• [KYUUBI #6644] Bump Paimon to 0.8
• [KYUUBI #5314] Support JDK 21
• [KYUUBI #6603] Upgrade Scala version from 2.13.8 to 2.13.14
• [KYUUBI #6620] [K8S][HELM] Update default Kyuubi version to 1.9.2
• [KYUUBI #6606][FOLLOWUP] Restore shell compatibility for build/mvn
• [KYUUBI #6584] Upgrade flip-tables from 1.0.2 to 1.1.1
• [KYUUBI #6593] bin/kyuubi supports kill command
• [KYUUBI #6607] Bump gRPC 1.65.1 Protobuf 3.25.4 Guava 32.1.3
• [KYUUBI #6606] build/mvn checks project local downloaded binary ahead
• [KYUUBI #6600] Correct usage of maven.scaladoc.skip
• [KYUUBI #6599] Disable Javadoc/Scaladoc by default
• [KYUUBI #6571] Bump apache parent pom from 31 to 33
• [KYUUBI #6580] [K8S][HELM] Improve metrics configuration
• [KYUUBI #6586] Bump commons-io to 2.16.1
• [KYUUBI #6364] Bump scala-maven-plugin from 4.8.0 to 4.9.2
• [KYUUBI #6558] Bump Iceberg 1.6.0
• [KYUUBI #6559] Various improvments for release scripts
• [KYUUBI #6550] Delete updateStrategy.rollingUpdate.maxUnavailable because it is alpha feature
• [KYUUBI #6543] Bump Kyuubi Shaded 0.4.1
• [KYUUBI #6548] Bump enforcer extra mojo rules to 1.8.0 for recognizing higher Java bytecode versions
• [KYUUBI #6523] Support configuration with --conf options
• [KYUUBI #6529] Upgrade kubernetes-client from 6.8.1 to 6.13.1
• [KYUUBI #6528] Bump Netty 4.1.108.Final
• [KYUUBI #6515] Bumping up dropwizard metrics to 4.2.26
• [KYUUBI #6506] kyuubi-beeline supports --conf
• [KYUUBI #6493] [K8S][HELM] Update default Kyuubi version to 1.9.1
• [KYUUBI #6484] Bump ws from 8.8.1 to 8.17.1 in web-ui
• [KYUUBI #6466] Bump minikube and kubernetes version of integration test
• [KYUUBI #6458] Remove commons-logging from binary release
• [KYUUBI #6457] Strip dependencies from REST client
• [KYUUBI #6456] Opens jdk.internal.ref
• [KYUUBI #6445] Normalize extra name for optional Python distribution dependencies
• [KYUUBI #6427] Extract data lake artifact names as maven properties
• [KYUUBI #6451] Bump Hudi 0.15.0 and enable Hudi authZ test for Spark 3.5
• [KYUUBI #6439] kyuubi-util-scala test jar leaked to compile scope
• [KYUUBI #6430] livenessProbe uses absolute path based on KYUUBI_HOME
• [KYUUBI #6419] Bump Jetty 9.4.54.v20240208
• [KYUUBI #6418] Bump Jersey to 2.40
• [KYUUBI #6416] Generate flattened POM
• [KYUUBI #6417] Remove scala compile args -Yno-adapted-args
• [KYUUBI #6397] Align Antlr version with Spark for Spark 4.0
• [KYUUBI #6392] Support javax.servlet and jakarta.servlet co-exist
• [KYUUBI #6391] Bump Arrow from 15.0.2 to 16.0.0
• [KYUUBI #6382] Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5
• [KYUUBI #6348] Upgrade iceberg from 1.5.0 to 1.5.2
• [KYUUBI #6328] Bump Hive 2.3.10
• [KYUUBI #6357] Bump org.bouncycastle:bcprov-jdk18on from 1.77 to 1.78
• [KYUUBI #6363] [INFRA] Increase CI concurrency
• [KYUUBI #6294] Simplify Netty and gRPC dependency management
• [KYUUBI #6312][HELM] Allow attaching annotations to service account
• [KYUUBI #6293] Bump Arrow from 12.0.0 to 15.0.2
• [KYUUBI #6298][HELM] Make ServiceMonitor use selector labels
• [KYUUBI #6251] Improve kyuubi-beeline help message
• [KYUUBI #6252] Upgrade hive-service-rpc 4.0.0
• [KYUUBI #6266] Kyuubi BeeLine supports JSON output format.
• [KYUUBI #5686][FOLLOWUP] Rename pyhive to python
• [KYUUBI #6180][FOLLOWUP] Update the setup JDK name in nightly.yml
• [KYUUBI #5686] Donate PyHive to Apache Kyuubi
• [KYUUBI #6268] Specify logDir for RollingFile filePattern
• [KYUUBI #6212] Added audit handler shutdown to the shutdown hook
• [KYUUBI #6267] Remove unused dependency management in POM
• [KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
• [KYUUBI #6239] Rename beeline to kyuubi-beeline
• [KYUUBI #6240] [INFRA] Add issue template for 2024H1 Kyuubi Code Contribution Task
• [KYUUBI #6185] Audit kyuubi operation state change log
• [KYUUBI #6196] Add log4j2 template for kyuubi repl commands
• [KYUUBI #6195] Update Helm Chart and playground to use 1.9.0
• [KYUUBI #6193] [INFRA] Add known_transkations
• [KYUUBI #6183] [K8S] KyuubiConf.getKubernetesConf should set namespace even if the Kubernetes context is empty
• [KYUUBI #6179] Bump iceberg from 1.4.3 to 1.5.0

Credits

Last but not least, this release would not have been possible without the following contributors:

• Alexander Rashed
• Binjie Yang
• Bogdan
• Bowen Liang
• Bruce Wong
• Cheng Pan
• Fei Wang
• Fu Chen
• George314159
• Harry
• Lennon Chin
• Liu Xiao
• Lucas Resch
• Mike Wilson
• PorterZhang2021
• Shilun Fan
• SteNicholas
• Wang, Fei
• Zhigang
• amanraj2520
• artilexx
• avishnus
• axiangzheng
• binjie yang
• camper42
• caoyu
• chengpeiming
• dependabot[bot]
• dnskr
• dupeng
• felixzh
• futureltl
• hang.huang
• hezhao2
• huangxiaoping
• huangzhir
• ic4y
• jeanlyn
• jialiang
• jiaoqingbo
• joey.ljy
• lifulong
• lijie0203
• liuxiao
• lllong33
• ludejiu
• madlnu
• minyk
• pengbei
• promising-forever
• ryosuke-hasebe
• senmiaoliu
• tatian
• taylor.fan
• vinoyang
• wangjunbo
• wankunde
• wenjie.wang01
• wforget
• xorsum
• yanghua
• yikaifei
• zRain
• zhang_yao
• zhouyifan279
• zml1206
• 吴梓溢
• 王龙

Changelogs

• [KYUUBI #6542] KyuubiBatchService should wait for HTTP server started before picking jobs
• [KYUUBI #6534] Fix checkEngineConnectionAlive may throw NPE
• [KYUUBI #6529] Upgrade kubernetes-client from 6.8.1 to 6.13.1
• [KYUUBI #6531] Fix SPARK-EngineTab stop/gracefulstop not work
• [KYUUBI #6518] Support extracting URL for Spark 4 on YARN
• [KYUUBI #6524] Trino engine supports insecure configuration
• [KYUUBI #6515] Bumping up dropwizard metrics to 4.2.26
• [KYUUBI #6516] Fix KyuubiSparkUtil.buildURI
• [KYUUBI #6508] Add the key-value pairs in optimizedConf to session conf
• [KYUUBI #6512] Improve docs for KSHC
• [KYUUBI #6507] Correct Kyuubi BeeLine help message
• [KYUUBI #6500] Remove service loader descriptor for relocated HMS DelegationTokenIdentifier
• [KYUUBI #6469] Lazily initialize RecordReaderIterator to avoid driver oom when fetching big result set
• [KYUUBI #6476] Fix incomplete app events deserialization in SHS
• [KYUUBI #6302] Skip spark job group cancellation on incremental collect mode
• [KYUUBI #6302] Call cancelJobGroup immediately after statement execution finished
• [KYUUBI #4847] Fix engine session never idle issue
• [KYUUBI #6466] Bump minikube and kubernetes version of integration test
• [KYUUBI #6463] Release semaphore immediately after startup process exit
• [KYUUBI #6458] Remove commons-logging from binary release
• [KYUUBI #6457] Strip dependencies from REST client
• [KYUUBI #5957] Flink engine should not load kyuubi-defaults.conf
• [KYUUBI #6353] Catch exception for closing flink internal session
• [KYUUBI #6251] Replace kyuubi-beeline with beeline in beeline help message
• [KYUUBI #6439] kyuubi-util-scala test jar leaked to compile scope
• [KYUUBI #6437] Fix Spark engine query result save to HDFS
• [KYUUBI #6440] Fix casing in kubectl commands for serviceaccount creation
• [KYUUBI #6441] Kyuubi Spark TPC-DS/H Connector cross version test
• [KYUUBI #6247] Make KSHC binary compatible with multiple Spark versions
• [KYUUBI #6493] [K8S][HELM] Update default Kyuubi version to 1.9.1

Credits

Last but not least, this release would not have been possible without the following contributors:

• Fu Chen
• Fei Wang
• Baokun Han
• Madhukar525722
• Cheng Pan
• Zhen Wang
• Z1Wu
• Yifan Zhou
• avishnus
• camper42
• felixzh2020
• huangzhir
• ic4y
• jeanlyn
• jiaoqingbo
• promising-forever
• zcx513566

Highlights

• Experimental support Spark 4.0-preview
• Web UI supports Basic Authentication

Features and Improvements

• [KYUUBI #6424] TPC-H/DS connector support Spark 4.0
• [KYUUBI #6413] SPARK_HOME detection supports Spark 4
• [KYUUBI #6404] HiveResult.toHiveString compatible to Spark 4.0
• [KYUUBI #6405] Spark engine supports both javax and jakarta ws.rs namespaces
• [KYUUBI #6399] Spark engine supports both javax and jakarta servlet namespaces
• [KYUUBI #6392] Support javax.servlet and jakarta.servlet co-exist
• [KYUUBI #6297] Package Spark SQL engine both Scala 2.12 and 2.13
• [KYUUBI #4847] Exclude the alive probe sessions in terminating checker
• [KYUUBI #6172] Support to interrupt the thrift request immediately after marking the engine not alive
• [KYUUBI #6315] Spark 3.5: MaxScanStrategy supports DSv2
• [KYUUBI #6049] Support to filter sessions/operations with session type
• [KYUUBI #6366] Improve log message for YARN getApplicationInfoByTag
• [KYUUBI #6298] Make ServiceMonitor use selector labels in Helm charts
• [KYUUBI #6251] Improve kyuubi-beeline help message
• [KYUUBI #6290] Add custom exception serialization for SparkOperationEvent
• [KYUUBI #6079] Web UI supports Basic authN
• [KYUUBI #6338] Support connecting Kyuubi using Hive JDBC driver 4.0
• [KYUUBI #6248] Web SQL Editor supports run SQL snippets
• [KYUUBI #6216] Support to deny some client ips to make connection
• [KYUUBI #6283] Improve the message of distribution built without enabling web ui
• [KYUUBI #6207] Support to retrieve Spark UserDefinedType result
• [KYUUBI #6205] Backport SPARK-47300: quoteIfNeeded should quote identifier starts with digits
• [KYUUBI #6215] Improve DropIgnoreNonexistent rule for Spark 3.5
• [KYUUBI #6211] Check memory offHeap enabled for CustomResourceProfileExec

Bug Fixes

• [KYUUBI #6430] livenessProbe uses absolute path based on KYUUBI_HOME
• [KYUUBI #6428] kyuubi-server/web-ui should not to set a timeout for axios requests
• [KYUUBI #6422] Merge standard error to output
• [KYUUBI #6396] Add caching for KerberosAuthentication using ticketCache key
• [KYUUBI #6410] Check if asyncRequestExecutor is initialized before shutdown
• [KYUUBI #6394] Fix example SQL in Spark TPC-DS connector docs
• [KYUUBI #6400] Fix memory leak when using saveToFile
• [KYUUBI #6344] FlinkProcessBuilder prioritizes user configurations
• [KYUUBI #6346] Change Kyuubi JDBC Driver default port and make it effective
• [KYUUBI #6212] Added audit handler shutdown to the shutdown hook
• [KYUUBI #6370] For saveToFile, checking idx to prevent IndexOutOfBoundsException
• [KYUUBI #6320] Fix terminated application pods not deleted issue when kyuubi server restarted
• [KYUUBI #6291] Avoid NPE in MySQLErrPacket
• [KYUUBI #5767] Fix spark batch conf not convert issue
• [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
• [KYUUBI #6287] Avoid missing match
• [KYUUBI #6183] KyuubiConf.getKubernetesConf should set namespace even if the Kubernetes context is empty
• [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
• [KYUUBI #6182] JDBC engine using the overlay conf as session conf
• [KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
• [KYUUBI #6228] Dropdown Selection and Parameter Parsing Issues in Kyuubi SQL Editor

Build, Test and Misc

• [KYUUBI #6432] Mention support for Spark 4.0 in docs
• [KYUUBI #6423] Fix Scalatest exclusion tags
• [KYUUBI #6419] Bump Jetty 9.4.54.v20240208
• [KYUUBI #6418] Bump Jersey to 2.40
• [KYUUBI #6416] Generate flattened POM
• [KYUUBI #6415] Manage Jersey in kyuubi-server module
• [KYUUBI #6417] Remove scala compile args -Yno-adapted-args
• [KYUUBI #6382] Bump Delta from 3.1.0 to 3.2.0 for Spark 3.5
• [KYUUBI #6348] Upgrade iceberg from 1.5.0 to 1.5.2
• [KYUUBI #6328] Bump Hive 2.3.10
• [KYUUBI #6373] Avoid NPE in get operation log api
• [KYUUBI #6377] Fix isCommand check and set min rows threshold for saveToFile
• [KYUUBI #6357] Bump org.bouncycastle:bcprov-jdk18on from 1.77 to 1.78
• [KYUUBI #6363] Increase CI concurrency
• [KYUUBI #6244] Fix test logs upload error
• [KYUUBI #6294] Simplify Netty and gRPC dependency management
• [KYUUBI #6252] Upgrade hive-service-rpc 4.0.0
• [KYUUBI #6268] Specify logDir for RollingFile filePattern
• [KYUUBI #6267] Remove unused dependency management in POM
• [KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
• [KYUUBI #6260] Clean up and improve comments for spark extensions
• [KYUUBI #6257] Bump vite from 4.5.2 to 4.5.3 in /kyuubi-server/web-ui
• [KYUUBI #6243] Add missing Impala alias for JDBC engine
• [KYUUBI #6234] Fix invalid code-block syntax in docs
• [KYUUBI #2782] Add missed UNION_TYPE back
• [KYUUBI #6206] Update comments for job group id of Spark broadcast jobs
• [KYUUBI #6197] Revise dependency management of Spark authZ plugin
• [KYUUBI #6196] Add log4j2 template for kyuubi repl commands
• [KYUUBI #6194] AuthZ shaded should include ranger-plugins-cred
• [KYUUBI #6188] Bump follow-redirects from 1.15.4 to 1.15.6 in web-ui
• [KYUUBI #6191] Update docs to mention support of Flink 1.19
• [KYUUBI #6134] Add profile and CI for Flink 1.19
• [KYUUBI #6193] Add known_transkations
• [KYUUBI #6179] Bump iceberg from 1.4.3 to 1.5.0

Credits

Last but not least, this release would not have been possible without the following contributors:

• Lennon Chin
• JiaLiangC
• Senmiao Liu
• Xiao Liu
• Cheng Pan
• PorterZhang2021
• TakawaAkirayo
• Zhen Wang
• Fei Wang
• Binjie Yang
• He Zhao
• Mingliang Zhu
• amanraj2520
• beryllw
• camper42
• dupen01
• lllong33
• mrpengbei
• sudohainguyen
• zRains

Changlogs

• [KYUUBI #5451] Ignore NoSuchFileException during OperationLog.close()
• [KYUUBI #6291] Avoid NPE in MySQLErrPacket
• [KYUUBI #2782] Add missed UNION_TYPE back
• [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
• [KYUUBI #6061] Delete svn staging dir before checking out
• [KYUUBI #6047] Free up disk space
• [KYUUBI #5953] Update NOTICE
• [KYUUBI #5951] Improve doc for Session Conf Advisor
• [KYUUBI #5933] Happy New Year 2024
• [KYUUBI #5922] Enhance YARN app state judgment
• [KYUUBI #5832] Always perform closing action in OperationLog to avoid fd leak
• [KYUUBI #5857] Dependencies CI check when
  dev/dependencyList
  changes
• [KYUUBI #5849] Incorrectly parse JDBC URL while variable includes colon
• [KYUUBI #5822] Fix the usage msg of KyuubiBeeLine
• [KYUUBI #5819] Fix rst syntax of missing blank line after code-block
• [KYUUBI #5785] Fix flaky test - JpsApplicationOperation with spark local mode
• [KYUUBI #5750] Fix the inaccurate issue of obtaining COLUMN_SIZE in the decimal field jdbc of spark engine
• [KYUUBI #5754] Fix engine id of the engine probe failures logging messages
• [KYUUBI #5317] Hive Connector throws NotSerializableException on reading Hive Avro partitioned table
• [KYUUBI #5720] Fix flaky test in TPCHQuerySuite
• [KYUUBI #5713] Backport HIVE-27271: Client connection to HS2 fails when transportMode=http, ssl=true, sslTrustStore specified without trustStorePassword in the JDBC URL
• [KYUUBI #5697] Fix incorrect url reference in code and docs
• [KYUUBI #5599] Correct the parameter name of fetching op log in RESTful API
• [KYUUBI #5499]Catch any exception when closing idle session
• [KYUUBI #5414] KSHC reader should not pollut the global hiveConf instance
• [KYUUBI #5432] Fix typo in README.md
• [KYUUBI #5347] Fix java code errors in quick_start_with_jdbc
• [KYUUBI #5333] Aaptive call fetchLaunchEngineResult
• [KYUUBI #5308] Ensure release using Java 8

Credits

Last but not least, this release would not have been possible without the following contributors:

• Shaoyun Chen
• Chao Chen
• Flyangz
• Pengqi Li
• Bowen Liang
• Omkar Makhare
• Drake Youngkun Min
• Cheng Pan
• Gianluca Principini
• Zeyu Wang
• Kang Wang
• Fei Wang
• Zhen Wang
• Kaifei Yi
• He Zhao
• mrtisttt

Changelogs

• [KYUUBI #6320] Fix terminated application pods not deleted issue when kyuubi server restarted
• [KYUUBI #6298] Make ServiceMonitor use selector labels
• [KYUUBI #6291] Avoid NPE in MySQLErrPacket
• [KYUUBI #6283] Improve the message of distribution built without enabling web ui
• [KYUUBI #6287] Avoid missing matche
• [KYUUBI #6268] Specify logDir for RollingFile filePattern
• [KYUUBI #6212] Added audit handler shutdown to the shutdown hook
• [KYUUBI #6271] Upgrade kafka-clients from 3.5.1 to 3.5.2
• [KYUUBI #5767] Fix spark batch conf not convert issue
• [KYUUBI #6223] Fix Scala interpreter can not access spark.jars issue
• [KYUUBI #6234] Fix invalid code-block syntax in docs
• [KYUUBI #6215] Improve DropIgnoreNonexistent rule for Spark 3.5
• [KYUUBI #6221] Fix parameter replacement issue caused by incorrect sql split
• [KYUUBI #2782] Add missed UNION_TYPE back
• [KYUUBI #6207] Support to retrieve Spark UserDefinedType result
• [KYUUBI #6206] Update comments for job group id of Spark broadcast jobs
• [KYUUBI #6204] Fix kyuubi session limiter leak when opening session failed
• [KYUUBI #6182] JDBC engine using the overlay conf as session conf
• [KYUUBI #6193] Add known_transkations
• [KYUUBI #6183] KyuubiConf.getKubernetesConf should set namespace even if the Kubernetes context is empty
• [KYUUBI #6175] Refactor copy path about SPARK_HOME in docker
• [KYUUBI #6166] Fix wrong copy path about SPARK_HOME in docker
• [KYUUBI #6170] Merge JDBC Trino TPC jobs and Style Check jobs in GitHub Actions workflow
• [KYUUBI #6164] Limit max parallel for each GitHub Actions workflow
• [KYUUBI #6156] Remove flink. prefix for create session configurations
• [KYUUBI #6173] Bump maven-download-plugin from 1.7.1 to 1.8.1
• [KYUUBI #6168] Check if forcedMaxOutputRows is negative
• [KYUUBI #6167] Drop JDK 11 on GitHub Actions
• [KYUUBI #6158] Remove confusing variable
• [KYUUBI #6152] Remove useless variable
• [KYUUBI #6130] Stop engine immediately after close session for CONNECTION level FlinkSQLEngine
• [KYUUBI #6147] Use the overlay conf as session conf
• [KYUUBI #6078] KSHC should handle the commit of the partitioned table as dynamic partition at write path
• [KYUUBI #6139] Expose env vars and sys props to JUnit tests too
• [KYUUBI #6126] Filter blank engine home env
• [KYUUBI #6114] Properly handle KyuubiStatement#getMoreResults(CLOSE_CURRENT_RESULT)
• [KYUUBI #5902] Bump doc build dependencies
• [KYUUBI #6081] RESTful client should retry on UnknownHostException
• [KYUUBI #6071] Add .java-version into git ignore
• [KYUUBI #6069] TFrontendServiceSuite - close expired operations
• [KYUUBI #5104] BatchesResourceSuite - expose the metrics with operation type and current state
• [KYUUBI #6061] Delete svn staging dir before checking out

Credits

Last but not least, this release would not have been possible without the following contributors:

• Shaoyun Chen
• Lennon Chin
• Fei Wang
• Bowen Liang
• Xiao Liu
• Senmiao Liu
• Tigran Manasyan
• Cheng Pan
• TakawaAkirayo
• Zhen Wang
• Binjie Yang
• Vino Yang
• Kaifei Yi
• He Zhao
• amanraj2520
• dupen01
• sudohainguyen

Highlight

• Fully verified with Java 8, 11, and 17, Scala 2.12 and 2.13
• Fully verified with Apache Spark 3.1 to 3.5, support for Spark 3.1 is deprecated and will be removed in the next release
• Fully verified with Apache Flink 1.16 to 1.19, support for Flink 1.16 is deprecated and will be removed in the next release
• Fully verified with Apache Hive 2.3, 3.1, and Cloudera Hive 2.1.1-cdh6, support running Hive engine on YARN (experimental)
• Spark engine supports save large query result to file before retrieving
• JDBC engine supports more dialects, including Doris, MySQL, PostgreSQL, Phoenix, StarRocks and Impala
• Kyuubis server metastore supports PostgreSQL as backend storage
• Introduce a basic SQL query editor in Kyuubi Web UI
• Continuously Spark AuthZ plugin improvements, including enhanced Iceberg support and new introduced Apache Hudi and Delta Lake support
• Continuously Kyuubi & Spark on Kubernetes improvements

Server

• [KYUUBI #5423] Support chaining SessionConfAdvisors
• [KYUUBI #5631][K8S][HELM] Set session affinity if needed in helm chart
• [KYUUBI #5674] Add PostgreSQL as backend database for kyuubi metadata
• [KYUUBI #5683] Add kyuubi.session.proxy.user as an alternative to hive.server2.proxy.user
• [KYUUBI #5704] Add unit tests for kyuubi.session.proxy.user in RESTful API
• [KYUUBI #5729] Use G1GC as Java option example in kyuubi-env template
• [KYUUBI #5797] Support to describe engine with command in current session
• [KYUUBI #5797][FOLLOWUP] Desc engine command support show engine registered attributes
• [KYUUBI #5873] Support to get operation progress with RESTful API
• [KYUUBI #5912] Cleanup unused config kyuubi.frontend.thrift.http.allow.user.substitution
• [KYUUBI #5921] EngineRef#deregister should return the value
• [KYUUBI #5934][K8S][HELM] Add Spark configuration support
• [KYUUBI #6003] Allow disabling user impersonation on launching engine
• [KYUUBI #6006][HELM] Support additional labels for service monitor
• [KYUUBI #6066] Fix PostgreSQL metastore migration script failure if indexes already exist
• [KYUUBI #6070] Improve perf on assembling row-based TRowSet
• [KYUUBI #6080] Improve instruction for deploy Kyuubi on K8s
• [KYUUBI #6081] RESTful client should retry on UnknownHostException
• [KYUUBI #6099] Remove support of Derby for Kyuubi metastore
• [KYUUBI #6126] Filter blank engine home env
• [KYUUBI #6140] Move getCodeSourceLocation to kyuubi-util

Spark Engine

• [KYUUBI #5377] Spark engine query result save to file
• [KYUUBI #5377][FOLLOWUP] Spark engine query result save to file
• [KYUUBI #5377][FOLLOWUP] Always try to cleanup session result path
• [KYUUBI #5377][FOLLOWUP] Get limit from more spark plan and regard result max rows
• [KYUUBI #5870] Directly mapping engine’s data type to Java type for TRowSet generation
• [KYUUBI #5877] Support Python magic syntax for notebook usage
• [KYUUBI #5877][FOLLOWUP] Add spark output mode to better support PySpark notebook
• [KYUUBI #5894] Separate closed and online sessions/statements in the SparkUI’s engine tab
• [KYUUBI #5899] Refine the python/scala interpreter error prompt
• [KYUUBI #6053][K8S] Show more details for pod/container status in application error
• [KYUUBI #6083] Bump Spark 3.3.3 to 3.3.4
• [KYUUBI #6091] Deprecate and remove building support for Spark 3.1
• [KYUUBI #6092] Enable DeltaTest for Spark 3.5
• [KYUUBI #6094] Bump Spark from 3.5.0 to 3.5.1
• [KYUUBI #6095] Enable PaimonTest for Spark 3.5
• [KYUUBI #6096] Enable PySpark tests for Spark 3.5
• [KYUUBI #6107][Spark] Collect and summarize the executorRunTime and executorCpuTime of the statement
• [KYUUBI #6107][FOLLOWUP] Add comments for the newly added parameters of the Spark event
• [KYUUBI #6108] Display the CPU time consumed by the statement in the Spark Engine tab
• [KYUUBI #6156] Remove flink. prefix for create session configurations
• [KYUUBI #6163] Set default Spark version to 3.5
• [KYUUBI #6166][MINOR] Fix wrong copy path about SPARK_HOME in docker
• [KYUUBI #6175][MINOR] Refactor copy path about SPARK_HOME in docker

Spark AuthZ Plugin

• [KYUUBI #5248][AUTHZ] Check previleges for Iceberg’s CALL to RewriteDataFilesProcedure procedure
• [KYUUBI #5255] Add an optional comment field to the authz specs for better recognition
• [KYUUBI #5284] Support Hudi Alter Table Command in Authz
• [KYUUBI #5323][AUTHZ] Drop Hive and Iceberg tables with PURGE option in tests
• [KYUUBI #5359][AUTHZ] Support Create Table Commands for Hudi
• [KYUUBI #5360] Support Hudi InsertIntoHoodieTableCommand
• [KYUUBI #5361][AUTHZ] Support Drop/Truncate Table Commands for Hudi
• [KYUUBI #5362] Remove Spark 3.0 support for Authz
• [KYUUBI #5399][AUTHZ] Cleanup Spark 3.0 specific implementation
• [KYUUBI #5407][AUTHZ] Tests for Iceberg system procedures of snapshot management
• [KYUUBI #5417] should not check in-subquery in permanent view
• [KYUUBI #5417] Authz should not check dependent subquery plan privilege
• [KYUUBI #5427][AUTHZ] Shade spark authz plugin
• [KYUUBI #5446][AUTHZ] Support Create/Drop/Show/Reresh index command for Hudi
• [KYUUBI #5447][AUTHZ] Support Hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand
• [KYUUBI #5447][FOLLOWUP] Remove unrelated debug prints in TableIdentifierTableExtractor
• [KYUUBI #5452][AUTHZ] Support Compaction table commands for Hudi
• [KYUUBI #5455][AUTHZ] Support hudi CompactionHoodiePathCommand & CompactionShowHoodiePathCommand
• [KYUUBI #5457][AUTHZ] Support RepairTable Commands for Hudi
• [KYUUBI #5462][AUTHZ] Support create table command for Paimon
• [KYUUBI #5472] Permanent View should pass column when child plan no output
• [KYUUBI #5472][AUTHZ][FOLLOWUP] Check permanent view also need support merge projection
• [KYUUBI #5475][FOLLOWUP] Authz check permanent view’s subquery should check view’s correct privilege
• [KYUUBI #5478][AUTHZ] Support Hudi ShowHoodieTablePartitionsCommand
• [KYUUBI #5479][AUTHZ] Support Hudi CallProcedureHoodieCommand for stored procedures
• [KYUUBI #5492][AUTHZ] saveAsTable create DataSource table miss db info
• [KYUUBI #5496][AUTHZ] Authz shaded include the transitive jackson libs
• [KYUUBI #5497][AuthZ] Simplify debug message for missing field/method in ReflectUtils
• [KYUUBI #5506][AUTHZ][MINOR] Reuse loaded table extractor in Hudi table extractors
• [KYUUBI #5512][AuthZ] Remove the non-existent query specs in Deletes and Updates
• [KYUUBI #5529][AUTHZ] Support create table command for Delta Lake
• [KYUUBI #5503][AUTHZ] Check plan auth checked should not set tag to all child nodes
• [KYUUBI #5503][FOLLOWUP][AUTHZ] Authz should skip inner plan that have been verified
• [KYUUBI #5529][AUTHZ][FOLLOWUP] Fix Create table ut for Delta Lake does not trigger a sql operation
• [KYUUBI #5529][AUTHZ][FOLLOWUP] Remove useless org.apache.spark.sql.delta.commands.CreateDeltaTableCommand
• [KYUUBI #5532][AUTHZ] Support Alter table commands of updating table schema for Delta Lake
• [KYUUBI #5533][AUTHZ] Support merge into table command for Delta Lake
• [KYUUBI #5534][AUTHZ] Support optimize table command for Delta Lake
• [KYUUBI #5535][AUTHZ] Support vacuum table command for Delta Lake
• [KYUUBI #5537][AuthZ] Order Command Spec by Command Class [group, classname]
• [KYUUBI #5543][AUTHZ] Generalize command specs and check duplicated class names of command spec
• [KYUUBI #5546][AUTHZ] Reorgnize the package names for rules
• [KYUUBI #5555][AUTHZ] Extractor common authorization rule class
• [KYUUBI #5556][AUTHZ] Support Alter table commands of set table properties
• [KYUUBI #5557][AUTHZ] Refactor code about handle PermanentViewMarker
• [KYUUBI #5565][AUTHZ] Support Delete/Insert/Update table command for Delta Lake
• [KYUUBI #5575][AUTHZ] Support InsertIntoDataSourceDir/InsertIntoHiveDirCommand path check
• [KYUUBI #5575][AUTHZ] Check path privilege should use correct action type
• [KYUUBI #5576][Bug] Fix wrong code in test case of dir command
• [KYUUBI #5579][AUTHZ] Support LogicalRelation don’t have CatalogTable but have HadoopFsRelation
• [KYUUBI #5594][AUTHZ] BuildQuery should respect normal node’s input
• [KYUUBI #5611][AUTHZ] Authz support path privilege for SaveIntoDataSourceCommand
• [KYUUBI #5628][AUTHZ] Support path privilege check for resource command
• [KYUUBI #5630][Authz] Support path check of LoadDataCommand
• [KYUUBI #5638][AUTHZ] URI related privileges shall be READ/WRITE
• [KYUUBI #5638][FOLLOWUP] Remove unnecessary actionType in uri desc
• [KYUUBI #5642][AUTHZ] Support path privilege of CreateDatabaseCommand/AlterDatabaseSetLocationCommand/CreateNamespace/SetNamespaceLocation
• [KYUUBI #5642][FOLLOWUP] Fix unit test
• [KYUUBI #5652][AUTHZ] CreateTable related command support path privilege
• [KYUUBI #5657][AUTHZ] Support path privilege of AlterTableSetLocationCommand/AlterTableAddPartitionCommand
• [KYUUBI #5664][AUTHZ] Support create path-based table for Delta Lake
• [KYUUBI #5677][AUTHZ] Typeof expression miss column information
• [KYUUBI #5684][AUTHZ] Support delete/update path-based table for Delta Lake
• [KYUUBI #5690][AUTHZ] Support insert into/overwrite path-based table for Delta Lake in Authz
• [KYUUBI #5692][Bug] Authz not skip explain command
• [KYUUBI #5707][AUTHZ] Support merge into path-based table for Delta Lake in Authz
• [KYUUBI #5715][MINOR] Replace with .isDefined and .isEmpty
• [KYUUBI #5726][AUTHZ] Support optimize path-based table for Delta Lake in Authz
• [KYUUBI #5735][AUTHZ] Support vacuum path-based table for Delta Lake
• [KYUUBI #5743][AUTHZ] Improve AccessControlException verification of RangerSparkExtensionSuite
• [KYUUBI #5768][AUTHZ] Authz internal place holder should skip privilege check
• [KYUUBI #5780][AUTHZ] Treating PermanentViewMarker as LeafNode make code simple and got correct privilege object
• [KYUUBI #5780][AUTHZ][FOLLOWUP] Format PermanentViewMarker tree string
• [KYUUBI #5793][AUTHZ][BUG] PVM with nested scalar-subquery should not check src table privilege
• [KYUUBI #5884] PVM should inherit MultiInstance and wrap with new exprId
• [KYUUBI #5903] PVM should override computeStats method
• [KYUUBI #5913][Bug] After resolve PVM should mark all nodes as checked
• [KYUUBI #5937] PVM cause cache table not work
• [KYUUBI #5964][BUG] Avoid check not fully optimized query for InsertIntoDataSourceDirCommand and InsertIntoDataSourceCommand
• [KYUUBI #5965][AUTHZ] Supports check hoodie procedures show_commits resource privileges
• [KYUUBI #5985][AUTHZ][MINOR] Remove incorrect getUri method
• [KYUUBI #5997][AUTHZ] Avoid unnecessary loop of RuleEliminateTypeOf
• [KYUUBI #6007] AuthZ should check hoodie procedures path resource privileges

Other Spark Plugins

• [KYUUBI #5580] Support generate and window operators for InferRebalanceAndSortOrders
• [KYUUBI #5786] Disable spark script transformation
• [KYUUBI #5816] Change spark rule class to object or case class
• [KYUUBI #6125] Drop Kyuubi extension for Spark 3.1
• [KYUUBI #6168] Check if forcedMaxOutputRows is negative

Flink Engine

• [KYUUBI #5264][FLINK] Force disable Flink’s session timeout
• [KYUUBI #5489] Adjust shuffle partitions dynamically
• [KYUUBI #5505][FLINK] Support HELP command
• [KYUUBI #6097][KYUUBI #6098] Upgrade Flink 1.16.3 and 1.18.1
• [KYUUBI #6130] Stop engine immediately after close session for CONNECTION level FlinkSQLEngine
• [KYUUBI #6134] Support Flink 1.19
• [KYUUBI #6142] Deprecate Flink 1.16
• [KYUUBI #6152] Remove useless variable
• [KYUUBI #6158][MINOR] Remove confusing variable
• [KYUUBI #6171] Drop building support for Flink 1.16
• [KYUUBI #6171][FOLLOWUP] Restore flink-1.16 profile to recover Flink IT

Hive Engine

• [KYUUBI #5830] Test Hive engine with CDH Hive 2.1.1-cdh6.3.2
• [KYUUBI #5865] Support get SQL keywords from Hive engine through API
• [KYUUBI #5867] HiveEngine support run on YARN mode
• [KYUUBI #5867][FOLLOWUP] Remove unused config
• [KYUUBI #5867][FOLLOWUP] Fix generated configuration docs and typo

Trino Engine

• [KYUUBI #5968] Support set authentication user for Trino engine
• [KYUUBI #6147][TRINO] Use the overlay conf as session conf

JDBC Engine

• [KYUUBI #5375] JDBC Engine supports PostgreSQL
• [KYUUBI #5382][JDBC] Duplication cleanup improvement in JdbcDialect and schema helpers
• [KYUUBI #5404] JDBC Engine supports StarRocks
• [KYUUBI #5464] JDBC Engine supports MySQL
• [KYUUBI #5509] Add Apache Impala JDBC engine dialect
• [KYUUBI #5582] JDBC Engine supports configurable default fetchSize
• [KYUUBI #5851] Generalize TRowSet generators
• [KYUUBI #5861] Generalize TRowSet generator for JDBC engine with dialects
• [KYUUBI #5862] Use TestContainerForAll for testing JDBC engine with testcontainers
• [KYUUBI #5906][JDBC] Rebase Doris dialect implementation
• [KYUUBI #6037] Fix the JDBC engine provider name

Other Engine

• [KYUUBI #5384][KSCH] Hive connector supports Spark 3.5
• [KYUUBI #5709] Avoid exposing client keystore and truststore passwords in the JDBC URL
• [KYUUBI #5783] Switch to kyuubi-relocated-hive-service-rpc
• [KYUUBI #5828][CHAT] Kyuubi chat engine supports ernie bot(文心一言)
• [KYUUBI #6078] KSHC should handle the commit of the partitioned table as dynamic partition at write path
• [KYUUBI #6114] Properly handle KyuubiStatement#getMoreResults(CLOSE_CURRENT_RESULT)

Web UI

• [KYUUBI #5366][UI] Add submit SQL page in Kyuubi Web UI
• [KYUUBI #5517][UI] Initial implement the SQL Lab page
• [KYUUBI #5958][WEBUI] Bump follow-redirects from 1.15.1 to 1.15.4
• [KYUUBI #6002] Bump vite from 4.3.9 to 4.5.2 in web UI

Documentation

• [KYUUBI #5265][DOCS][FLINK] Add notice about Jobmanager HA to Flink deployment docs
• [KYUUBI #5500] Add Kyuubi Code Program to Doc
• [KYUUBI #5688][DOC] Update supported spark 3.5.x in authz doc
• [KYUUBI #5762][DOC] Introduce Kyuubi Web UI Engine Page
• [KYUUBI #5893][DOC] Update logos in layer picture on welcome page
• [KYUUBI #5905][DOC] Workaround for long inlined code sample in config page
• [KYUUBI #6010] Improve Gluten usage documentation
• [KYUUBI #6119][DOC] Update doc for HA/Zookeeper Configuration

Building, Infrastructure and Dependency

• [KYUUBI #4453][FOLLOW] Delete LICENSE for removed kubernetes client dependencies
• [KYUUBI #5104][TEST] BatchesResourceSuite - expose the metrics with operation type and current state
• [KYUUBI #5241] Bump Commons Lang3 from 3.12.0 to 3.13.0
• [KYUUBI #5245][INFRA] Daily deploy snapshot version on branch-1.8
• [KYUUBI #5266] Upgrade playground to Kyuubi 1.7.1
• [KYUUBI #5322][INFRA] Enable GitHub Projects
• [KYUUBI #5324] Bump latest 1.7.3 in playground and helm charts
• [KYUUBI #5342] Add label hacktoberfest to project
• [KYUUBI #5356][INFRA] GitHub Issue template for 2023 Kyuubi Code Contribution Program
• [KYUUBI #5395] Bump netty from 4.1.93.Final to 4.1.100.Final
• [KYUUBI #5449] Bump Delta Lake 3.0.0
• [KYUUBI #5501] Update codecov token and fix codecov reporting on PRs
• [KYUUBI #5509][FOLLOWUP] JDBC IT should always depends on kyuubi-hive-jdbc-shaded
• [KYUUBI #5600] Fix flaky test SessionsResourceSuite
• [KYUUBI #5602] Improve issue template for 2023 Kyuubi Code Contribution Program
• [KYUUBI #5660][TEST] Ensure SPI provider file content sorted in alphabet order
• [KYUUBI #5671] Bump axios from 0.27.2 to 1.6.0 in /kyuubi-server/web-ui
• [KYUUBI #5674][LICENSE][FOLLOWUP] Update license files
• [KYUUBI #5676] Bump Apache parent pom from 30 to 31
• [KYUUBI #5701] Enhance the PR Template to make PR desc informative
• [KYUUBI #5716][TEST] Fix match may not be exhaustive in Scala 2.13
• [KYUUBI #5774][CI] Bump actions/checkout from v3 to v4
• [KYUUBI #5779][CI] Upgrade Github actions for Docker
• [KYUUBI #5800][KYUUBI#5467] Integrate Intel Gluten with Spark engine
• [KYUUBI #5835] Remove issue template for 2023 Kyuubi Code Contribution Task
• [KYUUBI #5838][BUILD] Bump actions/setup-python from v4 to v5
• [KYUUBI #5839][BUILD] Bump actions/setup-java from v3 to v4
• [KYUUBI #5841][BUILD] Bump actions/stale from v7 to v9
• [KYUUBI #5842][BUILD] Bump actions/setup-node from v3 to v4
• [KYUUBI #5874][INFRA] Bump github labeler action to v5
• [KYUUBI #5879] Bump dropwizard metrics from 4.2.8 to 4.2.23
• [KYUUBI #5902] Bump doc build dependencies
• [KYUUBI #5914] Update layer diagram on welcome page
• [KYUUBI #5925] Kyuubi TPC-DS support running benchmark with skipping some queries
• [KYUUBI #5939] Bump Gluten version to recover Gluten IT
• [KYUUBI #5941] Drop Kubernetes Block Cleaner Tool from Kyuubi
• [KYUUBI #5944] Introduce Prometheus and Grafana for Kyuubi Playground
• [KYUUBI #5955][TEST][GLUTEN] Disable gluten ui when running integration test
• [KYUUBI #5973] Bump gRPC from 1.53.0 to 1.60.1
• [KYUUBI #5974] Bump jetcd 0.7.7
• [KYUUBI #5975] Bump trino-client 411
• [KYUUBI #5975][FOLLOWUP][TEST] Fix trino - get type info
• [KYUUBI #6043][TEST][GLUTEN] Gluten-it gluten package add arch suffix
• [KYUUBI #6061] Delete svn staging dir before checking out
• [KYUUBI #6063] Bump iceberg 1.4.3
• [KYUUBI #6067] Bump org.postgresql:postgresql from 42.6.0 to 42.7.2
• [KYUUBI #6068] Remove community section from user docs
• [KYUUBI #6069][TEST] TFrontendServiceSuite - close expired operations
• [KYUUBI #6071] Add .java-version into git ignore
• [KYUUBI #6074] Add a script to simplify the process of creating release notes
• [KYUUBI #6075] Update known_translations
• [KYUUBI #6076] Update Helm Chart and playground to use 1.8.1
• [KYUUBI #6085] Bump gRPC from 1.60.1 to 1.60.2
• [KYUUBI #6089] Bump Paimon from 0.5.0-incubating to 0.7.0-incubating
• [KYUUBI #6090] Bump Delta from 3.0.0 to 3.1.0 for Spark 3.5
• [KYUUBI #6101] Bump Jackson from 2.15.0 to 2.15.4
• [KYUUBI #6109] Fork beeline module from Apache Hive 3.1.3
• [KYUUBI #6117] Bump Kyuubi Shaded 0.3.0
• [KYUUBI #6118] Migrate to shaded HMS client for getting delegation token on server
• [KYUUBI #6129][BUILD] Remove the extension spark jars before build to avoid include unspecified version
• [KYUUBI #6131] Simplify Maven dependency management after dropping building support for Spark 3.1
• [KYUUBI #6139] Expose env vars and sys props to JUnit tests too
• [KYUUBI #6143] Add Dockerfile.HMS for testing
• [KYUUBI #6162] Cut out hive-common deps in beeline module
• [KYUUBI #6164] Limit max parallel for each GitHub Actions workflow
• [KYUUBI #6145] Use HiveMetaStore Docker for testing
• [KYUUBI #6145][FOLLOWUP] Remove unused dependency commons-lang
• [KYUUBI #6151] Remove invalid matrix spark 3.1 in GA
• [KYUUBI #6153] Cut out Hive JDBC and Hive Service deps in beeline module
• [KYUUBI #6160] Fix beeline test KyuubiBeeLineTest.testKyuubiBeelineComment
• [KYUUBI #6167] Drop JDK 11 on GitHub Actions
• [KYUUBI #6170] Merge JDBC Trino TPC jobs and Style Check jobs in GitHub Actions workflow
• [KYUUBI #6173] Bump maven-download-plugin from 1.7.1 to 1.8.1
• [KYUUBI #6174] Remove unused dependency management from POM
• [KYUUBI #6177] Bump BouncyCastle from 1.67 to 1.77
• [KYUUBI #6178] Improve Gluten integration test

Credits

Last but not least, this release would not have been possible without the following contributors:

• GitHub Bot
• Shaoyun Chen
• GGHWS
• Deepali Gupta
• Baokun Han
• Denis Krivenko
• Kwafoor
• Pengqi Li
• Bowen Liang
• Paul Lin
• Senmiao Liu
• Chenliang Lu
• Tigran Manasyan
• Hai Nguyen
• Cheng Pan
• XH Qian
• Zhen Wang
• Fei Wang
• Kang Wang
• Junjie Xu
• Binjie Yang
• Kent Yao
• Xianxun Ye
• Kaifei Yi
• David Yuan
• He Zhao
• Yangming Zhao
• Yifan Zhou
• Mingliang Zhu
• Yi Zhu
• Yaogai Zhu
• huangxiaopingRD
• kevinjmh
• mrtisttt
• BingKun Pan
• rhh777
• silencily
• sincewhen
• Vino Yang