ZenML Release Notes

This release smooths out many rough edges: fewer database-is-locked errors for local SQLite stores, correct project scoping for artifact deletion and prefix lookups, complete exception tracebacks, and no more unnecessary Kubernetes pod retries for already-finished runs. MLflow tracking behaves better on Databricks and managed runtimes, cloudpickle artifacts are validated with a SHA-256 hash on load, service account tokens and adopted API keys keep working through migrations, and Helm deployments can supply the ZenML Pro enrollment key from an existing Kubernetes Secret.

Runtime and orchestration

Multi-pod Kubernetes step operator jobs: Command steps can now run across multiple Kubernetes pods with the Kubernetes step operator. Set pod_count in KubernetesStepOperatorSettings to launch the step as an indexed job, making it easier to distribute command-style workloads across pods. PR #5104

Local Docker sandbox: ZenML now includes a local Docker sandbox, plus a unified settings model for containerized sandboxes. Local sandbox workflows also support file upload and download, making it easier to test containerized ZenML behavior locally before moving to remote infrastructure. PR #5102

Deployment, security, and artifact integrity

Enrollment keys from Kubernetes Secrets: The Helm chart now supports server.pro.enrollmentKeySecretRef, so ZenML Pro enrollment keys can be injected from an existing Kubernetes Secret instead of being stored inline in Helm release values. The secret reference is applied consistently to the server, migration, and worker containers. PR #5123

Cloudpickle artifact hash validation: Cloudpickle-materialized artifacts now store a SHA-256 hash when written and validate that hash before loading. This helps detect corrupted or unexpectedly modified artifact files earlier and fail with a clearer integrity signal. PR #5103

CLI and dashboard UX

Select a stack during login: zenml login now accepts --stack, allowing you to connect to a server and immediately set the active stack in one command. When used together with --project, ZenML applies the project first so the stack is resolved in the intended project context. PR #5125

Improved secret value display: Secret values in the dashboard no longer expand indefinitely in the UI and are truncated for readability. A direct copy action is now available, making it easier to work with long secret values without disrupting the page layout. PR #1104

Timeline updates for cancelled runs: The dashboard timeline now shows steps that were not started because a run was cancelled. The timeline filter also supports filtering for Not Started, making cancelled or partially executed runs easier to inspect. PR #1108

Fixed

Fewer SQLite lock failures for local stores: Local SQL stores backed by SQLite now wait up to 60 seconds for write locks instead of using Python’s 5-second default. This reduces sqlite3.OperationalError: database is locked failures when concurrent steps, such as mapped dynamic pipeline steps, finish and publish artifacts at the same time. PR #5096

Artifact deletion on deployed servers: Artifact version deletion now correctly applies project scope when checking whether an artifact is unused. This fixes deletion through the API on remote deployed servers where the default project is disabled. PR #5100

Complete exception tracebacks: Exception reporting now includes the full traceback lineage instead of only the final exception traceback. This makes chained failures easier to debug because the original cause is preserved alongside the final error. PR #5098

Smarter Kubernetes dynamic pipeline retries: Kubernetes orchestrator pods are no longer retried when the pipeline run is already in a finished state that cannot be retried. This avoids unnecessary pod restarts that would immediately exit with Run is already finished. PR #5107

Correct scoped prefix lookups: Prefix-based lookups now keep scoped and default filters properly constrained when matching by ID or name prefix. This prevents unrelated entities from being returned, for example when looking up a schedule trigger by a prefix that does not match its name or ID. PR #5126

Generic token provenance and service account auth: Generic and stack-deployment tokens now preserve the original service account or device provenance used to create them. This fixes authentication failures for pipeline workloads using server-issued JWTs for workspace-local service accounts, especially with external authentication enabled. PR #5127

MLflow tracking with managed runtimes and Databricks: ZenML’s MLflow experiment tracker now behaves more reliably when managed runtimes inject MLflow environment variables such as MLFLOW_RUN_ID. This avoids accidentally resuming an inherited run when ZenML needs to create its own run, improving compatibility with Databricks-backed MLflow setups. PR #5122

API keys after service account adoption: Existing workspace-level API keys remain valid when a workspace service account is adopted by an organization-level service account with the same name. This allows teams to migrate service accounts gradually without interrupting workloads that still use older API keys. PR #5138

What's Changed

Add version 0.96.1 to legacy docs by @github-actions[bot] in #5093

Add 0.96.2 to the migration tests by @github-actions[bot] in #5095

Bump the minor-and-patch group across 1 directory with 9 updates by @dependabot[bot] in #5076

Raise sqlite busy timeout for local stores by @htahir1 in #5096

Fix missing project scope when deleting artifact versions by @schustmi in #5100

Fix exception info cutoff by @schustmi in #5098

Fix flag syntax in service-connector register help example by @strickvl in #5101

Stop format_name_template from mutating the caller's substitutions dict by @chuenchen309 in #5088

Count an installed prerelease as installed by @chuenchen309 in #5089

Bump the minor-and-patch group with 6 updates by @dependabot[bot] in #5105

Docker sandbox by @schustmi in #5102

Bump the minor-and-patch group with 6 updates by @dependabot[bot] in #5115

Bump actions/setup-python from 6.3.0 to 7.0.0 by @dependabot[bot] in #5116

Bump astral-sh/setup-uv from 8.3.2 to 9.0.0 by @dependabot[bot] in #5117

Examples linting/formatting due to the new ruff version by @bcdurak in #5130

Support zenml enrollment key as a kubernetes secrets by @amitvikramraj in #5123

Upgrading the mypy dependency by @bcdurak in #5131

Validate cloudpickle content hash by @schustmi in #5103

Fix clean_requirements dropping > and != version specifiers by @chuenchen309 in #5085

Improved Kubernetes dynamic pipeline retries by @schustmi in #5107

Ability to select active stack upon login by @bcdurak in #5125

Add multi-node support for Kubernetes step operator by @schustmi in #5104

Fix scoped filters in prefix lookups by @bcdurak in #5126

Scope generic tokens to the original service account or device used to create them by @stefannica in #5127

MLFlow Experiment Tracker Databricks Improvements by @bcdurak in #5122

Fix broken trigger filtering by associate ids by @Json-Andriopoulos in #5132

Fix flaky schedule deletion test by @Json-Andriopoulos in #5135

Allow old API keys to be used for adopted service accounts by @stefannica in #5138

Pin JAX to avoid TensorFlow ml-dtypes conflict by @Json-Andriopoulos in #5137

Fix Haystack agent example for Haystack 3 by @strickvl in #5142

Prepare release 0.96.3 by @github-actions[bot] in #5143

New Contributors

@chuenchen309 made their first contribution in #5088

Full Changelog: 0.96.2...0.96.3

Dynamic pipelines

Explicit start ordering for dynamic steps: Dynamic pipelines now support start_after=... when calling steps, letting you control which concurrently launched steps should wait for others before starting. This makes it easier to model ordering constraints without turning concurrent parts of a dynamic pipeline into fully synchronous execution. Note that start_after is now a reserved step keyword, so steps that previously used a parameter with this name will need to be updated. PR #4995

More flexible dynamic step inputs: You can now configure whether JSON-serializable raw values passed to steps in dynamic pipelines should be treated as parameters instead of artifacts. The new environment-variable threshold defaults to 0 to preserve existing behavior, while explicit APIs such as with_options(parameters=...) and ExternalArtifact(...) remain available when you want to force either behavior. PR #5079

Improved dynamic execution semantics: Dynamic pipelines now support CONTINUE_ON_FAILURE execution mode, allowing already queued or asynchronous work to continue when an async step fails. ZenML also models implicit dependencies from newly launched steps to the last completed sync step, making mixed sync/async dynamic pipelines execute in a more predictable order. PR #5052

Integrations and deployment

DigitalOcean integration: ZenML now includes a first-class digitalocean integration with support for DigitalOcean Spaces artifact stores and DigitalOcean Container Registry stack components. Spaces support builds on the existing S3-compatible implementation while handling DigitalOcean regions and endpoint generation for you. PR #5054

Helm chart logging and OpenTelemetry configuration: The ZenML Helm chart now exposes server logging options and OpenTelemetry settings directly in values. You can configure console or JSON logging, service names, OTEL endpoints, and enable or disable traces, metrics, and logs without custom chart modifications. PR #5048

Data and metadata management

Server-side artifact data deletion: ZenML can now delete artifact version data through the server API, not only from a full client with direct stack access. This enables artifact metadata and backing data to be deleted from the UI or from thin clients that do not have the artifact store stack component locally available. PR #5034

Project metadata: Projects now support arbitrary project_metadata on create, update, and hydrated response models. Metadata is stored as portable JSON, preserved when omitted, replaced when explicitly supplied, and can be cleared by sending an empty object. PR #5086

Performance and scalability

Lower-memory cross-filesystem copies: fileio.copy() now streams cross-filesystem copies in bounded chunks instead of reading the entire file into memory. This significantly reduces peak memory usage for local-to-remote and remote-to-local artifact operations, including PathMaterializer, directory copies, integration materializers, and code archive upload/download flows. PR #5031

Fewer server requests during runs: ZenML now caches commonly reused project, store, stack, pipeline run, and completed step run responses in process where safe. Local and in-process execution paths make substantially fewer server requests, which improves responsiveness for pipelines with many steps. PR #5036 PR #5038

Faster DAG endpoint on large runs: The DAG endpoint now does less unnecessary parsing and object construction when serving large pipeline graphs. In benchmarks on a DAG with thousands of nodes and edges, endpoint latency was reduced by roughly half. PR #5051

Security and dependencies

FastAPI, Starlette, and OpenTelemetry updates: ZenML now supports FastAPI 0.138.0, raises the lower Starlette bound to 0.46.0 to pick up security fixes, and updates OpenTelemetry packages for compatibility. The update also removes an unused fastapi_utils dependency and cleans up deprecated FastAPI response and lifespan usage. PR #5017 PR #5060

Fixed

Token invalidation after credential changes: User tokens issued before a password change are now rejected, and tokens derived from API keys are tied to the key generation so rotated keys no longer leave stale sessions usable. The dashboard also correctly rotates API keys when a non-zero retention period is configured. PR #5025 PR #1088

Docker credentials supplied at runtime: ZenML now properly handles in-memory Docker credentials when a local Docker credential store already has entries for the same registry. This prevents the Docker Python client from silently preferring stale local credentials over credentials passed through ZenML. PR #5023

Remote image builds with restrictive .dockerignore files: ZenML no longer includes the root .dockerignore in generated build context archives for remote builders. This fixes builds on AWS CodeBuild, GCP Cloud Build, Kaniko, and similar builders when allowlist-style ignore patterns would otherwise exclude ZenML-generated files. PR #5033

Cleaner replay configurations: When replaying a pipeline run, ZenML now removes step parameters from the configuration if they are overridden by an input artifact. The displayed configuration no longer contains outdated values that were not actually used as step inputs. PR #5040

Deployment invocation with dict parameters: Dict-valued pipeline parameters sent to a deployment /invoke endpoint now replace compiled defaults instead of being recursively merged with them. This matches normal pipeline invocation behavior and prevents default keys from leaking into step inputs. PR #5042

Dynamic pipeline DAG race condition: ZenML now avoids a race where a step run could be visible in the database before its configuration was committed. This prevents intermittent 500 errors from the DAG endpoint while dynamic pipeline steps are being created. PR #5053

SSH orchestrator re-runs: SSH orchestrator container and Compose service names now avoid collisions across multiple runs of the same snapshot. Re-running from the dashboard, templates, or concurrent triggers no longer fails because an old container with the same name still exists on the host. PR #5082

What's Changed

• Add version 0.96.0 to legacy docs by @github-actions[bot] in #5021
• Remove placeholder data on queue full errors by @Json-Andriopoulos in #5027
• Support secret references in secret strings by @schustmi in #5024
• Bump FastAPI version by @schustmi in #5017
• Expire tokens after password changes and API key rotations by @stefannica in #5025
• Fix the in-memory docker credentials properly by @stefannica in #5023
• Remove deprecated workspace-local service accounts in ZenML Pro by @stefannica in #5032
• Stream cross-filesystem file copies in chunks by @kounelisagis in #5031
• Misc query improvements by @schustmi in #5035
• Don't include dockerignore in build context archive by @schustmi in #5033
• Reduce unnecessary server requests by @schustmi in #5036
• Remove shadowed step parameters when replaying by @schustmi in #5040
• Allow workspace service accounts for a little while longer by @stefannica in #5041
• Fix dict parameter overrides when invoking deployments by @kounelisagis in #5042
• Move task-specific guidance out of root CLAUDE.md by @strickvl in #5049
• Trim Codex agent instructions by @strickvl in #5050
• Fix duplicated word in service connector login output by @kounelisagis in #5046
• feat: added logging and otel support to helm charts by @amitvikramraj in #5048
• Support deleting artifact data from server API by @Json-Andriopoulos in #5034
• Fix dynamic pipeline step config race condition by @schustmi in #5053
• Fix broken docstring checks & instructions by @Json-Andriopoulos in #5058
• Updated docs to include K8s distros by @AlexejPenner in #5061
• fix: decouple the zenml logging level setting from ZENML_DEBUG env var by @amitvikramraj in #5062
• Always track pipeline image when using skip_build=True by @schustmi in #5069
• Improve DAG endpoint performance by @schustmi in #5051
• Fix truncated dashboard URLs in CLI output by @kounelisagis in #5045
• Dynamic pipeline start ordering by @schustmi in #4995
• Allow requesting extra step metadata in dag endpoint by @schustmi in #5070
• Narrow exception handling in list projects CLI to RuntimeError by @qubeena07 in #5074
• feat: bumped otel version to support fastapi 0.138.0 version by @amitvikramraj in #5060
• Allow passing raw values as step parameters in dynamic pipelines by @schustmi in #5079
• Dynamic pipeline improvements by @schustmi in #5052
• Fix/mlflow model registry crash by @Json-Andriopoulos in #5072
• Fix SSH orchestrator container name collisions on pipeline re-runs by @kounelisagis in #5082
• In-process execution caching by @schustmi in #5038
• Add DigitalOcean integration (Spaces artifact store, DOCR container registry) by @htahir1 in #5054
• Add project metadata by @strickvl in #5086
• Prepare release 0.96.2 by @github-actions[bot] in #5092

New Contributors

@kounelisagis made their first contribution in #5031

Full Changelog: 0.96.1...0.96.2