ZenML Release Notes

Run pipelines and steps over SSH: ZenML now includes an SSH orchestrator and SSH step operator for executing workloads on remote machines accessible via SSH (PR #4953). This makes it easier to use existing servers or on-prem infrastructure as execution targets without adopting a full cluster-based backend, while still managing runs through ZenML.

What's Changed

• Add version 0.95.1 to legacy docs by @github-actions[bot] in #5014
• Remove unused release notes generation by @schustmi in #5013
• Add missing group info import by @schustmi in #5012
• Add 0.96.0 to the migration tests by @github-actions[bot] in #5016
• Always log in to the local docker SDK by @stefannica in #5018
• Add OAuth service connector logo by @schustmi in #5019
• SSH Orchestrator + Step Operator by @htahir1 in #4953
• Prepare release 0.96.1 by @github-actions[bot] in #5020

Full Changelog: 0.96.0...0.96.1

Known Issues

If the Docker credential helper contains invalid credentials for a registry that ZenML is trying to use, those will have precendence over the credentials set by the service connector and will cause the following error: RuntimeError: Docker error: denied: Your authorization token has expired. Reauthenticate and try again... This will be fixed in the next release, in the meantime docker logout <REGISTRY> should fix it.

Breaking Changes

The minimum supported transformers version has been raised. If you use ZenML with Hugging Face/transformers, update your environment and dependency pins to a newer compatible transformers release before upgrading ZenML. PR #4976

The Azure integration now requires newer Azure dependency versions, and support for the deprecated azureml-core library has been fully removed. If you use ZenML on Azure, update your Azure-related dependency pins and migrate any remaining azureml-core usage to the currently supported Azure SDK packages before upgrading. PR #4987

In open-source ZenML server deployments without RBAC enabled, service account and API key management is now restricted to admins only. Non-admin users will no longer be able to manage service accounts or API keys they previously created, so move any required credentials and automation to admin-managed accounts as part of your upgrade. PR #5007

New integrations and execution backends

Trackio experiment tracking: ZenML now includes a Trackio experiment tracker integration, allowing pipelines to log experiment data through Trackio’s public API. This makes it easier to manage trial results and connect ZenML runs with Hugging Face-backed Trackio workflows such as datasets, spaces, and buckets. PR #4841

Backblaze B2 artifact store: You can now configure Backblaze B2 as a ZenML artifact store. This adds another S3-compatible storage option for teams that want to store pipeline artifacts in Backblaze infrastructure. PR #4791

Baseten step operator: ZenML now supports a baseten step operator flavor for running GPU workloads as Baseten Training jobs. It supports regular single-node steps with ZenML artifacts and logs, as well as multi-node distributed training through command steps that can consume Baseten’s distributed training environment variables. PR #4973

Generic OAuth2 service connector: A new OAuth2 service connector lets you authenticate external services using a static token, client credentials, or a client ID with refresh token. This provides a reusable connector option for services that expose OAuth2-based authentication. PR #4992

Workflow controls and platform operations

Replay input overrides by step name: When replaying a run, you can now use step_default_input_overrides to override a step input for every invocation of a step with the same name. Per-invocation step_input_overrides still take precedence, giving you both broad and targeted control during replay. PR #4978

Trigger cycle protection: ZenML now detects execution loops in Platform Event Trigger chains at the pipeline level. Cyclic trigger dispatches are skipped with the new SKIPPED_TRIGGER_CYCLE status, while unrelated downstream dispatches can continue normally and the affected cycle can be inspected through the SDK. PR #4971

Optional sandbox cleanup on exit: Sandbox sessions can now be configured to destroy the sandbox automatically when the session exits. The option defaults to False, preserving the behavior from previous releases unless you opt in. PR #4986

Configurable Kubernetes API retries: Kubernetes-based deployments can now configure retry behavior for Kubernetes API calls. This gives operators more control over resilience in clusters where transient API failures or throttling can occur. PR #5004

Dashboard filtering and connector selection improvements: The dashboard Timeline View now has additional filtering options, including more status filters. Component creation also gets a more efficient connector selector, making setup flows smoother in larger workspaces. PR #1084

Performance and scalability

Faster pipeline sorting by latest run: Listing pipelines sorted by latest run is now more efficient on large deployments. ZenML changed the query shape and supporting database indexing so the server no longer has to scan all runs for all pipelines just to compute the latest run timestamp. PR #4969

More efficient run and artifact queries: Several common server queries now load only the data they need and fetch related metadata more efficiently. This improves performance for DAG, pipeline run, step run, artifact version, and model version views, especially in workspaces with many entities. PR #4994

Catch-up cleanup for expired API transactions: Expired API transaction cleanup now works through bounded catch-up passes instead of a single fixed delete per interval. This helps servers recover from cleanup backlogs while keeping each database operation bounded, and also allows completed expired idempotency transactions to be safely reclaimed. PR #4943

Fixed

Artifact store caching prevents server OOMs: The ZenML server now caches artifact store instances used for operations such as reading logs and visualizations. This avoids repeatedly rebuilding heavy storage clients and helps prevent memory growth that could previously lead to OOM kills on busy servers. PR #4974

Docker credentials for image builds and pushes: ZenML now configures Docker credentials correctly when building and pushing container images. This fixes cases where username/password credentials were not applied to the expected registry URI, which could cause authentication failures in build and push workflows. PR #5005

Secret backup and restore authorization: Secret backup and restore endpoints now enforce an explicit admin check when RBAC is disabled. This closes an authorization gap where an authenticated non-admin user could access admin-only secret operations in the default non-RBAC setup. PR #5009

Safer custom flavor loading: Custom flavor sources are now validated before server-side hydration. ZenML ensures the configured source resolves to a Flavor subclass before instantiation, preventing arbitrary zero-argument callables from being invoked during flavor loading. PR #5008

What's Changed

Add version 0.95.0 to legacy docs by @github-actions[bot] in #4967
Improve Modal orchestrator docs by @strickvl in #4970
Bump the minor-and-patch group with 2 updates by @dependabot[bot] in #4962
Improve listing pipelines sorted by latest run by @schustmi in #4969
Fix CI-blocking dead docs links and transformers version floor by @htahir1 in #4976
Bump anthropics/claude-code-action from 1.0.143 to 1.0.150 in the minor-and-patch group by @dependabot[bot] in #4981
Add Trackio Integration for ZenML by @ParagEkbote in #4841
Document multi-node distributed training via CommandStep by @htahir1 in #4980
Fix databricks test leaking the global custom source root by @htahir1 in #4979
Cache artifact store in server to avoid OOM kills by @schustmi in #4974
Make the docs agent-aware: Agents guide in Learn, Kitaru routing across spaces by @htahir1 in #4897
Extract shared Modal settings/credentials mixins by @htahir1 in #4954
Add Backblaze B2 artifact store flavor by @goanpeca in #4791
Restore the source-root env var in the databricks test isolation fixture by @htahir1 in #4985
Add Missing Doc Links for Trackio by @ParagEkbote in #4989
Honesty pass on Replay and improve docs (drop non-existent cohort/Recipe/diff APIs) by @htahir1 in #4984
Misc DB improvements by @schustmi in #4982
Add option to destroy sandbox on session exit by @schustmi in #4986
Add Baseten step operator (single-node + multi-node distributed training) by @htahir1 in #4973
Improved snapshot fetching by @schustmi in #4991
Allow overwriting step inputs for all invocations when replaying by @schustmi in #4978
Add isolated step cleanup hooks by @schustmi in #4990
Update azure dependencies by @Json-Andriopoulos in #4987
Feature/prevent trigger execution loops by @Json-Andriopoulos in #4971
More query improvements by @schustmi in #4994
Configurable Kubernetes API retries by @schustmi in #5004
Async snapshot execution by @Json-Andriopoulos in #5003
Fix configuration of docker credentials for building/pushing container images by @stefannica in #5005
Bump actions/checkout from 6.0.3 to 7.0.0 by @dependabot[bot] in #5002
Bump the minor-and-patch group across 1 directory with 3 updates by @dependabot[bot] in #5000
Bump actions/cache from 5.0.5 to 6.1.0 by @dependabot[bot] in #5001
OAuth2 service connector by @schustmi in #4992
Changed default otel-service-name as workspace name for zenml-pro workspaces by @amitvikramraj in #5006
Add catch-up cleanup for expired API transactions by @safoinme in #4943
Add admin permissions check to the secret backup/restore endpoints by @stefannica in #5009
Validate custom flavors at loading time by @stefannica in #5008
Enforce admin-only permissions for service account management in non-RBAC deployments by @stefannica in #5007
Prepare release 0.96.0 by @github-actions[bot] in #5010

New Contributors

@ParagEkbote made their first contribution in #4841
@goanpeca made their first contribution in #4791

Full Changelog: 0.95.1...0.96.0

Dynamic pipelines with step operators

Dynamic pipeline execution is more reliable when steps use step operators.

Fixed

Fixed an issue where running a step with a step operator in a dynamic pipeline could fail unless that step was explicitly listed in pipeline.depends_on.

ZenML now falls back to the orchestrator image in this case, matching the behavior already used for isolated steps without step operators. PR #4960

Faster pipeline and step run queries

Common pipeline run and step run views should now load more efficiently, especially on larger deployments.

Improved database query performance by adjusting how related data is loaded for common pipeline run and step run queries.

This helps avoid expensive query plans in MySQL for paginated run listings, making these queries more scalable. PR #4965

Logging stability

Logging shutdown is now safer when using artifact-backed log stores.

Fixed

Fixed a deadlock that could happen during logging context shutdown when using fsspec-based artifact log stores with debug logs enabled.

ZenML now avoids writing back into the log store during shutdown, improving reliability for pipeline and step log collection. PR #4964

What's Changed

Add version 0.94.6 to legacy docs by @github-actions[bot] in #4957

Add 0.95.0 to the migration tests by @github-actions[bot] in #4959

Exclude zenml from its own dependency audit by @strickvl in #4961

Fallback to orchestrator image by @schustmi in #4960

Fix logging shutdown deadlock by @schustmi in #4964

Improve DB query plans by using selectinload by @schustmi in #4965

Prepare release 0.95.1 by @github-actions[bot] in #4966

Full Changelog: 0.95.0...0.95.1