NOTE

PyPI and Docker artifacts for this release were built from different SHAs, but are expected to be functionally the same. PyPI was built from d0c8667 and Docker from fa647f7.

BREAKING CHANGES

Audit logging now defaults to on for proxies running an Enterprise license. store_audit_logs becomes tri-state: unset plus a premium license resolves to enabled, replacing roughly fifteen scattered gate checks with one resolver, and LITELLM_STORE_AUDIT_LOGS is now honoured by the key hooks that previously ignored it. An enterprise proxy that never set the flag starts writing audit rows on upgrade, which will grow the audit table. Set store_audit_logs: false explicitly to keep the prior behavior. See PR #37518.

Shadow eval jobs are budgeted in dollars instead of turns. POST /auto_router/shadow_eval/start now takes max_budget in USD per key; a request that still sends max_turns gets a 422 naming the replacement. The sampler, sweep, stop guard, and derived status all gate on the billed response_cost of the shadow arm and the judge, and admission re-checks the proxy's cross-pod spend counter. Jobs created before the migration keep their configured turn budget, since max_budget stays NULL on those rows. See PR #37555.

A shadow eval job now covers a list of keys rather than one. api_key_ids replaces api_key_id on the start payload, results come back pooled and per key, and sibling rows are tied together by a new group_id column. A caller still sending the singular api_key_id must move to the list form. See PR #37251.

Key Highlights

• The Admin UI's migration off antd and Tremor is complete - @tremor/react, antd, and @ant-design/icons are all removed from the dashboard's dependencies, with the last components, the shared primitives, and every form ported onto shadcn (base-vega) and react-hook-form. 130 UI pull requests land in this window, and the dashboard is now on React 19
• Dark mode ships - a light/dark/system toggle in the top bar, semantic status tokens for success, warning and info, a dark variant of the LiteLLM logo, and an admin-supplied dark variant of a custom logo. Hardcoded Tailwind palette classes across the dashboard are mapped onto theme tokens so surfaces, form controls, inline styles and code blocks all follow the theme
• lite login is a real OAuth flow - the CLI now authenticates with an authorization code plus PKCE against the proxy, stores the credential and the refresh token in the OS keychain rather than a token.json on disk, and lite login --config-claude wires Claude Code up at login
• Batch spend is accounted for end to end - enqueued-token rate limiting admits batches against a token budget and refunds on completion or cancellation, cost rows are claimed atomically so multi-pod polling cannot double-bill, cancelled and failed batches that still produced output are billed, a single undecodable output line no longer zeroes a batch's spend, and Bedrock batches can be cancelled through POST /v1/batches/{id}/cancel
• Provisioned throughput can be declared in config.yaml - a PTU reservation no longer has to be created through the API; the rollup accrues flat cost for config-declared deployments, refuses an incomplete reservation the way the endpoints do, requires an operator-declared id, and warns when a config declares PTU while attribution is switched off
• The complexity router is operator-configurable - operator-defined tier sets for the LLM classifier, custom classifier plugins via classifier_type: custom, a plan-mode tier floor for coding-agent clients, a business classification rubric preset, and per-model reasoning effort in the tier editor

Included from v1.99.0-rc.2

This stable also carries everything that landed on the release line after the rc.1 cut, shipped in v1.99.0-rc.2 and included here:

• Anthropic
  • Translate tool_result document blocks in the /v1/messages bridge - PR #38251
• Dashboard (dark mode follow-ups)
  • Make provider logos readable in dark mode - PR #38588
  • One-click theme toggle and matching Docs/Blog styling in the top bar - PR #38601
  • Make code blocks and the logs JSON viewer follow the theme in dark mode - PR #38771, PR #38778
  • Make playground chat bubbles theme-aware, and theme the created-key box - PR #37978, PR #37985
• Dashboard (bug fixes)
  • Open select popups below the trigger instead of over it - PR #38554
  • Stop server-searched comboboxes from clobbering picks and queries, and let the paginated search select keep what the user types - PR #38574, PR #38475
  • Keep a deleted-from search query instead of blanking the box - PR #38830
  • Keep focus in the add model public name input while typing, and restore its tooltip layout - PR #38366, PR #37986
  • Restore the reopen control for the log drawer's trace sidebar - PR #38782
  • Stack the policy flow builder below the popup layer so guardrail options render - PR #38273
  • Drop stray text next to Close in the model connection test dialog - PR #38852
• Docker
  • Pin the image builds' apk python to 3.13 and bump wolfi-base for glibc 2.44 - PR #38917, PR #38973
• End-to-End Testing
  • De-flake the select-anchoring, router-fallback, vertex realtime and vision fixture specs (test-only) - PR #38848, PR #38862

New Providers and Endpoints

New Providers (5 new providers)

Provider: SCX.ai
Supported LiteLLM Endpoints: /chat/completions
Description: JSON-configured OpenAI-compatible provider, registered in the provider enum, config map, pricing map and dashboard, shipping GLM-5.2 and Qwen3.8-Max

Provider: Cognition
Supported LiteLLM Endpoints: /chat/completions
Description: Cognition gets its own provider identity rather than riding an OpenAI-compatible alias, with swe-1.6, swe-1.7 and swe-1.7-lightning priced

Provider: AWS Bedrock AgentCore
Supported LiteLLM Endpoints: /search
Description: Bedrock AgentCore registered as a search provider

Provider: Amazon Comprehend Medical
Supported LiteLLM Endpoints: /* passthrough
Description: Passthrough provider for Comprehend Medical, routed and credentialed through the proxy

Provider: Valkey
Supported LiteLLM Endpoints: Vector stores
Description: Valkey as a managed vector store provider alongside the existing backends

New LLM API Endpoints (4 new endpoints)

Endpoint: /model/deprecations
Method: GET
Description: Lists deployments whose models carry a provider-announced deprecation_date, backing proactive deprecation alerts
Documentation: Proxy Config

Endpoint: /team/daily/activity/aggregated
Method: GET
Description: Pre-aggregated team activity for the Usage tab, replacing the per-row scan it used to do client-side
Documentation: Cost Tracking

Endpoint: /team/{team_id}/callback/{callback_name}
Method: DELETE
Description: Removes a single team-scoped logging callback without rewriting the whole callback set
Documentation: Team Logging

Endpoint: /auto_router/validate_complexity_router_config
Method: POST
Description: Dry-runs a complexity-router config against the write gate so an invalid tier set is rejected before it is saved
Documentation: Auto Router

New Models / Updated Models

New Model Support (136 new models)

[Table of providers, models, context windows, pricing, and features including OpenAI, Azure, Amazon Bedrock, Databricks, Google Gemini, Fireworks AI, Mistral, Moonshot, OpenRouter, Perplexity, Voyage, SCX.ai, Cognition, fal.ai, and others with detailed pricing and capabilities]

The fal.ai entry expands to 57 registry keys: the two base slugs plus every low/medium/high quality crossed with six output resolutions, for both generation and /edit. The Fireworks entries each carry an accounts/fireworks/models/ alias, and the two glm-5p2-fast and two kimi-k3 fast/US variants also carry an accounts/fireworks/routers/ alias, so the slug you already use keeps resolving.

Beyond the new entries, this release is another large cost-map maintenance pass over 288 existing entries. 219 entries gain or correct a provider-announced deprecation_date, 25 gain Batch API input and output rates alongside a regional_endpoint_uplift_multiplier, and 15 gain a prompt_cache_min_tokens floor, including 4096 on Gemini 3.5, 3.6 and 3.7 Flash and 3.1 Pro Preview. Pricing moves in both directions: Gemini 3.6 Flash halves to $0.75 / $3.75 per 1M on both Gemini and Vertex, gpt-5.6 and gpt-5.6-sol drop to $4.00 / $20.00 on a promotional cut, mistral/codestral-latest falls to $0.30 / $0.90 and Vertex DeepSeek V3.1 MaaS to $0.60 / $1.70, while gemini-3.1-flash-image doubles to $0.50 / $3.00 and mistral/mistral-small-latest rises to $0.15 / $0.60. The GPT-5.6 family's max input tokens is corrected from 1.05M to 922K across the Azure, US and EU entries. Flex and priority service-tier rates land on 8 and 5 entries respectively, thinking_always_on on 11, and supports_prompt_cache_breakpoint on 4. No pricing entries were removed.

Features

• Amazon Bedrock
  • Serve GPT-5.6 cross-region inference profiles on Bedrock runtime - PR #37821
  • Forward LiteLLM identity and metadata into Bedrock requestMetadata - PR #36861
  • Add a config toggle to disable agent-runtime pass-through - PR #37386
  • Day-0 pricing for Grok 4.6 on Bedrock - PR #37517
• Cognition
  • Give Cognition its own provider identity, then price swe-1.7 at the standard tier and add swe-1.7-lightning - PR #37743, PR #37763
• SCX.ai
  • Add SCX.ai as a JSON-configured OpenAI-compatible provider - PR #34752
• Mistral
  • Add zai-glm-5-2 and glm-5-2 pricing - PR #37110
• Perplexity
  • Add the Agent API third-party models - PR #37112
• Databricks
  • Add cost map entries for 14 newer Databricks models - PR #28501
• Moonshot
  • Add moonshot/kimi-k3 to the cost map - PR #37552, PR #37753
• Anthropic
  • Map cache_control_injection_points to the OpenAI prompt_cache_breakpoint on GPT-5.6+ targets - PR #37628
  • Map metadata.user_id to prompt_cache_key on the /v1/messages bridge - PR #37623
• General
  • Route /chat/completions through the Rust core for Anthropic and Bedrock - PR #37241
  • Surface TinyFish response headers and top-level response extras - PR #32448
  • Authenticate to Azure Postgres with Microsoft Entra ID tokens - PR #37663

Bug Fixes

• Amazon Bedrock
  • Preserve cache token usage when invocationMetrics replaces the usage block, and report provider thinking tokens instead of classifying them as text - PR #36878, PR #35998
  • Forward provider response headers on chat completions, degrade gracefully on malformed tool-call arguments, and validate file-content retrieval against the configured output bucket - PR #37003, PR #33842, PR #31435
  • Read batch usage by payload shape rather than provider name, and report uploaded size in the FileObject returned by managed batch uploads - PR #37078, PR #36392
• Anthropic
  • Emit tool_use content_block_start without awaiting the next chunk, and resolve the provider exactly once on /v1/messages - PR #37310, PR #37757
  • Fold guardrail-modified leading system rows into the top-level system param - PR #37231
  • Stop emitting empty thinking blocks on the Responses adapter, and preserve optional Responses tool properties - PR #36033, PR #36979
  • Log partial stream spend when a /v1/messages client disconnects mid-stream - PR #37558
  • Gate sampling params on /v1/messages the way /chat/completions does, omit thinking.type=disabled for always-on thinking models, and accept a bool thinking param instead of raising AttributeError - PR #37868, PR #37510, PR #37423
• Google Vertex AI
  • Apply the regional endpoint uplift to cost tracking, resolving the served location from the passthrough URL, optional_params, and the deployment for native /v1/messages calls - PR #37543
  • Only fall back to a placeholder thought signature on the first parallel function call - PR #37541
  • Convert messages to contents in Gemini count_tokens - PR #36981
• Google Gemini
  • Price Gemini 3.6 Flash at Google's introductory rates on every service tier, and correct gemini-3.1-flash-lite-image capabilities while deduping its entries - PR #37197, PR #36849
• Azure
  • Rename max_tokens to max_completion_tokens for gpt-5-chat deployments - PR #36857
  • Strip non-OpenAI-spec message fields before the request on Azure AI - PR #34445
• Fireworks AI
  • Skip the accounts/ rewrite for FW-* Foundry deployment ids - PR #37242
• SageMaker
  • Send the inference component header and honor hf_model_name - PR #37766
• General
  • Stop forwarding a client Anthropic OAuth token to Bedrock and Vertex - PR #37905
  • Stop the Rust flag and the client_side_timeout marker from leaking into upstream provider request bodies - PR #37218, PR #37346
  • Resolve the advisor sub-call through the proxy router - PR #36246
  • Surface provider errors from the container file content endpoint - PR #37737

[Further detailed sections on LLM API Endpoints, Management Endpoints/UI, AI Integrations, Logging, Guardrails, Prompt Management, Secret Managers, Spend Tracking, Budgets and Rate Limiting, MCP Gateway, Performance/Loadbalancing/Reliability improvements, Documentation Updates, PR roll-up by ownership area, End-to-End Testing, New Contributors, and Full Changelog with numerous PR references and detailed technical notes]

End-to-End Testing

We are investing heavily in end-to-end testing to cut regressions and make LiteLLM more stable release over release. Every version is exercised by a live suite that runs against a real deployed proxy and hits real provider endpoints, not mocks, so the behavior we validate is the behavior you get in production.

This window added 73 test-only pull requests, 14 of them touching the live e2e suite. The headline change is a record-and-replay transport seam: a fixture bundle format, canonical content-based match keys, query params and multipart form fields pinned as part of replay identity, and the seam moved down to the provider edge, so the non-streaming provider flows can be replayed deterministically while the live suite still runs against real providers. New permanent regressions pin prompt-cache, service-tier and cost-header billing, OpenAI passthrough routing and file-list isolation, the OpenAI WebSocket passthrough prefixes, and twelve previously closed issues.

The suite itself was hardened against the failures that were masking real signal: response-cache cross-talk, slow providers and single upstream blips are now tolerated deliberately rather than by luck, blind sleeps are replaced with deadline waits, and live suites were repointed off retired Gemini, Groq, Together AI and Vertex image models. On the ratchet side, six ruff rules now reject tests that cannot fail, pytest.raises(Exception) is banned outright, PT011/PT012/PT014/PT017/RUF043 are enforced so a broad raises block cannot pass on the wrong error, and F811 and F821 are enforced so a duplicate or undefined name cannot silently replace the first. Thirty test files stranded in a second mirror now actually run, tests/old_proxy_tests was retired, and the conftest save/restore inventory is frozen so it can only shrink; the global-state cleanup unwound 182 leaking writes out of the cost-calc suites alone.

New Contributors

@ChenluJi made their first contribution in PR #32448
@Sujithr07 made their first contribution in PR #33195
@MUSE-CODE-SPACE made their first contribution in PR #34067
@ayaangazali made their first contribution in PR #34445
@bhuvan2134686 made their first contribution in PR #34752
@shivijain2323 made their first contribution in PR #35110
@Scott-Wilson-ZocDoc made their first contribution in PR #36032
@LHMQ878 made their first contribution in PR #36151
@harryzhou2000 made their first contribution in PR #36355
@irosh-colombage-ZocDoc2 made their first contribution in PR #36482
@itaimodi made their first contribution in PR #36764
@brian5021 made their first contribution in PR #36878
@oneKn8 made their first contribution in PR #36968
@sailikhithk made their first contribution in PR #36981
@bruno-olivia made their first contribution in PR #37242
@longwind48 made their first contribution in PR #37821

Three fixes in this release reached the repository as maintainer-pushed copies so the full CI pipeline could run against them, with commit authorship preserved: PR #37867, PR #37868, and PR #37869 are @mihidumh's work, and PR #37219 is @marty-sullivan's.

Full Changelog

https://github.com/BerriAI/litellm/compare/v1.98.0...v1.99.0

Original source

BREAKING CHANGES

The Langfuse metadata blob is now sourced from the StandardLoggingPayload allowlist instead of raw request metadata. Roughly 20 fields no longer appear on the generation, measured live at 52 keys down to 38, so any saved Langfuse filter, dashboard, or alert built on model_group, model_info, deployment, deployment_model_name, model_group_alias, model_group_size, litellm_api_version, litellm_received_at, litellm_parent_otel_span, queue_time_seconds, attempted_retries, max_retries, agent_id, caller_tags, inherited_tags, global_max_parallel_requests, user_api_key, or the remaining user_api_key_* budget and permission-id fields will stop matching. model_group and the deployment id are still recoverable from hidden_params. Direct-SDK callers passing flat custom metadata must nest it, metadata={"metadata": {"my_key": "v"}}, and it arrives under requester_metadata; proxy callers are unaffected. debug_langfuse now emits caller scalars rather than the raw metadata dump. See PR #36744.

The Global Control Plane worker registry now requires an Enterprise license. A proxy configured with worker_registry and no valid LITELLM_LICENSE refuses to boot instead of silently running unlicensed. Set a valid license to keep the control plane, or remove the worker_registry key to start without it. See PR #36996.

A litellm_settings.callbacks entry whose dotted path names a class now fails config load. Those entries used to be accepted and silently inert: the proxy booted, served traffic, and never ran the hook. Point the entry at an instance or a function, for example custom_callbacks.proxy_handler_instance, to start again. See PR #36858.

Key Highlights

• Reserved capacity is billed as reserved capacity - a deployment can now carry ptu_count and cost_per_ptu_per_hour with an effective window, the daily rollup writes per-model flat cost by active hour, and per-token billing is switched off entirely on that deployment so a team paying for provisioned throughput is no longer charged twice for the same traffic. Attribution is opt-in behind an env var, and a price sent alongside PTU config is rejected with a 400.
• You can measure the auto-router before you adopt it - a shadow eval job samples a slice of one key's successful traffic, replays it through the auto-router in a detached task that never serves a response or adds latency, and has an LLM judge compare both answers blind with randomized A/B labels. Counts, status, judge spend, and win rates by tier and incumbent model are all derived at read time from one append-only row per sample, and the job also runs in reverse.
• Routing groups are callable models - model= now routes across the union of member deployments using the group's own strategy, group names appear in /v1/models so Claude Code and Codex discovery surface them, and they are grantable on keys and teams. The Create Group modal has promised this since day one.
• Every response can state its own cost breakdown - six x-litellm-response-cost-* headers ship next to the total, where input, cache read, cache creation, output, and tool usage sum exactly to the total and reasoning is a subset of output, so a platform team attributes spend per component with no local pricing table.
• TPM reservations follow declared output size - expected output tokens are now declarable per key, per team, and per model instead of one static floor for every tenant, so concurrent requests stop overrunning a team's TPM limit and teams whose models emit far less stop being throttled. No config means byte-identical behavior and no migration.
• The Admin UI's move off antd and Tremor took its largest step yet - 75 UI pull requests in this window carry the navbar, playground, guardrails, usage, cost tracking, models and endpoints, team and user surfaces, the log details drawer, the AI Hub, and much of the shared component library onto shadcn (base-vega) primitives. The migration is not finished; both libraries remain dependencies and still back parts of the dashboard.

New Providers and Endpoints

New Providers (1 new provider)

Provider: Nimble
Supported LiteLLM Endpoints: /search
Description: Nimble's Search API as the 18th native search provider, registered in the provider enum, config map, pricing map, and dashboard, priced at $0.005 per query.

New LLM API Endpoints (2 new endpoints)

Endpoint: /v1/indexes
Method: GET
Description: Admin-only listing of every registered vector store index, newest first, so indexes created through POST /v1/indexes can be audited.

Endpoint: /auto_router/shadow_eval/{start,stop,{job_id}}
Method: POST, GET
Description: Start, stop, and read a pre-adoption shadow eval job for an auto-router, returning derived counts, judge spend, latest error, and win rates.

New Models / Updated Models

New Model Support (52 new models)

[Table of providers, models, context windows, pricing, and features omitted for brevity]

Beyond the new entries, this release is a large cost-map maintenance pass: 270 existing entries gain or correct a provider-announced deprecation_date across Bedrock, Mistral, Cohere, Gemini, OpenAI, and xAI, and 85 gain a search_context_cost_per_query block. xAI pricing is corrected in both directions: the grok-4.20 reasoning and non-reasoning variants drop from $2.00 / $6.00 to $1.25 / $2.50 per 1M and their context window is corrected from 2M to 1M, while grok-code-fast rises from $0.20 / $1.50 to $1.00 / $2.00. The Bedrock Mantle openai.gpt-5.6-sol, -terra, and -luna entries move from a 272K to a 1M context window with matching above-272K tiers, gpt-5-pro max output rises from 128K to 272K, the DeepSeek V4 entries go from 8K to 393K max output, and the Groq llama-3.1-8b-instant, llama-3.3-70b-versatile, and gpt-oss entries are resynced with Groq's own docs. Native structured output is flagged on 17 entries, supports_tool_search on 43, and reasoning-effort flags on a further 29. No pricing entries were removed.

Features

• Fireworks AI
  • Support router slugs via the routers/ prefix - PR #34257
  • Translate NIM and vLLM extra params to Fireworks-native arguments - PR #35969
• Azure AI
  • Add Fireworks FW model pricing on Azure AI Foundry and Grok 4.3 metadata - PR #35613, PR #27932
• Google Gemini
  • Day-0 pricing for gemini-3.7-flash - PR #36792
• xAI
  • Day-0 pricing for grok-4.6 - PR #36805
• DashScope
  • Add the latest Model Studio models to the cost map - PR #36496
• Meta
  • Add meta/muse-spark-1.2 and its contributor tier - PR #36717
• OpenRouter
  • Add NVIDIA Nemotron 3.5 Lightning on OpenRouter and DeepInfra - PR #36696
• OpenAI
  • Enable xhigh reasoning support for the gpt-5.4-mini models - PR #26909

Bug Fixes

• Amazon Bedrock
  • Use the deployment's credentials for AWS requests, and resolve aliases in batch file records - PR #36160, PR #36159
  • Resolve the managed-batch output bucket on every path that reads it, and stop managed-batch litellm_params leaking to the provider - PR #37047, PR #37048
  • Drop toolSpec.strict for Claude Sonnet 5 on Converse - PR #33196
  • Add a text block to Converse user messages carrying documents - PR #36499
  • Send the tool-search beta header for Haiku 4.5 on Invoke /v1/messages - PR #36502
  • Preserve adaptive thinking effort through the /v1/messages bridge - PR #36507
  • Hoist custom.defer_loading before dropping custom on invoke tools - PR #36855
  • Reject the Anthropic server-side web_search tool with an actionable error instead of a provider failure - PR #36473
  • Enable native structured output for GLM 5 and DeepSeek V3.2, and advertise it on every Bedrock id - PR #35669, PR #36597
  • Give the Bedrock Mantle GPT-5.6 Sol, Terra, and Luna entries a 1M context window with long-context pricing - PR #36698
• Anthropic
  • Preserve midturn system corrections - PR #34290
  • Preserve speed=fast in usage for /v1/messages and pass-through - PR #36447
  • Flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5 - PR #35930
• Google Vertex AI
  • Translate /v1/embeddings batch rows to the Gemini embedding shape - PR #35092
• Azure AI
  • Recognize real Search doc endpoints so teams can read and write through passthrough - PR #36798
• Databricks
  • Surface provider usage, including prompt-cache counts, in streaming chunks - PR #36943
• NVIDIA NIM
  • Preserve image passages and stop sending top_k to /v1/ranking - PR #34177
• Groq
  • Sync the Groq registry with Groq's docs - PR #36664
• OpenAI
  • Return a length-truncated 200 when the output budget fits no token - PR #36859
• General
  • Ship boto3 with the base SDK so Bedrock works out of the box on pip install litellm - PR #36568
  • Let an explicit provider outrank a known OpenAI model name - PR #36800
  • Refresh deprecation dates, correct xAI pricing, add missing provider models, and correct DeepSeek V4 max output - PR #36403, PR #36538, PR #36788, PR #36925

LLM API Endpoints

• Anthropic /v1/messages
  • Serve an Anthropic-native /v1/models so Claude Code's gateway discovery populates its model picker - PR #35455
• Vector Stores
  • Admin-only GET /v1/indexes listing every registered vector store index - PR #36289

Bugs

• Responses API
  • Preserve Codex namespace tool calls - PR #32536
  • Initialize completed_response on the bridge streaming iterator - PR #35413
  • Map interaction step and turn history onto Responses API roles and content types - PR #36733
• Batches
  • Stop forwarding custom_llm_provider twice in list and cancel - PR #32813
• Managed files
  • Stop /{provider}/v1/files capturing /openai_passthrough - PR #36092
  • Scope file list pagination cursors to the caller, and report has_more false on caller-scoped pages - PR #36093, PR #36326
• Passthrough
  • Stop forwarding the client's Accept-Encoding upstream, which garbled Claude Code's /v1/models and count_tokens bodies on stock Docker images once Anthropic began brotli-compressing JSON - PR #37058
• Anthropic /v1/models
  • Always emit the token limits, null when unknown - PR #36961
• Web search
  • Stop leaking interception control fields to providers - PR #36480
• General
  • Make tool_result images visible to OpenAI-compatible providers - PR #34462
  • Stop a bare 429 in an error body outranking the status code in exception mapping - PR #36705

Management Endpoints / UI

Features

• Virtual Keys
  • Per-key prompt caching toggle via enable_prompt_caching, with an Admin UI switch on key create and edit - PR #36466
  • Add a config_updated_at audit timestamp - PR #36488
• Proxy CLI
  • Make the hidden lite command list configurable through ~/.litellm/config.json - PR #36816
• Auto-router screens
  • Shadow evals tab beside auto-router usage, with a direction picker and reverse-mode display - PR #36588, PR #36994
  • Deployment affinity toggle, and models shown under each tier in the routing benchmark chart - PR #36302, PR #36291
  • Highlight Auto Router in the navbar announcement - PR #36315
• Dashboard
  • Render request metrics on the /ui/chat surface, including provider prompt cache tokens in the chat response metrics - PR #36845, PR #36827
  • Show vector store indexes on the Vector Stores page - PR #36306
  • Surface PTU inputs on the model form and flat cost on the Usage page - PR #35393
  • Migrate the playground chat controls to shadcn - PR #36129
  • Warn in the Admin UI when no Redis is configured - PR #36495
  • Link user detail team names to their team pages - PR #37022

Bugs

• Auth & roles
  • Carry team grants in lite login session tokens so team-bound CLI users stop seeing the whole proxy on /v1/models - PR #36826
  • Expand config-defined model access groups when resolving team models for /v2/model/info - PR #34211
  • Treat SAML as configured in UI SSO detection - PR #36196
  • Restore management_v1 query-param validation under fastapi>=0.140.7 - PR #35773
• Teams & access groups
  • Sync assigned_team_ids and assigned_key_ids from the team and key write paths - PR #36825, PR #36843
  • Sweep dangling team references and cache on team delete - PR #36819
  • Resolve member_delete cleanup by user id rather than the addressed email - PR #36839
  • Stop the duplicate legacy invitation email and fix its onboarding link - PR #36455
• Proxy CLI
  • Launch agents as a child process on Windows - PR #36822
• Dashboard
  • Hide admin-only Logs tabs from roles that cannot call their endpoints, and restore the Deleted Teams tab for organization admins - PR #36333, PR #36478
  • Gate organization and agent usage views, policy and prompt lookups, the Old Usage page, and four sidebar pages behind the roles their endpoints allow - PR #36334, PR #36335, PR #36469, PR #36475
  • Scope the Virtual Keys and Logs team lists to the caller - PR #36472
  • Show and edit key-level router settings on a virtual key - PR #36674
  • Stop a deselected MCP server keeping its grant on a virtual key, and match the MCP servers count badge to its sibling permission badges - PR #36840, PR #36984
  • Restore playground model filtering by endpoint, keep mode: completion models in the chat dropdown, and distinguish hosted from local vLLM in the provider dropdown - PR #36130, PR #37954, PR #36974
  • Add NVIDIA Riva to the model provider list - PR #36769
  • Align the spend and budget columns, and rename the models table Status column to Source - PR #35176, PR #37021
  • Show zeroed auto-router usage stats when a window has no sessions, and open the classifier prompt editor above the edit auto-router form - PR #36868, PR #36438
  • Keep the cost tracking removal confirmation open until it settles, de-duplicate the reset budget option, stop the models tab strip scrolling vertically, and anchor chips-combobox popups to the field - PR #36960, PR #37010, PR #36993, PR #36995
• Dashboard internals: the shadcn migration
  • Page-level migrations off antd and Tremor: guardrails-monitor, projects and logs, cost-optimization, cost-tracking, admin-panel, team settings, users dashboard, prompts, models-and-endpoints, guardrails, usage, playground, and the AI Hub - PR #34606, PR #36629, PR #36631, PR #36635, PR #36641, PR #36642, PR #36643, PR #36648, PR #36832, PR #36834, PR #36838, PR #36847, PR #36908
  • Navbar, log details drawer, settings page and bulk user invite, key info and permissions views, router settings and shared badges, MCP permission panels, model hub and model select, shared dropdowns and selectors, root-level dashboard components, cost tracking components, shared common_components, and ten remaining small files - PR #36902, PR #36904, PR #36936, PR #36913, PR #36915, PR #36964, PR #36918, PR #36924, PR #36927, PR #36955, PR #36910, PR #36966
  • Log viewer internals: TokenFlow and JsonViewer, SimpleMessageBlock and SimpleToolCallBlock, HistoryTree and CollapsibleMessage, TruncatedValue and OutputCard, SectionHeader and ToolsSection, the guardrail and duration controls, search and user controls, team detail controls, and the policy impact popover - PR #36735, PR #36737, PR #36738, PR #36739, PR #36793, PR #36693, PR #36694, PR #36695, PR #36653
  • Move the usage, guardrails content, and guardrails monitor tables onto the shared DataTable, make illegal DataTable prop combinations unrepresentable, migrate the access group create modal to RHF and zod, re-sync badge and skeleton onto the base-vega style, and declare DateRangePickerValue locally instead of importing it from Tremor - PR #36707, PR #36708, PR #36709, PR #36470, PR #37033, PR #36991, PR #36962

AI Integrations

Logging

• Langfuse
  • Source the emitted metadata blob from the StandardLoggingPayload, so a team's own Langfuse credentials stop arriving inside that team's own traces - PR #36744
  • Restrict trace steering keys to real Langfuse trace fields, and coerce header-sourced mask and trace-update steering values - PR #36862, PR #36740
  • Emit the OTel trace version and release on the keys Langfuse v4 reads - PR #36702
• Arize
  • Trace MCP tool calls instead of crashing on CallToolResult - PR #36453
• Slack alerting
  • Dedupe scheduled Slack spend reports across pods - PR #36489
• General
  • Opt-in session_id and trace_id correlation on JSON log records via contextvars - PR #34418
  • Fail config load when a callbacks entry is not dispatchable, instead of booting with a silently inert hook - PR #36858
  • Log requests rejected for an unparsable body in spend logs - PR #36673

Guardrails

• Palo Alto Networks Prisma AIRS
  • Return the full scan response on blocked requests, surface scan_id on allowed requests, and scan tool call arguments as plain text rather than a tool event - PR #37036, PR #37037, PR #37038
• Bedrock Guardrails
  • Skip ApplyGuardrail when there is no content to scan, so a tool-only turn stops failing the whole request - PR #36441
  • Scan and re-emit raw Anthropic SSE streams in the post-call hook - PR #36598
• General
  • Isolate guardrail load failures per row so one bad entry stops taking the rest with it - PR #36432
  • Report real token usage on guardrail-blocked /v1/responses replies - PR #36907

Spend Tracking, Budgets and Rate Limiting

• Provisioned throughput (PTU)
  • Configure ptu_count, cost_per_ptu_per_hour, and an effective window on a model deployment - PR #35341
  • Write per-model PTU flat cost by active hour in the daily rollup, and surface it on the daily activity read path - PR #35343, PR #35391
  • Gate PTU flat-cost attribution behind an opt-in env var - PR #36138
  • Stop per-token billing, and stop billing for grounded search, on a PTU-configured deployment - PR #36829, PR #37043
• Rate limiting
  • Declare expected output tokens per key, team, and model so TPM reservation stops using one static floor - PR #36143
  • Reserve the larger declared output budget for TPM limits - PR #37001
• Cost tracking
  • Emit six per-component x-litellm-response-cost-* headers next to the total - PR #36965
  • Track OpenAI and Azure web search tool cost per call, and bill xAI web search from server_side_tool_usage_details - PR #35286, PR #30817
  • Support cache creation cost in tiered pricing, all-or-nothing - PR #36720
  • Net prompt-caching savings against the cache-write premium - PR #36452
  • Apply the Anthropic regional geo uplift to cached tokens, and aggregate the 5m and 1h cache-write split across the iterations path - PR #34850, PR #34860
  • Price dict-shaped image input token details at the image rate - PR #33490
  • Stop a zero output rate from zeroing transcription cost - PR #36914
  • Forward resolved provider and deployment pricing in /cost/estimate - PR #35880
  • Never price a strategy-router alias, and stop get_router_model_info wiping cached pricing - PR #36691, PR #36985
• Passthrough and batches
  • Track spend for OpenAI passthrough /v1/embeddings, inject streaming usage cost on OpenAI passthrough streams, and track streamed passthrough Responses cost - PR #36660, PR #36503, PR #36529
  • Carry the budget reservation into passthrough request metadata, so successful requests stop leaking reservations into Redis and tripping false BudgetExceededError - PR #36592
  • Attribute Vertex and Anthropic passthrough batch cost to the creating key, team, and tags - PR #34456, PR #36468
  • Account a managed batch's cost exactly once, give a batch's cost row a primary key of its own, mark a terminal batch with no output file as processed, and stop uncostable batches starving the cost poll page - PR #37050, PR #36876, PR #35360, PR #36714
  • Strip NUL bytes from passthrough batch tags before the managed object write - PR #36688
• Spend logs and budgets
  • Requeue Redis spend buffer transactions when the DB commit fails, requeue spend logs on a transport error, and stop losing rows when a flush is cancelled - PR #33881, PR #36716, PR #34826
  • Atomic budget cascade with chunked reset scans in the reset budget job - PR #36287
  • Honor an explicit null budget_duration on team and key create, with clearable UI dropdowns - PR #36699
  • Tolerate a concurrent creator when creating spend views - PR #36824

MCP Gateway

• Scope gateway session bearers to the RFC 8707 resource - PR #35045
• Serve the aggregate MCP endpoint on bare /mcp instead of a 307 redirect - PR #34845
• Resolve admin OAuth sessions from any worker via DB-backed drafts, and keep admin-entered OAuth endpoints in management reads - PR #36844, PR #36888
• Bound MCP client requests with a session read timeout - PR #36675
• Expose client HTTP headers to logging callbacks and hooks - PR #36724
• Drop the caller host and configured upstream headers from logged metadata - PR #36901

Performance / Loadbalancing / Reliability improvements

• Router & auto-router
  • Make routing groups callable as virtual models and list them in /v1/models - PR #36519
  • Pre-adoption shadow eval for the auto-router with a blind pairwise judge, extended to sample /v1/messages and /v1/responses traffic and to run reverse-direction jobs - PR #36587, PR #36830, PR #36865
  • Calibrate the complexity classifier rubric with worked examples, selectable per router, and stop scoring system prompt text for code and technical complexity - PR #36578, PR #36721
  • Add a required-AND (&) tag prefix and an allow_fail_open flag, let untagged requests bypass a tagged pre-routing strategy on shared model names, and stop re-applying router-selecting request tags to the routed tier's deployments - PR #36193, PR #36627, PR #36628
  • Forward auto-router alias params from the marker entry rather than the first same-name deployment - PR #36626
  • Per-deployment allowed_fails_policy and cooldown_time overrides, cooldown for failed fallback deployments, and a corrected cooldown TTL after Redis backfill - PR #34416, PR #35104
  • Keep batch fallbacks inside the model group that owns the file - PR #36181
  • Warn when a deployment's credentials contradict its provider - PR #36486
• Streaming & connections
  • Global litellm_settings.sse_keepalive_ping_interval_seconds, on top of the per-deployment keepalive_seconds SSE heartbeat, so silent streams stop being killed by ingress idle timeouts - PR #36154, PR #34423
  • Refactor HTTP handler initialization with client support - PR #30952
  • Cache Anthropic /v1/messages responses, including streaming - PR #34581
  • Unwrap decorated __init__s when deriving the Redis from_url kwargs allowlist - PR #36654
• Database & background jobs
  • Write each daily spend batch in one upsert statement - PR #36448
  • Bound spend-logs retention cleanup so one run cannot saturate the database - PR #36594
  • Stagger scheduled background jobs across jobs and pods - PR #36589
  • Force a Prisma recreate on a Postgres cached-plan error - PR #36428
  • Serialize model reconciles so concurrent model writes stop evicting each other - PR #36687
  • Skip prisma-dependent hooks when no database is attached - PR #36273
• Deployment
  • Make VPC, Aurora, and Redis optional in the AWS Terraform module, so a locked-down account can reuse its own networking, Postgres, and Redis - PR #36676
  • Add startupProbe and hpa.behavior knobs to the componentized Helm chart - PR #36382

Documentation Updates

• Describe the Terraform provider release as automatic - PR #36467
• Require a user flow and a live-proxy proof in bug reports, and a user flow plus a stuck-at proof in feature requests - PR #36498, PR #36500
• Show only the latest run as Before/After in the proof-of-fix section, with nested cases - PR #37063
• Replace the Changes PR template section with Caveats - PR #36423
• Rewrite the CLAUDE.md comment rule with explicit any-of exceptions, require ReadOnly on every TypedDict field, tell agents to let heavy gates queue for machine-wide slots, and drop the @ prefix from the PR template path - PR #36301, PR #36421, PR #37005, PR #37057, PR #36726

PR roll-up by ownership area

PRs by ownership area (total: 277)
UI: 75
Models & Providers: 39
Spend / Budgets / Rate Limits: 37
Other (CI / chore / tests / build / version bumps): 36
Performance: 27
Auth & Management: 16
LLM API Endpoints: 14
Docs: 10
Logging: 9
Guardrails: 7
MCP: 7

End-to-End Testing

We are investing heavily in end-to-end testing to cut regressions and make LiteLLM more stable release over release. Every version is exercised by a live suite that runs against a real deployed proxy and hits real provider endpoints, not mocks, so the behavior we validate is the behavior you get in production.

This window added 18 test-only pull requests, 10 of them touching the live e2e suite. Vendor API coverage was hardened so that provider denials, disconnects, server errors, and missing credentials hard-fail instead of passing, and new cells cover Google-native generateContent framing, Prometheus queue time, the Anthropic web_search server tool on Bedrock, and the model allow-list's positive case rather than only its denials. Admin UI e2e tests no longer stop at a success toast: every mutating flow now asserts the outgoing request body and reads the resource back from the API, with new coverage for Logs, Playground, Usage, and MCP edit and delete. Three reproducers for open gateway gaps in passthrough headers and per-model budgets are checked in skipped, with the product gap named in each skip reason, so removing the skip becomes the regression test once a fix lands. On the maintenance side, OTel assertions now target the attempt that served the stream rather than the span count, live Bedrock tests were repointed off the retired Claude 3 Sonnet, and the stale antd selectors that were failing the UI suite on every run were repointed at what the migrated dashboard actually renders.

New Contributors

@Praveen11558 made their first contribution in PR #30952
@geraint0923 made their first contribution in PR #30817
@dcadenas made their first contribution in PR #32536
@anxkhn made their first contribution in PR #32813
@kr0k made their first contribution in PR #33196
@vairodp made their first contribution in PR #33490
@atomic made their first contribution in PR #34177
@heathriel made their first contribution in PR #34257
@eugene-yao-zocdoc made their first contribution in PR #34290
@alexshtf made their first contribution in PR #35669
@HuanQian571 made their first contribution in PR #35773
@milesadkins made their first contribution in PR #35969
@daleselaji-dev made their first contribution in PR #36160
@fancybear-dev made their first contribution in PR #36196
@ilchemla made their first contribution in PR #36347
@Louis-Vauterin made their first contribution in PR #36382
@william-xue made their first contribution in PR #36529
@lostmartian made their first contribution in PR #36660
@FahimaGold made their first contribution in PR #36705
@guptaishaan made their first contribution in PR #36907
@pokepoke81 made their first contribution in PR #36943
@erensh27 made their first contribution in PR #36965

Full Changelog

https://github.com/BerriAI/litellm/compare/v1.97.0...v1.98.0

Original source

BREAKING CHANGES

Request-parameter checks now apply to path and form inputs, not only the request body. A request that supplies a deployment name through the URL path, or metadata through bracket-notation form fields, is now subject to the same destination check the body already went through, so calls that previously slipped past it will start being rejected. Configured credentials also stay with the endpoint they belong to, so a connection test can no longer borrow another model's credentials. Three admin opt-outs are preserved for legitimate overrides. See PR #36011.

Auto-router deployment_affinity now defaults to on. An auto-router whose callers send a session id will pin that session to one deployment inside the routed model group instead of spreading its turns across every deployment. Set deployment_affinity: false on the auto-router config to restore the previous behavior. See PR #36146.

Key Highlights

• Guardrails can be pointed at tool results alone - a new per-guardrail scan_only_tool_results flag scans and masks tool output while system, user, and assistant content pass through untouched, so an agent platform can keep injection detection on untrusted tool results without its own harness prompts tripping the filter. Works on both /v1/messages and /v1/chat/completions
• The auto-router now sticks to a deployment, not just a model group - deployment_affinity is on by default, so a conversation returning to a model group lands on the deployment it used there before and the provider prompt cache stays warm, while every turn is still classified on its own merits. session_affinity implies it, and session pins are now scoped by the caller's hashed API key
• Successful and failed request counts come from the gateway, not the spend logs - a new LiteLLM_DailyGatewayRequests table written by the ASGI request-metrics middleware backs the Usage tiles, so the number no longer drops to zero when spend logging is off or the database is unavailable, and a by-endpoint breakdown chart the SpendLogs path could never produce comes with it
• Proxy admin viewers can finally see the proxy - about fifteen read endpoints that compared against PROXY_ADMIN exactly now use the viewer-inclusive check, and the UI presents a viewer as an admin for gating purposes while the server still rejects every write
• Keys, users, teams, and organizations can pull their own spend report - four new caller-scoped spend/report endpoints auto-scope a non-admin caller to their own identity, return 403 on a mismatched scope override, and cap date ranges at 366 days
• Managed files and batches get a correctness sweep - unified output file ids are now derived deterministically so concurrent registrations converge, are returned from GET /batches, from unscoped file listing, and on cancel, and unparseable rows no longer take down a listing
• Admins can broadcast a banner to the dashboard - GET /get/user_banner and PATCH /update/user_banner back a dismissible markdown banner rendered on every page, published from Admin Settings with a live preview and stored in the existing LiteLLM_UISettings table with no migration

New Models / Updated Models

New Model Support (2 new models)

Provider: Google Gemini

Models:

• gemini/gemini-robotics-er-2-preview, Context Window: 131K, Input: $2.00/1M tokens, Output: $10.00/1M tokens, Features: Reasoning, vision, audio input, video input, PDF input, function calling, tool choice, prompt caching, response schema, web search, URL context
• gemini/gemini-robotics-er-1.6-preview, Context Window: 131K, Input: $1.00/1M tokens, Output: $5.00/1M tokens, Features: Reasoning, vision, audio input, video input, PDF input, function calling, tool choice, prompt caching, response schema, web search, URL context

Beyond the new entries, this release carries OpenAI's GPT-5.6 price cut onto the Azure cost map: azure/gpt-5.6-terra falls from $2.50 / $15.00 to $2.00 / $12.00 per 1M and azure/gpt-5.6-luna from $1.00 / $6.00 to $0.20 / $1.20 per 1M, with the same reductions applied to the regional azure/us/* and azure/eu/* variants and to every cache, flex, priority, and above-272K tier. Flex and priority tier keys are added to the dated OpenAI snapshot variants that were missing them (gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14, gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4o-mini-2024-07-18, gpt-5-nano-2025-08-07, o3-2025-04-16, o4-mini-2025-04-16). The Groq gpt-oss models gain a web search context price of $0.005 per query, the Bedrock claude-sonnet-5 entries are marked as not supporting Converse strict tools, and a mangled replicateopenai/gpt-oss-20b key is corrected to replicate/openai/gpt-oss-20b. No pricing entries were removed.

Features

• Google Gemini: Add gemini-robotics-er-2-preview and gemini-robotics-er-1.6-preview to the model cost map - PR #35555

Bug Fixes

• Amazon Bedrock:
  • Include the batch and S3 fields plus the model in deployment credentials - PR #24548
  • Grant bedrock:CountTokens in the OIDC session policy - PR #33145
  • Pass the SSE-KMS key through to the batch input-file S3 upload - PR #35148
  • Normalize /v1/completions and /v1/responses batch records - PR #35675
  • Stop forwarding a no-op toolSpec.strict to Converse - PR #35688
  • Drop a conflicting tool_choice.type when toolConfig.toolChoice is already set - PR #35738
  • Sign managed-file S3 requests with S3SigV4Auth - PR #35983
• Anthropic:
  • Stop indexing choices[0] on choiceless streaming chunks in the adapter - PR #35314
  • Coerce an explicit additionalProperties to false in the output_format schema - PR #35811
• Google Vertex AI:
  • Surface the real error and status on batch create instead of an IndexError 500 - PR #35141
• OpenAI:
  • Apply OpenAI's gpt-5.6 terra and luna price cut to the Azure cost map - PR #35481
• Groq:
  • Translate web_search_options to the browser_search tool - PR #34971
• Replicate:
  • Correct the mangled gpt-oss-20b model key - PR #34800
• AI21:
  • Resolve the documented AI21_API_KEY instead of a misspelled name - PR #35985
• Jina AI:
  • Resolve the documented JINA_API_KEY as a fallback - PR #35992
• General:
  • Rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion - PR #36010

LLM API Endpoints Features

• Anthropic /v1/messages: Send keepalive pings on SSE streams during upstream silence - PR #36024
• Cursor:
  • Make /cursor/chat/completions work with Cursor agent mode - PR #34029
  • Resolve Cursor thinking and fast model-name suffixes - PR #35554
• Claude Code:
  • Create-only skill registration with a PUT update route - PR #31752

Bugs

• Responses API:
  • Forward client headers to the provider on /v1/responses - PR #34531
  • Forward allowed_openai_params through the chat completions bridge - PR #35885
• Batches:
  • Register managed batch output files on terminal retrieve - PR #34092
  • Account for Responses API usage - PR #35367
  • Prevent managed file fallbacks across providers - PR #35371
  • Register managed output files on batch cancel, and persist managed file ids for cancelled, failed, and expired batches - PR #36034, PR #36048
• Managed files:
  • Skip rows without file objects, and skip unparseable rows when listing - PR #35365, PR #36021
  • Enforce require_managed_files on every route that accepts a raw provider id - PR #35551
  • Derive unified output file ids deterministically so concurrent registrations converge - PR #36019
  • Return unified ids from unscoped file listing and unified output file ids from GET /batches - PR #36031, PR #36049
• Passthrough:
  • Resolve pass-through credentials live from router deployments - PR #35916
  • Stop forwarding the client's Accept-Encoding upstream - PR #37058
• Web search:
  • Restore snippet text in native web_search_tool_result blocks - PR #36228
• General:
  • Map a generic error finish reason to stop - PR #33972

Management Endpoints / UI Features

• Spend reports:
  • Caller-scoped key, user, team, and organization spend/report endpoints - PR #35725
• Teams & Users:
  • Custom metadata validation hook for team create and update - PR #33353
  • Apply the default organization to new teams from default team settings - PR #35540
• Dashboard:
  • Admin-configurable, dismissible markdown user banner - PR #35729
  • Role capability gating, with the Tool Policies route migrated onto it - PR #35812
  • Non-streaming response toggle in the playground - PR #35560
  • Show the user email or alias in the usage data export - PR #36232
• Auto-router screens:
  • Show auto-router savings and add an auto-router usage tab on the cost-optimization dashboard - PR #35522, PR #35995
  • Reorder Add Auto Router into name plus template with a collapsible detailed config - PR #35746
  • Add Test Routing to the auto router create form - PR #35859
  • Match preset models against deployments' underlying model IDs - PR #35972

Bugs

• Auth & roles:
  • Give proxy_admin_viewer read parity with proxy_admin across roughly fifteen read endpoints and the UI gating that shadowed them - PR #35851
  • Apply request-parameter checks consistently across body, path, and form inputs - PR #36011
  • Propagate user_email and bind api_key on JWT auth attribution paths, and backfill null user_email on existing users - PR #34331, PR #34588
  • Name enable_jwt_auth when a JWT-shaped key is rejected - PR #35831
  • Return 403 from the OAuth2 enterprise gate - PR #35838
  • Re-assert the authenticated identity on passthrough requests - PR #36121
  • Stop resolving the UI session sentinel team on /search_tools/list - PR #36061
  • Let non-admins reach /user/daily/activity/aggregated - PR #36062
• Keys & credentials:
  • Apply key_alias and key_hash filters to all /key/list visibility branches - PR #35840
  • Return the real status code when a credential update is rejected - PR #36166
• Agents & access groups:
  • Derive config agent ids from agent_name so grants survive secret rotation - PR #36020
  • Deny agent access when key and team grants resolve to nothing - PR #36221
  • Resolve entity access groups in the model listing endpoints - PR #36230
• Config & projects:
  • Apply key and team router_settings.model_group_alias - PR #35486
  • Let a YAML store_prompts_in_spend_logs take precedence over the DB cached value - PR #35769
  • Persist the periodic reload schedule so status survives restarts and fires without store_model_in_db - PR #35165
  • Invalidate the cached project object on project update and delete - PR #36028
• Dashboard:
  • Hide guardrail review buttons from non-admin users, and block the Playground page for viewer roles on direct URL access - PR #27535, PR #35676
  • Render the Responses API request and response in the logs drawer - PR #35718
  • Push ?project= when opening a project, link project keys to their virtual key detail, and sync the projects list page index to ?page= - PR #36001, PR #36002, PR #36003
  • Allow clearing a key's budget reset from the Edit Key form - PR #36140
  • Let access groups be a team's only model source, with hover provenance - PR #36234
  • Show team BYOK models in team fallback settings - PR #36241
  • Reject an auto-router keyword rule left empty instead of dropping it - PR #35705
  • Match auto-router preset models against wildcard-expanded model groups - PR #36111
  • Correct the expired-miss share to run over all measured turns, and fix the cost-optimization tab labels - PR #36037
  • Update the Anthropic model presets - PR #35896
  • Note Google's Agent Platform rename in the vector store setup copy - PR #28076
• Dashboard internals:
  • Replace hand-rolled query-param routing with nuqs - PR #35871
  • Inject the fetch client's base url instead of reading it at import - PR #35802
  • Route MCP session tokens through the shared storage helper - PR #35835
  • Extract the MCP create form's logic and field groups, and rename the create MCP server component to PascalCase - PR #35694, PR #35686

AI Integrations Logging

• OpenTelemetry:
  • Mark v2 server spans as failed for pre-call errors - PR #34546
  • Stamp service tier attributes on inference spans - PR #35679
  • Name the RPC system and upstream on MCP tool-call spans - PR #35857
• Arize Phoenix:
  • Lowercase the OTLP/gRPC auth metadata key - PR #34883
• DataDog:
  • Read team callback dd_* params from kwargs instead of blocked dynamic params - PR #35687
• Langfuse:
  • Stop a collected httpx handler from closing a shared client - PR #35981
• s3:
  • Sign S3 object URLs with S3SigV4Auth so encoded paths verify - PR #35726
• Azure Storage:
  • Honor AZURE_STORAGE_ENDPOINT_SUFFIX for sovereign clouds - PR #35806
• Azure Sentinel:
  • Respect AZURE_AUTHORITY_HOST for the Entra token and derive the Azure Monitor audience per cloud, with AZURE_SENTINEL_AUTHORITY_HOST as a scoped override - PR #36137, PR #36165
• General:
  • Actually stop logging when a team callback calls disable_logging - PR #35520
  • Redact credential headers from request logging copies, and extend secret redaction to records litellm does not emit directly - PR #35678, PR #35977
  • Promote caller metadata trace fields into litellm_metadata, and fall back to litellm_metadata when metadata is empty - PR #35866, PR #36105

Guardrails

• General:
  • New scan_only_tool_results flag scopes a unified guardrail to tool results - PR #36014
  • Scan /v1/messages tool traffic - PR #35999
  • Scan model output on the /openai/v1/responses alias - PR #35818
  • Let litellm_content_filter run on post_mcp_call - PR #35980
• Bedrock Guardrails:
  • Chunk oversized ApplyGuardrail requests instead of failing - PR #36119
• Rubrik:
  • Prompt moderation, response-text blocking, a streaming buffer, and failure logging - PR #35722
  • Attribute blocked requests to the caller that made them - PR #35734
• Zscaler AI Guard:
  • Honor the configured timeout - PR #36110
  • Return the right HTTP code when input is blocked - PR #31948
• Compresr / Headroom:
  • Improve the /v1/compress HTTP 404 diagnostics - PR #35952

Spend Tracking, Budgets and Rate Limiting

• Gateway request accounting:
  • Make the gateway middleware the source of truth for successful requests, with a new by-endpoint breakdown - PR #35717
• Reporting:
  • Caller-scoped key, user, team, and organization spend report endpoints - PR #35725
  • Include today's UTC bucket when a daily activity range ends at the caller's current day - PR #36051
• Auto-router savings:
  • Add net auto-router savings to the cost-optimization dashboard and derive a default baseline from the hardest tier - PR #35521, PR #35907
  • Rebuild the auto-router benchmarks backend as a per-session rollup, and track turns per complexity tier - PR #35910, PR #36209
  • Report the LLM classifier cost per request via routing_decision and an x-litellm-classifier-cost header - PR #36015
• Budgets:
  • New opt-in apply_user_budget_to_team_keys, default off, reaching all three personal-budget gates - PR #36102
  • Enforce per-model budgets against resolved Cursor model variants - PR #35834
  • Warn at startup when max_budget is set but no database is connected - PR #36041
• Cost tracking:
  • Read what a request cost from the record instead of pricing it again - PR #35736
  • Bill gpt-5.6 prompt cache reads at the cache read rate, and keep OpenAI prompt cache token details through usage reassembly - PR #34957, PR #34812
  • Bill reasoning tokens at the service tier output rate, and sync flex and priority tier keys to dated OpenAI snapshot variants - PR #35925, PR #35923
  • Bill intercepted web searches to the calling key - PR #35708
  • Stop token-pricing the placeholder input on file content calls - PR #35140
  • Track cost for managed batches with no attributable key or user - PR #35468
  • Fetch background responses through the router in CheckResponsesCost - PR #35137

Performance / Loadbalancing / Reliability improvements

• Router & auto-router:
  • Independent, default-on deployment_affinity for the auto-router - PR #36146
  • Let operators replace the LLM classifier's system prompt and rename the four complexity tiers - PR #35855, PR #35893
  • Default session affinity off and expose it in the UI - PR #35714
  • Make the reminder marker pair configurable, and accept every pair a harness emits - PR #35874, PR #36029
  • Match CJK keyword_tier_rules that regex word boundaries miss - PR #35984
  • Stop the embedding model's context window from failing long requests - PR #35956
  • Bound fallback-walk work and error-log volume - PR #36148
  • Redact fallback tracebacks at the call site and cover the sync deferred stream - PR #35843
  • Eagerly fetch the Vertex AI deferred stream so HTTP errors surface in the _acompletion fallback path - PR #34627
  • Keep custom model_info across a price data reload - PR #35491
• Connections & caching:
  • Self-heal handler clients closed after cache eviction, then re-land closing of evicted LLM clients on top of it - PR #35862, PR #35870
  • Stop pooled clients persisting cookies on the aiohttp jar - PR #36149
  • Stop writing per-caller state onto the shared cached A2A httpx client - PR #35978
  • Install hiredis so redis-py parses replies with its C parser - PR #35709
• Throughput:
  • Build log messages lazily so filtered-out log records cost nothing - PR #35703
  • Assemble streamed tool-call arguments in linear time - PR #35826
• Database & startup:
  • Only treat a recoverable database outage as grounds to serve without one, and keep the connected DB client when a startup health check fails - PR #35864, PR #35837
  • Stop alerting on health probes that lose the planned engine-restart race - PR #36141
  • Retry the model cost map fetch with Retry-After-aware backoff, keep the current map on reload failure, and log that failure lazily - PR #35739, PR #35750
• Migrations & images:
  • Recover from an interrupted Prisma toolchain install, and keep the heal from raising on an unreadable nodeenv cache - PR #35832, PR #35986
  • Bake the prisma engines at a world-readable path in the pip image and at /opt/prisma in the componentized images so any uid can start - PR #35976, PR #35989
• Dependencies & maintenance:
  • Upgrade cryptography to 50.0.0 - PR #35803
  • Bump gitpython to 3.1.58, h2 to 4.4.1, js-yaml to 4.3.1, nanoid to 3.3.17, brace-expansion, and postcss - PR #36212, PR #36147, PR #36227, PR #35692
  • Move the Admin UI toolchain to Node 24 - PR #35801
  • Sync the Terraform provider 0.3.0 from the mirror and cut 0.4.0 - PR #36098

Documentation Updates

• Document the /key/info fields and clarify that budget_reset_at is the next reset - PR #36127
• Replace the classic Helm chart's 128Mi resource example with the documented 4Gi sizing - PR #35830
• Add a User Flow section with authoring instructions to the PR template - PR #36162
• Cap all GitHub comments at 15-25 words and curb semicolon splices in the contributor guide - PR #36059
• Prefer commas over semicolons when replacing em dashes, and clarify guideline priority ordering - PR #35825, PR #36296

PR roll-up by ownership area

PRs by ownership area (total: 253)

• Other (CI / chore / tests / build / version bumps): 86
• Performance: 30
• UI: 28
• LLM API Endpoints: 21
• Auth & Management: 20
• Spend / Budgets / Rate Limits: 20
• Models & Providers: 17
• Logging: 15
• Guardrails: 10
• Docs: 6

End-to-End Testing

We are investing heavily in end-to-end testing to cut regressions and make LiteLLM more stable release over release. Every version is exercised by a live suite that runs against a real deployed proxy and hits real provider endpoints, not mocks, so the behavior we validate is the behavior you get in production.

This window added 18 test-only PRs, 11 of them touching the live e2e suite. New and repaired coverage lands on the legacy text /completions endpoint, control-plane writes settled across every replica rather than just one, provider-transient statuses retried at the transport with bounded backoff, and the UI suite self-seeding its own password-login users in global setup. Load and performance testing moved out of the main suite so a slow lane no longer gates correctness, and the vendor API strategy coverage added mid-window was reverted after it proved unstable. View-backed global spend probes are parked behind LIT-5211 rather than left flaking.

New Contributors

• @hMED22 made their first contribution in PR #34971
• @AkashNaickar made their first contribution in PR #34800
• @Souravrajvi0 made their first contribution in PR #34092
• @rimysore made their first contribution in PR #35367
• @elinacse made their first contribution in PR #35468
• @aayush598 made their first contribution in PR #35952

Full Changelog

https://github.com/BerriAI/litellm/compare/v1.96.0...v1.97.0

Original source

BREAKING CHANGES

Mock testing request params are gated behind one config flag. Six mock_* request params previously had three different behaviors, and a dropped param returned a normal success, so a fallback drill could pass without ever running. All six now reject with a 400 naming the params and the key unless an admin sets general_settings.dangerously_allow_mock_testing_request_params: true in config.yaml; the flag cannot be changed from the Admin UI or the API. See PR #35423.

Keyless gateway OAuth now admits session bearers at any MCP scope. Session-bearer admission and RFC 9728 WWW-Authenticate challenges fire on per-server MCP URL paths, not only the aggregate /mcp/ scope, so a per-server path that previously fell through to a plain rejection now issues a challenge. See PR #34856.

Key Highlights

• MCP entitlements reach the person, not just the key - an internal user's object_permission now acts as an MCP entitlement level that intersects the key, team, agent, and org scopes, is read at both tools/list and tools/call time, is persisted by /user/new and /user/update, returned by /v2/user/info, and editable from the internal user page.
• Guardrails can finally see MCP tool results - a new post_mcp_call mode routes tool result text through the unified apply_guardrail seam, so a guardrail can mask values inside a result or reject it outright; previously a tool returning sensitive data bypassed every guardrail.
• Config changes propagate to every pod immediately - management writes publish an invalidation event on coordination Redis and every pod resyncs on receipt, debounced with jitter and capped at one resync per 10s, replacing the 30s poll lag on models, credentials, and settings. Without Redis both sides no-op and polling behavior is unchanged.
• OpenAI cuts GPT-5.6 prices - gpt-5.6-terra drops 20% and gpt-5.6-luna drops 80%, mirrored onto Bedrock Mantle, plus new flex long-context (above 272K) rates across the gpt-5.6 family and a correction to the advertised gpt-5.4-mini and gpt-5.4-nano context windows.
• The auto-router learns what it is actually routing - the complexity classifier now sees prior turns and assistant turns, rates what a short reply approves, closes its rubric on the window it was given, and records its tier decision, request body, and its own classifier calls in spend logs and the log drawer.
• Budgets become a first-class management surface - a generic /management/v1 list contract lands with GET /management/v1/budgets on top of it, and the budgets page gains sorting, filtering, and search.
• Operational hardening for large deployments - opt-in database_statement_timeout and database_lock_timeout, opt-in REPLICA IDENTITY FULL re-asserted after every migration, pod-hardening and migration-Job knobs on the componentized Helm chart, and an unreachable Redis that no longer blocks every request.

New Models / Updated Models

New Model Support (2 new models)

Provider: DashScope

Models:

• dashscope/qwen3.7-max: 991.8K context window, $2.50 input / $7.50 output per 1M tokens, features: Reasoning, function calling, tool choice, prompt caching, response schema
• dashscope/qwen3.7-plus: 991.8K context window, $0.40 (tiered) input / $1.60 (tiered) output per 1M tokens, features: Reasoning, vision, function calling, tool choice, prompt caching, response schema

Pricing note: dashscope/qwen3.7-plus is priced in two tiers: $0.40 / $1.60 per 1M up to 256K tokens, and $1.20 / $4.80 per 1M above 256K.

Beyond the new entries, this release applies OpenAI's price cut to the GPT-5.6 family: gpt-5.6-terra falls from $2.50 / $15.00 to $2.00 / $12.00 per 1M and gpt-5.6-luna from $1.00 / $6.00 to $0.20 / $1.20 per 1M, with the same reductions mirrored on bedrock_mantle/openai.gpt-5.6-terra and bedrock_mantle/openai.gpt-5.6-luna and applied consistently to the batch, flex, priority, and cache variants. Flex long-context rates above 272K tokens are added to gpt-5.6, gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna. The advertised context window on gpt-5.4-mini and gpt-5.4-nano is corrected from 1.05M to 272K on OpenAI, Azure, and Azure AI Foundry, and the Azure AI above-272K tiers are removed with it. Fireworks AI Kimi K2.5, K2.6, and K2.7 max output tokens drop from 262,144 to 32,768. No pricing entries were removed.

Features

• DashScope
  • Add qwen3.7-plus and qwen3.7-max to the model cost map - PR #35123

Bug Fixes

• OpenAI
  • Correct GPT-5.6 prices for OpenAI, Bedrock, and flex long context - PR #35270
  • Adjust gpt-5.6-terra and gpt-5.6-luna prices per OpenAI's published rates - PR #35258
  • Correct gpt-5.4-mini and gpt-5.4-nano token limits - PR #35182
• Google Vertex AI
  • Forward function_call id on Vertex Gemini 3+ tool turns - PR #34603
  • Stop sending duplicate thoughtSignature copies to Gemini - PR #35004
  • Skip context caching when the cached block ends on a model turn - PR #35172
• Anthropic
  • Split mixed stream chunks by payload kind - PR #35289
• Amazon Bedrock
  • Cache AssumeRole credentials per attributed identity - PR #35467
• Fireworks AI
  • Correct Kimi K2.5, K2.6, and K2.7 max output token limits - PR #35174

LLM API Endpoints

Features

• General
  • Request stream usage upstream by default and strip it from client streams - PR #35290

Bugs

• Anthropic /v1/messages
  • Open the first content block with the real upstream type so reasoning-first streams start with thinking - PR #34433
  • Translate stop_sequences and disabled thinking for non-Claude targets - PR #34589
• Responses API
  • Map all documented in-stream error codes to real HTTP statuses - PR #35307
• Batches
  • Encode the public model group on background-created output file ids - PR #35406
• Managed files
  • Source the Vertex AI managed-file read bucket and credentials from per-model litellm_params - PR #34847
• Passthrough
  • Decide Vertex rawPredict passthrough streaming from the request body - PR #34672
• A2A
  • Keep config-defined agents registered and accept the documented agents: key - PR #35163

Management Endpoints / UI

Features

• Management API
  • Generic list contract for /management/v1 entity lists - PR #35308
  • GET /management/v1/budgets - PR #35310
  • Let AI API keys read /model/info - PR #35473
• Virtual Keys & CLI
  • Identify a key by key_alias on /key/update - PR #34851
  • Read base_url from a persistent CLI config file - PR #35015
• Models + Endpoints
  • Give auto-routers their own tab - PR #35009
  • Expose classifier context window fields on the Auto-Router screens - PR #35315
  • Expose the assistant-turn classifier context switch on the Auto-Router screens - PR #35500
• Dashboard
  • Sorting, filtering, and search on the budgets page - PR #35309
  • Deep-link the team and organization detail pages via ?team= and ?org= query params, and link organization teams to their team detail pages - PR #35112, PR #35117, PR #35120
  • Shareable log links via a log_id query param on the logs page - PR #34879
  • Split failed requests into their own series on the cache dashboard - PR #34862
  • Show which log rows are the auto-router's own classifier calls - PR #35304

Bugs

• Teams & Users
  • Stop serving a stale team model allowlist after /team/update - PR #34266
  • Align team member add with existing user provisioning rules - PR #35435
  • Skip team model aliases that point at deleted deployments - PR #34993
  • Show pass-through route selections and match team id substrings in team search - PR #35319
  • Report API-registered callbacks from GET /team/{team_id}/callback - PR #35512
• Auth & SSO
  • Grant only /v1/messages routes to JWT teams by default rather than all Anthropic routes - PR #34222
  • Resolve a managed batch or file deployment model_id to a model name for team access checks - PR #32587
• SCIM
  • Stop provisioning nested group ids as internal users - PR #34997
• Model writes
  • Reject model writes that corrupt an auto-router pseudo-model - PR #34151
  • Report when a model write does not survive the post-write reload - PR #34861
  • Stop model writes 500ing on another pod's delete - PR #35400
• Config & credentials
  • Resolve named credentials on provider-only batch and files calls - PR #35028
  • Drop an unsupported prisma select kwarg from the tag-management key lookup and the tool-management team lookup - PR #35288, PR #35293
• Policy engine
  • Preserve config-defined policies across DB sync and expose them via the list APIs - PR #35263
• Dashboard
  • Let the internal user and organization forms save sub-cent budgets - PR #35302
  • Stop clamping the budgets Budget ID column at 15 characters - PR #35268
  • Keep the session view open when selecting a log inside it - PR #35399
  • Show public model names in usage breakdowns - PR #35107
  • Land general login on the keys dashboard and send MCP consent to /ui/connect - PR #35523
  • Point the navbar and sidebar logos at the dashboard home route - PR #35041
  • Size the Object Permissions card grid by container width - PR #35019
  • Nest the source object in the Claude Code marketplace settings snippet - PR #35322

AI Integrations

Logging

• OpenTelemetry
  • Make OTLP export work against Grafana Cloud - PR #35060
  • Cap tool-definition attributes so they cannot evict gen_ai.* from the LLM span - PR #34828
  • Label retrieval and agent metrics correctly and emit gen_ai.provider.name - PR #35151
  • Record the GenAI duration metric on failed requests - PR #35152
  • Add a Grafana dashboard for the OTel GenAI metrics to the cookbook - PR #35159
• Prometheus
  • Global exclude_metrics and exclude_labels options - PR #34201
  • Add a service_tier label to the latency and spend metrics - PR #34966
• s3
  • Support SSE-KMS encryption params on both S3 logging paths - PR #35291
• General
  • Bind litellm_metadata by reference in function_setup so guardrail information reaches spend logs - PR #35292

Guardrails

• General
  • Scan and mask MCP tool results via a new post_mcp_call mode - PR #35155
  • Serve config-defined guardrails from the list and info endpoints without a DB, with stable ids - PR #35259
  • Run post_call guardrails on /v1/messages streaming via unified guardrail translation - PR #35260
• Bedrock Guardrails
  • Run the Bedrock guardrail on MCP tool calls in during_mcp_call mode - PR #35149
• Compresr / Headroom
  • Stop compressing the turn the model must act on - PR #35294

Spend Tracking, Budgets and Rate Limiting

Budgets

• Stop enforcing user budgets on team keys, reverting the earlier hierarchy change - PR #35271
• Only enforce budgets on routes that can spend - PR #35274

Rate limiting

• Enforce token limits when the pre-call increment is zero - PR #35422
• Move the v3 limiter per-request stash off request metadata onto a ContextVar - PR #35278
• Keep the v3 limiter out of provider-facing metadata on Responses routes - PR #35207

Cost tracking

• Bill the fast service tier at the priority rate - PR #35320
• Calculate cost and usage for completed Vertex AI batches, and aggregate batch output cost, usage, and models in a single pass - PR #35186, PR #35205
• Stamp the provider on embedding cache-hit spend logs - PR #35282

MCP Gateway

Entitlements

• Enforce per-user MCP tool-call entitlements in the auth module - PR #35146
• Enforce tool entitlements on every MCP tool dispatch path - PR #35156
• Deny MCP access when a named entitlement cannot be read - PR #35160
• Recover the tool-name prefix boundary from registered prefixes - PR #34673

OAuth

• Extend the keyless gateway OAuth flow to per-server MCP URL paths - PR #34856
• Source the ID-JAG subject from the user's stored SSO assertion - PR #35147
• Manual authorization-code delivery for headless MCP clients - PR #34848

Servers & discovery

• Never write discovery results to the row, heal rows a release already stamped, and retry failed discovery with backoff - PR #34990
• Annotate connected-app reachability on the gateway connect page - PR #34867

Performance / Loadbalancing / Reliability improvements

Config propagation

• Push config sync to pods via Redis pub/sub instead of waiting on the 30s poll - PR #35436

Database

• Bound DB statement and lock time via general_settings - PR #35496
• Opt-in REPLICA IDENTITY FULL re-asserted after prisma migrations - PR #35267
• Bound each spend-log write statement by payload bytes - PR #34956

Router & auto-router

• Give the ComplexityRouter LLM classifier prior-turn context - PR #35185
• Let the classifier see assistant turns and rate what a short reply approves - PR #35471
• Drop the tier-rubric override and close the rubric on the window it was given - PR #35504
• Record why the auto-router picked a tier, capture the classifier request body, and mark the auto-router's own classifier calls in spend logs - PR #35016, PR #35164, PR #35300
• Honor request-level num_retries over a deployment's litellm_params value - PR #35483
• Serialize latency for non-chat responses in lowest-latency routing - PR #33290

Connections & caching

• Stop an unreachable Redis from blocking every request - PR #35273
• Dispose recycled aiohttp client sessions deterministically, and keep the keep-alive connector config when a session is rebuilt - PR #33428, PR #34962

Rust core

• Make litellm-core the callable messages() SDK and drop the ai-gateway handler - PR #35044

Deployment & Helm

• Pod-hardening and migration-Job knobs on the componentized chart - PR #35489
• Render pod-level securityContext on the migration Job - PR #35482
• Give the gateway and backend probes an explicit timeoutSeconds - PR #35497
• Honor USE_DDTRACE in the componentized gateway and backend deployments - PR #35490
• Bake prisma offline in the componentized migrations image - PR #35485
• Pin the bundled postgres and redis to the bitnamilegacy images - PR #34963

Dependencies & maintenance

• Raise the aiohttp floor to 3.14.2 to clear pooled-connection timeouts - PR #35337
• Move pydantic-settings into the base dependencies - PR #35518
• Remove the dead BedrockLLM invoke code path - PR #35188

Documentation Updates

• Require e2e proof on all three LLM endpoints when applicable in the PR template - PR #35280
• Require 15-25 word human-readable replies to AI PR review bots - PR #35266

PR roll-up by ownership area

PRs by ownership area (total: 151)

• Other (CI / chore / tests / build / version bumps): 35
• Performance: 24
• Auth & Management: 20
• UI: 19
• Models & Providers: 11
• Logging: 9
• MCP: 9
• Spend / Budgets / Rate Limits: 9
• LLM API Endpoints: 8
• Guardrails: 5
• Docs: 2

End-to-End Testing

We are investing heavily in end-to-end testing to cut regressions and make LiteLLM more stable release over release. Every version is exercised by a live suite that runs against a real deployed proxy and hits real provider endpoints, not mocks, so the behavior we validate is the behavior you get in production.

This window added 18 test-only PRs, 10 of them against the live e2e suite. New and repaired coverage lands on MCP tool polling across multi-worker lag, budget-reset timing polled to a deadline rather than a fixed sleep, the team-key budget hierarchy after the revert, and a throughput SLO derived per replica with locust's error breakdown surfaced. The coverage registry now excludes skipped tests from its numerator, so the reported number reflects what actually ran, and several suites are parked behind named Linear tickets (LIT-5027, LIT-5052, LIT-5054, LIT-5118, LIT-5119) rather than left flaking.

New Contributors

• @ljogeiger made their first contribution in PR #34603
• @lihugang made their first contribution in PR #35258

Full Changelog

https://github.com/BerriAI/litellm/compare/v1.95.0...v1.96.0

Original source

BREAKING CHANGES

User budgets are no longer enforced on team keys. This reverts PR #32005, which shipped in v1.94.0 and made a user's personal max_budget stack on top of the team and team-member budgets. Team keys are back to team budgets only, and the skip_user_budget_on_team_key opt-out introduced alongside it is removed; leaving it in general_settings is now a no-op. If you added that flag to restore the old behavior, you can drop it. See PR #35271.

Key Highlights

• Claude Opus 5 on day zero, everywhere - the new 1M-context Opus lands simultaneously on Anthropic, Amazon Bedrock (including the us, eu, au, jp, and global inference profiles), Google Vertex AI, and Azure AI Foundry, with adaptive thinking, xhigh reasoning effort, computer use, PDF input, and prompt caching all recorded in the cost map.
• Gemini 3.6 Flash and Gemini 3.5 Flash Lite - day-0 pricing on both Google AI Studio and Vertex AI, at $1.50/$7.50 and $0.30/$2.50 per 1M tokens respectively.
• MCP gateway grows a real front door - an always-on aggregate DCR discovery endpoint, identity-only session tokens, RFC 8707 resource indicators on upstream OAuth legs, MCP server support on the Anthropic /v1/messages API, and a standalone /connect route that no longer depends on the Chat UI flag.
• The Rust gateway takes over /v1/messages - native Anthropic Messages now routes through the axum gateway behind LITELLM_RUST, joined by a 1:1 port of the Responses API WebSockets surface, BaseAWSLLM credential resolution and SigV4 in litellm-core, and Bedrock audio transcription over the Python-to-Rust bridge.
• SAML 2.0 SSO for the admin UI - a second enterprise SSO path alongside the existing OIDC flow.
• The dashboard finishes its shadcn and DataTable migration - roughly twenty routes move onto shadcn and the shared composable DataTable, plus new react-hook-form and zod form infrastructure behind Organization Settings and Create Organization.
• Budget resets get correct - a configurable budget_reset_time of day, word-form budget_duration values that no longer silently collapse to daily, and repair for users and teams whose budget_reset_at was left NULL.
• Provider-level abuse controls per end user - with overwrite_user_with_key_hash, LiteLLM resolves the identity behind an incoming request, stamps the resolved virtual key hash into the outgoing user param, and forwards that to the provider. Providers can then ban or rate-limit at the level of an individual user rather than your entire organization. See PR #34417.

New Models / Updated Models

New Model Support (16 new pricing entries)

Beyond the new entries, this release raises the advertised context window on azure_ai/claude-opus-4-6, claude-opus-4-7, and claude-opus-4-8 from 200K to 1M, marks the DeepSeek V4 flash and pro families as reasoning models, and records supports_mid_conversation_system across the Claude Fable 5, Sonnet 5, and Opus 4.8 entries on Vertex AI and Azure AI. No pricing entries were removed.

Features

• Anthropic
  • Add Claude Opus 5 - PR #34518
• Google AI Studio / Vertex AI
  • Day-0 pricing for gemini-3.6-flash and gemini-3.5-flash-lite - PR #34106

Bug Fixes

• Anthropic
  • Only inject cache_control when the request does not already carry one - PR #33886
  • Strip uniqueItems and other unsupported array and object constraints from the output_format schema - PR #34313
  • Strip the remaining output_format schema keywords Anthropic rejects - PR #34319
• Amazon Bedrock
  • Hoist Codex additional_tools input items to top-level tools on Bedrock Mantle - PR #33228
• Google Vertex AI
  • Incrementally parse accumulated Gemini stream JSON so a multi-value payload no longer wedges the stream - PR #34320
  • Handle an explicit outputInfo: null in the Vertex AI batch response - PR #34473
• Azure AI Foundry
  • Advertise the 1M context window for Claude Opus 4.6 and newer on Foundry - PR #34556
• Fireworks AI
  • Restore the application/json Content-Type header, fixing a 415 response - PR #33929

LLM API Endpoints

Features

• Responses API
  • 1:1 port of the OpenAI Responses API WebSockets surface to litellm-rust - PR #33849
• Anthropic /v1/messages
  • Route native Anthropic /messages through Rust behind the LITELLM_RUST env var - PR #33848
  • Expose the Anthropic Messages route on the axum gateway - PR #33880
• Batches
  • Forward bedrock_tags to CreateModelInvocationJob for batch jobs - PR #33733
• General
  • Auto-enable drop_params for Codex user agents - PR #34068

Bugs

• Responses API
  • Preserve reasoning content through prompt hooks - PR #33422
  • Keep one chat completion id per stream and always stream completed responses - PR #34539
  • Strip include_usage from stream_options instead of dropping the whole param - PR #34549
• Anthropic /v1/messages
  • Model-aware mid-conversation system handling for Claude on Vertex AI and Azure - PR #33807
  • Route agentic-hook /messages requests back to Python for all stream modes - PR #34126
  • Backfill usage on non-streaming Bedrock Mantle /v1/messages responses - PR #34446
• Batches
  • Paginate the managed batch list by unified_object_id cursor - PR #34192
  • Resolve a managed unified input_file_id to a storage URL with an ownership check before dispatch - PR #34474
  • Make managed-file resolution additive, restoring the fall-back for missing-row and lookup errors - PR #34584
• Realtime
  • Emit the Nova Sonic session.created event on connect and session.updated on session.update - PR #34133
  • Install the bedrock-realtime extra for Nova Sonic realtime - PR #34426
• A2A
  • Accept semver protocolVersion values such as 0.3.0 in agent cards - PR #34154
  • Allow optional securityScheme fields so /public/agent_hub no longer 500s - PR #33897
  • Route /a2a through the gateway component - PR #34958
• General
  • Return 400 instead of 500 for chat completions sent without messages - PR #34547
  • Add queued to the Interaction status enum - PR #34135

Management Endpoints / UI

Features

• Auth & SSO
  • SAML 2.0 SSO for the admin UI - PR #31429
  • Stamp the outgoing user param with the key hash via overwrite_user_with_key_hash - PR #34417
• Organizations
  • RESTful PATCH /v2/organization/{organization_id} - PR #32350
  • Rebuild Organization Settings on react-hook-form and zod with a dirty-field PATCH - PR #34324
  • Migrate the Create Organization form to shadcn and react-hook-form - PR #34552
• Virtual Keys
  • Block and unblock a key directly from the key info page - PR #34116
  • Deep-link the virtual key detail view via a ?key= query param - PR #34591
  • Surface a key's budget_reset_at in key info and the keys table - PR #34113
• Models + Endpoints
  • Give each Models + Endpoints tab its own path - PR #34327
  • Extract shared tab-routing helpers and adopt them in Models + Endpoints - PR #34435
  • Make the DB config-reload interval configurable from config.yaml and the Admin UI - PR #34130
• Shared DataTable migration
  • Credentials, available teams, memory, audit logs, organizations, and agents tables - PR #34053, PR #34070, PR #34079, PR #34080, PR #34081, PR #34089
  • Tool Policies, users and model health checks, request logs, models and endpoints, and routing groups tables - PR #34176, PR #34182, PR #34343, PR #34363, PR #34571
  • Controlled row selection on the shared DataTable - PR #34167
• shadcn migration
  • api-reference, prompts list, transform-request, old-usage, search-tools, agents, memory, and workflow runs - PR #34263, PR #34289, PR #34303, PR #34304, PR #34323, PR #34365, PR #34366, PR #34370
  • mcp-servers, tag-management, tool-policies, logging-and-alerts, caching, policies, budgets, skills, ui-theme, access-groups, and vector-stores - PR #34469, PR #34468, PR #34465, PR #34466
  • react-hook-form and zod form infrastructure - PR #34170
  • Migrate inline provider logo lookups and MCP, callback, guardrail, SSO, and search-tool logos onto the shared Logo component - PR #34141, PR #34169
• Typed management API
  • Type the PATCH /team/{team_id} request body - PR #34195
  • Derive the dashboard object_permission type from the generated schema - PR #34454

Bugs

• SCIM
  • Use members_with_roles as the source of truth for group membership - PR #34162
  • Prune a deleted user from teams' members_with_roles - PR #34180
  • Parse the membership id from the filtered PATCH path when value is omitted - PR #34181
  • Sync the team roster and dedup teams on existing-user email upsert - PR #34183
• Teams & Users
  • Make team member add atomic so a concurrent add cannot lose a member - PR #34185
  • Restore the atomic user upsert when adding team members - PR #34457
  • Route the JWT default-team into memberships instead of the create payload - PR #33082
  • Populate user_email on UserAPIKeyAuth for JWT auth - PR #34174
  • Validate default team values in Default User Settings - PR #34815
• Config & credentials
  • Prevent provider key exposure through URL-valued model destinations and fallbacks - PR #34189
  • Stop save_config from snapshotting environment_variables into the DB - PR #34119
  • Hash a caller-supplied key in the key update audit log object_id - PR #34632
  • Surface SSO and SMTP settings supplied as process env vars - PR #33576
  • Reflect REDIS_* env cache config and stop the UI overwriting the stored password - PR #34160
• CLI
  • Stable port and persisted master key for lite autoroute up - PR #34026
  • Discover models via /v1/models so an AI-API-only key works for autoroute - PR #34259
• Dashboard
  • Find logs by request id across pages and dates - PR #31743
  • Scope and bound the End User filter on the logs page - PR #34579
  • Bind the key duration input to a single Form.Item so a pre-filled expiry submits - PR #34521
  • Stop cloning body-carrying requests into stream uploads in the fetchClient middleware - PR #34122
  • Distinguish the response cache from provider prompt caching - PR #34138
  • Keep entity usage tabs aligned with their panels - PR #34573
  • Add a tooltip to the Active key status badge - PR #34109
  • Restore the Add MCP Server dialog size and header spacing, and match MCP Servers tabs to the dashboard line tab pattern - PR #34679, PR #34685
  • Center vertical toolbar dividers and truncate long team names in the models table team dropdown - PR #34684, PR #34689
  • Serve /ui/assets from the nginx image instead of the SPA fallback, and bundle provider logos as static imports - PR #34066, PR #34125, PR #34163
  • Return Models + Endpoints tabs to in-memory routing while keeping the ?model drill-in - PR #34629
  • Land general login on the keys dashboard and send MCP consent to /ui/connect - PR #35523

AI Integrations

Logging

• OpenTelemetry
  • Keep an MCP tool call in one trace, anchored to its own request - PR #34537
  • Stamp an MCP tool failure on the request that carried it - PR #34551
• Langfuse
  • Send the v4 ingestion header for the otel callback - PR #33907
• Prometheus
  • Populate cache write token metrics for OpenAI-style usage - PR #34803
• General
  • Record cost and usage reported by the upstream target on passthrough routes - PR #34590
  • Move the logs end-user filter onto /management/v1 - PR #34691
  • Sanitize per-key callback config out of logged metadata - PR #32583
  • Match the exact class in callback dedup so a custom subclass no longer blocks a built-in logger - PR #34804
  • Stop scheduling the sync failure_handler concurrently with async_failure_handler - PR #34306
  • Surface env-var-sourced theme and logging-callback settings, and drop the misleading os.environ tooltip - PR #34156, PR #34305

Guardrails

• General
  • Add only_scan_new_messages for per-session incremental scanning - PR #33278
  • Add a run_in_parallel opt-in for concurrent pre_call and post_call guardrails - PR #33770
  • Add DeepKeep as a custom guardrail - PR #33844
  • Merge model-level guardrails before pre_call_hook - PR #29654
  • Classify all 4xx HTTPException guardrail blocks as intervened - PR #33821
  • Stop reporting a no-op guardrail as applied on passthrough - PR #34411
  • Keep guardrail information in spend logs when the caller sends its own metadata - PR #34458
  • Resolve judge_model credentials via a lazy Router lookup in llm_as_a_judge - PR #34509
• Model Armor
  • Sanitize error details by default - PR #33908
  • Handle None metadata in the post_call response processor - PR #34405
• Compresr / Headroom
  • Compress content-parts messages in the headroom guardrail for Anthropic traffic - PR #34586
  • Derive tokens_saved when the compression service omits it - PR #34578
  • Preserve cache_control breakpoints in the compresr write-back - PR #34660
• Straiker
  • Add /v1/messages support - PR #34548
• Bedrock Guardrails
  • Stop replaying expired Google OIDC tokens to STS on guardrail auth - PR #34637
• Dashboard
  • Hide guardrail group headers when only one group has entries - PR #33885

Spend Tracking, Budgets and Rate Limiting

• Budgets
  • Configurable budget_reset_time of day - PR #31007
  • Reset users and teams whose budget_reset_at is NULL - PR #33623
  • Resolve word-form budget_duration so it no longer silently resets daily - PR #34250
  • Enforce a global max_budget against the resettable proxy budget row so budget_duration is honored - PR #33732
  • Set budget_reset_at when a JWT upsert seeds a budget_duration - PR #34050
  • Reject failed atomic budget reservations under fail_closed_budget_enforcement - PR #34429
  • Stop enforcing user budgets on team keys, reverting the hierarchy change that shipped in v1.94.0 - PR #35271
  • Handle a tz-aware temp_budget_expiry, apply temp_budget_increase for cache-hit keys, and derive the increase without mutating the token - PR #33840, PR #33841, PR #34121
  • Raise the dashboard session budget default to $1 and make it configurable in config and the Admin UI - PR #34146
• Cost tracking
  • Map OpenAI cache_write_tokens for prompt cache creation billing - PR #34046
  • Track prompt compression saved tokens in daily spend aggregates - PR #33810
  • Attribute org spend for team-linked credentials minted without an org_id - PR #34577
  • Gate an unsupported service_tier on drop_params for the Bedrock Mantle Responses API - PR #34058
  • Raise the /spend/logs/v2 page_size cap to 1000 - PR #33994
• Cost Optimization page
  • Add spend-by-tool and cache leakage views - PR #33978
  • Add configuration tabs - PR #33899
  • Mark Cost Optimization as beta in the left nav - PR #34984
  • Anchor the savings line at a $0 range start and swap the methodology Collapse for a shadcn HoverCard - PR #34453, PR #34598
  • Keep the cache leakage time range picker inline at narrow widths and the date picker on the right - PR #34439, PR #34885
• Tool spend
  • Roll up tool spend daily instead of scanning SpendLogs - PR #34675
  • Cap the /v1/tool/spend window at 30 days and bound every SpendLogs read - PR #34582
• Dashboard
  • Stop the key-edit form 403ing on non-budget saves - PR #34112

MCP Gateway

Dynamic Client Registration

• Always-on aggregate gateway DCR discovery front door - PR #33174
• Identity-only session tokens for the gateway DCR front door - PR #33182
• Admit gateway DCR session bearers at the aggregate /mcp scope - PR #33190
• Return the DCR client's own redirect_uris to stop the /callback self-redirect loop - PR #33756
• Fall through to an ephemeral DCR mint when passthrough authorize has no client_id - PR #33884

OAuth

• Send RFC 8707 resource indicators on upstream OAuth legs - PR #34265
• Migrate client_credentials (M2M) onto the v2 resolver arm - PR #32259
• Delete the unreachable v1 OBO handler and gate REST OAuth on the v2 resolver - PR #34407
• Let an admin-pinned issuer drive OAuth discovery for url-less servers - PR #34065
• Log actionable OAuth discovery failures for misconfigured server urls - PR #34225
• Store the enterprise IdP identity assertion at SSO login for EMA egress - PR #34072
• Standalone /connect route for MCP OAuth, decoupled from the Chat UI flag - PR #34334

Servers & tools

• Support MCP servers on the Anthropic /v1/messages API - PR #33631
• Add Google Sheets, Drive, Calendar, and Docs to the OpenAPI registry, and move Drive to the official streamable HTTP MCP server - PR #34059, PR #34322
• Attach resolved OAuth credentials to OpenAPI spec_path tool calls - PR #34063
• Stop leaking upstream server credentials in a tool-call 403 - PR #34340
• Use a toolset row's stored tool name as written - PR #34559
• Keep a key's MCP toolsets when saving an edit - PR #34452
• Consolidate exception-tree walkers into one shared faults traversal - PR #33183

Performance / Loadbalancing / Reliability improvements

• Rust core
  • Port BaseAWSLLM auth, credential resolution, and SigV4 to litellm-core as a base provider - PR #33888
  • Bedrock audio transcription via the Rust core over the Python-to-Rust bridge - PR #33990
  • Honor pre-computed Entra ID auth for Azure /messages - PR #34107
• Streaming & core
  • Build the per-chunk Delta directly instead of setattr / delattr churn - PR #33992
  • Fast-path SafeAttributeModel.delattr for declared fields - PR #33993
  • Forward SageMaker stream events as they arrive to cut TTFT - PR #34338
• Router
  • Edit fallback chains from router settings - PR #32841
  • return_raw_model_name toggle for the complexity router's response model field - PR #33875
  • Show in the log drawer and session sidebar when an auto-router served a request - PR #34434
  • Don't cool down the parent deployment on an advisor sub-call failure - PR #33792
  • Stop custom model_info leaking onto the shared backend cost map key, and propagate capability flags to it - PR #34041, PR #34047
  • Honor request-level num_retries over litellm_settings.num_retries, and stop per-deployment num_retries double-counting as provider max_retries - PR #34124, PR #34129
  • Release the pre-routing strategy slot when a deployment is replaced or deleted - PR #34564
  • Treat malformed cost-map token limits as absent on /v1/models - PR #33903
• Proxy reliability
  • Make in-memory and disk cache increments atomic - PR #34013
  • Share CLI SSO login sessions across workers without enable_redis_auth_cache - PR #33261
  • Avoid a DB outage during planned RDS IAM rotation - PR #34749
  • Stop litellm/proxy from shadowing installed packages on sys.path - PR #34656
  • Bake non-root prisma engines at /opt/prisma so migrations run offline for any uid - PR #34325
  • Pass an explicit Python version request to uv tool install - PR #34750
• Dependencies & build
  • Migrate the litellm-rust workspace to Rust edition 2024 - PR #33940
  • Advisory-clear bumps for gitpython, pypdf, pyasn1, js-yaml, brace-expansion, postcss, sharp, and Next.js - PR #34056, PR #34148, PR #34168, PR #34193, PR #34329, PR #34634, PR #34798

Documentation Updates

• Add a TLDR section to the PR template - PR #34203
• Ask for a numbered list of reproduction steps in the issue template - PR #34207

PR roll-up by ownership area

PRs by ownership area (total: 294)

• UI: 55
• Other (CI / chore / tests / build / version bumps): 46
• Spend / Budgets / Rate Limits: 34
• LLM API Endpoints: 31
• MCP: 29
• Auth & Management: 28
• Performance: 22
• Guardrails: 19
• Logging: 14
• Models & Providers: 14
• Docs: 2

End-to-End Testing

We are investing heavily in end-to-end testing to cut regressions and make LiteLLM more stable release over release. Every version is exercised by a live suite that runs against a real deployed proxy and hits real provider endpoints, not mocks, so the behavior we validate is the behavior you get in production.

This window added 62 test PRs, the largest single-release investment in the suite so far. New coverage lands on live A2A agents, /v1/images/edits, /openai, and /vllm chat passthrough cost logging, credential-backed /v1/messages, Azure AI Foundry and Anthropic /v1/messages through the Rust bridge, MCP access-group tool selection at key creation, a real Linear OAuth MCP driven through chat completions, budget and rate-limit resets across personal, team, and team-member keys, and a weekly session-anomaly load test against real providers. The Admin UI Playwright suite moved under tests/e2e/ui, and a run of hardening PRs removed cross-suite races on control-plane writes and data-plane sync lag.

New Contributors

• @lyb0307 made their first contribution in PR #33228
• @jyeung-r7 made their first contribution in PR #33623
• @vineetpuranik made their first contribution in PR #33940

Full Changelog

See https://github.com/BerriAI/litellm/compare/v1.94.0...v1.95.0

Original source

Key Highlights

• GPT-5.6 and more new models - day-0 pricing and metadata for OpenAI GPT-5.6 (sol/terra/luna) on OpenAI and Azure, xAI Grok-4.5, OpenAI Realtime 2.1 (and -mini), Google Cloud Chirp 3 speech-to-text, and the jp regional inference profile for Bedrock Claude Opus 4.8.
• Meta Model API provider - a new OpenAI-compatible provider (meta) serving muse-spark-1.1 on day-0 across Chat Completions, /v1/messages, and Responses.
• Client-forwarded MCP credentials - new true_passthrough and oauth_delegate auth modes plus a dcr_bridge sealed-envelope path let clients hold their own upstream MCP credentials, with PKCE S256 enforced on both authorize arms and upstream discovery bound to each server.
• shadcn / Base UI dashboard migration - the shared DataTable, charts (recharts), and the full-height sidebar shell move onto Base UI primitives, with a redesigned account menu and reusable filter/column-visibility controls.
• Smarter complexity router - keyword tier overrides, semantic keyword matching, an optional LLM-based classifier, and per-decision routing logs for the auto router.

New Providers and Endpoints

New Providers (1 new provider)

Provider Supported LiteLLM Endpoints Description Meta Model API (meta) Chat Completions, /v1/messages, Responses OpenAI-compatible Meta Model API provider serving muse-spark-1.1 on day-0 - PR #32701

New Models / Updated Models

New Model Support (31 new pricing entries)

Provider Model Context Window Input ($/1M tokens) Output ($/1M tokens) Features OpenAI gpt-5.6 (and -sol / -terra / -luna) 1.05M $5.00 (terra $2.50, luna $1.00) $30.00 (terra $15.00, luna $6.00) Reasoning, vision, function calling, prompt caching, PDF input, web search Azure azure/gpt-5.6 (and us / eu regional + -sol / -terra / -luna) 1.05M $5.00 $30.00 Reasoning, vision, function calling, prompt caching, PDF input, web search Azure azure/{us,eu}/gpt-5.4, azure/{us,eu}/gpt-5.5 (data-zone + long-context) 1.05M $2.75 / $5.50 $16.50 / $33.00 Reasoning, vision, function calling, prompt caching, PDF input OpenAI gpt-realtime-2.1, gpt-realtime-2.1-mini 128K $4.00 / $0.60 $24.00 / $2.40 Audio input/output, function calling xAI xai/grok-4.5, xai/grok-4.5-latest 500K $2.00 $6.00 Reasoning, vision, function calling, web search Meta meta/muse-spark-1.1 1.05M $1.25 $4.25 Reasoning, vision, function calling, prompt caching, PDF input, web search Amazon Bedrock jp.anthropic.claude-opus-4-8 1M $5.50 $27.50 Reasoning, computer use, vision, PDF input, adaptive thinking Google Vertex AI vertex_ai/chirp_3 (Speech-to-Text) n/a $0.00026667 / sec n/a Audio transcription

GPT-5.6 ships priority, flex, batch, and above-272k long-context pricing tiers on both OpenAI and Azure - PR #32659, PR #32678. Azure gpt-5.4 / gpt-5.5 gained data-zone and long-context entries - PR #32279, and Bedrock regional inference profiles now resolve to their regional pricing in get_model_info - PR #32389.

Features

OpenAI

• Add GPT-5.6 (sol/terra/luna) pricing and metadata - PR #32659
• Add gpt-realtime-2.1 models with regional processing uplift - PR #32387
• Forward the verbosity param to chat completion providers - PR #32254

Azure

• Add Azure GPT-5.6 (sol/terra/luna) pricing and metadata - PR #32678
• Add Azure data-zone and long-context pricing for gpt-5.4 / gpt-5.5 - PR #32279

xAI

• Add xai/grok-4.5 pricing and metadata - PR #32549

Meta

• Add the Meta Model API provider and muse-spark-1.1 on day-0 - PR #32701

Amazon Bedrock

• Add jp.anthropic.claude-opus-4-8 to the model cost map - PR #32840
• Retain clear_tool_uses_20250919 context-management edits and emit the context-management-2025-06-27 beta for Claude Invoke - PR #32658
• Flag mapped Claude 4.8+ entries with supports_mid_conversation_system - PR #32882

Google Vertex AI

• Add Google Cloud Speech-to-Text Chirp 3 transcription support - PR #32274

Anthropic

• Translate adaptive thinking/effort to pre-4.6 model support and thread the real provider through capability probes - PR #32867, PR #32874
• Resolve @default Vertex AI models to adaptive thinking by stripping the @version suffix in lookup candidates - PR #32833

Bug Fixes

Amazon Bedrock

• Preserve the stream param and decode SSE for Bedrock Mantle streaming - PR #32141
• Emit an SSE error event when an invoke Messages stream ends without message_stop - PR #32159
• Stop stale SigV4 headers clobbering a fresh signature on strip-and-retry re-sign - PR #32371
• Honor cache_control ttl on message-level cache points - PR #32538, PR #32551
• Keep mid-conversation system messages in place for Claude Invoke, gated on model support - PR #32578, PR #32831
• Honor AWS auth params in the realtime handler - PR #32275
• Forward AWS credential kwargs into litellm_params so the chat -> Responses bridge keeps Web Identity Federation auth - PR #32956

Google Vertex AI

• Build the full request path when a custom api_base has no path - PR #32367
• Return the create_vertex_url result directly for openai-path partner models with a custom api_base - PR #32380
• Forward realtime health check params - PR #32550

Azure

• Build the responses input_items url with the path before the query string - PR #32270

Anthropic

• Drop an incompatible pinned temperature when downgrading adaptive thinking for pre-4.6 models on /v1/messages - PR #33244

General

• Forward provider response headers on streaming /v1/messages responses - PR #32160
• Surface in-body error payloads on OpenAI-compatible streams - PR #32237
• Resolve os.environ/ references universally in DB-sourced models - PR #32405
• Stop per-request custom pricing from clobbering shared model_cost pricing - PR #32163

LLM API Endpoints

Features

Responses API

• Preserve custom-tool round-trips and allowlists for the Codex CLI in the responses bridge - PR #32258
• Preserve reasoning_tokens through the chat -> responses usage translation - PR #32837

Pass-through

• Stream non-SSE pass-through responses instead of buffering them in memory - PR #32386

General

• Merge websearch tool params - PR #32162
• Stamp completion_start_time on the first chunk for /v1/messages and /v1/responses - PR #32284

Bugs

Responses API

• Make response-id encoding idempotent to prevent double-encoding - PR #32034
• Map an upstream 4xx on cancel to a client error instead of a 500, and surface the upstream error status on get - PR #32271, PR #32287
• Raise APIError on in-stream error events and widen ErrorEventError.param to accept a dict - PR #32835
• Stop scheduling the sync success_handler concurrently with async_success_handler - PR #32239
• Decrypt response ids for input_items follow-ups - PR #32269

Rerank

• Log optional_rerank_params at debug so request content is not written to logs - PR #32533

General

• Stop request params from clobbering merged target query params in pass-through - PR #32404

Management Endpoints / UI

Features

UI (shadcn / Base UI migration)

• Switch the shadcn primitives from Radix to Base UI - PR #32124
• Add a shared composable DataTable and reskin it onto shadcn table primitives, with a filter drawer, column visibility, and search - PR #32680, PR #32209, PR #32856
• shadcn charts foundation with tremor-compatible wrappers and conversion of the caching, projects, activity, usage, and per-user charts to shadcn/recharts - PR #32668, PR #32721, PR #32722, PR #32725, PR #32726, PR #32729
• Full-height sidebar shell with a content-scoped top bar, a redesigned account menu, and a shared CopyButton - PR #32793, PR #32931, PR #32945
• Typed openapi-fetch foundation (fetchClient) with a first typed caller - PR #29884

Dashboard

• Enterprise license expiry banner on the admin dashboard - PR #32540
• Session id filter and duration/start-time sort on request logs and the session sidebar - PR #32568, PR #32432
• Cost optimization feedback banner on the models page - PR #32174
• Expose MCP max_concurrent_requests in the server create and edit forms - PR #32397
• Root the gateway breadcrumb in the AI Gateway selector - PR #32886
• Back the Redis URL and Database Index UI fields end-to-end - PR #32075

Auth & Management

• RESTful PATCH /team/{team_id} with JSON merge patch semantics, reachable by org admins - PR #32883
• JWT auth falls back to DB team memberships when the token has no team claims - PR #31356
• lite auth print-token for Claude Code apiKeyHelper support - PR #32846
• Add an expires filter to GET /key/list - PR #32953
• Make the Microsoft Graph endpoint configurable for GCC High - PR #32517

Bugs

UI

• Scope the key models dropdown options to the key's team - PR #32382
• Reflect the persisted "Store Prompts in Spend Logs" toggle on load - PR #32145
• Prevent the reasoning block from expanding the chat playground layout - PR #32485
• Rename the Virtual Keys "Key Hash" filter label to "Key ID" - PR #32672
• Forward refs through UI primitives and fail tests on swallowed refs - PR #32401
• Strip a trailing slash from --base-url in the lite CLI - PR #32845

Auth & Management

• Surface OAuth error params in the SSO callback - PR #32433
• Resolve team org from team_id so org admins can update team budgets - PR #32560
• Honor os.environ/ references for all AWS auth params in DB-sourced models - PR #32256
• Hash Bearer-prefixed API keys before writing spend logs - PR #31799

AI Integrations

Logging

• OpenTelemetry
  • Stamp gen_ai.response.time_to_first_chunk on streaming LLM spans - PR #32236
  • Emit the gen_ai.client.operation.exception event on failed LLM calls and restore error.* span attributes on v2 error spans - PR #32655, PR #32524
  • Move litellm error detail keys under the litellm.* namespace - PR #32591
• Prometheus
  • Skip budget metric DB/cache lookups when the gauges are a NoOpMetric - PR #32834
• DataDog
  • Split log batches proactively to stay under intake payload limits - PR #32860
• General
  • Classify allm_passthrough_route as async to prevent duplicate success callbacks - PR #32265
  • Capture logging_obj before post_call_failure_hook pops it on the streaming ModifyResponseException path - PR #32665

Guardrails

• Model Armor
  • Scan MCP tool calls in pre_mcp_call / during_mcp_call modes - PR #32296
  • Pass reference-only attachments through with skip_unscannable_attachments and remove the attachment count cap - PR #33554
• Content Filter
  • Add pre_mcp_call support to Content Filter - PR #32936
• CrowdStrike AIDR
  • Send only new messages since the last assistant turn - PR #31974
• GraySwan
  • Forward the GraySwan scan id header - PR #32544
• General
  • Walk the Responses-API text taxonomy in the shared content helpers - PR #32542
  • Honor disable_exception_on_block by raising ModifyResponseException for Bedrock guardrails - PR #32289
  • Match a multi-mode guardrail_mode without a false-COMPLIANT result - PR #32832
  • Filter the Add-Guardrail mode dropdown per provider - PR #32712
  • Mask credentials embedded in a guardrail_response before persisting it - PR #32687
  • Log real token/compression stats on the headroom guardrail - PR #32158

Secret Managers

• General
  • Harden external secret-manager name validation - PR #32092

Spend Tracking, Budgets and Rate Limiting

Budgets

• Throttle keys after a spend limit instead of revoking access - PR #31300
• Resolve team org from team_id so org admins can update team budgets - PR #32560

Rate Limiting

• Per-tag RPM limiting on a single key - PR #31502
• Separate ITPM/OTPM deployment rate limits on the router - PR #31952
• Trigger gateway fallbacks on local rate limit errors - PR #31788
• Populate x-ratelimit-* remaining/limit values in the standard logging object for streaming - PR #32711

Cost Tracking

• Enforce budget and cost tracking for Dashscope tiered pricing - PR #32942
• Price Anthropic pass-through message batches correctly in the batch cost job - PR #32307
• Filter /global/spend/report by team_id when group_by=team - PR #32170
• Sum multi-round session cost in the logs UI, and bound the logs-tab pagination count - PR #32796, PR #31825
• Honor store_prompts_in_spend_logs for guardrail_information - PR #32688

MCP Gateway

Client-Forwarded Credentials (true_passthrough / oauth_delegate / dcr_bridge)

• Add the true_passthrough and oauth_delegate auth modes, with upstream OAuth discovery bound to each server - PR #31989, PR #32414
• Add the dcr_bridge column and plumbing, a sealed-envelope module for client-held credentials, and the discovery facade plus register/token relay with mandatory PKCE S256 - PR #32745, PR #32748, PR #32753, PR #32747
• Admit dcr_bridge oauth_delegate clients via a single envelope bearer, with consumer helpers for key derivation - PR #32824, PR #32794
• Relay an upstream 401 on client-forwarded pass-through tool calls - PR #32556

OAuth 2.0 (On-Behalf-Of)

• oauth2_token_exchange auth type via the REST API and dashboard, plus an entra_obo token_exchange profile selectable in the UI and API - PR #31772, PR #31983, PR #32144
• Persist oauth2_flow explicitly on create, backfill legacy null rows on startup, and read it verbatim from DB rows - PR #32288, PR #32290, PR #32292
• Add an OAuth flow selector on the MCP edit page - PR #32298
• Apply the outbound concurrency limit to OBO tool calls - PR #32071

Semantic Filter

• Apply the semantic filter to expanded litellm_proxy tools and show the filtered-out count - PR #32285
• Keep tool names whole in the filter response header - PR #32282
• Fail closed and surface semantic-filter context-window errors - PR #32715

Bug Fixes

• Log MCP tool calls returning isError=true as failures - PR #32238
• Re-register a DCR client when the proxy origin no longer matches its registered redirect_uri - PR #32527
• Invalidate a browser-authorized upstream token when a mint-relevant field changes - PR #32652
• Defer the proxy import so completion(tools=...) works without proxy extras - PR #32339
• Alias/display-name tool routing, REST filters, and BYOK auth - PR #32320

Performance / Loadbalancing / Reliability improvements

Routing

• Keyword tier overrides and semantic keyword matching for the complexity auto router - PR #32859
• Optional LLM-based classifier for the complexity router, plus custom technical keywords and per-decision routing logs - PR #32169, PR #32262, PR #32943

Reliability

• Recover the Prisma DB reconnect loop when the client is disconnected - PR #32323
• Configure a coordination Redis independently of the response cache, building the usage cache from REDIS_* env - PR #32661, PR #32635
• Only use SSLConnection when ssl is truthy in the connection-pool kwargs - PR #32825
• Keep the Prometheus /metrics Mount in the gateway route trim - PR #32317
• Wire the general_settings request allowlist to litellm globals - PR #32243

Hot paths

• Negative-cache missing user/key lookups on the request hot path - PR #32368
• Stop CacheCodec dropping null fields on a cache round-trip - PR #32207
• Source /v1/models token limits from the cost map instead of per-model router aggregation, removing wildcard deep copies - PR #33721
• Treat malformed configured token limits as absent on /v1/models - PR #33864

Build

• Restore the litellm-proxy-extras source dir in runtime images - PR #33592
• Bake the prisma CLI and engines at a fixed path so fresh-database migrations work for any uid offline - PR #33853
• Raise the requires-python cap to <3.15 so Python 3.14 installs current releases - PR #33438
• Allow redisvl, pypdf, and openapi-core on Python 3.14 - PR #33801
• Raise pyo3 to 0.29 so the native bridge compiles on Python 3.14 - PR #33798
• Update ddtrace to the 4.x line - PR #33484
• Bump mcp to 1.28.1, pillow to 12.3.0, and pin httplib2/setuptools floors - PR #33803, PR #33093, PR #33233

Documentation Updates

• Point OSS contributors at the daily OSS branch - PR #32830

PR roll-up by ownership area

PRs by ownership area (total: 254)

• Other (CI / chore / tests / refactors / version bumps): 65
• UI: 42
• MCP: 36
• Models & Providers: 33
• Performance: 14
• Spend / Budgets / Rate Limits: 13
• LLM API Endpoints: 15
• Guardrails: 11
• Auth & Management: 9
• Logging: 9
• Docs: 6
• Secret Managers: 1

New Contributors

• @thibault-linktree made their first contribution in PR #32034
• @akapur99 made their first contribution in PR #32829

Full Changelog

https://github.com/BerriAI/litellm/compare/v1.92.0...v1.93.0

Original source

Key Highlights

• Claude Sonnet 5 - first-class support across Anthropic, Amazon Bedrock (including the regional inference profiles), Vertex AI, and Azure AI, with a 1M-token context window, reasoning, computer use, PDF input, and introductory pricing through 2026-08-31.
• Production-ready MCP OAuth (On-Behalf-Of) - the token_exchange arm moves onto the v2 resolver with RFC 9728 -> RFC 8414 endpoint discovery (no IdP guessing), persisted Dynamic Client Registration, per-server outbound concurrency limits, and a mcp_tool_search virtual tool for large tool catalogs.
• Two new providers - Tencent (DeepSeek V4 flash/pro via TokenHub, chat and /v1/messages) and Google Distributed Cloud (GDC) Gemini for on-prem/sovereign deployments.
• Access-control hardening - admin-gating of permissions and allowed_routes across the key, user, and team endpoints, AES-256-GCM at-rest credential encryption with a versioned re-encryption migration, and redaction of secrets from startup and router error logs.
• Faster hot paths - spend-counter increments and pre-call budget reads are now gathered concurrently, the cost-callback deepcopy moved off the request event loop, OTel runtime imports are memoized, and Redis-cluster reconnect plus read-replica boot resilience keep the proxy serving during infra blips.

New Providers and Endpoints

New Providers (2 new providers)

Provider | Supported LiteLLM Endpoints | Description

Tencent (tencent) | Chat Completions, /v1/messages | Tencent TokenHub provider serving DeepSeek V4 (flash and pro) with reasoning, prompt caching, and Anthropic Messages support - PR #31903

Google Distributed Cloud - GDC (gdc) | Chat Completions | Google Distributed Cloud Gemini provider for on-prem and sovereign-cloud deployments - PR #31895

New Models / Updated Models

New Model Support (13 new pricing entries across 4 models)

Provider | Model | Context Window | Input ($/1M tokens) | Output ($/1M tokens) | Features

Anthropic / Bedrock / Vertex / Azure AI | claude-sonnet-5 | 1M | $2.00 | $10.00 | Reasoning, vision, function calling, prompt caching, computer use, PDF input, adaptive thinking

Bedrock Mantle | bedrock_mantle/xai.grok-4.3 | 131K | $1.25 | $2.50 | Reasoning, vision, function calling

Tencent | tencent/deepseek-v4-flash | 1M | $0.14 | $0.28 | Reasoning, function calling, prompt caching

Tencent | tencent/deepseek-v4-pro | 1M | $0.435 | $0.87 | Reasoning, function calling, prompt caching

Claude Sonnet 5 ships with pricing entries for Anthropic (claude-sonnet-5), Amazon Bedrock (anthropic.claude-sonnet-5 plus the us / eu / au / jp / global regional inference profiles), Vertex AI (vertex_ai/claude-sonnet-5), and Azure AI (azure_ai/claude-sonnet-5) - PR #31740, with introductory pricing applied through 2026-08-31 - PR #31917.

Features

• Anthropic
  • Add Claude Sonnet 5 with reasoning, computer use, and PDF input - PR #31740
  • Keep context_management working when drop_params is enabled - PR #32020
  • Preserve x-anthropic-billing-header system blocks for first-party Anthropic - PR #29584
• Amazon Bedrock
  • Forward strict and additionalProperties to the Converse toolSpec - PR #29814
  • Add xai.grok-4.3 to the model cost map for Bedrock Mantle SigV4 auth - PR #31916
  • SigV4/IAM auth on the Bedrock Mantle Responses API route - PR #29788
  • Honor ttl for tool_config cache injection points - PR #31929
  • Map guardrailConfig to InvokeModel guardrail headers - PR #31985
• Google Vertex AI
  • Pass the full imageConfig dict for Gemini image generation - PR #31811
  • Propagate Vertex AI metadata in streaming success callbacks - PR #29899
  • Single media upload for batch files to fix 499s on large uploads - PR #31653
• Google Gemini
  • Support Gemini TTS languageCode parameters - PR #29623
  • New Google Distributed Cloud (GDC) Gemini provider - PR #31895
• Tencent
  • Add Tencent TokenHub as a provider serving DeepSeek V4 - PR #31903
• Fireworks AI
  • Enable tool calling for glm-5p1 in the model cost map - PR #29697
• Databricks
  • Split parallel tool calls so each tool message follows its tool_calls - PR #31633
• General
  • Add Parasail as a JSON-configured OpenAI-compatible provider - PR #29842

Bug Fixes

• Amazon Bedrock
  • Drop toolSpec.strict for Opus 4.7/4.8 to unblock tool calls - PR #31923
  • Drop strict / additionalProperties from toolSpec for Claude Sonnet 4 - PR #31943
  • Omit empty additionalModelRequestFields and system from the Converse payload - PR #29565
  • Expand os.environ/ references for AWS auth params, then all fields, in DB-sourced models - PR #32256, PR #32405
• Anthropic
  • Bill streaming 1h prompt-cache writes at the 1h rate - PR #32073
  • Drop unsignable thinking blocks and allow a null signature in logging - PR #31654
  • Require a caller api_key and validate api_base in the advisor tool - PR #32093

LLM API Endpoints

Features

• Responses API
  • Passthrough /v1/messages to native provider endpoints via supported_endpoints - PR #31685
  • Route GitHub Copilot /v1/messages to the native Anthropic endpoint - PR #31802
  • Add cache-control injection support for the /v1/messages endpoint - PR #31778
• /v1/messages
  • Drop top-level additional_drop_params on /v1/messages - PR #31645
• Image Generation
  • Azure AI MAI-Image-2.5 image generation support - PR #29688
• A2A
  • Support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents - PR #30950
• Sandbox
  • Reuse the e2b container across requests when metadata.session_id is set - PR #31688
• General
  • Extend the response-headers hook to streaming, TTS, image generation, and pass-through - PR #24232
  • websearch_interception agentic-loop fixes for chat completions and Anthropic messages - PR #31669
  • Make the TinyFish search provider permissive and attribute errors - PR #31997

Bugs

• Responses API
  • Preserve forced-function tool_choice name in the Responses-to-Chat transform - PR #29812
  • Map a system-only chat request to a system input item in the Responses bridge - PR #29817
  • Merge metadata.tags into litellm_metadata on the /v1/responses route - PR #31793
  • Route per-model on GitHub Copilot /v1/responses based on model info - PR #29747
  • Drop unmappable Bedrock Responses tools instead of failing the request - PR #31663
• Realtime API
  • Stop a second Gemini Live setup, retry a hung handshake, and close a guardrail bypass - PR #31519
  • Trigger Nova Sonic generation on response.create so realtime sessions stop hanging - PR #31924
  • Route realtime HTTP endpoints through the router for credential resolution - PR #32077
• OCR
  • Preserve content, tables, and keyValuePairs in Azure AI doc-intelligence /v1/ocr - PR #32018
• A2A
  • Populate response usage in the A2A chat transformation - PR #31980
• General
  • Return upstream error bodies unchanged in pass-through - PR #32133
  • Normalize Anthropic pass-through server tool usage - PR #29827
  • Redact provider credentials embedded in fallback error messages - PR #32083
  • Support Responses input in the Redis semantic cache - PR #29581

Management Endpoints / UI

Features

• Virtual Keys & Access Control
  • Admin-gate permissions on /key/update, /key/regenerate, /user/new, /user/update, and bulk key updates - PR #31810, PR #31998, PR #32002
  • Admin-gate allowed_routes presence on /key/update and /key/regenerate - PR #31987
  • Gate non-admin /key/generate budget_limits and permissions - PR #31469
  • Reject team-scoped object_permission on personal keys for non-admins - PR #31471
  • Support object_permission in default_key_generate_params - PR #31776
  • Reject non-finite budget_limits windows and hard-reject CLI session-token personal-key budgets on /key/generate - PR #31630, PR #31631
  • Tighten role gating on the /get/config/callbacks response and extend banned-params + admin-clear lists - PR #31745, PR #31742
  • Audit default-user-settings and remaining system-wide-settings updates - PR #31753, PR #31754
  • JWT auth opt-in fallback to the DB team on an unresolved claim - PR #28913
  • Reject non-existent team/key/model scope entries on policy attachment create - PR #32131
• UI
  • shadcn migration foundation: Tailwind v4, shadcn init, and antd cascade fix - PR #31995
  • Migrate the chat UI from antd to shadcn/ui and add key-management and usage panels - PR #32074
  • Rotate model credentials in a dedicated modal so a normal save can't overwrite secrets - PR #28089
  • Disclaim that the Update API Key modal only rotates api_key - PR #31805
  • Add budget duration to the edit-team-member form - PR #29717
  • Render provider icons on the public model hub - PR #29958

Bugs

• Virtual Keys & Models
  • Show team projects to internal users on key creation - PR #28855
  • Label the default key type as "Full Access" on the key edit page - PR #29870
  • Keep virtual-keys filters across delete and refresh - PR #31533
  • Allow deleting a BYOK model after its team is deleted, and delete a team's BYOK models on team deletion - PR #29875, PR #29977
  • Count only legacy function_call.arguments in the token counter - PR #31741
  • Fix the typo generic_role_mappoings -> generic_role_mappings - PR #29753
• UI
  • Stop the Request Logs page from overflowing horizontally and size its columns - PR #31426
  • Fix the Router Settings Loadbalancing tab save - PR #31735
  • Allow any git host on the skills add form - PR #31652
  • Include cache token columns in the usage export - PR #32015
  • Unify migrated-route URLs and migrate the API Reference page - PR #29953
  • Make the workflow runs page fill full width - PR #29868

AI Integrations

Logging

• Prometheus
  • Add an api_provider label to token, latency, request, and cache metrics - PR #32126
  • Add litellm_overhead_with_guardrails_latency_metric - PR #31593
  • Expose project_alias in custom metadata labels and expose MCP tool metadata - PR #31784, PR #31899
  • Bound per-request budget metric emission with a timeout - PR #31632
• OpenTelemetry
  • Stamp gen_ai.response.time_to_first_chunk on streaming LLM spans - PR #32236
  • Restore error.* span attributes on v2 error spans - PR #32524
• S3
  • Send Content-MD5 on PUT and support optional server-side encryption in the s3 v2 logger - PR #31928
• Microsoft Sentinel
  • Resolve the audit stream from AZURE_SENTINEL_AUDIT_STREAM_NAME - PR #32010
• FOCUS Export
  • Include organization metadata in Vantage FOCUS Tags export and add a GCS destination for FOCUS export - PR #28184, PR #29751
• General
  • Route realtime success logging through the bounded worker - PR #31733
  • Restore the admin key/team callback_vars.turn_off_message_logging override - PR #31905
  • Resolve model_map_value for proxy custom pricing in standard logging - PR #31940
  • Log hashed cache keys - PR #29890
  • Add a Galileo health check for the UI callback test - PR #29908

Guardrails

• Model Armor
  • Scan file and document attachments with Model Armor - PR #31655
• CrowdStrike AIDR
  • Capture user and model metadata, reading identity from both metadata bags - PR #29517, PR #29991
• Headroom
  • Add CCR (compress-cache-retrieve) via an agentic loop - PR #31681
  • Add an unreachable_fallback fail-open option to the headroom guardrail - PR #32026
• General
  • Buffer streamed responses until moderated, with a clean SSE on block - PR #31389
  • Expose streaming knobs on generic_guardrail_api - PR #31730
  • Keep the create-guardrail modal open on outside click and default the guardrails page to the Guardrails tab - PR #29871, PR #29872

Secret Managers

• General
  • AES-256-GCM at-rest credential encryption with a versioned format and a re-encryption migration - PR #31215

Spend Tracking, Budgets and Rate Limiting

Budgets & Fallbacks

• Key-level budget_fallbacks to reroute requests when a per-model budget is exceeded - PR #31783, with UI configuration on the key create/edit forms - PR #32072
• Add a disable_budget_reservation general setting - PR #29493
• Reserve team-budget raises for proxy admins and don't block /team/update on an unchanged budget - PR #30030, PR #29525
• Prevent duplicate budget alert emails on concurrent threshold crossings and apply EMAIL_SIGNATURE to them - PR #32011, PR #31712

Cost Tracking

• Standardize rate-limit errors with category, rate_limit_type, model, and llm_provider fields - PR #27687
• Store the cost breakdown for /v1/realtime sessions - PR #30069
• Track cost for unmanaged Vertex AI batch jobs - PR #31442
• Report real token usage on blocked responses - PR #31217
• Emit the x-litellm-response-cost header on /messages and /generateContent - PR #31675
• Recognize *.cognitiveservices.azure.com as OpenAI-compatible in pass-through cost tracking - PR #29730
• Record agent cost_per_query and input tokens on the A2A native send path - PR #31979
• Count only active users toward the license seat limit - PR #31227
• Log per-token-type reasoning and cache cost breakdown - PR #31623

MCP Gateway

OAuth 2.0 (On-Behalf-Of) v2

• Migrate the token_exchange (OBO) arm to the v2 resolver and make it production-ready with discovery threading, audit hardening, and an RFC 9728 challenge - PR #31526, PR #31622
• Discover the OBO token endpoint via RFC 9728 -> RFC 8414 instead of guessing the IdP - PR #31762
• Persist the DCR client_id so interactive OAuth token refresh works, including on-create Authorize & Fetch - PR #31912, PR #31920
• Resolve per-user OAuth identity authoritatively at the token endpoint - PR #31657
• Support client_secret_basic for upstream OAuth token endpoints and add a token-endpoint auth-method selector in the UI - PR #31635, PR #31739
• Gate OAuth authorize/token/register/discovery on auth_type=oauth2 - PR #31736
• Mirror the upstream token lifetime instead of forcing a 1h OBO expiry - PR #29951
• Reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session - PR #30000
• Let non-creator users OAuth into OBO-mode servers and allow team access-group grants in the authorize/token check - PR #29867, PR #30041

Server Management & Tools

• Add mcp_tool_search virtual tools for large tool catalogs - PR #31777
• Bound outbound tool-call concurrency per MCP server - PR #31641
• Add an all-proxy-mcpservers sentinel to grant teams every MCP server - PR #32012
• Roll up MCP tool spend to user counters and the usage UI - PR #31576
• Hydrate the MCP server registry from the DB on startup when store_model_in_db is false - PR #31775
• Emit a tools/list CLIENT span for MCP discovery under otel_v2 - PR #31525

Bug Fixes

• Stop one unauthenticated server from emptying the aggregate tools/list - PR #31684
• Surface tools/list auth failures as a 401 challenge on single-server routes - PR #31921
• Load MCP tool configuration tools via the OBO/passthrough-aware GET path - PR #29960
• Tighten role-based visibility on /v1/mcp/server/submissions - PR #31932
• Highlight MCP cards red when the logged-in user is missing per-user env vars - PR #29856
• BYOM visibility, preview UX, and admin-settings gating - PR #31809
• Re-add the chat UI and allow a simple UI for MCP OBO auth - PR #31893
• Keep an in-flight OAuth resume from resetting when the create-server modal mounts closed - PR #32416

Performance / Loadbalancing / Reliability improvements

Spend & Auth hot paths

• Gather independent per-scope spend-counter increments - PR #31578
• Move the cost-callback payload deepcopy off the request event loop - PR #31579
• Gather independent pre-call budget-enforcement reads in common_checks - PR #31604
• Memoize the per-request lazy import of OTel runtime hooks - PR #31707
• Load the virtual-keys team filter from the fast v2 endpoint - PR #31638

Reliability

• Keep serving reads from the read replica when the primary DB is down at startup - PR #31951
• Re-establish async Redis-cluster connections after a node restart - PR #31577
• Isolate poison spend-log rows so one bad record can't drop the whole batch - PR #31705
• Stop leaking master_key and database_url in startup DEBUG logs - PR #31944

Routing

• Tag-routing denylist support via a ! prefix - PR #31728
• Declarative fallback generalizations for unknown models - PR #29718
• Skip the health check for semantic auto_router deployments - PR #31668

Build

• Bump the wolfi-base digest for glibc 2.43-r10 - PR #32277

Documentation Updates

• Require a reproduction video for reported issues in the contribution guidelines - PR #30063

PR roll-up by ownership area

PRs by ownership area (total: 226)

• LLM API Endpoints: 30
• UI: 28
• MCP: 28
• Models & Providers: 27
• Other (CI / chore / tests / version bumps): 24
• Auth & Management: 23
• Logging: 22
• Spend / Budgets / Rate Limits: 18
• Guardrails: 12
• Performance: 11
• Docs: 2
• Secret Managers: 1

New Contributors

• @roytev made their first contribution in PR #29565
• @balcsida made their first contribution in PR #29581
• @PigeonMark made their first contribution in PR #29584
• @johngarrido made their first contribution in PR #29623
• @arnav-144p made their first contribution in PR #29753
• @Kaihuang724 made their first contribution in PR #29842
• @fengjikui made their first contribution in PR #29890
• @fernando-izar made their first contribution in PR #31632

Full Changelog

v1.91.0...v1.92.0

Original source

Key Highlights

• MCP Gateway OAuth 2.0 v2 resolver - a new shared OAuth token foundation with cross-replica single-flight refresh, an outbound-credentials package with typed results, and the first authorization_code migration onto the v2 resolver.
• Rust OCR gateway - a new LiteLLM Rust workspace ships an async-first Mistral OCR bridge, packaged directly into the LiteLLM wheel, alongside an experimental Axum-based realtime AI gateway.
• Realtime API performance - upstream connection-pool pre-warming and client-disconnect cancellation cut session-establishment latency and stop wasted upstream work.
• Least-privilege MCP defaults - team keys can now default to least-privilege MCP access, scope to zero MCP servers via a sentinel, and harden client-IP resolution with trusted X-Forwarded-For hop counts.
• ~48 new models - a large Cloudflare Workers AI batch, Gemini 3 image models, Mistral Medium 3.5 / OCR 3 & 4, GLM/zai, SambaNova, and AI/ML image models.

New Providers and Endpoints

New Providers (2 new providers)

Provider Supported LiteLLM Endpoints Description Amazon Bedrock Mantle (bedrock_mantle) Chat Completions Bedrock Mantle support with VPC endpoint routing via api_base, surfaced as its own Add Model provider - PR #31034, PR #31141 OpenSandbox (opensandbox) Sandbox / code interpreter New sandbox provider for the code-interpreter loop - PR #31024

New LLM API Endpoints

Capability Description Documentation Rust OCR (Mistral) A new LiteLLM Rust workspace ships an async-first Mistral OCR bridge, packaged into the LiteLLM wheel - PR #31033, PR #31253, PR #31267 OCR Code interpreter Sandbox code-interpreter interceptor on the Responses API and a chat-completions code-interpreter loop - PR #30905, PR #31027 Sandbox

New Models / Updated Models

New Model Support (~48 new models)

Provider Model Context Input ($/1M) Output ($/1M) Features Gemini / Vertex AI gemini-3-pro-image, gemini-3.1-flash-image (+ gemini/, vertex_ai/ variants) 1M per-image per-image Image generation, GA pricing AI/ML aiml/openai/gpt-image-2 - per-image per-image Image generation Cloudflare Workers AI ~28 text-generation models (Llama 3.x/4, Qwen 2.5/3/QwQ, GLM 4.7/5.2, Kimi K2.6/K2.7, gpt-oss 20b/120b, Gemma, Granite, Nemotron, DeepSeek-R1 distill, Mistral, Llama Guard) varies varies varies Native Workers AI via OpenAI-compatible endpoint Mistral mistral-medium-2508, mistral-medium-2604, mistral-medium-latest (Medium 3.5), mistral-ocr-2512 (OCR 3), mistral-ocr-4-0 (OCR 4) varies varies varies Chat, OCR SambaNova sambanova/DeepSeek-V3.2, sambanova/gemma-4-31B-it varies varies varies Chat zai / OpenRouter zai/glm-4.7-flash, zai/glm-5.1, openrouter/z-ai/glm-5.1 varies varies varies Chat Bedrock amazon.titan-embed-g1-text-02 - embedding - Embeddings Darkbloom darkbloom/gemma-4-26b, darkbloom/gpt-oss-20b varies varies varies Chat

Exact per-model context windows and prices are in model_prices_and_context_window.json.

Features

• Fireworks AI
  • Sync chat completions endpoint with the full Fireworks API surface - PR #30885
• Cloudflare
  • Add current Workers AI text-generation models to the cost map - PR #31051
  • Route the native Workers AI provider through the OpenAI-compatible endpoint - PR #31053
• Mistral
  • Support Mistral OCR 4 (mistral-ocr-4-0) - PR #31353
  • Add mistral/mistral-ocr-2512 (OCR 3) to the cost map - PR #31463
  • Retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases - PR #31373
• AI/ML
  • Add the openai/gpt-image-2 image model - PR #31323
• Rerank
  • Rerank transformation refresh across ~15 providers (Cohere v1/v2, Voyage, Jina, Vertex, Bedrock, Hugging Face, hosted vLLM, DashScope, DeepInfra, NVIDIA NIM, Fireworks, Watsonx) - PR #31185
• DeepSeek / GitHub Copilot / Moonshot
  • Chat transformation updates - PR #31185

Bug Fixes

• Anthropic
  • Sanitize tool_use ids on the native /v1/messages path - PR #31094
  • Drop the unsupported speed param under drop_params - PR #31152
  • Normalize the Messages system role and adaptive-thinking for Claude Invoke - PR #31364
• Bedrock
  • Only expand config-sourced AWS credential references - PR #30867
  • Prevent key-level metadata.tags from leaking into the Bedrock passthrough body - PR #30985
  • Surface web-identity token aud / iss on InvalidIdentityToken - PR #31412
  • Drop the unsupported toolSpec.strict field for Converse on Claude Opus 4.7/4.8 - PR #31582
  • Honor the cache TTL for tool_config cache injection points - PR #31929
• Vertex AI
  • Prevent a stale Vertex bearer token from causing a /v1/messages 401 after token expiry - PR #31276
  • Append the rawPredict suffix for a custom api_base - PR #31529

LLM API Endpoints

Features

• Responses API
  • Code-interpreter interceptor (sandbox) on the Responses API - PR #30905
  • Chat-completions code-interpreter loop - PR #31027
• Realtime API
  • Add an OpenAI realtime translation layer to litellm-rust (1/2) - PR #31129
  • Add a minimal Rust router + Axum AI-gateway calling router.realtime (2/2) - PR #31135
• OCR
  • Make the Rust OCR bridge async-first - PR #31253
  • Add Rust OCR providers - PR #31272
  • Thin Rust OCR Python bridge - PR #31368
• Batches
  • Stream OpenAI to Vertex batch JSONL uploads - PR #31036
• Pass-through
  • Forward all multipart files with repeated field names - PR #31391
  • Schedule spend logging via the durable logging worker - PR #31485
• Web Search
  • Sync tool_choice when converting web_search tools - PR #31375
  • Wrap the agentic-loop response in a fake stream for streaming requests - PR #31484

Bugs

• Realtime API
  • Fix post-tool-call function_response id omission - PR #30446
  • Stop revalidating realtime events at the logging boundary - PR #31054
• General
  • Skip the model override when the response has no model field - PR #31183
  • Recover cost on interrupted and agentic Anthropic streams - PR #31035

Management Endpoints / UI

Features

• Virtual Keys & Teams
  • Scope team BYOK models by key team_id in /model/info - PR #31009
  • Restore wildcard expansion in /v1/model/info - PR #31444
  • Expand the all-proxy-models sentinel in direct-access lookup - PR #31153
  • Persist budget_duration on /team/member_add member budgets - PR #31443
  • Persist budget-window deletion on virtual keys - PR #31107
• SCIM
  • Ingest enterprise-extension attributes into user metadata - PR #30893
  • Drive the global proxy role from a SCIM admin group - PR #30895
• Proxy CLI / Auth
  • Mint a per-session agent credential on lite login - PR #31072
• Config & Plugins
  • LiteLLM plugin architecture v2 - PR #30688
  • Persist the global retry_policy via /config/update - PR #29540
  • Tighten role-based visibility of config and MCP fields - PR #30587
• UI
  • Show an agent's attached virtual key in the UI - PR #29619
  • Add Amazon Bedrock Mantle to the Add Model provider dropdown - PR #31034
  • Clarify OpenAI-compatible provider dropdown labels (chat vs legacy completions) - PR #31046
  • Render logos under a custom server_root_path - PR #31156

Bugs

• UI
  • Keep team Organization optional for proxy admins in single-org setups - PR #30861
  • Stop per-model usage export from duplicating user spend across models - PR #30980
  • Resolve user_id to email in the Spend Per User usage chart - PR #30992
  • Label the request-logs column "Key Alias" to match the filter - PR #31037
  • Stop listing bedrock_mantle models under the Bedrock provider - PR #31478
• Auth & Management
  • Resolve caller identity once into a Principal at the auth seam - PR #30887
  • Cache the auth-path team object under the canonical team_id key - PR #31418
  • Honor user_api_key_cache_ttl for management-object cache writes - PR #31504
  • Reject model_list in the proxy body and gate advisor client credentials - PR #30585
  • Redact the API key from key/info client error messages - PR #31342
  • Stop double-decrypting email/slack alerting env vars in get_config - PR #31117
  • Serialize team budget_limits to JSON in jsonify_team_object - PR #31045
  • Block a server credential leak to a caller-supplied api_base - PR #30682
  • Restore teamless-key access to all-team models - PR #32032

AI Integrations

Logging

• Prometheus
  • Add a requested_model label to spend and request metrics - PR #31410
  • Add a per-team litellm_team_members_metric gauge - PR #31506
• OpenTelemetry
  • Resolve the LITELLM_OTEL_V2 flag once instead of rebuilding settings per call - PR #30989
  • Use a hashable scope for _emit_once when guardrail_mode is a list - PR #31262
  • Point the AgentOps OTLP exporter at otlp.agentops.ai - PR #31490
• General
  • Add POST /v1/callbacks/logs to replay logging payloads through callbacks - PR #31134

Guardrails

• Bedrock Guardrails
  • Select the latest user message by original role in apply_guardrail - PR #30482
• General
  • Add a headroom guardrail for message compression - PR #31407
  • Instrument during-call and post-call guardrail latency - PR #31414
  • Match the policy-pipeline block response to a direct guardrail attachment - PR #31421
  • Make the Generic Guardrail resilient to built-in tools and errors - PR #31461

Spend Tracking, Budgets and Rate Limiting

• Cost tracking
  • Store litellm_call_id on spend logs for DB-to-trace correlation - PR #31344
  • Preserve Anthropic server_tool_use web-search usage in cost tracking - PR #31355
  • Restore per-query Gemini 3.x web-search billing - PR #31363
  • Preserve Gemini Embedding 2 usageMetadata for cost tracking - PR #31354
  • Correct the regional processing uplift to the gpt-5.4/5.5 series only - PR #31136
  • Isolate all per-deployment pricing overrides from sibling deployments - PR #31021
• Spend UI and endpoints
  • Fold the logs-tab total into the page query to avoid a separate COUNT(*) - PR #31423
  • Spend-management endpoint and OpenAI image-generation cost-calculator updates - PR #31185

MCP Gateway

• OAuth 2.0 v2 resolver
  • Shared OAuth token foundation: challenge, store seam, expiry-aware cache, single-flight refresh - PR #31275
  • Scaffold the outbound_credentials package with a typed Result - PR #31047
  • Add a resolve_credentials dispatch skeleton - PR #31056
  • Graft the v2 resolver onto _create_mcp_client (none + api_key static family) - PR #31058
  • Migrate authorization_code MCP to the v2 resolver (single-replica) [1/2] - PR #31473
  • Cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] - PR #31493
  • Challenge delegate-auth OAuth servers with upstream resource_metadata - PR #31255
  • Support client_secret_basic for upstream OAuth token endpoints - PR #31635
• Access control
  • Opt-in least-privilege default for team-key MCP access - PR #31380
  • Scope a key to zero MCP servers with a no-mcp-servers sentinel - PR #31029
  • Allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools - PR #31031
  • Let proxy admins assign MCP servers to teamless keys - PR #31126
  • Resolve config-defined servers in per-user credential and env-var endpoints - PR #31171
• X-Forwarded-For hardening
  • Add mcp_xff_num_trusted_hops to harden X-Forwarded-For client-IP resolution - PR #31257
  • Correct the misleading no-trusted-proxy warning for XFF access control - PR #31264
  • Warn loudly when X-Forwarded-For is present but use_x_forwarded_for is off - PR #31266
• Bug fixes
  • Stop exposing MCP server URLs on the AI Hub and public hub API - PR #30902
  • Stop auth failures on the /mcp path surfacing as cancelled tool calls - PR #31011
  • Resolve toolset tools by the server's known prefix - PR #31254
  • Stop logging tool-call input in the MCP client - PR #31393
  • Persist the DCR client_id from on-create MCP OAuth Authorize & Fetch - PR #31920
  • Persist the DCR client_id so interactive OAuth token refresh works - PR #31912
  • Surface tools/list 401 auth failures as a challenge on single-server routes - PR #31921

Performance / Loadbalancing / Reliability improvements

• Streaming and realtime
  • Pre-warm the upstream realtime connection pool to cut session-establishment latency - PR #31163
  • Cancel the upstream LLM stream when the client disconnects during time-to-first-token - PR #31499
  • Word-sliced cache replay for stream=true cache hits - PR #30216
  • Stop the O(n^2) re-parse of accumulated Gemini stream JSON - PR #31297
  • Surface a clean RateLimitError on a mid-stream 429 with no fallbacks - PR #31298
• Router and timeouts
  • Honor litellm_settings.request_timeout as an independent per-attempt timeout - PR #31119
  • Guard num_retries=None in async_function_with_retries - PR #30036
• Caching and proxy
  • Apply the Redis namespace to all key operations - PR #31288
  • Loop-scope async Lua script registration - PR #31501
  • Memoize _get_all_llm_api_params, rebuilt per request - PR #31430
  • Precompute service-tier cost-key suffixes - PR #31431
  • Bound event-loop blocking from oversized requests - PR #31497
  • Stop the pass-through route registry growing on every reload - PR #31314
  • Strip NUL bytes in safe_dumps only when present - PR #31424
  • Semantic-caching (Redis/Qdrant) and embedding-router updates - PR #31305
• Supply chain and build
  • Bump osv-flagged dependencies to clear known CVEs - PR #31122
  • Bump the wolfi-base digest to patch openssl CVE-2026-34182 - PR #31133
  • Add a Grype image scan for OS and library CVEs - PR #31151
  • Harden cargo fetches during maturin builds - PR #31348
  • Build the Admin UI from source in a build-platform-pinned stage - PR #31130

Documentation Updates

• Add MCP server change guidelines - PR #31038

New Contributors

This release contains changes from existing maintainers only; there are no new contributors in this window.

Full Changelog

v1.90.0...v1.91.0

Original source

v1.89.0 builds on v1.88.0

• Claude Fable 5 is supported across Anthropic, Bedrock, Azure AI, and Vertex at 1M-token context with adaptive thinking and computer use.
• Agent-to-agent (A2A) gains two new agent providers - watsonx Orchestrate and LangFlow (with A2A session bridging) - plus OAuth M2M for Databricks Apps agents.
• MCP gateway adds per-server environment variables with global and per-user scopes, per-server RPM rate limiting for keys and teams, OAuth passthrough with issuer-scoped JWT auth, and oauth2_flow persistence on server registration.
• Observability lands OpenInference rendering parity for Arize/Phoenix (tool calls, cost, passthrough I/O, sessions, multimodal, cache tokens), MCP semantic conventions on the typed OTel v2 spans, and a Galileo logger that uses the ingest-traces API.
• New search and transcription providers - APISerpent, You.com, and Soniox - join the gateway, alongside the dashboard's migration to fully typed, OpenAPI-generated API clients.

MCP Credential Store

This release lets you securely store per-server credentials for MCP servers directly on the gateway. Define variables once on a server, scoped either as Instance (shared across all users) or Per-user (each user supplies their own value), and reference them in static headers or authentication using ${VAR_NAME} syntax (for example, ${DB_PROTOCOL}://${CORP_USERNAME}:${CORP_PASSWORD}@${DB_HOSTNAME}), letting each user connect their own identity.

New Providers and Endpoints

New Providers (3 new providers)

Provider Supported LiteLLM Endpoints Description
APISerpent (apiserpent) Search Web search and deep-search API
You.com (you_com) Search You.com web search API
Soniox (soniox) Audio Transcription Async speech-to-text (stt-async-v4)

New Models / Updated Models

New Model Support (selected)

Provider Model Context Window Input ($/1M tokens) Output ($/1M tokens) Features
Anthropic claude-fable-5 1,000,000 $10.00 $50.00 Adaptive thinking, computer use, function calling, prompt caching, vision
Vertex AI vertex_ai/claude-fable-5 1,000,000 $10.00 $50.00 Same as Anthropic direct
Azure AI azure_ai/claude-fable-5 1,000,000 $10.00 $50.00 Same as Anthropic direct
Bedrock anthropic.claude-fable-5 (+ global. / us. / eu. routes) 1,000,000 $10.00 $50.00 Same as Anthropic direct
Bedrock Mantle bedrock_mantle/openai.gpt-5.5 272,000 $5.50 $33.00 Responses API, reasoning, function calling, prompt caching
Bedrock Mantle bedrock_mantle/openai.gpt-5.4 272,000 $2.75 $16.50 Responses API, reasoning, function calling, prompt caching
Azure AI azure_ai/kimi-k2.6 262,144 $0.95 $4.00 Reasoning, vision, function calling, tool choice
MiniMax minimax/MiniMax-M3 512,000 $0.60 $2.40 Reasoning, prompt caching, function calling
Inception inception/mercury-2 (+ mercury-edit-2) 128,000 $0.25 $0.75 Function calling, prompt caching, response schema

Additional model-map additions: fal.ai Nano Banana and Gemini 2.5 Flash Image generation - PR #29798; mistral/ministral-8b-latest - PR #29453; a batch of new Snowflake Cortex model entries (Claude, GPT, Llama, embeddings); vertex_ai/google/gemma-4-26b-a4b-it-maas; APISerpent, You.com, and Soniox catalog entries; and a jp. regional route for Claude Opus 4.7.

Features

• Anthropic
  • Route future Claude models to the Anthropic provider via pattern matching - PR #29239
  • Route Claude Opus 4.8 through adaptive thinking - PR #29702
  • Emit a thinking block for reasoning_content -only streaming chunks in the Anthropic adapter - PR #29600
  • Inline legacy $ref defs in tool schemas (Anthropic and Fireworks) - PR #28646
• Gemini
  • Support googleSearch with server-side tools and googleMaps JSON schema - PR #29582
  • Use GA event names for Pipecat 1.3.x compatibility on Gemini realtime - PR #29662
• Vertex AI
  • Use a user-supplied api_base as-is for the Model Garden OpenAI-compatible path - PR #29530
  • Handle namespace tools and strip client_metadata for Codex compatibility on Vertex/Anthropic - PR #29489
• Azure AI
  • Strip tool-level extra fields on a 400 and retry - PR #29479

Bug Fixes

• General
  • Return a 400 (not 500) on Anthropic context overflow, and seed identity on failed auth - PR #29848
  • Omit the OpenAI [DONE] sentinel on google-genai streamGenerateContent - PR #29426

LLM API Endpoints

Features

• Batches
  • Skip unnecessary batch input-file reads - PR #29114
  • Resolve credentials correctly when cancelling a managed batch - PR #29734
• Vector Stores
  • Resolve vector-store file-list credentials from team deployments - PR #29739
  • Support an engines URL for Vertex AI Search - PR #27885
  • Forward per-request params to Vertex AI Search - PR #29459
• Realtime
  • Track realtime audio token cost - PR #29722
  • Allow null transcripts in stream logging payloads - PR #29625
  • WebSocket connection improvements - PR #29563

Agents (A2A)

• watsonx Orchestrate agent provider - PR #29410
• LangFlow agent provider with A2A session bridging - PR #28963
• OAuth M2M for Databricks Apps A2A agents - PR #29586
• A2A bug fixes - PR #29566

Management Endpoints / UI

Features

• Virtual Keys & Auth
  • JWT-to-virtual-key mapping - PR #28510
  • Let internal users view search tools - PR #29542
  • Expand the all-team-models sentinel in can_key_call_model for batch validation - PR #29746
• Dashboard
  • Generate dashboard API types from the proxy OpenAPI spec - PR #29816
  • Centralize proxy base-URL resolution into a tested resolver - PR #29793
  • Route networking calls through a shared, location-pinned apiClient - PR #29723, PR #29806, PR #29815
  • Migrate ESLint to flat config and bump eslint-config-next to 16 - PR #29626

Bug Fixes

• Use the resolved DB user_id for spend on legacy email match (JWT) - PR #29217
• Preserve the 401 status for expired JWTs in OTel traces - PR #29510
• Stop team BYOK model-name corruption on model edit - PR #29731
• Drop a deleted team BYOK model name from team.models - PR #29820
• Add default=None to LiteLLM_TeamMembership.litellm_budget_table - PR #29684
• Require a new expiration when regenerating an expired key - PR #29838
• Render caller-supplied filter options in caller order (LIT-3151) - PR #29462
• Make A2A skill tags enterable and validated - PR #29512
• Persist the Tools-tab MCP OAuth token to the DB - PR #29809
• Route MCP playground auth by OAuth2 mode instead of token_url - PR #29714
• Stop MCP playground tool calls from sending twice - PR #29821

AI Integrations

Logging

• Arize / Phoenix
  • OpenInference rendering parity: tool calls, cost, passthrough I/O, session/user, multimodal, and cache tokens - PR #28800
• Datadog
  • Split oversized batches on a 413 instead of re-queueing forever - PR #29444
• Galileo
  • Use the ingest-traces API and the standard logging payload - PR #29651
• OpenTelemetry
  • Allowlist team_metadata sub-keys promoted to baggage - PR #29442
  • Add MCP semantic conventions to OTel v2 - PR #29468
  • Capture 401 error details in management-endpoint spans - PR #29535
  • Emit the missing MCP span attributes - PR #29554
  • Emit a guardrail span on passthrough, including when a guardrail blocks - PR #29552, PR #29470

Guardrails

• Sensitive Data Routing
  • Route sensitive data to on-premise models - PR #29531

Spend Tracking, Budgets and Rate Limiting

• Strip NUL bytes from spend-log payloads to prevent PostgreSQL 22P05 errors - PR #29515
• Scope the session-token team-key budget exemption to a caller-supplied team_id - PR #29641

MCP Gateway

• Per-server environment variables with global and per-user scopes - PR #28917
• Per-MCP-server RPM rate limiting for keys and teams - PR #29482
• Support MCP OAuth passthrough and issuer-scoped JWT auth - PR #28356
• Persist oauth2_flow on MCP server registration - PR #29690
• Clear allowed_tools and tool overrides on MCP server edit - PR #29411
• Gate /public/mcp_hub strictly on litellm.public_mcp_servers - PR #27764

Performance / Loadbalancing / Reliability improvements

• Native /health/drain preStop hook for graceful shutdown - PR #29439
• Disable proxy buffering on streaming SSE responses - PR #29557
• Populate llm_provider on internal rate-limit errors - PR #27707
• Hot-reload .env in dev when running with --reload - PR #29783
• Enable the Helm backend deployment to mount the gateway config.yaml - PR #29605
• Convert the AWS and GCP Terraform stacks into reusable modules - PR #28103
• Terraform GCP: abandon the SQL user on destroy - PR #29855; prompt for image_registry in the DeployStack one-click - PR #29852
• Dependency bumps - PR #29860

Documentation Updates

• Clarify when to create new test files - PR #29472
• Remove fixed dimensions from the README hero image - PR #29496
• CLAUDE.md nits - PR #29504, PR #29749

PR roll-up by ownership area

PRs by ownership area (visible, non-vehicle set; total: 101)

• UI / Dashboard: 22
• General Proxy (testing / CI / build): 22
• Models & Providers: 13
• Performance / Reliability: 10
• Logging: 9
• LLM API Endpoints: 8
• MCP: 6
• Auth & Management: 5
• Agents (A2A): 4
• Docs: 4
• Spend / Budgets / Rate Limits: 2
• Models & Providers (new providers): 3
• Guardrails: 1

New Contributors

A list of contributors who made their first contribution linked to their PRs (not shown here in full).

Original source

v1.88.0 is the stable release, graduated from the v1.88.0 release candidates.

• Claude Opus 4.8 is supported across Anthropic, Bedrock (including global / us / eu / au regional routes), Azure AI, and Vertex, at 1M-token context with adaptive thinking and output_config goal mode.
• MCP access-group authorization was reworked end to end: key and team access groups now resolve to MCP servers, grants are additive with opt-in member assignment, and clients can route through stateful or stateless sessions by session id.
• Typed OpenTelemetry instrumentation lands a semconv-aligned span model that carries team_metadata, http.route, and model names on inference spans.
• Streaming is ~30% cheaper per chunk on the Anthropic and Bedrock hot path.
• Agent-to-agent (A2A) gains well-known agent-card discovery and a LangGraph Platform mode.

New Models / Updated Models

Claude Opus 4.8 across 9 provider routes including Anthropic, Vertex AI, Azure AI, Bedrock (global/regions).

Plus a reasoning-effort flag cleanup across existing Claude catalog entries: supports_minimal_reasoning_effort removed where unsupported, supports_max_reasoning_effort normalized, and a new bedrock_output_config_effort_ceiling (high / xhigh / max) field on Bedrock entries.

Features

highlight additions and improvements across Anthropic, Bedrock, Vertex AI, Azure, OpenAI, Realtime API, A2A, Context Management, Video, Pass-through, Virtual Keys & Teams management, Deployment with Helm, and more.

Bug fixes

include numerous corrections across Anthropic, Vertex AI, Bedrock, Azure, UI, Realtime API, Moderations, Batches, Vector Stores, Video, Pass-through, etc.

LLM API Endpoints

updated with tool calling support and pruning stale flags.

Management Endpoints/UI

improvements including virtual keys and teams, deployment improvements.

AI Integrations

like Logging (DataDog, Galileo, OpenTelemetry semantic conventions) and Guardrails enhancements.

Spend tracking, budgets, and rate limiting

adjusted with regional processing costs uplift, new caps, budgeting enforcement, and counter fixes.

MCP Gateway

enhancements including session routing, access-group resolution, virtual keys management, and CRUD improvements.

Performance/load balancing/reliability

improvements targeting streaming path efficiency, Docker system node usage, and dependency bumps.

Documentation updates

including CLAUDE.md and agent guidance.

General proxy improvements

focusing on testing, CI, build hardening, and deterministic replay.

The release aggregates 97 PRs with a detailed roll-up by ownership area.

Release candidate changelog notes mostly rc.1 contents, with minor patches in rc.2 and rc.3 addressing fixes like video model authorization, key creation on org-scoped teams, output_config flags stripping, pass-through log deduplication, session token budget exemptions, and registry route fixes.

No new contributors this release; all 11 authors are returning contributors.

Full changelog available: https://github.com/BerriAI/litellm/compare/v1.87.0...v1.88.0

Original source

v1.88.0rc3 is the current release candidate for 1.88.0.

New Models / Updated Models

• Claude Opus 4.8 is supported across Anthropic, Bedrock (including global/us/eu/au regional routes), Azure AI, and Vertex, at 1M-token context with adaptive thinking and output_config goal mode.
• MCP access-group authorization was reworked end to end: key and team access groups now resolve to MCP servers, grants are additive with opt-in member assignment, and clients can route through stateful or stateless sessions by session id.
• Typed OpenTelemetry instrumentation lands a semconv-aligned span model that carries team_metadata, http.route, and model names on inference spans.
• Streaming is ~30% cheaper per chunk on the Anthropic and Bedrock hot path.
• Agent-to-agent (A2A) gains well-known agent-card discovery and a LangGraph Platform mode.

New Model Support (Claude Opus 4.8 across 9 provider routes) includes Anthropic claude-opus-4-8, Vertex AI vertex_ai/claude-opus-4-8, Azure AI azure_ai/claude-opus-4-8, Bedrock anthropic.claude-opus-4-8 (+ global./us./eu./au. routes) with context windows up to 1,000,000 tokens (200,000 for Azure AI), input $5.00/1M tokens, output $25.00/1M tokens, and features like vision, function calling, prompt caching, reasoning (adaptive + max/xhigh effort), PDF input, computer use, response schema, tool choice, output_config, and native structured output for Bedrock.

Additional updates include reasoning-effort flag cleanup across existing Claude catalog entries, removal of unsupported supports_minimal_reasoning_effort, normalization of supports_max_reasoning_effort, and a new bedrock_output_config_effort_ceiling (high/xhigh/max) field on Bedrock entries (PR #29238).

Features

• Anthropic: Add Claude Opus 4.8 and prune stale reasoning-effort flags (PR #29238).
• Bedrock: Claude Code goal mode via output_config for Bedrock Opus (PR #28898), support tool search results and chat annotations (PR #29120).

Bug Fixes

• Anthropic: Stop injecting unsupported output_config.effort=xhigh for Claude Code on Sonnet/Opus 4.6 (PR #29304).
• Vertex AI: Strip output_config.effort for Vertex Claude models that reject it (Haiku 4.5) (PR #29585).
• Bedrock: Align toolUse/toolSpec names and allow hyphens (PR #28874).
• Azure: Preserve AD token refresh in the v1 OpenAI client path (PR #28627).
• OpenAI: Fix the double provider-prefix bug on model names (PR #28661).
• General: Hydrate wildcard model-discovery credentials (PR #28284).

LLM API Endpoints Features

• Realtime API: Tool calling for the Gemini and Vertex AI live API (PR #26590).
• A2A: Well-known agent-card discovery and LangGraph Platform mode (PR #28860).
• Context Management: compact_20260112 polyfill so non-Anthropic providers get context compaction (PR #28868).
• Video: Vertex Veo video edit, using DB credentials in the video handlers (PR #29098).
• Pass-through: Extend passthrough_managed_object_ids to Azure (PR #29160).

Bugs

• Realtime API: Send TEXT frames and a valid guardrail session.update (PR #28848).
• Moderations: Wire streaming flags through to the unified dispatcher (PR #27324).
• Batches: Strip LiteLLM policy tracking from OpenAI batch metadata (PR #28425), map the stripped batch body.model back to the proxy alias for auth (PR #29264).
• Vector Stores: Restrict vector store index create/delete to proxy admins (PR #29202).
• Video: Resolve managed video model ids for auth (PR #29545).
• Pass-through: Bedrock Knowledge Base pass-through: preserve SigV4 headers and the signed request body (PR #27526), enforce allowed_passthrough_routes for auth=true pass-through (PR #29256), de-duplicate pass-through endpoint logs (PR #29598), match pass-through registry routes bare-to-bare when SERVER_ROOT_PATH is set, fixing pass-through 404s (PR #29658).

Management Endpoints / UI Features

• Virtual Keys & Teams: Expose keys_count on /v2/team/list and wire the UI Resources badge (PR #28502), allow team members to create keys on org-scoped teams (PR #29310), exempt UI and CLI session tokens from team-key budget ceilings, hardened so custom default_key_generate_params cannot re-impose them (PR #29612, PR #29639), record ownership for service-account keys, plus a Prisma JSON serialization fix (PR #28990).
• Deployment: Helm: split per-component ServiceAccounts for gateway, backend, and UI (PR #28712), Enterprise: RESEND_FROM_EMAIL for self-hosted Resend sends (PR #28830).

Management Endpoints / UI Bugs

• Virtual Keys & Teams: Refresh the team cache on team_model_add/team_model_delete (PR #28683), keep the team_alias cache in sync on _cache_team_object writes (PR #28737), fix spend-logs v2 route permissions (PR #28705), normalize the Bearer prefix in the safe-hash helper (PR #29343).
• UI: Allow clearing custom pricing on wildcard models (PR #28719), stop vertex_ai-anthropic_models from leaking into the Anthropic dropdown (PR #28723), route API Reference back to the query-param page (PR #28726), show 2-decimal precision for max_budget on the key overview (PR #28809), break the logout redirect loop across dev and proxy origins (PR #29360), internal refactors: extract auth state into AuthContext, remove dead App Router scaffolding (PR #28910, PR #28891).

AI Integrations Logging

• DataDog: Drain the cost-management queue and add an opt-in FinOps tag allowlist (PR #28487).
• Galileo: Support the hosted v2 spans API and string output extraction (PR #28771).
• OpenTelemetry: Typed, semconv-aligned instrumentation (PR #28909), add team_metadata, http.route, and model names to inference spans (PR #29319), export the SERVER span on management-endpoint success without an http_request (PR #28794), link pass-through success spans to the SERVER root span (PR #29315).
• General: Exclude proxy_server_request from its own body snapshot (PR #28618), fix duplicate Claude Code traces (PR #29311).

Guardrails

• General: Return HTTP 400 for LiteLLM content-filter blocks (PR #28418), wire apply_guardrail into proxy logging callbacks (PR #28970), persist disable_global_guardrails on keys (PR #29233).

Spend Tracking, Budgets and Rate Limiting

• Cost Tracking — OpenAI regional-processing cost uplift for EU/US data residency (PR #28626).
• Rate Limiting — Cap the no-max_tokens TPM floor at the smallest configured limit (v3 limiter) (PR #28805).
• Budgets — Enforce tag budgets for key-level tags (PR #29108), enforce deployment budgets for dynamically added models (PR #29273), reset_budget writes only {spend, budget_reset_at} and stops pre-zeroing the counter (PR #29358).

MCP Gateway

• Session Routing — Stateless and stateful clients via session-id routing (PR #26857).
• Access Groups — Additive key access-group grants with opt-in member assignment (PR #29313), resolve team access_group_ids to MCP servers (PR #28997), resolve key access_group_ids to MCP servers (ungated) (PR #29195), extend the key access-group union to MCP servers (PR #28890).
• Discovery — Allow llm_api_routes virtual keys to list MCP servers (PR #28442).
• Server CRUD — Preserve source_url on GET /v1/mcp/server list responses (PR #29249), preserve omitted fields on PUT /v1/mcp/server partial updates (PR #29253).
• Virtual Keys — Ignore stale ids on key save (PR #29128).

Performance / Loadbalancing / Reliability improvements

• Streaming hot path — ~30% lower per-chunk overhead on the Anthropic and Bedrock streaming path (PR #28720).
• Docker — Use system Node in the componentized builders and retry apk add (PR #28888).
• Dependencies — Routine dependency bumps, including a Starlette bad-host fix (PR #29208, PR #29373).

Documentation Updates

• Hand-written CLAUDE.md; remove AGENTS.md and point GEMINI.md at it (PR #29252).
• Agent guidance: require consent before writing new third-party names (PR #28908).
• Cookbook: bump the Go directive to 1.26.3 in the gollem example (PR #29234).

General Proxy Improvements

Testing, CI & build hardening:
• UI e2e coverage across roles and flows — Team-BYOK add-model, Router fallback, MCP add-server, AI Hub make-public, Team Admin, Internal User / Viewer, logout and navbar identity (PR #29068, #29069, #29070, #29071, #29072, #29074, #29075, #29076, #29077, #29080, #29083, #28652).
• Pass-through SERVER_ROOT_PATH login-redirect trailing-slash e2e (PR #29369).
• Behavior-pinning harnesses for proxy_server.py (PR #28827, #29309).
• Deterministic Redis cassette replay and live Google OAuth token minting for VCR (PR #28826, #29229).
• Reasoning-effort grid test covering Claude Opus 4.8 across provider routes (PR #29327).
• Bedrock CI account moves and restore (PR #28728, #29326, #29245).
• Keep litellm_internal_staging green (PR #29344).
• Regenerate the admin-ui static export with trailingSlash: true (PR #28112).

PR roll-up by ownership area (total: 97):
• Other (CI / tests / build hardening): 23
• UI / Auth & Management: 18
• LLM API Endpoints: 15
• MCP: 9
• Models & Providers: 9
• Logging: 8
• Spend / Budgets / Rate Limits: 5
• Performance: 4
• Documentation: 3
• Guardrails: 3

Release candidate changelog (rc.1 → rc.2 → rc.3)

Almost everything above shipped in rc.1. The later candidates are small, targeted patches cut by cherry-pick.

rc.2 added six fixes:
• Resolve managed video model ids for auth (PR #29545).
• Allow team members to create keys on org-scoped teams (PR #29310).
• Strip output_config.effort for Vertex Claude Haiku 4.5 (PR #29585).
• De-duplicate pass-through endpoint logs (PR #29598).
• Exempt UI/CLI session tokens from team-key budget ceilings (PR #29612).
• Harden that exemption against custom default_key_generate_params (PR #29639).
rc.3 added one fix:
• Match pass-through registry routes bare-to-bare when SERVER_ROOT_PATH is set, fixing pass-through 404s (PR #29658).

New Contributors

No new contributors this release; all 11 authors are returning contributors.

Full Changelog: https://github.com/BerriAI/litellm/compare/v1.87.0-rc.1...v1.88.0-rc.3

06/04/2026 (v1.88.0rc3)

• New Models / Updated Models: 9
• LLM API Endpoints: 15
• Management Endpoints / UI: 18
• AI Integrations (Logging / Guardrails): 11
• Spend Tracking, Budgets and Rate Limiting: 5
• MCP Gateway: 9
• Performance / Loadbalancing / Reliability improvements: 4
• General Proxy Improvements (testing / CI / build): 23
• Documentation Updates: 3
Total: 97 PRs

Original source

Key Highlights

• OCI Generative AI as a first-class provider — production-ready chat, embeddings, streaming, reasoning and tool use across Cohere Command-A, Meta Llama 3.1/3.2/3.3/4, xAI Grok 3/4, Google Gemini 2.5, and OpenAI GPT-5 hosted on OCI; full model-pricing catalog included.
• Gemini 3.5 Flash Day-0 support — gemini-3.5-flash and gemini-3.1-flash-lite ship on Vertex AI, Google AI Studio, and OpenRouter with full pricing, function calling, web search, code execution, and managed-agents support.
• MCP UI for OAuth tool calls — the dashboard now resolves tool list and tool call against OAuth-protected MCP servers directly, plus native MCP OAuth support for Cursor and clearer OAuth error messages.
• Codex CLI auth hardening — JWT-derived team aliases and SSO form-URL flow for the OpenAI Codex CLI, plus allowlisted OIDC-claim persistence across the CLI SSO poll.
• Anthropic streaming hot-path perf — ~90% lower TTFT overhead and higher sustained throughput on the proxy's Anthropic /v1/messages SSE path, measured on a real 4-pod deployment against both Anthropic and Bedrock Invoke (wire output is parity-tested); plus lazy-loaded response streaming for Bedrock SageMaker.

New Providers and Endpoints

New Providers (1 new provider)

Provider: OCI Generative AI
Supported LiteLLM Endpoints: /v1/chat/completions, /v1/embeddings
Description: Official Oracle Cloud Infrastructure Generative AI integration. Production-ready support for chat, streaming, reasoning, tool calling, and embeddings across Cohere Command-A (incl. Reasoning + Vision), Meta Llama 3.1 / 3.2 / 3.3 / 4, xAI Grok 3 / 4, Google Gemini 2.5, and OpenAI GPT-5. Includes full model-pricing catalog. - PR #28223

New Models / Updated Models

New Model Support (22 new models) including Gemini gemini-3.5-flash and gemini-3.1-flash-lite with extensive features such as audio input, function calling, PDF input, vision, web search, and more.

Features

• Gemini
  • Day-0 support for gemini-3.5-flash - PR #28268
  • Add gemini-3.1-flash-lite model cost map - PR #28320
  • Additional gemini-3.1-flash-lite pricing entry - PR #27933
  • Gemini managed-agents support - PR #28270
• Azure
  • Add Azure Speech STT config support - PR #27482
• OpenRouter
  • Add Xiaomi MiMo-V2.5 and MiMo-V2.5-Pro model entries - PR #27700
  • Add openrouter/google/gemini-3.1-flash-lite pricing entry - PR #28280

Bug Fixes

• Vertex AI
  • Omit function_call.id on Vertex Gemini 3.5+ tool turns (the field is rejected by the new schema) - PR #28324
  • vertex_gemma: strip context_management from the request body - PR #28438
• Bedrock
  • bedrock/cohere: send embedding_types as a JSON array, not a string - PR #28172
  • Sanitize batch metadata to prevent Pydantic ValidationError - PR #28202
  • Decouple STS region from Bedrock aws_region_name - PR #28245
• SageMaker
  • Send the native Cohere embed payload to Cohere SageMaker endpoints - PR #28613
• DeepSeek
  • Use the native /anthropic/v1/messages endpoint and sanitize tools - PR #28200
• Azure
  • Decouple Azure OpenAI deployment ID from model name via base_model so GPT-5 model routing works on custom deployment names - PR #28490
  • Router: use the forwarded model_id for native Azure container IDs - PR #27921
• vLLM
  • Fix Anthropic tool-call transformation on vLLM deployments - PR #28549

LLM API Endpoints

• Interactions API
  • Migrate to the Google Interactions API steps schema (May 2026 revision) - PR #28153
• Google-native passthrough
  • Decode bytes and pass through SSE for Google-native streamGenerateContent (no more b'...' literals on the wire) - PR #28213
• Responses API
  • Forward timeout on the completion-transformation path for Anthropic, Bedrock, and Vertex - PR #28133
  • Accept dict-shape reasoning_effort from the Anthropic Responses bridge - PR #28201
  • Wrap aresponses streaming iterator for mid-stream router fallbacks - PR #28215
  • Unblock staging — mypy + coverage for aresponses streaming fallback - PR #28318
  • Strip Anthropic cache_control from OpenAI Responses API requests - PR #28431
  • Use the OpenAI SSEDecoder for Responses API streaming - PR #28566
  • Replay openai/responses bridge cache hits as chat streams - PR #28158
• Interactions API
  • Never drop streamed text deltas; always emit the terminal completion - PR #28394
• Batch API
  • Normalize batch file IDs before the ManagedObjectTable write - PR #28339

Management Endpoints / UI

• Models + Endpoints
  • Add a pause/resume Switch on the models table - PR #28151
• Spend Logs
  • Consolidate filter state and extract components in the UI - PR #25847
• Playground
  • Interactions API endpoint in the Playground with SSE streaming - PR #28156
• Passthrough Routes
  • Team passthrough routes — create parity + edit-load fix - PR #28098
  • Gate team.allowed_passthrough_routes writes to proxy admins - PR #28097
• Auth / Codex CLI
  • Codex CLI JWT team alias propagation - PR #28621
  • Codex CLI SSO form-URL flow - PR #28271
  • Persist allowlisted OIDC claims in the CLI SSO poll - PR #28463
• Virtual Keys
  • Encrypt callback_vars in key/team metadata at rest in the DB - PR #27141

AI Integrations

• Logging
  • Prometheus
    • Emit per-token-type detail metrics — five sparse counters that break out usage.prompt_tokens_details / usage.completion_tokens_details fields providers already report (LIT-3220) - PR #28372
    • Add user_email and user_alias labels to user budget metrics - PR #28155
  • OpenTelemetry
    • Propagate team_id and team_alias to all child OTEL spans - PR #28273
    • Emit a guardrail span on violations and surface status + categories - PR #28364
    • Serialize guardrail_response to JSON in OTEL traces - PR #28362
    • Stamp http.response.status_code on all error responses - PR #28405
• Guardrails
  • Microsoft Purview DLP
    • New guardrail integration for Microsoft Purview DLP - PR #24966

Spend Tracking, Budgets and Rate Limiting

• Spend Counter — Seed the Redis counter via SET NX to prevent cross-pod double-seed on cold start - PR #27854
• Cost Tracking — Recalculate cost after router retry failures so the logged cost reflects the actual attempt that succeeded - PR #28476
• Cost Tracking — Treat litellm_provider=None as a wildcard in _check_provider_match so cost lookup works for catalog entries that omit the provider field - PR #28523

MCP Gateway

• OAuth in the UI — Add tool-call and tool-list support via the dashboard for OAuth-protected MCP servers - PR #28454
• Cursor OAuth — Allow native MCP OAuth support for Cursor - PR #28327
• Auth Resolution — JWT on tools/list and REST tools/call server resolution - PR #28227
• Cold-Start Init — Forward upstream initialize instructions on cold gateway init - PR #28231
• OAuth Errors — Add error_description and hint to OAuth flow error responses - PR #28471
• Inspector — Trim whitespace from MCP inspector tool-call inputs - PR #28203

Performance / Loadbalancing / Reliability improvements

• Anthropic /v1/messages streaming hot path — cut per-request and per-chunk overhead on the proxy's Anthropic streaming path, with byte-identical wire output guaranteed by parity tests that diff the logged and billed payloads between the fast and legacy paths. Measured on a real 4-pod m7i.xlarge deployment (no HPA) streaming 256 text_delta chunks per request, against both Anthropic and Bedrock Invoke — TTFT overhead ~90% lower with higher sustained throughput (full numbers below) - PR #28289
  • Skip work that's a no-op in the default config: the per-chunk Datadog span when tracing is off, the per-chunk streaming hook when no callback / guardrail / cost-injection is active, and the agentic post-processing wrapper when no callback overrides its hook (it otherwise buffers every chunk and rebuilds the response from SSE just to call hooks that all return (False, {})).
  • Stop doing the same work twice per request: serialize the request body once and reuse it for the pre-call log and the wire, memoize the optional-params type-hint resolution (~80µs/request), and skip the redundant strip_empty_text_blocks scan when the async wrapper already sanitized.
  • Cheaper end-of-stream reconstruction: collapse the homogeneous run of content_block_delta text events into a single equivalent SSE event before stream_chunk_builder, removing O(output-token) ModelResponseStream constructions; tool-use / thinking / citations streams fall back to the unchanged legacy path.
  • Cheaper hot-path logging: gate debug f-string evaluation behind isEnabledFor(DEBUG), hoist cost_injection_active out of the per-chunk loop, and drop one async-generator layer per chunk in async_sse_data_generator.
• Bedrock / SageMaker — Switch to lazy loading for response streaming - PR #28189
• Granian ASGI — Add Granian as a supported ASGI server for better throughput stability - PR #26027
• Prisma — Expose Prisma idle/connect timeout + extra DB URL params so production deployments can tune connection pools - PR #28395
• Proxy auth — Strict media-type match for form bodies (defensive against ambiguous Content-Type) - PR #27939
• Proxy auth — Carry the ASGI path into the WebSocket auth synthetic Request so auth resolves the right route - PR #27940
• Docker — Restore npm to the non-root builder image so UI builds run there - PR #28519
• Helm — Drop the main- prefix from the default image tag - PR #28710
• License check — Read PEP 639 license-expression metadata in check_licenses - PR #28529

Documentation Updates

• Fix the incorrect /v1/agents request example - PR #28131
• Fix misleading credential-passing examples in Gemini-agents GET/DELETE docstrings - PR #28293

General Proxy Improvements

Testing, CI & build hardening:

• Behavior-pinning harness + Key Tier-1 matrix (and tier-2/3 + team management endpoints + phase-4 payload matrix) - PR #28321, PR #28441, PR #28620, PR #28681
• Stabilize image-edit VCR cassettes to stop live gpt-image-1 spend - PR #28110
• Migrate realtime + rerank tests off shut-down upstream models; replace gpt-4o-audio-preview with gpt-audio-1.5; expect session.created as xAI realtime initial event - PR #28191, PR #28281, PR #28424
• Harden the flaky proxy callback-leak detector - PR #28195
• E2E runner migrated to uv; add an "All Proxy Models" key test - PR #28313
• UI-e2e: admin key creation with a specific proxy model; forward LITELLM_LICENSE to the UI e2e proxy - PR #28365, PR #28398
• Vertex AI grounding test tolerates transient 500; streaming test tolerates Vertex 429 wrapped in MidStreamFallbackError - PR #28503, PR #28669
• Bump black to 26.3.1 and reapply formatting; one-shot lint fix - PR #28525, PR #28639
• Allow audio_transcription_config in the model-prices schema - PR #28708
• Remove the dead old Playwright e2e suite - PR #28632
• Routine dependency/CI bumps - PR #28287, PR #28524, PR #28528, PR #27665, PR #28296, PR #28303, PR #28707

PR roll-up by ownership area

PRs by ownership area (total: 93)

• Other (CI / tests / build hardening): 25
• Models & Providers (incl. new provider): 18
• UI / Auth & Management: 12
• LLM API Endpoints: 11
• Performance: 9
• Logging: 6
• MCP: 6
• Spend / Budgets / Rate Limits: 3
• Docs: 2
• Guardrails: 1

New Contributors

• @IshaMeera made their first contribution in #28131
• @TorvaldUtne made their first contribution in #27700
• @adityasingh2400 made their first contribution in #28523
• @cwang-otto made their first contribution in #28133
• @ro31337 made their first contribution in #28280
• @withomasmicrosoft made their first contribution in #28490

Full Changelog: https://github.com/BerriAI/litellm/compare/v1.86.0...v1.87.0

Original source

Key Highlights

• OCI Generative AI as a first-class provider — production-ready chat, embeddings, streaming, reasoning and tool use across Cohere Command-A, Meta Llama 3.1/3.2/3.3/4, xAI Grok 3/4, Google Gemini 2.5, and OpenAI GPT-5 hosted on OCI; full model-pricing catalog included.
• Gemini 3.5 Flash Day-0 support — gemini-3.5-flash and gemini-3.1-flash-lite ship on Vertex AI, Google AI Studio, and OpenRouter with full pricing, function calling, web search, code execution, and managed-agents support.
• MCP UI for OAuth tool calls — the dashboard now resolves tool list and tool call against OAuth-protected MCP servers directly, plus native MCP OAuth support for Cursor and clearer OAuth error messages.
• Codex CLI auth hardening — JWT-derived team aliases and SSO form-URL flow for the OpenAI Codex CLI, plus allowlisted OIDC-claim persistence across the CLI SSO poll.
• Anthropic streaming hot-path perf — ~90% lower TTFT overhead and higher sustained throughput on the proxy's Anthropic /v1/messages SSE path, measured on a real 4-pod deployment against both Anthropic and Bedrock Invoke (wire output is parity-tested); plus lazy-loaded response streaming for Bedrock SageMaker.

New Providers and Endpoints

New Providers (1 new provider)

Provider: OCI Generative AI
Supported LiteLLM Endpoints: /v1/chat/completions, /v1/embeddings
Description: Official Oracle Cloud Infrastructure Generative AI integration. Production-ready support for chat, streaming, reasoning, tool calling, and embeddings across Cohere Command-A (incl. Reasoning + Vision), Meta Llama 3.1 / 3.2 / 3.3 / 4, xAI Grok 3 / 4, Google Gemini 2.5, and OpenAI GPT-5. Includes full model-pricing catalog. - PR #28223

Features

• Gemini
  • Day-0 support for gemini-3.5-flash - PR #28268
  • Add gemini-3.1-flash-lite model cost map - PR #28320
  • Additional gemini-3.1-flash-lite pricing entry - PR #27933
  • Gemini managed-agents support - PR #28270
• Azure
  • Add Azure Speech STT config support - PR #27482
• OpenRouter
  • Add Xiaomi MiMo-V2.5 and MiMo-V2.5-Pro model entries - PR #27700
  • Add openrouter/google/gemini-3.1-flash-lite pricing entry - PR #28280

Bug Fixes

• Vertex AI
  • Omit function_call.id on Vertex Gemini 3.5+ tool turns (the field is rejected by the new schema) - PR #28324
  • vertex_gemma: strip context_management from the request body - PR #28438
• Bedrock
  • bedrock/cohere: send embedding_types as a JSON array, not a string - PR #28172
  • Sanitize batch metadata to prevent Pydantic ValidationError - PR #28202
  • Decouple STS region from Bedrock aws_region_name - PR #28245
• SageMaker
  • Send the native Cohere embed payload to Cohere SageMaker endpoints - PR #28613
• DeepSeek
  • Use the native /anthropic/v1/messages endpoint and sanitize tools - PR #28200
• Azure
  • Decouple Azure OpenAI deployment ID from model name via base_model so GPT-5 model routing works on custom deployment names - PR #28490
  • Router: use the forwarded model_id for native Azure container IDs - PR #27921
• vLLM
  • Fix Anthropic tool-call transformation on vLLM deployments - PR #28549

LLM API Endpoints

• Interactions API
  • Migrate to the Google Interactions API steps schema (May 2026 revision) - PR #28153
• Google-native passthrough
  • Decode bytes and pass through SSE for Google-native streamGenerateContent (no more b'...' literals on the wire) - PR #28213

Bugs

• Responses API
  • Forward timeout on the completion-transformation path for Anthropic, Bedrock, and Vertex - PR #28133
  • Accept dict-shape reasoning_effort from the Anthropic Responses bridge - PR #28201
  • Wrap aresponses streaming iterator for mid-stream router fallbacks - PR #28215
  • Unblock staging — mypy + coverage for aresponses streaming fallback - PR #28318
  • Strip Anthropic cache_control from OpenAI Responses API requests - PR #28431
  • Use the OpenAI SSEDecoder for Responses API streaming - PR #28566
  • Replay openai/responses bridge cache hits as chat streams - PR #28158
• Interactions API
  • Never drop streamed text deltas; always emit the terminal completion - PR #28394
• Batch API
  • Normalize batch file IDs before the ManagedObjectTable write - PR #28339

Management Endpoints / UI

Features

• Models + Endpoints
  • Add a pause/resume Switch on the models table - PR #28151
• Spend Logs
  • Consolidate filter state and extract components in the UI - PR #25847
• Playground
  • Interactions API endpoint in the Playground with SSE streaming - PR #28156
• Passthrough Routes
  • Team passthrough routes — create parity + edit-load fix - PR #28098
  • Gate team.allowed_passthrough_routes writes to proxy admins - PR #28097
• Auth / Codex CLI
  • Codex CLI JWT team alias propagation - PR #28621
  • Codex CLI SSO form-URL flow - PR #28271
  • Persist allowlisted OIDC claims in the CLI SSO poll - PR #28463
• Virtual Keys
  • Encrypt callback_vars in key/team metadata at rest in the DB - PR #27141

Bugs

• Auth / Discovery
  • Hydrate wildcard discovery credentials so OIDC discovery works against wildcarded providers - PR #28284
• Spend Logs
  • Restore the log-filter loading indicator - PR #28282
• End-User Logs
  • Fix end-user logs surfacing - PR #27758

AI Integrations

Logging

• Prometheus
  • Emit per-token-type detail metrics — five sparse counters that break out usage.prompt_tokens_details / usage.completion_tokens_details fields providers already report (LIT-3220) - PR #28372
  • Add user_email and user_alias labels to user budget metrics - PR #28155
• OpenTelemetry
  • Propagate team_id and team_alias to all child OTEL spans - PR #28273
  • Emit a guardrail span on violations and surface status + categories - PR #28364
  • Serialize guardrail_response to JSON in OTEL traces - PR #28362
  • Stamp http.response.status_code on all error responses - PR #28405

Guardrails

• Microsoft Purview DLP
  • New guardrail integration for Microsoft Purview DLP - PR #24966

Spend Tracking, Budgets and Rate Limiting

• Spend Counter — Seed the Redis counter via SET NX to prevent cross-pod double-seed on cold start - PR #27854
• Cost Tracking — Recalculate cost after router retry failures so the logged cost reflects the actual attempt that succeeded - PR #28476
• Cost Tracking — Treat litellm_provider=None as a wildcard in _check_provider_match so cost lookup works for catalog entries that omit the provider field - PR #28523

MCP Gateway

• OAuth in the UI — Add tool-call and tool-list support via the dashboard for OAuth-protected MCP servers - PR #28454
• Cursor OAuth — Allow native MCP OAuth support for Cursor - PR #28327
• Auth Resolution — JWT on tools/list and REST tools/call server resolution - PR #28227
• Cold-Start Init — Forward upstream initialize instructions on cold gateway init - PR #28231
• OAuth Errors — Add error_description and hint to OAuth flow error responses - PR #28471
• Inspector — Trim whitespace from MCP inspector tool-call inputs - PR #28203

Performance / Loadbalancing / Reliability improvements

• Anthropic /v1/messages streaming hot path — cut per-request and per-chunk overhead on the proxy's Anthropic streaming path, with byte-identical wire output guaranteed by parity tests that diff the logged and billed payloads between the fast and legacy paths. Measured on a real 4-pod m7i.xlarge deployment (no HPA) streaming 256 text_delta chunks per request, against both Anthropic and Bedrock Invoke — TTFT overhead ~90% lower with higher sustained throughput (full numbers below) - PR #28289
• Bedrock / SageMaker — Switch to lazy loading for response streaming - PR #28189
• Granian ASGI — Add Granian as a supported ASGI server for better throughput stability - PR #26027
• Prisma — Expose Prisma idle/connect timeout + extra DB URL params so production deployments can tune connection pools - PR #28395
• Proxy auth — Strict media-type match for form bodies (defensive against ambiguous Content-Type) - PR #27939
• Proxy auth — Carry the ASGI path into the WebSocket auth synthetic Request so auth resolves the right route - PR #27940
• Docker — Restore npm to the non-root builder image so UI builds run there - PR #28519
• Helm — Drop the main- prefix from the default image tag - PR #28710
• License check — Read PEP 639 license-expression metadata in check_licenses - PR #28529

Documentation Updates

• Fix the incorrect /v1/agents request example - PR #28131
• Fix misleading credential-passing examples in Gemini-agents GET/DELETE docstrings - PR #28293

General Proxy Improvements

Testing, CI & build hardening:

• Behavior-pinning harness + Key Tier-1 matrix (and tier-2/3 + team management endpoints + phase-4 payload matrix) - PR #28321, PR #28441, PR #28620, PR #28681
• Stabilize image-edit VCR cassettes to stop live gpt-image-1 spend - PR #28110
• Migrate realtime + rerank tests off shut-down upstream models; replace gpt-4o-audio-preview with gpt-audio-1.5; expect session.created as xAI realtime initial event - PR #28191, PR #28281, PR #28424
• Harden the flaky proxy callback-leak detector - PR #28195
• E2E runner migrated to uv; add an "All Proxy Models" key test - PR #28313
• UI-e2e: admin key creation with a specific proxy model; forward LITELLM_LICENSE to the UI e2e proxy - PR #28365, PR #28398
• Vertex AI grounding test tolerates transient 500; streaming test tolerates Vertex 429 wrapped in MidStreamFallbackError - PR #28503, PR #28669
• Bump black to 26.3.1 and reapply formatting; one-shot lint fix - PR #28525, PR #28639
• Allow audio_transcription_config in the model-prices schema - PR #28708
• Remove the dead old Playwright e2e suite - PR #28632
• Routine dependency/CI bumps - PR #28287, PR #28524, PR #28528, PR #27665, PR #28296, PR #28303, PR #28707

PR roll-up by ownership area

PRs by ownership area (total: 93)

• Other (CI / tests / build hardening): 25
• Models & Providers (incl. new provider): 18
• UI / Auth & Management: 12
• LLM API Endpoints: 11
• Performance: 9
• Logging: 6
• MCP: 6
• Spend / Budgets / Rate Limits: 3
• Docs: 2
• Guardrails: 1

New Contributors

• @IshaMeera made their first contribution in #28131
• @TorvaldUtne made their first contribution in #27700
• @adityasingh2400 made their first contribution in #28523
• @cwang-otto made their first contribution in #28133
• @ro31337 made their first contribution in #28280
• @withomasmicrosoft made their first contribution in #28490

Full Changelog: https://github.com/BerriAI/litellm/compare/v1.86.0-rc.1...v1.87.0-rc.1

Original source

New Models / Updated Models

New Model Support

Provider Model Context Window Input ($/1M tokens) Output ($/1M tokens) Deprecation Date OpenAI gpt-audio, gpt-audio-2025-08-28 128K $32/1M audio tokens, $2.5/1M text tokens $64/1M audio tokens, $10/1M text tokens - OpenAI gpt-audio-mini, gpt-audio-mini-2025-08-28 128K $10/1M audio tokens, $0.6/1M text tokens $20/1M audio tokens, $2.4/1M text tokens - Deepinfra, Vertex AI, Google AI Studio, OpenRouter, Vercel AI Gateway gemini-2.0-flash-001, gemini-2.0-flash - - - 2026-03-31 Groq openai/gpt-oss-120b 131K 0.075/1M cache read 0.6/1M output tokens - Groq groq/openai/gpt-oss-20b 131K 0.0375/1M cache read, $0.075/1M text tokens 0.3/1M output tokens - Vertex AI gemini-2.5-computer-use-preview-10-2025 128K $1.25 $10 - Azure AI claude-haiku-4-5 $1.25/1M cache read, $2/1M cache read above 1 hr, $0.1/1M text tokens $5/1M output tokens - Azure AI claude-sonnet-4-5 $3.75/1M cache read, $6/1M cache read above 1 hr, $3/1M text tokens $15/1M output tokens - Azure AI claude-opus-4-5 $6.25/1M cache read, $10/1M cache read above 1 hr, $0.5/1M text tokens $25/1M output tokens - Azure AI claude-opus-4-1 $18.75/1M cache read, $30/1M cache read above 1 hr, $1.5/1M text tokens $75/1M output tokens -

Features

• OpenAI
  • Add gpt-audio and gpt-audio-mini models to pricing - PR #19509
  • correct audio token costs for gpt-4o-audio-preview models - PR #19500
  • Limit stop sequence as per openai spec (ensures JetBrains IDE compatibility) - PR #19562
• VertexAI
  • Docs - Google Workload Identity Federation (WIF) support - PR #19320
• Agentcore
  • Fixes streaming issues with AWS Bedrock AgentCore where responses would stop after the first chunk, particularly affecting OAuth-enabled agents - PR #17141
• Chatgpt
  • Adds support for calling chatgpt subscription via LiteLLM - PR #19030
  • Adds responses API bridge support for chatgpt subscription provider - PR #19030
• Bedrock
  • support for output format for bedrock invoke via v1/messages - PR #19560
• Azure
  • Add support for Azure OpenAI v1 API - PR #19313
  • preserve content_policy_violation details for images (#19328) - PR #19372
  • Support OpenAI-format nested tool definitions for Responses API - PR #19526
• Gemini(Vertex AI, Google AI Studio)
  • use responseJsonSchema for Gemini 2.0+ models - PR #19314
• Volcengine
  • Support Volcengine responses api - PR #18508
• Anthropic
  • Add Support for calling Claude Code Max subscriptions via LiteLLM - PR #19453
  • Add Structured output for /v1/messages with Anthropic API, Azure Anthropic API, Bedrock Converse - PR #19545
• Brave Search
  • New Search provider - PR #19433
• Sarvam ai
  • Add support for new sarvam models - PR #19479
• GMI
  • add GMI Cloud provider support - PR #19376

Bug Fixes

• Anthropic
  • Fix anthropic-beta sent client side being overridden instead of appended to - PR #19343
  • Filter out unsupported fields from JSON schema for Anthropic's output_format API - PR #19482
• Bedrock
  • Expose stability models via /image_edits endpoint and ensure proper request transformation - PR #19323
  • Claude Code x Bedrock Invoke fails with advanced-tool-use-2025-11-20 - PR #19373
  • deduplicate tool calls in assistant history - PR #19324
  • fix: correct us.anthropic.claude-opus-4-5 In-region pricing - PR #19310
  • Fix request validation errors when using Claude 4 via bedrock invoke - PR #19381
  • Handle thinking with tool calls for Claude 4 models - PR #19506
  • correct streaming choice index for tool calls - PR #19506
• Ollama
  • Fix tool call errors due with improved message extraction - PR #19369
• VertexAI
  • Removed optional vertex_count_tokens_location param before request is sent to vertex - PR #19359
• Gemini(Vertex AI, Google AI Studio)
  • Supports setting media_resolution and fps parameters on each video file, when using Gemini video understanding - PR #19273
  • handle reasoning_effort as dict from OpenAI Agents SDK - PR #19419
  • add file content support in tool results - PR #19416
• Azure
  • Fix Azure AI costs for Anthropic models - PR #19530
• Giga Chat
  • Add tool choice mapping - PR #19645

AI API Endpoints (LLMs, MCP, Agents)

Features

• Files API
  • Add managed files support when load_balancing is True - PR #19338
• Claude Plugin Marketplace
  • Add self hosted Claude Code Plugin Marketplace - PR #19378
• MCP
  • Add MCP Protocol version 2025-11-25 support - PR #19379
  • Log MCP tool calls and list tools in the LiteLLM Spend Logs table for easier debugging - PR #19469
• Vertex AI
  • Ensure only anthropic betas are forwarded down to LLM API (by default) - PR #19542
  • Allow overriding to support forwarding incoming headers are forwarded down to target - PR #19524
• Chat/Completions
  • Add MCP tools response to chat completions - PR #19552
  • Add custom vertex ai finish reasons to the output - PR #19558
  • Return MCP execution in /chat/completions before model output during streaming - PR #19623

Bugs

• Responses API
  • Fix duplicate messages during MCP streaming tool execution - PR #19317
  • Fix pickle error when using OpenAI's Responses API with stream=True and tool_choice of type allowed_tools (an OpenAI-native parameter) - PR #17205
  • stream tool call events for non-openai models - PR #19368
  • preserve tool output ordering for gemini in responses bridge - PR #19360
  • Add ID caching to prevent ID mismatch text-start and text-delta - PR #19390
  • Include output_item, reasoning_summary_Text_done and reasoning_summary_part_done events for non-openai models - PR #19472
• Chat/Completions
  • fix: drop_params not dropping prompt_cache_key for non-OpenAI providers - PR #19346
• Realtime API
  • disable SSL for ws:// WebSocket connections - PR #19345
• Generate Content
  • Log actual user input when google genai/vertex endpoints are called client-side - PR #19156
• /messages/count_tokens Anthropic Token Counting
  • ensure it works for Anthropic, Azure AI Anthropic on AI Gateway - PR #19432
• MCP
  • forward static_headers to MCP servers - PR #19366
• Batch API
  • Fix: generation config empty for batch - PR #19556
• Pass Through Endpoints
  • Always reupdate registry - PR #19420

Management Endpoints / UI

Features

• Cost Estimator
  • Fix model dropdown - PR #19529
• Claude Code Plugins
  • Allow Adding Claude Code Plugins via UI - PR #19387
• Guardrails
  • New Policy management UI - PR #19668
  • Allow adding policies on Keys/Teams + Viewing on Info panels - PR #19688
• General
  • respects custom authentication header override - PR #19276
• Playground
  • Button to Fill Custom API Base - PR #19440
  • display mcp output on the play ground - PR #19553
• Models
  • Paginate /v2/models/info - PR #19521
  • All Model Tab Pagination - PR #19525
  • Adding Optional scope Param to /models - PR #19539
  • Model Search - PR #19622
  • Filter by Model ID and Team ID - PR #19713
• MCP Servers
  • MCP Tools Tab Resetting to Overview - PR #19468
• Organizations
  • Prevent org admin from creating a new user with proxy_admin permissions - PR #19296
• Teams
  • Reusable Model Select - PR #19543
  • [Fix] Team Update with Organization having All Proxy Models - PR #19604
• Logs
  • Include tool arguments in spend logs table - PR #19640
• Fallbacks / Loadbalancing
  • New fallbacks modal - PR #19673
  • Set fallbacks/loadbalancing by team/key - PR #19686

Bugs

• Playground
  • increase model selector width in playground Compare view - PR #19423
• Virtual Keys
  • Sorting Shows Incorrect Entries - PR #19534
• General
  • UI 404 error when SERVER_ROOT_PATH is set - PR #19467
  • Redirect to ui/login on expired JWT - PR #19687
• SSO
  • Fix SSO user roles not updating for existing users - PR #19621
• Guardrails
  • ensure guardrail patterns persist on edit and mode toggle - PR #19265

AI Integrations

Logging

• General Logging
  • prevent printing duplicate StandardLoggingPayload logs - PR #19325
  • Fix: log duplication when json_logs is enabled - PR #19705
• Langfuse OTEL
  • ignore service logs and fix callback shadowing - PR #19298
• Langfuse
  • Send litellm_trace_id - PR #19528
  • Add Langfuse mock mode for testing without API calls - PR #19676
• GCS Bucket
  • prevent unbounded queue growth due to slow API calls - PR #19297
  • Add GCS mock mode for testing without API calls - PR #19683
• Responses API Logging
  • Fix pydantic serialization error - PR #19486
• Arize Phoenix
  • add openinference span kinds to arize phoenix - PR #19267
• Prometheus
  • Added new prometheus metrics for user count and team count - PR #19520

Guardrails

• Bedrock Guardrails
  • Ensure post_call guardrail checks input+output - PR #19151
• Prompt Security
  • fixing prompt-security's guardrail implementation - PR #19374
• Presidio
  • Fixes crash in Presidio Guardrail when running in background threads (logging_hook) - PR #19714
• Pillar Security
  • Migrate Pillar Security to Generic Guardrail API - PR #19364
• Policy Engine
  • New LiteLLM Policy engine - create policies to manage guardrails, conditions - permissions per Key, Team - PR #19612
• General
  • add case-insensitive support for guardrail mode and actions - PR #19480

Prompt Management

• General
  • fix prompt info lookup and delete using correct IDs - PR #19358

Secret Manager

• AWS Secret Manager
  • ensure auto-rotation updates existing AWS secret instead of creating new one - PR #19455
• Hashicorp Vault
  • Ensure key rotations work with Vault - PR #19634

Spend Tracking, Budgets and Rate Limiting

• Pricing Updates
  • Add openai/dall-e base pricing entries - PR #19133
  • Add input_cost_per_video_per_second in ModelInfoBase - PR #19398

Performance / Loadbalancing / Reliability improvements

• General
  • Fix date overflow/division by zero in proxy utils - PR #19527
  • Fix in-flight request termination on SIGTERM when health-check runs in a separate process - PR #19427
  • Fix Pass through routes to work with server root path - PR #19383
  • Fix logging error for stop iteration - PR #19649
  • prevent retrying 4xx client errors - PR #19275
  • add better error handling for misconfig on health check - PR #19441
• Router
  • Fix Azure RPM calculation formula - PR #19513
  • Persist scheduler request queue to redis - PR #19304
  • pass search_tools to Router during DB-triggered initialization - PR #19388
  • Fixed PromptCachingCache to correctly handle messages where cache_control is a sibling key of string content - PR #19266
• Memory Leaks/OOM
  • prevent OOM with nested $defs in tool schemas - PR #19112
  • fix: HTTP client memory leaks in Presidio, OpenAI, and Gemini - PR #19190
• Non root
  • fix logfile and pidfile of supervisor for non root environment - PR #17267
  • resolve Read-only file system error in non-root images - PR #19449
• Dockerfile
  • Redis Semantic Caching - add missing redisvl dependency to requirements.txt - PR #19417
  • Bump OTEL versions to support a2a dependency - resolves modulenotfounderror for Microsoft Agents by @Harshit28j in #18991
• DB
  • Handle PostgreSQL cached plan errors during rolling deployments - PR #19424
• Timeouts
  • Fix: total timeout is not respected - PR #19389
• SDK
  • Field-Existence Checks to Type Classes to Prevent Attribute Errors - PR #18321
  • add google-cloud-aiplatform as optional dependency with clear error message - PR #19437
  • Make grpc dependency optional - PR #19447
  • Add support for retry policies - PR #19645
• Performance
  • Cut chat_completion latency by ~21% by reducing pre-call processing time - PR #19535
  • Optimize strip_trailing_slash with O(1) index check - PR #19679
  • Optimize use_custom_pricing_for_model with set intersection - PR #19677
  • perf: skip pattern_router.route() for non-wildcard models - PR #19664
  • perf: Add LRU caching to get_model_info for faster cost lookups - PR #19606

General Proxy Improvements

Doc Improvements

• new tutorial for adding MCPs to Cursor via LiteLLM - PR #19317
• fix vertex_region to vertex_location in Vertex AI pass-through docs - PR #19380
• clarify Gemini and Vertex AI model prefix in json file - PR #19443
• update Claude Code integration guides - PR #19415
• adjust opencode tutorial - PR #19605
• add spend-queue-troubleshooting docs - PR #19659
• docs: add litellm-enterprise requirement for managed files - PR #19689

Helm

• Add support for keda in helm chart - PR #19337
• sync Helm chart version with LiteLLM release version - PR #19438
• Enable PreStop hook configuration in values.yaml - PR #19613

General

• Add health check scripts and parallel execution support - PR #19295

New Contributors

• @dushyantzz made their first contribution in PR #19158
• @obod-mpw made their first contribution in PR #19133
• @msexxeta made their first contribution in PR #19030
• @rsicart made their first contribution in PR #19337
• @cluebbehusen made their first contribution in PR #19311
• @Lucky-Lodhi2004 made their first contribution in PR #19315
• @binbandit made their first contribution in PR #19324
• @flex-myeonghyeon made their first contribution in PR #19381
• @Lrakotoson made their first contribution in PR #18321
• @bensi94 made their first contribution in PR #18787
• @victorigualada made their first contribution in PR #19368
• @VedantMadane made their first contribution in #19266
• @stiyyagura0901 made their first contribution in #19276
• @kamilio made their first contribution in PR #19447
• @jonathansampson made their first contribution in PR #19433
• @rynecarbone made their first contribution in PR #19416
• @jayy-77 made their first contribution in #19366
• @davida-ps made their first contribution in PR #19374
• @joaodinissf made their first contribution in PR #19506
• @ecao310 made their first contribution in PR #19520
• @mpcusack-altos made their first contribution in PR #19577
• @milan-berri made their first contribution in PR #19602
• @xqe2011 made their first contribution in #19621

Full Changelog

View complete changelog on GitHub

Original source

v1.85.1 is a patch release on top of v1.85.0. It adds day-0 support for Gemini 3.5 Flash and ships two reliability fixes — cross-pod spend accuracy and Vertex AI tool calling.

New Models / Updated Models

New Model Support (1 new model)

Provider Model Context Window Input ($/1M tokens) Output ($/1M tokens) Features Gemini / Vertex AI gemini/gemini-3.5-flash, vertex_ai/gemini-3.5-flash 1M $1.50 $9.00 Reasoning, vision, audio input, PDF input, prompt caching, web search, function calling, response schema

Features

• Gemini / Vertex AI
  • Day-0 support for Gemini 3.5 Flash on both Google AI Studio and Vertex AI - PR #28268

Bug Fixes

• Vertex AI
  • Omit the function_call / function_response id on Vertex Gemini 3.5+ tool turns, fixing HTTP 400 Unknown name "id" errors. Google AI Studio (gemini provider) still forwards the id on Gemini 3.5+ for strict tool-call matching - PR #28324

Spend Tracking, Budgets and Rate Limiting

• Seed the Redis spend counter via SET NX instead of INCRBYFLOAT to prevent cross-pod double-seeding. On multi-pod deployments this previously caused team spend to jump to ~Nx the pod count after a Redis cache miss / TTL expiry, triggering false "Budget Crossed" alerts - PR #27854

Full Changelog

https://github.com/BerriAI/litellm/compare/v1.85.0...v1.85.1

Original source