AssemblyAI Release Notes

LLM Gateway completions now support a post-processing pipeline, and the first step available is json-repair — an optional pass that automatically fixes malformed JSON returned by a model before it reaches your application. Enable it with a single new parameter on your existing request.

Anyone working with structured output or tool calling has seen the failure mode: the model returns JSON with a trailing comma, an unescaped quote, a missing brace, or a stray markdown fence — and your downstream parser blows up on a response that was 99% correct. json-repair catches these errors at the Gateway layer and returns clean, parseable JSON to your client, so you don't have to ship your own repair logic, retry the call, or wrap every parse in a try/except.

The new post_processing_steps field is designed to be extensible — JSON repair is the first transformation we support, with more steps to come. Steps run in order on the model's completion before the response is returned, so you can compose them into a deterministic post-processing pipeline that works the same across every model in the Gateway.

How to use it

Add a post_processing_steps array to your LLM Gateway request with {"type": "json-repair"}:

{
  "model": "gemini-2.5-flash-lite",
  "messages": [
    {
      "role": "user",
      "content": "return exactly with no extra characters, do not fix the json: {\"name\":\"extra comma\",}"
    }
  ],
  "post_processing_steps": [{"type": "json-repair"}]
}

• Works with every model available through LLM Gateway — no model-specific configuration needed
• Steps execute in the order they appear in the array, so future steps will compose predictably
• Available now for all LLM Gateway users in every region

AssemblyAI's LLM Gateway gives you a single API to access 20+ models from Claude, GPT, Gemini, and more — swap models with a single parameter change, with built-in fallbacks, prompt caching, and now post-processing baked in.

How to use it

We've shipped a major upgrade to streaming speaker diarization, with significant accuracy gains and a refined API that delivers per-word speaker labels. The new model is live now in production across both US and EU regions for Universal-3 Pro Streaming and Universal-Streaming — no integration changes required to benefit from the accuracy improvements.

Across our internal benchmarks, the upgrade reduces false-alarm speakers by 66% and phantom turn rate by 60%, while improving cpWER by 12% overall and 24% on 2-speaker conversations. Against the closest competitive alternative (Deepgram Nova-3), the new model delivers 2x better cpWER on 2-speaker telephony, 13% better cpWER on 4-speaker meetings, 43% fewer false-alarm speakers, and 91% fewer phantom turns and words attributed to non-existent speakers.

Alongside the accuracy gains, each word object within a Turn now carries its own speaker label, enabling much more refined mid-turn speaker change detection. Previously, every word in a Turn inherited the Turn's speaker_label; now, when a different speaker briefly cuts in mid-turn, the individual word objects reflect that change — and words the model can't confidently attribute are tagged UNKNOWN rather than rolled into the dominant speaker. This unlocks accurate attribution in fast back-and-forths, brief interjections, and noisy multi-speaker calls where speakers overlap or trade off mid-sentence.

• Live now in production across US and EU regions for Universal-3 Pro Streaming and Universal-Streaming — no config changes required to get the accuracy improvements
• Each word in a Turn message now includes a speaker field alongside start, end, text, confidence, and word_is_final
• Words the model cannot confidently attribute to a known speaker are labeled UNKNOWN — opt into per-word attribution by reading from words[].speaker
• The Turn-level speaker_label field is unchanged, so existing integrations continue to work without modification
• For best-in-class diarization accuracy, we recommend Universal-3 Pro Streaming ("speech_model": "u3-rt-pro")

AssemblyAI's Universal-Streaming API is the most accurate, lowest-latency way to build real-time voice applications — and with this upgrade, it now delivers the most precise speaker attribution in production speech AI.

The Voice Agent API is now available — a complete voice agent pipeline built on AssemblyAI's own models, delivered through a single WebSocket. Stream audio in, get audio back, and pay one all-in rate of $4.50/hr that covers speech understanding, LLM reasoning, and voice generation.

The API runs on Universal-3 Pro Streaming, the same speech model that already powers production voice stacks — accurate on names, account numbers, domain terminology, and accented speech across six languages. Turn detection runs server-side with configurable thresholds, so the agent knows the difference between a thinking pause and an end-of-turn, and interruptions stop the agent immediately. Listening that actually works is the foundation; everything downstream gets better when the transcription and turn-taking are right.

The developer experience is designed to get out of the way. No SDK to install, no framework to learn — the entire API surface is JSON over WebSocket and most teams ship a working agent the same afternoon they start. Live configuration lets you update system prompts, tools, or turn detection mid-conversation with no reconnect. Tool calling with JSON Schema lets the agent take real actions through your custom functions. Session resumption restores full context if a WebSocket drops within 30 seconds.

How to use it

• Open a WebSocket connection to the Voice Agent API endpoint and stream audio in; receive audio and event messages back as JSON
• Configure agent behavior at session start or mid-conversation — system prompt, tools, turn detection thresholds — via standard JSON message types
• Register custom functions with JSON Schema for tool calling; reconnect within 30 seconds with session resumption to preserve context on dropped connections
• Single billing line at $4.50/hr covering STT, LLM, and TTS — measured in audio hours, no separate metering for each pipeline stage
• Available now to all customers; works end-to-end with Claude Code for scaffolding integrations directly from your terminal when using our

AssemblyAI Docs MCP

The Voice Agent API is invisible infrastructure for production voice products — accurate listening, natural turn-taking, and a developer surface small enough to read in 10 minutes. Your customers should feel like you built it for them, not like they're using a platform.

You can now retrieve both the redacted and unredacted versions of a transcript in a single PII Redaction request. Set the new redact_pii_return_unredacted flag to true in your POST /v2/transcript body, and the response will include the original text, words, and utterances alongside the redacted output — no second API call required.

The new fields are purely additive. text, words, and utterances stay fully redacted as before, and three new top-level fields — unredacted_text, unredacted_words, and unredacted_utterances — are returned alongside them with the original PII intact. The unredacted word and utterance arrays mirror the exact shape of their redacted counterparts (text, start, end, confidence, speaker, channel).

This is an opt-in convenience for workflows that need both versions in the same place — for example, a UI that toggles between redacted-first and unredacted views, or a dual-pipeline that stores compliance-grade redacted output for sharing while preserving the original in a trusted environment. It removes the need for previously brittle workarounds like sending two API requests, doing client-side redaction via Entity Detection, or post-hoc LLM-based redaction.

How to use it

Add redact_pii_return_unredacted: true alongside the existing PII parameters in your transcription request:

{
  "audio_url": "YOUR_AUDIO_URL",
  "redact_pii": true,
  "redact_pii_return_unredacted": true,
  "redact_pii_policies": ["person_name", "phone_number", "email_address"],
  "redact_pii_sub": "entity_name"
}

• Requires redact_pii: true — sending redact_pii_return_unredacted: true on its own returns HTTP 400
• Defaults to false; when off or absent, responses are unchanged and the three unredacted_* fields are not returned
• Works with all existing PII params, including redact_pii_policies, redact_pii_sub, and redact_pii_audio
• Available now on Pre-recorded transcription, with SDK support live across Python and JavaScript

AssemblyAI's PII Redaction automatically detects and removes sensitive information from both transcripts and audio — giving you compliant, production-ready output without extra processing steps.

LLM Gateway now supports automatic model fallbacks, giving your application resilience against model failures without changing your integration. If a model returns a server error, the Gateway will automatically retry with a fallback — or retry the same model after 500ms by default.

This is available now in Public Beta for all LLM Gateway users.

How to use it

Add a fallbacks array and optional fallback_config to your request. All fields from the original request are copied over to the fallback automatically — you only need to specify what you want to override.

Simple fallback — fall back to a different model, inheriting all original parameters:

{
  "model": "kimi-k2.5",
  "messages": [
    {
      "role": "user",
      "content": "Summarize this meeting: ..."
    }
  ],
  "temperature": 0.2,
  "fallbacks": [
    {
      "model": "claude-sonnet-4-6"
    }
  ]
}

Advanced fallback — override specific parameters when falling back (e.g., different prompt or temperature for a different model's behavior):

{
  "model": "kimi-k2.5",
  "messages": [
    {
      "role": "user",
      "content": "Summarize this meeting: ..."
    }
  ],
  "temperature": 0.2,
  "fallbacks": [
    {
      "model": "claude-sonnet-4-6",
      "messages": [
        {
          "role": "user",
          "content": "Summarize this meeting concisely, key info only: ..."
        }
      ],
      "temperature": 0.3
    }
  ]
}

Fallback config options:

"fallback_config": {
  "depth": 1,
  "retry": true
}

By default, if no fallbacks are set, the API will automatically retry a failed request after 500ms. For more control, set fallback_config.retry to false and implement your own exponential backoff.

AssemblyAI's LLM Gateway gives you a single API to access leading models from every major provider — with built-in resilience, load balancing, and cost tracking.