What's coming next across the AssemblyAI platform — from the speech models we build
to the APIs and tooling developers integrate — and what shipped in the last 90 days.
The next Universal-3.x release. Native-language coverage will grow from 18 to over 30
languages, with accuracy gains on already supported languages. A new model option
priced the same as Universal-3.5.
A standalone text-to-speech model for production voice workloads. Low
time-to-first-byte, voice prompting, and accurate delivery of phone numbers, emails,
and named entities that today’s TTS struggles with.
The next major accuracy and capability release after Universal-3.x, across
pre-recorded and realtime audio. Targets the lowest turn latency and the strongest
handling of voice-agent audio (noise, interruptions, hesitation, accented speech),
with instruction-following strong enough to replace today’s STT + LLM + TTS stack. The
foundation for our speech-to-speech architecture.
A single native model that replaces today’s Voice Agent pipeline (STT, LLM, TTS) with
a unified Realtime Speech LLM. Tighter latency, better prosody, and more natural
interruption handling than orchestrated stacks.
Entity-Aware EndpointingEndpointing now detects when a turn ends mid-entity — a phone number, an account
ID, a spelled-out name — and waits instead of cutting, with a new end-of-turn
confidence field. Turn-taking benchmarks improved 12%, phone-number endpointing
40.7%, and emails 14.4%. On by default.
In-House TTS VoicesOur first AssemblyAI-built TTS voices are live in production behind the Voice Agent
API voice catalog, selectable by voice_id in the docs voice reference and the
playground.
Universal-Async 3.5 & Universal-Realtime 3.5Our flagship speech model, on pre-recorded and realtime audio. Native-language
coverage grew from 6 to 18 languages with mid-sentence code-switching (WER dropped
from 9.07% to 7.69%, the best accuracy among all models tested) and stronger speaker
diarization performance. Realtime runs at sub-300ms latency with fully formatted
final transcripts, takes direction from your agent via context carryover, and is the
only model in the Human Parity Zone on Coval’s independent benchmark. New languages:
Japanese, Vietnamese, Arabic, Dutch, Swedish, Hindi, Norwegian, Finnish, Danish,
Hebrew, Mandarin, Turkish.
Context CarryoverUniversal-Realtime 3 carries prior finalized turns forward as context to improve
accuracy, on by default. Optionally pass your voice agent’s spoken reply via
agent_context so the model knows the question the user is answering.
Voice FocusRealtime noise suppression for voice agents and telephony, so accuracy holds up in
real call-center conditions with no separate preprocessor.
Streaming Speaker RevisionAn end-of-stream SpeakerRevision message returns corrected speaker labels at
async-parity cpWER, for roughly 400ms of added latency.
Ongoing catalog expansion across Anthropic, OpenAI, Google, and Qwen models, with
DeepSeek and other open-weight models through new inference providers next.
qwen3.5-4b-32k-fastNew open source model in the LLM Gateway catalog optimized for fast voice AI
workloads.
Gemini 3.6 Flash, Gemini 3.5 Flash-LiteTwo new Gemini models in the LLM Gateway catalog, served from the regional
endpoints.
Compressed Audio InputThe streaming WebSocket accepts compressed Opus and AAC audio alongside PCM. AAC
streams are roughly six times lighter on network I/O than PCM for the same
audio.
Faster Time-to-FinalFinal transcripts return roughly twice as fast on forced endpoints — median 427ms
to 285ms, P99 899ms to 443ms — with no accuracy cost, via speculative decoding. Now
#1 on time-to-first-token on Coval’s public benchmark.
Sync STT APISynchronous HTTP transcription: POST a short audio clip and get the finished
Universal-Async 3.5 transcript back in the same response, ~134ms median latency.
Clips up to two minutes, conversation context for multi-turn voice flows, and a
/warm endpoint to keep connection setup off the critical path.
Claude Sonnet 5, GPT-5.6 Luna & TerraThree new frontier models added to the LLM Gateway catalog, with prompt-cache
support on the GPT-5.6 models.
Faster Turnaround TimeUniversal-Async 3 transcription is significantly faster end-to-end. Between early
April and the end of June, median turnaround time dropped 11-43% across
audio-duration cohorts and P99 turnaround time dropped 26-59%.
Streaming Modesmin_latency, balanced, and max_accuracy presets to tune the latency/accuracy
trade-off per workload.
Global RoutingAn opt-in model_region: global setting that routes calls to lower-cost
capacity.
Claude Opus 4.8, Gemini 3.5 Flash, Gemini 3.1 Flash Lite (GA)Three new models added to the LLM Gateway catalog, available through the Gateway on
day one.
Purchase phone numbers directly through the AssemblyAI API — no third-party
vendor required — and handle inbound and outbound calls with DTMF keypad
support. A SIP integration guide with no bridge required is already live.
Make backend requests while the phone is still ringing, before the call connects. The
caller's phone number is included in the request, so you can run a CRM lookup and use
the result to decide whether to answer the call at all — or to greet the caller
by name with a dynamic, personalized greeting.
Session Recordings and TranscriptsA sessions API returns recordings and transcripts for completed voice agent
sessions, so you can surface call history in your own application instead of storing
it yourself.
Agents as CodeCreate, update, and version agent configurations through a REST management API,
subscribe to session and call lifecycle webhooks, and let AssemblyAI execute your
HTTP tools server-side, with JSON-Schema parameter hints that validate spoken values
before a tool runs.
Accuracy improvements to Speaker ID, Translation, and Custom Formatting. Translation
covers both streaming and pre-recorded audio, for workflows where the spoken language
differs from the output.
Regional English LocalizationLanguage detection can render English transcripts in a regional spelling variant
(en_au, en_uk, and more) and return the region-aware language code, without giving
up automatic detection or code-switching.
Language-Aware Text NormalizationNumbers, dates, and abbreviations are normalized per language across all six
supported TTS languages, rather than applying English rules everywhere.
Per-Session STT ControlsTune speech recognition per voice agent session: transcription modes for the
latency/accuracy trade-off, a contextual transcription prompt, language steering via
language_codes, and Voice Focus noise suppression. Agent replies also stream word by
word for live captions.
Word-Level TTS TimestampsStreaming TTS returns per-word start and end times for generated audio, so a voice
agent knows exactly how much of its reply the caller heard before
interrupting.
Summarization and Action Items (Open Beta)Chaptered summaries with timestamps, and follow-up tasks with supporting quotes,
powered by frontier models — replacing legacy Summarization and Auto Chapters. An
effort control handles harder transcripts. Available in US and EU regions.
Bring Your Own LLMPoint a voice agent at any OpenAI-compatible endpoint — your own model, fine-tune,
or provider account — as the brain of the agent.
Static Entity RedactionRedact your own terms — codenames, project names, account identifiers — by passing
entity names with example strings. Matching spans are removed from both the
transcript and the audio, on top of standard PII redaction.
Speaker ID Effort ControlSpeaker identification requests accept an effort parameter to trade processing time
for accuracy on harder cases.
Python SDK 1.0One interface across pre-recorded, streaming, and sync transcription, replacing
three separate client shapes. Adds SDK coverage for the Sync API, and
AsyncTranscriber runs transcriptions concurrently as awaitable coroutines instead of
blocking a thread.
Dashboard Refresh and Integrated PlaygroundA visual refresh across the dashboard — new palette, typography, and retuned light
and dark modes — and the standalone playground now lives inside the dashboard.
Single Sign-On (SSO)Self-serve SAML with just-in-time provisioning by email domain, managed from the
dashboard. Available on contract and pay-as-you-go accounts as an add-on.
Self-Service Onboarding WizardGuided setup for new accounts: model selection, API key creation, a first-request
walkthrough, and best-practice defaults.
Self-Serve Data ControlsModel-training opt-out, transcript retention (TTL), and BAA initiation are now
self-serve from the dashboard, instead of going through support.