Overview
AssemblyAI provides two speech-to-text services:-
AssemblyAISTTService: Real-time streaming via WebSocket. Opens a persistent connection and transcribes audio as it arrives, with support for interim results, end-of-turn detection, and continuous transcription. -
AssemblyAISyncSTTService: Segmented transcription via HTTP. Uses VAD to detect speech segments and transcribes each complete segment (up to 120 seconds) in a single HTTP request. No persistent session to manage. Ideal for dictation, scribe workflows, and voice agents where speech is detected locally.
AssemblyAI STT API Reference
Pipecat’s API methods for AssemblyAI STT integration
Example Implementation
Example with AssemblyAI built-in turn detection
Universal 3.5 Pro Realtime
Universal 3.5 Pro Realtime documentation and features
Universal 3.5 Pro Realtime API Reference
Complete Universal 3.5 Pro Realtime API reference
AssemblyAI Console
Access API keys and transcription features
Installation
To use AssemblyAI services, install the required dependency:Prerequisites
AssemblyAI Account Setup
Before using AssemblyAI STT services, you need:- AssemblyAI Account: Sign up at AssemblyAI Console
- API Key: Generate an API key from your dashboard
- Configuration: Configure transcription settings and features for your use case
Required Environment Variables
ASSEMBLYAI_API_KEY: Your AssemblyAI API key for authentication
Configuration
AssemblyAISTTService
str
required
AssemblyAI API key for authentication.
Language
default:"Language.EN"
deprecated
Language code for transcription. Deprecated in v0.0.105. Use
settings=AssemblyAISTTService.Settings(...) instead.str
default:"wss://streaming.assemblyai.com/v3/ws"
WebSocket endpoint URL. Override for custom or proxied deployments.
int | None
default:"None"
Audio sample rate in Hz. If None, uses the input sample rate from the start
frame.
str
default:"pcm_s16le"
Audio encoding format.
AssemblyAIConnectionParams
default:"None"
deprecated
Connection configuration parameters. Deprecated in v0.0.105. Use
settings=AssemblyAISTTService.Settings(...) instead. See
Settings below.bool
default:"True"
Controls turn detection mode. When
True (Pipecat mode, default): Forces
AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart
Turn) decides when the user is done. VAD stop sends ForceEndpoint as ceiling.
No UserStarted/StoppedSpeakingFrame emitted from STT. When False (AssemblyAI
turn detection mode): AssemblyAI’s model controls turn endings using built-in
turn detection. Uses AssemblyAI API defaults for all parameters unless
explicitly set. Emits UserStarted/StoppedSpeakingFrame from STT.bool
default:"True"
Whether to interrupt the bot when the user starts speaking in AssemblyAI turn
detection mode (
vad_force_turn_endpoint=False). Only applies when using
AssemblyAI’s built-in turn detection. See User Turn
Strategies
if you pass your own user_turn_strategies.str | None
default:"None"
Optional format string for speaker labels when diarization is enabled. Use
{speaker} for speaker label and {text} for transcript text. Example:
"<{speaker}>{text}</{speaker}>" or "{speaker}: {text}". If None, transcript
text is not modified.AssemblyAISTTService.Settings
default:"None"
Runtime-configurable settings for the STT service. See Settings
below.
float
default:"ASSEMBLYAI_TTFS_P99"
P99 latency from speech end to final transcript in seconds. Override for your
deployment.
AssemblyAISyncSTTService
str
required
AssemblyAI API key for authentication.
aiohttp.ClientSession
required
aiohttp ClientSession for HTTP requests. Pre-warming only helps when the warm
and transcribe requests share this session’s connection pool, so keep one
session for the service.
str
default:"https://sync.assemblyai.com"
Base URL for the Sync API. Override for a data-residency endpoint (e.g.
https://sync.us.assemblyai.com or https://sync.eu.assemblyai.com).int | None
default:"None"
Audio sample rate in Hz. If None, uses the pipeline’s rate.
bool
default:"True"
Whether to open the connection when the user starts speaking so the
transcription request avoids the connection handshake.
int
default:"5"
Number of prior conversation turns — user transcripts and agent replies
together, in one chronological buffer — automatically carried as
conversation_context on each request, so the model transcribes each turn
with the surrounding dialogue. Set to 0 to disable automatic context. Ignored
when conversation_context is set explicitly in settings.int
default:"1500"
Character budget for the automatic context buffer; the oldest turns are
dropped first once either cap is exceeded.
AssemblyAISyncSTTService.Settings
default:"None"
Runtime-configurable settings for the Sync STT service. See Sync
Settings below.
float
default:"ASSEMBLYAI_SYNC_TTFS_P99"
P99 latency from speech end to final transcript in seconds. Broadcast at
pipeline start for downstream turn timing; set it to your measured value.
Settings
Runtime-configurable settings passed via thesettings constructor argument using AssemblyAISTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.
Sync Settings
Runtime-configurable settings passed via thesettings constructor argument using AssemblyAISyncSTTService.Settings(...). The Sync service has a simpler settings surface than the streaming service — it supports prompting, key terms, and conversation context, but not turn detection or real-time features like voice focus.
Usage
Basic Setup
With Custom Settings
With AssemblyAI Built-in Turn Detection
AssemblyAI’s Universal-3 Pro models (universal-3-5-pro and universal-3-6-pro) support built-in turn detection for more natural conversation flow. When vad_force_turn_endpoint=False, the service automatically requests ExternalUserTurnStrategies, so you don’t need to configure turn strategies manually:
With Speaker Diarization
Enable speaker identification for multi-party conversations:With Context Carryover
Context carryover improves transcription by giving the model memory of recent conversation turns. It’s enabled by default for Universal-3 Pro models — the service automatically feeds each of the agent’s completed replies to AssemblyAI as carryover context, so no configuration is required. The example below only tunes how many prior turns are retained:previous_context_n_turns=0 to disable automatic carryover.
With Voice Focus
Voice focus isolates the primary speaker and suppresses background noise:"near-field" for close-talking mics (headsets, handsets) or "far-field" for distant capture (conference rooms, laptop mics).
With Language Steering
Steer transcription toward specific languages usinglanguage_codes:
Language.ES_MX) resolve to their base code ("es"). At most 10 distinct languages can be declared. Unlike most settings, language_codes can be updated mid-session without reconnecting using STTUpdateSettingsFrame.
AssemblyAISyncSTTService Usage
The Sync service transcribes VAD-detected speech segments via HTTP requests rather than a persistent WebSocket connection.Basic Setup
With Custom Settings
With Custom Context Management
Data Residency
Methods
update_agent_context()
text(str): The agent’s spoken reply text. Clipped to ~1500 characters.
It is automatically invoked each time an assistant turn is completed.
warm()
When enable_prewarming=True (the default), the connection is warmed automatically when the user starts speaking.
Notes
- Model: Use
universal-3-5-prooruniversal-3-6-pro, AssemblyAI’s flagship Universal-3 Pro streaming models.universal-3-6-proisuniversal-3-5-proupgraded with the same feature set. Both support built-in turn detection, prompting, continuous partials, context carryover, and voice focus.universal-3-5-prois the default. - Turn detection modes:
- Pipecat mode (
vad_force_turn_endpoint=True, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. The service sends aForceEndpointmessage when VAD detects the user has stopped speaking. - AssemblyAI mode (
vad_force_turn_endpoint=False, U3 Pro models only): AssemblyAI’s model controls turn endings using built-in turn detection. The service proposes each turn boundary and automatically requestsExternalUserTurnStrategies, so you don’t need to configure turn strategies manually. See User Turn Strategies for the interruption controls.
- Pipecat mode (
- Context carryover (U3 Pro models only): Seed the agent’s most recent reply, improving transcription of the user’s next turn — short answers, spelled-out entities, disambiguation.
update_agent_context()is automatically invoked each time an assistant turn is completed. Control the window size withprevious_context_n_turns(0–100, default 3); set to 0 to disable carryover entirely. - Voice focus (U3 Pro models only): Set
voice_focusto"near-field"or"far-field"to isolate the primary voice and suppress background noise. Tune suppression strength withvoice_focus_threshold(0.0–1.0, higher values suppress more). - Speaker diarization: Enable
speaker_labels=Truein Settings to automatically identify different speakers. Final transcripts will include a speaker field (e.g., “Speaker A”, “Speaker B”). Use thespeaker_formatparameter to format transcripts with speaker labels. - Prompting (U3 Pro models only): The
promptparameter allows you to guide transcription for specific names, terms, or domain vocabulary. May be combined withkeyterms_prompton U3 Pro models. AssemblyAI recommends testing without a prompt first. - Dynamic settings updates: Most settings can be updated at runtime using
STTUpdateSettingsFrame.agent_contextandlanguage_codesare hot-updatable without reconnecting; other settings require a reconnect.
AssemblyAISyncSTTService
- Segment limit: Audio segments must be at most 120 seconds. The API rejects longer segments. This suits dictation, scribe workflows, and voice-agent turns where the client detects speech locally.
- No persistent connection: The Sync API is stateless. Each speech segment is transcribed in a single HTTP POST request with no upload step, no polling, and no session to manage.
- Automatic conversation context: Recent conversation turns — user transcripts and agent replies together — are automatically carried as
conversation_contexton each request. Control the buffer size withmax_context_turns(default 5, set to 0 to disable) andmax_context_chars(default 1500). Settingconversation_contextexplicitly in Settings turns off the automatic buffer and sends exactly that value. - Pre-warming: When
enable_prewarming=True(the default), the connection is opened when the user starts speaking so the transcription request skips the DNS, TCP, and TLS handshake. Callwarm()directly to warm the connection at any other time. - aiohttp session: Pre-warming only helps when the warm and transcribe requests share the same aiohttp session’s connection pool, so keep one session for the service.
- Data residency: Override
base_urlfor data-residency endpoints (e.g.https://sync.us.assemblyai.comorhttps://sync.eu.assemblyai.com).
Event Handlers
AssemblyAI STT supports the standard service connection events, plus turn-level events for conversation tracking:on_end_of_turn event receives (service, transcript) where transcript is the final transcript text. This event fires after the final transcript is pushed, providing a reliable hook for end-of-turn logic that doesn’t race with TranscriptionFrame. Works in both Pipecat and AssemblyAI turn detection modes.