Skip to main content

Overview

AssemblyAI provides two speech-to-text services:
  • AssemblyAISTTService: Real-time streaming via WebSocket. Opens a persistent connection and transcribes audio as it arrives, with support for interim results, end-of-turn detection, and continuous transcription.
  • AssemblyAISyncSTTService: Segmented transcription via HTTP. Uses VAD to detect speech segments and transcribes each complete segment (up to 120 seconds) in a single HTTP request. No persistent session to manage. Ideal for dictation, scribe workflows, and voice agents where speech is detected locally.

AssemblyAI STT API Reference

Pipecat’s API methods for AssemblyAI STT integration

Example Implementation

Example with AssemblyAI built-in turn detection

Universal 3.5 Pro Realtime

Universal 3.5 Pro Realtime documentation and features

Universal 3.5 Pro Realtime API Reference

Complete Universal 3.5 Pro Realtime API reference

AssemblyAI Console

Access API keys and transcription features

Installation

To use AssemblyAI services, install the required dependency:

Prerequisites

AssemblyAI Account Setup

Before using AssemblyAI STT services, you need:
  1. AssemblyAI Account: Sign up at AssemblyAI Console
  2. API Key: Generate an API key from your dashboard
  3. Configuration: Configure transcription settings and features for your use case

Required Environment Variables

  • ASSEMBLYAI_API_KEY: Your AssemblyAI API key for authentication

Configuration

AssemblyAISTTService

str
required
AssemblyAI API key for authentication.
Language
default:"Language.EN"
deprecated
Language code for transcription. Deprecated in v0.0.105. Use settings=AssemblyAISTTService.Settings(...) instead.
str
default:"wss://streaming.assemblyai.com/v3/ws"
WebSocket endpoint URL. Override for custom or proxied deployments.
int | None
default:"None"
Audio sample rate in Hz. If None, uses the input sample rate from the start frame.
str
default:"pcm_s16le"
Audio encoding format.
AssemblyAIConnectionParams
default:"None"
deprecated
Connection configuration parameters. Deprecated in v0.0.105. Use settings=AssemblyAISTTService.Settings(...) instead. See Settings below.
bool
default:"True"
Controls turn detection mode. When True (Pipecat mode, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. VAD stop sends ForceEndpoint as ceiling. No UserStarted/StoppedSpeakingFrame emitted from STT. When False (AssemblyAI turn detection mode): AssemblyAI’s model controls turn endings using built-in turn detection. Uses AssemblyAI API defaults for all parameters unless explicitly set. Emits UserStarted/StoppedSpeakingFrame from STT.
bool
default:"True"
Whether to interrupt the bot when the user starts speaking in AssemblyAI turn detection mode (vad_force_turn_endpoint=False). Only applies when using AssemblyAI’s built-in turn detection. See User Turn Strategies if you pass your own user_turn_strategies.
str | None
default:"None"
Optional format string for speaker labels when diarization is enabled. Use {speaker} for speaker label and {text} for transcript text. Example: "<{speaker}>{text}</{speaker}>" or "{speaker}: {text}". If None, transcript text is not modified.
AssemblyAISTTService.Settings
default:"None"
Runtime-configurable settings for the STT service. See Settings below.
float
default:"ASSEMBLYAI_TTFS_P99"
P99 latency from speech end to final transcript in seconds. Override for your deployment.

AssemblyAISyncSTTService

str
required
AssemblyAI API key for authentication.
aiohttp.ClientSession
required
aiohttp ClientSession for HTTP requests. Pre-warming only helps when the warm and transcribe requests share this session’s connection pool, so keep one session for the service.
str
default:"https://sync.assemblyai.com"
Base URL for the Sync API. Override for a data-residency endpoint (e.g. https://sync.us.assemblyai.com or https://sync.eu.assemblyai.com).
int | None
default:"None"
Audio sample rate in Hz. If None, uses the pipeline’s rate.
bool
default:"True"
Whether to open the connection when the user starts speaking so the transcription request avoids the connection handshake.
int
default:"5"
Number of prior conversation turns — user transcripts and agent replies together, in one chronological buffer — automatically carried as conversation_context on each request, so the model transcribes each turn with the surrounding dialogue. Set to 0 to disable automatic context. Ignored when conversation_context is set explicitly in settings.
int
default:"1500"
Character budget for the automatic context buffer; the oldest turns are dropped first once either cap is exceeded.
AssemblyAISyncSTTService.Settings
default:"None"
Runtime-configurable settings for the Sync STT service. See Sync Settings below.
float
default:"ASSEMBLYAI_SYNC_TTFS_P99"
P99 latency from speech end to final transcript in seconds. Broadcast at pipeline start for downstream turn timing; set it to your measured value.

Settings

Runtime-configurable settings passed via the settings constructor argument using AssemblyAISTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.

Sync Settings

Runtime-configurable settings passed via the settings constructor argument using AssemblyAISyncSTTService.Settings(...). The Sync service has a simpler settings surface than the streaming service — it supports prompting, key terms, and conversation context, but not turn detection or real-time features like voice focus.

Usage

Basic Setup

With Custom Settings

With AssemblyAI Built-in Turn Detection

AssemblyAI’s Universal-3 Pro models (universal-3-5-pro and universal-3-6-pro) support built-in turn detection for more natural conversation flow. When vad_force_turn_endpoint=False, the service automatically requests ExternalUserTurnStrategies, so you don’t need to configure turn strategies manually:

With Speaker Diarization

Enable speaker identification for multi-party conversations:

With Context Carryover

Context carryover improves transcription by giving the model memory of recent conversation turns. It’s enabled by default for Universal-3 Pro models — the service automatically feeds each of the agent’s completed replies to AssemblyAI as carryover context, so no configuration is required. The example below only tunes how many prior turns are retained:
Context carryover helps with short answers, spelled-out entities (emails, IDs), and similar-sounding words. Set previous_context_n_turns=0 to disable automatic carryover.

With Voice Focus

Voice focus isolates the primary speaker and suppresses background noise:
Use "near-field" for close-talking mics (headsets, handsets) or "far-field" for distant capture (conference rooms, laptop mics).

With Language Steering

Steer transcription toward specific languages using language_codes:
Language steering is prompt-based and applies to U3 Pro models only. Regional variants (e.g. Language.ES_MX) resolve to their base code ("es"). At most 10 distinct languages can be declared. Unlike most settings, language_codes can be updated mid-session without reconnecting using STTUpdateSettingsFrame.

AssemblyAISyncSTTService Usage

The Sync service transcribes VAD-detected speech segments via HTTP requests rather than a persistent WebSocket connection.

Basic Setup

With Custom Settings

With Custom Context Management

Or provide explicit context:

Data Residency

Methods

update_agent_context()

Send the agent’s latest spoken reply to AssemblyAI as carryover context. Improves transcription of the user’s next turn — short answers, spelled-out entities, disambiguation. Parameters:
  • text (str): The agent’s spoken reply text. Clipped to ~1500 characters.
Example:
It is automatically invoked each time an assistant turn is completed.

warm()

Pre-warm the connection to the Sync API (AssemblyAISyncSTTService only). Establishes the connection (DNS, TCP, TLS) ahead of a transcription request so the next segment starts uploading audio immediately. The warmed connection is reused only by requests that share this service’s aiohttp session and base URL. Best-effort: failures are logged and swallowed, since a failed warm-up only forfeits the latency saving. Example:
When enable_prewarming=True (the default), the connection is warmed automatically when the user starts speaking.

Notes

  • Model: Use universal-3-5-pro or universal-3-6-pro, AssemblyAI’s flagship Universal-3 Pro streaming models. universal-3-6-pro is universal-3-5-pro upgraded with the same feature set. Both support built-in turn detection, prompting, continuous partials, context carryover, and voice focus. universal-3-5-pro is the default.
  • Turn detection modes:
    • Pipecat mode (vad_force_turn_endpoint=True, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. The service sends a ForceEndpoint message when VAD detects the user has stopped speaking.
    • AssemblyAI mode (vad_force_turn_endpoint=False, U3 Pro models only): AssemblyAI’s model controls turn endings using built-in turn detection. The service proposes each turn boundary and automatically requests ExternalUserTurnStrategies, so you don’t need to configure turn strategies manually. See User Turn Strategies for the interruption controls.
  • Context carryover (U3 Pro models only): Seed the agent’s most recent reply, improving transcription of the user’s next turn — short answers, spelled-out entities, disambiguation. update_agent_context() is automatically invoked each time an assistant turn is completed. Control the window size with previous_context_n_turns (0–100, default 3); set to 0 to disable carryover entirely.
  • Voice focus (U3 Pro models only): Set voice_focus to "near-field" or "far-field" to isolate the primary voice and suppress background noise. Tune suppression strength with voice_focus_threshold (0.0–1.0, higher values suppress more).
  • Speaker diarization: Enable speaker_labels=True in Settings to automatically identify different speakers. Final transcripts will include a speaker field (e.g., “Speaker A”, “Speaker B”). Use the speaker_format parameter to format transcripts with speaker labels.
  • Prompting (U3 Pro models only): The prompt parameter allows you to guide transcription for specific names, terms, or domain vocabulary. May be combined with keyterms_prompt on U3 Pro models. AssemblyAI recommends testing without a prompt first.
  • Dynamic settings updates: Most settings can be updated at runtime using STTUpdateSettingsFrame. agent_context and language_codes are hot-updatable without reconnecting; other settings require a reconnect.

AssemblyAISyncSTTService

  • Segment limit: Audio segments must be at most 120 seconds. The API rejects longer segments. This suits dictation, scribe workflows, and voice-agent turns where the client detects speech locally.
  • No persistent connection: The Sync API is stateless. Each speech segment is transcribed in a single HTTP POST request with no upload step, no polling, and no session to manage.
  • Automatic conversation context: Recent conversation turns — user transcripts and agent replies together — are automatically carried as conversation_context on each request. Control the buffer size with max_context_turns (default 5, set to 0 to disable) and max_context_chars (default 1500). Setting conversation_context explicitly in Settings turns off the automatic buffer and sends exactly that value.
  • Pre-warming: When enable_prewarming=True (the default), the connection is opened when the user starts speaking so the transcription request skips the DNS, TCP, and TLS handshake. Call warm() directly to warm the connection at any other time.
  • aiohttp session: Pre-warming only helps when the warm and transcribe requests share the same aiohttp session’s connection pool, so keep one session for the service.
  • Data residency: Override base_url for data-residency endpoints (e.g. https://sync.us.assemblyai.com or https://sync.eu.assemblyai.com).
The connection_params= / InputParams / params= pattern is deprecated as of v0.0.105. Use Settings / settings= instead. See the Service Settings guide for migration details.

Event Handlers

AssemblyAI STT supports the standard service connection events, plus turn-level events for conversation tracking:
The on_end_of_turn event receives (service, transcript) where transcript is the final transcript text. This event fires after the final transcript is pushed, providing a reliable hook for end-of-turn logic that doesn’t race with TranscriptionFrame. Works in both Pipecat and AssemblyAI turn detection modes.