AssemblyAI

Overview

AssemblyAISTTService provides real-time speech recognition using AssemblyAI’s WebSocket API with support for interim results, end-of-turn detection, and configurable audio processing parameters for accurate transcription in conversational AI applications.

AssemblyAI STT API Reference

Pipecat’s API methods for AssemblyAI STT integration

Example Implementation

Example with AssemblyAI built-in turn detection

Universal 3.5 Pro Realtime

Universal 3.5 Pro Realtime documentation and features

Universal 3.5 Pro Realtime API Reference

Complete Universal 3.5 Pro Realtime API reference

AssemblyAI Console

Access API keys and transcription features

Installation

To use AssemblyAI services, install the required dependency:

uv add "pipecat-ai[assemblyai]"

Prerequisites

AssemblyAI Account Setup

Before using AssemblyAI STT services, you need:

AssemblyAI Account: Sign up at AssemblyAI Console
API Key: Generate an API key from your dashboard
Configuration: Configure transcription settings and features for your use case

Required Environment Variables

ASSEMBLYAI_API_KEY: Your AssemblyAI API key for authentication

Configuration

AssemblyAISTTService

api_key

str

required

AssemblyAI API key for authentication.

language

Language

default:"Language.EN"

deprecated

Language code for transcription. Deprecated in v0.0.105. Use settings=AssemblyAISTTService.Settings(...) instead.

api_endpoint_base_url

str

default:"wss://streaming.assemblyai.com/v3/ws"

WebSocket endpoint URL. Override for custom or proxied deployments.

sample_rate

int

default:"16000"

Audio sample rate in Hz.

encoding

str

default:"pcm_s16le"

Audio encoding format.

connection_params

AssemblyAIConnectionParams

default:"None"

deprecated

Connection configuration parameters. Deprecated in v0.0.105. Use settings=AssemblyAISTTService.Settings(...) instead. See Settings below.

vad_force_turn_endpoint

bool

default:"True"

Controls turn detection mode. When True (Pipecat mode, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. VAD stop sends ForceEndpoint as ceiling. No UserStarted/StoppedSpeakingFrame emitted from STT. When False (AssemblyAI turn detection mode): AssemblyAI’s model controls turn endings using built-in turn detection. Uses AssemblyAI API defaults for all parameters unless explicitly set. Emits UserStarted/StoppedSpeakingFrame from STT.

should_interrupt

bool

default:"True"

Whether to interrupt the bot when the user starts speaking in AssemblyAI turn detection mode (vad_force_turn_endpoint=False). Only applies when using AssemblyAI’s built-in turn detection.

speaker_format

str | None

default:"None"

Optional format string for speaker labels when diarization is enabled. Use {speaker} for speaker label and {text} for transcript text. Example: "<{speaker}>{text}</{speaker}>" or "{speaker}: {text}". If None, transcript text is not modified.

settings

AssemblyAISTTService.Settings

default:"None"

Runtime-configurable settings for the STT service. See Settings below.

ttfs_p99_latency

float

default:"ASSEMBLYAI_TTFS_P99"

P99 latency from speech end to final transcript in seconds. Override for your deployment.

Settings

Runtime-configurable settings passed via the settings constructor argument using AssemblyAISTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.

Parameter	Type	Default	Description
`model`	`str`	`"universal-3-5-pro"`	STT model identifier. Set to `"universal-3-5-pro"`. (Inherited from base STT settings.)
`language`	`Language \| str`	`Language.EN`	Language for speech recognition. (Inherited from base STT settings.)
`formatted_finals`	`bool`	`True`	Whether to enable transcript formatting.
`word_finalization_max_wait_time`	`int`	`None`	Maximum time to wait for word finalization in milliseconds.
`end_of_turn_confidence_threshold`	`float`	`None`	Confidence threshold for end-of-turn detection.
`min_turn_silence`	`int`	`None`	Silence duration (ms) before a speculative end-of-turn check. If terminal punctuation is found, the turn ends; otherwise a partial is emitted and the turn continues. Clamped to 50–10000 ms. Server default is mode-dependent (set by the `mode` preset).
`max_turn_silence`	`int`	`None`	Maximum silence (ms) before the turn is forced to end, regardless of punctuation.
`keyterms_prompt`	`List[str]`	`None`	A list of words and phrases to improve recognition accuracy for. Maximum 100 terms.
`prompt`	`str`	`None`	A contextual prompt describing what the audio is about — its domain, scenario, or conversation details — so the model better recognizes likely vocabulary. Carries context about your audio, not transcription instructions; formatting or behavioral commands are not supported. Maximum ~1500 characters.
`language_code`	`str`	`None`	Steers transcription toward a specific language on a per-token basis. Pass a single ISO code to bias output toward that language; when unset, no steering is applied and the model code-switches natively across supported languages. Supported codes: `en`, `es`, `fr`, `de`, `it`, `pt`, `tr`, `nl`, `sv`, `no`, `da`, `fi`, `hi`, `vi`, `ar`, `he`, `ja`, `zh`. Distinct from language detection, which only controls whether the detected language is reported.
`speaker_labels`	`bool`	`None`	Whether to enable streaming speaker diarization. When enabled, each turn includes a `speaker_label` and each final word includes a `speaker` field for word-level attribution.
`vad_threshold`	`float`	`None`	Confidence threshold (0.0–1.0) for classifying audio frames as silence. Frames with VAD confidence below this value are considered silent; increase for noisy environments to reduce false speech detection. Server default is mode-dependent (set by the `mode` preset).
`domain`	`str`	`None`	Enable a domain-specific model for specialized terminology. Set to `"medical-v1"` for Medical Mode (improved accuracy on medications, procedures, conditions, and dosages). Supported languages: `en`, `es`, `de`, `fr`.
`continuous_partials`	`bool`	`True`	Whether to emit additional partial transcripts during long turns at a steady ~3 second cadence. When enabled, partials covering the full turn transcript are emitted about every 3 seconds while speech continues; when disabled, only one early partial is emitted near turn start. The first partial (at 750 ms) is unaffected.
`interruption_delay`	`int`	`None`	How soon the first partial is emitted, in milliseconds (0–1000). Useful for tuning barge-in responsiveness or emitting earlier partials for LLM inference; larger values are more confident on interruptions, smaller values give faster time to first partial. The server adds a fixed 256 ms, so `0` yields an effective 256 ms and `500` yields 756 ms. Server default is mode-dependent (set by the `mode` preset).
`agent_context`	`str`	`None`	Your voice agent’s spoken text (TTS reply), used as context for the next user turn — improves accuracy on short or ambiguous replies and spelled-out entities like emails or IDs. Set at connection to seed the agent’s greeting and/or update after each reply; each update replaces the previous value. Maximum ~1500 characters.
`previous_context_n_turns`	`int`	`None`	Advanced. Maximum number of prior conversation entries (user transcripts and any `agent_context` values) carried forward as context. Range 0–100; set to `0` to disable automatic carryover. Most integrations should leave this unset.
`voice_focus`	`Literal["near-field", "far-field"]`	`None`	Enable Voice Focus to isolate the primary voice and suppress background noise before transcription. Set to `"near-field"` for close-talking mics (headsets, phones) or `"far-field"` for distant mics (conference rooms). Off when unset.
`voice_focus_threshold`	`float`	`None`	Controls how aggressively Voice Focus suppresses background audio, from `0.0` (least) to `1.0` (most). Requires `voice_focus` to be set, otherwise a validation error is returned.
`mode`	`Literal["min_latency", "balanced", "max_accuracy"]`	`None`	Latency and accuracy preset controlling turn-detection and partial-emission defaults. `max_accuracy` favors quality, `min_latency` favors speed, and `balanced` trades off between them. When omitted, the server applies its own preset, which sets the defaults for mode-dependent fields such as `interruption_delay`, `min_turn_silence`, `vad_threshold`, `previous_context_n_turns`, and `continuous_partials`.

Usage

Basic Setup

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
    ),
)

With Custom Settings

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        keyterms_prompt=["Pipecat", "AssemblyAI"],
    ),
    vad_force_turn_endpoint=True,
)

With AssemblyAI Built-in Turn Detection

AssemblyAI’s universal-3-5-pro model supports built-in turn detection for more natural conversation flow. When vad_force_turn_endpoint=False, the service automatically requests ExternalUserTurnStrategies, so you don’t need to configure turn strategies manually:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    vad_force_turn_endpoint=False,  # Use AssemblyAI's built-in turn detection
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        # Optional: Tune turn detection timing
        min_turn_silence=100,  # Silence duration (ms) before a speculative end-of-turn check.
        max_turn_silence=1000,  # Maximum silence (ms) before forcing end-of-turn
    ),
)

With Speaker Diarization

Enable speaker identification for multi-party conversations:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        speaker_labels=True,  # Enable speaker diarization
    ),
    speaker_format="{speaker}: {text}",  # Format transcripts with speaker labels
)

With Context Carryover

Context carryover improves transcription by giving the model memory of recent conversation turns. It’s enabled by default for universal-3-5-pro — the service automatically feeds each of the agent’s completed replies to AssemblyAI as carryover context, so no configuration is required. The example below only tunes how many prior turns are retained:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        previous_context_n_turns=4,  # Keep last 4 turns (if None, server default is 3. Set to 0 to disable.)
    ),
)

Context carryover helps with short answers, spelled-out entities (emails, IDs), and similar-sounding words. Set previous_context_n_turns=0 to disable automatic carryover.

With Voice Focus

Voice focus isolates the primary speaker and suppresses background noise:

from pipecat.services.assemblyai.stt import AssemblyAISTTService

stt = AssemblyAISTTService(
    api_key=os.getenv("ASSEMBLYAI_API_KEY"),
    settings=AssemblyAISTTService.Settings(
        model="universal-3-5-pro",
        voice_focus="near-field",  # Or "far-field" for distant capture
        voice_focus_threshold=0.7,  # 0.0-1.0, higher suppresses more
    ),
)

Use "near-field" for close-talking mics (headsets, handsets) or "far-field" for distant capture (conference rooms, laptop mics).

Methods

update_agent_context()

async def update_agent_context(text: str)

Send the agent’s latest spoken reply to AssemblyAI as carryover context. Improves transcription of the user’s next turn — short answers, spelled-out entities, disambiguation. Parameters:

text (str): The agent’s spoken reply text. Clipped to ~1500 characters.

Example:

# After the agent finishes speaking
await stt.update_agent_context("What is your order ID?")

It is automatically invoked each time an assistant turn is completed.

Notes

Model: Use universal-3-5-pro, AssemblyAI’s flagship streaming model. It supports built-in turn detection, prompting, continuous partials, context carryover, and voice focus.
Turn detection modes:
- Pipecat mode (vad_force_turn_endpoint=True, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. The service sends a ForceEndpoint message when VAD detects the user has stopped speaking.
- AssemblyAI mode (vad_force_turn_endpoint=False, universal-3-5-pro only): AssemblyAI’s model controls turn endings using built-in turn detection. The service emits UserStartedSpeakingFrame and UserStoppedSpeakingFrame and automatically requests ExternalUserTurnStrategies, so you don’t need to configure turn strategies manually. Pass your own user_turn_strategies only to override this.
Context carryover (universal-3-5-pro only): Seed the agent’s most recent reply, improving transcription of the user’s next turn — short answers, spelled-out entities, disambiguation. update_agent_context() is automatically invoked each time an assistant turn is completed. Control the window size with previous_context_n_turns (0–100, default 3); set to 0 to disable carryover entirely.
Voice focus (universal-3-5-pro only): Set voice_focus to "near-field" or "far-field" to isolate the primary voice and suppress background noise. Tune suppression strength with voice_focus_threshold (0.0–1.0, higher values suppress more).
Speaker diarization: Enable speaker_labels=True in Settings to automatically identify different speakers. Final transcripts will include a speaker field (e.g., “Speaker A”, “Speaker B”). Use the speaker_format parameter to format transcripts with speaker labels.
Prompting: The prompt parameter allows you to guide transcription for specific names, terms, or domain vocabulary. AssemblyAI recommends testing without a prompt first.
Dynamic settings updates: Most settings can be updated at runtime using STTUpdateSettingsFrame. agent_context is hot-updatable without reconnecting; other settings require a reconnect.

The connection_params= / InputParams / params= pattern is deprecated as of v0.0.105. Use Settings / settings= instead. See the Service Settings guide for migration details.

Event Handlers

AssemblyAI STT supports the standard service connection events, plus turn-level events for conversation tracking:

Event	Description
`on_connected`	Connected to AssemblyAI WebSocket
`on_disconnected`	Disconnected from AssemblyAI WebSocket
`on_end_of_turn`	End of turn detected (fires after final transcript is pushed)

@stt.event_handler("on_connected")
async def on_connected(service):
    print("Connected to AssemblyAI")

@stt.event_handler("on_end_of_turn")
async def on_end_of_turn(service, transcript):
    print(f"Turn ended: {transcript}")

The on_end_of_turn event receives (service, transcript) where transcript is the final transcript text. This event fires after the final transcript is pushed, providing a reliable hook for end-of-turn logic that doesn’t race with TranscriptionFrame. Works in both Pipecat and AssemblyAI turn detection modes.

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

Overview

AssemblyAI STT API Reference

Example Implementation

Universal 3.5 Pro Realtime

Universal 3.5 Pro Realtime API Reference

AssemblyAI Console

Installation

Prerequisites

AssemblyAI Account Setup

Required Environment Variables

Configuration

AssemblyAISTTService

Settings

Usage

Basic Setup

With Custom Settings

With AssemblyAI Built-in Turn Detection

With Speaker Diarization

With Context Carryover

With Voice Focus

Methods

update_agent_context()

Notes

Event Handlers

​Overview

AssemblyAI STT API Reference

Example Implementation

Universal 3.5 Pro Realtime

Universal 3.5 Pro Realtime API Reference

AssemblyAI Console

​Installation

​Prerequisites

​AssemblyAI Account Setup

​Required Environment Variables

​Configuration

​AssemblyAISTTService

​Settings

​Usage

​Basic Setup

​With Custom Settings

​With AssemblyAI Built-in Turn Detection

​With Speaker Diarization

​With Context Carryover

​With Voice Focus

​Methods

​update_agent_context()

​Notes

​Event Handlers

Overview

Installation

Prerequisites

AssemblyAI Account Setup

Required Environment Variables

Configuration

AssemblyAISTTService

Settings

Usage

Basic Setup

With Custom Settings

With AssemblyAI Built-in Turn Detection

With Speaker Diarization

With Context Carryover

With Voice Focus

Methods

update_agent_context()

Notes

Event Handlers