Audio Pipeline

How speech recognition, emotion detection, filler speech, barge-in detection, and TTS work together in real time.

Browser Recording Sessions

Clinical recording sessions support a browser-based capture path where the provider's microphone audio streams to the platform over a resumable WebSocket connection. The protocol supports pause, resume, and reconnect so that transient network interruptions do not lose accumulated transcript state.

Key characteristics:

• Resumable. If the connection drops, the browser SDK can reconnect to the same session. The worker rehydrates prior transcript segments and resumes from the last acknowledged audio position.
• Pause and resume. A provider can pause recording, which flushes a transcript snapshot and releases the upstream transcription resource. Resuming opens a fresh transcription connection while preserving segment ordering.
• Structured lifecycle. Each connection validates session ownership and state before accepting audio. Typed close codes distinguish clean completion, recoverable disconnects, and fatal errors so the client can respond appropriately.
• Fenced state transitions. Concurrent or stale connections cannot overwrite state owned by a newer attach, preventing split-brain scenarios during reconnect sequences.

The audio pipeline converts a caller's voice into text, processes the transcript through the agent's reasoning, and converts the response back to speech. Speech recognition, optional emotion analysis, tool execution, and speech output have separate responsibilities, but required model, tool, or transport work can still delay or fail a turn.

Each voice session manages four user-visible concerns:

• Listen - Capture caller speech and detect when the caller has finished a turn.
• Understand - Produce transcripts and emotional context for the agent.
• Decide - Use the current context graph, patient context, and tool results to choose the next response.
• Speak - Deliver the response with the right timing, voice, and interruption behavior.

This design lets Amigo update recognition, emotion detection, and response generation independently while preserving a consistent call experience for the patient.

Signal Capture

Audio arrives from the telephony layer as a standard telephony audio stream. The system splits it into two parallel paths the moment it arrives:

1. Speech-to-text - Converts audio to transcript text in real time
2. Emotion detection - Analyzes vocal qualities for emotional signals (covered in Emotion Detection)

The optional emotion path is isolated from speech recognition, so its failure does not stop transcription. Emotion analysis may still receive audio when transcription fails, but a live agent turn generally cannot proceed without usable caller text.

Progressive Initialization

The runtime prepares configuration, tools, integrations, and greeting audio before the agent begins the live exchange. Speech recognition and the optional emotion stream initialize alongside the rest of the session.

For outbound calls, preparation can run while the destination is dialing or ringing. For inbound conference calls, the caller can remain on hold while preparation completes and the agent leg is created. These paths reduce startup delay but do not guarantee zero wait under all network or dependency conditions.

The runtime connects speech recognition before it begins forwarding caller audio. Emotion analysis connects after STT and is non-fatal: if that optional stream is unavailable, the call continues without emotion-derived steering.

Greeting

The agent can prepare and play the greeting while the speech recognition setup task is running. Caller audio waits for the STT-ready signal before it is forwarded, preventing a partially initialized recognition stream from receiving audio.

For conference-mode calls where the agent leg is created during ring time, the platform waits for the caller to actually join the conference before releasing the greeting. This prevents the greeting from playing into an empty conference.

Speech Handling

While the agent plays its greeting, any caller speech is discarded. The caller is "not heard" until the greeting finishes. This prevents the agent from interpreting ambient noise, simultaneous "hello" responses, or partial utterances as meaningful input before the conversation has properly started. Once the greeting completes, the speech-to-text pipeline begins processing caller audio normally.

Speech-to-Text

The speech-to-text stage converts caller audio into text transcripts in real time. On the multilingual recognition path that supports language consolidation, detected language observations can narrow recognition hints and update the speech-output language after enough consistent evidence is available. Other speech-recognition providers do not expose the same mid-call controls.

Amigo uses streaming speech recognition for real-time transcription. Latency varies with provider, language, network conditions, audio quality, and end-of-turn detection; the platform does not promise a fixed per-segment transcription latency.

The default recognition path can accept mid-call updates to keyterms, end-of-turn sensitivity, and language hints. Support is provider-dependent: a routed provider may accept only a subset of these controls or ignore a control it cannot apply.

Provider Routing

The platform supports multiple speech-to-text engines and selects one at session setup using resolved language, deployment availability, audio encoding, and an optional workspace preference. A workspace preference is honored only when that provider is enabled and compatible; otherwise routing falls through to the deployment's normal selection chain. This is setup-time selection, not a guarantee of live provider failover after a recognition session has started. Operator audio follows a fixed policy rather than the caller's workspace override.

Language Selection

The platform supports English-optimized and multilingual STT models, selected per-service:

Setting	Model	Best For
English	English-optimized (lowest error rate)	Monolingual English-speaking populations
Multilingual	Multi-language with code-switching	Populations that switch between languages mid-conversation
Auto	Multi-language with auto-detection	Unknown caller language; narrows automatically once detected

In auto mode on the compatible multilingual recognition path, the platform tracks reported languages across turns. Once one language has enough consistent support, it narrows language hints and notifies the speech-output path. This can improve recognition focus, but it is not a formal language-identification guarantee.

When an already-resolved entity has a language value in the world model, session setup can use it before speech begins. The priority order is entity language, then workspace voice language, then the deployment default. A stored language is a routing hint and should not be treated as proof of the caller's preference for the current interaction.

Keyterm Boosting

Medical terminology, provider names, medication names, and organization-specific vocabulary are difficult for general-purpose speech recognition. Three layers of keyterm boosting improve recognition accuracy:

Level	Managed By	Scope
Service-level	Workspace administrators	Applied to all calls for a given service
Workspace-level	API configuration	Per-workspace vocabulary (clinic names, local terminology)
System defaults	Amigo engineering	Baseline medical and scheduling vocabulary

The configured layers are merged and deduplicated during session setup. When an existing caller entity resolves, selected medication, allergy, and condition terms can be added to the active recognition session. Whether those keyterms affect transcription depends on the selected provider's capabilities.

End-of-Turn Detection

The system must determine when the caller has finished speaking so the agent can respond. This uses configurable confidence thresholds that balance two concerns:

• Responding too early cuts the caller off mid-sentence
• Responding too late creates awkward silence

Base end-of-turn controls come from the deployment and optional per-service voice settings. Context graph states can add compatible turn-policy overrides:

• Data collection states (collecting a date of birth, spelling a name) use higher thresholds and longer silence timeouts, because the caller is thinking and pausing between pieces of information
• Action states (confirming an appointment, answering a yes/no question) use default thresholds for snappy turn-taking

These overrides take effect on state transitions - when the agent moves to a data collection state, the STT engine reconfigures mid-call to be more patient with pauses.

Final Turn Boundary

Navigation starts only after the speech recognizer emits its final end-of-turn event. Earlier eager-end and resumed-turn signals are retained for observability, but they do not launch or cancel speculative navigation. This avoids spending a model call on an incomplete utterance while keeping turn-detection behavior measurable.

Text-to-Speech

The TTS engine converts generated text into spoken audio. Response and filler utterances can carry their own emotion and speed values, while voice identity, volume, language, and other settings come from the resolved provider configuration. Speech synthesis is streamed, so audio can begin before the complete response text is available.

Provider Selection

The platform supports multiple TTS providers through layered workspace, agent, service, and per-language configuration. Provider capabilities differ in latency, voice selection, expressive controls, pronunciation support, language coverage, and timing metadata. The Developer Console can expose available voices and previews for configured providers, while the API and agent configuration carry the provider-specific settings.

Provider-specific settings are resolved only for the selected provider, which prevents configuration fields from one provider leaking into another. Selecting an unknown provider name falls back to the deployment default during setup, but an outage in the selected live provider can still delay or fail speech. Provider choice can change voice quality, language behavior, sample rate, and call experience; recording metadata preserves the actual output format used.

Per-Language Provider Routing

The platform supports routing text-to-speech to different providers based on the caller's detected language. This is configured through a language-provider map that associates language codes with specific TTS providers and voice configurations.

When a caller's language is detected, the platform resolves the TTS provider through a priority matrix:

1. Exact language match - e.g., ar-SA matches an Arabic (Saudi Arabia) entry
2. Base language match - e.g., ar-SA falls back to an ar entry
3. Multilingual fallback - a catch-all entry for any language not explicitly mapped

At each level, service configuration takes priority over agent configuration, which takes priority over workspace configuration. The first match wins.

If no language-specific entry matches, the platform uses the standard TTS provider selection (service > agent > workspace > default). Per-language configuration is isolated - when a language-specific provider is selected, only that entry's voice settings are used, preventing configuration for one provider from affecting another.

TTS Tone Resolution

The response utterance first uses the emotion selected by navigation, when navigation returns one. Otherwise, the runtime uses the computed voice context:

1. After at least two acoustic segments, a sufficiently strong rolling emotion maps to an empathetic provider tone. Anger and Disgust map to calm delivery, Fear and Sadness map to sympathetic delivery, and Joy maps to enthusiastic delivery.
2. If the current rolling signal is weak or unmapped, tone momentum retains the previous successfully mapped emotion-derived tone.
3. If no emotion-derived tone exists, a current context-graph action that matches a configured sensitive topic can select sympathetic delivery.
4. A configured workspace tone overrides the computed voice-context tone.
5. If no turn-specific tone is set, the TTS provider keeps its configured baseline or default.

There is no separate vocal-burst model or burst-first tone rule in the live pipeline. Tone momentum also does not carry arbitrary navigation choices forward; it stores only the last tone derived by the rolling emotional state.

Conversation Dynamics

Recognizing speech and synthesizing it are only part of the job. The pipeline also manages the moment-to-moment dynamics of the conversation: which models handle each turn, how the agent adapts to the caller's state, what the caller hears while tools run, when the agent yields the floor, and how the call ends.

Split Model Architecture

Navigation and response generation can use different model preferences:

• Navigation returns a constrained state-routing decision and is commonly tuned for latency and consistency.
• Response generation produces caller-facing text and tool calls and is commonly tuned for response quality.

The configured models, observed latency, and output quality depend on the active version set. Model-backed routing and response generation remain probabilistic.

Situation-Response Adaptation

The voice pipeline adapts across four independent dimensions simultaneously. Each dimension operates on different output channels, so the agent can change what it says, how it says it, and whether it fills silence - all independently and in real time.

Emotion → Voice Tone

The agent mirrors empathy, not the caller's emotion. An angry caller hears a calm voice (de-escalation), not an angry one. An anxious caller hears a sympathetic voice (reassurance). A happy caller hears enthusiasm (matching energy).

Emotion → Filler Behavior

Filler speech adapts to the caller's emotional state. Anxious callers hear reassuring fillers ("Of course," "I'm here to help"). Frustrated callers with high arousal hear no fillers at all - the system suppresses them because frustrated callers want answers, not acknowledgments. Happy callers hear warm, matching fillers.

Emotion → Response Content

Once at least two acoustic segments are available, non-neutral rolling state or tracked behavioral signals can add emotional trend and adaptation guidance to the response prompt. Deteriorating negative state prioritizes empathy and resolution speed. Neutral, stable state without behavioral signals adds no emotion guidance.

Behavioral Signals → Response Content

Three behavioral signals - interruption count, short response streaks, and silence gaps - are tracked in real time and injected into prompts when thresholds are crossed, augmenting the emotion detection system with patterns that never appear in vocal tone. See Behavioral Signals for the signals, thresholds, and how the agent responds.

Response Micro-Behaviors

The voice channel prompt instructs the response model to use communication patterns such as:

• Speech rhythm mirroring - Short bursts from the caller produce concise responses; conversational callers get warmer, flowing replies
• Emotional name usage - The caller's name is used at moments of emotional significance, not mechanically
• Pause injection - When delivering difficult information, the agent pauses naturally before the key detail
• Pace inversion - When the caller is rushing, the agent slows down with longer sentences and gentle transitions
• Completion inference - When a caller trails off mid-sentence, the agent acknowledges what they were trying to say

The prompt instructs the agent not to mention emotion detection. Because response generation is model-backed, teams should validate this behavior in simulations and production review rather than treat a prompt instruction as an absolute control.

Voice Timeline

The voice pipeline applies the same cut/navigate/engage pattern that drives conversation-level reasoning - but within each turn, managing what the caller hears and when.

Fillers, responses, empathy pauses, and tool progress narration are treated as states in one timeline. "Let me check on that" followed by "Her appointment is Thursday" is one conversation trajectory in two parts, not two unrelated audio events.

Three Operations

1. Cut - A supported timing or session signal creates a new voice-timeline boundary. Signals that do not require a transition leave the current timeline state unchanged.
2. Navigate - Given the compressed state and the trajectory of previous states, select the next voice state. Navigation does not directly produce audio; engagement turns the selected state into an utterance.
3. Engage - Prepare an utterance with its own emotion, pace, and timing policy. If the main response is still pending, the voice timeline can move through filler, progress, or hold states without losing the caller's place.

Signal-to-State Mapping

Each signal produces a specific voice state:

Signal	Voice State	What Happens
Caller finished speaking	Breath	Brief pause before the agent responds
Navigation complete	Transition	Filler window opens if the response is not ready
Tool started	Progress	Tool wait narration on a repeating interval ("Let me check on that...")
Tool finished	Response	Agent delivers the tool result
All audio finished	Listen	Silence deadline starts - check-ins escalate if the caller stays quiet
Empathy tier shifted	Hold	Intentional silence - the agent pauses to give the caller space
Caller started speaking	Listen	Pending fillers drain - the caller has the floor
Deadline expired	Next state	Self-signal - the actor re-enters cut/navigate/engage

Timing policies are what make this self-driving. A transition state can wait briefly for the real response, play a filler when the response is not ready, or hold silence when empathy calls for it. If the response arrives first, the pending filler is skipped.

Per-Utterance Voice Parameters

Each utterance carries its own emotion and speed, set when the utterance is prepared. This keeps filler speech, progress narration, and final responses from overwriting one another when they are close together in time.

Voice Timing Configuration

The voice timeline exposes two categories of configuration per service:

When - timing policies for pauses, filler windows, progress narration, empathy holds, and cooldowns.

What - vocabulary and style:

Parameter	What It Controls
Filler style	Phrase, backchannel, or silent (see below)
Filler vocabulary	Custom backchannel words ("Mm," "Yeah," "Mhm")
Progress vocabulary	Custom tool-wait phrases ("One moment...," "Let me check...")

These settings shape the timeline, while the runtime still applies bounded queues, cooldowns, lifecycle checks, and error handling.

Filler Styles

The service schema accepts three filler-style values, but their current runtime behavior is not fully symmetric:

Style	Behavior	Best For
Phrase	Currently advisory; it follows the normal autonomous filler path	Reserved for phrase-oriented configuration
Backchannel	Currently follows the same normal autonomous filler path	Default non-silent behavior
Silent	Suppresses autonomous navigation backchannels and ordinary tool-wait acknowledgements	Services that prefer no autonomous filler

silent removes autonomous filler prompt pressure and audio in the normal path. Explicit deterministic progress phrases authored on a context-graph tool binding remain intentional script and can still play. Receipt and working fillers inherit the resolved utterance tone and speed when those values are present. Service filler vocabulary affects navigation backchannels; tool-wait vocabulary is configured separately.

When navigation is skipped - typically in single-action context graphs where the agent always stays in the same state - the orchestrator starts a short timer (configurable per service). If the response has not produced audio by the time the timer fires, a backchannel sound plays to hold the conversational rhythm. If the response arrives first, the timer is cancelled. Services using the "silent" filler style suppress this timer entirely.

Empathy-Gated Filler Behavior

Filler behavior is controlled by the caller's empathy tier. Higher tiers can add an intentional hold or suppress fillers:

• T0-T1 - Normal filler emission. At T1, the filler type is set to "empathy" (warmer, acknowledging) rather than "receipt" or "working."
• T2 Full Empathy - The voice timeline applies an empathy hold and the response prompt must lead with empathy. An empathy filler may still be used.
• T3 Hold Space - Fillers are suppressed entirely. The agent pauses, then delivers a pure empathy response.

The tier is assigned at the caller's final turn boundary and sent to the voice timeline before navigation completes for that turn.

Empathy fillers are marked separately from receipt and working fillers so the speaker can use slower delivery when no per-utterance speed is already present. Per-utterance tone and speed from the voice timeline take precedence over the speaker's fallback filler adjustments.

Principle-Based Filler Generation

Navigation can return a contextual filler using the current action and available emotional guidance. When navigation is skipped or retried, the runtime uses bounded backchannel or retry vocabularies instead. Tool-wait progress uses authored phrases or deterministic templates rather than making another model call on the voice latency path.

Autonomous navigation acknowledgements use cooldowns and a bounded retry count to prevent a burst of overlapping fillers. Tool-progress narration follows its own trigger and interval policy and can continue while a long-running tool remains active; later attempts reuse the final configured or templated pattern.

Tool-Wait Progress Hints

Fillers emitted while a tool is running can be shaped per state and per tool, not just per service. A progress hint describes the shape of the wait rather than supplying a phrase list: what kind of work the tool is doing (record lookup, write, external call, computation, multi-step workflow), roughly how long it is expected to take, and how the agent should cover the wait (auto, verbal, backchannel, or silent).

The orchestrator turns the hint into utterances using authored phrases, progress class, attempt count, and the active voice style. Later attempts use attempt-aware templates such as an acknowledgement followed by a continuing-status update. A tool-level hint field-merges with the state's channel-level hint, so individual tools need to override only the fields that differ.

For tools with expected latency of four seconds or more, a custom phrase can override the generated progress text. This gives agent engineers precise control over what callers hear during long waits - for example, a tool that queries multiple external systems. The custom phrase is bounded to 30 words and requires a progress class as fallback.

Tool-start signals hand progress narration to the voice timeline, which avoids a second independent tool-wait filler loop. Authored deterministic phrases remain part of that same timeline.

Result Persistence Modes

Tool call specs support a result_persistence setting that controls how tool results accumulate in the agent's prompt context:

Mode	Behavior	Best For
accumulate (default)	Every tool result is retained in the prompt history	Tools called once or a few times per conversation
override	Only the latest result per tool name is retained; previous results for the same tool are replaced	Polling tools called repeatedly (availability checks, status lookups) where only the most recent result matters

Override mode prevents context bloat from tools that the agent calls multiple times during a conversation. A scheduling agent that checks availability five times during a complex multi-provider booking only carries the most recent availability snapshot in its context, not all five results stacked up. This keeps the prompt focused and reduces token usage without losing the information the agent actually needs.

Context Window Management

The engine compares the most recent model request's input-token count with the configured context-window size. This measures the prompt that must fit now rather than cumulative usage across earlier calls.

When the current prompt reaches the compaction threshold (80% by default), the engine ratchets the session into compact history mode and caps retained prompt-history entries. Recent context stays detailed while older state groups are summarized. The policy does not create a separate warning tier or automatically escalate the call to an operator.

Barge-In Detection

If the caller starts speaking while the agent is talking, the system needs to decide whether to stop the agent's audio. Barge-in uses semantic confirmation - it requires actual recognized words rather than just acoustic energy. This filters out coughs, breathing, background conversation, and echo from the agent's own audio that would otherwise cause false interruptions.

The decision is based on four conditions evaluated together:

1. Whether the caller's speech contains actual recognized words from the speech-to-text engine (not breathing, echo, or background noise). Voice activity detection alone is not sufficient - the system requires at least one recognized word before triggering a barge-in.
2. Whether the speech has lasted long enough with recognized words (minimum duration is configurable per service). The default threshold is low enough that short responses like "yes," "no," and "okay" (200-400ms of speech) can trigger barge-in.
3. Whether the cooldown period has elapsed since the last barge-in (configurable per service, prevents rapid false triggers)
4. Whether the agent is currently speaking

When all conditions are met, the agent's audio stops and the system returns to listening mode. This prevents the agent from talking over a caller who is trying to ask a question or correct a misunderstanding.

There is also a fast path for end-of-turn interrupts. When the speech engine produces a complete transcript with an end-of-turn signal while the agent is speaking, the system interrupts the agent's audio immediately from the recognition listener rather than waiting for the transcript to pass through the processing queue. This eliminates queue latency on short-phrase interrupts where the caller finishes speaking quickly.

Response Length Enforcement

Supported TTS paths enforce per-service sentence and word caps while consuming the streamed model response. Once a cap is detected, the runtime stops accepting further response text for speech. Counting occurs on streamed text fragments, so the boundary is a practical output guard rather than a linguistic parser with perfect sentence segmentation.

Call Completion

When the agent reaches a terminal state in the context graph and decides to end the call, it signals its intent to hang up but does not disconnect immediately. The system waits for signal convergence: the agent's closing utterance must finish playing and any in-flight tool results must resolve before the call disconnects. If the caller speaks during this window (barge-in) or a transfer is initiated, the hangup intent is retracted and the conversation continues.

This is designed to let the closing message and pending tool work settle before disconnect. Transport failure or caller hangup can still end the call earlier.

Per-Service Voice Configuration

Voice behavior is configurable at the service level, allowing different services within the same workspace to have different voice characteristics. Unset fields do not all fall through one three-level hierarchy: provider and TTS fields use their field-specific service, agent, workspace, and environment rules; service tuning fields use documented runtime defaults or supported fallbacks; and Context Graph TurnPolicy controls separate per-state behavior. See Voice Control Plane. Per-service settings cover:

• Filler behavior - Silent versus normal autonomous filler behavior, navigation vocabulary, tool-progress vocabulary, and backchannel timing
• Barge-in sensitivity - Minimum speech duration and cooldown period
• Response limits - Maximum sentences and words per response
• End-of-turn detection - Eagerness threshold and timeout
• TTS settings - Model selection and buffer delay
• Voice timing - The "when" and "what" knobs described in Voice Timing Configuration above
• Call forwarding - Whether the agent can transfer calls to external numbers (opt-in, disabled by default). Supports both pre-configured forwarding numbers (from EHR location data or workspace settings) and dynamic forwarding to any E.164 number provided at call time

These settings are managed through the Platform API and the Agent Forge CLI.

After the Call

The real-time STT stream prioritizes speed over accuracy. Post-call re-transcription catches words the live stream may have missed.

Transcript verification. When enabled and a recording is available, a background pass creates another caller transcript with word timing and compares it with the live transcript. The platform stores the result and comparison measurements when processing succeeds. The second transcript is review evidence, not guaranteed ground truth, and it does not automatically change configured keyterms.

Quality analysis. When enabled, recording-based analysis scores five dimensions - task completion, information accuracy, conversation flow, error recovery, and caller experience - and can return an outcome, observed issues, and suggested recognition keyterms. Processing is asynchronous and best-effort; suggestions do not automatically update voice configuration.

Terminal intelligence. At call end, the primary voice runtime can derive a summary from observed session state: available emotion evidence, latency, conversation shape, tool performance, and operator involvement. A rule-based 0-100 quality score summarizes operational friction from the evidence available on that runtime path; it is not a clinical score or a complete measure of outcome quality.

The scoring dimensions, intelligence summaries, and the APIs that expose them are covered in the Call Intelligence section:

Intelligence Overview