Voice Agent
Real-time voice pipeline with emotion detection, context graph engine, tool execution, and post-call analysis.
The Amigo voice agent powers real-time, emotionally intelligent voice conversations. It handles inbound and outbound phone calls, executing context graph logic (based on a Hierarchical State Machine architecture) with speech understanding, text-to-speech, tool execution, safety monitoring, and continuous emotional adaptation. Every call connects to the world model, reads live patient context, writes confidence-gated clinical events, and adapts its behavior based on real-time vocal emotion analysis.
Reliability target. This system handles healthcare scheduling calls where callers may be in distress, pain, or crisis. Every design decision prioritizes graceful degradation: if any intelligence layer fails, the call continues with the next-best behavior, never silence.
Voice settings and Classic API differences. Voice settings (tone, speed, keyterms, sensitive topics, post-call flags) are configured at the workspace level; see Workspaces, Voice Settings. Classic API offers WebSocket voice streaming for text-based apps; Platform API voice is phone-based with emotion detection, EHR context, and operator escalation.
Audio Pipeline Architecture
Every voice call flows through a five-layer pipeline that transforms the caller's audio into emotionally adaptive agent speech, while simultaneously reading from and writing to the world model.
Layer 1: Signal Capture
Two parallel streams process the caller's audio simultaneously, and neither blocks the other. This dual-stream architecture is fundamental: speech recognition and emotion detection are completely independent. A failure in one never impacts the other.
Speech-to-Text: real-time streaming transcription with sub-300ms latency. Three layers of domain vocabulary boost recognition accuracy:
Service-level keyterms: managed by workspace administrators, applied to all calls for that service.
Workspace voice settings keyterms: API-configurable per workspace (see voice settings).
System defaults: engineering-level fallback vocabulary.
All sources are merged and deduplicated per call. Configurable end-of-turn detection with tunable confidence thresholds determines when the caller has finished speaking, balancing responsiveness against cutting off mid-sentence.
Emotion Detection: parallel audio analysis with zero impact on the voice pipeline. Three concurrent models analyze every audio segment:
Prosody
2-second audio segments
48 emotions from vocal tone, pitch, rhythm, timbre
Real-time voice quality analysis
Vocal Burst
Same audio segments
67 non-speech vocal types (laughs, sighs, cries, gasps, groans)
Captures sounds that transcription loses entirely
Language
Final STT transcripts
53 emotions + 9-point sentiment + 6-category toxicity
Detects sarcasm, tiredness, annoyance, disapproval, enthusiasm (5 emotions unavailable from audio alone)
Dual-payload multiplexing: audio segments request prosody and burst models; text transcripts request the language model. Responses are unambiguous: each contains only the models requested. This separation is architecturally important: the language model requires text input, not audio, and runs on STT output rather than raw audio, ensuring it analyzes what the caller said, not just how they sounded.
Audio buffering: audio is buffered in 2-second segments on bounded non-blocking queues. The emotion pipeline never blocks the voice pipeline, and dropped segments gracefully degrade to slightly less precise emotion detection rather than failure.
Circuit breaker protection: emotion detection is protected by a circuit breaker that briefly disables it after repeated failures, then automatically recovers. If the emotion pipeline is degraded, calls continue smoothly with workspace defaults; the circuit breaker prevents cascading latency from affecting the critical voice path.
Layer 2: Intelligence
The Emotional State maintains a rolling 30-second window (~15 segments) of recent caller signals with recency-weighted linear averaging, so the most recent signals have the highest influence. The agent responds to the caller's current emotional state, not an average of the whole call.
Valence/Arousal computation: every emotion maps to a (valence, arousal) coordinate via a complete emotion-dimension mapping. Weighted sums across all detected emotions per segment, then recency-weighted across the rolling window, produce stable yet responsive emotional tracking.
Trend detection: compares first-half vs second-half valence of the rolling window. A delta > 0.1 → improving; delta < -0.1 → deteriorating; otherwise stable. This powers the call-phase escalation system: deteriorating trends trigger increasingly urgent adaptation.
Coherence (prosody vs language agreement): Measures whether what the caller says matches how they sound:
Same valence sign (both positive or both negative)
High (0.7-1.0)
Normal adaptation
Opposite valence (words say fine, voice says distressed)
Low (0.0-0.3)
Trust the vocal tone over the words
One signal neutral
Mild (0.8)
Use the available signal
When coherence < 0.4: the caller may be masking their true state. The agent's emotional steering instruction shifts: "Respond to how they sound, not what they claim." This is injected into the system prompt without the agent ever explicitly mentioning the discrepancy.
Behavioral signal tracking: updated in real-time from the session and turn controller:
barge_in_count
Caller interrupts agent speech
≥ 2
Frustration (agent talking too much)
short_response_streak
Consecutive responses ≤ 4 words
≥ 3
Disengagement (caller withdrawing)
silence_gap_count
Gaps ≥ 5 seconds
≥ 2
Confusion, hesitation, or distress
Semantic barge-in detection. Barge-in detection uses semantic confirmation: it requires actual recognized words from the STT engine (not just voice activity detection). This filters false triggers from coughs, breathing, echo, and background noise. Barge-in requires both recognized words (interim STT output, not just voice activity) AND a minimum speech duration (default 0.25s, configurable per service via barge_in_min_speech_s). There is no duration-only fallback - duration alone never triggers a barge-in, which prevents false interrupts from coughs, breathing, echo, and background noise.
These behavioral signals are injected into the system prompt alongside emotional steering, so the LLM receives a complete picture of both how the caller sounds and how they're behaving.
Layer 3: Context Graph Engine
Each turn processes through a two-stage LLM pipeline:
Navigator: selects the next action or exit condition from the current context graph state. Also generates a filler phrase to cover processing latency. Uses structured output validation with automatic retry (up to 3 attempts) and fallback to first valid action.
Engage LLM: generates the caller-facing response, informed by the selected action, full conversation history with per-message emotion annotations (
[VOICE: EmotionName, valence=V.VVV]), emotional steering context, ambient patient context, and available tools.
Emotion reaches the LLM via two independent paths:
Per-message annotations
Every user message
Inline [VOICE: Anxiety, valence=-0.312], so the LLM sees the emotional trajectory across the full conversation
Session-level steering
System prompt
Dominant emotion + trend, quadrant-specific adaptation instructions, behavioral signals, call-phase urgency, coherence warnings
Communication micro-behaviors: the engage template contains hardcoded guidelines that instruct the LLM on micro-level conversational behaviors that are always active, not gated by emotion:
Speech rhythm mirroring
If the caller speaks in short bursts, respond concisely; if conversational, match warmth
Emotional name usage
Use the caller's name at moments of emotional significance, not mechanically
Pause injection
When delivering difficult information, pause naturally before the key detail
Pace inversion
When the caller is rushing, slow the pace with longer sentences and gentle transitions
Completion inference
When the caller trails off mid-sentence, acknowledge what they were trying to say
Emotion concealment
Never explicitly mention that the system can detect emotions
Natural laughter
Contextual laughter available for naturally warm moments, used sparingly
Layer 4: Audio Output
The engage LLM's text streams to the TTS engine for speech synthesis with per-turn dynamic controls:
Emotion: derived from the voice tone priority chain.
Speed: from workspace voice settings.
Volume: from workspace voice settings.
Word-level timestamps are collected for every generated word (start time and end time), enabling transcript-to-audio scrubbing in the call playback UI. This is critical for the review queue workflow where operators need to jump to specific moments in a call.
Layer 5: Post-Call Intelligence
Two optional analyses run after every call (controlled via voice settings):
Transcript verification: re-transcribes the full call audio with a high-accuracy batch model and computes Word Error Rate (WER) against the real-time transcript. Produces verified_transcript, verified_words, and transcript_accuracy, enabling quality comparisons between the real-time and batch transcription.
Quality analysis: listens to the full stereo recording (caller and agent) and scores on 5 dimensions (1-5 each):
Task Completion
Did the agent achieve the caller's goal?
Information Accuracy
Was the information provided correct?
Conversation Flow
Was the conversation natural and smooth?
Error Recovery
How well did the agent recover from mistakes?
Caller Experience
How did the caller feel at the end?
Call Intelligence Persistence
Alongside the LLM-based quality analysis, the voice agent computes a structured intelligence summary from in-memory session state at call end. This runs synchronously during session cleanup (before the session is torn down) and captures operational telemetry that the async quality analysis cannot see.
Each call intelligence record contains:
quality_score
float
Rule-based composite score (0-100), penalty-based
emotion_summary
object
dominant_emotion, average_valence, arousal, peak negative, shifts, final trend
latency_summary
object
Engine response time (avg/p50/p95), audio TTFB (avg/p50/p95), silence ratio
conversation_summary
object
turn_count, states_visited_count, unique_states, loop_count, barge-in count, completion reason
tool_summary
object
Total calls, success/failure counts, failure rate, per-tool breakdown
operator_summary
object
escalated (boolean), operator connect time, resolution
completion_reason
string
Why the call ended (hangup, terminal state, silence, etc.)
final_state
string
Last context graph state at call end
Quality score penalties:
High latency
p95 audio TTFB > 1000ms
-5 to -15
Silence
Silence ratio > 0.2
-10 to -20
Barge-ins
> 2
-5 to -15
Agent loops
> 0 revisited states
-10 to -20
Escalation
Any
-10
Tool failures
Failure rate > 5%
-5 to -15
The computation is pure (no I/O, no external calls); all data comes from in-memory session state. If the write fails, the error is logged but does not affect the caller or post-call processing.
Call Intelligence Endpoints
Two endpoints expose intelligence data for completed and active calls:
GET /v1/{workspace_id}/calls/{call_id}/intelligence. Full intelligence profile for a completed call.
call_id, call_sid
Call identifiers
direction, duration_seconds, created_at
Call metadata
quality_score
Composite 0-100 score
quality_breakdown
Per-signal breakdown of the quality score
key_moments
Notable moments detected during the call
emotion_summary, latency_summary, conversation_summary, tool_summary, operator_summary
Persisted intelligence summaries (see the persistence table above)
completion_reason, final_state
Why the call ended and the last context graph state
Returns 404 if the call or intelligence data is not found.
GET /v1/{workspace_id}/calls/active/intelligence. Active calls with live intelligence overlay.
Enriches the active call listing with per-turn intelligence from cached snapshots:
current_emotion
Live cache
Current detected emotion
current_valence
Live cache
Current emotional valence
turn_count
Live cache
Number of turns completed
escalation_active
Live cache
Whether operator escalation is active
current_state
Live cache
Current context graph state
Live Intelligence Pipeline
The voice agent writes a compact intelligence snapshot after each caller speech turn. The snapshot includes current emotion, turn count, escalation status, and current state.
Intelligence data is refreshed alongside the active call heartbeat. If the session ends or is lost, the live data expires automatically.
The active intelligence endpoint reads all live intelligence for active calls in a single operation for efficient dashboard polling.
Self-improving feedback loop: quality analysis also produces stt_suggestions, words the STT misheard, formatted as recognition keywords for future calls. This creates a closed loop:
How Calls Work
Every call runs inside a conference architecture, a multi-party audio bridge that lets the caller, AI agent, and optionally a human operator all participate simultaneously.
Inbound Call Flow (Instant Greeting)
The system eliminates dead air at call start through parallel pre-warming: the engine, greeting, and agent connection all initialize while the phone is still ringing.
Key insight: the telephony conference API accepts friendly names, not just IDs. The conference name is known at webhook time. The agent leg is created immediately; the conference is created on-demand when the agent joins. The agent can be fully connected and waiting before the caller even picks up.
Timeline comparison:
Webhook → Engine ready
After pickup (+1-3s)
During ring (hidden)
Agent leg creation
After pickup (+200ms)
During ring (hidden)
WebSocket connection
After pickup (+200ms)
During ring (hidden)
Greeting generation
After pickup (+500ms)
During ring (hidden)
Total dead air
~1200ms
~200-300ms (TTS streaming only)
Safety guarantees:
If the caller hangs up during ring, pre-warmed resources are cleaned up automatically.
If the pre-warmed state is unavailable at pickup, the call initializes normally with no degradation.
If pre-warm does not finish before pickup, the call proceeds with standard initialization.
Session capacity is NOT consumed during pre-warm (no active session yet).
Pre-warm is best-effort. If initialization takes longer than expected, the system falls back to standard initialization: no degradation in call quality, just a slightly longer time to first greeting.
Outbound Call Flow
Outbound calls are world-model-native: scheduled as outbound_task entities in the world model (by triggers, dashboards, or automated rules), then dispatched by the connector runner when they become due.
Five business logic patterns can produce outbound tasks:
Scheduled
Decision made, execution deferred
"I'll call you back tomorrow at 2pm"
Event-reactive
Trigger → evaluate → maybe act
New lab result → is it critical? → call patient
Continuous monitoring
Periodic population sweep
Patients with no contact in 30 days
Conversational follow-through
Track preconditions from agent promises
"I'll call after the doctor reviews" → pending on doctor event
Orchestrated campaign
Achieve outcome for population over time
"Get all 200 patients to complete annual wellness by Q4"
Each outbound task carries a patient reference, reason, goal, priority (1-10), business-hours window (timezone-aware), retry config (max attempts with configurable backoff), and rich context from the patient's world model projection. The dispatch loop enriches the system prompt so the agent starts the call with full patient knowledge. The agent never needs to "look up" the patient.
Outbound prewarm: outbound calls use the same parallel pre-warming as inbound calls. During the dialing/ringing phase (typically 5-15 seconds), the engine initializes, loads the context graph, resolves patient context, and generates the greeting. When the patient answers, the engine and greeting are already cached, so the patient hears an instant greeting instead of several seconds of silence. Prewarm is best-effort: if initialization takes longer than the ring time, the system falls back to standard cold initialization.
Conference Architecture
Conference architecture: telephony details
The conference architecture supports multiple simultaneous audio participants with independent per-participant streams:
Caller
Person who called or was called
PSTN
Dedicated per-participant stream
Agent
AI voice agent
Bidirectional WebSocket
Main session STT
Operator
Human monitor/takeover (optional)
PSTN or browser WebRTC
Dedicated per-participant stream
Three-party speaker resolution: when multiple parties are on the call, speaker attribution uses a priority chain: operator STT → caller STT → default (caller). Every turn in the call record carries speaker_id and speaker_role for accurate attribution in the transcript.
Context Graph Engine
The voice agent executes a Hierarchical State Machine loaded from the service's version set. Each call gets its own engine instance with an in-memory state database for zero-latency state tracking, flushed to persistent storage after the call ends.
State Types
ActionState
Agent performs actions and evaluates exit conditions to transition
Yes (Engage LLM)
DecisionState
Agent evaluates conditions and chooses a transition
Yes (Navigator only)
ReflectionState
Agent reasons deeply over a problem with optional tool calls
Yes (deep reasoning)
ToolCallState
Enforces execution of a designated tool before transitioning
No (automatic)
RecallState
Retrieves information from memory before transitioning
No (automatic)
AnnotationState
Injects an inner thought and transitions immediately
No (automatic)
Per-Turn Flow
The navigator handles multi-state traversal automatically. Decision states, annotation states, and recall states are resolved without user interaction before landing on an action state for the engage LLM.
Navigator resilience: structured output validation with automatic retry (up to 3 total attempts). When all retries are exhausted, the engine falls back to the first valid action or exit. Filler text from earlier attempts is preserved across retries (first-wins), so the caller never hears silence even during recovery.
Action State Extensions
Action states support three optional extension fields for asynchronous workflows:
wait_for
string
Pause navigation until an async condition clears. Values: surface_submission, human_approval.
channel_overrides
object
Per-channel overrides keyed by channel kind (voice, sms). Each override can set objective, action_guidelines, and progress (a progress hint for tool waits in that state).
surface_spec_template
object
Surface spec auto-created on state entry. Uses the same field schema as POST /surfaces. The entity ID defaults to the session's primary entity.
Wait conditions: when the navigator returns a waiting_for value, the engine skips context graph navigation on subsequent turns. The engage prompt includes a WAITING_FOR_CONDITION block that constrains the agent to empathetic small-talk until the condition clears. For voice sessions, clearance comes via the real-time event stream. For text sessions, the session blocks on a dedicated event listener.
Channel overrides: the channel_kind is set on the engine session (voice for calls, sms for text sessions). Prompt rendering merges the channel-specific objective and guidelines into the engage prompt. For voice, the override can also carry a progress hint that shapes how the agent covers tool waits in that state (see Tool-Wait Progress Hints).
Surface templates: on state entry, if the new state has a surface_spec_template, the engine creates the surface via the platform API and tracks the surface_id in the session's active surface set. This enables deterministic surface creation as part of the context graph design rather than relying on agent tool calls.
Terminal State & Auto-Hangup
When the context graph reaches its terminal state (an ActionState with one action and zero exits), the agent speaks its goodbye and automatically ends the call:
Navigator lands on terminal state →
is_terminal = true.Agent speaks the goodbye response.
Waits for TTS to finish plus a grace period (audio buffer flush).
Terminates the call via telephony API.
Silence detection: when the caller goes silent, the silence monitor fires check-ins at increasing intervals (10s → 20s → 40s). After 3 unanswered check-ins, the agent says a brief goodbye and auto-disconnects.
Session shutdown contract: every code path that stops the session must also stop the audio speaker; otherwise the speaker blocks indefinitely. This is enforced across all shutdown triggers: hangup, STT failure, WebSocket disconnect, and terminal state.
World Model Integration
The voice agent connects to the workspace's world model through three data channels. This architecture is informed by the Liquid World Model thesis where the distinction between data infrastructure and intelligence dissolves.
Channel 1: Ambient (Pushed)
Data the LLM should always have without asking. Injected into the system prompt at session start and refreshed as the conversation evolves:
Patient demographics: name, DOB, MRN, phone, email, address.
Clinical context: active conditions, medications, allergies (filtered to text-only for LLM consumption).
Upcoming appointments: with patient entity references for cross-referencing.
Insurance coverage: active plans and subscriber info.
Location context: clinic details, available appointment types, hours (resolved from the inbound phone number).
Design principle: ambient over queried. If the LLM will almost certainly need this data, push it into context. Don't make it ask. A voice agent that already has the patient's insurance in context doesn't need to dispatch a tool call to look it up.
Channel 2: Queried (Pulled)
Data that can't be ambient because the search space is too large. The agent calls built-in clinical tools to retrieve specific information.
Key simplification: queried tools return human-readable results, not database internals. Slot search returns doctor names and times, not template IDs and slot UUIDs. When the agent says "book the 1:45 with Dr. Jones," the system resolves scheduling internals from cached slot data. The LLM never touches scheduling internals.
Channel 3: Extracted (Captured)
Structured data mentioned in conversation (insurance details, contact information, preferences) is automatically captured and written to the world model without requiring explicit tool calls. This eliminates the mode switch where the LLM stops being a conversationalist and becomes a database operator. The conversation IS the data entry.
Extracted data is written with moderate confidence (below verified threshold). The LLM can still use explicit write tools for high-stakes data where precision matters. Extraction is a complement, not a replacement.
Confidence-Gated Writes
All data written by the voice agent during calls carries a confidence level: a source-class ranking, not a subjective score. Confidence gates what is allowed to sync to external systems. This is the trust architecture for autonomous agents acting on noisy phone audio.
0.0
Rejected
Human-rejected or system-invalidated
0.3
Agent raw
Raw voice agent inference (unverified)
0.5
Self report
Patient-confirmed or patient-submitted data
0.7
Verified
Verified or EHR-ingested data
0.9
High
Operator-verified or high-quality source
0.95
Human approved
Human review approved
1.0
Authoritative
Manual entry or authoritative API
Confirmation-gated confidence: the agent chooses a confirmation level per write based on how the information surfaced in conversation - confirmed (the caller explicitly confirmed the fact, written at self-report confidence), mentioned (mentioned in passing, written at agent-raw confidence), or inferred (inferred from context, written below agent-raw confidence).
Outbound sync gate: the connector runner only syncs world events to external systems (such as EHRs) when their confidence meets the verified threshold. Low-confidence agent writes stay in the world model until they are corroborated or approved.
Approval-gated external writes: writes to external integrations can additionally require explicit human approval before egress. Pending writes appear as external write proposals in the Review Queue, where reviewers approve or reject them.
Patient Safety Isolation
A write scope is enforced per session: write tools can only target the patient identified in the current call. This prevents cross-patient data errors. Write tools also have in-flight deduplication: if the LLM re-calls the same write tool with identical parameters while the first call is still executing, the duplicate is suppressed instead of creating a second record.
Emotional Adaptation
The voice agent adapts across four independent output channels simultaneously based on real-time caller emotion. Each row in the matrix below is a detected situation; columns show how each output channel responds. All adaptation is automatic; workspace managers control only the baseline via voice settings.
Valence-Arousal Model
Every detected emotion maps to a two-dimensional (valence, arousal) coordinate. The system tracks these coordinates across a rolling window to build a stable yet responsive picture of the caller's emotional state:
High-arousal negative (anger, frustration)
De-escalate
calm
Direct, concise, acknowledge frustration, skip pleasantries, match urgency
Low-arousal negative (sadness, disappointment)
Comfort
sympathetic
Warm, patient, gentle language, give extra space, do not rush
High-arousal positive (excitement, joy)
Match energy
enthusiastic
Enthusiastic language, keep momentum, match positive energy
Low-arousal positive (contentment, relief)
Maintain
content
Warm and steady, reinforce positive outcome, conversational
Confusion (high confidence)
Clarify
calm
Simplify explanations, break into small pieces, check understanding, offer to repeat
Anxiety (high confidence)
Reassure
sympathetic
Calm and reassuring, provide clear next steps, avoid uncertainty
Voice Tone Priority Chain
The agent's voice tone is determined by a six-level priority chain. Each layer fires only if the previous returned no signal:
Why this ordering matters:
Bursts are the highest-priority signal because they capture the most immediate emotional state. A caller who just laughed should hear warmth immediately, not the rolling average of the last 30 seconds. Burst detection (within last 5 seconds, confidence ≥ 0.5) overrides everything.
Tone momentum (layer 4) prevents jarring voice tone changes. When the current emotional signal is weak (score < 0.25) or doesn't map to a tone, the previous turn's tone persists. Only a strong contradictory signal changes the tone, making the voice feel continuous across the conversation:
Proactive topic sensitivity (layer 3) fires before the caller shows distress. When the agent is about to discuss test results, billing, surgery, or other loaded topics, the voice tone preemptively shifts to sympathetic, even without an emotion signal.
Emotion → Response Matrix
All four adaptation channels respond simultaneously to each caller state. The agent mirrors empathy, not the caller's emotion:
Anger, Annoyance, Contempt
calm
Suppressed
Direct, concise, acknowledge frustration
De-escalate without mirroring aggression
Anxiety, Fear, Distress
sympathetic
Reassuring
Calm, clear next steps, avoid uncertainty
Reassure with a steady presence
Sadness, Disappointment, Guilt
sympathetic
Warm
Patient, supportive, don't rush
Warm empathy, give space
Confusion
calm
Simple
Simplify, small pieces, check understanding
Patient clarity
Excitement, Joy, Enthusiasm
enthusiastic
Warm, matching
Match positive energy, keep momentum
Mirror positive energy
Contentment, Relief, Gratitude
content
Warm
Steady, reinforce outcome
Warm and grounding
Interest, Concentration
curious
Engaged
Engaged tone, match intellectual focus
Show interest
Embarrassment, Doubt
calm
Encouraging
Non-judgmental, encouraging
Put at ease
Boredom, Tiredness
enthusiastic
Concise
Re-engage with energy, be efficient
Re-energize
Sarcasm
calm
Professional
Respond to underlying concern, not surface tone
Stay professional
Burst-to-experience mapping: 25 vocal burst types are mapped to specific agent tones and caller state interpretations:
Laugh, Giggle
enthusiastic
Amused
Sigh
sympathetic
Weary
Cry, Sob, Whimper
sympathetic
Distressed
Gasp
calm
Alarmed
Groan, Ugh
sympathetic / calm
Frustrated
Growl, Tsk
calm
Angry
Hmm, Mhm
calm
Thinking / Acknowledging
Aww
sympathetic
Touched
Filler Speech
Fillers cover processing latency so the caller never hears silence. The system uses principle-based guidance rather than hardcoded phrase lists, generating contextually appropriate fillers from emotional context, the current action, and the expected latency.
Three-layer filler generation:
Latency adaptation
Always
Filler length matches expected processing time (typically 2-4 words)
Emotional attunement
When emotion data available
Emotional register matches the caller's state. Not specific phrases, but principles like "gentle and reassuring" or "a verbal hand on the shoulder".
Action context
Always
Current context graph action description injected so the filler hints at what the agent is about to do
Per-action filler hints: context graph actions can include optional PM-configured filler suggestions. These are weak steering; emotion-adaptive principles always dominate. The LLM sees hints as suggestions to draw from, not commands.
Suppression rule: when valence < -0.2 AND arousal > 0.4 AND emotion is NOT Anxiety/Fear/Distress → fillers disabled entirely. Frustrated callers don't want acknowledgments, they want the answer. Exception: anxious callers still receive reassuring fillers, because anxiety benefits from reassurance while frustration does not.
Tool-Wait Progress Hints
When a tool is running, the agent narrates the wait based on a ProgressHint declared on the state's channel override and/or on the individual tool binding. The hint describes the shape of the wait, not a phrase list - the engine composes the actual utterance from tool semantics, turn emotion, and conversation context.
mode
"auto" | "silent" | "backchannel" | "verbal"
How the agent covers the wait. auto lets the engine decide from class and latency; silent drops filler text; backchannel emits a single token ("Mm."); verbal produces a contextual filler.
progress_class
"lookup" | "write" | "external_call" | "compute" | "multi_step" | null
Semantic category of the work the tool is doing. Drives the templated language the engine picks for retries.
expected_latency_ms
integer (0-60000) | null
How long the tool is expected to take. Used to choose filler length and pacing.
trigger_delay_ms
integer (0-30000) | null
How long to wait after a tool starts before playing the first filler. Set to 0 for immediate playback. When unset, the delay is calculated from expected_latency_ms; when unset and deterministic phrases are configured, the first filler fires immediately.
interval_ms
integer (100-30000) | null
Milliseconds between subsequent fillers. Overrides the global progress interval for this hint.
deterministic
boolean (default false)
Whether phrases play verbatim. true: scripted - the engine plays the phrases exactly as written, in order. false: the engine picks - phrases are hints.
phrases
array of strings (1-10 items) | null
Ordered filler phrases, each at most 30 words. With deterministic: true, phrases play verbatim in order, cycling back to the start for attempts beyond the list length. With deterministic: false (the default), phrases are used as hints for early attempts - attempt N plays phrases[N-1] while within the list, and later attempts fall back to class templates, vocabulary, and retry fillers. backchannel mode preempts phrases. When set, phrases supersedes custom_phrase.
custom_phrase
string (max 500 chars) | null
Operator-authored filler utterance for the first filler attempt. In auto mode: requires expected_latency_ms >= 4000 and progress_class to be set. In verbal mode: always honored (word cap still applies). Ignored when phrases is set. Subsequent attempts use class templates.
Placement: progress can be set on channel_overrides[channel].progress (state-level default for that channel) and on each entry of action_tool_call_specs or exit_condition_tool_call_specs (per tool). Per-tool fields merge field-wise over the channel override, so a state can declare the default wait shape once and individual tools only override fields that actually differ.
Retry narration: when a tool retries, the engine uses deterministic attempt-aware templates keyed on progress_class - acknowledgement on attempt 1, brief apology on attempt 2, a "still working" update from attempt 3 - so retry audio stays off the LLM hot path and latency stays bounded. With deterministic: true, the engine skips templates and plays the scripted phrases in order, cycling through the list; with deterministic: false, phrases cover early attempts and templates take over for attempts beyond the list.
Deterministic vs non-deterministic: scripted flows (demo flows, regulated disclosures) that need exact control over what the agent says during tool waits must set phrases with deterministic: true. Deterministic fillers require the tool binding's execution: "background" - write-time validation rejects a blocking binding combined with deterministic: true. Use custom_phrase + progress_class when you want to steer the first filler but let the engine handle retries with contextual templates.
Call Phase Escalation
The system automatically increases urgency as calls extend with negative sentiment:
Early
< 5 min
Any
Standard emotional adaptation
Mid
5-10 min
Trend deteriorating
"Focus on resolution speed. Shorten responses."
Late
≥ 10 min
Negative valence
URGENCY. "Prioritize resolution. Be maximally concise. Escalate if unable to resolve."
Proactive Intelligence
The system detects emotionally sensitive topics from the current context graph action before the caller shows distress:
sensitive_topics is configurable via voice settings. Falls back to healthcare defaults: test results, diagnosis, billing, payment, insurance, denial, emergency, referral, specialist, surgery, procedure, medication.
This fires at priority level 3 in the TTS emotion chain, below burst and prosody (which have actual data about the caller's current state) but above tone momentum and workspace defaults.
Coherence Detection
When what the caller says doesn't match how they sound (coherence < 0.4), the system shifts its steering: "The caller's words suggest X but voice sounds Y. Trust the vocal tone over the words, respond to how they sound, not what they claim."
This is injected into the system prompt without the agent ever explicitly mentioning the discrepancy to the caller.
Control Plane ↔ Adaptation
How each workspace voice setting interacts with the automatic emotion adaptation system:
tone
Baseline voice emotion for neutral callers
Emotion-derived tone replaces it
Any non-neutral emotion detected (score ≥ 0.25)
speed
Base speech rate
Never overridden
Your choice is always respected
volume
Base volume
Never overridden
Your choice is always respected
voice_id
Voice persona
Per-agent voice config overrides
Agent version has voice config set
keyterms
Domain vocabulary for STT boost
Merged with service keyterms
Always additive, never overridden
sensitive_topics
Topics for proactive tone softening
Falls back to healthcare defaults
Preemptive, not reactive
post_call_analysis_enabled
Quality scoring on/off
None
Full PM control
transcript_correction_enabled
Re-verification on/off
None
Full PM control
Key principle: workspace managers control the baseline experience and domain knowledge. The emotion intelligence system overrides the baseline only when it detects a strong signal, and always in the direction of more empathy, never less.
Graceful Degradation
Every intelligence layer is best-effort with an explicit fallback. A failed intelligence layer must never fail a call.
Emotion connection
Auth error, billing, timeout
Session continues without emotion detection
No emotional adaptation, workspace defaults used
Emotion segment
Processing error, connection close
Consecutive failure counter → disable after 5
Degrades gracefully to less data
Emotion detection
Insufficient data (< 2 segments)
No emotional steering, default fillers
First few seconds may lack adaptation
Burst detection
No burst events
Falls through to prosody-derived emotion
Loses immediate reaction, uses rolling average
Language model
No language results
Coherence defaults to 1.0 (agreement assumed)
Loses word-vs-tone disagreement detection
Voice settings
Parse error
Defaults (filler on, emotion on)
Baseline experience still works
Post-call analysis
Any error
Logged, not raised (fire-and-forget)
Quality data missing, call unaffected
TTS connection
Close/error mid-stream
Auto-reconnect on next turn
Brief silence, then recovery
STT connection
Connection loss
Exponential backoff reconnect (max 3 attempts)
Brief gap in transcription
Context graph engine
Backend unavailable
Falls back to static prompt mode (without context graph navigation)
Agent still converses, just without state machine navigation
Tool Execution
Skills configured in the context graph execute asynchronously during calls. The agent acknowledges the action and continues speaking while tools run in the background. Results arrive as continuation turns.
Execution Model
Tool calls are routed by tool family, and each state-level tool binding declares how the call executes and how its result is delivered:
World tools
world_*
Built-in reads/writes against the world model (patient, appointment, insurance, typed entity lookups)
Skills
(by slug)
LLM-backed companion micro-agents: multi-turn reasoning with the skill's configured integration and static tools (see Skills)
System tools
-
forward_call, complete_transfer, resume_patient, get_tool_call_status, cancel_background_task; handled inline by the turn controller
Blocking vs background execution: each state x tool binding declares an execution axis. Blocking tools are awaited inline. Background tools are fired without awaiting - the agent acknowledges and keeps talking, and the result arrives later as a continuation turn. The binding's delivery axis controls how that result lands: interrupt (immediate continuation turn, cutting current speech) or queue (folded in silently on the next user turn). A separate failure_delivery axis can route failures differently from successes.
State gating: tool calls are validated against the tools actually offered to the LLM for the current context graph state. A hallucinated call to a tool that is not available in the current state is rejected with an error result rather than executed.
In-flight write deduplication: if the LLM re-calls the same write tool with identical parameters while the first call is still running, the duplicate is suppressed.
Built-in Clinical Tools
Workspaces get a catalog of built-in world_* tools automatically, with no integration configuration required. Context graphs reference these tools by ID; the same catalog is validated when a context graph is saved.
Read tools:
world_patient_lookup
Search patients by DOB, name, phone, or MRN (DOB preferred for accuracy)
world_persons_lookup
Generic person-entity lookup with server-side filters
world_slot_search
Available appointment slots by location and date. Returns human-readable times and doctor names; slot internals are cached and resolved server-side
world_appointment_lookup
Patient's existing appointments, with references for cancel/confirm/reschedule
world_payer_search
Look up insurance payers/carriers
world_reference_data_lookup
Look up reference data (locations, appointment types, providers)
world_get_medications
Patient's medications and prescription references
world_get_encounters
Patient's encounter history
world_queue_lookup
Read the queue of pending outbound call tasks
world_conditions_lookup, world_allergies_lookup, world_observations_lookup, world_medication_statements_lookup, world_clinical_notes_lookup, world_family_history_lookup, world_questionnaire_responses_lookup, world_intake_uploads_lookup, world_care_plans_lookup, world_goals_lookup, world_tasks_lookup, world_service_requests_lookup, world_consents_lookup, world_diagnostic_reports_lookup
Typed read tools over projected entity shapes
world_conversations_lookup, world_calls_lookup
Conversation and call record lookups
world_state_resolve
Batch entity-ID to current world-state resolver
Write tools:
world_patient_create
Create a patient with automatic deduplication (name + DOB)
world_patient_update
Update contact info (phone, email, address); requires an entity reference
world_save_patient
Create-or-update with dedup check; accepts natural field names and flexible date formats
world_schedule_appointment
Book from slot search results; auto-resolves booking details from cached slot data
world_cancel_appointment
Cancel by appointment reference
world_confirm_appointment
Confirm a booked appointment
world_reschedule_appointment
Move an appointment to a new slot