> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/developer-guide/platform-api/conversations/voice-agent.md). # Voice Agent The Amigo voice agent powers real-time voice conversations. It handles inbound and outbound phone calls with speech understanding, text-to-speech, tool execution, safety monitoring, and context-graph or native/realtime runtime behavior. Supported service configurations can load projected patient context, emit source-attributed world events, and adapt responses from best-effort acoustic analysis; those capabilities vary by runtime and configured tools. {% hint style="warning" %} **Reliability target.** This system handles healthcare scheduling calls where callers may be in distress, pain, or crisis. Optional intelligence features use best-effort fallbacks where the active runtime supports them. Failures in required telephony, speech, model, or runtime dependencies can still delay or end a call, so monitor call and session outcomes. {% endhint %} {% hint style="info" %} **Voice settings and Classic API differences.** Voice settings (tone, speed, keyterms, sensitive topics, post-call flags) are configured at the workspace level; see [Workspaces, Voice Settings](/developer-guide/platform-api/workspaces.md#voice-settings). Classic API offers [WebSocket voice streaming](/developer-guide/classic-api/core-api/conversations/conversations-voice.md) for text-based apps. Platform API voice supports phone calls and, when configured, acoustic emotion analysis, projected context, and operator escalation. {% endhint %} ## Audio Pipeline Architecture The standard phone-call runtime uses the five-layer pipeline below. Optional emotion, world-model, and post-call stages depend on service configuration, available data, and runtime support. ```mermaid %%{init: {"theme": "base", "themeVariables": {"primaryColor": "#D4E2E7", "primaryTextColor": "#100F0F", "primaryBorderColor": "#083241", "lineColor": "#575452", "textColor": "#100F0F", "clusterBkg": "#F1EAE7", "clusterBorder": "#D7D2D0"}}}%% flowchart TB subgraph Input["1. Signal Capture"] A["Caller Audio\n(telephony stream)"] A --> B["Speech-to-Text\n(streaming)"] A --> C["Emotion Detection\n(9 acoustic classes + V/A)"] end subgraph Intel["2. Intelligence Layer"] D["Emotional State\n(4-segment roll +\n5-turn compounds)"] E["Voice Context\n(per-turn steering)"] C --> D --> E end subgraph Engine["3. Context Graph Engine"] F["Navigator\n(select action + filler)"] G["Engage LLM\n(generate response)"] B --> F --> G E -->|"emotional steering\n+ filler guidelines"| F E -->|"emotional steering\n+ micro-behaviors"| G end subgraph Output["4. Audio Output"] H["Text-to-Speech\n(emotion-adaptive,\nprovider timing metadata)"] G --> H E -->|"emotion + speed\n+ volume"| H end subgraph Post["5. Post-Call Intelligence"] I["Transcript Verification\n(batch re-transcription)"] J["Quality Analysis\n(5-dimension scoring)"] JJ["STT Suggestions\n(review evidence)"] J --> JJ end subgraph World["World Model"] K["Patient Context\n(ambient injection)"] L["Source-Attributed Events\n(confidence metadata)"] end K -.->|"ambient context\n(3 channels)"| G G -.->|"tool results\n(confidence-gated writes)"| L ``` ### Layer 1: Signal Capture Speech recognition and optional acoustic emotion analysis use separate processing paths. Backpressure on the emotion path can drop analysis segments rather than blocking transcription; shared session, audio, or infrastructure failures can still affect both paths. **Speech-to-Text**: real-time streaming transcription. Three sources of domain vocabulary can improve recognition of configured terms: 1. **Service-level keyterms**: managed by workspace administrators, applied to all calls for that service. 2. **Workspace voice settings keyterms**: API-configurable per workspace (see [voice settings](/developer-guide/platform-api/workspaces.md#voice-settings)). 3. **System defaults**: engineering-level fallback vocabulary. All sources are merged and deduplicated per call. Configurable end-of-turn detection with tunable confidence thresholds determines when the caller has finished speaking, balancing responsiveness against cutting off mid-sentence. **Emotion detection** is a best-effort parallel stream. The runtime converts supported caller audio and sends approximately 2-second segments to the emotion service. Each result contains a nine-class acoustic score distribution plus valence and arousal. Raw classes are mapped to runtime labels such as Anger, Disgust, Fear, Joy, Calmness, Sadness, Surprise, and Contemplation. Final transcripts can be sent separately for sentiment and toxicity analysis. Acoustic inference and transcript analysis are independent inputs; the current contract does not include a separate vocal-burst taxonomy. The emotion service can return a dominance value for observer telemetry, but dominance is not used in the rolling state or turn-level compound resolver. Runtime adaptation uses the mapped acoustic scores, valence/arousal, transcript signals when available, and behavioral/contextual evidence. **Backpressure behavior**: acoustic analysis uses approximately 2-second audio segments. Under load, a segment can be dropped rather than delaying the transcription and call path. **Failure behavior**: an initial connection failure leaves the call running without acoustic adaptation. Repeated receive failures disable the emotion stream for that session after the configured limit; the contract does not promise automatic reconnection during the same call. ### Layer 2: Intelligence The **Emotional State** maintains the latest four acoustic segments (about 8 seconds at the normal segment size) with linear recency weighting. At each caller turn, the controller snapshots that state and keeps up to five turn snapshots for compound-emotion resolution. ```mermaid %%{init: {"theme": "base", "themeVariables": {"primaryColor": "#D4E2E7", "primaryTextColor": "#100F0F", "primaryBorderColor": "#083241", "lineColor": "#575452", "textColor": "#100F0F", "clusterBkg": "#F1EAE7", "clusterBorder": "#D7D2D0"}}}%% flowchart TB subgraph Inputs["Signal Sources"] P["Acoustic inference\n(9 classes + valence/arousal)"] L["Transcript analysis\n(sentiment + toxicity)"] BH["Behavioral\n(barge-ins, silences,\nshort responses)"] end subgraph State["Emotional State (Latest 4 Segments)"] V["Valence: -1.0 to +1.0"] AR["Arousal: 0.0 to 1.0"] DOM["Dominant emotion + score"] TR["Trend: improving/stable/deteriorating"] COH["Coherence: acoustic vs transcript"] PH["Call phase: early/mid/late"] end subgraph Turns["Turn Window (Latest 5 Caller Turns)"] CE["Compound emotions\n(acoustic + linguistic + behavioral + context)"] end subgraph Output["Per-Turn Voice Context"] TTS["TTS: emotion + speed + volume"] FG["Filler: guidelines + suppression"] ES["Emotional steering → system prompt"] FE["Filler enabled/disabled"] end P --> State L --> State BH --> State State --> Turns State --> Output Turns --> Output ``` **Valence/arousal aggregation**: the emotion engine supplies valence and arousal per segment. The runtime linearly weights the latest four segments, derives an internal dominant label from the aggregate, and uses the latest acoustic distribution plus the five-turn window to resolve compound labels. Dominance is currently fixed to `0.0` in those turn snapshots and does not affect resolution. **Trend detection**: compares first-half vs second-half valence of the rolling window. A delta > 0.1 → **improving**; delta < -0.1 → **deteriorating**; otherwise **stable**. This powers the call-phase escalation system: deteriorating trends trigger increasingly urgent adaptation. **Acoustic/transcript coherence**: when transcript sentiment is available, the runtime compares it with acoustic valence and accounts for toxicity. Near-neutral voice is treated as ambiguous rather than automatic agreement. Coherence remains telemetry and steering input; it is absent or less informative when transcript analysis is unavailable. **Behavioral signal tracking**: updated in real-time from the session and turn controller: | Signal | Detection | Threshold | Meaning | | ----------------------- | ------------------------------- | --------- | ------------------------------------ | | `barge_in_count` | Caller interrupts agent speech | ≥ 2 | Frustration (agent talking too much) | | `short_response_streak` | Consecutive responses ≤ 4 words | ≥ 3 | Disengagement (caller withdrawing) | | `silence_gap_count` | Gaps ≥ 5 seconds | ≥ 2 | Confusion, hesitation, or distress | {% hint style="info" %} **Semantic barge-in detection.** Barge-in detection uses semantic confirmation: it requires actual recognized words from the STT engine (not just voice activity detection). This filters false triggers from coughs, breathing, echo, and background noise. Barge-in requires both recognized words (interim STT output, not just voice activity) AND a minimum speech duration (default 0.25s, configurable per service via `barge_in_min_speech_s`). There is no duration-only fallback - duration alone never triggers a barge-in, which prevents false interrupts from coughs, breathing, echo, and background noise. {% endhint %} These behavioral signals can be injected into the system prompt alongside emotional steering, giving the LLM additional evidence about both *how the caller sounds* and *how they are behaving*. ### Layer 3: Context Graph Engine Each turn processes through a **two-stage LLM pipeline**: 1. **Navigator**: selects the next action or exit condition from the current context graph state. Also generates a filler phrase to cover processing latency. Uses structured output validation with automatic retry (up to 3 attempts) and fallback to first valid action. 2. **Engage LLM**: generates the caller-facing response, informed by the selected action, available conversation history with per-message emotion annotations (`[VOICE: EmotionName, valence=V.VVV]`), emotional steering context, ambient patient context, and available tools. **Emotion reaches the LLM via two independent paths:** | Path | Scope | What It Contains | | --------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | **Per-message annotations** | Annotated user messages retained in context | Inline `[VOICE: Anxiety, valence=-0.312]`, so the LLM can use the emotional trajectory in the available history | | **Session-level steering** | System prompt | Dominant emotion + trend, quadrant-specific adaptation instructions, behavioral signals, call-phase urgency, coherence warnings | **Communication micro-behaviors**: when the standard engage template is used, it includes guidelines for micro-level conversational behaviors independently of whether emotion data is available: | Behavior | Description | | --------------------------- | ---------------------------------------------------------------------------------------- | | **Speech rhythm mirroring** | If the caller speaks in short bursts, respond concisely; if conversational, match warmth | | **Emotional name usage** | Use the caller's name at moments of emotional significance, not mechanically | | **Pause injection** | When delivering difficult information, pause naturally before the key detail | | **Pace inversion** | When the caller is rushing, slow the pace with longer sentences and gentle transitions | | **Completion inference** | When the caller trails off mid-sentence, acknowledge what they were trying to say | | **Emotion concealment** | Instruct the model not to mention emotion detection explicitly | | **Natural laughter** | Contextual laughter available for naturally warm moments, used sparingly | ### Layer 4: Audio Output The engage LLM's text streams to the TTS engine for speech synthesis with **per-turn dynamic controls**: * **Emotion**: selected through the [voice tone selection](#voice-tone-selection) path. * **Speed**: from workspace [voice settings](/developer-guide/platform-api/workspaces.md#voice-settings). * **Volume**: from workspace voice settings. When the active TTS provider supplies word timing metadata, the runtime can retain start/end offsets for transcript-to-audio alignment. Clients must not assume every provider or utterance includes complete word-level timestamps. ### Layer 5: Post-Call Intelligence When enabled in [voice settings](/developer-guide/platform-api/workspaces.md#voice-settings) and the required recording is available, two background analyses can run after a call. Both are best-effort and can be absent on an otherwise completed call. **Transcript verification**: re-transcribes the inbound caller recording with a batch model and computes whole-call Word Error Rate (WER) against the real-time caller transcript. Produces `verified_transcript`, `verified_words`, and `transcript_accuracy`. Per-turn accuracy values remain unavailable because the two timing domains do not align reliably. **Quality analysis**: listens to the full stereo recording (caller and agent) and scores on 5 dimensions (1-5 each): | Dimension | What It Measures | | ------------------------ | --------------------------------------------- | | **Task Completion** | Did the agent achieve the caller's goal? | | **Information Accuracy** | Was the information provided correct? | | **Conversation Flow** | Was the conversation natural and smooth? | | **Error Recovery** | How well did the agent recover from mistakes? | | **Caller Experience** | How did the caller feel at the end? | #### Call Intelligence Persistence Alongside the quality analysis, the platform records a structured intelligence summary at call end. It captures operational signals that are not available from transcript scoring alone. Fully analyzed call intelligence records contain: | Field | Type | Description | | -------------------------------- | ------------- | --------------------------------------------------------------------------------------------------------------------- | | `quality_score` | float or null | Rule-based operational quality score (0-100), penalty-based | | `emotion_summary` | object | `dominant_emotion`, `average_valence`, arousal, peak negative, shifts, final trend | | `latency_summary` | object | Engine response time (avg/p50/p95), audio TTFB (avg/p50/p95), silence ratio | | `conversation_summary` | object | `turn_count`, `states_visited_count`, `unique_states`, `loop_count`, barge-in count, completion reason | | `tool_summary` | object | Total calls, success/failure counts, failure rate, per-tool breakdown | | `operator_summary` | object | Whether the call escalated and, when it did, whether an operator was active | | `risk_summary`, `safety_summary` | object | Compatibility fields that are empty in the current runtime; they are not active risk-scoring or safety-triage outputs | | `completion_reason` | string | Why the call ended (hangup, terminal state, silence, etc.) | | `final_state` | string | Last context graph state at call end | Some external voice runtimes persist only a terminal call envelope. Those records retain call identifiers, duration, direction, completion reason, final state, and turn count, but leave `quality_score` null and the analysis summaries empty. **Quality score penalties:** | Signal | Threshold | Penalty | | ------------- | ----------------------- | ---------- | | High latency | p95 audio TTFB > 1000ms | -5 to -15 | | Silence | Silence ratio > 0.2 | -10 to -20 | | Barge-ins | > 2 | -5 to -15 | | Agent loops | > 0 revisited states | -10 to -20 | | Escalation | Any | -10 | | Tool failures | Failure rate > 5% | -5 to -15 | #### Call Intelligence Endpoints The call intelligence endpoint exposes the persisted terminal summary for a completed call: {% openapi src="" path="/v1/{workspace\_id}/calls/{call\_id}/intelligence" method="get" %} {% endopenapi %} Returns 404 if the call or intelligence data is not found. **Recognition suggestions**: quality analysis can return `stt_suggestions` for words it believes the live recognizer misheard. The runtime publishes up to five suggestions with the post-call completion event for review. It does not automatically add them to workspace or service keyterms; an administrator must validate and configure any change explicitly. ## How Calls Work Supported phone-call flows use a **conference architecture**, a multi-party audio bridge that lets the caller, AI agent, and optionally a human [operator](/developer-guide/platform-api/conversations/operators.md) participate simultaneously. ### Inbound Call Preparation Supported inbound telephony paths can reduce startup delay through best-effort **parallel pre-warming**: engine initialization, greeting preparation, patient-context resolution, and the agent connection can begin while the phone is ringing. Completion before pickup is not guaranteed. ```mermaid %%{init: {"theme": "base", "themeVariables": {"actorBkg": "#083241", "actorTextColor": "#FFFFFF", "actorBorder": "#083241", "signalColor": "#575452", "signalTextColor": "#100F0F", "labelBoxBkgColor": "#F1EAE7", "labelBoxBorderColor": "#D7D2D0", "labelTextColor": "#100F0F", "loopTextColor": "#100F0F", "noteBkgColor": "#F1EAE7", "noteBorderColor": "#D7D2D0", "noteTextColor": "#100F0F", "activationBkgColor": "#E8E2EB", "activationBorderColor": "#083241", "altSectionBkgColor": "#F1EAE7", "altSectionColor": "#100F0F"}}}%% sequenceDiagram participant Caller participant Tel as Telephony participant Agent as Voice Agent Caller->>Tel: Dials phone number Tel->>Agent: Inbound webhook Note over Agent: Resolve: phone → workspace → service → version set par Pre-warm during ring time (parallel) Agent->>Agent: Initialize engine + load context graph + load tools Agent->>Agent: Generate greeting text via LLM Agent->>Agent: Resolve caller → patient context from world model Agent->>Tel: Create agent conference leg (via conference name) Tel->>Agent: Agent WebSocket connects Note over Agent: Greeting ready if preparation completes end Note over Tel: Phone rings... Caller->>Tel: Picks up Tel->>Agent: Caller joins conference Agent-->>Caller: Greeting begins when ready loop Conversation Turns Caller->>Agent: Speech audio (bidirectional stream) par Signal Processing Agent->>Agent: STT (transcript + end-of-turn) Agent->>Agent: Emotion (acoustic + transcript signals) end Agent->>Agent: Navigator → Filler → Engage LLM → TTS Agent-->>Caller: Emotionally adaptive audio response end Note over Agent: Call ends (terminal state, silence, or hangup) Note over Agent: Persist call record + emotional summary Note over Agent: Post-call analysis (background) ``` The conference name is available at webhook time in the standard telephony implementation, allowing the runtime to begin creating the agent leg before pickup. Whether the agent connection, context, and greeting are ready at answer time depends on ring duration and downstream dependencies. {% hint style="info" %} **Pre-warm is best-effort.** Startup work that has not finished by pickup continues through the ordinary initialization path. Time to first audio varies by provider, runtime, configuration, and dependency health; pre-warming is not a latency guarantee. {% endhint %} ### Outbound Call Flow Outbound voice has two entry paths. [On-demand outbound calls](/developer-guide/platform-api/conversations/calls.md) start directly through the Calls API. Scheduled outreach is represented as an `outbound_task` entity and dispatched by the [connector runner](/developer-guide/platform-api/data-world-model/connector-runner.md) after its asynchronous projection becomes due. This section describes the scheduled path. **Common workflow patterns** that an application can implement with outbound tasks include: | Pattern | Description | Example | | --------------------------------- | ---------------------------------------- | -------------------------------------------------------------- | | **Scheduled** | Decision made, execution deferred | "I'll call you back tomorrow at 2pm" | | **Event-reactive** | Trigger → evaluate → maybe act | New lab result → is it critical? → call patient | | **Continuous monitoring** | Periodic population sweep | Patients with no contact in 30 days | | **Conversational follow-through** | Track preconditions from agent promises | "I'll call after the doctor reviews" → pending on doctor event | | **Orchestrated campaign** | Achieve outcome for population over time | "Get all 200 patients to complete annual wellness by Q4" | ```mermaid %%{init: {"theme": "base", "themeVariables": {"primaryColor": "#D4E2E7", "primaryTextColor": "#100F0F", "primaryBorderColor": "#083241", "lineColor": "#575452", "textColor": "#100F0F", "clusterBkg": "#F1EAE7", "clusterBorder": "#D7D2D0"}}}%% flowchart TB subgraph Producers["Task Producers"] A["Trigger action"] B["Application workflow"] C["Authorized event producer"] end subgraph WM["World Model"] E["outbound_task\nentity"] end subgraph Dispatch["Dispatch Loop"] F{"Due?\nBusiness hours?\nRetry budget?"} G["Build available context\nfrom patient projection"] H["Dispatch call"] end A --> E B --> E C --> E E --> F F -->|Yes| G --> H F -->|No| I["Wait for\nnext window"] H --> J["Voice agent\nexecutes with available\nprojected context"] ``` Each outbound task can carry a patient reference, reason, goal, priority (1-10), business-hours window (timezone-aware), retry config, and context from the patient's world-model projection. The dispatch loop can enrich the system prompt with context resolved at dispatch time. Missing, stale, or non-projected facts can still require a tool lookup or caller confirmation. **Outbound preparation**: the current standard outbound path prepares the first agent audio before dialing the human leg. If that preparation fails or exceeds its budget, the request fails instead of falling through to an unprepared dial. Provider connection and playback can still add latency after answer, so this is not a time-to-first-audio guarantee. ### Conference Architecture
Conference architecture: telephony details The conference architecture supports multiple simultaneous audio participants with independent per-participant streams: ```mermaid %%{init: {"theme": "base", "themeVariables": {"primaryColor": "#D4E2E7", "primaryTextColor": "#100F0F", "primaryBorderColor": "#083241", "lineColor": "#575452", "textColor": "#100F0F", "clusterBkg": "#F1EAE7", "clusterBorder": "#D7D2D0"}}}%% flowchart TB subgraph Conference["Telephony Conference"] C["Caller\n(PSTN - phone network)"] A["AI Agent\n(WebSocket stream)"] O["Operator\n(PSTN or WebRTC)\n[optional]"] end subgraph STT["Per-Participant Speech-to-Text"] CS["Caller STT\n(speaker attribution)"] OS["Operator STT\n(human transcript capture)"] AS["Agent STT\n(turn processing)"] end C --> CS O --> OS A --> AS subgraph Resolution["Speaker Resolution"] R["Priority: Operator > Caller > Default"] end CS --> Resolution OS --> Resolution ``` | Participant | Role | Audio Transport | STT | | ------------ | --------------------------------- | ----------------------- | -------------------------------- | | **Caller** | Person who called or was called | PSTN | Dedicated per-participant stream | | **Agent** | AI voice agent | Bidirectional WebSocket | Main session STT | | **Operator** | Human monitor/takeover (optional) | PSTN or browser WebRTC | Dedicated per-participant stream | **Three-party speaker resolution**: when multiple parties are on the call, speaker attribution uses a priority chain: operator STT → caller STT → default (caller). Recorded turns can carry `speaker_id` and `speaker_role` to support transcript attribution; clients should tolerate missing or ambiguous metadata when a stream or persistence step is incomplete.
## Context Graph Engine The standard HSM voice runtime executes a **Hierarchical State Machine** loaded from the service's version set. Each call maintains isolated conversation state and attempts to persist the resulting call record when the call ends. Native and realtime runtimes can use the same Context Graph as workflow guidance without following the identical navigator path. ### State Types | State Type | Purpose | LLM Call? | | ----------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------- | | **ActionState** | Produces user-facing responses, eligible tool calls, and exit-condition evaluation | Yes (engagement and navigation as needed) | | **DecisionState** | Selects among authored exits without producing a user-facing turn | Yes (navigation only) | | **AnnotationState** | Injects authored internal guidance and advances to its fixed next state | No additional model call | | **DataCollectionState** | Collects declared fields across turns, with turn limits and optional Surface fallback | Yes (engageable) | ### Per-Turn Flow ```mermaid %%{init: {"theme": "base", "themeVariables": {"actorBkg": "#083241", "actorTextColor": "#FFFFFF", "actorBorder": "#083241", "signalColor": "#575452", "signalTextColor": "#100F0F", "labelBoxBkgColor": "#F1EAE7", "labelBoxBorderColor": "#D7D2D0", "labelTextColor": "#100F0F", "loopTextColor": "#100F0F", "noteBkgColor": "#F1EAE7", "noteBorderColor": "#D7D2D0", "noteTextColor": "#100F0F", "activationBkgColor": "#E8E2EB", "activationBorderColor": "#083241", "altSectionBkgColor": "#F1EAE7", "altSectionColor": "#100F0F"}}}%% sequenceDiagram participant Caller participant STT as Speech-to-Text participant Emo as Emotion Detection participant Nav as Navigator participant Engage as Engage LLM participant TTS as Text-to-Speech participant Tools as Tool Executor Caller->>STT: Speech audio par Signal Processing STT->>Nav: Transcript + end-of-turn Caller->>Emo: Audio when analysis is enabled Emo->>Nav: Emotional state when available end Nav->>Nav: Select state/action + optional filler opt Filler selected Nav->>TTS: Filler phrase TTS-->>Caller: Filler audio plays end Nav->>Engage: State guidance + available context Engage->>TTS: Response text (streaming) TTS-->>Caller: Response audio with supported tone controls opt Tool calls in response Engage->>Tools: Dispatch eligible tool Tools-->>Engage: Result or later continuation when successful end ``` The navigator handles multi-state traversal automatically. Decision and annotation states can resolve without user interaction before landing on an action state for the engage LLM. **Navigator resilience**: structured output validation retries up to 3 total attempts. When all retries are exhausted, the standard runtime can fall back to the first valid action or exit. Filler text from an earlier successful attempt is preserved across retries (first-wins), which can reduce quiet gaps but does not guarantee continuous audio during recovery. ### Action State Extensions Action states support three optional extension fields for asynchronous workflows: | Field | Type | Description | | ----------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `wait_for` | string | Pause navigation until an async condition clears. Values: `surface_submission`, `human_approval`. | | `channel_overrides` | object | Per-channel overrides keyed by channel kind (`voice`, `sms`). Each override can set `objective`, `action_guidelines`, and `progress` (a [progress hint](#tool-wait-progress-hints) for tool waits in that state). | | `surface_spec_template` | object | Surface spec auto-created on state entry. Uses the [Surfaces](/developer-guide/platform-api/conversations/surfaces.md) field schema. The entity ID defaults to the session's primary entity. | **Wait conditions**: when the navigator returns a `waiting_for` value, the engine skips context graph navigation on subsequent turns. The engage prompt includes a `WAITING_FOR_CONDITION` block that constrains the agent to empathetic small-talk until the condition clears. For voice sessions, clearance comes via the real-time event stream. For text sessions, the session blocks on a dedicated event listener. **Channel overrides**: the `channel_kind` is set on the engine session (`voice` for calls, `sms` for text sessions). Prompt rendering merges the channel-specific objective and guidelines into the engage prompt. For voice, the override can also carry a `progress` hint that shapes how the agent covers tool waits in that state (see [Tool-Wait Progress Hints](#tool-wait-progress-hints)). **Surface templates**: on state entry, if the new state has a `surface_spec_template`, the engine requests a surface through the Platform API and tracks the `surface_id` when creation succeeds. This makes surface creation configuration-driven rather than dependent on a model-selected tool call. ### Terminal State & Auto-Hangup When the standard runtime reaches the Context Graph's configured terminal state, it can speak a final response and schedule an automatic hangup: 1. Navigator lands on terminal state → `is_terminal = true`. 2. Agent speaks the goodbye response. 3. Waits for TTS to finish plus a grace period (audio buffer flush). 4. Terminates the call via telephony API. **Silence detection**: when the caller goes silent, the silence monitor fires check-ins at increasing intervals (10s → 20s → 40s). After 3 unanswered check-ins, the agent says a brief goodbye and auto-disconnects. **Session shutdown behavior**: the standard shutdown handlers stop both the session and audio speaker for hangup, STT failure, WebSocket disconnect, and terminal-state paths. Clients should rely on the reported terminal outcome rather than assuming cleanup timing. ## World Model Integration The voice agent connects to the workspace's [world model](/developer-guide/platform-api/data-world-model.md) through three data channels. See [Storage and Projection Boundaries](/developer-guide/platform-api/data-world-model.md#storage-and-projection-boundaries) for the distinction between durable events and asynchronously projected entity state. ```mermaid %%{init: {"theme": "base", "themeVariables": {"primaryColor": "#D4E2E7", "primaryTextColor": "#100F0F", "primaryBorderColor": "#083241", "lineColor": "#575452", "textColor": "#100F0F", "clusterBkg": "#F1EAE7", "clusterBorder": "#D7D2D0"}}}%% flowchart TB subgraph WM["World Model"] EV["Source-attributed events\n(confidence metadata)"] EN["Entities\n(asynchronous projected state)"] EG["Entity Graph\n(relationships)"] EV -.->|"asynchronous projection"| EN EN --- EG end subgraph Channels["Three Data Channels"] direction TB A["Ambient (pushed)\nPatient state in system prompt\nLocation context\nRelated entities"] B["Queried (pulled)\nSlot search, patient lookup\nTyped entity lookup"] C["Extracted (captured)\nInsurance details from speech\nContact info from conversation"] end subgraph LLM["LLM Context"] CTX["System prompt + conversation history\n+ tool results + emotional steering"] end EN -->|"At session start +\nmid-call refresh"| A B <-->|"Tool calls\n↔ results"| EN C -->|"Transcript extraction\n(moderate confidence)"| EV A --> CTX B --> CTX C -.->|"implicit capture"| EV ``` ### Channel 1: Ambient (Pushed) Supported runtimes can inject selected ambient data without an explicit agent lookup. Available fields depend on identity resolution, projected state, service configuration, and successful prompt construction, and can include: * **Patient demographics**: name, DOB, MRN, phone, email, address. * **Clinical context**: active conditions, medications, allergies (filtered to text-only for LLM consumption). * **Upcoming appointments**: with patient entity references for cross-referencing. * **Insurance coverage**: active plans and subscriber info. * **Location context**: clinic details, available appointment types, hours (resolved from the inbound phone number). **Design principle: ambient over queried.** If the LLM will almost certainly need this data, push it into context. Don't make it ask. A voice agent that already has the patient's insurance in context doesn't need to dispatch a tool call to look it up. ### Channel 2: Queried (Pulled) Data that can't be ambient because the search space is too large. The agent calls [built-in clinical tools](#built-in-clinical-tools) to retrieve specific information. **Key simplification**: standard queried tools return human-readable results rather than database internals. Slot search can return doctor names and times while the runtime carries the selected scheduling identifiers into a later booking request, so raw template IDs and slot UUIDs do not need to be exposed to the LLM. ### Channel 3: Extracted (Captured) Supported voice runtimes can extract structured observations such as insurance details, contact information, and preferences from conversation evidence. When extraction and event publication succeed, those observations enter the world model without a separate conversational tool call. Extraction is model-generated, can be incomplete, and is not enabled identically on every runtime. Extracted data uses a source-appropriate confidence class below authoritative records. Explicit write tools remain the controlled path for supported high-stakes mutations. Extraction complements those tools; it does not prove that every mentioned fact was captured or externally delivered. ### Confidence-Gated Writes All data written by the voice agent during calls carries a **confidence level**: a source-class ranking, not a subjective score. Confidence gates what is allowed to sync to external systems. This is the trust architecture for autonomous agents acting on noisy phone audio. | Confidence | Level | Source Class | | ---------- | -------------- | ------------------------------------------- | | 0.0 | Rejected | Human-rejected or system-invalidated | | 0.3 | Agent raw | Raw voice agent inference (unverified) | | 0.5 | Self report | Patient-confirmed or patient-submitted data | | 0.7 | Verified | Verified or EHR-ingested data | | 0.8 | EHR trusted | Selected trusted clinical data | | 0.9 | High | Operator-verified or high-quality source | | 0.95 | Human approved | Human review approved | | 1.0 | Authoritative | Manual entry or authoritative API | **Confirmation-gated confidence**: the agent chooses a confirmation level per write based on how the information surfaced in conversation - `confirmed` (the caller explicitly confirmed the fact, written at self-report confidence), `mentioned` (mentioned in passing, written at agent-raw confidence), or `inferred` (inferred from context, written below agent-raw confidence). **Outbound sync gate**: configured connectors evaluate confidence together with the event source, supported mapping, dependency state, destination policy, and any review configuration. Meeting the configured threshold can be necessary for unattended delivery but is not sufficient by itself. Low-confidence observations remain available as source evidence unless a separate supported workflow acts on them. **External write proposals**: a connector workflow can stage a write for review before destination delivery where the private-preview proposal capability is enabled. Reviewers decide those proposals through the [Review Queue](/developer-guide/platform-api/integrations/review-queue.md). This is separate from an integration's conversation-scoped `approval_policy`, which applies to supported text conversations and is not a voice-call gate. ### Patient Safety Isolation Supported entity-anchored write tools enforce a **write scope** for the session and reject a target outside the resolved patient anchor. This is an authorization guard, not proof that the selected field, value, or patient anchor is clinically correct. Write tools also have **in-flight deduplication**: if the model re-calls the same write tool with identical parameters while the first call is still executing, the concurrent duplicate is suppressed. Once the first execution finishes, the same parameters can execute again; this guard is not general request idempotency or an exactly-once guarantee. ## Emotional Adaptation When acoustic data is available, the rolling valence/arousal state can influence prompt steering, TTS tone, filler behavior, and empathy pacing. These effects are best-effort and depend on the active runtime and speech provider; workspace and service settings can override or disable them. ### Valence-Arousal Model The emotion engine returns valence and arousal for each acoustic segment. The runtime aggregates the latest four segments and derives an internal label from that two-dimensional state: ``` High Arousal (1.0) │ ANGER ───┼─── EXCITEMENT Frustration│ Joy Fear │ Enthusiasm │ ───────────┼───────────── Valence Negative │ Positive (-1.0) │ (+1.0) │ SADNESS ──┼─── CONTENTMENT Disappointment Relief Boredom │ Gratitude │ Low Arousal (0.0) ``` | Quadrant | Agent Strategy | Voice Tone | LLM Behavior | | ------------------------- | -------------- | ----------------------------- | ----------------------------------------------------------- | | **High-arousal negative** | De-escalate | `calm` when supported | Direct, concise, acknowledge frustration, skip pleasantries | | **Low-arousal negative** | Comfort | `sympathetic` when supported | Warm, patient, gentle language, give extra space | | **High-arousal positive** | Match energy | `enthusiastic` when supported | Keep momentum and match positive energy | | **Low-arousal / neutral** | Stay steady | `content` when supported | Conversational and grounded | ### Voice Tone Selection The standard voice-context path chooses a TTS tone in this order: 1. With at least two acoustic segments and aggregate signal strength of at least `0.25`, map the rolling internal label to a supported TTS tone. 2. If no new mapped tone is available, reuse the previous mapped tone as momentum. 3. If neither produces a tone and the current action matches a configured sensitive topic, select `sympathetic`. 4. If workspace `tone` is configured, it explicitly overrides the computed tone. Tone momentum prevents a weak or unmapped segment from snapping the voice back to a default: ``` Turn 1: Sadness aggregate (score 0.72) → "sympathetic" → stored as momentum Turn 2: weak aggregate (score 0.18) → momentum returns "sympathetic" Turn 3: Joy aggregate (score 0.65) → "enthusiastic" → replaces momentum ``` The turn-level compound resolver can also attach labels derived from acoustic scores, sentiment/toxicity, barge-in and silence behavior, empathy tier, and conversation context. Those compound labels are stored with turn/call intelligence; they are not a separate vocal-burst model or an independent first-priority TTS signal. ### Filler Speech Fillers can cover processing latency. The system uses **principle-based guidance** rather than relying only on hardcoded phrase lists, generating contextually appropriate fillers from available emotional context, the current action, and expected latency. **Three-layer filler generation:** | Layer | When | What It Controls | | ------------------------ | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | **Latency adaptation** | When filler generation runs | Filler length can reflect expected processing time (typically 2-4 words) | | **Emotional attunement** | When emotion data available | Emotional register matches the caller's state. Not specific phrases, but principles like "gentle and reassuring" or "a verbal hand on the shoulder". | | **Action context** | When an action description is available | The action description can steer a filler toward what the agent is about to do | **Per-action filler hints**: context graph actions can include optional PM-configured filler suggestions. In non-deterministic mode, the LLM sees them as suggestions alongside emotional and contextual guidance, not commands. Explicit deterministic phrases follow the separate contract below. **Suppression rule**: when `valence < -0.2 AND arousal > 0.4 AND emotion is NOT Anxiety/Fear/Distress` → fillers disabled entirely. Frustrated callers don't want acknowledgments, they want the answer. **Exception**: anxious callers still receive reassuring fillers, because anxiety benefits from reassurance while frustration does not. #### Tool-Wait Progress Hints When a tool is running, the agent narrates the wait based on a `ProgressHint` declared on the state's channel override and/or on the individual tool binding. The hint describes the *shape* of the wait, not a phrase list - the engine composes the actual utterance from tool semantics, turn emotion, and conversation context. | Field | Type | Description | | --------------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mode` | `"auto"` \| `"silent"` \| `"backchannel"` \| `"verbal"` | How the agent covers the wait. `auto` lets the engine decide from class and latency; `silent` drops filler text; `backchannel` emits a single token ("Mm."); `verbal` produces a contextual filler. | | `progress_class` | `"lookup"` \| `"write"` \| `"external_call"` \| `"compute"` \| `"multi_step"` \| null | Semantic category of the work the tool is doing. Drives the templated language the engine picks for retries. | | `expected_latency_ms` | integer (0-60000) \| null | How long the tool is expected to take. Used to choose filler length and pacing. | | `trigger_delay_ms` | integer (0-30000) \| null | How long to wait after a tool starts before playing the first filler. Set to `0` for immediate playback. When unset, the delay is calculated from `expected_latency_ms`; when unset and deterministic phrases are configured, the first filler fires immediately. | | `interval_ms` | integer (100-30000) \| null | Milliseconds between subsequent fillers. Overrides the global progress interval for this hint. | | `deterministic` | boolean (default `false`) | Whether `phrases` play verbatim. `true`: scripted - the engine plays the phrases exactly as written, in order. `false`: the engine picks - phrases are hints. | | `phrases` | array of strings (1-10 items) \| null | Ordered filler phrases, each at most 30 words. With `deterministic: true`, phrases play verbatim in order, cycling back to the start for attempts beyond the list length. With `deterministic: false` (the default), phrases are used as hints for early attempts - attempt N plays `phrases[N-1]` while within the list, and later attempts fall back to class templates, vocabulary, and retry fillers. `backchannel` mode preempts phrases. When set, `phrases` supersedes `custom_phrase`. | | `custom_phrase` | string (max 500 chars) \| null | Operator-authored filler utterance for the first filler attempt. In `auto` mode: requires `expected_latency_ms` >= 4000 and `progress_class` to be set. In `verbal` mode: always honored (word cap still applies). Ignored when `phrases` is set. Subsequent attempts use class templates. | **Placement**: `progress` can be set on `channel_overrides[channel].progress` (state-level default for that channel) and on each entry of `action_tool_call_specs` or `exit_condition_tool_call_specs` (per tool). Per-tool fields merge field-wise over the channel override, so a state can declare the default wait shape once and individual tools only override fields that actually differ. **Retry narration**: when a tool retries, the engine uses deterministic attempt-aware templates keyed on `progress_class` - acknowledgement on attempt 1, brief apology on attempt 2, a "still working" update from attempt 3 - so retry audio stays off the LLM hot path and latency stays bounded. With `deterministic: true`, the engine skips templates and plays the scripted phrases in order, cycling through the list; with `deterministic: false`, phrases cover early attempts and templates take over for attempts beyond the list. **Deterministic vs non-deterministic**: scripted flows (demo flows, regulated disclosures) that need exact control over what the agent says during tool waits must set `phrases` with `deterministic: true`. Deterministic fillers require the tool binding's `execution: "background"` - write-time validation rejects a blocking binding combined with `deterministic: true`. Use `custom_phrase` + `progress_class` when you want to steer the first filler but let the engine handle retries with contextual templates. ### Call Phase Escalation The system automatically increases urgency as calls extend with negative sentiment: | Phase | Duration | Condition | Adaptation | | --------- | -------- | ------------------- | ------------------------------------------------------------------------------------------ | | **Early** | < 5 min | Any | Standard emotional adaptation | | **Mid** | 5-10 min | Trend deteriorating | "Focus on resolution speed. Shorten responses." | | **Late** | ≥ 10 min | Negative valence | **URGENCY.** "Prioritize resolution. Be maximally concise. Escalate if unable to resolve." | ### Proactive Intelligence The system detects emotionally sensitive topics from the current context graph action **before the caller shows distress**: ```mermaid %%{init: {"theme": "base", "themeVariables": {"primaryColor": "#D4E2E7", "primaryTextColor": "#100F0F", "primaryBorderColor": "#083241", "lineColor": "#575452", "textColor": "#100F0F", "clusterBkg": "#F1EAE7", "clusterBorder": "#D7D2D0"}}}%% flowchart TB A["Current Action:\n'Discuss test results'"] --> B{"Matches\nsensitive_topics?"} B -->|Yes| C["Preemptive shift to\nsympathetic tone\n(priority level 3\nin tone chain)"] B -->|No| D["Normal tone\npriority chain"] ``` `sensitive_topics` is configurable via [voice settings](/developer-guide/platform-api/workspaces.md#voice-settings). Falls back to healthcare defaults: test results, diagnosis, billing, payment, insurance, denial, emergency, referral, specialist, surgery, procedure, medication. This fallback applies only when acoustic selection and tone momentum produced no tone. An explicitly configured workspace `tone` then overrides the result. ### Coherence Detection When what the caller *says* doesn't match how they *sound* (coherence < 0.4), the system shifts its steering: *"The caller's words suggest X but voice sounds Y. Trust the vocal tone over the words, respond to how they sound, not what they claim."* This is injected into the system prompt without the agent ever explicitly mentioning the discrepancy to the caller. ### Control Plane ↔ Adaptation How each workspace voice setting interacts with the automatic emotion adaptation system: | Voice Setting | What You Control | What the System Overrides | Override Condition | | ------------------------------- | ----------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------- | | `tone` | Explicit workspace TTS tone | Replaces acoustic, momentum, and sensitive-topic selection when configured | Whenever the setting is present | | `speed` | Base speech rate | Can be reduced by supported empathy pacing logic | When an empathy baseline is active; configured service floors apply | | `volume` | Base volume | No adaptive override in the standard voice-context path | Provider support still applies | | `voice_id` | Voice persona | Per-agent voice config overrides | Agent version has voice config set | | `keyterms` | Domain vocabulary for STT boost | Merged with service keyterms and deduplicated | Subject to selected STT provider support | | `sensitive_topics` | Topics for proactive tone softening | Falls back to healthcare defaults | Preemptive, not reactive | | `post_call_analysis_enabled` | Request quality scoring on/off | None | Analysis remains best-effort and dependency-dependent | | `transcript_correction_enabled` | Request re-verification on/off | None | Analysis remains best-effort and recording-dependent | **Key principle**: automatic acoustic selection is best-effort. Explicit workspace and service settings remain authoritative where the active runtime supports them. ### Graceful Degradation The standard runtime implements the best-effort behaviors below. Optional emotion and post-call failures can leave those features unavailable without ending an otherwise healthy call, but failures in required telephony, speech, model, or context-engine dependencies can interrupt or end the session. | Layer | Failure Mode | Fallback | Impact | | ------------------------ | --------------------------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | **Emotion connection** | Auth error, timeout, unavailable service | Session continues without emotion detection | No acoustic adaptation; configured defaults remain | | **Emotion stream** | Repeated processing error or connection close | Disable for the remainder of the session after the configured consecutive-failure limit | No automatic same-call recovery guarantee | | **Emotion detection** | Insufficient data (< 2 segments) | No emotional steering, default fillers | First few seconds may lack adaptation | | **Transcript analysis** | No sentiment/toxicity result | Continue with acoustic and behavioral inputs | Compound/coherence evidence is reduced | | **Voice settings** | Parse error | Defaults (filler on, emotion on) | Baseline experience still works | | **Post-call analysis** | Any error | Logged, not raised (fire-and-forget) | Quality data missing, call unaffected | | **TTS connection** | Close/error mid-stream | A later turn can attempt reconnection | Audio gap; session can fail if recovery does not succeed | | **STT connection** | Connection loss | Reconnection is attempted where supported | Transcription gap; session can fail after recovery is exhausted | | **Context graph engine** | Backend unavailable | Any configured runtime fallback is attempted | The response or session can fail when no fallback is available | ## Tool Execution Tool bindings can execute inline or in the background during calls. Background bindings let the agent acknowledge the action and continue speaking while results arrive through the configured continuation-delivery mode. ```mermaid %%{init: {"theme": "base", "themeVariables": {"primaryColor": "#D4E2E7", "primaryTextColor": "#100F0F", "primaryBorderColor": "#083241", "lineColor": "#575452", "textColor": "#100F0F", "clusterBkg": "#F1EAE7", "clusterBorder": "#D7D2D0"}}}%% flowchart TB A["LLM returns\ntool call"] --> B["Filler plays\nwhile tool runs"] B --> C["Tool executes\n(async, background)"] C --> D["Result arrives"] D --> E["Agent relays\nresult to caller"] ``` ### Execution Model Tool calls are routed by tool family, and each state-level tool binding declares how the call executes and how its result is delivered: | Tool Family | Prefix | Execution Model | | -------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **World tools** | `world_*` | Built-in reads/writes against the world model (patient, appointment, insurance, typed entity lookups) | | **Surface tools** | `surface_*` | Create, deliver, and track [surfaces](/developer-guide/platform-api/conversations/surfaces.md) | | **Skills** | (by slug) | LLM-backed companion micro-agents: multi-turn reasoning with the skill's configured integration and static tools (see [Skills](/developer-guide/platform-api/workspaces/skills.md)) | | **Platform functions** | `fn_*` | Registered [platform functions](/developer-guide/platform-api/functions.md) executed against the data warehouse | | **Workspace data queries** | `wsq_*` | Parameterized SQL against [workspace tables](/developer-guide/platform-api/functions/workspace-data-queries.md) | | **System tools** | - | `forward_call`, `complete_transfer`, `resume_patient`, `get_tool_call_status`, `cancel_background_task`; handled inline by the turn controller | **Blocking vs background execution**: each state x tool binding declares an `execution` axis. Blocking tools are awaited inline. Background tools start without being awaited, so the agent can acknowledge and keep talking. When background execution and delivery succeed, the result can arrive as a continuation turn. The binding's `delivery` axis controls how that result lands: `interrupt` (a continuation turn that can cut current speech) or `queue` (folded in on the next user turn). A separate `failure_delivery` axis can route reported failures differently from successes. **State gating**: in the standard HSM runtime, tool calls are validated against the tools offered for the current context-graph state, and an unavailable call is rejected. Native and realtime runtimes can also attach shared platform or system tools globally to each phase. In every runtime, authorization, explicit tool enablement, and patient/write scope remain the security boundary; state membership alone is not one. **In-flight write deduplication**: an identical write call is suppressed only while the first call is still running. It does not deduplicate a later retry after completion and does not provide request-level exactly-once semantics. ### Built-in Clinical Tools The Platform API publishes a catalog of built-in `world_*` tools. A supported runtime can expose eligible catalog entries after applying service configuration, Context Graph bindings, authorization, and entity scope. Catalog presence is therefore discovery information, not proof that a tool is callable in every session. **Read tools:** | Tool | Purpose | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `world_patient_lookup` | Search patients by DOB, name, phone, or MRN (DOB preferred for accuracy) | | `world_persons_lookup` | Generic person-entity lookup with server-side filters | | `world_slot_search` | Available appointment slots by location and date. Returns human-readable times and doctor names while retaining the identifiers needed for a later scheduling call | | `world_appointment_lookup` | Patient's existing appointments, with references for cancel/confirm/reschedule | | `world_payer_search` | Look up insurance payers/carriers | | `world_reference_data_lookup` | Look up reference data (locations, appointment types, providers) | | `world_get_medications` | Patient's medications and prescription references | | `world_get_encounters` | Patient's encounter history | | `world_queue_lookup` | Read the queue of pending outbound call tasks | | `world_conditions_lookup`, `world_allergies_lookup`, `world_observations_lookup`, `world_medication_statements_lookup`, `world_clinical_notes_lookup`, `world_family_history_lookup`, `world_questionnaire_responses_lookup`, `world_intake_uploads_lookup`, `world_care_plans_lookup`, `world_goals_lookup`, `world_tasks_lookup`, `world_service_requests_lookup`, `world_consents_lookup`, `world_diagnostic_reports_lookup` | Typed read tools over projected entity shapes | | `world_conversations_lookup`, `world_calls_lookup` | Conversation and call record lookups | | `world_state_resolve` | Batch entity-ID to current world-state resolver | **Write tools:** | Tool | Purpose | | ------------------------------------ | ----------------------------------------------------------------------------------------------------------- | | `world_patient_create` | Create a patient after the supported duplicate checks; ambiguous matches still require reconciliation | | `world_patient_update` | Update contact info (phone, email, address); requires an entity reference | | `world_save_patient` | Create or update after the supported duplicate check; accepts natural field names and flexible date formats | | `world_schedule_appointment` | Book from a supported slot-search result and resolve its scheduling identifiers | | `world_cancel_appointment` | Cancel by appointment reference | | `world_confirm_appointment` | Confirm a booked appointment | | `world_reschedule_appointment` | Move an appointment to a new slot | | `world_insurance_create` | Create an insurance event that an eligible configured connector can consider for delivery | | `world_refill_prescription` | Submit a refill request for a specific prescription | | `world_log_call`, `world_log_triage` | Record call and triage outcomes | | `world_ticket_create` | Create a support-ticket event for a supported downstream workflow | World write results carry source and confidence metadata. Only eligible events with a supported connector mapping are considered for external delivery, where additional source, confidence, review, and destination policy applies. Entity-anchored tools also apply the available [write scope](#patient-safety-isolation); catalog membership alone is not a delivery or safety guarantee. ### Call Forwarding A built-in `forward_call` tool transfers the caller to a human. Two modes: * **Static forwarding**: workspace-configured fallback for the source number. * **Location-based forwarding**: the agent selects from location phone numbers in the patient's context. The tool does not accept an arbitrary phone number from the model; the destination comes from resolved configuration or location entity state. Agent policy instructs the model to invoke the tool when a caller requests a human, but a transfer occurs only when the tool is called and the telephony operation succeeds. {% hint style="info" %} **Deferred transfer.** Call transfers are deferred until the agent's goodbye message finishes playing. The transfer is cancellable by barge-in or operator join. {% endhint %} ## Safety & Escalation Safety behavior comes from the agent definition, Context Graph guidance and turn policy, eligible tools, and operator workflows. Model reasoning can still miss or misclassify a concern, so these controls require scenario testing and a human fallback. Supported escalation paths include: * **Call forwarding**: the agent invokes the built-in [`forward_call` tool](#call-forwarding) to transfer the caller to a human. * **Operator join**: a human operator joins in listen or takeover mode. Takeover suspends agent speech while the operator drives; listen mode does not. When a supported escalation record is created and its live run remains visible, it can appear in the Console's **Runs > Live** view, reached through the **Takeover** navigation item. See [Operators](/developer-guide/platform-api/conversations/operators.md) for operator resources and [Safety & Monitoring](/developer-guide/platform-api/safety.md) for the safety model, audit trail, and compliance reporting. ## Observer WebSocket Monitor active calls in real time via a WebSocket connection: ``` WS /agent/observe/{call_sid}?token={api_key} ``` Requires a valid workspace API key. Any observer can monitor any active call in the workspace, regardless of which server handles the call. **Late-join replay**: observers connecting mid-call receive a buffered replay of recent events before transitioning to the live stream. Events carry monotonic sequence numbers for ordering. ### Observer Event Types The observer stream emits session lifecycle, transcript, state transition, tool, timing, emotion, and call control events. The full catalog with field lists and PHI annotations lives on the observer events page: {% content-ref url="/pages/Qm4yWkfviMbMOVLEbCzJ" %} [Observer Events](/developer-guide/platform-api/conversations/observer-events.md) {% endcontent-ref %} ## Session Event Injection External systems can inject events into active voice sessions. When delivery and response generation succeed, the runtime handles the event without Context Graph navigation and can speak a response. Injection acceptance does not guarantee response timing, exact wording, or that a response will be produced. ### Injection Paths The Voice Agent HTTP route (`POST /agent/sessions/{call_sid}/event`) supports direct backend injection with a bearer token. It is not published in the Platform OpenAPI document. The `/test-call` WebSocket also accepts control frames for developer playground testing. #### Platform Session Injection Use workspace authentication for frontend or third-party injection. {% openapi src="" path="/v1/{workspace\_id}/sessions/{call\_sid}/inject" method="post" %} {% endopenapi %} #### Operator Guidance Use operator-scoped guidance when the caller should be identified and checked for `Operator:Update` permission. {% openapi src="" path="/v1/{workspace\_id}/operators/{operator\_id}/send-guidance" method="post" %} {% endopenapi %} ### Injected Event Types | Type | Behavior | Example | | ---------------- | ------------------------------------------------------ | ---------------------------------------------- | | `external_event` | Queues behind current speech. Cancels silence monitor. | "Appointment confirmed for 2pm tomorrow" | | `guidance` | Interrupts current speech and cancels silence monitor. | "Ask for their insurance ID before confirming" | The distinction matters: external events carry factual information that can wait for the agent to finish speaking, while guidance is prioritized as an instruction and can interrupt current speech. Delivery can still be delayed or fail, and the model is not guaranteed to follow the instruction verbatim. ### Request Format ``` POST /agent/sessions/{call_sid}/event Authorization: Bearer ``` ```json { "workspace_id": "a1b2c3d4-...", "message": "The patient's insurance has been verified", "sender": "ehr_system", "event_type": "external_event" } ``` The `workspace_id` field is required and must match the workspace of the call being injected into; a mismatch is rejected with 403. The `event_type` field accepts `"external_event"` or `"guidance"` (default `"external_event"`). The `message` field is limited to 5,000 characters. The `sender` field (max 200 characters, default `"System"`) can be retained with the injected event when processing and call persistence succeed. Returns 404 if no active session exists for the `call_sid`. ### Response The endpoint returns delivery status indicating whether the event was received by the session: ```json { "status": "delivered", "call_sid": "CA1234..." } ``` A `status` of `"queued_no_subscriber"` indicates the event was published but no active session was listening. This can happen during a brief window when a session is initializing or if the call has already ended. A `status` of `"deduplicated"` indicates that an identical injection (same message, sender, and event type) for the same call was suppressed within a 30-second window. This reduces duplicate processing for ordinary retries but is not an end-to-end delivery or exactly-once guarantee. ### Delivery Resilience Injection can cross runtime instances to reach the active call. Inspect the returned status: `delivered` confirms that a session listener received the publication, not that the model followed it or that a caller heard a response. Runtime interruptions can delay or prevent later processing. ### Active Sessions List currently active sessions via the platform API: {% openapi src="" path="/v1/{workspace\_id}/sessions/active" method="get" %} {% endopenapi %} Returns a best-effort real-time list of active sessions with call metadata. A live session can appear late or be temporarily absent, so use the canonical Runs surface for durable inventory. ### WebSocket Control Channel Test calls and direct streams accept text-frame control messages for injection and session control: ```jsonc // Inject an external event {"type": "inject_event", "message": "...", "sender": "..."} // Inject operator guidance (interrupts speech) {"type": "inject_guidance", "message": "..."} // Force context refresh (reloads patient data) {"type": "refresh_context"} // Stop the session {"type": "stop"} ``` ### Test-Call Scenarios The `/test-call` WebSocket endpoint supports scenario-based testing: | Parameter | Default | Description | | ------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------- | | `scenario` | `inbound` | `inbound` (agent greets first), `outbound` (task context greeting), `silent` (no greeting) | | `caller_id` | empty string | Simulated caller phone number. Test calls are tagged with source `playground` independently of `caller_id`. | | `outbound_task_entity_id` | - | Entity ID for outbound task context (required for `outbound` scenario) | | `system_prompt` | - | Freeform prompt override (takes precedence over scenario-derived prompts) | ``` WS /agent/test-call?token={api_key}&scenario=outbound&outbound_task_entity_id=123&caller_id=+15551234567 ``` ## Call Record & Persistence When call persistence succeeds, the record can include: * **Turns**: each turn carries a timing model with four field groups (all fields in milliseconds): * **User speech boundaries**: `user_speech_start_ms`, `user_speech_end_ms` (when the caller's speech started and ended). * **Processing breakdown**: `engine_ms`, `nav_ms`, `render_ms`, `audio_ttfb_ms` (reasoning-engine, navigation, render, and audio time-to-first-byte latency). * **Agent speech boundaries**: `agent_speech_start_ms`, `agent_speech_end_ms` (when agent audio playback started and ended). * **Audio window**: `audio_window_start_ms`, `audio_window_end_ms` (the turn's bounding audio interval). * **Tool calls**: name, input, output, duration, success/failure. * **State transitions**: recorded context graph navigation events. * **Emotional summary**: see [below](#emotional-summary). * **Escalation history**: recorded escalation lifecycle events when an operator joined. * **Config snapshot**: version set, agent version, context graph version used. ## Calls API ### Voice Run Inventory {% openapi src="" path="/v1/{workspace\_id}/runs" method="get" %} {% endopenapi %} `/runs` is the canonical workspace inventory for current clients. Filter by live status, conversation kind, and voice channel for active calls; omit the status filter for history. Active-call visibility is best-effort, so clients should tolerate a live call appearing late or being temporarily absent. ### Compatibility Voice Listing {% openapi src="" path="/v1/{workspace\_id}/calls" method="get" %} {% endopenapi %} This voice-specific compatibility listing is deprecated and returns `Deprecation: true` and `Sunset: 2026-07-01` response headers. Migrate inventory reads to `/runs`. ### Compatibility Call Detail {% openapi src="" path="/v1/{workspace\_id}/calls/{call\_id}" method="get" %} {% endopenapi %} Available call-record fields can include turns with timing data, tool calls, state transitions, an emotional summary, escalation history, safety state, and a configuration snapshot. This compatibility detail route returns the same deprecation and sunset headers. Use `/runs/{run_id}` for the canonical run record; call-specific subordinate analytics and recording routes remain documented below where no run-scoped replacement is available. ### Recordings #### Get Recording Metadata {% openapi src="" path="/v1/{workspace\_id}/recordings/{call\_sid}/metadata" method="get" %} {% endopenapi %} #### Get Recording URLs {% openapi src="" path="/v1/{workspace\_id}/recordings/{call\_sid}/urls" method="get" %} {% endopenapi %} #### Download a Recording {% openapi src="" path="/v1/{workspace\_id}/recordings/{call\_sid}/download/{filename}" method="get" %} {% endopenapi %} ### Outbound Calls Outbound calls are placed through the Platform API outbound call endpoint, which handles use-case-based caller ID selection, an explicit startup instruction, response reuse for some same-key retries, and best-effort world-model lineage. The patient identifier validates and anchors lineage but is not passed as a dedicated voice-runtime context field: {% content-ref url="/pages/B2fvwH4YLDTlmhMXUy84" %} [Calls](/developer-guide/platform-api/conversations/calls.md) {% endcontent-ref %} ## Text Sessions Text conversations - REST, WebSocket, SMS, and WhatsApp - can use the same Context Graph definitions and many of the same tool families as voice calls, but transport behavior, runtime tool exposure, turn handling, safety fallbacks, and persistence details differ by channel. The canonical documentation for text transports lives on [Conversations](/developer-guide/platform-api/conversations.md): WebSocket connection and close codes, the `tool_events` default, SSE streaming event shapes, message queuing and coalescing, outbound SMS parameters, and inbound SMS and WhatsApp behavior. {% content-ref url="/pages/js8kvebm1nrcYWgMPlI0" %} [Conversations](/developer-guide/platform-api/conversations.md) {% endcontent-ref %} ### WhatsApp Voice Notes {% openapi src="" path="/v1/{workspace\_id}/services/{service\_id}/voice-turn" method="post" %} {% endopenapi %} Handles voice note conversations on WhatsApp (and any channel that sends completed audio recordings rather than real-time streams). The platform transcribes the audio, runs the same reasoning engine pipeline as voice calls and text sessions, synthesizes the agent's spoken reply, and returns it as OGG Opus audio. **Response**: `200 OK` with `audio/ogg` body (OGG Opus). `204 No Content` if the agent has nothing to say. `409 Conflict` if a turn for this phone number is already in progress. Sessions are server-managed - no session ID is required. Voice-note state is maintained across requests sharing the same workspace, service, and phone number until the flow terminates or the stored state expires. This key is distinct from the text-turn session namespace, so the endpoint does not by itself provide one shared thread across text and voice notes. ## Desktop Sessions Remote desktop automation sessions - screenshots, mouse and keyboard actions, and session lifecycle - are documented on their own page: {% content-ref url="/pages/ugvrMoRTCVQoelwZdaSU" %} [Desktop Sessions](/developer-guide/platform-api/conversations/desktop-sessions.md) {% endcontent-ref %} ## Agent Trace & Debugging Three endpoints provide deep inspection into agent reasoning and call quality. ### Execution Trace ``` GET /agent/calls/{call_id}/trace ``` Per-turn execution log showing recorded agent decisions. Each turn can include: | Field | Type | Description | | --------------- | -------------- | -------------------------------------------------------------------------------------- | | `action` | string | The action taken (speak, navigate, tool call, escalate) | | `signal_kind` | string | What triggered this turn (caller speech, tool result, state transition, barge-in) | | `effect_kind` | string | What the agent did (spoke, invoked tool, navigated, escalated) | | `tools` | array | Tool calls with name, parameters, result, and duration | | `state` | string | Context graph state at this turn | | `emotion` | object | Detected caller emotion at this turn | | `inner_thought` | string or null | Agent's internal reasoning (when available) | | `latency_ms` | integer | Response latency for this turn | | `barge_in` | object or null | Barge-in details if the agent was interrupted (interrupted text, discarded utterances) | ### Prompt Trace ``` GET /agent/calls/{call_id}/prompts ``` Captured LLM prompt data can include the system prompt, conversation history, tool definitions, and model response for a turn. Availability depends on runtime capture and retention, and the data is intended for debugging unexpected agent behavior. ### Call Trace Analysis {% openapi src="" path="/v1/{workspace\_id}/calls/{call\_id}/trace-analysis" method="get" %} {% endopenapi %} Audio-native intelligence analysis computed asynchronously after call completion. See [Call Trace Analysis](/developer-guide/platform-api/safety/call-trace-analysis.md) for the full endpoint reference. Returns `status: pending` while analysis is in progress and `status: unavailable` if analysis is not possible for this call. ## Emotional Summary When acoustic and behavioral telemetry is available and persistence succeeds, the call detail response can include an emotional summary: ```json { "dominant_emotion": "Sadness", "average_valence": -0.312, "average_arousal": 0.654, "peak_negative_valence": -0.587, "peak_negative_emotion": "Fear", "emotional_shifts": 3, "final_trend": "improving", "segment_count": 42, "barge_in_count": 2, "short_response_streak": 0, "silence_gap_count": 1, "coherence": 0.72, "language_sentiment": 0.45, "language_toxicity": null, "compound_emotions": [ {"name": "Concern", "score": 0.61} ] } ``` ## Emotional Intelligence Direction The emotional intelligence system continues to deepen along four axes: audio-level prosodic planning (breath-like pauses, per-word pacing), sub-second emotional adaptation within a conversational beat, cross-call emotional profiles surfaced proactively from the world model, and blended emotion expressed through TTS-level control. For the conceptual treatment of emotion detection and adaptation, see [Emotion Detection](https://docs.amigo.ai/channels/voice/emotion-detection) in the platform docs. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.amigo.ai/developer-guide/platform-api/conversations/voice-agent.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.