> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/platform-overview/how-it-works.md).
# How It Works
This page walks through a complete interaction lifecycle using a voice call as the example, since voice is the most complex modality. Every step maps to a real system component.
The platform's [reasoning engine](/agent/reasoning-engine.md) is modality-independent - it processes typed signals (utterances, emotion, tool results) and emits effects (respond, execute tool, escalate). Voice, SMS, and simulation are modality adapters that convert channel-specific I/O into these signals and execute the resulting effects. The call lifecycle below shows how the voice adapter feeds signals to the engine and delivers effects as speech. Text conversations follow the same reasoning pipeline but skip audio processing and deliver effects as messages.
## Call Lifecycle
## System Architecture
Channels feed into the reasoning engine under the active runtime safety controls. The engine navigates context graphs, reads selected memory and world-model context, and executes eligible tools. Conversation observations and connector data converge in the world model; outbound connector policy separately evaluates eligible event payloads for external delivery. Intelligence records supported interaction evidence for analytics, and operators monitor eligible work from the unified Runs surface.
## Phase by Phase
The phases below describe the voice modality in detail. Text channels replace the audio phases with message I/O and apply their own channel capabilities, timing, and approval behavior while reusing the core reasoning path.
### 1. Greeting Preparation During Ring Time
On the supported conference-based voice path, preparation can begin while the call rings. Depending on the resolved service and available context, the runtime can:
* Prepare the agent leg and conference
* Resolve an existing caller binding from the phone number
* Load authorized projected context that is available for the resolved entity
* Load the Context Graph and other published service configuration
Preparation during ring time reduces the delay before the greeting. If preparation or capacity fails, the call follows the configured failure path rather than guaranteeing immediate first audio.
### 2. Parallel Audio Processing
Caller audio can feed separate speech-recognition and acoustic-analysis paths:
**Speech-to-Text** converts supported audio into interim and final transcripts. Configured keyterms can improve recognition of domain vocabulary, and end-of-turn detection determines when the caller has finished speaking. Actual latency and word-timing coverage depend on the selected provider and network path.
**Emotion Detection** classifies voiced two-second acoustic segments into nine categories plus valence and arousal. A four-segment rolling state weights newer observations more heavily, while final transcripts can separately contribute sentiment and toxicity evidence. A per-call acoustic profile is exposed for observability after warmup but does not recalibrate live inference or empathy control.
Emotion analysis is not on the speech-recognition critical path. It can be disabled after repeated failures while the call continues, although failures in required voice or transport dependencies can still interrupt the session.
### 3. Context Graph Navigation
The context graph is a hierarchical state machine that defines what the agent should accomplish at each point in the call. A navigation step evaluates the current transcript, emotional state, and conversation history to select the next action.
This is not a fixed script. The context graph defines goals and constraints. The agent determines how to achieve them based on the live conversation. If a patient brings up insurance while the agent is in a scheduling flow, the state machine can handle the transition.
The navigation step also selects filler phrases ("Let me check that for you") that keep the conversation flowing while the system processes the next response.
### 4. Tool Execution
The agent can call built-in clinical tools, read-only Platform Functions, configured REST integration endpoints, and prompt-driven Skills that orchestrate declared tools. Context Graph states determine which tools are eligible for each part of the conversation. Blocking and background execution behavior is configured on the state-to-tool binding rather than through a Skill execution tier.
Voice fillers and progress narration can cover supported tool waits. When a tool records a requested external mutation, the world model preserves that intent as a source-attributed event. Connector delivery has a separate outcome, and recording the event is not confirmation that the external system accepted the change.
Conversation-scoped integration approval currently applies to supported text conversations, not live voice calls. Voice-derived events can still be routed through outbound policy and, where enabled, a separate private-preview external write proposal before connector delivery.
### 5. Response Generation and TTS
The response generation step produces the agent's reply using the full context: patient data, conversation history, tool results, emotional state, and the current context graph action. Emotion detection results directly influence the response through micro-behaviors (pacing, word choice, acknowledgment phrases).
Text-to-speech converts the response to audio using the resolved voice configuration and any supported tone or speed steering. Barge-in requires recognized interim speech plus the configured minimum speech duration; voice activity alone does not interrupt the agent. When the TTS provider returns word timing, the runtime can retain it for transcript-to-audio alignment, but not every provider or utterance supplies complete timing.
### 6. Operator Escalation
When a situation exceeds the agent's scope (clinical judgment calls, upset callers requesting a human, safety triggers), the system can escalate to a human operator. The **Runs > Live** view ranks active work by urgency and opens takeover controls for the selected run in place.
For voice, the architecture is conference-first: the patient, agent, and operator join the same call. The operator can:
* Review the live transcript for the selected run
* Join in listen mode without transferring or reconnecting the caller
* Send guidance while the agent continues responding
* Explicitly confirm before speaking directly to the caller
* Hand control back without re-initializing the session
### 7. Post-Call Processing
Data captured during a live phone call is inherently uncertain. A patient might misspeak, speech recognition might mishear, or the agent might misinterpret. The platform preserves that uncertainty rather than treating every extracted value as authoritative.
After the call ends:
1. **Transcript and call events** remain associated with the call when those artifacts were produced and retained.
2. **Extracted observations** enter the world model at source-appropriate confidence.
3. **Entity projections** resolve new observations against stronger or more recent evidence.
4. **Eligible quality evaluations** can score the interaction when their required evidence and processing path are available.
5. **Post-call summaries and intelligence** become available when their processing succeeds.
6. **External write proposals** are created only for connector workflows configured to use the private-preview review path.
7. **Eligible writes** are validated by the connector and, when the operation supports confirmation, checked through a target response or read-back.
The connector runner handles delivery to the external system. Operations that support confirmation are not presented as complete unless the target response or read-back confirms the change; other delivery failures are reported explicitly.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.amigo.ai/platform-overview/how-it-works.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.