How It Works
Walk through a complete interaction lifecycle from patient contact to post-call analysis, mapping each step to real system components.
This page walks through a complete interaction lifecycle using a voice call as the example, since voice is the most complex modality. Every step maps to a real system component.
The platform's reasoning engine is modality-independent - it processes typed signals (utterances, emotion, tool results) and emits effects (respond, execute tool, escalate). Voice, SMS, and simulation are modality adapters that convert channel-specific I/O into these signals and execute the resulting effects. The call lifecycle below shows how the voice adapter feeds signals to the engine and delivers effects as speech. Text conversations follow the same reasoning pipeline but skip audio processing and deliver effects as messages.
Call Lifecycle
System Architecture
Channels feed into the reasoning engine under the active runtime safety controls. The engine navigates context graphs, reads selected memory and world-model context, and executes eligible tools. Conversation observations and connector data converge in the world model; outbound connector policy separately evaluates eligible event payloads for external delivery. Intelligence records supported interaction evidence for analytics, and operators monitor eligible work from the unified Runs surface.
Phase by Phase
The phases below describe the voice modality in detail. Text channels replace the audio phases with message I/O and apply their own channel capabilities, timing, and approval behavior while reusing the core reasoning path.
1. Greeting Preparation During Ring Time
On the supported conference-based voice path, preparation can begin while the call rings. Depending on the resolved service and available context, the runtime can:
Prepare the agent leg and conference
Resolve an existing caller binding from the phone number
Load authorized projected context that is available for the resolved entity
Load the Context Graph and other published service configuration
Preparation during ring time reduces the delay before the greeting. If preparation or capacity fails, the call follows the configured failure path rather than guaranteeing immediate first audio.
2. Parallel Audio Processing
Caller audio can feed separate speech-recognition and acoustic-analysis paths:
Speech-to-Text converts supported audio into interim and final transcripts. Configured keyterms can improve recognition of domain vocabulary, and end-of-turn detection determines when the caller has finished speaking. Actual latency and word-timing coverage depend on the selected provider and network path.
Emotion Detection classifies voiced two-second acoustic segments into nine categories plus valence and arousal. A four-segment rolling state weights newer observations more heavily, while final transcripts can separately contribute sentiment and toxicity evidence. A per-call acoustic profile is exposed for observability after warmup but does not recalibrate live inference or empathy control.
Emotion analysis is not on the speech-recognition critical path. It can be disabled after repeated failures while the call continues, although failures in required voice or transport dependencies can still interrupt the session.
3. Context Graph Navigation
The context graph is a hierarchical state machine that defines what the agent should accomplish at each point in the call. A navigation step evaluates the current transcript, emotional state, and conversation history to select the next action.
This is not a fixed script. The context graph defines goals and constraints. The agent determines how to achieve them based on the live conversation. If a patient brings up insurance while the agent is in a scheduling flow, the state machine can handle the transition.
The navigation step also selects filler phrases ("Let me check that for you") that keep the conversation flowing while the system processes the next response.
4. Tool Execution
The agent can call built-in clinical tools, read-only Platform Functions, configured REST integration endpoints, and prompt-driven Skills that orchestrate declared tools. Context Graph states determine which tools are eligible for each part of the conversation. Blocking and background execution behavior is configured on the state-to-tool binding rather than through a Skill execution tier.
Voice fillers and progress narration can cover supported tool waits. When a tool records a requested external mutation, the world model preserves that intent as a source-attributed event. Connector delivery has a separate outcome, and recording the event is not confirmation that the external system accepted the change.
Conversation-scoped integration approval currently applies to supported text conversations, not live voice calls. Voice-derived events can still be routed through outbound policy and, where enabled, a separate private-preview external write proposal before connector delivery.
5. Response Generation and TTS
The response generation step produces the agent's reply using the full context: patient data, conversation history, tool results, emotional state, and the current context graph action. Emotion detection results directly influence the response through micro-behaviors (pacing, word choice, acknowledgment phrases).
Text-to-speech converts the response to audio using the resolved voice configuration and any supported tone or speed steering. Barge-in requires recognized interim speech plus the configured minimum speech duration; voice activity alone does not interrupt the agent. When the TTS provider returns word timing, the runtime can retain it for transcript-to-audio alignment, but not every provider or utterance supplies complete timing.
6. Operator Escalation
When a situation exceeds the agent's scope (clinical judgment calls, upset callers requesting a human, safety triggers), the system can escalate to a human operator. The Runs > Live view ranks active work by urgency and opens takeover controls for the selected run in place.
For voice, the architecture is conference-first: the patient, agent, and operator join the same call. The operator can:
Review the live transcript for the selected run
Join in listen mode without transferring or reconnecting the caller
Send guidance while the agent continues responding
Explicitly confirm before speaking directly to the caller
Hand control back without re-initializing the session
7. Post-Call Processing
Data captured during a live phone call is inherently uncertain. A patient might misspeak, speech recognition might mishear, or the agent might misinterpret. The platform preserves that uncertainty rather than treating every extracted value as authoritative.
After the call ends:
Transcript and call events remain associated with the call when those artifacts were produced and retained.
Extracted observations enter the world model at source-appropriate confidence.
Entity projections resolve new observations against stronger or more recent evidence.
Eligible quality evaluations can score the interaction when their required evidence and processing path are available.
Post-call summaries and intelligence become available when their processing succeeds.
External write proposals are created only for connector workflows configured to use the private-preview review path.
Eligible writes are validated by the connector and, when the operation supports confirmation, checked through a target response or read-back.
The connector runner handles delivery to the external system. Operations that support confirmation are not presented as complete unless the target response or read-back confirms the change; other delivery failures are reported explicitly.
Last updated
Was this helpful?