> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/agent/reasoning-engine.md).
# Reasoning Engine
The reasoning engine is the core intelligence layer of the Amigo platform. It gives supported voice, text, simulation, and API interactions a shared contract for context-graph navigation, tool execution, and response generation. Channel adapters still own transport-specific behavior such as audio streaming, message delivery, and connection lifecycle.
## Why a Unified Engine Matters
Early voice AI systems tightly couple reasoning logic with audio transport. The agent's decision-making is interleaved with speech-to-text timing, filler audio generation, and WebSocket management. This coupling means every new channel (SMS, simulation, API webhooks) must reimplement the reasoning loop from scratch, and bugs fixed in one channel don't propagate to others.
Amigo separates these concerns. Modality adapters convert channel input into typed signals and translate engine output into channel behavior, keeping audio and messaging transport out of the core reasoning contract.
## Cut / Navigate / Engage
Amigo uses three operations to coordinate the voice timeline and signal-driven text flow:
1. **Cut** - Decide whether an incoming signal creates a boundary in the current interaction phase.
2. **Navigate** - Select the next context-graph state or channel-level unit of work from the current session state.
3. **Engage** - Carry out that selection by generating a response, executing tools, scheduling a deadline, or delivering channel output.
The operations appear at more than one layer:
| Scale | Cut | Navigate | Engage |
| ------------------ | -------------------------------------------------- | ------------------------------------------------------------ | -------------------------------------------- |
| **Conversation** | A user turn or qualifying event prompts evaluation | Select a path through the context graph | Generate a response and execute tools |
| **Voice timeline** | A timing or session signal creates a boundary | Select breath, filler, hold, response, or listening behavior | Schedule or play the selected audio behavior |
| **Text flow** | A queued message or event becomes the next item | Run the shared reasoning path | Deliver or persist the resulting effects |
This coordination keeps fillers, silence handling, empathy pauses, tool progress narration, and barge-in recovery on one voice timeline. The [voice timeline](/channels/voice/audio-pipeline.md#voice-timeline) describes how it operates within each turn.
## Signals and Effects
The engine communicates through two primitives.
**Signals** represent something that happened. Supported adapters normalize relevant input into typed signals before it enters the shared flow:
| Signal | What It Represents |
| ------------------ | ----------------------------------------------------------------------------- |
| **Utterance** | The caller or user said something (text, from any source) |
| **Emotion** | An emotional state update from acoustic or transcript analysis when enabled |
| **Tool result** | A tool execution completed with a result |
| **Silence** | The caller has been silent beyond the configured threshold |
| **Barge-in** | The caller interrupted the agent mid-speech |
| **External event** | An injected event from an operator, surface submission, or external system |
| **System** | A system-level event such as a timeout, connection change, or error condition |
**Effects** represent something the engine wants to happen. The modality adapter decides how to execute each one:
| Effect | Voice | SMS | Simulation |
| -------------- | -------------------------------------------------------- | ----------------------- | -------------------------------------------------- |
| **Respond** | Stream the generated response through the voice pipeline | Send as an SMS message | Capture as simulation output |
| **Filler** | Play filler audio ("Let me check on that...") | No-op | No-op |
| **Pause** | Hold deliberate silence | No caller-facing output | Capture the effect when applicable |
| **Tool call** | Execute tool, feed result back as signal | Same | Same real tool path, with simulation-tagged writes |
| **Terminate** | Hang up after final speech | End session | Return final state |
| **Transition** | Records a state change; no caller-facing output | Same | Capture in trace log |
| **Observe** | Emits an analytics event; no caller-facing output | Same | Capture in trace log |
## The Pipeline
Each signal flows through three stages.
**Perceive.** The modality adapter converts raw input into typed signals. A voice adapter produces utterance signals from speech-to-text and emotion signals from prosody analysis. An SMS adapter produces utterance signals from message text. A simulation adapter injects both from test parameters.
**Reason.** The engine's core loop implements cut/navigate/engage at the conversation level:
1. **Navigate** - The context graph engine determines the current state, evaluates transition conditions, and selects the appropriate action.
2. **Engage** - The response generation model produces a reply, drawing on the agent's persona, current state guidance, selected memory, patient data from the world model, and the emotional context described below.
3. **Execute** - If the model calls tools, the engine executes them, feeds results back as tool result signals, and re-engages. This loop continues until a final text response is produced.
**Act.** The engine emits effects. The modality adapter executes each one according to channel capabilities. For voice, the [voice timeline](/channels/voice/audio-pipeline.md#voice-timeline) applies cut/navigate/engage within each turn to coordinate fillers, empathy pauses, and tool progress narration - the same three operations at a smaller scale.
The engine supports two processing styles. **Streaming mode** lets the voice adapter begin response generation and speech delivery without waiting for a complete text response. **Completed-effect mode** lets text, simulation, and API consumers receive materialized effects. Both use the shared navigation and tool-execution contract, while voice-specific timing and acoustic adaptation remain in the voice path.
Filler handling is channel-aware. Voice can play a short acknowledgement while work continues. Asynchronous messaging channels suppress voice-style filler and deliver results through their channel-specific completion flow. See [Email](/channels/email.md#long-running-tools-and-reply-delivery) for a non-live example.
Navigation can also be re-evaluated after a successful tool result rather than waiting for another user message. This occurs only for tool bindings configured for completion-gated navigation; the navigator may remain in the current state or select another valid state. See [Action State Extensions](/agent/context-graphs.md#action-state-extensions).
## Emotional Adaptation
When a voice session has usable emotion data, the engine can add that context through two paths. Simulations may also supply emotion signals for testing, but ordinary text messages do not produce acoustic evidence.
**Per-message annotations.** When acoustic evidence is available, the user message can retain the current detected emotion and valence alongside its transcript. These annotations are model-derived signals, not verified statements about the caller's internal state.
**Session-level steering.** Once the voice runtime has enough evidence, it can add a rolling summary to response prompts:
* **Dominant emotion and trend** - Is the caller improving, stable, or deteriorating?
* **Adaptation instructions** - Targeted guidance based on the caller's emotional quadrant (high-arousal negative callers need de-escalation; low-arousal negative callers need patience)
* **Behavioral signals** - Patterns like repeated interruptions, short response streaks, or extended silences that indicate disengagement or frustration independent of vocal emotion
* **Call-phase urgency** - After extended calls with deteriorating mood, the engine instructs the model to become more direct and resolution-focused
* **Coherence warnings** - When what the caller says and how they sound disagree, the engine flags the ambiguity so the model does not over-commit to a single interpretation
The combination gives the response model recent evidence and adaptation guidance. It does not make emotion classification definitive, and safety or clinical decisions should not rely on emotion inference alone.
## Per-State Configuration (TurnPolicy)
Each context graph state can configure the pipeline independently. A medication verification state behaves differently than a general scheduling state - not because the reasoning logic changes, but because the state's turn policy tunes the pipeline for that context.
Five areas are configurable per state:
* **Barge-in** - Enable or disable caller interruptions. A greeting state can suppress caller barge-in for a configurable shield duration; transport failure or other session termination can still interrupt playback. A quick-answer state can keep barge-in enabled for faster turn-taking.
* **Safety response** - What happens when a safety rule fires. Options: stay in the conversation and respond with empathy, suspend the agent and route to an operator, or log an alert without interrupting.
* **Context strategy** - Choose full history or compact past state groups. A per-state threshold can activate compaction, and the engine can also ratchet to compact mode when prompt usage approaches the configured context limit.
* **Tool controls** - The state defines its actions. Turn policy can additionally block call forwarding entirely or after a configured number of turns.
* **STT sensitivity** (voice only) - End-of-turn thresholds and silence timeouts. Data collection states use higher thresholds and longer timeouts because callers pause between pieces of information. Quick-answer states use lower thresholds for faster responses.
## Model Configuration
The platform supports separate model preferences for navigation and engagement. Navigation selects a path through the context graph; engagement handles tool calling and response generation. Configuring them independently lets teams tune cost, latency, and response quality, but model-backed navigation is not inherently deterministic.
## Graceful Degradation
Selected failure paths have bounded fallbacks so an optional subsystem or a single model timeout does not automatically end a session.
| Component | Failure | Fallback |
| --------------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------ |
| **Emotion detection** | Connection failure or repeated receive errors | Continues without new emotion-derived steering |
| **Navigation model** | Timeout or unavailable configured model | Tries a configured fallback; otherwise remains in the current engageable state |
| **Context pressure** | Estimated prompt usage approaches the configured limit | Compacts past state groups and caps retained history |
These paths reduce failure propagation, but they are not a zero-failure guarantee. Transport loss, exhausted fallbacks, session limits, or failures in required dependencies can still interrupt or end an interaction.
## Voice Control Plane
Voice calls combine service, agent, workspace, and environment settings that control vocal identity and delivery. They do not use one universal override hierarchy. Each field family has its own resolution rule.
| Setting Family | Resolution |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Service pipeline tuning** | A service value applies when set; otherwise the field uses its documented runtime default or supported fallback |
| **Session provider** | Service, then agent, then environment |
| **TTS provider** | Service, then agent, then workspace, then environment. Per-language maps check the exact language, base language, and `multilingual` key, with service, agent, and workspace precedence within each key |
| **TTS provider configuration and model** | Workspace provider configuration forms the base, then agent and service configuration override it. A service model is the final model override; unresolved values use the environment configuration. A selected per-language entry can replace the merged base provider configuration |
| **Workspace voice settings** | Provide voice identity, tone, speed, volume, language, domain vocabulary, sensitive topics, and other workspace baselines for the fields that consult them |
| **Per-state `TurnPolicy`** | Controls separate state behavior such as STT sensitivity, barge-in, safety response, and context strategy; it is not another provider-inheritance tier |
When the emotion system has usable evidence, it can derive a turn-level delivery tone and prompt guidance. Navigation can select a turn-specific tone, an explicit workspace tone can take precedence over computed voice context, and provider defaults apply when no turn tone is available. This tone choice is separate from provider and TTS configuration resolution. See [Emotion Detection](/channels/voice/emotion-detection.md) and the [Audio Pipeline](/channels/voice/audio-pipeline.md).
## Concurrency
Voice timeline signals are processed through a per-session mailbox, and live text conversations use a per-conversation signal queue with an ownership lease. These boundaries serialize the work each actor accepts while still allowing external tools, providers, and background tasks to run independently.
Queueing reduces re-entrant state changes and gives each consumer an ordered input stream. It does not make the complete system deterministic or replayable: concurrent producers can race to enqueue, networks can retry or reorder delivery, and model output can vary. Durable traces support inspection, but they are not a bit-exact replay artifact.
The same model applies within voice turns. Fillers, responses, empathy pauses, and tool progress narration are not separate subsystems competing for the audio stream. They are effects emitted by the same actor into a single timeline, scheduled by cut/navigate/engage at the turn scale. One actor, one timeline.
## Modality Adapters
Each adapter handles the channel-specific concerns that the reasoning engine does not touch:
| Adapter | Signal Production | Effect Execution |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Voice** | STT produces utterance signals; prosody analysis produces emotion signals; silence and barge-in detectors produce their respective signals | Respond effects stream through TTS with emotion-adaptive delivery; fillers play audio; pauses hold silence; terminate effects hang up after final speech |
| **Text (SMS)** | Incoming messages produce utterance signals | Respond effects send SMS messages; terminate effects end the session |
| **Simulation** | Test parameters inject utterance and emotion signals | Effects are captured as simulation evidence; tool writes use simulation source tagging rather than a separate database branch |
New modalities can reuse the signal-and-effect contract through an adapter. Channel-specific delivery, lifecycle, authorization, and failure handling still require integration and testing.
{% hint style="info" %}
**Related sections** - See [Context Graphs](/agent/context-graphs.md) for how the engine navigates problem spaces, [Functional Memory](/agent/memory.md) for bounded cross-session context, and [Voice Agent](/channels/voice.md) for voice-specific pipeline details. [Dynamic Behaviors](/agent/context-graphs/dynamic-behaviors.md) documents a separate Classic API capability.
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.amigo.ai/agent/reasoning-engine.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.