> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/testing/testing.md).

# Testing Overview

The platform provides a testing and evaluation framework for verifying agent behavior before deployment and measuring quality in production. This section covers simulations for repeatable behavior checks, the playground for interactive testing, tiered voice testing, metrics for scoring conversation quality, and a practical process for finding degradation in those results over time.

## Testing Philosophy

Healthcare workflows are long and multi-step. A single patient interaction might span 20 or more steps across scheduling, insurance verification, EHR writeback, and outbound follow-up calls. These workflows touch multiple external systems, each with its own availability characteristics, rate limits, and failure modes.

Testing every branch through live calls is slow and makes reasoning failures difficult to separate from audio or integration failures. The platform uses three principles:

1. **Use controlled workspace context.** Simulations read current workspace state. Dedicated test workspaces or entities provide stable fixtures when comparison across runs matters.
2. **Exercise the real reasoning and tool path.** Text simulations run context graphs and tools rather than replacing external calls with generic stubs. Simulation provenance keeps their world-model effects out of production EHR delivery and analytics, while unsupported effects such as surface delivery remain disabled.
3. **Validate audio and integrations at their own boundaries.** Real-audio tests cover speech and transport behavior. Connector and approval workflows need an appropriately isolated integration environment when their live behavior is under test.

This structure keeps broad reasoning regression tests fast while preserving separate evidence for media and integration behavior.

```mermaid
flowchart LR
    S["Simulations\n(pre-deployment)"] --> M["Metrics\n(per-interaction scoring)"]
    M --> D["Trend Review\n(compare periods)"]
    D -->|Change investigated| S
    M -->|Quality data| Dash["Dashboards +\nTeam review"]
```

## Simulations

Simulations test an agent against synthetic users in controlled scenarios. You define personas, situations, and evaluation criteria; the platform runs the conversations and scores the results. Saved cases group into suites for repeatable regression testing.

### What You Can Validate

* **Conversation behavior** - Exercise context graph branches, escalation rules, tool selection, and expected outcomes.
* **World-model context** - Ground simulated callers in current workspace data and use dedicated test entities when a fixed starting state matters.
* **Tool behavior** - Exercise real tool execution while simulation-tagged writes remain excluded from production EHR delivery and analytics. Surface delivery and approval workflows are not available in simulations.
* **Authorization-scoped parameters** - SQL tools with authorization-bound parameters can use their configured test values in simulations when the launching workspace actor holds the data-query permission, matching the grant required by the text playground. The Developer Console displays configured test values on tool detail pages and disables inline testing for authorization-bound tools, directing users to the playground instead.
* **Quality output** - Shared evaluation definitions make simulation and production results comparable, while simulation evidence remains in simulation-specific views and stays out of production analytics.

## Production Evaluations

Completed conversations - voice and text - are automatically evaluated against the workspace's active evaluation definitions. A workspace without active definitions incurs no evaluation cost. The platform enforces a per-workspace daily cap on production evaluations to bound spend.

New production evaluation definitions must use the metric type. Legacy assertion definitions remain readable and executable while they are migrated, but new assertion definitions can no longer be created. Workspaces should express new evaluation criteria as metrics.

This means production grading is governed by the definitions a workspace configures, not by a separate opt-in flag. Workspaces that add definitions mid-day are not penalized: conversations that complete before any definitions are active do not consume the daily cap.

Production evaluation is best-effort. A transient failure in the evaluation path does not affect the conversation itself, and evaluations can be re-run. The trigger fires at conversation teardown on both voice and text channels, so every completed conversation - regardless of channel - is eligible for grading.

### Reliable Suite Execution

Long suites are designed to reach a reportable result even when execution is interrupted:

* Interrupted runs are retried automatically up to a bounded attempt limit.
* Runs that cannot be recovered end with a failure reason, rather than remaining indefinitely in progress.
* Each suite run records the number of cases selected at launch, making incomplete starts visible even if a case never produced a run record.
* Summary views report run, session, and turn totals, while normalized checks give metrics and assertions a consistent verdict, score, and rationale shape.

Coverage APIs bring states, observed paths, gaps, and session evidence together so teams can identify untested branches. The Developer Console coverage visualization is currently an internal preview.

The Simulations landing page includes a performance overview organized by metric, with pass rates, score distributions, and trends visible before drilling into individual runs. Each conversation within a run shows its evaluation verdicts - assertion outcomes, metric scores, and justification text - so reviewers can trace a result back to the session that produced it.

Simulations answer the question: **does the agent handle this situation correctly?**

{% content-ref url="/pages/ge01EpnrtOpPWn6hpB7M" %}
[Simulations](/testing/testing/simulations.md)
{% endcontent-ref %}

## Playground and Interactive Testing

Open **Prove > Playground** to test an eligible service through a browser voice call or turn-by-turn text. Text mode uses conversation creation plus request-scoped SSE streaming; it is not a separate realtime or WebSocket mode.

### Prove Workspace

The **Prove** section groups the tools used while building and validating agents:

* **Interactive playground** - Try voice and text behavior against a selected service.
* **Tool testing** - Run individual tools with controlled inputs before using them in a conversation.
* **Cases, suites, and simulations** - Move from one-off exploration to repeatable regression coverage.
* **Framework runs** - Dispatch supported partner-framework agents, including customer-authored native definitions, and inspect their normalized trajectories.
* **Test traffic controls** - Keep designated test calls and credentials out of production billing, analytics, and downstream writes.

The playground's takeover controls use the same operator identity and confirmation rules as production Runs. An operator joins voice in listen mode and must explicitly confirm before speaking to the caller.

When a service is configured to use a session-owning voice provider, Playground voice calls run through that provider rather than the default in-house pipeline. The provider generates its own opening audio, so the platform does not pre-render a separate greeting. Tools available to the provider are limited to those referenced in the service's context graph - the same authorization boundary that applies to phone calls.

{% hint style="info" %}
**Session-owning voice provider tool behavior.** When a service uses a session-owning speech-to-speech voice provider, the provider's tool contract is aligned with the tools actually available at runtime. The provider receives only the tools authorized by the service's context graph, and its system prompt describes the exact execution model configured for the call - including whether tools run sequentially or in parallel, and whether write-capable tools are present. Services without tools receive no tool instructions, avoiding unnecessary model confusion.
{% endhint %}

### Unified Runs

The **Operate > Runs** page provides one surface for conversation and framework activity. The **Operate > Takeover** shortcut opens the **Live** tab of that same Runs page:

* **Live** - An urgency-ranked operator queue containing only live runs.
* **All runs** - A filterable history across run kind, channel, and status.
* **Overview** - Workspace totals and status distributions.

Live voice calls appear while they are in progress. Filters accept multiple values within an axis, so teams can review combinations such as failed and timed-out runs without losing workspace-wide pagination.

Opening a run composes the appropriate detail view:

* Voice runs show audio, transcript, analysis, and eligible operator controls.
* Text, SMS, email, and web runs show the conversation transcript, memory context, and eligible close or takeover actions.
* Framework runs show metadata and an ordered structural trajectory of decisions, tool calls, and completion steps.

Eligible voice runs support listen and live takeover modes. Eligible non-voice runs support authored-turn takeover, where the operator pauses the agent, sends the next reply, and hands control back explicitly. Historical views include persisted successful, failed, and timed-out framework runs. Persistence is best-effort, so a run can still reach its correct terminal outcome even if its historical marker or trajectory is unavailable.

The playground answers the question: **what does the agent actually do when I talk to it?**

{% content-ref url="/pages/4ZM02YLTTpkYAA4li1u9" %}
[Playground and Interactive Testing](/testing/testing/playground.md)
{% endcontent-ref %}

## Voice Simulation

Voice testing separates **text-tier quality** (reasoning, tool selection, context graph coverage, and response content) from **voice-path health** (audio attachment, transcription, speech rendering, turn timing, latency, and capacity). Teams can run broad text-tier simulations for behavior regressions and reserve real-audio checks for the media path.

Voice testing answers the question: **is the agent behavior correct, and does the audio path deliver it correctly?**

{% content-ref url="/pages/KaEECAS9Ciko9oisktO5" %}
[Voice Simulation](/testing/testing/voice-simulation.md)
{% endcontent-ref %}

## Metrics

Metrics measure the quality of agent conversations across the dimensions that matter to your organization - safety, clinical accuracy, empathy, goal completion, and other use-case-specific criteria. Eligible production conversations and simulations can be evaluated through their supported definitions and execution paths, while human review provides separate calibration evidence. Do not assume that every session is evaluated or that one definition shape applies unchanged across Classic, Platform production-eval, and simulation APIs.

The platform supports two evaluation modes for production conversations:

* **On-demand evaluation.** Trigger evaluation for a specific completed conversation through the API. Useful for spot-checking individual conversations or integrating evaluation into custom workflows.
* **Eager post-conversation evaluation.** Where the feature is enabled, completed voice, text, SMS, email, and web conversations can be evaluated against active definitions after they end. Triggering is best-effort and bounded by task and per-workspace daily caps, so this path does not guarantee a verdict for every conversation. Use persisted eval results to distinguish evaluated, pending, skipped, and error outcomes.

Both modes use the same production-eval definitions and verdict model. Trigger mode alone does not change the rubric, but channel modality, retained evidence, model execution, and missing artifacts can still affect whether two results are meaningfully comparable.

Metrics answer the question: **how well is the agent performing?**

{% content-ref url="/pages/R2FZnfpyXlTJjCCoPYwH" %}
[Metrics and Quality](/testing/testing/metrics.md)
{% endcontent-ref %}

## Drift Detection

Use metric trends, run history, and saved simulation baselines to look for sustained changes in inputs or outcomes. Teams choose the comparison windows, investigate representative interactions, update test cases, and make an explicit release decision. The platform does not currently turn a trend threshold into an automatic alert, promotion block, or rollback.

Drift detection answers the question: **is the agent getting worse?**

{% content-ref url="/pages/ogceEl19CRp2g6SKFaRb" %}
[Drift Detection](/testing/testing/drift-detection.md)
{% endcontent-ref %}

***

## How the Pillars Work Together

These capabilities support a repeatable team workflow:

1. **During development**, the playground gives interactive feedback on agent behavior, and tool testing exercises individual tools outside a conversation with the side-effect policy documented for that tool family.
2. **Before deployment**, simulations exercise target scenarios and measure configured criteria, while targeted real-audio tests provide separate evidence about speech, timing, transport, and capacity changes.
3. **In production**, configured evaluations and analytics provide ongoing quality evidence.
4. **Over time**, teams compare equivalent periods, investigate material changes, and add newly discovered failure modes to the test set.

When the evidence indicates drift, update the relevant simulations and re-verify before releasing a change. This keeps the test set aligned with production without allowing an uncertain trend to change production automatically.

{% hint style="info" %}
**For Developers**: See the [REST API reference](https://docs.amigo.ai/developer-guide/classic-api/core-api/metrics) and [Simulations reference](https://docs.amigo.ai/developer-guide/classic-api/core-api/simulations/) for endpoint details, request/response schemas, and SDK code examples.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.amigo.ai/testing/testing.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.