Testing Overview
Simulations, interactive testing, voice-path checks, metrics, and trend review support deliberate verification before and after deployment.
The platform provides a testing and evaluation framework for verifying agent behavior before deployment and measuring quality in production. This section covers simulations for repeatable behavior checks, the playground for interactive testing, tiered voice testing, metrics for scoring conversation quality, and a practical process for finding degradation in those results over time.
Testing Philosophy
Healthcare workflows are long and multi-step. A single patient interaction might span 20 or more steps across scheduling, insurance verification, EHR writeback, and outbound follow-up calls. These workflows touch multiple external systems, each with its own availability characteristics, rate limits, and failure modes.
Testing every branch through live calls is slow and makes reasoning failures difficult to separate from audio or integration failures. The platform uses three principles:
Use controlled workspace context. Simulations read current workspace state. Dedicated test workspaces or entities provide stable fixtures when comparison across runs matters.
Exercise the real reasoning and tool path. Text simulations run context graphs and tools rather than replacing external calls with generic stubs. Simulation provenance keeps their world-model effects out of production EHR delivery and analytics, while unsupported effects such as surface delivery remain disabled.
Validate audio and integrations at their own boundaries. Real-audio tests cover speech and transport behavior. Connector and approval workflows need an appropriately isolated integration environment when their live behavior is under test.
This structure keeps broad reasoning regression tests fast while preserving separate evidence for media and integration behavior.
Simulations
Simulations test an agent against synthetic users in controlled scenarios. You define personas, situations, and evaluation criteria; the platform runs the conversations and scores the results. Saved cases group into suites for repeatable regression testing.
What You Can Validate
Conversation behavior - Exercise context graph branches, escalation rules, tool selection, and expected outcomes.
World-model context - Ground simulated callers in current workspace data and use dedicated test entities when a fixed starting state matters.
Tool behavior - Exercise real tool execution while simulation-tagged writes remain excluded from production EHR delivery and analytics. Surface delivery and approval workflows are not available in simulations.
Authorization-scoped parameters - SQL tools with authorization-bound parameters can use their configured test values in simulations when the launching workspace actor holds the data-query permission, matching the grant required by the text playground. The Developer Console displays configured test values on tool detail pages and disables inline testing for authorization-bound tools, directing users to the playground instead.
Quality output - Shared evaluation definitions make simulation and production results comparable, while simulation evidence remains in simulation-specific views and stays out of production analytics.
Production Evaluations
Completed conversations - voice and text - are automatically evaluated against the workspace's active evaluation definitions. A workspace without active definitions incurs no evaluation cost. The platform enforces a per-workspace daily cap on production evaluations to bound spend.
New production evaluation definitions must use the metric type. Legacy assertion definitions remain readable and executable while they are migrated, but new assertion definitions can no longer be created. Workspaces should express new evaluation criteria as metrics.
This means production grading is governed by the definitions a workspace configures, not by a separate opt-in flag. Workspaces that add definitions mid-day are not penalized: conversations that complete before any definitions are active do not consume the daily cap.
Production evaluation is best-effort. A transient failure in the evaluation path does not affect the conversation itself, and evaluations can be re-run. The trigger fires at conversation teardown on both voice and text channels, so every completed conversation - regardless of channel - is eligible for grading.
Reliable Suite Execution
Long suites are designed to reach a reportable result even when execution is interrupted:
Interrupted runs are retried automatically up to a bounded attempt limit.
Runs that cannot be recovered end with a failure reason, rather than remaining indefinitely in progress.
Each suite run records the number of cases selected at launch, making incomplete starts visible even if a case never produced a run record.
Summary views report run, session, and turn totals, while normalized checks give metrics and assertions a consistent verdict, score, and rationale shape.
Coverage APIs bring states, observed paths, gaps, and session evidence together so teams can identify untested branches. The Developer Console coverage visualization is currently an internal preview.
The Simulations landing page includes a performance overview organized by metric, with pass rates, score distributions, and trends visible before drilling into individual runs. Each conversation within a run shows its evaluation verdicts - assertion outcomes, metric scores, and justification text - so reviewers can trace a result back to the session that produced it.
Simulations answer the question: does the agent handle this situation correctly?
Playground and Interactive Testing
Open Prove > Playground to test an eligible service through a browser voice call or turn-by-turn text. Text mode uses conversation creation plus request-scoped SSE streaming; it is not a separate realtime or WebSocket mode.
Prove Workspace
The Prove section groups the tools used while building and validating agents:
Interactive playground - Try voice and text behavior against a selected service.
Tool testing - Run individual tools with controlled inputs before using them in a conversation.
Cases, suites, and simulations - Move from one-off exploration to repeatable regression coverage.
Framework runs - Dispatch supported partner-framework agents, including customer-authored native definitions, and inspect their normalized trajectories.
Test traffic controls - Keep designated test calls and credentials out of production billing, analytics, and downstream writes.
The playground's takeover controls use the same operator identity and confirmation rules as production Runs. An operator joins voice in listen mode and must explicitly confirm before speaking to the caller.
When a service is configured to use a session-owning voice provider, Playground voice calls run through that provider rather than the default in-house pipeline. The provider generates its own opening audio, so the platform does not pre-render a separate greeting. Tools available to the provider are limited to those referenced in the service's context graph - the same authorization boundary that applies to phone calls.
Session-owning voice provider tool behavior. When a service uses a session-owning speech-to-speech voice provider, the provider's tool contract is aligned with the tools actually available at runtime. The provider receives only the tools authorized by the service's context graph, and its system prompt describes the exact execution model configured for the call - including whether tools run sequentially or in parallel, and whether write-capable tools are present. Services without tools receive no tool instructions, avoiding unnecessary model confusion.
Unified Runs
The Operate > Runs page provides one surface for conversation and framework activity. The Operate > Takeover shortcut opens the Live tab of that same Runs page:
Live - An urgency-ranked operator queue containing only live runs.
All runs - A filterable history across run kind, channel, and status.
Overview - Workspace totals and status distributions.
Live voice calls appear while they are in progress. Filters accept multiple values within an axis, so teams can review combinations such as failed and timed-out runs without losing workspace-wide pagination.
Opening a run composes the appropriate detail view:
Voice runs show audio, transcript, analysis, and eligible operator controls.
Text, SMS, email, and web runs show the conversation transcript, memory context, and eligible close or takeover actions.
Framework runs show metadata and an ordered structural trajectory of decisions, tool calls, and completion steps.
Eligible voice runs support listen and live takeover modes. Eligible non-voice runs support authored-turn takeover, where the operator pauses the agent, sends the next reply, and hands control back explicitly. Historical views include persisted successful, failed, and timed-out framework runs. Persistence is best-effort, so a run can still reach its correct terminal outcome even if its historical marker or trajectory is unavailable.
The playground answers the question: what does the agent actually do when I talk to it?
Voice Simulation
Voice testing separates text-tier quality (reasoning, tool selection, context graph coverage, and response content) from voice-path health (audio attachment, transcription, speech rendering, turn timing, latency, and capacity). Teams can run broad text-tier simulations for behavior regressions and reserve real-audio checks for the media path.
Voice testing answers the question: is the agent behavior correct, and does the audio path deliver it correctly?
Metrics
Metrics measure the quality of agent conversations across the dimensions that matter to your organization - safety, clinical accuracy, empathy, goal completion, and other use-case-specific criteria. Eligible production conversations and simulations can be evaluated through their supported definitions and execution paths, while human review provides separate calibration evidence. Do not assume that every session is evaluated or that one definition shape applies unchanged across Classic, Platform production-eval, and simulation APIs.
The platform supports two evaluation modes for production conversations:
On-demand evaluation. Trigger evaluation for a specific completed conversation through the API. Useful for spot-checking individual conversations or integrating evaluation into custom workflows.
Eager post-conversation evaluation. Where the feature is enabled, completed voice, text, SMS, email, and web conversations can be evaluated against active definitions after they end. Triggering is best-effort and bounded by task and per-workspace daily caps, so this path does not guarantee a verdict for every conversation. Use persisted eval results to distinguish evaluated, pending, skipped, and error outcomes.
Both modes use the same production-eval definitions and verdict model. Trigger mode alone does not change the rubric, but channel modality, retained evidence, model execution, and missing artifacts can still affect whether two results are meaningfully comparable.
Metrics answer the question: how well is the agent performing?
Drift Detection
Use metric trends, run history, and saved simulation baselines to look for sustained changes in inputs or outcomes. Teams choose the comparison windows, investigate representative interactions, update test cases, and make an explicit release decision. The platform does not currently turn a trend threshold into an automatic alert, promotion block, or rollback.
Drift detection answers the question: is the agent getting worse?
How the Pillars Work Together
These capabilities support a repeatable team workflow:
During development, the playground gives interactive feedback on agent behavior, and tool testing exercises individual tools outside a conversation with the side-effect policy documented for that tool family.
Before deployment, simulations exercise target scenarios and measure configured criteria, while targeted real-audio tests provide separate evidence about speech, timing, transport, and capacity changes.
In production, configured evaluations and analytics provide ongoing quality evidence.
Over time, teams compare equivalent periods, investigate material changes, and add newly discovered failure modes to the test set.
When the evidence indicates drift, update the relevant simulations and re-verify before releasing a change. This keeps the test set aligned with production without allowing an uncertain trend to change production automatically.
For Developers: See the REST API reference and Simulations reference for endpoint details, request/response schemas, and SDK code examples.
Last updated
Was this helpful?