Simulations
Automated agent testing with personas, scenarios, unit tests, and configurable success criteria.
Amigo's simulation system is an evaluation and testing framework for validating agent behavior before deploying to production. You define simulated users (personas), test scenarios, and success criteria, then run automated conversations to measure how your agent performs.
How Simulations Work
The simulation system uses five building blocks that compose together:
Building Blocks
Simulated user profiles with a background, role, and preferred language. Versioned so you can iterate on persona definitions without breaking existing tests.
Conversation scripts that define the objective, instructions for the simulated user, and how the conversation starts. Also versioned.
Combine a persona, a scenario, a service (with version set), and success criteria (metrics with thresholds) into a single test case.
Group multiple unit tests together, each with a configurable run count, to form a test suite.
Execute a unit test set. The platform runs all unit tests, evaluates metrics, and produces downloadable artifacts with the results.
Typical Workflow
Define personas that represent different user archetypes (for example, "confused new user", "expert power user", "frustrated customer").
Define scenarios that describe what the simulated user is trying to accomplish and how the conversation should start.
Create unit tests that pair a persona with a scenario, target a specific service and version set, and set success criteria based on conversation metrics.
Group unit tests into sets with run counts (for example, run each test 5 times for statistical significance).
Execute runs and review artifacts to see whether your agent meets the defined success criteria.
Versioning
Personas and scenarios are versioned independently. When you update a persona's background or a scenario's instructions, you create a new version. Unit tests reference the persona and scenario by ID and always use the latest version at run time. This lets you iterate on test definitions without recreating unit tests.
Tool Execution Modes
During simulations, tools are invoked with invocation_mode: "conversation-simulation" instead of "regular". This lets your tools mock external calls and avoid side effects. See Tools: Execution Modes for implementation details.
API Categories
Personas
Simulation Personas: create, list, search, update, delete, and version simulated user profiles.
Scenarios
Simulation Scenarios: create, list, search, update, delete, and version conversation test scenarios.
Unit Tests
Simulation Unit Tests: create, list, search, update, and delete individual test cases.
Unit Test Sets
Simulation Unit Test Sets: create, list, search, update, and delete grouped test suites.
Unit Test Set Runs
Simulation Unit Test Set Runs: execute test suites, monitor progress, cancel runs, and download result artifacts.
CLI Testing Tools (Agent Forge)
The Agent Forge SDK provides CLI commands that build on top of the simulation APIs for automated testing:
forge simulation run: coverage-optimized multi-session simulation that scores recommended responses against the context graph to systematically explore states, behaviors, and tools.forge simulation bridge: Claude-driven multi-scenario testing from a natural language objective, with pass^k consistency testing.forge simulation plan: generate target specs from natural language objectives or metric stress tests.forge simulation evaluate: compare metric scores across simulation runs (before/after diff mode).forge conversation simulate-step: agent-driven step-by-step simulation with interaction insights (current state, behaviors, tools called).
These CLI commands use ephemeral test users for parallel execution. See the Agent Forge CLI reference for setup and usage.
Related
Classic API -> Services
Classic API -> Tools
Data Access → Simulation Tables
Getting Started → Authentication
Last updated
Was this helpful?