For the complete documentation index, see llms.txt. This page is also available as Markdown.

Simulation Coverage

Branch-and-bound simulation coverage API for systematic context graph state space exploration.

The simulation coverage API provides branch-and-bound exploration of context graph state space. It replaces random sampling with targeted exploration that steers conversations toward untested states, tools, and transitions.

Conceptual overview. For background on how coverage testing works, see the Simulation Coverage conceptual documentation.

Core Model

Coverage is organized into three levels:

Resource
Description

Coverage Run

A campaign that owns an isolated test branch for write isolation. Contains sessions.

Coverage Session

A conversation path within a run. Sessions can be forked from other sessions at any turn.

Coverage Turn

A single agent turn within a session, recording context graph state, tool calls, and scores.

Sessions form a tree structure through the fork mechanism. Each session tracks its parent_session_id and fork_turn_index, so the full exploration tree is reconstructable from the session list.

Endpoints

All endpoints are scoped to a workspace and require a workspace API key.

Create Coverage Run

Create Simulation Run

post

Create a simulation coverage run with an optional Lakebase branch.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
Body
service_idstring · uuidRequired
branch_namestring · max: 64 · nullableOptional
tagsstring[] · max: 20Optional
Responses
200

Successful Response

application/json
Other propertiesanyOptional
post/v1/{workspace_id}/simulations/runs

Returns the created run with ID, status (running), tags, and branch metadata.

Create Coverage Session

Creates a new conversation session within a run. The session starts with a fresh agent state.

Create Simulation Session

post

Create a simulation session within a run. Proxies to agent-engine.

entity_id ownership: agent-engine's _resolve_caller queries the entity with workspace_id scoping — a workspace-A entity_id resolves to None in workspace-B, so no cross-tenant data is exposed.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
run_idstring · uuidRequired
Body
service_idstring · uuidRequired
branch_namestring · max: 64 · nullableOptional
tagsstring[] · max: 20Optional
caller_idstring · max: 64 · nullableOptional

Simulated caller phone number used to resolve patient context when entity_id is not provided (or does not match a row). Omitted/blank values are normalized to the sim-orchestrator sentinel, which yields no phone match and an empty caller context — same default the voice test-call WebSocket uses, so patient lookups resolve consistently across modalities.

entity_idstring · uuid · nullableOptional

Optional world entity UUID to bind caller context directly. Precedence rules: (1) when this resolves to a world entity in the workspace, it WINS — caller_id is not used for identity resolution, and the resolution provenance is entity_id_lookup; (2) when this is a well-formed UUID with no matching entity (stale, deleted, wrong workspace), the session falls back to phone lookup against caller_id (provenance phone_lookup) — no error is raised, the sim is survivable; (3) malformed UUIDs are rejected with HTTP 422 before the request reaches agent-engine. The supplied caller_id is still recorded on the active call and surfaced in the greeting metadata in case (1).

Responses
200

Successful Response

application/json
session_idstringRequired
greetingstringRequired
is_terminalbooleanRequired
post/v1/{workspace_id}/simulations/runs/{run_id}/sessions

Step a Session

Sends a simulated user message and records the agent's response as a turn.

Simulation Step

post

Step a simulation session and auto-store the turn observation.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
session_idstringRequired
Body
session_idstring · max: 64Required
caller_textstring · max: 2000Required
emotionstring · max: 50 · nullableOptional
valencenumber · min: -1 · max: 1 · nullableOptional
Responses
200

Successful Response

application/json
post/v1/{workspace_id}/simulations/sessions/{session_id}/step

Returns the turn observation including agent response, context graph state, tool calls, and turn index.

Fork a Session

Atomically clones the parent session state N times, steps each clone with a different simulated message, and returns all observations. This is the core branch-and-bound primitive.

Simulation Fork

post

Fork a session into N branches — clone + step each alternative atomically.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
session_idstringRequired
Body
Responses
200

Successful Response

application/json
Other propertiesanyOptional
post/v1/{workspace_id}/simulations/sessions/{session_id}/fork

Each child session inherits the parent's full conversation history up to the fork point. The parent session remains unchanged.

Score a Session

Evaluates the session against configured metrics. Scores are attributed per-session, not per-state-visit.

Score Simulation Session

post

ASI assigns a score to a completed simulation session.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
session_idstringRequired
Body
scorenumber · max: 100Required
score_rationalestring · max: 2000 · nullableOptional
Responses
200

Successful Response

application/json
Other propertiesstringOptional
post/v1/{workspace_id}/simulations/sessions/{session_id}/score

Returns a confirmation status ({ "status": "scored" }). The supplied score and rationale are persisted; this endpoint does not return computed metric scores. Aggregated metric and coverage scores are read back through the coverage graph, the session list, or the benchmark results endpoint.

Get Coverage Graph

Returns the knowledge graph for the run. The graph has two layers:

Layer
Description

Observed turns

Aggregated per context graph state: session count, pass rate, total turns

Topology overlay

Ghost nodes for context graph states that exist in the state machine but were never reached. These represent coverage gaps.

Get Simulation Coverage Graph

get

Get the B&B knowledge graph for a service — topology + observations merged.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
service_idstring · uuidRequired
Query parameters
run_idstring · uuid · nullableOptional
include_turnsbooleanOptionalDefault: false
Responses
200

Successful Response

application/json
Other propertiesanyOptional
get/v1/{workspace_id}/simulations/services/{service_id}/graph

Complete a Coverage Run

Marks the run as finished and cleans up the isolated test branch. Sessions and turns remain available for read access.

Complete Simulation Run

post
Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
run_idstring · uuidRequired
Query parameters
errorstring · nullableOptional
Responses
200

Successful Response

application/json
Other propertiesstringOptional
post/v1/{workspace_id}/simulations/runs/{run_id}/complete

Get Coverage Run

Returns the full run payload, including the stored bridge_request (original request body) and scenarios (generated persona/scenario list) for runs created through the Simulation Bridge. The list runs endpoint omits these fields to keep responses compact.

The response also includes a checks array alongside the raw eval_results - a normalized, read-only projection that flattens each metric and assertion verdict into one uniform shape (source_type, key, label, expected/actual with display strings, verdict, optional score, rationale, references). Per-case assertion detail responses carry the same checks array. Use it for consistent rendering; the raw fields are unchanged. See the Simulations API reference for the full schema.

Get Simulation Run

get

Fetch a single run including objective / bridge_request / scenarios.

Use the bridge_request payload to re-POST /bridge and replay the run.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
run_idstring · uuidRequired
Responses
200

Successful Response

application/json

Full run payload — used by GET /runs/{run_id} for replay flows.

idstring · uuidRequired
service_idstring · uuidRequired
branch_namestring · max: 64 · nullableOptional
statusstring · enumRequiredPossible values:
tagsstring[]Optional
total_sessionsintegerOptionalDefault: 0
total_turnsintegerOptionalDefault: 0
objectivestring · max: 2000 · nullableOptional
total_scenarios_requestedinteger · nullableOptional
created_atstring · nullableOptional
completed_atstring · nullableOptional
metric_statusstring · enumOptionalDefault: pendingPossible values:
metric_result_countintegerOptionalDefault: 0
metrics_last_checked_atstring · date-time · nullableOptional
eval_result_countintegerOptionalDefault: 0
eval_pass_countintegerOptionalDefault: 0
eval_fail_countintegerOptionalDefault: 0
eval_error_countintegerOptionalDefault: 0
get/v1/{workspace_id}/simulations/runs/{run_id}

Simulation Bridge

The simulation bridge turns a natural language test objective into a set of autonomous test scenarios and runs them end-to-end against a service.

Simulation Bridge

post

NL objective to N autonomous scenarios against the tracked simulation primitives.

Returns immediately with run_id + generated scenarios. Scenarios run in the background via BridgeExecutor (lifecycle-managed, shutdown-drained). Completion events are emitted to Delta via world_emitter for SDP observability.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
Body
service_idstring · uuidRequired
objectivestring · min: 1 · max: 2000Required
num_scenariosinteger · min: 1 · max: 50OptionalDefault: 5
max_turnsinteger · min: 1 · max: 40OptionalDefault: 20
concurrencyinteger · min: 1 · max: 20OptionalDefault: 3
branch_namestring · max: 64 · nullableOptional
entity_idstring · uuid · nullableOptional

Optional world entity UUID to bind caller context for every scenario session created by this bridge run. The value is forwarded to each scenario's session-create call and inherited by any forks; identical precedence rules apply per-session (entity match wins over phone, stale UUID falls back to phone lookup, malformed UUID returns 422). Use this to pin an entire regression suite to a specific test patient.

tagsstring[] · max: 20Optional
auto_target_specbooleanOptionalDefault: false
async_generationbooleanOptionalDefault: false
Responses
200

Successful Response

application/json
run_idstring · uuidRequired
statusstring · enumRequiredPossible values:
case_idsstring · uuid[]Optional
inferred_target_spec_rationalestring · max: 1000 · nullableOptional
post/v1/{workspace_id}/simulations/bridge

The request supports these notable fields:

  • objective (string) - natural language description of what to test (e.g., "test appointment scheduling with difficult patients who change their mind")

  • num_scenarios (integer, default 5) - number of scenario variations to generate

  • max_turns (integer, default 20) - maximum turns per scenario before stopping

  • target_spec (object) - optional scoring criteria for the run. See Target Spec below.

  • exploration (object) - optional controls for coverage-driven behavior. See Exploration below.

  • auto_target_spec (boolean) - when true and target_spec is absent, infers a target spec from the objective before running. The inferred spec is returned in the response.

  • async_generation (boolean) - when true, returns the run ID immediately while target-spec inference, scenario generation, and scenario execution continue in the background. Poll Get Coverage Run for progress.

Target Spec

The target_spec object defines structured scoring criteria for evaluating scenario outcomes.

Field
Type
Description

desired_states

array of strings

Context graph states the conversation should reach

non_desired_states

array of objects

States the conversation should avoid. Each object has state (string) and mode ("hard" or "soft" - hard failures fail the scenario immediately, soft failures are penalized in scoring)

ordered_pathways

array of objects

Expected state sequences. Each object has sequence (array of strings) representing the ordered states the conversation should traverse

completion_criteria

object

Conditions that define a successful conversation. Contains optional state_based (array of strings - states that count as completed) and tool_based (array of strings - tool calls that count as completed)

Exploration

The exploration object controls how the bridge explores the state space.

Field
Type
Description

coverage_driven_selection

boolean

When true, steers scenario generation toward untested or low-coverage states

enable_forking

boolean

When true, allows sessions to fork mid-conversation to explore alternative paths

max_forks_per_scenario

integer

Maximum number of forks allowed per scenario

recommend_candidates

integer

Number of fork candidates to recommend at each decision point

Response

Returns the created coverage run with generated scenarios. Each scenario includes a persona (name, background, temperament) and conversation strategy (initial message, instructions).

When async_generation is true, the response returns status: running with an empty scenarios array. Use the returned run_id with Get Coverage Run to read generated scenarios and final status as they become available.

When auto_target_spec was set to true, the response also includes:

Field
Type
Description

inferred_target_spec

object

The target spec that was inferred from the objective (same structure as the target_spec request field)

inferred_target_spec_rationale

string

Explanation of why each state and tool was included in the inferred spec (max 1000 characters)

The bridge response itself carries no execution summary. Read total_sessions and per-session results back through Get Coverage Run once execution finishes.

Each session is tagged with metadata for joining sessions back to their source scenarios: scenario_index:N identifies which generated scenario produced the session, and temperament:X records the persona temperament used.

The bridge creates a tracked coverage run, so results are visible through the existing coverage graph, session list, and scoring endpoints.

This is the REST API equivalent of forge platform sim bridge - same scenario generation and autonomous execution, accessible without the CLI.

Saved Case Benchmarks

Saved-case benchmarks run a tag-selected batch of simulation cases and queue each selected case as its own bridge run. The benchmark response returns immediately with a stable batch_id, the selected case IDs, started run IDs, skipped cases, failed-to-start cases, and an aggregate start summary.

Run Simulation Benchmark

post

Run a suite or tag-selected benchmark batch as one saved-case run per case.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
Body
suite_idstring · uuid · nullableOptional
required_tagsstring[] · max: 20Optional
service_idstring · uuid · nullableOptional
entity_idstring · uuid · nullableOptional

Optional world entity UUID to bind caller context for every selected saved-case run. When omitted, each case falls back to fixtures.case_specific.entity_id if present.

max_casesinteger · min: 1 · max: 200OptionalDefault: 20
max_turnsinteger · min: 1 · max: 40OptionalDefault: 20
concurrencyinteger · min: 1 · max: 20OptionalDefault: 1
branch_namestring · max: 64 · nullableOptional
tagsstring[] · max: 20Optional
Responses
200

Successful Response

application/json
batch_idstring · uuidRequired
suite_idstring · uuid · nullableOptional
required_tagsstring[]Required
selected_case_idsstring · uuid[]Required
run_idsstring · uuid[]Required
post/v1/{workspace_id}/simulations/benchmarks/run

Results

Use POST /v1/{workspace_id}/simulations/benchmarks/results with the returned run_ids to aggregate completion status, scores, pass counts, metric availability, per-run summaries, and capability breakdowns.

Bridge Plan

Generates a target spec from a natural language objective without running any simulations. Use this to preview and edit the inferred spec before passing it to the bridge endpoint.

Simulation Bridge Plan

post

Infer a TargetSpec from an NL objective — preview before running /bridge.

UX: user types "test scheduling failures in 20 different ways", we return a structured target_spec + rationale. The frontend shows both, lets the user edit (add/remove states, tighten pathways), then POSTs the final spec to /bridge. Same LLM, same Vertex cost structure as /bridge, but no run is created — pure inference.

Authorizations
AuthorizationstringRequired

API key issued via POST /v1/{workspace_id}/api-keys. Pass the returned api_key value as a Bearer token.

Path parameters
workspace_idstring · uuidRequired
Body
service_idstring · uuidRequired
objectivestring · min: 1 · max: 2000Required
Responses
200

Successful Response

application/json
rationalestring · max: 1000Required
available_statesstring[] · max: 200Optional
available_toolsstring[] · max: 200Optional
post/v1/{workspace_id}/simulations/bridge/plan

The response includes the inferred target_spec (same structure as the bridge request target_spec field), a rationale explaining why each state and tool was included, and available_states and available_tools listing every state and tool in the service's context graph for UI autocomplete when editing the spec.

Write Isolation

Coverage runs use isolated write branches. All tool writes during simulation sessions go to the test branch, not production data. When the run completes, the branch is cleaned up. This allows coverage testing to exercise tools that create appointments, update patient records, or write to the world model without contaminating live data.

Last updated

Was this helpful?