Production Evals
Define eval criteria for production calls and evaluate completed calls against them.
Production evals let you run the same assertion and metric checks used in simulations against your workspace's live completed calls. You define eval definitions (what to check), evaluate completed calls on demand, and retrieve the verdicts.
Metric evals reference metrics defined in the metric store, so define the metric there first:
Production Eval Definitions
A production eval definition specifies an eval that runs against live calls. Definitions can be scoped to a specific service or applied workspace-wide (service ID omitted). When a call is evaluated, workspace-wide definitions apply to all calls, and service-scoped definitions apply only to calls handled by that service. If both a workspace-wide and a service-scoped definition share the same eval key, the service-scoped definition takes priority for that service's calls.
Create a Production Eval Definition
POST /v1/{workspace_id}/production-eval-definitions
Creates a new production eval definition.
Request body:
eval_key
string (1-128 chars)
Yes
Unique key identifying this eval within its scope
eval_type
string
Yes
assertion or metric
service_id
string (UUID)
No
Scope to a specific service. Omit for workspace-wide
assertion_kind
string
No
For assertions: transcript_contains, must_contain, transcript_not_contains, must_not_contain, tool_called, final_state, or llm_judge (default). Unrecognized kinds are routed to the AI judge
metric_key
string
Conditional
Required when eval_type is metric. Must match an active AI-evaluated call intelligence metric in the workspace (see Metric Store)
expected
any
No
For assertions: a phrase string or expected value. For metrics: a threshold object (e.g. {"gte": 0.8}, {"lte": 5}, {"equals": true}, {"contains": "keyword"})
params
object
No
Additional parameters (e.g. {"phrase": "..."}, {"tool_name": "..."}, {"state": "..."})
active
boolean
No
Whether this definition is active. Defaults to true
Response: 201 Created with the created definition object.
Errors:
409 Conflict- A definition with this eval key already exists in this scope422 Unprocessable Entity- Validation error (e.g.metrictype withoutmetric_key, orservice_idnot in this workspace)
List Production Eval Definitions
GET /v1/{workspace_id}/production-eval-definitions
Returns a paginated list of production eval definitions.
Query parameters:
service_id
string (UUID)
Filter by service
active
boolean
Filter by active status
limit
integer
Page size (max 100, default 10)
continuation_token
integer
Offset for pagination
Response: Paginated response with items, total, and pagination metadata.
Get a Production Eval Definition
GET /v1/{workspace_id}/production-eval-definitions/{definition_id}
Response: The definition object, or 404 if not found.
Update a Production Eval Definition
PATCH /v1/{workspace_id}/production-eval-definitions/{definition_id}
Partial update. Include only the fields you want to change.
Request body: Same fields as create, all optional. At least one field must be provided.
Response: The updated definition object.
Errors:
404 Not Found- Definition does not exist409 Conflict- Update would create a duplicate eval key in this scope422 Unprocessable Entity- No fields provided, orservice_idnot in this workspace
Delete a Production Eval Definition
DELETE /v1/{workspace_id}/production-eval-definitions/{definition_id}
Response: 204 No Content on success, 404 if not found.
Evaluating a Call
POST /v1/{workspace_id}/calls/{conversation_id}/evaluate
Runs all active production eval definitions against one completed call and persists the verdicts. The call must exist in the workspace. Definitions scoped to the call's service and workspace-wide definitions are both applied. If a workspace-wide and a service-scoped definition share the same eval key, the service-scoped definition takes priority.
This endpoint is synchronous - the caller waits while the eval judge processes. Assertion evals (transcript contains, tool called, final state) are deterministic and fast. AI judge assertions and metric evals invoke an AI model and may take several seconds per eval.
Re-evaluating a call overwrites prior verdicts for the same eval keys rather than creating duplicates.
Response:
Errors:
404 Not Found- Conversation does not exist in this workspace
Notes:
An empty
resultsarray means the call was found but no active definitions apply to it.Each eval definition that encounters an error (e.g. AI model unavailable) produces an individual
errorstatus verdict rather than failing the entire evaluation.
Retrieving Call Eval Results
GET /v1/{workspace_id}/calls/{conversation_id}/eval-results
Returns all persisted eval verdicts for a completed call.
Response: Same shape as the evaluate response.
Eval Result Fields
id
string (UUID)
Unique verdict identifier
workspace_id
string (UUID)
Workspace the verdict belongs to
service_id
string (UUID), nullable
Service that handled the call (from the definition)
conversation_id
string (UUID)
The evaluated call
call_sid
string, nullable
Telephony call identifier, if available
eval_key
string
The eval definition's key
eval_type
string
assertion or metric
assertion_kind
string, nullable
The assertion kind (for assertion evals)
metric_key
string, nullable
The metric key (for metric evals)
status
string
passed, failed, pending, skipped, or error
passed
boolean, nullable
Whether the eval passed
score
number, nullable
Numeric score (0-1 for AI judge assertions; metric value for metric evals)
expected
any, nullable
The configured expectation
actual
any, nullable
What was observed
rationale
string, nullable
Human-readable explanation of the verdict
justification
string, nullable
For AI-evaluated metrics: the model's explanation of the metric value
cited_turns
array of integers
Turn indices in the conversation that support the verdict
computed_at
string (datetime)
When the verdict was computed
created_at
string (datetime)
When the verdict was first persisted
Permissions
All production eval endpoints require workspace authentication.
List, get definitions
Service view
Create, update, delete definitions
Service update
Evaluate a call
Service update
Get eval results
Service view
Assertion Kinds
transcript_contains / must_contain
Passes if the expected phrase appears in the transcript (case-insensitive)
transcript_not_contains / must_not_contain
Passes if the expected phrase does NOT appear in the transcript
tool_called
Passes if the expected tool was called during the conversation
final_state
Passes if the conversation ended in the expected state
llm_judge (default)
An AI judge evaluates the transcript against the assertion's expected text
For transcript_contains and tool_called, the expected value can be provided as the expected field or within params (as phrase, text, tool_name, or state depending on the kind).
Metric Eval Expectations
Metric evals compute the metric value for the call and compare it against the expected threshold:
{"gte": N} or {"min": N}
Passes if value >= N
{"lte": N} or {"max": N}
Passes if value <= N
{"equals": V}
Passes if value equals V
{"contains": S}
Passes if string representation contains S
Bare value
Passes if value equals the expected value
null / omitted
No threshold check; the metric value is computed and returned without a pass/fail judgment
Multiple threshold keys can be combined (e.g. {"gte": 0.5, "lte": 1.0}).
Last updated
Was this helpful?