> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/developer-guide/platform-api/safety/production-evals.md).

# Production Evals

Production evals let you run the same assertion and metric checks used in simulations against your workspace's live completed calls. You define eval definitions (what to check), evaluate completed calls on demand, and retrieve the verdicts.

Metric evals reference metrics defined in the metric store, so define the metric there first:

{% content-ref url="/pages/NA5V0i3dF76la165YCPP" %}
[Metric Store](/developer-guide/platform-api/safety/metric-store.md)
{% endcontent-ref %}

## Production Eval Definitions

A production eval definition specifies an eval that runs against live calls. Definitions can be scoped to a specific service or applied workspace-wide (service ID omitted). When a call is evaluated, workspace-wide definitions apply to all calls, and service-scoped definitions apply only to calls handled by that service. If both a workspace-wide and a service-scoped definition share the same eval key, the service-scoped definition takes priority for that service's calls.

### Create a Production Eval Definition

`POST /v1/{workspace_id}/production-eval-definitions`

Creates a new production eval definition.

**Request body:**

| Field            | Type                 | Required    | Description                                                                                                                                                                                                 |
| ---------------- | -------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `eval_key`       | string (1-128 chars) | Yes         | Unique key identifying this eval within its scope                                                                                                                                                           |
| `eval_type`      | string               | Yes         | `assertion` or `metric`                                                                                                                                                                                     |
| `service_id`     | string (UUID)        | No          | Scope to a specific service. Omit for workspace-wide                                                                                                                                                        |
| `assertion_kind` | string               | No          | For assertions: `transcript_contains`, `must_contain`, `transcript_not_contains`, `must_not_contain`, `tool_called`, `final_state`, or `llm_judge` (default). Unrecognized kinds are routed to the AI judge |
| `metric_key`     | string               | Conditional | Required when `eval_type` is `metric`. Must match an active AI-evaluated call intelligence metric in the workspace (see [Metric Store](/developer-guide/platform-api/safety/metric-store.md))               |
| `expected`       | any                  | No          | For assertions: a phrase string or expected value. For metrics: a threshold object (e.g. `{"gte": 0.8}`, `{"lte": 5}`, `{"equals": true}`, `{"contains": "keyword"}`)                                       |
| `params`         | object               | No          | Additional parameters (e.g. `{"phrase": "..."}`, `{"tool_name": "..."}`, `{"state": "..."}`)                                                                                                                |
| `active`         | boolean              | No          | Whether this definition is active. Defaults to `true`                                                                                                                                                       |

**Response:** `201 Created` with the created definition object.

**Errors:**

* `409 Conflict` - A definition with this eval key already exists in this scope
* `422 Unprocessable Entity` - Validation error (e.g. `metric` type without `metric_key`, or `service_id` not in this workspace)

### List Production Eval Definitions

`GET /v1/{workspace_id}/production-eval-definitions`

Returns a paginated list of production eval definitions.

**Query parameters:**

| Parameter            | Type          | Description                     |
| -------------------- | ------------- | ------------------------------- |
| `service_id`         | string (UUID) | Filter by service               |
| `active`             | boolean       | Filter by active status         |
| `limit`              | integer       | Page size (max 100, default 10) |
| `continuation_token` | integer       | Offset for pagination           |

**Response:** Paginated response with `items`, `total`, and pagination metadata.

### Get a Production Eval Definition

`GET /v1/{workspace_id}/production-eval-definitions/{definition_id}`

**Response:** The definition object, or `404` if not found.

### Update a Production Eval Definition

`PATCH /v1/{workspace_id}/production-eval-definitions/{definition_id}`

Partial update. Include only the fields you want to change.

**Request body:** Same fields as create, all optional. At least one field must be provided.

**Response:** The updated definition object.

**Errors:**

* `404 Not Found` - Definition does not exist
* `409 Conflict` - Update would create a duplicate eval key in this scope
* `422 Unprocessable Entity` - No fields provided, or `service_id` not in this workspace

### Delete a Production Eval Definition

`DELETE /v1/{workspace_id}/production-eval-definitions/{definition_id}`

**Response:** `204 No Content` on success, `404` if not found.

## Evaluating a Call

`POST /v1/{workspace_id}/calls/{conversation_id}/evaluate`

Runs all active production eval definitions against one completed call and persists the verdicts. The call must exist in the workspace. Definitions scoped to the call's service and workspace-wide definitions are both applied. If a workspace-wide and a service-scoped definition share the same eval key, the service-scoped definition takes priority.

This endpoint is synchronous - the caller waits while the eval judge processes. Assertion evals (transcript contains, tool called, final state) are deterministic and fast. AI judge assertions and metric evals invoke an AI model and may take several seconds per eval.

Re-evaluating a call overwrites prior verdicts for the same eval keys rather than creating duplicates.

**Response:**

```json
{
  "conversation_id": "uuid",
  "results": [
    {
      "id": "uuid",
      "workspace_id": "uuid",
      "service_id": "uuid | null",
      "conversation_id": "uuid",
      "call_sid": "string | null",
      "eval_key": "string",
      "eval_type": "assertion | metric",
      "assertion_kind": "string | null",
      "metric_key": "string | null",
      "status": "passed | failed | pending | skipped | error",
      "passed": true,
      "score": 0.95,
      "expected": "...",
      "actual": "...",
      "rationale": "string | null",
      "justification": "string | null",
      "cited_turns": [0, 3, 7],
      "computed_at": "2026-07-01T12:00:00Z",
      "created_at": "2026-07-01T12:00:00Z"
    }
  ]
}
```

**Errors:**

* `404 Not Found` - Conversation does not exist in this workspace

**Notes:**

* An empty `results` array means the call was found but no active definitions apply to it.
* Each eval definition that encounters an error (e.g. AI model unavailable) produces an individual `error` status verdict rather than failing the entire evaluation.

## Retrieving Call Eval Results

`GET /v1/{workspace_id}/calls/{conversation_id}/eval-results`

Returns all persisted eval verdicts for a completed call.

**Response:** Same shape as the evaluate response.

## Eval Result Fields

| Field             | Type                    | Description                                                                |
| ----------------- | ----------------------- | -------------------------------------------------------------------------- |
| `id`              | string (UUID)           | Unique verdict identifier                                                  |
| `workspace_id`    | string (UUID)           | Workspace the verdict belongs to                                           |
| `service_id`      | string (UUID), nullable | Service that handled the call (from the definition)                        |
| `conversation_id` | string (UUID)           | The evaluated call                                                         |
| `call_sid`        | string, nullable        | Telephony call identifier, if available                                    |
| `eval_key`        | string                  | The eval definition's key                                                  |
| `eval_type`       | string                  | `assertion` or `metric`                                                    |
| `assertion_kind`  | string, nullable        | The assertion kind (for assertion evals)                                   |
| `metric_key`      | string, nullable        | The metric key (for metric evals)                                          |
| `status`          | string                  | `passed`, `failed`, `pending`, `skipped`, or `error`                       |
| `passed`          | boolean, nullable       | Whether the eval passed                                                    |
| `score`           | number, nullable        | Numeric score (0-1 for AI judge assertions; metric value for metric evals) |
| `expected`        | any, nullable           | The configured expectation                                                 |
| `actual`          | any, nullable           | What was observed                                                          |
| `rationale`       | string, nullable        | Human-readable explanation of the verdict                                  |
| `justification`   | string, nullable        | For AI-evaluated metrics: the model's explanation of the metric value      |
| `cited_turns`     | array of integers       | Turn indices in the conversation that support the verdict                  |
| `computed_at`     | string (datetime)       | When the verdict was computed                                              |
| `created_at`      | string (datetime)       | When the verdict was first persisted                                       |

## Permissions

All production eval endpoints require workspace authentication.

| Operation                          | Required Permission |
| ---------------------------------- | ------------------- |
| List, get definitions              | Service view        |
| Create, update, delete definitions | Service update      |
| Evaluate a call                    | Service update      |
| Get eval results                   | Service view        |

## Assertion Kinds

| Kind                                           | Behavior                                                                   |
| ---------------------------------------------- | -------------------------------------------------------------------------- |
| `transcript_contains` / `must_contain`         | Passes if the expected phrase appears in the transcript (case-insensitive) |
| `transcript_not_contains` / `must_not_contain` | Passes if the expected phrase does NOT appear in the transcript            |
| `tool_called`                                  | Passes if the expected tool was called during the conversation             |
| `final_state`                                  | Passes if the conversation ended in the expected state                     |
| `llm_judge` (default)                          | An AI judge evaluates the transcript against the assertion's expected text |

For `transcript_contains` and `tool_called`, the expected value can be provided as the `expected` field or within `params` (as `phrase`, `text`, `tool_name`, or `state` depending on the kind).

## Metric Eval Expectations

Metric evals compute the metric value for the call and compare it against the `expected` threshold:

| Expected Format              | Behavior                                                                                   |
| ---------------------------- | ------------------------------------------------------------------------------------------ |
| `{"gte": N}` or `{"min": N}` | Passes if value >= N                                                                       |
| `{"lte": N}` or `{"max": N}` | Passes if value <= N                                                                       |
| `{"equals": V}`              | Passes if value equals V                                                                   |
| `{"contains": S}`            | Passes if string representation contains S                                                 |
| Bare value                   | Passes if value equals the expected value                                                  |
| `null` / omitted             | No threshold check; the metric value is computed and returned without a pass/fail judgment |

Multiple threshold keys can be combined (e.g. `{"gte": 0.5, "lte": 1.0}`).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.amigo.ai/developer-guide/platform-api/safety/production-evals.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.