> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/developer-guide/platform-api/safety/production-evals.md). # Production Evals Production evals let you run the same assertion and metric checks used in simulations against your workspace's live completed calls. You define eval definitions (what to check), evaluate completed calls on demand, and retrieve the verdicts. Metric evals reference metrics defined in the metric store, so define the metric there first: {% content-ref url="/pages/NA5V0i3dF76la165YCPP" %} [Metric Store](/developer-guide/platform-api/safety/metric-store.md) {% endcontent-ref %} ## Production Eval Definitions A production eval definition specifies an eval that runs against live calls. Definitions can be scoped to a specific service or applied workspace-wide (service ID omitted). When a call is evaluated, workspace-wide definitions apply to all calls, and service-scoped definitions apply only to calls handled by that service. If both a workspace-wide and a service-scoped definition share the same eval key, the service-scoped definition takes priority for that service's calls. ### Create a Production Eval Definition {% openapi src="" path="/v1/{workspace\_id}/production-eval-definitions" method="post" %} {% endopenapi %} Creates a new production eval definition. **Errors:** * `409 Conflict` - A definition with this eval key already exists in this scope * `422 Unprocessable Entity` - Validation error (e.g. `metric` type without `metric_key`, or `service_id` not in this workspace) ### List Production Eval Definitions {% openapi src="" path="/v1/{workspace\_id}/production-eval-definitions" method="get" %} {% endopenapi %} Returns a paginated list of production eval definitions. ### Get a Production Eval Definition {% openapi src="" path="/v1/{workspace\_id}/production-eval-definitions/{definition\_id}" method="get" %} {% endopenapi %} ### Update a Production Eval Definition {% openapi src="" path="/v1/{workspace\_id}/production-eval-definitions/{definition\_id}" method="patch" %} {% endopenapi %} Partial update. Include only the fields you want to change. **Errors:** * `404 Not Found` - Definition does not exist * `409 Conflict` - Update would create a duplicate eval key in this scope * `422 Unprocessable Entity` - No fields provided, or `service_id` not in this workspace ### Delete a Production Eval Definition {% openapi src="" path="/v1/{workspace\_id}/production-eval-definitions/{definition\_id}" method="delete" %} {% endopenapi %} ## Evaluating a Call {% openapi src="" path="/v1/{workspace\_id}/calls/{conversation\_id}/evaluate" method="post" %} {% endopenapi %} Runs all active production eval definitions against one completed call and persists the verdicts. The call must exist in the workspace. Definitions scoped to the call's service and workspace-wide definitions are both applied. If a workspace-wide and a service-scoped definition share the same eval key, the service-scoped definition takes priority. This endpoint is synchronous - the caller waits while the eval judge processes. Assertion evals (transcript contains, tool called, final state) are deterministic and fast. AI judge assertions and metric evals invoke an AI model and may take several seconds per eval. Re-evaluating a call overwrites prior verdicts for the same eval keys rather than creating duplicates. **Errors:** * `404 Not Found` - Conversation does not exist in this workspace **Notes:** * An empty `results` array means the call was found but no active definitions apply to it. * Each eval definition that encounters an error (e.g. AI model unavailable) produces an individual `error` status verdict rather than failing the entire evaluation. ## Retrieving Call Eval Results {% openapi src="" path="/v1/{workspace\_id}/calls/{conversation\_id}/eval-results" method="get" %} {% endopenapi %} Returns all persisted eval verdicts for a completed call. ## Permissions All production eval endpoints require workspace authentication. | Operation | Required Permission | | ---------------------------------- | ------------------- | | List, get definitions | Service view | | Create, update, delete definitions | Service update | | Evaluate a call | Service update | | Get eval results | Service view | ## Assertion Kinds | Kind | Behavior | | ---------------------------------------------- | -------------------------------------------------------------------------- | | `transcript_contains` / `must_contain` | Passes if the expected phrase appears in the transcript (case-insensitive) | | `transcript_not_contains` / `must_not_contain` | Passes if the expected phrase does NOT appear in the transcript | | `tool_called` | Passes if the expected tool was called during the conversation | | `final_state` | Passes if the conversation ended in the expected state | | `llm_judge` (default) | An AI judge evaluates the transcript against the assertion's expected text | For `transcript_contains` and `tool_called`, the expected value can be provided as the `expected` field or within `params` (as `phrase`, `text`, `tool_name`, or `state` depending on the kind). ## Metric Eval Expectations Metric evals compute the metric value for the call and compare it against the `expected` threshold: | Expected Format | Behavior | | ---------------------------- | ------------------------------------------------------------------------------------------ | | `{"gte": N}` or `{"min": N}` | Passes if value >= N | | `{"lte": N}` or `{"max": N}` | Passes if value <= N | | `{"equals": V}` | Passes if value equals V | | `{"contains": S}` | Passes if string representation contains S | | Bare value | Passes if value equals the expected value | | `null` / omitted | No threshold check; the metric value is computed and returned without a pass/fail judgment | Multiple threshold keys can be combined (e.g. `{"gte": 0.5, "lte": 1.0}`). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.amigo.ai/developer-guide/platform-api/safety/production-evals.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.