Metrics
Define and evaluate conversation quality metrics - boolean, numerical, or categorical - with automated scoring.
Overview
Metric Value Types
// Boolean: "Did the agent follow the safety protocol?"
{ "type": "boolean" }
// Numerical: "Rate empathy on a 1-10 scale"
{ "type": "numerical", "lower_bound": 1, "upper_bound": 10 }
// Categorical: "Classify the conversation outcome"
{ "type": "categorical", "categories": ["resolved", "escalated", "abandoned"] }Evaluation Sources
Typical Lifecycle
Create a Metric
Create a metric.
Permissions
Metric:CreateMetricfor the metric to create.
The username should be set to {org_id}_{user_id}, and the password should be the Amigo issued JWT token that identifies the user.
Amigo issued JWT token that identifies an user. It's issued either after logging in through the frontend, or manually through the SignInWithAPIKey endpoint.
An optional organization identifier that indicates from which organization the token is issued. This is used in rare cases where the user to authenticate is making a request for resources in another organization.
The Mongo cluster name to perform this request in. This is usually not needed unless the organization does not exist yet in the Amigo organization infra config database.
[]The name of the metric. Must be unique within the organization.
The description of the metric.
The services that the metric is applied to.
Additional notes about the metric.
Succeeded
Invalid authorization credentials.
Missing required permissions.
Specified organization is not found.
A metric with this name already exists
Invalid request path parameter or request body failed validation.
The user has exceeded the rate limit of 100 requests per minute for this endpoint.
The service is going through temporary maintenance.
List Metrics
Retrieve metrics that match the given filters.
Permissions
This endpoint may be impacted by the following permissions:
- Only metrics that the authenticated user has
Metric:GetMetricpermission for will be retrieved.
The username should be set to {org_id}_{user_id}, and the password should be the Amigo issued JWT token that identifies the user.
Amigo issued JWT token that identifies an user. It's issued either after logging in through the frontend, or manually through the SignInWithAPIKey endpoint.
An optional organization identifier that indicates from which organization the token is issued. This is used in rare cases where the user to authenticate is making a request for resources in another organization.
The IDs of the metrics to retrieve.
[]The maximum number of metrics to return.
50The continuation token from the previous request used to retrieve the next page of metrics.
0The IDs of the services that the metric is applied to.
[]Whether the metric is deleted.
falseThe creators of the dynamic behavior sets.
[]The tags of the dynamic behavior sets. Must be specified using the syntax key:value, which means to match all sets with the given key and value pair among its tags. If value is *, it means the value does not matter. If value is empty, it matches against when the value is None.
[]The fields to sort the sets by. Supported fields are updated_at. Specify a + before the field name to indicate ascending sorting and - for descending sorting. Multiple fields can be specified to break ties.
[]The Mongo cluster name to perform this request in. This is usually not needed unless the organization does not exist yet in the Amigo organization infra config database.
[]Succeeded
Whether there are more metrics to retrieve.
The continuation token to use to retrieve the next set of metrics.
For each filter that this endpoint supports that can take on dynamic values, this field includes what these values are. This is only provided for the first page in the pagination results.
Note that the values are counted assuming the authenticated user has access to all the metrics, so they might differ from how many are actually retrieved.
Invalid authorization credentials.
Missing required permissions.
Specified organization is not found.
Invalid request path parameter or request query parameter failed validation.
The user has exceeded the rate limit of 20 requests per minute for this endpoint.
The service is going through temporary maintenance.
Search Metrics
Search for metrics that match the given filters and contain the given query in its name or description. Only the top 50 results will be returned. The results will be sorted by the relevance of the search query.
Permissions
This endpoint may be impacted by the following permissions:
- Only metrics that the authenticated user has
Metric:GetMetricpermission for will be retrieved.
The username should be set to {org_id}_{user_id}, and the password should be the Amigo issued JWT token that identifies the user.
Amigo issued JWT token that identifies an user. It's issued either after logging in through the frontend, or manually through the SignInWithAPIKey endpoint.
An optional organization identifier that indicates from which organization the token is issued. This is used in rare cases where the user to authenticate is making a request for resources in another organization.
The query to search for. Any metrics containing the terms in its name and description would be returned.
The IDs of the services that the metric is applied to.
[]The creators of the metrics. Each value must be of the format org_id,user_id.
[]The tags of the metrics. Must be specified using the syntax key:value, which means to match all metrics with the given key and value pair among its tags.
If value is *, it means the value does not matter. If value is empty, it matches against when the value is None.
[]The Mongo cluster name to perform this request in. This is usually not needed unless the organization does not exist yet in the Amigo organization infra config database.
[]Succeeded.
Invalid authorization credentials.
Missing required permissions.
Specified organization is not found.
Invalid request path parameter or request query parameter failed validation.
The user has exceeded the rate limit of 20 requests per minute for this endpoint.
The service is going through temporary maintenance.
Update a Metric
Update properties of a metric. The metric's name and metric values cannot be updated.
Permissions
This endpoint requires the following permissions:
Metric:ModifyMetricfor the metric to modify.
The username should be set to {org_id}_{user_id}, and the password should be the Amigo issued JWT token that identifies the user.
Amigo issued JWT token that identifies an user. It's issued either after logging in through the frontend, or manually through the SignInWithAPIKey endpoint.
An optional organization identifier that indicates from which organization the token is issued. This is used in rare cases where the user to authenticate is making a request for resources in another organization.
The ID of the metric to update.
^[a-f0-9]{24}$The Mongo cluster name to perform this request in. This is usually not needed unless the organization does not exist yet in the Amigo organization infra config database.
[]The description of the metric. Only updated if set.
The services that the metric is applied to. Only updated if set.
Additional notes about the metric. Only updated if set.
{}A specific type to indicate that a field is not set in the request.
The tags of the metric. Only updated if set.
Succeeded
Invalid authorization credentials.
Missing required permissions.
Specified organization or metric is not found.
Invalid request path parameter or request body failed validation.
The user has exceeded the rate limit of 20 requests per minute for this endpoint.
The service is going through temporary maintenance.
No content
Delete a Metric
Delete a metric. The metric is no longer modifiable or usable, but remains in the database. One can still retrieve it using its ID to view metrics that were evaluated in previous conversations.
This endpoint will error if the metric is used in any simulation unit tests.
Permissions
This endpoint requires the following permissions:
Metric:DeleteMetricfor the metric to delete.
The username should be set to {org_id}_{user_id}, and the password should be the Amigo issued JWT token that identifies the user.
Amigo issued JWT token that identifies an user. It's issued either after logging in through the frontend, or manually through the SignInWithAPIKey endpoint.
An optional organization identifier that indicates from which organization the token is issued. This is used in rare cases where the user to authenticate is making a request for resources in another organization.
The ID of the metric to update.
^[a-f0-9]{24}$The Mongo cluster name to perform this request in. This is usually not needed unless the organization does not exist yet in the Amigo organization infra config database.
[]Succeeded.
Metric is used in a simulation unit test.
Invalid authorization credentials.
Missing required permissions.
Specified organization or metric is not found.
Invalid request path parameter failed validation.
The user has exceeded the rate limit of 100 requests per minute for this endpoint.
The service is going through temporary maintenance.
No content
Evaluate Metrics
Evaluate the specified metrics for a completed conversation. The results will be stored and retrievable through the GetMetricEvaluationResults
endpoint with a special source type of manual.
If the same conversation has been manually evaluated for the same metric, the previous evaluation result will be overwritten.
Permissions
This endpoint requires the following permissions:
Metric:GetMetricfor the metrics.Metric:EvaluateMetricfor the metrics.Metric:GetMetricEvaluationResultfor the metric results.
The username should be set to {org_id}_{user_id}, and the password should be the Amigo issued JWT token that identifies the user.
Amigo issued JWT token that identifies an user. It's issued either after logging in through the frontend, or manually through the SignInWithAPIKey endpoint.
An optional organization identifier that indicates from which organization the token is issued. This is used in rare cases where the user to authenticate is making a request for resources in another organization.
The Mongo cluster name to perform this request in. This is usually not needed unless the organization does not exist yet in the Amigo organization infra config database.
[]The IDs of the metrics to evaluate.
The ID of the conversation to evaluate the metrics for.
^[a-f0-9]{24}$If specified, only messages up to (and including) this interaction will be evaluated.
^[a-f0-9]{24}$Succeeded.
The conversation has no interactions.
Invalid authorization credentials.
Missing required permissions.
Specified organization, conversation, or metric is not found.
Invalid request path parameter or request body failed validation.
The user has exceeded the rate limit of 50 requests per minute for this endpoint.
The service is going through temporary maintenance.
Get Evaluation Results
Retrieve metric evaluation results that match the given filters.
Permissions
This endpoint may be impacted by the following permissions:
- Only metric evaluation results that the authenticated user has
Metric:GetMetricEvaluationResultpermission for will be retrieved.
The username should be set to {org_id}_{user_id}, and the password should be the Amigo issued JWT token that identifies the user.
Amigo issued JWT token that identifies an user. It's issued either after logging in through the frontend, or manually through the SignInWithAPIKey endpoint.
An optional organization identifier that indicates from which organization the token is issued. This is used in rare cases where the user to authenticate is making a request for resources in another organization.
The maximum number of metrics to return.
500The continuation token from the previous request used to retrieve the next page of metrics.
0An ISO8601 timestamp. Only results evaluated on or after this moment is returned.
An ISO8601 timestamp. Only results evaluated on or before this moment is returned.
The ID of the metric to retrieve.
[]For metric evaluation sources with type post-session or manual, this field filters the metric evaluation results to only include those that were computed from the given conversation IDs.
[]For metric evaluation sources with type simulation, this field filters the metric evaluation results to only include those that were computed from the given simulation unit test set run IDs.
[]The fields to sort the metrics by. Supported fields are created_at. Specify a + before the field name to indicate ascending sorting and - for descending sorting.
Multiple fields can be specified to break ties.
[]The Mongo cluster name to perform this request in. This is usually not needed unless the organization does not exist yet in the Amigo organization infra config database.
[]Succeeded
Whether there are more metric evaluation results to retrieve.
The continuation token to use to retrieve the next set of metric evaluation results.
For each filter that this endpoint supports that can take on dynamic values, this field includes what these values are. This is only provided for the first page in the pagination results.
Note that the values are counted assuming the authenticated user has access to all the metric evaluation results, so they might differ from how many are actually retrieved.
Invalid authorization credentials.
Missing required permissions.
Specified organization is not found.
Invalid request path parameter or request query parameter failed validation.
The user has exceeded the rate limit of 10 requests per minute for this endpoint.
The service is going through temporary maintenance.
Related
Last updated
Was this helpful?