> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/intelligence-and-analytics/metric-store.md). # Metric Store ## Semantic Foundation The Metric Store is built on a governed semantic layer that enforces consistency, traceability, and workspace-scoped security across all metric definitions and computations. ### Canonical Models Metrics are computed against a fixed set of canonical models that represent the platform's core analytical domains: * **Conversation fact** - conversation and run lifecycle * **Interaction fact** - user, agent, and system interactions * **Tool execution fact** - decision, state-transition, and tool-execution evidence * **Conversation outcome fact** - conversation-to-outcome relationships * **Evaluation fact** - call-quality, safety, latency, and voice-judge evaluations * **Topic classification fact** - conversation-topic classifications * **Appointment workflow fact** - appointment, slot, workflow, and state transitions * **Entity dimension** - workspace-scoped entity and identity relationships * **Event evidence fact** - typed event, evidence, and review records * **Model health fact** - connector, event-throughput, and metric-freshness health observations * **Surface interaction fact** - workspace-scoped surface and form interaction events Each canonical model declares its grain (for example, one conversation or one tool execution), its key columns, its timestamp columns, and an explicit freshness target. These declarations make the contract between raw platform data and computed metrics auditable. ### Governed Relationships Canonical models are connected through governed relationships that define how models can be joined. Each relationship declares its parent and child model, join keys, cardinality (such as one-to-many), and allowed join direction. Relationships are versioned and certified independently of the models they connect. For example, a conversation can have many interactions, many tool executions, many outcomes, many topic classifications, many evaluations, and many surface interactions. An outcome can have many appointment workflow events. These relationships are enforced at definition time so that metric authors cannot construct unsupported joins. ### Semantic Metric Definitions A semantic metric definition wraps a standard metric definition with additional governance: * **Base model** - which canonical model the metric is computed against * **Dimensions** - the approved dimensional axes (workspace and reporting period are always required; service and channel are optional) * **Time field** - which timestamp column drives the reporting period * **Owner** - the responsible party for the metric * **Security classification** - currently workspace-scoped, ensuring metrics never cross workspace boundaries * **Version and status** - definitions move through draft, certified, and deprecated states * **Required relationships** - which governed relationships the metric depends on * **Validation tests** - references to tests that verify the metric's correctness Definitions are compiled into deterministic execution plans before computation. The compilation step validates that the base model exists in the certified registry, that all required relationships are certified, and that the requested dimensions and time field are supported by the base model. The execution plan includes a content hash so that identical definitions produce identical plans. ### Freshness Tracking Each canonical model and metric definition declares an explicit freshness target in minutes. The platform classifies each materialized result as: * **Fresh** - computed within the freshness target * **Stale** - computed but older than the freshness target * **Never computed** - no successful computation has occurred * **Failed** - the most recent computation attempt failed This classification drives the freshness indicators visible in dashboards and health monitoring. ### Computation Certified metric definitions are computed periodically against their canonical models. Each computation run: 1. Reads the certified execution plan for each active metric definition 2. Queries the canonical model within a configurable time window (up to 7 days) 3. Aggregates values by the declared dimensions using the plan's aggregation type (count, count distinct, sum, average, min, max, ratio, or rate) 4. Merges results into the metric values store using an upsert strategy keyed by workspace, metric, version, period, and dimension combination 5. Records the computation run's status, timing, and any error for observability Computation is workspace-scoped: a metric definition owned by one workspace can only read data from that workspace's partition of the canonical model. ## Setup and Refresh The Metric Store provides two operational endpoints for workspace administrators: * **Setup** triggers a one-time schema and registry initialization for the workspace. This is an idempotent operation - running it more than once has no adverse effect. * **Refresh** triggers an on-demand refresh of the canonical Metric Store models. Use this when you need updated metrics before the next scheduled pipeline run. Both operations are asynchronous. The endpoint returns a run identifier that you can poll for status. The run status reports lifecycle state, result state, and a human-readable message. Setup and refresh require the workspace update permission. Each endpoint returns `202 Accepted` with a run identifier. If the backing job is not yet configured for the environment, the endpoint returns `503 Service Unavailable`. ### Workspace Scoping All canonical Metric Store models enforce workspace-level scoping. Rows without a valid workspace identifier are excluded during materialization, ensuring that every record in the serving layer is attributable to a single workspace. ## Canonical Models The Metric Store includes a set of canonical models that normalize data from multiple platform sources into consistent, workspace-scoped views. These models are materialized as governed projections and serve as the foundation for metrics, dashboards, and downstream analytics. Canonical models use a pointer-driven design: physical source tables are resolved through configuration rather than hard-coded references, so environment changes do not require editing transformation logic. ### Available Models | Model | Purpose | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Entity Dimension | Workspace-scoped entity identity and governed dimensions. Joins entity attributes with identity mappings to produce a single row per entity with canonical identifiers, display names, service assignments, patient and provider references, facility assignments, and channel information. | | Conversation Fact | Cross-channel conversation lifecycle at one row per workspace conversation. Merges run-level metadata with session and voice activity to produce start and end timestamps, status, run counts, and interaction counts. | | Interaction Fact | Normalized text, voice, and world-event interactions. Unions session-based text turns, voice turns, and non-session event interactions into a common shape with interaction type, channel, timestamps, and source evidence. | | Decision and Tool Execution Fact | Decision, state-transition, and tool-execution evidence. Captures tool name, success status, state transitions, and timing from both traced execution paths and world-event evidence. | | Conversation Outcome Fact | Outcome and handoff evidence attributable to conversations when source identity permits. Unions outcome events, current outcome projections, and resolved outcome projections. | | Call Intelligence Fact | Call-quality, safety, latency, and evaluation results. Merges call intelligence records with evaluation scores into a unified shape covering quality scores, duration, completion reason, emotion analysis, risk assessment, and tool execution summaries. | | Topic Classification Fact | Conversation topic classifications joined with taxonomy metadata. Includes primary and secondary category assignments, confidence, reasoning, taxonomy version, and model information. | | Appointment Workflow Fact | Appointment lifecycle, status, and workflow evidence. Unions appointment-specific events with general workflow evidence that carries appointment data. | | Event and Clinical Evidence Fact | Typed event, clinical, and review evidence records. Unions raw, current, and resolved event states with FHIR resource references, review status, and domain classification. | | Data Source and Model Health Fact | Connector health, event throughput, and metric freshness observations. Provides a unified view of ingestion recency, event volume, and metric computation timestamps. | | Metric Output | Existing metric values and freshness normalized for downstream consumers. Joins metric values with freshness metadata to expose computation timestamps, period boundaries, and value fields in a single row. | Each model includes a canonicalization timestamp indicating when the projection was last materialized. Models that join multiple sources use full or left joins to preserve rows even when one source lacks a matching record. ### Design Principles * **Environment-independent logic.** Source table references are resolved from configuration at runtime. The same transformation code runs across staging, production, and other catalog environments without modification. * **Union-then-join structure.** Models that combine multiple source streams first union rows from the same grain, then join across grains. This keeps the transformation predictable and avoids fan-out. * **Coalesce-based fallbacks.** When a preferred field may be null, models fall back to alternative fields using ordered preference. For example, display names prefer an explicit display name over a raw name, and timestamps prefer the most specific available timestamp. * **Workspace scoping.** Every canonical model is keyed by workspace, ensuring tenant isolation in all downstream queries. The metric store provides one value model for operational, quality, and evaluation metrics. Values carry a metric key, type, source class, scope identifiers, period, event count, optional confidence, unit, and computation time. The active Platform API catalog contains 41 built-in definitions across six categories. A catalog entry describes a metric the platform knows how to represent; it does not guarantee that every workspace or interaction will produce a value for that key. {% hint style="info" %} Metric values distinguish **production** and **simulation** sources and **aggregate** and **entity** scopes. Simulations compute only the metrics selected by their evaluation configuration; they do not automatically produce all active workspace metrics. {% endhint %} ## How Values Become Available Metric values can arrive through different paths: 1. The platform loads active built-in definitions and supported workspace custom definitions. 2. Eligible call-intelligence, surface, world-event, voice-quality, production-eval, or simulation-eval evidence is evaluated or aggregated. 3. Recent per-interaction values can become available through a low-latency projection, while scheduled analytical processing produces aggregate and AI-evaluated history. 4. The metric API combines recent values with durable history. Dashboard panels query durable analytical views. These paths can have different freshness. A definition update takes effect when the applicable execution path next processes eligible evidence. It does not promise atomic historical replacement, immediate recomputation, or backfill of earlier interactions. ## Developer Console The Developer Console **Quality** page currently includes: * The built-in **Realtime Metric Store** dashboard. * A production-eval quality panel. * Latest values and editable custom evaluation metrics. * The 41 built-in definitions, with an active or inactive toggle. The Console's custom-metric form creates conversation-summary `ai_query` definitions using the fast, balanced, or max model tier. The settings API exposes a broader definition schema than this form. Built-in definitions are read-only in the Console except for activation. Through the API, accepted built-in override fields are limited to activation, freshness target, hourly or daily period granularity, and numerical valid-range bounds. Built-in keys, types, sources, and extraction behavior are platform-owned. The settings response merges saved overrides into the built-in definitions, and the low-latency call projection reads the active workspace settings. The scheduled built-in aggregate pipeline remains code-defined, and the catalog endpoint returns the platform base catalog rather than saved built-in overrides. A toggle therefore does not delete history or guarantee that every scheduled aggregate stops being produced. ### Realtime Metric Store Dashboard The `metrics-overview` template refreshes every five minutes by default and contains: * **Metric store summary** - active metric count, numerical metric count, value points, scored events, average confidence, and latest computation. * **Average metric scores** - numerical metric averages. * **Daily scored events** - daily event volume, active metric count, and value points. * **Production versus simulation scores** - numerical values grouped by source. * **Categorical distribution** - event counts for categorical values. * **Entity-level coverage** - distinct entities and value counts by metric. * **Metric freshness** - last computation, latest covered period, value count, and observed minutes stale. * **Per-call metric scores** - recent entity-scoped values with source and period. Dashboard filters cover time window, source, aggregate or entity scope, and metric type. A panel can be empty when its source has not produced values or the analytical query path is unavailable. ## Built-In Catalog The 41 built-ins are grouped as follows: | Category | Count | Evidence | | ------------------------ | ----: | ----------------------------------------- | | Voice intelligence | 10 | Call-intelligence fields | | Surface intelligence | 5 | Surface lifecycle evidence | | Data quality | 3 | World-event fields | | Cross-channel | 2 | Call outcome and surface lifecycle events | | Standard quality | 11 | AI evaluation of conversation summaries | | Voice quality evaluation | 10 | Audio-native Voice Judge results | ### Voice Intelligence | Metric | Type | What It Measures | | --------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------- | | **Voice quality score** | Numerical | Average composite call-quality score from 0 to 100 | | **Voice duration** | Numerical | Average call duration in seconds | | **Voice escalation** | Numerical | Fraction of calls marked as escalated | | **Voice completion reason** | Categorical | Counts by canonical terminal reason | | **Voice risk level** | Categorical | Compatibility key. The current voice producer leaves the source field empty, so it does not provide a current risk distribution. | | **Voice average TTFB** | Numerical | Average audio time-to-first-byte in milliseconds | | **Voice tool failure rate** | Numerical | Average fraction of failed tool calls | | **Voice barge-in count** | Numerical | Average caller interruptions per call | | **Voice loop count** | Numerical | Average context-graph state revisits per call | | **Voice silence ratio** | Numerical | Average fraction of call duration classified as silence | These metrics use hourly periods and a near-realtime latency classification. They require a terminal call-intelligence artifact and the relevant source field. Missing source evidence produces no meaningful value; it should not be interpreted as zero. ### Surface Intelligence | Metric | Type | What It Measures | | ---------------------------- | ----------- | ---------------------------------------------------------- | | **Surface completion rate** | Numerical | Submitted surfaces divided by created surfaces | | **Surface open rate** | Numerical | Opened surfaces divided by delivered surfaces | | **Surface abandonment rate** | Numerical | Opened surfaces not submitted, relative to opened surfaces | | **Surface channel** | Categorical | Counts by observed delivery channel | | **Surface time to complete** | Numerical | Average hours from creation to submission | Surface metrics depend on the corresponding lifecycle events. Missing delivery or open events affect the available denominator and should be considered when comparing channels. ### Data Quality | Metric | Type | What It Measures | | --------------------------- | --------- | ---------------------------------------------------------- | | **Average data confidence** | Numerical | Average confidence on eligible current world events | | **Event volume** | Numerical | Count of eligible current world events | | **Review approval rate** | Numerical | Compatibility definition for earlier event-review statuses | The review-approval key has no active producer for the current external-write review workflow. Do not use it for present-day governance reporting. External write proposal decisions have a separate lifecycle. ### Cross-Channel | Metric | Type | What It Measures | | ------------------------- | --------- | -------------------------------------------------------------------------- | | **Patient contact count** | Numerical | Eligible call-outcome and surface-delivery contacts | | **Patient response rate** | Numerical | Eligible call outcomes and submitted surfaces relative to tracked contacts | These definitions cover the event families named above. They are not a universal count of every message or contact channel. ### Standard Quality Five definitions apply to every eligible service. Six outcome definitions apply only when the service carries the corresponding product-type tag. | Metric | Type | Scope | What It Evaluates | | ------------------------------ | --------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | **Patient sentiment** | Categorical | All eligible services | The patient's apparent attitude toward the interaction: positive, neutral, or negative | | **Conversational naturalness** | Numerical, 1-10 | All eligible services | Scriptedness, recovery, empathy, pacing, turn-taking, and related conversational qualities | | **Conciseness** | Numerical, 1-10 | All eligible services | Whether responses were appropriately brief without penalizing necessary detail | | **Information accuracy** | Boolean or null | All eligible services | Whether agent claims match tool responses. It does not verify that the tool data itself was correct. Null means there were no tool calls to compare. | | **Safety** | Categorical | All eligible services | `no_event`, `handled`, `warning`, or `critical` according to the evaluation rubric | | **Scheduling outcome** | Categorical | Scheduling | `escalated`, `resolved`, or `no_action` | | **Outbound outcome** | Categorical | Outbound | `escalated`, `resolved`, `opted_out`, `no_action`, or `no_answer` | | **Coaching outcome** | Categorical | Coaching | `escalated`, `resolved`, or `no_action` | | **Intake outcome** | Categorical | Intake | `escalated`, `resolved`, or `no_action` | | **Triage outcome** | Categorical | Triage | `escalated`, `resolved`, or `no_action` | | **Support outcome** | Categorical | Support | `escalated`, `resolved`, or `no_action` | Standard quality metrics use batch AI evaluation over eligible call-intelligence summaries. Their output is non-deterministic, can be incomplete, and does not replace authoritative record retrieval or human review. ### Voice Quality Evaluation Ten daily metrics aggregate the [Voice Judge](/intelligence-and-analytics/intelligence/call-intelligence.md#voice-judge) dimensions: latency and dead air, pronunciation, clarity, filler and silence, interruption handling, audio consistency, pacing, warmth and tone, accent quality, and voice identity. Values range from `0.0` to `1.0`. These metrics require a usable recording and a completed Voice Judge result. Recording or analysis failures can leave a call without these values. ## Custom Metrics The settings API accepts up to 50 custom definitions per workspace. Custom keys cannot reuse built-in keys, and custom definitions cannot claim built-in status. ### Paved Console Workflow The Console currently creates AI-query evaluation metrics against conversation summaries. Authors provide a name, rubric, result type, optional categories, and model tier. The transcript or conversation summary is supplied by the evaluation path. This is a model-based judge, not a deterministic rule engine. Test new rubrics against representative conversations and calibrate them with human reviewers before using the results for deployment or safety decisions. ### API Extraction Modes The definition schema contains five extraction modes, but their execution boundaries differ: | Mode | Current Boundary | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **`static`** | Reads a configured JSON path from matching world events. The current custom static aggregate path is suited to numerical values. | | **`ai_classify`** | Classifies selected event content into configured labels. Labels are required by the current execution path. | | **`ai_extract`** | Extracts configured fields from event content. The current execution path requires labels. | | **`ai_query`** | Runs a rubric against a call-intelligence summary or matching world-event data using an allowed managed tier or explicitly permitted custom model configuration. | | **`sql_expr`** | Reserved for platform definitions. The settings API rejects it for custom metrics. | The API accepts numerical, categorical, and boolean metric types. Not every type, aggregation, and extraction-mode combination has an active executor. Prefer the Console's AI-query workflow or a combination explicitly verified in the current API and pipeline before deploying a custom definition. ### Model Tiers | Tier | Current Use | | ------------ | ------------------------------------------------------------------------------------------------------ | | **Free** | Platform-managed classification and extraction modes; it is not valid for `ai_query` | | **Fast** | Managed AI-query tier for simpler evaluations | | **Balanced** | Managed AI-query tier used by the built-in standard quality judges | | **Max** | Managed AI-query tier for more complex evaluation | | **Custom** | API-only custom model configuration when explicitly permitted; not exposed by the current Console form | Tier names describe routing and intended complexity. They are not a latency, accuracy, or cost guarantee. ### Source and Channel Scoping Custom definitions accept matching event types and a channel scope of `all`, `voice`, `text`, `surface`, `inbound`, or `outbound`. `ai_query` treats `call_intelligence` as a conversation-summary source and other supported configurations as world-event evaluation. Current scheduled custom execution does not apply every channel-scope value consistently: the static custom path does not apply the field, AI event paths handle voice, text, and surface, and inbound or outbound scoping is not implemented across those paths. Use narrow event types as the reliable filter and verify the current executor before relying on channel scope. A broad wildcard can evaluate unrelated events, increase cost, and produce values that are difficult to interpret. ### No Generic Preview Endpoint The Platform API does not currently expose a generic, non-persisting **evaluate metric** endpoint. The embedded [production-eval operation](https://docs.amigo.ai/developer-guide/platform-api/safety/production-evals#evaluating-a-call) runs the workspace's active eval definitions for one completed conversation and persists the verdicts. It is not an inline preview of an arbitrary metric definition. ## Processing and Freshness ### Latency Classification | Tier | Intended Processing Class | | ----------------- | ------------------------------------------------------------- | | **Streaming** | A producer and projection capable of seconds-level processing | | **Near realtime** | Triggered or low-latency processing measured in minutes | | **Batch** | Scheduled processing, typically hourly or daily | `latency_tier` is definition metadata and processing intent. It does not itself create a streaming producer, runtime alert, or contractual delivery time. ### Period Granularity Definitions support hourly or daily periods. Built-in voice operational metrics use hourly periods. Standard quality and Voice Judge metrics use daily periods. Changing the setting affects future applicable processing; it does not guarantee an immediate rewrite of existing history. ### Freshness Target `freshness_sla_minutes` accepts values from 5 minutes to 24 hours and defaults to 60 minutes. It is a configured staleness target, not an externally enforced SLA. The Realtime Metric Store dashboard shows observed computation freshness from its analytical freshness view. There is no dedicated public metric-freshness endpoint in the current Platform API. ## Query Surface The public metric routes provide four reads: | Read | Current Behavior | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **List metric values** | Latest values by default, or history with `latest_only=false`; supports source, scope, entity, service, run, session, date, limit, and offset filters | | **Get one metric** | Values for a metric key with source, scope, entity, service, run, session, and date filters | | **Metric trend** | Up to 365 days for one metric key with the same scope filters | | **Metric catalog** | Active built-ins plus active custom definitions, including type, source, extraction mode, value granularity, period granularity, latency tier, model tier, unit, prompt presence, and built-in status | `GET` and `PUT` on metric settings manage the complete definitions list. Reads require authenticated workspace access; updates require admin or owner permission. The list is replaced when supplied, while built-ins are merged back with only supported overrides. A historical call route also returns recent per-call metric values for call detail. Use the unified Runs surface as the canonical interaction inventory, and use the general metric filters for run- and session-scoped analysis. ### Missing and Delayed Values Preserve these distinctions in downstream reporting: * A configured metric with no value can mean no eligible source evidence, pending processing, a failed evaluator, or an unavailable analytical dependency. * Numerical `0`, boolean `false`, categorical values, and null are different outcomes. * Recent API values can appear before the durable dashboard view catches up. * Production and simulation values are separate source classes and should not be combined unless the analysis explicitly requests `all`. * Per-entity values should not be treated as workspace aggregates without an explicit aggregation step. ## Production Evals and Metric Values Production eval definitions and metric definitions are related but distinct. A production eval can reference an active AI-query metric and persist both an eval verdict and a metric value for the evaluated conversation. Assertion evals have their own verdict semantics. Automatic eager evaluation is best-effort and feature-dependent. It can be skipped when disabled, when the conversation lacks a durable identifier, when capacity controls apply, or when an evaluator fails. Manual call evaluation persists results. Simulation runs evaluate only definitions captured and selected in the run configuration. See [Metrics and Quality](/testing/testing/metrics.md) for simulation assertions, score interpretation, and deployment-gate guidance. ## Billing Meters Are Separate Billing metering is a separate pipeline and API surface from the metric store. Its default lifecycle catalog currently includes: | Meter | Unit | Current Evidence | | ---------------------- | -------------- | --------------------------------------------------------------- | | Voice minutes | minutes | Reported voice-call duration | | Call count | calls | Voice-call events | | LLM input tokens | tokens | Emitted model-usage records | | LLM output tokens | tokens | Emitted model-usage records | | SMS messages | messages | SMS message events | | Call recording minutes | minutes | Reported recording duration | | Completed calls | calls | Selected completed call-intelligence outcomes | | Quality-weighted calls | quality points | Quality scores for those selected completed outcomes | | Surface submissions | submissions | Submitted surface events | | Message count | messages | Recorded agent engage-message events | | Action count | actions | Reported tool counts on eligible voice and companion executions | | Conversation count | conversations | Eligible call-session and companion-execution events | Meter rows are partitioned by production or simulation source when the emitter supplies that classification. The metering emit API can also introduce authorized custom meter keys. Meter projection is asynchronous, so an accepted emission is not proof that a billing rollup or invoice already contains it. {% hint style="info" %} **Developer Guide** - For the current metric routes, settings schema, filters, and production-eval distinction, see the [Metric Store developer guide](https://docs.amigo.ai/developer-guide/platform-api/safety/metric-store). {% endhint %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.amigo.ai/intelligence-and-analytics/metric-store.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.