Metric Store

The current metric catalog, custom-definition boundaries, dashboard behavior, query surface, and freshness semantics.

Semantic Foundation

The Metric Store is built on a governed semantic layer that enforces consistency, traceability, and workspace-scoped security across all metric definitions and computations.

Canonical Models

Metrics are computed against a fixed set of canonical models that represent the platform's core analytical domains:

• Conversation fact - conversation and run lifecycle
• Interaction fact - user, agent, and system interactions
• Tool execution fact - decision, state-transition, and tool-execution evidence
• Conversation outcome fact - conversation-to-outcome relationships
• Evaluation fact - call-quality, safety, latency, and voice-judge evaluations
• Topic classification fact - conversation-topic classifications
• Appointment workflow fact - appointment, slot, workflow, and state transitions
• Entity dimension - workspace-scoped entity and identity relationships
• Event evidence fact - typed event, evidence, and review records
• Model health fact - connector, event-throughput, and metric-freshness health observations
• Surface interaction fact - workspace-scoped surface and form interaction events

Each canonical model declares its grain (for example, one conversation or one tool execution), its key columns, its timestamp columns, and an explicit freshness target. These declarations make the contract between raw platform data and computed metrics auditable.

Governed Relationships

Canonical models are connected through governed relationships that define how models can be joined. Each relationship declares its parent and child model, join keys, cardinality (such as one-to-many), and allowed join direction. Relationships are versioned and certified independently of the models they connect.

For example, a conversation can have many interactions, many tool executions, many outcomes, many topic classifications, many evaluations, and many surface interactions. An outcome can have many appointment workflow events. These relationships are enforced at definition time so that metric authors cannot construct unsupported joins.

Semantic Metric Definitions

A semantic metric definition wraps a standard metric definition with additional governance:

• Base model - which canonical model the metric is computed against
• Dimensions - the approved dimensional axes (workspace and reporting period are always required; service and channel are optional)
• Time field - which timestamp column drives the reporting period
• Owner - the responsible party for the metric
• Security classification - currently workspace-scoped, ensuring metrics never cross workspace boundaries
• Version and status - definitions move through draft, certified, and deprecated states
• Required relationships - which governed relationships the metric depends on
• Validation tests - references to tests that verify the metric's correctness

Definitions are compiled into deterministic execution plans before computation. The compilation step validates that the base model exists in the certified registry, that all required relationships are certified, and that the requested dimensions and time field are supported by the base model. The execution plan includes a content hash so that identical definitions produce identical plans.

Freshness Tracking

Each canonical model and metric definition declares an explicit freshness target in minutes. The platform classifies each materialized result as:

• Fresh - computed within the freshness target
• Stale - computed but older than the freshness target
• Never computed - no successful computation has occurred
• Failed - the most recent computation attempt failed

This classification drives the freshness indicators visible in dashboards and health monitoring.

Computation

Certified metric definitions are computed periodically against their canonical models. Each computation run:

1. Reads the certified execution plan for each active metric definition
2. Queries the canonical model within a configurable time window (up to 7 days)
3. Aggregates values by the declared dimensions using the plan's aggregation type (count, count distinct, sum, average, min, max, ratio, or rate)
4. Merges results into the metric values store using an upsert strategy keyed by workspace, metric, version, period, and dimension combination
5. Records the computation run's status, timing, and any error for observability

Computation is workspace-scoped: a metric definition owned by one workspace can only read data from that workspace's partition of the canonical model.

Setup and Refresh

The Metric Store provides two operational endpoints for workspace administrators:

• Setup triggers a one-time schema and registry initialization for the workspace. This is an idempotent operation - running it more than once has no adverse effect.
• Refresh triggers an on-demand refresh of the canonical Metric Store models. Use this when you need updated metrics before the next scheduled pipeline run.

Both operations are asynchronous. The endpoint returns a run identifier that you can poll for status. The run status reports lifecycle state, result state, and a human-readable message. Setup and refresh require the workspace update permission.

Each endpoint returns 202 Accepted with a run identifier. If the backing job is not yet configured for the environment, the endpoint returns 503 Service Unavailable.

Workspace Scoping

All canonical Metric Store models enforce workspace-level scoping. Rows without a valid workspace identifier are excluded during materialization, ensuring that every record in the serving layer is attributable to a single workspace.

Canonical Models

The Metric Store includes a set of canonical models that normalize data from multiple platform sources into consistent, workspace-scoped views. These models are materialized as governed projections and serve as the foundation for metrics, dashboards, and downstream analytics.

Canonical models use a pointer-driven design: physical source tables are resolved through configuration rather than hard-coded references, so environment changes do not require editing transformation logic.

Available Models

Model	Purpose
Entity Dimension	Workspace-scoped entity identity and governed dimensions. Joins entity attributes with identity mappings to produce a single row per entity with canonical identifiers, display names, service assignments, patient and provider references, facility assignments, and channel information.
Conversation Fact	Cross-channel conversation lifecycle at one row per workspace conversation. Merges run-level metadata with session and voice activity to produce start and end timestamps, status, run counts, and interaction counts.
Interaction Fact	Normalized text, voice, and world-event interactions. Unions session-based text turns, voice turns, and non-session event interactions into a common shape with interaction type, channel, timestamps, and source evidence.
Decision and Tool Execution Fact	Decision, state-transition, and tool-execution evidence. Captures tool name, success status, state transitions, and timing from both traced execution paths and world-event evidence.
Conversation Outcome Fact	Outcome and handoff evidence attributable to conversations when source identity permits. Unions outcome events, current outcome projections, and resolved outcome projections.
Call Intelligence Fact	Call-quality, safety, latency, and evaluation results. Merges call intelligence records with evaluation scores into a unified shape covering quality scores, duration, completion reason, emotion analysis, risk assessment, and tool execution summaries.
Topic Classification Fact	Conversation topic classifications joined with taxonomy metadata. Includes primary and secondary category assignments, confidence, reasoning, taxonomy version, and model information.
Appointment Workflow Fact	Appointment lifecycle, status, and workflow evidence. Unions appointment-specific events with general workflow evidence that carries appointment data.
Event and Clinical Evidence Fact	Typed event, clinical, and review evidence records. Unions raw, current, and resolved event states with FHIR resource references, review status, and domain classification.
Data Source and Model Health Fact	Connector health, event throughput, and metric freshness observations. Provides a unified view of ingestion recency, event volume, and metric computation timestamps.
Metric Output	Existing metric values and freshness normalized for downstream consumers. Joins metric values with freshness metadata to expose computation timestamps, period boundaries, and value fields in a single row.

Each model includes a canonicalization timestamp indicating when the projection was last materialized. Models that join multiple sources use full or left joins to preserve rows even when one source lacks a matching record.

Design Principles

• Environment-independent logic. Source table references are resolved from configuration at runtime. The same transformation code runs across staging, production, and other catalog environments without modification.
• Union-then-join structure. Models that combine multiple source streams first union rows from the same grain, then join across grains. This keeps the transformation predictable and avoids fan-out.
• Coalesce-based fallbacks. When a preferred field may be null, models fall back to alternative fields using ordered preference. For example, display names prefer an explicit display name over a raw name, and timestamps prefer the most specific available timestamp.
• Workspace scoping. Every canonical model is keyed by workspace, ensuring tenant isolation in all downstream queries.

The metric store provides one value model for operational, quality, and evaluation metrics. Values carry a metric key, type, source class, scope identifiers, period, event count, optional confidence, unit, and computation time.

The active Platform API catalog contains 41 built-in definitions across six categories. A catalog entry describes a metric the platform knows how to represent; it does not guarantee that every workspace or interaction will produce a value for that key.

Metric values distinguish production and simulation sources and aggregate and entity scopes. Simulations compute only the metrics selected by their evaluation configuration; they do not automatically produce all active workspace metrics.

How Values Become Available

Metric values can arrive through different paths:

1. The platform loads active built-in definitions and supported workspace custom definitions.
2. Eligible call-intelligence, surface, world-event, voice-quality, production-eval, or simulation-eval evidence is evaluated or aggregated.
3. Recent per-interaction values can become available through a low-latency projection, while scheduled analytical processing produces aggregate and AI-evaluated history.
4. The metric API combines recent values with durable history. Dashboard panels query durable analytical views.

These paths can have different freshness. A definition update takes effect when the applicable execution path next processes eligible evidence. It does not promise atomic historical replacement, immediate recomputation, or backfill of earlier interactions.

Developer Console

The Developer Console Quality page currently includes:

• The built-in Realtime Metric Store dashboard.
• A production-eval quality panel.
• Latest values and editable custom evaluation metrics.
• The 41 built-in definitions, with an active or inactive toggle.

The Console's custom-metric form creates conversation-summary ai_query definitions using the fast, balanced, or max model tier. The settings API exposes a broader definition schema than this form.

Built-in definitions are read-only in the Console except for activation. Through the API, accepted built-in override fields are limited to activation, freshness target, hourly or daily period granularity, and numerical valid-range bounds. Built-in keys, types, sources, and extraction behavior are platform-owned.

The settings response merges saved overrides into the built-in definitions, and the low-latency call projection reads the active workspace settings. The scheduled built-in aggregate pipeline remains code-defined, and the catalog endpoint returns the platform base catalog rather than saved built-in overrides. A toggle therefore does not delete history or guarantee that every scheduled aggregate stops being produced.

Realtime Metric Store Dashboard

The metrics-overview template refreshes every five minutes by default and contains:

• Metric store summary - active metric count, numerical metric count, value points, scored events, average confidence, and latest computation.
• Average metric scores - numerical metric averages.
• Daily scored events - daily event volume, active metric count, and value points.
• Production versus simulation scores - numerical values grouped by source.
• Categorical distribution - event counts for categorical values.
• Entity-level coverage - distinct entities and value counts by metric.
• Metric freshness - last computation, latest covered period, value count, and observed minutes stale.
• Per-call metric scores - recent entity-scoped values with source and period.

Dashboard filters cover time window, source, aggregate or entity scope, and metric type. A panel can be empty when its source has not produced values or the analytical query path is unavailable.

Built-In Catalog

The 41 built-ins are grouped as follows:

Category	Count	Evidence
Voice intelligence	10	Call-intelligence fields
Surface intelligence	5	Surface lifecycle evidence
Data quality	3	World-event fields
Cross-channel	2	Call outcome and surface lifecycle events
Standard quality	11	AI evaluation of conversation summaries
Voice quality evaluation	10	Audio-native Voice Judge results

Voice Intelligence

Metric	Type	What It Measures
Voice quality score	Numerical	Average composite call-quality score from 0 to 100
Voice duration	Numerical	Average call duration in seconds
Voice escalation	Numerical	Fraction of calls marked as escalated
Voice completion reason	Categorical	Counts by canonical terminal reason
Voice risk level	Categorical	Compatibility key. The current voice producer leaves the source field empty, so it does not provide a current risk distribution.
Voice average TTFB	Numerical	Average audio time-to-first-byte in milliseconds
Voice tool failure rate	Numerical	Average fraction of failed tool calls
Voice barge-in count	Numerical	Average caller interruptions per call
Voice loop count	Numerical	Average context-graph state revisits per call
Voice silence ratio	Numerical	Average fraction of call duration classified as silence

These metrics use hourly periods and a near-realtime latency classification. They require a terminal call-intelligence artifact and the relevant source field. Missing source evidence produces no meaningful value; it should not be interpreted as zero.

Surface Intelligence

Metric	Type	What It Measures
Surface completion rate	Numerical	Submitted surfaces divided by created surfaces
Surface open rate	Numerical	Opened surfaces divided by delivered surfaces
Surface abandonment rate	Numerical	Opened surfaces not submitted, relative to opened surfaces
Surface channel	Categorical	Counts by observed delivery channel
Surface time to complete	Numerical	Average hours from creation to submission

Surface metrics depend on the corresponding lifecycle events. Missing delivery or open events affect the available denominator and should be considered when comparing channels.

Data Quality

Metric	Type	What It Measures
Average data confidence	Numerical	Average confidence on eligible current world events
Event volume	Numerical	Count of eligible current world events
Review approval rate	Numerical	Compatibility definition for earlier event-review statuses

The review-approval key has no active producer for the current external-write review workflow. Do not use it for present-day governance reporting. External write proposal decisions have a separate lifecycle.

Cross-Channel

Metric	Type	What It Measures
Patient contact count	Numerical	Eligible call-outcome and surface-delivery contacts
Patient response rate	Numerical	Eligible call outcomes and submitted surfaces relative to tracked contacts

These definitions cover the event families named above. They are not a universal count of every message or contact channel.

Standard Quality

Five definitions apply to every eligible service. Six outcome definitions apply only when the service carries the corresponding product-type tag.

Metric	Type	Scope	What It Evaluates
Patient sentiment	Categorical	All eligible services	The patient's apparent attitude toward the interaction: positive, neutral, or negative
Conversational naturalness	Numerical, 1-10	All eligible services	Scriptedness, recovery, empathy, pacing, turn-taking, and related conversational qualities
Conciseness	Numerical, 1-10	All eligible services	Whether responses were appropriately brief without penalizing necessary detail
Information accuracy	Boolean or null	All eligible services	Whether agent claims match tool responses. It does not verify that the tool data itself was correct. Null means there were no tool calls to compare.
Safety	Categorical	All eligible services	no_event, handled, warning, or critical according to the evaluation rubric
Scheduling outcome	Categorical	Scheduling	escalated, resolved, or no_action
Outbound outcome	Categorical	Outbound	escalated, resolved, opted_out, no_action, or no_answer
Coaching outcome	Categorical	Coaching	escalated, resolved, or no_action
Intake outcome	Categorical	Intake	escalated, resolved, or no_action
Triage outcome	Categorical	Triage	escalated, resolved, or no_action
Support outcome	Categorical	Support	escalated, resolved, or no_action

Standard quality metrics use batch AI evaluation over eligible call-intelligence summaries. Their output is non-deterministic, can be incomplete, and does not replace authoritative record retrieval or human review.

Voice Quality Evaluation

Ten daily metrics aggregate the Voice Judge dimensions: latency and dead air, pronunciation, clarity, filler and silence, interruption handling, audio consistency, pacing, warmth and tone, accent quality, and voice identity. Values range from 0.0 to 1.0.

These metrics require a usable recording and a completed Voice Judge result. Recording or analysis failures can leave a call without these values.

Custom Metrics

The settings API accepts up to 50 custom definitions per workspace. Custom keys cannot reuse built-in keys, and custom definitions cannot claim built-in status.

Paved Console Workflow

The Console currently creates AI-query evaluation metrics against conversation summaries. Authors provide a name, rubric, result type, optional categories, and model tier. The transcript or conversation summary is supplied by the evaluation path.

This is a model-based judge, not a deterministic rule engine. Test new rubrics against representative conversations and calibrate them with human reviewers before using the results for deployment or safety decisions.

API Extraction Modes

The definition schema contains five extraction modes, but their execution boundaries differ:

Mode	Current Boundary
static	Reads a configured JSON path from matching world events. The current custom static aggregate path is suited to numerical values.
ai_classify	Classifies selected event content into configured labels. Labels are required by the current execution path.
ai_extract	Extracts configured fields from event content. The current execution path requires labels.
ai_query	Runs a rubric against a call-intelligence summary or matching world-event data using an allowed managed tier or explicitly permitted custom model configuration.
sql_expr	Reserved for platform definitions. The settings API rejects it for custom metrics.

The API accepts numerical, categorical, and boolean metric types. Not every type, aggregation, and extraction-mode combination has an active executor. Prefer the Console's AI-query workflow or a combination explicitly verified in the current API and pipeline before deploying a custom definition.

Model Tiers

Tier	Current Use
Free	Platform-managed classification and extraction modes; it is not valid for ai_query
Fast	Managed AI-query tier for simpler evaluations
Balanced	Managed AI-query tier used by the built-in standard quality judges
Max	Managed AI-query tier for more complex evaluation
Custom	API-only custom model configuration when explicitly permitted; not exposed by the current Console form

Tier names describe routing and intended complexity. They are not a latency, accuracy, or cost guarantee.

Source and Channel Scoping

Custom definitions accept matching event types and a channel scope of all, voice, text, surface, inbound, or outbound. ai_query treats call_intelligence as a conversation-summary source and other supported configurations as world-event evaluation.

Current scheduled custom execution does not apply every channel-scope value consistently: the static custom path does not apply the field, AI event paths handle voice, text, and surface, and inbound or outbound scoping is not implemented across those paths. Use narrow event types as the reliable filter and verify the current executor before relying on channel scope. A broad wildcard can evaluate unrelated events, increase cost, and produce values that are difficult to interpret.

No Generic Preview Endpoint

The Platform API does not currently expose a generic, non-persisting evaluate metric endpoint. The embedded production-eval operation runs the workspace's active eval definitions for one completed conversation and persists the verdicts. It is not an inline preview of an arbitrary metric definition.

Processing and Freshness

Latency Classification

Tier	Intended Processing Class
Streaming	A producer and projection capable of seconds-level processing
Near realtime	Triggered or low-latency processing measured in minutes
Batch	Scheduled processing, typically hourly or daily

latency_tier is definition metadata and processing intent. It does not itself create a streaming producer, runtime alert, or contractual delivery time.

Period Granularity

Definitions support hourly or daily periods. Built-in voice operational metrics use hourly periods. Standard quality and Voice Judge metrics use daily periods. Changing the setting affects future applicable processing; it does not guarantee an immediate rewrite of existing history.

Freshness Target

freshness_sla_minutes accepts values from 5 minutes to 24 hours and defaults to 60 minutes. It is a configured staleness target, not an externally enforced SLA.

The Realtime Metric Store dashboard shows observed computation freshness from its analytical freshness view. There is no dedicated public metric-freshness endpoint in the current Platform API.

Query Surface

The public metric routes provide four reads:

Read	Current Behavior
List metric values	Latest values by default, or history with latest_only=false; supports source, scope, entity, service, run, session, date, limit, and offset filters
Get one metric	Values for a metric key with source, scope, entity, service, run, session, and date filters
Metric trend	Up to 365 days for one metric key with the same scope filters
Metric catalog	Active built-ins plus active custom definitions, including type, source, extraction mode, value granularity, period granularity, latency tier, model tier, unit, prompt presence, and built-in status

GET and PUT on metric settings manage the complete definitions list. Reads require authenticated workspace access; updates require admin or owner permission. The list is replaced when supplied, while built-ins are merged back with only supported overrides.

A historical call route also returns recent per-call metric values for call detail. Use the unified Runs surface as the canonical interaction inventory, and use the general metric filters for run- and session-scoped analysis.

Missing and Delayed Values

Preserve these distinctions in downstream reporting:

• A configured metric with no value can mean no eligible source evidence, pending processing, a failed evaluator, or an unavailable analytical dependency.
• Numerical 0, boolean false, categorical values, and null are different outcomes.
• Recent API values can appear before the durable dashboard view catches up.
• Production and simulation values are separate source classes and should not be combined unless the analysis explicitly requests all.
• Per-entity values should not be treated as workspace aggregates without an explicit aggregation step.

Production Evals and Metric Values

Production eval definitions and metric definitions are related but distinct. A production eval can reference an active AI-query metric and persist both an eval verdict and a metric value for the evaluated conversation. Assertion evals have their own verdict semantics.

Automatic eager evaluation is best-effort and feature-dependent. It can be skipped when disabled, when the conversation lacks a durable identifier, when capacity controls apply, or when an evaluator fails. Manual call evaluation persists results. Simulation runs evaluate only definitions captured and selected in the run configuration.

See Metrics and Quality for simulation assertions, score interpretation, and deployment-gate guidance.

Billing Meters Are Separate

Billing metering is a separate pipeline and API surface from the metric store. Its default lifecycle catalog currently includes:

Meter	Unit	Current Evidence
Voice minutes	minutes	Reported voice-call duration
Call count	calls	Voice-call events
LLM input tokens	tokens	Emitted model-usage records
LLM output tokens	tokens	Emitted model-usage records
SMS messages	messages	SMS message events
Call recording minutes	minutes	Reported recording duration
Completed calls	calls	Selected completed call-intelligence outcomes
Quality-weighted calls	quality points	Quality scores for those selected completed outcomes
Surface submissions	submissions	Submitted surface events
Message count	messages	Recorded agent engage-message events
Action count	actions	Reported tool counts on eligible voice and companion executions
Conversation count	conversations	Eligible call-session and companion-execution events

Meter rows are partitioned by production or simulation source when the emitter supplies that classification. The metering emit API can also introduce authorized custom meter keys. Meter projection is asynchronous, so an accepted emission is not proof that a billing rollup or invoice already contains it.

Developer Guide - For the current metric routes, settings schema, filters, and production-eval distinction, see the Metric Store developer guide.