Cost and Latency Optimization

How prompt caching, model routing, and pipeline design keep costs low and responses fast.

Cost to Serve Visibility

The platform provides workspace-level cost-to-serve data through two separate views, each covering a distinct cost domain:

• LLM inference cost. The workspace's own token usage, priced per call against tiered vendor rates. Because the tokens are metered directly to the workspace, this figure is near-exact. Each model's input, output, and cached token counts are reported alongside the dollar cost. Models that lack a rate-card entry for a given month are surfaced with their token counts but no dollar amount, and the response flags that the total understates actual spend.
• Infrastructure cost. The workspace's usage-weighted share of pooled compute and storage resources. Pipelines, warehouse, storage, and other components are shared across workspaces, so the figure is an apportionment rather than a metered quantity - a weaker claim than the LLM inference number. The two are kept as separate endpoints so neither figure inherits the other's caveats.

Both views are sourced from a monthly cost-allocation close that reconciles the full bill to zero residual. Data is available after the close job completes (typically within a few days of month end). Account-level overhead that cannot be attributed to any single workspace is excluded.

Cost-to-serve data is restricted to Amigo staff and is not available to workspace administrators. These figures represent what Amigo pays to serve a workspace - they are not amounts owed and not contracted prices.

For endpoint details, see the Cost to Serve endpoints in the Developer Guide.

Running LLM-powered agents at scale requires deliberate engineering to control both cost and latency. Amigo's in-house conversation runtime uses a stable prompt prefix, per-task model routing, and a latency budget for real-time responses. The exact optimization path varies for native framework agents, provider-owned speech-to-speech sessions, and other runtimes; voice has the tightest latency constraints.

Stable-Prefix Prompt Caching

The in-house conversation runtime builds prompts in two parts:

• Static prefix - Session-constant instructions such as the agent identity, service description, capabilities, context-graph topology, and output format.
• Dynamic suffix - Per-turn state, objectives, active guidance, current time, and bounded conversation history.

When the selected model provider supports prompt-prefix caching, sending the stable content first lets the provider reuse work across turns with the same prefix. The dynamic suffix still changes as the conversation advances, and conversation history remains part of that per-turn content.

This split reduces repeated processing without changing what the model receives. A provider that does not support compatible prefix caching can process the complete prompt normally; correctness does not depend on a cache hit.

Per-Task Model Routing

Different parts of the pipeline use different models based on what the task demands:

Task	Model Selection	Why
Navigation	Separately configured low-latency model	Produces a small, structured routing decision on each turn
Response generation	Separately configured engagement model	Produces the caller-facing response, where language quality matters most
Filler speech	No separate model call	Short time-buying phrases come from prepared templates, while context-aware fillers are produced inline by the same model that generates the response - so no extra model round trip is added
Post-call data review	Separately configured offline processing	Runs outside the live turn path, so its latency and model choice can differ from conversation generation
Metric evaluation	Definition- and pipeline-specific model tier	Eligible evaluations can run eagerly or in scheduled processing without blocking the live conversation

Separate model roles let teams and platform defaults balance latency, quality, and cost by task. Actual usage depends on the resolved service configuration, runtime, fallbacks, and enabled evaluation paths.

Pipeline Latency Budget

For voice interactions, the pipeline targets sub-300ms speech-to-text latency. End-to-end response time varies with the selected models, speech provider, tools, and network path. Three mechanisms reduce the delay callers perceive:

Filler speech covers the gap between when the caller finishes speaking and when the agent's full response is ready. The caller hears "Let me check on that" or a similar phrase while the LLM generates the real response.

Prompt caching can reduce repeated input processing when the selected provider supports it. The session-constant prefix stays stable while the current state and conversation history remain in the dynamic suffix.

Two-phase initialization can prepare the agent during ring time on supported conference-based voice paths. It loads available context and can begin generating the greeting before answer. Provider-side prompt caching and preparation can still miss, so this reduces expected delay rather than guaranteeing instant audio.

Usage Visibility

The Platform API exposes workspace-scoped billing reads for current-period usage, usage trends, invoice lists, and invoice detail. Values depend on the meters that producers emitted and on asynchronous projection, so an accepted interaction is not proof that every billing rollup has already refreshed.

Teams can use those API responses for their own reporting and reconciliation. Invoice detail includes the line items generated for that billing period, with meter key, quantity, unit price, and amount; the actual catalog depends on configured prices and emitted usage rather than a fixed list of channel examples.