> For the complete documentation index, see [llms.txt](https://docs.amigo.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.amigo.ai/platform-overview/cost-and-latency.md). # Cost and Latency Optimization ## Cost to Serve Visibility The platform provides workspace-level cost-to-serve data through two separate views, each covering a distinct cost domain: * **LLM inference cost.** The workspace's own token usage, priced per call against tiered vendor rates. Because the tokens are metered directly to the workspace, this figure is near-exact. Each model's input, output, and cached token counts are reported alongside the dollar cost. Models that lack a rate-card entry for a given month are surfaced with their token counts but no dollar amount, and the response flags that the total understates actual spend. * **Infrastructure cost.** The workspace's usage-weighted share of pooled compute and storage resources. Pipelines, warehouse, storage, and other components are shared across workspaces, so the figure is an apportionment rather than a metered quantity - a weaker claim than the LLM inference number. The two are kept as separate endpoints so neither figure inherits the other's caveats. Both views are sourced from a monthly cost-allocation close that reconciles the full bill to zero residual. Data is available after the close job completes (typically within a few days of month end). Account-level overhead that cannot be attributed to any single workspace is excluded. Cost-to-serve data is restricted to Amigo staff and is not available to workspace administrators. These figures represent what Amigo pays to serve a workspace - they are not amounts owed and not contracted prices. For endpoint details, see the [Cost to Serve endpoints](https://docs.amigo.ai/developer-guide/platform-api/cost-to-serve) in the Developer Guide. Running LLM-powered agents at scale requires deliberate engineering to control both cost and latency. Amigo's in-house conversation runtime uses a stable prompt prefix, per-task model routing, and a latency budget for real-time responses. The exact optimization path varies for native framework agents, provider-owned speech-to-speech sessions, and other runtimes; voice has the tightest latency constraints. ## Stable-Prefix Prompt Caching The in-house conversation runtime builds prompts in two parts: * **Static prefix** - Session-constant instructions such as the agent identity, service description, capabilities, context-graph topology, and output format. * **Dynamic suffix** - Per-turn state, objectives, active guidance, current time, and bounded conversation history. When the selected model provider supports prompt-prefix caching, sending the stable content first lets the provider reuse work across turns with the same prefix. The dynamic suffix still changes as the conversation advances, and conversation history remains part of that per-turn content. This split reduces repeated processing without changing what the model receives. A provider that does not support compatible prefix caching can process the complete prompt normally; correctness does not depend on a cache hit. ## Per-Task Model Routing Different parts of the pipeline use different models based on what the task demands: | Task | Model Selection | Why | | ------------------------- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Navigation** | Separately configured low-latency model | Produces a small, structured routing decision on each turn | | **Response generation** | Separately configured engagement model | Produces the caller-facing response, where language quality matters most | | **Filler speech** | No separate model call | Short time-buying phrases come from prepared templates, while context-aware fillers are produced inline by the same model that generates the response - so no extra model round trip is added | | **Post-call data review** | Separately configured offline processing | Runs outside the live turn path, so its latency and model choice can differ from conversation generation | | **Metric evaluation** | Definition- and pipeline-specific model tier | Eligible evaluations can run eagerly or in scheduled processing without blocking the live conversation | Separate model roles let teams and platform defaults balance latency, quality, and cost by task. Actual usage depends on the resolved service configuration, runtime, fallbacks, and enabled evaluation paths. ## Pipeline Latency Budget For voice interactions, the pipeline targets sub-300ms speech-to-text latency. End-to-end response time varies with the selected models, speech provider, tools, and network path. Three mechanisms reduce the delay callers perceive: **Filler speech** covers the gap between when the caller finishes speaking and when the agent's full response is ready. The caller hears "Let me check on that" or a similar phrase while the LLM generates the real response. **Prompt caching** can reduce repeated input processing when the selected provider supports it. The session-constant prefix stays stable while the current state and conversation history remain in the dynamic suffix. **Two-phase initialization** can prepare the agent during ring time on supported conference-based voice paths. It loads available context and can begin generating the greeting before answer. Provider-side prompt caching and preparation can still miss, so this reduces expected delay rather than guaranteeing instant audio. ## Usage Visibility The Platform API exposes workspace-scoped billing reads for current-period usage, usage trends, invoice lists, and invoice detail. Values depend on the meters that producers emitted and on asynchronous projection, so an accepted interaction is not proof that every billing rollup has already refreshed. Teams can use those API responses for their own reporting and reconciliation. Invoice detail includes the line items generated for that billing period, with meter key, quantity, unit price, and amount; the actual catalog depends on configured prices and emitted usage rather than a fixed list of channel examples. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.amigo.ai/platform-overview/cost-and-latency.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.