KB Agent: Retrieval & Build Methodology

How to choose a retrieval architecture for a knowledge-base agent, and a measurement-driven process for building one.

This is the deep-dive. It is the depth behind step 3 of Build a Knowledge-Base Agent End to End. Start there for the full path (load data, assign roles, build, test); come here for how retrieval and the build loop work.

A knowledge-base (KB) agent answers questions over a body of documents. On every turn it has to find the document that actually contains the answer (and only that one), ground its reply in it, and say so when the KB has no answer instead of guessing.

This guide covers two things:

1. How to choose a retrieval architecture for your corpus, and why.
2. How to build the agent: a measurement-driven process that produces exactly the agent your tests require and nothing more.

The recommendations come from building KB agents on Amigo and align with published retrieval research.

TL;DR

1. For a small KB (roughly 800 documents or fewer), preload the whole catalog instead of doing RAG. The agent reads a complete index of titles and ids once, then fetches the exact documents it needs. This avoids any ranking step that could miss the right document, is cache-friendly, and lets the agent answer "not in the KB" honestly. It matches published guidance: when the corpus fits in context, put it in the prompt and rely on prompt caching.
2. When you outgrow the catalog, switch to hybrid search (BM25 + vector, fused), not pure vector and not pure keyword. Hybrid finds the right document most reliably, but it will always return something, so it has to ship with an explicit "decline when nothing is a good match" behavior.
3. Retrieval is the easy part; the corpus and the integration are the hard parts. Data quality is the single biggest lever, and an approach that looks best on retrieval alone can still do worse once it's inside the full agent.
4. Evaluate before you optimize. The most valuable thing you build is not the agent, it's the question set that turns every future change into a quick re-run instead of a guess.

Pick the retrieval architecture by corpus size

Corpus	Approach	Notes
~800 docs or fewer / index fits in context	Catalog-preload	The agent loads an index of titles + ids once, then fetches exact documents by id. Cache-friendly, no ranking step to lose a document, honest about misses.
Larger, or the index exceeds the result-size limit	Hybrid search (BM25 + vector, fused with RRF)	Beats pure vector on retrieval quality. Must ship with a decline-when-no-good-match behavior, because search always returns something.
Real scale	Hybrid + metadata/scope filtering + reranking	Filtering is the biggest precision lever at scale; reranking adds roughly 15% quality on top, at a latency cost.

Why catalog-preload works for a small KB

When the index of every document's title and id fits in the prompt, you don't need a retrieval pipeline at all. The agent sees the whole corpus, picks the documents it needs by title, and fetches them by id. There is no ranking step to lose the right document, no similarity threshold to tune, and the index is a stable prefix that prompt caching makes cheap. It is also honest about "no answer": if nothing in the catalog matches, the agent can say so, rather than being handed the nearest few documents and tempted to stretch.

In practice, catalog-preload matches search on finding the right document, does better at declining when the KB genuinely has no answer, and composes cleanly with the rest of the agent.

The catalog ceiling

The catalog has to physically reach the agent. A single tool result is truncated once it exceeds 128 KB: an oversized catalog is cut off, with a truncation notice, rather than delivered whole. Keep the index well under that. A title-and-id index of a few hundred documents fits with comfortable headroom; per-row summaries eat the budget fast. Two rules follow:

• Keep the catalog lean. Document id, title, page group, and scope tags are enough. Make the titles carry the selection signal; drop per-row summaries.
• Around 800 documents, switch to search. That is the trigger to move off catalog-preload.

When you do need search

Past the catalog ceiling, fuse keyword and semantic retrieval rather than choosing one. Use real BM25 for the keyword leg (term-frequency saturation, inverse document frequency, and length normalization), not a plain term-count, which ranks the wrong document first whenever a rare exact term matters (a fee amount, a policy number). Fuse the keyword and vector result lists with reciprocal rank fusion (RRF).

Better retrieval makes the honesty problem harder, not easier: search always returns its nearest documents, which tempts the agent to answer when it shouldn't. So any search-based agent needs an explicit "decline when nothing clears the similarity threshold" behavior before it goes live. At real scale, add metadata/scope filtering (the biggest precision lever) and then reranking.

Build it measurement-first: two principles

The build process rests on two principles. Everything else (the coverage matrix, the briefs, the iteration loop) follows from them.

Principle 1: start from a bare-bones agent; add complexity only when a test demands it

Your first version is the minimum agent that handles the first turn of the first conversation. Nothing more: no speculative behaviors, no preemptive guardrails, no conversation states for scenarios you haven't tested. Then run the tests, and for each turn that fails, apply the single smallest change that fixes it and re-run.

Behaviors, guardrails, states, and tools get pulled in by test failures, not pushed in by upfront planning. The agent ends with exactly the complexity its tests require. Every piece traces to a test, rollback is cheap because each version is a measurable artifact, and the agent ships in hours instead of weeks.

Principle 2: start from real conversations, then build the agent to satisfy them

Don't write the agent configuration first. Write the conversations first. A test conversation is a contract: this is what the user says, and this is how the agent must respond. The user message is fixed (the same every run); the expected behavior is captured as semantic judge criteria plus exact-match checks for any literal value that matters.

Once the conversations exist, the agent design isn't a separate step, it emerges from the iteration loop. Each failure points to exactly one thing to add:

What the failing test shows	What you add
The agent doesn't know a piece of session context	One context field
The agent can't fetch information the reply needs	One tool (name, args, return shape)
The agent needs structured per-conversation data	One entity enrichment field
The agent says the wrong thing across turns	One behavior
The agent violates a "must never"	One guardrail
The agent loses the topic or re-escalates when it shouldn't	One conversation-state transition

The conversations are the spec; the agent configuration is the implementation; you reverse-engineer it one piece at a time. There is no separate "design the agent" stage: the conversations are written upfront, the agent is not.

The build process, end to end

Stage 0: understand the requirements

Read the requirements and build a shared understanding of the user flows, guardrails, acceptance tests, capabilities, KB topics, and role/access scopes. This stage produces understanding, not configuration. Flag ambiguities now (names, escalation destinations, category values); they otherwise surface mid-build.

Stage 1: coverage matrix

Find the smallest set of conversations that exercises every requirement.

• 8 to 12 conversations. More becomes unmaintainable; fewer misses coverage.
• Boundary cases need a conversation on both sides. A scope-isolation rule needs a positive case (the in-scope user gets the content) and a negative case (the out-of-scope user is refused). An escalation rule needs a must-escalate case and a must-not-re-escalate case.
• Persona variety drives role-scope coverage. A single-persona set tests no role scoping. Writing a conversation for each role forces you to decide what that role can reach.
• Mark requirements you intentionally don't cover, so a gap is a decision rather than an oversight.

Stage 2: write a brief per conversation

For each conversation, write the persona (name, role, scopes, region), the per-turn context, two to five turn summaries, what it must prove, and the acceptance criteria (exact-match checks plus judge criteria).

The most important authoring rule: every judge criterion must be answerable from the reply text alone. The judge sees the user message and the agent's reply, not the persona, the context, or any scene metadata.

Instead of "Did the reply name the caller's assigned supervisor?", write "Did the reply name 'Dr. Alex Rivera' as the routing destination?" The verbatim expected value is verifiable from the reply; the persona reference is not. Getting this wrong undermines the whole judging step.

Stage 3: write the conversations

Turn the summaries into verbatim user messages, and make the phrasing realistic. Synthetic phrasing won't catch production failures.

• Sanitized: "What is the no-show fee policy?"
• Realistic: "Hey, quick one. Had a same-day no-show this morning, a client in my 10am slot. They messaged about 30 min before saying they had a family emergency. Do I still apply the standard no-show fee, or does the emergency waive it?"

Real messages have specific names, amounts, multi-sentence framing, and embedded follow-ups. That is what production traffic looks like. Stages 0 through 3 are the only stages that produce artifacts before you deploy anything.

The iteration loop

There is no "write the agent configuration" stage. Deploy the bare-bones agent: a name, a one-line description, one conversation state, no behaviors, no tools. It will fail almost every turn, and that's the point.

Then iterate. Run the test set; for each failing turn, ask "what is the smallest change that fixes this turn?", apply only that change, re-run, and save a baseline at each version so you can see regressions. Add the smallest pieces in rough order from mechanical to judgmental:

1. A tool (name, args, return shape inferred from how the reply uses the result)
2. A session context field
3. A guardrail (one binary "must never")
4. A constraint on phrasing in a specific state
5. A behavior (one cross-turn pattern)
6. A conversation-state or transition (most likely to interact with others, so add these last)

Add one piece per failure. Adding several at once loses track of which one fixed the turn.

Test the whole agent, not just retrieval

Good retrieval in isolation is not enough. The approach has to hold up inside the full agent, with all its safety behaviors, on a harder multi-turn test set. It is common for an approach that edges ahead on raw retrieval to fall behind once it's inside the full agent, where safety behaviors and multi-turn flow interact. That is exactly why you test the whole agent: the best retriever is a candidate, the agent you ship is whatever survives the full test set with safety intact.

The evaluation harness (the asset worth keeping)

The test set is the thing worth keeping. With it, any future change (a model upgrade, a prompt edit, a new retrieval mechanism) becomes a quick re-run against a fixed set of questions, not a leap of faith.

• A fixed question set with known-good answers and category coverage: exact-fact questions, multi-step workflows, paraphrase (the wording differs from the document), multi-document (the answer needs two or more docs), and no-answer (the KB genuinely can't answer; the agent must decline). The no-answer category is non-negotiable; it is how you measure honesty.
• Grade two ways. Mechanical document-selection metrics (did the agent fetch the right document?) and an LLM judge on answer quality (binary, verbatim-anchored criteria). They catch different failures.
• Verify your judge. A weak judge model silently corrupts the comparison. Spot-check it on a criterion you can verify by eye; if it's noisy, use a stronger judge model rather than tuning the prompt around the noise.
• Run it a few times and account for variance. Agents are non-deterministic. Run each version several times, accept only if every run clears the threshold, and use baseline diffs to catch regressions against the prior version.

Practical rules

1. Evaluate first. Build the question set before touching retrieval.
2. Grade two ways. Mechanical document-selection and a judge on answer quality.
3. Verify your judge. Use a stronger model if it's noisy; don't tune around the noise.
4. Refusal is a first-class behavior. Any approach that always returns documents needs an explicit decline. Better retrieval makes this more important, not less.
5. Put trigger conditions in tool descriptions. "Call this once before answering" works better than only describing what the tool does (the agent under-calls) or assuming a result is already present (the agent never calls it).
6. Prompt with goals and constraints, not scripts. Over-prescriptive prompts reduce output quality on current models. If you change the model, re-run the tests before editing any prompt, and never combine a model change and a prompt change in one version.
7. Test the full agent, not just retrieval.
8. The corpus is usually the bottleneck. When half your documents are empty or placeholder pages, no retrieval technique recovers the missing content. Fix data quality before chasing the last retrieval point.

Role-scoped access

If the KB is segmented by who may see what, scope retrieval to the user's role rather than building a separate agent per segment. One way to do this is a behavior that reads the caller's role from session context and only fetches in-scope documents. Registering roles, anchoring a conversation to the user's entity, and passing identity through to per-user tools is covered in step 2 of Build a Knowledge-Base Agent End to End, with the full reference in External Principals. The two guides are halves of one build: this one is how the agent finds and grounds answers, that one is how access is scoped.

What a finished small-KB agent looks like

A small-KB agent built this way stays deliberately simple, because the test set only ever pulls in what it needs. A typical end state:

• Retrieval: catalog-in-prompt plus fetch-by-id. No separate search tool. The catalog (id, scope, title) goes in the prompt, and the agent fetches documents by id.
• Tools: two. Fetch a document by id, and escalate a safety signal. No search tool and no extra integrations unless a test demanded them.
• Conversation states: three. A chat state (answer via catalog, then fetch, then cite), a safety_escalation state (emit only the escalation message, no tools), and an end state.
• A handful of behaviors, each added for a specific failing test: answer operational questions only, catalog-then-fetch retrieval, escalate safety signals immediately with no KB content, isolate scope at the catalog level, read the caller's role from context, refuse general-knowledge lookups, quote numbered procedures verbatim, prefer the canonical policy source, and don't mix different populations' policies in one answer.
• One communication pattern: a peer-to-peer register (the user is a professional; skip the "great question" preambles).

A plan written from requirements alone might call for eight tools, a dozen states, and a search-and-rerank pipeline. Built measurement-first, the same agent often collapses to a couple of tools and a few states, because the test set never demands the speculative pieces.

Guardrails tend to live on conversation states, not on the agent. It's natural to expect agent-wide guardrails, but in practice each guardrail attaches to a specific state (a safety check before responding in chat, "no KB content" in safety_escalation). Expect to place guardrails per state.

Platform notes

• A tool result is truncated past 128 KB. See The catalog ceiling above: keep the catalog lean and confirm the whole index reaches the agent.
• Refusal must be explicit for any search-based agent. Search always returns its nearest matches; without a decline-when-no-good-match behavior the agent will stretch to answer.
• Give each document a stable composite id, not the URL alone. Pages can legitimately share a source URL; keying identity on the URL alone collapses distinct documents into one. Derive the id from the source URL plus a per-document discriminator such as the filename.
• Cap the result set and the per-row excerpt in the query itself, so a broad query over a large corpus can't overflow the context window. Carry citation fields (source URL, document id, page group) on each row.
• Data quality is the real ceiling. Audit the corpus before optimizing retrieval. Empty or placeholder documents are invisible to every retrieval method.