How to Choose Your Retrieval Strategy

When an agent can traverse the source directly, a codebase or a well-structured file corpus, live agentic search wins: find the candidates, grep for the term, read what matters, repeat. There is no index to build, so there is no index to go stale, and no second copy of the customer's source sitting in another system waiting to be secured. That last point lands hard with Japanese security reviewers. If your product works on customer code or files and still ships a pre-built index, prepare a specific answer for why the index earns its keep.

Some questions require joining several facts before an answer exists at all: which supplier feeds which component in which affected product line, for example. For these compositional workloads, an explicit graph structure combined with hybrid retrieval wins clearly. An agent looping over flat search results does not close that gap, however many iterations you give it, because the relationships were never captured. If your product serves this workload, say so with a graph story. If it does not, say that too, and name the workloads where your simpler design is the better fit.

Most knowledge-base traffic is ordinary. Someone wants the policy, the procedure, the paragraph in the manual. For general or semi-structured lookup, a lean agentic approach, or plain retrieval with a single augmentation hop, usually covers it. Proposing a heavy graph platform here reads as over-engineering, and Japanese buyers notice cost that outruns the workload. Reserve the expensive machinery for the questions that need it, and be able to explain the boundary.

Embeddings compress meaning, and rare proper nouns are exactly what the compression loses. A part number, a person's name, an internal system code: pure vector search will miss these, and in Japan the corpus is full of them, from company registrations to ministry form numbers. The fix is old and unglamorous. Keep lexical keyword search in the mix, always, and blend it with semantic retrieval as a hybrid. A buyer who tests your demo with their own product codes will find this gap in minutes if you left it open.

If the corpus is small, bounded, and the use case tolerates latency, you can sometimes skip retrieval and place the whole thing in the model's context window. Do this with open eyes. Answer quality degrades on material buried deep in a long context, and feeding the model everything relevant does not convert one-to-one into correct answers. It is a legitimate design for contract review, a single manual, a bounded case file. Present it as a deliberate scoped choice, with its failure mode named, and it strengthens your credibility rather than spending it.

For regulated, permissioned data at scale, the deciding question in Japan is rarely recall quality. It is location. Run retrieval inside the residency boundary, beside the systems of record, honoring the permissions those systems already enforce. Shipping a second indexed copy of regulated data to an outside service creates a new data store the buyer's governance team must now defend, and many will simply decline. A vendor who arrives with an in-boundary retrieval architecture has answered the CISO's first question before it was asked.

When a retrieval deployment underperforms, the chunking is a likelier cause than the embedding model, and it is far cheaper to fix. Cut too small and a chunk loses the context that made it meaningful. Cut too large and the retrieved passage is mostly padding that dilutes the match. The uncomfortable part is that no universal best value exists: it depends on the corpus, and it has to be found by experiment. Overlap between adjacent chunks recovers the sentences a boundary would otherwise split. Ask a vendor what their chunk size is and why, and the answer tells you whether anyone measured it or whether a default shipped.

The production shape is two stages. A cheap retriever pulls a generous candidate set, twenty or fifty passages, optimising for not missing the right one. A reranker, slower and more accurate, then scores those candidates against the query and keeps the best few for the model. Running the expensive scoring across every document in the corpus would be unaffordable. Running it over fifty candidates is routine. A design with only one stage is trading recall against precision with no way to hold both.

Conversation breaks retrieval quietly. Someone asks about a supplier, then follows with what is their lead time, and the search runs on a query with no subject in it. Rewriting that follow-up into a self-contained question before it reaches the retriever fixes a whole class of failures that otherwise look like a broken index. The cost is one small model call. Japanese enterprise deployments hit this early, because the assistant patterns buyers ask for are conversational by design rather than one-shot search.