Choosing a retrieval strategy by testing the workload
Compare search, structured queries, graphs and long context with representative questions.
Can the application find the right passage, respect the user’s permissions and answer in time? Those questions matter more than the name of the retrieval method. Compare approaches on the prospect’s own documents and questions, recording both useful results and failures.
Choose retrieval through evaluation
SC-MX-01 · REV A · 2026.07
01Task and corpus
questions and evidence
02Candidate methods
search / SQL / graph / context
03Evaluation
quality / access / freshness
04Operating choice
cost and remaining gaps
No universal numerical ranking of retrieval architectures is implied.
Build a useful evaluation set
Include exact identifiers, Japanese terminology, ambiguous questions, follow-ups, changed documents and questions with no answer in the corpus. Establish expected evidence and acceptable responses with domain owners. Measure retrieval separately from answer generation so a fluent response does not hide missing or incorrect evidence.
Compare direct search, lexical and semantic retrieval
Direct file or source search can be useful when the application has safe, authorised access and an effective search path. Keyword search is worth testing for identifiers and exact terms; semantic retrieval may help with paraphrases. Hybrid systems combine signals, but their benefit depends on the corpus and query set. Neither indexing nor live search guarantees freshness or complete coverage.
Add structure when the task needs it
Structured SQL can suit exact calculations over governed tables. Graph approaches can help explore relationships and summaries when the required entities and links are represented well. Microsoft’s GraphRAG research studies particular question-answering and summarisation settings; it does not establish that every multi-step question requires a graph or that an agent cannot solve one.
Test long context with a defined document set
For a limited corpus, supplying selected documents directly may be simpler than maintaining an index. Check context limits, permissions, cost, latency and how answer quality varies with length and evidence position. A large context window is a capacity limit, not a guarantee that all relevant evidence will be used correctly. Avoid sending an entire corpus without an access decision.
Tune chunking and ranking with evidence
Where documents are indexed, compare chunk boundaries, overlap and metadata on the evaluation set. Consider reranking a candidate set if its measured benefit justifies the cost. Query rewriting can help resolve conversational references but can also change the user’s meaning; test it. None of these components is mandatory for every system.
Test document updates, access changes and deletions
Map source updates, index refresh, caches and derived summaries. Confirm how revoked access and removed documents stop reaching answers. Review every external service in that path. Record measured quality, delay, cost and remaining gaps, then repeat the evaluation after changing the model, corpus or retrieval configuration.
// Key Takeaways
What to remember
- Compare methods on representative questions and expected evidence.
- Choose graphs, reranking or long context only when they improve the actual task.
- Test permission changes, updates and deletions across the complete path.
// FAQ
Frequently asked questions
Does every multi-step question require GraphRAG?
No. The data, available tools and evaluation determine the fit. A graph needs suitable entity and relationship extraction and has its own maintenance cost.
Should every retrieval system use hybrid search and reranking?
No. They are options to compare with a baseline. Measure evidence quality, latency, cost and failure cases on representative queries before adding them.
Last updated:
Demonstrate retrieval quality with a Japanese prospect’s documents.
Share the search methods, access controls and evaluation tools your product supports. StrategyCore can help organise a Japanese technical evaluation that compares relevant answers, permissions and response time against agreed business tasks.