Run Both: The AI-Era Data Architecture That Keeps Your Systems of Record

The common approach to AI retrieval has been to embed an entire corpus into a vector database and search it by similarity. It works in a demo. At production scale it carries three costs that rarely appear in the first proof of concept. The index is a second copy of your data, sitting outside the system that governs it, with its own access controls to keep in sync. That copy drifts out of date the moment the source changes, so answers quietly reflect yesterday's records. And permission logic that was clean in the source becomes tangled once it has to be mirrored into a separate store. None of these are reasons to abandon retrieval. They are reasons to stop treating one retrieval method as the answer to every question.

As model context windows grow, it is tempting to skip retrieval and put the whole corpus in front of the model. The evidence does not support that as a general strategy. Studies through 2025 and 2026 show a consistent gap between what a system retrieves and what the model actually uses, with attention falling off sharply for material placed later in a long input. Beyond a bounded, latency-tolerant corpus, feeding everything into context degrades answer quality and raises cost per call. Retrieval stays faster and cheaper at scale. The useful reframing is to treat retrieval as one input to a broader context system rather than a fixed pipeline.

The strongest teams have stopped asking which retrieval method wins and started matching the method to the workload. For structured or code-like corpora that a system can navigate directly, live search over the source often beats a pre-built index, with nothing to keep fresh. When a question requires connecting several facts across documents, explicit graph structure still delivers large and consistent gains that simpler methods do not close. Where a lookup turns on exact names or rare identifiers, keyword search belongs in the mix, because pure similarity search misses them. For a small, bounded corpus, long context can be enough on its own. The decision rule matters more than any single tool.

The instinct to migrate off legacy databases in the name of AI is usually the wrong one. Systems of record run the business, hold the source of truth, and carry compliance history that took years to build. The better move is to keep them and expose them. Change data capture, streaming, and open table formats let an AI layer read fresh state from the operational systems without owning a stale second copy of it. That bridge is what prevents the staleness problem from returning at enterprise scale. The result is two planes that coexist: the systems you already trust, and a newer layer built for AI workloads, connected by a fresh flow of data between them.

For government bodies and regulated enterprises, where data can physically sit is not a detail. It shapes which retrieval design is even permissible. Running inference close to the data, inside the same governed boundary, keeps sensitive information from leaving the residency zone on every query. Live search that reads the source directly avoids creating an indexed second copy of regulated material at all. External managed vector services deserve the hardest scrutiny here, because they are precisely that off-boundary copy. In Japan and other jurisdictions with firm residency expectations, this is often the deciding factor rather than raw performance.

The honest conclusion is that this is a transition, and transitions are run with both stances active at once. Keep the operational databases. Add an AI-native layer beside them. Choose the retrieval method for each workload from a clear rule rather than the trend of the quarter, and keep the freedom to change it as models and costs move. Standardizing the decision process, rather than betting the estate on one product, is what lets an organization adopt AI quickly without stranding the systems that already work. For vendors, the lesson is symmetric: the buyers who move fastest are the ones shown how to add value without being asked to tear down what they depend on.

The reflex of copying all content into a vector index is fading as a universal default. The stronger pattern keeps operational systems of record in place and exposes them to an AI-native layer, choosing retrieval per workload. For a vendor entering Japan, matching that shift is what makes a pitch land.

It means running the operational stack and the AI-native stack together through the transition, rather than replacing one with the other. Systems of record stay authoritative, and change-data-capture and open table formats expose their data to AI without a wholesale migration.

Keeping data in its system of record lets a business hold regulated and sovereign data inside its residency boundary while still feeding AI. The AI layer reads from the source under the same controls, so residency and AI adoption stop competing.

No. Retrieval should be chosen per workload: some need vector search, some need a direct query against live operational data, some need a graph or keyword path. Defaulting every workload to one index adds cost and staleness without improving answers.