Sovereign vs Hosted LLMs: When to Self-Host in Japan

Teams often treat hosted-versus-self-hosted as an identity choice, but at any real scale it is a routing decision made workload by workload. A public-facing marketing assistant and a system that reasons over patient records have different requirements on every axis that matters, so sending both to the same place is a compromise for one of them. The mature pattern is a router that sends each request to the model that fits it, hosted or self-hosted, and treats the two as tiers of one system.

Five axes usually settle the choice. Data sensitivity: whether the prompt can legally and safely leave the customer's environment. Cost profile: how spend behaves as volume grows. Latency and availability: whether you can tolerate a network hop and a third party's uptime. Control: whether you need the model to be un-revocable and unchanging. And capability: whether a frontier hosted model is meaningfully better for the task than the best open-weight model you can run. Most workloads are decided by the first two; the rest break ties.

The cost profiles are shaped differently. A hosted API is pure usage-based operating cost: nearly free to start, and it scales linearly with every token you send, forever. Self-hosting is closer to a fixed cost: you provision GPUs whether they are busy or idle, so it is expensive at low volume and cheap per token at high sustained volume. The crossover is real and worth calculating. A bursty, low-volume, experimental workload is usually cheaper hosted. A steady, high-volume production workload is often dramatically cheaper self-hosted, especially once GPU utilisation is optimised.

For most Japanese enterprises, data sensitivity is the axis that decides first, because it is a legal constraint rather than an optimisation. Classify workloads by the data they touch. Non-sensitive, public, or already-published content can route to a hosted frontier model for its capability. Regulated, personal, or confidential data routes to a self-hosted open-weight model, where it never leaves the environment. A classifier and a router at the front of the system apply this automatically, so a developer does not decide residency per feature and the policy is enforced consistently.

Put together, the framework for a Japanese enterprise is straightforward. Start every workload with a data-sensitivity classification, because that sets the hard floor. Route regulated data to a sovereign, self-hosted model on-premise or in an in-country cloud. Route non-sensitive, spiky, or experimental work to a hosted model for speed and capability. Recalculate the cost crossover as production volume grows, because a workload that started hosted can become cheaper self-hosted once it is steady. The result is a hybrid by design, which is what almost every serious Japanese AI deployment converges on.

It depends on the workload, and most enterprises do both. Route by data sensitivity first: regulated or personal data to a self-hosted open-weight model, non-sensitive work to a hosted API. Then weigh cost, latency, and control. Steady high-volume production often favours self-hosting; bursty experimentation favours hosted.

When volume is high and steady. A hosted API costs per token forever, so linear scaling adds up; self-hosting is a fixed GPU cost that is expensive at low volume and cheap per token at high sustained volume, especially with optimised utilisation. Calculate the crossover for your actual production volume.

A sovereign LLM is an open-weight model you run on infrastructure you control, so your data never leaves and no vendor can revoke, throttle, or change it. Regulated Japanese enterprises choose it to keep personal and confidential data inside the residency boundary and to remove dependency on an external provider.

Yes, and it is the common mature pattern. A router at the front of the system classifies each request by data sensitivity and sends it to the model that fits: hosted for non-sensitive capability, self-hosted for regulated data, applying governance consistently across both.

It often decides it. Under the APPI, sending regulated or personal data to a hosted model's servers abroad can breach residency rules, so that data must route to a self-hosted model in-country. Non-sensitive data has no such constraint and can use a hosted model freely.