Running an Open-Weight LLM On-Premise in Japan

The reason to self-host is that the data and the model stay in one environment you control. For regulated Japanese data, a prompt that never leaves the building cannot breach a residency rule, and a model whose weights you hold cannot be revoked, throttled, or changed under you. That is the whole value of a sovereign deployment. Everything that follows is the engineering required to earn it, and it is more approachable than it first appears because the open-weight ecosystem has matured fast.

The first practical question is how much GPU memory the model needs. As a rough guide, a model's weights in full precision take about two bytes per parameter, so a model with tens of billions of parameters needs tens of gigabytes of GPU memory just to load, before the working memory each request adds. Quantisation, storing the weights at lower precision such as eight or four bits, cuts that requirement substantially with a small quality trade, and it is what lets a capable model run on a single accessible GPU rather than a cluster. Sizing is a memory calculation first and a throughput calculation second.

A raw model file is not a service. An inference server sits in front of it and does the work that makes it usable at scale: batching concurrent requests, managing the key-value cache that makes generation efficient, streaming tokens back, and exposing an API. Mature open-source servers handle this well, so the task is running and operating one, not building it. This is the main operational cost a hosted API hides from you, and it is the price of keeping the model in-house. On Kubernetes it also gains the autoscaling and fractional-GPU techniques that keep utilisation and cost under control.

Sovereignty is only real if it is enforced, not assumed. A sovereign deployment runs in a network that denies outbound connections, so the model, the data, and the logs physically cannot leave, and that denial is verifiable rather than promised. This matters because a self-hosted model that quietly calls out to an external service for any part of its work is no longer sovereign. Egress control, ideally in an air-gapped or egress-locked environment, is what turns the sovereignty claim into a property you can audit.

A common worry about self-hosting is falling behind the frontier. In practice the open-weight field moves fast, and strong new models are released constantly, so staying current is a matter of process: pull a new open-weight model, validate it against your own tasks, and swap it in behind the same serving stack and orchestration layer. Because the application talks to the orchestration layer instead of the model directly, that swap does not touch the application. Self-hosting and staying current are compatible when the architecture treats the model as replaceable, which is the model-independence principle applied to operations.

Roughly two bytes per parameter at full precision, so a model with tens of billions of parameters needs tens of gigabytes just to load, plus working memory per request. Quantisation to eight or four bits cuts that substantially with a small quality trade, often letting a capable model run on a single accessible GPU.

An inference server that sits in front of the model file and makes it a usable service: batching concurrent requests, managing the KV cache, streaming tokens, and exposing an API. Mature open-source servers handle this, so the work is operating one, and building it is not required. On Kubernetes it also gains autoscaling and fractional-GPU efficiency.

By enforcing egress control. Run the deployment in a network that denies outbound connections, ideally air-gapped or egress-locked, so the model, data, and logs cannot leave and that denial is verifiable. A self-hosted model that calls out to an external service is not truly sovereign.

No, when the architecture treats the model as replaceable. The open-weight field moves fast; staying current is a process of pulling a new model, validating it on your tasks, and swapping it behind the same serving and orchestration layer, which leaves the application untouched.

Yes, and increasingly common for regulated data. The open-weight ecosystem has matured, quantisation lets capable models run on accessible GPUs, and mature serving software handles the operations. The main constraints are GPU supply and the operational effort, both manageable with the right stack.