Operating an open-weight LLM on customer infrastructure
Check licensing, memory, network dependencies and change control before deployment.
Who will maintain the model, patch its runtime, manage the GPUs and investigate a failed request? An on-premise proposal needs to answer those questions alongside data handling and performance. Running model weights on customer equipment gives direct control over parts of the service, with corresponding operating responsibilities.
A controlled on-premise design
SC-BD-04 · REV A · 2026.07
Model and runtime
licence and versions
Data and keys
access and recovery
Packages and updates
approved import
Tools and support
explicit access review
Illustrative intended controls. Verify the actual configuration and dependencies.
Review the model and its licence
Identify the exact model release, weights, tokenizer, adapters and intended use. Check the licence and any use or redistribution restrictions; downloadable weights are not automatically open-source AI under the OSI definition. Confirm how packages, model files and security updates will enter the environment and who will verify them.
Size memory and throughput separately
For P parameters represented at b bits each, P × b / 8 gives a nominal raw-weight byte count. Real formats may mix precision or add metadata. Serving also needs runtime memory, cache and room for concurrent requests. Quantisation can reduce memory but its quality and speed effects depend on the method, model and hardware. Validate the proposed configuration under load.
Verify network and access controls
Inventory model downloads, embedding services, tools, licence checks, telemetry, logs, backups and remote support. If the design requires restricted egress, configure and test it explicitly. An air-gapped design also needs an approved update and incident process. Self-hosting does not prove that every component is offline, and local processing does not remove access-control or privacy obligations.
Record model versions and test updates before release
Record the weights, runtime, drivers, prompts and settings used in evaluation. Test model updates, vulnerability fixes, restart and recovery before release. A common serving interface may reduce application changes, but behaviour and resource demand can still change. Preserve a known working configuration and define who approves a rollback.
// Key Takeaways
What to remember
- Review the exact model licence and release.
- Raw weights are only part of serving memory.
- Test egress, updates, failure and model changes explicitly.
// FAQ
Frequently asked questions
Does downloading weights grant unrestricted use?
No. Check the licence for the exact release and intended use, including redistribution and any restrictions. Open weights and open-source AI are not interchangeable terms.
Does placing a model on premises prove data cannot leave?
No. Inspect and test dependencies, tools, logging, telemetry, backups and support access. Apply and verify network controls required by the design.
Last updated:
Plan the local delivery of your self-hosted AI product.
Explain the hardware, model licence, update process and support access your product needs. We can coordinate Japanese prospect evaluations and agree which installation and ongoing support tasks sit with your team, StrategyCore and delivery partners.