Evaluating LLM inference serving on Kubernetes
Test model readiness, memory demand and latency under realistic traffic.
Kubernetes provides building blocks for operating a model service, but it does not make inference cheap, resilient or automatically scalable. The serving engine, accelerator, model and traffic pattern determine much of the result. Evaluate the complete request path and the operating work it creates.
A model-serving request path
SC-FL-01 · REV A · 2026.07
01Request checks
identity and limits
02Routing and queue
capacity and timeout
03Ready model replica
runtime and accelerator
04Response
quality and telemetry
Replica scaling, GPU availability and cold-start behaviour require separate validation.
Define the service and memory requirements
Specify input and output lengths, concurrency, response targets and acceptable failures. Model weights are only part of memory use: caches, activations, runtime overhead and concurrent requests can add demand. Check support for the selected model, numerical format, accelerator and serving-engine version. A small pilot does not establish production capacity.
Benchmark batching and model placement
Batching can improve throughput but can affect request latency and memory demand. Parallelism, quantisation or shared accelerators may help some workloads and hurt others. Use representative prompts, long requests and concurrent tenants. Compare time to first output, continuing output speed, tail latency and errors rather than reporting only a best-case throughput figure.
Test scaling from request to ready replica
Kubernetes HPA can scale from supported metrics, but the application needs appropriate metrics and adapters. Loading weights, scheduling a GPU and warming the model can delay a replica’s readiness. Scaling replicas does not guarantee node or accelerator availability. Scale-to-zero requires a suitable serving layer and acceptable cold-start behaviour; it is not an automatic property of HPA.
Exercise failure and operating controls
Test overload, timeouts, cancelled requests, replica or node loss and recovery. Define authentication, request limits, logging and model-version rollback. Include every service that receives prompts or outputs in the data review. Self-managed Kubernetes does not itself prove data stays in Japan or that external telemetry and support access are disabled.
// Key Takeaways
What to remember
- Size for weights, runtime state and concurrent requests.
- Measure latency and errors under representative load.
- Test startup, scaling, failure and data flows end to end.
// FAQ
Frequently asked questions
Does Kubernetes HPA automatically scale a model to zero?
No. Scale-to-zero depends on the serving or event-driven scaling implementation and its configuration. Evaluate cold starts and readiness for the intended service.
Is model-weight size enough to size inference memory?
No. Include runtime state, KV cache, input/output lengths, concurrency and overhead, then validate the configuration with load tests.
Last updated:
Talk to us about selling your product in Japan.
Tell us about your product, the buyers you want to reach and the technical support they will need. We can discuss sales, evaluations and support in Japan.