StrategyCore
Back to Resources
18/ai / ml 2 min read

Evaluating LLM inference serving on Kubernetes

Test model readiness, memory demand and latency under realistic traffic.

A model service must answer requests within the application’s time and quality limits, including when traffic changes or a component fails. Kubernetes supplies operating tools, but the model, serving engine, hardware and traffic determine the result. Test them together and record the work the operating team will need to do.

A model-serving request path

SC-FL-01 · REV A · 2026.07

01Request checks

identity and limits

02Routing and queue

capacity and timeout

03Ready model replica

runtime and accelerator

04Response

quality and telemetry

Replica scaling, GPU availability and cold-start behaviour require separate validation.

An authorised request reaches a ready model replica and returns a bounded response.
01

Define the service and memory requirements

Specify input and output lengths, concurrency, response targets and acceptable failures. Model weights are only part of memory use: caches, activations, runtime overhead and concurrent requests can add demand. Check support for the selected model, numerical format, accelerator and serving-engine version. A small pilot does not establish production capacity.

02

Benchmark batching and model placement

Batching can improve throughput but can affect request latency and memory demand. Parallelism, quantisation or shared accelerators may help some workloads and hurt others. Use representative prompts, long requests and concurrent tenants. Compare time to first output, continuing output speed, tail latency and errors rather than reporting only a best-case throughput figure.

03

Test scaling from request to ready replica

Kubernetes HPA can scale from supported metrics, but the application needs appropriate metrics and adapters. Loading weights, scheduling a GPU and warming the model can delay a replica’s readiness. Scaling replicas does not guarantee node or accelerator availability. Scale-to-zero requires a suitable serving layer and acceptable cold-start behaviour; it is not an automatic property of HPA.

04

Exercise failure and operating controls

Test overload, timeouts, cancelled requests, replica or node loss and recovery. Define authentication, request limits, logging and model-version rollback. Include every service that receives prompts or outputs in the data review. Self-managed Kubernetes does not itself prove data stays in Japan or that external telemetry and support access are disabled.

Sources and further reading

// Key Takeaways

What to remember

  • Size for weights, runtime state and concurrent requests.
  • Measure latency and errors under representative load.
  • Test startup, scaling, failure and data flows end to end.

// FAQ

Frequently asked questions

Q1

Does Kubernetes HPA automatically scale a model to zero?

No. Scale-to-zero depends on the serving or event-driven scaling implementation and its configuration. Evaluate cold starts and readiness for the intended service.

Q2

Is model-weight size enough to size inference memory?

No. Include runtime state, KV cache, input/output lengths, concurrency and overhead, then validate the configuration with load tests.

Last updated:

Prepare Japanese evaluations of your inference platform.

Discuss supported models, batching, capacity limits and recovery behaviour. Our engineers can organise Japanese workload tests with your team, while our sales team develops accounts, proposals and the local support arrangement.