StrategyCore
Back to Resources
29/ai / ml 2 min read

Model versions, quantisation and reproducible evaluation

Record the full inference configuration and test changes against the application’s requirements.

When an AI application changes its answer, the model name alone may not explain why. The result also depends on the model release, prompts, numerical settings and software running it. Record the configuration you tested, then decide whether the task needs identical output or a consistent level of quality.

A recorded inference configuration

SC-LY-04 · REV A · 2026.07

Model artefacts

weights / tokenizer / adapters

T1

Numerical settings

precision / quantisation / sampling

T2

Runtime

libraries / hardware / kernels

T3

Application context

prompts / tools / retrieved data

T4

Pinning inputs supports comparison; it does not guarantee cross-platform identical output.

Model artefacts, numerical settings, runtime and application context all contribute to evaluated behaviour.
01

Identify the artefacts and configuration

Record the exact model release and, where available, artifact hashes, tokenizer, adapters, prompt template and generation settings. For a hosted service, check the provider’s version identifiers and change policy. For self-managed weights, also record runtime, libraries, drivers and hardware. Neither a model family label nor a stable endpoint name proves the configuration has stayed fixed.

02

Distinguish inference from training

Model weights are learned parameters. Ordinary inference uses them to compute outputs; prompts and retrieved documents affect the context rather than retraining the base model. Systems can also include adapters, external memory or explicit online updates, so document those separately. An application’s persistent state is not limited to its base weight file.

03

Evaluate quantisation on the actual task

Lower-precision representations can reduce memory, but quality and speed depend on the method, model, runtime and hardware. A lower bit-width is not a universal ranking of usefulness, and a quantised variant is not necessarily worse on every metric. Compare representative Japanese and English tasks, structured outputs, tool calls and long inputs. Record the quantisation method as part of the tested configuration.

04

Define the reproducibility target

PyTorch warns that complete reproducibility is not guaranteed across releases, platforms or devices. Seeds and deterministic settings help within defined conditions, but do not establish bit-identical behaviour everywhere. Decide whether the application needs identical tokens, bounded statistical variation or a quality threshold. Test repeated requests and deployment changes accordingly.

05

Treat model replacement as a regression exercise

A common API or external knowledge store can reduce coupling, but it does not make models interchangeable. Recheck evidence extraction, retrieval use, prompts, tool permissions, format handling and safety behaviour when the model changes. A separate model used as an evaluator also needs validation; it is not an independent source of truth simply because it is different.

06

Record the model configuration and evaluation results for each release

Retain the evaluation dataset, expected evidence, scoring method, configuration and failures. Compare quality, latency, memory and cost rather than selecting from a single benchmark. Record model licensing and update responsibilities. Release only after the defined criteria are met and a rollback to the known configuration has been tested.

Sources and further reading

// Key Takeaways

What to remember

  • Version the model, prompt, runtime and numerical configuration together.
  • Measure quantisation effects rather than assuming a universal quality loss.
  • Seeds and fixed weights do not guarantee identical output across all environments.
  • Re-evaluate extraction and answer behaviour when changing models.

// FAQ

Frequently asked questions

Q1

Do fixed weights and temperature zero guarantee identical outputs?

No. Runtime, numerical behaviour, hardware and other conditions can still affect results. Define the required reproducibility target and test the exact environment.

Q2

Does lower-bit quantisation always make a model less useful?

No universal conclusion follows from bit-width alone. Evaluate the chosen method on task quality, speed and memory for the actual hardware and runtime.

Last updated:

Prepare a reproducible model evaluation for Japan.

Discuss the model files, licences, runtime and test records your engineers can supply. StrategyCore can coordinate Japanese technical discussions and agree the evidence a prospect will receive during the product evaluation.