Why the Model You Run Still Matters: Weights, Quantization, and Reproducibility

Weights are the parameters learned during training. They are the model's entire persistent state, its parametric memory: grammar, world facts, reasoning, and coding skill, all encoded implicitly in these numbers. Inference does not change them. A forward pass reads the weights and never writes back, so in-context learning, retrieval, and long prompts change the context and the activations, not the parameters. What mutates weights is training: pretraining, then instruction tuning, then preference alignment. A base model, its instruction-tuned version, its preference-aligned version, and an adapter-merged variant are therefore different artifacts even inside one lineage. Same model family is not the same weights.

Even byte-identical weights do not guarantee identical output. Sampling above temperature zero adds randomness, and greedy decoding removes that. Two harder sources remain. Core GPU kernels are not batch-invariant, so a result depends on the batch size a request happens to run in, which shifts with server load. Hardware and numeric precision change the arithmetic path as well. The evidence is stark. At temperature zero, one 235B model produced 80 unique completions across 1000 identical requests, first diverging at token 103; with batch-invariant kernels all 1000 became bitwise identical, at roughly a 1.6 to 2 times slowdown (Thinking Machines Lab, 2025). Under 16-bit precision, changing only GPU type, GPU count, and batch size with fixed seeds produced up to 9 percent accuracy variation, and 90 percent of examples diverged across hardware; under 32-bit, only 2.2 percent did (Yuan et al., 2025). Bitwise reproducibility is an engineering achievement, not a free property of fixing the seed.

Quantization stores weights at a lower bit-width to shrink the memory footprint. The number is the compression level, roughly N bits per weight for Q-N, and lower means heavier compression and lower fidelity. On an 8B model measured against a 16-bit baseline, 8-bit and 6-bit are near-lossless, while 4-bit carries a small but real cost and anything below 4-bit degrades sharply. The loss is not evenly spread. Mathematical reasoning is the most sensitive, falling from about 77.6 percent to 68.3 percent at 3-bit and recovering near baseline at 4-bit; exact structured output and numeric precision degrade first; commonsense is barely touched. Larger models tolerate compression better than small ones. The working rule is to run the least compression that fits the hardware, and to treat the quant as part of the model version. A quant change is a different model.

A model served behind a stable API name is not a fixed artifact. Comparing two snapshots of the same named model three months apart, one study measured its accuracy on a fixed task falling from about 84 percent to 51 percent, with more formatting errors and weaker instruction-following, under the same name and with no version bump (Chen, Zaharia, Zou, 2023). The vendor had silently re-tuned or replaced the weights behind the name. Every evaluation baseline and downstream pipeline built on that endpoint breaks quietly when it happens. Real reproducibility requires self-hosted, version-locked weights, not a vendor endpoint. This is the plain technical case for a sovereign, self-hosted posture, before cost or privacy enter the argument.

The real axis is not one model versus many. It is controlled versus uncontrolled variation, and complementary versus diluting variation. A single pinned, self-hosted model gives strong reproducibility and one evaluation suite to gate everything, capped at that model's capability. A mix of models can beat any single one and can save cost through routing and cascades, but only when the models are individually strong and genuinely complementary. The nuance is sharp: ensembling several models beat a single premium model on one benchmark, 65.1 against 57.5 percent, yet aggregating several samples from the single best model beat the mixed ensemble by 6.6 percent. A weak model dilutes the mix. Two costs of heterogeneity are usually under-priced: prompts do not transfer cleanly between models, and every model added is another version to pin, evaluate, and monitor for drift. One place to use a different model on purpose is verification, since a model judging its own output carries a self-preference bias that a separate judge reduces.

Knowledge can sit inside the weights or be supplied at inference from an external store. The split is the founding idea of retrieval-augmented generation, and the evidence that external knowledge substitutes for parametric knowledge is strong. One retrieval-augmented model matched a model class 25 times larger on a knowledge-intensive benchmark by conditioning on a large external datastore. Others decouple the store from the generator so knowledge updates without retraining, and when both are available, retrieval-augmented models prefer the external context over their own memory. For the knowledge dimension, then, the model is a commodity and a swappable read and write head: you update the store, not the weights. This is the architecture we design toward, because it lets the reasoning engine improve underneath a durable, model-independent knowledge asset.

Externalizing knowledge is real, but it leaks at two seams the store cannot insulate. On the write side, extraction fidelity is set by the model that populates the store, and that error is baked in. A benchmark across 22 models found that value fidelity, not structural compliance, is the bottleneck: wrong-value errors account for roughly half of imperfect runs even at 70B, and the dangerous failure is a confident, schema-valid, factually wrong output that downstream systems trust because it parses. On the read side, two models given identical external state diverge in how faithfully they use it, and stronger models are more stubborn about their own priors, sometimes trusting their own incorrect knowledge over the supplied evidence. The practical consequence: a knowledge store is portable across a model swap, but extraction and faithful read-back must be re-validated whenever the model changes. Getting the extractor right at the start matters, because a write-once store inherits its error distribution permanently.