How Would You Know If It Was Working

Classical machine learning came with a labelled test set and a metric everyone accepted. Foundation models broke that arrangement in four places. Outputs are open-ended, so there is rarely one correct string to compare against. The model is a black box, so you cannot inspect why its behaviour moved. Public benchmarks saturate quickly and leak into training data, which makes a high score evidence of contamination roughly as often as evidence of capability. What survives is narrow and specific: your own task, your own data, your own written definition of a good answer.

Two kinds of check live in every serious eval suite, and confusing them wastes months. The exact kind asks whether the output is well formed. Valid JSON, required fields present, the number inside the allowed range, the citation pointing at a document that exists. These are cheap, deterministic, and belong in continuous integration. The subjective kind asks whether the content is right, which needs a written rubric and a judge. Build the exact layer first. It catches a surprising share of production failures for almost no effort, and it never argues with itself.

For content quality there is an order worth following. Exact match works where the answer is a fact with one form. Semantic similarity handles paraphrase, at the cost of a scoring model you now also have to trust. A model acting as judge against a written rubric covers the rest, and it is where most teams end up. Move up the ladder only when the rung below stops discriminating between a good answer and a bad one. Every step up adds cost, and adds one more thing that can drift without telling you.

A model scoring outputs is a measurement instrument with characteristic errors. It rates its own family's output higher. It is sensitive to the order candidates appear in. It keeps rewarding length past the point where length helps. None of that makes the method unusable, and all of it makes an unaudited judge misleading. Randomise position, score a held-out set by hand on a schedule, and check the judge against human ratings before quoting a number that came out of it. A judge nobody has calibrated produces a metric nobody should cite.

The strongest argument for an eval pipeline is not technical rigour. It is that the metric maps onto money. Factual consistency at eighty percent might let the system close three in ten support tickets without a person; at ninety percent, five in ten. Now a model improvement has a value, a budget, and a stopping point. Bring that mapping into a Japanese enterprise conversation and the discussion moves from whether AI can be trusted to which threshold funds the deployment, which is a far better meeting to be in.

The first eval set is guesswork, written by the team that built the feature, and it goes stale fast. Real queries never look like the imagined ones. Log them, sample the failures, promote them into the suite, and the suite grows toward the traffic instead of away from it. User feedback works the same way: a thumbs-down with the conversation attached is a labelled example. Wire that path before launch, because retrofitting it means asking a customer for permission to keep data you already hold, and in Japan that is a conversation with the privacy officer rather than the project sponsor.

Start with checks you can run exactly: valid JSON, required fields present, numbers in range, citations pointing at documents that exist. Those are cheap and belong in continuous integration. Content quality needs a written rubric and a judge, which costs more. Build the exact layer first, because a Japanese risk committee asks for evidence early and that layer is the fastest thing to show it.

Foundation models broke three assumptions. Outputs are open-ended, so there is rarely one correct string to compare against. The model is a black box, so you cannot inspect why its behaviour moved. Public benchmarks saturate and leak into training data, which makes a high score evidence of contamination about as often as capability.

It is a model scoring outputs against a written rubric, and it works if you treat it as an instrument with known errors. It favours its own model family, the first candidate position, and longer answers. Randomise order and calibrate against human ratings before quoting a number that came out of it.

Map the metric onto a business number. Factual consistency at eighty percent might close three in ten support tickets without a person; at ninety percent, five in ten. That turns a model improvement into a value, a budget, and a stopping point, which is the form a Japanese enterprise investment case takes before it clears an internal approval round.

Production. The first set is guesswork by the team that built the feature and goes stale fast, because real queries never look like imagined ones. Log them, sample the failures, promote them into the suite. In Japan, wire the consent path for that before launch rather than negotiating it afterwards.