Read the code that made the number

This is a sample teardown, run against my own evaluation harness rather than a client's, so you can see the shape of the deliverable before you commission one. It takes a single metric, states what it claimed, lists what was checked and in what order, names the confound, gives the steps to reproduce it, and states what the number became once the confound was removed. That is one written finding, in the format every finding in a real teardown arrives in.

The metric as received

For months, one capability measured zero. Not low, zero, across every routing evaluation I ran. The number was stable, it reproduced, and it was the kind of clean result you stop questioning. So I did not question it. I built theories about why the capability was hard, and I built mechanisms to fix a problem that did not exist. A confidently wrong zero had become the floor the rest of the work was standing on.

The attack list

The rule is that the instrument gets attacked before the result does. In order:

1. Start at the instrument, not at the result. That habit is the first of the four the work runs on, and everything below is it applied to one metric.
2. Read the load, not just the score. The load returns the keys it could not place, and the load line prints how many were dropped against how many were loaded. Read the top of the run, not only the accuracy at the bottom.
3. Confirm the model under test is the model on disk. Check that the trained weights and the routing they are supposed to use both actually loaded before a single question is scored.

The finding

The evaluation harness loaded checkpoints with strict=False. That one argument tells the loader to accept a checkpoint even where it does not match the model, and to drop the keys it cannot place rather than raise. It dropped 112 trained routing keys in silence.

result = model.load_state_dict(state, strict=False)
# result.missing_keys held 112 entries. Nothing ever printed it.

Every one of those evaluations had been scoring a model whose routing was random: the weights were trained, and the routing they were supposed to use never loaded. The capability was not failing. It was never being measured. The tell was on screen the whole time, in the load count at the top of every run, sitting above an accuracy number I was reading at the bottom.

The same check, on your harness

The checkpoint and the logs behind this example are private and are not published, so nobody can re-run my case. That is the wrong thing to be running anyway. Here is the check itself, four steps, on your harness rather than mine:

1. Capture the return value of load_state_dict instead of discarding it.
2. Print missing_keys. Here it held 112 entries, and they were the routing keys. Whatever it holds on yours is the number that matters.
3. Add a hard assertion that the keys you care about loaded, and re-run one evaluation. If it refuses to score, that is the correct behavior: it was never scoring the model you meant to test.
4. Load again with those keys present, and re-score. If the number moves, it was never measuring what you thought it was.

What the number was actually worth

Nothing. The zero was not a capability floor, it was an artifact of a harness scoring a randomly routed model, and no amount of re-running or retraining would have moved it, because none of that touched the load. The fix was not a better model. It was one assertion that the routing keys had actually loaded, and a refusal to score anything until they had. That assertion is still in the harness I run today, guarding every score against a load that only looked like it worked.

With the load finally verified, the same evaluation stopped returning zero and returned a real, low reading. A dead zero tells you nothing to act on; this number pointed at a concrete, fixable failure instead: the model reciting the first fact it had stored, regardless of what the query asked. That is the shape of the deliverable. A teardown does not end at "your number was wrong." It ends at the real number and the specific reason it is what it is.

This is also why every number on this site has a row in a public ledger naming the source it was checked against, why the build refuses to compile a row that has drifted from the wording it was locked in, and why a result whose evidence status is a proxy rather than a measurement cannot ship as copy at all. The ledger publishes what that check does not catch as well as what it does. Honesty holds better when a check enforces it than when a person remembers to.

This is what the Evaluation Teardown does, pointed at your harness instead of mine. Before you defend a number in a room you do not control, I attack the instrument that produced it, and you keep one written finding per metric: the confound named, the exact steps to reproduce it, and what the number becomes once the confound is gone. This one took me months to find in my own code. The point of the engagement is that it should take days to find in yours.