Reproduce the as-built model before you correct it

You cannot audit what you cannot regenerate

The most useful thing the July 5th audit session did for Quantum Caddy happened before any corrected physics existed. Asked to audit a deprecated training-data simulator, the session's first move was to rebuild that simulator from the outside, using nothing but the data it had shipped. No corrections and no judgments, just a reproduction pass.

The logic deserves a slow walk. An audit that skips straight to building the correct model ends the day holding two models that disagree, and disagreement between two models is nearly information-free on its own. Maybe the reference exposed a real defect. Maybe the auditor misread what the original computes. You cannot tell, because nothing has pinned down what the original actually does, as opposed to what its documentation or its authors believe it does.

Reproduction is the pin. Once you hold a model that demonstrably regenerates the artifact's outputs, you know what the artifact computes at the level of mechanism, whatever its source claims. Every later disagreement acquires an address. Reproduction converts "these two disagree" into "they disagree because of this specific mechanism," and that conversion is the difference between an audit and an argument.

There is a second payoff, and it is cheaper than it sounds: reproduction finds defects on its own, before any reference model exists. Fitting an artifact from the outside forces you to stare at its outputs harder than its authors ever did, and internal inconsistencies fall out of that stare.

The shipped data didn't match the shipped code

Here is how the pass ran on our simulator. The grid of simulated throws, tens of millions of rows deep, had shipped, but the script that generated it was not in the repo. Two sibling generator scripts were, and the natural assumption is that one of them produced the data. The session tested the assumption instead of adopting it. Neither sibling reproduced the shipped outputs.

So the session treated the grid as a black box and fit it from the outside. What came back was a closed-form model that matched the simulator's flight outputs to about 1.5 percent, published as a fidelity table with a number for every field it fit. That table carried the rest of the audit. From then on, any claim about what the simulator does was backed by a model that regenerates the simulator's own numbers, not by a reading of orphaned code.

The reproduction pass also caught rot before correction ever started. One output field turned out to be inflated by a third in one regime of throws, an inconsistency internal to the artifact, visible with no reference physics at all. Findings like that went on their own list, separate from the correctness disagreements that came later. The separation matters because the two kinds of finding carry different weight: a defect the artifact has against itself is undeniable, while a disagreement with your new model is only as strong as your new model.

Fit the simplest model that regenerates the outputs

Stripped of the physics, the method is four steps, and none of them requires a science agent.

• Treat the artifact as a black box. Audit what it emits, not what its code says. Code describes intent; the outputs are the behavior you have been relying on.
• Fit the simplest model that could regenerate the outputs. Start with the dumbest closed form that might work and add structure only when the residuals demand it. The simpler the recovered model, the more interpretable every later divergence.
• Publish the fidelity numbers. A table, one number per output field, of how closely the recovered model matches, fit on the fields people act on. This is the evidence that you understand the thing you are about to criticize.
• Keep two defect lists. One for what the artifact gets wrong against itself, found during reproduction. One for what it gets wrong against your reference, found later. Never merge them.

An honest limit sits under all of this: reproduction is not validation. Our 1.5 percent covered the flight outputs. The simulator's scoring stage never reproduced cleanly from the outside, the audit said so plainly, and every conclusion about that stage was held to a weaker standard as a result. A fidelity table earns its authority by being allowed to contain bad numbers; a table that only ever reports success is decoration.

Three ways reproduction fails

Skipping to the reference model is the common one. Under deadline, reproduction feels like ceremony, a delay before the real work of building the better model. The cost arrives two weeks later as ambiguous divergence: the models disagree, nobody can say which mechanism is responsible, and the meeting about it turns into an argument about intent instead of a review of evidence.

Reproducing only the average is subtler. A recovered model that matches the artifact's headline aggregate can still be wrong about every field that feeds a decision. Averages are where errors go to cancel. Fit the fields you will act on, at the granularity you will act on them, and report fidelity for each one separately.

Trusting the repo to be the generator is the one that got us. Version control describes what somebody intended to ship at some moment in the project's history; whether that code produced the dataset in front of you is an empirical question. In our case both candidate generators failed the test, and an audit that had read either one as ground truth would have spent the night auditing a simulator that no longer existed. Verify code against outputs before you cite it.

Apply this

Pick one generated dataset your team acts on. A synthetic training set, a demand forecast, a risk model's scored backtest, a game economy's simulated payouts. One artifact, attempted this week.

Before you fit anything, write down the fidelity number that would satisfy you, per output field, dated, somewhere you cannot quietly revise.

A fidelity target chosen after the fit is a rationalization with a decimal point. Write the passing number down before you touch the data.

Then run the pass:

• List the output fields that feed real decisions, and fit only those.
• Treat the artifact as a black box; leave the generator code closed for now.
• Fit the simplest model that could regenerate those fields, and publish the fidelity table somewhere your team will see it. A table in the repo beats a claim in a meeting.
• Log every internal inconsistency you hit on its own list, labeled as reproduction defects, before you form any opinion about correctness.
• Only then open the generator code, and check whether it can produce the data you actually have.

If the table comes in worse than your pre-written number, that is the finding, not a failed exercise. You have learned that your team acts on an artifact nobody can regenerate, and you learned it for the price of a week.