Run a falsification session before you trust your own asset

For most of a quarter, Quantum Caddy's throw simulator looked like an asset. Tens of millions of simulated cornhole throws, built in a weekend to be a training-data engine while we waited on real footage. A quarter of sitting in the repo had promoted it from experiment to fact. On the evening of July 5th I pointed Claude at it with a one-page brief written like a lab contract and went to do something else. Three hours later, every maximum-score throw in the grid was exposed as the product of one modelling shortcut, and 48.3 percent of the outcomes we cared about had flipped.

Field note 0023 is the teardown of that session. This piece is about the discipline underneath it, because the session was cheap, the finding was existential, and the only reason it happened at all was a scheduling decision. Every operator I know is carrying assets like this: simulators, demand models, dashboards, forecasts. Built under deadline, checked by their builder, promoted into load-bearing status by nothing but time.

Nobody attacks their own work under deadline

Self-audit fails for social reasons before it fails for technical ones. The person best positioned to falsify an asset is the person who built it, and that person has a deadline, a plan that depends on the asset being fine, and a weekend of sunk pride. When I looked at our simulator I saw the effort it cost and the model it was about to feed. I did not look hard at the slide stage, because the slide stage was where the exciting outcomes came from, and nobody points a gun at the part of their work that produces the results they want.

Handing the audit to a teammate barely helps. Attacking a colleague's work spends relationship capital, so peer review converges on the polite version: a read-through, a few questions, an approval. And on a two-person team there is no spare week for Curtis to rebuild my asset from first principles, which is what a real audit costs a human.

A contract-bound agent has no stake in your asset being right

What made the July session a genuine adversary was the absence of every social force above, plus a contract. The agent had no pride in the simulator, no deadline that depended on it, and no relationship to protect. The brief held it to an experimentalist's standard: no claim survives without a number, an error bar, and a plot; write code another engineer can run; when the data can't support a claim, say so and name the measurement that would settle it.

Under that contract the agent did the thing no human on a small team will do. It spent its opening moves reproducing our simulator from the outside, matching its flight numbers to about 1.5 percent, before criticizing anything. Every defect it reported afterward traced to a specific mechanism rather than a difference of opinion. It never got tired or defensive, and it left behind scripts and CSVs that Curtis can re-run without trusting me or the agent. The artifacts are the point: an adversary whose attack you can reproduce is an adversary you can learn from.

Three hours against a season of wrong training

The accounting on the session: the brief took about forty minutes to write, and the run took three hours while I did other work. That was the entire cost.

Against it, what the session prevented. The simulator existed to generate training labels, and a scoring model trained on its outcome labels would have learned a rule that does not exist in the world. Under corrected physics, 48.3 percent of outcomes flip, and all 2,000 maximum-score throws in the audited sample turn out to ride an unphysical slide of about three feet into the hole. Zero survive. We would have discovered that months later, wrapped in inference code, at many times the cleanup cost, if we discovered it at all.

The session also told us what to keep. Flight prediction was validated to about plus or minus two inches, while the stage after board contact carries about fifteen inches of uncertainty, and one coefficient drives roughly three times more of that error than any other. The demolition came with a salvage map: keep the flight stage, measure friction first, trust nothing after the bag touches the board.

The right time is before the asset feeds a decision

An audit after the decision is a damage report. The July session's whole value was that it ran before we trained anything, so the finding cost us a plan instead of a product. Falsification sessions get scheduled by decision proximity, not by suspicion — the assets that most need an adversary are the ones nobody suspects, because unsuspected is how they became load-bearing.

Three triggers that put an asset at the front of the queue:

• Before it generates training data. Once a model learns from generated labels, the defect stops being visible in any single output.
• Before a number it produced goes into a fundraise deck or a loan package. Investor diligence is an adversarial audit whose timing you don't control.
• Before it graduates from draft to dependency. The day another system starts consuming an asset's output is the day its errors start compounding somewhere you don't look.

The loop that makes it a discipline

One session is an anecdote. The practice worth adopting is a loop:

• **Inventory:** list the assets your decisions currently rest on, with the date each was last adversarially examined. For most of mine the honest date was never.
• **Queue:** rank by decision proximity. The asset feeding next month's decision outranks the one you happen to be suspicious of.
• **Session:** one evening each. An agent, an experimentalist contract, the raw data, and a mandate to reproduce the asset before correcting it.
• **Output:** require a ranked list of missing measurements, not just a verdict. Ours came back with a ranked list of six experiments, ordered by how much prediction error each one retires.
• **Re-audit:** any change to the asset or its use puts it back in the queue. An audit certifies a version, not a thing.

We still don't have a validated scoring predictor, and the honest status is that nothing downstream of the flight stage earns trust until real throws are filmed and measured. What changed is posture. The simulator is now the best-understood asset in the portfolio precisely because it's the only one that has faced a hostile expert. The rule we adopted is the scheduling one: a falsification session goes on the calendar before its asset feeds a decision, not after the decision starts to look wrong.