I asked Claude to audit my simulator. It killed every high score.

We built a throw simulator for Quantum Caddy last quarter. A parameter sweep tens of millions of rows deep, then an overnight Monte-Carlo run nearly three times larger that cooked while I slept. The plan was to use it as a training-data engine: teach a model which throws score before we had enough real footage to learn from.

On the evening of July 5th I wrote a prompt that told Claude to treat that simulator as a defendant, handed it the data, and went to get a drink. Three hours later it had reproduced our simulator from the outside, built a corrected physics model from first principles, and shown that every single maximum-score throw in the entire grid was an artifact of one modelling shortcut.

This is the teardown of that session — what the agent did, what it found, and what I now believe about simulators as training data.

Tens of millions of assumptions

Quantum Caddy is our AR scoring system for cornhole. A bag gets thrown about 27 feet onto an inclined board with a hole in it. Three points in the hole, one on the board. A miss is worth nothing. The physics splits into two regimes: flight, which is a slow arcing lob with some air drag, and the landing, where the bag either sticks, slides up the incline toward the hole, or bounces off. All the interesting scoring behavior lives in that second regime.

The simulator swept eight input parameters (everything from launch state to board geometry to the air) and emitted where the bag landed, how far it slid, and how it scored. Tens of millions of combinations. It looked authoritative. Big numbers usually do.

What we did not have was a single measured throw. No high-speed video and no tracked trajectories. Every "parameter" in that grid was a dialed-in assumption. I knew that abstractly. I had not sat with what it meant.

A contract, not a question

The brief I gave the session was one page. The line that did the work:

“You work like an experimentalist, not a theorist with a whiteboard: no physical claim survives without a number, an error bar, and a plot. When the data can't support a claim, you say so and state what measurement would settle it.”

That sentence changed the character of everything that came back. No hand-waving about "the model suggests." Every finding arrived with a reproduction script, a CSV, and a figure. When the agent hit the edge of what the data could support, it said so and put the missing measurement on a list instead of guessing.

I've written a lot of agent prompts in the past year. This contract clause is going in all of them.

Reproduce before you correct

The first thing the agent did surprised me: it ignored my physics questions and reverse-engineered the simulator itself. The generator script for the grid wasn't in the repo — two sibling generators were, and neither reproduced the shipped data. So the agent fit the grid from the outside and recovered a closed-form ballistic model that matched our simulator's flight numbers to about 1.5 percent.

That step earned everything after it. If it had jumped straight to "here's the correct physics," I'd have had two models and no way to say what the difference meant. Because it reproduced the as-built model first, every later disagreement was traceable to a specific defect rather than a vague vibe of wrongness.

It found the defects fast. Our drag only acted horizontally — a bag in our simulator fell like a vacuum object vertically while getting slowed sideways. The flight-time field was inflated a third for short throws. And the slide model had no impact energy loss at all: a bag hitting the board at sixty degrees still slid 1.7 feet, where a real bag smacks, deforms, and stops.

Midnight, minus two thousand hole-ins

Then the divergence map. The agent ran both models, our recovered simulator and its corrected one, over a balanced six-thousand-throw sample and compared outcomes.

In the air, the two models agree almost perfectly. Median landing difference: 0.01 feet. Apex within a quarter inch. If you plotted the trajectories on top of each other you would not tell them apart.

On the ground, 48.3 percent of outcomes flip. And the number that made me put the drink down: the sample held two thousand hole-in throws, the max-score outcomes the whole product exists to detect, and zero survive. Every one becomes a board score or a miss under corrected physics.

The mechanism is almost embarrassing once you see it. Our simulator's hole-ins land near the front of the board and then ride a three-foot slide up the incline into the hole. Real contact physics arrests those bags almost immediately — steep descent means the impact eats the tangential energy, and the slide is close to zero. The median ratio between our slide and the corrected slide on those throws was a factor of several hundred. Our best outcomes weren't physics. They were a slide bug wearing a scoring rule as a costume.

The disagreement isn't localized either. It spans the middle of the launch-angle range, the middle of the speed range, the middle of the friction range. There is no single knob to turn that patches it.

Salvage

Here's what kept the night from being a pure loss: the flight model is fine. Better than fine — the corrected model puts a landing point inside about two inches given a launch state. The agent's uncertainty budget was explicit about where the error lives. Flight prediction: plus or minus two inches. Rest position after contact: plus or minus fifteen. The entire failure of the simulator is concentrated in the four inches of physics after the bag touches the board.

That budget came with a sensitivity ranking, and it reads like a shopping list. Board-to-bag friction swings the rest position by fifteen inches across its plausible range — three times more than any other coefficient. So the single most valuable thing we can do for scoring accuracy isn't more simulation and isn't a bigger model. It's a Saturday with a regulation board, a stack of bags, and a high-speed camera, measuring friction.

One more finding I want on the record because it reversed my intuition. I assumed spin mattered the way it matters on a golf ball — lift. It doesn't. Magnus force on a bag tops out around three inches of effect. But whether the bag flies flat or on edge changes its air resistance enough to move the landing point twenty-three inches. Spin matters through presentation, not lift. I'd have spent our measurement budget on the wrong quantity.

What I now believe about simulators

The uncomfortable takeaway is that a simulator can be perfectly credible in every observable you check and useless in the one you built it for. We validated flight against intuition and it looked great. Nobody validated the slide, because the slide is where the interesting outcomes come from, and interesting outcomes are the reason you look away from the boring physics that produces them.

So the rules we adopted:

- Reproduce the as-built model first. An audit that can't regenerate your simulator's numbers can't tell you which differences matter. - Never train an outcome head on simulated labels until the sim has survived a physics audit at the regime boundaries. The stages the audit validates are the only stages you may train on. - Spend measurement budget by sensitivity, not curiosity. The tornado chart says friction first, drag second, everything else noise. - Ask the agent for the missing-measurement list, not just the findings. The audit's real output was six experiments, ranked by how much prediction error each one retires.

The simulator took a GPU and a weekend to build. The audit took one prompt and three hours, and it was the better investment. We still don't have a validated scoring predictor — the honest status is that nothing downstream of flight can be trusted until we measure friction and film real throws. But we now know exactly which two measurements buy back the fifteen inches, and we know it before we trained a model on two thousand hole-ins that never happened.