Uncertainty budgets tell you which stage to distrust

One stage held almost all the error

When the July 5th physics audit came back on Quantum Caddy's throw simulator, my first instinct was wholesale: write the asset off. Nearly half the scored outcomes flipped under corrected physics, and every simulated hole-in died. The simulator was a training-data engine, since deprecated, never the production scoring path, but we had planned real work on top of it, and the audit read like a demolition order.

The uncertainty budget in the audit's last section is what stopped me. The session had propagated the uncertainty in every physical coefficient through the model stage by stage instead of end to end. The in-flight prediction, launch to the moment the bag reaches the board, carried about ±2 inches. The prediction of where the bag comes to rest after contact carried ±15. The entire failure of the asset was concentrated in one stage. The flight stage wasn't merely salvageable; it was good, and we kept it as the prior for everything we do next.

I had been doing the opposite for years, and I suspect most teams do. Pipelines get trusted or distrusted as a unit. A model "works" or "doesn't". A data source is "reliable" or "garbage". Error is never actually distributed that way. It clumps, and until you make the clumping visible per stage, you either trust the whole thing or burn the good parts along with the bad.

A budget gives distrust an address

An uncertainty budget is a per-stage accounting of how wrong your pipeline's output could be, given honest ranges on the things each stage doesn't know. Take every coefficient, prior, or assumption a stage leans on. Put an honest interval on each one. Push those intervals through the stage's transformation and record the spread of the output. Repeat for the next stage, then print the intervals next to each other.

Each stage answers one question:

If this stage had perfect inputs, how wrong could its output still be?

That framing separates inherited error from manufactured error. A stage that turns clean inputs into wide outputs is where distrust belongs. A stage that passes wide inputs through at roughly the same width is carrying someone else's problem, and fixing it fixes nothing.

The mechanics don't need to be fancy. Monte-Carlo is fine: sample from each coefficient's range, run the stage a few thousand times, measure the spread. The QC audit did exactly that with the physical coefficients the model leaned on and produced the ±2-versus-±15 split in one evening, with no new data collected.

Three things the budget bought us

• We stopped discarding good stages along with bad ones. Without the budget, the honest summary of the audit is "the simulator is wrong" and the flight model goes in the bin with the contact model. With it, the summary is "the physics is trustworthy until the bag touches the board". The sound stage stayed usable.
• Product claims got phased by stage. We can predict where a throw lands, within a couple of inches, from its launch state. We cannot predict whether it scores until the contact stage is measured. That is a roadmap we can say out loud, X now and Y after measurement, instead of one blended claim that is half true.
• We know where the next data pays. Inside the distrusted stage, the same propagation ranks coefficients by how much output swing each contributes. For us, board-to-bag friction drove roughly three times more rest-position uncertainty than any other coefficient. The first measurement campaign aims there, not at whatever is easiest to instrument.

Stage boundaries live at the hand-offs

Where do the stages come from? Draw the boundary wherever one computation's output becomes the next computation's initial condition, the physical or logical hand-off points. For the throw simulator that seam is the instant the bag first touches the board: ballistics on one side, surface mechanics on the other. The two regimes obey different physics, depend on different coefficients, and deserve different levels of trust, which is exactly what a whole-pipeline error bar erases.

Your pipeline has the same seams even if nothing in it flies. A growth stack hands off from acquisition to scoring to intervention. A data product hands off from ingestion to features to model to decision. Each hand-off is a place where one stage's output gets treated as fact by the next stage, and each is a candidate address for error to live.

Wholesale trust feels natural because the stages share a codebase, a launch date, and an owner. They still don't share epistemic status. Some stages rest on measured quantities, some on literature priors, some on a constant somebody set two years ago and forgot. The budget makes that difference visible on one page.

Two ways budgets rot

The first: computed once and never re-run. A budget is a snapshot of what you didn't know on the day you ran it. Stages get rewritten, coefficients get measured, input distributions drift, and the printed intervals quietly stop describing the pipeline that exists. Ours lives next to the model code as a script and a CSV, so re-running it is one command, and the rule we adopted is that a change to any stage re-runs the budget. A budget that is expensive to refresh will not be refreshed.

The second: budgets computed on toy inputs. Propagating uncertainty through a flattering input distribution produces a flattering budget. Run it on inputs that exercise the regime you'll actually act in. Our flight interval is tight inside the throw envelope we care about; quoted outside that envelope it would be fiction.

And the honest limits, because a budget has them. Ours came from a model-vs-model audit: the coefficient ranges are physics priors and literature values, not measurements, so the intervals are themselves estimates and reality gets a vote later. A budget also can't price an assumption you never named. If a stage hides a choice you don't know you made, no amount of propagation will surface it. The budget locates risk among the uncertainties you listed. It does not retire that risk, and it does not certify that the list is complete.

Apply this

Do this today, on paper, for the pipeline you rely on most. Demand model, risk model, game sim, scoring stack: the shape is the same.

• Draw the stages as boxes. Put a boundary wherever one stage's output becomes the next stage's initial condition. If you can't find the seams, that is itself a finding.
• Under each box, list what the stage assumes: coefficients, priors, thresholds, unmeasured inputs, constants inherited from a tutorial.
• For each stage, answer the budget question in writing: given perfect inputs, how wrong could this stage's output be? Rough orders of magnitude are fine. A gut interval written down beats a precise one never attempted, and ours started as exactly this kind of estimate before the Monte-Carlo confirmed it.
• Print the intervals side by side and look for the clump. The error is never uniform, and the widest stage is rarely the one the team argues about.
• Act on the clump: phase your external claims by stage, aim the next measurement or data purchase at the widest interval, and date the budget so you notice when it goes stale.

If you want the numeric version later, Monte-Carlo through each stage is an afternoon of work. The paper version is twenty minutes and buys most of the insight.