Run the sensitivity tornado before you measure anything

Friction moved the answer three times more than anything else

After the July 5th audit killed our simulator's outcome labels, the obvious next move for Quantum Caddy was a measurement program, and my instinct for building one was terrible. I wanted to measure everything at once, starting with whatever was easiest, and weighing bags is very easy. The audit's final artifact replaced that instinct with an ordering. The session had taken every coefficient the corrected physics model relied on but could not measure, propagated an honest uncertainty range for each through the model one at a time, and ranked the resulting swings in the observable we actually cared about: where the bag comes to rest on the board.

Friction swung the rest position by 14.9 inches, and the entire uncertainty budget on that observable was about plus or minus 15 inches. The runner-up coefficient moved it roughly a third as much. Mass, the quantity easiest to measure and the one everyone instinctively reaches for first, was negligible.

Sorted and drawn as bars, that ranking is a tornado chart, and a tornado chart reads as a spending plan. The first dollar of our measurement budget goes to a friction rig, not to more simulation and not to a scale. Without the chart I would have spent a week measuring things that could not change the prediction and called it progress.

The tornado in four steps

Nothing in the method needs physics.

• Pick the decision observable. Not an intermediate quantity, the number a real decision turns on. For us that was rest position, because scoring turns on rest position. The landing point in the air was already tight to about two inches, and no decision improved by tightening it further.
• Write an honest range for every uncertain coefficient. The range you would defend to a skeptical reviewer, sourced from literature, physical limits, or the spread across whatever references you trust. Write the source next to the range.
• Propagate one at a time. Hold everything else at its nominal value, run the model at both ends of one coefficient's range, and record how far the decision observable moves. Repeat per coefficient.
• Rank the swings and draw the chart. The top bar is where your next measurement dollar goes. The bottom bars are precision you already have and should stop paying for.

If the model runs at all, this is an afternoon. Ours came free, as a by-product of an audit session that had the model built anyway.

Your other models have tornadoes too

The word "coefficient" makes this sound like lab equipment, but the tornado is a property of any model with assumed numbers in it, whether or not anyone has drawn it:

• A venue demand model runs on conversion-rate and repeat-visit assumptions, and one of them probably dominates the revenue projection. We carry exactly this kind of model for Mile High Golf. I have not run its tornado yet; writing this lesson put it on the calendar.
• A retention model runs on churn elasticity, meaning how much a price change or a feature loss actually moves cancellations. If that range is wide, precision anywhere else in the model is wasted.
• A credit book runs on default correlation. Teams polish individual default probabilities to three decimals while the correlation assumption, which drives the tail loss, stays a round number nobody has defended.

The chart does two jobs at once. It names the assumption that is load-bearing, and it names the precision that is decoration. Most measurement programs get budgeted by curiosity or convenience. The tornado budgets by consequence.

Three ways to build a garbage tornado

• Optimistic ranges. Propagate the ranges you hope are true and the tornado ranks your hopes. The ranges have to be the defend-them-out-loud kind, which is why writing a source next to each one is part of the method and not paperwork.
• Ranking on an intermediate observable. Run our sweep on the in-air landing point instead of the rest position and every bar shrinks, friction never dominates, and the chart is tidy and useless. Rank on the number the decision consumes, even when the intermediate one is easier to compute.
• Treating the chart as static. A tornado describes one version of the model. Change the model's structure and the sensitivities move, sometimes drastically, because fixing one stage can push all the remaining uncertainty into another. Re-run the sweep after any structural change; a stale tornado directs money with last quarter's map.

Where one-at-a-time lies to you

One honest limit, and it matters. Moving one coefficient while holding the rest at nominal misses interactions. If two coefficients are strongly coupled, meaning the effect of one depends on where the other sits, one-at-a-time bars can understate their combined contribution, and in bad cases the true dominant pair never shows up as a dominant bar. In our model, friction and restitution both act in the same instant of contact, so coupling was plausible. We trusted the simple sweep anyway because the separation was roughly threefold, and no plausible interaction closes a gap that wide.

When your top bars sit close together, or you have a physical or economic reason to expect coupling, upgrade the analysis: vary all the coefficients at once across their ranges and attribute the output variance to each input and to their interactions. The runs cost more; the idea is the same. The tornado is the cheap screening pass that tells you whether the expensive version is worth commissioning.

And one boundary on the whole method. The tornado ranks uncertainty inside your model. If the model's structure is wrong, the chart will confidently rank the coefficients of a fiction. We had exactly that problem one layer down, which is what lesson 32 is about.

Apply this

Build one tornado this week. One model, five coefficients.

• Pick the model whose output feeds your nearest real decision: the demand model behind a hire, or the sim behind a training set.
• Name the decision observable and write one sentence on which decision consumes it. If you cannot write the sentence, you picked an intermediate observable; go one step downstream.
• List the model's five most uncertain coefficients and write a range for each that you could defend out loud, with the source noted beside it.
• Run both ends of each range, one coefficient at a time, and record the swing in the decision observable. Ten runs.
• Sort the swings, draw the bars, and act on both ends of the chart: schedule a measurement or data pull for the top bar, and cancel whatever effort was queued for the bottom one.