Reading a diff before you accept it

Why diff review beats trust

When an agent produces a diff and tests pass, the easy move is to ship. Tests are green; agent says done; you move on. That habit is wrong about 20-30% of the time, and the 20-30% includes most of the bugs that bite in production.

Tests cover what somebody thought to test. They don’t cover the global log-level that just got set to DEBUG, the fixed buffer that doesn’t bound-check, the watchdog that didn’t get fed, the dead event that nobody consumes. Those bugs survive the test suite intact. They surface in production, where the cost of debug is 100x what it would have been at review.

Diff review isn’t a code-quality concern; it’s a cost-of-rework concern. Every minute spent on the 4-pass review is buying you 10-100 minutes of avoided field debug.

The QC ESP32 firmware diff in this lesson’s transcript is real. It passed all 24 tests. The 4-pass review found 6 issues, 3 of which would have hard-faulted boards in the field. Reading the diff took 7 minutes.

The 4-pass review

Four passes, each looking for something different. Run them in order — earlier passes scope the later ones.

1. Pass 1 — File list. Does the diff touch the files you expected? Anything unexpectedly missing (a Kconfig entry, a migration, a config file)? Anything unexpectedly present (a vendored dependency, a config-file mutation that looks unrelated)? This pass is mechanical and takes 30 seconds.
2. Pass 2 — Control flow. For each changed function, walk entry → body → exit. Are error returns checked? Are loops bounded? Are watchdogs fed in tight loops? Are exceptions either handled or propagated, not swallowed? This is where most logic bugs live.
3. Pass 3 — Side effects. What changes in the world? Files written, events emitted, global state set, network calls, flash writes. Are emitted events actually consumed? Are global settings (log level, timezone, locale) being changed quietly? Side effects are where production surprises live.
4. Pass 4 — Security. New secrets, credentials, env reads. New external endpoints. Auth boundaries crossed. Input bounds and injection points. This pass is fast on most diffs but catches the highest-severity findings when it does catch.

The 4-pass review takes 5-10 minutes on a normal diff. On firmware, ML training pipelines, or schema migrations, it can take longer. It’s the cheapest defense against shipping the agent’s confidence as your own.

Three diff-review failures

The pattern across all three: review is structural work, not vibes work. Skimming for “does this look right” misses the bugs that look exactly like what was already there.

The “what changed” prompt

The 4-pass review as a paste-able prompt:

Run a 4-pass review of this diff. For each pass,
list findings with severity (BLOCK / FLAG / NIT).

**Pass 1 — File list**
- Files changed (count + paths)
- Match against the plan / intent of this change?
- Anything unexpectedly missing or unexpectedly present?

**Pass 2 — Control flow**
- For each new or changed function: entry → body → exit
- Are error returns checked? Are loops bounded? Are watchdogs fed?
- Any swallowed exceptions or silently-ignored return values?

**Pass 3 — Side effects**
- What changes in the world? (files written, events emitted, state mutated)
- Are emitted events consumed? Are global settings changed?
- Network, flash, db writes — anything new?

**Pass 4 — Security**
- New secrets, credentials, env reads?
- New external endpoints, auth boundaries crossed?
- Buffer / bounds / injection issues — any unchecked input sizes?

Output as a markdown report. Final line: BLOCK / APPROVE-WITH-NITS / APPROVE.

The trick is the BLOCK / FLAG / NIT severity labels. The agent has to commit to a final-line verdict. That commitment surfaces issues that a vibes-y summary would have hidden in qualifications.

QC ESP32 firmware diff — 4-pass review

A real diff: 161 lines of new sensor stubs for the QC radar/ToF sensor strategy reset. All 24 tests passed. The 4-pass review surfaced 6 issues, 3 blocking.

Prompt kit

Three prompts: the 4-pass review, function-level deltas, and the “what tests didn’t catch” challenge.

Run a 4-pass review of this diff. For each pass,
list findings with severity (BLOCK / FLAG / NIT).

**Pass 1 — File list**
- Files changed (count + paths)
- Match against the plan / intent of this change?
- Anything unexpectedly missing or unexpectedly present?

**Pass 2 — Control flow**
- For each new or changed function: entry → body → exit
- Are error returns checked? Are loops bounded? Are watchdogs fed?
- Any swallowed exceptions or silently-ignored return values?

**Pass 3 — Side effects**
- What changes in the world? (files written, events emitted, state mutated)
- Are emitted events consumed? Are global settings changed?
- Network, flash, db writes — anything new?

**Pass 4 — Security**
- New secrets, credentials, env reads?
- New external endpoints, auth boundaries crossed?
- Buffer / bounds / injection issues — any unchecked input sizes?

Output as a markdown report. Final line: BLOCK / APPROVE-WITH-NITS / APPROVE.

For the diff at <branch> vs main, walk every changed function and
output:

| function | file | change-type (added/modified/deleted) | one-line behavior delta |

A behavior delta describes what the function does NOW that's different
from what it did BEFORE. If the change is purely cosmetic (rename,
formatting), say "no behavior change."

For the changes in this diff, list 5 plausible bugs that:
- The tests would NOT catch (because the test mocks them out, doesn't
  exercise the path, or doesn't assert the right thing)
- Could plausibly happen in production
- Would be expensive to debug after deploy

For each, suggest the smallest test or assertion that would catch it.

Apply this — install the 4-pass habit

Ongoing exercise. Habit installs in ~10 reviews. Aim for at least 5 this week.