Reading agent output critically

Why this matters

Claude is a confident writer. That’s mostly a feature: it produces fluent prose, structured plans, and answers that read like they came from a senior teammate. But the same fluency that makes the output useful also makes errors invisible — the prose flows past the wrong number without tripping the reader.

The most expensive operator mistake I see is treating fluent output as evidence of correct output. The two are uncorrelated. An agent can fabricate a number with the same conviction it quotes a real one. Without a verification reflex, the wrong number ships.

On Parley Notebook 02, my agent confidently quoted a 92% accuracy number from the field’s easier random-split benchmark when the actual signer-holdout number we cared about was 0.4467 — a 50-point error. The error sounded smart. It would have shipped to a Substack post. The fix was 90 seconds of operator pushback using the prompts in this lesson.

This lesson is the verify-this reflex. It applies anywhere agents produce specific facts: metric reports, datasheet quotes, market sizing, code that “all tests pass,” research summaries.

The four signals of fabrication

Read agent output looking for these four signals. Any one of them should trigger the verify-this reflex.

1. The plausible number. A specific value (8.4%, $12M, 47ms) cited without an artifact path. The number is in the right shape for the domain — that’s exactly why it’s suspicious. Real numbers come with sources.
2. The summarized success. “All N items succeeded.” “All tests pass.” “Everything is working.” Compression hides failure modes. Force enumeration.
3. The confident citation. A reference to a spec, paper, or doc, given with author and year but no quoted sentence. The agent may have inferred from related sources. Quote-or-retract.
4. The smooth handoff. “Done — I’ve updated the file.” No diff shown, no line counts, no verification step. The done-claim sounds confident; the verification is missing. (See Operating 01: When your agent says “done”.)

Each signal has the same root cause: output that summarizes faster than it sources. The agent’s training data rewards fluency, and fluency compresses. Your job as operator is to push back until the compression is undone.

Three ways operators get fooled

The patterns where the verify-this reflex didn’t fire and the wrong thing shipped.

The fix in all three: treat any specific claim as a hypothesis until an artifact backs it. Confidence in the prose is not confidence in the fact.

The verify-this prompt

The reflex you’ll fire dozens of times a week. Save it as a snippet.

For each non-trivial claim in your last response,
provide:

1. The artifact (file path, URL, or command output) that supports it
2. The exact quote or numeric value from that artifact
3. A confidence label:
   - VERIFIED — quoted from artifact
   - INFERRED — derived from related sources, not directly verified
   - GUESSED — your prior, no specific source

If a claim is INFERRED or GUESSED, mark it explicitly. Don't
re-state the claim with new confidence — flag it.

The trick is the three-way confidence label. Most agents will label honestly when the labels are required. The act of writing “INFERRED” or “GUESSED” surfaces what was hidden in the original confident-prose answer.

Parley ASR transcript — fabricated metrics

A real session. Agent confidently quotes a 92% number. Operator pushes back. Real number is ~45%. The whole exchange takes ~90 seconds and saves a public Substack post from shipping wrong.

Prompt kit

Three prompts: verify-this, enumerate-don’t-summarize, cite-or-retract.

For each non-trivial claim in your last response,
provide:

1. The artifact (file path, URL, or command output) that supports it
2. The exact quote or numeric value from that artifact
3. A confidence label:
   - VERIFIED — quoted from artifact
   - INFERRED — derived from related sources, not directly verified
   - GUESSED — your prior, no specific source

If a claim is INFERRED or GUESSED, mark it explicitly. Don't
re-state the claim with new confidence — flag it.

You said "<X> all worked" or "<X> all <verb>." That's a summary.

Re-answer as an enumerated list:
- One row per item
- One column for the metric or outcome of interest
- Mark any row where the outcome was actually a failure / partial /
  unverified

Don't compress to a summary until I've seen the rows.

You cited a specific spec, datasheet, paper, or doc. Quote the
exact sentence and provide the section number or URL.

If you can't quote it directly, retract the claim and tell me
what we'd need to read to verify it.

Apply this — install the verify-this reflex

Ongoing exercise. The reflex is built by repetition. Aim for at least 5 verify-this fires this week.