Skip to content
SearchSubscribe
Expert · Lesson 02 — Real ROI from agent hierarchies
E02Expert
Expert · Lesson 02● live

Real ROI from agent hierarchies

When 6 agents are right, when 1 is enough, when more agents make you slower.

20 min read · 60 min auditprereq: Operating 10 (agent routing model)

Why agent ROI is hard to see

Agent-mediated work feels productive in a specific way that’s easy to mistake for output. Sessions feel like progress. The conversation reads engaging. The artifacts pile up in folders. Months in, the question is hard to answer honestly. Am I shipping more, or am I shipping the same amount with more steps?

Three illusions make the answer unclear. Activity gets counted as output. Specialist count gets read as capability. Token cost gets treated as the only cost. Each illusion produces a different kind of waste — and each requires a different kind of measurement to catch.

The discipline below is what separates real ROI from apparent ROI: shipped artifacts per session, scope-cap on specialists, wall-clock comparison to solo work.Three numbers, one comparison, run quarterly. That’s the audit.

Three tiers of agent ROI

Where each agent in your stack tends to land. The tier shifts over time as scope, briefs, and your own habits evolve — that’s why the audit is recurring, not one-shot.

TierSignalAction
Carrying weight2-4 sessions per artifact, <10% net-negative, 5+ shipped artifacts/monthKeep. Document why it’s working — that’s reusable knowledge for future agents.
Marginal4-6 sessions per artifact, 10-20% net-negative, 2-5 shipped artifacts/monthTighten the brief. Re-audit in 2 weeks. If unchanged, fold scope into a stronger agent.
Net-negative>6 sessions per artifact, >20% net-negative, or <2 artifacts/monthRetire or fold immediately. The agent is costing you more than it ships.

Most operators have one of each. Two carrying-weight agents handling the bulk; two marginal that need brief tightening; one net-negative that should be folded. That distribution is healthy. Six carrying-weight agents is rare and usually means measurement is too generous.

Three ROI illusions

The patterns that make agent stacks feel productive while underperforming. Hover any card to see the diagnosis.

№ 01

The activity-counts-as-output illusion

claim looks like“"I had 14 agent sessions this week — clearly the system is paying back."”
what’s missingSessions aren't output. Output is shipped artifacts (commits merged, documents finalized, decisions logged). 14 sessions with 2 shipped artifacts is a 7:1 thrash ratio — usually a sign the agents are spinning, not producing.
the moveTrack shipped artifacts per session, not session count. Healthy ratio is 1-2 sessions per shipped artifact. Above 4:1 means something is wrong — agents redoing work, briefs unclear, or scope drift.
№ 02

The specialist sprawl illusion

claim looks like“"Adding more specialists makes me faster." Operator goes from 3 agents to 9 over six months.”
what’s missingPast ~6 specialists, routing overhead exceeds specialization gain. You spend more time deciding which agent to invoke than the agent saves you. Worse, the agents start to overlap — and now there's ambiguity about which one owns what.
the moveHard-cap specialists at 6-7. Each must have a non-overlapping scope and a trigger word that's unambiguous. If you can't write a one-sentence description that excludes other agents' scopes, you don't need a new agent — you need a clearer one of the existing ones.
№ 03

The hidden-cost illusion

claim looks like“"Agents are basically free — I'm on a flat plan."”
what’s missingToken cost is one component. The real costs are: time spent briefing, time spent reviewing, time spent fixing what shipped wrong, and the opportunity cost of work the agent did that you'd have skipped if you'd known its true price. Most agent-heavy operators underestimate review time by 3-5x.
the moveTrack wall-clock time per shipped artifact: brief + render + review + revise + ship. Compare to the same artifact done solo. If agent-mediated takes longer, the agent isn't paying back on that class of work.

The fix in all three: measure shipped artifacts, cap specialists at 6, count wall-clock time honestly. Without these, agent ROI is whatever feels true — which drifts toward optimistic over time.

The agent-cost audit

The diagnostic. One prompt, run quarterly. Surfaces which agents are carrying weight, which are marginal, and which should be folded.

The agent-cost audit
Run an honest agent-cost audit on the last 30 days.

Pull these numbers per agent:
1. Shipped artifacts (commits merged, docs finalized,
   decisions logged) — count
2. Sessions — count from token logs
3. Sessions-per-artifact ratio (healthy: 1-4)
4. Estimated wall-clock review time per artifact

Then spot-check 5 random sessions per agent and grade:
- Net positive (faster / better than solo)
- Net neutral (similar to solo)
- Net negative (slower / worse than solo)

For any agent with >20% net-negative sessions, propose
either a brief tightening or a scope-folding into an
existing agent. Show your reasoning so I can decide.

Three numbers, a spot-check, and a recommendation per agent. ~1 hour total.

The audit doesn’t produce decisions on its own — it produces evidence. The decision is yours, but with the audit done, the decision is informed instead of vibe-based. The discipline is to actually act on the audit findings.Operators who run the audit but don’t fold or tighten end up with the same agent stack at every audit, slightly worse each time.

Annotated transcript

A real conversation: operator runs the audit, finds 3:1 healthy thrash but discovers two net-negative sessions in a recently-added specialist. Click any pinned annotation to expand the “why this turn matters” commentary.

Honest ROI audit on a 6-agent stack

Three operator turns, three agent turns. Result: 5 healthy agents validated, 1 marginal agent flagged for brief tightening + 2-week re-audit.

  1. operator

    I'm trying to figure out if my 6-agent setup is actually paying back. Feels productive but I haven't measured. How do I do this honestly?

  2. agent

    Three numbers, then a comparison.

    Numbers (last 30 days):
    1. Shipped artifacts (commits merged, documents finalized, decisions logged) — count per agent
    2. Sessions per agent — count from your token usage logs
    3. Wall-clock review time per artifact — your honest estimate, not stopwatch

    Comparison: for the highest-volume artifact class, estimate how long the same work would have taken solo. Be honest — "I couldn't have done it at all" is sometimes the right answer; "I would have done it in half the time" is sometimes also the right answer.

    This takes about an hour. The numbers will surprise you.

  3. operator

    Ran the numbers. Last 30 days: 47 shipped artifacts across 6 agents, 142 sessions total, ~3 sessions per artifact. Review time average 25 minutes per artifact. Solo comparison for the highest-volume class (postmortems): I would have shipped 4-6 instead of 14, and they'd be lower quality.

  4. agent

    Two real findings:

    1. 3 sessions per shipped artifact is healthy. Below 2:1 means you're under-iterating; above 4:1 means agents are spinning. You're in the sweet spot.
    2. Postmortems are paying back 2-3x. Same operator, same hours, ~3x the artifacts at higher quality. That's the agent system carrying real weight.

    One caveat — you didn't measure the failure mode artifacts. Sessions that produced worse output than solo would have. If those exist, they reduce the apparent ROI. Spot-check 5 random sessions: were any of them net-negative? If so, what did they have in common?

  5. operator

    Spot-checked 5. Two were net-negative — agent ran far past scope and I had to redo significant portions. Both were the same agent (a specialist I added 6 weeks ago).

  6. agent

    That's actionable. Two paths:

    1. The specialist's brief is too broad. Tighten the scope and the trigger words. Net-negative sessions usually come from agents trying to cover too much ground.
    2. The specialist isn't needed. If the work fits inside an existing agent's scope, fold it in. Specialist count above ~6 starts paying negative — you're hitting that boundary.

    My recommendation: tighten the brief first, audit again in 2 weeks. If net-negative sessions drop, keep the agent. If they don't, fold the role into the closest existing specialist and watch the routing overhead drop.

ROI accounting template

A spreadsheet (or markdown table) you fill in once a quarter. Save the template; the data accumulates over time and the trends are more useful than any single audit.

Audit period: <YYYY-MM-DD to YYYY-MM-DD>

Per-agent table:

AGENT      | SHIPPED | SESSIONS | RATIO | NET-NEG % | TIER
-----------+---------+----------+-------+-----------+----------------
agent-A    |    14   |    32    |  2.3  |    7%     | carrying weight
agent-B    |     9   |    28    |  3.1  |    11%    | carrying weight
agent-C    |     6   |    21    |  3.5  |    14%    | marginal
agent-D    |     4   |    19    |  4.8  |    21%    | marginal
agent-E    |     3   |    22    |  7.3  |    27%    | NET-NEGATIVE
agent-F    |    11   |    20    |  1.8  |    5%     | carrying weight

Comparison classes (per highest-volume artifact type):
- <type>: solo would have shipped ~<N>; with agents shipped <M>
- Quality delta: <better / same / worse>; brief explanation

Decisions this audit:
- Keep: <agents>
- Tighten brief on: <agents>; specific changes:
- Fold/retire: <agents>; absorbing into:

Re-audit date: <2 weeks for tightenings; next quarter otherwise>

The trend across audits is more useful than any single audit.An agent that’s gone marginal-to-net-negative across three quarters has a structural problem; one that bounces around is responding to your work shifts and is probably fine.

Prompt kit

Three prompts for auditing, evaluating new agent proposals, and retiring underperformers. Save in your CLAUDE.md or a personal snippets file.

The agent-cost audit
Run an honest agent-cost audit on the last 30 days.

Pull these numbers per agent:
1. Shipped artifacts (commits merged, docs finalized,
   decisions logged) — count
2. Sessions — count from token logs
3. Sessions-per-artifact ratio (healthy: 1-4)
4. Estimated wall-clock review time per artifact

Then spot-check 5 random sessions per agent and grade:
- Net positive (faster / better than solo)
- Net neutral (similar to solo)
- Net negative (slower / worse than solo)

For any agent with >20% net-negative sessions, propose
either a brief tightening or a scope-folding into an
existing agent. Show your reasoning so I can decide.
Test whether a new specialist is needed
I'm thinking about adding a new specialist for <role>.

Before I add it, walk me through:
1. What's the one-sentence scope description? Does it
   exclude every existing agent's scope?
2. What are the unambiguous trigger words / phrases? Do
   any overlap with existing agents?
3. What's the smallest experiment that would test
   whether this role pays back? (Usually: route <type
   of work> to an existing agent for 2 weeks, compare
   shipped artifacts.)
4. If the experiment doesn't pay back, what's the
   reversal cost?

If scope or triggers overlap, or if the experiment is
hard to run, the answer is probably "tighten an
existing agent" — not "add a new one."
Decide whether to retire an agent
Audit <agent name> over the last 60 days.

Three signals that argue for retirement:
1. Net-negative session rate >20% even after one
   round of brief tightening
2. Shipped-artifacts count <5 per month (low utilization)
3. Sessions-per-artifact >5 (unhealthy thrash)

If two or more signals are present, propose retirement
plan: which existing agent absorbs the scope, what
moves into that agent's brief, what the trigger
remapping looks like. Don't actually retire — just
draft the plan so I can review.

Apply this — your first ROI audit

~1 hour exercise, run quarterly. Three numbers per agent, one spot-check, one decision per marginal agent.

Your first agent ROI audit

Each step takes 10-15 minutes. Progress saves automatically.

0/5
  1. 01Pull your shipped-artifact count and session count for each agent over the last 30 days.Honest counts. "Shipped" means merged / finalized / logged — not drafted.
  2. 02Compute sessions-per-artifact per agent. Flag any above 4 or below 1.5.High thrash means agents are spinning. Low ratio means you're under-iterating; quality may be lower than you think.
  3. 03Spot-check 5 random sessions per agent. Grade each net-positive, net-neutral, or net-negative vs solo.Honest grading. Net-negative sessions are the hidden cost most operators miss.
  4. 04For any agent with >20% net-negative or >5 sessions-per-artifact, draft a brief-tightening or scope-folding proposal.Use the third prompt in the kit. Don't act yet — let the proposal sit a day, then review with fresh eyes.
  5. 05Run the audit again in 2-4 weeks. Confirm the changes moved the numbers, or revert.Without re-measurement, every audit becomes an opinion. The re-measurement is what makes this real engineering.
Expert tier · what's next

After this lesson