
Real ROI from agent hierarchies
When 6 agents are right, when 1 is enough, when more agents make you slower.
Why agent ROI is hard to see
Agent-mediated work feels productive in a specific way that’s easy to mistake for output. Sessions feel like progress. The conversation reads engaging. The artifacts pile up in folders. Months in, the question is hard to answer honestly. Am I shipping more, or am I shipping the same amount with more steps?
Three illusions make the answer unclear. Activity gets counted as output. Specialist count gets read as capability. Token cost gets treated as the only cost. Each illusion produces a different kind of waste — and each requires a different kind of measurement to catch.
The discipline below is what separates real ROI from apparent ROI: shipped artifacts per session, scope-cap on specialists, wall-clock comparison to solo work.Three numbers, one comparison, run quarterly. That’s the audit.
Three tiers of agent ROI
Where each agent in your stack tends to land. The tier shifts over time as scope, briefs, and your own habits evolve — that’s why the audit is recurring, not one-shot.
| Tier | Signal | Action |
|---|---|---|
| Carrying weight | 2-4 sessions per artifact, <10% net-negative, 5+ shipped artifacts/month | Keep. Document why it’s working — that’s reusable knowledge for future agents. |
| Marginal | 4-6 sessions per artifact, 10-20% net-negative, 2-5 shipped artifacts/month | Tighten the brief. Re-audit in 2 weeks. If unchanged, fold scope into a stronger agent. |
| Net-negative | >6 sessions per artifact, >20% net-negative, or <2 artifacts/month | Retire or fold immediately. The agent is costing you more than it ships. |
Most operators have one of each. Two carrying-weight agents handling the bulk; two marginal that need brief tightening; one net-negative that should be folded. That distribution is healthy. Six carrying-weight agents is rare and usually means measurement is too generous.
Three ROI illusions
The patterns that make agent stacks feel productive while underperforming. Hover any card to see the diagnosis.
The activity-counts-as-output illusion
The specialist sprawl illusion
The hidden-cost illusion
The fix in all three: measure shipped artifacts, cap specialists at 6, count wall-clock time honestly. Without these, agent ROI is whatever feels true — which drifts toward optimistic over time.
The agent-cost audit
The diagnostic. One prompt, run quarterly. Surfaces which agents are carrying weight, which are marginal, and which should be folded.
Run an honest agent-cost audit on the last 30 days.
Pull these numbers per agent:
1. Shipped artifacts (commits merged, docs finalized,
decisions logged) — count
2. Sessions — count from token logs
3. Sessions-per-artifact ratio (healthy: 1-4)
4. Estimated wall-clock review time per artifact
Then spot-check 5 random sessions per agent and grade:
- Net positive (faster / better than solo)
- Net neutral (similar to solo)
- Net negative (slower / worse than solo)
For any agent with >20% net-negative sessions, propose
either a brief tightening or a scope-folding into an
existing agent. Show your reasoning so I can decide.Three numbers, a spot-check, and a recommendation per agent. ~1 hour total.
The audit doesn’t produce decisions on its own — it produces evidence. The decision is yours, but with the audit done, the decision is informed instead of vibe-based. The discipline is to actually act on the audit findings.Operators who run the audit but don’t fold or tighten end up with the same agent stack at every audit, slightly worse each time.
Annotated transcript
A real conversation: operator runs the audit, finds 3:1 healthy thrash but discovers two net-negative sessions in a recently-added specialist. Click any pinned annotation to expand the “why this turn matters” commentary.
Three operator turns, three agent turns. Result: 5 healthy agents validated, 1 marginal agent flagged for brief tightening + 2-week re-audit.
- operator
I'm trying to figure out if my 6-agent setup is actually paying back. Feels productive but I haven't measured. How do I do this honestly?
- agent
Three numbers, then a comparison.
Numbers (last 30 days):
1. Shipped artifacts (commits merged, documents finalized, decisions logged) — count per agent
2. Sessions per agent — count from your token usage logs
3. Wall-clock review time per artifact — your honest estimate, not stopwatchComparison: for the highest-volume artifact class, estimate how long the same work would have taken solo. Be honest — "I couldn't have done it at all" is sometimes the right answer; "I would have done it in half the time" is sometimes also the right answer.
This takes about an hour. The numbers will surprise you.
- operator
Ran the numbers. Last 30 days: 47 shipped artifacts across 6 agents, 142 sessions total, ~3 sessions per artifact. Review time average 25 minutes per artifact. Solo comparison for the highest-volume class (postmortems): I would have shipped 4-6 instead of 14, and they'd be lower quality.
- agent
Two real findings:
1. 3 sessions per shipped artifact is healthy. Below 2:1 means you're under-iterating; above 4:1 means agents are spinning. You're in the sweet spot.
2. Postmortems are paying back 2-3x. Same operator, same hours, ~3x the artifacts at higher quality. That's the agent system carrying real weight.One caveat — you didn't measure the failure mode artifacts. Sessions that produced worse output than solo would have. If those exist, they reduce the apparent ROI. Spot-check 5 random sessions: were any of them net-negative? If so, what did they have in common?
- operator
Spot-checked 5. Two were net-negative — agent ran far past scope and I had to redo significant portions. Both were the same agent (a specialist I added 6 weeks ago).
- agent
That's actionable. Two paths:
1. The specialist's brief is too broad. Tighten the scope and the trigger words. Net-negative sessions usually come from agents trying to cover too much ground.
2. The specialist isn't needed. If the work fits inside an existing agent's scope, fold it in. Specialist count above ~6 starts paying negative — you're hitting that boundary.My recommendation: tighten the brief first, audit again in 2 weeks. If net-negative sessions drop, keep the agent. If they don't, fold the role into the closest existing specialist and watch the routing overhead drop.
ROI accounting template
A spreadsheet (or markdown table) you fill in once a quarter. Save the template; the data accumulates over time and the trends are more useful than any single audit.
Audit period: <YYYY-MM-DD to YYYY-MM-DD>
Per-agent table:
AGENT | SHIPPED | SESSIONS | RATIO | NET-NEG % | TIER
-----------+---------+----------+-------+-----------+----------------
agent-A | 14 | 32 | 2.3 | 7% | carrying weight
agent-B | 9 | 28 | 3.1 | 11% | carrying weight
agent-C | 6 | 21 | 3.5 | 14% | marginal
agent-D | 4 | 19 | 4.8 | 21% | marginal
agent-E | 3 | 22 | 7.3 | 27% | NET-NEGATIVE
agent-F | 11 | 20 | 1.8 | 5% | carrying weight
Comparison classes (per highest-volume artifact type):
- <type>: solo would have shipped ~<N>; with agents shipped <M>
- Quality delta: <better / same / worse>; brief explanation
Decisions this audit:
- Keep: <agents>
- Tighten brief on: <agents>; specific changes:
- Fold/retire: <agents>; absorbing into:
Re-audit date: <2 weeks for tightenings; next quarter otherwise>
The trend across audits is more useful than any single audit.An agent that’s gone marginal-to-net-negative across three quarters has a structural problem; one that bounces around is responding to your work shifts and is probably fine.
Prompt kit
Three prompts for auditing, evaluating new agent proposals, and retiring underperformers. Save in your CLAUDE.md or a personal snippets file.
Run an honest agent-cost audit on the last 30 days.
Pull these numbers per agent:
1. Shipped artifacts (commits merged, docs finalized,
decisions logged) — count
2. Sessions — count from token logs
3. Sessions-per-artifact ratio (healthy: 1-4)
4. Estimated wall-clock review time per artifact
Then spot-check 5 random sessions per agent and grade:
- Net positive (faster / better than solo)
- Net neutral (similar to solo)
- Net negative (slower / worse than solo)
For any agent with >20% net-negative sessions, propose
either a brief tightening or a scope-folding into an
existing agent. Show your reasoning so I can decide.I'm thinking about adding a new specialist for <role>.
Before I add it, walk me through:
1. What's the one-sentence scope description? Does it
exclude every existing agent's scope?
2. What are the unambiguous trigger words / phrases? Do
any overlap with existing agents?
3. What's the smallest experiment that would test
whether this role pays back? (Usually: route <type
of work> to an existing agent for 2 weeks, compare
shipped artifacts.)
4. If the experiment doesn't pay back, what's the
reversal cost?
If scope or triggers overlap, or if the experiment is
hard to run, the answer is probably "tighten an
existing agent" — not "add a new one."Audit <agent name> over the last 60 days.
Three signals that argue for retirement:
1. Net-negative session rate >20% even after one
round of brief tightening
2. Shipped-artifacts count <5 per month (low utilization)
3. Sessions-per-artifact >5 (unhealthy thrash)
If two or more signals are present, propose retirement
plan: which existing agent absorbs the scope, what
moves into that agent's brief, what the trigger
remapping looks like. Don't actually retire — just
draft the plan so I can review.Apply this — your first ROI audit
~1 hour exercise, run quarterly. Three numbers per agent, one spot-check, one decision per marginal agent.
Your first agent ROI audit
Each step takes 10-15 minutes. Progress saves automatically.
- 01Pull your shipped-artifact count and session count for each agent over the last 30 days.Honest counts. "Shipped" means merged / finalized / logged — not drafted.
- 02Compute sessions-per-artifact per agent. Flag any above 4 or below 1.5.High thrash means agents are spinning. Low ratio means you're under-iterating; quality may be lower than you think.
- 03Spot-check 5 random sessions per agent. Grade each net-positive, net-neutral, or net-negative vs solo.Honest grading. Net-negative sessions are the hidden cost most operators miss.
- 04For any agent with >20% net-negative or >5 sessions-per-artifact, draft a brief-tightening or scope-folding proposal.Use the third prompt in the kit. Don't act yet — let the proposal sit a day, then review with fresh eyes.
- 05Run the audit again in 2-4 weeks. Confirm the changes moved the numbers, or revert.Without re-measurement, every audit becomes an opinion. The re-measurement is what makes this real engineering.