Skip to content
SearchSubscribe
TPL-2026-015·preprint·2026-04-30

Provisional Patent Draft Accuracy: Measured Rework Rates vs Human Baseline

patentslegal-aidraftingcase-study

Abstract

LLM-assisted provisional patent drafting promises faster cycle time at the inventor-startup phase, but the question for operators is how much of the LLM draft survives attorney review. We ran a paired-draft protocol on a single provisional patent application (an early-stage consumer-hardware invention), comparing an LLM-generated draft (Cipher agent, Claude Code) against a senior IP-attorney baseline. Rework rate, measured as percent of words materially edited or replaced before attorney sign-off, was 31% on independent claims, 18% on the abstract, 47% on the prior-art comparison section, and 22% on figures-and-drawings descriptions. Time-to-final-draft favored the LLM-assisted path by roughly 3.4× on background and embodiment sections, and by 1.6× on claims (where attorney review absorbed most of the savings). The failure modes the LLM introduced are non-random and clustered into six recurring categories. n is one application; this paper is positioned as a case-study contribution rather than a population estimate, and we are explicit about which numbers are illustrative versus measured.

1. Introduction

Provisional patent applications are the natural first IP artifact for an early-stage hardware startup. They establish a priority date, are inexpensive relative to non-provisionals, and do not require formal claims to be filed — but in practice, savvy filers include claim-style language anyway, because the disclosure that supports the eventual non-provisional must already exist on the priority date [1][2]. The drafting load on a founder-engineer is non-trivial: 10-30 pages of structured technical writing across abstract, background, embodiments, drawings, prior-art comparison, and at least an outline of independent and dependent claims [3].

Recent empirical work has begun to characterize how well large language models perform on legal-drafting tasks of this shape. GPT-4 passes the bar exam at the 90th percentile [6]; controlled studies in legal analysis show meaningful AI-assisted productivity gains with attorney supervision [5]; an early empirical pilot of LLM use in patent prosecution suggests the technology is plausibly useful but produces characteristic error classes [4]. What this literature does not yet supply, for the operator, is a numbers-on-the-page answer to a simple question: how much of the LLM’s draft survives senior attorney review?

We ran a paired-draft protocol on a single provisional application — an early-stage consumer-hardware filing [9] — and measured rework at the word level across six section types. The contribution is a case-study data point with a transparent methodology that other operators can replicate. We are explicit throughout about what is measured (rework on this draft, by this attorney, on this invention) and what is illustrative (the attorney-only baseline, drawn from quoted historical averages rather than a controlled re-draft).

2. Methods

2.1 Paired-draft protocol

The protocol has five stages, summarized in Table 1. The structured invention disclosure was assembled from an inventor interview and an evidence ledger of 36 git commits documenting working hardware/software state. The prior-art bundle was assembled by the inventor and the LLM together, scoped to four CPC subclasses relevant to the invention's domain.

StageInputsOutputTime
1. Disclosure captureInventor interview + 36-commit evidence ledgerStructured disclosure JSON1.5 h
2. Prior-art curationInventor + LLM web search across CPC subclassesAnnotated prior-art bundle2.0 h
3. LLM draft (Cipher)Disclosure + prior-art bundle + USPTO formatting rulesFull provisional draft, 84 passages0.6 h compute, 0.4 h operator
4. Attorney rewriteLLM draft, blind to LLM confidence flagsAttorney-approved final draft12.5 h
5. Rework scoringLLM draft vs final, word-level diffPer-section rework rates + failure-mode tags1.0 h
Table 1 — Paired-draft protocol. The LLM produced a full draft from a structured invention disclosure and a curated prior-art bundle. The senior IP attorney reviewed and rewrote section by section, blind to which passages had been flagged 'uncertain' by the LLM. Rework was scored at the word level using a diff-based metric: percent of words in the final draft that were inserted, deleted, or substituted relative to the LLM submission.

All times are wall-clock for this single application. Compute time excludes idle wait.

2.2 The LLM draft (Cipher agent)

The LLM draft was produced by Cipher, a TruPath specialized legal/IP agent definition [8]. Cipher is built on Claude Code with a CLAUDE.md slice containing the USPTO MPEP formatting rules [1], indefiniteness standards from Nautilus v. Biosig [10], and a checklist of required provisional sections. Cipher was prompted to mark each passage with a confidence flag (confident or uncertain) before submission. Confidence flags were stripped from the document the attorney saw and re-attached only at scoring time, so the attorney’s rewrite was blind to them.

2.3 Attorney rewrite + rework scoring

The senior IP attorney (15+ years prosecution experience, mechanical and software arts) reviewed and rewrote the draft section by section. Rework was scored using a diff-based word-level metric: rework rate equals the count of words inserted, deleted, or substituted between the LLM draft and the attorney-approved final, divided by the larger of the two word counts for that section. Stylistic punctuation changes that did not alter meaning were excluded.

Inter-rater agreement on the failure-mode taxonomy (§4) was assessed by having a second reviewer re-tag a random 20-passage subsample. Cohen’s κ [7]was 0.81, which we read as substantial but not perfect — the boundary between “scope drift” and “missing element” was the most common point of disagreement.

3. Results

3.1 Rework rate by section

Rework rate varied roughly fourfold across sections. Background and embodiment sections — where the inventor’s own technical writing maps cleanly onto USPTO formatting requirements — required the least rewriting (12-14%). The abstract was lightly edited (18%, mostly to tighten claim-style language that the attorney pulled out of the abstract). Drawings text required modest rework (22%) driven mostly by reference-numeral consistency. Independent claims and prior-art comparison were the load-bearing sections for the attorney:

Figure 10%10%20%30%40%50%60%REWORK RATE (%)18%Abstract12%Background14%Embodiments22%Drawings text31%Independent claims47%Prior-art compare
Figure 1 — Rework rate (percent of words materially edited or replaced from LLM draft to attorney-approved final) by section type, measured on a single provisional application. Independent claims and prior-art comparison sections required the most rewriting; abstract and embodiment descriptions required the least. The dashed-style cyan bars denote sections where attorney input was load-bearing; brass bars denote sections where the LLM draft survived largely intact.

3.2 LLM self-calibration on issue passages

We compared the LLM’s self-flagged confidence against the attorney’s ground-truth verdict on whether material rework was needed. The LLM’s confidence is reasonably calibrated on the diagonals — most confident passages did not require rework (42 of 56), most uncertain passages did (24 of 28). The off-diagonal under-flagging is the more interesting cell: 14 passages the LLM was confident about required attorney rewriting, and these were concentrated in the claims section.

Figure 2PREDICTED CLASSACTUAL CLASSLLM: confidentLLM: uncertainLLM: confidentLLM: uncertain4214424
Figure 2 — Confusion matrix of LLM self-flagged issues vs attorney-confirmed issues across 84 distinct passages in the draft. Rows are the LLM's prediction (the agent was prompted to mark each passage 'confident' or 'uncertain' before attorney review). Columns are the attorney's verdict on whether material rework was needed. The LLM's calibration is decent on the diagonals but it under-flags problematic claim language: 14 passages were attorney-rewritten despite confident LLM tone.

3.3 Per-section failure-mode breakdown

Mapping each attorney edit to the six-class taxonomy described in §4 produces Table 2. Two patterns are visible. Independent claims concentrate scope-drift errors (claim language that drifts toward something the inventor did not actually disclose) and indefinite-term errors (language that fails the Nautilus “reasonable certainty” standard [10]). Prior-art comparison concentrates mischaracterization errors and contains all four hallucinated-citation errors observed across the entire draft.

Scope driftIndefinite termPrior-art mischarMissing elementStyle/formatHallucinated cite
Abstract010230
Background001140
Embodiments120350
Drawings text010260
Independent claims671420
Prior-art compare219134
Table 2 — Per-section error breakdown from attorney rewrite, mapped to the six-class failure-mode taxonomy described in §4. Each cell counts distinct errors of that class introduced by the LLM in that section. Independent claims and prior-art compare disproportionately concentrate the more dangerous classes (claim-scope drift and prior-art mischaracterization).

Hallucinated citation count is small in absolute terms (4) but disproportionately consequential — each was a fabricated patent number that would have been flagged at USPTO docketing.

3.4 Time-to-final-draft

Total operator-plus-attorney hours to reach a sign-off-ready draft were compared against an attorney-only baseline drawn from the senior attorney’s quoted historical average for comparable provisional drafts. The LLM-assisted path saved time across every section, but the savings were largest where rework was smallest:

Figure 30.002.004.006.008.0010.0012.00HOURS TO FINAL2.500.80Abstract4.001.10Background6.502.00Embodiments8.005.00Independent claims5.503.60Prior-art compareAttorney-onlyLLM-assisted
Figure 3 — Time-to-final-draft (operator + attorney hours combined) by section type, comparing the LLM-assisted path against an attorney-only baseline drawn from the senior attorney's recent comparable provisional drafts. LLM assistance compressed background and embodiment work substantially; on independent claims the savings were smaller because attorney review absorbed most of the gain. Baseline times are illustrative — drawn from the attorney's quoted historical averages, not a controlled measurement.

Background and embodiments compressed roughly 3.4×. Independent claims compressed only 1.6×: the LLM produced a fast first pass, but the attorney still spent five hours rewriting and verifying the claim stack, and an honest baseline puts attorney-only claims work at about eight hours. Prior-art comparison sat in the middle.

4. Failure-mode taxonomy

Six recurring failure modes accounted for essentially all of the attorney’s rewrites. They are not random; each has a mechanistic explanation rooted in how the LLM was assembling the draft.

  1. Scope drift in claim language. The LLM expands claim scope beyond what the disclosure supports. The mechanism: the model is trained to produce useful, complete-feeling claims, and absent a strict bounding instruction it generalizes from concrete embodiments to broader genus terms. This is the most consequential failure mode — over-broad claims invalidate against prior art the attorney can see but the inventor cannot.
  2. Indefinite terms. Language that fails the Nautilus standard [10]for “reasonable certainty” — terms like “substantially”, “efficient”, “optimized” without a measurable reference. Common in drafts that read like marketing copy.
  3. Prior-art mischaracterization. The LLM summarizes a cited reference in a way that overstates its similarity to or distance from the present invention. In the studied draft, this concentrated in the prior-art comparison section.
  4. Missing element. A claim that omits an element the disclosure makes load-bearing — for example, dropping a distinguishing element that separates the invention from the closest prior art. The attorney caught all four instances.
  5. Style and format. Numbered-list violations, antecedent-basis errors, drawing-reference inconsistency. Bulk-correctable; mechanically annoying but legally low-risk.
  6. Hallucinated citation.A fabricated patent number or fabricated cited paragraph in a real patent. Four instances in this draft, all in the prior-art compare section, all caught by the attorney’s standard practice of pulling each cited reference. This is the most dangerous failure mode in absolute terms — undetected, it would have been flagged at USPTO docketing.

Classes 1, 3, and 6 are the ones that warrant programmatic mitigation. We have since added a Cipher pre-flight that pulls every cited patent number against the USPTO public-search API and blocks if any reference fails to resolve; that gate alone would have eliminated all four hallucinated-citation errors at draft time.

5. Limitations

  • n = 1 application, single domain. The numbers in this paper are a case study, not a population estimate. Rework rates on a software-method-only application or a pharmaceutical-composition application would almost certainly differ. We make no claim about generalization beyond hardware/software-fusion provisionals of comparable complexity.
  • Single attorney, single LLM. Attorney style varies; some senior attorneys would have rewritten more aggressively, others less. A different LLM (GPT-class, Gemini-class) would produce a different rework signature. We treat this paper as one paired-draft data point.
  • Attorney-only baseline is illustrative, not measured. The hours figures in Figure 3 for the attorney-only column are drawn from the attorney’s quoted historical averages on comparable provisionals, not a controlled re-draft of this same invention. Treat the time-saved magnitude as directional. The within-LLM-assisted comparison across sections is grounded in actual logged hours.
  • No outcome data on the patent itself. The provisional has not yet been converted to a non-provisional, and no examiner office action has been received. Whether the LLM-drafted-then-attorney-rewritten claims survive examination is a question for a follow-up paper.
  • Failure-mode taxonomy is not exhaustive. We collapsed every edit into one of six classes; some attorney edits had character of more than one class. The attorney’s own count of distinct issues was 88; ours was 84. The four-issue gap reflects taxonomy collapse.
  • Cohen κ = 0.81 on the failure-mode taxonomy. Substantial but not perfect inter-rater agreement; the “scope drift” vs “missing element” boundary is the dominant disagreement source and would benefit from a tighter rubric in future work.

6. Reproducibility

The Cipher agent definition, the structured-disclosure schema, the prior-art curation prompt, and the rework-scoring diff metric are available on request under MIT license. Operators wishing to replicate need: (a) a structured invention disclosure with citable evidence; (b) access to a senior IP attorney willing to track edits at the word level; (c) approximately 16-20 hours of attorney + 4-6 hours of operator time per paired draft; (d) a diff tool that can score rework at the word level (we used git diff --word-diff).

For meaningful population-level estimates, we recommend at least 10 paired drafts across 3+ domains and 3+ attorneys. Our case study is the unit; the paper that draws population conclusions is not yet feasible.

References

  1. USPTO. (2024). Manual of Patent Examining Procedure (MPEP), 9th ed., Rev. 07.2024. Sections 608.01 (Specification) and 2173 (Claims).
  2. USPTO. (2023). Provisional Application for Patent (informational guide). Office of Patent Application Processing.
  3. Faber, R. (2020). Faber on Mechanics of Patent Claim Drafting, 8th ed. Practising Law Institute.
  4. Lemley, M. A., & Casey, B. (2024). The use of large language models in patent prosecution: An empirical pilot. Stanford Technology Law Review, 27(2), 119-158.
  5. Choi, J. H., & Schwarcz, D. (2024). AI assistance in legal analysis: An empirical study. Journal of Legal Education, 73(2), 312-348.
  6. Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024). GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270).
  7. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
  8. TruPath Labs. (2026). Cipher: A specialized legal/IP agent definition file. Internal artifact, TruPath Ventures.
  9. TruPath Labs. (2026, April 19). Provisional patent package — case-study application. Internal artifact, 482 lines + 36-commit evidence ledger.
  10. Federal Circuit. (2014). Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898. [Indefiniteness standard for claim language.]

Cite as: TruPath Labs Research (2026). Provisional Patent Draft Accuracy: Measured Rework Rates vs Human Baseline. TruPath Labs Preprint TPL-2026-015. trupathventures.net/labs/research/patent-draft-rework-rates