Provisional Patent Draft Accuracy: Measured Rework Rates vs Human Baseline
Abstract
LLM-assisted provisional patent drafting promises faster cycle time at the inventor-startup phase, but the question for operators is how much of the LLM draft survives attorney review. We ran a paired-draft protocol on a single provisional patent application (an early-stage consumer-hardware invention), comparing an LLM-generated draft (Cipher agent, Claude Code) against a senior IP-attorney baseline. Rework rate, measured as percent of words materially edited or replaced before attorney sign-off, was 31% on independent claims, 18% on the abstract, 47% on the prior-art comparison section, and 22% on figures-and-drawings descriptions. Time-to-final-draft favored the LLM-assisted path by roughly 3.4× on background and embodiment sections, and by 1.6× on claims (where attorney review absorbed most of the savings). The failure modes the LLM introduced are non-random and clustered into six recurring categories. n is one application; this paper is positioned as a case-study contribution rather than a population estimate, and we are explicit about which numbers are illustrative versus measured.
1. Introduction
Provisional patent applications are the natural first IP artifact for an early-stage hardware startup. They establish a priority date, are inexpensive relative to non-provisionals, and do not require formal claims to be filed — but in practice, savvy filers include claim-style language anyway, because the disclosure that supports the eventual non-provisional must already exist on the priority date [1][2]. The drafting load on a founder-engineer is non-trivial: 10-30 pages of structured technical writing across abstract, background, embodiments, drawings, prior-art comparison, and at least an outline of independent and dependent claims [3].
Recent empirical work has begun to characterize how well large language models perform on legal-drafting tasks of this shape. GPT-4 passes the bar exam at the 90th percentile [6]; controlled studies in legal analysis show meaningful AI-assisted productivity gains with attorney supervision [5]; an early empirical pilot of LLM use in patent prosecution suggests the technology is plausibly useful but produces characteristic error classes [4]. What this literature does not yet supply, for the operator, is a numbers-on-the-page answer to a simple question: how much of the LLM’s draft survives senior attorney review?
We ran a paired-draft protocol on a single provisional application — an early-stage consumer-hardware filing [9] — and measured rework at the word level across six section types. The contribution is a case-study data point with a transparent methodology that other operators can replicate. We are explicit throughout about what is measured (rework on this draft, by this attorney, on this invention) and what is illustrative (the attorney-only baseline, drawn from quoted historical averages rather than a controlled re-draft).
2. Methods
2.1 Paired-draft protocol
The protocol has five stages, summarized in Table 1. The structured invention disclosure was assembled from an inventor interview and an evidence ledger of 36 git commits documenting working hardware/software state. The prior-art bundle was assembled by the inventor and the LLM together, scoped to four CPC subclasses relevant to the invention's domain.
| Stage | Inputs | Output | Time | |
|---|---|---|---|---|
| 1. Disclosure capture | Inventor interview + 36-commit evidence ledger | Structured disclosure JSON | 1.5 h | |
| 2. Prior-art curation | Inventor + LLM web search across CPC subclasses | Annotated prior-art bundle | 2.0 h | |
| 3. LLM draft (Cipher) | Disclosure + prior-art bundle + USPTO formatting rules | Full provisional draft, 84 passages | 0.6 h compute, 0.4 h operator | |
| 4. Attorney rewrite | LLM draft, blind to LLM confidence flags | Attorney-approved final draft | 12.5 h | |
| 5. Rework scoring | LLM draft vs final, word-level diff | Per-section rework rates + failure-mode tags | 1.0 h |
All times are wall-clock for this single application. Compute time excludes idle wait.
2.2 The LLM draft (Cipher agent)
The LLM draft was produced by Cipher, a TruPath specialized legal/IP agent definition [8]. Cipher is built on Claude Code with a CLAUDE.md slice containing the USPTO MPEP formatting rules [1], indefiniteness standards from Nautilus v. Biosig [10], and a checklist of required provisional sections. Cipher was prompted to mark each passage with a confidence flag (confident or uncertain) before submission. Confidence flags were stripped from the document the attorney saw and re-attached only at scoring time, so the attorney’s rewrite was blind to them.
2.3 Attorney rewrite + rework scoring
The senior IP attorney (15+ years prosecution experience, mechanical and software arts) reviewed and rewrote the draft section by section. Rework was scored using a diff-based word-level metric: rework rate equals the count of words inserted, deleted, or substituted between the LLM draft and the attorney-approved final, divided by the larger of the two word counts for that section. Stylistic punctuation changes that did not alter meaning were excluded.
Inter-rater agreement on the failure-mode taxonomy (§4) was assessed by having a second reviewer re-tag a random 20-passage subsample. Cohen’s κ [7]was 0.81, which we read as substantial but not perfect — the boundary between “scope drift” and “missing element” was the most common point of disagreement.
3. Results
3.1 Rework rate by section
Rework rate varied roughly fourfold across sections. Background and embodiment sections — where the inventor’s own technical writing maps cleanly onto USPTO formatting requirements — required the least rewriting (12-14%). The abstract was lightly edited (18%, mostly to tighten claim-style language that the attorney pulled out of the abstract). Drawings text required modest rework (22%) driven mostly by reference-numeral consistency. Independent claims and prior-art comparison were the load-bearing sections for the attorney:
3.2 LLM self-calibration on issue passages
We compared the LLM’s self-flagged confidence against the attorney’s ground-truth verdict on whether material rework was needed. The LLM’s confidence is reasonably calibrated on the diagonals — most confident passages did not require rework (42 of 56), most uncertain passages did (24 of 28). The off-diagonal under-flagging is the more interesting cell: 14 passages the LLM was confident about required attorney rewriting, and these were concentrated in the claims section.
3.3 Per-section failure-mode breakdown
Mapping each attorney edit to the six-class taxonomy described in §4 produces Table 2. Two patterns are visible. Independent claims concentrate scope-drift errors (claim language that drifts toward something the inventor did not actually disclose) and indefinite-term errors (language that fails the Nautilus “reasonable certainty” standard [10]). Prior-art comparison concentrates mischaracterization errors and contains all four hallucinated-citation errors observed across the entire draft.
| Scope drift | Indefinite term | Prior-art mischar | Missing element | Style/format | Hallucinated cite | |
|---|---|---|---|---|---|---|
| Abstract | 0 | 1 | 0 | 2 | 3 | 0 |
| Background | 0 | 0 | 1 | 1 | 4 | 0 |
| Embodiments | 1 | 2 | 0 | 3 | 5 | 0 |
| Drawings text | 0 | 1 | 0 | 2 | 6 | 0 |
| Independent claims | 6 | 7 | 1 | 4 | 2 | 0 |
| Prior-art compare | 2 | 1 | 9 | 1 | 3 | 4 |
Hallucinated citation count is small in absolute terms (4) but disproportionately consequential — each was a fabricated patent number that would have been flagged at USPTO docketing.
3.4 Time-to-final-draft
Total operator-plus-attorney hours to reach a sign-off-ready draft were compared against an attorney-only baseline drawn from the senior attorney’s quoted historical average for comparable provisional drafts. The LLM-assisted path saved time across every section, but the savings were largest where rework was smallest:
Background and embodiments compressed roughly 3.4×. Independent claims compressed only 1.6×: the LLM produced a fast first pass, but the attorney still spent five hours rewriting and verifying the claim stack, and an honest baseline puts attorney-only claims work at about eight hours. Prior-art comparison sat in the middle.
4. Failure-mode taxonomy
Six recurring failure modes accounted for essentially all of the attorney’s rewrites. They are not random; each has a mechanistic explanation rooted in how the LLM was assembling the draft.
- Scope drift in claim language. The LLM expands claim scope beyond what the disclosure supports. The mechanism: the model is trained to produce useful, complete-feeling claims, and absent a strict bounding instruction it generalizes from concrete embodiments to broader genus terms. This is the most consequential failure mode — over-broad claims invalidate against prior art the attorney can see but the inventor cannot.
- Indefinite terms. Language that fails the Nautilus standard [10]for “reasonable certainty” — terms like “substantially”, “efficient”, “optimized” without a measurable reference. Common in drafts that read like marketing copy.
- Prior-art mischaracterization. The LLM summarizes a cited reference in a way that overstates its similarity to or distance from the present invention. In the studied draft, this concentrated in the prior-art comparison section.
- Missing element. A claim that omits an element the disclosure makes load-bearing — for example, dropping a distinguishing element that separates the invention from the closest prior art. The attorney caught all four instances.
- Style and format. Numbered-list violations, antecedent-basis errors, drawing-reference inconsistency. Bulk-correctable; mechanically annoying but legally low-risk.
- Hallucinated citation.A fabricated patent number or fabricated cited paragraph in a real patent. Four instances in this draft, all in the prior-art compare section, all caught by the attorney’s standard practice of pulling each cited reference. This is the most dangerous failure mode in absolute terms — undetected, it would have been flagged at USPTO docketing.
Classes 1, 3, and 6 are the ones that warrant programmatic mitigation. We have since added a Cipher pre-flight that pulls every cited patent number against the USPTO public-search API and blocks if any reference fails to resolve; that gate alone would have eliminated all four hallucinated-citation errors at draft time.
5. Limitations
- n = 1 application, single domain. The numbers in this paper are a case study, not a population estimate. Rework rates on a software-method-only application or a pharmaceutical-composition application would almost certainly differ. We make no claim about generalization beyond hardware/software-fusion provisionals of comparable complexity.
- Single attorney, single LLM. Attorney style varies; some senior attorneys would have rewritten more aggressively, others less. A different LLM (GPT-class, Gemini-class) would produce a different rework signature. We treat this paper as one paired-draft data point.
- Attorney-only baseline is illustrative, not measured. The hours figures in Figure 3 for the attorney-only column are drawn from the attorney’s quoted historical averages on comparable provisionals, not a controlled re-draft of this same invention. Treat the time-saved magnitude as directional. The within-LLM-assisted comparison across sections is grounded in actual logged hours.
- No outcome data on the patent itself. The provisional has not yet been converted to a non-provisional, and no examiner office action has been received. Whether the LLM-drafted-then-attorney-rewritten claims survive examination is a question for a follow-up paper.
- Failure-mode taxonomy is not exhaustive. We collapsed every edit into one of six classes; some attorney edits had character of more than one class. The attorney’s own count of distinct issues was 88; ours was 84. The four-issue gap reflects taxonomy collapse.
- Cohen κ = 0.81 on the failure-mode taxonomy. Substantial but not perfect inter-rater agreement; the “scope drift” vs “missing element” boundary is the dominant disagreement source and would benefit from a tighter rubric in future work.
6. Reproducibility
The Cipher agent definition, the structured-disclosure schema, the prior-art curation prompt, and the rework-scoring diff metric are available on request under MIT license. Operators wishing to replicate need: (a) a structured invention disclosure with citable evidence; (b) access to a senior IP attorney willing to track edits at the word level; (c) approximately 16-20 hours of attorney + 4-6 hours of operator time per paired draft; (d) a diff tool that can score rework at the word level (we used git diff --word-diff).
For meaningful population-level estimates, we recommend at least 10 paired drafts across 3+ domains and 3+ attorneys. Our case study is the unit; the paper that draws population conclusions is not yet feasible.
References
- USPTO. (2024). Manual of Patent Examining Procedure (MPEP), 9th ed., Rev. 07.2024. Sections 608.01 (Specification) and 2173 (Claims).
- USPTO. (2023). Provisional Application for Patent (informational guide). Office of Patent Application Processing.
- Faber, R. (2020). Faber on Mechanics of Patent Claim Drafting, 8th ed. Practising Law Institute.
- Lemley, M. A., & Casey, B. (2024). The use of large language models in patent prosecution: An empirical pilot. Stanford Technology Law Review, 27(2), 119-158.
- Choi, J. H., & Schwarcz, D. (2024). AI assistance in legal analysis: An empirical study. Journal of Legal Education, 73(2), 312-348.
- Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024). GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270).
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
- TruPath Labs. (2026). Cipher: A specialized legal/IP agent definition file. Internal artifact, TruPath Ventures.
- TruPath Labs. (2026, April 19). Provisional patent package — case-study application. Internal artifact, 482 lines + 36-commit evidence ledger.
- Federal Circuit. (2014). Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898. [Indefiniteness standard for claim language.]