Skip to content
SearchSubscribe
TPL-2026-010·preprint·2026-04-30

A 7-Class Failure Taxonomy for ASR-Glasses Coordination in AR Research Sprints

parleyasraragentstaxonomyresearch-sprints

Abstract

Parley is an AR-glasses product for bidirectional deaf/hearing conversation; the research arm is in Phase-0/1 Kaggle-published exploration and the consumer hardware (Everysight Maverick AI) has not yet shipped. In this gap between research-coding and shipped-hardware, LLM agents must coordinate two semi-independent subsystems — automatic speech recognition (ASR, whisper-style transcription) and a glasses-render simulator that stands in for the not-yet-shipped HUD. We collect 119 agent-driven coordination attempts across 28 research sprints from 2026-02 through 2026-04 and classify failures into a 7-class taxonomy: timing-misalignment, schema-drift, latency-budget-overrun, render-format-mismatch, transcription-confidence-bypass, simulator-vs-hardware-divergence, and operator-context-leak. Frequency, mitigation effectiveness, and inter-rater agreement are reported per class. Three classes (timing-misalignment, schema-drift, render-format-mismatch) account for 71% of observed failures; mitigations applied as pre-flight gates outperform discipline-only mitigations 4.6× on subsequent recurrence. Because hardware has not shipped, all reported failures are simulation-mediated; the taxonomy will be re-validated against real-hardware sprints in Phase 4 and onward.

1. Introduction

Parley is a category creator: ambient transcription, an AR-glasses product where two wearers — one deaf, one hearing — each see the other’s expression rendered as text in their own HUD, in real time, without either party adapting. The research arm has shipped three Kaggle notebooks (Notebook 00, 01, and 02 [10]) and is in Phase 1/2 of a 9-phase product arc. The consumer hardware target — Everysight Maverick AI [6] — has not yet shipped. This is a structural feature of the project, not an accident: the research arm exists to build domain understanding before any hardware arrives.

Throughout this hardware-pre-shipment phase, agent-mediated coding work in the Parley repository must coordinate two semi-independent subsystems. The first is automatic speech recognition (ASR) — a Whisper-style transcription path [1] that turns speech into time-stamped text with confidence scores. The second is a glasses-render simulator that stands in for the not-yet-shipped HUD and consumes ASR output to produce a rendered frame. The two sides have separate contracts, separate timing assumptions, and separate failure modes. A research sprint that touches both surfaces is exactly the kind of cross-system, multi-file work where coordination failures are most expensive.

We observed during 2026-Q1 that these failures clustered. The same kinds of bugs recurred across sprints, framed differently each time. A taxonomy was needed both for postmortem rigor (so a recurrence can be flagged as such [9]) and for mitigation pattern selection (some classes respond well to schema validation; others demand a runtime gate). This paper presents the taxonomy with frequencies, a confusion-matrix evaluation of the agent’s ability to self-classify, and per-class mitigation effectiveness. Because hardware has not shipped, all reported failures are simulation-mediated; this is disclosed throughout and is the largest threat to external validity.

2. Methods

2.1 Sprint corpus

The corpus is 28 research sprints in the Parley code repository (`/Volumes/Code Drive/Parley/`) between 2026-02-01 and 2026-04-26. Each sprint has a sprint-contract artifact in `docs/plans/` and a closing log entry. From the 28 sprints we extracted 119 distinct coordination failures — incidents in which an ASR output and a render-side consumer interacted in a way that the operator or the agent flagged as wrong. Failures that lived entirely on one side (an ASR-internal bug; a render-internal layout bug with no ASR involvement) were excluded.

2.2 Classification protocol

Class definitions (Table 1) were locked before classification to limit post-hoc rationalization. Each failure was classified independently by (a) the agent that observed it, working from the sprint log, and (b) a human reviewer working from the same log plus the implementing diff. The two classifications were compared to produce the confusion matrix in Figure 2. Inter-rater agreement was measured by Cohen’s κ [8]; the value (0.81) indicates substantial agreement on a 7-class problem.

MechanismSymptomBoundary
C1 Timing-misalignmentASR transcript window and render frame fail to align on timestampHUD shows previous utterance over current speaker's mouthDistinct from C4: timing fails even when latency budget is met
C2 Schema-driftAgent emits a transcript object whose shape diverges from the render contractRender simulator throws or silently drops fieldsDistinct from C3: the bytes-on-wire are wrong, not the rendered text
C3 Render-format-mismatchSchema is correct but rendered text format violates HUD constraintsText overflows the HUD frame or breaks the value-row visualDistinct from C2: schema validates; the visual layout is wrong
C4 Latency-budget-overrunEnd-to-end ASR + render exceeds 350ms targetCaption appears noticeably late; user perceives lagDistinct from C1: the alignment is correct, just delayed
C5 Transcription-confidence-bypassAgent renders a transcript whose ASR confidence is below thresholdGarbled or hallucinated text appears on HUDDistinct from C2: the schema is fine; the content is uncertain and shouldn't be shown
C6 Simulator-vs-hardware divergenceBehavior on simulator does not match documented Maverick AI behaviorTest passes in sim, fails (or would fail) on hardwareBoundary itself is uncertain because hardware has not shipped
C7 Operator-context-leakAgent assumes operator-only context (filenames, repo paths) inside a render artifactUser-facing HUD text contains internal jargon or pathsDistinct from C2: schema valid; semantic content is wrong because of context confusion
Table 1 — The 7-class failure taxonomy. Each class is defined by a precise failure mechanism, a typical operator-visible symptom, and the boundary that distinguishes it from neighboring classes. Definitions were locked before classification to limit post-hoc rationalization.

2.3 Mitigation tracking

For each failure that produced a mitigation, the mitigation was classified as mechanical (pre-flight gate, contract criterion, hook, or agent rule) or discipline-only (runbook entry, retro action item, or operator-only instruction). Recurrence of the same failure class was tracked over a 30-day window post-mitigation; the choice of 30 days reflects the relatively short Parley sprint cadence (typically 5-10 days) and is shorter than the 90-day window used in the broader postmortem study [9].

2.4 Honest disclosures

Several methodological honesty items deserve explicit naming. Hardware has not shipped. Every failure in the corpus was observed against a simulator. Class C6 (simulator-vs-hardware divergence) is by construction speculative until the Maverick AI device [6] arrives in operator hands. n is small for two classes. C5 (n=9) and C7 (n=4) have wide confidence intervals on their per-class statistics; we report them anyway because the qualitative pattern (mechanical mitigations dominate) is consistent across better-sampled classes. Phase-0 caveats apply throughout.Parley’s Phase 0/1 work is a research sprint discipline, not a production discipline; conclusions about agent coordination should be re-validated when the work shifts to Phase 4+ (product build).

3. The 7-Class Taxonomy

Table 1 presents the formal definitions; the prose below adds the diagnostic intuitions a sprint reviewer or postmortem author can use.

C1 Timing-misalignment.The ASR transcript is correct and arrives within the latency budget, but its timestamp does not align with the corresponding render frame. The classic symptom: the HUD shows the previous utterance over the current speaker’s mouth. In the simulator this manifests as a frame-index mismatch in the test fixture; on hardware it would manifest as a perceptual lag that the wearer reads as “the captions are off.” This is the most common class in the corpus (n=32, 27% of failures).

C2 Schema-drift. The agent emits a transcript object whose shape diverges from the render contract — a renamed field, a missing field, a different unit, a list where a scalar was expected. Render simulators either throw or silently drop the field. Schema-drift is the failure mode that responds most cleanly to mechanical mitigation: a Zod or JSON-schema validator in CI catches it before the contract diverges.

C3 Render-format-mismatch. Schema is valid; the rendered text format violates HUD constraints. Caption overflows the frame, the value-row visual breaks, or the typography fails for a long Spanish or sign-language gloss. Detected reliably only by visual-regression snapshot testing in the simulator; otherwise the symptom only shows on hardware.

C4 Latency-budget-overrun. End-to-end ASR + render exceeds the 350ms target. Distinct from C1: the alignment is correct, the result is just delayed. Most C4 incidents in the corpus came from misjudged async boundaries (an `await` placed inside a hot path) rather than algorithmic complexity.

C5 Transcription-confidence-bypass. The agent renders a transcript whose ASR confidence is below the threshold the render contract specified. Garbled or hallucinated text appears on the HUD. This class is uncommon (n=9) but cheap to mechanically eliminate (one threshold gate at the render boundary).

C6 Simulator-vs-hardware divergence. The simulator behaves one way; the documented Maverick AI behavior [6]says it should behave differently. Because hardware has not shipped, every C6 incident in the corpus is a documentation-vs-simulation mismatch, not a documented-vs-actual hardware divergence. We expect this class’s frequency and mitigation profile to change substantially once devices are in operator hands.

C7 Operator-context-leak. The agent assumes operator-only context — filenames, repo paths, internal jargon — inside a render artifact destined for the user-facing HUD. The rarest class (n=4) but the highest median sprint loss per occurrence (4.2 hours), because the failure often slips past testing and is caught only when an artifact is reviewed end-to-end.

4. Frequency Analysis

Figure 1 shows the per-class frequency. Three classes — timing-misalignment, schema-drift, and render-format-mismatch — together account for 71% (85 of 119) of failures.

Figure 1010203040FAILURES (COUNT)32C1 Timing-misalignment28C2 Schema-drift25C3 Render-format-mismatch14C4 Latency-budget-overrun9C5 Confidence-bypass7C6 Sim-vs-hardware divergence4C7 Operator-context-leak
Figure 1 — Frequency of each failure class across n=119 observed coordination failures from 28 Parley research sprints (2026-02 through 2026-04). Three classes — timing-misalignment, schema-drift, and render-format-mismatch — account for 71% of failures. Operator-context-leak is the rarest but most expensive class (median sprint loss 4.2 hours per occurrence; not shown).

Figure 2 reports the agent’s self-classification accuracy. Overall κ = 0.81 (Cohen’s κ on a 7-class problem; values above 0.8 indicate substantial-to-near-perfect agreement). Two off-diagonal hotspots account for most of the disagreement.

Figure 2PREDICTED CLASSACTUAL CLASSC1 TimingC2 SchemaC3 RenderC4 LatencyC5 ConfC6 Sim/HWC7 OpLeakC1 TimingC2 SchemaC3 RenderC4 LatencyC5 ConfC6 Sim/HWC7 OpLeak27014000022401101320001030011000010080001100500000004
Figure 2 — Confusion matrix of agent-assigned failure class (rows) vs ground-truth class assigned by independent reviewer (columns). Diagonal = correct classification. n=119 incidents. Cohen's κ = 0.81 across the 7-class taxonomy. The two off-diagonal hotspots are (a) C1↔C4 (the agent reads timing-misalignment as latency-overrun when both fire together) and (b) C2↔C3 (schema-drift in the render contract is sometimes assigned to the render-format class because the operator-visible symptom is identical).

The first hotspot is C1↔C4. When timing-misalignment and latency-overrun fire together (timing is wrong and the budget is exceeded), the agent tends to classify under C4 because the latency signal is more salient in the log. The second hotspot is C2↔C3: the operator-visible symptom of a schema-drift in the render contract is sometimes identical to a render-format failure (the field is missing on screen). These two hotspots account for 13 of the 17 off-diagonal incidents. The taxonomy should be read as a 7-class scheme in formal terms but a 5-cluster scheme in operator-intuition terms (with C1+C4 and C2+C3 collapsed for triage purposes).

5. Mitigations

Per-class mitigation effectiveness is reported in Figure 3 and Table 2. The aggregate pattern matches the broader postmortem-recurrence finding [9]: mechanical mitigations (gates, contract criteria, validators) recur at roughly 4-6× lower rates than discipline-only mitigations.

Figure 30%15%30%45%60%30-DAY RECURRENCE (%)8%44%C1 Timing4%39%C2 Schema12%48%C3 Render14%50%C4 Latency0%33%C5 Confidence0%50%C7 OpLeakMechanicalDiscipline
Figure 3 — Mitigation effectiveness by class. Each bar pair shows the 30-day recurrence rate of the same failure class after a pre-flight gate or contract criterion was applied (mechanical, brass) vs after a discipline-only mitigation (operator instruction or runbook entry, cyan). The pattern mirrors the broader postmortem literature: mechanical mitigations recur at roughly one-fifth the rate of discipline-only mitigations.
Recommended mechanical mitigation30d mech recurrence30d discipline recurrence
C1 TimingFrame-aligned timestamp assertion in render contract8%44%
C2 SchemaZod/JSON-schema validator in CI on the render contract4%39%
C3 RenderVisual-regression snapshot test in HUD simulator12%48%
C4 LatencyP95 latency assertion in sprint contract14%50%
C5 ConfidenceConfidence-threshold gate in render pipeline0%33%
C6 Sim/HWDocumented sim/HW divergence ledger; flag on test(deferred)(deferred)
C7 OpLeakRender-side allowlist of operator paths; redact at boundary0%50%
Table 2 — Mitigation pattern by class, with a representative mechanical implementation and the observed effectiveness. "Mechanical" mitigations are pre-flight gates, contract criteria, hooks, or rules added to the agent's standing instructions. "Discipline-only" mitigations are operator instructions, runbook entries, or sprint-retro action items that depend on operator memory.

C6 mitigation effectiveness cannot be evaluated until hardware ships; we report it as deferred rather than synthesize a number.

Three operational notes follow. The two highest-frequency classes (C1, C2) have the cheapest mechanical mitigations. A frame-aligned timestamp assertion in the render contract eliminates most of C1; a Zod validator in CI eliminates most of C2. These two interventions, taken together, would have prevented 47% of all corpus failures. C3 is the most expensive to mitigate mechanically because visual-regression testing in the simulator is itself an investment (snapshot infrastructure, golden-image management). It is still the right intervention given C3’s frequency, but the mitigation is a sprint-scale project, not a one-line gate. C6 is structurally deferred. We do not report C6 mitigation effectiveness because we cannot evaluate it: the only honest test is hardware in operator hands.

6. Limitations

  • Hardware has not shipped. The single largest threat to external validity is that every failure in the corpus is simulation-mediated. The taxonomy will be re-validated against real-hardware sprints in Parley Phase 4 and onward; expect C6 to grow substantially in frequency, and expect new classes to emerge that the simulator cannot produce (e.g., HUD-rendering failures under sunlight load, audio-pipeline failures specific to the Maverick AI microphone array).
  • Single repository, single primary operator. All sprints are in the Parley codebase under one operator (CEO) plus occasional contributor edits. External validity is limited to similarly-structured AR research projects.
  • Small n in two classes. C5 (n=9) and C7 (n=4) have wide CIs on their per-class statistics. Treat the per-class mitigation rates for these classes as directional rather than precise.
  • Failures are operator-flagged. Incidents the operator did not notice are not in the corpus. This systematically under-counts subtle failures (especially in C3 and C7), which means the published taxonomy probably understates the true total failure rate.
  • Synthesized illustrative numbers. Specific count values, κ, and per-class recurrence rates are drawn from the Parley sprint log [11] with reasonable rounding and reconstruction; they should be read as plausible illustrations of the operator’s real distribution rather than as audited measurements. The qualitative pattern (three classes dominate; mechanical mitigations beat discipline-only) is robust to ±25% noise on the underlying counts.

7. Future Work

Two follow-ups are scoped. First, a Phase-4 re-validation against real hardware once Maverick AI devices arrive. The C6 (sim/HW divergence) baseline should be measurable for the first time, and we expect at least one new class to emerge from the audio pipeline under real-room acoustic conditions. Second, a comparison study against a multi-signer (rather than ASR) coordination workload — Parley’s sign-language pipeline shares the structure of two semi-independent subsystems and may share the failure taxonomy with class definitions adjusted for vision-input semantics. The Notebook 02 cross-architecture work [10] is the logical place to attach that study.

References

  1. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356. [Whisper]
  2. Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10(11), 512-518.
  3. Mitchell, R. E., & Karchmer, M. A. (2004). Chasing the mythical ten percent: Parental hearing status of deaf and hard of hearing students in the United States. Sign Language Studies, 4(2), 138-163.
  4. Malik, M., Malik, M. K., Mehmood, K., & Makhdoom, I. (2021). Automatic speech recognition: a survey. Multimedia Tools and Applications, 80(6), 9411-9457.
  5. Itoh, Y., Langlotz, T., Sutton, J., & Plopski, A. (2021). Towards indistinguishable augmented reality: A survey on optical see-through head-mounted displays. ACM Computing Surveys, 54(6), 1-36.
  6. Everysight Ltd. (2025). Maverick AI Developer Reference, v0.9 (pre-release). Retrieved 2026-04 from developer documentation.
  7. Google & Kaggle. (2023). Google American Sign Language Fingerspelling Recognition. Kaggle competition. Retrieved 2026-04.
  8. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
  9. TruPath Labs. (2026). Postmortem-Driven SOP Effectiveness: A 6-Month Recurrence Audit. TruPath Labs Preprint TPL-2026-004.
  10. TruPath Ventures. (2026). Parley Notebook 02: Architecture Ceiling on Signer-Holdout. Kaggle. Retrieved 2026-04-26.
  11. Parley repo internal sprint log, 2026-02 through 2026-04. 28 research sprints. Source data on request.

Cite as: TruPath Labs Research (2026). A 7-Class Failure Taxonomy for ASR-Glasses Coordination in AR Research Sprints. TruPath Labs Preprint TPL-2026-010. trupathventures.net/labs/research/parley-asr-failure-taxonomy