A 7-Class Failure Taxonomy for ASR-Glasses Coordination in AR Research Sprints
Abstract
Parley is an AR-glasses product for bidirectional deaf/hearing conversation; the research arm is in Phase-0/1 Kaggle-published exploration and the consumer hardware (Everysight Maverick AI) has not yet shipped. In this gap between research-coding and shipped-hardware, LLM agents must coordinate two semi-independent subsystems — automatic speech recognition (ASR, whisper-style transcription) and a glasses-render simulator that stands in for the not-yet-shipped HUD. We collect 119 agent-driven coordination attempts across 28 research sprints from 2026-02 through 2026-04 and classify failures into a 7-class taxonomy: timing-misalignment, schema-drift, latency-budget-overrun, render-format-mismatch, transcription-confidence-bypass, simulator-vs-hardware-divergence, and operator-context-leak. Frequency, mitigation effectiveness, and inter-rater agreement are reported per class. Three classes (timing-misalignment, schema-drift, render-format-mismatch) account for 71% of observed failures; mitigations applied as pre-flight gates outperform discipline-only mitigations 4.6× on subsequent recurrence. Because hardware has not shipped, all reported failures are simulation-mediated; the taxonomy will be re-validated against real-hardware sprints in Phase 4 and onward.
1. Introduction
Parley is a category creator: ambient transcription, an AR-glasses product where two wearers — one deaf, one hearing — each see the other’s expression rendered as text in their own HUD, in real time, without either party adapting. The research arm has shipped three Kaggle notebooks (Notebook 00, 01, and 02 [10]) and is in Phase 1/2 of a 9-phase product arc. The consumer hardware target — Everysight Maverick AI [6] — has not yet shipped. This is a structural feature of the project, not an accident: the research arm exists to build domain understanding before any hardware arrives.
Throughout this hardware-pre-shipment phase, agent-mediated coding work in the Parley repository must coordinate two semi-independent subsystems. The first is automatic speech recognition (ASR) — a Whisper-style transcription path [1] that turns speech into time-stamped text with confidence scores. The second is a glasses-render simulator that stands in for the not-yet-shipped HUD and consumes ASR output to produce a rendered frame. The two sides have separate contracts, separate timing assumptions, and separate failure modes. A research sprint that touches both surfaces is exactly the kind of cross-system, multi-file work where coordination failures are most expensive.
We observed during 2026-Q1 that these failures clustered. The same kinds of bugs recurred across sprints, framed differently each time. A taxonomy was needed both for postmortem rigor (so a recurrence can be flagged as such [9]) and for mitigation pattern selection (some classes respond well to schema validation; others demand a runtime gate). This paper presents the taxonomy with frequencies, a confusion-matrix evaluation of the agent’s ability to self-classify, and per-class mitigation effectiveness. Because hardware has not shipped, all reported failures are simulation-mediated; this is disclosed throughout and is the largest threat to external validity.
2. Methods
2.1 Sprint corpus
The corpus is 28 research sprints in the Parley code repository (`/Volumes/Code Drive/Parley/`) between 2026-02-01 and 2026-04-26. Each sprint has a sprint-contract artifact in `docs/plans/` and a closing log entry. From the 28 sprints we extracted 119 distinct coordination failures — incidents in which an ASR output and a render-side consumer interacted in a way that the operator or the agent flagged as wrong. Failures that lived entirely on one side (an ASR-internal bug; a render-internal layout bug with no ASR involvement) were excluded.
2.2 Classification protocol
Class definitions (Table 1) were locked before classification to limit post-hoc rationalization. Each failure was classified independently by (a) the agent that observed it, working from the sprint log, and (b) a human reviewer working from the same log plus the implementing diff. The two classifications were compared to produce the confusion matrix in Figure 2. Inter-rater agreement was measured by Cohen’s κ [8]; the value (0.81) indicates substantial agreement on a 7-class problem.
| Mechanism | Symptom | Boundary | |
|---|---|---|---|
| C1 Timing-misalignment | ASR transcript window and render frame fail to align on timestamp | HUD shows previous utterance over current speaker's mouth | Distinct from C4: timing fails even when latency budget is met |
| C2 Schema-drift | Agent emits a transcript object whose shape diverges from the render contract | Render simulator throws or silently drops fields | Distinct from C3: the bytes-on-wire are wrong, not the rendered text |
| C3 Render-format-mismatch | Schema is correct but rendered text format violates HUD constraints | Text overflows the HUD frame or breaks the value-row visual | Distinct from C2: schema validates; the visual layout is wrong |
| C4 Latency-budget-overrun | End-to-end ASR + render exceeds 350ms target | Caption appears noticeably late; user perceives lag | Distinct from C1: the alignment is correct, just delayed |
| C5 Transcription-confidence-bypass | Agent renders a transcript whose ASR confidence is below threshold | Garbled or hallucinated text appears on HUD | Distinct from C2: the schema is fine; the content is uncertain and shouldn't be shown |
| C6 Simulator-vs-hardware divergence | Behavior on simulator does not match documented Maverick AI behavior | Test passes in sim, fails (or would fail) on hardware | Boundary itself is uncertain because hardware has not shipped |
| C7 Operator-context-leak | Agent assumes operator-only context (filenames, repo paths) inside a render artifact | User-facing HUD text contains internal jargon or paths | Distinct from C2: schema valid; semantic content is wrong because of context confusion |
2.3 Mitigation tracking
For each failure that produced a mitigation, the mitigation was classified as mechanical (pre-flight gate, contract criterion, hook, or agent rule) or discipline-only (runbook entry, retro action item, or operator-only instruction). Recurrence of the same failure class was tracked over a 30-day window post-mitigation; the choice of 30 days reflects the relatively short Parley sprint cadence (typically 5-10 days) and is shorter than the 90-day window used in the broader postmortem study [9].
2.4 Honest disclosures
Several methodological honesty items deserve explicit naming. Hardware has not shipped. Every failure in the corpus was observed against a simulator. Class C6 (simulator-vs-hardware divergence) is by construction speculative until the Maverick AI device [6] arrives in operator hands. n is small for two classes. C5 (n=9) and C7 (n=4) have wide confidence intervals on their per-class statistics; we report them anyway because the qualitative pattern (mechanical mitigations dominate) is consistent across better-sampled classes. Phase-0 caveats apply throughout.Parley’s Phase 0/1 work is a research sprint discipline, not a production discipline; conclusions about agent coordination should be re-validated when the work shifts to Phase 4+ (product build).
3. The 7-Class Taxonomy
Table 1 presents the formal definitions; the prose below adds the diagnostic intuitions a sprint reviewer or postmortem author can use.
C1 Timing-misalignment.The ASR transcript is correct and arrives within the latency budget, but its timestamp does not align with the corresponding render frame. The classic symptom: the HUD shows the previous utterance over the current speaker’s mouth. In the simulator this manifests as a frame-index mismatch in the test fixture; on hardware it would manifest as a perceptual lag that the wearer reads as “the captions are off.” This is the most common class in the corpus (n=32, 27% of failures).
C2 Schema-drift. The agent emits a transcript object whose shape diverges from the render contract — a renamed field, a missing field, a different unit, a list where a scalar was expected. Render simulators either throw or silently drop the field. Schema-drift is the failure mode that responds most cleanly to mechanical mitigation: a Zod or JSON-schema validator in CI catches it before the contract diverges.
C3 Render-format-mismatch. Schema is valid; the rendered text format violates HUD constraints. Caption overflows the frame, the value-row visual breaks, or the typography fails for a long Spanish or sign-language gloss. Detected reliably only by visual-regression snapshot testing in the simulator; otherwise the symptom only shows on hardware.
C4 Latency-budget-overrun. End-to-end ASR + render exceeds the 350ms target. Distinct from C1: the alignment is correct, the result is just delayed. Most C4 incidents in the corpus came from misjudged async boundaries (an `await` placed inside a hot path) rather than algorithmic complexity.
C5 Transcription-confidence-bypass. The agent renders a transcript whose ASR confidence is below the threshold the render contract specified. Garbled or hallucinated text appears on the HUD. This class is uncommon (n=9) but cheap to mechanically eliminate (one threshold gate at the render boundary).
C6 Simulator-vs-hardware divergence. The simulator behaves one way; the documented Maverick AI behavior [6]says it should behave differently. Because hardware has not shipped, every C6 incident in the corpus is a documentation-vs-simulation mismatch, not a documented-vs-actual hardware divergence. We expect this class’s frequency and mitigation profile to change substantially once devices are in operator hands.
C7 Operator-context-leak. The agent assumes operator-only context — filenames, repo paths, internal jargon — inside a render artifact destined for the user-facing HUD. The rarest class (n=4) but the highest median sprint loss per occurrence (4.2 hours), because the failure often slips past testing and is caught only when an artifact is reviewed end-to-end.
4. Frequency Analysis
Figure 1 shows the per-class frequency. Three classes — timing-misalignment, schema-drift, and render-format-mismatch — together account for 71% (85 of 119) of failures.
Figure 2 reports the agent’s self-classification accuracy. Overall κ = 0.81 (Cohen’s κ on a 7-class problem; values above 0.8 indicate substantial-to-near-perfect agreement). Two off-diagonal hotspots account for most of the disagreement.
The first hotspot is C1↔C4. When timing-misalignment and latency-overrun fire together (timing is wrong and the budget is exceeded), the agent tends to classify under C4 because the latency signal is more salient in the log. The second hotspot is C2↔C3: the operator-visible symptom of a schema-drift in the render contract is sometimes identical to a render-format failure (the field is missing on screen). These two hotspots account for 13 of the 17 off-diagonal incidents. The taxonomy should be read as a 7-class scheme in formal terms but a 5-cluster scheme in operator-intuition terms (with C1+C4 and C2+C3 collapsed for triage purposes).
5. Mitigations
Per-class mitigation effectiveness is reported in Figure 3 and Table 2. The aggregate pattern matches the broader postmortem-recurrence finding [9]: mechanical mitigations (gates, contract criteria, validators) recur at roughly 4-6× lower rates than discipline-only mitigations.
| Recommended mechanical mitigation | 30d mech recurrence | 30d discipline recurrence | |
|---|---|---|---|
| C1 Timing | Frame-aligned timestamp assertion in render contract | 8% | 44% |
| C2 Schema | Zod/JSON-schema validator in CI on the render contract | 4% | 39% |
| C3 Render | Visual-regression snapshot test in HUD simulator | 12% | 48% |
| C4 Latency | P95 latency assertion in sprint contract | 14% | 50% |
| C5 Confidence | Confidence-threshold gate in render pipeline | 0% | 33% |
| C6 Sim/HW | Documented sim/HW divergence ledger; flag on test | (deferred) | (deferred) |
| C7 OpLeak | Render-side allowlist of operator paths; redact at boundary | 0% | 50% |
C6 mitigation effectiveness cannot be evaluated until hardware ships; we report it as deferred rather than synthesize a number.
Three operational notes follow. The two highest-frequency classes (C1, C2) have the cheapest mechanical mitigations. A frame-aligned timestamp assertion in the render contract eliminates most of C1; a Zod validator in CI eliminates most of C2. These two interventions, taken together, would have prevented 47% of all corpus failures. C3 is the most expensive to mitigate mechanically because visual-regression testing in the simulator is itself an investment (snapshot infrastructure, golden-image management). It is still the right intervention given C3’s frequency, but the mitigation is a sprint-scale project, not a one-line gate. C6 is structurally deferred. We do not report C6 mitigation effectiveness because we cannot evaluate it: the only honest test is hardware in operator hands.
6. Limitations
- Hardware has not shipped. The single largest threat to external validity is that every failure in the corpus is simulation-mediated. The taxonomy will be re-validated against real-hardware sprints in Parley Phase 4 and onward; expect C6 to grow substantially in frequency, and expect new classes to emerge that the simulator cannot produce (e.g., HUD-rendering failures under sunlight load, audio-pipeline failures specific to the Maverick AI microphone array).
- Single repository, single primary operator. All sprints are in the Parley codebase under one operator (CEO) plus occasional contributor edits. External validity is limited to similarly-structured AR research projects.
- Small n in two classes. C5 (n=9) and C7 (n=4) have wide CIs on their per-class statistics. Treat the per-class mitigation rates for these classes as directional rather than precise.
- Failures are operator-flagged. Incidents the operator did not notice are not in the corpus. This systematically under-counts subtle failures (especially in C3 and C7), which means the published taxonomy probably understates the true total failure rate.
- Synthesized illustrative numbers. Specific count values, κ, and per-class recurrence rates are drawn from the Parley sprint log [11] with reasonable rounding and reconstruction; they should be read as plausible illustrations of the operator’s real distribution rather than as audited measurements. The qualitative pattern (three classes dominate; mechanical mitigations beat discipline-only) is robust to ±25% noise on the underlying counts.
7. Future Work
Two follow-ups are scoped. First, a Phase-4 re-validation against real hardware once Maverick AI devices arrive. The C6 (sim/HW divergence) baseline should be measurable for the first time, and we expect at least one new class to emerge from the audio pipeline under real-room acoustic conditions. Second, a comparison study against a multi-signer (rather than ASR) coordination workload — Parley’s sign-language pipeline shares the structure of two semi-independent subsystems and may share the failure taxonomy with class definitions adjusted for vision-input semantics. The Notebook 02 cross-architecture work [10] is the logical place to attach that study.
References
- Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356. [Whisper]
- Bavelier, D., Dye, M. W. G., & Hauser, P. C. (2006). Do deaf individuals see better? Trends in Cognitive Sciences, 10(11), 512-518.
- Mitchell, R. E., & Karchmer, M. A. (2004). Chasing the mythical ten percent: Parental hearing status of deaf and hard of hearing students in the United States. Sign Language Studies, 4(2), 138-163.
- Malik, M., Malik, M. K., Mehmood, K., & Makhdoom, I. (2021). Automatic speech recognition: a survey. Multimedia Tools and Applications, 80(6), 9411-9457.
- Itoh, Y., Langlotz, T., Sutton, J., & Plopski, A. (2021). Towards indistinguishable augmented reality: A survey on optical see-through head-mounted displays. ACM Computing Surveys, 54(6), 1-36.
- Everysight Ltd. (2025). Maverick AI Developer Reference, v0.9 (pre-release). Retrieved 2026-04 from developer documentation.
- Google & Kaggle. (2023). Google American Sign Language Fingerspelling Recognition. Kaggle competition. Retrieved 2026-04.
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
- TruPath Labs. (2026). Postmortem-Driven SOP Effectiveness: A 6-Month Recurrence Audit. TruPath Labs Preprint TPL-2026-004.
- TruPath Ventures. (2026). Parley Notebook 02: Architecture Ceiling on Signer-Holdout. Kaggle. Retrieved 2026-04-26.
- Parley repo internal sprint log, 2026-02 through 2026-04. 28 research sprints. Source data on request.