Recipe, Not Architecture: Three "Failed" Sign-Recognition Models Recover Under Per-Architecture Training
Abstract
Architecture comparisons in sign-language recognition are routinely run under a single shared training recipe, and the resulting ranking is reported as a property of the architectures. We test that assumption directly. In an earlier signer-holdout study on the Google ISLR dataset (250 signs, 21 signers), three landmark-sequence architectures appeared to fail under a shared recipe: Squeezeformer-small collapsed to chance, and BiGRU and SPOTER were lottery-ticket, with one of three seeds training and the other two collapsing. Here we re-run each architecture under both the original shared recipe and a per-architecture tuned recipe (linear learning-rate warmup plus gradient clipping), with the full ladder on one hardware (CUDA, RTX 4090), three seeds each, every rung trained to early-stop convergence. Under a pre-registered recovery criterion (seed dispersion std/mean <= 0.10 and tuned >= 2x old), all three architectures recover. Two of them, SPOTER (45.1%) and Squeezeformer-small (46.4%), reach the competition-winning frame-transformer ceiling (45.8%) on the same honest split. A permuted-label control trains to 0.39% (chance), confirming no label leak. The reported architecture failures were substantially training-recipe artifacts, not architecture limits, and the original ranking measured the recipe more than the model. We make no state-of-the-art claim; the contribution is a methodological correction with a worked example.

1. The question this paper closes
This is the fourth installment from Parley's research arm on the Google ISLR dataset. The first three are public: a hand-shape baseline, an architecture ceiling-search that put honest cross-signer accuracy at 44.7% against the roughly 83% random-split leaderboard number [9], and a 21-fold leave-one-signer-out study that found a 38.57-point fairness spread across signers.
The ceiling-search left a loose end. Of the seven architectures on the ladder, several did not behave. Squeezeformer-small collapsed to chance and stayed there. BiGRU and SPOTER were what the lottery-ticket literature [1] would call untrainable-by-init: one seed of three found a working basin, the other two sat at chance. Under a shared training recipe, one learning rate, one schedule, one set of regularization choices applied uniformly to every architecture, these looked like architecture failures. The honest version of that finding raised a question instead of answering one: were these architecture limits, or were they recipe artifacts that a shared recipe could not have distinguished from limits?
The cast, briefly: BiGRU is a bidirectional gated-recurrent sequence model that reads the landmark stream forward and backward in time; SPOTER [6] is a transformer built for sign language from body and hand pose; and Squeezeformer-small [5] is a convolution-augmented, downsample-recover transformer adapted from automatic speech recognition. The reference ceiling is the competition-winning frame-level landmark transformer.

That question is the whole paper. It matters beyond Parley because shared-recipe architecture comparisons are the default in applied sign-language work, and a shared recipe that happens to suit one architecture's optimization geometry will systematically flatter it and bury the others.

2. Data and method
We use the Google ISLR landmark data: 250 sign classes, 21 signers, MediaPipe Holistic landmarks (543 points per frame across face, pose, and both hands). Chance and majority-class baseline are both 1/250 = 0.4%. The split is signer-holdout throughout: signers in test never appear in train (17 / 2 / 2 train / val / test, seed 42 for the split), fixed before training and never changed within the experiment.
The old recipe is the shared recipe from the ceiling-search: identical optimizer, learning rate, schedule, and regularization for every architecture. The tuned recipe adds two changes per architecture, a linear learning-rate warmup and gradient clipping at norm 1.0, with the base learning rate set per architecture (Table 2). Both recipes are declared in advance in a contract, not adjusted after seeing results.

| Base LR | Batch | Warmup epochs | Grad-clip norm | |
|---|---|---|---|---|
| Squeezeformer-small | 0.0003 | 32 | 8 | 1.0 |
| SPOTER | 0.0005 | 32 | 5 | 1.0 |
| BiGRU | 0.001 | 64 | 2 | 1.0 |
Permuted-label control: tuned SPOTER trained on deliberately shuffled labels reaches top-1 0.39%, on top of the 0.40% (1/250) chance baseline. No label leak, so the recovered accuracy is real signal.
An earlier version of this comparison ran across mixed hardware (Apple MPS and CUDA), which introduced a confound: a collapse on one backend could be a numerical-stability artifact of that backend rather than the recipe [3]. To remove it, the full ladder here ran on a single hardware (CUDA, RTX 4090), three seeds each, every rung trained to early-stop convergence. We pre-registered the recovery verdict before the runs: an architecture recovers only if its tuned-recipe seed dispersion is std/mean ≤ 0.10 (it trains stably across seeds) and its tuned mean is ≥ 2x its old-recipe mean (the recipe, not luck, moved it). Both conditions must hold [4].
3. Results
3.1 All three recover
Under the tuned recipe, every architecture that failed under the shared recipe passes the pre-registered criterion (Table 1). Squeezeformer-small moves from 1.25% to 46.4%, a 37x gain with seed dispersion of 0.016. SPOTER moves from 3.26% to 45.1%, a 14x gain at dispersion 0.013 [6]. Both clear the bar decisively, and both land on the frame-transformer ceiling of 45.8% [5]. The architectures reported as failures, given a warmup and a gradient clip, match the competition-winning model on the honest split.
| Old recipe | Tuned recipe | Ratio | Tuned std/mean | Recovered? | |
|---|---|---|---|---|---|
| Squeezeformer-small | 1.25% +/- 0.24 | 46.4% +/- 0.76 | 37.2x | 0.016 | Yes |
| SPOTER | 3.26% +/- 2.03 | 45.1% +/- 0.61 | 13.8x | 0.013 | Yes |
| BiGRU | 11.4% +/- 15.5 | 30.0% +/- 2.72 | 2.6x | 0.091 | Yes (marginal) |
| frame_transformer (ceiling) | — | 45.8% +/- 0.13 | — | — | Reference |
Recovery is a pre-registered binary: tuned std/mean <= 0.10 AND tuned >= 2x old. All three failed architectures pass. Old-recipe BiGRU dispersion was 1.36 (lottery-ticket); the tuned recipe fixed the instability.

BiGRU is the marginal pass. It moves from an unstable 11% to a stable 30%, just inside the 0.10 dispersion bar. The recipe fixed its instability (old-recipe dispersion was 1.36, the signature of a lottery ticket) and roughly tripled its mean, but it recovers to a lower ceiling than the other two. BiGRU genuinely is a weaker architecture for this task; it is not, however, the chance-level failure the shared recipe implied.
3.2 The mechanism is early-training stability
With everything else held constant, same architecture, same hardware, same seed, only the recipe differs, the cause is visible in the loss curves. Under the shared recipe the training loss rises and validation top-1 sits at chance: the model never leaves its initialization basin because the first optimizer steps, unwarmed and unclipped, destabilize it [2]. Under the tuned recipe the loss descends smoothly and validation climbs to the ceiling. The collapse was an early-training instability the shared recipe could not survive, not an inability to represent the task.

3.3 The recovery is not a leak
A recovery this large invites the worry that the tuned pipeline introduced label leakage. It did not. Training tuned SPOTER on shuffled labels reaches top-1 0.39%, on top of the 0.40% chance floor. The honest split has no leak; the recovered accuracy is real signal.

4. Two honest nuances
On an earlier MPS run, BiGRU's old-vs-tuned ratio was 1.69x, a fail under the same criterion that gives 2.6x (a pass) on CUDA. A lottery-ticket architecture is sensitive to both seed and hardware, which is itself a small confirmation of this paper's thesis: instability masquerading as a result. It is exactly why the verdict is computed on a single hardware. SPOTER and Squeezeformer, which collapse to chance under the old recipe, recover decisively and are stable across hardware; BiGRU is the one to treat with caution.

And SPOTER and Squeezeformer reaching the frame-transformer ceiling means the tuned recipe is sound and the ceiling holds across all three architectures. It does not mean we beat anything. The honest cross-signer ceiling on this task is about 45% regardless of which of these three architectures you pick once the recipe is right.
5. What it means
The narrow finding: on Google ISLR, the architecture ranking drawn under a shared recipe was largely a recipe artifact. Per-architecture warmup plus gradient clipping rescues all three failed architectures, and two reach the ceiling.
The general finding, and the reason this is worth a paper: an architecture comparison that holds the training recipe fixed across architectures does not measure the architectures; it measures how well a single recipe happens to suit each one's optimization geometry. The fix is cheap and requires no new data. Declare a per-architecture recipe in advance, run on one hardware to convergence, report seed dispersion alongside the mean, and include a permuted-label control. Without those, a published "Architecture X fails on this task" may be reporting nothing more durable than "the shared recipe didn't suit Architecture X." The same penalty for honest evaluation replicates across sign languages and datasets [7], and standardized open baselines make the discipline cheap to adopt [8].

References
- Frankle, J., & Carbin, M. (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. ICLR.
- Picard, D. (2021). torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision. arXiv:2109.08203.
- Bouthillier, X., Delaunay, P., Bronzi, M., et al. (2021). Accounting for Variance in Machine Learning Benchmarks. MLSys, 3.
- Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A. (2019). Show Your Work: Improved Reporting of Experimental Results. EMNLP-IJCNLP.
- Kim, S., Gholami, A., Shaw, A., et al. (2022). Squeezeformer: An Efficient Transformer for Automatic Speech Recognition. NeurIPS, 35.
- Bohacek, M., & Hruz, M. (2022). Sign Pose-based Transformer for Word-level Sign Language Recognition (SPOTER). WACV Workshops.
- Sincan, O. M., & Keles, H. Y. (2020). AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods. IEEE Access, 8. [Random vs. signer-independent split.]
- Selvaraj, P., NC, G., Kumar, P., & Khapra, M. (2021). OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages. arXiv:2110.05877.
- TruPath Labs. (2026). The Honest Ceiling on Landmark-Only Isolated-Sign Recognition: A Signer-Holdout Audit. TruPath Labs Preprint TPL-2026-023.