The Honest Ceiling on Landmark-Only Isolated-Sign Recognition: A Signer-Holdout Audit
Abstract
Isolated-sign recognition results are routinely reported in the 80-90% range, but most are measured on signer-mixed splits where the same signers appear in both training and test data. We audit the real landmark-only ceiling on a strict signer-holdout split of the Google ISLR dataset (250 signs, a 17/2/2 train/val/test signer split, mean and standard deviation over three seeds). The best stable architecture, a frame-level transformer, reaches 44.7% top-1 (+/- 1.0); hand-shape features alone reach 36.4%, so the full body-and-face landmark set adds only about eight points over the hands. Three of seven architecture families collapsed to near-random accuracy on two or more of three seeds, an instability that single-seed reporting would hide entirely. We argue that signer-holdout evaluation with multi-seed reporting is the minimum honest protocol for this task, and that the gap between leaky-split and signer-holdout numbers is the single most important figure a sign-recognition result can disclose.



1. Introduction
A reader scanning isolated-sign recognition results comes away with the impression that the problem is mostly solved. Accuracy numbers in the 80-90% range are common, both in published work and on public leaderboards for datasets like Google ISLR [1]. The impression is misleading, and the reason is the evaluation split. Most of those numbers are measured on signer-mixed data, where the same people who appear in training also appear in test. The model is rewarded for memorizing how a specific person signs, and the score it reports is not the score a deaf user in the wild would experience [3].
The honest version of the question is: how well does a landmark-only model recognize signs from people it has never seen? We answer it on the Google ISLR dataset, restricted to 250 signs, with a strict signer-holdout split and results reported as mean and standard deviation over three random seeds. We are not proposing a new architecture. We are establishing a credible ceiling and documenting the failure modes an optimistic evaluation would have hidden.
The headline result is that the best stable model we trained reaches 44.7% top-1 on held-out signers, less than half what signer-mixed reporting suggests. Two further findings matter as much as the number itself: hand-shape features alone account for most of that accuracy, and several otherwise-reasonable architectures collapsed to near-random performance on a majority of seeds.
2. Methods
2.1 Data and split
We use the Google ISLR landmark data [1], which provides per-frame body, hand, and face landmarks extracted with MediaPipe Holistic [2] rather than raw video. We restrict to a fixed vocabulary of 250 signs. The split is signer-holdout: signers are partitioned 17 / 2 / 2 into train, validation, and test, so no signer in the test set contributes any training example. This is the design decision that separates an honest ceiling from accuracy theater, and it is the one most commonly skipped [4].
Every configuration is trained on three seeds (42, 7, 13) and reported as mean and standard deviation. A result that survives on one seed and collapses on the others is not a result; it is a lottery ticket, and reporting only the winning ticket is the most common way an unstable model is made to look reliable.
2.2 Models
Two families were evaluated. Hand-shape models operate on static hand features only, to measure how much of the signal lives in the hands: a linear probe, a multilayer perceptron, and a small 1D convolution. Full-landmark models operate on the complete landmark set with temporal modeling: a dilated temporal convolutional network [7], a bidirectional GRU, a graph convolutional baseline, a sign-pose transformer in the SPOTer family [5], a conformer [6], a squeezeformer, and a frame-level transformer. We report top-1 and top-5 accuracy and macro-F1, the last to avoid rewarding a model that does well only on frequent signs.
3. Results
3.1 The ceiling
On held-out signers, the best stable model is a frame-level transformer at 44.7% top-1 (standard deviation 1.0 over three seeds). The conformer reaches a similar mean but with much higher seed variance. Everything else lands lower. Figure 1 places each model against the linear-probe baseline and random chance; Table 1 gives the full breakdown.
| Top-1 (mean +/- sd) | Top-5 | Macro-F1 | Stable seeds | |
|---|---|---|---|---|
| Frame transformer | 44.7% +/- 1.0 | 70.7% | 43.9% | 3 / 3 |
| Conformer | 41.2% +/- 6.4 | 68.4% | 40.6% | 3 / 3 |
| Hand-shape (conv) | 36.4% +/- 1.5 | 63.6% | 35.7% | 3 / 3 |
| Hand-shape (MLP) | 31.5% +/- 1.4 | 61.7% | 29.9% | 3 / 3 |
| Dilated TCN | 28.4% +/- 1.8 | 57.2% | 26.4% | 3 / 3 |
| GCN-lite | 10.9% +/- 0.5 | 29.3% | 8.7% | 3 / 3 |
| BiGRU | ~33% (1 seed) | 62.8% | 32.2% | 1 / 3 |
| SPOTer | ~36% (1 seed) | 65.1% | 34.4% | 1 / 3 |
| Squeezeformer | ~1.5% | 6.1% | 0.2% | 0 / 3 |
Collapsed runs (near-0.4% top-1) are reported, not discarded. Reporting only the surviving seed for BiGRU or SPOTer would overstate a non-reproducible result.
3.2 How much the hands carry
A linear probe on static hand-shape features alone reaches 25.3%, and a small convolution over the same features reaches 36.4%. The best full-landmark model, which additionally sees body pose, face landmarks, and motion over time, reaches 44.7%. Hand-shape alone therefore delivers roughly four-fifths of the achievable accuracy, and the entire rest of the pipeline buys about eight points. For anyone deciding where to spend engineering effort, that ratio is the practical finding.
3.3 Seed instability
The most important thing a single-seed result would have hidden is in Figure 2. The frame transformer is stable: 45.7%, 44.6%, 43.7% across the three seeds. BiGRU and SPOTer each trained successfully on exactly one seed and collapsed to 0.4%, indistinguishable from random, on the other two. The squeezeformer failed on all three. Reporting the best run for BiGRU would let us write "BiGRU reaches 33%," a claim that does not reproduce two times out of three.
We report the collapsed runs rather than discarding them. The collapses are not noise to be cleaned away; they are evidence that some of these architectures are too fragile to deploy without stability work that the headline accuracy never mentions.
Work in progress · updated 2026-06-01
Was the collapse the architecture, or the recipe? A pre-registered follow-up (Notebook 04) is running now. It re-runs each failed architecture with a per-architecture recipe — adding linear learning-rate warmup and gradient clipping — and compares against a baseline re-run on the same hardware. That last detail matters: a lottery-ticket collapse is numerics-sensitive, and moving from cloud CUDA to local Apple Silicon can shift accuracy by several points on identical code and seeds, so the old-recipe baseline is measured locally rather than read from the table above.
Preliminary result (BiGRU, complete). Under the old shared recipe, one of three seeds collapsed to chance (0.4%) and the architecture was unstable. Under the tuned recipe, all three seeds trained stably to roughly 37% top-1, with about 1.5% relative spread and no collapse. For BiGRU, the collapse was a recipe artifact, not an architecture limit. SPOTer and Squeezeformer are still training; the full pre-registered verdict — whether at least two of the three recover — is pending, and this section will be updated when the ladder finishes. The follow-up is about stability and fair comparison, not a new ceiling: the 44.7% frame-transformer result is not expected to change.
3.4 Baselines and the leaky-split gap
Random chance and majority-class prediction both sit at 0.4%, confirming the 250-way task is genuinely hard. The honest ceiling of 44.7% should be read against the 80-90% routinely reported on signer-mixed splits of this same dataset. Table 2 lays the contrast out directly. The gap between those two numbers, not either number alone, is the figure that tells you whether a sign-recognition result will survive contact with a new user.
| Top-1 | What it measures | |
|---|---|---|
| Random chance | 0.4% | Floor for 250 balanced classes |
| Majority class | 0.4% | Predict the most frequent sign every time |
| Linear probe (hand features) | 25.3% | What a trivial model extracts from hands alone |
| Best signer-holdout model | 44.7% | Frame transformer, full landmarks, held-out signers |
| Typical signer-mixed report | ~85-90% | Same dataset, signers shared across train and test |
The last row is illustrative of commonly reported leaky-split performance, included for contrast; it is not a measurement from this study. The measured signer-holdout ceiling is 44.7%.
4. What this implies
The minimum honest protocol for isolated-sign recognition has two parts, and both are cheap. Evaluate on held-out signers, because signer-mixed accuracy measures memorization of individuals rather than recognition of signs. Report mean and standard deviation over several seeds, because a single seed cannot distinguish a stable model from one that happened to train. Neither costs anything but discipline, and skipping either is how a 45% capability gets sold as a 90% one.
None of this says landmark-only sign recognition is a dead end. It says the real starting line is lower than the literature implies, the hands carry most of the load, and architecture choice is as much about training stability as about peak accuracy. Those are useful things to know before building on top of a number.
5. Limitations
This is isolated-sign classification on pre-extracted landmarks, not continuous signing and not raw video, so the ceiling here is an upper bound on an easier problem than real conversation. The vocabulary is 250 signs; a larger label set would lower the numbers. Landmark-only input discards facial grammar and non-manual markers that carry meaning in the language, a limitation of the input modality, not of the models. And the comparison row for signer-mixed performance is illustrative of commonly reported figures rather than a re-measurement under identical conditions. The measured contribution of this paper is the 44.7% signer-holdout ceiling and the stability profile behind it.
References
- Google. (2023). Isolated Sign Language Recognition (Google - ASL Signs). Kaggle competition. https://www.kaggle.com/competitions/asl-signs
- Lugaresi, C., Tang, J., Nash, H., et al. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv:1906.08172.
- Li, D., Rodriguez, C., Yu, X., & Li, H. (2020). Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison. WACV. [Discusses signer-independent vs signer-dependent splits.]
- Joze, H. R. V., & Koller, O. (2019). MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language. BMVC.
- Boháček, M., & Hrúz, M. (2022). Sign Pose-based Transformer for Word-level Sign Language Recognition. WACV Workshops.
- Gulati, A., Qin, J., Chiu, C.-C., et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech.
- Bai, S., Kolter, J. Z., & Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv:1803.01271.