Skip to content
SearchSubscribe
TPL-2026-023·preprint·2026-05-31

The Honest Ceiling on Landmark-Only Isolated-Sign Recognition: A Signer-Holdout Audit

CVsign-languageevaluationsigner-holdoutmethodology

Abstract

Isolated-sign recognition results are routinely reported in the 80-90% range, but most are measured on signer-mixed splits where the same signers appear in both training and test data. We audit the real landmark-only ceiling on a strict signer-holdout split of the Google ISLR dataset (250 signs, a 17/2/2 train/val/test signer split, mean and standard deviation over three seeds). The best stable architecture, a frame-level transformer, reaches 44.7% top-1 (+/- 1.0); hand-shape features alone reach 36.4%, so the full body-and-face landmark set adds only about eight points over the hands. Three of seven architecture families collapsed to near-random accuracy on two or more of three seeds, an instability that single-seed reporting would hide entirely. We argue that signer-holdout evaluation with multi-seed reporting is the minimum honest protocol for this task, and that the gap between leaky-split and signer-holdout numbers is the single most important figure a sign-recognition result can disclose.

The accuracy illusion: signer-mixed (leaky) splits report 80-90% by measuring user memorization, while strict signer-holdout evaluation reveals a real ceiling near 45%; hand-shape features alone reach 36%; several architectures collapse on some seeds, so multi-seed reporting is mandatory.
Visual summary of the accuracy gap. Synthesized from the TruPath Labs sign-language research substrate. Download the slide deck (PDF).
The signer firewall: a flawed random split lets the same signers leak from training into testing, while a strict signer-holdout split keeps training signers and test signers entirely separate.
The signer firewall. A random split lets the same signers leak across training and test; a strict signer-holdout split keeps them separate, which is what makes the accuracy honest.
Architectural fragility and model collapse: under strict holdout, some training runs climb to high validation accuracy while others collapse toward the random-chance floor.
Architectural fragility. The same model can reach a strong result or collapse to near-random depending only on the random seed, which is why we report mean and standard deviation over several seeds.

1. Introduction

A reader scanning isolated-sign recognition results comes away with the impression that the problem is mostly solved. Accuracy numbers in the 80-90% range are common, both in published work and on public leaderboards for datasets like Google ISLR [1]. The impression is misleading, and the reason is the evaluation split. Most of those numbers are measured on signer-mixed data, where the same people who appear in training also appear in test. The model is rewarded for memorizing how a specific person signs, and the score it reports is not the score a deaf user in the wild would experience [3].

The honest version of the question is: how well does a landmark-only model recognize signs from people it has never seen? We answer it on the Google ISLR dataset, restricted to 250 signs, with a strict signer-holdout split and results reported as mean and standard deviation over three random seeds. We are not proposing a new architecture. We are establishing a credible ceiling and documenting the failure modes an optimistic evaluation would have hidden.

The headline result is that the best stable model we trained reaches 44.7% top-1 on held-out signers, less than half what signer-mixed reporting suggests. Two further findings matter as much as the number itself: hand-shape features alone account for most of that accuracy, and several otherwise-reasonable architectures collapsed to near-random performance on a majority of seeds.

2. Methods

2.1 Data and split

We use the Google ISLR landmark data [1], which provides per-frame body, hand, and face landmarks extracted with MediaPipe Holistic [2] rather than raw video. We restrict to a fixed vocabulary of 250 signs. The split is signer-holdout: signers are partitioned 17 / 2 / 2 into train, validation, and test, so no signer in the test set contributes any training example. This is the design decision that separates an honest ceiling from accuracy theater, and it is the one most commonly skipped [4].

Every configuration is trained on three seeds (42, 7, 13) and reported as mean and standard deviation. A result that survives on one seed and collapses on the others is not a result; it is a lottery ticket, and reporting only the winning ticket is the most common way an unstable model is made to look reliable.

2.2 Models

Two families were evaluated. Hand-shape models operate on static hand features only, to measure how much of the signal lives in the hands: a linear probe, a multilayer perceptron, and a small 1D convolution. Full-landmark models operate on the complete landmark set with temporal modeling: a dilated temporal convolutional network [7], a bidirectional GRU, a graph convolutional baseline, a sign-pose transformer in the SPOTer family [5], a conformer [6], a squeezeformer, and a frame-level transformer. We report top-1 and top-5 accuracy and macro-F1, the last to avoid rewarding a model that does well only on frequent signs.

3. Results

3.1 The ceiling

On held-out signers, the best stable model is a frame-level transformer at 44.7% top-1 (standard deviation 1.0 over three seeds). The conformer reaches a similar mean but with much higher seed variance. Everything else lands lower. Figure 1 places each model against the linear-probe baseline and random chance; Table 1 gives the full breakdown.

Figure 10%10%20%30%40%50%TOP-1 ACCURACY (SIGNER-HOLDOUT)45%Frame transformer41%Conformer36%Hand-shape (conv)32%Hand-shape (MLP)28%Dilated TCN25%Linear probe (hands)11%GCN-lite
Figure 1 — Top-1 accuracy on the strict signer-holdout split, by model, mean over three seeds. The dashed reference is the linear-probe-on-hand-features baseline (25.3%); random chance on 250 classes is 0.4%. The best stable model (frame transformer) reaches 44.7%. Note how far every result sits below the 80-90% figures commonly reported for this dataset on signer-mixed splits.
Top-1 (mean +/- sd)Top-5Macro-F1Stable seeds
Frame transformer44.7% +/- 1.070.7%43.9%3 / 3
Conformer41.2% +/- 6.468.4%40.6%3 / 3
Hand-shape (conv)36.4% +/- 1.563.6%35.7%3 / 3
Hand-shape (MLP)31.5% +/- 1.461.7%29.9%3 / 3
Dilated TCN28.4% +/- 1.857.2%26.4%3 / 3
GCN-lite10.9% +/- 0.529.3%8.7%3 / 3
BiGRU~33% (1 seed)62.8%32.2%1 / 3
SPOTer~36% (1 seed)65.1%34.4%1 / 3
Squeezeformer~1.5%6.1%0.2%0 / 3
Table 1 — Full results by architecture on the signer-holdout split (Google ISLR, 250 signs). Top-1 and Macro-F1 are mean +/- standard deviation over seeds 42, 7, and 13. 'Stable seeds' counts how many of the three seeds trained to a usable result rather than collapsing to near-random. Hand-shape models (rows 3-4) use static hand features only; all others use the full landmark set with temporal modeling.

Collapsed runs (near-0.4% top-1) are reported, not discarded. Reporting only the surviving seed for BiGRU or SPOTer would overstate a non-reproducible result.

3.2 How much the hands carry

A linear probe on static hand-shape features alone reaches 25.3%, and a small convolution over the same features reaches 36.4%. The best full-landmark model, which additionally sees body pose, face landmarks, and motion over time, reaches 44.7%. Hand-shape alone therefore delivers roughly four-fifths of the achievable accuracy, and the entire rest of the pipeline buys about eight points. For anyone deciding where to spend engineering effort, that ratio is the practical finding.

Figure 30%10%20%30%40%50%TOP-1 ACCURACY (SIGNER-HOLDOUT)25%Hand-shape (linear probe)36%Hand-shape (conv)45%All landmarks + time
Figure 3 — How much of the signal the hands carry. A linear probe on static hand-shape features reaches 25.3%; a small conv over the same hand features reaches 36.4%; the best full-landmark temporal model (frame transformer, using hands plus body plus face plus motion) reaches 44.7%. Hand-shape alone delivers about four-fifths of the achievable accuracy. The remaining landmarks and temporal modeling add roughly eight points, not the dominant share a full-body pipeline's complexity might suggest.

3.3 Seed instability

The most important thing a single-seed result would have hidden is in Figure 2. The frame transformer is stable: 45.7%, 44.6%, 43.7% across the three seeds. BiGRU and SPOTer each trained successfully on exactly one seed and collapsed to 0.4%, indistinguishable from random, on the other two. The squeezeformer failed on all three. Reporting the best run for BiGRU would let us write "BiGRU reaches 33%," a claim that does not reproduce two times out of three.

Figure 20%10%20%30%40%50%TOP-1 ACCURACY46%45%44%Frame transformer0%33%0%BiGRU0%0%36%SPOTer2%1%1%Squeezeformerseed 42seed 7seed 13
Figure 2 — Per-seed top-1 accuracy for four architectures, same data and split, only the random seed changed (42, 7, 13). The frame transformer is stable across seeds. BiGRU and SPOTer each trained successfully on exactly one of three seeds and collapsed to near-random (0.4%) on the other two; squeezeformer collapsed on all three. A paper that reported only the best seed could claim BiGRU 'reaches 33%' or SPOTer '35%' while hiding that the result does not reproduce. Multi-seed reporting is what makes the instability visible.

We report the collapsed runs rather than discarding them. The collapses are not noise to be cleaned away; they are evidence that some of these architectures are too fragile to deploy without stability work that the headline accuracy never mentions.

Work in progress · updated 2026-06-01

Was the collapse the architecture, or the recipe? A pre-registered follow-up (Notebook 04) is running now. It re-runs each failed architecture with a per-architecture recipe — adding linear learning-rate warmup and gradient clipping — and compares against a baseline re-run on the same hardware. That last detail matters: a lottery-ticket collapse is numerics-sensitive, and moving from cloud CUDA to local Apple Silicon can shift accuracy by several points on identical code and seeds, so the old-recipe baseline is measured locally rather than read from the table above.

Preliminary result (BiGRU, complete). Under the old shared recipe, one of three seeds collapsed to chance (0.4%) and the architecture was unstable. Under the tuned recipe, all three seeds trained stably to roughly 37% top-1, with about 1.5% relative spread and no collapse. For BiGRU, the collapse was a recipe artifact, not an architecture limit. SPOTer and Squeezeformer are still training; the full pre-registered verdict — whether at least two of the three recover — is pending, and this section will be updated when the ladder finishes. The follow-up is about stability and fair comparison, not a new ceiling: the 44.7% frame-transformer result is not expected to change.

3.4 Baselines and the leaky-split gap

Random chance and majority-class prediction both sit at 0.4%, confirming the 250-way task is genuinely hard. The honest ceiling of 44.7% should be read against the 80-90% routinely reported on signer-mixed splits of this same dataset. Table 2 lays the contrast out directly. The gap between those two numbers, not either number alone, is the figure that tells you whether a sign-recognition result will survive contact with a new user.

Top-1What it measures
Random chance0.4%Floor for 250 balanced classes
Majority class0.4%Predict the most frequent sign every time
Linear probe (hand features)25.3%What a trivial model extracts from hands alone
Best signer-holdout model44.7%Frame transformer, full landmarks, held-out signers
Typical signer-mixed report~85-90%Same dataset, signers shared across train and test
Table 2 — Where the honest ceiling sits relative to baselines and to the numbers commonly reported for this dataset. Random and majority-class baselines confirm the 250-way task is hard. The linear probe shows how much a trivial model extracts from hand features alone. The best signer-holdout result is 44.7%. Published results on signer-mixed splits of the same dataset routinely land far higher, which is the gap this paper exists to flag.

The last row is illustrative of commonly reported leaky-split performance, included for contrast; it is not a measurement from this study. The measured signer-holdout ceiling is 44.7%.

4. What this implies

The minimum honest protocol for isolated-sign recognition has two parts, and both are cheap. Evaluate on held-out signers, because signer-mixed accuracy measures memorization of individuals rather than recognition of signs. Report mean and standard deviation over several seeds, because a single seed cannot distinguish a stable model from one that happened to train. Neither costs anything but discipline, and skipping either is how a 45% capability gets sold as a 90% one.

None of this says landmark-only sign recognition is a dead end. It says the real starting line is lower than the literature implies, the hands carry most of the load, and architecture choice is as much about training stability as about peak accuracy. Those are useful things to know before building on top of a number.

5. Limitations

This is isolated-sign classification on pre-extracted landmarks, not continuous signing and not raw video, so the ceiling here is an upper bound on an easier problem than real conversation. The vocabulary is 250 signs; a larger label set would lower the numbers. Landmark-only input discards facial grammar and non-manual markers that carry meaning in the language, a limitation of the input modality, not of the models. And the comparison row for signer-mixed performance is illustrative of commonly reported figures rather than a re-measurement under identical conditions. The measured contribution of this paper is the 44.7% signer-holdout ceiling and the stability profile behind it.

References

  1. Google. (2023). Isolated Sign Language Recognition (Google - ASL Signs). Kaggle competition. https://www.kaggle.com/competitions/asl-signs
  2. Lugaresi, C., Tang, J., Nash, H., et al. (2019). MediaPipe: A Framework for Building Perception Pipelines. arXiv:1906.08172.
  3. Li, D., Rodriguez, C., Yu, X., & Li, H. (2020). Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison. WACV. [Discusses signer-independent vs signer-dependent splits.]
  4. Joze, H. R. V., & Koller, O. (2019). MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language. BMVC.
  5. Boháček, M., & Hrúz, M. (2022). Sign Pose-based Transformer for Word-level Sign Language Recognition. WACV Workshops.
  6. Gulati, A., Qin, J., Chiu, C.-C., et al. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. Interspeech.
  7. Bai, S., Kolter, J. Z., & Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv:1803.01271.