Detecting Eval-Train Overlap in Production CV Pipelines: A Lightweight Audit Protocol
Abstract
Computer vision systems shipped to production frequently report misleading evaluation metrics due to undetected overlap between training and held-out eval data. We present a sub-second hash-based audit protocol that runs adversarially in the eval pipeline and alarms on any F1 measurement above a configurable suspicion threshold. Applied retroactively to four production CV systems, the audit identified silent overlap in two of four pipelines with overlap rates between 6.2% and 14.8%, each producing inflated F1 scores of 0.97-0.99 that collapsed to 0.71-0.83 once overlap was removed.
1. Introduction
Production computer vision systems are routinely shipped on the basis of held-out evaluation metrics that, on close inspection, derive from datasets that overlap with the training corpus. The most common form of overlap is silent — neither the team nor the eval pipeline detects it — and the resulting F1, mAP, or accuracy figures are inflated by a margin large enough to mask catastrophic real-world failures.
Existing literature covers train-test overlap extensively in academic ML (see Kaufman et al. 2012; Recht et al. 2019), but production CV teams rarely run overlap audits. The gap is partly cultural — production engineers are not trained as statisticians — and partly tooling: there is no widely deployed audit tool that runs as a pre-publication gate on eval results.
We present a lightweight audit protocol that runs adversarially in any CV evaluation pipeline. The protocol uses perceptual hashing of training and eval images, computes pairwise similarity, and flags F1 measurements above a configurable threshold (default 0.95) for human review. Applied retroactively to four production CV pipelines across two organizations, the audit identified silent overlap in two of four systems, with overlap rates between 6.2% and 14.8% producing inflated F1 scores of 0.97-0.99 that fell to 0.71-0.83 once overlap was removed.
Our contributions:
- A sub-second hash-based audit protocol suitable for inline use in any eval pipeline.
- An empirical study of four production CV systems showing the prevalence and magnitude of silent overlap.
- A configurable F1-suspicion threshold whose calibration we discuss in §6.
2. Methods
2.1 Perceptual hashing
We use 64-bit pHash [1] over the resized 32×32 luminance channel for both training and eval images. pHash is invariant to scale, JPEG compression, mild crops, and brightness shifts — all transformations that frequently produce silent overlap when datasets are constructed by manual collection from a shared source pool.
Hamming distance between hashes serves as the similarity metric. Thresholds are calibrated empirically (§3.2): a Hamming distance of ≤ 6 corresponds to a near-duplicate pair in our dataset; ≤ 12 captures most cropped/recompressed variants. We default to ≤ 8 for production audits.
2.2 Audit protocol
The audit runs in three stages, each gated on the prior:
- Hash both corpora — compute pHash for every train and eval image. O(n) time; embarrassingly parallelizable.
- Pairwise comparison via locality-sensitive hashing — bucket hashes by 16-bit prefix; only compare within-bucket pairs. Reduces the O(n×m) naive comparison to O((n+m) log(n+m)) in expectation.
- Flag and report — emit any train-eval pair with Hamming distance ≤ threshold, alongside the eval image's contribution to the headline metric.
The protocol completes in under one second on corpora of up to 50,000 image pairs on a single CPU core, which is well within the budget of any eval pipeline.
2.3 F1-suspicion threshold
In addition to the hash audit, we report any F1 score above a configurable suspicion threshold. The default of 0.95 is empirically calibrated against four production systems where a ground-truth F1 of 0.95 was rare even after extensive optimization, and where exceeding 0.95 in a fresh eval was strongly correlated with overlap or schema-leakage events. F1 ≥ 0.99 should always trigger an audit, regardless of the model's apparent maturity.
3. Experimental setup
We evaluated four production CV pipelines, anonymized as Pipeline A through Pipeline D. Each pipeline had been independently developed and was reporting an F1 score above 0.92 prior to our audit. Hardware, datasets, and architectures were standardized as follows.
| Architecture | Train images | Eval images | Reported F1 | Hardware | |
|---|---|---|---|---|---|
| Pipeline A | Detector A (transformer) | 8,420 | 1,054 | 0.978 | RTX 4090 |
| Pipeline B | Detector B (compact CNN) | 12,300 | 1,540 | 0.991 | RTX 4090 |
| Pipeline C | Detector C (compact CNN) | 5,210 | 650 | 0.924 | T4 (cloud) |
| Pipeline D | Detector D (transformer) | 9,850 | 1,234 | 0.943 | A100 (cloud) |
All F1 scores reported by original teams, replicated by us before audit.
3.1 Threshold calibration
We constructed a calibration set of 1,000 known-paired images (the same source image with controlled augmentations: scale, crop ≤ 20%, JPEG quality 70-95, brightness ±15%) and 1,000 unrelated pairs. The Hamming distance distribution clearly separates the two populations:
4. Results
4.1 Overlap detection across the four pipelines
The audit identified silent overlap in two of four pipelines. Pipelines A and C — the systems whose teams had explicit deduplication steps in their dataset construction — showed near-zero overlap. Pipelines B and D — both of which had aggregated training and eval data from a shared upstream source — showed substantial overlap.
4.2 F1 collapse after overlap removal
For each pipeline showing non-trivial overlap, we re-evaluated against a clean held-out set with all near-duplicates removed. F1 scores for Pipelines B and D fell substantially:
| Reported F1 | Clean F1 | Drop | Overlap rate | |
|---|---|---|---|---|
| Pipeline A | 0.978 | 0.974 | 0.4% | 0.4% |
| Pipeline B | 0.991 | 0.711 | 28.3% | 14.8% |
| Pipeline C | 0.924 | 0.918 | 0.6% | 1.1% |
| Pipeline D | 0.943 | 0.834 | 11.6% | 6.2% |
Clean F1 reported with 95% bootstrap CI of ±0.018-0.024 (n=1,000 resamples).
4.3 Confusion matrix shift on Pipeline B
To illustrate the practical impact of overlap on real model behavior, we present the confusion matrix for Pipeline B before and after dedup. The reported matrix (left) suggested high precision and recall for both classes. The clean matrix (right) reveals that Pipeline B systematically misclassifies the minority class once it cannot rely on memorized eval examples.
4.4 Audit runtime
The audit completes well under one second per pipeline at the corpus sizes encountered, validating the claim that this is a free addition to any eval pipeline.
5. Discussion
The results suggest that silent eval-train overlap is common in production CV systems and produces metric inflation large enough to invalidate deployment decisions. Two findings are worth emphasizing:
F1 ≥ 0.99 should alarm. In our four-pipeline study, the only pipeline reporting F1 above 0.99 was Pipeline B — which turned out to have the largest overlap (14.8%) and the largest collapse on dedup (to 0.711). A heuristic of "F1 above 0.99 is a smell, not a victory" would have flagged this system without any audit infrastructure at all. We recommend wiring this heuristic as a hard alarm in any eval pipeline producing such scores.
Manual dedup is unreliable; mechanical dedup is sufficient. Pipelines A and C — the two with explicit dedup steps in their dataset construction — were essentially clean (overlap rates 0.4% and 1.1%, both below the false-positive rate of our default threshold). Pipelines B and D — without dedup — were not. The lesson is structural: relying on team discipline to prevent overlap fails routinely; running an automated audit catches what discipline misses.
The audit protocol presented here is deliberately simple. We do not claim novelty in the use of perceptual hashing for near-duplicate detection (cf. Zauner 2010 [1]); the contribution is in demonstrating that this audit, run as a hard gate before any eval result is published, would have caught two of four production failures we examined. The deployment friction is near zero. The cost of not deploying it, as Pipelines B and D demonstrate, can be a shipped product whose advertised performance is fiction.
6. Limitations
- n=4 production pipelines is too small to claim a population-level overlap-rate distribution. The 50% silent-overlap rate observed in our sample is consistent with anecdotal reports from production teams but should not be cited as a generalized statistic without a larger study.
- Perceptual hashing has known failure modes. Heavy crops (>30% area removed), large geometric warps, and severe color-channel manipulations can defeat pHash and produce false negatives. Teams working in domains where such transformations are common (medical imaging with reformatting; cross-modality eval) should validate pHash thresholds on their own data.
- This audit only catches one class of leakage. Schema leakage, distribution leakage, and label leakage all bypass pHash entirely. The audit is necessary, not sufficient. We recommend pairing it with: (a) hard-isolating eval data sources; (b) checking schema consistency across pipeline boundaries; (c) verifying class-distribution similarity between train and eval.
- F1 ≥ 0.95 is empirical. The 0.95 default suspicion threshold is calibrated against four pipelines in two organizations. Some legitimate eval results exceed 0.95 honestly. The threshold is a smoke detector, not an oracle — flagged scores require human review, not automatic rejection.
7. Reproducibility
We provide enough detail for an independent team to replicate the audit protocol and the calibration study.
| Detail | Value | |
|---|---|---|
| pHash implementation | imagehash 4.3.1, 32×32 DCT, 8-bit hash | |
| LSH bucket size | 16-bit prefix, ~65,536 buckets | |
| Hamming threshold (default) | ≤ 8 | |
| F1 suspicion threshold (default) | ≥ 0.95 | |
| Hardware (audit runtime) | Apple M2 Pro, 32 GB, single-threaded | |
| Framework versions | Python 3.11.6, NumPy 1.26.4, Pillow 10.3.0 | |
| Random seed (calibration set) | 1729 for image sampling; deterministic augmentations | |
| Bootstrap CI runs | 1,000 resamples, BCA method |
Source code for the audit protocol will be released upon publication. Calibration dataset construction recipe is in Appendix A (forthcoming with the published version).
References
- Zauner, C. (2010). Implementation and Benchmarking of Perceptual Image Hash Functions.Master’s thesis, Upper Austria University of Applied Sciences, Hagenberg.
- Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: Formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), 1-21.
- Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). Do ImageNet classifiers generalize to ImageNet? Proceedings of ICML 2019.
- Lin, T.-Y., Maire, M., Belongie, S., et al. (2014). Microsoft COCO: Common Objects in Context. ECCV 2014.
- Carion, N., Massa, F., Synnaeve, G., et al. (2020). End-to-End Object Detection with Transformers. ECCV 2020. [DETR]
- Lv, W., Xu, S., Zhao, Y., et al. (2024). DETRs Beat YOLOs on Real-time Object Detection. arXiv:2304.08069.
- Jocher, G., Chaurasia, A., & Qiu, J. (2024). Ultralytics YOLO11. github.com/ultralytics/ultralytics.
- Allspaw, J. (2012). Blameless PostMortems and a Just Culture. Code as Craft / Etsy Engineering Blog.
- Sculley, D., Holt, G., Golovin, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. NIPS 2015.
- Beery, S., Van Horn, G., & Perona, P. (2018). Recognition in Terra Incognita. ECCV 2018. [domain shift]
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press, Chapter 5: Machine Learning Basics.