Data-Centric CV Pipeline Engineering: Lessons from 15 Training Iterations on a Custom 3-Class Detection Domain

1. Introduction

The dominant frame for improving computer vision model performance is architectural: choose a better backbone, a more expressive detection head, a richer pre-training corpus. Dataset construction is treated as a prerequisite to get right once and then forget. This paper argues that frame is inverted for custom-domain fine-tuning. In production systems targeting narrow, domain-specific object classes, label quality and sensor coverage are the primary performance variables — architecture is nearly irrelevant once a reasonable baseline is selected.

We report on a 15-iteration training arc for a custom 3-class detection system built on a single fixed compact-detector architecture. The arc spans April 2025 through April 2026 and includes a controlled natural experiment: v11 and v15 used identical architecture, identical training code, identical hyperparameters, and the same evaluation protocol. v11 was trained on 4,398 noisily-labeled frames and achieved ~73% mAP50. v15 was trained on 505 clean, hand-annotated frames and achieved over 99% mAP50. The roughly 27-point gap is not attributable to model selection, hyperparameter tuning, or augmentation strategy. It is attributable to annotation quality, class coverage, and sensor gap elimination.

The secondary contribution of this paper is a practical analysis of the 2024–2026 toolchain for scaling annotation throughput without sacrificing quality: Grounding DINO 1.5 [3] and SAM 2 [4] for pseudo-label generation, DINOv2 [1] for self-supervised feature extraction and outlier detection, and active learning [6] for sample selection under budget constraint. We treat these tools as components of a human-in-the-loop annotation pipeline, not as replacements for human review.

Total GPU compute spend across 15 training iterations: less than $5. The cost constraint is not incidental — it shaped every data decision in the arc. The lessons documented here are most applicable to practitioners operating under similarly tight compute budgets on proprietary or narrow-domain datasets where off-the-shelf zero-shot detectors (Grounding DINO: 52.5% zero-shot COCO AP [2]) fall well short of production requirements.

2. Domain Description and Problem Setup

The target domain is real-time detection of three object classes in a fixed-scene sports environment: a small fast-moving object, a fixed structured target, and a negative class representing background clutter and hard-negative objects. These classes are visually demanding for general-purpose detectors: the moving object is small, motion-blurred at typical capture rates, and visually similar to background noise at lower resolutions. The target is a structured geometric object with variable lighting across environments. Hard negatives include spectators, equipment, and reflective surfaces that generate false positive detections in untuned models.

The production inference runtime imposes a specific export constraint: model weights must pass through the runtime’s conversion pipeline, which — as documented in Section 3.3 — introduced a systematic class index remapping failure that cost multiple training iterations to diagnose.

The evaluation metric throughout this arc is mAP50 (mean Average Precision at IoU threshold 0.50), consistent with COCO evaluation conventions. All mAP50 values reported are computed on a held-out validation set withheld from training data in all versions. The validation set was re-examined at v11 to confirm no annotation leakage between training and validation splits after dataset expansion.

3. Training Arc v4–v15

3.1 Infrastructure

All training was performed on cloud GPU infrastructure (A100 or equivalent), using a standard open-source training framework with a single fixed detector architecture throughout the arc. Hyperparameters — learning rate, batch size, augmentation policy, number of epochs — were fixed after v4 and held constant for all subsequent versions unless explicitly noted. The decision to hold hyperparameters constant was deliberate: it isolates data quality as the experimental variable and prevents attribution confusion.

Annotation tooling evolved across the arc. Early versions (v4–v6) used a mix of managed annotation tooling with semi-automated assistance. v7 onward used a stricter manual review protocol. v15 used fully hand-annotated frames with no automated label generation in the final training set — a decision made after the v11 experiment demonstrated the cost of label noise.

3.2 Version History

Table 1 summarizes the key milestones across the training arc. The version history is not monotonically improving — v6 represents a significant regression caused by a systematic labeling error, and the arc from v6 to v11 is a long recovery period whose cost is attributable entirely to data management failures, not modeling choices.

3.3 Ground Zero Failure

The most consequential event in the training arc was the discovery of a systematic class index mapping error between the training export and the production inference backend. The model was trained with a fixed three-class ordering, but the export — without explicit class ordering enforcement — silently reordered classes, producing an inference backend that emitted detections under swapped class IDs.

This failure was masked through v4 because early validation was performed using a secondary export path rather than the production path. The two paths agreed numerically on mAP50 but diverged on class assignment — a discrepancy invisible to aggregate metrics. The failure surfaced only when end-to-end production testing revealed that target-class detections were being reported under the moving-object class in downstream logic.

The Ground Zero response was a full dataset reset: all prior annotations were discarded, the class ordering was re-specified with explicit enforcement in the export pipeline, and annotation was restarted from scratch. This decision — painful in the short term — is the direct cause of the data quality improvement that produced v15. A decision to patch rather than reset would have preserved annotation volume at the cost of perpetuating label ambiguity.

Hard rule derived: Every class in every detection task must have a canonical, stable integer index enforced at annotation time, training time, and export time. The same index map must be validated end-to-end in the production inference path before any training run is considered complete.

3.4 v11 to v15 Transition

v11 represented the post-Ground-Zero high-water mark for dataset scale: 4,398 frames across multiple sensor configurations, with semi-automated annotation. The ~73% mAP50 result was disappointing relative to the annotation investment and led to a systematic audit of label quality. The audit identified three failure categories in the v11 dataset: (1) inconsistent bounding box tightness across annotators, particularly for motion-blurred moving-object frames; (2) missing hard negative examples for the most common false-positive trigger objects; and (3) sensor coverage gaps where certain capture angles and lighting conditions were overrepresented in training but underrepresented in the validation set.

The v11-to-v15 transition was not an incremental improvement. It was a deliberate dataset contraction: instead of adding more frames to address the quality issues, the team removed frames that failed a per-frame quality gate and replaced them with a smaller number of carefully selected, hand-annotated frames chosen to address the identified coverage gaps. The result was a training set 88% smaller than v11 that produced a model roughly 27 mAP50 points higher.

4. Technical Analysis

4.1 Data Quality as Primary Variable

Figure 1 presents the central controlled experiment of this paper. Both v11 and v15 used the identical detector architecture with identical training code and hyperparameters. The only variable is the training dataset. The roughly 27-point mAP50 improvement (~73% → over 99%) cannot be attributed to architectural change, hyperparameter tuning, or augmentation strategy, because none of those variables changed between v11 and v15.

Figure 2 shows the dataset size comparison. v15 is 88% smaller than v11 — 505 frames versus 4,398. This result challenges the common practitioner intuition that more data is always better. In this domain, at this dataset scale, the marginal value of an additional noisily-labeled frame is negative: it introduces label inconsistency that degrades model calibration more than the added coverage improves it.

The mechanism is consistent with findings in the transfer learning literature [5]: fine-tuned models on small, high-quality datasets can match or exceed models trained on large, noisily-labeled datasets. The pre-training compute on the pretrained base checkpoint provides a strong initialization; fine-tuning is sensitive to label noise in proportion to how narrow the target domain is relative to the pre-training distribution.

4.2 Sensor Coverage

Sensor coverage failures — training distributions that do not match inference distributions — are the second major failure mode in this arc. The v11 dataset was collected primarily in one capture configuration: a fixed camera angle, controlled lighting, a narrow range of object trajectories. When the model was evaluated on frames from a different capture angle, performance collapsed to near-chance for the moving-object class.

The v15 dataset was constructed with explicit sensor coverage auditing: each frame selected for the training set was annotated with its capture angle, lighting condition, and trajectory class. Stratified sampling was used to ensure that the training set included frames from all production capture configurations. This annotation-time bookkeeping cost approximately 15% additional annotation time but is estimated to have contributed 8–10 mAP50 points on out-of-distribution evaluation subsets.

Hard rule derived: Before finalizing any training set, enumerate all production sensor configurations. Verify that the training set contains a minimum of N examples from each configuration. If any configuration is missing, collect before training — do not train and patch.

4.3 Hard Negatives

The v11 model produced a high false-positive rate on two background object categories: reflective playing surfaces under direct lighting, and small hand-held objects in the frame periphery. Both were present in the inference environment but absent from the training set. The model had no negative examples to anchor against these object types and defaulted to emitting them under the moving-object class.

v15 included a dedicated hard-negative mining pass: 80 frames were selected specifically because they contained the false-positive trigger objects without any positive class instances. These frames were annotated with the background class label only and added to the training set. The false-positive rate on these trigger objects dropped to near-zero in v15 evaluation.

Hard rule derived: Run the in-progress model against a representative sample of production inference frames before each training iteration. Identify the top false-positive trigger objects. Add hard negative examples for those triggers to the next training set. Hard negative mining is not a one-time operation; it is a recurring step in the training loop.

4.4 Automated Labeling Toolchain

The v15 dataset was hand-annotated, but the annotation rate of 60 frames per hour at domain-expert quality is not scalable for larger dataset targets. Figure 3 shows the throughput comparison between manual annotation from scratch and a human-review pipeline built on Grounding DINO 1.5 [3] + SAM 2 [4] pseudo-label generation with human review of each generated label.

The Grounding DINO 1.5 + SAM 2 pipeline operates as follows: Grounding DINO 1.5 generates open-vocabulary bounding box candidates given natural-language class prompts; SAM 2 refines the bounding boxes into precise segmentation masks that are converted back to axis-aligned bounding boxes for detection training. The human reviewer inspects each generated annotation and either accepts, corrects, or rejects it. Rejection rate on the target domain was approximately 15–20%, meaning 80–85% of generated annotations were accepted with minor corrections.

The critical constraint: no pseudo-label is accepted into the training set without human review. This is not a quality recommendation — it is a pipeline invariant. Automated label generation at scale produces systematic failure modes that are not detectable from aggregate accuracy metrics on held-out sets. Human review is the quality gate that prevents systematic error propagation of the type that produced the v6 regression.

4.5 DINOv2 Self-Supervised Features

DINOv2 [1] was evaluated as a component of the dataset quality pipeline rather than as a training backbone. Specifically, DINOv2 patch features were used to identify near-duplicate frames in the training set (cosine similarity above 0.97 in the patch feature space) and to flag potential distribution outliers (frames whose nearest-neighbor distance in feature space exceeded two standard deviations from the training set mean).

Near-duplicate detection removed approximately 12% of the v11 candidate frames — frames that added annotation cost without adding distributional diversity. Outlier detection flagged 8 frames that, on manual inspection, were found to contain annotation errors. Both applications of DINOv2 are data curation operations, not training operations: the DINOv2 features were used to improve the quality of the training set fed to the production detector, not to replace it.

4.6 Active Learning

The active learning literature [6] reports that uncertainty sampling combined with diversity sampling can achieve greater than 85% of fully-supervised performance at 10% annotation budget. This finding is directionally consistent with the v11-to-v15 result: 505 frames (11.5% of 4,398) substantially outperformed the full v11 dataset on the target domain. The mechanism differs — v15 used human-driven coverage auditing rather than a formal active learning algorithm — but the underlying principle is the same: the informationally relevant portion of a training dataset is highly non-uniform. Strategic sample selection dominates random sample collection.

For future dataset expansion beyond the v15 baseline, a formal active learning pipeline is planned: the current v15 model is used to score candidate frames by prediction uncertainty (entropy of the class probability distribution), and high-uncertainty frames are prioritized for human annotation. This approach is expected to maintain annotation efficiency as the dataset grows beyond the manually curatable scale.

5. Performance Results

Table 2 situates the v15 production result in the context of recent published benchmarks for related methods. v15’s over-99% mAP50 on the target domain exceeds all zero-shot baselines by a wide margin, which is expected: zero-shot detection systems are not optimized for narrow, proprietary object classes. The meaningful comparison is against the v11 baseline (same architecture, same domain) and against the active learning efficiency results (annotation budget versus performance).

The v15 result should be interpreted as an upper-bound estimate for in-distribution performance: the training and validation sets share the same sensor configurations, capture conditions, and annotator. Out-of-distribution generalization — to new venues, new sensor positions, or new object varieties — was not evaluated in this study and is a known limitation of the current dataset scope.

6. Comparative Analysis

The roughly 27-point mAP50 gap between v11 and v15 is larger than any architectural improvement documented in the real-time-detector literature: backbone scaling within a model family typically produces a 3–5 mAP improvement on COCO. Domain adaptation techniques typically produce 5–15 mAP improvements on narrow domains. A roughly 27-point improvement from data curation alone, without any architectural change, represents an order of magnitude more leverage than is available through model selection at this scale.

This result is consistent with the broader shift toward data-centric AI in the applied ML community [5]. The academic literature on model architecture continues to optimize for benchmark performance on large, well-curated public datasets (COCO, LVIS, ImageNet). Practitioners working on narrow custom domains are in a structurally different situation: their baseline data quality is not comparable to COCO quality, and the marginal return on data curation investment is correspondingly higher.

The zero-shot detection results from Grounding DINO 1.5 [3] (55.7% zero-shot LVIS-minival AP) confirm that even the strongest open-vocabulary detectors cannot bridge the domain gap for highly specialized object categories without domain-specific training data. Zero-shot results on the target domain in this work (not reported in Table 2, available on request) were below 30% mAP50 — well below the production threshold. The practical implication is that custom-domain detection tasks at production quality levels require custom training data, and the quality of that data is the primary performance lever.

7. Open Problems

• Scaling annotation beyond 505 frames without quality regression. The v15 dataset is hand-annotated at a rate that is not sustainable for 10x dataset growth. The Grounding DINO 1.5 + SAM 2 pipeline (Section 4.4) addresses throughput but requires formal validation that pseudo-label quality is consistent with hand-annotation quality on the target domain. A controlled experiment comparing models trained on hand-annotated vs. reviewed pseudo-labeled datasets at matched frame counts is needed before scaling the automated pipeline.
• Out-of-distribution venue generalization. The current v15 model is evaluated only on the training venue sensor configuration. Production deployment to new venues with different lighting, court surfaces, and camera positions requires a venue adaptation protocol. Whether this is best handled by few-shot fine-tuning per venue, by expanding the training set to cover venue diversity, or by test-time domain adaptation is an open question.
• Occlusion and partial visibility. Frames where the object is partially occluded by a player or fixed structure are systematically underrepresented in the current training set. These frames are also the highest-stakes detection cases in production (occlusion-heavy moments are often the contested events). Targeted collection of occlusion frames and development of an occlusion-aware annotation protocol is a high-priority gap.
• Formal active learning integration. The strategic sample selection in v15 was human-driven. Replacing it with a formal active learning loop using v15 as the seed model is planned but not yet validated. The primary risk is uncertainty sampling bias: high-uncertainty frames may be high-uncertainty because they are genuinely hard examples (useful) or because they are corrupted or ambiguous (harmful). A diversity constraint is required to prevent the active learning loop from concentrating annotation budget on a narrow subset of the difficulty distribution.

8. Conclusion

Fifteen training iterations and one controlled natural experiment converge on a single, operationally useful conclusion: for narrow-domain detection tasks at fine-tuning scale, data curation discipline is the dominant performance variable. Architecture selection, hyperparameter optimization, and augmentation strategy are secondary by at least an order of magnitude in the regime where label quality is low. Practitioners should allocate their iteration budget accordingly — investing in annotation tooling, coverage auditing, and hard negative mining before investing in model search or hyperparameter sweeps.

The three hard rules derived from this arc — class index enforcement end-to-end, sensor coverage auditing before each training run, and recurring hard negative mining — are not novel claims. They are documentation of well-understood principles that are nonetheless routinely violated in production training pipelines because the cost of violation is not visible until it materializes as a dramatic performance regression. This paper provides the quantitative evidence that makes that cost visible.

The 2024–2026 automated labeling toolchain (Grounding DINO 1.5, SAM 2, DINOv2) is mature enough for production use in human-in-the-loop annotation pipelines. The critical constraint is that human review must remain mandatory regardless of automation level. Automated label generation accelerates annotation throughput; it does not eliminate the need for human quality gates. That constraint applies whether the annotation budget is 500 frames or 50,000.

References

1. Oquab, M., et al. (2024). DINOv2: Learning Robust Visual Features without Supervision. TMLR 2024; arXiv:2304.07193.
2. Liu, S., et al. (2024). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. ECCV 2024; arXiv:2303.05499.
3. Ren, T., et al. (2024). Grounding DINO 1.5: Advance the ‘Edge’ of Open-Set Object Detection. arXiv:2405.10300.
4. Ravi, N., et al. (2024). SAM 2: Segment Anything in Images and Videos. arXiv:2408.00714. Meta AI.
5. Chlap, P., et al. (2024). Transfer learning with generative models for object detection on limited datasets. arXiv:2402.06784.
6. Li, Z., et al. (2025). Uncertainty Meets Diversity: A Comprehensive Active Learning Framework for Indoor 3D Object Detection. CVPR 2025; arXiv:2503.16125.