The performance of your artificial intelligence model is limited by the quality of the labels it is trained on. A model published with a claim of 95% accuracy will perform far worse in a real clinical setting if its labeling protocol is flawed. This situation is described in the medical AI literature as "garbage in, garbage out," and it is one of the field's biggest methodological problems.
Why Is Ground Truth Not Simply "an Expert's Annotation"?
The most common misconception is this: "If a radiologist or surgeon annotates the images, the ground truth is ready."
No. Ground truth is a protocol, not one person's annotation.
A data set labeled by a single expert has several serious problems. The expert's own inconsistency (intra-rater variability) changes over time, and they may label the same image differently two weeks apart. Specialty-specific biases (specialty bias) are transferred to the model. Most importantly: a study conducted with a single labeler is regarded as a methodological weakness in high-impact journals and is flagged by reviewers.
Disagreement Between Experts Is Real
Studies conducted in radiology reveal that experienced experts show a significant level of disagreement when they independently evaluate the same image. This is similar in pathology, dermatology, and orthopedics.
This disagreement does not mean poor expertise. There is genuine ambiguity in the interpretation of medical images, and when this ambiguity is transferred to the model, it causes the model to behave ambiguously as well.
The ground truth protocol must define in advance how it will manage this disagreement. What is done in the case of disagreement? Does a third expert step in? Is the decision made by voting? Are ambiguous cases removed from the data set? Each of these decisions affects the model's behavior.
The Most Common Mistakes and Their Consequences
Starting without a labeling guideline: If two experts understand the same term differently, for example if they use different criteria for "grade 2 OA," the data set becomes systematically inconsistent. The model learns these inconsistencies.
Not reporting inter-rater agreement: AI studies submitted for publication without calculating Cohen's kappa or ICC are directly rejected by reviewers. The sentence "Two independent radiologists annotated the images" is no longer sufficient; an agreement coefficient is requested.
Ignoring labeler fatigue: The attention quality of an expert who labels hundreds of images one after another declines. If session duration, break protocol, and quality control checkpoints are not defined, the second half of the data set may be systematically different from the first half.
No protocol for borderline cases: Every data set has "borderline" cases that make diagnosis difficult. If it has not been determined in advance how these will be handled systematically, each labeler makes a different decision.
Where Do People Get Stuck in This Process?
The points where researchers struggle the most are these: Selecting and setting up the labeling software. Transferring DICOM files to the labeling platform. Coordinating and tracking expert time. Automating the quality control process. The resolution protocol for disagreement cases.
Each of these requires a separate technical and organizational decision. If these decisions are not determined before data collection begins, correcting them afterward is very costly.
Request a free 30-minute consultation to design the ground truth protocol for your medical AI study together.
Where Do People Get Stuck Most in This Analysis?
- Two experts labeled the same image differently, inter-rater agreement is low, and you cannot decide which one is correct.
- You created an annotation protocol, but when applied to real data, too many borderline cases appear.
- The ground truth creation process took 5 times longer than you expected and is still not finished.