Medical Dataset Curation: Ground Truth, Annotation, and KVKK Compliance

The dataset comes before the model

The most frequently heard sentence in the first AI study meeting with a physician is: "Which model will we use?" The answer is this: before we reach that stage, we will work on the dataset for 2-3 months. Seventy percent of the success of an AI study is not the model, it is dataset quality.

With modern transfer learning, reasonable performance can be obtained even on small datasets. But "the dataset may be small" and "the dataset may be sloppy" are not the same thing. A rigorous dataset of 100 patients produces a better model than a sloppy dataset of 1000 patients. The reproducibility crisis in clinical AI comes not from the model, but from dataset inadequacy.

In this article, I describe the six stages from clinical data to a model-ready dataset, the typical mistakes at each stage, and where KVKK compliance fits in.

Stage 1: PHI removal and pseudonymization

Clinical data is sensitive personal data. It falls into the category of special categories of personal data under KVKK Article 6. No AI study should contain identifying information.

PHI cleaning on DICOM:

DICOM files contain rich metadata. Patient name, date of birth, national ID number, patient ID, institution name, operator name, study description, study dates. All of them need to be cleaned or pseudonymized.

Standard approach: DICOM De-identification Profile (DICOM PS 3.15 Annex E).

• Basic Profile: cleaning of core PHI tags
• Retain Patient Characteristics: age, sex, weight are retained (critical for clinical meaning)
• Retain Longitudinal Temporal Information: dates are shifted with a patient-level offset (study ordering is preserved, the real date is lost)

Pseudonymization vs anonymization:

• Anonymization: identity cannot be reversed. If your dataset is fully anonymous, KVKK does not apply.
• Pseudonymization: identity can be reversed via a mapping table. KVKK applies, but it complies with the data minimization principle.

In a clinical AI study, pseudonymization is generally preferred, because:

• In dataset quality control, the question "could this image belong to the same patient?" can be asked
• In the train/test split, separation can be done at the patient level (prevents data leakage)
• In ethics committee revision, the patient can be recalled

Mapping table security: The Patient_ID -> pseudonym_ID table is stored separately from the dataset. In an encrypted vault. Access only for the study owner.

Common mistake: While cleaning DICOM tags, overlooking the "burned-in PHI" in the image pixel data. In some modalities (especially older ultrasound devices) patient information is written onto the image. This pixel must also be cleaned, otherwise PHI remains in the model input.

Stage 2: DICOM standardization

Even images of the same modality coming from different devices of the same hospital are not identical. Voxel spacing, intensity range, orientation, and slice thickness vary depending on the manufacturer, model, and parameter set. Training does not begin without standardizing the model input.

Standardization steps:

1. Orientation: RAS, LPS, ITK convention. All images must be brought to the same orientation. If one image is RAS and another is LPS, the model gets confused about right and left.
2. Voxel spacing: All images must be resampled to the same isotropic or defined anisotropic spacing.
3. Intensity normalization: For CT, Hounsfield Unit clipping (example: between -200 and 1500 HU for bone), for MR, z-score normalization or histogram matching.
4. ROI cropping or padding: The model input must be a fixed size. Anatomy-centered crop or zero padding.

The pipeline for these steps is generally built with Python + SimpleITK + nibabel + monai. Which parameter each step was run with must be kept under version control, because a single parameter change requires regenerating the entire dataset.

Stage 3: Ground Truth Protocol

Ground truth is the "correct answer" the model will learn. How this answer is produced is the foundation of the scientific value of the study. The TRIPOD and TRIPOD-AI reporting guidelines hold this point very strictly.

Ground truth types:

• Clinician annotation (manual): One or more experts label the image/data. The most common approach. It contains subjective variability; the number of experts and the consensus protocol are critical.
• Clinical evidence (clinical gold standard): Histopathology, surgical finding, clinical follow-up. To be able to say "there is a tumor" on an image, the biopsy result must confirm it. This approach is more reliable, but the data collection cost is high.
• Mixed: Manual annotation + clinical evidence are combined. Example: manual mask for segmentation + clinical diagnosis for classification.

Protocol questions:

• How many experts will annotate?
• Which specialty, how many years of experience?
• How will consensus be established? Multi-rater voting, third-arbiter, atlas-conformance?
• What happens to uncertain cases? Exclusion, or a separate category?
• How will inter-rater agreement be measured (Cohen's Kappa, ICC, Dice, IoU)?
• Intra-rater agreement (how consistent is the same expert when re-labeling the same case)?

Common mistake: Single-expert annotation. Acceptable for a pilot study, but in a study going to publication there should be at least two, ideally three experts, and inter-rater agreement should be reported.

Stage 4: Annotation Workflow

The transition from protocol to execution. Here the triangle of software infrastructure + expert time + quality control is established.

Annotation tools:

• 3D Slicer: The most widely used open source for medical image segmentation. Supports multi-rater workflows.
• ITK-SNAP: Lighter, single-user. Sufficient for classic segmentation.
• MD.ai, Labelbox, V7 Darwin: Cloud-based, multi-user, audit log included. There is a commercial cost, but team coordination becomes easier on large datasets.
• OHIF + viewer + custom backend: Preferred for institution-specific annotation platforms.

Pre-annotation pilot: All experts label the first 10-20 cases. Inter-rater agreement is calculated. If it is low, the protocol is reviewed, training is conducted, and a calibration meeting is held. If this step is skipped, when the entire 300-case dataset has been annotated, agreement comes out low and you start over.

Estimated durations (average):

• 2D radiological image (X-ray, mammography) bounding box: 30-60 seconds
• 2D segmentation mask: 2-5 minutes
• 3D MR/CT segmentation (example: a single anatomical structure): 10-30 minutes
• 3D multi-class segmentation (4-8 classes): 30-90 minutes

For a 500-patient dataset, 3D multi-class segmentation is 250-750 hours of work for a single expert. Double that with two experts. This calculation is done before the pilot and budgeted.

Stage 5: Quality Control

After annotation is complete, automated and manual quality control.

Automated checks:

• Unexpected voxel spacing, orientation, label class
• Empty annotation (all voxels = 0)
• Excessively small annotation (example: a lesion mask smaller than 5 voxels, either an error or an extreme edge case)
• Class imbalance report

Manual checks:

• Review of 10% of cases via random sampling
• Separate examination of outlier cases (the smallest and largest annotations)
• Separate reporting of cases with high inter-rater disagreement

Re-annotation rule: Cases where a clear error is caught are re-labeled. A log is kept of how many cases, for what reason, and on what date they were changed.

Stage 6: Train / Validation / Test Split

A common mistake in modern AI studies: hospital-level or random split. The correct split is patient-level and stratified.

Patient-level split: If different images of the same patient are not kept in different sets, there is data leakage. 80 of Hospital A's 100 patients are in train, 20 in test. If a follow-up image of a training patient leaks into the test set, the model appears to have "learned," when in reality it recognizes the same patient.

Stratified split: The class distribution is preserved. If there are 20% positive cases in train, there should also be 20% in test. If there is outcome imbalance, stratification is essential.

External validation set: If possible, an independent test set from a different institution. A model trained on data from the same institution and tested with data from the same institution may not generalize to an external institution (selection bias, equipment bias).

Typical ratios: 70% train, 15% validation (for hyperparameter tuning), 15% test (hold-out, one-time after the model is finalized). If the dataset is small, k-fold cross-validation is preferred.

TRIPOD-AI 2024 reporting: How the train, validation, and external test sets were split, according to which characteristics they were stratified, and the patient count + demographic summary must be reported. Incomplete reporting is a cause for major revision.

The KVKK compliance chain

KVKK compliance throughout the entire process:

1. Data collection: Ethics committee approval + KVKK-compliant explicit consent or public interest basis (in retrospective studies)
2. Data storage: Encrypted storage, access control, audit log
3. Data processing: Only within the scope of the study, not used for any other purpose
4. Data sharing: An anonymized dataset may be shared in a publication or repository; the mapping table is not shared
5. Data destruction: When the study ends, the mapping table is destroyed within the specified period (generally 5 years, according to the ethics committee decision)

Information notice and explicit consent: Prepared according to the study protocol, passes through ethics committee approval. In retrospective studies, it is evaluated within the framework of the public interest + scientific research exception.

Common mistakes and their cost

Roadmap

Typical timeline for a first AI study:

• Weeks 1-2: Protocol writing (study question, ground truth definition, ethics committee)
• Weeks 3-4: Preparation of the ethics committee application, contracts
• Weeks 5-12 (while awaiting the ethics committee): Testing the DICOM pipeline and annotation tool on a pilot dataset
• Weeks 12-24: Data collection + annotation (in parallel)
• Weeks 24-28: Quality control, train/val/test split
• Week 28+: Model training, reporting

Most physicians think "let's write the model, let's finish it in three months." The reality: a serious AI study takes 6 months at the dataset stage, and 3-6 months more for the model + reporting. The only thing that shortens this time is giving the dataset its due early.

For support with your research data infrastructure, you can review the academic consulting services.