Clinical AI Validation Pipeline: TRIPOD/DECIDE-AI

ROC-AUC is not a model report, it is a starting point

The two numbers you most often see when reading a clinical AI model publication: sensitivity and specificity. "Sensitivity 92%, specificity 88%, AUC 0.94, our model is very successful." The text ends with that sentence.

The problem is this: ROC-AUC alone does not indicate clinical benefit. Two models with the same AUC can have different calibration curves, one usable and the other not. With the same AUC they can produce different net benefit in different populations. Depending on the clinical decision-making threshold, 92% sensitivity may be meaningful or meaningless for all your patients.

In 2024-2025, the TRIPOD-AI and DECIDE-AI guidelines were released. They redefined how AI studies should be reported. The old "AUC + sensitivity + specificity" format is no longer sufficient. Correct reporting: calibration, decision curve analysis, subgroup analyses, fairness metrics, prospective deployment.

In this article I explain the four stages of the clinical AI validation pipeline, what needs to be done at each stage, ethics committee and TITCK compliance, and reporting standards.

Stage 1: Internal Validation

At the model development stage. On your own data set.

Data split

• Train (70%): Trains the model.
• Validation (15%): Hyperparameter optimization (learning rate, batch size, regularization).
• Test (15%): Only to measure the performance of the final model. Used once.

Critical rule: Split at the patient level. The same patient cannot be in both train and test. A naive approach in a CT image split creates data leakage, yielding 15-25% inflated performance.

Cross-validation alternative

For small data sets (n < 500), k-fold cross-validation is preferred. 5-fold is common. Again stratified at the patient level.

What needs to be reported (TRIPOD-AI 2024)

Why calibration is mandatory

If a model says "disease present" with 85% probability, disease should truly be present in 85% of the patients. When a poorly calibrated model says 85%, the true rate could be 60% or 95%. If the clinician trusts the model's probability, poor calibration leads to the wrong decision.

Calibration plot: The X axis is the model predictions (divided into bins), the Y axis the true rates. The ideal is the y = x line. Deviation is calibration error.

Correction: Post-hoc calibration with Platt scaling or isotonic regression. No retraining is required.

Stage 2: External Validation

The same model, a different data source. This is the most important and most frequently skipped stage of clinical AI.

Three types of external validation

1. Temporal: Same center, different time period (model trained on 2022 data, tested on 2024 data). For capturing drift.

2. Geographic: Different city / center. Same patient typology but different device, protocol, clinician.

3. Domain: Different population (adult model, pediatric patients). Different device manufacturer (trained on GE, tested on Siemens).

Typical performance drop

That is, a model with an internal AUC of 0.92 may show performance between 0.75 and 0.84 externally. This is normal. The problem is overstating the internal and not doing the external.

Cohort reporting

For each external set:

• N patients
• Age, sex, disease severity distribution
• Device / protocol differences
• Outcome incidence
• Ground truth standard used

Stage 3: Prospective Deployment Study (DECIDE-AI)

Measures how the model works in the real clinical workflow. Three main study designs:

Design A: Silent deployment

The model is not shown to the clinician. It makes predictions in the background and does not affect the clinical decision process. Afterwards, the clinical decision is compared with the model output.

Purpose: How the model works in the real world, how the clinical decision-making rate changes.

Duration: 3-6 months, n ≥ 500 patients.

Design B: Side-by-side (clinician + AI)

The model output is shown to the clinician. The clinician makes the decision. The rate of decision change + its quality is measured.

Purpose: Does the model produce value as clinical decision support?

Duration: 6-12 months, n ≥ 1000 patients.

Design C: Randomized clinical trial (RCT)

Patients are randomized: one arm classic workflow, the other arm AI-assisted workflow. Clinical outcome comparison.

Purpose: Does AI improve clinical outcome (mortality, morbidity, length of stay, readmission)?

Duration: 1-3 years, n ≥ 5000 patients. The highest level of evidence.

DECIDE-AI reporting (2024)

The DECIDE-AI checklist contains 27 items. The important ones:

• Definition of the clinical workflow before and after AI
• Clinician-AI interaction design (override option, explainability)
• Measurement of clinician trust and acceptance
• Error modes (false positive vs false negative implications)
• Early stopping rules

Stage 4: Post-Market Surveillance

Continuous follow-up of the model after it goes into live use. The part that has been mandatory since 2023 with the FDA's "Predetermined Change Control Plan" (PCCP) concept.

Drift monitoring

Three types of drift:

1. Data drift: The distribution of the input data changes (new device, new protocol).
2. Concept drift: The relationship between input and output changes (new disease subtype, population change).
3. Performance drift: The performance of the model output declines.

Practical technique: Monthly monitoring dashboard.

• Input feature distribution (KL divergence vs baseline)
• Prediction distribution (entropy)
• Subset performance (as feedback data is received)

Retraining triggers

Automatic retraining when the performance drop threshold is exceeded. This plan must be predefined in the PCCP.

Adverse event tracking

A reporting mechanism when there is an error stemming from the model (false positive → unnecessary biopsy, false negative → missed diagnosis).

The Regulatory Framework in Turkey

TITCK SaMD (Software as a Medical Device)

Three classes:

• Class A: For informational purposes (pre-assessment suggestion in radiology). Lowest risk.
• Class B: Diagnostic support (cardiac rhythm anomaly detection). Medium risk.
• Class C: Diagnosis + treatment guidance (automatic bolus dosage calculation). Highest risk.

Depending on the class, the validation evidence need, the number of clinical studies, and the reporting burden increase.

KVKK Art. 6 special categories of personal data

Health data is a special category. Informed consent + explicit consent are required. For AI training data:

• If anonymization is possible: KVKK does not apply (Art. 28/1).
• Pseudonymization: KVKK applies, but complies with the data minimization principle.
• Fully identified data: Explicit consent is mandatory.

Ethics committee requirements

• Retrospective data study: Local ethics committee approval is sufficient (usually 1-2 months).
• Prospective study: Local ethics committee + TITCK clinical research approval.
• Multi-center prospective: Ministry coordination.

The headings added since 2024 for the technical section of the ethics committee application for AI studies:

• Algorithm description (architecture, training data, hyperparameters)
• Ground truth standard + number of expert evaluators
• Data security plan (which server, which encryption, who has access)
• Drift monitoring + retraining plan

Typical Mistakes and Their Corrections

Practical Pipeline Example

The timeline of a typical radiology AI study (example: pulmonary nodule detection on chest CT):

A total of 24+ months. This can be accelerated but cannot be skipped over.

Conclusion

Clinical AI is not an academic prototype, it is a medical device. The validation discipline resembles that of drug clinical research but has additional layers specific to AI: the external validation requirement, calibration reporting, drift monitoring, the DECIDE-AI prospective study.

Producing a publication with internal validation alone is still possible (60% of medical journals accept it) but these articles rarely turn into clinical use. If one works toward a goal of "clinical impact" instead of a publication goal, external validation + DECIDE-AI + TITCK compliance are planned from the very beginning.

Two critical traps for medical AI ventures in Turkey: (1) chasing only academic AUC, (2) leaving the regulatory process to the end. The right approach: TRIPOD-AI compliant study design + parallel TITCK + advancing the ethics committee + an 18-24 month realistic roadmap. You can also review how a SaMD clinical-evidence study is structured end to end.

If your organization has a similar artificial intelligence or data engineering need, we can evaluate it together within the scope of professional consulting.