TRIPOD-AI and PROBAST-AI: Reporting AI Diagnostic Models

A reviewer reading an AI diagnostic model publication or technical file, a Notified Body assessor, or a TITCK expert holds two separate questions in hand. The first: "Has this study reported enough information transparently?" The second: "In light of what has been reported, can I trust this model's performance claim?" Separating these two questions from each other is the foundation of modern AI assessment.

The first question is answered by TRIPOD-AI, the second by PROBAST-AI. The two were published simultaneously in BMJ in 2024 and are the adaptations of the classic TRIPOD (2015) and PROBAST (2019) guidelines to machine learning models. If you are going to develop and publish a diagnostic/prognostic model, or place it in a device file, these two documents should be on your desk.

What TRIPOD-AI is and what it reports

TRIPOD-AI is the expansion of the abbreviation "Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis - Artificial Intelligence." It is a checklist of 27 main items and reports the model's life cycle end to end:

• Data source and participants: Which center, which date range, which inclusion/exclusion criteria, how many patients, how many images/records. How the training and evaluation sets were separated.
• Predictors (inputs): The definition of the model inputs, the time of measurement, how missing data was handled (imputation strategy).
• Outcome (reference standard / ground truth): How the target was defined, by whom, with which method it was labeled, inter-reader agreement.
• Model development: Architecture, hyperparameter selection, class imbalance management, the software used and its versions.
• Performance: Not only discrimination (AUC-ROC), but also calibration (calibration plot, slope, intercept) and, where possible, clinical utility (decision curve analysis, net benefit).

The most frequently skipped aspect of TRIPOD-AI is this: the guideline asks not for a performance number, but for enough information to reproduce that number. The sentence "AUC 0.94" does not comply with TRIPOD-AI; what complies is in which population, against which reference standard, and with which uncertainty interval (95% CI) it was measured.

What PROBAST-AI is and what it assesses

PROBAST-AI ("Prediction model Risk Of Bias ASsessment Tool - AI") is not a reporting tool but an assessment tool. That is, you report the study, and someone else (a reviewer, systematic review author, regulator) judges your study with PROBAST-AI. It works across four domains:

1. Participants: Is the data source appropriate, does the population represent the intended use?
2. Predictors: Are the inputs defined as they are in clinical use, independently of outcome information?
3. Outcome: Is the reference standard reliable, determined independently of the predictors?
4. Analysis: Is the sample size sufficient, was missing data handled correctly, was overfitting/optimism corrected, was calibration assessed?

Each domain is scored from two perspectives: risk of bias (is the result distorted) and applicability (does this study overlap with my question). The "analysis" domain of PROBAST-AI is the place that most often comes out "high risk" in machine learning models, because it is exactly here that data leakage, small sample size, and missing calibration accumulate.

The difference between TRIPOD-AI and PROBAST-AI: what answers what

Confusing these two is a very common mistake. The clear distinction:

• TRIPOD-AI = reporting standard. For the author. "Did I write my study completely and transparently?"
• PROBAST-AI = risk-of-bias tool. For the assessor. "Can I trust the result of this study?"

Their relationship is one-directional: a study reported in full compliance with TRIPOD-AI makes a PROBAST-AI assessment possible. In an incompletely reported study, the assessor marks most domains "no information," which in practice means low trust. So good reporting is a precondition for low risk-of-bias but is not sufficient on its own: a poorly designed but honestly reported study can also come out "high risk" in PROBAST-AI.

The gaps reviewers and regulators look at most often

Over the years, the same weaknesses recur in both review processes and file assessments. The most common gaps that undermine an AI diagnostic model claim:

• Data leakage: The train-test split being done at the image or slice level, not at the patient level. If one slice of the same patient is in the training set and another slice is in the test set, performance is artificially inflated by 15-25%. PROBAST-AI marks this directly as "high risk."
• Claim based on a single metric: Only AUC-ROC being reported. AUC measures discrimination, not calibration. A model with excellent AUC may have calibration that cannot be used in the clinic. TRIPOD-AI asks for a calibration plot; its absence is the first question a regulator asks.
• Absence of external validation: The model being tested only in the center where it was developed, on the same device and the same population. In EU-MDR and FDA files, an independent validation done on a different center/device/population (ideally standalone) is expected.
• Unreported subgroup performance: Performance not being provided by sex, age, device manufacturer, or imaging protocol. EU AI Act Article 10 (for high-risk systems as of August 2026) explicitly requires the representativeness of training data and bias management; subgroup performance is the evidence of this.
• Skipping optimism correction: Optimism arising from overfitting not being corrected with bootstrapping or internal validation on a small sample. This is the classic breaking point of the PROBAST-AI "analysis" domain.

To scan your diagnostic model's TRIPOD-AI compliance and PROBAST-AI risk-of-bias profile with an independent eye before publication or submission, request a 15-minute free scoping.

Where these guidelines stand in the regulatory file

TRIPOD-AI and PROBAST-AI were born as academic reporting tools, but in practice they also feed the performance section of device files. In the TITCK CDSS (clinical decision support software) assessment, the transparent reporting of standalone clinical validation is expected; TRIPOD-AI is the skeleton of this report. The performance evaluation within the EU-MDR Clinical Evaluation Report (CER) follows the MEDDEV 2.7/1 Rev 4 methodology, and here the model's analytical/clinical performance evidence is strengthened by a PROBAST-AI type bias assessment. In the FDA AI/ML framework, too, documenting the development and validation of the algorithm in a transparent, reproducible manner is a fundamental expectation.

Here it is necessary to draw the scope boundary clearly: as a named methodologist and biostatistician, I design the validation study, run its statistics, and write the TRIPOD-AI compliant report. ISO 13485/QMS documentation, product registration paperwork, and device law are separate areas of expertise; your regulatory consultant handles those tasks. What I provide is the scientific core of the file: the study being statistically defensible.

Common Mistakes

• Thinking of TRIPOD-AI as a "writing rule." The guideline defines content coverage, not format: a missing calibration or sample size calculation will bring down even a beautifully formatted article at the reviewer's desk.
• Thinking PROBAST-AI is specific only to systematic reviews. Applying the tool to your own study, as a pre-publication self-assessment, is the cheapest risk reduction method; you close the "high risk" domains before the reviewer.
• Equating external validation with temporal validation. A different time period in the same center is not true external validation; a different center, different device, different population is needed. The regulator knows this difference.
• Thinking of fairness and subgroup analysis as "extra." Within the framework of EU AI Act Article 10, representativeness and bias management are a legal expectation for high-risk systems, not an optional bonus.

Related Articles

• How to Design a SaMD Clinical Validation Study
• How to Prepare a TITCK-CDSS Clinical Validation Report
• Data Leakage and Train-Test Split in AI Validation

TRIPOD-AI and PROBAST-AI institutionalize the difference between an AI diagnostic model "looking good" and "being trustworthy." When used correctly, what convinces the publication reviewer and the regulator is not the size of the AUC, but the transparency about how that AUC was produced and the control of bias. Solidifying this core before publication or submission is many times cheaper than dealing with a major revision or a file deficiency afterward. Scope, duration, and budget differ for every file; we clarify these in a free scoping call.

Free Scoping Call