How to Design a SaMD Clinical Validation Study

For SaMD, the most common reason for failure is not technical: while the model shows excellent performance on the internal test set, the statistical skeleton of a real validation study has never been built. When the regulator or notified body opens the file, it looks for a pre-locked analysis plan, a defined reference standard, and evidence of external validity. These cannot be produced post-hoc.

This article addresses the methodological design of a SaMD clinical validation study step by step: from selecting the study type to sample size, from defining the reference standard to the distinction between internal and external validation. The aim is to produce evidence that is both clinically meaningful and defensible in a regulatory file.

Statement of Intent First: Clinical Purpose and Intended Use

The first step of the design is not statistics, it is scope. The validation study must be constructed to test the device's intended use statement exactly. The questions that need to be clarified here:

• What is the clinical function? Triage, diagnostic aid (CADx), detection (CADe), quantification, or risk prediction?
• Who is the target population? Age range, clinical indication, imaging modality, device/manufacturer diversity; inclusion and exclusion criteria.
• What is the level of autonomy? Is it making a standalone decision or offering a recommendation to the clinician? This determines whether the study will be standalone or a reader study (MRMC).
• Where is the decision threshold? If the output is a probability score, whichever cut-off will be applied in clinical use must also be fixed in advance in the validation.

This statement of intent is written into the protocol in a way that cannot be changed later. Under EU-MDR, this framework forms the basis of the clinical evaluation plan (MEDDEV 2.7/1 Rev 4 structure); since the TITCK Medical Device Regulation transposes EU-MDR exactly, the same standalone performance expectation also applies to the Turkey file.

Architecture of the Standalone Performance Study

For most SaMD, the primary evidence is a standalone performance study: the model output is measured against a fixed reference standard without any clinician intervention. Design decisions:

• Study type: Usually a retrospective, cross-sectional diagnostic accuracy study. Reporting should sit within the STARD 2015 framework and its artificial intelligence extension (STARD-AI in preparation).
• Pre-locking the data flow: Model weights, preprocessing steps, and the decision threshold are frozen before the validation data is seen (locked model). No hyperparameter tuning is done during validation; otherwise the study is not "validation" but "development".
• Blinded evaluation of index test and reference standard: The experts who set the reference standard must not see the model output, and those who run the model must not see the reference (blinding).
• Pre-defined analysis population: How missing data, technical failure (uninterpretable cases), and exclusions will be handled is specified in the protocol; removing cases afterward produces bias.

Standalone results show the device's raw discriminative power, separated from the clinical workflow effect. This distinction is the first place both the regulator and the reviewers look.

Primary Endpoint and Performance Metrics

The choice of metric depends on the clinical purpose, and the primary endpoint should be single:

• AUC-ROC: Summarizes threshold-independent discrimination; it is common as the primary metric in devices intended for screening and ranking. The confidence interval (usually by the DeLong method) is reported.
• Sensitivity and specificity: Reported at a pre-fixed clinical threshold. In detection (CADe) devices, high sensitivity is usually the priority; in that case specificity becomes a secondary endpoint.
• PPV/NPV: Sensitive to the true prevalence of the target population; if the prevalence of the study sample differs from the population, these values cannot be directly generalized, and this limitation must be stated clearly.
• Calibration: In risk-predicting models it is as important as discrimination; the calibration curve and the observed/expected ratio are reported. TRIPOD-AI explicitly requests these items.

The primary hypothesis is constructed by testing the lower performance bound (for example, a pre-specified performance goal for AUC or sensitivity) against the lower end of the confidence interval. Point estimates of the "our model achieved 95% accuracy" type are not evidence for the regulator without a confidence interval and a pre-defined threshold.

To clarify which metric should be the primary endpoint for your SaMD device and to set the performance target according to the clinical purpose, request a 15-minute free scoping.

Reference Standard (Ground Truth) Design

The quality of the validation study cannot exceed the quality of the reference standard. Design decisions:

• Independence: The reference standard must be separate from the labeling process used in training the model. The situation where the same reader produces both the training and the reference label creates hidden leakage.
• Level of the standard: Where possible, a "hard" reference such as histopathology, follow-up outcome, or multi-expert consensus is preferred. If expert consensus is used, the number of readers, level of experience, and disagreement resolution rule are defined in advance.
• Inter-observer agreement: The agreement between the experts who define the reference (for example, Cohen/Fleiss kappa or ICC) is measured and reported; low agreement indicates that the reference itself is noisy.
• Imperfect reference standard problem: If the reference is not perfect (which it often is not), its effect on the performance estimate must be discussed, and a sensitivity analysis performed if necessary.

Sample Size: Not an Estimate, but Power Analysis

In SaMD validation, sample size is determined not with the logic of "all the cases we have" but with a formal calculation that will estimate the primary endpoint at the targeted precision:

• For AUC: The number of patients and controls is calculated from the targeted confidence interval width and the expected AUC (for example, the Hanley-McNeil approach). The number of positive cases (event count) is often more binding than the total N.
• For sensitivity/specificity: A separate precision target is set for each; with rare target findings, collecting enough positive cases is the hardest part and determines the study duration.
• Subgroup analyses: The device must work in clinically meaningful subgroups (sex, age band, device manufacturer, disease severity). EU AI Act Article 10 (data governance for high-risk systems, binding from August 2026) and FDA AI/ML guidances explicitly expect performance to be reported in subgroups as well and expect data representativeness. This is a constraint that pushes the total sample upward.

Power analysis is a part of the protocol, and therefore of the statistical analysis plan (SAP), and is written before the data is looked at.

Internal vs External Validation: The Real Distinction

This distinction is the backbone of SaMD validation and is the most frequently skipped point:

• Internal validation: This is the evaluation done with samples from the data source on which the model was developed (hold-out test set, k-fold, or bootstrap cross-validation). It reduces optimism but cannot capture distribution shift. It is not sufficient on its own for the regulator.
• External validation: This is testing on independent data from a different source (another center, another device/manufacturer, another time window, another geography) without retraining the model. The main area where PROBAST-AI flags a high risk of bias is the absence of external validation.
• Temporal vs geographic external validation: Future cases from the same center (temporal) are the weakest form of external validity; independent center data (geographic) is stronger. Which one was done must be stated clearly.
• Single-lock rule: The external validation set must be run only once, with the locked model. Looking at the result, making changes to the model, and running the set again invalidates the external validation.

TRIPOD-AI and PROBAST-AI directly assess the presence and quality of external validation; therefore, locking an independent external data source at the design stage is a requirement that cannot be remedied afterward.

Common Mistakes

• Train-test leakage: Having different images of the same patient in both the training and test sets, or performing preprocessing/feature selection on all the data, artificially inflates performance and collapses in external validation.
• Selecting the threshold by looking at the data: Optimizing the decision threshold for sensitivity/specificity on the validation set drowns the reported performance in optimism. The threshold must be fixed in advance.
• Presenting internal validation as external: A randomly split test set from the same center is not external validation; showing this as external validity in the file comes back during notified body review.
• Writing the analysis plan afterward: Determining the primary endpoint, subgroups, or exclusion criteria after seeing the performance (HARKing) makes the study unverifiable. The SAP must be locked and dated.

Related Articles

• How to Prepare a SaMD Clinical Validation Report for TITCK CDSS
• AI Diagnostic Model Reporting with TRIPOD-AI and PROBAST-AI
• Data Leakage and Train-Test Split in AI Validation

A well-designed SaMD clinical validation study starts with a locked model, an independent reference standard, a pre-written analysis plan, and a real external validation set. Once this skeleton is built, both a peer-reviewed publication and a TITCK-CDSS, EU-MDR, or FDA file feed from the same evidence. Scope, timeline, and budget differ for each file; we clarify these in a free scoping call. Setting the design up correctly from the start is always cheaper than collecting data again afterward.

Free Scoping Call