How to Prepare a TITCK CDSS Clinical Validation Report

The CDSS clinical validation report is the most frequently returned and most underestimated document in the TITCK dossier. Most teams support the sentence "our model is 94% accurate" with a table and think that is validation. Yet the agency asks for a chain of evidence that is independent, pre-planned, and measured in the target population. Validation is not "the model works well"; it is the statistical proof of the claim that "the model met pre-defined acceptance criteria, in the claimed intended use, in the claimed population."

In this article I explain, from a methodologist's viewpoint, what the report must contain, which standards it rests on, who prepares what, and where the dossier most often collapses.

Do not confuse clinical validation with analytical/technical validation

IMDRF's SaMD evidence framework defines three separate layers, and the TITCK dossier is read with this logic too:

• Valid clinical association: Demonstrating that the model's output is genuinely associated with the clinical condition it targets (literature + logical rationale).
• Analytical/technical validation: That the software processes the input correctly and produces the output in a technically reliable way (verification). This is engineering testing, not clinical validation.
• Clinical validation: Measuring, on independent data, that the output is clinically meaningful and accurate in the target population.

A common error: the team performs analytical validation (code is tested, reproducibility is ensured) and presents it as clinical validation. TITCK asks for these two separately. The clinical validation report is the third layer, and the statistical core is here.

The mandatory core of the report: a pre-defined validation protocol

A valid validation report is the application of a protocol written before the results. Seeing the results and then choosing metrics (HARKing) invalidates the report. The protocol must fix the following:

• Intended use: A complete, single-sentence definition. Clinical condition, target population, user (physician or patient), and the device's role in the decision process (informational / triggering / autonomous). The performance target is derived from this sentence.
• Primary and secondary endpoints: For classification, sensitivity/specificity and AUC-ROC; for segmentation, Dice/Hausdorff; for risk scores, calibration (calibration-in-the-large, slope, Brier score). AUC alone is not sufficient; calibration and clinical utility (decision curve analysis) must be added.
• Pre-calculated sample size: An n justified by expected performance, an acceptable lower bound, and power. The answer "as much data as we have" is not accepted.
• Acceptance criteria: Thresholds written before the study begins (e.g., the lower 95% confidence bound of sensitivity is above the specified clinical threshold).
• Ground-truth (reference standard) definition: How, by whom, and with how many readers it was created; whether it was characterized with inter-reader agreement (ICC, kappa).

Standalone performance and data leakage: the dossier's breaking point

The heart of CDSS validation is standalone performance: the model's output measured on a fixed test set, isolated from the clinical workflow. Two rules are non-negotiable:

• No data leakage: The train/validation/test split must be done at the patient level. Different images/slices of the same patient cannot appear in both training and test; otherwise performance typically inflates in an unrealistic way. This is a bias source that TRIPOD-AI and PROBAST-AI audit directly.
• The test set is single-use: It is run once on the final model; the model is not selected by looking at the test set.

If standalone performance is not sufficient in the real world, a multi-reader multi-case (MRMC) design is needed for performance together with the user (for example, radiologist + AI). If the decision-support claim is "it improves physician performance," this is proven not by a standalone metric but by MRMC.

To independently assess whether the sample size, endpoints, and data-leakage controls in your CDSS validation protocol withstand TITCK audit, request a free 15-minute scoping call.

Training-data representativeness and subgroup performance

The report must show that the training and test data represent the target Turkish clinical population. This is a central topic both for TITCK and for upcoming regulations:

• Data characterization: Age, sex, and disease-severity distribution; device/manufacturer/protocol diversity (for example, different CT/MRI devices, different slice thicknesses, different centers).
• Domain shift risk: A model trained at a single center with a single device may drop at another center. External validation (different center/device/population) determines the strength of the report.
• Subgroup (fairness) analysis: Reporting performance separately across sex, age, and clinical subgroups. The EU Artificial Intelligence Act Article 10 (data governance and representativeness obligations for high-risk systems, which begin to apply in August 2026) hardens this expectation; if your TITCK dossier is advancing together with EU-MDR, it is sensible to build it in this structure from now.

The link to risk management: the report cannot be disconnected from the device dossier

The clinical validation report must be linked to the ISO 14971 risk management file. Medical device risk assessment itself is not in our lane, but the validation statistics intersect with risks at the following points:

• Cost of false positives/false negatives: The clinical threshold must be chosen not only by a statistical optimum such as the Youden index, but by the clinical consequence of the error. If the cost of a missed pathology is higher than that of an unnecessary further test, the study prioritizes sensitivity.
• Residual risk: The lower bound of the performance confidence intervals must support the accepted residual-risk level.
• Traceability: The validation results must be referenced as evidence for the effectiveness of the risk control measures in the risk file; this chain must be consistent in the CER and the technical file.

Who prepares what?

The report is not the work of a single person but of defined roles:

• Manufacturer / quality team: Intended-use definition, ISO 13485 quality system, ISO 14971 risk file, product marketing-authorization paperwork. (These are regulatory/QMS work, not the biostatistics lane.)
• Responsible/coordinating investigator (physician): Ethics committee responsibility on prospective data, clinical conduct, and patient safety. The principal investigator of a clinical study is a physician.
• Methodologist / biostatistician: Design of the validation protocol, sample size, endpoints and acceptance criteria, statistical analysis plan, results analysis, and the TRIPOD-AI/PROBAST-AI-compliant statistics section of the report. The signature as named methodologist is placed here.

Serteser Consulting takes on the third role: it designs and signs the statistical core of the dossier, the part looked at first in audit but most often overlooked. Physician-led principal investigatorship, therapeutic areas that create a conflict of interest (where the consultant develops its own product), and QMS documentation are out of scope.

Common Mistakes

• Mistaking analytical validation for clinical validation: Code testing and reproducibility are not clinical evidence; performance on independent data is required.
• Writing the protocol afterwards: Looking at the results and choosing metrics and thresholds (HARKing) collapses the report in audit; the protocol must be locked before the data and, if possible, pre-registered.
• Overlooking data leakage: When an image/slice-based split is confused with a patient-based split, performance inflates misleadingly; PROBAST-AI flags this directly as high bias.
• Assuming single-center data is "sufficient": Without external validation and subgroup representativeness, TITCK and EU-MDR/AI Act expectations are not met; the sentence "94% accuracy" is not, on its own, a validation report.

Related Articles

• How to Design a SaMD Clinical Validation Study
• MRMC Multi-Reader Study: AI Diagnostic Performance
• Data Leakage and Train/Test Split in AI Validation

The success of a TITCK CDSS validation report depends not on the regulatory language but on whether the statistical core behind it withstands audit. A pre-locked protocol, a justified sample size, leakage-free standalone performance, and representative data turn your dossier from a "returned" document into a "passing" one. Scope, timeline, and budget differ for every dossier; we clarify these in a free scoping call.

Free Scoping Call