RWE (Real-World Evidence) Medical-AI Protocol

Assuming "the job is done" once the pivotal validation study is finished is the most common mistake in medical-AI projects. The pivotal study demonstrates your model on a selected, curated, balanced cohort. The real world is not like that: the patient mix shifts, device brands change, clinician usage is off-standard, and data is missing. Real-world evidence (RWE) is precisely the type of evidence that shows whether your device works in this noisy environment.

RWE is no longer just "nice to have." It is a mandatory part of the total product lifecycle approach in FDA AI/ML submissions, and of the PMCF (post-market clinical follow-up) and PMS plans in the EU-MDR. However, producing "evidence" from retrospective observational data is a far more dangerous statistical exercise than a randomized study: if you set up the method wrong, you can produce any result you want. This article explains how to build an RWE protocol through the eyes of a methodologist.

The Difference Between RWD and RWE: Data Is Not Evidence

Let us first clarify the terminology, because this is where submissions most often go off track.

• RWD (Real-World Data): Raw real-world data. Electronic health records (EHR), imaging archives (PACS/DICOM), billing/claims data, patient registry systems, wearable device data. On its own, it is not a claim.
• RWE (Real-World Evidence): The clinical evidence produced by processing this data with an appropriate study design and analysis. It is an inference about the benefit or risk of a treatment/device.

The distinction is critical because the regulator audits RWD quality (relevance and reliability) separately from the methodology that produces RWE. The FDA's real-world data/evidence guidance package makes exactly this distinction: the suitability of EHR and claims data, data standards, external control arms, and study design are each addressed separately. In your submission too, these two layers must be documented separately.

Study Design: The Target Trial Emulation Framework

When working with retrospective observational data, the most disciplined approach is Hernán and Robins' "target trial emulation" framework: you first design on paper the ideal randomized trial you cannot run, then emulate it with observational data. This kills most of the immortal time bias and selection bias at the design stage.

The components that must be explicitly defined in the protocol:

• Eligibility criteria: Inclusion/exclusion, assessed at the index date (retrospective filtering "with future knowledge" is prohibited).
• Treatment/exposure strategies: Here, definitions such as "AI-assisted workflow" vs "standard workflow" or "triage based on AI output."
• Assignment procedure: When there is no actual randomization, how confounder balancing will be done.
• Start of follow-up (time zero): The exposure definition, eligibility, and start of follow-up must be aligned at the SAME point. If this alignment is not made, immortal time bias is inevitable.
• Outcome and estimand: Which effect is being measured (intention-to-treat-like or per-protocol).

Writing the estimand (the estimation target) from the very start is essential. The estimand framework of the ICH E9(R1) addendum requires answering the question, "in which population, in which comparison, which outcome, and with intercurrent events handled how, are we measuring?" A typical intercurrent event in medical AI: the clinician overriding the AI output. If you do not define this in advance, the analysis becomes arbitrary afterward.

Protocol and SAP: Two Separate, Pre-Locked Documents

The single biggest determinant of RWE's reliability is that the analysis is locked before seeing the data. Otherwise, "p-hacking" and the problem of picking the nicest result from countless analytic branches in observational data (the garden of forking paths) invalidate the submission.

• Protocol: The research question, design, population, data sources, operational definitions of exposure/outcome, sample size justification, confounder list.
• SAP (Statistical Analysis Plan): A separate, more technical document from the protocol. Primary/secondary estimand, model specification, missing data strategy, propensity score method, subgroup analyses, sensitivity analyses, multiple comparison control.

Ideal practice: register the study in a registry (for example, a preregistration platform suitable for observational studies) and fix the SAP with a timestamp before the data is locked. This multiplies the value of your submission as "evidence," because it removes the suspicion of post-hoc analysis both for TRIPOD-AI reporting and in the eyes of the regulator.

To set up confounding and bias control as a written strategy in the SAP from the very start, request a free 15-minute scoping.

Confounding and Bias: RWE's Real Battle

In observational RWE, the effect may be produced not by reality but by bias. The methodological arsenal:

• Measured confounding: Multivariable regression, propensity score (matching, IPTW, stratification). As PROBAST-AI also points out, the handling of confounders in model development directly enters the risk-of-bias assessment.
• Unmeasured confounding: The most dangerous. It is bounded with negative control outcomes, E-value calculation (which measures how strong a hidden confounder would need to be to explain the observed effect), and sensitivity analyses.
• Selection bias: Who enters the data, who drops out of follow-up. In an EHR, patients who are "recorded because they were measured" are already a different population (informative presence).
• Immortal time bias: The exposure definition overlapping with the follow-up window. Prevented with target trial emulation.
• Spectrum bias / distribution shift: Specific to medical AI. If the population the model was trained on differs from the patient mix of the RWE cohort, AUC-ROC drops in the real world. This needs to be reported with subgroup performance.
• Data leakage: When building the RWE cohort, the same patient being present in the training set, or temporal leakage, artificially inflates performance.

Within the EU AI Act framework as well, data governance (Article 10) for high-risk AI systems requires representativeness and bias control; the related obligations come into force as of August 2026. The representativeness of your RWE cohort is also evidence of this obligation.

FDA RWE Guidance and Regulatory Position

The FDA runs a guidance program for real-world data and evidence and keeps these guidances up to date. In practice, the headings that flow into your submission:

• RWD relevance and reliability: The suitability of the data source to the question, data capture accuracy, missingness rates, traceability.
• Use of EHR and claims data: Linking operational definitions to code systems (ICD, SNOMED, LOINC) and validating them.
• External control arms: The conditions for establishing a real-world comparison arm in single-arm studies.
• Total product lifecycle for AI/ML: A predetermined change control plan (PCCP) and the expectation of continuous monitoring.

A caution here: the FDA-AI guidance package is updated frequently. When writing a submission, always confirm the date and number of the version in force; your methodology should be justified with reference to the guidance, not resting on the assumption that "the guidance said so." On the EU side, the same RWE is mostly embedded within the CER/PMCF and PMS plan (in the logic of MEDDEV 2.7/1 Rev 4).

Post-Market: RWE Is Not an Event, It Is a System

In medical AI, the most natural home for RWE is post-market surveillance. Because while a model is live, three things shift:

• Data drift: The distribution of input data changes (new device, new protocol, new population).
• Concept drift: The relationship between input and outcome changes.
• Performance drift: Calibration and discrimination degrade over time.

What a post-market RWE plan should include: predefined performance thresholds, periodic recalibration triggers, calibration shift monitoring (Brier score, calibration slope), subgroup fairness metrics, and an adverse event reporting mechanism. This plan must be consistent with the PCCP; the answer to the question "what happens when the model is updated" must be written in the submission in advance.

Common Mistakes

• Writing the SAP after seeing the data: The most fatal mistake. An analysis selected after seeing the results almost always comes out "significant" in observational data, and the evidentiary value is reset to zero. The protocol and SAP must be signed BEFORE the data lock and timestamped.
• Getting past confounding by relying on regression alone: A multivariable model does not solve unmeasured confounding. Without negative controls, E-value, and sensitivity analyses, the effect cannot be claimed to be causal.
• Entering the analysis without defining the estimand: Doing statistics on "does the AI work" questions without translating them into ICH E9(R1) estimand components (population, comparison, outcome, intercurrent events, summary measure) leads to unanswerable questions later.
• Mixing train/test with the RWE cohort: If a patient or center from the model's training data leaks into the RWE evaluation, performance rises artificially; this destroys external validity.

Related Articles

• How to Design a SaMD Clinical Validation Study
• EU AI Act Article 10: Data Representativeness and Bias in Medical AI
• TRIPOD-AI and PROBAST-AI: Reporting and Risk Assessment of AI Diagnostic Models

RWE is not the art of producing evidence from retrospective data; it is disciplined engineering. Set up correctly, it becomes the strongest evidence layer of your submission; set up wrong, it becomes the first part the regulator invalidates. In practice, who sets this work up methodologically (who defines the estimand, who signs the confounding strategy) directly determines acceptability. Scope, duration, and budget differ for every submission; we clarify these in a free scoping call.

Free Scoping Call