AI in Systematic Reviews and RoB 2

Systematic review + AI: a dangerous pairing, when used correctly

A systematic review typically requires 1,500-3,000 hours of human labor. Title-abstract screening alone takes 80-150 hours with two independent reviewers. Data extraction takes 100-200 hours. Risk of bias assessment takes 30-60 hours. Total: roughly one academic-year.

AI can cut this time by 40-60%. But if used at the wrong point, the review gets rejected, the PROSPERO registration is deleted, and a retraction is issued by the journal. In 2024, Lancet Digital Health and BMJ Evidence-Based Medicine published successive editorials clarifying the minimum requirements for AI-assisted systematic reviews.

In this article, I lay out an AI usage map compliant with the Cochrane Handbook, PRISMA 2020, and TRIPOD-AI guidance, for both academic teams and CROs / clinical research companies. Which tool at which stage, how much you can trust it, and where the line for the second independent reviewer begins.

Stage 1: Question Formulation and Search Strategy

PICO (Population, Intervention, Comparator, Outcome) formulation is open to human + AI collaboration.

The correct usage pattern:

I am going to formulate the PICO for a systematic review. Clinical question:
"The difference in 5-year functional score (KSS) between robotic-assisted vs
conventional approaches after total knee arthroplasty"

Please fill in the following structure:
- Population: inclusion/exclusion age, primary/revision, OA/RA, BMI limit
- Intervention: which robotic platforms (Mako, ROSA, NAVIO, Cori)
- Comparator: conventional + computer-assisted (CAS) distinction
- Outcome: primary (KSS, FJS) + secondary (revision rate, ROM, complication)
- Study design: include RCT, prospective cohort, retrospective cohort?
- Time frame: minimum follow-up 24 months?

Then suggest the MeSH and Emtree terms.

The output is your draft. You approve it, you revise it. An expert librarian (medical research librarian) validates the final search. An AI suggestion cannot be the search strategy on its own, because Chapter 4 of the Cochrane Handbook explicitly states that every systematic review requires a librarian-validated search strategy.

Do not:

• Say "write the PubMed search string" and copy it directly for use. AI sometimes produces incorrect syntax, sometimes the wrong MeSH.
• Tell ChatGPT to "summarize the literature" and never run a search at all. This is not replicable and is not acceptable.

Stage 2: Title-Abstract Screening (The Biggest Efficiency Gain)

This is AI's most powerful and safest use case. Two main tools:

ASReview (open source, free)

It works on an active learning principle. First you manually mark 10-20 articles as include/exclude, the model learns, and it ranks the remaining thousands of articles by priority. Those with high probability of inclusion move to the top, those with low probability move to the bottom.

Practical workflow:

1. Export 3,000-8,000 abstracts from the PubMed/Embase search as Endnote/RIS.
2. Upload to ASReview and choose a BERT or Naive Bayes model.
3. First 20 manual screening + then sequential review with continuous learning.
4. Early stopping criterion: If there are no includes in the last 200 abstracts, with 95% probability the rest are also excludes.

Gain: In a 5,000-abstract screen, typically a 50-70% reduction. Instead of 5,000, reading 1,500-2,500 manually is enough.

Critical rule: Cochrane requires two independent reviewers. ASReview is not counted as a reviewer on its own. Two humans screen in parallel with ASReview support; then the results of the two are reconciled.

Rayyan AI

Web-based, optimized for team collaboration. Automatic duplicate detection, blind dual screening, AI-suggested include/exclude.

Advantage: The most practical for parallel screening by multiple team members. Disadvantage: The free tier is limited; for institutional use, 50-300 USD per year.

PRISMA 2020 reporting for this stage

PRISMA 2020 items 8 and 9: "Used any automation tool? Specify the tool, who validated, and the level of human oversight." AI use must be explicitly reported. In general, the following sentence in the Methods section is sufficient:

"Title-abstract screening was performed by two independent reviewers using Rayyan AI active learning support. Each reviewer manually validated AI suggestions; final inclusion was determined by consensus, with a third reviewer resolving discrepancies."

Stage 3: Full-Text Triage

The 200-400 articles that pass the title-abstract stage go to full-text reading. Here AI comes in for a second time, but in a different role: rapid triage summarization.

The correct usage:

I have read the full text of this article (pasting it). According to my PICO
criteria below, should it be included? Please answer in the following format:

1. Population match (yes/no + why)
2. Intervention match (yes/no + why)
3. Comparator match (yes/no + why)
4. Outcome match (yes/no + why)
5. Study design match (yes/no + why)
6. Follow-up duration (X months)
7. RECOMMENDATION: Include / Exclude / Uncertain
8. If Exclude, due to which criterion (for the PRISMA flow)

PICO:
[paste your own PICO here]

Gain: You can assess a full-text article in 3-5 minutes instead of 15-20 minutes. Before accepting the AI output, you must have seen the article yourself as well (checking a missed table, supplementary material).

Critical: AI saying "Uncertain" is common (if you deliberately included it). In that case, full reading + second reviewer deliberation is a must.

Stage 4: Data Extraction

The riskiest but most time-saving stage. Typical data items to extract from an RCT:

• Study characteristics (year, country, multicenter or not)
• Sample (n, mean age, sex ratio, BMI, baseline severity)
• Intervention (device/drug name, dose, duration, training)
• Comparator (control group details)
• Outcomes (mean, SD, n for each outcome at each time point)
• Risk factors and subgroup analyses

Automated tools

RobotReviewer: Automatic PICO + RoB extraction from RCTs. You upload a PDF, it returns JSON. Works only with RCTs, not used for observational studies.

Distiller AI (managed service): Comprehensive data extraction templates, multi-reviewer reconcile UI. Annual license 1,500-4,000 USD.

Your own LLM-based pipeline: Reading long-context PDFs with Claude Sonnet 4 or GPT-4.1 and extracting structured JSON. A practical prompt:

I have read this RCT article (pasting it). Extract data according to the JSON
schema below. For fields not found, write "NR" (not reported).

{
  "study_id": "first_author_year",
  "design": "parallel_RCT | crossover_RCT | cluster_RCT",
  "country": "...",
  "n_randomized": {"intervention": 0, "control": 0},
  "n_analyzed": {"intervention": 0, "control": 0},
  "age_mean_sd": {"intervention": [0, 0], "control": [0, 0]},
  "intervention_description": "...",
  "control_description": "...",
  "follow_up_months": 0,
  "primary_outcomes": [
    {
      "name": "KSS",
      "timepoint_months": 24,
      "intervention": {"mean": 0, "sd": 0, "n": 0},
      "control": {"mean": 0, "sd": 0, "n": 0}
    }
  ],
  "funding": "...",
  "conflicts_of_interest": "..."
}

Return only JSON, no other explanation.

Critical rule: AI data extraction does not enter meta-analysis on its own. A second independent reviewer (human) compares all figures against the PDF. Typical error rate: AI makes 3-8% figure errors (especially SE ↔ SD confusion, subgroup ↔ total confusion).

Performance practice: 100% manual verification for every 10th article. Continuous quality control.

Stage 5: Risk of Bias (RoB 2 and ROBINS-I)

RoB 2 (for RCTs) assesses across 5 domains:

1. Randomization process
2. Deviations from intended interventions
3. Missing outcome data
4. Measurement of outcome
5. Selection of reported result

For each domain a "Low / Some concerns / High" decision is made, structured through signaling questions.

AI's role

RobotReviewer: Automatic RoB 2 recommendation + rationale paragraph.

LLM-based approach:

I have read this RCT article (pasting it). Using the RoB 2 tool, do a 5-domain
assessment. For each domain:
1. Which answer to which signaling question
2. Domain judgment (Low/Some/High)
3. Rationale (1-2 sentences, with a quote from the article)

At the end, give the overall RoB 2 judgment (Low/Some/High).

Performance: Between LLM RoB 2 and an expert reviewer, kappa = 0.55-0.65 (moderate-good agreement). This kappa does not mean "AI is enough on its own"; it means "AI is a good starting point."

Practical workflow:

1. AI does a preliminary assessment, decision + rationale for each domain.
2. The human reviewer can override the AI output + add to or change the rationale.
3. The second independent reviewer runs the same process in parallel.
4. The two human reviewers reconcile, with a third for disagreements.

Stage 6: GRADE Certainty of Evidence

GRADE evaluation assesses 5 main factors (risk of bias, inconsistency, indirectness, imprecision, publication bias) and produces an outcome-based certainty rating.

Here too AI acts as a draft assistant, but the final decision belongs to the methodologist. The GRADEpro GDT software is used as the official tool; AI only produces the suggested certainty level + a draft rationale.

Cost vs Time: A Practical Scenario

For a typical 30-article RCT meta-analysis:

These figures are calculated from real studies (Hip OA SR CRD420261324092 and other practices).

PRISMA 2020 Reporting and Ethics

The PRISMA 2020 checklist + AI extension (PRISMA-AI 2024) makes reporting of all AI use mandatory:

• Which AI tool was used (including version)
• At which stage
• How human oversight was performed
• Performance metrics (precision, recall, kappa)

ICMJE Vancouver: AI is not an author. It cannot appear in the author list. Its use is stated in the Methods + Acknowledgments sections. Calling ChatGPT or Claude a "co-author" has been grounds for retraction at all major journals since 2024.

When not to use AI

There are three situations:

1. Very small review (fewer than 200 abstracts): The AI learning curve is a waste of time. Manual is faster.
2. Sensitive pathologies / rare diseases: AI is weak in Turkish rare-disease terminology. Manual is safer.
3. Official Cochrane review: Cochrane's own approved pipeline (Cochrane RevMan, EPPI Reviewer) is mandatory. External AI tools are not accepted.

Conclusion

AI does not kill the systematic review, it scales it. Used correctly, time is cut by 40-60%, the two-independent-reviewer principle is preserved, and PRISMA + Cochrane standards are not breached.

Used incorrectly, it produces hallucinated data extraction, fake references, incomplete screening, and journal retraction. The difference lies in understanding that AI is an assistive layer, not a decision maker.

Practical advice for teams: first set up the workflow with a small pilot review (50-100 articles), then scale up. AI output is always used within the process of two independent human reviewers, as a first draft or supporting material.

For support in designing your first AI study, you can review the study design service within academic consulting.