"I am going to do a systematic review, search the literature and combine it": this description is only the visible face of a systematic review. The invisible face is a meticulous protocol preparation, methodological consistency, and a systematic process that takes months. An error at any stage of this process can result in publication rejection months later.
Why Can the Protocol Not Be Changed Afterward?
A systematic review protocol requires that all methodological decisions be determined before looking at the data. This principle is the foundation of preventing publication bias.
Deciding after the search that "if we include these studies too, the result would come out stronger" means changing the methodology by looking at the results. This is the systematic review equivalent of p-hacking and is regarded as a violation of scientific integrity.
For this reason, PROSPERO registration is prospective; it should be done before the literature search begins. A systematic review registered after the search is not accepted by many journals.
The Critical Decisions Most Often Skipped in the Protocol
The points researchers most often overlook while preparing the protocol are as follows:
Insufficient operationalization of PICO: "Knee OA patients" is not specific enough. Which diagnostic criteria? Is there a KL grade limit? Bilateral or unilateral? Age range? These ambiguities lead to inconsistency in inclusion decisions made later on.
When outcome measures are not specified in advance: If primary and secondary outcomes are not defined in the protocol, one can decide by looking at the results, "let's take this as primary, this one came out more meaningful." This is called outcome switching and is a serious methodological violation.
Lack of justification for the language restriction decision: "Only English studies were included" is common but not always a defensible decision. If the justification for this restriction is not explained in the protocol, the reviewer will question it.
Ambiguity of the gray literature policy: Will unpublished studies, conference proceedings, and theses be included? This decision directly affects publication bias.
The Two-Researcher Requirement: Why Is It So Important?
Having each step done by two independent researchers is the quality assurance of a systematic review. Title-abstract screening, full-text screening, data extraction, quality assessment.
In practice, this requirement creates serious organizational difficulty. Who will the second researcher be? How will the timeline be coordinated? How will disagreements be resolved? The answers to these should be included in the protocol.
Screening done without calculating and reporting Cohen's kappa does not meet the systematic review standard.
Managing Deviations from the Protocol
No systematic review can be completed without deviating from the protocol. Unexpected search results, difficulties in definition, unforeseen gaps in the literature: these are inevitable.
What matters is managing deviations transparently. Every protocol deviation should be documented: What changed, why it changed, and how this change might affect the results?
PROSPERO registration can be updated afterward, but which changes were accepted is recorded by the platform. This transparency ensures the legitimacy of methodological adjustments made later.
The Meta-Analysis Decision: When Should It Be Made?
The answer to the question "will we do a meta-analysis?" should be given in the protocol. However, this decision has a critical nuance.
Whether a meta-analysis will be possible cannot be known without knowing the results of the literature search. Heterogeneous studies, different outcome measures, or an insufficient number of studies can make a meta-analysis impossible.
For this reason, two scenarios should be defined in the protocol: the conditions under which a meta-analysis can be applied, and how a narrative synthesis will be conducted when it cannot.
Where Do People Get Stuck in This Process?
Researchers struggle most at the following points: Operationalizing PICO in all its dimensions. Optimizing the search strategy for all databases. Coordinating the second researcher and calculating kappa. Transparently documenting protocol deviations. Defining the meta-analysis decision as conditional.
Request a 30-minute free consultation to prepare your systematic review protocol together.
Where Do People Get Stuck Most in This Analysis?
- You wrote a search strategy in the protocol but the actual search gave very different results, and you do not know how to report the deviation from the protocol.
- You registered with PROSPERO but then changed the PICO question, and it is unclear whether an amendment is needed.
- Two researchers will do the screening but agreement was very low in the pilot screening, and you need to clarify the inclusion criteria.