Your meta-analysis returned I²=78%. The reviewer wrote "High heterogeneity undermines the validity of the pooled estimate." What will you do?
Heterogeneity is one of the most misunderstood concepts in meta-analysis. Many researchers are content to calculate and report the I² value. Yet managing heterogeneity is the point at which the most critical methodological decisions of a meta-analysis are made.
Why Is Heterogeneity Not Merely a "Problem"?
High heterogeneity is not a sign of a poor study. On the contrary, it may be a reflection of clinical reality.
Different patient populations, different intervention doses, different follow-up durations, different outcome measures, these are all differences that exist in the real world. Ignoring these differences and reducing them to a single number (the pooled estimate) may be clinically meaningless and potentially misleading.
The real question is this: Where does this heterogeneity come from and what does it mean from the clinician's perspective?
Why Is I² Not Enough?
The I² statistic shows how much of the variation between studies stems from factors other than chance. The commonly used thresholds (25%, 50%, 75%) are reference points, not fixed boundaries.
One problem with I² is that it depends on sample size. In a meta-analysis that includes very large studies, even clinically negligible small differences can produce a high I². In a meta-analysis that includes small studies, on the other hand, true heterogeneity can be hidden.
For this reason, tau² (the heterogeneity variance) and the prediction interval should be reported alongside I². The prediction interval is especially important: even if the average effect is significant, if the prediction interval is wide and includes zero, it means that the effect could turn out in the opposite direction in a subsequent study.
Subgroup Analysis: When Is It Meaningful, When Is It Misleading?
In the face of high heterogeneity, researchers' first reflex is to perform subgroup analysis. This may be a correct approach, but if it is not done correctly it leads to serious methodological problems.
First, subgroup analyses not specified in the registered protocol (post-hoc subgroup analysis) create serious credibility problems. "I looked at the data and grouped it" is not a transparent methodology, and reviewers evaluate it that way.
The multiple comparison problem in subgroup analyses also cannot be overlooked. A result that turns out significant by chance across ten different subgroup tests may not reflect a real difference.
Meta-Regression: Powerful but Risky
Meta-regression serves to examine the effect of study-level variables that could explain heterogeneity. The relationship between the effect size and variables such as mean age, treatment dose, and follow-up duration can be tested.
However, meta-regression faces a serious power problem in small meta-analyses. As a rule, at least 10 studies are recommended per covariate. Performing a meta-regression with two covariates in a five-study meta-analysis cannot produce statistically meaningful results.
Where Do People Get Stuck in This Process?
Researchers struggle most at these points: deciding which model (fixed vs random effects) is appropriate when I² is high. Calculating the prediction interval and interpreting it correctly. Methodologically defending whether subgroup analyses were pre-specified or post-hoc. Writing an evidence-based response to the reviewer's heterogeneity criticisms.
To get methodological support for your meta-analysis with high heterogeneity, request a 30-minute free consultation.
Where Do People Get Stuck Most in This Analysis?
- Your I² came out at 85%, the reviewer says "you cannot pool with this heterogeneity," but you used a random-effects model.
- You want to perform subgroup analysis but each subgroup has 3-4 studies, and the statistical power is insufficient.
- You are performing a sensitivity analysis but the result changes each time you remove a study, and it is unclear which result to report.