You want to test whether two different measurement devices can be used interchangeably. Or you need to demonstrate the agreement between an AI model you developed and expert measurements. In this case, what you need is not correlation analysis but Bland-Altman analysis.
Why Is Correlation Not Enough?
"There is an r=0.95 correlation between the two methods, so they can be used interchangeably." This is a common misconception.
Correlation shows whether two methods change together, not the actual difference between them. Even when there is systematic error (bias), the correlation can still be high. Bland-Altman, on the other hand, directly examines the differences between the methods.
How Is a Bland-Altman Plot Created?
Two measurements are taken for each participant. The calculations are:
- X axis: The mean of the two measurements [(Method1 + Method2) / 2]
- Y axis: The difference between the two measurements [Method1 - Method2]
The plot contains three horizontal lines:
Bias (Mean difference): An indicator of the systematic error between the methods. It is expected to be close to zero. If it differs significantly from zero, it means that one method systematically exceeds or underestimates the other.
Upper Limit of Agreement: Bias + 1.96 × SD
Lower Limit of Agreement: Bias - 1.96 × SD
95% of the measurements are expected to fall between these two limits.
How Is It Interpreted?
Step 1: Evaluate the bias
Is the mean difference clinically meaningful? Even if it is not statistically significant, a systematic error of 5 mmHg may still be clinically unacceptable.
Step 2: Evaluate the limits of agreement
Are the limits of agreement clinically acceptable? The answer to this question is given by the clinician, not the statistician. The researcher must define in advance the question "Is a difference this large important in clinical practice?"
Step 3: Is there a trend?
Do the differences increase systematically with the magnitude of the measurement? If the points on the plot show a slope instead of forming a horizontal band, this is a problem.
Step 4: Outlying values
Are there points that fall outside the limits? These may be potential outliers or may indicate measurement disagreement in certain patient groups.
Bland-Altman with SPSS
There is no automatic menu for Bland-Altman in SPSS. However, the calculations can be done manually:
-
New variable: Mean = (Method1 + Method2) / 2
-
New variable: Difference = Method1 - Method2
-
Mean of the difference variable = Bias
-
SD of the difference variable → Upper/lower limit calculation
-
Scatter plot: X=Mean, Y=Difference
-
Reference line: Bias, Bias+1.96SD, Bias-1.96SD
In R, an automatic plot can be created with the BlandAltmanLeh or blandr packages.
Bland-Altman in AI Studies
To show the agreement between a new AI model and expert measurements, Bland-Altman is the most appropriate method. In a "model vs expert" comparison, the second x axis of the plot is usually taken as the expert measurement.
Reporting it together with the ICC (Intraclass Correlation Coefficient) has become standard.
Request a free consultation for measurement agreement and Bland-Altman analysis.
Where Do People Get Stuck Most in This Analysis?
- You have calculated the limits of agreement, but you do not know how to justify whether these limits are "clinically acceptable."
- The points on the plot show a systematic trend, but it is unclear how you should report this.
- You are asked to report the ICC and Bland-Altman together, but the two give different results.