A note on confidence intervals and statistical significance

If two 95% confidence intervals overlap this does not imply that the two statistics on which they are based (e.g. means, odds ratios) differ at the 5 percent level. In other words it is possible for the difference between two statistics to be statistically non-zero and for their respective confidence intervals still to overlap. This is usually the case when the difference between the means has moderate significance.

It is true, however, that if a pair of confidence intervals do not overlap the difference between the two statistics is statistically non-zero.

The rationale behind the above discrepancy is explained in this article taken from the Cornell University website. See also here.

As a special case if two independent group means have the same standard error (se) then the standard error of the difference in the two means equals sqrt(se² + se² ) = sqrt(2 se² ) = sqrt(2)se.

Now if both groups have large sample sizes then the group difference is approximately statistically significant if the abs[difference in group means] / (sqrt(2) se) is greater than 2 ie if the abs(difference in group means) > 2 x sqrt(2) se = approximately 2.8 se. Now, 2.8 se is greater than 2 se which suggests that in this special case of equal mean standard errors two intervals about two statistically non-significant means could not overlap if the interval widths about the mean are equal to one standard error of each mean.

Reference

Wolfe R and Hanley J (2002) If we're so different, why do we keep overlapping? When 1 plus 1 doesn't make 2. CMAJ 166 65-66