On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals

Excerpts

The overlap method is simple, and it is convenient when lists or graphs of confidence intervals are presented. It can be useful as a quick and relatively rough method for exploratory data analysis. It should not be regarded as an optimal method for significance testing, however, given its conservatism and low power relative to the standard method in the common situation that we have considered. Thus, the overlap method should not be used for formal significance testing unless the data analyst is aware of its deficiencies and unless the information needed to carry out a more appropriate procedure is unavailable.

Abstract

To judge whether the difference between two point estimates is statistically significant, data analysts often examine the overlap between the two associated conŽ dence intervals. We compare this technique to the standard method of testing signiŽ cance under the common assumptions of consistency, asymptotic normality, and asymptoticindependenceof the estimates. Rejection of the null hypothesis by the method of examining overlap implies rejection by the standard method, whereas failure to reject by the method of examining overlap does not imply failure to reject by the standard method. As a consequence, the method of examining overlap is more conservative (i.e., rejects the null hypothesis less often) than the standard method when the null hypothesis is true, and it mistakenly fails to reject the null hypothesis more frequently than does the standard method when the null hypothesis is false. Although the method of examining overlap is simple and especially convenient when lists or graphs of confidence intervals have been presented, we conclude that it should not be used for formal significance testing unless the data analyst is aware of its deficiencies and unless the information needed to carry out a more appropriate procedure is unavailable.

Reference

Nathaniel Schenker, Jane F Gentleman “On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals” (2001) DOI: 10.1198/000313001317097960

@Article{schenker2001,
  title = {On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals},
  abstract = {To judge whether the difference between two point estimates is statistically significant, data analysts often examine the overlap between the two associated conŽ dence intervals. We compare this technique to the standard method of testing signiŽ cance under the common assumptions of consistency, asymptotic normality, and asymptoticindependenceof the estimates. Rejection of the null hypothesis by the method of examining overlap implies rejection by the standard method, whereas failure to reject by the method of examining overlap does not imply failure to reject by the standard method. As a consequence, the method of examining overlap is more conservative (i.e., rejects the null hypothesis less often) than the standard method when the null hypothesis is true, and it mistakenly fails to reject the null hypothesis more frequently than does the standard method when the null hypothesis is false. Although the method of examining overlap is simple and especially convenient when lists or graphs of confidence intervals have been presented, we conclude that it should not be used for formal significance testing unless the data analyst is aware of its deficiencies and unless the information needed to carry out a more appropriate procedure is unavailable.},
  volume = {55},
  issn = {1537-2731},
  url = {http://dx.doi.org/10.1198/000313001317097960},
  doi = {10.1198/000313001317097960},
  number = {3},
  journal = {The American Statistician},
  publisher = {Informa UK Limited},
  author = {Schenker, Nathaniel and Gentleman, Jane F},
  year = {2001},
  month = {aug},
  pages = {182–186}
}