Why Most Discovered True Associations Are Inflated

• Disclaimer
  • The debate on whether research findings are credible is out of the scope.
  • Assumption: considered research findings are true.
• First example is based on the previous work of the author: ioannidis2007a.
  • While the original study van2002 is based on 117 patients (98 initial + 19 confirmatory), the author highlights only 19 patients from the second trial (to show the small sample size in the pioneer work).
• On page 645, left column, third line from the bottom, the number 256 should be 461. The same correction should be made in the legend for Figure 2.

Abstract

Newly discovered true (non-null) associations often have inflated effects compared with the true effect sizes. I discuss here the main reasons for this inflation. First, theoretical considerations prove that when true discovery is claimed based on crossing a threshold of statistical significance and the discovery study is underpowered, the observed effects are expected to be inflated. This has been demonstrated in various fields ranging from early stopped clinical trials to genome-wide associations. Second, flexible analyses coupled with selective reporting may inflate the published discovered effects. The vibration ratio (the ratio of the largest vs. smallest effect on the same association approached with different analytic choices) can be very large. Third, effects may be inflated at the stage of interpretation due to diverse conflicts of interest. Discovered effects are not always inflated, and under some circumstances may be deflated-for example, in the setting of late discovery of associations in sequentially accumulated overpowered evidence, in some types of misclassification from measurement error, and in conflicts causing reverse biases. Finally, I discuss potential approaches to this problem. These include being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery phase, using strict protocols for analyses, pursuing complete and transparent reporting of all results, placing emphasis on replication, and being fair with interpretation of results.

Reference

John PA Ioannidis “Why Most Discovered True Associations Are Inflated” (2008) DOI: 10.1097/ede.0b013e31818131e7

@Article{ioannidis2008,
  title = {Why Most Discovered True Associations Are Inflated},
  abstract = {Newly discovered true (non-null) associations often have inflated effects compared with the true effect sizes. I discuss here the main reasons for this inflation. First, theoretical considerations prove that when true discovery is claimed based on crossing a threshold of statistical significance and the discovery study is underpowered, the observed effects are expected to be inflated. This has been demonstrated in various fields ranging from early stopped clinical trials to genome-wide associations. Second, flexible analyses coupled with selective reporting may inflate the published discovered effects. The vibration ratio (the ratio of the largest vs. smallest effect on the same association approached with different analytic choices) can be very large. Third, effects may be inflated at the stage of interpretation due to diverse conflicts of interest. Discovered effects are not always inflated, and under some circumstances may be deflated-for example, in the setting of late discovery of associations in sequentially accumulated overpowered evidence, in some types of misclassification from measurement error, and in conflicts causing reverse biases. Finally, I discuss potential approaches to this problem. These include being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery phase, using strict protocols for analyses, pursuing complete and transparent reporting of all results, placing emphasis on replication, and being fair with interpretation of results.},
  volume = {19},
  issn = {1044-3983},
  url = {http://dx.doi.org/10.1097/EDE.0b013e31818131e7},
  doi = {10.1097/ede.0b013e31818131e7},
  number = {5},
  journal = {Epidemiology},
  publisher = {Ovid Technologies (Wolters Kluwer Health)},
  author = {Ioannidis, John PA},
  year = {2008},
  month = {sep},
  pages = {640–648}
}