Why Even More Clinical Research Studies May Be False: Effect of Asymmetrical Handling of Clinically Unexpected Values

Sequel for ioannidis2005. At the beginning, there is a wonderful questionnaire for clinical readers with subsequent sarcastic discussion of the state of the art.

Excerpts

Believing there is a difference between groups, a well-intentioned clinician researcher addresses unexpected values. We tested how much removal, remeasurement, or reclassification of patients would be needed in most cases to turn an otherwise-neutral study positive. Remeasurement of 19 patients out of 200 per group was required to make most studies positive. Removal was more powerful: just 9 out of 200 was enough. Reclassification was most powerful, with 5 out of 200 enough. The larger the study, the smaller the proportion of patients needing to be manipulated to make the study positive: the percentages needed to be remeasured, removed, or reclassified fell from 45%, 20%, and 10% respectively for a 20 patient-per-group study, to 4%, 2%, and 1% for an 800 patient-per-group study. Dot-plots, but not bar-charts, make the perhaps-inadvertent manipulations visible.

— Page 1

In medical practice, clinically unexpected measurements might be quite properly handled by the remeasurement, removal, or reclassification of patients. If these habits are not prevented during clinical research, how much of each is needed to sway an entire study?

— Page 1

Imagine three researchers in separate laboratories around the world who believe that patients in two groups differ in their values of a variable. Each researcher proclaims high clinical and research standards. Dr A is particularly fastidious, taking care to remeasure any initial measurements where they are inconsistent with the clinical picture. Dr B is especially scrupulous about bias in research and tries to prevent even a few patients who have other intercurrent diseases from distorting results. Dr C realises that unaided clinical judgement may be poor at classifying patients and that test results may be better guidance.

— Page 2

Reference

Matthew James Shun-Shin, Darrel P. Francis, Andrew R. Dalby “Why Even More Clinical Research Studies May Be False: Effect of Asymmetrical Handling of Clinically Unexpected Values” (2013) DOI: 10.1371/journal.pone.0065323

@Article{shun-shin2013,
  title = {Why Even More Clinical Research Studies May Be False: Effect of Asymmetrical Handling of Clinically Unexpected Values},
  volume = {8},
  issn = {1932-6203},
  url = {http://dx.doi.org/10.1371/journal.pone.0065323},
  doi = {10.1371/journal.pone.0065323},
  number = {6},
  journal = {PLoS ONE},
  publisher = {Public Library of Science (PLoS)},
  author = {Shun-Shin, Matthew James and Francis, Darrel P.},
  editor = {Dalby, Andrew R.},
  year = {2013},
  month = {jun},
  pages = {e65323}
}