The Philosophy of Multiple Comparisons

Excerpts

Statisticians classically asked the wrong question — and were willing to answer with a lie, one that was often a downright lie. They asked “Are the effects of A and B different?” and they were willing to answer “no.”

All we know about the world teaches us that the effects of A and B are always difference — in some decimal place — for any A and B. Thus asking “Are the effects different?” is foolish.

What we should be answering first is “Can we tell the direction in which the effects of A differ from the effects of B?” In other words, can we be confident about the direction from A to B? Is it “up,” “down” or “uncertain”?

The third answer to this first question is that we are “uncertain about the direction” — it is not, and never should be, that we “accept the null hypothesis.”

Reference

John W Tukey “The Philosophy of Multiple Comparisons” (1991) DOI: 10.1214/ss/1177011945

@Article{tukey1991,
  title = {The Philosophy of Multiple Comparisons},
  volume = {6},
  issn = {0883-4237},
  url = {http://dx.doi.org/10.1214/ss/1177011945},
  doi = {10.1214/ss/1177011945},
  number = {1},
  journal = {Statistical Science},
  publisher = {Institute of Mathematical Statistics},
  author = {Tukey, John W},
  year = {1991},
  month = {feb}
}