Of beauty, sex, and power: statistical challenges in estimating small effects
Excerpts
This is a Type M (magnitude) error (Gelman and Tuerlinckx, 2000): the study is constructed in such a way that any statistically-significant finding will almost certainly be a huge overestimate of the true effect. In addition there will be Type S (sign) errors, in which the estimate will be in the opposite direction as the true effect.
Reference
A Gelman, D Weakliem “Of beauty, sex, and power: statistical challenges in estimating small effects” (2009) DOI: 10.1511/2009.79.310
@Article{gelman2009,
title = {Of beauty, sex, and power: statistical challenges in estimating small effects},
volume = {97},
issn = {1545-2786},
url = {http://dx.doi.org/10.1511/2009.79.310},
doi = {10.1511/2009.79.310},
number = {4},
journal = {American Scientist},
publisher = {Sigma Xi},
author = {A Gelman and D Weakliem},
year = {2009},
pages = {310},
custom-url-pdf = {http://www.stat.columbia.edu/~gelman/research/unpublished/power.pdf}
}