In the previous posts, I discussed the idea of nonparametric effect size measures consistent with Cohen’s d under normality. However, Cohen’s d is not always the best effect size measure, even in the normal case.
In this post, we briefly discuss a case study in which a nonparametric version of Glass’s delta is preferable than the previously suggested Cohen’s d-consistent measure.
Nonparametric versions of Cohen’s d and Glass’s delta
In the scope of this post, we use the following nonparametric modifications of Cohen’s d and Glass’s delta:
\[d(x, y) = \frac{\operatorname{median}(y) - \operatorname{median}(x)}{\operatorname{PMAD}(x, y)}, \]
\[\Delta(x, y) = \frac{\operatorname{median}(y) - \operatorname{median}(x)}{\operatorname{MAD}(x)}, \]
where \(\operatorname{MAD}\) is the median absolute deviation:
\[\operatorname{MAD}(x) = \operatorname{median}(x - |\operatorname{median}(x)|), \]
\(\operatorname{PMAD}(x, y)\) is the pooled version of \(\operatorname{MAD}\):
\[\operatorname{PMAD}(x, y) = \sqrt{\frac{(n_x - 1) \operatorname{MAD}^2_x + (n_y - 1) \operatorname{MAD}^2_y}{n_x + n_y - 2}}, \]
\(\operatorname{median}\) is the traditional sample median,
Case study
Let us consider the following three samples (inspired by a real set of data samples):
\[\begin{split} x = \{ & 298, 297, 314, 312, 299, 301, 295, 295, 293, 293, 293, 293, 293, 292, 295,\\ & 293, 295, 293, 292, 295, 293, 293, 293, 299, 295, 304, 301, 296, 327, 294,\\ & 294, 293, 293, 293, 293, 293, 293, 292, 293, 292, 293, 294, 292, 294, 294,\\ & 294, 293, 293, 293, 293, 292, 294, 293, 296, 294, 299, 292, 293, 293, 294,\\ & 292, 293, 293, 292, 294, 292, 292, 293, 293, 292, 292, 292, 294, 293, 293\}, \end{split} \]
\[\begin{split} y_A = & \{ 2641, 30293, 27648 \},\\ y_B = & \{ 2641, 175631, 532991 \}. \end{split} \]
We may expect that the effect size between \(x\) and \(y_B\) should be larger than the effect size between \(x\) and \(y_A\). However, it is not true for the previously defined \(d(x, y)\) measure:
\[d(x, y_A) \approx 63.8, \quad d(x, y_B) \approx 6.2. \]
Surprisingly, \(d(x, y_B)\) is actually ten times smaller than \(d(x, y_A)\). Moreover, \(d(x, y_B) \approx 6.2\) does not always mean a truly significant change between medians in the nonparametric case. Such a situation arises because of the enormous median absolute deviation of \(y_B\): \(\operatorname{MAD}(y_B) = 172990\). Regardless of the small size of \(y_B\), the pooled median absolute deviation is heavily affected: \(\operatorname{PMAD}(x, y_B) \approx 28062.68\). Since it is used as the denominator in \(d(x, y)\) equation, a huge value of \(\operatorname{PMAD}\) leads to a small value of estimated effect size.
The described problem can often be observed when the second sample contains a small number of elements from a heavy-tailed distribution. In such cases, it is better to use the previously defined Glass’s delta-consistent approach. Here are the corresponding results:
\[\Delta(x, y_A) \approx 27355, \quad \Delta(x, y_B) \approx 175338. \]
As we can see, \(\Delta(x, y_A) < \Delta(x, y_B)\), which matches our expectation. Moreover, \(\Delta(x, y_B) \approx 175338\), which is much more impressive than \(d(x, y_B) \approx 6.2\): the estimated change is truly significant.