Nonparametric effect size: Cohen's d vs. Glass's delta


In the previous posts, I discussed the idea of nonparametric effect size measures consistent with Cohen’s d under normality. However, Cohen’s d is not always the best effect size measure, even in the normal case.

In this post, we briefly discuss a case study in which a nonparametric version of Glass’s delta is preferable than the previously suggested Cohen’s d-consistent measure.

Nonparametric versions of Cohen’s d and Glass’s delta

In the scope of this post, we use the following nonparametric modifications of Cohen’s d and Glass’s delta:

\[d(x, y) = \frac{\operatorname{median}(y) - \operatorname{median}(x)}{\operatorname{PMAD}(x, y)}, \]

\[\Delta(x, y) = \frac{\operatorname{median}(y) - \operatorname{median}(x)}{\operatorname{MAD}(x)}, \]

where \(\operatorname{MAD}\) is the median absolute deviation:

\[\operatorname{MAD}(x) = \operatorname{median}(x - |\operatorname{median}(x)|), \]

\(\operatorname{PMAD}(x, y)\) is the pooled version of \(\operatorname{MAD}\):

\[\operatorname{PMAD}(x, y) = \sqrt{\frac{(n_x - 1) \operatorname{MAD}^2_x + (n_y - 1) \operatorname{MAD}^2_y}{n_x + n_y - 2}}, \]

\(\operatorname{median}\) is the traditional sample median,

Case study

Let us consider the following three samples (inspired by a real set of data samples):

\[\begin{split} x = \{ & 298, 297, 314, 312, 299, 301, 295, 295, 293, 293, 293, 293, 293, 292, 295,\\ & 293, 295, 293, 292, 295, 293, 293, 293, 299, 295, 304, 301, 296, 327, 294,\\ & 294, 293, 293, 293, 293, 293, 293, 292, 293, 292, 293, 294, 292, 294, 294,\\ & 294, 293, 293, 293, 293, 292, 294, 293, 296, 294, 299, 292, 293, 293, 294,\\ & 292, 293, 293, 292, 294, 292, 292, 293, 293, 292, 292, 292, 294, 293, 293\}, \end{split} \]

\[\begin{split} y_A = & \{ 2641, 30293, 27648 \},\\ y_B = & \{ 2641, 175631, 532991 \}. \end{split} \]

We may expect that the effect size between \(x\) and \(y_B\) should be larger than the effect size between \(x\) and \(y_A\). However, it is not true for the previously defined \(d(x, y)\) measure:

\[d(x, y_A) \approx 63.8, \quad d(x, y_B) \approx 6.2. \]

Surprisingly, \(d(x, y_B)\) is actually ten times smaller than \(d(x, y_A)\). Moreover, \(d(x, y_B) \approx 6.2\) does not always mean a truly significant change between medians in the nonparametric case. Such a situation arises because of the enormous median absolute deviation of \(y_B\): \(\operatorname{MAD}(y_B) = 172990\). Regardless of the small size of \(y_B\), the pooled median absolute deviation is heavily affected: \(\operatorname{PMAD}(x, y_B) \approx 28062.68\). Since it is used as the denominator in \(d(x, y)\) equation, a huge value of \(\operatorname{PMAD}\) leads to a small value of estimated effect size.

The described problem can often be observed when the second sample contains a small number of elements from a heavy-tailed distribution. In such cases, it is better to use the previously defined Glass’s delta-consistent approach. Here are the corresponding results:

\[\Delta(x, y_A) \approx 27355, \quad \Delta(x, y_B) \approx 175338. \]

As we can see, \(\Delta(x, y_A) < \Delta(x, y_B)\), which matches our expectation. Moreover, \(\Delta(x, y_B) \approx 175338\), which is much more impressive than \(d(x, y_B) \approx 6.2\): the estimated change is truly significant.