In our of the previous post, I explored the distribution of observed p-values for the Mann–Whitney U test in the finite case when the null hypothesis is true. It is time to repeat the experiment for the Brunner–Munzel test.
We generate $100\,000$ pairs of samples of size $n$ from the standard normal distribution, calculate the p-value using the two-sided Brunner–Munzel test, and build the density plot for the observed p-values. We use a test implementation that is extended for the corner case with values $0$ and $1$. Here is the result for $n=3$:
Similarly to other rank-based tests, we get a discrete distribution with a limited set of different p-values. The probabilities of each p-value are the following:
$$ \mathbb{P}(p = 0.0000) = 0.1, $$ $$ \mathbb{P}(p \approx 0.0686) = 0.1, $$ $$ \mathbb{P}(p \approx 0.3465) = 0.2, $$ $$ \mathbb{P}(p \approx 0.5734) = 0.1, $$ $$ \mathbb{P}(p \approx 0.6667) = 0.2, $$ $$ \mathbb{P}(p \approx 0.8683) = 0.1, $$ $$ \mathbb{P}(p \approx 0.8727) = 0.2. $$As we can see, the observed distribution is not as nice as in the the Mann–Whitney U test case. In particular, the expectations of influence of the specified statistical significance level to the actual false positive rate ($\mathbb{P}(p \leq \alpha) = \alpha$) are distorted:
$$ \mathbb{P}(p \leq \alpha)= 0.1 \quad\textrm{for}\quad \alpha \in [0.0000;0.0686), $$ $$ \mathbb{P}(p \leq \alpha) = 0.2 \quad\textrm{for}\quad \alpha \in [0.0686;0.3465), $$ $$ \mathbb{P}(p \leq \alpha) = 0.4 \quad\textrm{for}\quad \alpha \in [0.3465;0.5734), $$ $$ \mathbb{P}(p \leq \alpha) = 0.5 \quad\textrm{for}\quad \alpha \in [0.5734;0.6667), $$ $$ \mathbb{P}(p \leq \alpha) = 0.7 \quad\textrm{for}\quad \alpha \in [0.6667;0.8683), $$ $$ \mathbb{P}(p \leq \alpha) = 0.9 \quad\textrm{for}\quad \alpha \in [0.8683;0.8727), $$ $$ \mathbb{P}(p \leq \alpha) = 1.0 \quad\textrm{for}\quad \alpha \in [0.8727;1.0000). $$Therefore, the test should be used cautiously when considering small samples. For example, if we set the statistical significance level $\alpha = 0.07$, the actual false-positive rate will be $\mathbb{P}(p \leq \alpha) = 0.2$.
Now let us look at the same distribution for $n=5$, $n=7$, and $n=15$:
Asymptotically, it becomes uniform as for other statistical tests. However, on small samples, it has a strange sawtooth-like shape that alter our expectations of the false-positive rate.