Optimal Quantile Absolute Deviation


We consider the quantile absolute deviation around the median defined as follows:

$$ \newcommand{\E}{\mathbb{E}} \newcommand{\PR}{\mathbb{P}} \newcommand{\Q}{\operatorname{Q}} \newcommand{\OQAD}{\operatorname{OQAD}} \newcommand{\QAD}{\operatorname{QAD}} \newcommand{\median}{\operatorname{median}} \newcommand{\Exp}{\operatorname{Exp}} \newcommand{\SD}{\operatorname{SD}} \newcommand{\V}{\mathbb{V}} \QAD(X, p) = K_p \Q(|X - \median(X)|, p), $$

where $\Q$ is a quantile estimator, and $K_p$ is a scale constant which we use to make $\QAD(X, p)$ an asymptotically consistent estimator for the standard deviation under the normal distribution.

In this post, we get the exact values of the $K_p$ values, derive the corresponding equation for the asymptotic Gaussian efficiency of $\QAD(X, p)$, and find the point in which $\QAD(X, p)$ achieves the highest Gaussian efficiency.

Read more


Quantile Absolute Deviation of the Pareto Distribution


In this post, we derive the exact equation for the quantile absolute deviation around the median of the Pareto(1,1) distribution.

Read more


Quantile Absolute Deviation of the Exponential Distribution


In this post, we derive the exact equation for the quantile absolute deviation around the median of the Exponential distribution.

Read more


Quantile Absolute Deviation of the Uniform Distribution


In this post, we derive the exact equation for the quantile absolute deviation around the median of the Uniform distribution.

Read more


Quantile Absolute Deviation of the Normal Distribution


In this post, we derive the exact equation for the quantile absolute deviation around the median of the Normal distribution.

Read more


Standard Quantile Absolute Deviation


The median absolute deviation (MAD) is a popular robust replacement of the standard deviation (StdDev). It’s truly robust: its breakdown point is $50\%$. However, it’s not so efficient when we use it as a consistent estimator for the standard deviation under normality: the asymptotic relative efficiency against StdDev (we call it the Gaussian efficiency) is only about $\approx 37\%$.

In practice, such robustness is not always essential, while we typically want to have the highest possible efficiency. I already described the concept of the quantile absolute deviation which aims to provide a customizable trade-off between robustness and efficiency. In this post, I would like to suggest a new default option for this measure of dispersion called the standard quantile absolute deviation. Its Gaussian efficiency is $\approx 54\%$ while the breakdown point is $\approx 32\%$

Read more


Asymptotic Gaussian Efficiency of the Quantile Absolute Deviation


I have already discussed the concept of the quantile absolute deviation in several previous posts. In this post, we derive the equation for the relative statistical efficiency of the quantile absolute deviation against the standard deviation under the normal distribution (so call Gaussian efficiency).

Read more


Finite-Sample Efficiency of the Rousseeuw-Croux Estimators


The Rousseeuw-Croux $S_n$ and $Q_n$ estimators are robust and efficient measures of scale. Their breakdown points are equal to $0.5$ which is also the breakdown point of the median absolute deviation (MAD). However, their statistical efficiency values are much better than the efficiency of MAD. To be specific, the MAD asymptotic relative Gaussian efficiency against the standard deviation is about $37\%$, whereas the corresponding values for $S_n$ and $Q_n$ are $58\%$ and $82\%$ respectively. Although these numbers are quite impressive, they are only asymptotic values. In practice, we work with finite samples. And the finite-sample efficiency could be much lower than the asymptotic one. In this post, we perform a simulation study in order to obtain the actual finite-sample efficiency values for these two estimators.

Read more


Caveats of Using the Median Absolute Deviation


The median absolute deviation is a measure of dispersion which can be used as a robust alternative to the standard deviation. It works great for slight deviations from normality (e.g., for contaminated normal distributions or slightly skewed unimodal distributions). Unfortunately, if we apply it to distributions with huge deviations from normality, we may experience a lot of troubles. In this post, I discuss some of the most important caveats which we should keep in mind if we use the median absolute deviation.

Read more


Preprint Announcement: 'Finite-Sample Bias-Correction Factors for the Median Absolute Deviation Based on the Harrell-Davis Quantile Estimator and Its Trimmed Modification'


I have just published a preprint of a paper ‘Finite-sample bias-correction factors for the median absolute deviation based on the Harrell-Davis quantile estimator and its trimmed modification’. It’s based on a series of my research notes that I have been writing since February 2021.

The paper preprint is available on arXiv: arXiv:2207.12005 [stat.ME]. The paper source code is available on GitHub: AndreyAkinshin/paper-mad-factors. You can cite it as follows:

  • Andrey Akinshin (2022) “Finite-sample bias-correction factors for the median absolute deviation based on the Harrell-Davis quantile estimator and its trimmed modification,” arXiv:2207.12005

Abstract:

The median absolute deviation is a widely used robust measure of statistical dispersion. Using a scale constant, we can use it as an asymptotically consistent estimator for the standard deviation under normality. For finite samples, the scale constant should be corrected in order to obtain an unbiased estimator. The bias-correction factor depends on the sample size and the median estimator. When we use the traditional sample median, the factor values are well known, but this approach does not provide optimal statistical efficiency. In this paper, we present the bias-correction factors for the median absolute deviation based on the Harrell-Davis quantile estimator and its trimmed modification which allow us to achieve better statistical efficiency of the standard deviation estimations. The obtained estimators are especially useful for samples with a small number of elements.

Read more