Outlier Detection with Mahalanobis Distance
https://www.steffenruefer.com/2016/12/outlier-detection-with-mahalanobis-distance/
In this tutorial I will discuss how to detect outliers in a multivariate dataset without using the response variable. I will first discuss about outlier detection through threshold setting, then about using Mahalanobis Distance instead.
The Problem
Outliers are data points that do not match the general character of the dataset. Often they are extreme values that fall outside of the “normal” range; one way of dealing with such values is to take out the highest and the lowest values of a variable. This can work quite well, but does not take into account variable combinations.
What is the problem with having outliers in the data? Sometimes there is no problem with it at all, in fact, outliers can be beneficial to understanding special characteristics of the data. In other cases, outliers might be simply mistakes in the data (i.e. noise); if you do not identify them, your predictive model will be less accurate in making predictions.
A Simple Example
The Height-Weight Dataset
To illustrate I will use a sample dataset containing height and weight data of male adults. Below is the scatter plot showing height vs weight. The data points appear normal distributed and there are some extreme values visible that are not part of the “cloud” of points around the center.
Remove Outliers with Feature Thresholds
As mentioned already, one way to deal with outliers is to set minimum and maximum thresholds to mark outliers. In this case, after visual inspection, I set the following limits (for example purpose only – no science involved!):
- height outliers: above 187 cm or below 160 cm
- weight outliers: above 72 kg or below 41 kg
Notice that many outliers were detected by using thresholds – however, most of them are closer to the regression line than the outliers that were missed.
Why should the “missed outliers” be outliers in the first place? From a model perspective, they are far from the regression line, which means such outliers will cause larger errors.
But it is more interesting to interpret these data points directly. Being a tall person does not make someone an exception – but being tall and having a very low weight does! One of the marked data points represents a person that is approx 1.85 m (6 ft 8 inches) tall, with a body weight of only 45 kg (99 pounds). Clearly this person is seriously under weight, and yet it slipped through the detection threshold.
Use Mahalanobis Distance
The Mahalanobis distance is a measure of the distance between a point P and a distribution D, as explained here. I will not go into details as there are many related articles that explain more about it. I will only implement it and show how it detects outliers. The complete source code in R can be found on my GitHub page.
# Calculate Mahalanobis Distance with height and weight distributions
m_dist <- mahalanobis(df[, 1:2], colMeans(df[, 1:2]), cov(df[, 1:2]))
df$m_dist <- round(m_dist, 2)
# Mahalanobis Outliers - Threshold set to 12
df$outlier_maha <- "No"
df$outlier_maha[df$m_dist > 12] <- "Yes"
# Scatterplot with Maha Outliers
ggplot(df, aes(x = weight, y = height, color = outlier_maha)) +
geom_point(size = 5, alpha = 0.6) +
labs(title = "Weight vs Height",
subtitle = "Outlier Detection in weight vs height data - Using Mahalanobis Distances",
caption = "Source: http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data_Dinov_020108_HeightsWeights") +
ylab("Height in cm") + xlab("Weight in kg") +
scale_y_continuous(breaks = seq(160, 200, 5)) +
scale_x_continuous(breaks = seq(35, 80, 5))
The scatterplot shows that all previously missed outliers were detected this time, and plenty of single feature “extreme” values were not declared as outliers.
Some data points that were previously also found as being outliers were still detected though – the ones on the very far end of the scale. While these might not have a large effect on the regression model, they are still outliers. Very tall or very short people, even without being over or under weight, are still rare and therefore fall into this category.
In this dataset I was using the response variable to detect outliers – this is usually not the case. In the next example I will use a dataset with more variables and try to detect outliers by using only the available predictor variables.
A Multivariate Example
The Housing Dataset
I will use a simplified version of the housing dataset, provided by Kaggle. The original data contains 80 predictor variables, but for the purpose of this tutorial I will reduce it to 4 predictor variables and the response variable:
- Response Variable: SalePrice
- Predictor Variables: GrLivArea, GarageYrBuilt, LotArea, LotFrontage
# Housing Dataset
df <- read.csv("train.csv")
# Select only 5 features - SalePrice is the response variable
df <- df %>%
select(SalePrice, GrLivArea, GarageYrBlt, LotArea, LotFrontage)
head(df)
# SalePrice GrLivArea GarageYrBlt LotArea LotFrontage
# 1 208500 1710 2003 8450 65
# 2 181500 1262 1976 9600 80
# 3 223500 1786 2001 11250 68
# 4 140000 1717 1998 9550 60
# 5 250000 2198 2000 14260 84
# 6 143000 1362 1993 14115 85
When plotting the data and adding the linear model regression line, it shows how strongly a few outliers distort the model.
We will try to remove these outliers by using Mahalanobis without including the response variable.
Calculating Mahalanobis Distance
By setting a threshold to the Mahalanobis Distance values calculated below, I am creating a binary outlier variable.
# Calculate Mahalanobis with predictor variables df2 <- df[, -1] # Remove SalePrice Variable m_dist <- mahalanobis(df2, colMeans(df2), cov(df2)) df$MD <- round(m_dist, 1) # Binary Outlier Variable df$outlier <- "No" df$outlier[df$MD > 20] <- "Yes" # Threshold set to 20
After marking outliers, we can see that the detection rate is quite good. Most major outliers were detected.
Rebuilding the Model
When removing the detected outliers and drawing a new regression line, the result is much better then before, though far from perfect.
Some more experimenting with the detection threshold might help. Additional things you can try to improve results are:
- Experiment with different predictor variables
- Try single feature thresholds – in that case that might have worked quite well!
Conclusion
Dealing with outliers is a critical part of model development. Both discussed methods – single feature thresholds and Mahalanobis Distance – provide good tools to detect point outliers. There are many others – so read up on it to make your predictive models more accurate.
Further References and Links
Wikipedia Article about Outliers
Wikipedia Article about Mahalanobis Distance
Eureka Statistics Article
Source Code and Data on my GitHub Page
Housing Data from Kaggle
