Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
30,876 characters · 29 sections · 29 citation commands
1.2STATISTICALLY 1.2SIGNIFICANT 1.2LINEAR 1.2REGRESSION 1.2COEFFICIENTS 1.2SOLELY 1.2DRIVEN 1.2BY 1.2OUTLIERS 1.2IN 1.2FINITE1.2-SAMPLE 1.2INFERENCE
\noindentKeywords: outliers, finite-sample inference, regression diagnostics, robust regression, t-statistics, p-values, p-hacking, linear models, single outlier test, CLT, normality assumption
\noindentJEL Codes: C10, C12, C13, C80, C81
License: This work is licensed under a \href{https://creativecommons.org/licenses/by-sa/4.0/}{Creative Commons Attribution-ShareAlike 4.0 International License (CC BY-SA 4.0)}. You are free to share and adapt the material, provided appropriate credit is given and any derivatives are distributed under the same license.
Linear regression is a foundational method widely used for modeling due to its simplicity, interpretability, and strong theoretical underpinnings. The classical simple linear regression (SLR) model in scalar notation is given by:
where \( \beta_0 \) (the intercept) and \( \beta_1 \) (the slope) are the coefficients capturing the linear relationship between the predictor \( X_i \) and the dependent/outcome/response variable \( Y_i \), for \( i = 1, \dots, n \). Estimation via OLS yields interpretable, closed-form solutions and enables simple statistical inference using t-tests and confidence intervals, assuming that the classical linear model conditions hold. As shown in wooldridge2019introductory, under assumptions SLR.1 through SLR.4 for simple linear regression, the OLS estimates remain unbiased; SLR.5, which assumes normally distributed errors, is only required for valid finite-sample inference using the t-distribution, but can in general be relaxed in large samples due to the Central Limit Theorem. (CLT)
One of the major advantages of linear regression is the interpretability of the estimated coefficient \( \hat{\beta}_1 \), representing the expected change in the response \( Y \) for a one-unit (assuming no re-scaling e.g. standard-units) increase in the predictor \( X \). Additionally, hypothesis testing on coefficients is conducted using Student’s t-statistic:
which provides measures of statistical significance with the associated p-values, assuming the normality of the error term or a (sufficiently) large sample size.
Despite its upsides, a critical limitation of OLS is its strong sensitivity to outliers. The use of the squared loss function \( \sum_{i=1}^n (Y_i - \hat{Y}_i)^2 \) for model fittings causes observations with large residuals to have a disproportionately high influence on the estimated coefficients. Although the sensitivity can in general be informative, it can also lead to highly misleading subsequent inferences and effect calculations. Even a single outlier can distort the estimated coefficient \( \hat{\beta}_1 \), resulting in underestimated standard errors and thus false statistical significance of the predictor variable belsley1980regression.
This paper investigates how outliers can lead to misleading conclusions in regression analysis. Through simulation using R, we demonstrate the fragility of OLS-based inference when a finite sample inference is manipulated by insertion of a single outlier. To mitigate this issue, we additionally fit a robust Huber regression huber1964robust.
In doing so, we aim to demonstrate the limitations of classical OLS regression based on the quadratic loss function for statistical inference and advocate the use of additional robust alternatives and model diagnostic tools.
These include regression diagnostics (for example, residual analysis, leverage, studentized residuals, Cook’s distance cook1977influential), model fit statistics (for example, \( R^2 \), F tests), and formal outlier detection methods such as single outlier statistical tests barnett1978outliers, rstudentbonferroni, Ugah2021 using residuals, to ensure reliable and transparent communication of statistically significant results.
We begin by recalling the standard form of the linear regression model in matrix notation:
where \( Y \in \mathbb{R}^n \) is the dependent/outcome/response variable, \( X \in \mathbb{R}^{n \times p} \) is the covariate matrix of predictors (assumed to have full column rank), \( \beta \in \mathbb{R}^p \) is the vector of coefficients to be estimated, and \( \varepsilon \in \mathbb{R}^n \) is its error term, assumed to be independent and identically distributed with zero mean and constant variance \( \sigma^2 \) wooldridge2019introductory. These assumptions are known as the classical linear model assumptions denoted as MLR.1–MLR.5 in wooldridge2019introductory in the context of multiple regression for example.
The coefficients \( \beta \) are estimated using OLS, which minimizes the sum of squared residuals:
The solution to this optimization problem is given by:
This formula is valid under assumption MLR.3 (no perfect multicollinearity), which ensures that \( X^\top X \) is a invertible matrix product wooldridge2019introductory.
Under optional assumption MLR.5, which states that the error term \( \varepsilon \) is normally distributed, the OLS estimator \( \hat{\beta} \) also follows a multivariate normal distribution:
This result forms the basis for inference procedures such as hypothesis testing and confidence interval construction wooldridge2019introductory.
Because \( \sigma^2 \) is unknown in practice, it is estimated using the residuals from the fitted model. Under assumption MLR.4 (homoskedasticity), an unbiased estimator of the variance is given by:
where \( \hat{\varepsilon} = Y - X\hat{\beta} \) is the vector of residuals wooldridge2019introductory.
The standard error of the estimated coefficient \( \hat{\beta}_j \) is computed as:
To test the null hypothesis \( H_0: \beta_j = 0 \), we compute the corresponding \( t \)-statistic as:
Under the assumptions of the classical linear model—and particularly MLR.5—this statistic follows a Student’s \( t \)-distribution with \( n - p \) or written as \( n - p -1\) (depending on intercept inclusion) degrees of freedom:
The two-sided p-value can be calculated as:
which tests whether the coefficient \( \beta_j \) is statistically different from zero in the population regression model wooldridge2019introductory.
OLS is sensitive to outliers. One unusual point can strongly affect the estimated slope, especially if the point has high leverage—that means, if it lies far from the center of the predictor distribution. This can make a non-significant result appear statistically significant belsley1980regression,cook1977influential.
We can show this with a simple example. We generate 100 observations $(x,y)$ in R using the seed: {123} with no true relationship. Model 1, based on this clean data, shows no significant coefficient. In Model 2, we add one extreme outlier. This one point changes the slope to 1.62 and creates a highly significant result. Model 3 uses robust regression (Huber’s M-estimator), which reduces the outlier’s impact and gives a smaller slope again (see Table (ref)).
We can also illustrate the effect mathematically. When a single value in the data changes, the approximate change (using the result of a first-order Taylor approximation) in the OLS estimate is approximately given by:
where \( x_i \) is the row vector for the \( i \)-th observation. This shows that the change in the estimated slope depends on both the residual size (\( \Delta y_i \)) and the leverage of the (outlier) point. Leverage is defined as:
where \( h_i \) measures how far \( x_i \) is from the center of the predictor space. High-leverage points can have a disproportionate effect on the fit belsley1980regression.
Robust methods help reduce this sensitivity. Huber’s M-estimator huber1964robust uses a different loss function that grows quadratically near zero but linearly in the tails of the distribution, limiting the influence of large residuals. Another robust approach is Least Trimmed Squares (LTS), which fits the model using only the subset of observations with the smallest residuals rousseeuw1987robust.
Robust regression methods provide more stable and reliable estimates when datasets contain outliers.
Outlier detection in linear regression models typically revolves around residuals, i.e., differences between observed and fitted values. Several test statistics target unusually large residuals to identify potential outliers.
The internally studentized residual for observation \( i \) is:
Here, \( h_{ii} \) is the leverage (diagonal of the so-called hat matrix \( H := X(X^\top X)^{-1}X^\top \)). The denominator rescales residuals the by the local variance.
Under the assumption of normally distributed errors (MLR.5), \( R_i \) approximately follows a standard normal distribution for large \( n \), but does not exactly follow a \( t \)-distribution. The externally studentized residual, which removes the \( i \)th observation when estimating variance, follows a \( t_{n - p - 1} \) distribution.
The test statistic for detecting a single outlier is the maximum studentized residual:
Under the null hypothesis \( H_0 \) (no outliers), an approximate critical value can be obtained using the Bonferroni correction. To control the family-wise error rate at level \( \alpha \), we compare \( R_n \) to the quantile of the Student's t-distribution with \( n - p - 1 \) degrees of freedom:
An alternative approach avoids using the hat matrix \( H \) and instead relies on the unadjusted, or raw, residuals. The corresponding test statistic is the normalized maximum ordinary residual:
This statistic highlights large absolute residuals relative to the overall residual magnitude. Since it does not account for leverage, it can be more sensitive to large deviations in response values. However, this also means it may overlook influential outliers associated with high-leverage points.
The exact distributions of \( R_n \) (maximum studentized residual) and \( R_n^* \) (normalized maximum ordinary residual) are not analytically tractable. Therefore, critical values are typically estimated using one or more of the following methods:
Ugah et al. Ugah2021 derive an upper bound as identical for both as:
Recall again MLR.5 for exact finite-sample inference. Under this assumption, the least squares estimator
is unbiased, efficient, and follows:
This enables valid inference using standard test statistics.
The t-statistic for testing \( H_0: \beta_j = 0 \) is:
where
The residuals \( \hat{\varepsilon}_i \), however, are correlated due to their dependence on the fitted values via the hat matrix \( H = X (X^\top X)^{-1} X^\top \), where \( \hat{\mathbf{y}} = H \mathbf{y} \) and \( \hat{\boldsymbol{\varepsilon}} = (I - H) \mathbf{y} \).
While OLS remains unbiased and consistent under weaker assumptions (e.g., finite variance and exogeneity), the exact distribution of test statistics like \( t_j \) requires normality.
Outliers violate this assumption in two key ways:
Under outlier contamination, the OLS estimator becomes:
where \( \Delta \) increases with leverage \( h_{oo} \) and residual magnitude \( |e_o| \). Thus, \( \hat{\boldsymbol{\beta}} \) can become biased in finite samples and may become inconsistent, particularly if the outlier introduces dependence between the covariates and the errors.
By the Gauss–Markov theorem, OLS is known as the Best Linear Unbiased Estimator (BLUE) under classical conditions, regardless of normality. In large samples, the Central Limit Theorem (CLT) implies that \( \hat{\boldsymbol{\beta}} \) is approximately normal, allowing for asymptotic inference even when the error distribution is not normal.
Linear regression relies on the Gauss–Markov assumptions, which ensure that the OLS estimator is the BLUE. These assumptions include linearity of the model, meaning the outcome variable \( \mathbf{y} \) is expressed as a linear combination. The covariate matrix \( X \) must have full column rank so that \( X^\top X \) is invertible, ensuring that the parameter estimates are uniquely defined. Exogeneity is also required, meaning the regressors are uncorrelated with the error term: \( \mathbb{E}[\boldsymbol{\varepsilon} \mid X] = 0 \), which implies \( \operatorname{Cov}(X, \boldsymbol{\varepsilon}) = 0 \). The error terms must be homoscedastic, following a constant variance \( \operatorname{Var}(\boldsymbol{\varepsilon}) = \sigma^2 I \), and they must be uncorrelated across observations, i.e., \( \operatorname{Cov}(\varepsilon_i, \varepsilon_j) = 0 \) for all \( i \neq j \). While these assumptions are sufficient for OLS to be unbiased and efficient, an additional assumption of normally distributed errors, \( \boldsymbol{\varepsilon} \sim \mathcal{N}(0, \sigma^2 I) \), is required for exact finite-sample inference using t-tests and F-tests.
In practice, these conditions are often violated. To address such limitations, various extensions and robust methods have been developed. Below, we outline some of the most widely used regression methods and highlight their relation to the underlying Gauss–Markov assumption.
Deming regression accounts for measurement error in both \(x\) and \(y\), minimizing orthogonal distances:
Useful when both variables are noisy cornbleet1989deming.\\ Assumptions addressed: Violates fixed \(x\); assumes homoscedastic, independent errors.
Ridge regression applies an \(L_2\) penalty to control variance from multicollinearity:
hoerl1970ridge.\\ Assumptions addressed: Mitigates multicollinearity.
Lasso uses an \(L_1\) penalty to induce sparsity:
Some coefficients may be exactly zero tibshirani1996regression.\\ Assumptions addressed: Mitigates multicollinearity.
Elastic Net combines \(L_1\) and \(L_2\) penalties:
Balances sparsity and stability zou2005regularization.\\ Assumptions addressed: Mitigates multicollinearity.
Robust regression reduces sensitivity to outliers by using a different loss function \(\rho(\cdot)\) less sensitive than squared error:
Huber’s loss is a common default choice for example huber1964robust.\\ Assumptions addressed: Handles non-normality and heteroscedasticity.
Quantile regression estimates conditional quantiles by minimizing asymmetric loss:
koenker1978regression.\\ Assumptions addressed: Handles heteroscedasticity and non-normality.
PCR applies PCA to \(X\), then regresses \(y\) on the top components. It reduces multicollinearity and variance jolliffe2002principal.\\ Assumptions addressed: Mitigates multicollinearity.
PLS projects \(X\) onto components maximizing covariance with \(y\), often outperforming PCR when predictors are correlated with the outcome wold1984pls.\\ Assumptions addressed: Mitigates multicollinearity.
LOESS fits local linear models weighted by distance:
Flexible and nonparametric cleveland1979robust.\\ Assumptions addressed: Relaxes linearity; locally handles heteroscedasticity.
Spline regression uses piecewise polynomials with a smoothness penalty:
The smoothing parameter \(\lambda\) controls complexity de1978practical.\\ Assumptions addressed: Relaxes linearity.
GLMs generalize linear models via a link function:
Includes logistic and Poisson models mccullagh1989generalized.\\ Assumptions addressed: Handles non-normality and heteroscedasticity.
GAMs allow additive nonlinear effects:
Each \(f_j\) is estimated nonparametrically (e.g., splines) hastie1990generalized.\\ Assumptions addressed: Relaxes linearity; handles non-normality and mild heteroscedasticity.
Outliers can inflate the significance of regression coefficients, leading to incorrect inferences. The normality assumption of the residuals is critical for valid t-tests, and its violation---commonly due to outliers---necessitates careful diagnostic checks or the use of robust methods. It is advised to always conduct residual diagnostics and robust regression techniques to assess and mitigate these issues, even post outlier-removal, especially in finite-sample inference.
Also, the use of outlier diagnostics utilizing both residuals and leverage is recommendable, such as standardized residuals, leverage plots, Cook's distance, DFBETAS, and influence plots. Normality of residuals can be assessed using tools like the Q–Q plot, Shapiro–Wilk test, or histogram of residuals.