Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
24,217 characters · 12 sections · 36 citation commands
The Frisch--Waugh--Lovell Theorem for Standard Errors
\onehalfspacing
The Frisch--Waugh--Lovell Theorem is a standard result in most econometrics textbooks, although it is less common to appear formally in statistics textbooks on linear models. The theorem states that the coefficient of $X_2$ in the full ordinary least squares (OLS) fit of $Y$ on $X = (X_1, X_2)$ equals the coefficient of $\tilde{X}_2$ in the partial OLS fit of $\tilde{Y}$ on $\tilde{X}_2$, where $\tilde{Y}$ and $\tilde{X}_2$ are the residuals from the OLS fits of $Y$ and $X_2$ on $X_1$, respectively. It is rare to see further discussions on the standard errors of the coefficients based on these two OLS fits. I shall show that the standard errors assuming homoskedasticity differ only due to a correction for degrees of freedom, and more interestingly, the Eicker--Huber--White (EHW) standard errors mackinnon1985some, angrist2008mostly allowing for heteroskedasticity are the same in full and partial OLS fits.
I shall further extend the results to the standard errors allowing for heteroskedasticity and clustering liang1986longitudinal, angrist2008mostly, and the standard errors allowing for heteroskedasticity and autocorrelation newey1987simple, andrews1991heteroskedasticity, andrews1992improved, newey1994automatic, lumley1999weighted.
Finally, I shall illustrate the results with an example from the regression analysis of stratified experiments. The example will show the discrepancy between the standard errors based on OLS and the correct standard error based on the design of experiments.
Consider the linear regression $Y = X_1 \beta_1 + X_2 \beta_2 + \varepsilon$, where $Y$ is an $n\times 1$ response vector, $X_1$ is an $n\times K$ matrix, $X_2$ is an $n\times L$ matrix, and $ \varepsilon$ is an $n\times 1$ error vector. Let $\hat{\beta}_1$ and $ \hat{\beta}_2$ be the OLS coefficients, and $\hat{ \varepsilon } = Y - X_1 \hat{\beta}_1 - X_2 \hat{\beta}_2$ be the residual vector. The second component $ \hat{\beta}_2$ has estimated covariance $\hat{V}$ equaling the $(2,2)$th block of $ \hat{\sigma}^2 (X'X)^{-1} $ assuming homoskedasticity, or $\hat{V}_{\textsc{ehw}}$ equaling the $(2,2)$th block of $ (X'X)^{-1} X' \hat{\Omega}_\textsc{ehw} X (X'X)^{-1} $ allowing for heteroskedasticity, where $\hat{\sigma}^2 = \| \hat{ \varepsilon} \|_2^2/(n-K-L)$ is the unbiased estimator of the common variance of the errors and $\hat{\Omega}_\textsc{ehw} = \text{diag}\{ \hat{ \varepsilon}^2\}$ is the diagonal matrix consisting of the squared residuals.
Consider the following partial regression. First, from the OLS fit of $Y$ on $X_1$ we obtain the residual vector $\tilde{Y}$. Then, from the column-wise OLS fit of $X_2$ on $X_1$ we obtain the residual matrix $\tilde{X}_2$. Finally, from the OLS fit of $\tilde{Y}$ on $\tilde{X}_2$ we obtain the coefficient $\tilde{\beta}_2$ and the residual vector $\tilde{\varepsilon} = \tilde{Y} -\tilde{X}_2 \tilde{\beta}_2 $. Then $\tilde{\beta}_2$ has estimated covariance $ \tilde{V} = \tilde{\sigma}^2 ( \tilde{X}_2 ' \tilde{X}_2)^{-1} $ assuming homoskedasticity, or $ \tilde{V}_{\textsc{ehw}} = ( \tilde{X}_2 ' \tilde{X}_2)^{-1} \tilde{X}_2' \tilde{\Omega}_\textsc{ehw} \tilde{X}_2 ( \tilde{X}_2 ' \tilde{X}_2)^{-1} $ allowing for heteroskedasticity, where $\tilde{\sigma}^2 = \| \tilde{ \varepsilon} \|_2^2/(n-L)$ and $\tilde{\Omega}_\textsc{ehw} = \text{diag}\{ \tilde{ \varepsilon}^2\}$.
The following theorem is well known.
Econometricians often formally call it the Frisch--Waugh--Lovell (FWL) Theorem although it is also known in statistics ever since yule1907theory. See agresti2015foundations for a textbook discussion.
Few textbooks comment on the estimated covariances based on the full and partial regressions. In fact, we have the following theorem.
amemiya1985advanced and greene2011econometric gave elegant proofs of Theorem (ref). lovell1963seasonal highlighted the importance of correcting for degrees of freedom in the partial regression, and hayashi2000econometrics hinted at the first result in Theorem (ref). I am not aware of a formal proof of the second result in Theorem (ref), and I shall prove both theorems for completeness. Two key facts ensure $\hat{V}_{\textsc{ehw}} = \tilde{V}_{\textsc{ehw}}$: first, the residual vectors from the full and partial regressions are the same; second, the $(2,1)$th block of $(X'X)^{-1} X$ equals $( \tilde{X}_2'\tilde{X}_2 )^{-1}\tilde{X}_2'$. See the Appendix for more details.
It is important to note that both Theorems (ref) and (ref) are pure linear algebra facts without any probabilistic assumptions on the data generating process.
Another known form of the FWL Theorem is that $\hat{\beta}_2 = ( \tilde{X}_2'\tilde{X}_2 )^{-1} \tilde{X}_2' Y$, which also follows from (ref) in the Proof of Theorem (ref). So $\hat{\beta}_2$ equals the coefficient of $\tilde{X}_2$ in the OLS fit of $Y$ on $\tilde{X}_2$. While this OLS fit can give the correct point estimator, the corresponding variance estimators are not the same as the correct one even after adjusting for degrees of freedom. This is because the residual vector from this regression does not equal the correct residual vector from the full regression.
The second result of Theorem (ref) holds only for the original form of the EHW (also called “HC0”), but not for other variants, for example, “HC1,” “HC2,”, “HC3,” and “HC4” mackinnon1985some, angrist2008mostly, zeileis2004econometric. HC1 has a correction for degrees of freedom, so the corresponding variance estimates are $\hat{V}_{\textsc{ehw}}\times n/(n-K-L)$ and $\tilde{V}_{\textsc{ehw}}\times n/(n-L)$, which satisfy the same relationship as the covariance estimators assuming homoskedasticity. However, other variants depend on the leverage scores of the OLS fits, and there is no simple relationship between the HC$j\ (j=2,3,4)$ obtained from the full and partial regressions. The equivalence of the EHW covariance estimators (HC0) obtain by these two regressions can also be a problem especially when the dimension of $X_1$ is high and the correction for degrees of freedom becomes important baltagi2008econometric.
From $\hat{V}_\textsc{ehw} = \tilde{V}_\textsc{ehw}$ and its proof, a general equivalence relationship holds as long as the $\hat{\Omega}$ matrix depends only on the residual vector. However, it does not hold if $\hat{\Omega}_\textsc{ehw}$ (or $\tilde{\Omega}_\textsc{ehw}$) depends on covariate matrix $X$ (or $\tilde{X}_2$) including its dimension, leverage scores, etc. This observation allows us to extend Theorem (ref) to other robust standard errors. I give two important examples.
When observations are clustered, for example, by states, villages, or classrooms, we can still use the OLS estimator, but it is more reasonable to use the cluster-robust variance estimator. Assume the observations are sorted based on the cluster indicators, and let $\hat{\varepsilon}_g$ be the residual vector for the observations in cluster $g\ (g=1,\ldots, G)$. Based on the full regression, $\hat{\beta}_2$ has cluster-robust covariance estimator $\hat{V}_\textsc{lz}$ liang1986longitudinal, angrist2008mostly, cameron2015practitioner equaling the $(2,2)$th block of $ (X'X)^{-1} X' \hat{\Omega}_\textsc{lz} X (X'X)^{-1} , $ where $ \hat{\Omega}_\textsc{lz} = \text{diag}\{ \hat{\varepsilon}_1 \hat{\varepsilon}_1', \ldots, \hat{\varepsilon}_G \hat{\varepsilon}_G' \} $ is the block diagonal matrix of the outer products of the residual vectors within clusters. The partial regression gives point estimator $\tilde{\beta}_2$, residual vectors $\tilde{\varepsilon}_g \ (g=1,\ldots, G)$, and cluster-robust covariance estimator equaling $\tilde{V}_\textsc{lz} = ( \tilde{X}_2 ' \tilde{X}_2)^{-1} \tilde{X}_2' \tilde{\Omega}_\textsc{lz} \tilde{X}_2 (\tilde{X}_2' \tilde{X}_2)^{-1} , $ where $\tilde{\Omega}_\textsc{lz} = \text{diag}\{ \tilde{\varepsilon}_1 \tilde{\varepsilon}_1', \ldots, \tilde{\varepsilon}_G \tilde{\varepsilon}_G' \} $.
We can easily extend Theorem (ref) and show that these two cluster-robust covariance estimators are identical numerically.
cameron2015practitioner stated but did not give a proof for a special case of Theorem (ref) in panel data regression, where $X_1$ is the matrix consisting of the fixed effects dummy variables. In fact, Theorem (ref) works for general full and partial regressions, and I shall give a formal proof in the Appendix.
Similar to the EHW covariance estimator, the LZ covariance estimator has several variants cameron2015practitioner. Sometimes, it is further multiplied by $G/(G-1)$. This will not change Theorem (ref). Sometimes, it is further multiplied by $G/(G-1)\times (n-1)/(n-K-L)$ in the full regression and by $G/(G-1)\times (n-1)/(n-L)$ in the partial regression. In this case, Theorem (ref) holds up to a correction for degrees of freedom. Other variants depend on the leverage scores mccaffrey2002bias, angrist2008mostly, and do not satisfy Theorem (ref) in general.
The consistency and asymptotic Normality of the OLS estimator hold even when the error terms are serially correlated hansen1982large. However, we must correctly estimate the asymptotic covariance matrix for statistical inference. Assume that the observations are ordered so that $i=1,\ldots, n$ index the time. Based on the full regression, $\hat{\beta}_2$ has heteroskedasticity and autocorrelation (HAC) robust covariance estimator $\hat{V}_\textsc{hac}$ equaling the $(2,2)$th block of $ (X'X)^{-1} X' \hat{\Omega}_\textsc{hac} X (X'X)^{-1} , $ where $\hat{\Omega}_\textsc{hac} = ( \hat{w}_{|i-j|} \hat{\varepsilon}_i \hat{\varepsilon}_j )_{1\leq i,j \leq n}$ have many different forms depending on the choice of the weight $\hat{w}_{|i-j|}$. Based on the partial regression, the HAC robust covariance estimator $\tilde{V}_\textsc{hac}$ equals $( \tilde{X}_2 ' \tilde{X}_2)^{-1} \tilde{X}_2' \tilde{\Omega}_\textsc{hac} \tilde{X}_2 (\tilde{X}_2' \tilde{X}_2)^{-1}$, where $\tilde{\Omega}_\textsc{hac} = ( \tilde{w}_{|i-j|} \tilde{\varepsilon}_i \tilde{\varepsilon}_j )_{1\leq i,j \leq n}$.
The condition in Theorem (ref) holds for newey1987simple's estimator with a fixed lag, and lumley1999weighted's weighted empirical adaptive variance estimator. It is common to adjust for the degrees of freedom, so Theorem (ref) holds up to a minor modification. We have $(n-K-L)\hat{V}_\textsc{hac} = (n-L) \tilde{V}_\textsc{hac}$ for those estimators, for example, the default choice of the newey command in stata. When the weight depends on the features of the covariate matrix andrews1991heteroskedasticity, andrews1992improved, newey1994automatic, Theorem (ref) does not hold even after correcting for the degrees of freedom. zeileis2004econometric's sandwich package in R implements many HAC robust covariance estimators.
In this section, I shall apply Theorems (ref) and (ref) to the regression analysis of stratified experiments. Consider an experiment stratified on a discrete covariate $X_i\in \{1, \ldots, K\}$, with a binary treatment $Z_i$ and an outcome $Y_i$, for unit $i \in \{ 1,\ldots, n \}$. Let $[k]$ be the set of indices, $n_{[k]}$ be the sample size, and $\pi_{[k]} = n_{[k]}/n$ be the proportion of units within stratum $k$. Within stratum $k$, let $n_{[k]1}$ and $n_{[k]0}$ be the number of units under treatment and control, respectively, and define $e_{[k]} = n_{[k]1}/n_{[k]}$ as the propensity score, i.e., the probability of receiving the treatment. Assume that $0< e_{[k]} < 1$ for all $k$. In general, different strata can have different probabilities of receiving the treatment and therefore the $e_{[k]}$'s can vary across strata.
A question of interest is the treatment effect of $Z$ on the $Y$. A standard estimator is the coefficient of $Z_i$ in the OLS fit of $Y_i$ on the treatment indicator $Z_i$ and $K$ stratum indicators:
where the intercept is omitted to avoid overparametrization duflo2007using, angrist2014opportunity, ImbensRubin15, gibbons2018broken.
Interestingly, we can use Theorem (ref) to obtain an explicit formula for this OLS estimator.
We can prove Theorem (ref) by directly solving the Normal equation of OLS in (ref). But using Theorem (ref) yields a more elegant proof. I give a sketch below and present the details in the supplementary material.
Theorem (ref) states that $\hat{\tau}_{\textsc{ols}}$ is a weighed average of the difference-in-means of the outcomes within stratum $k$, with weights proportional to the product of $\pi_{[k]} $ and the variance of the treatment $ e_{[k]} (1-e_{[k]})$. It is a numerical result of the OLS fit, which holds without any probabilistic assumptions on the data generating process. angrist1998estimating and angrist2008mostly, ImbensRubin15, aronow2016does, and gibbons2018broken gave similar forms of Theorem (ref), assuming certain superpopulation sampling and asymptotic scheme.
Within stratum $k$, we have a completely randomized experiment, and therefore we can estimate the variance of $\hat{\tau}_{[k]}$ by $\hat{V}_{[k]} = s_{[k]1}^2 / n_{[k]1} + s_{[k]0}^2 / n_{[k]0}$, where $s_{[k]1}^2 = n_{[k]1}^{-1} \sum_{i\in [k], Z_i=1} (Y_i - \bar{Y}_{[k]1})^2$ and $s_{[k]0}^2 = n_{[k]0}^{-1} \sum_{i\in [k], Z_i=0} (Y_i - \bar{Y}_{[k]0})^2$ are the sample variances of the outcomes under treatment and control, respectively. The usual definitions of the sample variances have denominators $ n_{[k]1}-1$ and $ n_{[k]0}-1$, but we adopt these definitions to simplify the expressions below. The difference is minor when the sample sizes $n_{[k]1}$ and $ n_{[k]0}$ are large. The variance estimator $\hat{V}_{[k]}$ is the standard choice assuming either a superpopulation or a finite population ImbensRubin15, ding2017bridging. Then based on Theorem (ref), a variance estimator for $\hat{\tau}_{\textsc{ols}}$ is $ \hat{V}_0 = \sum_{k=1}^K \omega_{[k]} ^2 \hat{V}_{[k]}. $ It is curious to see whether this variance estimator is similar to the ones based on the OLS fit. Unfortunately, the answer is no. We can use Theorem (ref) to derive the following results.
I give the sketch of the proof below and present the details in the supplementary material.
The variance estimator assuming homoskedasticity is quite different from $\hat{V}_0$. The label switching of $n_{[k]1}$ and $n_{[k]0}$ in $\hat{V}_{[k]} $ and $\hat{\Lambda}_{[k]}$ appeared in the regression analysis of completely randomized experiments Ding:2014. In general, inference based on $\hat{V} $ can be either conservative or anti-conservative. The variance estimator allowing for heteroskedasticity is close to $\hat{V}_0$ when there is no treatment effect heterogeneity across strata, i.e., $\hat{\tau}_{[k]} - \hat{\tau}_{\textsc{ols}}$ is close to zero. Otherwise, $\hat{V}_{\textsc{ehw}}$ is more conservative than $\hat{V}_0$ because the positive term $\Delta_{[k]} ( \hat{\tau}_{[k]} - \hat{\tau}_{\textsc{ols}} )^2 / n_{[k]} $ has the same order as $\hat{V}_{[k]}$ when $n_{[k]1}$ and $n_{[k]0}$ are large.
Theorem (ref) shows that the coefficient of the treatment in the OLS fit (ref) is a weighted average of the within-stratum difference-in-means estimators, with weights proportional to $ \pi_{[k]} e_{[k]} (1-e_{[k]})$ rather than $\pi_{[k]}.$ If the weights were $\pi_{[k]}$, then the estimator would be unbiased for the usual average treatment effect for the whole population. Theorem (ref) suggests that $\hat{\tau}_\textsc{ols}$ is not unbiased for the usual estimand of interest unless the propensity scores are constant.
Sometimes $\hat{\tau}_\textsc{ols}$ may estimate the parameter of interest, in the cases, for example, where the propensity scores $e_{[k]} $'s are constant, or we target a population weighted by $\pi_{[k]} e_{[k]} (1-e_{[k]})$. Even in these cases, from Theorem (ref), we recommend neither of the variance estimators based on OLS because both can be quite different from $\hat{V}_0$. The one assuming homoskedasticity $\hat{V}$ can be anti-conservative, and the one allowing for heteroskedasticity $\hat{V}_\textsc{ehw}$ are overly conservative in the presence of treatment effect heterogeneity. Recently, abadie2020sampling highlighted the difference between model-based and design-based uncertainties.
Overall, Theorems (ref) and (ref) are negative results on the regression estimator for stratified experiments, which extend freedman2008regression_b's critique on the regression estimator for completely randomized experiments. The first critique is on the point estimator and the second critique is on the standard errors based on OLS. miratrix2013adjusting and gibbons2018broken discussed alternative estimators in stratified experiments, suggesting including all the interaction terms of the treatment and stratum indicators in the OLS fit (ref). This is also a strategy proposed by lin2013agnostic in analyzing completely randomized experiments with covariates. With this fully interacted regression, the EHW standard error is nearly identical to the correct standard error.
Finally, because Theorems (ref) and (ref) are purely numeric results without assuming any probabilistic model, they also hold for other studies including completely randomized experiment and observational studies with a discrete covariate $X$. lin2013agnostic and miratrix2013adjusting discussed the former; angrist2008mostly discussed the latter.
The Frisch--Waugh--Lovell Theorem is a powerful tool to understand regression coefficients from full and partial regressions. This note provides some further results on the associated covariance estimators that assume homoskedasticity and allow for heteroskedasticity, clustering, and autocorrelation.
I illustrate these results with an example from the regression analysis of stratified experiments, showing that the commonly-used OLS estimator may not be consistent for the targeted parameter of interest, and moreover, even if it is consistent under strong assumptions, the corresponding variance estimators can be either conservative or anti-conservative, assuming homoskedasticity or not. These results are also useful for understanding many other regression estimators in randomized experiments. I will report additional results in a longer paper.