What Estimators Are Unbiased For Linear Models?

What Estimators Are Unbiased For Linear Models?

Introduction

The celebrated Gauss-Markov theorem plays a central role in econometrics theory for fixed-design linear models. It states that the ordinary least squares (OLS) estimator is the best linear unbiased estimator (BLUE) when the variance-covariance has a scalar form. The result was later generalized by aitkin1935least, who proved that the generalized least squares (GLS) estimator is BLUE under a linear model with the covariance matrix known up to a multiplicative constant. Notably, both efficiency results are proved within the class of linear estimators, raising the possibility that there might exist unbiased estimators nonlinear in the response variable that outperform the OLS/GLS estimators.

To the best of our knowledge, the problem of finding unbiased, nonlinear estimators with smaller variances was first studied by Theodore W. Anderson in the early 1960s anderson1962least, though the work seems to have received little attention. Anderson found that the OLS estimator does not have the minimal variance among all estimators that are unbiased under homoskedastic linear models with independent errors and an essentially unrestricted design matrix. In particular, he pointed out that linear-plus-quadratic (LPQ) estimators can outperform the OLS estimator for certain error distributions.

Very recently, hansen2022modern proved that, among all estimators that are unbiased under any linear model with a finite covariance matrix, the OLS estimator achieves the minimal variance under a scalar covariance matrix. In fact, Hansen proved the more general result that the GLS estimator achieves the minimal variance when the covariance matrix is proportional to the weight matrix (Theorem 4 therein). In this sense, the OLS/GLS estimator is the best unbiased estimator (BUE), a stronger property than BLUE. Hansen's proof relies on a clever combination of the distribution-tilting technique, which is often applied in semiparametric statistics to study asymptotic efficiency bickel1993efficient, and the Cram\'{e}r-Rao lower bound, which is used to study finite-sample efficiency (among unbiased estimators). These two results are not contradictory because anderson1962least examines a larger class of candidate estimators which are only required to be unbiased under a scalar covariance matrix while hansen2022modern restricts the candidates to be unbiased without constraints on the covariance matrix.

The broader model class considered by hansen2022modern only restricts the first moment of the dependent variable, casting doubt on existence of nonlinear unbiased estimators under such weak restrictions. In fact, since the acceptance of hansen2022modern, there has been a lively online discussion on this question led by the current authors\footnote{See e.g., \url{https://twitter.com/jmwooldridge/status/1492990971218440197?s=21} and

\url{https://twitter.com/lihua_lei_stat/status/1493291015129550849?s=21}} and investigated by other researchers potscher2022modern, portnoy2022linear. They unanimously conclude, with different proof strategies, that only linear estimators are unbiased under every linear model with finite covariance matrix. This, unfortunately, implies that BUE and BLUE are equivalent and hence Hansen's new results are restatements of the classical Gauss-Markov theorem.

In an attempt to address the issue, hansen2022modern added new results in the latest version (Theorems 5 and 6). The new results imply that, among all estimators that are unbiased under linear models with independent (rather than uncorrelated) errors, the OLS estimator is BUE under homoskedasticity and independence. This class of unbiased estimators surely includes nonlinear ones. Apparently, this is not the only way to enrich the set of unbiased estimators. In the 1980s, koopmann1982parameterschatzung studied the class of linear models with a covariance matrix known up to a multiplicative constant and proved that all unbiased estimators under this class are LPQ estimators with certain restrictions on the coefficients. Building upon this remarkable representation theorem, gnot1992nonlinear proved that the OLS/GLS estimator is no longer BUE among these LPQ estimators under the same class of linear models. Again, this result does not contradict hansen2022modern because they consider non-nested classes of unbiased estimators and different model classes for which the BUE is proved.

The differences among the above results are subtle --- any statement that the OLS/GLS estimator is or is not BUE hinges on (a) the set of competing estimators and (b) the class of linear models under examination. To address the nuances, we introduce a framework that allows restatement of all existing results anderson1962least, gnot1992nonlinear, hansen2022modern without ambiguity. This can help clear up the confusion that appears to be prevalent in the community and pave the way for further technical discussion.

After that, unlike the other commentaries, we will mainly focus on the characterization of unbiased estimators under different model classes, instead of conditions under which the OLS/GLS estimator is (not) BUE. Our main contributions are new representation theorems for unbiased estimators under various classes of linear models. Our results (Theorem (ref) and Theorem (ref)) substantially generalize potscher2022modern, portnoy2022linear, and koopmann1982parameterschatzung by allowing the coefficients and covariance matrix to take only a finite number of values, the higher moments of the estimator and the dependent variable to exist, and the error distribution to be discrete, absolutely continuous, or dominated by another probability measure. They are based on (a slightly stronger version of) a generic result in ruschendorf1987unbiased, which unifies and generalizes a line of work on unbiasedness and completeness halmos1946theory, fraser1954completeness, hoeffding1977more, fisher1982unbiased, koopmann1982parameterschatzung. Roughly speaking, it states that the class of unbiased estimators of zero under a class of distributions defined by a set of moment conditions are linear combinations of these moments. We believe the result can be applied to study other model classes and provide a user-friendly proof in Appendix (ref) for readers who are less familiar with functional analysis in abstract Banach spaces. Finally, we apply the representation theorem to detect a new case where the OLS/GLS estimator is BUE. It is analogous to Theorem 5 of hansen2022modern but closer to the spirit of classical Gauss-Markov theorem that only constrains the moments.

Notation and literature review without ambiguity

Families of linear models

We start by formally defining several families of linear models that have been investigated in the literature and will be studied in this paper. We require an intricate notation in order to address the nuances that have caused considerable confusion.

Let $X\in \mathbb{R}^{n\times k}$ denote the design matrix and $Y\in \mathbb{R}^{n}$ denote the vector of the dependent variable. Throughout the paper we treat $X$ as fixed and assume the measurable space of $Y$ is equipped with the standard Borel $\sigma$-algebra on $R^{n}$. For any $r\ge 1$ and coefficient vector $\beta\in \mathbb{R}^{k}$, we denote by $\mathbf{F}_r(X; \beta)$ the class of distributions of $Y\in \mathbb{R}^{n}$ with mean $X\beta$ and finite $r$-th moment,

where $\|\cdot\|$ denotes the Euclidean norm in $\mathbb{R}^{n}$. The second condition $\mathbb{E}_{F}[\|Y\|^r] < \infty$ becomes redundant when $r = 1$. The standard (fixed-design) linear model can be characterized as

often with $r = 2$ and sometimes with $r = 1$ jensen1979linear. It is clear that $\mathbf{F}_s(X)\subset \mathbf{F}_r(X)$ if $s \ge r$.

In the econometrics literature, the covariance matrix is often assumed to be known or estimable. We further define the stratum $\mathbf{F}_r(X; \beta, \Lambda)$ of $\mathbf{F}_r(X; \beta)$ by fixing the covariance matrix $\Lambda$, i.e.,

Note that $\mathbf{F}_r(X; \beta, \Lambda) = \mathbf{F}_2(X; \beta, \Lambda)$ for any $r \in [1, 2]$ and, for any $r\ge 2$, \[\bigcup_{\Lambda\succeq \textbf{0}_{k\times k}}\mathbf{F}_r(X; \beta, \Lambda) = \mathbf{F}_r(X; \beta),\] where $\textbf{0}_{k\times k}$ denotes the $k\times k$ zero matrix and $A \succeq B$ (resp. $\succ$) iff $A - B$ is positive semidefinite (resp. positive definite). Using this notation, the linear model considered in Aitken's theorem can be expressed as $\mathbf{F}_2^\Sigma(X)$ where

In particular, the sets $\mathbf{F}_2^0$ and $\mathbf{F}_2$ defined in hansen2022modern are equivalent to $\mathbf{F}_2^{I_{n}}(X)$, where $I_{n}$ denotes the $n\times n$ identity matrix, and $\bigcup_{\Sigma\succeq \textbf{0}_{k\times k}}\mathbf{F}_2^\Sigma(X) = \mathbf{F}_2(X)$ in our notation; see Remark 2.1 of potscher2022modern for a similar clarification.

All above classes only restrict the moments of $Y$. Sometimes one is willing to impose further independence assumptions on the error terms $Y - X\beta$. Specifically, let

Similarly, we can further restrict the errors to be identically distributed:

Analogous to (ref) and (ref), we can define

Note that $\mathbf{F}_{2, \mathrm{indep}}^{I_{n}}(X)$ (resp. $\mathbf{F}_{2, \mathrm{iid}}^{I_{n}}(X)$) gives all linear models with homoskedastic and independent errors (resp. with i.i.d. errors). All have been previously studied in the literature anderson1962least, gnot1992nonlinear, hansen2022modern and will be examined in later sections.

Sometimes we further restrict the above model classes by intersecting them with a dominating class:

or a subset of $\mathcal{P}(\mu)$ that only includes distributions with bounded Radon-Nikodym derivatives:

Note that the set of absolutely continuous distributions in $\mathbb{R}^{n}$, denoted by $\mathcal{P}_{\mathrm{cont}}$, is given by $\mathcal{P}(\mu)$ with $\mu$ chosen as the Lebesgue measure\footnote{$\mathcal{P}(\mu)$ would include mixed distributions if $\mu$ is mixed.}. Furthermore, as with potscher2022modern and portnoy2022linear, we also consider the class of discrete distributions\footnote{All results in the paper continue to hold if we redefine $\mathcal{P}_{\mathrm{disc}}$ to include discrete distributions with a countable support.}:

Unbiased estimators for fixed-design linear models

Unbiased estimators exist only if $\beta$ is identifiable. A sufficient and necessary condition, which will be assumed throughout, is \[\mathrm{rank}(X) = k \le n,\] in which case the coefficient $\beta$ can be identified as \[\beta(F) = (X' X)^{-1}X'\mathbb{E}_{F}[Y].\] where $'$ denotes the transpose. For notational convenience, we will simply write $\beta(F)$ as $\beta$, though it should be kept in mind that $\beta$ is a functional of the distribution of $Y$.

For any $r\ge 1$ and family of linear models $\mathbf{F}(X)$ with a full-rank $X$, denote by $\mathbf{U}_{r}(\mathbf{F}(X))$ as the class of estimators (i.e., measurable functions of $Y$) that are unbiased for $\beta$ with finite $r$-th moments under every distribution in $\mathbf{F}(X)$, i.e.,

where $\mathcal{L}^{r}(F)$ denotes the set of measurable functions that have finite $r$-th moments under $F$. The recent work hansen2022modern, potscher2022modern, portnoy2022linear studied the special case $r = 1$. Clearly, $\mathbf{U}_r(\cdot)$ satisfies the following monotonicity property, which will be repeatly invoked.

In the following, we will consider the class of linear estimators

and the class of linear unbiased estimators (LUE):

It is well-known that all LUE estimators are unbiased.

Most recent discussions about hansen2022modern are essentially about when $\mathbf{U}_1(\mathbf{F}_1(X))\subset \mathbf{H}_{\mathrm{lin}}^{n, k}$. The following result shows that whenever it is true, the LUE estimators represent all unbiased estimators.

The main result of potscher2022modern \footnote{The authors posted a proof on social media that coincides with the first proof by potscher2022modern two weeks before; see \url{https://twitter.com/lihua_lei_stat/status/1493291015129550849?s=21}.} can be paraphrased as follows.

Together with Proposition (ref), Theorem (ref) implies that only LUE estimators can be unbiased under linear models without further constraints on the covariance matrix, even if the model only involves discrete distributions with finite second moments, i.e., \[\mathbf{U}_1(\mathbf{F}_2(X)\cap \mathcal{P}_{\mathrm{disc}}) = \mathbf{H}_{\mathrm{LUE}}(X);\] see also the footnote 7 of potscher2022modern.

While only linear estimators can be unbiased under $\mathbf{F}_2(X)$, the conclusion fails for $\mathbf{F}_{2, \mathrm{iid}}^{I_{k}}(X)$.

hansen2022modern provides a simple example that adds a mean-zero term, $Y_i$ multiplied by the leave-$i$-th-observation-out residual, onto the OLS estimator.

Theorem (ref) also fails if further assumptions are imposed on the covariance matrix. Consider the class of LPQ estimators:

where $\mathbb{R}_{\mathrm{sym}}^{n\times n}$ is the set of all symmetric matrices in $\mathbb{R}^{n\times n}$. This class was first studied by anderson1962least who showed that, in the model class $\mathbf{F}_{2, \mathrm{iid}}^{I_{k}}$, an LPQ estimator with appropriately chosen $B_j$'s can have a smaller variance for estimating a specific linear contrast of $\beta$ than the best LUE estimator, unless $n = k$ or the errors are Gaussian brudno1957dispersion, kagan1969.

Subsequently, koopmann1982parameterschatzung proved a remarkable representation theorem that all unbiased estimators under $\mathbf{F}_2^\Sigma(X)$ are given by a subset of $\mathbf{H}_{\mathrm{LPQ}}^{n, k}$: \footnote{koopmann1982parameterschatzung does not require the symmetry of $B_j$, though he commented in a remark that $B_j$ can be assumed to be symmetric without loss of generality because $y'B_j y = y'(B_j + B_j')y / 2$. }

koopmann1982parameterschatzung only presented a proof sketch for this result in his book (Section 3 and 4, Chapter I). It is relatively easy to prove that $\mathbf{H}_{\mathrm{Km}}^{\Sigma}(X)\subset \mathbf{U}_1(\mathbf{F}_2^\Sigma(X))$ via simple linear algebra. The other direction is much more challenging and we will generalize Theorem (ref) in Section (ref) with a self-contained proof.

Revisiting B(L)UE for fixed-design linear models

Define the the GLS estimator associated with a covariance matrix $\Sigma$,

where $A^{-}$ denotes the generalized inverse of $A$. Note that the OLS estimator $\hat{\beta}_{\mathrm{OLS}}(Y) = (X'X)^{-1}X'Y$ is equivalent to $\hat{\beta}_{\mathrm{GLS}}^{I_k}(Y)$.

The confusion often arises when $\mathbf{H}(X)$ and $\mathbf{F}(X)$ are not explicitly defined. We summarize the existing results on BUE and non-BUE of the OLS and GLS estimators separately in the following two theorems. By paraphrasing them on an equal footing, we hope to reveal the nuances.

Contrasting Theorem (ref) (b) (3) with Theorem (ref) (c) or Theorem (ref) (b) (1) with Theorem (ref) (c), we can see that the choice of $\mathbf{H}(X)$ is crucial --- while the OLS/GLS can be BUE in a smaller class of estimators, they may no longer be optimal in a larger class.

Unbiased estimators for fixed-design linear models

A master theorem

For any class of fixed-design linear models $\mathbf{F}(X)\subset \mathbf{F}_r(X)$, let

Since $\hat{\beta}_{\mathrm{OLS}}$ is linear, by Rosenthal's inequality rosenthal1970subspaces, \[\mathbb{E}_{F}[\|Y\|^r] < \infty\Longrightarrow \hat{\beta}_{\mathrm{OLS}}\in \mathcal{L}^{r}(F).\] Thus, \[u\in \mathbf{U}_r(\mathbf{F}(X)) \Longleftrightarrow u - \hat{\beta}_{\mathrm{OLS}} \in \{(u_1, \ldots, u_{k})': u_j\in \mathbf{U}^0_r(F(X))\} \triangleq \mathbf{U}^0_{r}(\mathbf{F}(X))^{\otimes k},\] where $\otimes k$ denotes the $k$-th order tensor product, and thus,

As a consequence, to find all unbiased estimators of $\beta$, it suffices to characterize all unbiased estimators of $0$.

It is not hard to see that both $\mathbf{F}_{r}(X; \beta)$ and $\mathbf{F}_{r}(X; \beta, \Lambda)$ can be formulated as

where

and $x_i'$ denotes the $i$-th row of $X$.

Intuitively, if we treat $F$ as an element in the “dual space” of real-valued measurable functions, though this is incorrect in the rigorous sense, $\mathbf{F}(X)$ is the orthogonal complement of $\mathbf{G}$ in the “dual space” with respect to the bilinear form $(u, F)\mapsto \mathbb{E}_{F}[u(Y)]$. Similarly, $\mathbf{U}^0_r(\mathbf{F}(X))$ can be regarded as the orthogonal complement of $\mathbf{F}(X)$. Then we should expect, in an unrigorous sense, that $\mathbf{U}^0_{r}(\mathbf{F}(X)) = (\mathbf{G}^{\perp})^{\perp}$, which is the closure of the span of $\mathbf{G}$ in an appropriately defined normed vector space. This intuitive argument, together with (ref), suggests that all unbiased estimators under $\mathbf{F}_r(X; \beta)$ are linear and those under $\mathbf{F}_r(X; \beta, \Lambda)$ are LPQ, hence implying the results by potscher2022modern, portnoy2022linear, and koopmann1982parameterschatzung.

We formalize the above heuristics in the following theorem. Below, for a given class of distributions $\mathbf{F}$, we write $\mathcal{L}^{r}(\mathbf{F})$ for $\bigcap_{F\in \mathbf{F}}\mathcal{L}^{r}(F)$. Moreover, for any subsets of functions $\mathcal{G}$ and $\mathcal{H}$, we say $\mathcal{G} \subset \mathcal{H}$ almost surely under $F$ if for each $g\in \mathcal{G}$ there exists $h\in \mathcal{H}$ such that $g - h = 0$ almost everywhere under $F$, and say $\mathcal{G} = \mathcal{H}$ if $\mathcal{G}\subset \mathcal{H}$ and $\mathcal{H}\subset \mathcal{G}$ almost surely under $F$.

When $\mathbf{G}$ only contains a finite number of functions, as in (ref), it is well-known that $\mathrm{span}(\mathbf{G})$ is closed (see Proposition (ref)). Then Theorem (ref) implies that

On the other hand, for any $F \in \mathbf{F}_{r, \mathbf{G}}$, \[\mathbf{G}\subset \mathcal{L}^{r}(F)\Longrightarrow \mathrm{span}(\mathbf{G})\subset \mathcal{L}^{r}(F), \quad F\in \mathbf{F}_{r, \mathbf{G}}\Longrightarrow F\in \mathbf{F}_{r, \mathrm{span}(\mathbf{G})}.\] As a consequence,

where the second step follows from Proposition (ref). (ref) and (ref) imply the following simplified version of Theorem (ref).

Theorem (ref) generalizes Theorem 1A and 2A of hoeffding1977more and Theorem 3.1 of koopmann1982parameterschatzung. It will be our key tool to study unbiased estimators for fixed-design linear models.

Unbiased estimators without second moment constraints

Much of the recent discussion on hansen2022modern is about $\mathbf{F}_{2}(X)$, which solely imposes restrictions on the first moment of $Y$. In this section, we will substantially generalize potscher2022modern and portnoy2022linear, showing that $\mathbf{U}_{r}(\mathbf{F}(X)) = \mathbf{H}_{\mathrm{GLS}}(X)$ even when $r > 1$ and $\mathbf{F}(X)$ is much smaller than $\mathbf{F}_{r}(X)$. We start with a surprising result that all unbiased estimators must be linear even if the domain of $\beta$ has only two distinct values with the same direction and the error distribution is restricted into a dominating class.

With Lemma (ref), we can generalize Theorem (ref) by restricting the values of $\beta$ to a finite set that spans $\mathbb{R}^{k}$ and allowing the error distribution to be absolutely continuous with respect to the Lebesgue measure or any probability measure.

Unbiased estimators with second moment constraints

Next, we will consider the case where the covariance matrix is constrained. Perhaps surprisingly, even if $\mathrm{Cov}(Y)$ can only take two distinct values that differ by a multiplicative constant and $\beta$ can only take three distinct values pointing to the same direction, any unbiased estimator must be LPQ.

We can then generalize Koopmann's representation theorem (Theorem (ref)). Throughout the rest of the paper, we will define the vectorization of any symmetric matrix $\Omega\in \mathbb{R}_{\mathrm{sym}}^{k\times k}$ as

By definition, for any $\Omega_1, \Omega_2\in \mathbb{R}_{\mathrm{sym}}^{k\times k}$, $\mathrm{tr}(\Omega_1\Omega_2) = \mathrm{vec}^{*}(\Omega_1)'\mathrm{vec}^{*}(\Omega_2)$.

As mentioned earlier, koopmann1982parameterschatzung only presented a proof sketch for Theorem (ref), which is a special case of Theorem (ref) with $\mathbf{F}(X) = \mathbf{F}_{2}^\Sigma(X)$. Therefore, we confirm the remarkable result by koopmann1982parameterschatzung and hence address the concerns raised by potscher2022modern.

When $r\ge 1$ and no constraint is imposed on $\beta$, i.e., $\mathcal{B} = \mathbb{R}^{k}$, Theorem (ref) and Theorem (ref) show that the constraint on the covariance matrix matters because \[\mathbf{H}_{\mathrm{GLS}}(X) = \mathbf{U}_r(\mathbf{F}_{2r}(X)) = \mathbf{U}_r\left( \bigcup_{\Sigma \succeq \textbf{0}_{k\times k}}\mathbf{F}_{2r}^\Sigma(X)\right), \quad \mathbf{H}_{\mathrm{Km}}^\Sigma(X) = \mathbf{U}_{r}\left(\mathbf{F}_{2r}^\Sigma(X)\right),\] where $\mathbf{F}_{2r}^{\Sigma}(X)$ is defined in (ref). The following result shows that the quadratic estimators would be excluded even if the domain of $\beta$ and $\Sigma$ are both finite.

A new nontrivial case where OLS/GLS is BUE

With the powerful representation theorem for $\mathbf{U}_{r}(\mathbf{F}_{2r}^\Sigma(X)\cap \mathcal{P})$ discussed in Theorem (ref), we can show that the OLS/GLS estimator is BUE in a sufficiently rich class that does not rule out quadratic estimators with respect to a subset of $\mathbf{F}_{2r}^\Sigma(X)$.

When $\Sigma$ is diagonal, Theorem (ref) is analogous to Theorem 5 of hansen2022modern (see also Theorem (ref) (b) (2) for a restatement in our notation), though we impose restrictions on the third moments instead of independence. It is worth emphasizing that neither result implies the other.