Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
107,524 characters · 17 sections · 99 citation commands
Nuclear Norm Regularized Estimation of Panel Regression Models
In this paper we consider a linear panel regression model of the form
where $i=1\ldots N$ and $t=1\ldots T$ label the cross-sectional units and the time periods, respectively, $Y_{it}$ is an observed dependent variable, $X_{k,it}$ are observed regressors, $\beta_0=(\beta_{0,1},\ldots,\beta_{0,K})' $ are unknown regression coefficients, $f_{0,tr}$ and $\lambda_{0,ir}$ are unobserved factors and factor loadings, $E_{it}$ is an unobserved idiosyncratic error term, $R_0$ denotes the number of factors, and $K$ denotes the number of regressors. The factors and loadings are also called interactive fixed effects. They parsimoniously represent heterogeneity in both dimensions of the panel, and they contain the conventional additive error components as a special case. We assume that $R_0 \ll \min ( N,T )$, and for our asymptotic results we will consider $R_0$ and $K$ as fixed, as $N,T \rightarrow \infty$. We can rewrite this model in matrix notation as
where $\beta_0 \cdot X := \sum_{k=1}^K X_k \beta_{0,k}$ and $\Gamma_0 := \lambda_0 f_0'$, and $Y$, $X_k$, $\Gamma_0$ and $E$ are $N \times T$ matrices, while $\lambda_0$ and $f_0$ are $N \times R_0$ and $T \times R_0$ matrices, respectively. The parameters $\beta_0$ and $\Gamma_0$ are treated as non-random throughout the whole paper, that is, all stochastic statements are implicitly conditional on their realization. Without loss of generality we assume $R_0 = {\rm rank}(\Gamma_0)$.
One widely used estimation technique for interactive fixed effect panel regressions is the least squares (LS) method,\footnote{ The LS estimator in this context is also sometimes called concentrated least squares estimator, and was originally proposed by Kiefer1980. } which treats $\lambda$ and $f$ as parameters to estimate (fixed effects).\footnote{ Other estimation methods of panel regressions with interactive fixed effects include the quasi-difference approach (e.g., HoltzEakin-Newey-Rosen1988), generalized method of moments estimation (e.g. AhnLeeSchmidt2001,AhnLeeSchmidt2013), the common correlated random effect method (e.g., Pesaran2006), the decision theoretic approach (e.g., Chamberlain2009), and Lasso type shrinkage methods on fixed effects (e.g., ChengLiaoSchorfheide2016, LuSu2016, SuShiPhillips2016). } Let the Frobenius norm of an $N \times T$ matrix $A$ be $\|A\|_2 := \left( \sum_{i=1}^N \sum_{t=1}^T A_{it}^2 \right)^{1/2}$. Then, the LS estimator for $\beta$ reads
where $R$ is the number of factors chosen in the estimation. A matrix $\Gamma \in \mathbb{R}^{N \times T}$ can be written as $\Gamma = \lambda f'$, for some $\lambda \in \mathbb{R}^{N \times R}$ and $f \in \mathbb{R}^{T \times R}$, if and only if ${\rm rank}(\Gamma) \leq R $. The profiled least square objective function $ L_R(\beta) $ can therefore equivalently be expressed as
It is known that under appropriate regularity conditions (including exogeneity of $X_{k,it}$ with respect to $E_{it}$), for $R \geq R_0$, and as $N,T \rightarrow \infty$ at the same rate, the LS estimator $\widehat{\beta}_{{\rm LS},R}$ is $\sqrt{NT}$-consistent and asymptotically normal, with a bias in the limiting distribution that can be corrected for (e.g., Bai2009, MoonWeidner2015, MoonWeidner2017).
The LS estimation approach is convenient, because it does not restrict the relationship between the unobserved heterogeneity ($\Gamma_0$) and the observed explanatory variables ($X_1,...,X_K$). However, the calculation of $\widehat{\beta}_{{\rm LS},R} $ requires solving a non-convex optimization problem. While $\left\| Y -\beta \cdot X- \Gamma \right\|_2^2$ is a convex function of $\beta$ and $\Gamma$, the profiled objective function $L_R(\beta)$ is in general not convex in $\beta$, and can have multiple local minima, as will be discussed in Section (ref) in more detail. The reason for the non-convexity is that the constraint ${\rm rank}(\Gamma) \leq R$ is non-convex. In practice, researchers often use iterative methods that alternate between estimating $\beta$ given $\Gamma$ and updating $\Gamma$ given $\beta$. However, due to the non-convexity, such iterative procedures may converge to local minima or inconsistent estimators if initialized with arbitrary values. The nuclear norm regularization approach proposed in this paper provides a consistent initial estimator, which can then be used to initialize iterative procedures that are guaranteed to converge to the correct solution. This non-convexity can also become a serious computational obstacle, for example, when the number of regressors is too large to allow for a simple grid-search over $\beta$ (for a given $\beta$, the optimization over $\Gamma$ is a principal components problem that can generally be solved quickly in the linear regression case). For generalizations to non-linear panel regression models (e.g., chen2014estimation, ChenFernandezValWeidner2014), the optimization of the non-convex objective function with respect to the high-dimensional factors and loadings becomes even more challenging.
In this paper, we make several important contributions to the literature of panel regression with interactive fixed effects:
Our first contribution is to overcome the non-convexity issue of the least squares estimation problem above by considering a panel regression with nuclear norm regularization, which provides a convex relaxation of the rank constraint in (ref). To be more specific, let $s(\Gamma) := [ s_1(\Gamma) , \allowbreak s_2(\Gamma) , \allowbreak \ldots , s_{\min(N,T)}(\Gamma)]$ be the vector of singular values of $\Gamma$.\footnote{ The non-zero singular values of $\Gamma$ are the square roots of non-zero eigenvalues of $\Gamma \Gamma'$. Singular values are non-negative by definition. } The rank of a matrix is equal to the number of non-zero singular values, that is, ${\rm rank}(\Gamma) = \left\| s(\Gamma) \right\|_0$, where $\left\| v \right\|_0$ equals the number of non-zero elements of the vector $v$ (sometimes calles the “$\ell^0$-norm” of $v$). The nuclear norm of $\Gamma$ is defined by $\| \Gamma \|_1 := \left\| s(\Gamma) \right\|_1 = \sum_{r=1}^{\min(N,T)} s_r( \Gamma )$, that is, the nuclear norm of the matrix $\Gamma$ is simply the $\ell^1$-norm of the vector $s(\Gamma)$.\footnote{ The nuclear norm $\| \Gamma \|_1$ is the convex envelope of ${\rm rank}(\Gamma)$ over the set of matrices with spectral norm at most one, see e.g.\ RechtFazelParrilo2010. The nuclear norm is also sometimes called trace norm, Schatten 1-norm, or Ky Fan n-norm. Our index notation for the nuclear norm $ \| \Gamma \|_1$, Frobenius norm $ \| \Gamma \|_2$, and spectral norm $ \| \Gamma \|_\infty = \lim_{q \rightarrow \infty} \| \Gamma \|_q$ is motivated by the unifying formula $\| \Gamma \|_q = \left\{ \sum_{r=1}^{\min(N,T)} \left[s_r(\Gamma)\right]^q \right\}^{1/q}$. } A convex relaxation of (ref) can then be obtained by replacing the non-convex constraint ${\rm rank}(\Gamma) \leq R$ by the convex constraint $\| \Gamma \|_1 \leq c$, for some constant $c$. This gives
where in the second line we replace the constraint on the nuclear norm by a nuclear-norm penalty term.\footnote{The normalizations with $1/(2NT)$ and $1/\sqrt{NT}$ in (ref) are somewhat arbitrary, but turn out to be convenient for our purposes.} Choosing a particular penalization parameter $\psi > 0 $ is equivalent to choosing a particular value for $c= c_\psi$, and we find it more convenient to parameterize the convex relaxation $Q_\psi(\beta) $ of $L_R(\beta)$ by $\psi$ instead of $c$.
For a given $\psi>0$ the nuclear-norm regularized estimator reads
We also define $\widehat{\beta}_{*} := \lim_{\psi \rightarrow 0} \widehat{\beta}_{\psi}$ for fixed $N$ and $T$.\footnote{ Here, the limit $\psi \rightarrow 0$ is for fixed $N$ and $T$, and has nothing to do with our large $N$, $T$ asymptotic considerations. } We will show in Section (ref) that $ \widehat{\beta}_{*} = \operatorname*{argmin}_{\beta} \left\| Y - \beta \cdot X \right\|_1$, that is, $\widehat{\beta}_{*}$ can alternatively be obtained by minimizing the nuclear norm of $Y - \beta \cdot X$. The nuclear norm minimizing estimator $\widehat{\beta}_{*} $ is novel to the literature as far as we know.
The second contribution of the paper is to derive asymptotic properties of $\widehat{\beta}_{\psi} $ and $\widehat{\beta}_{*} $. We establish asymptotic results for $\widehat \beta_\psi$ and $\widehat \beta_*$ when both panel dimensions become large. Under appropriate regularity conditions we show $\sqrt{\min ( N,T )}$-consistency of these estimators. We also show how to use $\widehat \beta_\psi$ and $\widehat \beta_*$ as initial values for a finite iteration procedure (avoiding a non-convex optimization) that gives improved estimates that are asymptotically equivalent to the LS estimator.
The third contribution of the paper is to solve a potential identification problem for interactive fixed effect models by employing the nuclear norm penalization. Notice that without restrictions on the parameter matrix $\Gamma_0$ in (ref), we cannot separate $\beta_0 \cdot X$ and $\Gamma_0$ uniquely, because for any other parameter $\beta$ we can write
implying that $(\beta_0, \Gamma_0)$ and $(\beta, \Gamma(\beta,X))$ are observationally equivalent. If any non-trivial linear combination of the regressors $X_k$ is a high-rank matrix, then the assumption that $R_0 = {\rm rank}( \Gamma_0) \ll \min(N,T)$ is sufficient to identify $\beta_0$, because ${\rm rank}[\Gamma(\beta,X)]$ will be large for any other value of $\beta$. However, if some of the regressors $X_k$ have low rank (as, for example, in GobillonMagnac2016, where $X_k$ is a panel of treatment variables) and the true number of factors $R_0$ is unknown, then there is an identification problem, and some regularization device is needed to resolve this. In Section (ref) we show that, under appropriate assumptions on the covariates, the nuclear norm penalization indeed provides such a regularization device to uniquely identify $\beta_0$.
Nuclear norm penalized estimation has been widely studied in the machine learning and statistical learning literature. There, the parameter of interest is usually the matrix that we call $\Gamma$ in our model. In particular, there are many papers that use this penalization method in matrix completion (e.g., RechtFazelParrilo2010 and Hastieetal2015 for recent surveys), and for reduced rank regression estimation (e.g., RohdeTsybakov2011). More recently, nuclear norm penalization has also been used in the econometrics literature: bai2019rank use it to improve estimation in a pure factor model. Atheyetal2017 apply nuclear norm penalization to treatment effect estimation with unbalanced panel data due to missing observations together with a regularization on the high dimensional regression coefficients -- their primary interest is to predict the left-hand side variable using the regularization. chernozhukov2018inference consider panel regression models with heterogeneous coefficients, while in this paper we focus on panel regression with homogenous coefficients. To the best of our knowledge, our results here on the estimates of the common regression coefficients $\beta_0$ are new in this literature, and the nuclear norm minimizing estimator $\widehat \beta_*$ has also not been proposed previously.
Since the 2018 working paper version of this paper, many related studies have used nuclear norm regularization and similar methods in panel data, factor models, and network models. In network settings, alidaee2020recovering and ma2022detecting use penalized methods to recover structure in networks. In panel data models, chernozhukov2018inference use low-rank methods to study models with varying slopes, and belloni2019high, wang2022low, and feng_2023 extend this to quantile regressions, which relates to our work in Appendix (ref). miao2020panel and miao2023high look at threshold models and VARs with factor structures. chen2022unified gives a general framework for conditional factor models using nuclear norm regularization, and also shows how to choose tuning parameters and prove consistency. beyhum2022factor allow for weak (non-strong) factors, and fernandez2021low use nuclear norm penalties to estimate nonseparable panel models. beyhum2019square treat the low-rank structure as an approximation, allowing the rank to grow with sample size. chetverikov2022spectral and mugnier2025simple study grouped fixed effects, and use nuclear norm methods in the first step of their slope estimation. armstrong2022robust also use the nuclear norm to make slope estimates more robust in interactive fixed effects models. hong2023profile develop a profile GMM method that uses nuclear norm regularization and allows for endogeneity. vogt2022cce build a high-dimensional version of the CCE estimator, combining factor projections with Lasso. gao2024robust study robust inference when the errors are not Gaussian and may be dependent over time and across units, using thresholded long-run covariance estimation. Finally, chen2022unified (already mentioned above) develop a flexible method that applies to many types of conditional factor models. Overall, this recent work shows how nuclear norm regularization and related convex methods are useful for many problems involving panel data, factor models, and networks.
The paper is organized as follows. Section (ref) discusses theoretical motivations for employing nuclear norm regularization instead conventional rank restrictions. In Section (ref) we derive consistency results on $\widehat{\beta}_{\psi}$ and $\widehat{\beta}_{*}$. Section (ref) shows how to use these two estimates as a preliminary step to construct an estimator through iterations that achieves asymptotic equivalence to the fixed effect estimator. Section (ref) investigates finite sample properties via simulations, and Section (ref) concludes the paper. All technical derivations and proofs are presented in the appendix or supplementary appendix.
In this section we provide further motivation and explanation of the nuclear norm regularized estimation method. This estimation approach comes with the computational advantage of having a convex objective function, and it also provides a solution to the identification problem of interactive fixed effect models with low-rank regressors.
We have already introduced the profile LS objective function $L_R(\beta)$ and its convex relaxation $Q_\psi(\beta)$ in the introduction. Here, we explain those objective functions further. Firstly, we want to briefly explain why $Q_\psi(\beta)$ is indeed convex. We have introduced the nuclear norm as $\| \Gamma \|_1 := \sum_{r=1}^{\min(N,T)} s_r( \Gamma )$, but it is not obvious from this definition that $\| \Gamma \|_1$ is convex in $\Gamma$, because the singular values $s_r( \Gamma )$ themselves are generally not convex functions of $\Gamma$, except for $r=1$. A useful alternative definition of the nuclear norm is
that is, the nuclear norm is dual to the spectral norm $\| \cdot \|_\infty$. From this it is easy to see that $\| \cdot \|_1$ is indeed a matrix norm, and thus convex in $\Gamma$.\footnote{ Let $B$ and $C$ be matrices of the same size. Then, by (ref) there exists a matrix $A$ of the same size with $\|A\|_{\infty} \leq 1$ such that $\|B+C\|_1 = {\rm Tr}[A' \, (B+C)] = {\rm Tr}(A' B) + {\rm Tr}(A' C) \leq \|B\|_1 + \|C\|_1 $, which is the triangle inequality for the nuclear norm. Together with absolute homogeneity of $\left\| \cdot \right\|_1$ this implies convexity. } Therefore, the nuclear norm regularized objective function \[ \frac{1}{2NT} \left\| Y -\beta \cdot X- \Gamma \right\|_2^2 + \frac{ \psi } {\sqrt{NT}} \left\| \Gamma \right\|_1 \] as a function of $(\beta,\Gamma)$ is convex. Profiling with respect to $\Gamma$ preserves convexity, that is, $Q_\psi(\beta)$ is also convex.
By contrast, the least squares objective function $\frac{1}{2NT} \left\| Y -\beta \cdot X- \lambda f^{\prime} \right\|_2^2$ is generally non-convex in the parameters $\beta$, $\lambda$ and $f$. However, the non-convexity of the LS minimization over $\lambda$ and $f$ is actually not a serious problem in computing the profile objective function $L_R(\beta)$, as long as the regression model is linear and one of the dimensions $N$ or $T$ is not too large.\footnote{The optimal $\widehat \lambda$ and $\widehat f$ are simply given by the leading $R$ principal components of $Y - \beta \cdot X$. Calculating them requires finding the eigenvalues and eigenvectors of either the $N\times N$ matrix $(Y - \beta \cdot X) (Y - \beta \cdot X)'$ or the $T \times T$ matrix $(Y - \beta \cdot X)' (Y - \beta \cdot X)$, which takes at most a few seconds on modern computers, as long as $\min(N,T) \lessapprox 5.000$, or so. The non-zero eigenvalues of $(Y - \beta \cdot X) (Y - \beta \cdot X)'$ and $(Y - \beta \cdot X)' (Y - \beta \cdot X)$ are identical, and are equal to the square of the non-zero singular values of $Y - \beta \cdot X$.} Recall that $s_r(Y - \beta \cdot X)$ is the $r^{th}$ largest singular value of the matrix $(Y - \beta \cdot X)$, for $r=1,...,\min(N,T)$. One can show (see MoonWeidner2017) that the profile least squares objective function is
where the largest $R$ singular values are omitted in the sum - because they are absorbed by the principal component estimates $\widehat \lambda$ and $\widehat f$. The remaining problem in calculating $\widehat \beta_{{\rm LS},R}$ is the generally non-convex minimization of $L_R(\beta) $ over $\beta$.\footnote{ In our discussion here we focus on the calculation of $\widehat \beta_{{\rm LS},R}$ via minimization of the profile objective function $L_R(\beta)$. More generally, $\widehat \beta_{{\rm LS},R}$ can be obtained by any method that minimizes $ \left\| Y -\beta \cdot X- \lambda f^{\prime} \right\|_2^2$ over $\beta$, $\lambda$, $f$, see e.g.\ Bai2009 or the supplementary appendix in MoonWeidner2015. For any such method the non-convexity of the objective function is a potential problem, because the algorithm may converge to a local minimum, or potentially even to a critical point that is not a local minimum. } To illustrate the potential difficulty caused by this non-convexity, in Figure (ref) we plot $L_R(\beta)$ for the simple example described in Appendix (ref). In this example $L_R(\beta) $ is non-convex and has two local minima, one of which (the global one) is close to the true parameter $\beta_0=2$. The figure also shows that $Q_\psi(\beta)$ is convex and only has a single local minimum.
\graphicspath{ {Figures/LS_Comp_Example/} }
For any $\psi > 0$ define the functions $\ell_\psi: [0,\infty) \mapsto [0,\infty)$ and $q_\psi: [0,\infty) \mapsto [0,\infty)$ by
For an $N \times T$ matrix $A$ let $ \ell_\psi(A) := \sum_{r=1}^{\min(N,T)} \ell_\psi(s_r(A))$ and $ q_\psi(A) := \sum_{r=1}^{\min(N,T)} q_\psi(s_r(A))$. We can then rewrite (ref) as
where $\psi(\beta,R)$ satisfies
Here, the normalization with $1/\sqrt{NT}$ is natural, because under standard assumptions the largest singular value of $Y - \beta \cdot X$ is of order $\sqrt{NT}$, as $N$ and $T$ grow. The formulation (ref) is interesting for us, because the following lemma shows that we have a very similar representation for $Q_\psi(\beta) $.
\graphicspath{ {Figures/} }
The proof is given in the appendix. We note that Lemma (ref) implies convexity of $Q_\psi(\beta)$ in $\beta$, since $(Y - \beta \cdot X)/\sqrt{NT}$ is linear in $\beta$ and $A \mapsto \sum_{r} q_\psi(s_r(A))$ is convex in $A$. The latter follows from a standard result in convex analysis: if $g: \mathbb{R} \rightarrow \mathbb{R}$ is convex and nondecreasing on $[0,\infty)$, then $A \mapsto \sum_{r} g(s_r(A))$ is convex in the matrix $A$ (see, e.g., lewis1995convex). Figure (ref) shows the functions $q_\psi(s)$ and $\ell_\psi(s)$ for real valued arguments $s$ and $\psi=1$.
Comparing $L_R(\beta)$ and $Q_\psi(\beta)$ we see that the parameter $R$ that counts the number of factors is replaced by the parameter $\psi$ that characterizes the magnitude at which the singular values of $(Y -\beta \cdot X) / \sqrt{NT}$ are considered to be factors, and for a given $\beta$ the relationship between $R$ and $\psi$ is given by (ref). Large $R$ corresponds to small $\psi$, and vice versa. Furthermore, $\widehat{\Gamma}_{\psi}(\beta) := \operatorname*{argmin}_{\Gamma} \frac{1}{2NT} \left\| Y -\beta \cdot X- \Gamma \right\|_2^2 + \frac{ \psi } {\sqrt{NT}} \left\| \Gamma \right\|_1$ has singular values\footnote{See Lemma (ref) in the supplementary appendix for details.} \[ s_r\left( \widehat{\Gamma}_{\psi}(\beta) \right) = \max \left(s_r\left( (Y -\beta \cdot X) / \sqrt{NT} \right) - \psi,0 \right) \quad r = 1,...,\min(N,T), \] that is, the nuclear norm penalization shrinks the singular values of $Y -\beta \cdot X$ towards zero by a fixed amount.
Fixing $\psi$ as opposed to fixing $R$ already changes the functional form of the profile objective function, because according to (ref) their relationship depends on $\beta$. In addition, the objective function is convexified by replacing the function $ \ell_\psi(s)$ that is applied to the singular values of $(Y - \beta \cdot X) / \sqrt{NT}$ with the function $q_\psi(s)$, as defined in (ref). The function $q_\psi(s)$ provides a convex continuation of $ \ell_\psi(s)$ for $s \geq \psi$.
Using the closed-form expression for $Q_{\psi}(\beta)$ in Lemma (ref), and noticing that it is convex in $\beta$, one can compute the minimizer $\widehat{\beta}_{\psi}$ of $Q_{\psi}(\beta)$ using various optimizing algorithms for a convex function (see chapter 5 of Hastieetal2015). If the dimension of $\beta$ is small, then one may even use a simple grid search method to find $\widehat{\beta}_{\psi}$. We will discuss a data dependent choice of the penalty parameter $\psi$ in Section (ref).
When estimating the interactive fixed effect model (ref), in practice both $\beta_0$ and $R_0$ are unknown. Showing that $\beta_0$ and $R_0$ can be consistently estimated jointly is a difficult problem in general.\footnote{ The problem of joint identification of $\beta_0$ and $R_0$ is often avoided in the literature. Some papers (e.g. Bai2009, li2016panel, MoonWeidner2017) assume that the number of factors $R_0$ is known when showing consistency for an estimator of $\beta_0$. Alternatively, LuSu2016 allow for unknown $R_0$, but assume consistency of their estimator for $\beta_0$. } Within the interactive fixed effects estimation framework this joint inference problem has only been successfully addressed when both of the following assumptions are satisfied:\footnote{ Some existing estimation methods avoid specifying $R$ when estimating $\beta_0$, but always at the cost of some additional assumptions on the data generating process. For example, the common correlated effects estimator of Pesaran2006 avoids choosing $R$, but requires assumptions on how the factors $f_0$ enter into the observed regressors $X_k$, and requires all regressors of interest to be high-rank. }
Under those assumptions (and other regularity conditions) the consistency proofs of Bai2009 and MoonWeidner2015 are applicable to the LS estimator for $\beta$ that uses $R=R_{\max} \geq R_0$ factors in the estimation, and one can also show the convergence rate result $\big\| \widehat \beta_{{\rm LS},R_{\max}} - \beta_0 \big\| = O_P \left( \min(N,T)^{-1/2} \right)$, as $N,T \rightarrow \infty$. To obtain a consistent estimator for $R_0$ one can then apply inference methods from pure factor models without regressors (e.g.\ BaiNg2002, Onatski2010, AhnHorenstein2013) to the matrix $Y - \widehat \beta_{{\rm LS},R_{\max}} \cdot X$.
The condition (C2) above is particularly strong, because “low-rank regressors” are quite common in practice. If we can write $X_{k,it} = w_{k,i} v_{k,t}$, then we have ${\rm rank}(X_k) = 1$, and the condition (C2) is violated. For example, GobillonMagnac2016 estimate an interactive fixed effects model in a panel treatment effect setting, where the main regressor of interest indeed can be multiplicatively decomposed in this way, with $w_{k,i}$ being the treatment indicator of unit $i$, and $v_{k,t}$ being the time indicator of treatment. Those interactive fixed effects models for panel treatment effect applications have grown very popular recently.\footnote{ Other recent applications in the same vein as GobillonMagnac2016 are chan2016policy, powell2017synthetic, gobillon2017local, adams2017identification, piracha2017immigration, li2018inference, to list just a few. This literature is also related to the synthetic control method (abadie2003economic, abadie2010synthetic, abadie2015comparative; see also hsiao2012panel). } However, when $R_0$ is unknown, then the presence of such low-rank regressors creates an identification problem, as illustrated by the following example.
The example shows that regression coefficients of low-rank regressors are not identified if $R_0$ is unknown, because $\beta \cdot X$ could simply be absorbed into the factor structure $\lambda f'$, which is also a low-rank matrix. Therefore, without some additional assumption or regularization device, the two low-rank matrices $ \beta_0 \cdot X$ and $\Gamma_0 = \lambda_0 f_0'$ cannot be uniquely disentangled, which is what we mean by “unique matrix separation” in the title of this section.
Both the nuclear norm regularized estimator $\widehat{\beta}_{\psi}$ and the nuclear norm minimizing estimator $\widehat{\beta}_{*}$ can be computed without specifying the number of factors $R_0$, thus avoiding the restriction (C1). In this subsection we characterize $\widehat{\beta}_{*}$ and provide intuition for why these estimators can achieve consistency even when $R_0$ is unknown. We already introduced $\widehat{\beta}_{*} = \lim_{\psi \rightarrow 0} \widehat{\beta}_{\psi}$ in Section (ref). Using Lemma (ref) we can now characterize $\widehat{\beta}_{*}$ differently. It is easy to see that $ \lim_{\psi \rightarrow 0} \psi^{-1} q_\psi(s) = s$, for $s \in [0,\infty)$, and therefore $ \lim_{\psi \rightarrow 0} \psi^{-1} q_\psi(A) = \| A\|_1$, for $A \in \mathbb{R}^{N \times T}$. Lemma (ref) thus implies that $\lim_{\psi \rightarrow 0} \psi^{-1} Q_\psi(\beta) = \| (Y - \beta \cdot X) / \sqrt{NT} \|_1$. Another way to see this is as follows. According to (ref), the limit $\psi \rightarrow 0$ corresponds to choosing $R$ very large, i.e., $R = \min(N,T)$. In this case, $\widehat{\Gamma}_{\psi}(\beta) = Y - \beta \cdot X$, and the profile objective function is $Q_{\psi}(\beta) = \frac{\psi}{\sqrt{NT}} \| \widehat{\Gamma}_{\psi} (\beta) \|_1 = \frac{\psi}{\sqrt{NT}} \| Y - \beta \cdot X \|_1.$ From this we deduce $\lim_{\psi \rightarrow 0} \psi^{-1} Q_{\psi}(\beta) = \frac{\| Y - \beta \cdot X \|_1}{\sqrt{NT}}.$
Notice that for $\psi=0$ we trivially have $Q_0(\beta) = 0$, but the rescaled objective function $ \psi^{-1} Q_\psi(\beta) $ nevertheless has a non-trivial limit as $\psi \rightarrow 0$. Since rescaling the objective function by a constant does not change the minimizer we thus find that
that is, the small $\psi$ limit of the nuclear norm regularized estimator $ \widehat{\beta}_{\psi}$ is the nuclear norm minimizing estimator $\widehat{\beta}_{*}$. The objective function $\left\| Y - \beta \cdot X \right\|_1$ is convex in $\beta$.
We cannot expect the LS estimator $\widehat{\beta}_{{\rm LS},R} $ to have good properties (in particular consistency) if we choose the number of factors equal to, or close to, its maximum possible value $R=\min(N,T)$. It is therefore somewhat surprising that $ \widehat{\beta}_{\psi}$ has a well-defined limit as $\psi \rightarrow 0$, and that we are able to show consistency of the limiting estimator $\widehat{\beta}_{*}$ under appropriate regularity conditions in the following sections, because the resulting estimator for $\Gamma$ is certainly not consistent for $\Gamma_0$ in that limit.\footnote{ The $\psi \rightarrow 0$ limit (for fixed $N$, $T$) of the optimal $\Gamma$ in (ref) is $Y - \widehat{\beta}_{*} \cdot X$, which as $N$ and $T$ grow converges to $\lambda_0 f_0' + E$ for consistent $\widehat{\beta}_{*}$, that is, the estimator for $\Gamma$ that corresponds to $\widehat{\beta}_{*}$ is not consistent for $\lambda_0 f_0' $. }
The main significance of $\widehat{\beta}_{*}$ is that it provides an estimator for $\beta$ that does not require any choice of “bandwidth parameter”, because neither $R$ nor $\psi$ needs to be specified. It thus provides a method to estimate $\beta_0$ consistently without requiring knowledge of an upper bound on $R_0$ as in the condition (C1) above. In a second step we can then estimate $R_0$ consistently by applying, for example, the BaiNg2002 method for pure factor models without regressors to the matrix $Y - \widehat{\beta}_{*} \cdot X$.
Notice that the pooled OLS estimator $\beta_0$ minimizes $\| Y - \beta \cdot X \|_2^2 = \sum_{r=1}^{\min(N,T)} s_r( Y - \beta \cdot X)^2$, the $\ell^2$-norm of the singular values of the residual matrix, $Y - \beta \cdot X$, while the nuclear norm minimizing estimator $\widehat{\beta}_*$ minimizes the $\ell^1$-norm, $\| Y - \beta \cdot X \|_1 = \sum_{r=1}^{\min(N,T)} s_r( Y - \beta \cdot X),$ of the residual matrix. The relationship between these two estimators is therefore analogous to that of the OLS estimator and the LAD (least absolute deviation) estimator for cross-sectional samples. $\widehat{\beta}_*$ is robust with respect to the unobserved factors, which are “outliers" in the singular value spectrum, while the pooled OLS estimator is not robust towards the presence of those unobserved factors (because they may be correlated with the regressors).
To understand intuitively why $\widehat{\beta}_{*}$ can be consistent, consider that at any $\beta \neq \beta_0$ the residual matrix is $Y - \beta \cdot X = \Gamma_0 + E + X \cdot (\beta_0 - \beta)$. For $\widehat{\beta}_{*}$ to correctly identify $\beta_0$, we need that $\| Y - \beta \cdot X \|_1 > \| Y - \beta_0 \cdot X \|_1 = \| \Gamma_0 + E \|_1$ for all $\beta \neq \beta_0$, which requires that the term $X \cdot (\beta_0 - \beta)$ does not cancel out singular values of $\Gamma_0 + E$. Since $\Gamma_0$ is low-rank and $E$ is generically full-rank, this imposes different requirements depending on the rank of $X$: (i) if $X$ is low-rank, then $X$ must operate under sufficiently different row and column spaces than $\Gamma_0$, so that $X \cdot (\beta_0 - \beta)$ cannot be absorbed into the factor structure; (ii) if $X$ is high-rank, then $X$ must be sufficiently different from $E$, so that $X \cdot (\beta_0 - \beta)$ does not simply offset the idiosyncratic errors. The formal conditions in Section (ref) make these requirements precise. Notice also that even when $\widehat{\beta}_{*}$ is consistent for $\beta_0$, the corresponding residual $Y - \widehat{\beta}_{*} \cdot X$ estimates $\Gamma_0 + E$, not $\Gamma_0$ alone.
Next, we provide an identification result showing how the nuclear norm regularization approach helps to overcome the restrictions (C2) above, that is, how to estimate regression coefficients for low-rank regressors when $R_0$ is unknown. The goal is to provide conditions on the regressors $X_k$ under which the nuclear norm penalization approach indeed solves the matrix separation problem for low-rank regressors and interactive fixed effects.
To build intuition, we first present an identification result based on the expected objective function for fixed $N$ and $T$. We consider
Here, the expectation is conditional on all the regressors $(X_1,\ldots,X_K)$, and also implicitly on all the parameters $\beta_0$ and $\Gamma_0$, because those are treated as non-random.\footnote{ $\bar \beta_\psi$ can be viewed as a population version of $\widehat \beta_\psi$ for an appropriately defined population distribution of $Y$ conditional on $X$. Proposition (ref) below establishes identification of $\beta_0$ for this population objective function. The sample analog results for $\widehat{\beta}_{\psi}$ and $\widehat{\beta}_{*}$ are established in Section (ref) using different proof techniques. }
For a matrix $A$, let $\mathbf{P}_A := A (A'A)^{\dagger} A'$ and $\mathbf{M}_A := \mathbf{I} - \mathbf{P}_A$ be the projectors onto and orthogonal to the column span of $A$, where $\mathbf{I} $ is the identity matrix of appropriate dimensions, and $\dagger$ refers to the Moore-Penrose generalized inverse. Remember also our notation $\alpha \cdot X := \sum_{k=1}^K \alpha_k X_k$ for $\alpha \in \mathbb{R}^K$. For vectors $v$ we write $\|v\|$ for the Euclidian norm.
The proof is given in the appendix. The proposition considers fixed $N$, $T$, with only $\psi \rightarrow 0$.\footnote{ Display (ref) in the appendix provides a bound on $\| \bar \beta_\psi - \beta_0 \|$ for finite $\psi$, but the limit $\psi \rightarrow 0$ is what matters most to us, because that limit allows to identify $\beta_0$. } The statement of the proposition implies that $\lim_{\psi \rightarrow 0} \bar \beta_\psi = \beta_0 $. Thus, the proposition provides conditions under which the nuclear norm regularization approach identifies the true parameter $\beta_0$. The proposition does not restrict the rank of the regressors, so the result is applicable to both low-rank and high-rank regressors. The assumption $\mathbb{E}( \left. E_{it} \, \right| \, X ) = 0$ requires strict exogeneity of all regressors, but we will allow for pre-determined regressors in consistency results of Section (ref) below.
The beauty of Proposition (ref) is that it provides a very easy to interpret non-collinearity condition on the regressors $X_k$. It requires that for any linear combination of the regressors the part $ \mathbf{M}_{\lambda_0} (\alpha \cdot X) \mathbf{M}_{f_0}$, which cannot be explained by neither $\lambda_0$ nor $f_0$, is larger in terms of nuclear norm than the part $ \mathbf{P}_{\lambda_0} (\alpha \cdot X) \mathbf{P}_{f_0}$, which can be explained by both $\lambda_0$ and $f_0$. For a single ($K=1$) regressor with $X_{1,it}=v_i w_t$, as in Example (ref), the condition simply becomes $\| \mathbf{M}_{\lambda_0} v \| \| \mathbf{M}_{f_0} w \| > \| \mathbf{P}_{\lambda_0} v \| \| \mathbf{P}_{f_0} w \| $. Here, $\| \mathbf{M}_{\lambda_0} v \|^2$ and $\| \mathbf{P}_{\lambda_0} v \|^2$ are the residual sum of squares and the explained sum of squares, respectively, of a regression of $v_i$ on the $\lambda_{0,i}$ and analogously for $\| \mathbf{M}_{f_0} w \|^2 $ and $\| \mathbf{P}_{f_0} w \|^2 $. In Example (ref) we obviously have $\| \mathbf{M}_{\lambda_\bigstar} v \| =0$ and $ \| \mathbf{M}_{f_\bigstar} w \| =0$, that is, the parameters $R_{\bigstar}$, $\beta_{\bigstar}$, $\lambda_{\bigstar}$, $f_{\bigstar}$ are ruled out by the condition on the regressors in Proposition (ref).
Related to the regularity condition ((ref)) of Proposition (ref), it is possible to show (see Bai2009, MoonWeidner2017) that the weaker condition $ \mathbf{M}_{\lambda_0} (\alpha \cdot X) \mathbf{M}_{f_0} \neq 0$ for any linear combination $\alpha \neq 0$ is sufficient for local identification of $\beta$ in a sufficiently small neighbourhood around $\beta_0$. However, that weaker condition is not sufficient for global identification of $\beta_0$, as illustrated by the examples in the supplementary appendix S.3 of MoonWeidner2017. The stronger condition ((ref)) in Proposition (ref) guarantees global identification of $\beta_0$ when using the nuclear norm penalization approach as a regularization device.
Providing such global identification conditions for models with low-rank regressors and unknown $R_0$ is a new contribution to the interactive fixed effects literature.\footnote{ If the model would not have any idiosyncratic errors (i.e.\ $E=0$), then $Y - \beta \cdot X = (\beta_0 - \beta) \cdot X + \Gamma_0$, and a natural solution to this identification problem would be to choose $\beta$ as the solution to the rank minimization problem $ \min_{\beta \in \mathbb{R}^K} \; {\rm rank}\left( Y - \beta \cdot X \right) , $ where at the true parameters we have ${\rm rank}\left( Y - \beta_0 \cdot X \right)= {\rm rank}(\Gamma_0) = R_0$, that is, we are minimizing the number of factors required to describe the data. However, once idiosyncratic errors $E$ are present, then this rank minimization does not work, because $Y - \beta\cdot X$ is of large rank for all $\beta$. } Our approach here is similar to the “Identification via a Strict Convex Penalty” proposed in chen2012estimation. As noted above, the proof of Proposition (ref) applies to the population objective function $\bar{\beta}_\psi$ for fixed $N$ and $T$. The consistency results for the sample estimators $\widehat{\beta}_{\psi}$ and $\widehat{\beta}_{*}$ in Section (ref) are established using different asymptotic arguments, in particular relying on Lemma S.3 in the supplementary appendix rather than on Proposition (ref) directly.
Proposition (ref) above provides an identification result for $\beta_0$ for fixed $N$ and $T$, based on the expected objective function. We now turn to the actual estimates $\widehat \beta_\psi$ and $\widehat \beta_*$ and investigate their properties as $N,T \rightarrow \infty$.
All our consistency results for $\widehat \beta_\psi$ are for asymptotic sequences where $\psi = \psi_{NT} \rightarrow 0$, as $N,T \rightarrow \infty$, but we do not usually make the dependence of $\psi$ on the sample size explicit. In addition, we assume that the number of the regressors $K$ and the true number of factors $R_0 = {\rm rank}(\Gamma_0)$ are both fixed. However, we do not restrict whether the factors are strong or weak, nor do we restrict the magnitude of $\Gamma_0$ in any matrix norm.
Here, we consider a special case where the regressors $X_1,...,X_K$ are of low rank. This section is short, because the results here are relatively straightforward extensions of Section (ref). The more general case that allows both high-rank and low-rank regressors will be discussed in the following subsection.
Various examples of DGP's for $E$ that satisfy the assumption $\| E \|_\infty = O_P(\sqrt{\max(N,T)})$ can be found in the supplementary appendix S.2 of MoonWeidner2017. Loosely speaking, that condition is satisfied as along as the entries $E_{it}$ have zero mean, some appropriately bounded moments, and are not too strongly correlated across $i$ and over $t$. The condition $\sum_{k=1}^K {\rm rank}(X_k) = O_P(1)$ requires all regressors to be low-rank. The interpretation of condition (ref) is the same as for condition (ref) in Proposition (ref), and Theorem (ref) is indeed a sample version of that proposition, except that low-rank regressors are required here.
The theorem shows that both $\widehat \beta_*$ and $\widehat \beta_\psi $, for $\psi = \psi_{NT} = O\left( 1 / {\sqrt{\min(N,T)}} \right)$, converge to $\beta_0$ at a rate of at least $\sqrt{\min(N,T)}$. The proof of the theorem is provided in the appendix, and is a relatively easy generalization of the proof of Proposition (ref).
We note that condition (ref) requires a lower bound of order $c > 0$ (a positive constant), whereas condition (ref) in Proposition (ref) only requires a strict inequality. This difference reflects the distinction between a population identification condition (Proposition (ref), for fixed $N$ and $T$) and a sample condition for asymptotic analysis (Theorem (ref), as $N, T \rightarrow \infty$). The nuclear norms appearing in (ref) are of order $\sqrt{NT}$, so requiring their difference to be bounded below by a positive constant $c$ uniformly over all sample sizes ensures the condition does not become slack asymptotically.
Returning to Example (ref), the DGP with low-rank regressor $X_1 = vw'$ satisfies condition (ref) when the vectors $v$ and $w$ are sufficiently different from the factor loadings $\lambda_0$ and factors $f_0$. Specifically, if $v$ is not well-approximated by linear combinations of the columns of $\lambda_0$, and similarly $w$ is not well-approximated by linear combinations of the columns of $f_0$, then the projection $\mathbf{M}_{\lambda_0} X_1 \mathbf{M}_{f_0} = (\mathbf{M}_{\lambda_0} v)(\mathbf{M}_{f_0} w)'$ retains substantial norm, while $\mathbf{P}_{\lambda_0} X_1 \mathbf{P}_{f_0} = (\mathbf{P}_{\lambda_0} v)(\mathbf{P}_{f_0} w)'$ remains small. In the extreme case where $v$ lies in the column space of $\lambda_0$ or $w$ lies in the column space of $f_0$, condition (ref) fails and $\beta_0$ is not identified, consistent with the non-identification illustrated in Example (ref).
The previous subsection considered the case where all regressor matrices $X_k$ are low-rank. We now study situation where all or some of the regressor matrices $X_k$ are high-rank.
Applying Lemma (ref) and the model for $Y$ we have
The proof strategy for Theorem (ref) requires that both $\Gamma$ and $X_k$ are low-rank, which allows to (approximately) separate off $E$ in this expression for $Q_\psi(\beta) $. But if one of the regressors $X_k$ is a high-rank matrix that proof strategy turns out not to work anymore, because the singular value spectrum of the sum of two high-rank matrices $E$ and $X_k$ does not decompose (or approximately decompose) into a contribution from $E$ and from $X_k$, but instead all singular values depend on both of those high-rank matrices in a complicated non-linear way.
We therefore now follow a different strategy, where instead of studying the objective function after profiling out $\Gamma$, we now explicitly study the properties of the estimator for $\Gamma$. Let
For the results in this subsection we are going to first show consistency of $\widehat{\Gamma}_{\psi}$, and afterwards use that to obtain consistency of $\widehat{\beta}_{\psi}$. This is a very different logic than in the preceding section, where consistency of $\widehat{\Gamma}_{\psi}$ is usually not achieved, because we do not impose any lower bound on $\psi$. In order to achieve consistency of $\widehat{\Gamma}_{\psi}$ one requires $\psi$ not be too small. The approach here is much more similar to the machine learning literature (e.g., Negahbanetal2012), where the matrix that we call $\Gamma$ is usually the object of interest, and correspondingly a lower bound on the penalization parameter is required. We also follow that literature here by imposing a so-called “restricted strong convexity” condition below, which is critical to show consistency of $\widehat{\Gamma}_{\psi}$ and consequently of $\widehat{\beta}_{\psi}$ in the following.
It is convenient to introduce some additional notation: Let $\rm vec(A)$ be the vector that vectorizes the columns of $A$. Denote $\rm mat(\cdot)$ as the inverse operator of $\rm vec(\cdot)$, so for $a = \rm vec(A)$ we have $\rm mat(a) = A$. We use small letters to denote vectorized variables and parameters. Let $y = {\rm vec}(Y), x_k ={\rm vec}(X_k), \gamma_0 = {\rm vec}(\Gamma_0),$ and $e = {\rm vec}(E)$. Define $x = (x_1,...,x_k)$. Using this, we express the model ((ref)) as $y = x \beta_0 + \gamma_0 + e$, where all the summands are $NT$-vectors, and the least-squares objective function reads $L(\beta,\Gamma) = \frac{1}{2NT} (y - x \beta - \gamma)' (y - x \beta - \gamma)$.
The intuitive interpretation of Assumption (ref) is very similar to condition (ref) in Proposition (ref): The cone $\mathbb{C}$ contains matrices $\Theta$ that are close to $\Gamma_0 = \lambda_0 f_0'$, in the sense that the part $\mathbf{M}_{\lambda_0} \Theta \mathbf{M}_{f_0}$ of $\Theta$ that cannot be explained by $\lambda_0$ and $f_0$ is small compared to the remaining part of $\Theta$, in terms of nuclear norm. The assumption then imposes that all those matrices $\Theta \in \mathbb{C}$ in the cone are sufficiently different from the regressors, in the sense that $\theta = {\rm vec}(\Theta)$ cannot be perfectly explained by $ x_k ={\rm vec}(X_k)$.
Specifically, the condition assumes that the quadratic term, $\frac{1}{2NT} (\gamma - \gamma_0)'\mathbf{M}_x(\gamma - \gamma_0)$, of the profile likelihood function, $\min_{\beta} L(\beta,\Gamma)$, is bounded below by a strictly convex function, $\frac{\mu}{2NT} (\gamma - \gamma_0)'(\gamma - \gamma_0)$, if $\Theta = \Gamma - \Gamma_0$ belongs in the cone $\mathbb{C}$. Notice that without any restriction on the parameter $\theta = \gamma - \gamma_0$, we cannot find a strictly positive constant $\mu > 0$ such that $\min_{\Gamma} (\gamma - \gamma_0)'\mathbf{M}_x(\gamma - \gamma_0) \geq \mu (\gamma - \gamma_0)'(\gamma - \gamma_0)$. Assumption (ref) imposes that if we restrict the parameter set to be the cone $\mathbb{C}$, then we can find a strictly convex lower bound of the quadratic term of the profile likelihood. Assumption (ref) corresponds to the restricted strong convexity condition in Negahbanetal2012, and it plays the same role as the restricted eigenvalue condition in recent LASSO literature (e.g., see CandesTao2007 and Bickeletal2009).
Notice that for $R_0=0$ we have $\mathbf{M}_{\lambda_0} = \mathbb{I}_N$ and $\mathbf{M}_{f_0} =\mathbb{I}_N $, and therefor $\mathbb{C} = \{ 0_{N \times T} \}$, implying that Assumption (ref) is trivially satisfied for any $\mu>0$.
Assumption (ref) requires that for ${\rm mat}(\theta) \in \mathbb{C}$ a lower bound of $\theta'\mathbf{M}_x \theta$ is given by the strictly convex function $\mu \, \theta' \theta $. To have some intuition, suppose that the regressor is scalar and assume that $\| X \|_2 = (x'x)^{1/2} = 1$ without loss of generality because the projection operator $\mathbf{M}_x$ is invariant to the scale change. Also assume that $\theta \neq 0$. Then,
In this case, if the limit of the distance between the regressor and the restricted parameter set is positive, Assumption (ref) is satisfied if $\mu := \liminf_{N,T}\min_{\theta \in \mathbb{C}} \| x - \theta \|^2$, the distance of the normalized regressor $x$ and convex cone $\mathbb{C}$, is positive. An obvious necessary condition for this is that the normalized regressor does not belong in the cone $\mathbb{C}$, that is, \[ \| \mathbf{M}_{\lambda_0} X \mathbf{M}_{f_0} \|_1 > 3 \| X - \mathbf{M}_{\lambda_0} X \mathbf{M}_{f_0} \|_1 . \] For example, if $X$ has an approximate factor structure \[ X = \lambda_x f_x' + E_{x}, \] with $E_{x,it} \sim i.i.d. {\cal N}(0,\sigma^2)$, then we can use random matrix theory results to show that Assumption (ref) is satisfied.
The lemma shows that once we impose restricted strong convexity and a lower bound on $\psi$, then we can indeed bound the difference between $\widehat{\Gamma}_{\psi} $ and $\Gamma_0$. This lemma is obviously key to obtain a consistency result for $\widehat{\Gamma}_{\psi} $. Notice furthermore that
that is, once we have a consistency result for $\widehat{\Gamma}_{\psi} $ (or equivalently $\widehat{\gamma}_{\psi} $), then we can also show consistency of $\widehat{\beta}_{\psi}$. Using that derivation strategy we obtain the following theorem, which provides a consistency result for both $\widehat{\Gamma}_{\psi}$ and $\widehat{\beta}_{\psi} $.
The additional regularity conditions imposed in Theorem (ref) are weak and quite general. As mentioned before, various examples of $E$ that satisfy (i) can be found in the supplementary appendix S.2 of MoonWeidner2017; these include weakly dependent errors, and nonidentical but independent sequences of errors. Condition (ii) is satisfied if the regressors are exogenous with respect to the error, $\mathbb{E}(x_{it}e_{it}) = 0$, and $x_{it}e_{it}$ are weakly correlated over $t$ and across $i$ so that $\frac{1}{NT} \sum_{i,j=1}^N \sum_{t,s=1}^T \allowbreak \mathbb{E}(x_{k,it}x_{l,js}e_{it}e_{js})$ is bounded asymptotically. Condition (iii) is the standard non-collinearity condition for the regressors. Condition (iv) restricts the choice of the regularization parameter $\psi$, which has to converge to zero (as discussed before for identification and consistency of $\beta_0$), but not too quickly (if $\psi$ is too small, then $\widehat{\Gamma}_{\psi} $ picks up all the noise $E$ and cannot be consistent). The conditions (i) and (iv) are sufficient regularity conditions for ((ref)). To see this in more detail, since ${\rm mat}(\mathbf{M}_x e) = E - \sum_{k=1}^K\widehat{E}_k$ with $\widehat{E}_k = X_k (x_k'x_k)^{-1}(x_k' e)$, we have
Then, choosing $\psi \geq \frac{2}{\sqrt{NT}} \| E \|_\infty \left( 1 + \frac{O_P(1)}{\| E \|_{\infty}}\right)$ makes $\psi$ satisfy $(\ref{def.psi})$ with probability approaching one, and the rate condition in (iv) guarantees this.
Theorem (ref) requires $\psi = \psi_{NT}$ to grow faster than $1 / \sqrt{\min(N,T)} $. By choosing $\psi$ appropriately we can therefore obtain a convergence rate of $\widehat{\beta}_{\psi}$ that is just below $\sqrt{\min(N,T)}$, which is essentially the same convergence rate that we found in Section (ref) for the case of only low-rank regressors.
For the special case $R_0=0$ we have $\Gamma_0 = \mathbf{0}_{N \times T}$, and if $\psi_{NT}$ then satisfies ((ref)), one can show that
with probability approaching one (wpa1), see the appendix for a proof of this. In this case, the regularized estimator of $\beta$ becomes the pooled OLS estimator, $ \widehat{\beta}_{\psi} = (x'x)^{-1}x'y, $ wpa1.
Here, we establish consistency of the nuclear norm minimization estimator $\widehat{\beta}_*$ for high-rank regressors. For simplicity we only discuss the case of a single regressor ($K=1$) in the main text, and we simply write $X$ for the $N \times T$ regressor matrix $X_1$ in this subsection. The general case of multiple regressors ($K>1$) is discussed in Appendix (ref).
Remember that $\widehat{\beta}_*$ is the minimizer of the objective function $ \| Y - \beta \cdot X \|_1 = \| E + (\beta_0-\beta) X + \Gamma_0 \|_1 = \sum_{r} s_r\left( E + (\beta_0-\beta) X + \Gamma_0 \right)$. Asymptotically separating the contribution of the low-rank matrix $\Gamma_0$ to the singular values of the sum $E + (\beta_0-\beta) X + \Gamma_0 $ is possible under a strong factor assumption.\footnote{ In MoonWeidner2015,MoonWeidner2017 we use the perturbation theory of linear operator to do exactly that.} However, characterizing the singular values of the sum of two high-rank matrices $ E + (\beta_0-\beta) X $ requires results from random matrix theory that are usually only shown under relatively strong assumptions on the distribution of the matrix entries. We therefore first provide a theorem under high-level assumptions, and afterwards discuss how to verify those assumptions using results from random matrix theory. We write SVD for “singular value decomposition” in the following.
The theorem considers the case $N>T$, because the two panel dimensions are not treated symmetrically in the assumptions and proof of this theorem. Alternatively, we could consider $T<N$, but then we also need to swap $N$ and $T$, and replace $X$ by $X'$ and $E$ by $E'$ in all the assumptions (the case $T=N$ is ruled out here for technical reasons). For both $N>T$ and $T<N$ the statement of theorem can be written as $\sqrt{\min(N,T)} \left( \widehat{\beta}_{*} - \beta_0 \right) = O_P(1)$, that is, we have the same convergence rate result here for $\widehat{\beta}_{*} $ as in Theorem (ref) above.
Condition (i) in the theorem is quite weak, we already discussed the rate restriction on $\| E \|_{\infty}$ above, and we have $\| X \|_\infty \leq \|X\|_2 = \sqrt{\sum_i \sum_t X_{it}^2} = O_P(\sqrt{NT})$ as long as $\sup_{it} \mathbb{E} (X_{it}^2)$ is finite. Condition (ii) almost follows from $\| E \|_{\infty} = O_P(\sqrt{N})$, because we have $ \| E \|_1 \leq {\rm rank}(E) \, \| E \|_{\infty} \leq T \| E \|_{\infty} =O_P(T \sqrt{N}) $, and the assumption is only slightly stronger than this in assuming a fixed upper bound with probability approaching one, which can also be verified for many error distributions. Condition (iii) is a high level condition and will be satisfied if
for some finite constant $M$, where $U_{E,r}$ and $V_{E,r}$ are the $r^{th}$ columns of $U_{E,r}$ and $V_{E}$, respectively. An example of DGP's of $X$ and $E$ that satisfies condition ((ref)) is given by Assumption LL (i) and (ii) in MoonWeidner2015. Condition (iv) rules out “low-rank regressors”, for which we typically have $ \| \mathbf{M}_{\lambda_0} X \mathbf{M}_{f_0} \|_1 = O_P(\sqrt{NT})$, but is satisfied generically for “high-rank regressors”, for which $\mathbf{M}_{\lambda_0} X \mathbf{M}_{f_0} $ has $T$ singular values of order $\sqrt{N}$, so that $ \| \mathbf{M}_{\lambda_0} X \mathbf{M}_{f_0} \|_1$ is of order $T \sqrt{N}$. Condition (v) requires that the singular vectors of $\mathbf{M}_{\lambda_0} X \mathbf{M}_{f_0}$ are sufficiently different from the singular vectors $\mathbf{M}_{\lambda_0} E \mathbf{M}_{f_0}$. If $X$ and $E$ are independent, then we expect that assumption to hold quite generally.
In Section (ref) we have shown that $\widehat{\beta}_{\psi}$ and $\widehat{\beta}_{*}$ are consistent for $\beta_0$ at a $\sqrt{\min(N,T)}$-rate, which is a slower convergence rate than the $\sqrt{NT}$-rate at which the LS estimator $\widehat{\beta}_{{\rm LS},R}$ converges to $\beta_0$ under appropriate regularity conditions. Our Monte Carlo results in Section (ref) confirm this relatively slow rate of convergence of $\widehat{\beta}_{\psi}$ and $\widehat{\beta}_{*}$, that is, those rates are not an artifact of our proof strategy, but are a genuine property of those estimators. In this section we investigate how to establish an estimator that is asymptotically equivalent to the LS estimator, and yet avoids minimizing any non-convex objective function. Our suggestion is to use either $\widehat{\beta}_{\psi}$ or $\widehat{\beta}_{*}$ as a preliminary estimator and iterate estimating $\Gamma_0 = \lambda_0 f_0'$ and $\beta_0$ a finite number of times.
The conditions that are usually needed to show that the global minimizer $\widehat{\beta}_{{\rm LS},R}$ of the objective function $L_R(\beta)$ is consistent for $\beta_0$ (i.e.\ Assumption A in Bai2009, or Assumption 4 in MoonWeidner2017) are not required here, because we have already shown consistency of $\widehat{\beta}_{\psi}$ or $\widehat{\beta}_{*}$ under different conditions (our discussion in Section (ref) highlights those differences). It is therefore convenient to introduce a local version of the LS estimator in (ref) as
where $r_{NT}$ is a sequence of positive numbers such that $r_{NT} \rightarrow 0$ and $\sqrt{NT} \, r_{NT} \rightarrow \infty$. Those rate conditions guarantee that $\widehat{\beta}^{\rm \, local}_{{\rm LS},R} $ is an interior point of $ {\cal B}(\beta_0, r_{NT})$, wpa1, under the assumptions of Theorem (ref) below. If the global minimizer $\widehat{\beta}_{{\rm LS},R}$ is consistent, then we expect $\widehat{\beta}_{{\rm LS},R} = \widehat{\beta}^{\rm \, local}_{{\rm LS},R} $ wpa1, but $\widehat{\beta}^{\rm \, local}_{{\rm LS},R} $ is consistent by definition even if $\widehat{\beta}_{{\rm LS},R}$ is not. Our goal in the following is to obtain an estimator that is asymptotically equivalent to $\widehat{\beta}^{\rm \, local}_{{\rm LS},R} $.
For simplicity, we first discuss the case where the number of factors $R_0$ is known. For unknown $R_0$ we recommend using a consistent estimator of $R_0$ instead, and we discuss estimation of $R_0$ in Section (ref) below. Starting from our initial nuclear norm regularized or minimized estimators we consider the following iteration procedure to obtain improved estimates of $\beta$:
The following theorem shows that if the initial estimator $\widehat{\beta}^{(0)}$ is consistent, then $\widehat{\beta}^{(s)}$ gets close to $\widehat{\beta}^{\rm \, local}_{{\rm LS},R_0} $ as the number of iteration $s$ increases. This result is very similar to the quadratic convergence result of a Newton-Raphson algorithm for minimizing a smooth objective function, and the above iteration step is indeed very similar to performing a Newton-Raphson step to minimize $L_{R_0}(\beta)$.
Here, assumption (i) is a strong factor condition, and is often used in the literature on interactive fixed effects. The conditions in assumption (ii) of the theorem have been discussed in previous sections and are quite weak (remember that $\| X_k \|_{\infty} \leq \| X_k \|_{2} = \sqrt{x_k' x_k}$). Assumption (iii) guarantees that $ L_R(\beta)$ is locally convex around $\beta_0$ -- that condition can equivalently be written as $\operatorname*{plim}_{N,T \rightarrow \infty} \| \mathbf{M}_{\lambda_0} (\alpha \cdot X) \mathbf{M}_{f_0} \|_2 > 0$ for any $\alpha \in \mathbb{R}^K \setminus \{0\}$, which connects more closely to our discussion in Section (ref). This is a non-collinearity condition on the regressors after profiling out both $\lambda_0$ and $f_0$. Only the true values $\lambda_0$ and $f_0$ appear in that non-collinearity condition, and it is therefore much weaker than the corresponding assumptions required for consistency of $\widehat{\beta}_{{\rm LS},R_0}$ in Bai2009 and MoonWeidner2017. Our results from the previous sections show that $\| \widehat{\beta}^{(0)} - \beta_0 \| = O_P(c_{NT})$ for both $\widehat{\beta}^{(0)} = \widehat{\beta}_{\psi}$ and $\widehat{\beta}^{(0)} =\widehat{\beta}_{*}$, under appropriate assumptions, where $c_{NT}$ is typically either $c_{NT} = 1 / \sqrt{\min(N,T)}$ or slightly slower than this, if $\psi = \psi_{NT}$ is chosen appropriately.
The following corollary is an immediate consequence of Theorem (ref).
The first statement of the corollary shows that if the initial estimators $ \widehat{\beta}_{\psi}$ and $ \widehat{\beta}_{*}$ satisfy typical convergence rates results derived in the previous sections, then the iterated estimator $ \widehat{\beta}^{(s)}$ is asymptotically equivalent to $ \widehat{\beta}^{\rm \, local}_{{\rm LS},R_0} $ after $s=2$ iterations or more. Remember that if $ \widehat{\beta}_{{\rm LS},R_0} $ is consistent, then we have $ \widehat{\beta}^{\rm \, local}_{{\rm LS},R_0} = \widehat{\beta}_{{\rm LS},R_0} $ wpa1, but by showing asymptotic equivalence with $ \widehat{\beta}^{\rm \, local}_{{\rm LS},R_0} $ here we avoid imposing conditions that require consistency of $ \widehat{\beta}_{{\rm LS},R_0} $.
From the results in Bai2009 and MoonWeidner2017 we also know that $ \widehat{\beta}^{\rm \, local}_{{\rm LS},R_0} $ is asymptotically normally distributed, but potentially with a bias in the limiting distribution. According to the corollary the same is therefore true for $\widehat{\beta}^{(s)}$ for $s \geq 2$. Asymptotic bias corrections could then also be applied to $\widehat{\beta}^{(s)}$, $s \geq 2$, to eliminate the bias in the limiting distribution and allow for inference on $\beta_0$. See Bai2009 and MoonWeidner2017 for details.
To implement the nuclear norm regularized estimator we need to choose the regularization parameter $\psi$, and for the post estimator $\widehat{\beta}^{(s)}$ we need to determine the number of factors $R_0$. In this section we suggest a data dependent choice of $\psi$ as well as an estimate of $R_0$. We assume that an upper bound $R_{\rm max} \geq R_0$ is known.
We suggest the following procedure to choose $\psi$.
This choice of $\widehat{\psi}$ is motivated by the condition ((ref)) in Lemma (ref), which guarantees that $\psi$ is sufficiently large to obtain estimates $\widehat \Gamma_\psi$ that are close to $\Gamma_0$. Notice also that the nuclear norm minimizing estimator $\widehat{\beta}_{*}$ in step 1 does not require any regularization parameter to be specified.
The post nuclear norm regularized estimator introduced above assumes that the number of factors $R_0$ is known. In practice $R_0$ needs to be estimated, for example, by applying a consistent estimation method for the number of the factors in a pure factor model to the residuals $\widehat{E}_*$, see e.g. BaiNg2002, Onatski2010 and AhnHorenstein2013.
For our Monte Carlo simulations below we use an alternative estimation method that thresholds the singular values of $\widehat{E}_*$ using the estimate $\widehat{\psi}$ introduced above. Namely, we estimate $R_0$ by \[ \widehat{R} = \sum_{r=1}^{\min(N,T)} \mathbbm{1} \left\{ s_r\left( \widehat{E}_* \right) \geq 2 \, \sqrt{NT} \, \widehat{\psi} \right\}. \] The motivation behind this estimator is that those singular values of $\widehat{E}_*$ that are significantly larger than $\sqrt{NT} \, \widehat{\psi} $ should correspond to factors, while singular values close to $\sqrt{NT} \, \widehat{\psi} $ and smaller should originate from idiosyncratic noise. The choice of the factor 2 in the formula for $\widehat{R}$ is somewhat arbitrary, any alternative factor larger than one would also be plausible here.
We generate data from the following linear panel model regression model with two regressors (including the intercept) and two factors:
where $f_{0,tr} \sim i.i.d. \, {\cal N}(0,1)$; $\lambda_{0,ir},\lambda_{x,ir} \sim i.i.d. \, {\cal N}(1,1)$; $E_{x,it}, E_{it} \sim i.i.d. \, {\cal N}(0,1)$; all mutually independent. Table (ref) reports the bias and standard deviation for the various estimators for different combinations of $N$ and $T$.
As shown in Table (ref), the nuclear norm regularized estimator $\widehat{\beta}_{\psi}$ and the nuclear norm minimization estimator $\widehat{\beta}_*$ have biases due to the regularization which vanish slowly as the sample size increases. This confirms that those estimators are indeed not $\sqrt{NT}$ consistent, but only have a $\sqrt{\min(N,T)}$ convergence rate to $\beta_0$. The table also shows that the post nuclear norm regularized estimation ($\widehat{\beta}^{(s)}$) quickly reduces the bias, and essentially agrees with the LS estimator (which is a consistent estimator in this MC design) after two iterations, as the theory predicts. The columns ALT(1) - ALT(5) in that table contain the results for an alternative bias corrected estimator that is presented in the appendix. It turns out that the alternative bias correction method is less effective in reducing the bias, and we therefore do not discuss it in the main text. Our recommendation in practice for inference on $\beta_0$ is the iteration procedure for $\widehat{\beta}^{(s)}$ explained in the previous section.
In this paper we analyze two new estimation methods for interactive fixed effect panel regressions that are based on convex objective functions: (i) nuclear norm penalized estimation, and (ii) nuclear norm minimizing estimation. The resulting estimators can also be applied in situations where the LS estimator may not be consistent, in particular when low-rank regressors are present and the true number of factors is unknown. We provide consistency and convergence rate results for the new estimators of the regression coefficients, and we show how to use them as a preliminary estimator to achieve asymptotic equivalence to the local version of the LS estimator. We have focused on the linear model with homogenous coefficients, which is a natural starting point to understand the usefulness of nuclear norm penalization approach for panel regression models, but there are several ongoing extensions, including developing a unified method to deal with non-linear models, heterogeneous coefficients, treatment effect estimation, nonparametric sieve estimation, and high-dimensional regressors --- see Atheyetal2017 and chernozhukov2018inference. In Appendix (ref) we briefly discuss how the nuclear norm regularization approach extends to nonlinear panel models, including maximum likelihood, weighted least squares, and quantile regression settings. We note that sharper convergence rates than those presented in the appendix have been established for specific nonlinear models in the subsequent literature, including belloni2019high, wang2022low, and feng_2023 for quantile regression.
\ifx\undefined\leavevmode\rule[.5ex]{3em}{.5pt}\ \fi \ifx\undefined\textsc \let\tmpsmall\tmpsmall\sc \fi