Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
35,664 characters · 6 sections · 48 citation commands
Performance of Empirical Risk Minimization For Principal Component Regression
\symbolfootnote[0]{\\ $^{\dag}$ Department of Economics and Business, Universitat Pompeu Fabra and Barcelona GSE;\\ e-mail: [email removed], [email removed].\\ $^{\ddag}$ Department of Economics and Business Economics, Aarhus University; \\ e-mail: [email removed].\\ $^*$ Corresponding author. \\ We have benefited from discussions with Gabor Lugosi, David Rossell and Piotr Zwiernik.\\ Christian Brownlees acknowledges support from the Spanish Ministry of Science and Technology (Grant MTM2012-37195); the Ayudas Fundaci\'on BBVA Proyectos de Investigación Cient\`ifica en Matemáticas 2021; the Spanish Ministry of Economy and Competitiveness through the Severo Ochoa Programme for Centres of Excellence in R&D (SEV-2011-0075).\\ Gu\dh mundur Stef\'an Gu\dh mundsson acknowledges financial support from the Danish National Research Foundation (DNRF Chair grant number DNRF154).}
\doublespacing
Principal component regression (PCR) is a regression methodology with a long and well established tradition that can be traced back to at least Hotelling:1957 and Kendall:1957. In a nutshell, PCR consists in forecasting a prediction target of interest on the basis of the principal components extracted from a potentially large set of predictors. PCR is a popular tool for forecasting in macroeconomics where it is documented to perform favourably relative to a number of competing approaches Stock:Watson:2012.
In this paper we study the properties of PCR from a learning theory perspective. Our main contribution consists in establishing nonasymptotic prediction performance guarantees for PCR. Our main result may be interpreted as a nonasymptotic analogue of classic asymptotic results on the prediction properties of PCR obtained in Stock:Watson:2002. Our analysis shows that, under appropriate conditions, PCR achieves near-optimal performance. An important feature of our analysis is that we treat PCR as a regularization procedure and we do not assume that the data are generated by a factor model. In particular, as is customary in learning theory, the relation between the prediction target and the predictors is not specified. That being said, as the factor model feinschmecker will recognize, our framework relies on assumptions and proof strategies analogous to the ones used in the factor model literature. In particular we build upon classic contributions such as Bai:Ng:2002, Bai:2003, Fan:Liao:Mincheva:2011,Fan:Liao:Mincheva:2013 and Onatski:2012 among others.
PCR may be described as a two-step procedure. Let $ \mathcal D = \{ (Y_t, \bm X_t')' \}_{t=1}^T $ be a stationary sequence of zero-mean random vectors taking values in $\mathcal Y \times \mathcal X \subset \mathbb R \times \mathbb R^p$. The goal is to forecast the prediction target $Y_t$ using the $p$-dimensional vector of predictors $\bm X_t=(X_{1\,t},\ldots,X_{p\,t})'$. The first step of PCR consists in computing the $T \times K$ principal components matrix $\widehat{\bm P} = (\widehat{\bm P}_1, \ldots, \widehat{\bm P}_T)'$ associated with the $T \times p$ predictor matrix $\bm X = (\bm X_1, \ldots, \bm X_T)'$, for some appropriate choice of $K$. This may be defined as the solution of the constrained least squares problem \[ ( \widehat{\bm B} , \widehat{\bm P} ) = \arg\min_{ \substack{\bm B \in \mathbb R^{p \times K} \\ \bm P \in \mathbb R^{T \times K}} } \| \bm X - \bm P \bm B' \|_F^2 ~~\text{s.t.}~{1\over T} \bm P' \bm P = \bm I_K, {1 \over p} \bm B' \bm B \text{ is diagonal}, \] where $\| \cdot \|_F$ denotes the Frobenius norm. As is well known, $\widehat {\bm P}$ is given by $\sqrt{T}$ times the first $K$ eigenvectors of the matrix $\bm X \bm X'$. It is useful to remark here that the principal components allow us to express the vector of predictors $\bm X_t$ as a linear combination of the matrix of coefficients $\widehat{\bm B}$ with the $K$ principal components $\widehat{\bm P}_t$ plus a residual vector $\widehat{\bm u}$, that is
where $\widehat{\bm u}_t = \bm X_t - \widehat{\bm B} \widehat{\bm P}_t$. The second step of PCR consists in computing the $K\times 1$ least squares coefficients vector $\hat{\bm \vartheta}$ associated with the regression of the $T\times 1$ target variable vector $\bm Y = (Y_1, \ldots, Y_T)'$ on the principal components matrix $\widehat{\bm P}$. This is the solution to the least squares problem \[ \hat{\bm \vartheta} = \arg \min_{\bm \vartheta \in \mathbb R^K } \| \bm Y - \widehat{\bm P} {\bm \vartheta} \|^2_2 ~, \] where $\| \cdot \|_2$ denotes the Euclidean norm. It is straightforward to check that $\hat{\bm \vartheta} =\widehat{\bm P}' \bm Y / T$.
PCR may be interpreted as regularized empirical risk minimization.\footnote{ We remark that in this paper we use the expression “empirical risk minimization for principal component regression” only for simplicity. A more appropriate name for the procedure we study would be “regularized empirical risk minimization based on principal component analysis”.} Consider the class of prediction rules indexed by $\bm \theta \in \mathbb R^p$ given by \( f_{\bm \theta\,t} = \bm \theta ' \bm X_{t} \). Then PCR can be cast as the regularized empirical risk minimization problem given by
where \[ R_T(\bm \theta) = {1 \over T} \sum_{t=1}^{T} ( Y_{t} - f_{\bm \theta\,t} )^2 \] is the empirical risk and \[ \widehat{\bm V}_{R} = (\widehat {\bm v}_{K+1} , \ldots ,\widehat {\bm v}_p ) ~, \] where $\widehat {\bm v}_i$ is the eigenvector associated with the $i$-th largest eigenvalue of the sample covariance matrix of the predictors $\widehat{\bm \Sigma} = {\bm X'\bm X}/T $. The vector $\hat{\bm \theta}_{PCR}$ defined in (ref) is the solution to a least squares problem subject to a set of linear constraints. It is straightforward to verify that \( \hat f^{PCR}_t = \hat{\bm \theta}_{PCR}'\bm X_t = \hat{\bm \vartheta}'\widehat{\bm P}_t \), which implies that the forecasts produced by PCR may be equivalently expressed as linear forecasts based on constrained least squares.
One of the main objectives of learning theory is to obtain a bound on the predictive performance of the ERM relative to the optimal risk that can be achieved within the given class of prediction rules. We define the risk of a prediction rule as \( R( \bm \theta ) = \mathbb E \left[ ( Y_{t} - f_{\bm \theta\,t} )^2 \right] \). The optimal risk is defined as the risk of a prediction rule associated with a $\bm \theta^*$ such that \[ R( \bm \theta^* ) = \min_{\bm \theta \in \mathbb R^p} \mathbb E \left[ ( Y_{t} - f_{\bm \theta\,t} )^2 \right] ~. \] The conditional risk of PCR is used to measure predictive performance. This is given by
where $(Y_t,\bm X_t)'$ in (ref) denotes an element of the process assumed to be drawn independently of $\mathcal D$. The performance measure in (ref) can be interpreted as the risk of the ERM obtained from the “training” sample $\mathcal D$ over the “validation” observation $(Y_t,\bm X_t)'$. Then our aim is to find a pair $(B_T(p,K),\delta_T)$ such that $B_T(p,K) \rightarrow 0$ and $\delta_T\rightarrow 0$ as $T \rightarrow \infty$ for which
holds with probability at least $1 - \delta_T$ for any $T$ sufficiently large. The inequality in (ref) is commonly referred to as an oracle inequality. Oracle inequalities such as (ref) provide non-asymptotic guarantees on the performance of the ERM and imply that the ERM asymptotically performs as well as the best linear predictor. We remark that the performance measure in (ref) allows us to keep our analysis close to the bulk of the contributions in the learning theory literature (which typically focus on the analysis of i.i.d. data) and facilitates comparisons. We remark that Brownlees:Gudmundsson:2021 and Brownlees:LlorensTerrazas:2021 consider alternative performance measures such as the conditional out-of-sample average risk of the ERM, which has a more attractive interpretation for time series applications. It turns out that these alternative measures lead to essentially the same theoretical analysis at the expense of introducing additional notation. We therefore focus on the performance measure defined in (ref) for clarity.
This paper is related to various strands of the literature. First, the vast literature on approximate factor models, principal component analysis and spiked covariance models, which includes Fan2017, Donoho_2018, Forni:Hallin:Lippi:Reichlin:2000,Forni:Hallin:Lippi:Reichlin:2005, Bai:Ng:2006,Bai:Ng:2019, Bai_2012, Bai_2016, Lam2011, Fan2016, Gon_alves_2020, Barigozzi2020, Su2017 and Fan:Masini:Medeiors:2024. In particular, this work is close to the important contribution of Fan:Masini:Medeiors:2024 that studies the properties of a large class of high-dimensional models, which includes factor models, and establishes results on the predictive performance of such a class. Second, it is related to the literature on the small-ball method, which includes LecueMendelson:2016, LecueMendelson:2017, Mendelson:2018 and LecueMendelson:2018. Third, the literature on empirical risk minimization for linear regression, which includes BirgeMassart:1998 and Tsybakov:2003 among others. In particular, our contribution is close to the subset of the literature that deals with dependent data, as in JiangTanner:2010,Brownlees:Gudmundsson:2021.
The rest of the paper is structured as follows. Section (ref) introduces additional basic notation, assumptions and preliminary results. Section (ref) contains the main result of the paper and its proof is outlined in Section (ref). Concluding remarks follow in Section (ref). All the remaining proofs are given in the appendix.
In this section we lay out the assumptions required for our analysis. Our assumptions may be interpreted as the union of standard conditions employed in the approximate factor model literature Bai:Ng:2002,Fan:Liao:Mincheva:2011 and conditions employed in the learning literature based on the small-ball method Mendelson:2018,LecueMendelson:2018,Brownlees:Gudmundsson:2021.
We introduce some basic notation. For a generic vector $\bm x \in \mathbb R^d$ we define $\| \bm x \|_{r}$ as $[ \sum_{i=1}^d |x_i|^r ]^{1/r}$ for $ 1 \leq r < \infty$ and $\max_{i=1,\ldots,d} |x_i| $ for $r=\infty$. For a generic random variable $X \in \mathbb R$ we define $\| X \|_{L_r}$ as $[\mathbb E ( |X|^r ) ]^{1/r}$ for $ 1 \leq r < \infty$ and $\inf \{ a : \mathbb P( |X|> a ) = 0 \}$ for $r=\infty$. For a positive semi-definite matrix $\mathbf M$ we use $\mathbf M^{ {1/2} }$ to denote the positive semi-definite square root matrix of $\mathbf M$ and $\mathbf M^{ -{1/2} }$ to denote the generalized inverse of $\mathbf M^{1/2}$. For a generic matrix $\mathbf M$ we use $\| \mathbf M \|_2$ to denote the spectral norm of $\mathbf M$.
We begin with a preliminary lemma that establishes the population analogue of the principal components decomposition of the prediction vector $\bm X_t$ introduced in (ref).
Parts $(i)$ of Lemma (ref) states that $\bm X_t$ can decomposed into a linear combination of the matrix of coefficients $\mathbf B$ with the random vector $\bm P_t$ plus the residual random vector $\bm u_t$, where $\bm P_t$ and $\bm u_t$ are orthogonal. We call $\bm P_t$ the population principal components and $\bm u_t$ the idiosyncratic component. Part $(ii)$ implies that the best linear predictor for $Y_t$ can be alternatively represented as a function of the population principal components $\bm P_t$ and the idiosyncratic component $\bm u_t$ with the coefficient vectors $\bm \vartheta^*$ and $\bm \gamma^*$. Note that since $\bm P_t$ and $\bm u_t$ are orthogonal, $\bm \vartheta^*$ may be interpreted as the best linear predictor based on the population principal components $\bm P_t$. We remark that, clearly, in the definitions of $\bm u_t$ and $\bm \gamma^*$ the matrix $\bm V_R (\bm V_R' \bm V_R)^{-1} \bm V_R'$ may be simplified to $\bm V_R \bm V_R'$ but we prefer to express it in this way to emphasize that this is a projection matrix. Last note that the decomposition established in part $(ii)$ holds for any vector in $\mathbb R^p$ but for our purposes it suffices to focus on the vector of coefficients of the best linear predictor. Also notice that the vector of coefficients of the best linear predictor may not be unique, but this raises no issues in our setup.
We lay out the assumptions of our analysis. We say that the $d$-dimensional random vector $\bm U$ is sub-Gaussian with parameters $C_m > 0$ if, for any $\varepsilon>0$, it holds that \[ \mathbb P\left( \sup_{\bm v : \| \bm v \|_2=1} | \bm v'\bm U | > \varepsilon \right) \leq \exp( -C_m \varepsilon^{2} ) ~. \] For a univariate random variable $U$ this is equivalent to \( \mathbb P( | U | > \varepsilon ) \leq \exp( -C_m \varepsilon^{2} ) \).
Part $(i)$ states that the tails of the data decay exponentially. More precisely it assumes that the prediction target $Y_t$, the prediction error of the best linear predictor based on the principal components $\bm P_t$ given by $Y_t - \bm P_t'\bm \vartheta^*$ and the standardized predictors $\bm Z_t$ have sub-Gaussian tails. We remark that such a condition is fairly standard in the analysis of large-dimensional factor models Fan:Liao:Mincheva:2011. We also remark that the sub-Gaussian condition may be replaced by a sub-Weibull condition at the expense of longer proofs WongTewari:2020. Part $(ii)$ is a regularity condition on the density of the standardized predictors $\bm Z_t$ that is required to establish upper bounds on the probability of a certain event associated with the vector of predictors $\bm X_t$ in one of the intermediate propositions of our analysis. The same condition is assumed in Brownlees:Gudmundsson:2021.
Let $\mathcal{F}_{-\infty}^t$ and $\mathcal{F}_{t+l}^{\infty}$ be the $\sigma$-algebras generated by $\lbrace (Y_s, \bm X_s')': -\infty \leq s \leq t\rbrace$ and $\lbrace (Y_s, \bm X_s')': t + l \leq s \leq \infty\rbrace$ respectively for some $t \in \mathbb Z$ and define the $\alpha$-mixing coefficients
(ref) states that the process $\{ (Y_t, \bm X_t')'\}$ has geometrically decaying strong mixing coefficients, which is a fairly standard assumption in the analysis of large dimensional time series models JiangTanner:2010,Fan:Liao:Mincheva:2011,Kock:Callot:2015.
(ref) states that the first $K$ eigenvalues of the covariance matrix $\bm \Sigma$ diverge as the cross-sectional dimension $p$ becomes large. The rate of divergence is determined by $\alpha$. We distinguish between two regimes that depend on the value of this constant. When $\alpha=1$ we say that we are in the strong signal regime, which is analogous to (classic) factor models with pervasive factors Stock:Watson:2002,Bai:Ng:2002,Bai:2003,Fan:Liao:Mincheva:2013. When $\alpha \in (1/2,1)$ we say that we are in the weak signal regime, which is analogous to weak factor models Onatski:2012,Bai:Ng:2021. We remark that this assumption is weaker than Fan:Liao:Mincheva:2013, which only allows for strong signals. We also point out that $K$ here is assumed to be known and that there is a large literature devoted to the estimation of this quantity Bai:Ng:2002,Amengual2007,Onatski_2010,Lam:Yao:2012,Ahn:Horenstein:2013,Yu_2019. Last, it is important to emphasize that the assumption allows for the non-diverging eigenvalues of $\bm \Sigma$ to be zero, so our framework allows $\bm \Sigma$ to be singular.
(ref) states that the number of predictors and the number of principal components are allowed to increase as a function of the sample size $T$. The rate of growth of these quantities depends on the rate of decay of the strong mixing coefficients (as measured by $r_\alpha$) and the strength of the signal (as measured by $\alpha$). The less dependence and the stronger the signal, the larger the numbers of allowed predictors and principal components. The assumption allows the number of predictors to be larger than the sample size $T$ whereas the number of principal components is at most $T^{1/3}$ (up to a proportionality constant). It is important to emphasize that when the signal is weaker, the maximum rate of growth of the number of predictors is smaller. The condition on the maximum rate of growth of the number of predictors $\overline{r}_p$ is analogous to condition (9) in Uematsu2022, who study the properties of factor models in the weak signal regime.
(ref) is the so-called small-ball assumption, and it is stated here as it is formulated in LecueMendelson:2016. This assumption can be interpreted as an identification condition. If we define $\bm \delta=(\bm \theta_1-\bm \theta_2)$ then the condition is equivalent to $\mathbb P \left( | \bm \delta' \bm X_t | \geq \kappa_1 \| \bm \delta' \bm X_t \|_{L_2} \right) \geq \kappa_2 $, which can be seen as requiring the random variable $\bm \delta' \bm X_t$ to not have excessive mass in a neighbourhood around zero. We remark that the constants $\kappa_1$ and $\kappa_2$ measure the strength of the identification in the sense that the larger the value of these constants the stronger the identification condition. We also remark that Brownlees:Gudmundsson:2021 discuss conditions that imply (ref).
We now state our main result on the performance of empirical risk minimization.
The theorem establishes a regret bound on the excess risk of PCR relative to the best linear predictor that can be obtained on the basis of the predictors $\bm X_t$. The gap is made up of two terms. The first can interpreted as the approximation error of PCR and the second as the estimation error. The approximation error measures the gap between the performance of the best linear predictor based on the population principal components $\bm P_t$ and the best linear predictor based on the predictors $\bm X_t$. The estimation error measures the gap between the performance of PCR relative to the best linear predictor based on the population principal components $\bm P_t$.
A number of additional remarks on Theorem (ref) are in order. First, it is insightful to provide an alternative representation of the approximation error of PCR. This may be equivalently expressed as \[ (\bm \theta^*)' \bm V_R \bm \Lambda_R \bm V_R' \bm \theta^* = \| \bm \Lambda^{1/2}_R \bm V_R' \bm \theta^* \|^2_2 = \| \bm \Lambda^{1/2}_R \bm V_R' \bm V_R ( \bm V_R' \bm V_R)^{-1} \bm V_R' \bm \theta^* \|^2_2 = (\bm \gamma^*)' \bm V_R \bm \Lambda_R \bm V_R' \bm \gamma^* ~. \] This highlights that the approximation error of PCR is small when the projection of the best linear predictor $\bm \theta^*$ on the subspace spanned by the population eigenvectors $\bm v_{K+1}, \ldots, \bm v_p$ is small. Differently put, if the contribution of the idiosyncratic component vector $\bm u_t$ is negligible then PCR has a negligible approximation error. Clearly, when $\bm \gamma^* = \bm 0$ the best linear predictor based on $\bm X_t$ coincides with the best linear predictor based on $\bm P_t$ and the approximation error is zero.
Second, it is interesting to provide some comments on the behaviour of the approximation error. Under the rate conditions of (ref) the approximation error is asymptotically negligible as $T \rightarrow \infty$. We remark that the condition ${p /( p^{\alpha} T) } \rightarrow 0$ as $T\rightarrow \infty$ is also required by Bai:Ng:2021 in the analysis of approximate weak factor models. The approximation error is also influenced by the degree of persistence in the data, as measured by $r_\alpha$, and in particular the more dependent the data are the slower the convergence of the approximation error to zero. We remark that the results of Bai:Ng:2021, among others, do not depend on the degree of persistence of the data. This is due to the fact that their analysis relies on higher level conditions that imply sharp rates of convergence of certain key estimators in the analysis. It is also interesting to note that the approximation error is made up of three terms that can be easily associated with the different estimation problems embedded in PCR. The first two terms capture the estimation error of the principal components whereas the third one can be interpreted as the estimation error the principal component regression if the population principal components were observed. Last, an important question concerning the approximation error is whether the rate obtained for it in the theorem is optimal. We provide insights on this question below in Section (ref).
A natural question that arises upon inspection of Theorem (ref) is whether the learning rate for the approximation error is, in some appropriate sense, optimal. We compare the learning rate established by the theorem with the optimal learning rate that could be achieved if the population principal components were observed. In this case it is well known that the optimal rate is of the order $K/T$ Tsybakov:2003, which is achieved by the least squares estimator based on the population principal components. In this section we provide conditions under which the optimal rate is achieved up to a logarithmic factor. We call this the near-optimal rate. For ease of exposition we assume that the approximation error is zero throughout this section.
For a given choice of the signal strength $\alpha$, degree of dependence $r_\alpha$ and the rate of growth of the number of principal components $r_K$, it is straightforward to verify that the near-optimal rate can be recovered in Theorem (ref) provided that \[ {1-r_K \over 2\alpha-1} < {1+r_K \over 2-2\alpha+2/r_\alpha } ~, \] and that the rate of growth of the number of parameters allowed is such a scenario is \[ r_p \in \left[ {1-r_K \over 2\alpha-1} , {1+r_K \over 2-2\alpha+2/r_\alpha}\right] ~. \] In particular we note that the near-optimal rate can be achieved in the both the strong signal ($\alpha=1$), and the weak signal ($\alpha<1$) cases, provided that $\alpha > 2/3$. The larger the values of $\alpha$, $r_\alpha$ and $r_K$, the larger the range of admissible growth rates for the number of predictors $r_p$.
It is interesting to provide concrete examples of these conditions. In the presence of a strong signal ($\alpha=1$), a fixed number of principal components ($r_K=0$) and independent data ($r_{\alpha} = \infty$)\footnote{The proofs of this manuscript assume that $r_\alpha$ is finite.} we obtain that
when $r_p \in [1,\infty)$. In the presence of a weak signal ($\alpha<1$), a fixed number of principal components ($r_K=0$) and independent data ($r_{\alpha} = \infty$) we obtain that
when $r_p \in [ 1/(2\alpha-1) , 1/(2-2\alpha) ] $. In particular when $\alpha = 3/4$ the rate of growth of the number of parameters required to achieve near-optimal performance is $r_p=2$.
We introduce some additional notation. In this section and in the appendices we use the subscript $s$ to denote the index of an observation in the training sample $\mathcal D$ and the subscript $t$ to denote the index of a validation observation. Also, for some random variable $X$ we use $ \| X \|_{L_2}$ to denote $\sqrt{ \mathbb E( X^2 | \mathcal D ) }$.
The proof of Theorem (ref) begins with a decomposition of an upper bound of the excess risk of the empirical risk minimizer. Define the approximate rotation matrix $\bm H = \widehat{\bm \Lambda}_K ^{-1/2} \widehat{\bm V}_K' \bm V_K \bm \Lambda_K ^{1/2}$ Bai:Ng:2002 and the infeasible PCR estimator \( \tilde{\bm \vartheta} = \arg \min_{\bm \vartheta \in \mathbb R^K } {1 \over T} \sum_{t=1}^{T} ( Y_{t} - \bm P_t'\bm \vartheta )^2 \). The following lemma provides a useful bound for the excess risk. We remark that the lemma only requires stationarity and finite variance.
Lemma (ref) implies that in order to control the excess risk of PCR it suffices to provide appropriate bounds on the four terms on the right hand side of (ref). The following four propositions establish such bounds.
The first two propositions control the terms $\mathbb E( \| \widehat {\bm P}_t - \bm H {\bm P}_t \|^2_2 | \mathcal D )$ and $\| \tilde{\bm \vartheta} - \bm H ' \hat{\bm \vartheta} \|^2_2 $. We remark that these two terms capture the risk of PCR that is due to the estimation of the principal components. The proof of the propositions is based on standard arguments from the approximate factor model literature Bai:Ng:2002,Fan:Liao:Mincheva:2011.
The next proposition controls the term $\| \bm \vartheta^* - \tilde{\bm \vartheta} \|^2_{2}$. We remark that this term may be interpreted as the risk of the least squares estimator of PCR based on the population principal components. The proof of the proposition is based on the so-called small-ball method, which is the same proof strategy used in Brownlees:Gudmundsson:2021.
The following proposition controls the error $\| \bm u_t'\bm \gamma^* \|^2_{L_2}$, which can be interpreted as the approximation error of PCR.
The claim of Theorem (ref) follows from Propositions (ref) to (ref) together with the implication rule, the union bound and the fact that the sub-Gaussian assumption on $Y_t$ ((ref)) implies that for each $\eta>0$ there is a $C>0$ such that \[ \mathbb P\left( \max_{1 \leq s \leq T}\{ Y^2_s \} \leq C \log(T) \right) \leq 1-T^{-\eta} ~. \]
This paper establishes predictive performance guarantees for principal component regression. Our analysis has a number of highlights. First, the analysis we carry out is nonparametric, in the sense that the relation between the prediction target and the predictors is not specified and, in particular, we do not assume that the prediction target is generated by a factor model. Second, our framework considers both the cases in which the largest eigenvalues of the covariance matrix of the predictors diverge linearly in the number of predictors (strong signal regime) or sublinearly (weak signal regime). A highlight of our results is that we show that, under appropriate conditions, PCR achieves optimal performance (up to a logarithmic factor) in both the strong signal and weak signal regimes.