Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
42,338 characters · 6 sections · 68 citation commands
Performance of Empirical Risk Minimization for Linear Regression with Dependent Data
\symbolfootnote[0]{\\ $^{\dag}$ Department of Economics and Business, Universitat Pompeu Fabra and Barcelona GSE;\\ e-mail: [email removed].\\ $^{\ddag}$ Department of Economics and Business Economics, Aarhus University; \\ e-mail: [email removed].\\ $^*$ Corresponding author. \\ We have benefited from discussions with Liudas Giraitis, Emmanuel Guerre, Petra Laketa, Gabor Lugosi, Stanislav Nagy, Jordi Llorens-Terrazas, Yaping Wang and Geert Mesters as well as seminar participants at the Granger Center, Nottingham University; School of Economics and Finance, Queen Mary University of London. Christian Brownlees acknowledges support from the Spanish Ministry of Science and Technology (Grant MTM2012-37195) and the Spanish Ministry of Economy and Competitiveness through the Severo Ochoa Programme for Centres of Excellence in R&D (SEV-2011-0075). Gu\dh mundur Stef\'an Gu\dh mundsson acknowledges financial support from the Danish National Research Foundation (DNRF Chair grant number DNRF154). }
\doublespacing
Let $ \mathcal D = \{ (Y_t, \bm X_t')' \}_{t=1}^T $ be a sequence of dependent random vectors taking values in $\mathcal Y \times \mathcal X$ with $\mathcal Y \subset \mathbb R$ and $\mathcal X \subset \mathbb R^p$. The $p$-dimensional vector $\bm X_t=(X_{1\,t},\ldots,X_{p\,t})'$ is used to predict the variable $Y_t$ through the class of linear forecasts given by
where $(\theta_1,\ldots,\theta_p)' = \bm \theta \in \mathbb R^p$. As is customary in learning theory, the relation between the regressand $Y_t$ and the regressors $X_{1\,t},\ldots,X_{p\,t}$ is not specified, and (ref) should be interpreted as a class of prediction rules indexed by $\bm \theta \in \mathbb R^p$.
A prediction rule is to be chosen from the data. The precision of a prediction rule is measured by its average risk defined as \[ R( \bm \theta ) = \mathbb E \left[ {1 \over T} \sum_{t=1}^{T} ( Y_{t} - f_{\bm \theta\,t} )^2 \right] ~. \] Thus, a natural strategy for choosing a prediction rule from the data consists in minimizing the empirical risk. The empirical risk minimizer (ERM) is defined as
If more than one prediction rule achieves the minimum we may pick one arbitrarily. Clearly, the ERM in (ref) corresponds to the classic least squares estimator. We sometimes denote $\hat{ \bm \theta}$ as $\hat{ \bm \theta}(\mathcal D)$ to emphasize that the ERM is a function of the data $\mathcal D$. The problem we have described so far is known as linear regression in statistics and econometrics whereas in learning theory it is known as linear aggregation Nemirovski:2000.
The accuracy of the ERM is measured by its conditional average risk defined as
where $\hat f_t = \hat \theta_1 X_{1\,t} + \ldots + \hat \theta_p X_{p\,t}$ and $\mathcal D'$ denotes an independent copy of the data $ \mathcal D $. The performance measure in (ref) can be interpreted as the risk of the ERM obtained from the “training data” $\mathcal D'$ over the “validation data” $\mathcal D$. This performance measure allows us to keep our analysis close to the bulk of contributions in the learning theory literature (which typically focus on the analysis of i.i.d. data) and facilitates comparisons. We also consider as an alternative accuracy measure the conditional out-of-sample average risk of the ERM, which is more attractive for time series applications. The alternative measure leads a to similar result at the expense of introducing additional notation.
The main objective of this paper is to obtain a bound on the performance of the ERM relative to the optimal risk that can be achieved within the given class of prediction rules. We aim to establish a bound $B_T(p)$ such that $B_T(p) \rightarrow 0$ as $T \rightarrow \infty$ for which
holds, with high probability, for all (sufficiently large) $T$. The inequality in (ref) is commonly referred to as an oracle inequality. Oracle inequalities such as (ref) provide non-asymptotic guarantees on the performance of the ERM. The inequality in (ref) implies that empirical risk minimization achieves asymptotically the best performance that is possible to attain in the class. We emphasize that in this paper we study the performance of the ERM for large-dimensional linear regression, meaning that in our analysis we assume that the number of predictors $p$ is not negligible relative to $T$ (in a sense to be spelled out precisely below). Establishing bounds on the performance of the ERM is a classic problem in learning theory. There is a fairly extensive literature that has studied this problem in the i.i.d. setting AudibertCantoni:2010. The literature conveys that the best possible rate for $B_T(p)$ is of the order $p/T$, which is referred to as the optimal rate of linear aggregation Tsybakov:2003.
The main contribution of this paper consists in establishing oracle inequalities for the ERM when the data are dependent and heavy-tailed. Our analysis covers both the cases of identically and heterogeneously distributed observations (using the jargon of White:2001). In particular, our main results establish that the ERM achieves the optimal rate of linear aggregation (up to a $\log(T)$ factor) in a dependent data setting. Our analysis highlights a trade-off between the dependence and moment properties of the data on the one hand, and the number of predictors on the other. In particular we show that the higher the dependence and the lower the number of moments of the data, the lower the maximum rate of growth allowed for the number of predictors. We emphasize that our analysis is nonparametric, in the sense that the relationship between the regressand and the regressors is assumed to be unknown. Lastly, we remark that the performance bound we recover depends transparently on constants that are straightforward to interpret.
Four remarks are in order before we proceed. {First, this work establishes prediction performance guarantees for empirical risk minimization/least squares estimation with dependent data in a large-dimensional setting. These results allow us to determine under which conditions least squares estimation is a reliable estimation strategy in a large-dimensional setup and to appraise more precisely the gains of estimation methodologies specifically designed for such a setup. It is important to acknowledge that estimation methodologies designed for large-dimensional settings (for instance, LASSO) typically achieve substantially better performance guarantees than the ones obtained here. However, these gains come at the expense of additional assumptions. In fact, the performance guarantees obtained here are optimal (up to a logarithmic factor) Tsybakov:2003. }
{ Second, this paper has a number of connections with the nonparametric literature and, in particular, with nonparametric series methods Stone:1985,Andrews:1991,Newey:1997,Chen:Shen:1998,Chen:2006,Tsybakov:2014,Belloni:2015. Among these papers we remark that Chen:Shen:1998 is the only one that considers a non i.i.d. data setup. Let $ \{ (Y_t, \bm W_t')' \}_{t=1}^T $ be a strictly stationary sequence of random vectors in $\mathcal Y \times \mathcal W \subset \mathbb R \times \mathbb R^d$. Then our framework subsumes the problem of estimating the conditional mean of $Y_t$ given $\bm W_t$ on the basis of the approximation given by
where $\{ f_i \}$ with $f_i : \mathcal W \rightarrow \mathbb R$ is a collection of functions (e.g. B-splines) called a dictionary. We emphasize that, in some sense, our framework is more general since our focus lies on the estimation of the optimal linear prediction rule rather than the conditional mean. }
Third, the literature on empirical risk minimization and oracle inequalities for dependent data has been rapidly developing in recent years. Notable contributions in this area include the works of JiangTanner:2010, Fan:Liao:Mincheva:2011, CanerKnight:2013, LiaoPhillips:2015 and MiaoPhillipsSu:2020. We remark that one of the challenges of this literature is that it is not straightforward to apply the theoretical machinery used in learning theory in a dependent data setting. In fact, as forcefully argued in Mendelson:2015, several of the standard results on empirical risk minimization used in learning theory assume i.i.d. bounded data and cannot be extended beyond this setup. In this work we rely on a proof strategy based on the so-called small-ball method developed by Shahar Mendelson and Guillaume Lecu\'e Mendelson:2015,LecueMendelson:2016. The small-ball method allows us to establish sharp bounds on the performance of the ERM under fairly weak moment and dependence assumptions.
Fourth, our analysis aims to provide large-dimensional analogues of some of the classic results of White:2001 for fixed-dimensional linear regression with dependent data. We shall point out the differences between those results and the ones established here.
This paper is related to various strands of the literature. First, it is related to the literature on empirical risk minimization for linear aggregation, which includes BirgeMassart:1998, BuneaTsybakovWegkamp:2007, AudibertCantoni:2011 and LecueMendelson:2016. Second, it is related to the literature on empirical risk minimization for heavy-tailed data, which includes AudibertCantoni:2011 and BrownleesJolyLugosi:2015. Third, it is related to the literature on empirical risk minimization for dependent data. In particular this contribution is close to JiangTanner:2010. Fourth, this paper is related to the vast literature on nonparametric estimation and nonparametric series methods, which includes Chen:2006 and Belloni:2015. Li:Racine:2006 contains a number of important results and references to this literature. Fifth, it is related to the literature on the small-ball method, which includes Mendelson:2018, LecueMendelson:2017 and LecueMendelson:2018. Sixth, it is related to the vast literature on machine learning and large-dimensional modeling, which includes (in econometrics) Kock:Callot:2015, Medeiros:Mendes:2016, Garcia:2017 and Babii:Ghysels:Striaukas:2021. Hastie:2001 and wainwright_2019 contain a number of important results and references to this literature.
The rest of the paper is structured as follows. Section (ref) contains preliminaries, additional notation and assumptions. Section (ref) contains an oracle inequality for linear regression with heterogeneously distributed observations. Section (ref) contains an analogous result for identically distributed observations. Section (ref) contains extensions of the baseline results. Concluding remarks follow in Section (ref). All proofs are in the Appendix.
We introduce the notation used in the remainder of the paper. For a generic vector $\bm x \in \mathbb R^d$ we define $\| \bm x \|_{r}$ as $[ \sum_{i=1}^d |x_i|^r ]^{1/r}$ for $ 1 \leq r < \infty$ and $\max_{i=1,\ldots,d} |x_i| $ for $r=\infty$. For a generic random variable $X \in \mathbb R$ we define $\| X \|_{L_r}$ as $[\mathbb E ( |X|^r ) ]^{1/r}$ for $ 1 \leq r < \infty$ and $\inf \{ a : \mathbb P( |X|> a ) = 0 \}$ for $r=\infty$. For a positive semi-definite matrix $\mathbf M$ we use $\mathbf M^{ {1 \over 2} }$ to denote the positive semi-definite square root matrix of $\mathbf M$ and $\mathbf M^{ -{1 \over 2} }$ to denote the generalized-inverse of $\mathbf M^{1\over 2}$.
In this section we establish a preliminary result and introduce the main assumptions required in our analysis. All results and assumptions are stated for the case of heterogeneously distributed observations. Clearly, these simplify in a straightforward manner if the observations are identically distributed.
We begin by establishing the existence of the optimal prediction rule, that is the oracle. Lemma (ref) states that there exists an optimal $\bm \theta^*$ that satisfies a Pythagorean-type identity. We remark that the assumptions of Lemma (ref) are fairly weak and, in particular, weaker than what we require for the analysis of the ERM.
Next, we lay out the assumptions we require to establish the properties of the ERM.
Assumption (ref) states that the regressand, predictors, and the product of the predictors and the forecast error of the optimal prediction rule have a number of moments strictly larger than two. The assumption also states that the $r_m$-th moments are bounded by a constant $K_m$ uniformly in $t$. A few comments are in order. First, this moment assumption is formulated as in White:2001 in the analysis of linear regression with heterogeneous data. Alternatively, we may state this assumption for the forecast error of the optimal prediction rule and the predictors separately and require at least four moments to exist and to be uniformly bounded. {Second, we assume $K_m \geq 1$ to obtain simpler expressions of some of the constants that appear in our analysis. Note that this is without loss of generality. } Lastly, we emphasize that this assumption is weaker than what is assumed in a number of contributions on oracle inequalities for dependent data for large dimensional models such as JiangTanner:2010, Fan:Liao:Mincheva:2011 and Kock:Callot:2015 which assume that all moments exist. We remark that assuming that the moments are uniformly bounded is fairly standard in the analysis of regression models with heterogeneous dependent data and that requiring more than two moments to exist is also required to establish consistency of the least squares estimator for fixed-dimensional linear regression White:2001.
{Assumption (ref) states that the sequence $\{ (Y_t, \bm X_t')'\}_{t=1}^T$ is strongly mixing with geometrically decaying mixing coefficients. The definition of the mixing coefficients is as in White:2001 and does not hinge on the data generating process being stationarity. See also SuWhite:2010 for the analysis of $\alpha$-mixing processes that are not required to be stationary. Note that while this is a stronger assumption than what is required by classical results for consistency and asymptotic normality for the (finite-dimensional) linear regression model that rely on polynomial $\alpha$-mixing White:2001, geometric $\alpha$-mixing is commonly used in the analysis of large dimensional time series models JiangTanner:2010,Fan:Liao:Mincheva:2011,Kock:Callot:2015. Moreover, geometric $\alpha$-mixing is satisfied by many commonly encountered processes such as ARMA and GARCH Meitz:Saikkonen:2008.}
Assumption (ref) states that the number of predictors is a function of $T$. This assumption allows the number of predictors to be constant or to grow sublinearly in $T$. Importantly, the bound on the rate of growth of the number of predictors $p$ depends on the number of moments and the amount of dependence of the data. The more moments and the less dependence, the higher the maximum rate of growth of the number of predictors. If the data have at least four moments, then the number of predictors is only constrained by the amount of dependence in the data.
Assumption (ref) states that the eigenvalues of the covariance matrix of the predictors are bounded from above and bounded away from zero uniformly in $t$. The assumption that the smallest eigenvalue is bounded away from zero is fairly standard Newey:1997. Notice that $\bm \theta^*$ is unique when (ref)$(i)$ holds, by Lemma (ref)$(iii)$. Assuming that the largest eigenvalue of the covariance matrix of the predictors is bounded above uniformly in $t$ is more restrictive. We remark that, as we shall see in detail below, under the additional assumption of identically distributed observations these constraints can be relaxed. In what follows we shall also use the constant \( K_{\bm \Sigma} = { \overline{\lambda} / \underline{\lambda} } \), which is an upper bound on the condition number of the matrices $\{ \bm \Sigma_t \}_{t=1}^T$ and measures the maximum degree of collinearity between the predictors.
Assumption (ref) is required to establish upper bounds on the probability of a certain event associated with the vector of predictors $\bm X_t$ in one of the intermediate propositions of our analysis. The probability of this event boils down to a multiple integral that can be expressed using $n$-spherical coordinates. The spherical distribution bound in (ref) makes it easy to compute such an integral after the $n$-spherical coordinates transformation. We conjecture that the assumption could be relaxed, however this would be at the expense of more tedious computations. That being said, the family of spherical distributions is fairly large and includes the appropriately standardized versions of the multivariate Gaussian, Student $t$, Cauchy and uniform\footnote{To be precise, the multivariate uniform distribution over the sphere.} distributions.\footnote{For more details on the class of spherical distributions we refer to Fang:Kotz:Ng:1990.} Moreover, finite mixtures of spherical distributions are also spherical. Assumption (ref) may be interpreted as a generalization of the bounded density assumption typically encountered in the nonparametric literature Newey:1997,Li:Racine:2006,Hansen:2008. Bounded density assumptions are also formulated in JiangTanner:2010 in the analysis of empirical risk minimization for time series data with bounded support. Last, we remark that this assumption allows for weaker moment conditions than what is imposed by Assumption (ref).
Assumption (ref) is the so-called small-ball assumption, and it is stated here as it is formulated in LecueMendelson:2016. This assumption can be interpreted as an identification condition. If we define $\bm v=(\bm \theta_1-\bm \theta_2)$ then the condition is equivalent to \( \mathbb P \left( | \bm v' \bm X | \geq \kappa_1 \| \bm v' \bm X \|_{L_2} \right) \geq \kappa_2 \), which can be seen as requiring that the random variable $\bm v' \bm X$ does not have excessive mass in a neighbourhood around zero. We remark that the constants $\kappa_1$ and $\kappa_2$ measure the strength of the identification in the sense that the larger the value of these constants the stronger the identification condition is. In Section (ref) we establish alternative identification assumptions that in turn imply Assumption (ref).
The ERM performance bound that we derive in this section depends on a constant related to the variance of the gradient of the empirical risk evaluated at the optimal prediction rule (after an appropriate rescaling), that is $\operatorname{Var} \left( {1\over \sqrt{T} } \sum_{t=1}^T (Y_t - f^*_t) \bm X_t \right)$. As is well known, in the standard large sample analysis of linear regression the asymptotic variance of the least squares estimator is typically expressed as a function of the limit of this quantity White:2001. In our analysis, the ERM performance depends on an upper bound on the diagonal elements of this quantity that is given by \[ K_{\sigma^2} = K_m^2 \left( 1 + 128 {r_m \over r_m -2} \sum_{l=1}^\infty \alpha(l)^{1-{2\over r_m} } \right) ~. \] It is possible to make substantially smaller choices of this constant if we make simplifying assumptions on the setup of our analysis. We explore this in more detail in Section (ref).
We can now state the main result of this section.
The theorem establishes that the ERM for large-dimensional linear regression with heterogeneous dependent data achieves the optimal rate of linear aggregation (up to a $\log (T)$ factor). {We remark that (ref) implies that $( p \log (T) )/ T \rightarrow 0$ as $T \rightarrow \infty$, which makes the inequality in the theorem an oracle inequality.} Note that the bound on the performance of the ERM is proportional to quantities that are associated with a larger asymptotic variability of the least squares estimator. {We remark that Theorem (ref) may be seen as a non-asymptotic version of classic asymptotic results in the series estimation literature, which establish optimality of the nonparametric least squares estimator. In fact, the convergence rate of $p/T$ (up to a $\log(T)$ factor) is the same as the rate obtained (for instance) in Belloni:2015.}
It is interesting to compare Theorem (ref) with an analogous result for i.i.d. data. The following result LecueMendelson:2016 is taken as benchmark.
\begin{thm-nonumber} Consider the linear regression model \[ Y_t = \bm X_t' \bm \theta^* + \epsilon_t, \qquad t=1,\ldots,T ~, \] where $\{ \bm X_t \}$ and $\{ \epsilon_t \}$ are sequences of i.i.d. random variables with $\mathbb E( \epsilon_t ) = 0$, $\operatorname{Var}( \epsilon_t ) = \sigma^2$, and $\epsilon_t$ is independent of $\bm X_t$. Assume that there are constants $\kappa_1$ and $\kappa_2$ such that \[ \mathbb P \left( | f_{\bm \theta_1\,t} - f_{\bm \theta_2\,t} | \geq \kappa_1 \| f_{\bm \theta_1\,t} - f_{\bm \theta_2\,t} \|_{L_2} \right) \geq \kappa_2 ~, \] for all $\bm \theta \in \mathbb R^p$. Then, for all $T > (400)^2 p / \kappa_2^2$ and $x>0$ we have that the empirical risk minimizer defined in (ref) satisfies \[ R( \hat{\bm \theta} ) \leq R( \bm \theta^* ) + { \sigma^2 } \left({ 16 \over \kappa_1^2 \kappa_2 }\right)^2 { {p \over T } } \, x ~, \] with probability at least $1-\exp( - \kappa_2 T/4 )- (1/x)$. \end{thm-nonumber}
As is immediate to see, we recover an analogous bound to what is established in LecueMendelson:2016. The constant that appears in our risk bound in (ref) is much larger than the one in this benchmark result. However, we remark that below we obtain a more favourable bound by simplifying the setup of our analysis. Also, we remark that the result above relies on assuming that the “true model” exists. LecueMendelson:2016 also have results that do not depend on such an assumption but rely on stronger assumptions on the prediction errors of the optimal forecast.
We conclude this section with a sketch of the proof. This is an elegant argument based on LecueMendelson:2016. Define the empirical risk differential for $\bm \theta \in \mathbb R^p$ as
The proof is based on showing that if the condition
holds, then we have that
with high probability. This, in turn, implies that for any $\bm \theta$ that satisfies (ref) we have $\widehat{\mathcal L}_{{\bm \theta}} > 0$. Since the empirical risk minimizer $\hat{\bm \theta}$ must satisfy $\widehat{\mathcal L}_{\hat {\bm \theta}} \leq 0$ then, conditional on the same events, we must have that \[ {1 \over T} \sum_{t=1}^T \|f^*_t - \hat f_{t} \|_{L_2} \leq { 48 K^{1\over 2}_{\sigma^2} K_{\bm \Sigma} \over \underline{\lambda}\kappa_1^2 \kappa_2 } \sqrt{ {p \log (T) \over T}} ~, \] which, in turn, implies that \[ R(\hat {\bm\theta}) - R(\bm\theta^*) = {1 \over T} \sum_{t=1}^T \| f^*_{t} - \hat f_{t} \|^2_{L_2} \leq { K_{\sigma^2} } \, { K^2_{\bm \Sigma} \over \underline{\lambda} } \,\left({ 48 \over \kappa_1^2 \kappa_2 }\right)^2 \, { {p \log (T) \over T } } ~, \] by an application of Lemma (ref).
The following two propositions are key in establishing that the inequality in (ref) holds with high probability and thus to determine the risk bound in Theorem (ref).
Both propositions exploit a Bernstein-type inequality for $\alpha$-mixing sequences from Liebscher:1996 (based on the famous covariance inequality of Rio:1995). Proposition (ref) uses a covering argument similar to the one used in JiangTanner:2010 and Hansen:2008. Proposition (ref) relies on the Bernstein-type inequality and a classic truncation trick used in, for instance, Hansen:2008. See also Dendramis:Giraitis:Kapetanios:2021 and Babii:Ghysels:Striaukas:2021 for recent developments on concentration inequalities for dependent data with applications to large-dimensional estimation problems.
The constant term in the bound of Theorem (ref) can be improved by assuming stationarity.
We remark that Assumptions (ref) and (ref) imply that the data are ergodic.
In the stationary case it is convenient to state the moment assumption differently.
The difference between (ref) and (ref) is that the former assumption bounds the $r_m$-th moment of $(Y_{t}-f^*_t) X_{i\,t}$ whereas the latter bounds the $r_m$-th moment of $(Y_{t}-f^*_t) Z_{i\,t}$.
In the stationary case the assumption on the eigenvalues of $\bm \Sigma$, Assumption (ref), can be dropped. In fact, as we show in the proof of Theorem (ref), (ref) implies that $\lambda_{\max} \left( \bm \Sigma \right) \leq K_m^2 p $. This allows the set of predictors to be generated by a factor model Forni:Hallin:Lippi:Reichlin:2000,Stock:Watson:2002,Bai:Ng:2002,Onatski:2012. Additionally, $\lambda_{\min}(\bm \Sigma)$ is allowed to be zero. This allows the set of predictors to contain some predictors that are perfectly correlated.
Before stating the main result of this section we introduce a new constant \[ K'_{\sigma^2} = K_m^2 \left( 1 + 32 {r_m \over r_m -2} \sum_{l=1}^\infty \alpha(l)^{1-{2\over r_m} } \right) ~. \] This constant plays the same role as $K_{\sigma^2}$ and can be interpreted as an upper bound on the diagonal elements of $\operatorname{Var}\left( {1\over \sqrt{T} } \sum_{t=1}^T (Y_t - f^*_t) \bm Z_t \right)$. Note that $K'_{\sigma^2} \leq K_{\sigma^2}$.
We can now state the main result of this section.
Note that the bound in Theorem (ref) does not depend on the eigenvalues of $\bm \Sigma$. Inspection of the proof shows that if $\{ (Y_t,\bm X_t')' \}$ is an i.i.d. sequence, $Y_t-f^*_t$ is independent of $\bm X_t$ and $\operatorname{Var}( Y_t - f^*_t ) = \sigma^2$ the bound in Theorem (ref) becomes \[ R( \bm \theta^* ) + \sigma^2 \left({ 48 \over \kappa_1^2 \kappa_2 }\right)^2 {p \log (T) \over T} ~, \] which is close to the bound established in LecueMendelson:2016.
\paragraph{Alternative Risk Definition.} It is important to emphasize that the performance measure defined in (ref) is the average risk of the prediction rule over the data $\mathcal D$ when $\hat {\bm\theta}$ is estimated using an independent copy of the data $\mathcal D'$. This measure may have limited appeal for time series applications since a forecaster typically does not have access to an independent copy of the data. Alternative more appropriate risk measures may be introduced to evaluate the performance of the risk minimizer in a time series context.
Assume that we are interested in predicting the out-of-sample observations $\{ (Y_{t}, \bm X_t')' \}_{t = T+1}^{T+H}$ on the basis of the prediction rule estimated from the in-sample observations $\{ (Y_t, \bm X_t')' \}_{t=1}^T$. For simplicity, here we focus only on the case of identically distributed observations. We define the average out-of-sample risk of $\bm \theta$ as \[ R_\mathsf{oos}(\bm \theta) = \mathbb E\left[ {1 \over H} \sum_{t=T+1}^{T+H} (Y_t - f_{\bm \theta\,t} )^2 \right] ~, \] and we measure the accuracy of the empirical risk minimizer $\hat {\bm \theta}$ using the conditional out-of-sample average risk defined as \[ R_\mathsf{oos}(\hat {\bm \theta}) = \mathbb E\left[ \left. {1 \over H} \sum_{t=T+1}^{T+H} (Y_t - f_{\hat {\bm \theta}\,t} )^2 \right| (Y_T,\bm X_T')' , \ldots , (Y_1,\bm X_1')' \right] ~. \]
For the following result, a slightly stronger version of Assumption (ref) is needed.
\begin{asmredef2}{1}{Moments} The sequences $\{Y_t\}_{t=1}^T$, $\{f^*_t\}_{t=1}^T$ and $\{ \bm X_{t} \}_{t=1}^T$ satisfy $\sup_{1 \leq i \leq p} \sup_{1\leq t\leq T} \| Y_{t}-f^*_t \|_{L_{r_m}} \leq K_m $ $\sup_{1\leq t\leq T} \| Y_t \|_{L_{r_m}} \leq K_m $ and $ \sup_{1 \leq i \leq p} \sup_{1\leq t\leq T} \| X_{i\,t} \|_{L_{r_m}} \leq K_m $ for some $K_m \geq 1$ and $r_m > 4$. \end{asmredef2}
If we define $K_H = 24 (K_m^2/\underline \lambda) \sum_{l=1}^{\infty} \alpha(l)^{1\over 2}$, we can establish the following theorem.
A key ingredient in the proof of Theorem (ref) is Ibragimov's inequality Ibragimov:1962, which bounds the expected value of the difference between the conditional and unconditional expectation as a function of the $\alpha$-mixing coefficients. {It is important to remark that the theorem requires $ ( p \log(T) ) / H \rightarrow 0$ in order to have that $ R_\mathsf{oos}( \hat {\bm\theta} ) - R_\mathsf{oos}( \bm \theta^* ) \rightarrow 0$. In other words there exists a “wedge” between $R_\mathsf{oos}( \hat {\bm\theta} )$ and $R_\mathsf{oos}( {\bm\theta}^* )$ that only vanishes as the forecast horizon $H$ grows large. This may be intuitively explained as follows. The empirical risk minimizer $\hat{\bm \theta}$ is consistent for $\bm \theta^*$, the minimizer of $R(\bm \theta)$. However, the minimizers of $R(\bm \theta)$ and $R_{oos}(\bm \theta)$ are not guaranteed to be same for finite $H$ and the difference between the two only vanishes as the forecast horizon $H$ grows large.}
\paragraph{Small-Ball Assumption.} It is possible to introduce alternative assumptions that imply the small-ball condition stated in (ref). For example, as LecueMendelson:2016 remark, the small-ball condition holds when the $L_2$ and $L_4$ norms of $f_{\bm \theta\,t}$ are equivalent. More precisely, if for each $\bm \theta \in \mathbb R^p$ (and all $t=1,\ldots,T$) it holds that \(\| f_{\bm \theta\,t} \|_{L_4} \leq C \| f_{\bm \theta\,t} \|_{L_2} \), for some constant $C$ (that does not depend on $\bm \theta$ or $t$). We remark that norm equivalence conditions are commonly used in the literature to establish the properties of empirical risk minimization, see for instance AudibertCantoni:2011.
To give a concrete example, below we show that if the distribution of the standardized predictors $\bm Z_t = \bm \Sigma^{-{1\over 2}} \bm X_t$ is spherical then the small-ball assumption is satisfied.
The proof uses the Paley-Zygmund inequality. (ref) can replace (ref) in Theorem (ref) and Theorem (ref).
This paper establishes oracle inequalities for the prediction risk of the empirical risk minimizer for large-dimensional linear regression. We generalize existing results by allowing the data to be dependent and heavy-tailed. Our main results show that the empirical risk minimizer achieves optimal performance (up to a logarithmic factor). The results have been established using the small-ball method, which is a powerful technique to obtain oracle inequalities. Future research includes extending these results to regularized empirical risk minimization, analogously to LecueMendelson:2017,LecueMendelson:2018.