Variable Selection in High Dimensional Linear Regressions with Parameter Instability

Variable Selection in High Dimensional Linear Regressions with Parameter Instability

{\setstretch{1.1}

{ Keywords: Lasso, One Covariate at a time Multiple Testing, OCMT, Parameter instability, Variable selection, Forecasting}

{ JEL Classifications: C22, C52, C53, C55} } \thispagestyle{empty} \pagenumbering{arabic}

Introduction

Models fitted to statistical relationships could be subject to parameter instabilities. In an extensive early study, StockWatson1996 find that a large number of time series regressions in economics are subject to breaks. ClementsHendry1998 consider parameter instability to be one of the main sources of forecast failure. This problem has been addressed at the estimation/forecasting stage for a given set of selected regressors. Typical solutions are either to use rolling windows or exponential down-weighting. For instance, pesaran2007selection, pesaran2011forecast and inoue2017rolling consider the choice of an observation window, and hyndman2008forecasting and pesaran2013optimal, respectively consider exponential and non-exponential down-weighting of the observations. There are also Bayesian approaches to prediction that allow for the possibility of breaks over the forecast horizon, such as chib1998estimation, koop2004forecasting, and pesaran2006forecasting. rossi2013advances provides a review of the literature on forecasting under instability. There are also related time varying parameter and regime switching models that are used for forecasting. See, for example, hamilton1988rational and dangl2012predictive . This literature does not address the problem of variable selection and takes the model specification as given.

The theory of variable selection in the presence of parameter instability is still largely underdeveloped. The application of penalized regression methods to variable selection is often theoretically justified under two key parameter stability assumptions: the stability of the coefficients in the data generating process and the stability of the correlation matrix of the covariates in the active set. Under these assumptions, the penalized regression methods can proceed using the full sample without down-weighting or separating the variable selection from the estimation stage. However, in the presence of parameter instability penalized regression methods must be adapted to simultaneously deal with selection and parameter change. There are a number of recent studies that use machine learning techniques to allow for parameter instability, in particular penalized regression, especially the Least Absolute Shrinkage and Selection Operator (Lasso) initially proposed by tibshirani1996regression. For example, qian2016shrinkage consider a linear regression model with a finite number of covariates but allow for an unknown number of breaks and use group fused Lasso by 10.1007/978-3-642-40728-4_9 to consistently estimate the number of breaks and their locations. lee2016lasso have proposed a Lasso procedure that allows for threshold effects. kapetanios2018time have proposed a time-varying Lasso procedure, where all the parameters of the model vary locally. fan2014nonparametric suggest an extension of the screening procedure initially proposed by fan2008sure to the case where the regression coefficients vary smoothly with an observable exposure variable. Also recently, yousuf2019boosting propose an interesting boosting procedure for the estimation of high-dimensional models with locally time varying parameters. These studies focus on specific forms of discrete or continuous time varying parameter models, and often carry out variable selection and estimation simultaneously using the penalized regression or boosting procedures.

This paper proposes the use of One Covariate at a Time Multiple Testing (OCMT) procedure proposed by chudik2018one which is readily adapted to the task of variable selection under parameter instability. The key insight comes from the fact that coefficients of the noise variables that do not enter the data generating process are zero at all times. Consequently, using unweighted observations at the variable selection stage will be most effective in removing noise variables, while using weighted observations at the estimation stage can provide gains in terms of mean squared forecast errors. In this study, we allow the marginal effects of signals on the target variable, as well as the correlation of the covariates under consideration, to vary over time, assuming time variations in the marginal effects are not correlated with the signals. We establish the conditions required for OCMT with unweighted observations to select a model that contains all the signal variables and none of the noise variables with probability approaching one as the sample size, $T$, and the number of covariates under consideration, $N$, tend to infinity.

Clearly, it is also possible to use penalized regression methods with unweighted observations for the purpose of variable selection, and then estimate the selected model by the least squares method using weighted observations. However, as far as we know, there are no studies that consider the choice of the penalty term to achieve variable selection consistency under parameter instability. It is hoped that the present paper provides an impetus for further theoretical analysis of penalized regression techniques under parameter instability. Although at this stage a comparison of the assumptions required for variable selection consistency of OCMT and Lasso under parameter instability is not possible, in Section (ref) we provide a discussion of the assumptions required for the variable selection consistency of Lasso under parameter stability that are comparable with the those required for the validity of the OCMT procedure.

The OCMT procedure selects variables based on the statistical significance of the net effect of the covariates in the active set on the target variable, one-at-a-time subject to the multiple testing nature of the inferential problem involved. The idea of using one-at-a-time regressions is not unique to OCMT and has been used in boosting as well as in screening approaches. See, for example, buhlmann2006boosting and fan2018sure as prominent examples of these approaches. What is unique about the OCMT procedure is its inferentially motivated stopping rule without resorting to the use of information criteria, or penalized regression after the initial stage.\ In the case of models with stable parameters, chudik2018one establish that OCMT asymptotically selects an approximating model that includes all the signals and none of the noise variables. This model can contain covariates that do not enter the data generating process for the target variable but exhibit non-zero correlation with at least one signal, known as pseudo-signals.

Lasso and OCMT exploit different aspects of the low-dimensional structure assumed for the underlying data generating process. Lasso restricts the magnitude of the correlations within signals as well as the correlations between signals and the remaining covariates in the active set. OCMT limits the rate at which the number of pseudo-signals, $k_{T}^{\ast }$, rises with the sample size, $T$. Under parameter stability, the variable selection consistency of Lasso has been investigated by zhao2006model, meinshausen2006high and more recently by lahiri2021necessary. These conditions, and how they compare with the conditions that underlie OCMT, are discussed in Section (ref) of the paper. Although Lasso does not directly impose any restrictions on $k_{T}^{\ast }$, its Irrepresentable Condition (IRC), by restricting the magnitude of correlations within and between the signals and pseudo-signals, does have implications for the number of pseudo-signals that Lasso selects. OCMT requires $k_{T}^{\ast }$\ not to rise faster than $\sqrt{T}$. When this condition is violated, then the true signals must end up as common factors for the pseudo-signals, and what matters is the number of residuals (from the regressions of pseudo-signals on the common factors) that are correlated with the residuals of the true signals from the same set of common factors. sharifvaghefi2023variable shows that such common factors can be estimated from the principal components of the covariates in the active set and the OCMT\ condition on the number pseudo-signals, now defined in terms of the correlation of the residuals, is no longer restrictive.\footnote{ Another extension of OCMT\ is provided by su2023one who allow for unknown potentially non-linear relationship between the signals and the target variable.} Once the model is selected, Theorem (ref) establishes how the convergence rate of estimated coefficients of the selected variables depends on $k_{T}^{\ast }$. The regular convergence rate of $\sqrt{T}$ is achieved only if $k_{T}^{\ast }$ is fixed in $T$. A similar issue also arises for Lasso, as shown by lahiri2021necessary who establishes that the Lasso procedure cannot achieve both variable selection consistency and $\sqrt{T}$-consistency in coefficient estimation. As noted above, the focus of the present paper is on the application of OCMT to variable selection in the presence of parameter instability, broadly defined. To the best of our knowledge, there are no studies that investigate the variable selection properties of Lasso under parameter instability.

To take account of the time variations in the coefficients of the signals, we consider their time averages and distinguish between strong signals whose average marginal effects go to a non-zero value, semi-strong signals whose average marginal effects converge to zero, but sufficiently slow, and weak signals whose average marginal effects approach to zero quite fast. In this way we allow for variety of time variations that could arise in practice. Strong signals tend to have non-zero effects at all times, semi-strong signals could have zero effects during some periods, with weak signals enter the model relatively rarely. Weak signals are often indistinguishable from noise variables. In our theoretical analysis we will focus on selection of strong and semi-strong signals.

We provide three main theorems in support of our proposed variable selection method. Under certain fairly general regularity conditions we show that the probability of OCMT selecting the approximating model that contains all the signals (strong and semi-strong) and none of the noise variables approaches to one as $T $ goes to infinity. Our results apply both when $N$ is fixed as well as when $N$ goes to infinity jointly with $T$, covering the case where $ N \gg T$. We also establish conditions under which (a) least squares estimates of the coefficients of selected covariates converge to zero unless they are signals, and (b) the average squared residuals of the selected model achieves the oracle rate for regression models with time-varying coefficients. These theoretical findings provide a formal justification for application of statistical techniques from the time-varying parameters literature to the post OCMT selected model. Our Monte Carlo experiments show that the OCMT procedure with weighted observations only at the estimation stage outperforms, in terms of mean squared forecast errors, Lasso and Adaptive Lasso (A-Lasso by zou2006adaptive), as well as boosting by buhlmann2006boosting, under many different settings.

Finally, we provide three empirical applications, forecasting monthly rates of price changes of 28 stocks in Dow Jones using large number of financial, economic and technical indicators, forecasting output growths across 33 countries using a large number of macroeconomic indicators, and forecasting euro area output growth using ECB surveys of 25 professional forecasters. To save space the third application is included in the online supplement. We generate a large number of forecasts using OCMT with and without down-weighting of the observations at the selection stage and compare the results with the forecasts obtained using Lasso, A-Lasso and boosting. The empirical results are in line with our theoretical and MC findings and suggest that using down-weighted observations at the selection stage of the OCMT procedure worsens forecast performance in terms of mean squared forecast errors and mean directional forecast accuracy. The empirical results also provide that OCMT with no down-weighting at the selection stage outperforms, in terms of mean squared forecast errors, boosting, Lasso and A-Lasso.

The rest of the paper is organized as follows: Section (ref) sets out the model specification. Section (ref) explains the basic idea behind the OCMT procedure for variable selection without down-weighting in the presence of parameter instability. {Section (ref) provides a discussion of key assumptions of Lasso and OCMT under parameter stability.} Section (ref) discusses the technical assumptions and the asymptotic properties of the OCMT procedure under parameter instability. Section (ref) provides the details of the Monte Carlo experiments and a summary of the main results. Section (ref) presents the empirical applications, and Section (ref) concludes. The paper is also accompanied with three online supplements. A theory supplement contains the mathematical proofs of the theorems and related lemmas. A Monte Carlo supplement provides additional summary tables, the full set of Monte Carlo results, as well as the description of the algorithms used for Lasso, A-Lasso and boosting. Further details of the empirical applications are given in an empirical supplement.

Notations: Generic finite positive constants are denoted by $C_{i}$ for $i=1,2,\cdots $. $\lVert \mathbf{A}\rVert _{2}$ and $\lVert \mathbf{A} \rVert _{F}$ denote the spectral and Frobenius norms of matrix $\mathbf{A}$, respectively. $\text{tr}(\mathbf{A})$ and $\lambda _{i}(\mathbf{A})$ denote the trace and the $i^{th}$ eigenvalue of a square matrix $\mathbf{A}$, respectively. $\left\Vert \mathbf{x}\right\Vert $ denotes the $\ell _{2}$ norm of vector $\mathbf{x}$. {\ If $\{f_{n}\}_{n=1}^{\infty }$ and $ \{g_{n}\}_{n=1}^{\infty }$ are both positive sequences of real numbers, then we say $f_{n}=\ominus (g_{n})$ if there exist $n_{0}\geq 1$ and positive constants $C_{0}$ and $C_{1}$, such that $\inf_{n\geq n_{0}}\left( f_{n}/g_{n}\right) \geq C_{0}$ and $\sup_{n\geq n_{0}}\left( f_{n}/g_{n}\right) \leq C_{1}$. }Similarly, if $f_{iT}$\ and $g_{iT}$\ are positive double sequences of real numbers for $i=1,2,3,\cdots $; and $ T=1,2,3,\cdots $, then $f_{iT}=\ominus (g_{iT})$\ if there exist $T_{0}\geq 1 $\ and positive constants $C_{0}$\ and $C_{1}$, such that $\inf_{T\geq T_{0}}\left( f_{iT}/g_{iT}\right) \geq C_{0}$\ and $\sup_{T\geq T_{0}}\left( f_{iT}/g_{iT}\right) \leq C_{1}$.

Model specification under parameter instability

We consider the following data generating process (DGP) for the target variable, $y_{t}$, in terms of the signal variables $(x_{it}$, for $ i=1,2,...,k$)

with time-varying parameters, $\left\{ \beta _{it}\text{, } i=1,2,...,k\right\} $, and an error term, $u_{t}$. Intercepts and other pre-selected variables can also be included.\footnote{ See the working paper version of the paper available at \url{https://doi.org/10.24149/gwp394r2}.} Since the parameters are time-varying we refer to the covariate $i$ as \textquotedblleft signal" if its average marginal effect, $\bar{\beta}_{i,T}=T^{-1} \sum_{t=1}^{T}\mathbb{E}(\beta _{it})$, is not equal to zero. The strength of the signal can be captured by the exponent coefficient $\vartheta _{i}$ in $\bar{\beta}_{i,T}=\ominus (T^{-\vartheta _{i}})$. For $\vartheta _{i}=0$ , the signal is strong and the average marginal effect, $\bar{\beta}_{i,T}$, does not converge to zero. For $0<\vartheta _{i}<1/2$, the signal is semi-strong and the average marginal effect converges to zero, but not too fast. For $\vartheta _{i}\geq 1/2$, the average marginal effect tends to zero very fast, making it infeasible for the OCMT procedure to distinguish such weak signals from noise, unless weak signals are sufficiently correlated with at least one strong or semi-strong signal. In this paper, we do not impose any restrictions on the correlations among signals, and we focus only on the covariates with strong and semi-strong signals, where $ 0\leq \vartheta _{i}<1/2$. For simplicity of exposition, unless specified otherwise, we will refer to both strong and semi-strong signals simply as signals.

The identity of the $k$ signals are unknown, and the task facing the investigator is to select the signals from a set of covariates under consideration, $\mathcal{S}_{Nt}=\{x_{1t},x_{2t},\cdots ,x_{Nt}\}$, known as the active set, with $N$, the number of covariates in the active set, possibly much larger than $T$, the number of data points available for estimation prior to forecasting. The time variations in $\beta _{it}$, for $ i=1,2,...,k$, are assumed to be exogenous, in the sense that $\beta _{it}$ are distributed independently of the covariates in the active set $\mathcal{S }_{Nt}$. This assumption rules out correlated time variations that can arise in non-linear regressions where $y_{t}$ is a non-linear function of the signals. One important example is given by the bilinear model

where it is assumed that $\beta _{it}$ systematically varies with $x_{it}$. Nevertheless, in the context of linear regressions, our assumptions about parameter instability includes many models of parameter instability studied in the literature. Specifically, our analysis accommodates cases where the coefficients vary continuously following a stochastic process as in the standard random coefficient model,

or could change at discrete time intervals, as

where $T_{0}=1$ and $T_{S}=T$.

In this paper we follow chudik2018one and consider the application of the OCMT procedure for variable selection even when the parameters are time-varying, and provide theoretical arguments in favour of using the full sample of data available without down-weighting. We first recall that OCMT's variable selection is based on the net effect of $x_{it}$ on $y_{t}$. However, when the regression coefficients and/or the correlations across the covariates in the active set are time-varying, the net effects will also be time-varying and we need to base our selection on average net effects. The average net effect of the covariate $x_{it}$ on $y_{t}$ can be defined as

By substituting $y_{t}$ from ((ref)) we can further write $\bar{ \theta}_{i,T}$ as (noting that $\beta _{jt}$ and $x_{it}$ are assumed to be independently distributed)

where $\sigma _{ij,t}=\mathbb{E}(x_{it}x_{jt})$, and $\bar{\sigma} _{iu,T}=T^{-1}\sum_{t=1}^{T}\mathbb{E}(x_{it}u_{t})$. In what follows we allow for a mild degree of correlation between $x_{it},$ and $u_{t},$ by assuming that $\bar{\sigma}_{iu,T}=O(T^{-\epsilon _{i}}),$ for some $ \epsilon _{i} \geq 1/2$. In this case the average net effect of the $i^{th}$ covariate simplifies to

In line with our assumption about the average marginal effects, namely that $ \bar{\beta}_{i,T}=\ominus (T^{-\vartheta _{i}}),$ for some $0\leq \vartheta _{i}<1/2$, we distinguish between covariates with strong and semi-strong net effects, and the noise variables whose net effects, averaged over time, tend to zero sufficiently fast. Specifically, for covariates with strong or semi-strong net effects we set $\bar{\theta}_{i,T}=\ominus (T^{-\vartheta _{i}}),$ for some $0\leq \vartheta _{i}<1/2$, and for the noise variables we shall assume that $\bar{\theta}_{i,T}=\ominus (T^{-\epsilon _{i}}),$ for some $\epsilon _{i}\geq 1/2$.

In what follows, we first describe the OCMT procedure and then discuss the conditions under which the approximating model (that includes all the signals and none of the noise variables) is selected with probability approaching one by OCMT.

Parameter instability and OCMT

The OCMT procedure begins with $N$ separate regressions, for each of the $N$ covariates in the active set $\mathcal{S}_{Nt}$. Specifically, the focus is on the statistical significance of $\phi _{i,T}$ in the following simple regressions:

where

with $\bar{\sigma}_{ii,T}=$ $T^{-1}\sum_{t=1}^{T}\sigma _{ii,t}$. Due to non-zero cross-covariate correlations, knowing whether $\phi _{i,T}$ $\ ($or equivalently $\bar{\theta}_{i,T}$) is zero does not necessarily allow us to establish whether $\bar{\beta}_{i,T}$ is sufficiently close to zero or not. There are four possibilities:

{

}

{ {$\dagger $ The signals are assumed to be (semi) strong such that $0\leq \vartheta _{i}<1/2$.}}

for some $0\leq \vartheta _{i}<1/2$, and $\epsilon _{i}\geq 1/2$. { To simplify the exposition, we consider the covariates $x_{it}$, for $ i=1,2,\cdots ,k$, as signals, and for $i=k+1,k+2,\cdots ,k+k_{T}^{\ast }$, as pseudo-signals. The remaining covariates in the active set, }$\left\{ { x_{it},}\text{ for }{i=k+k_{T}^{\ast }+1,k+k_{T}^{\ast }+2,\cdots ,N} \right\} ${, are classified as (pure) noise variables. We assume that the number of signals, $k$, is a finite fixed integer but we allow the number of pseudo-signals, denoted by $k_{T}^{\ast }$, to grow with $N$ and $T$.} Notice, if the covariate $x_{it}$ is a noise variable, then $\bar{\theta} _{i,T}$ converges to zero very fast. Therefore, down-weighting of observations at the variable selection stage is likely to be inefficient for eliminating the noise variables. Moreover, for a signal to remain hidden, we need the terms of higher order, $\ominus (T^{-\vartheta _{j}})$ with $0\leq \vartheta _{i}<1/2$, to exactly cancel out such that $\theta _{i,T}$ becomes a lower order, i.e. $\ominus (T^{-\varepsilon _{i}})$, that tends to zero at a sufficiently fast rate (with $\epsilon _{i}\geq 1/2)$. This combination of events seem quite unlikely, and to simplify the theoretical derivations in what follows we abstract from such a possibility and assume that there are no hidden signals and consider a single stage version of the OCMT procedure for variable selection. To allow for hidden signals, chudik2018one extend the OCMT method to have multiple stages.

{OCMT uses the t-ratio of $\phi _{i,T}$, defined by ((ref)), to select the signals (strong as well as semi-strong), $\{x_{it}:i=1,2,\cdots ,k\}$, and none of the noise variables, $\{x_{it}:k+k_{T}^{\ast }+1,k+k_{T}^{\ast }+2,\cdots ,N\}$. The selected model is referred to as an approximating model since it can include pseudo-signals, $ \{x_{it}:k+1,k+2,\cdots ,k+k_{T}^{\ast }\}$, that proxy for the true signals. To deal with the multiple testing nature of the problem, the critical value $c_{p}(N,\delta )$ used for the separate-induced tests is chosen to be an appropriately increasing function of $N$, by setting $\delta >0$. The choice of $\delta $ is guided by our theoretical derivations, to be discussed below in Section (ref). }

Before presenting our technical assumptions and theoretical results under parameter instability, it is instructive to discuss and compare the key conditions under which Lasso and OCMT lead to consistent model selection under parameter stability.

Lasso and OCMT under parameter stability

{As formally established by zhao2006model and meinshausen2006high, three main conditions are required for the Lasso variable selection to be consistent. Here we follow lahiri2021necessary who also considers the convergence of Lasso estimated coefficients to their true values. The key condition is the \textquotedblleft Irrepresentable Condition" (IRC) that places restrictions on the magnitudes of the sample correlations across the signals, }${\mathbf{x }_{1t}=(x_{1t},x_{2t},\cdots ,x_{kt})^{\prime }}$, and the rest of the { covariates in the active set, namely $\mathbf{x}_{2t}=(x_{k+1,t},x_{k+2,t}, \cdots ,x_{Nt})^{\prime }$. Let

be the $N\times N$ matrix of sample correlations of the covariates in the active set, partitioned conformably to $\mathbf{x}_{t}=(\mathbf{x} _{1t}^{\prime },\mathbf{x}_{2t}^{\prime })^{\prime }$. The IRC can be written as

where $\left\Vert .\right\Vert _{\infty }$ is the $\ell _{\infty }$ norm of a vector, $\text{sign}(.)$ is the sign function, and $\boldsymbol{\beta } _{0}=\left( \beta _{01},\beta _{02},\cdots ,\beta _{0k}\right) ^{\prime }$ is the $k\times 1$ vector of the coefficients of the signals. The following example provides more intuition on how IRC imposes restrictions on the magnitudes of the sample correlations between the covariates in the active set. }

From the above example, it is clear that IRC places restrictions on the magnitude of sample correlation among signals ($\hat{\rho}$ in the above example), as well as the magnitude of sample correlation between signals and pseudo-signals ($\hat{\rho}_{i1}$ and $\hat{\rho}_{i2}$). Notably, the IRC is met for noise variables but need not hold for pseudo-signals. OCMT also has no difficulty in dealing with noise variables, and is very effective at eliminating them. However, for consistent estimation of the approximate model, post OCMT selection, it is necessary to restrict the number of selected covariates relative to the sample size, $T$. To this end, OCMT assumes that the number of pseudo-signals, $k_{T}^{\ast }$, could grow at an order less than the square root of the number of observations, namely

It is important to note that OCMT does not place any restrictions on the magnitude of correlations of signals and pseudo-signals. Instead, it limits the number of covariates that are correlated with the signals ($k_{T}^{\ast } $). Clearly, the IRC could be violated even when the number of pseudo-signals grows at an order less than $\sqrt{T}$. Hence the OCMT's requirement on the number of pseudo-signals allows for cases where the IRC does not hold, and vice versa.

The condition on the number of pseudo-signals ($k_{T}^{\ast }$) in the OCMT framework has been recently relaxed by sharifvaghefi2023variable. To illustrate how this is done, suppose there are no noise variables and hence the signals, $\mathbf{x}_{1t}=\left( x_{1t},x_{2t},\cdots ,x_{kt}\right) ^{\prime }$, are correlated with all the remaining covariates in the active set. In this case if $N \gg \sqrt{T} $, a straightforward application of OCMT will not be valid. But, we can model the correlation between the signals, $\mathbf{x}_{1t}$, and the remaining covariates, {$\mathbf{x}_{2t}$, } as

The signals thus act as strong factors for the pseudo-signals. Given that the identity of signals and pseudo-signals are unknown and the number of pseudo-signals is large, it is reasonable to propose the existence of latent factors, $\mathbf{f}_{t}$, that are common across the covariates in the active set. This idea can be formally expressed as:

where $\boldsymbol{\psi }_{i}$ is vector of factor loadings, and $ \varepsilon _{it}$ refers to the idiosyncratic components that are weakly cross-correlated such that

Substituting $x_{it}$ into the DGP for $y_{t}$, given by ((ref)), we obtain:

with $\boldsymbol{\delta }_{0}=\sum_{i=1}^{k}\beta _{i0}\boldsymbol{\psi } _{i}$. When the common factors, $\mathbf{f}_{t}$, and idiosyncratic components, $\varepsilon _{it}$ , are known, this model would correspond to that presented in working paper version of our work, where common factors $ \mathbf{f}_{t}$ can be used as preselected variables. Since $\mathbf{f}_{t}$ and $\varepsilon _{it}$ are not known, sharifvaghefi2023variable shows that when both $N$ and $T$ are large the OCMT selection can be carried out using the principal component estimators of $\mathbf{f}_{t}$ and $ \varepsilon _{it}$, denoted by $\hat{\mathbf{f}}_{t}$ and $\hat{\varepsilon} _{it}$, using all the covariates in the active set. The large $N$ is required for consistent estimation of the common factors. As a result, the OCMT\ condition on the number of pseudo-signals now relates to the number of $\varepsilon _{it}$ for $i=k+1,k+2,...,N$ that are correlated with $ \varepsilon _{it}$ for $i=1,2,...,k$, which is bounded under condition ((ref)).

For variable selection consistency of Lasso under parameter stability, the literature further requires the penalty term, $\lambda _{T}$, to grow at an order greater than $\sqrt{T}$ such that:

where $\mathbf{x}_{2t}^{\perp} $ is the part of variation in $\mathbf{x} _{2t} $ that is orthogonal to $\mathbf{x}_{1t} $ and $u_{t}$ is the error term in the data generating process. The exact choice of $\lambda _{T}$ in practice is often unclear, with practitioners typically relying on cross-validation methods.

A third condition required by Lasso for variable selection consistency is the beta-min condition:

where $\left\vert .\right\vert _{j}$ denotes the absolute value of the $ j^{th}$ element of a vector. Given that $\lambda _{T}$ must grow at an order greater than $\sqrt{T}$, we can conclude from the beta-min condition that $ \beta _{i0}\gg \frac{1}{\sqrt{T}}$ for $i=1,2,\cdots ,k$. For example, lahiri2021necessary assumes that $\beta _{i0}\gg \sqrt{\frac{k\log (T)}{T}} $. The OCMT's requirement on the strength of signals (under parameter stability) is given by $\beta _{i0}=\ominus (T^{-\vartheta _{i}})$, for some $0\leq \vartheta _{i}<1/2$. This condition is essentially very similar to the Lasso's beta-min condition.

Asymptotic properties of OCMT under parameter instability

We establish the asymptotic properties of the OCMT procedure for variable selection assuming the time variations in $\beta _{it}$ for $i=1,2,...,k$ are distributed independently of the regressors in the active set. We also make additional assumptions that bound the degree of time variations in $ \beta _{it}$ and $x_{it}$, in addition to assuming the exponentially decaying tail probabilities for $\beta _{it}$ and $x_{it}$. Our assumptions on $x_{it}$, $i=1,2,...,k$ and their correlations with the other variables in the active set are in line with those assumed in the literature. A formal statement of these assumptions are set out in Section (ref). Theorem (ref) establishes that OCMT continues to asymptotically select an approximating model that includes all the signals and none of the noise variables. Additional assumptions are required for investigating the asymptotic properties of the least squares estimates of the post OCMT selected model. These assumptions and the related theorems are provided in Section (ref). Theorem (ref) establishes the rate at which the least squares estimates of the coefficients of the selected model converge to their true time averages. It is shown that the regular convergence rate of $\sqrt{T}$ is achieved only if $k_{T}^{\ast }$ (the number of selected covariates) is fixed in $T$. Irregular convergence rates result when $k_{T}^{\ast }$ rises in $T$. Theorem (ref) shows that the sum of squared residuals of the estimated model converges in probability to its limiting value at the oracle rate of $\sqrt{T}$. The limiting value consists of two components: the first is the unavoidable uncertainty due to the unobserved error term, $u_{t}$, and the second is the cost (in terms of fit) of ignoring the time variations in the coefficients of the signals.

Suppose the target variable, $y_{t}$, is generated by ((ref)) in terms of $x_{it}$ for $i=1,2,...,k$, and $\mathbf{x}_{t}=(x_{1t},x_{2t}, \cdots ,x_{kt},x_{k+1,t},....,x_{Nt})^{\prime }$ is the $N\times 1$ vector of covariates in the active set ($N \gg k$). Let $\bar{\beta}_{i,T}\equiv T^{-1}\sum_{t=1}^{T}\mathbb{E}(\beta _{it})$, for $i=1,2,...,k$, and $\bar{ \theta}_{i,T}=\sum_{j=1}^{k}\left( T^{-1}\sum_{t=1}^{T}\mathbb{E}(\beta _{jt})\sigma _{ij,t}\right) +\bar{\sigma}_{iu,T},$ for $i=1,2,...,N$, where $ \sigma _{ij,t}=\mathbb{E}(x_{it}x_{jt})$, and $\bar{\sigma} _{iu,T}=T^{-1}\sum_{t=1}^{T}\mathbb{E}(x_{it}u_{t})$. Define the filtrations $\mathcal{F}_{t}^{u}=\sigma (u_{t},u_{t-1},\cdots )$, $\mathcal{F} _{t}^{x}=\sigma (\mathbf{x}_{t},\mathbf{x}_{t-1},\cdots )$, and $\mathcal{F} _{jt}^{\beta }=\sigma (\beta _{jt},\beta _{j,t-1},\cdots )$, for $ j=1,2,\cdots ,k$. Set $\mathcal{F}_{t}^{\beta }=\cup _{j=1}^{k}\mathcal{F} _{jt}^{\beta }$ and $\mathcal{F}_{t}=\mathcal{F}_{t}^{q}\cup \mathcal{F} _{t}^{\mathrm{a}}\cup \mathcal{F}_{t}^{\beta }\cup \mathcal{F}_{t}^{u}$, and consider the following assumptions:

Assumptions

{Before presenting the theoretical results, we briefly discuss the rationale behind our assumptions and compare them with the assumptions typically made in the high-dimensional linear regressions and the parameter instability literature. }

Assumption (ref)(a) posits that the number of signals is a fixed integer. This is crucial to ensure that the random variable $y_{t}$ has a distribution with an exponentially decaying probability tail. Under the premise that the covariates $x_{it}$ for all $i$ and $t$ are non-random and fixed, which is a common assumption in the penalized regression setting, it becomes permissible for the number of signals to grow with the sample size at an order slower that $\sqrt{T}$. Assumption (ref)(b) is common in the literature under parameter instability and restrict the distribution of time-varying parameters to be independent of the covariates. Assumption (ref)(c) is an identification assumption needed to distinguish signals from noise variables and is similar to the beta-min condition already discussed in Section (ref). Finally, Assumption (ref)(d) ensures that there are no hidden signals. As discussed in Section (ref), we make this assumption to simplify the theoretical derivations, and one can use the multi-stage OCMT procedure suggested by chudik2018one to allow for hidden signals.

To establish that the OCMT procedure with the critical value function $ c_{p}(N,\delta )=\Phi ^{-1}\left( 1-\frac{p}{2N^{\delta }}\right) $ does not select any of the noise variables with a probability approaching one as $N$ and $T$ go to infinity, we need to show that the t-statistic given by ((ref)) follows a distribution with exponentially decaying tails. We utilize the concentration inequality of an exponential decaying rate to accomplish this goal. Assumptions (ref) and (ref) place constraints on the sequence of random variables, $x_{it}$ for $i=1,2,\ldots ,N$, $\beta _{jt}$ for $j=1,2,\ldots ,k$, and $u_{t}$ such that they adhere to a martingale difference process and exhibit exponential decaying probability tails. These assumptions are sufficient to establish the exponential decaying concentration inequality, as provided in Lemma (ref) in the online theory supplement. Notably, these assumptions could be relaxed provided that the exponential decaying concentration inequality holds. For example, Theorem 1 of merlevede2011bernstein and Lemma D1 of the online theory supplement for chudik2018one establishes that this inequality can be achieved while allowing for weak time-series dependence. In penalized regression literature, a commonly held assumption is that the covariates are non-random and fixed. Moreover, error terms $\{u_{t}\}_{t=1}^{T}$ are typically assumed to be serially independent. See, for example, see zhao2006model, javanmard2013model, lee2015model, belloni2014inference, javanmard2020flexible, and lahiri2021necessary. Additionally, in the Lasso literature it is often assumed that $u_{t}$ possesses an exponentially decaying probability tail. See, for example, javanmard2018debiasing, hansen2019factor, fan2020factor, and javanmard2020flexible.

Variable selection consistency

As mentioned in Section (ref), the purpose of this paper is to provide the theoretical argument for applying the OCMT procedure with no down-weighting at the variable selection stage in linear high-dimensional settings subject to parameter instability. We now show that under the assumptions set out in Section (ref), the OCMT procedure selects the approximating model that contains all the signals; $ \{x_{it}:i=1,2,\cdots ,k\}$; and none of the noise variables; $ \{x_{it}:k+k_{T}^{\ast }+1,k+k_{T}^{\ast }+2,\cdots ,N\}$. The event of choosing the approximating model is defined by

Note that the approximating model can contain pseudo-signals. In what follows, we show that $\Pr (\mathcal{A}_{0})\rightarrow 1$, as $ N,T\rightarrow \infty $.

This theorem shows that the probability of selecting the approximating model is unaffected by parameter instability, so long as the average net effects of the signals are non-zero or converge to zero sufficiently slowly in $T$, as defined formally by Assumption (ref). The theorem also highlights the importance of an appropriate choice of $\delta $ for model selection consistency. Corollary (ref) in the online theory supplement shows that if the covariates in the active set are generated by a stationary process and the noise variables are independent of $y_{t}$ then $ \mathcal{X}_{NT}=1$. As a result, for any $\delta >1$, OCMT consistently selects the approximating model, $\mathcal{A}_{0}$. Notably, $c_{p}(N,\delta )$ is reasonably stable with respect to small increases in $\delta $ in the neighborhood of $\delta =1$ and the extensive Monte Carlo studies in chudik2018one also suggest that setting $\delta =1$ performs well in practice.\footnote{ One could also use the heteroscedasticity and/or autocorrelation robust standard errors in computation of t-statistics given by ((ref)) to ensure the consistent selection of the approximating model for any $\delta >1 $ in a more general setup.}

Properties of the post OCMT selected model

To investigate the asymptotic properties of the least squares estimates of the selected model (post OCMT) we require the following additional assumption:

This assumption ensures that the post OCMT selected model can be consistently estimated subject to certain regularity conditions to be discussed below. The post OCMT selected model can be written as

where $\hat{\mathcal{J}}_{i}=\mathbf{\mathit{I}}\left[ \lvert t_{i,T}\rvert >c_{p}(N,\delta )\right] ,\ $defined by ((ref)). Also $ \sum_{i=1}^{N}\hat{\mathcal{J}}_{i}=\hat{k}_{T}$, where $\hat{k}_{T}$ is the number of covariates selected by OCMT. By Theorem (ref) the probability that the selected model contains the signals tends to unity as $T\rightarrow \infty $. We can further write

where $\mathbf{w}_{t}=\left( w_{1t},w_{2t},\cdots ,w_{\hat{k}_{T}t}\right) ^{\prime }$. The least squares (LS) estimator of selected coefficients, $ \boldsymbol{\gamma }_{T}=\left( \gamma _{1},\gamma _{2},\cdots ,\gamma _{ \hat{k}_{T}}\right) ^{\prime }$, is given by

In establishing the rate of convergence of $\hat{\mathbf{\gamma }}_{T}$ we distinguish between two cases: when the vector of signals, $\mathbf{x} _{k,t}=\left( x_{1t},x_{2t},\cdots ,x_{kt}\right) ^{\prime }$ is included in $\mathbf{w}_{t}$ as a subset, and when this is not the case. But we know from Theorem (ref) that the probability of the latter tends to zero at a sufficiently fast rate. The following theorem provides the conditions under which the estimates of the coefficients of the selected signals and pseudo-signals of the approximating model tend to their true mean values, defined formally below.

Lastly, we consider the fit of the post OCMT selected regression in terms of its residuals given by

It is worth noting that even when all the signal variables are correctly selected, the forecasts based on the selected model will be biased due to parameter instability. The implications of parameter instability for the in-sample fit of the selected regression is derived in Proposition (ref) of the online theory supplement, abstracting from variable selection uncertainty. In what follows we derive the asymptotic properties of the sum of squared residuals (SSR) of the selected model, namely $\sum_{t=1}^{T}\hat{\eta}_{t}^{2}$, taking account of the costs associated with variable selection uncertainty and parameter instability. To this end we need the following assumption on the cross correlation of parameter heterogeneity.

The following theorem establishes the limiting property of SSR of the post OCMT selected model.

Results ((ref)) and ((ref)) in Theorem (ref) show that the SSR of the selected model depends on ($i$) the unavoidable uncertainty due to the unobserved error term, $u_{t}$, given by the term $ \bar{\sigma}_{u,T}^{2}$, ($ii$) the cost (in terms of fit) of ignoring the time variation in the coefficients of the signals, $\boldsymbol{\beta }_{it}$ , $i=1,2,\cdots ,k$, as given by the term $\bar{\Delta}_{\beta ,T}$ and $ \bar{\Delta}_{\beta ,T}^{\ast }$, respectively, and ($iii$) the $O_{p}\left( T^{-1/2}\right) $ term due to sampling uncertainty (which will be present even in the absence of variable selection uncertainty), and $(iv)$ the $ O_{p}\left( T^{d-1}\right) $ term which is due to variable selection uncertainty, and will be dominated by $O_{p}\left( T^{-1/2}\right) $ when $ d<1/2$. Therefore, the cost of variable selection can be controlled when using OCMT if the number of pseudo-signals, $k_{T}^{\ast }$, do not rise faster than $\sqrt{T}$. However, to reduce the cost associated with parameter instability more information about the nature of time variations in $\boldsymbol{\beta }_{it}\boldsymbol{\ }$and $\sigma _{ijt,x}$ are required. For example, $\bar{\Delta}_{\beta ,T}$ (or $\bar{\Delta}_{\beta ,T}^{\ast }$) could be lower if $\boldsymbol{\Omega }_{\beta ,t}$ is close to zero in some periods, or if there are cancelling effects from negative $ \sigma _{ijt,x}$ ($\sigma _{ijt,x}^{\ast }$) when $\sigma _{ijt,\beta }$ is positive, namely $\sigma _{ijt,x}\sigma _{ijt,\beta }<0$ ($\sigma _{ijt,x}^{\ast }\sigma _{ijt,\beta }<0$), for some $i\neq j$ and some $t$. This finding for the in-sample fit is similar to the results for mean squared forecast errors in the presence of breaks in the literature, such as Proposition 2 of pesaran2007selection or equation (20) of pesaran2013optimal, where the main focus is to minimize the MSFE by mitigating the cost of parameter instability at the expense of increased sampling uncertainty by appropriate weighting of the observations.

Monte Carlo evidence

We use Monte Carlo (MC) techniques to compare finite sample performance of OCMT with and without down-weighting at the selection stage, as well as comparing the OCMT\ results with those of Lasso, A-Lasso, and boosting. In these comparisons we consider the number of selected covariates ($\hat{k} _{T} $), the true positive rate (TPR), the false positive rate (FPR), and the one-step-ahead mean squared forecast error (MSFE) of the selected models. Sub-section (ref) outlines the MC designs, sub-section (ref) provides a summary of how the OCMT, Lasso, A-Lasso, and boosting procedures are implemented, and finally sub-section (ref) presents the main MC findings. {Details of Lasso, A-Lasso, and boosting procures and how they are implemented are provided in Section (ref) of the online Monte Carlo supplement.}

Simulation design

We consider the following data generating process (DGP):

where the four signals $\tilde{x}_{jt},$ $j=1,2,3,4$ have non-zero, time-varying means $\mu _{jt}=\mathbb{E}\left( \tilde{x}_{jt}\right) $. To simplify the exposition of the DGP we consider the demeaned covariates, $ x_{jt}=\tilde{x}_{jt}-\mu _{jt}$ (so that $\mathbb{E}\left( x_{jt}\right) =0$ ), and write the DGP equivalently as

where

Since $c_{t}$ is a free parameter, without loss of generality we also treat $ \left\{ d_{t}\,,\text{ }t=1,2,...,T\right\} $ as free parameters.

For each MC replication, $r=1,2,...,R$, the target variable, $y_{t}$, is generated as random draws using ((ref)). The signal variables $ x_{jt} $, $j=1,2,3,4$, are unknown and belong to a set $\mathcal{S} _{Nt}=\left\{ x_{1t},x_{2t},\cdots ,x_{Nt}\right\} $. The vector of covariates $\mathbf{x}_{t}=\left( x_{1t},x_{2t},\cdots ,x_{Nt}\right) ^{\prime }$ is generated as $\mathbf{x}_{t}=\boldsymbol{R}_{t}^{1/2} \boldsymbol{\varepsilon }_{t}$, where $\boldsymbol{\varepsilon }_{t}=\left( \varepsilon _{1t},\varepsilon _{2t},\cdots ,\varepsilon _{Nt}\right) ^{\prime }$. $\left\{ \varepsilon _{it}\right\} $ are generated as AR(1) processes with GARCH(1,1) innovations

using the starting values $\varepsilon _{i,0}\sim IIDN\left( 0,1\right) $. The parameters were generated heterogeneously as independent draws, $ \rho_{i\varepsilon }\sim IIDU\left( 0,0.95\right) $. $e_{\varepsilon _{i}t}\sim IIDN\left( 0,\sigma_{\varepsilon _{i},t}^{2}\right) $, with $ \sigma_{\varepsilon_{i},t}^{2}$ given by

where $\alpha _{1\varepsilon _{i}}\sim IIDU(0,0.2)$, and $\alpha _{2\varepsilon _{i}}\sim IIDU(0.6,0.75)$. The error terms, $\left\{ u_{t}\right\} _{t=1}^{T}$, in ((ref)) are generated as $ IIDN(0,\sigma _{ut}^{2})$ with $\sigma _{ut}^{2}$ following the GARCH(1,1) specification

using $u_{0}\sim \mathcal{N}(0,1)$, $\alpha _{1u}=0.2$ and $\alpha _{2u}=0.75 $.

As our baseline DGP we consider a model with stable parameters, and set $ \beta _{jt}=1$ for $j=1,2,3,4$. We also set $c_{t}=0$ and $\mu _{jt}=1$ in ( (ref)), which yields $d_{t}=4$. In addition, we set $\rho _{y,t}=0$ when the baseline model is static and $\rho _{y,t}=0.3$ when the baseline model is dynamic. In the dynamic case we set $y_{0}=(1-\rho _{y,1})^{-1}d_{1}$. In the case of models with parameter instability we consider a mixed deterministic-stochastic model and generate $\beta _{jt}$ as

where $b_{jt}$ are deterministic and $\eta _{jt}$ are AR(1) processes with GARCH(1,1) innovations,

using the starting values $\eta _{j,0}\sim IID\mathcal{N}\left( 0,1\right) $ , and $\rho _{\eta j}=0.5$, for all $j$. $\left\{ e_{\eta _{j}t}\right\} $ follows a normal distribution with mean zero, and variance $\sigma _{\eta _{j}t}^{{2}}$ given by

where $\alpha _{1\eta _{j}}=0.2$ and $\alpha _{2\eta _{j}}=0.75$. We set $ \tau _{\eta _{j}}$ such that deterministic variations in $\beta _{jt}$ are quite large relative to the stochastic variations. To this end we set $\tau _{\eta _{j}}$ (using simulations) so that

For the deterministic components of the slope coefficients ($b_{jt}$, for $ j=1,2,3,4$), we consider the following specifications

and

where $[.]$ is the nearest integer function.

We \ also set $c_{t}=0$ in ((ref)) and generate the intercept as $ d_{t}=\sum_{j=1}^{k}\beta _{jt}\mu _{jt},$ where

and

In this design, the jumps in $b_{jt}$ and $\mu _{jt}$, for $j=1,2$, have opposite signs and the jumps in $b_{jt}$ and $\mu _{jt}$, for $j=3,4$, have the same sign.

The $N\times N$ correlation matrix of the covariates, $\mathbf{R}_{t}\equiv (r_{ij,t})$, are set as $r_{ij,t}=r_{t}^{|i-j|}$, for all $i,j=1,2,\cdots ,N$ . We allow for a break in the correlation matrix and set $r_{t}$ equal to 0.9 in the first half of the sample and 0.4 in the second half of the sample. Also, we consider two possibilities for $\rho _{y,t}$. In the static scenario we set $\rho _{y,t}=0$ for all $t$. In the dynamic scenario we allow for a switch in $r_{y,t}$ and set it as

For the static and dynamic models with parameter instabilities, the parameter $\tau _{u}$ is calibrated by simulations to ensure that the R-squared of the linear regression of $y_{t}$ on a constant term, the signal variables $\left\{ x_{1t},x_{2t},x_{3t},x_{4t}\right\} $, and (in experiments with $\rho _{y,t}\neq 0$) the lagged dependent variable is equal to $30\%$ (low fit) and $50\%$ (high fit). The same value of $\tau _{u}$ is used for the corresponding static and dynamic models without parameter instabilities.

We base the MC results on $R=2,000$ replications, and consider $N\in \left\{ 20,40,100\right\} $ and $T\in \left\{ 100,200,500\right\} $, combinations. These choices of $\left( N,T\right) $ cover our empirical applications. For each pair of $(N,T)$, there are four experiments in case of the models with no parameter instabilities, and four experiments in the case of models with parameter instabilities, corresponding to the two choices of $\tau _{u}$ (low and high fit), $\rho _{yt}$ (static to dynamic). In total, we carry out eight different experiments.

Selection and estimation methods using weighted and unweighted observations

Let $\mathbf{w}_{t}=(\mathbf{x}_{t}^{\prime },y_{t})^{\prime }$, $ t=1,2,\cdots ,T$ be the (unweighted) set of available observations, and denote the corresponding set of down-weighted observations by $\hat{\mathbf{w }}_{t}(\lambda )=\lambda ^{T-t}\mathbf{w}_{t}$ where $0<\lambda \leq 1$ is the down-weighting coefficient. We are not arguing for the use of exponential down-weighting -- but use it as an example. There are also non-exponential type down-weighting schemes that one can use, e.g. pesaran2013optimal. We will consider the following selection/estimation methods: (1) OCMT with down-weighted observations $\{\hat{\mathbf{w}} _{t}(\lambda )\}_{t=1}^{T}$ used at both selection and estimation stages; (2) OCMT with the unweighted observations, $\{\mathbf{w}_{t}\}_{t=1}^{T}$, used at the selection stage and down-weighted observations, $\{\hat{\mathbf{w }}_{t}(\lambda )\}_{t=1}^{T}$, used at the estimation stage; (3) OCMT using unweighted observations, $\{\mathbf{w}_{t}\}_{t=1}^{T}$, at both selection and estimation stages; (4,5 & 6) Lasso, A-Lasso, and boosting also using unweighted observations, $\{\mathbf{w}_{t}\}_{t=1}^{T}$; and (7,8 & 9) Lasso, A-Lasso, and boosting with down-weighted observations, $\{\hat{ \mathbf{w}}_{t}(\lambda )\}_{t=1}^{T}$ used as inputs.

We also implement a two-step procedures based on Lasso, A-Lasso and boosting. In the first step, we apply Lasso, A-Lasso and boosting to the original (unweighted) observations and select the variables with non-zero coefficients. In the second step, we estimate the corresponding post-selected model by LS using the weighted observations. Overall, the MSFEs of these procedures were higher than that of direct application of Lasso, A-Lasso and boosting to the weighted observations. The results are available in Section S-2 of the online MC supplement.

We consider two sets of values for the down-weighting coefficient, $\lambda $ : (1) Light down-weighting with $\lambda =\left\{ 0.975,0.98,0.985,0.99,0.995,1\right\} $, and (2) Heavy down-weighting with $ \lambda =\left\{ 0.95,0.96,0.97,0.98,0.99,1\right\} $. For each of the above two sets of exponential down-weighting schemes (light/heavy) we focus on simple average forecasts computed over the individual forecasts obtained for each value of $\lambda $ in the set under consideration.

Simulation results

A summary of the main results are provided in Tables (ref) to (ref), with additional summary tables highlighting the effects of down-weighting at the selection stage, and the differences between static versus dynamic models provided in the online MC supplement. Table (ref) give the number of selected covariates ($\hat{k}_{T}$), TPR and FPR of OCMT, Lasso, A-Lasso and boosting without down-weighting. Panel A of this table reports the results for different $N$ and $T$ combinations, averaged across the four experiments without parameter instabilities, and panel B of the table gives the corresponding results for the four experiments with parameter instabilities. The results show that all the methods under consideration have higher average TPR for models with stable parameters compared to the ones with parameter instabilities. This is to be expected, as the models with parameter instabilities are subject to an additional source of uncertainty.

We further observe that the lower average TPR of OCMT in the models with parameter instabilities is associated with a lower average number of selected covariates, and hence a lower average FPR. On the other hand, the other procedures tend, on average, to select more covariates in the models with parameter instabilities and hence have a higher average FPR relative to the models without parameter instabilities. Lastly, OCMT most of the times selects fewer covariates relative to Lasso, A-Lasso, and boosting , while maintaining the TPR at a similar level. As a result, OCMT has mostly the lowest average FPR among the selection methods under consideration. Summary Tables (ref) and (ref) in the online MC supplement provide further results on the effects of down-weighting on TPR and FPR. The results consistently show that down-weighting of observations provides no gains for OCMT in terms of average TPR and FPR. This is also true for other methods in majority but not all cases.

Table (ref) focusses on the one-step-ahead MSFEs and provides comparative results on the effects of down-weighting across the methods (OCMT, Lasso, A-Lasso and boosting). As in Table (ref), Panel A of Table (ref) gives average MSFEs for the four experiments without parameter instabilities, and Panel B gives the corresponding results for the experiments with parameter instabilities. As expected, in the absence of parameter instabilities, using unweighted observations gives the lowest MSFE across all the methods. Moreover, for all $N$ and $T$ combinations and different down-weighting scenarios, the average MSFE of each method is lower in the case of models with stable parameters as compared to those with parameter instabilities. This observation is consistent with our finding in Theorem (ref) about the cost of time-variation in the coefficients on the in-sample fit of the estimated model. As can be seen, for models with parameter instabilities, down-weighting does improve the forecasting performance of OCMT (with and without down-weighting in the selection stage), Lasso, and A-Lasso. However, by comparing the MSFEs of OCMT with and without down-weighting at the selection stage, we see that the down-weighting at the selection stage always results in deterioration of the forecast accuracy of OCMT, which is in line with our main theoretical result. Last but not least, the results in Table (ref) show that OCMT with down-weighting only at the estimation stage almost always has the lowest average MSFE among all the methods for all choices of $N$, $T$, and different down-weighting scenarios. In fact, in the case of experiments with parameter instabilities OCMT with down-weighting (light or heavy) at the estimation stage only, always beats Lasso, A-Lasso and boosting with light or heavy down-weighting in terms of the one-step-ahead MSFE.

Table (ref) compares the performance of OCMT with the down-weighting option at the estimation stage to that of the other procedures, using the same set of down-weighting parameter ($\lambda $). Specifically, we report the MSFE of Lasso, A-Lasso, and boosting relative to that of OCMT. Since the relative MSFE ranking of OCMT, Lasso, A-Lasso, and boosting does not appear to be affected by no/light/heavy down-weighting options, as a summary measure, we simply average relative MSFE values across individual experiments and the three (no/light/heavy) down-weighting options. However, we provide the relative MSFE results for the models without and with parameter instabilities separately, on left and right panels of Table (ref). Two observations stand out from this table. First, the reported average relative MSFEs are almost always greater than one for all the $N$ and $T$ choices, indicating that OCMT outperforms Lasso, A-Lasso, and boosting. Second, the degree to which OCMT outperforms Lasso and A-Lasso tends to increase with the degree of parameter instability. This is less so if we compare OCMT\ with boosting.

Tables (ref), (ref), and (ref) in the online MC supplement provide further details about the performance of the methods under consideration in static and dynamic experiments. In Table (ref), we compare the number of selected covariates, the TPR, and the FPR of each method without down-weighting across static and dynamic models. For various $N$ and $T$ combinations the reported results are averaged across four experiments (with/without parameter instabilities and with/without high-fit). The results show that all the methods tend to select fewer covariates in the dynamic models relative to the static ones, and hence have a lower TPR and FPR. This is expected, as in the dynamic models, part of the variation in the target variable is explained by its own lag rather than the signal variables. Consequently, in Tables (ref) and (ref), which are about the MSFE in static and dynamic models, respectively, we see that all the methods have a higher MSFE in dynamic models relative to the static ones. Additionally, the results in Tables (ref) and (ref) show that the MSFE for models with stable parameters is always lower than the ones with parameter instabilities, regardless of whether the model is static or not.

Overall, the results of our MC studies suggest that the OCMT procedure without down-weighting at the selection stage is a useful method to deal with variable selection in linear regression settings with parameter instability.

Empirical applications

The rest of the paper considers empirical applications whereby the forecast performance of the proposed OCMT approach with no down-weighting at the selection stage is compared with those of Lasso and A-Lasso. In particular, we consider the following two applications:\footnote{ We also consider forecasting euro area quarterly output growth using the European Central Bank (ECB) survey of professional forecasters as our third application. The results of this application can be found in Section (ref) of the online empirical supplement.}

In each application, we first compare the performance of OCMT with and without down-weighted observations at the selection stage. We then consider the comparative performance of OCMT (with variable selection carried out without down-weighting) relative to Lasso and A-Lasso, with and without down-weighting. For down-weighting we make use of exponentially down-weighted observations, namely $\hat{x}_{it}(\lambda )=\lambda ^{T-t}x_{it}$, and $\hat{y}_{t}(\lambda )=\lambda ^{T-t}y_{t}$, where $y_{t}$ is the target variable to be forecasted, $x_{it}$, for $i=1,2,...,N$ are the covariates in the active set, and $\lambda $ is the exponential decay coefficient. We consider the same two sets of values for the degree of exponential decay, $\lambda $, as in the MC section: (1) Light down-weighting with $\lambda =\left\{ 0.975,0.98,0.985,0.99,0.995,1\right\} $ , and (2) Heavy down-weighting with $\lambda =\left\{ 0.95,0.96,0.97,0.98,0.99,1\right\} $. For each of the above two sets of exponential down-weighting schemes we again focus on simple average forecasts computed over the individual forecasts obtained for each value of $ \lambda $ in the set under consideration.

For forecast evaluation we consider Mean Squared Forecasting Error (MSFE) and Mean Directional Forecast Accuracy (MDFA), together with related pooled versions of\ Diebold-Mariano (DM), and Pesaran-Timmermann (PT) test statistics. A panel version of diebold2002comparing test is proposed by pesaran2009forecasting. Let $q_{lt}\equiv e_{ltA}^{2}-e_{ltB}^{2}$ be the difference in the squared forecasting errors of procedures $A$ and $B$ , for the target variable $y_{lt}$ ($l=1,2,...,L)$ and $t=1,2,...,T_{l}^{f}$ , where $T_{l}^{f}$ is the number of forecasts for target variable $l$ (could be one or multiple step ahead) under consideration. Suppose $ q_{lt}=\alpha _{l}+\varepsilon _{lt}$ with $\varepsilon _{lt}\sim \mathcal{N} (0,\sigma _{l}^{2})$. Then under the null hypothesis of $H_{0}:\alpha _{l}=0$ for all $l$ we have

Note that $V(\bar{q})$ needs to be modified in the case of multiple-step ahead forecast errors, due to the serial correlation that results in the forecast errors from the use of over-lapping observations. There is no adjustment needed for one-step ahead forecasting, since it is reasonable to assume that in this case the loss differentials are serially uncorrelated. However, to handle possible serial correlation for $h$-step ahead forecasting with $h>1$, we can modify the panel DM test by using the Newey-West type estimator of $\sigma _{l}^{2}$.

The $MDFA$ statistic compares the accuracy of forecasts in predicting the direction (sign) of the target variable, and is computed as

where $\mathbf{1}(w>0)$ is the indicator function takes the value of $1$ when $w>0$ and zero otherwise, $\text{sgn}(w)$ is the sign function, $y_{lt}$ is the actual value of dependent variable at time $t$ and $y_{lt}^{f}$ is its corresponding predicted value. To evaluate statistical significance of the directional forecasts for each method, we also report a pooled version of the test suggested by pesaran1992simple:

where $\hat{P}$ is the estimator of the probability of correctly predicting the sign of $y_{lt}$, computed by

Finally,$\ \hat{V}(\hat{P})=T_{Lf}^{-1}\hat{P}^{\ast }(1-\hat{P}^{\ast }),$ and

The last term of $\hat{V}(\hat{P^{\ast }})$ is negligible and can be ignored. Under the null hypothesis, that prediction and realization are independently distributed, PT is asymptotically distributed as a standard normal distribution.

Forecasting monthly returns of stocks in Dow Jones

In this application the focus is on forecasting one-month ahead stock returns, defined as monthly change in natural logarithm of stock prices. We consider stocks that were part of the Dow Jones index in 2017m12, and have non-zero prices for at least 120 consecutive data points (10 years) over the period 1980m1 and 2017m12. We ended up forecasting 28 blue chip stocks. \footnote{ Visa and DowDuPont are excluded since they have less than 10 years of historical price data.} Daily close prices for all the stocks are obtained from Data Stream. For stock $i$, the price at the last trading day of each month is used to construct the corresponding monthly stock prices, $P_{it}$. Finally, monthly returns are computed by $r_{i,t+1}=100\ln(P_{i,t+1}/P_{it}) $, for $i=1,2,...,28$. For all 28 stocks we use an expanding window starting with the observations for the first 10 years ($T=120$). The active set for predicting $r_{i,t+1}$ consists of 40 financial, economic, and technical variables.\footnote{ All regressions include the intercept as the only conditioning (pre-selected) variable.} The full list and the description of the indicators considered can be found in Section (ref) of online empirical supplement.

Overall we computed 8,659 monthly forecasts for the 28 target stocks. The results are summarized as average forecast performances across the different variable selection procedures. Table (ref) reports the effects of down-weighting at the selection stage of the OCMT procedure. It is clear that down-weighting worsens the predictive accuracy of OCMT. From the Panel DM tests, we can also see that down-weighting at the selection stage worsens the forecasts significantly. Panel DM test statistics is -5.606 (-11.352) for light (heavy) versus no down-weighing at the selection stage. Moreover, Table (ref) shows that the OCMT procedure with no down-weighting at the selection stage dominates Lasso, A-Lasso and boosting in terms of MSFE and the differences are statistically highly significant.

Further, OCMT outperforms Lasso, A-Lasso and boosting in terms of Mean Directional Forecast Accuracy (MDFA), measured as the percent number of correctly signed one-month ahead forecasts across all the 28 stocks over the period 1990m2-2017m12. See Table (ref). As can be seen from this table, OCMT\ with no down-weighting performs the best; correctly predicting the direction of 56.057% of 8,659 forecasts, as compared to 55.769%, which we obtain for Lasso, A-Lasso and boosting forecast, at best. This difference is highly significant considering the very large number of forecasts involved. It is also of interest that the better of performance of OCMT is achieved with a much fewer number of selected covariates as compared to Lasso, A-Lasso and boosting. As can be seen from the last column of Table (ref), Lasso, A-Lasso and boosting on average select many more covariates than OCMT (1-15 variables as compared to 0.072 for OCMT).

So far we have focused on average performance across all the 28 stocks. Table (ref) provides the summary results for individual stocks, showing the relative performance of OCMT in terms of the number of stocks, using MSFE and MDFA criteria. The results show that OCMT performs better than Lasso, A-Lasso and boosting in the majority of the stocks in terms of MSFE and MDFA. OCMT outperforms Lasso, A-Lasso and boosting in at least 22 out of 28 stocks in terms of MSFE, under no down-weighting, and almost universally when Lasso, A-Lasso and boosting are implemented with down-weighting. Similar results are obtained when we consider MDFA criteria, although the differences in performance are somewhat less pronounced. Overall, we can conclude that the better average performance of OCMT (documented in Tables (ref) and (ref)) is not driven by a few stocks and holds more generally.

Forecasting quarterly output growth rates across 33 countries

We consider one and two years ahead predictions of output growth for 33 countries (20 advanced and 13 emerging). We use quarterly data from $1979Q2$ to $2016Q4$ taken from the GVAR dataset.\footnote{ The GVAR dataset is available at \url{https://sites.google.com/site/gvarmodelling/data}.} We predict $ \Delta_{4}y_{it}=y_{it}-y_{i,t-4}$, and $\Delta _{8}y_{it}=y_{it}-y_{i,t-8},$ where $y_{it}$, is the log of real output for country $i$. We adopt the following direct forecasting equations:

where we consider $h=4$ (one-year-ahead forecasts) and $h=8$ (two-years-ahead forecasts). Given the known persistence in output growth, in addition to the intercept in the present application we also condition on the most recent lagged output growth, denoted by $ \Delta_{1}y_{it}=y_{it}-y_{i,t-1}$, and confine the variable selection to list of variables set out in Table (ref) in the online empirical supplement. Overall, we consider a maximum of 15 covariates in the active set covering quarterly changes in domestic variables such as real output growth, real short term interest rate, and long-short interest rate spread and quarterly change in the corresponding foreign variables.

We use expanding samples, starting with the observations on the first 15 years (60 data points), and evaluate the forecasting performance of the three methods over the period 1997Q2 to 2016Q4.

Tables (ref) and (ref), respectively, report the MSFE of OCMT for one-year and two-year ahead forecasts of output growth, with and without down-weighting at the selection stage. Consistent with the previous application, down-weighting at the selection stage worsens the forecasting accuracy. Moreover, in Tables (ref) and (ref), we can see that OCMT (without down-weighting at the selection stage) outperforms Lasso, A-Lasso and boosting in two-year ahead forecasting. In the case of one-year ahead forecasts, OCMT and Lasso are very close to each other and both outperform A-Lasso and boosting. Table (ref) summarizes country-specific MSFE and DM findings for OCMT relative to Lasso, A-Lasso and boosting. The results show OCMT under-performs Lasso in more than half of the countries for one-year ahead horizon, but outperforms Lasso, A-Lasso and boosting in more than 70 percent of the countries in the case of two-year ahead forecasts. It is worth noting that while Lasso generally outperforms OCMT in the case of one-year ahead forecasts, overall its performance is not statistically significantly better. See Panel DM test of Table (ref). On the other hand we can see from Table (ref) that overall OCMT significantly outperforms Lasso in the case of the two-year ahead forecasts.

Finally in Tables (ref) and (ref) we reports MDFA and PT test statistics for OCMT, Lasso, A-Lasso and boosting. Overall, OCMT has a slightly higher MDFA and hence predicts the direction of real output growth better than Lasso, A-Lasso and boosting in most cases. The PT test statistics suggest that while all the methods perform well in forecasting the direction of one-year ahead real output growth, none of the methods considered are successful at predicting the direction of two-year ahead output growth.

It is also worth noting that as with the previous applications, OCMT selects very few variables from the active set (0.1 on average for both horizons, with the maximum number of selected variables being 2 for $h=4$ and $8$). On the other hand, Lasso on average selects 2.7 variables from the active set for $h=4$, and $1$ variable on average for $h=8$. Maximum number of variables selected by Lasso is $9$ and $13$ for $h=4$, $8$, respectively (out of possible $15$). Again as to be expected, A-Lasso selects a fewer number of variables as compared to Lasso (2.3 and 0.8 on average for $h=4,8$ , respectively), but this does not lead to a better forecast performance in comparison with Lasso. Boosting on average selects $2.7$ variables from the active set for $h=4$, and $1.4$ variables on average for $h=8$.

In conclusion, down-weighting at both selection and forecasting stages deteriorates OCMT's MSFE for both one--year and two-years ahead forecast horizons, as compared to down-weighting only at the forecasting stage. Moreover, light down-weighting at the forecasting stage improves forecasting performance for both horizons. Statistically significant evidence of forecasting skill is found for OCMT relative to Lasso only in the case of two-years ahead forecasts. However, it is interesting that none of the big data methods can significantly beat the simple (light down-weighted) AR(1) baseline model.

Concluding remarks

The penalized regression approach has become the de facto benchmark in the literature on variable selection in the context of linear regression models. But, barring a few exceptions (such as kapetanios2018time ), these studies focus on models with stable parameters, and do not consider the implications of parameter instabilities for variable selection. Recently, chudik2018one proposed OCMT as an alternative procedure to penalized regression. One feature of the OCMT procedure is the fact that the problem of variable selection is separated from the forecasting stage, in contrast to the penalized regression techniques where the variable selection and estimation are carried out simultaneously. Using OCMT one can decide whether to use the weighted observations at the variable selection stage or not, without preempting whether to down-weight and how to down-weight the observations at the forecasting stage.

We have provided theoretical arguments for using the unweighted observations at the selection stage of OCMT, and down-weighted observations at the forecasting stage of OCMT. Our MC results as well as empirical applications uniformly suggest that OCMT without down-weighting at the selection stage outperforms, in terms of mean squared forecast errors, Lasso, Adaptive Lasso, boosting, as well as when OCMT is applied with down-weighted observations.

{\ { \setstretch{1.15} }}

\setcounter{section}{0} \setcounter{table}{0} \setcounter{page}{1} \setcounter{equation}{0} \setcounter{proposition}{0} \setcounter{corollary}{0} \setcounter{proofcorollary}{0} \setcounter{proofproposition}{0}

\quad

This online theory supplement has three sections. Section (ref) provides the proofs of Theorems 1 to 3, and additional propositions and corollaries. Section (ref) establishes the main lemmas needed for the proof of the theorems in Section (ref). Section (ref) contains the complementary lemmas needed for the proofs of the main lemmas in the previous section.

Notations: Generic finite positive constants are denoted by $C_{i}$ for $i=1,2,\cdots $ and $c $. They can take different values in different instances. $\lVert \mathbf{A}\rVert _{2}$, $\lVert \mathbf{A}\rVert _{F}$, $ \lVert \mathbf{A}\rVert _{\infty}$ and $\lVert \mathbf{A}\rVert _{1}$ denote the spectral, Frobenius, row, and column norms of matrix $\mathbf{A}$, respectively. $\lambda_{i}(\mathbf{A}) $ denotes the $i^{th} $ eigenvalue of a square matrix $\mathbf{A} $. $\text{tr}(\mathbf{A})$ and $\text{det}( \mathbf{A})$ are the trace and determinant of a square matrix $\mathbf{A}$, respectively. $\left\| \mathbf{x} \right\| $ denotes the $\ell_{2} $ norm of vector $\mathbf{x} $. If $\{f_{n}\}_{n=1}^{\infty }$ is any real sequence and $\{g_{n}\}_{n=1}^{\infty }$ is a sequence of positive real numbers, then $f_{n}=O(g_{n})$, if there exists a positive constant $C_{0}$ and $n_0 $ such that $\lvert f_{n}\rvert /g_{n}\leq C_{0}$ for all $n > n_0$. $ f_{n}=o(g_{n})$ if $f_{n}/g_{n}\rightarrow 0$ as $n\rightarrow \infty $. If $ \{f_{n}\}_{n=1}^{\infty }$ and $\{g_{n}\}_{n=1}^{\infty }$ are both positive sequences of real numbers, then $f_{n}=\ominus (g_{n})$ if there exist $ n_{0}\geq 1$ and positive constants $C_{0}$ and $C_{1}$, such that $ \inf_{n\geq n_{0}}\left( f_{n}/g_{n}\right) \geq C_{0}$ and $\sup_{n\geq n_{0}}\left( f_{n}/g_{n}\right) \leq C_{1}$. If $\{f_{n}\}_{n=1}^{\infty }$ is a sequence of random variables and $\{g_{n}\}_{n=1}^{\infty }$ is a sequence of positive real numbers, then $f_{n}=O_p(g_{n})$, if for any $ \varepsilon > 0 $, there exists a positive constant $B_{\varepsilon}$ and $ n_{\varepsilon} $ such that $\Pr \left(\lvert f_{n}\rvert > g_{n} B_{\varepsilon} \right) < \varepsilon $ for all $n > n_{\varepsilon}$.

Proof of the Theorems

This section provides the proofs of Theorems 1 to 3. The proofs are based on lemmas presented in Section (ref). Among these, Lemmas (ref) and (ref) are key. For each covariate $i=1,2,\cdots ,N$, Lemmas (ref) establishes exponential probability inequalities for the t-ratio multiple tests conditional on the average net effect, $\bar{\theta}_{i,T}$, being either of the order $\ominus (T^{-\varepsilon _{i}})$ for some $\varepsilon _{i}>1/2$, or of the order $ \ominus (T^{-\vartheta _{i}})$, for some $0\leq \vartheta _{i}<1/2$. For DGP given by ((ref)), Lemma (ref) provides asymptotic properties of LS estimator of coefficients and SSR of a regression model that includes all the signals and pseudo-signals. This lemma establishes that the coefficients of pseudo-signals estimated by LS converges to zero so long as $k_{T}^{\ast }=\ominus (T^{d})$ grows at a slow rate relative to $T$ , i.e. $0\leq d<1/2$. This lemma also shows that the SSR of the regression model converges to that of the oracle model, which includes only the signals.

Proof of Theorem (ref)

{\ To establish result ((ref)), first note that $ \mathcal{A}_{0}^{c}=\mathcal{H}^{c}\cup \mathcal{G}^{c}$ and hence ($ \mathcal{H}^{c}$ denotes the complement of $\mathcal{H}$)

where $\mathcal{H}$ and $\mathcal{G}$ are given by ((ref)). We also have $\mathcal{H}^{c}=\{\sum_{i=1}^{k}\hat{\mathcal{J}}_{i}<k\}$ and $ \mathcal{G}^{c}=\{\sum_{i=k+k_{T}^{\ast }+1}^{N}\hat{\mathcal{J}}_{i}>0\}$. Let's consider $\Pr \left( \mathcal{H}^{c}\right) $ and $\Pr \left( \mathcal{ \ G}^{c}\right) $ in turn. We have $\textstyle\Pr (\mathcal{H}^{c})\leq \sum_{i=1}^{k}\Pr (\hat{\mathcal{J}} _{i}=0)$. But for any signal

where $0\leq \vartheta _{i}<1/2$ and hence by Lemma (ref), we can conclude that there exist sufficiently large finite positive constants $ C_{0}$ and $C_{1}$ such that $\textstyle\Pr (\hat{\mathcal{J}}_{i}=0)=O\left[ \exp (-C_{0}T^{C_{1}}) \right] $. Since by Assumption (ref), the number of signals is finite we can further conclude that

for some finite positive constants $C_{0}$ and $C_{1}$. In the next step note that

But for any noise variable $\textstyle\Pr (\hat{\mathcal{J}}_{i}=1)=\Pr \left[ \lvert t_{i,T}\rvert >c_{p}(N,\delta )|\bar{\theta}_{i,T}=\ominus (T^{-\epsilon _{i}})\right] ,$ where $\epsilon _{i}\geq 1/2$ and hence by Lemma (ref), we can conclude that there exist sufficiently large finite positive constants $C_{0}$ and $C_{1}$ such that for any $0<\pi <1$, $\textstyle\Pr (\hat{\mathcal{J}}_{i}=1)\leq \exp \left[ -\frac{(1-\pi )^{2}\bar{\sigma}_{\eta _{i},T}^{2}\bar{\sigma}_{x_{i},T}^{2}c_{p}^{2}(N, \delta )}{2\bar{\omega}_{iy,T}^{2}(1+d_{T})^{2}}\right] +\exp (-C_{0}T^{C_{1}})$, in which $\bar{\sigma}_{x_{i},T}^{2}=T^{-1}\sum_{t=1}^{T} \mathbb{E}(x_{it}^{2})$, $\bar{\omega}_{iy,T}^{2}=T^{-1}\sum_{t=1}^{T} \mathbb{E}(x_{it}^{2}y_{t}^{2}|\mathcal{F}_{t-1})$, $\bar{\sigma}_{\eta _{i},T}^{2}=T^{-1}\sum_{t=1}^{T}\mathbb{E}(\eta _{it}^{2})$, }$\eta _{it}=y_{t}-\phi _{i,T}x_{it}$, and $\phi _{i,T}$ is defined in ((ref)). {Therefore,

where $\mathcal{X}_{NT}=\inf_{i\in \{k+k^{\ast }+1,k+k^{\ast }+2,\cdots ,N\}} \frac{\bar{\sigma}_{\eta _{i},T}^{2}\bar{\sigma}_{x_{i},T}^{2}}{\bar{\omega} _{iy,T}^{2}}$. By result (II) of Lemma (ref) in online theory supplement we can further conclude that for any $0<\pi <1$,

Using ((ref)) and ((ref)) in ((ref)), we obtain $\Pr (\mathcal{A} _{0}^{c})=O\left( N^{1-\mathcal{X}_{NT}\left( \frac{1-\pi }{1+d_{T}}\right) ^{2}\delta }\right) +O\left[ N\exp (-C_{0}T^{C_{1}})\right] $ and $\Pr ( \mathcal{A}_{0})=1-O\left( N^{1-\mathcal{X}_{NT}\left( \frac{1-\pi }{1+d_{T}} \right) ^{2}\delta }\right) -O\left[ N\exp (-C_{0}T^{C_{1}})\right] $, which completes the proof. }

Proof of Theorem (ref)

For any $B>0$,

Since $\Pr \left( T^{\frac{1-d}{2}}\left\Vert \hat{\boldsymbol{\gamma }}_{T}- \boldsymbol{\gamma }_{T}^{\ast }\right\Vert >B|\mathcal{A}_{0}^{c}\right) $ and $\Pr \left( \mathcal{A}_{0}\right) $ are less than or equal to one, we can further write,

By conditioning on $\mathcal{A}_{0}$ the dimension of vector $\hat{ \boldsymbol{\gamma }}_{T}$ is at most equal to $k+k_{T}^{\ast}$ and by assumption $k_{T}^{\ast}=\ominus (T^{d})$ where $0\leq d<1/2$. Therefore, by Lemma (ref) in online theory supplement, conditional on $\mathcal{A} _{0} $, $\left\Vert \hat{\boldsymbol{\gamma }}_{T}-\boldsymbol{\gamma } _{T}^{\ast }\right\Vert $ is $O_{p}\left( T^{\frac{d-1}{2}}\right) $. By Theorem 1, we also have $\lim_{T\rightarrow \infty }\Pr \left( \mathcal{A} _{0}^{c}\right) =0$. Hence, for any $\varepsilon >0$, there exists $ B_{\varepsilon }>0$ and $T_{\varepsilon }>0$ such that

Therefore, $\Pr \left( T^{\frac{1-d}{2}}\left\Vert \hat{\boldsymbol{\gamma }} _{T}-\boldsymbol{\gamma }_{T}^{\ast }\right\Vert >B_{\varepsilon }\right) <\varepsilon \text{ for all }T>T_{\varepsilon }, $ and we conclude that

as required. Similar lines of arguments can be used to show that if $\mathbb{ E}\left( \mathbf{x}_{\tilde{k}_{T},t}\mathbf{x}_{\tilde{k}_{T},t}^{\prime }\right) $ is a fixed time-invariant matrix, then $\left\Vert \hat{\boldsymbol{\gamma }}_{T}-\boldsymbol{\gamma } _{T}^{\diamond }\right\Vert =O_{P}\left( T^{\frac{d-1}{2}}\right)$, which completes the proof.

Proof of Theorem (ref)

Let $D_{T}=T^{-1}\sum_{t=1}^{T}\hat{\eta}_{t}^{2}-\left(\bar{\Delta} _{\beta,T} +\bar{\sigma}_{u,T}^{2}\right) $. For any $B>0$,

Since $\Pr \left( T^{\frac{1}{2}}\left\vert D_{T}\right\vert >B|\mathcal{A} _{0}^{c}\right) $ and $\Pr \left( \mathcal{A}_{0}\right) $ are less than or equal to one, we can further write,

By conditioning on $\mathcal{A}_{0}$, the number of selected covariates is at most equal to $k+k_{T}^{\ast }$ and by assumption $k_{T}^{\ast }=\ominus (T^{d})$, where $0\leq d<1/2$. Therefore, by Lemma (ref) in online theory supplement, conditional on $\mathcal{A}_{0}$, $D_{T}$ is $O_{p}\left( T^{- \frac{1}{2}}\right) $. By Theorem 1, we also have $\lim_{T\rightarrow \infty }\Pr \left( \mathcal{A}_{0}^{c}\right) =0$. Hence, for any $ \varepsilon >0$, there exists $B_{\varepsilon }>0$ and $T_{\varepsilon }>0$ such that $\Pr \left( T^{\frac{1}{2}}\left\vert D_{T}\right\vert >B_{\varepsilon }| \mathcal{A}_{0}\right) +\Pr \left( \mathcal{A}_{0}^{c}\right) <\varepsilon, \text{ for all }T>T_{\varepsilon }$. Therefore, $\Pr \left( T^{\frac{1}{2}}\left\vert D_{T}\right\vert >B_{\varepsilon }\right) <\varepsilon \text{ for all }T>T_{\varepsilon }, $ and we conclude that

Furthermore, by Lemma (ref), $\bar{\Delta}_{\beta,T} $ is non-negative. Following similar lines of argument we get that if $\mathbb{E} \left( \mathbf{x}_{\tilde{k}_{T},t}\mathbf{x}_{\tilde{k}_{T},t}^{\prime }\right) $ is a fixed time-invariant matrix, then,

with $\bar{\Delta}_{\beta,T}^{\ast} \geq 0 $ which completes the proof.

Propositions and corollaries

Proof of propositions and corollaries

Since $\pi $ is an arbitrary constant between zero and one, result ((ref)) of Corollary (ref) implies that for any $\delta >1$, we can select an approximating model with probability approaching one as $N$ and $T$ grows to infinity.

Main lemmas