Regression Model Selection Under General Conditions

Regression Model Selection Under General Conditions

1\quadIntroduction

Model selection criteria are one of the most important tools in statistics and are used for a variety of things such as model selection, model comparison, model averaging, or to evaluate the forecasting ability of models. Model selection criteria are used in a wide range of models from parametric, semiparametric, to nonparametric. Though model selection criteria have been around for decades and are a lynchpin in statistics, the behavior of these methods are complex and not fully understood Bates2024. Asymptotic optimality of these criteria (their ability to select the true model or the best approximating model if the true model is not in the set), have only been shown for certain types of models in a limited number of situations.

Model selection criteria can be broken down into three major classes: Cross Validation (CV), information criteria, and pseudo-out-of-sample forecasting.\footnote{Bayesian model selection via the marginal likelihood is a class not discussed in this paper in part because Bayesian and frequentist notions of model selection do not necessarily agree Moreno2015 (see Chib2016 and references therein for more on model selection consistency in the Bayesian case). It is common for forecasting models estimated using Bayesian methods to use frequentist model selection criteria such as pseudo-out-of-sample forecasting.}\textsuperscript{,}\footnote{General-to-specific and specific-to-general testing are not a classes discussed in this paper. Though they have some popularity, unlike the other major classes they lack wide applicability (e.g. you cannot use it to find the optimal tuning parameters or things not directly involving variable selection).} Though CV is considered by many to be the gold standard for model selection, asymptotic optimality results for CV have been limited. Li1987 shows that leave-one-out Cross-Validation (LOO CV) is asymptotically optimal for a class of linear smoother models under the assumption that the data are i.i.d. Andrews1991c extends the results to handle regressions with independent but heteroskedastic residuals. Extending these results to time series, however, has been spotty at best. Results for CV either assume strict exogeneity, which is well known to be an unrealistic assumption for time series Stock2007, or if they allow for lagged dependent variables, the assumptions for these models end up being highly restrictive such as requiring the residuals to be i.i.d. which rules out models with heteroskedasticity of any kind as well as models with autocorrelated residuals (see for example Zhang2013,Sun2021).\footnote{This rules out most time series models which include but are not limited to: Local Projections and direct forecast regressions Jorda2005 and Vector Autoregressions or autoregressive distributed lag models with heteroskedasticity of any kind.} As noted in Hansen2012, extending the proofs to time-series regression would be quite challenging.\footnote{To my knowledge there have been scant results for panel data with Gao2016,Yu2025 being exceptions.} Moving away from linear smoothers, assumptions are even more restrictive and things covered are not that broad.\footnote{Chetverikov2021 show the validity of $k$-fold cross validation for Lasso in high dimensions assuming the errors are i.i.d. and neither sub-Gaussian nor sub-exponential. As noted in their paper, the results do not cover LOO CV.} Popular methods that are not covered at all or only covered under restrictive conditions include but are not limited to: Lasso and its variants (adaptive lasso, elastic net, square root lasso, etc.), bridge estimators more generally, least angle regression (LARS) and other stepwise methods, quantile regressions, partially linear regression, nonlinear regressions, generalized linear models, neural networks, regression trees, ensemble methods, and other machine learning methods.

Asymptotic optimality of information criteria such as AIC and BIC have typically only been shown under more stringent assumptions such as i.i.d. data or i.i.d. residuals (see for example Shao1997,Claeskens2016,Ding2018 for extensive reviews). An exception is Sin1996 who consider information criteria for dependent processes, but they do not account for nonparametric or time-varying parameter models. The paper also assumes restrictive conditions such as the models being finite dimensional as well as continuous differentiability of the log quasi-likelihood function which rules out: quantile regressions, many penalized regressions (such as lasso, its variants, most bridge estimators, and trend filtering), robust regressions (Huber loss functions), LARS and other stepwise methods, and regression trees just to name a few. Sin1996 acknowledge that generalizing their results to a wider class of data generating processes and models represents an interesting and challenging area for future research, but general progress has not been made.\footnote{Sin1996 show asymptotic optimality by showing the model selection criteria choose the model with lowest Kullback-Leibler divergence from the data generating process. In this paper asymptotic optimality is demonstrated by showing model selection criteria choose the model with the lowest integrated mean squared error. Sin1996 also prove strong consistency of certain information criteria. Strong consistency arguments are not attempted in this paper. See the discussion in section 4 for the reasoning.}

Real-time out of sample forecasting exercises (also known as pseudo-out-of-sample forecasting) in economics and finance are regarded by many researchers as the \textquotedblleft ultimate test of a forecasting model\textquotedblright in time series (see Stock2007 page 571).\footnote{This is particularly the case in economics and finance. For example, central banks routinely evaluate their forecasting models using pseudo-out-of-sample exercises Faust2013, and asset pricing studies typically require pseudo-out-of-sample evidence for predictability claims to be taken seriously (see for example Welch2008). The preference for pseudo-out-of-sample testing reflects the high cost of forecast errors in policy and investment decisions.} These methods are the standard way to choose/evaluate forecast models. Despite the influx of several new and more complicated methods over the past few decades (e.g. penalized regression methods, model averaging, machine learning methods, etc.), asymptotic optimality of pseudo-out-of-sample forecast methods such as rolling window and recursive methods, are generally limited to least square estimators or standard time-varying parameter models (see Rossi2021 and references therein).

Currently, researchers spend entire papers or substantial parts of papers showing asymptotic optimality of a selection procedure, if it is shown at all, for a limited class of estimators under restrictive assumptions. In this paper, I derive the asymptotic optimality of model selection criteria under fairly general conditions. The proofs do not rely on a specific estimation method and encompass a wide array of data generating processes. Section 2 reviews CV, information criteria, and pseudo-out-of-sample forecasting. Section 3 presents the main proofs for CV, information criteria, and pseudo-out-of-sample forecasting. Section 4 provides implications and a broader discussion of results in the literature. Section 5 concludes.

Some notation: $\xrightarrow{p}$ is converges in probability, $\parallel\cdot\parallel$ is the Frobenius/Euclidean norm, $O_{p}(\cdot)$ is big O probability notation, and $o_{p}(\cdot)$ is little o probability notation.

2\quadPreliminaries for Cross Validation, Information Criteria, and Pseudo-Out-Of-Sample Forecasting

Let the true model be \[ y_{i}=\mu_{i}+\varepsilon_{i}\;for\;i=1,2,\ldots,T, \] where $y_{i}$ is the dependent variable, $\mu_{i}$ is the conditional mean, and $\varepsilon_{i}$ is the residual. In the special case of linear regressions \[ y_{i}=x_{i}\beta+\varepsilon_{i}\;for\;i=1,2,\ldots,T, \] where $x_{i}$ is a $1\times p$ vector of regressors, $\beta$ is a $p\times1$ vector of regression coefficients. The Leave-one-out (LOO) residual for a regression model is calculated by estimating $\beta$ for all but one observation, and predicting the residual for the left out observation. More formally for observation $i$, \[ \widetilde{\beta}_{-i}=(\sum_{j\neq i}x_{j}'x_{j})^{-1}\sum_{j\neq i}x_{j}'y_{j},\quad\widetilde{\mu}_{-i}=x_{i}\widetilde{\beta}_{-i},\quad\widetilde{\varepsilon}_{i}=y_{i}-\widetilde{\mu}_{-i}, \] where $\widetilde{\beta}_{-i}$ is the leave $i$ out OLS estimate of $\beta$, $\widetilde{\mu}_{-i}$ is the leave $i$ out estimate of $\mu_{i}$, and $\widetilde{\varepsilon}_{i}$ is the leave $i$ out estimate of $\varepsilon_{i}$. LOO CV is calculated by finding the model that minimizes the mean-squared error (MSE) of the LOO residuals, that is, by finding the model that minimizes $T^{-1}\sum_{i=1}^{N}\widetilde{\varepsilon}_{i}^{2}$, though other loss functions such as the absolute value can be used.

LOO CV was developed and is mainly used for independent data. It is well known that LOO CV can perform poorly in finite samples when used for dependent data due to the residuals and regressors being autocorrelated. This can lead to overfitting or underfitting Opsomer2001,Arlot2010,Hansen2010. One solution is to use block CV methods Burman1994. Block methods work the same way as the LOO method, but instead of leaving out just observation $i$, one would additionally leave out $h$ observations on both sides of $i$ to break up the dependence. More formally, one would leave out observations $i-h,i-h+1,\ldots,i,\ldots i+h-1,i+h$, so for $h$-block CV via OLS estimation \[ \widetilde{\beta}_{-i}=(\sum_{j\neq i-h:i+h}x_{j}'x_{j})^{-1}\sum_{j\neq i-h:i+h}x_{j}'y_{j}. \] Though fairly popular, the block methods have not been asymptotically justified.

Another major strand of model selection criteria are information criteria. They generally take the form of \[ log\Bigg(\frac{1}{T}\sum_{i=1}^{T}\{y_{i}-\hat{\mu}_{i}\}^{2}\Bigg)+\frac{\lambda_{T}p}{T} \] where in the standard linear regression case $\hat{\mu}_{i}=x_{i}\hat{\beta}$, $\hat{\beta}$ is the estimate of $\beta$ based on all of the data, $p$ is the number of predictors and $\lambda_{T}>0$ is the penalty coefficient. For the AIC $\lambda_{T}=2$, for the BIC $\lambda_{T}=log(T)$, for HQIC $\lambda_{T}=clog(log(T))$ where $c>2$, for the RIC $\lambda_{T}=2log(p)$, etc. Models are chosen by choosing the model that minimizes the information criteria of choice.

The last strand of model selection criteria we will discuss is pseudo-out-of-sample forecasting. The two most popular methods are rolling window forecast and recursive forecasts Clark2009a.\footnote{In the least squares case, recursive forecasts are also known as predictive least squares Wei1992.} In the standard linear regression case estimated using OLS, rolling window forecasts estimate a model on the last $R$ observations so that \[ \ddddot{\beta_{i}}=(\sum_{j=i-R}^{i-1}x_{j}'x_{j})^{-1}\sum_{j=i-R}^{i-1}x_{j}'y_{j},\quad\ddddot{\mu_{i}}=x_{i}\ddddot{\beta_{i}},\quad and\quad\ddddot{\varepsilon_{i}}=y_{i}-\ddddot{\mu_{i}}. \] Recursive models alternatively estimate the model based on all observations up to that point with \[ \ddddot{\beta_{i}}=(\sum_{j=1}^{i-1}x_{j}'x_{j})^{-1}\sum_{j=1}^{i-1}x_{j}'y_{j}. \] The optimal rolling window and recursive forecast models are chosen by choosing the model that minimizes the MSE of the forecasted residuals. Note that even though the above examples estimate the standard linear regression model using OLS, the proofs of asymptotic optimality in this paper are general and can handle most types of regression models and estimation methods.

Ideally, the goal of these methods is to minimize the integrated mean squared error (IMSE). The IMSE is a distance or loss function between the true conditional mean and the estimated conditional mean for model $\alpha$ with models and is a measure of how well the conditional mean for model $\alpha$ approximates the true conditional mean. A model with a smaller IMSE means that model is closer to the truth in the squared error loss sense. Define $\mathcal{A}{}_{T}$ as the set of all models being compared, and $\alpha$ is the model index.\footnote{Note that the models being compared may be individual models or combination/model averaged models.} The IMSE is defined as \[ \widetilde{L}_{T}(\alpha)=\frac{1}{T}\sum_{i=1}^{T}(\mu_{i}-\widetilde{\mu}_{-i}(\alpha))^{2}\qquad and\qquad L_{T}(\alpha)=\frac{1}{T}\sum_{i=1}^{T}(\mu_{i}-\hat{\mu}_{i}(\alpha))^{2} \] for leave out and full sample estimates respectively. For recursive and rolling window estimates, the IMSE is define as \[ \ddddot{L}_{T}(\alpha)=\frac{1}{T-t_{0}}\sum_{i=t_{0}+1}^{T}(\mu_{i}-\ddddot{\mu_{i}}(\alpha))^{2}\qquad and\qquad\ddddot{L}_{T}(\alpha)=\frac{1}{T-R}\sum_{i=R+1}^{T}(\mu_{i}-\ddddot{\mu_{i}}(\alpha))^{2}, \] where $t_{0}$ is the minimum number of observations needed so $\ddddot{\mu_{i}}(\alpha)$ is uniquely defined for all models and all observations, $i$.\footnote{For least squares models, $t_{0}$ would be the max $p(\alpha)$ for all models in the recursive case. In the rolling window case, $R$ would need to be large enough that $\ddddot{\mu_{i}}(\alpha)$ is uniquely defined.} To show asymptotic optimality, I follow the standard in the literature and show asymptotically optimality in the sense that \[ \frac{L_{T}(\widehat{\alpha})}{\underset{\alpha\in\mathcal{A}_{T}}{inf}L_{T}(\alpha)}\overset{p}{\rightarrow}1 \] where $\hat{\alpha}$ is the model in the set of $\mathcal{A}{}_{T}$ selected by CV, information criteria, or pseudo out sample forecasting methods. Intuitively, the above formula says that as the sample size tends toward infinity, the probability of the model selection procedure choosing the model with the smallest IMSE converges to 1.\footnote{Note that asymptotic optimality is shown in terms of the full sample IMSE.}

3\quadOptimality of Model Selection Under General Conditions

This section shows the proofs for asymptotic optimality. Section 3.1 shows the proofs for CV, section 3.2. for information criteria, and section 3.3. for pseudo-out-of-sample forecasting.

3.1\quadCross-Validation

I use the following assumptions to show asymptotic optimality for CV:

First note that $\varepsilon_{i}$ is the true residual and $\varepsilon_{i}(\alpha)$ is the residual for model $\alpha$. Assumption 1(a) should be an uncontroversial assumption since it is made up of standard first order moment conditions and requires that the second moments for the true residual and the regression score exist and are bounded. The first two parts follow directly from the exogeneity assumption, $E(\varepsilon_{i}|x_{i})=0$, and depending on the models being considered could be strictly (past, present, and future) exogenous, past and present exogenous, and past exogenous. Most time series models would be past or past and present exogenous, while non-time series models are mostly assumed to be strictly exogenous. Note that the dimension of $x_{i}(\alpha)\varepsilon_{i}$ is $1\times p(\alpha)$. The dimensions are suppressed in the proofs to make the notation simpler, but the dimensions are taken into account in the proofs. Assumption 1(a) can even apply to quantile regressions, though it does not apply to quantile regressions in all situations (see discussions Machado2019).\footnote{In the context of this paper, for quantile regressions one would want to minimize the IMSE between the conditional quantile for model $\alpha$ and the true conditional quantile. Other loss functions have been used in the literature for quantile regressions (see for example Lu2015). } Note that 1(a) would apply even in the case of omitted variable bias or classical measurement error since $\varepsilon_{i}$ is the true residual. If, however, there is simultaneity, 1(a) would not apply unless one were to first instrument the endogenous variable(s).

The first part of Assumption 1(b) is a more general version of the assumption made in Li1987 and Hansen2012 that the leverage values for each model $\alpha$ dissipate to zero as $T\rightarrow\infty$. The second part is just a uniform boundness assumption which rules out razors edge cases where, for example, the difference grows as a function of the sample size. As argued in Li1987, it should be the case that the impact of leaving out an observation has on an estimate should dissipate asymptotically for any reasonable estimator.\footnote{This would clearly apply to M-estimators that satisfy standard regularity conditions and would apply more generally to extremum estimators that satisfy certain regularity conditions (see for example Theorem 2.1 in Newey1994). It is also consistent with convergence of misspecified models (see for example White1981,White1982,White1994 and references therein).} Note that in the case of $h$-block CV, one can allow $h$ to grow with the sample size, but to ensure that the condition is satisfied, one can let $\frac{h}{T}\rightarrow0$ as $h,T\rightarrow\infty$.\footnote{This is standard in block bootstrapping Lahiri2003. It may also be possible to set $h$ as a constant fraction of the sample size.}

Assumption 1(c) and 1(d) are standard model assumptions that are used when proving consistency. There are a wide range of law of large numbers (LLN) for different data including: i.i.d., independent but not identically distributed, stationary, mixing, mixingale, near-epoch dependent, and locally stationary just to name a few. For a more exhaustive list for different types of cross sectional and times series data, see for example chapter 3 in White2000 or part 4 in Davidson2022. Assumption 1(e) places a restriction on the the number of variables that can be included in a model. For example, many law of large numbers converge at rate $O_{p}(T^{-1/2})$ for each element in the vector, so assuming $\frac{p(\alpha)^{2}}{T}\rightarrow0$, as $p(\alpha),T\rightarrow\infty$ would be sufficient in many cases. Assuming that $\frac{p(\alpha)^{2}}{T}\rightarrow0$ or $\frac{p(\alpha)^{3}}{T}\rightarrow0$ is standard in infinite dimensional time series models, infinite dimensional semiparametric models, as well as high dimensional models Lutkepohl2005,Chen2007,Fan2004.\footnote{Note that high-dimensional models with a diverging number of parameters such that $\frac{p(\alpha)}{T}\in(0,\infty)$ as $p(\alpha),T\rightarrow\infty$ (which are sometimes called proportional asymptotics) are excluded.} This assumption also rules out the razors edge case of perfectly predicting $y_{i}$ by including as many variables as there are observations.

Assumption 1(f) is standard in the literature, and can be imposed for example by assuming that either $\parallel\mu_{i}\parallel$ or $\parallel\mu_{i}-\hat{\mu}_{i}(\alpha)\parallel$ are bounded above for all $i$ and $\alpha$ in addition to Assumption 1(b) (see Lemma 1 in the appendix). Assumption 1(g) assumes that the candidate models can be written as linear regressions and is standard assumption in model selection.\footnote{See for example White1981,White1982,White1994 and references therein for more about pseudo-true models.} Though 1(g) is set up for variable/specification selection, the proofs also apply to bandwidth selection or other tuning parameters where $\alpha$ denotes the model for a specific tuning parameter. Assumption 1(g) is a broad assumption that includes a broad range of regression models, however, it excludes nonparametric kernel regression models (e.g. Nadaraya-Watson or Gasser-Muller types) as well as time-varying parameter models. These models will require modifications to the proof and will be addressed in Theorem 2.

Assumption 1(h) is a standard assumption in the literature (see Li1987,Shao1997,Hansen2012 and references therein). It is consistent with the statistical adage from George Box that “All models are wrong” or “All models are wrong, but some are useful” as a way to point out that in science, all models are approximate, so we should not believe the true model is in the set. This assumption is satisfied for infinite dimensional models since the true model cannot be in the set. For finite dimensional models, people do not think the true model is in the set anyway.\footnote{If one includes multiple true models in the set, the model selection procedures will still select a true model since asymptotically the model selection procedure will choose a model whose IMSE converges to zero, though it may not select the most parsimonious version of the true model in the set whose IMSE will converge to faster than the other true models Shao1997. So it is essentially a razors edge case where a true model is selected (and the true model will have a IMSE which converges to zero), but the true model that is selected does not necessarily have the IMSE which converges to zero the fastest, so the model selection procedure may not asymptotic optimality in this case.}

\textcompwordmark

As was just shown, setting up the problem by decomposing the LOO MSE in terms of the MSE of the true residuals, the IMSE, and what are essentially standard first order moment conditions, allows us to leverage standard statistical results without having to specify the type of estimation method and easily allows for generality for the type of model and type of data. The above proof also shows $h$-block CV is asymptotically optimal, which to my knowledge, no one has actually shown Burman1994,Racine2000. These proofs could also be applied to $k$-fold CV, but it would require assumptions on the fold size similar to $h$-block CV (e.g. if you have $k$ folds with $h$ observations in each fold, $h$ can grow with the sample size but $\frac{h}{T}\rightarrow0$).

As mentioned earlier, Theorem 1 does not allow for observation dependent coefficients, so it does not account for time-varying parameter models or Nadaraya-Watson type nonparametric models. To address this shortcoming, we need a slightly different set of assumptions.

Assumption 2 is essentially the same as Assumption 1, but there are two differences. The first difference is it allows the model parameters to vary with the observation or covariate. For time-varying parameter models, $\beta_{i}(\alpha)$ is a continuous function in time and follows standard infill asymptotics arguments Robinson1989,Cai2007,Chen2012,Dahlhaus2012. Note that for time-varying parameter case, the continuous function can be deterministic or stochastic.\footnote{The standard in nonparametric time-varying parameter estimation is to use nonparametric kernels Robinson1989,Cai2007,Dahlhaus2012, and it is typically assumed that the function is deterministic and differentiable. It has been shown that kernel methods can be used for certain stochastic processes Giraitis2014,Giraitis2021. As long as the continuous functions can be written as an infinite order of orthogonal basis functions (whether it be by the Stone-Weierstrass Theorem or Karhunen-Loève Theorem), the time-varying parameters can alternatively be estimated using basis function approximations Huang2002,Huang2004.} The continuity assumption for nonparametric kernel regressions is also standard Fan1996. Assumption 2(c) just says as the sample size grows, the difference between estimates in the same block should decrease asymptotically. Note that the construction of the $m$ blocks of size $\ell$ is in line with consistency assumptions and intuition behind nonparametric kernel regressions and infill asymptotics of time-varying parameter models. In the case where the estimators are consistent/pseudo consistent, this condition is satisfied.\footnote{It is fairly standard in nonparametric regression literature to assume that the class of estimators being compared are consistent/pseudo consistent (see for example Li1985,Hardle1994,Hart1994,Leung2005,Sun2021 and references therein).} There are also cases where this condition can be satisfied even if the estimators are not consistent/pseudo consistent (e.g. rolling window regression where the window is a constant fraction of the sample size or more generally a kernel regression where the bandwidth does not go to zero fast enough to satisfy the consistency condition).

The second difference from Assumption 1 is that the dimension, $p(\alpha)$, is not growing with the sample size. This is standard in the time-varying parameter and nonparametric kernel regression literature. As will be seen in the proofs, the proofs may be able to handle $p(\alpha)$ growing with the sample size, but it require being specific about the rates at which $p(\alpha)$ and $\ell$ grow.

3.2\quadInformation Criteria

I use the following assumptions to show asymptotic optimality for information criteria:

Assumptions 1(b) and 1(f) are dropped since information criteria use all of the data when estimating the residuals unlike CV. The penalty term tending toward zero asymptotically occurs for most if not all of the popular information criteria including but not limited to: AIC, BIC, HQIC, RIC, and Generalized information criteria.

So despite AIC, BIC, and other information criteria being derived under fairly restrictive conditions (i.i.d. data or independent residuals), they are asymptotically optimal under more general conditions. This result holds for any penalty that converges to 0 asymptotically, thus indicating the arbitrariness of the penalty from an asymptotic optimality standpoint. In finite samples, the chosen model may vary drastically due to the penalty, but currently it is not clear what the penalty should be, and it may be the case that there is no best penalty for all situations.

3.3\quadPseudo-Out-Of-Sample Forecast Methods

3.3.1\quadConstant Parameter Case

To show asymptotic optimality of rolling window and recursive forecasts in the constant parameter case, we need updated assumptions.

Assumption 4 is the rolling window and recursive version of Assumption 1. Assumption 4(a), which requires that the window size, $R$, grows with the sample size but at a slower rate, is standard in the literature (see Inoue2017 and references therein). Letting the initial starting point for the recursive forecast, $t_{0}$, tend toward infinity is also common (e.g. West1996,West1998,Clark2009a), though many papers also assume that it is either constant or a constant fraction of the sample size.\footnote{The proofs for the recursive case can be written under the alternative assumptions for $t_{0}$, though it would require different proofs, and it is convenient to assume that it grows with the sample size as it allows the same proof to be used for both the rolling window and recursive forecast cases.}

3.3.1\quadTime-varying Parameter Case

To show asymptotic optimality for rolling window and recursive forecasts in the time-varying parameter case, we use the following assumption:

As long as $R$ and $t_{0}$ grow at suitable rates, the modified Assumption 2 should be satisfied. In the case of nonparametric kernel estimation with time-varying parameters, rolling window estimates are simply one sided kernel estimates of the time-varying parameters Inoue2017,Cai2023,Farmer2023, as opposed to the full sample estimates which are generally based on two sided kernels. Taking that into account along with the infill asymptotic framework, the recursive/rolling window version of Assumption 2(c) is not unrealistic.\footnote{This also makes sense if one uses the basis function approximation approach to estimating time-varying parameter models Huang2002,Huang2004.} Note that the rate at which the block size, $\ell$, grows may be dependent on the rate at which $R$ and $t_{0}$ grow. Assumption 5(a) is the same as 4(a) and follows the same reasoning.

4\quadImplications and Discussion

Under fairly general conditions, CV, the most popular information criteria, and the most popular pseudo-out-of-sample forecasting methods will asymptotically all choose the model with the same conditional mean, so from an asymptotic optimality standpoint, there is no benefit from using one procedure versus another. In regards to what model selection procedure one should use in finite samples, the arguments in this paper cannot speak to them. I do think the results help highlight why recommendations for certain model selection procedures are not as strong as they first appear.

Many people over the last few decades have advocated the use of model selection criteria such as BIC and HQIC on the grounds that they are consistent (see for example Shao1997,Claeskens2016,Ding2018 and references therein). What does not get talked about enough is consistency is a distinction without a difference. Intuitively, consistency of a model selection procedure simply means that if the most parsimonious version of the true model is in a set, then the model selection procedure will choose the most parsimonious version of the true model asymptotically. What does not get brought up enough is that it could be the case that in finite samples, the researcher would prefer a criteria that is not consistent. Consistent model selection criteria tend to choose a more parsimonious version of the model, which may not be ideal. There is a host of Monte Carlo evidence indicating this is the case. For example Lutkepohl2005 (page 155) shows in a Monte Carlo where the true model is a vector autoregression (VAR) with two lags that AIC selected the true model more often than BIC and HQIC, despite the latter two criteria being consistent. Burnham2004 shows in a Monte Carlo that when the true model is denser (has more variables), AIC does a better job of selecting a model closer to the truth than BIC, but the results depend on the size of the parameters. Ng2005 and Ng2013 show that neither AIC or BIC dominate, but again, the results depend on the size of the parameters.

If the true model is not in the set (which would be the case for infinite dimensional models), a model selection procedure cannot be consistent since the true model can never be in the set of models being compared. Even if the true model is finite dimensional, if the true model is not in the set, then we currently do not have a reason (from the asymptotic point of view) to prefer one model selection procedure over another. This is in line with the George Box argument that “All models are wrong” or “All models are wrong, but some are useful” as a way to point out that in science, all models are approximate, so we should not believe the true model is in the set.\footnote{Many researchers have realized this, and it is one of the reasons why model averaging has become popular over the past 30 years Steel2020. }

There has also been focus on model selection procedures, such as AIC, being minimax.\footnote{Conditions for minimax estimators are generally derived under i.i.d. data, and to my knowledge these results have not been extended.}\textsuperscript{,}\footnote{Yang2005 shows an estimator cannot be strongly consistent and minimax at the same time.} A model selection procedure being minimax simply means that in the worst case scenario, the minimax model selection procedure has a smaller IMSE than a non-minimax procedure, or alternatively, minimax procedures minimize downside risk. But for any particular situation a researcher is interested in, a procedure being minimax does not say anything about which criteria is preferred in finite samples because the situation may be far from the worst case scenario. Minimax procedures tend to choose larger models, and consistent procedures tend to choose smaller models, so depending how accurately and efficiently estimated parameters are and whether you are more worried about overfitting or underfitting, you may prefer one procedure over another. In a finite sample, there is a bias variance tradeoff because while parameters may be biased when the selected model is too small, the parameter estimates may be highly inefficient if the estimated model includes too many parameters Ng2013,Rossi2021. Furthermore, as long as $\lambda_{T}$ is a finite constant greater than 1, the information criteria is minimax Shao1997,Yang2005. So even within the class of minimax procedures, it is not clear which one should be chosen since as shown in Theorem 3, they are all asymptotically optimal.\footnote{The same holds for consistent model selection procedures. A sufficient condition for consistent model selection procedures require that $\lambda_{T}\rightarrow\infty$ but $\frac{\lambda_{T}p}{T}\rightarrow0$. Since there are an infinite number of ways to construct a consistent model selection procedures and they are asymptotically optimal under the conditions stated in this paper, the literature currently cannot distinguish between procedures within this class from an asymptotic standpoint.}

Based off the results in this paper and in the literature, I believe it is the case that the literature should focus more on the finite sample properties or maybe use local asymptotics or set up problems in such a way that the differences in model selection criteria show up in the limit.\footnote{An important topic of research in the literature that is beyond the scope of this paper is taking into account the impact model selection has on subsequent inference (see for example Leeb2005).}

5\quadConclusion

This paper provides general proofs showing optimality for a wide range of model selection criteria under fairly general conditions. This paper not only asymptotically justifies model selection criteria for most situations, but it also unifies and extends a range of previously disparate results.

The results from this paper should allow researchers to move on from showing the asymptotic optimality of model selection procedures for most situations and to potentially focus on showing the theoretical finite sample properties of these methods. Since it is the case that the most popular methods end up being asymptotically optimal under general conditions, the choice of which model selection procedure to choose from, like most things in statistics, appears to be a finite sample choice and not an asymptotic one.

Proof of Theorem 4

Proof of Theorem 5