Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
59,271 characters · 12 sections · 86 citation commands
\baselineskip=18pt
\thispagestyle{plain} \thispagestyle{empty}
\vskip 3mm \centerline{ ABSTRACT} Asymptotic optimality is a key theoretical property in model averaging. Due to technical difficulties, existing studies rely on restricted weight sets or the assumption that there is no true model with fixed dimensions in the candidate set. The focus of this paper is to overcome these difficulties. Surprisingly, we discover that when the penalty factor in the weight selection criterion diverges with a certain order and the true model dimension is fixed, asymptotic loss optimality does not hold, but asymptotic risk optimality does. This result differs from the corresponding result of fang2023 and reveals that using the discrete weight set of hansen2007mma can yield opposite asymptotic properties compared to using the usual weight set. Simulation studies illustrate the theoretical findings in a variety of settings.
\abst.arg
\vskip 3mm {\it Keywords:} Asymptotic loss/risk optimality, least square model averaging, Mallows model averaging, weight set. \key.arg
\setcounter{page}{1}
\baselineskip=23pt
Least squares model averaging has attracted much attention in both econometrics and statistics since the seminal work on Mallows model averaging (MMA) by hansen2007mma. Subsequently, numerous alternative methods have emerged to select the model averaging weights. These include optimal mean squared error averaging liang2011, jackknife model averaging hansen2012,zhang2013JMA,ando2014, heteroskedasticity-robust $C_p$ model averaging liu2013ma, prediction model averaging xie2015, Kullback-Leibler model averaging zhang2015, parsimonious model averaging zhang2020PMA, and so on.
Asymptotic optimality is a key theoretical goal pursued in model averaging research. It states that model averaging estimator yields the smallest possible prediction loss (or risk) among all such estimators as the sample size approaches infinity. Most of the aforementioned model averaging methods have been shown to be asymptotically optimal under some conditions. To illustrate, taking MMA as an example. hansen2007mma were the first to establish MMA's asymptotic optimality when the candidate models are nested and the weights are restricted to a discrete set. Subsequently, wan2010 and zhang2021 provided justification in a non-nested model setting with continuous weights. However, their works impose some conditions to limit the number of candidate models. More recently, peng.opt significantly improved these conditions by allowing more candidate models to be combined in a nested model setting. It is important to note that these results typically require that all candidate models with fixed dimensions are approximations, which is satisfied in the scenario where all candidate models are misspecified.
In the area of model averaging, there is another common scenario where the true model is among the candidate models. This scenario is also frequently encountered in model selection. In this context, existing works, such as zhang2015el, zhang.liu2019, fang2020, and zhang2020PMA, mainly focus on the asymptotic behavior of the selected weight and the asymptotic distribution of the model averaging estimator. However, theoretical results on asymptotic optimality in this setting have received limited attention in the literature. Recently, fang2023 made a significant contribution to filling this gap in the nested model setting. To the best of our knowledge, this is the sole work addressing this topic. A related work by shao1997 established the asymptotic loss efficiency of various model selection procedures, such as Mallows' $C_p$, AIC, BIC, and cross-validation, in the scenario where a true model exists.
Let us consider a weight selection criterion (ref) with a penalty factor $\phi_n$ defined in Section (ref). This criterion encompasses MMA ($\phi_n=2$) and PMA ($\phi_n\to \infty$ or $\phi_n=\log n$) as special cases. When considering the scenario where the true model is among the candidate models, the findings of fang2023 can be summarized as follows. When $\phi_n=\log n$, the least squares model averaging is asymptotically loss optimal by putting weight one to the true model. When $\phi_n=2$ (MMA) and the model dimensions are fixed, asymptotic loss optimality does not hold since the model averaging method puts weight on the over-fitted models with a positive probability. These results are very similar to the asymptotic behaviors of the traditional model selection methods AIC and BIC shao1997,ding2018. However, the aforementioned results of fang2023 restrict the model weights to a slightly special weight set due to technical difficulty. While they believed that these results remain valid for the general weight set, a formal theoretical justification is still lacking.
In this paper, we revisit the asymptotic optimality of least square model averaging in the nested model setting when the true model is among the candidate models. Compared to the results of fang2023, our main contributions are twofold. First, our results hold for the general weight set, rather than their restricted weight set. Second, we explore asymptotic risk optimality, a topic that they did not address. We show that when $\phi_n=\log n$ and the true model dimension is fixed, the asymptotic loss optimality does not hold, which differs from the corresponding result of fang2023. This also reveals that using the discrete weight set of hansen2007mma for model averaging can lead to opposite asymptotic properties compared to using the usual weight set. When $\phi_n=\log n$ and the true model dimension does not diverge too fast, the asymptotic risk optimality holds. Besides, MMA with fixed model dimensions is neither asymptotically loss optimal nor asymptotically risk optimal, which coincides with the corresponding result of fang2023.
The remainder of the paper is organized as follows. Section (ref) introduces a class of least squares model averaging methods, which includes MMA and PMA as special cases. Section (ref) reviews the existing results and provides a discussion. Section (ref) presents the main theoretical results. Section (ref) provides the results of finite sample simulation studies. Section (ref) concludes the paper. The proofs of the main results are relegated to the Appendix.
Suppose we have $n$ independent observations $\{(y_i, \mathbf{x}_i)\colon i=1,\ldots,n\}$, where $y_i$ is the scalar response and $\mathbf{x}_i=(x_{i1},x_{i2},\ldots)$ is a vector of countably infinite covariates. Consider the linear regression model
where $e_1,\ldots, e_n$ are independent and identically distributed (i.i.d.) errors with $E(e_i|\mathbf{x}_i)=0$ and $E(e_i^2|\mathbf{x}_i)=\sigma^2$ and $\beta_j$'s are unknown parameters. In matrix notation, (ref) can be written as $\mathbf{y}=\bm{\mu}+\mathbf{e}$, where $\mathbf{y}=(y_1,\ldots,y_n)^{\top}$, $\bm{\mu}=(\mu_1,\ldots,\mu_n)^{\top}$, and $\mathbf{e}=(e_1,\ldots,e_n)^{\top}$.
To estimate the true mean vector $\bm{\mu}$, we consider $M_n$ nested candidate models for model averaging, where the $m$th model uses the first $k_m$ covariates and $0<k_1<\cdots<k_{M_n}<\infty$. Both $k_{M_n}$ and $M_n$ can be diverging to infinity as $n\to \infty$. Let $\mathbf{X}_m$ be the $n \times k_m$ design matrix of the $m$th model. We assume that $\mathbf{X}_m$ is of full column rank for any $m \in \{1,\ldots,M_n\}$. Then, under the $m$th model, the estimator of $\bm{\mu}$ is $\hat{\bm{\mu}}_m=\mathbf{P}_m \mathbf{y}$, where $\mathbf{P}_m=\mathbf{X}_m(\mathbf{X}_m^{\top}\mathbf{X}_m)^{-1}\mathbf{X}_m^{\top}$. Let $\mathbf{w}=(w_1,\ldots,w_{M_n})^{\top}$ be a weight vector belonging to the unit simplex in $\mathbb{R}^{M_n}$:
Then, the model averaging estimator of $\bm{\mu}$ with weights $\mathbf{w}$ is $\hat{\bm{\mu}}(\mathbf{w})=\sum_{m=1}^{M_n} w_m \hat{\bm{\mu}}_m=\mathbf{P}(\mathbf{w})\mathbf{y}$, where $\mathbf{P}(\mathbf{w})=\sum_{m=1}^{M_n} w_m\mathbf{P}_m$. Let $\mathbf{I}_n$ be the identity matrix of size $n$ and $\|\cdot\|$ be the Euclidean norm. Following zhang2020PMA, the weight vector $\mathbf{w}$ is selected by minimizing the criterion
where $\hat{\sigma}^2=(n-k_{M_n})^{-1}\mathbf{y}^{\top}(\mathbf{I}_n-\mathbf{P}_{M_n})\mathbf{y}$ is an estimator of $\sigma^2$, $\mathbf{K}=(k_1,\ldots,k_{M_n})^{\top}$, and $\phi_n$ is a penalty factor which may depend on $n$. Two common choices for $\phi_n$ are $\phi_n=2$ and $\phi_n=\log n$. When $\phi_n=2$, (ref) is the Mallows criterion of hansen2007mma. When all elements of $\mathbf{w}$ are 0 or 1, the weight selection criterion (ref) corresponds to the AIC if $\phi_n=2$ and the BIC if $\phi_n=\log n$. Note that fang.EL2022 proposed a cross-validation procedure for selecting $\phi_n$ between $2$ and $\log n$. Finally, the selected weight vector is defined as $\hat{\mathbf{w}}=(\hat{w}_1,\ldots,\hat{w}_{M_n})^{\top}=\operatorname*{\arg \min}_{\mathbf{w}\in \mathcal{H}_n} \mathcal{G}_n(\mathbf{w})$.
Define the squared prediction loss as $L_n(\mathbf{w})=\|\bm{\mu}-\hat{\bm{\mu}}(\mathbf{w})\|^2$ and the corresponding prediction risk as $R_n(\mathbf{w})=E\{L_n(\mathbf{w})|\mathbf{x}_1,\ldots,\mathbf{x}_n\}$. In the context of model averaging research, two types of asymptotic optimality are commonly considered: asymptotic loss and risk optimality. Specifically, we say that $\hat{\mathbf{w}}$ is asymptotically loss optimal if
and we refer to $\hat{\mathbf{w}}$ as asymptotically risk optimal if
Here $\to_p$ denotes convergence in probability. All limiting processes in this paper are with respect to $n\to \infty$. Note that (ref) means that the squared prediction loss of the model averaging estimator is asymptotically as small as the squared prediction loss of the infeasible best possible averaging estimator, and (ref) is a similar statement about the squared prediction risk.
When $\phi_n=2$ (MMA), hansen2007mma established the asymptotic loss optimality of $\hat{\mathbf{w}}$. However, this requires weights to be confined to the discrete set
where $N$ is a fixed positive integer. Note that model averaging restricted to $\mathcal{H}_n(1)$ reduces to model selection. Subsequently, wan2010, zhang2021, and peng.opt demonstrated that $\hat{\mathbf{w}}$ is asymptotically loss optimal over the continuous weight set $\mathcal{H}_n$. Their proofs actually imply the asymptotic risk optimality of $\hat{\mathbf{w}}$ as well. When $\phi_n\to \infty$, zhang2020PMA justified both the asymptotic loss and risk optimality of $\hat{\mathbf{w}}$. A key prerequisite for these results is that $\xi_n=\inf_{\mathbf{w}\in \mathcal{H}_n} R_n(\mathbf{w})\to \infty$. This condition means that there are no candidate models with fixed dimensions for which the approximation error is zero; see also hansen2007mma. Therefore, the aforementioned results do not apply to the case where there exists a true candidate model with fixed dimension.
Our aim is to investigate both the asymptotic loss and risk optimality of $\hat{\mathbf{w}}$ in the scenario where the true model is included in the set of candidate models. Let $\mathcal{A}_m=\{1,\ldots,k_m\}$ and $\mathcal{A}_m^c$ denote its complement. We assume that there exists an $M_0$ such that $(\beta_j, j\in \mathcal{A}_{M_0+1}\cap \mathcal{A}_{M_0}^c)^{\top}\neq \mathbf{0}$ and $\beta_j=0$ for all $j>k_{M_0+1}$. Consequently, the first $M_0$ candidate models are under-fitted, the $(M_0+1)$th model is defined as the “true model", and all other models are over-fitted. Additionally, we assume that $M_0\geq 1$ and $M_n>M_0+1$, meaning there is at least one under-fitted and one over-fitted model. Note that $k_{M_0+1}$ is the true model dimension, and we allow for diverging $k_{M_0+1}$ and $M_0$.
Throughout this paper, we consider the case of deterministic covariates. When $\mathbf{x}_1,\ldots, \mathbf{x}_n$ are random, the results in this article are still valid in the almost sure sense, provided that the required conditions involving $\mathbf{x}_1,\ldots, \mathbf{x}_n$ hold almost surely.
In a recent study, fang2023 established conditions that determine whether $\hat{\mathbf{w}}$ achieves the asymptotic loss optimality (ref) when the true model is among the candidate models; see their Subsection 3.2 for the details. However, due to technical difficulty, their results rely on the following slightly special weight set
Here, $\delta$ is a positive constant which can be arbitrarily small, and $\tau_0$ is a positive constant that satisfies that $\sum_{j\in \mathcal{A}_{M_0+1}\cap \mathcal{A}_{M_0}^c} \beta_j^2\geq c_{\tau_0} n^{-\tau_0}$ for a positive constant $c_{\tau_0}$. Note that when $\delta$ is sufficiently small or $n$ is large, the set $\mathcal{H}_n^{\delta}$ includes the discrete weight set $\mathcal{H}_n(N)$ considered by hansen2007mma and hansen2012.
By an abuse of notation, in this subsection, we still use $\hat{\mathbf{w}}$ to denote $\operatorname*{\arg \min}_{\mathbf{w}\in \mathcal{H}_n^{\delta}} \mathcal{G}_n(\mathbf{w})$. The results of fang2023 are provided in their Theorem 5. Here, we summarize them as follows.
In the area of model selection, it is well known that in a parametric framework, AIC is not asymptotically loss optimal, whereas BIC is due to its consistency shao1997,ding2018. Consequently, Results (i) and (ii) are very similar to the corresponding asymptotic behaviors of BIC and AIC, respectively.
Denote by $\mathbf{w}_m^0$ an $M_n\times 1$ vector in which the $m$th element is one and the others are zeros. Let $\mathbf{w}^*=\operatorname*{\arg \min}_{\mathbf{w}\in \mathcal{H}_n^{\delta}} L_n(\mathbf{w})$. The proof of the above results of fang2023 is divided into the following two steps.
Steps 1 and 2 correspond to the results in their Lemma 1 and Theorem 5, respectively. The special weight set $\mathcal{H}_n^{\delta}$ is mainly utilized in Step 1. If $\Pr(\mathbf{w}^*=\mathbf{w}_{M_0+1}^0)\to 1$, we have $\Pr\{\inf_{\mathbf{w}\in \mathcal{H}_n^{\delta}} L_n(\mathbf{w})=L_n(\mathbf{w}_{M_0+1}^0)\}\to 1$, and thus $L_n(\hat{\mathbf{w}})/\inf_{\mathbf{w}\in \mathcal{H}_n^{\delta}} L_n(\mathbf{w})$ and $L_n(\hat{\mathbf{w}})/L_n(\mathbf{w}_{M_0+1}^0)$ have the same asymptotic behavior. Finally, after finishing Step 1, the subsequent work is Step 2, which relies on the asymptotic properties of $\hat{w}_m$ for the under-fitted and over-fitted models. These properties have been extensively studied in the literature such as zhang.liu2019 and zhang2020PMA.
Let $\mathbf{w}^L=\operatorname*{\arg \min}_{\mathbf{w}\in \mathcal{H}_n} L_n(\mathbf{w})$ denote the optimal weight vector in terms of prediction loss. Considering that the aforementioned results of fang2023 are restricted to the special weight set $\mathcal{H}_n^{\delta}$, two natural questions arise:
As discussed by fang2023, intuitively, Questions Q1 and Q2 should have affirmative answers. Unfortunately, this assertion may not hold true. In the following, we present a toy example to illustrate this.
We also conduct a simulation study to verify (ref) and (ref) in Example (ref). We simulate $10000$ data sets from the model in Example (ref), where $(\beta_1, \beta_2, \beta_3)=(-1, 0.1, 0)$, $x_{ij}$ are independently generated from $N(0, 1)$, and $e_i$ are i.i.d. from $N(0,1)$ and are independent of $x_{ij}$'s. Figure (ref)(a) shows the simulated values of $\Pr(\mathbf{w}^L=\mathbf{w}_2^0)$ from $n=100$ to $2000$ with an increment of $100$. These values converge to around $0.5$ when $n\geq 1000$, aligning with (ref). Figure (ref)(b) displays the kernel density estimates for the values of $L_n(\mathbf{w}^L)/L_n(\mathbf{w}_2^0)$ after excluding 1, for $n=100$, $1000$, and $2000$. The density estimates for $n=1000$ and $2000$ closely resemble the density curve of $\mathrm{Beta}(1/2, 1/2)$, which is consistent with (ref).
In Lemma (ref) and Theorem (ref) of the next section, we demonstrate that the above findings in Example (ref) remain consistent when the true model dimension is fixed. These results are opposite to the corresponding results obtained by fang2023 under the restricted weight set $\mathcal{H}_n^{\delta}$, which is somewhat surprising given our intuition. In the following section, we will also seek conditions that determine whether or not the asymptotic risk optimality (ref) of $\hat{\mathbf{w}}$ holds, a topic that has not been addressed by fang2023.
In this section, we explore both the asymptotic loss and risk optimality of $\hat{\mathbf{w}}$. As pointed out in Subsection (ref), the result $\Pr(\mathbf{w}^L=\mathbf{w}_{M_0+1}^0)\to 1$ may not hold. This suggests that the proof procedures used by fang2023, i.e., Steps 1 and 2 in Subsection (ref), may be not applicable. Observe that
Therefore, if $L_n(\hat{\mathbf{w}})/L_n(\mathbf{w}_{M_0+1}^0)\to_p 1$, then the asymptotic loss optimality holds if and only if $L_n(\mathbf{w}_{M_0+1}^0)/\inf_{\mathbf{w}\in \mathcal{H}_n} L_n(\mathbf{w})$ conveges to $1$ in probability. Similarly, if $R_n(\hat{\mathbf{w}})/R_n(\mathbf{w}_{M_0+1}^0)\to_p 1$, then the asymptotic risk optimality holds if and only if $R_n(\mathbf{w}_{M_0+1}^0)/\inf_{\mathbf{w}\in \mathcal{H}_n} R_n(\mathbf{w})$ conveges to $1$. It is worth mentioning that in Appendixes (ref) and (ref), we establish that $L_n(\hat{\mathbf{w}})/L_n(\mathbf{w}_{M_0+1}^0)\to_p 1$ and $R_n(\hat{\mathbf{w}})/R_n(\mathbf{w}_{M_0+1}^0)\to_p 1$ when $\phi_n=\log n$ under certain conditions. As a result, we can replace Step 1 in Subsection (ref) with the following:
Note that Step 1' does not depend on specific weight selection methods. In this section, we follow Step 1' and Step 2 to explore the asymptotic loss and risk optimality.
Throughout this section, we assume that $E|e_i|^{4+\zeta}<\infty$ for some $\zeta>0$. Let $\lambda_{\min}(\mathbf{M})$ be the minimum eigenvalue of a positive definite matrix $\mathbf{M}$. We first state the regularity conditions required for main results.
Condition (ref) is a standard moment condition in the model averaging literature wan2010,liu2016, which requires the average of $\mu_i^2$ is bounded. Condition (ref) assumes that the eigenvalues of $n^{-1}\mathbf{X}_{M_0+1}^{\top}\mathbf{X}_{M_0+1}$ are uniformly bounded away from zero for all $n$. This condition is commonly used for theoretical results with a diverging number of parameters zou2009,fang2023,zhang2020PMA. Obviously, if the true model dimension is fixed and $\mathbf{X}_{M_0+1}$ is of full rank, Condition (ref) is satisfied. Condition (ref) is from yuan.yang2005, zhang2020PMA, and fang2023, which essentially assumes that $\hat{\sigma}^2$ is neither too small nor too large. Note that Condition (ref) does not require that $\hat{\sigma}^2$ is consistent and is thus easily satisfied. Condition (ref) is adopted from liu2015joe and zhang.liu2019, which is quite mild. Note that this condition is not required if $k_{M_0+1}$ is diverging.
Let $\eta_n=\sum_{j\in \mathcal{A}_{M_0+1}\cap \mathcal{A}_{M_0}^c} \beta_j^2$. Note that $\eta_n$ measures the difference between the true model and the largest under-fitted model. Under Conditions (ref)--(ref) and the assumption that the $(M_0+1)$th model is the true model, it can be easily verified that $0< \eta_n<\infty$. If the true model dimension is diverging, $\eta_n$ could converge to $0$. On the other hand, if the true model dimension is fixed, $\eta_n^{-1}=O(1)$.
In this subsection, we aim to determine the conditions under which $\hat{\mathbf{w}}$ achieves the asymptotic loss optimality (ref), and the conditions under which it does not in the sense that
As mentioned at the beginning of this section, we first explore whether the true model is asymptotically loss optimal or not (Step 1').
Lemma (ref)(i) indicates that when the true model dimension is fixed, the true model is not asymptotically loss optimal. This result also provides a negative response to Question Q1 in the case of a fixed true model dimension. In the proof of this result in Appendix (ref), we actually derive a more general result. Specifically, we demonstrate that for any constant $z>1$,
where $\mathbf{Z}\sim N(\mathbf{0}, \mathbf{I}_{k_{M_0+1}})$ and $\mathbf{v}$ is a $k_{M_0+1}\times 1$ unit vector defined in (ref) of the Appendix. Given that $\|\mathbf{v}\|=1$, it is easy to see that the support of $(\mathbf{v}^{\top}\mathbf{Z})^2/\|\mathbf{Z}\|^2$ is $[0, 1]$. By using Theorems 3--4 of forchini2002, it is possible to derive an exact distribution function of $(\mathbf{v}^{\top}\mathbf{Z})^2/\|\mathbf{Z}\|^2$. In the special case where $\mathbf{Q}=\sigma_X^2 \mathbf{I}_{k_{M_0+1}}$ with some $\sigma_X^2>0$ and $k_{M_0+1}-k_{M_0}=1$, we obtain $\mathbf{v}=(0, \ldots, 0, \pm1)^{\top}$ and $(\mathbf{v}^{\top}\mathbf{Z})^2/\|\mathbf{Z}\|^2 \sim \mathrm{Beta}(1/2, k_{M_0}/2)$. Therefore, from (ref), we can establish Lemma (ref)(i).
In Lemma (ref)(ii), the condition $M_0/(k_{M_0+1}\eta_n)\to 0$ constrains the rate of increase in the number of under-fitted models $M_0$. Therefore, Lemma (ref)(ii) tells us that when the true model dimension diverges, but not too fast, and the number of under-fitted models does not diverge too fast, the true model is asymptotically loss optimal.
The following theorem establishes the asymptotic loss optimality of the least squares model averaging with $\phi_n=2$ and $\phi_n=\log n$.
The first part of Theorem (ref)(i) reveals that when $\phi_n=\log n$ and the true model dimension is fixed, the least squares model averaging is not asymptotically loss optimal because the asymptotic loss optimality of the true model does not hold. This finding is opposite to the corresponding result of fang2023 and thus partially provides a negative response to Question Q2. In the proof of this result in Appendix (ref), we derive a more general result. Specifically, we demonstrate that if $\phi_n\to \infty$ and $\phi_n^2/n \to 0$, then for any constant $z>1$,
where $\mathbf{Z}$ and $\mathbf{v}$ are the same as those in (ref). It follows from (ref) that the asymptotic loss optimality of $\hat{\mathbf{w}}$ does not hold.
The second part of Theorem (ref)(i) suggests that when $\phi_n=\log n$, the true model dimension diverges, but not too fast, and the number of under-fitted models does not diverge too fast, the least squares model averaging is asymptotically loss optimal. This result arises because the asymptotic loss optimality of the true model holds, and the model averaging method pushes all the weights to the true model.
When $\phi_n=2$ and the model dimensions are fixed, fang2023 established that $\sum_{m>M_0+1} \hat{w}_m^2 (\frac{\mathbf{e}^{\top}\mathbf{P}_m\mathbf{e}}{\mathbf{e}^{\top}\mathbf{P}_{M_0+1}\mathbf{e}}-1)$ does not converge to $0$ in probability. This is because $\frac{\mathbf{e}^{\top}\mathbf{P}_m\mathbf{e}}{\mathbf{e}^{\top}\mathbf{P}_{M_0+1}\mathbf{e}}\nrightarrow_p 1$, and $\{\hat{w}_m, m\geq M_0+1\}$ converges to a nondegenerate random weight vector in distribution zhang.liu2019. So the first part of Theorem (ref)(ii) actually shows that MMA is not asymptotically loss optimal with fixed model dimensions because it assigns nonnegligible weights to over-fitted models. This result coincides with the corresponding result of fang2023. It is important to note that another reason for this result is that the asymptotic loss optimality of the true model does not hold.
The second part of Theorem (ref)(ii) implies that when the true model dimension diverges, MMA is asymptotically loss optimal under two key conditons. The first condition is that $(k_{M_0+1}-k_1)/(n\eta_n)\to 0$ and $M_0/(k_{M_0+1}\eta_n)\to 0$, ensuring the asymptotic loss optimality of the true model. The other condition is that $k_{M_n}/k_{M_0+1}\to 1$, indicating that the over-fitted models have no significant difference from the true model in the large sample sense. Although MMA may assign nonnegligible weights to over-fitted models, the condition $k_{M_n}/k_{M_0+1}\to 1$ ensures that these over-fitted models can be considered negligible.
In this subsection, we aim to determine the conditions under which $\hat{\mathbf{w}}$ achieves the asymptotic risk optimality (ref), and the conditions under which it does not in the sense that
As mentioned at the beginning of this section, we first explore whether the true model is asymptotically risk optimal or not (Step 1').
The condition $(k_{M_0+1}-k_1)/(n\eta_n)\to 0$ is clearly satisfied when the true model dimension is fixed. Therefore, Lemma (ref) implies that as long as the true model dimension does not diverge too fast, the true model is asymptotically risk optimal. Comparing this result with Lemma (ref)(i), an intriguing observation is that when the true model dimension is fixed, the true model is not asymptotically loss optimal, but it is asymptotically risk optimal.
Now we explore the asymptotic risk optimality of the least squares model averaging with $\phi_n=2$ and $\phi_n=\log n$ in the following theorem.
Theorem (ref)(i) indicates that when $\phi_n=\log n$, as long as the true model dimension does not diverge too fast, the least squares model averaging is asymptotically risk optimal. This is because the true model achieves the asymptotic risk optimality, and the model averaging method pushes all the weights to the true model. Comparing this result with Theorem (ref)(i), a remarkable observation emerges: when $\phi_n=\log n$ and the true model dimension is fixed, the least squares model averaging is not asymptotically loss optimal, but it is asymptotically risk optimal.
When $\phi_n=2$ and the model dimensions are fixed, Theorem 2 of zhang.liu2019 established that $\{\hat{w}_m, m\geq M_0+1\}$ converges in distribution to a nondegenerate random weight vector. Combining this result with the fact that $\sum_{m>M_0+1} \hat{w}_m^2 (\frac{k_m}{k_{M_0+1}}-1)\geq (\frac{k_{M_0+2}}{k_{M_0+1}}-1) \sum_{m>M_0+1} \hat{w}_m^2$, it follows that $\sum_{m>M_0+1} \hat{w}_m^2 (\frac{k_m}{k_{M_0+1}}-1)$ does not converge to $0$ in probability. Therefore, the first part of Theorem (ref)(ii) actually shows that MMA is not asymptotically risk optimal with fixed model dimensions since it puts weights on the over-fitted models with a positive probability.
The second part of Theorem (ref) implies that when the true model dimension diverges, but not too fast, and $k_{M_n}/k_{M_0+1}\to 1$, MMA is asymptotically risk optimal. The condtion $k_{M_n}/k_{M_0+1}\to 1$ means that the over-fitted models do not significantly differ from the true model in the large sample sense. Therefore, in this case, MMA's asymptotic risk optimality holds because these over-fitted models can be considered negligible, even though MMA might assign non-negligible weights to them.
Last, we verify Lemma (ref) and Theorem (ref) in the following examples.
We summarize the main theoretical results in Table (ref). One remarkable result is that when $\phi_n=\log n$ and the true model dimension is fixed, the asymptotic loss optimality does not hold, while the asymptotic risk optimality does. This occurs because when the true model dimension is fixed, the true model is not asymptotically loss optimal but is asymptotically risk optimal. This result does not contradict the corresponding finding of fang2023, where they demonstrated the asymptotic loss optimality with a fixed true model dimension when $\phi_n=\log n$, since their result is confined to the special weight set $\mathcal{H}_n^{\delta}$.
From Table (ref), we can observe three cases where the asymptotic loss or risk optimality does not hold. These can be attributed to one or both of the following main reasons:
Specifically, when model dimensions are fixed, MMA is not asymptotically loss optimal due to both Reasons (R1) and (R2), and it is also not asymptotically risk optimal because of Reason (R2). When $\phi_n=\log n$ and the true model dimension is fixed, the asymptotic loss optimality does not hold due to Reason (R1).
When $\phi_n=\log n$ and the true model dimension diverges, the least squares model averaging is asymptotically both loss and risk optimal under certain conditons on $M_0$, $k_m$, and $\eta_n$. This can be attributed to the asymptotic loss and risk optimality of the true model, along with the model averaging method assigning all the weights to the true model.
In the case where the true model dimension diverges, MMA is asymptotically both loss and risk optimal under certain conditons. This result is mainly attributed to the condition that $k_{M_n}/k_{M_0+1}\to 1$, rendering the over-fitted models asymptotically negligible, although MMA may assign weights to them with a positive probability.
Last, as mentioned in Remark (ref), the results from wan2010, zhang2020PMA, zhang2021, and peng.opt also indicate asymptotic optimality in cases where the true model dimension diverges. Here, we compare their conditions with ours, using the recent work by peng.opt as an example. In the following discussion, we assume that $e_i$ is sub-Gaussian and Conditions (ref)--(ref) are satisfied. Define
where $\mathbf{P}_0=\mathbf{0}$ and $a\wedge b=\min(a, b)$ for any $a, b\in \mathbb{R}$. Under the conditions that $k_{M_0+1}\to \infty$ and $(k_{M_0+1}-k_1)/(n\eta_n)\to 0$, we can deduce from Lemma (ref) that $\xi_n/(\sigma^2 k_{M_0+1})\to 1$. Combining this with Corollary 1 of peng.opt, we conclude that if $k_{M_0+1}\to \infty$, $(k_{M_0+1}-k_1)/(n\eta_n)\to 0$, $k_{M_n} E\{(\hat{\sigma}^2-\sigma^2)^2\}/k_{M_0+1}\to 0$, and $\psi(\mathbf{K})/k_{M_0+1}\to 0$, then MMA is asymptotically both loss and risk optimal. In comparison with the second part of Theorem (ref)(ii) and the second part of Theorem (ref)(ii), their condition $\psi(\mathbf{K})/k_{M_0+1}\to 0$ imposes a weaker restriction for the over-fitted models than our condition $k_{M_n}/k_{M_0+1}\to 1$. However, their condition $k_{M_n} E\{(\hat{\sigma}^2-\sigma^2)^2\}/k_{M_0+1}\to 0$ requires the consistency of $\hat{\sigma}^2$, which is not necessary for our results from Condition (ref).
In this section, we conduct several simulation studies to illustrate the theoretical results presented in Section (ref). For each $i=1,\ldots,n$, we generate $y_i$ from the model (ref), where the covariates $(x_{i1}, x_{i2}, \ldots)$ are i.i.d. normal random vectors with zero mean and the covariance matrix between the $(k, l)$th element being $\rho^{|k-l|}$, and the random errors $e_i$ are i.i.d. from $N(0, \sigma^2)$ and are independent of $x_{ij}$'s. The population R-squared is denoted as $R^2=\mathrm{Var}(\mu_i)/\mathrm{Var}(y_i)$, which is controlled in the range of $\{0.05, 0.1, 0.5, 0.9\}$ via the parameter $\sigma^2$. For each configuration, the simulation is repeated 10000 times.
We consider least squares model averaging methods with $\phi_n=2$ and $\phi_n=\log n$. To assess the results in Lemma (ref) and Theorem (ref), we calculate the averages of $L_n(\mathbf{w}_{M_0+1}^0)/\inf_{\mathbf{w}\in \mathcal{H}_n} L_n(\mathbf{w})$ and $L_n(\hat{\mathbf{w}})/\inf_{\mathbf{w}\in \mathcal{H}_n} L_n(\mathbf{w})$. Similarly, to evaluate the results in Lemma (ref) and Theorem (ref), we calculate the averages of $R_n(\mathbf{w}_{M_0+1}^0)/\inf_{\mathbf{w}\in \mathcal{H}_n} R_n(\mathbf{w})$ and $R_n(\hat{\mathbf{w}})/\inf_{\mathbf{w}\in \mathcal{H}_n} R_n(\mathbf{w})$. We illustrate the performance through the following three examples, each involving different settings for $\beta_j$.
In this article, we study the asymptotic optimality of least squares model averaging with the weight set $\mathcal{H}_n$ for nested candidate models in the scenario where the true model is included in the set of candidate models. In contrast to the existing work by fang2023, our work is not confined to their special weight set $\mathcal{H}_n^{\delta}$, and we also explore the asymptotic risk optimality. A surprising finding is that when $\phi_n=\log n$ and the true model dimension is fixed, the asymptotic loss optimality does not hold. This result differs from the corresponding result of fang2023 and the asymptotic behavior of BIC. The reason behind this discrepancy is that the asymptotic loss optimality of the true model does not hold, while it holds for the restricted weight sets $\mathcal{H}_n(N)$ and $\mathcal{H}_n^{\delta}$.
In conclusion, we present several interesting directions for future research. First, the results in Lemma (ref)(i) and the first part of Theorem (ref)(i) were derived under the assumption of a fixed true model dimension. It remains an open question whether these results still hold in the case of a diverging true model dimension. Therefore, an appealing direction to relax the conditions of the results in this paper. Second, it is of interest to investigate the asymptotic optimality of jackknife model averaging hansen2012 in the scenario of this paper under heteroskedastic errors. Third, our current results are confined to nested models. It is desirable to extend the analysis to include non-nested models, but it is quite challenging as previously mentioned by hansen2014, zhang.liu2019, and fang2023. Last, it would be intriguing to explore similar issues in the context of other models, such as generalized linear models zhang2016jasa,yu.glm2024, quantile regression models lu2015, and time series models liao2021.joe. These problems warrant further investigations.
\setcounter{equation}{0} \setcounter{lemma}{0}