Minimax Semiparametric Learning With Approximate Sparsity

abstractEstimating linear, mean-square continuous functionals is a pivotal challenge in statistics. In high-dimensional contexts, this estimation is often performed under the assumption of exact model sparsity, meaning that only a small number of parameters are precisely non-zero. This excludes models where linear formulations only approximate the underlying data distribution, such as nonparametric regression methods that use basis expansion such as splines, kernel methods or polynomial regressions. Many recent methods for root-$n$ estimation have been proposed, but the implications of exact model sparsity remain largely unexplored. In particular, minimax optimality for models that are not exactly sparse has not yet been developed. This paper formalizes the concept of approximate sparsity through classical semi-parametric theory. We derive minimax rates under this formulation for a regression slope and an average derivative, finding these bounds to be substantially larger than those in low-dimensional, semi-parametric settings. We identify several new phenomena. We discover new regimes where rate double robustness does not hold, yet root-$n$ estimation is still possible. In these settings, we propose an estimator that achieves minimax optimal rates. Our findings further reveal distinct optimality boundaries for ordered versus unordered nonparametric regression estimation.

commentThis $\mathcal{M}_{C,\xi}$ is the set of functions $f(\cdot)$ such that $f(X)$ can be The idea is that $f(X)$ is the true function and $a$ are the coefficients of a sparse approximation to $\psi(X)^{\top}v$ with approximation rate $s^{-\xi}.$

itemize• Ordered \textcolor{black}{basis}: \textcolor{black}{Represented by area} $A\bigcup B\bigcup C\bigcup D$ in Figure (ref), \textcolor{black}{corresponding to } Eq ((ref)). • Non-ordered \textcolor{black}{basis} (approximate sparsity): \textcolor{black}{Represented by area} $A\bigcup B\bigcup C$ in Figure (ref), \textcolor{black}{corresponding to} Eq. ((ref)).

commentand their relationship to rate double robustness. Values of $(\xi_{1},\xi_{2})$ outside the box are those where the necessary condition of Equation ((ref)) is satisfied, for root-n consistency under approximate sparsity. Values outside the lower triangle are those where the necessary condition of Equation ((ref)) is satisfied, for root-n consistent estimation when the identity of the best regressors is known. The wedge below the box but above the triangle is the set of $\left( \xi_{1},\xi_{2}\right) $ where the necessary condition is satisfied for known identity of the best regressors but not for unknown identity. The existence of this wedge shows that it is more difficult to attain root-n consistency when the identity of the best regressors is unknown. {\color{red}Jelena: can you color these wedges in the plot as the current reading of the text is difficult ....}

comment\section*{Old Section 1} A regression slope is of interest in many settings. The coefficient of a binary treatment variable is a weighted average treatment effect in treatment models under the conditional independence assumptions of rosenbaum1983central. The coefficient of a price variable is important for economic demand models. There are also other important objects that depend on a regression, including average derivatives and average treatment effects. {\color{red} Jelena: the above paragraph is weird ... some examples are taken out of context - maybe rewrite for general, non econ audience - take a look at some AOS examples ...} We give necessary conditions for root-n consistent estimation of a regression slope or average derivative of an approximately sparse, high dimensional regression. We show that these conditions are substantially stronger than corresponding conditions for low dimensional regression. We also give estimators that are root-n consistent under minimal approximate sparsity of the regression or under the rate double robustness condition of the debiased machine learning literature. These estimators improve on existing estimators in being root-n consistent under more general conditions than previous estimators. {\color{red} JElena: need to add new results that were not present before} Approximate sparsity is the key condition here that determines whether it is possible to root-n consistently estimate a regression slope and other objects. Approximate sparsity refers to the minimum root mean square error in approximating a high dimensional function by a linear combination of $s$ regressors that are unknown elements of a known, large set of $p$ regressors, as in belloni2014inference. {\color{red} be more specific here - why only this paper .... there must be others not by the same group of people...Also it would be beneficial to motivate why we would be interested in studying ordered vs unordered settings ..are there practical significance or is this just a theorey concept that only we care about...} We consider sparse approximation rates of the form $s^{-\xi}$ for $\xi>0.$ {\color{red} Jelena: again - motive why this rate is interesting practicaly, why it is inovative theoretically, what does it allow that previous studies couldn't...} Our results depend on two such rates. The first rate $\xi_{1}$ is the sparse approximation rate for the regression and the second rate $\xi_{2}$ is the sparse rate for another function we refer to as the Riesz representer. {\color{red} maybe find an equaivalent name to be used here for the moment - in the introduction, without proper definition Riesz representer seems too artificial...} The key condition for root-n consistency is \begin{equation} \max\{\xi_{1},\xi_{2}\}>1/2. \end{equation} We show this condition is necessary for existence of a root-n consistent estimator of a regression slope or average derivative. We also show that, along with a few other regularity conditions, the minimal condition $\xi_{1}>1/2$ is sufficient for root-n consistent estimation of a regression slope, average derivative, average treatment effect, and any other linear functional that is mean-square continuous. The estimators are based on Lasso regression, Lasso estimation of a bias correction term, and special cross-fitting as explained in Section (ref). These estimators are also root-n consistent under the rate double robustness condition $\xi_{1}\xi_{2}>1/4,$ which is equivalent to the product of Lasso convergence rates for the regression and Riesz representer going to zero faster than $n^{-1/2},$ i.e. is equivalent to $\xi_{1}/(2\xi_{1}+1)+\xi_{2}/(2\xi_{2}+1)>1/2.$ These approximate sparsity conditions are more general than previously given in allowing for either $\xi_{1}>1/2$ or $\xi_{1}\xi_{2}>1/4$. In addition we give a regression slope estimator that attains root-n consistency under $\xi _{2}>1/2,$ where the approximate sparsity of the regression is not restricted, i.e. the regression can be "dense." This is a debiased machine learning estimator similar to chernozhukov2018double with another kind of special cross-fitting. {\color{red} this sentence and this paragraph must be strengthen: after reading that everytihg is the same as in chernozhukov2018double apart from cross-fitting, noone will accept the paper. Not all estimators need to be novel. But some are novel in this work. Our theory is completely novel. So I would emphasize only strong novelty - from theory and then add here and there some novel estimators and why are they needed - I would remove examples from this paragraph as they are artificial and difficult to parse right now ... } We are not aware of any existing estimator that only requires nice Riesz representer ($\xi_2>1/2$) and achieves the parametric rate for a regression slope under weak conditions allowing for non-Gaussian regressors and heteroskedastic errors. An important feature of sparse approximations is the unknown identity of the $s$ regressors that give the sparse approximation rate $s^{-\xi}$, which regressors we will refer to as the "best regressors." This feature distinguishes sparse approximations from series or sieve approximations for low dimensional regressions where the identity of the best regressors is known. We show that with known identity of the best regressors a necessary condition for root-n consistent estimation of a regression slope is \begin{equation} \xi_{1}+\xi_{2}\geq1/2. \end{equation} Equation ((ref)) is more restrictive than Equation ((ref)), showing increased requirements for attaining root-n consistency when the identity of the best regressors is unknown. {\color{red} can we emphasize this result much better - add lit on low dimensional sieve, motivate why we would care, why is theory different, what are practical implications...} Points above or to the right of the hyperbola in Figure (ref) are those where the rate double robustness condition $\xi_{1}\xi_{2}>1/4$ is satisfied, e.g. as in farrell2015robust and chernozhukov2018double. {\color{red}Jelena: Farell's conditions are much worse than many existing papers...there are a large number of other papers that discuss rate double robustness that need to be cited here..} We give estimators that attain root-n consistency on the union of points above the hyperbola with points to the right of the box where $\xi_{1}>1/2$, a larger set of $(\xi_{1},\xi_{2})$ values than previously obtained. The estimator of a regression slope that attains root-n consistency for a dense regression does so for points above the box where $\xi_{2}>1/2$. We also obtain a lower bound for convergence rates under approximate sparsity when $\tilde{\xi}:=\max\{\xi_{1},\xi_{2}\}\leq1/2$ that is $(\log (p)/n)^{2\tilde{\xi}/(2\tilde{\xi}+1)},$ which is illustrated in Figure (ref). This lower bound is also determined by the maximum of the sparse approximation rates. We can see here that approximate sparsity leads to convergence rate bounds that are slower by a power of $n$ than bounds where the best regressors are known. For example when $\xi_{1}=\xi_{2}=1/4$ Equation ((ref)) is satisfied so that the lower bound is $n^{-1/2}$ when the best regressors are known but under approximate sparsity the lower bound is $(\log(p)/n)^{1/3},$ that goes to zero slower than $n^{-1/3}.$ {\color{red} Jelena: This last point needs to be emphasized much more - its a bit buried here and there are no citations...} \begin{figure}[ptb] \caption{Minimal conditions for the root-n rate} \begin{centering} \end{centering} The above figure displays $2\tilde{\xi}/(2\tilde{\xi}+1)$, where $\tilde{\xi }=\max\{\xi_{1},\xi_{2}\}$. \end{figure} The first estimator of a regression coefficient we give is different than belloni2014inference and is more like the debiased Lasso estimators of javanmard2014confidence,javanmard2018debiasing, van2014asymptotically, zhang2014confidence, and cai2017confidence. {\color{red} Jelena: again the estimators vs theoretical conditions are conflated togetehr - we are either discusssing conditions or estiamtors ....ptherwise they'll just say we are the same as viktor's other papers and we will be in toruble...} To explain the relationship of the conditions here to the conditions in these papers it is helpful to relate the strict sparsity conditions of these papers to the size of $\xi_{1}$ and $\xi_{2}.$ The regression strict sparsity condition in these papers is $s_{0}(\log p)/\sqrt{n}\rightarrow0,$ where $s_{0}$ is the number of nonzero coefficients (and the second moment matrix of the regressors is unknown). This condition is equivalent to the root mean square convergence rate for Lasso regression being faster than $n^{-1/4}.$ With approximate sparsity Lasso will converge faster than $n^{-1/4}$ when $\xi_{1}>1/2,$ so that $\xi_{1}>1/2$ corresponds to the strict sparsity conditions in the previous papers. Except for cai2017confidence and javanmard2014confidence, these papers also impose the corresponding approximate or strict sparsity condition on the residual from regressing the variable of interest on the other regressors. Because the Riesz representer here is proportional to this residual that sparsity condition corresponds to $\xi_{2}>1/2.$ We do not require this condition. In addition we obtain root-n consistency under the doubly robust rate condition $\xi_{1} \xi_{2}>1/4$ that allows for $\xi_{1}<1/2$ if $\xi_{2}$ is large enough. The conditions here also differ from the previous literature, except for belloni2014inference, in providing standard errors that are robust to heteroskedasticity and in allowing for non Gaussian disturbances and regressors; see Corollary (ref) in Section (ref). {\color{red}Jelena: It cannot be that Bellonit et al is the only paper that discusses heterodscedastic errors...}The results here are also for approximately sparse specifications that may be more credible in some applications than strictly sparse specifications. The regression coefficient estimator that attains root-n consistency when $\xi_{2}>1/2$ and the regression is dense is a debiased machine learning estimator like chernozhukov2018double. This estimator builds on zhu2018linear and bradic2022testability. It differs from the previous literature in standard errors being robust to heteroskedasticity and in not requiring Gaussian regression or regressors. As far as we are aware the conditions given here are the weakest yet given for root-n consistent estimation of a slope when the regression is dense. {\color{red}Jelena: same comment as above - estimators or theory? Also I don't quite get what is new being said in this paragraph ...its quite weak and will kill the paper ..} The estimators of an average derivative, average treatment effect, and a general linear functional are automatic debiased machine learners similar to chernozhukov1802.08667 and chernozhukov2018learning with special cross-fitting. These attain root-n consistency under either $\xi_{1}>1/2$ or $\xi_{1}\xi_{2}>1/4$. athey2018approximate gave an estimator of the average treatment effect on the treated that is root-n consistency for a homoskedastic regression with sparsity corresponding to $\xi_{1}>1/2.$ Also, hirshberg2021augmented recently showed in independent work that their minimax estimator of a regression functional is root-n consistent for a high dimensional model under conditions similar to $\xi_{1}>1/2.$ In comparison the automatic debiased machine learner given here is relatively simple to compute, has simple heteroskedasticity robust standard errors, is doubly robust, and is root-n consistent under $\xi_{1}\xi_{2}>1/4$, as discussed in Section (ref). {\color{red} Jelena: words like automated DML are introduced without any context. Again the message here is - we are the same as all these other papers . Need to talk about how to rewrite this better ...} The minimal approximate sparsity condition of Equation ((ref)) is analogous to the minimal strict sparsity conditions of javanmard2018debiasing. A crucial difference is that our results do not depend on a quantity that is closely related to the $\ell_1$-norm of the rows of the precision matrix, i.e., the quantity $\rho$ in javanmard2018debiasing. Moreover, in contrast to Remark 4.3 therein, the lower and upper bounds here match, thereby establishing the minimax rate. Our work differs from cai2017confidence in involving the approximate sparsity of the Riesz representer, a feature that is not present for the regression functionals considered there. Targeted maximum likelihood van2006targeted based on machine learners has been considered by van2011targeted and large sample theory given by luedtke2016optimal, toth2016tmle, and zheng2018marginal. mackey2018orthogonal showed that weak sparsity conditions would suffice for $1/\sqrt{n}$ consistency of a certain estimator of a partially linear conditional mean when certain variables are independent and non Gaussian. The estimator given there will not be consistent for the objects and model we consider. {\color{red} Jelena: this needs to be elaborated - why is the last sentence true ?} Our model specification is like van2014asymptotically in being an explicit semiparametric model, or more precisely a nonparametric model where the object of interest is a functional of a regression. This modeling approach facilitates comparison with conditions for root-n consistent estimation in the semiparametric literature where the best approximating functions are known, especially conditions of robins2009semiparametric. robins2009semiparametric showed that Equation ((ref)) is a necessary conditions for root-n consistent estimation when unknown functions are in H\"older classes for a regression coefficient and the average treatment effect. In a H\"older class each of $\xi_{1}$ and $\xi_{2}$ will equal the ratio of the number of derivatives of the regression and Riesz representer, respectively, to the dimension of the regressors when the regressors are splines. We consider instead necessary conditions in terms of sparse approximation rates, which is more general but perhaps less intuitive. In this setting the series estimator of donald1994series attains root-n consistency under the minimal condition of Equation ((ref)). It is interesting to note that in the approximately sparse setting debiased machine learning based on regression that converges at the optimal nonparametric rate is root-n consistent under minimal conditions. In contrast, in the semiparametric, low dimensional setting a root-n consistent estimator can be obtained under the minimal condition $\xi_{1}=1/2$ without debiasing but seems to require undersmoothing where the convergence rate of the regression is not optimal, e.g. see newey2018cross. Thus we see that the debiasing of regression learners that converge at optimal nonparametric rates seems particularly well suited to approximately sparse models. Define $\mathcal{B}$ to be the set of mean square limits of sparse specifications, i.e. limits of linear combinations of a finite number of elements of the sequence $X$. {\color{red}Jelena: this B needs a math definition}Let $\mathrm{proj}(a(X)|\mathcal{B)}$ denote the linear projection of any $a(X)$ with finite second moment on $\mathcal{B}$. For any function $f$, we denote $\left\Vert f\right\Vert _{2}=\sqrt{\mathbb{E}[f(X)^{2}]}$. For any vector $v$, let $\Vert v\Vert_{0}$ denote the number of nonzero elements of $v$. For any vector $v=(v_1,...,v_p)^{\top}$ and $q \in [1,\infty)$, let $\|v\|_q=\left(\sum_{j=1}^{p} |v_j|^q \right)^{1/q}$ and $\|v\|_\infty=\max_{1\leq j\leq p} |v_j|$ . For $J \subset \{1,...,p\}$ and $v=(v_1,...,v_p)^{\top} \in\mathbb{R}^p$, $\left\vert J\right\vert $ denote the number of elements of $J,$ $v_{J}$ be the vector in $\mathbb{R}^p$ consisting of $(v_{J})_j=v_{j}$ for $j\in J$ and $(v_{J})_j=0$ otherwise, and $v_{J^{c}}$ be the corresponding vector for $J^{c}$ (so that $v=v_{J}+v_{J^{c}})$. The rest of this paper is organized as follows. In Section (ref) we describe the parameters of interest. Section (ref) explains the approximately sparse setting of the paper and gives minimax lower bounds. Section (ref) describes the estimators we give. Section (ref) derives their properties.

lemmaSuppose that $\mathbb{E}[\psi(X)\psi(X)^{\top}]=I_p$. Define the set \(\Theta(C, k)\) of sparse linear combinations of the basis functions with bounded $L_2$ norm: $$ \Theta(C, k) = \Biggl\{ \theta^\top \psi(\cdot) \mid \theta \in \mathbb{R}^p, \|\theta\|_0 \leq k, \|\theta\|_2 \leq C \Biggl\}, $$ for some constant $C>0$. \textcolor{black}{As} the sparsity $k$ and the dimension $p$ grow to $\infty$ there \textcolor{black}{do} not exist constants $D$ and $\xi>0$ such that $$\Theta(C,k) \subset \mathcal{M}_{D,\xi}.$$

cases1, & if k=\ell,\\ 0, & otherwise.

lemmaFor any $t\geq1$, consider the corresponding subset of basis functions $ \bigl\{\psi_k : k\in \mathcal{N}_t^D\bigr\}$. Then, there exists a constant $C>0$, depending only on $D,s,\Omega$, such that \begin{equation} \inf_{a_{k}\in\mathbb{C}} \Bigl\| f-\sum_{k\in\mathcal{N}_{t}^{D}}a_{k}\psi_{k}\Bigl\|_{L^{2}(\Omega)}\leq CQ \ t^{-s/D}, \qquad \forall f\in B_{H^{s}(\Omega)}(Q). \end{equation}

comment\textcolor{PantoneGreen}{Yinchu: Whitney thinks that Lemma (ref) is already known, see Theorem 1 of Gallant (1981), which in the dropbox folder. One idea is to just say that suppose that we have the rate in Lemma (ref) and we can construct the examples below (like Section 2.3.1). These examples are the more interesting ones anyway. } \textcolor{pantoneRed}{Jelena: Again, this is a problem - is theorem 1 though the same as this lemma - the theorem there has k going to infinity and has the limit to be zero - does not look the same to me...cProof of that theorem tells me that every function has an infinite expansion - we are dealing with finite one. Also Theorem 1 there has NO proof.} True. They cited a Polish paper in 1977, which is not really available. I guess the question is whether Lemma (ref) is a main contribution. I feel that the examples afterwards are the main point of having this section, right? Just giving examples. How about the following?\\

commentGiven $x=(x_{1},...,x_{D})^{\top}$ with $D>d$, a natural approximately sparse class is \[ \mathcal{F}_{1}=\left\{ f(x)=f_{0}(x_{l_{1}},...,x_{l_{d}}):\ f_{0}\in\mathcal{Q}\right\} . \] Here, $f_{0}$ is a function of $d$ variables although the dimension of $x$ is $D$, which can be much larger. Then we give other examples. We consider functions that can be approximated by ordered basis functions: \begin{equation} \mathcal{Q}=\left\{ f(\cdot):\ \left\Vert f-\sum_{j=1}^{s}a_{j}\psi_{j}\right\Vert _{2}\leq Cs^{-\xi}\right\} . \end{equation} for an ordered dictionary $\{\psi_{j}\}_{j=1}^{\infty}$.

assumptionConsider the model in ((ref)). The random vector $W_i=(Y_i,X_i^\top)^\top$ with $X_i=(D_i,Z_i^\top)^\top$ follows a Gaussian distribution with mean zero and $\mathbb{E}[Z_{i}Z_{i}^{\top}]=I_{p}$. Furthermore, $f(\tilde{Z}_i)=Z_i^\top \beta$ and $g(\tilde{Z}_i)=Z_i^\top \phi$.

theoremLet $\tilde{\xi}=\max\{\xi_1,\xi_2 \}$. Suppose that Assumption (ref) is satisfied and there exist constants $\kappa_1,\kappa_2>0$ such that $ \kappa_{1}\log p\leq n\leq\kappa_{2}p\log p$ for large enough $n$. If $\tilde{\xi}\leq 1/2$, then \textcolor{black}{\[ \mathcal{R}_{\xi_{1},\xi_{2}}\geq C(n^{-1}\log p)^{2\tilde{\xi}/(2\tilde{\xi}+1)}, \]} where $C>0$ is a constant depending only on $\tilde{\xi}$, $\kappa_1,\kappa_2,C_{0}$.

commentThe average derivative is a harder problem than partial linear models and hence the lower bound in Theorem (ref) applies to the problem of average derivative. To see this, consider a function of $(Z_{i},X_{i})$. A special case is when the partial derivative with respect to $Z_{i}$ is constant. In this special case, the average derivative problem becomes learning a coefficient in a partial linear model. By Theorem (ref), even in this special problem, $\max\{\tilde{\xi}_{1},\tilde{\xi}_{2}\}>1/2$ is a necessary condition for attaining root-n consistency. Therefore, in general, one needs to impose $\max\{\tilde{\xi}_{1},\tilde{\xi}_{2}\}>1/2$ to obtain the $1/\sqrt{n}$ rate for the average derivative.

commentWe observe $\{(Y_{i},D_{i},Z_{i})\}_{i=1}^{n}$ from \[ Y_{i}=D_{i}\beta+Z_{i}^{\top}\phi+\varepsilon_{i}\qquad{\text{and}}\qquad D_{i} =Z_{i}^{\top}\pi+u_{i}, \] where $\gamma,\pi\in{\mathbb{R}}^{p}$, $Z_{i}\sim N(0,I_{p})$ is independent of $(\varepsilon_{i},u_{i})^{\top}\sim N(0,{\rm diag}(\sigma_{\varepsilon}^{2},\sigma_{u}^{2}))$. The distribution of a data observation $W$ is determined by the parameter $\theta=(\beta ,\phi,\pi,\sigma_{\varepsilon}^{2},\sigma_{u}^{2})$. We are interested in $\beta$. We consider the ordered space \[ {\mathcal{S}}_{M_1,\xi}=\left\{ x=(x_{1},...,x_{p})^{\top}\in{\mathbb{R}} ^{p}:\ \sqrt{\sum_{j=t+1}^{p}x_{j}^{2}}\leq M_{1}t^{-\xi}\qquad\forall t\in\{1,...,p\}\right\} . \]

theorem\textcolor{black}{If Assumption (ref) holds, and $p\gtrsim n^{2}$, then, for all $\xi_{1}+\xi_{2}<1/2$, we have} $$\mathcal{R}^{\rm{Ordered}}_{\xi_{1},\xi_{2}}\gg n^{-1/2}.$$

remark\textcolor{black}{C}omparing Theorems (ref) and (ref) \textcolor{black}{highlights the crucial} role of ordering \textcolor{black}{in the dictionary of basis functions.} \textcolor{black}{In the {\bf{unordered}} case-- where} the most important terms might not be the first a few basis \textcolor{black}{functions--} the parametric \textcolor{black}{convergence} rate is only possible when $\max\{\xi_1,\xi_2\}>1/2 $. \textcolor{black}{Conversely,} for {\bf ordered} models \textcolor{black}{-- where the most significant terms appear first in the dictionary--} the parametric rate \textcolor{black}{can be achieved as long as} $\xi_1+\xi_2\geq 1/2$. As illustrated in Figure (ref), the \textcolor{black}{condition} $\xi_1+\xi_2\geq 1/2$ is \textcolor{black}{ considerably less restrictive} than $\max\{\xi_1,\xi_2\}>1/2 $ \textcolor{black}{-- this difference is depicted in the figure as a triangle versus a square}. \textcolor{black}{This disparity underscores the cost of} being unable to order the basis functions \textcolor{black}{according to} their approximation power.

comment\textcolor{black}{with the conditions} $ \mathbb{E}(\varepsilon_{i}\mid D_{i},Z_{i})=0$ and $\mathbb{E}(u_{i}\mid Z_{i})=0$. \textcolor{black}{Observe that covariance between $\varepsilon_i$ and $u_i$ is allowed to be non-zero}. \textcolor{PantoneGreen}{Yinchu: I think the covariance between $\varepsilon_i$ and $u_i$ is implied to be zero.} \textcolor{black}{Here, \(D_i \in \mathbb{R}\) is a scalar regressor, and \(Z_i \in \mathbb{R}^p\) is a \(p\)-dimensional vector of controls. For notational convenience, define the \((p+1)\)-dimensional regressor vector and parameter vector as} $X_{i}=(D_{i},Z_{i}^{\top})^{\top}\in\mathbb{R}^{p+1}$ and $\beta=(\theta,\gamma^{\top})^{\top}\in\mathbb{R}^{p+1}$. \textcolor{black}{Moreover, introduce the vector $\phi$ as } $ \sigma_u^2 \phi = (1, -\pi^\top )^\top \in \mathbb{R}^{p+1}, $ where \(\sigma_u^2 = \mathbb{E}[u_i^2]\). \textcolor{PantoneGreen}{Yinchu: Maybe we should say $Y_{i}=D_{i}\theta+f(Z_{i})+\varepsilon_{i}\quad\text{and}\quad D_{i}=g(Z_{i})+u_{i}$, where $f\in \mathcal{M}_{C,\xi_1}$ and $g \in \mathcal{M}_{C,\xi_2}$? Indeed, this is a good question. What model should we write here?} \textcolor{pantoneRed}{Jelena: I think this second one that looks nonparameteric but wouldn't that contradict the conditional mean assumption on the errors above ?} \textcolor{PantoneGreen}{Yinchu: What would be the problem for conditional mean zero of the error term? We can assume this nonparametric-looking model and still say $\mathbb{E} (\varepsilon_{i}\mid Z_{i},D_{i})=0$, no? }

comment\textcolor{pantoneRed}{Jelena: Yinchu: The above looks new to me - compare with existing literature -motivate and brag if novel} \textcolor{PantoneGreen}{Yinchu: I think it's new. Should we compare with Victor's papers? Like the one in econometrica?} \textcolor{pantoneRed}{Jelena: we shouldn't cite viktor everywhere - but in general that is one of the papers - this is also related to devbiasing literuature, one step estimators etc ...} \textcolor{PantoneGreen}{Yinchu: I cited some debiasing papers in the paragraph right before Section 5. I guess we can cite the same papers here. But perhaps the above discussion already shows that our estimator is clearly different from any existing estimator? }

assumptionConsider the model in ((ref)). The eigenvalues of $\Sigma=\mathbb{E}[X_{i}X_{i}^{\top}]$ and $\Sigma^{-1}$ bounded by constants. For some constant $\kappa>0$, $\mathbb{E}(\varepsilon_{i}^{2}\mid X_{i})$, $\mathbb{E}|\varepsilon_{i}|^{2+\kappa}$, $\mathbb{E} (u_{i}^{2}\mid Z_{i})$, $(\mathbb{E} u_{i}^{2})^{-1}$ and $\mathbb{E}|Y_{i}|^{2+\kappa}$ are also bounded by constants. Moreover, either \[ \|X_{i}\|_{\infty}\ \text{and }\mathbb{E}|u_{i}|^{2+\kappa}\text{ are bounded} \] or \[ X_{i}\text{ has bounded sub-Gaussian norm}. \] Furthermore, $p\gg n^{1/(2\min\{\xi_{1},\xi_{2}\})}$ and $\log p \ll n^{\kappa/(2\kappa+4)}$.

comment\textcolor{pantoneRed}{Jelena: Where in the proof do we need $p$ to be bigger than something? This seems new and it eliminates low-dimensional models in general} \textcolor{PantoneGreen}{Yinchu: So the approximation error is $p^{-\xi}$, which is from the definition of approximate sparsity. We need this to be smaller than $n^{-1/2}$, which gives $p\gg n^{1/(2\xi)}$. This is needed because we are not assuming that including all the $p$ terms would be the true model. Yes, it rules out low-dimensional models just because the idea of approximate sparsity assumes that the low-dimensional models are not accurate enough. } \textcolor{pantoneRed}{Jelena: If we are always requiring the approximation error to be smaller than root - n, then we are indeed not that different from what Viktor was doing in 2014(ish) papers with what he called approximate sparsity or approximately sparse model....$y=x\beta + r_n$ and $r_n=o(1/\sqrt{n})$... How do you think about this yourself ? this is different from Lemma 2... We just want to differentiate from those papers ..... why do we need this to be smaller than root - n: I assume it is because we are looking for root-n inference....} \textcolor{PantoneGreen}{Yinchu: Well, it's true that Victor's other papers kinda do that. For example, belloni2014inference assume that the approximation error with $p$ terms is $o(n^{-1/2})$ and the approximation error with $s$ terms is $O(\sqrt{s/n})$. In our case, $s= (n/log p)^{1/(2\xi+1)}$. Based on this, we check whether $\sqrt{s/n} \lesssim s^{-\xi}$. However, this looks like assuming whatever is needed to make the proof work, rather than starting from a well-defined function class and working out its properties. We tried to discuss this in Section 1.3. Perhaps we need to be more clear? BTW, which lemma 2 do you mean? You mean lemma 2.1? Well, Lemma 2.1 just says that we are different from strict sparsity.} \textcolor{PantoneGreen}{Yinchu: I changed assumption 2. Below is the old one. \begin{assumption}[Old Assumption 2] Consider the model in ((ref)). There exist constants $\kappa_{1},\kappa_{2}>0$ such that for $\ell\in\{1,2\}$, \[ P\left(\min_{|J|\leq\kappa_{1}n/(\log p)}\min_{\|v_{J^{c}}\|_{1}\leq\|v_{J}\|_{1}}\frac{v'\hat{\Sigma}_{(\ell)}v}{\|v_{J}\|_{2}^{2}}\geq\kappa_{2}\right)\geq1-o(1). \] The eigenvalues of $\Sigma=\mathbb{E}[X_{i}X_{i}^\top]$ are in $[\kappa_{3},\kappa_{4}]$ for some constants $\kappa_{3},\kappa_{4}>0$. There are constants $\kappa_{5},\kappa_{6},\kappa_{7},\kappa_{8}>0$ such that $\mathbb{P}(\|X_{i}\|_{\infty}\leq\kappa_{5})=1$, $\mathbb{P}\left(\mathbb{E}(\varepsilon_{i}^{2}\mid X_{i})\leq\kappa_{6}\right)=1$, $\mathbb{E}(Y_{i}^{2})\leq\kappa_{7}$ and $\mathbb{E}|\varepsilon|^{2+\kappa_{8}}\leq \kappa_{7}$. \end{assumption} } \textcolor{pantoneRed}{Jelena: We brag about non-gaussian stuff but then assume that the covariates are bounded - if they are bounded one can show RE condition right ? if bounded then sub-gaussian and then RS condition hold.... most statisticians don't like to see the condition above } \textcolor{PantoneGreen}{Yinchu: We can verify RE condition with bounded designs. For example Rudelson and Zhou paper. We are not really assuming it's sub-Gaussian. We are just saying it can be non-Gaussian. Basically, we only need to assume boudnedness and Rudelson and Zhou would give the RE condition.} \textcolor{pantoneRed}{Jelena: so precisely that - we either have bounded covariates or RE - not both as is written above ...} \textcolor{pantoneRed}{Jelena: Also why do we need l2 norm of beta to be bounded - is this new ? I thought we removed any norm condition on beta...} \textcolor{PantoneGreen}{Yinchu: Unfortunately, we need $\gamma$ to have bounded $L_2$ norm. Basically, our estimator uses things like $E_n X_i Y_i$. If $\|\gamma\|_2$ is unbounded, then entries of $E_n X_i Y_i$ can explode. } \textcolor{pantoneRed}{Jelena: we should really discuss this - I am not convinced this is necessary}

theorem\textcolor{black}{Consider the model in ((ref)). Let Assumption (ref) hold. }Suppose that $$\log p\ll n^{1/2-1/(4\max\{\xi_{1},\xi_{2}\})},$$ $f\in\mathcal{M}_{C_{0},\xi_{1}}$ and $g\in\mathcal{M}_{C_{0},\xi_{2}}$ for some constants $C_{0},\xi_{1},\xi_{2}>0$ such that $\max\{\xi_{1},\xi_{2}\}>1/2$. Then \[ \sqrt{n}(\tilde{\theta}-\theta)=n^{-1/2}\sigma_{u}^{-2}\sum_{i=1}^{n}u_{i}\varepsilon_{i}+o_{P}(1) \] and $\tilde{\theta}-\theta=O_{P}(n^{-1/2})$.

comment\textcolor{PantoneGreen}{Yinchu: Notice that Theorem (ref) allows for functions that have not been considered in existing literature. For example, belloni2014inference assume that there exists an $s$-approximation with $s\ll \sqrt{n}/\log p$ such that the approximation error is $O(\sqrt{s/n})$. This rules out certain functions in $\mathcal{M}_{C,\xi}$ for $\xi<1/2$. To see this, suppose that there exists $s$ such that $s\ll \sqrt{n}/\log p$ and $s^{-\xi}\lesssim \sqrt{s/n}$. Then $n^{1/(2\xi+1)}\lesssim s\ll \sqrt{n}/\log p$, which is impossible for $\xi<1/2$. On the other hand, Theorem (ref) allows one function to be in $\mathcal{M}_{C,\xi}$ for $\xi<1/2$ as long as the other function is in $\mathcal{M}_{C,\tilde{\xi}}$ for some $\tilde{\xi}>1/2$. Feel free to delete/rephrase/move this paragraph. }

commentFor the above to be valid, a standard assumption is the largest eigenvalues of $\Sigma$ and $\Sigma^{-1}$ are bounded by some constant $C>0$. For simplicity, we also assume that $P(\left\Vert \psi(X)\right\Vert _{\infty}\leq C)=1$ although sub-Gaussian properties would also be applicable. \begin{assumption} For some $c>0$, $m(W,\psi_{j})=\tilde{m}_{0}(W)\tilde{m}_{j}(W)$ such that $\mathbb{E}[|\tilde{m} _{0}(W)|^{2+c}]\leq C$ and $\max_{j\leq p}\left\vert \tilde{m}_{j}(W)\right\vert \leq C$. The largest eigenvalue of $\mathbb{E}[\tilde{m}_{0}^{2}(W)\tilde{m}(W,b)\tilde{m}(W,b)^\top] $ is bounded by $C$, where $\tilde{m}(W,b)=(\tilde{m}_{1}(W),...,\tilde{m}_{p}(W))^\top\in\mathbb{R}^{p}$. Moreover, the largest eigenvalue of $\mathbb{E}[\alpha_0(X)^{2} \psi(X)\psi(X)^{\top}]$ is bounded by a constant. \end{assumption} The decomposition of $m(W,\psi_j)=\tilde{m}_0(W)\tilde{m}_j(W)$ allows for unbounded $m(W,\cdot)$. This is particularly important in the context of learning the ATE, as it allows for unbounded propensity scores (see Remark (ref)), marking a new contribution to the literature. \begin{assumption} There exist constants $c,C>0$ with $EY^2\leq C$, $\mathbb{E} |Y-\mathbb{E}(Y|X)|^{2+c}\leq C$ and $\mathbb{E}((Y-\mathbb{E}(Y|X))^2|X)\leq C$ with probability one. \end{assumption} \textcolor{PantoneGreen}{Yinchu: I simplified the assumptions. We only need the following two assumptions: (ref) and (ref). I wrote Assumption (ref) like Assumption (ref). }

assumptionThe largest eigenvalues of $\Sigma$ and $\Sigma^{-1}$ are bounded by some constant $C>0$. There exists a constant $\kappa>0$ such that $\mathbb{E} |\alpha_{0}(X_{i})|^{2+\kappa}$, $\mathbb{E} |Y|^{2+\kappa}$, $Var(Y\mid X)$ and the largest eigenvalues of $\mathbb{E}[\alpha_0(X)^{2} \psi(X)\psi(X)^{\top}]$ and $\mathbb{E} [m(W,\psi)m(W,\psi)^{\top}]$ are bounded. Moreover, either \[ \|\psi(X_{i})\|_{\infty}\ \text{and }\mathbb{E}|u_{i}|^{2+\kappa}\text{ are bounded} \] or \[ \psi(X_{i})\text{ has bounded sub-Gaussian norm}. \] The sub-Gaussian norm of $m(W,\psi_{j})$ is bounded uniformly in $j\in \{1,...,p \}$. Furthermore, $p\gg n^{1/(2\min\{\xi_{1},\xi_{2}\})}$, $\log p \ll n^{\kappa/(2\kappa+4)}$ and $\log p\ll n^{1/2-1/(4\max\{\xi_{1},\xi_{2}\})}$.

commentNotice that we do not assume that the error $Y-\mathbb{E}(Y|X)$ is sub-Gaussian or bounded. We only require $2+c$ moments for some $c>0$. Therefore, we allow for heavy-tailed errors. \textcolor{PantoneGreen}{Yinchu: Maybe we want to brag about this? There are two recent papers in Annals of Stats that only solves this problem for nonparametric estimation, see han2019convergence and kuchibhotla2022least.} \textcolor{pantoneRed}{Jelena: YESS} \textcolor{PantoneGreen}{Yinchu: OK. I'll dig out these papers and see what we can brag about.} \textcolor{PantoneGreen}{Yinchu: The sub-Gaussian entries of $m(W,\psi)$ can be relaxed to $m(W,\psi)/q(W)$ having sub-Gaussian entries and $q(W)$ having $2+\kappa$ moments, but maybe we don't need to do this because we don't have an example. For ATE, entries $m(W,\psi)$ are either $\psi_j$ or zero, which are sub-Gaussian; see Remark (ref). }

assumptionApproximation parameters $\xi_1$ and $\xi_2$ satisfy \begin{equation} \xi_{1}>1/2, \quad n\gg (\log p)^{(2\xi_1)/(\xi_1-1/2)} \tag{i} \end{equation} or \begin{equation} \xi_{1}\xi_{2}>1/4, \quad n^{\xi_1\xi_2-1/4}\gg (\log p)^{2\xi_1\xi_2+(\xi_1+\xi_2)/2}. \tag{ii} \end{equation}

theoremLet Assumptions (ref) and (ref) hold. Then $ \hat{\theta}-\theta_{0}=O_{P}(n^{-1/2}), $ and in particular, \[ \hat{\theta}-\theta_{0}=n^{-1}\sum_{i=1}^{n}\tilde{\varUpsilon}_{n}(W_{i})+o_{P}(n^{-1/2}), \] where $\tilde{\varUpsilon}_{n}(W_i)=m(W_i,\rho_{n})-\theta_{n}+\alpha_{n}(X_i)[Y_i-\rho_{0}(X_i)]$ such that $\theta_n=\mathbb{E} m(W_i,\rho_n)$, $\mathbb{E}\tilde{\varUpsilon}_{n}(W_i)=0$ and $\mathbb{E}\tilde{\varUpsilon}_{n}(W_i)^2$ is bounded by a constant.

remarkFor a procedure that can be applied to a generic problem, this estimator achieves parametric rate under weaker conditions compared to some existing estimators that are designed for specific problems. In the problem of estimating ATE (Example (ref)), no existing estimator can guarantee the parametric rate in both $\xi_1>1/2$ and $\xi_1 \xi_2>1/4$. athey2018approximate only impose the assumptions on $\rho_0$, which would roughly correspond to $\xi_1>1/2$ here; however, whether their estimator achieves the parameter rate under $\xi_1 \xi_2>1/4$ is unknown. Works such as farrell2015robust, ning2020robust and chernozhukov2022debiased among others impose assumptions that imply $\xi_1 \xi_2>1/4$; there is guarantee of the parametric rate under $\xi_1>1/2$. \begin{comment} In the problem of estimating ATE (Example (ref)), athey2018approximate only impose the assumptions on $\rho_0$, which would roughly correspond to $\xi_1>1/2$ here; however, whether their estimator achieves the parameter rate under $\xi_1 \xi_2>1/4$ is unknown. \textcolor{pantoneRed}{Jelena: Much more details are needed - comparison with other papers needed - comparison with other reisz representers papers needed} \textcolor{PantoneGreen}{Yinchu: How about replacing this discussion with the following? } \textcolor{PantoneGreen}{Yinchu: However, this is an exception, which is our paper bradic2019sparsity. The recent paper (https://arxiv.org/abs/2305.04174) is another exception; they claim that they achieve double robustness, but they assume essentially bounded L1 norm of the parameters, which would give almost rate double robustness. (Under bounded L1 norm, we automatically get the rate $(n^{-1}\log p)^{-1/4}$ for both the propensity score and the regression. I don't know how to cite/discuss this paper. } \end{comment}

assumptionThere exist a constant $c>0$ such that $\mathbb{E} |m(W,\rho_{n})|^{2+c}$ is bounded.

theoremLet the Assumptions (ref), (ref) and (ref) hold. Then \[ \frac{n^{1/2}(\hat{\theta}-\theta_0)}{\sqrt{\mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2}}}\overset{d}{\rightarrow}N(0,1) \] and $\hat{V}=\mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2}+o_{P}(1)$.

comment\begin{example}[Individual Coefficients] Suppose that $X=(D,Z)$ and $W=(Y,D,Z)$. Then $\rho_0(d,z)=\mathbb{E}(Y\mid (D,Z)=(d,z))$. Suppose that the dictionary is $\psi_1(d,z)=d$ and $\psi_j(d,z)=\tilde{b}_{j-1}(z)$ for $j\geq 2$, where $\{\tilde{b}_j(z)\}_{j=1}^{\infty}$ is a dictionary of basis functions of $z$. Then this is a partial linear model where $\rho_0(X)=\gamma_0 \cdot D+h_0(Z)$ and $h_0(Z)$ is in the closure in $\left\Vert f\right\Vert _{2}$ of finite linear combinations of $\{\tilde{b}_j(z)\}_{j=1}^{\infty}$. Consider the functional $m(W,f)$ with \begin{equation} m(W,f)=\frac{f(\bar{d},Z)-f(d,Z)}{\bar{d}-d}, \end{equation} where $d$ and $\bar{d}$ are distinct values in the support of $D$. The functional $\mathbb{E}[m(W,f)]$ with $m(\cdot,\cdot)$ in ((ref)) is simply the coefficient $\gamma$ for $D$. The Riesz representer is $\alpha_0(D,Z)=U/\mathbb{E}(U^2) $, where $U=D-g(Z)$ and $g(Z)$ minimizes $\mathbb{E}(D-f(Z))^2$ over the closure in $\left\Vert f\right\Vert _{2}$ of finite linear combinations of $\{\tilde{b}_j(z)\}_{j=1}^{\infty}$. This example was not previously considered as a linear functional of a conditional mean in chernozhukov2022locally and hirshberg2021augmented. It is included here to show how the results of this Section apply to the coefficient of a partially linear regression. \begin{remark}[Example (ref)] For Example (ref), we can view $\tilde{m}_0(W)=1/(\bar{d}-d) $ and $\tilde{m}_j(W)=\psi_{j}(\bar{d},Z)-\psi_{j}(d,Z)$. Thus, $\tilde{m}_1(W)=\bar{d}-d$ and $\tilde{m}_j(W)=\tilde{b}_{j-1}(Z)$ for $j\geq 2$. Assumption (ref) holds whenever $U$ has bounded eigenvalues and $\mathbb{E}(U^2)$ is bounded below by a positive constant, where $V$ is the covariance of $(1, \tilde{b}_{1}(Z),...,\tilde{b}_{p-1}(Z))^{\top} $. We also notice that $\mathbb{E}[\alpha_0(X)^{2} \psi(X)\psi(X)^{\top}]=\mathbb{E}(U^2 \psi(X) \psi(X)\psi(X)^{\top})/\mathbb{E}(U^2)$ has bounded eigenvalues if, in addition, $\mathbb{E}(U^2|Z) $ is bounded. \end{remark} \end{example}

example[Average Treatment Effect] Suppose that we are interested in the average treatment effect of a binary $D\in\{0,1\}$ treatment variable: $\theta_0=\mathbb{E}[Y(1)-Y(0)]$, where $Y(1)$ and $Y(0)$ are potential outcomes. We observe $(Y,D,Z)$, where $Y=Y(1)\cdot D +Y(0)\cdot(1-D)$ and $Z$ is a vector of covariates. We work with the usual unconfoundedness condition: $(Y(1),Y(0))$ and $D$ are independent conditional on $Z$. Let $X=(D,Z)$ and $\psi(X)=\psi(D,Z)$. We define the basis functions as follows: $\psi_{2k-1}(d,z)=\tilde{b}_k(z) $ and $\psi_{2k}(d,z)=d\cdot \tilde{b}_k(z) $ for $k\geq 1$ where $\{\tilde{b}_j(z)\}_{j=1}^{\infty}$ is a dictionary of functions of $z$. Following the usual argument (e.g., rosenbaum1983central), we can write the average treatment effect as \[ \theta_{0}=\mathbb{E}[m(W,\rho_{0})], \] where $m(W,f)=f(1,Z)-f(0,Z)$. Let $\pi_{0}(Z)=P(D=1|Z)$ be the propensity score. Then by routine calculation the Riesz representer is $$ \alpha_0(X)=\frac{D}{\pi_{0}(Z)}-\frac{1-D}{1-\pi_{0}(Z)} $$ assuming that $\alpha_0(X) \in \mathcal{B}$. This example was also considered in chernozhukov2022locally and hirshberg2021augmented. We include it here because of its fundamental importance and because the estimator we give is root-n consistent without the doubly robust rate condition.

remark\textcolor{black}{For Example (ref), we notice that $m(W,\psi_{2k-1})=\psi_{2k-1}(1,Z)-\psi_{2k-1}(0,Z)=0$ and $m(W,\psi_{2k})=\psi_{2k}(1,Z)-\psi_{2k}(0,Z)=\tilde{b}_{k}(Z)$ for any $k\geq 1$. Most of the conditions in Assumption (ref) are easy to verify. We focus on the boundedness of $\mathbb{E}|\alpha_0(X)|^{2+\kappa}$ and eigenvalues of $\mathbb{E}[\alpha_0(X)^{2} \psi(X)\psi(X)^{\top}]$. The condition of $\mathbb{E} |\alpha_{0}(X_{i})|^{2+\kappa}$ is needed because the parametric rate is impossible with $\mathbb{E} |\alpha_{0}(X_{i})|^{2}=\infty$ even if the propensity score is known, see e.g., hahn1998role. Moreover, the largest eigenvalues of $\mathbb{E}[\alpha_0(X)^{2} \psi(X)\psi(X)^{\top}]$ and $\mathbb{E} [m(W,\psi)m(W,\psi)^{\top}]$ are bounded whenever the largest eigenvalues of $\mathbb{E}(\tilde{b}(Z)\tilde{b}(Z)^{\top}/\pi_0(Z))$ and $\mathbb{E}(\tilde{b}(Z)\tilde{b}(Z)^{\top}/(1-\pi_0(Z)))$ are bounded, where $\tilde{b}(Z)=(\tilde{b}_1(Z),...,\tilde{b}_p(Z))^\top \in \mathbb{R}^p$ is the basis function for $Z$ in the above example. This is significantly weaker than the commonly imposed strict overlap condition, which requires that $P(c \leq \pi_0(Z) \leq 1 - c) = 1$ for some constant $c > 0$. As noted by d2021overlap, the strict overlap assumption can be unduly restrictive, particularly in high‑dimensional settings. Our results show that the parametric $n^{-1/2}$ rate is achievable without strict overlap whenever $\max\{\xi_{1},\xi_{2}\}>1/2$. When $\max\{\xi_{1},\xi_{2}\}\le 1/2 $, the parametric rate remains out of reach even if strict overlap holds. }

example[Partially linear instrumental variable regression] \textcolor{black}{Consider the model $Y=\tau D+g(X)+U$ with $\mathbb{E}(U\mid X,Z)=0$, where $D\in\mathbb{R}$ is an endogenous variable and $Z\in\mathbb{R}$ is an instrumental variable. The goal is to learn the coefficient $\tau$ of the endogenous variable from observations of $W=(Y,D,Z,X)$. Notice that $\mathbb{E}(Y\mid X,Z)-\mathbb{E}(Y\mid X)=\tau\cdot[\mathbb{E}(D\mid X,Z)-\mathbb{E}(D\mid X)]$. Therefore, $$ \tau =\frac{\mathbb{E}\left\{ Z\cdot[\mathbb{E}(Y\mid X,Z)-\mathbb{E}(Y\mid X)]\right\} }{\mathbb{E}\left\{ Z\cdot[\mathbb{E}(D\mid X,Z)-\mathbb{E}(D\mid X)]\right\} } =\frac{\mathbb{E}(YZ)-\mathbb{E}[Z\cdot\mathbb{E}(Y\mid X)]}{\mathbb{E}(DZ)-\mathbb{E}[Z\cdot\mathbb{E}(D\mid X)]}. $$} \textcolor{black}{Notice that the term $\mathbb{E}[Z\cdot\mathbb{E}(Y\mid X)]$ is a linear functional of the conditional mean function $\mathbb{E}(Y\mid X=\cdot)$, i.e., $\mathbb{E} [m(W,f)]$ with $m(W,f)=Z\cdot f(X)$. Moreover, the term $\mathbb{E}[Z\cdot\mathbb{E}(D\mid X)]$ is the same linear functional of the conditional mean function $\mathbb{E}(D\mid X=\cdot)$. Since we can easily learn $\mathbb{E}(YZ)$ and $\mathbb{E}(DZ)$ at the parametric rate, the problem of learning $\tau$ becomes solving the problem of learning a linear functional of two conditional mean functions. }

example[Continuous Treatment Effect in Panel Data] \textcolor{black}{Suppose that $Y_t$ is an outcome variable and $X_t=(D_t,Z_t)$ is a vector of regressors for two time periods $(t=1,2)$, where $D_t$ is a continuous treatment variable for period $t$ that varies with $t$. Also suppose that \[ Y_t=g_t(X_t)+A+\epsilon_t, \mathbb{E}[\epsilon_t|X_1,X_2]=0, \] where $A$ is a time invariant unobserved confounder, i.e. a fixed effect, such that $\mathbb{E}[A|X_1,X_2]$ is unrestricted. Define $Y=Y_2-Y_1$ and $X=(X_1,X_2)$ so that \[ \rho_0(X)=\mathbb{E}[Y|X]=g_2(X_2)-g_1(X_1). \] A parameter of interest is \[ \theta_{0}=\mathbb{E}[m(W,\rho_{0})]=\mathbb{E}[\partial \rho_{0}(D_{1},Z_{1},D_{2},Z_{2})/\partial D_2], \] where $m(\cdot,\cdot)$ is defined by $m(w,f)=\partial f(x)/\partial d_{2}$. This parameter is a continuous average treatment effect for the second period as in for example klosin2023estimating. Consider a dictionary $\psi_{2k-1}(x)=\tilde{b}_k(x_2) $ and $\psi_{2k}(x)=-\tilde{b}_k(x_1) $ for $k\geq 1$ where $\{\tilde{b}_j(x)\}_{j=1}^{\infty}$ is a dictionary of functions of $x=(d,z)$ that are each differentiable in the first element of $x$. This dictionary imposes the same additive structure that $\rho_0(X)$ has while allowing the function $g_t(\cdot)$ to vary with $t$. This parameter was not considered as a linear functional of an approximately sparse specification in chernozhukov2022locally or hirshberg2021augmented. The estimator given here will be root-n consistent without the doubly robust rate condition. }

commentHere we can take $X=(Z_{1},Z_{2},...,DZ_{1},DZ_{2},...)$ and $\rho_{0}(x)=\rho_{0}(d,z)$, where $D\in\{0,1\}$ is a treatment indicator and $Z$ are covariates. The object of interest is \[ \theta_{0}=\mathbb{E}[\rho_{0}(1,Z)-\rho_{0}(0,Z)]=\mathbb{E}[m(W,\rho_{0})],\text{ } m(W,\rho)=\rho(1,Z)-\rho(0,Z). \] Let $\pi_{0}(Z)=\Pr(D=1|Z)$ be the propensity score. Note that $\mathbb{E}[\rho(1,Z)]=\mathbb{E}[\pi_{0}(Z)^{-1}D\rho(1,Z)]=\mathbb{E}[\pi_{0}(Z)^{-1}D\rho(X)]$ and similarly $\mathbb{E}[\rho(0,Z)]=\mathbb{E}[\{1-\pi_{0}(Z)\}^{-1}(1-D)\rho(X)].$ Then for any $\rho\in\mathcal{B}$ \[ \mathbb{E}[m(W,\rho)]=\mathbb{E}[\rho(1,Z)-\rho(0,Z)]=\mathbb{E}[\alpha_{0}(X)\rho(X)],\text{ }\alpha _{0}(X)=\frac{D}{\pi_{0}(Z)}-\frac{1-D}{1-\pi_{0}(Z)}. \] Similarly to Example (ref), Equation ((ref)) is satisfied for \[ \alpha_0(X)=\mathrm{proj}\left(\frac{D}{\pi_{0}(Z)}-\frac{1-D}{1-\pi_{0}(Z)} |\mathcal{B} \right). \]

commentThis result shows that the automatic debiased machine learning estimator given in Equation ((ref)) is asymptotically Gaussian and the asymptotic variance estimator is consistent under either $\xi_{1}>1/2$ or $\xi_{1}\xi _{2}>1/4$ and the other conditions we have imposed. Recall from Section (ref) that the parametric rate is impossible whenever $\max \{\xi_1,\xi_2\}\leq 1/2$ for a regression slope. Hence, the only hope for the parametric rate is either $\xi_1>1/2$ or $\xi_2>1/2$; in other words, $\max \{\xi_1,\xi_2\}> 1/2$. By Theorem (ref), the proposed estimator attains the parametric rate when $\xi_1>1/2$. In Theorem (ref), we show that the parametric rate can be achieved for $\xi_2>1/2$ using another estimator. Therefore, the requirement of $\max \{\xi_1,\xi_2\}> 1/2$ is indeed sufficient and necessary for the parametric rate for a regression slope. Another important property of the above estimators is that they are doubly robust and have specification robust standard errors under rate double robustness $\xi_{1}\xi_{2}>1/4$. Theorem (ref) does assume that $\rho_{0}(X)\in\mathcal{B}$, i.e. that the regression function is a linear combination of the dictionary $(\psi_{1} (X),\psi_{2}(X),...)$. This condition is not restrictive when $(\psi_{1} (X),\psi_{2}(X),...)$ can approximate any function of $X$ in mean square but otherwise is restrictive. We intend to explore in future research whether it is possible to attain root-n consistency when $\xi_{1}>1/2$ but $\rho_{0}(X)$ is not the conditional expectation. The estimator $\hat\theta$ is doubly robust, meaning that the estimator of $\theta_0$ is consistent if either $\hat\rho$ or $\hat\alpha$ converge to a correct limit; see Equations (2.5) and (2.6) of chernozhukov_new\mathbb{EY_singh_local}. For conciseness we omit here the demonstration of double robustness. Theorem (ref) applies to many interesting objects of interest including Examples (ref)-(ref). First, consider the estimator of a regression slope in Example (ref). \begin{corollary}[Example (ref)] If Assumptions (ref) and (ref)--(ref) are satisfied, the residual $U$ from the linear projection of $D$ on the mean square limits of finite linear combinations of $(Z_{1},Z_{2},...)$ is nonzero and bounded, and either a) Assumption (ref) i) and ii) are satisfied and $\sqrt{n}\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2} \rightarrow 0;$ or Assumption (ref) i), ii), and v) are satisfied, then $\sqrt {n}(\hat{\theta}-\theta_{0})\overset{d}{\rightarrow}N(0,V).$ If in addition, $\bar{\tau}_{n}\rightarrow\infty$ and $\bar{\tau}_{n}[(r/\varepsilon_{n})r^{2\xi_{1}/(2\xi_{1}+1)}+\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2} ]\rightarrow0$, then $\hat{V}\overset{p}{\rightarrow}V.$ \end{corollary} \textcolor{PantoneGreen}{Yinchu: We now have an estimator that achieves the good rate under the minimal conditions. Should we still write this?} This result differs from previous debiased Lasso results of javanmard2014confidence,javanmard2018debiasing, van2014asymptotically, zhang2014confidence, and cai2017confidence in several ways. The estimator of the bias correction is obtained from a Lasso like objective function, approximate sparsity is imposed here rather than strict sparsity, the regressors are not Gaussian, and standard errors are robust to heteroskedasticity. Also, the estimators here use cross-fitting that enables them to be root-n consistent under the double robust rate condition $\xi_{1}\xi_{2}>1/4$ as well as under the minimal regression approximate sparsity condition $\xi_{1}>1/2.$ Also, under $\xi _{1}>1/2$ no additional sparsity conditions are imposed, unlike javanmard2014confidence,javanmard2018debiasing, van2014asymptotically, zhang2014confidence, where additional sparsity conditions are imposed on the projection of the variable of interest on the covariates or/and the inverse second moment matrix of the regression dictionary. A crucial difference is that our results do not depend on a quantity that is closely related to the $\ell_1$-norm of the rows of the precision matrix, i.e., the quantity $\rho$ in javanmard2018debiasing. Moreover, in contrast to Remark 4.3 therein, the lower and upper bounds here match, thereby establishing the minimax rate. Here is an asymptotic efficiency result for the weighted average derivative of Example (ref): \begin{corollary}[Example (ref)] If Assumptions (ref) and (ref)--(ref) are satisfied, \[\mathbb{E}[\{1+S(D)^{4}\}\omega(D)^{2}f(D|Z)^{-2}]<\infty, \int S(u)^{2}\omega(u)du<\infty,\textit{ } S(D)^{2}\omega(D)f(D|Z)^{-1}\leq C,\] and either a) Assumption (ref) i) and ii) are satisfied, $\sqrt {n}\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2} \rightarrow0$, or b) Assumption (ref) i), ii), iii), and v) are satisfied and $\alpha_0(X)$ is bounded then $\sqrt{n}(\hat{\theta}-\theta _{0})\overset{d}{\rightarrow}N(0,V).$ If in addition, $\bar{\tau }_{n}\rightarrow\infty$ and $\bar{\tau}_{n}[(r/\varepsilon _{n})r^{2\xi_{1}/(2\xi_{1}+1)}+\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2}]\rightarrow0$, then $\hat{V}\overset{p}{\rightarrow}V.$ \end{corollary} This result allows weaker conditions on approximate sparsity than in chernozhukov_new\mathbb{EY_singh_local} by allowing for $\xi_{1}>1/2$ as in Assumption (ref) rather than requiring rate double robustness. Here is an asymptotic efficiency result for the average treatment effect of Example (ref). \begin{corollary}[Example (ref)] If Assumptions (ref) and (ref)--(ref) are satisfied, $\mathbb{E}[\pi_{0}(Z)^{-1}\{1-\pi_{0}(Z)\}^{-1}]<\infty,$ and either a) Assumption (ref) i) and ii) are satisfied, $\sqrt{n}\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2} \rightarrow0$, or $\rho_{0}(X)$ is bounded; or b) Assumption (ref) i), ii), iii), and v) are satisfied, $\pi_{0}(Z)$ is bounded away from zero and one, and $\bar{\alpha }(X)$ is bounded, then $\sqrt{n}(\hat{\theta}-\theta_{0} )\overset{d}{\rightarrow}N(0,V).$ If in addition $\bar{\tau} _{n}\rightarrow\infty$ and $\bar{\tau}_{n}[(r/\varepsilon _{n})r^{2\xi_{1}/(2\xi_{1}+1)}+\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2} ]\rightarrow0$, then $\hat{V}\overset{p}{\rightarrow} V$. \end{corollary} This result differs from other results for the ATE in the estimator being asymptotically normal under either the rate double robustness condition $\xi_{1}\xi_{2}>1/4$ or the minimal regression approximate sparsity condition $\xi_{1}>1/2.$ We are not aware of any other ATE estimator that is root-n consistent under either of these conditions. athey2018approximate root-n consistency result is for sparsity corresponding to $\xi_{1}>1/2.$ Corollary (ref) for $\xi_{1}>1/2$ was developed independently of a related result by hirshberg2021augmented. Also, the condition $\mathbb{E}[\pi_{0}(Z)^{-1}\{1-\pi_{0}(Z)\}^{-1}]<\infty$ is an overlap condition that is required for root-n consistency. Imposing this condition rather than a stronger overlap condition may be of interest in applications where keeping overlap conditions as weak as possible may be warranted. We also give root-n consistency and asymptotic normality for the estimator $\tilde{\theta}$ of a regression coefficient from Equation ((ref)) under the minimal condition $\xi_{2}>1/2$ for the Riesz representer that allows a dense regression. Let $\mathcal{Z}$ denote the set of mean square limits of finite linear combinations of $\tilde{Z}=(Z_{1},Z_{2},...),$ $\eta_{0} (\tilde{Z})=\mathrm{proj}(D|\mathcal{Z)},$ and $\eta(\tilde{Z})$ be the least squares projection of $D$ on $\psi(Z)$ with coefficients $\pi.$ \begin{theorem} If $\xi_{2}>1/2,$ Assumption (ref) is satisfied with $\xi_{2}$ and $\pi$ replacing $\xi_{1}$ and $\gamma$ respectively, Assumptions (ref)--(ref) are satisfied, $r=o(\varepsilon_{n}\delta_{n}^{-1})$, $\mathbb{E}[D|\tilde{Z}]\in \mathcal{Z},$ there is $C>0$ with $\mathbb{E}[D^{2}|\tilde{Z}]+$, $\mathbb{E}[Y^{2}|\tilde{Z}]+$ $\left\vert \mathrm{proj}(Y|\tilde{Z})\right\vert \leq C$ and $\sqrt{n}\left\Vert \tilde{\eta}-\eta_{0}\right\Vert _{2}\rightarrow0$, then $\sqrt{n}(\tilde{\theta}-\theta _{0})\overset{d}{\rightarrow}N(0,V)$. If in addition $\bar{\tau }_{n}\rightarrow\infty$ and $\bar{\tau}_{n}[(r/\varepsilon _{n})r^{2\xi_{2}/(2\xi_{2}+1)}+\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2} ]\rightarrow0$, then $\hat{V}\overset{p}{\rightarrow}V$. \end{theorem} This result differs from previous results on estimation with a dense regression by zhu2018linear, javanmard2018debiasing and bradic2022testability in several ways. The estimator has a relatively simple form of a double/debiased Lasso machine learner as in chernozhukov2018double, approximate sparsity is imposed here rather than strict sparsity, the regressors are not Gaussian, and the inference here is robust to heteroskedasticity. Also, the estimator here uses cross-fitting that enables it to be root-n consistent under the double robust rate condition $\xi_{1} \xi_{2}>1/4$ as well as under the minimal approximate sparsity condition $\xi_{2}>1/2$ for the projection $\bar{D}$ of $D$ on the covariates. This result combined with that of Corollary (ref) shows that it is possible to construct root-n consistent estimators of a coefficient of a high dimensional regression under either of the minimal approximate sparsity conditions $\xi_{1}>1/2$ or $\xi_{2}>1/2$. This is accomplished with a different estimator for the two cases. One estimator that attained root-n consistency under $\max \{\xi_{1},\xi_{2}\}>1/2$ would be good. We leave the search for such an estimator to future work. It would also be of interest to have estimators for linear functionals other than a regression coefficient that attain root-n consistency when Riesz representer is approximately sparse with $\xi_{2}>1/2.$ Such is given for the ATE by wangandshah for a sparse logit propensity score. We leave to future work the search for such for other functionals.

comment\section{A Regression Slope and Other Linear Functionals of a Regression} To describe a regression slope let $W$ denote a data observation, $Y$ an outcome of interest, $D$ a variable of interest, and $(Z_{1},Z_{2},...)$ an infinite sequence of potential covariates. A regression slope will be the coefficient $\theta_{0}$ of $D$ in a linear regression of $Y$ on $D$ and $(Z_{1},Z_{2},...),$ that is \begin{equation} \mathbb{E}[Y|D,Z_{1},Z_{2},...]=\theta_{0}D+\sum_{j=1}^{\infty}\gamma_{j0}Z_{j}, \mathbb{E}[D^{2}]<\infty, \mathbb{E}[Z_{j}^{2}]<\infty, (j=1,2,...). \end{equation} This model is a familiar and important multiple regression with $\theta_{0}$ being the coefficient of the regressor $D$ and $Z_{j},$ $(j=1,...,\bar{p}),$ being covariates, where we allow for an infinite number of covariates$.$ When all but a finite number of $\gamma_{j0}$ are zero this is a sparse specification. In general we regard the sum as a mean square limit of sparse specifications but keep the sum notation for expositional purposes. We do not require sparsity but do require that $\theta_{0} D+\sum_{j=1}^{\infty}\gamma_{j0}Z_{j}$ can be approximated in mean square by a linear combination of $s$ covariates at rate $s^{-\xi_{1}}$. This is the approximate sparsity condition highlighted in the Introduction and further discussed in the rest of this paper. As noted in Section (ref), the regression coefficient $\theta_{0}$ is of wide interest. To motivate the estimators and other results it is helpful to interpret $\theta_{0}$ as a functional of a regression. Let $X=(X_{1} ,X_{2},...)$ denote the regressor sequence $X_{1}=D$ and $X_{j}=Z_{j-1},$ $(j\geq2).$ Then Equation ((ref)) can be written as \begin{equation} \mathbb{E}[Y|X]=\rho_{0}(X), \rho_{0}(X)=\sum_{j=1}^{\infty}\gamma_{j0} X_{j}, \mathbb{E}[X_{j}^{2}]<\infty, (j=1,2,...), \end{equation} where $\gamma_{10}=\theta_{0}$ and $\gamma_{j0}=\gamma_{j-1,0}$, $(j\geq2).$ Equation ((ref)) (and Equation ((ref))) impose the condition that \[ \mathbb{E}[Y|X]\in\mathcal{B}, \] In general this condition restricts the conditional mean to be linear in elements of $X=(X_{1},X_{2},...).$ This condition is not restrictive when linear combinations of $X$ can approximate in mean square any function of $X$, but we do not limit ourselves to this case. We will maintain $\mathbb{E}[Y|X]\in \mathcal{B}$ throughout much of this paper. There are many other interesting examples of objects of interest that can be formulated as a linear, mean square continuous functional of an infinite dimensional linear regression $\rho_{0}(X)$ similar to Equation ((ref) ). A regression slope as just described constitutes our first example, that we will refer to as Example (ref). Two other examples we consider are an average derivative and the average treatment effect. In each example there will be a functional $m(W,\rho)$ that is linear in $\rho$ such that the object of interest $\theta_{0}$ is given by \[ \theta_{0}=\mathbb{E}[m(W,\rho_{0})]. \] Thus the parameter of interest is an expectation of some known formula $m(W,\rho)$ of a data observation $W$ and the regression function $\rho.$ We focus on objects where there is $\alpha_0(X)$ with $\mathbb{E}[\alpha_0 (X)^{2}]<\infty$ and $\alpha_0(X)\in\mathcal{B}$ such that \begin{equation} \mathbb{E}[m(W,\rho)]=\mathbb{E}[\alpha_0(X)\rho(X)]\text{\ for all }\rho(X)\in\mathcal{B}. \end{equation} By the Riesz representation theorem, existence of such a $\alpha_0(X)$ is equivalent to $\mathbb{E}[m(W,\rho)]$ being a mean-square continuous functional of $\rho,$ i.e. there is $C>0$ such that $\left\vert \mathbb{E}[m(W,\rho)]\right\vert \leq C\left\Vert \rho\right\Vert _{2}$ for all $\rho\in\mathcal{B}$. We will refer to this $\alpha_0(X)$ as the Riesz representer. Existence of the Riesz representer is equivalent to the semiparametric variance bound for $\theta_{0}$ being finite, as mentioned in newey1994asymptotic and shown in hirshberg2021augmented and auto_debias_1809_05224. {\color{red} Jelena: Provide some examples of riesz representer - ideally some nonobvious one. Or if they are listed below maybe some intuitive words for why it would exist... And maybe interpret the last sentence of what does it mean for the semiparametric variance to be finite .... } We turn now to the average derivative and average treatment effect examples. \begin{example}[Average Derivative] Here $X_{j}=X_{j}(D,Z)$ $(j\geq1)$ for a continuously distributed variable $D$ and covariates $Z=(Z_{1} ,Z_{2},...),$ $\rho_{0}(x)=\rho_{0}(d,z),$ $\omega(d)$ is a pdf, and \[ \theta_{0}=\mathbb{E}\left[ \int\omega(u)\frac{\partial\rho_{0}(u,Z)}{\partial d}du\right] =\mathbb{E}\left[ m(W,\rho_{0})\right] ,\text{ }m(W,\rho)=S(U)\rho (U,Z), \] where $S(u)=-\omega(u)^{-1}\partial\omega(u)/\partial u$ is the negative of the location score {\color{red} Jelena: what is exactly a location score here, and what is the intuition or meaning of U? Instead of this average derivative idea - perhaps look for some more intuitive parameters when D is a continuous random variable in causal literature...} for the pdf $\omega(u),$ $U$ is a random variable that is independent of $(Y,X)$ and has pdf $\omega(u),$ and the second equality follows by integration by parts. This $\theta_{0}$ can be interpreted as an average treatment effect on $Y$ of a continuous treatment $D;$ see chernozhukov2018learning. Multiplying and dividing by the conditional pdf $f(d|z)$ of $D=d$ given $Z=z$ we find that for any $\rho(X)$ with $\mathbb{E}[\rho(X)^{2}]<\infty$, \begin{align*} \mathbb{E}[m(W,\rho)] & =\mathbb{E}\left [\int S(u)\rho(u,Z)\omega(u)du \right]=\mathbb{E}[f(D|Z)^{-1} S(D)\omega(D)\rho(X)]=\mathbb{E}[\alpha_{0}(X)\rho(X)],\\ \alpha_{0}(X) & =f(D|Z)^{-1}S(D)\omega(D). \end{align*} It follows that Equation ((ref)) is satisfied for \[ \alpha_0(X)=\mathrm{proj}(f(D|Z)^{-1}S(D)\omega(D)|\mathcal{B)}. \] \end{example} \begin{example}[Average Treatment Effect] Here we can take $X=(Z_{1},Z_{2},...,DZ_{1},DZ_{2},...)$ and $\rho_{0}(x)=\rho_{0}(d,z)$, where $D\in\{0,1\}$ is a treatment indicator and $Z$ are covariates. The object of interest is \[ \theta_{0}=\mathbb{E}[\rho_{0}(1,Z)-\rho_{0}(0,Z)]=\mathbb{E}[m(W,\rho_{0})],\text{ } m(W,\rho)=\rho(1,Z)-\rho(0,Z). \] If potential outcomes are mean independent of treatment $D$ conditional on covariates $Z$, then $\theta_{0}$ is the average treatment effect rosenbaum1983central. Let $\pi_{0}(Z)=\Pr(D=1|Z)$ be the propensity score. Note that $\mathbb{E}[\rho(1,Z)]=\mathbb{E}[\pi_{0}(Z)^{-1}D\rho(1,Z)]=\mathbb{E}[\pi_{0}(Z)^{-1}D\rho(X)]$ and similarly $\mathbb{E}[\rho(0,Z)]=\mathbb{E}[\{1-\pi_{0}(Z)\}^{-1}(1-D)\rho(X)].$ Then for any $\rho\in\mathcal{B}$ \[ \mathbb{E}[m(W,\rho)]=\mathbb{E}[\rho(1,Z)-\rho(0,Z)]=\mathbb{E}[\alpha_{0}(X)\rho(X)],\text{ }\alpha _{0}(X)=\frac{D}{\pi_{0}(Z)}-\frac{1-D}{1-\pi_{0}(Z)}. \] Similarly to Example (ref), Equation ((ref)) is satisfied for \[ \alpha_0(X)=\mathrm{proj}\left(\frac{D}{\pi_{0}(Z)}-\frac{1-D}{1-\pi_{0}(Z)} |\mathcal{B} \right). \] \end{example} {\color{red} Jelena: is proj function defined somewhere ? Example 1 and Example 3 are natural examples but we should change Example 2. You can look for Mediation Analysis, Dynamic Treatment Effects, as mentioned before a more intuitive parameter when D is continuous, etc...} An important feature of the set up here is that the Riesz representer $\alpha_0(X)$ is an element of $\mathcal{B}$ and so can be approximated by a sparse linear combination of the regressors $X=(X_{1},X_{2},...).$ Consequently a sparse approximation for $\alpha_0(X)$ exists by the definition of $\mathcal{B}$ and does not require approximation of nonlinear functions by a linear regression. For instance in Example (ref) the Riesz representer $\alpha_0(X)$ is the linear projection of the inverse propensity score term $\alpha_{0}(X)$ on the set of linear combinations of $X=(Z_{1},Z_{2},...,DZ_{1},DZ_{2}...).$ Thus for the average treatment effect and in general a sparse approximation of $\alpha_0(X)$ exists by construction and does not require that the inverse propensity score be well approximated by linear regressors. \section{Estimation and Inference} The first estimator we give is a debiased Lasso estimator of a regression slope. For this estimator $\psi(X)=(D,Z_{1},...,Z_{p-1})^{\top}$ with first element being the regressor of interest. The estimator $\hat{\theta}$ of the coefficient $\theta_{0}$ of that regressor is \begin{align} \hat{\theta} & =\frac{n_{1}}{n}\hat{\theta}_{1}+\frac{n_{2}}{n}\hat{\theta }_{2},\text\\ \hat{\theta}_{1} & :=\hat{\gamma}_{21}+\frac{1}{n_{1}}\sum_{i\in I_{1}} \hat{\alpha}_{1}(X_{i})[Y_{i}-\hat{\rho}_{2}(X_{i})],\text\hat{\theta} _{2}:=\hat{\gamma}_{11}+\frac{1}{n_{2}}\sum_{i\in I_{2}}\hat{\alpha}_{2} (X_{i})[Y_{i}-\hat{\rho}_{1}(X_{i})].\nonumber \end{align} Here each $\hat{\alpha}_{\ell}(x)$ is a bias correction term given by \begin{equation} \hat{\alpha}_{\ell}(x)=\psi(x)^{\top}\hat{\pi}_{\ell},\text\hat{\pi}_{\ell }:=\arg\min_{\pi}\left\{ \pi^{\top}\hat{\Sigma}_{\ell}\pi-2\pi _{1}+2r\left\Vert \pi\right\Vert _{1}\right\} ,\text(\ell=1,2)\text{.} \end{equation} The estimator $\hat{\theta}$ is a bias-corrected Lasso similar to those of javanmard2014confidence,javanmard2018debiasing, van2014asymptotically, zhang2014confidence, and cai2017confidence. It differs in the form of the bias correction terms $\hat{\alpha}_{\ell}(x)$ and in the use of cross-fitting. In the supplementary materials, we provide an estimator without any cross-fitting. The cross-fitting in $\hat{\theta}$ sums over the same observations used to construct $\hat{\alpha}_{\ell}(x)$ but different observations than used to construct $\hat{\rho}_{\ell}(x)$. Using the same observations to construct $\hat{\alpha}_{\ell}(x)$ as in the sums enables the first order conditions for $\hat{\alpha}_{\ell}(x)$ to help bound a key quadratic remainder under weak conditions on Riesz representer $\alpha_0(x).$ As a result this estimator will attain root-n consistency under $\xi_{1}>1/2$ and some regularity conditions. Using different observations for $\hat{\rho}_{\ell}(x)$ than in the sums leads to root-n consistency under the rate double robustness condition $\xi_{1}\xi_{2}>1/4.$ In this way the special cross-fitting in Equation ((ref)) gives an estimator that is root-n consistent if either $\xi_{1}>1/2$ or $\xi_{1}\xi_{2}>1/4.$ To construct an estimator of the asymptotic variance of $\sqrt{n}(\hat{\theta }-\theta_{0})$ that allows for very weak conditions on $\alpha_0(X)$ it is theoretically convenient to trim the estimator $\hat{\alpha}_{\ell}(x)$. Let $\bar{\tau}_{n}>0$ be a constant that will be assumed to go to infinity with $n$ and for a scalar $a$ let \[ \tau_{n}(a):=\left\{ \begin{array} [c]{c} \bar{\tau}_{n},\text{ }a\geq\bar{\tau}_{n},\\ a,\text{ }-\bar{\tau}_{n}<a<\tau_{n},\\ -\bar{\tau}_{n},\text{ }a\leq-\bar{\tau}_{n}. \end{array} \right. \] Also let $\tilde{\alpha}_{\ell}(x)=\tau_{n}(\hat{\alpha}_{\ell}(x)),$ $(\ell=1,2).$ The asymptotic variance estimator is \begin{equation} \hat{V}=\frac{1}{n}\sum_{i=1}^{n}\hat{\psi}_{i}^{2},\text\hat{\psi} _{i}=\left\{ \begin{array} [c]{c} \hat{\gamma}_{21}+\tilde{\alpha}_{1}(X_{i})[Y_{i}-\hat{\rho}_{2}(X_{i} )]-\hat{\theta},\texti\in I_{1},\\ \hat{\gamma}_{11}+\tilde{\alpha}_{2}(X_{i})[Y_{i}-\hat{\rho}_{1}(X_{i} )]-\hat{\theta},\texti\in I_{2}. \end{array} \right. . \end{equation} This will be a heteroskedasticity consistent estimator of the asymptotic variance of $\sqrt{n}(\hat{\theta}-\theta_{0})$ under the conditions given in Section (ref). The estimator $\hat{\theta}$ of Equation ((ref)) is root-n consistent for those values of $(\xi_{1},\xi_{2})$ to the right of the box in Figure (ref), where $\xi_{2}>1/2.$ We now give another estimator that is root-n consistent for the rest of $(\xi_{1},\xi_{2})$ values outside the box where $\xi_{2}>1/2.$ Here the regression is allowed to be dense with no restriction placed on its rate of sparse approximation $\xi_{1} $. As before we partition the observations into $I_{1}$ and $I_{2}.$ Let $\psi(z)=(z_{1},...,z_{p-1})^{\top}$ denote the $(p-1)\times1$ vector of all dictionary functions except $d$ and \begin{align*} \hat{\zeta}_{\ell}(z) & :=\psi(z)^{\top}\tilde{\gamma}_{\ell}, \quad \text \tilde{\gamma}_{\ell}=\arg\min_{\gamma}\left\{ \gamma^{\top}\tilde{\Sigma }_{\ell}\gamma-2\tilde{\mu}_{\ell}^{\top}\gamma+2r\left\Vert \gamma \right\Vert _{1}\right\} , \quad \text\\ \hat{\eta}_{\ell}(z) & :=\psi(z)^{\top}\tilde{\pi}_{\ell},\text\tilde{\pi }_{\ell}=\arg\min_{\pi}\left\{ \pi^{\top}\tilde{\Sigma}_{\ell}\pi -2\tilde{M}_{\ell}^{\top}\pi+2r\left\Vert \pi\right\Vert _{1}\right\} ,\text\\ \tilde{\Sigma}_{\ell} & :=\frac{1}{n_{\ell}}\sum_{i\in I_{\ell}} \psi(Z_{i})\psi(Z_{i})^{\top},\text\tilde{\mu}_{\ell}:=\frac{1}{n-n_{\ell}} \sum_{i\notin I_{\ell}}\psi(Z_{i})Y_{i},\text\tilde{M}_{\ell}:=\frac {1}{n-n_{\ell}}\sum_{i\notin I_{\ell}}\psi(Z_{i})D_{i}. \end{align*} Here $\hat{\zeta}_{\ell}(z)$ and $\hat{\eta}_{\ell}(z)$ are Lasso like estimators where different data observations are used for the estimator $\tilde{\Sigma}_{\ell}$ of the second moment matrix of $\psi(z)$ than for the cross moment estimators $\tilde{\mu}_{\ell}$ and $\tilde{M}_{\ell}$. An estimator of the regression slope \begin{equation} \tilde{\theta}=\hat{H}^{-1}\sum_{\ell=1}^{2}\sum_{i\in I_{\ell}}\{D_{i} -\hat{\eta}_{\ell}(Z_{i})\}\{Y_{i}-\hat{\zeta}_{\ell}(Z_{i})\}/n,\text \hat{H}=\sum_{\ell=1}^{2}\sum_{i\in I_{\ell}}\{D_{i}-\hat{\eta}_{\ell} (Z_{i})\}^{2}/n. \end{equation} Here $\hat{\gamma}_{\ell}(z)$ and $\hat{\alpha}_{\ell}(z)$ are not based on different observations than in $I_{\ell}$. Instead $\hat{\gamma}_{\ell}$ and $\hat{\pi}_{\ell}$ use $\hat{\Sigma}_{\ell}$ that is constructed from observations in $I_{\ell}$. Using the same observations to construct $\hat{\Sigma}_{_{\ell}}$ as in the sums enables the first order conditions for $\tilde{\gamma}_{\ell}$ and $\tilde{\pi}_{\ell}$ to help bound a key quadratic remainder under weak conditions. Using a different samples for $\hat{\gamma}_{\ell}$ and $\hat{\pi}_{\ell}$ than for $\hat{\Sigma}_{\ell}$ also helps bound key remainders. We will show that $\tilde{\theta}$ is root-n consistent and asymptotically normal under the minimal approximate sparsity condition $\xi_{2}>1/2.$ An estimator of the asymptotic variance of $\sqrt{n}(\tilde{\theta}-\theta _{0})$ can be constructed as \begin{align*} \hat{V} & =\hat{H}^{-1}\hat{\Omega}\hat{H}^{-1},\text\hat{\Omega} =\sum_{\ell=1}^{2}\sum_{i\in I_{\ell}}\{D_{i}-\hat{\eta}_{\ell}(Z_{i} )\}^{2}\hat{u}_{i\ell}^{2}/n\\ \hat{u}_{i\ell} & =Y_{i}-\tau_{n}(\hat{\zeta}_{\ell}(Z_{i}))-\tilde{\theta }[D_{i}-\tau_{n}(\hat{\eta}_{\ell}(Z_{i}))]. \end{align*} This variance estimator will be heteroskedasticity consistent under conditions given in Section (ref). We also give estimators of an average derivative, average treatment effect, and other linear functionals that are root-n consistent under either the minimal approximate sparsity condition $\xi_{1}>1/2$ or the doubly robust rate condition $\xi_{1}\xi_{2}>1/4.$ To describe these estimators we adopt the linear functional notation of Section (ref) where the parameter of interest is $\theta_{0}=\mathbb{E}[m(W,\rho_{0})]$. A consistent asymptotic variance estimator can be constructed similar to the debiased Lasso variance estimator given above. For $\tilde{\alpha}_{\ell }(x)=\tau_{n}(\hat{\alpha}_{\ell}(x)),$ $(\ell=1,2),$ we let \begin{equation} \hat{V}=\frac{1}{n}\sum_{i=1}^{n}\hat{\psi}_{i}^{2},\text\hat{\psi} _{i}=\left\{ \begin{array} [c]{c} m(W_{i},\hat{\rho}_{2})+\tilde{\alpha}_{1}(X_{i})[Y_{i}-\hat{\rho}_{2} (X_{i})]-\hat{\theta},\texti\in I_{1},\\ m(W_{i},\hat{\rho}_{1})+\tilde{\alpha}_{2}(X_{i})[Y_{i}-\hat{\rho}_{1} (X_{i})]-\hat{\theta},\texti\in I_{2}. \end{array} \right. \end{equation} This estimator is heteroskedasticity robust for any $m(W,\rho).$ \textsc{Example (ref) continued:} (Regression Coefficient). Recall that $\psi(X)=(D,Z_{1} ,...,Z_{p-1})$ for the regression slope, i.e. the first component of $\psi(X)$ is the regressor of interest $D.$ Let $\hat{\theta}_{\ell}$ denote the Lasso regression coefficient of $d$ for the Lasso regressions in Equation ((ref)) with $\ell=1,2.$ Note that by Equation ((ref)) we have $m(W,\hat{\rho}_{\ell})=\hat{\theta}_{\ell}.$ Also note that $m(W,\psi_{1})=1$ and $m(W,\psi_{j})=0$ for all $j\geq2,$ so that $m(W,b)=e_{1}=(1,0,...,0)^{\top }$ is the first unit vector. Then the previously specified estimators of $\alpha_0(X)=U,$ $\theta_{0},$ and $V$ given in Equations ((ref)), ((ref)), and ((ref)), respectively, are the same as those of Equations ((ref)), ((ref)), and ((ref)). \textsc{Example (ref) continued:} (Weighted Average Derivative) Here $X_{j}=X_{j}(D,Z)$ $(j\geq1)$ where $D$ is continuously distributed and \[ m(W_{i},\hat{\rho}_{\ell})=S(U_{i})\hat{\rho}_{\ell}(U_{i},Z_{i}), \] where $U_{i}$ has pdf $\omega(u)$ and is independent of $(Y_{i},X_{i}).$ One can think of $U_{i}$ as a single simulation draw from the distribution with pdf $\omega(u).$ Also, we have \[ m(W_{i},\psi_{j})=S(U_{i})\psi_{j}(U_{i},Z_{i}),\text{ }(j=1,...,p). \] A high dimensional weighted average derivative estimator can then be obtained from Equations ((ref)) and ((ref)), with asymptotic variance estimator from Equation ((ref)). \textsc{Example (ref) continued: }(Average Treatment Effect) Here the dictionary can be chosen in a variety of ways. One way is to choose $\psi(x)$ to consist of products of a $p/2$ dimensional vector of covariates $z^{\top} =(z_{1},...,z_{p/2})^{\top}$ with the treatment and nontreatment dummies $d$ and $1-d$, that is \[ \psi(x)=(dz^{\top},(1-d)z^{\top})^{\top}. \] Here $m(W_{i},\hat{\rho}_{\ell})=\hat{\rho}_{\ell}(1,Z_{i})-\hat{\rho} (0,Z_{i}),$ $m(W_{i},b)=(Z_{i}^{\top},-Z_{i}^{\top})^{\top}.$ A high dimensional average treatment effect estimator can then be obtained from Equations ((ref)) and ((ref)), with asymptotic variance estimator from Equation ((ref)). As discussed in chernozhukov2018learning the resulting $\hat{\alpha}_{\ell}(x)$ can be interpreted as balancing weights for the covariates $Z_{i}.$ \section{Large Sample Properties of the Estimators}

appendix\section{Proof of Lemma (ref)} \begin{proof}[Proof of Lemma (ref)] We proceed by contradiction. Suppose that such $D,\xi$ exist. Consider $v\in\mathbb{R}^p$ with $v_1=v_2=\cdots=v_k=\sqrt{C/k} $ and $v_j=0$ for $j\geq k+1$. Clearly, $v\in\Theta(C,k)$. Consider $s=\left \lceil{k^{1/2}}\right \rceil$. Then $s\in \{1,...,k\}$ for large $k$. By $\mathbb{E}[\psi(X)\psi(X)^{\top}]=I_p$ and the definition of $ \mathcal{M}_{D,\xi}$, we have $(C/k)(k-s)\leq D^2 s^{-2\xi}$, which means that $1-s/k\leq (D^2/C) s^{-2\xi}$. This is impossible because the left-hand side tends to 1 and the right-hand side tends to zero as $k\rightarrow \infty$. \end{proof} \section{Proof of Lemma (ref)} \begin{proof} \textcolor{black}{We consider the periodic Sobolev space. Let $\Omega_{*}=[0,1]^{D}$ and \[ B_{H^{s}(\Omega_{*})}(Q_{*})=\left\{ f:\Omega_{*}\mapsto\mathbb{R}:\|f\|_{H^{s}(\Omega_{*})}\leq Q_{*}\right\} . \] } \textcolor{black}{Now we extend the Sobolev function $f\in H^{s}(\Omega)$ to a function in $H^{s}(\mathbb{R}^{D})$. Notice that the boundary of $\Omega$ is minimally smooth as defined on page 189 of stein1970singular; $\partial\Omega$ is minimally smooth since $\Omega$ is an open bounded set. By Theorem 5 on page 181 of stein1970singular, there exists a function $f_{*}\in H^{s}(\mathbb{R}^{D})$ such that $f_{*}(x)=f(x)$ for any $x\in\Omega$ and \begin{equation} \|f_{*}\|_{H^{s}(\mathbb{R}^{D})}\leq c_{1}\|f\|_{H^{s}(\Omega)}, \end{equation} where $c_{1}>0$ is a universal constant that depends only on $D,s,\Omega$. } \textcolor{black}{We now construct an extension of $f$ that is supported on $\Omega_{*}$. To do so, let $\eta(\cdot)$ be an infinitely smooth function such that $\eta$ is supported on ${\rm Int}(\Omega_{*})$ and $\eta(x)=1$ for $x\in\Omega$. This is possible by a smooth version of Urysohn's lemma, e.g., page 38 of lieb2001analysis. Then we define $f_{**}(x)=f_{*}(x)\cdot\eta(x)$. Clearly, $f_{**}(x)=f(x)$ for $x\in\Omega$ and $f_{**}$ is supported on ${\rm Int}(\Omega_{*})$. } \textcolor{black}{We now bound $\|f_{**}\|_{H^{s}(\Omega_{*})}$. By the product rule of differentiation, $D^{\alpha}(f_{**})=D^{\alpha}(f_{*}\eta)=\sum_{q}(D^{q}f_{*})\cdot(D^{\alpha-q}\eta)$, where the summation is taken over all $q\in\mathbb{N}_{0}^{D}$ such that $0\leq q_{l}\leq a_{l}$ for all $l\in\{1,...,D\}$. Since $\eta(\cdot)$ is a fixed smooth function with compact support, $\sup_{x\in\mathbb{R}^{D}}|D^{\alpha-q}\eta(x)|$ is bounded by a constant that depends only on $\alpha-q$. Therefore, there exist a constant $c_{2}>0$ depending only on $\alpha$ such that $\|D^{\alpha}(f_{**})\|_{L^{2}(\Omega_{*})}\leq c_{2}\sum_{q}\|D^{q}f_{*}\|_{L^{2}(\Omega_{*})}$. It follows that \[ \|f_{**}\|_{H^{s}(\Omega_{*})}=\sum_{|\alpha|\leq s}\|D^{\alpha}f_{**}\|_{L^{2}(\Omega_{*})}\leq c_{3}\sum_{|\alpha|\leq s}\|D^{\alpha}f_{*}\|_{L^{2}(\Omega_{*})}=c_{3}\|f_{*}\|_{H^{s}(\Omega_{*})}\leq c_{3}\|f_{*}\|_{H^{s}(\mathbb{R}^{D})}. \]} \textcolor{black}{Thus, by ((ref)) and $\|f\|_{H^{s}(\Omega)}\leq Q$, we have that \[ \|f_{**}\|_{H^{s}(\Omega_{*})}\leq c_{1}c_{3}Q. \]} \textcolor{black}{We now study the approximation property of $f_{**}$. } \textcolor{black}{For any function $g\in L^{2}(\Omega_{*})$ and any $k\in\mathbb{Z}^{D}$, define $s_{k}(g)=\int_{\Omega_{*}}g(x)\exp(2\pi\iota k'x)dx$, where $\iota=\sqrt{-1}$. Let $\psi_{k}(x)=\exp(2\pi\iota k'x)$. We know that \[ g=\sum_{k\in\mathbb{Z}^{D}}s_{k}(g)\cdot\psi_{k}\quad\text{and}\quad\|g\|_{L^{2}(\Omega_{*})}^{2}=\sum_{k\in\mathbb{Z}^{D}}|s_{k}(g)|^{2}. \]} \textcolor{black}{We apply integration by parts and obtain that for $k_{l}\neq0$ and $\alpha_{l}\geq1$, \begin{align*} & \int_{0}^{1}D^{\alpha}f_{**}(x)\exp(2\pi\iota k_{l}x_{l})dx_{l}\\ & =\int_{0}^{1}\exp(2\pi\iota k_{l}x_{l})dD^{\alpha-e_{l}}f_{**}(x)\\ & =D^{\alpha-e_{l}}f_{**}(x)\cdot\exp(2\pi\iota k_{l}x_{l})\Bigg|_{0}^{1}-(2\pi\iota k_{l})\int_{0}^{1}\exp(2\pi\iota k_{l}x_{l})\cdot D^{\alpha-e_{l}}f(x)dx_{l}\\ & \overset{(i)}{=}-(2\pi\iota k_{l})\int_{0}^{1}\exp(2\pi\iota k_{l}x_{l})\cdot D^{\alpha-e_{l}}f(x)dx_{l}, \end{align*} where (i) follows by the fact that $f_{**}$ is supported on ${\rm Int}(\Omega_{*})$. For $k_{l}=0$, $\int_{0}^{1}D^{\alpha}f_{**}(x)\exp(2\pi\iota k_{l}x_{l})dx_{l}=\int_{0}^{1}D^{\alpha}f_{**}(x)dx_{l}=0$ since $f_{**}$ is supported on ${\rm Int}(\Omega_{*})$. Thus, for $a_{l}\geq1$, we have \[ \int_{0}^{1}D^{\alpha}f_{**}(x)\exp(2\pi\iota k_{l}x_{l})dx_{l}=-(2\pi\iota k_{l})\int_{0}^{1}\exp(2\pi\iota k_{l}x_{l})\cdot D^{\alpha-e_{l}}f(x)dx_{l}. \]} \textcolor{black}{Therefore, for any $\alpha_{l}\geq1$, \begin{align*} s_{k}\left(D^{\alpha}f_{**}\right) & =\int_{\Omega_{*}}D^{\alpha}f_{**}(x)\exp(2\pi\iota k'x)dx\\ & =\int_{[0,1]^{D-1}}\left[\exp\left(2\pi\iota\sum_{j\neq l}k_{j}x_{j}\right)\left(\int_{0}^{1}D^{\alpha}f_{**}(x)\exp(2\pi\iota k_{l}x_{l})dx_{l}\right)\right]\prod_{j\neq l}dx_{j}\\ & =-(2\pi\iota k_{l})\int_{[0,1]^{D-1}}\left[\exp\left(2\pi\iota\sum_{j\neq l}k_{j}x_{j}\right)\left(\int_{0}^{1}\exp(2\pi\iota k_{l}x_{l})\cdot D^{\alpha-e_{l}}f(x)dx_{l}\right)\right]\prod_{j\neq l}dx_{j}\\ & =-(2\pi\iota k_{l})\int_{\Omega_{*}}D^{\alpha-e_{l}}f_{**}(x)\exp(2\pi\iota k'x)dx=-(2\pi\iota k_{l})s_{k}\left(D^{\alpha-e_{l}}f_{**}\right). \end{align*}} \textcolor{black}{By induction, for any $\alpha_{l}\geq1$, \[ s_{k}\left(D^{\alpha}f_{**}\right)=(-2\pi\iota k_{l})^{a_{l}}s_{k}\left(D^{\alpha-\alpha_{l}e_{l}}f_{**}\right) \] and thus \[ s_{k}\left(D^{\alpha}f_{**}\right)=\prod_{l=1}^{D}(-2\pi\iota k_{l})^{a_{l}}s_{k}\left(f_{**}\right). \]} \textcolor{black}{Therefore, \[ \|D^{\alpha}f_{**}\|_{L^{2}(\Omega_{*})}^{2}=\sum_{k\in\mathbb{Z}^{D}}\left|s_{k}\left(D^{\alpha}f_{**}\right)\right|^{2}=\sum_{k\in\mathbb{Z}^{D}}\prod_{l=1}^{D}|2\pi k_{l}|^{2a_{l}}\cdot\left|s_{k}\left(f_{**}\right)\right|^{2}. \]} \textcolor{black}{Notice that $\|D^{\alpha}f_{**}\|_{L^{2}(\Omega_{*})}^{2}\leq c_{1}^{2}c_{3}^{2}Q^{2}$ for any $\alpha$ with $|\alpha|\leq s$. For any $b\in\{1,...,D\}$, we choose $\alpha=se_{b}$. Thus, for any $b\in\{1,...,D\}$, \[ c_{1}^{2}c_{3}^{2}Q^{2}\geq\|D^{se_{b}}f_{**}\|_{L^{2}(\Omega_{*})}^{2}=\sum_{k\in\mathbb{Z}^{D}}|2\pi k_{b}|^{2s}\cdot\left|s_{k}\left(f_{**}\right)\right|^{2}=(2\pi)^{2s}\cdot\sum_{k\in\mathbb{Z}^{D}}|k_{b}|^{2s}\cdot\left|s_{k}\left(f_{**}\right)\right|^{2}. \]} \textcolor{black}{For any $b\in\{1,...,D\}$, let $k_{-b}\in\mathbb{Z}^{D-1}$ denote the elements of $k=(k_{1},...,k_{D})'$ other than $k_{b}$. For $k$ such that $k_{b}=t$, define $m_{b}(t)=\sum_{k_{-b}\in\mathbb{Z}^{D-1}}|s_{k}(f_{**})|^{2}$. Then the above means that for any $b\in\{1,...,D\}$, \[ c_{1}^{2}c_{3}^{2}Q^{2}\geq(2\pi)^{2s}\cdot\sum_{k_{b}\in\mathbb{Z}}|k_{b}|^{2s}\cdot\sum_{k_{-b}\in\mathbb{Z}^{D-1}}\left|s_{k}\left(f_{**}\right)\right|^{2}=(2\pi)^{2s}\cdot\sum_{k_{b}\in\mathbb{Z}}|k_{b}|^{2s}m_{b}(k_{b})=(2\pi)^{2s}\cdot\sum_{j\in\mathbb{Z}}|j|^{2s}m_{b}(j). \]} \textcolor{black}{Hence, for any $r\geq1$ and for any $b\in\{1,...,D\}$, \begin{multline} \sum_{j\in\mathbb{Z},|j|\geq r+1}m_{b}(j)\leq\sum_{j\in\mathbb{Z},|j|\geq t+1}\left|\frac{j}{r+1}\right|^{2s}m_{b}(j)=(r+1)^{-2s}\sum_{j\in\mathbb{Z},|j|\geq r+1}|j|^{2s}m_{b}(j)\\ \leq(r+1)^{-2s}\sum_{j\in\mathbb{Z}}|j|^{2s}m_{b}(j)\leq(r+1)^{-2s}\cdot(2\pi)^{-2s}c_{1}^{2}c_{3}^{2}Q^{2}. \end{multline}} \textcolor{black}{For $t\geq1$, let $\mathcal{N}_{t}=\{j\in\mathbb{Z}:|j|\leq\left\lfloor t^{1/D}\right\rfloor \}$. We observe that \begin{multline*} \left(\mathbb{Z}^{D}\right)\backslash\left(\mathcal{N}_{t}^{D}\right)=\left\{ k\in\mathbb{Z}^{D}:|k_{b}|\geq\left\lfloor t^{1/D}\right\rfloor +1\ for some b\in\{1,...,D\}\right\} \\ =\bigcup_{b=1}^{D}\left\{ k\in\mathbb{Z}^{D}:|k_{b}|\geq\left\lfloor t^{1/D}\right\rfloor +1\right\} . \end{multline*}} \textcolor{black}{It follows that \begin{align*} \sum_{k\in\left(\mathbb{Z}^{D}\right)\backslash\left(\mathcal{N}_{t}^{D}\right)}\left|s_{k}\left(f_{**}\right)\right|^{2} & =\sum_{k\in\bigcup_{b=1}^{D}\left\{ k\in\mathbb{Z}^{D}:|k_{b}|\geq\left\lfloor t^{1/D}\right\rfloor +1\right\} }\left|s_{k}\left(f_{**}\right)\right|^{2}\\ & \leq\sum_{b=1}^{D}\sum_{k\in\mathbb{Z}^{D}:|k_{b}|\geq\left\lfloor t^{1/D}\right\rfloor +1}\left|s_{k}\left(f_{**}\right)\right|^{2}\\ & =\sum_{b=1}^{D}\sum_{|k_{b}|\geq\left\lfloor t^{1/D}\right\rfloor +1}\sum_{k_{-b}\in\mathbb{Z}^{D-1}}\left|s_{k}\left(f_{**}\right)\right|^{2}\\ & =\sum_{b=1}^{D}\sum_{|k_{b}|\geq\left\lfloor t^{1/D}\right\rfloor +1}m_{b}(k_{b})\\ & \overset{(i)}{\leq}D\left(\left\lfloor t^{1/D}\right\rfloor +1\right)^{-2s}\cdot(2\pi)^{-2s}c_{1}^{2}c_{3}^{2}Q^{2}\\ & \overset{(ii)}{\leq}D\left(t^{1/D}\right)^{-2s}\cdot(2\pi)^{-2s}c_{1}^{2}c_{3}^{2}Q^{2}=(2\pi)^{-2s}c_{1}^{2}c_{3}^{2}Q^{2}D\cdot t^{-2s/D}, \end{align*} where (i) follows by ((ref)) and (ii) follows by $\left\lfloor t^{1/D}\right\rfloor +1\geq t^{1/D}$. Therefore, \begin{multline*} \left\Vert f_{**}-\sum_{k\in\mathcal{N}_{t}^{D}}s_{k}(f_{**})\psi_{k}\right\Vert _{L^{2}(\Omega_{*})}^{2}=\left\Vert \sum_{k\in\left(\mathbb{Z}^{D}\right)\backslash\left(\mathcal{N}_{t}^{D}\right)}s_{k}(f_{**})\psi_{k}\right\Vert _{L^{2}(\Omega_{*})}^{2}\\ =\sum_{k\in\left(\mathbb{Z}^{D}\right)\backslash\left(\mathcal{N}_{t}^{D}\right)}\left|s_{k}\left(f_{**}\right)\right|^{2}\leq(2\pi)^{-2s}c_{1}^{2}c_{3}^{2}Q^{2}D\cdot t^{-2s/D}. \end{multline*}} \textcolor{black}{Since $\Omega\subset\Omega_{*}$ and $f_{**}(x)=f(x)$ for $x\in\Omega$, the desired result follows. } \end{proof} \section{Proofs for Section (ref)} In this section, we adopt the notation of $\tau(\lambda)=\theta$ for $ \lambda=(\theta,\beta,\phi,\sigma_{u} ^{2},\sigma_{\varepsilon}^{2})$. \subsection{Proof of Theorem (ref)} We prove a stronger result than Theorem (ref) by deriving the optimal length of confidence intervals. Let $\mathcal{C}(\Lambda)$ be the set of $1-\alpha$ confidence intervals for $\theta$ that are valid uniformly over $\lambda\in\Lambda$. A confidence interval $CI=[L,U]$ consists of $L$ and $U$ measurable functions of the data $W_1,...,W_n$ and has length $|CI|=U-L$. We are interested in how the minimax length of confidence interval in $\mathcal{C}(\Lambda_{\xi_1,\xi_2})$ depends on $(\xi_1,\xi_2)$. We also would like to investigate the adaptivity to $(\xi_1,\xi_2)$ in that whether knowledge of $(\xi_1,\xi_2)$ changes the efficiency of inference. To study these two questions simultaneous, we introduce \[ \mathcal{L}(\Lambda_{\xi_1,\xi_2},\Lambda_{\tilde{\xi}_1,\tilde{\xi}_2})=\inf_{CI\in\mathcal{C}(\Lambda_{\tilde{\xi}_1,\tilde{\xi}_2} )}\sup_{\lambda\in \Lambda_{\xi_1,\xi_2}}\mathbb{E}_{\lambda}|CI|, \] where $\xi_1\geq \tilde{\xi}_1$ and $\xi_2\geq \tilde{\xi}_2$. Therefore, $\Lambda_{\xi_1,\xi_2}\subset \Lambda_{\tilde{\xi}_1,\tilde{\xi}_2}$. When $\xi_1= \tilde{\xi}_1$ and $\xi_2 =\tilde{\xi}_2$, then the above measures the minimax length of confidence intervals in $\Lambda_{\xi_1,\xi_2}$. When $\xi_1> \tilde{\xi}_1$ and $\xi_2 >\tilde{\xi}_2$, the above tells us the performance of confidence intervals that are valid in the enlarged class $\Lambda_{\tilde{\xi}_1,\tilde{\xi}_2}$. If a confidence interval that is valid in the enlarged class automatically achieves the same optimal length when the data distribution $\lambda$ is in the smaller $\Lambda_{\xi_1,\xi_2}$, then this confidence interval adapts to the true smoothness of the problem. If such a confidence interval does not exist, then we say that there is lack of adaptivity to $(\xi_1,\xi_2)$. Indeed, this is what we will show. It turns out that $\mathcal{L}(\Lambda_{\xi_1,\xi_2},\Lambda_{\tilde{\xi}_1,\tilde{\xi}_2})$ does not even depend on $(\xi_1,\xi_2)$. Therefore, if $\xi_1> \tilde{\xi}_1$ and $\xi_2 >\tilde{\xi}_2$, then $\Lambda_{\xi_1,\xi_2}$ is a very small subset of $\Lambda_{\tilde{\xi}_1,\tilde{\xi}_2}$ but no confidence interval with validity on $\Lambda_{\tilde{\xi}_1,\tilde{\xi}_2}$ can automatically exploit the extra smoothness of the data distribution. Our result is in the following statement. \begin{theorem} If Assumption (ref) is satisfied and there exist constants $\kappa_1,\kappa_2>0$ such that $ \kappa_{1}\log p\leq n\leq\kappa_{2}p\log p$ for large enough $n$, then for any $$\xi_{1}\geq\tilde{\xi}_{1}\geq 0, \ \xi_{2}\geq\tilde{\xi}_{2}\geq0, \textit{ and } \tilde{\xi} =\max\{\tilde{\xi}_{1},\tilde{\xi}_{2}\}\leq1/2, $$ \[ \mathcal{L}(\Lambda_{\xi_{1},\xi_{2}},\Lambda_{\tilde{\xi}_{1},\tilde{\xi} _{2}}) \geq C(n^{-1}\log p)^{2\tilde{\xi}/(2\tilde{\xi}+1)}, \] where $C>0$ is a constant depending only on $\tilde{\xi}_{1}$, $\tilde{\xi}_{2}$, $\kappa_1,\kappa_2,\alpha,C_{0}$. \end{theorem} { Theorem (ref) has two key implications. First, if $ \tilde{\xi} = \max\{\tilde{\xi}_{1},\tilde{\xi}_{2}\} $ is significantly smaller than \(1/2\), then the rate $ \mathcal{L}\left(\Lambda_{\xi_{1},\xi_{2}},\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}}\right) $ can be much slower than the standard parametric rate \(n^{-1/2}\). Second, Theorem (ref) shows that adaptivity to the sizes of \(\xi_{1}\) and \(\xi_{2}\) is impossible. Notice that the lower bound for \[ \mathcal{L}\left(\Lambda_{\xi_{1},\xi_{2}},\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}}\right) \] depends only on \(\max\{\tilde{\xi}_{1},\tilde{\xi}_{2}\}\) and not on the specific values of \((\xi_{1},\xi_{2})\). This implies that any confidence interval valid over \(\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}}\) with \(\max\{\tilde{\xi}_{1},\tilde{\xi}_{2}\}\le 1/2\) cannot achieve an expected width of \(n^{-1/2}\) even when restricted to a smaller subspace \(\Lambda_{\xi_{1},\xi_{2}}\), no matter how narrow that space is. In other words, no confidence interval can simultaneously (1) be valid over \(\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}}\) with \(\max\{\tilde{\xi}_{1},\tilde{\xi}_{2}\} < 1/2\) and (2) have an expected width of \(O(n^{-1/2})\) on a (possibly much smaller) subspace \(\Lambda_{\xi_{1},\xi_{2}}\). Consequently, it is impossible to distinguish between the cases \(\max\{\xi_{1},\xi_{2}\} < 1/2\) and \(\max\{\xi_{1},\xi_{2}\} > 1/2\) from the data. In particular, no test can yield a confidence interval of length \(1/\sqrt{n}\) when \(\max\{\xi_{1},\xi_{2}\} < 1/2\). } {We emphasize that although \(\mathcal{L}(\Lambda_{\xi_{1},\xi_{2}},\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}})\) is defined as the expected confidence interval length, our rates are not driven by rare extreme values. The interval lengths are bounded, so the expected length is not skewed by outliers. } \begin{lemma} Let $k\in\mathbb{N}$ and define $\mathcal{Q}_{k}=\{v\in\{0,1\}^{p}:\ \|v\|_{0}=k\}$. Let $v$ and $u$ be two independent vectors that have a uniform distribution on $\mathcal{Q}_{k}$. Then for any $D\geq0$, \[ \mathbb{E}\exp\left(Du'v\right)<\exp\left(\exp\left(D+\log(k^{2}/p)\right)\right). \] \end{lemma} \begin{proof}[Proof of Lemma (ref)] Let $N=|\mathcal{Q}_{k}|$. We list elements in $\mathcal{Q}_{k}$, i.e., $\mathcal{Q}_{k}=\{x_{1},...,x_{N}\}$. Then \[ \mathbb{E}\exp\left(Du'v\right)=N^{-2}\sum_{j_{2}=1}^{N}\sum_{j_{1}=1}^{N}\exp\left(Dx_{j_{1}}'x_{j_{2}}\right)\overset{\text{(i)}}{=}N^{-1}\sum_{j=1}^{N}\exp\left(Dx_{1}'x_{j}\right)=\mathbb{E}\exp(Dx_{1}'v), \] (i) follows by the observation that $\sum_{j_{1}=1}^{N}\exp(Dx_{j_{1}}'x_{j_{2}})$ does not depend on $j_{2}$. Without loss of generality, we take $x_{1}=(1,...,1,0,...,0)^{\top}$, i.e., the vector whose first $k$ entries are nonzero. Let $\mathcal{C}_{n,k}$ be the population that consists of $n$ elements with $n-k$ elements being 0 and the remaining $k$ being 1. Let $\{\xi_{i}\}_{i=1}^{k}$ be a random sample without replacement from the population of $\mathcal{C}_{n,k}$. We observe that $x_{1}'v$ has the same distribution as $\sum_{i=1}^{k}\xi_{i}$. Then \[ \mathbb{E}\exp(Dx_{1}'v)=\mathbb{E}\exp\left(D\sum_{i=1}^{k}\xi_{i}\right). \] Let $\{\zeta_{i}\}_{i=1}^{k}$ be a random sample with replacement from $\mathcal{C}_{n,k}$. In other words, $\{\zeta_{i}\}_{i=1}^{k}$ is i.i.d Bernoulli with $\mathbb{E}(\zeta_{i})=k/p$. Since $x\mapsto\exp(Dx)$ is a convex function, we can use Theorem 4 of hoeffding1963probability and obtain that \begin{multline*} \mathbb{E}\exp\left(D\sum_{i=1}^{k}\xi_{i}\right) \leq \mathbb{E}\exp\left(D\sum_{i=1}^{k}\zeta_{i}\right) =\left[\mathbb{E}\exp(D\zeta_{1})\right]^{k} \\ \overset{\text{(i)}}{=} \left(1-\frac{k}{p}+\frac{k}{p}\exp(D)\right)^{k}\overset{\text{(ii)}} {\leq}\exp\left(\frac{k^{2}}{p}[\exp(D)-1]\right)\\ <\exp\left(\frac{k^{2}}{p}\exp(D)\right)=\exp\left(\exp\left(D+\log(k^{2}/p)\right)\right) \hskip 90pt \end{multline*} where (i) follows by the moment generating function of Bernoulli distributions, (ii) follows by the elementary inequality $1+x\leq\exp(x)$ for $x\geq0$. The proof is complete. \end{proof} \begin{lemma} Let $k$ be a positive integer and define $\mathcal{Q}_{k}=\{v\in\{0,1\}^{p}:\ \|v\|_{0}=k\}$. Let $N=|\mathcal{Q}_{k}|$. We list elements in $\mathcal{Q}_{k}$, i.e., $\mathcal{Q}_{k}=\{\delta_{1},...,\delta_{N}\}$. Let \[ \Sigma_{j}=\begin{pmatrix}1 & 0 & q_{1}\delta_{j}'\\ 0 & 1 & q_{2}\delta_{j}'\\ q_{1}\delta_{j} & q_{2}\delta_{j} & I_{p} \end{pmatrix} \] and $\Sigma_{*}=I_{p+2}$. Let $P_{j}$ denote the distribution of $N(0,\Sigma_{j})$ and $P_{*}$ denote the distribution of $N(0,I_{p+2})$. If $n^{-1}k\log(p/k^{2})\leq3$ and $6n(q_{1}^{2}+q_{2}^{2})\leq\log(p/k^{2})$, then \[ {\rm TV}\left(N^{-1}\sum_{j=1}^{N}P_{j},\ P_{*}\right)\leq\sqrt{\exp\left(\sqrt{k^{2}/p}\right)-1}. \] \end{lemma} \begin{proof}[Proof of Lemma (ref)] Let $\mathbb{E}_{*}$ denotes the expectation under $P_{*}$. By Lemma 3 in cai2017confidence, we have that \begin{equation} \mathbb{E}_{*}\frac{dP_{j_{1}}}{dP_{*}}\frac{dP_{j_{2}}}{dP_{*}}=\left[\det\left(I_{p+2}-(\Sigma_{*}^{-1}\Sigma_{j_{1}}-I_{p+2})(\Sigma_{*}^{-1}\Sigma_{j_{2}}-I_{p+2})\right)\right]^{-n/2}. \end{equation} By the definitions of $\Sigma_{j}$ and $\Sigma_{*}$, we have that \[ \Sigma_{*}^{-1}\Sigma_{j}-I_{p+2}=\begin{pmatrix}0 & 0 & q_{1}\delta_{j}'\\ 0 & 0 & q_{2}\delta_{j}'\\ q_{1}\delta_{j} & q_{2}\delta_{j} & 0 \end{pmatrix}. \] Then we have \begin{align*} & I_{p+2}-(\Sigma_{*}^{-1}\Sigma_{j_{1}}-I_{p+2})(\Sigma_{*}^{-1}\Sigma_{j_{2}}-I_{p+2})\\ & =I_{p+2}-\begin{pmatrix}0 & 0 & q_{1}\delta_{j_{1}}'\\ 0 & 0 & q_{2}\delta_{j_{1}}'\\ q_{1}\delta_{j_{1}} & q_{2}\delta_{j_{1}} & 0 \end{pmatrix}\begin{pmatrix}0 & 0 & q_{1}\delta_{j_{2}}'\\ 0 & 0 & q_{2}\delta_{j_{2}}'\\ q_{1}\delta_{j_{2}} & q_{2}\delta_{j_{2}} & 0 \end{pmatrix}\\ & =I_{p+2}-\begin{pmatrix}q_{1}^{2}\delta_{j_{1}}'\delta_{j_{2}} & q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}} & 0\\ q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}} & q_{2}^{2}\delta_{j_{1}}'\delta_{j_{2}} & 0\\ 0 & 0 & (q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}\delta_{j_{2}}' \end{pmatrix}\\ & =\begin{pmatrix}1-q_{1}^{2}\delta_{j_{1}}'\delta_{j_{2}} & -q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}} & 0\\ -q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}} & 1-q_{2}^{2}\delta_{j_{1}}'\delta_{j_{2}} & 0\\ 0 & 0 & I_{p}-(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}\delta_{j_{2}}' \end{pmatrix}. \end{align*} Therefore, \begin{align*} & \det\left[I_{p+2}-(\Sigma_{*}^{-1}\Sigma_{j_{1}}-I_{p+2})(\Sigma_{*}^{-1}\Sigma_{j_{2}}-I_{p+2})\right]\\ & =\det\left(I_{p}-(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}\delta_{j_{2}}'\right)\times\det\begin{pmatrix}1-q_{1}^{2}\delta_{j_{1}}'\delta_{j_{2}} & -q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}}\\ -q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}} & 1-q_{2}^{2}\delta_{j_{1}}'\delta_{j_{2}} \end{pmatrix}\\ & \overset{\text{(i)}}{=}\left(1-(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}'\delta_{j_{2}}\right)\times\det\begin{pmatrix}1-q_{1}^{2}\delta_{j_{1}}'\delta_{j_{2}} & -q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}}\\ -q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}} & 1-q_{2}^{2}\delta_{j_{1}}'\delta_{j_{2}} \end{pmatrix}\\ & =\left(1-(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}'\delta_{j_{2}}\right)\times\left\{ \left(1-q_{1}^{2}\delta_{j_{1}}'\delta_{j_{2}}\right)\left(1-q_{2}^{2}\delta_{j_{1}}'\delta_{j_{2}}\right)-\left(-q_{1}q_{2}\delta_{j_{1}}'\delta_{j_{2}}\right)^{2}\right\} \\ & =\left(1-(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}'\delta_{j_{2}}\right)^{2}, \end{align*} where (i) follows by Sylvester's determinant identity. By ((ref)), we have that \begin{equation} \mathbb{E}_{*}\frac{dP_{j_{1}}}{dP_{*}}\frac{dP_{j_{2}}}{dP_{*}}=\left(1-(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}'\delta_{j_{2}}\right)^{-n}\overset{\text{(i)}}{\leq}\exp\left(3n(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}'\delta_{j_{2}}\right), \end{equation} where (i) follows by $(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}'\delta_{j_{2}}j\leq(q_{1}^{2}+q_{2}^{2})k\leq1/2$ and the fact that $(1-x)^{-n}<\exp(3xn)$ for any $x\in[0,1/2]$. To see the above, define $f(x)=-3x-\log(1-x)$. Notice that $f(\cdot)$ is convex on $[0,1/2]$ by checking $f''(\cdot)$. Also notice that $f(0)<0$ and $f(1/2)<0$. Hence, $f(x)<0$ on $[0,1/2]$. This means $-\log(1-x)\leq3x$. Multiplying both sides by $n$ and taking exponential, we obtain $(1-x)^{-n}\leq\exp(3xn)$. By ((ref)), we have \begin{align} \mathbb{E}_{*}\left(N^{-1}\sum_{j=1}^{N}\frac{dP_{j}}{dP_{*}}-1\right)^{2} & =N^{-2}\sum_{j_{2}=1}^{N}\sum_{j_{1}=1}^{N}\mathbb{E}_{*}\frac{dP_{j_{1}}}{dP_{*}}\frac{dP_{j_{2}}}{dP_{*}}-1 \\ &\leq N^{-2}\sum_{j_{2}=1}^{N}\sum_{j_{1}=1}^{N}\exp\left(3n(q_{1}^{2}+q_{2}^{2})\delta_{j_{1}}'\delta_{j_{2}}\right)-1\nonumber \\ & \overset{\text{(i)}}{<}\exp\left(\exp\left(3n(q_{1}^{2}+q_{2}^{2})+\log(k^{2}/p)\right)\right)-1 \nonumber \\ & \overset{\text{(ii)}}{=}\exp\left(\exp\left((1/2)\log\frac{k^{2}}{p}\right)\right)-1=\exp\left(\sqrt{k^{2}/p}\right)-1, \end{align} where (i) follows by Lemma (ref) and (ii) follows by $q_{1}^{2}+q_{2}^{2}\leq(6n)^{-1}\log(p/k^{2})$. The desired result follows by Equation (2.27) of Tsybakov2009. \end{proof} \begin{lemma} Suppose that $q\neq0$ and $\delta\in\{0,1\}^{p}$ with $\|\delta\|_{0}=k$. Let $\xi>0$. If $|q|k^{\xi+1/2}\leq\sqrt{C}$, then $(k-t)q^{2}\leq Ct^{-2\xi}$ for $1\leq t\leq k$. \end{lemma} \begin{proof}[Proof of Lemma (ref)] We shall show that for any $0\leq t\leq k$, we have \[ kt^{2\xi}-t^{2\xi+1}\leq Cq^{-2}. \] The maximum of this mapping on $[0,k]$ is either at the end points or at an interior point. Taking the first-order condition of the mapping $t\mapsto kt^{2\xi}-t^{2\xi+1}$ and setting it zero, we have that $t=\alpha k$ with $\alpha=2\xi/(2\xi+1)$, which corresponds to a function value of $k^{2\xi+1}(\alpha^{2\xi}-\alpha^{2\xi+1})$. Notice that the requirement of $k^{2\xi+1}(\alpha^{2\xi}-\alpha^{2\xi+1})\leq Cq^{-2}$ is satisfied by $k^{2\xi+1}\leq Cq^{-2}$ since $\alpha\in(0,1)$. Clearly, the value of the mapping $t\mapsto kt^{2\xi}-t^{2\xi+1}$ at the end points ($t=0$ and $t=k$) is less than $Cq^{-2}$. The desired result follows. \end{proof} \begin{proof}[Proof of Theorems (ref) and (ref)]Let \[ k=\left\lfloor c_{0}\left(n/\log p\right)^{1/(2\tilde{\xi}+1)}\right\rfloor , \] where $c_{0}>0$ is a constant to be chosen. Let $q_{n}=c_{1}\sqrt{n^{-1}\log p}$ and $c_{1}>0$ is a constant to be chosen. Define $\mathcal{Q}_{k}=\{v\in\{0,1\}^{p}:\ \|v\|_{0}=k\}$. Let $N=|\mathcal{Q}_{k}|$. Clearly, $N={p \choose k}$. We list elements in $\mathcal{Q}_{k}$, i.e., $\mathcal{Q}_{k}=\{\delta_{1},...,\delta_{N}\}$. For $1\leq j\leq N$, define $\beta_{j}=-2q_{n}\delta_{j}$ and $\phi_{j}=q_{n}\delta_{j}$. Clearly, we can choose $c_{0},c_{1}$ such that \[ 2q_{n}k^{\tilde{\xi}+1/2}\leq2c_{1}c_{0}^{\tilde{\xi}+1/2}\leq C_{0}\qquad{\rm and}\qquad\exp(c_{0}^{2}\kappa_{2})-1\leq\alpha^{2}. \] By Lemma (ref), this implies that $-2q_{n}\delta_{j},q_{n}\delta_{j}\in\mathcal{M}_{C_{0},\tilde{\xi}}$. We define $\lambda_{*}=(0,0,0,1,1)$ and for $1\leq j\leq N$, $\lambda_{j}=(2q_{n}^{2}k,-2q_{n}\delta_{j},q_{n}\delta_{j},1-q_{n}^{2}k,\sigma_{\varepsilon,1}^{2})$ with $\sigma_{\varepsilon,1}^{2}=1-4q_{n}^{2}k(1-q_{n}^{2}k)^{2}-4q_{n}^{4}k^{2}(1-q_{n}^{2}k)$. Notice that \[ q_{n}^{2}k\leq c_{1}^{2}c_{0}\left(n/\log p\right)^{-1+1/(2\tilde{\xi}+1)}=c_{1}^{2}c_{0}\left(n/\log p\right)^{-2\tilde{\xi}/(2\tilde{\xi}+1)}\leq c_{1}^{2}c_{0}\kappa_{1}^{-2\tilde{\xi}/(2\tilde{\xi}+1)}. \] Then we can choose $c_{1},c_{0}$ small enough such that $1-q_{n}^{2}k,\sigma_{\varepsilon,1}^{2}\in[1/2,2]\subseteq[M^{-1},M]$ (due to $M\geq2$). Hence, $\lambda_{*},\lambda_{j}\in\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}}$. Then under $P_{\lambda_{*}}$, the distribution of $(Y_{i},Z_{i},X_{i})$ is $N(0,I_{p+2})$. Under $P_{\lambda_{j}}$, $Y_{i}=Z_{i}\cdot(2q_{n}^{2}k)-2q_{n}X_{i}'\delta_{j}+\varepsilon_{i}$ and $Z_{i}=q_{n}X_{i}'\delta_{j}+u_{i}$, where $\mathbb{E}\varepsilon_{i}^{2}=\sigma_{\varepsilon,1}^{2}$ and $Eu_{i}^{2}=1-q_{n}^{2}k$. After simple calculations, we have that under $P_{\lambda_{j}}$, the distribution of $(Y_{i},Z_{i},X_{i})$ is $N(0,\Sigma_{j})$ with \[ \Sigma_{j}=\begin{pmatrix}1 & 0 & -q_{n}\delta_{j}'\\ 0 & 1 & q_{n}\delta_{j}'\\ -q_{n}\delta_{j} & q_{n}\delta_{j} & I_{p} \end{pmatrix}. \] Clearly, we can choose $c_{0},c_{1}$ small enough such that \begin{eqnarray*} n^{-1}k\log(p/k^{2}) &\leq n^{-1}k\log p\leq c_{0}n^{-1}\left(n/\log p\right)^{1/(2\tilde{\xi}+1)}\log p \\ &=c_{0}\left(n/\log p\right)^{-2\tilde{\xi}/(2\tilde{\xi}+1)}\leq c_{0}\kappa_{1}^{-2\tilde{\xi}/(2\tilde{\xi}+1)}\leq3 \end{eqnarray*} and \[ \frac{6n(q_{n}^{2}+q_{n}^{2})}{\log(p/k^{2})}\leq\frac{6n(q_{n}^{2}+q_{n}^{2})}{\log(p)}=12c_{1}^{2}\leq1. \] Hence, the assumptions of Lemma (ref) hold. Therefore, \begin{equation} {\rm TV}\left(N^{-1}\sum_{j=1}^{N}P_{\lambda_{j}},\ P_{\lambda_{*}}\right)\leq\sqrt{\exp\left(\sqrt{k^{2}/p}\right)-1}. \end{equation} Let $CI_{n}=[l_{n},u_{n}]$ be an arbitrary confidence interval for $\theta=\tau(\lambda)$ with nomimal coverage probability $1-\alpha$ on $\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}}$. In other words, \begin{equation} \inf_{\lambda\in\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}}}P_{\lambda}\left(l_{n}\leq\tau(\lambda)\leq u_{n}\right)=\inf_{\lambda\in\Lambda_{\tilde{\xi}_{1},\tilde{\xi}_{2}}}P_{\lambda}\left(\tau(\lambda)\in CI_{n}\right)\geq1-\alpha. \end{equation} We now define the random variable \[ \Psi_{n}=\mathbf{1}\left\{ 2q_{n}^{2}k\in CI_{n}\right\} . \] Thus, by ((ref)), $n\leq\kappa_{2}p\log p$ and $\exp(c_{0}^{2}\kappa_{2})-1\leq\alpha^{2}$, we have \begin{eqnarray*} \left|N^{-1}\sum_{j=1}^{N}\mathbb{E}_{\lambda_{j}}\Psi_{n}-\mathbb{E}_{\lambda_{*}}\Psi_{n}\right| &\leq&{\rm TV}\left(N^{-1}\sum_{j=1}^{N}P_{\lambda_{j}},\ P_{\lambda_{*}}\right)\leq\sqrt{\exp\left(\sqrt{k^{2}/p}\right)-1}\\ &\leq&\sqrt{\exp\left(\frac{c_{0}^{2}n}{p\log p}\right)-1}\leq\sqrt{\exp(c_{0}^{2}\kappa_{2})-1}\leq\alpha. \end{eqnarray*} Therefore, \begin{eqnarray*} P_{\lambda_{*}}\left(2q_{n}^{2}k\in CI_{n}\right) &=&\mathbb{E}_{\lambda_{*}}\Psi_{n}\geq N^{-1}\sum_{j=1}^{N}\mathbb{E}_{\lambda_{j}}\Psi_{n}-\alpha=N^{-1}\sum_{j=1}^{N}P_{\lambda_{j}}\left(2q_{n}^{2}k\in CI_{n}\right)-\alpha\\ &\overset{\text{(i)}}{=}&N^{-1}\sum_{j=1}^{N}P_{\lambda_{j}}\left(\tau(\lambda_{j})\in CI_{n}\right)-\alpha\overset{\text{(ii)}}{\geq}1-2\alpha, \end{eqnarray*} where (i) follows by $\tau(\lambda_{j})=2q_{n}^{2}k$ and (ii) follows by ((ref)). On the other hand, by $\tau(\lambda_{*})=0$ and ((ref)), we have \[ P_{\lambda_{*}}\left(0\in CI_{n}\right)\geq1-\alpha. \] The above two displays imply that $P_{\lambda_{*}}\left(\{0,2q_{n}^{2}k\}\subset CI_{n}\right)\geq1-3\alpha$. Since $CI_{n}$ is an interval, the event of $\{0,2q_{n}^{2}k\}\subset CI_{n}$ is the same as the event $[0,2q_{n}^{2}k]\subset CI_{n}$. Hence, we have $P_{\lambda_{*}}\left([0,2q_{n}^{2}k]\subset CI_{n}\right)\geq1-3\alpha$, which means \[ \frac{\mathbb{E}_{\lambda_{*}}|CI_{n}|}{2q_{n}^{2}k}\geq P_{\lambda_{*}}\left(|CI_{n}|\geq2q_{n}^{2}k\right)\geq1-3\alpha. \] Since $2q_{n}^{2}k\asymp(n/\log p)^{-2\tilde{\xi}/(2\tilde{\xi}+1)}$ (by the definitions of $q_{n}$ and $k$), we have $$ \mathbb{E}_{\lambda_{*}}|CI_{n}| \gtrsim (n^{-1} \log p )^{2\tilde{\xi}/(2\tilde{\xi}+1)}. $$ This proves Theorem (ref). Let $a=\mathcal{R}_{\xi_{1},\xi_{2}}$. Then there exists an estimator $\hat{\theta}$ such that $\mathbb{E}_{\lambda} | \hat{\theta}(W)-\theta|\leq 2a$ for any $\lambda=(\theta,\beta,\phi,\sigma_{\varepsilon}^{2},\sigma_{u}^{2})\in \Lambda(\xi_{1},\xi_{2})$. Consider the confidence interval $CI=[\hat{\theta}-2a/\alpha,\hat{\theta}+2a/\alpha]$. Then for any $\lambda=(\theta,\beta,\phi,\sigma_{\varepsilon}^{2},\sigma_{u}^{2})\in \Lambda(\xi_{1},\xi_{2})$, $$ P_{\lambda} \left( \theta\in CI\right)=P_{\lambda} \left( |\hat{\theta}-\theta|>2a/\alpha \right)\leq \mathbb{E}_{\lambda} | \hat{\theta}(W)-\theta|/(2a/\alpha)\leq 2a/(2a/\alpha)=\alpha. $$ Therefore, $CI$ is a confidence interval that has coverage probability $1-\alpha$. Thus, $|CI|\gtrsim (n^{-1} \log p )^{2\tilde{\xi}/(2\tilde{\xi}+1)}$. On the other hand, $|CI|=4a/\alpha$, which means that $a \gtrsim (n^{-1} \log p )^{2\tilde{\xi}/(2\tilde{\xi}+1)}$. This proves Theorem (ref). \end{proof} \subsection{Proof of Theorem (ref)} We next give several Lemmas that are useful in the proof of Theorem (ref). \begin{lemma} Let $C_{0},C_{1},C_{2},\xi_{1},\xi_{2}>0$ be constants. Define $k=\left\lfloor C_{0}n^{1/(\xi_{1}+\xi_{2} +1/2)}\right\rfloor $, $c_{1}=C_{1}k^{-\xi_{1}-1/2}$ and $c_{2}=C_{2}k^{-\xi_{2}-1/2}$, where $\left\lfloor \cdot \right\rfloor $ denotes the integer part. Let $\omega\sim N(0,I_{k})$ . There exist a constant $D>0$ depending only on $(\xi_{1},\xi_{2},M_{1})$ such that for any $C_{1},C_{2}\in (0,D)$ , we have $P(c_{1}\omega\in \mathcal{S}^{*}_{M_1,\xi_{1}})=1-o(1)$ and $P(c_{2}\omega\in \mathcal{S}^{*}_{M_1,\xi_{2}})=1-o(1)$. \end{lemma} \begin{proof}[Proof of Lemma (ref)] We now show $P(c_{1}\omega\in\mathcal{S}^{*}_{M_1,\xi_{1}} )=1-o(1)$. Let $\omega=(\omega_{1},...,\omega_{k})^{\top}\in{\mathbb{R} }^{k}$ follow $N(0,I_{k})$. The goal is to show that the following event occurs with probability $1-o(1)$, \[ \bigcap_{t=1}^{k-1}\left\{ c_{1}\sqrt{\sum_{j=t+1}^{k}\omega_{j}^{2}}\leq M_{1}t^{-\xi_{1}}\right\} . \] We can rewrite this event as \[ \bigcap_{t=1}^{k-1}\left\{ \sum_{j=t+1}^{k}(\omega_{j}^{2}-1)\leq\left( c_{1}^{-2}M_{1}^{2}t^{-2\xi_{1}}-(k-t)\right) \right\} . \] Notice that $\mathbb{E}(\omega_{j}^{2}-1)^{2}=3$. By Kolmogorov's maximal inequality, we have that for $x>0$, \[ P\left( \max_{1\leq t\leq k-1}\left| \sum_{j=t+1}^{k}(\omega_{j} ^{2}-1)\right| >x\right) \leq\frac{3k}{x^{2}}. \] Hence, \[ P\left( \max_{1\leq t\leq k-1}\left| \sum_{j=t+1}^{k}(\omega_{j} ^{2}-1)\right| >\sqrt{k\log k}\right) =o(1). \] Therefore, we only need to show \[ \min_{1\leq t\leq k}c_{1}^{-2}M_{1}^{2}t^{-2\xi_{1}}-(k-t)\geq\sqrt{k\log k}. \] Recall $c_{1}=C_{1}k^{-\xi_{1}-1/2}$ for a constant $C_{1}>0$. We need to show \[ \min_{1\leq t\leq k}M_{1}^{2}C_{1}^{-2}k^{2\xi_{1}+1}t^{-2\xi_{1}}+t\geq k+\sqrt{k\log k}. \] Notice that the left-hand side is the minimum of a convex function of $t$. Therefore, the minimum occurs at $t=t_{*}$ with $t_{*}$ being the solution of the first-order condition \[ -M_{1}^{2}C_{1}^{-2}k^{2\xi_{1}+1}t_{*}^{-2\xi_{1}-1}/(2\xi_{1})+1=0. \] This means $t_{*}=(2\xi_{1}M_{1}^{-2}C_{1}^{2})^{-1/(2\xi_{1}+1)}k$, which means \begin{eqnarray*} &&\min_{t\in{\mathbb{R}}} \left\{ M_{1}^{2}C_{1}^{-2}k^{2\xi_{1}+1}t^{-2\xi_{1} }+t \right\} \\ &=&\left[ (2\xi_{1}M_{1}^{-2})^{-1/(2\xi_{1}+1)}+M_{1}^{2}(2\xi_{1} M_{1}^{-2})^{2\xi_{1}/(2\xi_{1}+1)}\right] k\times C_{1}^{-2/(2\xi_{1}+1)}. \end{eqnarray*} Hence, the right-hand side is larger than $k+\sqrt{k\log k}$ for small enough $C_{1}$. Therefore, we have proved that for $c_{1}=C_{1}k^{-\xi_{1}-1/2}$ with a small constant $C_{1}>0$, $P(c_{1}\omega\in\mathcal{S}^{*}_{M_1,\xi_{1}} )=1-o(1)$. The result for $P(c_{2}\lambda\in\mathcal{S}^{*}_{M_1,\xi_{2}})=1-o(1)$ is analogous. \end{proof} \begin{lemma} Let $W\sim N(0,I_{k})$. Suppose that $A\in \mathbb{R}^{k\times k}$ is positive semi-definite and $b\in R^{k}$. Then \[ \mathbb{E}\operatorname{exp}\left( -\frac{1}{2}W^{\top}AW+b^{\top}W\right) =\left( \det(A+I_{k})\right) ^{-1/2}\operatorname{exp}\left( b^{\top }(A+I_{k})^{-1}b/2\right). \] \end{lemma} \begin{proof} [Proof of Lemma (ref)] Let $A=UBU^{\top}$, where $B$ is diagonal with $B_{jj}\geq0$ and $U^{\top}U=I_{k}$. This is possible because $A$ is positive semi-definite. Define $T=U^{\top}W$ and $c=U^{\top}b$. Clearly, $T$ is Gaussian with mean zero and variance $U^{\top}I_{k}U=I_{k}$. Then we need to compute \[ \mathbb{E}\operatorname{exp}\left( -\frac{1}{2}T^{\top}BT+c^{\top}T\right) =\mathbb{E}\operatorname{exp}\left( \sum_{j=1}^{k}(-T_{j}^{2}B_{jj}/2+c_{j} T_{j})\right) . \] Since $T_{j}\sim N(0,1)$ is independent across $j$, it follows that \begin{align*} \mathbb{E} \operatorname{exp}\left( -\frac{1}{2}T^{\top}BT+c^{\top}T\right) & =\mathbb{E} \operatorname{exp}\left( \sum_{j=1}^{k}(-T_{j}^{2}B_{jj}/2+c_{j} T_{j})\right) \\ & =\prod_{j=1}^{k}\mathbb{E} \operatorname{exp}\left( -T_{j}^{2}B_{jj}/2+c_{j} T_{j}\right) \\ & =\prod_{j=1}^{k}\left[ \int_{-\infty}^{\infty} \operatorname{exp}\left( -x^{2}B_{jj}/2+c_{j}x\right) \cdot\frac{1}{\sqrt{2\pi}} \operatorname{exp} (-x^{2}/2)dx\right] \\ & =\prod_{j=1}^{k}\left[ \frac{1}{\sqrt{B_{jj}+1}} \operatorname{exp}\left( \frac{c_{j}^{2}}{2(B_{jj}+1)}\right) \right] \\ & =\left[ \prod_{j=1}^{k}(B_{jj}+1)\right] ^{-1/2} \operatorname{exp} \left( \frac{1}{2}\cdot\sum_{j=1}^{k}\frac{c_{j}^{2}}{B_{jj}+1}\right) . \end{align*} Since $U^{\top}U=I_{k}$, we have \[ \prod_{j=1}^{k}(B_{jj}+1)=\det(B+I_{k})=\det(UBU^{\top}+I_{k})=\det (A+I_{k}). \] Moreover, since $B$ is diagonal, we have \[ \sum_{j=1}^{k}\frac{c_{j}^{2}}{B_{jj}+1}=c^{\top}(B+I_{k})^{-1}c=(U^{\top }b)^{\top}\left( U^{\top}AU+I_{k}\right) ^{-1}(U^{\top}b)=b^{\top }(A+I_{k})^{-1}b. \] The above three displays imply \[ \mathbb{E} \operatorname{exp}\left( -\frac{1}{2}T^{\top}BT+c^{\top}T\right) =\left( \det(A+I_{k})\right) ^{-1/2} \operatorname{exp}\left( b^{\top }(A+I_{k})^{-1}b/2\right) . \] The proof is complete. \end{proof} \begin{lemma} Let $C_{0},C_{1},C_{2},\xi_{1},\xi_{2}>0$ be constants. Define $k=\left\lfloor C_{0}n^{1/(\xi_{1}+\xi_{2} +1/2)}\right\rfloor $\textit{, }$c_{1}=C_{1}k^{-\xi_{1}-1/2}$ and $c_{2}=C_{2}k^{-\xi_{2}-1/2}$, where $\left\lfloor \cdot \right\rfloor $ denotes the integer part. For any $\omega\in \mathbb{R}^{k}$, let $\Phi_{1,\omega}$ denote the distribution of i.i.d $\{(Y_{i},D_{i},Z_{i})\}_{i=1}^{n}$ given by $Z_{i}\sim N(0,I_{k} )$, \[ \begin{pmatrix} Y_{i}\\ D_{i} \end{pmatrix} \mid Z_i \sim N\left( \begin{pmatrix} c_{1}Z_{i}^{\top}\omega\\ 0 \end{pmatrix} ,\ \Sigma\right) \qquad{\text{and}}\qquad\Sigma= \begin{pmatrix} 1 & c_{1}c_{2}k\\ c_{1}c_{2}k & 1 \end{pmatrix}. \] Let $\Phi_{2,\omega}$ denote the distribution of i.i.d $\{(Y_{i},D_{i},Z_{i})\}_{i=1}^{n}$ given by $Z_{i}\sim N(0,I_{k} )$, \[ \begin{pmatrix} Y_{i}\\ D_{i} \end{pmatrix} \mid Z_i \sim N\left( \begin{pmatrix} c_{1}Z_{i}^{\top}\omega\\ c_{2}Z_{i}^{\top}\omega \end{pmatrix} ,\ \begin{pmatrix} 1 \quad & 0\\ 0 \quad & 1 \end{pmatrix} \right) . \] Assume that $\xi_{2}\geq1/4>\xi_{1}$ and $\xi_{1}+\xi _{2}<1/2$. Let $Z_{i}$ be i.i.d from $N(0,I_{k})$. Then \[ {\text{TV}}(\bar{\phi}_{1},\bar{\phi}_{2})\leq2C_{1}C_{2}C_{0}^{-(\xi_{1} +\xi_{2}+1/2)}+o(1), \] where $TV$ denotes the total variation distance, $\bar{\phi }_{1}=\int_{{\mathbb{R}}^{k}}\Phi_{1,\omega}\mu(\omega)d\omega$, $\bar{\phi}_{2}=\int_{{\mathbb{R}}^{k}}\Phi_{2,\omega}\mu(\omega)d\omega $ and $\mu$ is the density function of $N(0,I_{k})$, i.e., $\mu(\omega)=(2\pi)^{-k/2}\operatorname{exp}(-\Vert\omega\Vert _{2}^{2}/2)$. \end{lemma} \begin{proof} [Proof of Lemma (ref)] For simplicity, we shall use $|\cdot|$ to denote the determinant of a square matrix. Since $\xi_{1}+\xi_{2}<1/2$, we have $k\gg n$. Let $\otimes$ denote the Kronecker product of matrices. Let $Y=(Y_1,...,Y_n)^{\top}\in\mathbb{R}^{n}$, $D=(D_1,...,D_n)^{\top}\in\mathbb{R}^{n}$, $\varepsilon=(\varepsilon_1,...,\varepsilon_n)^{\top}\in\mathbb{R}^{n}$, $u=(u_1,...,u_n)^{\top}\in\mathbb{R}^{n}$ and $Z=(Z_1,...,Z_n)^{\top}\in\mathbb{R}^{n\times k}$. For $j\in\{1,2\}$, let $f_{j,\omega}$ denote the conditional distribution of $(Y,D)$ given $Z$ under $\Phi_{j,\omega}$ and define $\bar{f}_{j} =\int_{{\mathbb{R}}^{k}}f_{j,\omega}\mu(\omega)d\omega$. Let ${\text{KL}}$ denote the Kullback-Leibler divergence. We will proceed in four steps. In the first three steps, we bound ${\text{KL}}(\bar{f}_{1},\bar{f}_{2})$; in the last step, we prove the final result. \textbf{Step 1:} Deriving the mixture $\bar{f}_{1}$ Notice that \[ R= \begin{pmatrix} Y\\ D \end{pmatrix} = \begin{pmatrix} c_{1}Z\\ 0 \end{pmatrix} \omega+ \begin{pmatrix} \varepsilon\\ u \end{pmatrix} =G\omega+ \begin{pmatrix} \varepsilon\\ u \end{pmatrix} , \] where $G= \begin{pmatrix} c_{1}Z\\ 0 \end{pmatrix} =g\otimes Z$ with $g= \begin{pmatrix} c_{1}\\ 0 \end{pmatrix} $ and \[ \mathbb{E} \begin{pmatrix} \varepsilon\\ u \end{pmatrix} \begin{pmatrix} \varepsilon\\ u \end{pmatrix} ^{\top}= \begin{pmatrix} \mathbb{E}\varepsilon\varepsilon^{\top} & \mathbb{E}\varepsilon u^{\top}\\ \mathbb{E} u\varepsilon^{\top} & \mathbb{E} uu^{\top} \end{pmatrix} = \begin{pmatrix} \Sigma_{1}I_{n} & \Sigma_{3}I_{n}\\ \Sigma_{3}I_{n} & \Sigma_{2}I_{n} \end{pmatrix} =\Sigma\otimes I_{n} \] with \[ \Sigma= \begin{pmatrix} 1 & c_{1}c_{2}k\\ c_{1}c_{2}k & 1 \end{pmatrix} . \] Let $S=\Sigma^{-1}$. Notice that \[ S= \begin{pmatrix} S_{1} & S_{3}\\ S_{3} & S_{2} \end{pmatrix} = \begin{pmatrix} 1 & -c_{1}c_{2}k\\ -c_{1}c_{2}k & 1 \end{pmatrix} \frac{1}{1-c_{1}^{2}c_{2}^{2}k^{2}} \] and $|\Sigma|=1-c_{1}^{2}c_{2}^{2}k^{2}$. Thus, under $f_{1,\omega}$, $R$ conditional on $Z$ is Gaussian mean $G\omega$ and variance $\Sigma\otimes I_{n}$. Hence, the density is \begin{align*} f_{1,\omega}&=f_{1,\omega}(R,Z) =(2\pi)^{-n}|\Sigma\otimes I_{n}|^{-1/2} \operatorname{exp}\left( -\frac{1}{2}(R-G\omega)^{\top}(\Sigma\otimes I_{n})^{-1}(R-G\omega)\right) \\ & =(2\pi)^{-n}|S\otimes I_{n}|^{1/2} \operatorname{exp}\left( -\frac{1} {2}(R-G\omega)^{\top}(S\otimes I_{n})(R-G\omega)\right) \\ & =(2\pi)^{-n}|S\otimes I_{n}|^{1/2} \operatorname{exp}\left( -\frac{1} {2}R^{\top}(S\otimes I_{n})R-\frac{1}{2}\omega^{\top}G^{\top}(S\otimes I_{n})G\omega+R^{\top}(S\otimes I_{n})G\omega\right) \end{align*} Hence, we can apply Lemma (ref) and obtain \begin{align*} \bar{f}_{1} & =\int f_{1,\omega}\mu(\omega)d\omega\\ & =\mathbb{E}_{\omega\sim N(0,I_{k})}f_{1,\omega}\\ & =(2\pi)^{-n}|S\otimes I_{n}|^{1/2}\operatorname{exp}\left( -\frac{1} {2}R^{\top}(S\otimes I_{n})R\right) \\ & \qquad \times \mathbb{E}_{\omega\sim N(0,I_{k} )}\operatorname{exp}\left( -\frac{1}{2}\omega^{\top}G^{\top}(S\otimes I_{n})G\omega+R^{\top}(S\otimes I_{n})G\omega\right) \\ & =(2\pi)^{-n}|S\otimes I_{n}|^{1/2}\operatorname{exp}\left( -\frac{1} {2}R^{\top}(S\otimes I_{n})R\right) \times|I+G^{\top}(S\otimes I_{n})G|^{-1/2}\\ & \qquad \times \operatorname{exp}\left( \frac{1}{2}R^{\top}(S\otimes I_{n})G(I+G^{\top}(S\otimes I_{n})G)^{-1}G^{\top}(S\otimes I_{n})R\right) \\ & =(2\pi)^{-n}|S\otimes I_{n}|^{1/2}\cdot|I+G^{\top}(S\otimes I_{n})G|^{-1/2}\operatorname{exp}\left( -\frac{1}{2}R^{\top}MR\right) , \end{align*} where \begin{align*} M & =S\otimes I_{n}-(S\otimes I_{n})G\left( I+G^{\top}(S\otimes I_{n})G\right) ^{-1}G^{\top}(S\otimes I_{n})\\ & =S\otimes I_{n}-(Sg\otimes Z)\left( I+g^{\top}SgZ^{\top}Z\right) ^{-1}(g^{\top}S\otimes Z^{\top}). \end{align*} We notice that \begin{align*} |M| & =\left| S\otimes I_{n}-(S\otimes I_{n})G\left( I+G^{\top}(S\otimes I_{n})G\right) ^{-1}G^{\top}(S\otimes I_{n})\right| \\ & \overset{{\text{(i)}}}{=}|S\otimes I_{n}|\cdot\left| I-G\left( I+G^{\top}(S\otimes I_{n})G\right) ^{-1}G^{\top}(S\otimes I_{n})\right| \\ & \overset{{\text{(ii)}}}{=}|S\otimes I_{n}|\cdot\left| I-\left( I+G^{\top}(S\otimes I_{n})G\right) ^{-1}G^{\top}(S\otimes I_{n})G\right| \\ & =|S\otimes I_{n}|\cdot\left| I-\left( I+G^{\top}(S\otimes I_{n})G\right) ^{-1}\left( I+G^{\top}(S\otimes I_{n})G-I\right) \right| \\ & =|S\otimes I_{n}|\cdot\left| \left( I+G^{\top}(S\otimes I_{n})G\right) ^{-1}\right| \\ & =|S\otimes I_{n}|\cdot\left| I+G^{\top}(S\otimes I_{n})G\right| ^{-1}, \end{align*} where (i) follows by the identity $|H-AB|=|H|\cdot|I-BH^{-1}A|$ with $H=A=S\otimes I_{n}$ and $B=G[ I+G^{\top}(S\otimes I_{n})G] ^{-1}G^{\top}(S\otimes I_{n})$ and (ii) follows by the same identity with $H=I$ and $A=G$. Therefore, \[ \bar{f}_{1}=(2\pi)^{-n}|M|^{1/2} \operatorname{exp}\left( -\frac{1} {2}R^{\top}MR\right) . \] \textbf{Step 2:} Deriving the mixture $\bar{f}_{2}$ We can write \[ R= \begin{pmatrix} Y\\ D \end{pmatrix} = \begin{pmatrix} c_{1}Z\\ c_{2}Z \end{pmatrix} \omega+ \begin{pmatrix} \varepsilon\\ u \end{pmatrix} =\tilde{G}\omega+ \begin{pmatrix} \varepsilon\\ u \end{pmatrix} , \] where $\tilde{G}= \begin{pmatrix} c_{1}Z\\ c_{2}Z \end{pmatrix} =\tilde{g}\otimes Z$ with $\tilde{g}= \begin{pmatrix} c_{1}\\ c_{2} \end{pmatrix} $. Observe that $E \begin{pmatrix} \varepsilon\\ u \end{pmatrix} \begin{pmatrix} \varepsilon\\ u \end{pmatrix} ^{\top}=I_{2n}$. Therefore, under $f_{2,\omega}$, $R$ conditional on $Z$ is Gaussian with mean $\tilde{G}\omega$ and variance $I_{2n}$. This means that the density is \begin{align*} f_{2,\omega}&=f_{2,\omega}(R,Z)=(2\pi)^{-n} \operatorname{exp}\left( -\frac {1}{2}(R-\tilde{G}\omega)^{\top}(R-\tilde{G}\omega)\right) \\ & =(2\pi)^{-n} \operatorname{exp}\left( -\frac{1}{2}R^{\top}R-\frac{1}{2}\omega^{\top }\tilde{G}^{\top}\tilde{G}\omega+R^{\top}\tilde{G}\omega\right) . \end{align*} By a similar calculation as in step 1, we have \begin{align*} \bar{f}_{2} & =\int f_{2,\omega}\mu(\omega)d\omega\\ & =\mathbb{E}_{\omega\sim N(0,I_{k})}f_{2,\omega}\\ & =(2\pi)^{-n} \operatorname{exp}\left( -\frac{1}{2}R^{\top}R\right) \mathbb{E}_{\omega\sim N(0,I_{k})} \operatorname{exp}\left( -\frac{1}{2} \omega^{\top}\tilde{G}^{\top}\tilde{G}\omega+R^{\top}\tilde{G} \omega\right) \\ & =(2\pi)^{-n} \operatorname{exp}\left( -\frac{1}{2}R^{\top}R\right) \times|I+\tilde{G}^{\top}\tilde{G}|^{-1/2} \operatorname{exp}\left( \frac {1}{2}R^{\top}\tilde{G}\left( I+\tilde{G}^{\top}\tilde{G}\right) ^{-1}\tilde{G}^{\top}R\right) \\ & =(2\pi)^{-n}|\tilde{M}|^{1/2} \operatorname{exp}\left( -\frac{1} {2}R^{\top}\tilde{M}R\right) , \end{align*} where \[ \tilde{M}=I_{2n}-\tilde{G}\left( I+\tilde{G}^{\top}\tilde{G}\right) ^{-1}\tilde{G}^{\top}=I-(\tilde{g}\otimes Z)\left( I+\|\tilde{g}\|_{2} ^{2}Z^{\top}Z\right) ^{-1}(\tilde{g}^{\top}\otimes Z^{\top}). \] \textbf{Step 3:} Computeing KL divergence between the two mixtures Hence, $\bar{f}_{1}$ and $\bar{f}_{2}$ represents $N(0,\Sigma_{1})$ and $N(0,\Sigma_{2})$, respectively, where $\Sigma_{1}=M^{-1}$ and $\Sigma _{2}=\tilde{M}^{-1}$. Now we consider the KL divergence conditional on $Z$ \[ {\text{KL}}(\bar{f}_{1}(\cdot,Z),\bar{f}_{2}(\cdot,Z))=\frac{1}{2}\left( \log\frac{|\Sigma _{2}|}{|\Sigma_{1}|}-2n+\mathrm{trace}(\Sigma_{2}^{-1}\Sigma_{1})\right) =\frac{1}{2}\left( \log\frac{|M|}{|\tilde{M}|}-2n+\mathrm{trace}(M^{-1} \tilde{M})\right) . \] Using the Woodbury's identity, we have \[ M^{-1}=\Sigma\otimes I_{n}+(gg^{\top})\otimes(ZZ^{\top}). \] Therefore, \begin{align*} & \mathrm{trace}(M^{-1}\tilde{M})\\ & =\mathrm{trace}\left[ \left( \Sigma\otimes I_{n}+(gg^{\top} )\otimes(ZZ^{\top})\right) \left( I-(\tilde{g}\otimes Z)\left( I+\|\tilde{g}\|_{2}^{2}Z^{\top}Z\right) ^{-1}(\tilde{g}^{\top}\otimes Z^{\top})\right) \right] \\ & =\mathrm{trace}\left[ \Sigma\otimes I_{n}+(gg^{\top})\otimes(ZZ^{\top })-\left( \Sigma\otimes I_{n}+(gg^{\top})\otimes(ZZ^{\top})\right) (\tilde{g}\otimes Z)\left( I+\|\tilde{g}\|_{2}^{2}Z^{\top}Z\right) ^{-1}(\tilde{g}^{\top}\otimes Z^{\top})\right] \\ & =n\cdot\mathrm{trace}(\Sigma)+\|g\|_{2}^{2}\mathrm{trace}(Z^{\top }Z) \\ & \qquad -\mathrm{trace}\left[ (\tilde{g}^{\top}\otimes Z^{\top})\left( \Sigma\otimes I_{n}+(gg^{\top})\otimes(ZZ^{\top})\right) (\tilde {g}\otimes Z)\left( I+\|\tilde{g}\|_{2}^{2}Z^{\top}Z\right) ^{-1}\right] \\ & =n\cdot\mathrm{trace}(\Sigma)+\|g\|_{2}^{2}\mathrm{trace}(Z^{\top }Z)-\mathrm{trace}\left[ \left( \tilde{g}^{\top}\Sigma\tilde{g}Z^{\top }Z+(g^{\top}\tilde{g})^{2}Z^{\top}ZZ^{\top}Z\right) \left( I+\|\tilde{g}\|_{2}^{2}Z^{\top}Z\right) ^{-1}\right] . \end{align*} By Sylvester's theorem, we have \begin{align*} |\tilde{M}| & =\left| I-(\tilde{g}\otimes Z)\left( I+\|\tilde{g}\|_{2} ^{2}Z^{\top}Z\right) ^{-1}(\tilde{g}^{\top}\otimes Z^{\top})\right| \\ & =\left| I-(\tilde{g}^{\top}\otimes Z^{\top})(\tilde{g}\otimes Z)\left( I+\|\tilde{g}\|_{2}^{2}Z^{\top}Z\right) ^{-1}\right| \\ & =\left| I-\|\tilde{g}\|_{2}^{2}Z^{\top}Z\left( I+\|\tilde{g}\|_{2} ^{2}Z^{\top}Z\right) ^{-1}\right| =\left| \left( I+\|\tilde{g}\|_{2} ^{2}Z^{\top}Z\right) ^{-1}\right| =\left| I+(c_{1}^{2}+c_{2} ^{2})Z^{\top}Z\right| ^{-1} \end{align*} and \begin{align*} |M| & =\left| S\otimes I_{n}-(Sg\otimes Z)\left( I+g^{\top}SgZ^{\top }Z\right) ^{-1}(g^{\top}S\otimes Z^{\top})\right| \\ & =\left| S\otimes I_{n}\right| \cdot\left| I-\left( I+g^{\top }SgZ^{\top}Z\right) ^{-1}(g^{\top}S\otimes Z^{\top-1}\otimes I_{n})(Sg\otimes Z)\right| \\ & =\left| S\otimes I_{n}\right| \cdot\left| I-\left( I+g^{\top }SgZ^{\top}Z\right) ^{-1}g^{\top}SgZ^{\top}Z\right| \\ & =\left| S\right| ^{n}\cdot\left| \left( I+g^{\top}SgZ^{\top }Z\right) ^{-1}\right| =\left| S\right| ^{n}\cdot\left| I+c_{1}^{2} S_{1}Z^{\top}Z\right| ^{-1}. \end{align*} Let $\sigma_{1},...,\sigma_{n}$ be the eigenvalues of $Z^{\top}Z$. By Corollary 5.35 of Vershynin (2012) (with $t=\sqrt{n}$), we have that \[ P\left( \max_{1\leq i\leq n}|\sqrt{\sigma}_{i}-\sqrt{k}|>2\sqrt{n}\right) \leq2 \operatorname{exp}(-n/2). \] Thus, with probability at least $1-2 \operatorname{exp}(-n/2)$, $\sqrt {\sigma_{i}}+\sqrt{k}\leq2\sqrt{k}+2\sqrt{n}$ for all $i$. By $k\gg n$, it follows that $P({\mathcal{A}})\geq1-o(1)$, where \[ {\mathcal{A}}=\left\{ \max_{1\le i\leq n}|\sigma_{i}-k|\leq\min\{k,4\sqrt {nk}\}\right\} . \] Now we notice \begin{align*} \log\frac{|M|}{|\tilde{M}|} & =n\log|S|+\log\frac{\left| I+(c_{1}^{2} +c_{2}^{2})Z^{\top}Z\right| }{\left| I+c_{1}^{2}S_{1}Z^{\top}Z\right| }\\ & =n\log|S|+\sum_{i=1}^{n}\log\frac{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i} }{1+c_{1}^{2}S_{1}\sigma_{i}}\\ & =n\log|S|+\sum_{i=1}^{n}\log\left( 1+\frac{(c_{1}^{2}(1-S_{1})+c_{2} ^{2})\sigma_{i}}{1+c_{1}^{2}S_{1}\sigma_{i}}\right) \\ & \overset{{\text{(i)}}}{\leq}n\log|S|+\sum_{i=1}^{n}\frac{(c_{1}^{2} (1-S_{1})+c_{2}^{2})\sigma_{i}}{1+c_{1}^{2}S_{1}\sigma_{i}}\\ & \overset{{\text{(ii)}}}{=}n\log\left( \frac{1}{1-c_{1}^{2}c_{2}^{2}k^{2} }\right) +\sum_{i=1}^{n}\frac{(c_{1}^{2}(1-S_{1})+c_{2}^{2})\sigma_{i} }{1+c_{1}^{2}S_{1}\sigma_{i}}\\ & \overset{{\text{(iii)}}}{\leq}nc_{1}^{2}c_{2}^{2}k^{2}+nc_{1}^{4}c_{2} ^{4}k^{4}+\sum_{i=1}^{n}\frac{(c_{1}^{2}(1-S_{1})+c_{2}^{2})\sigma_{i} }{1+c_{1}^{2}S_{1}\sigma_{i}}, \end{align*} where (i) follows by $\log(1+x)\leq x$ for $x\geq-1$ and (ii) follows by $|S|=(1-c_{1}^{2}c_{2}^{2}k^{2})^{-1}$ and (iii) follows by the elementary inequality $\log(\frac{1}{1-x})\leq x+x^{2}$ for $x\in[0,1/2]$ and $c_{1} ^{2}c_{2}^{2}k^{2}\asymp k^{-2\xi_{1}-2\xi_{2}}=o(1)$. Observe that on the event ${\mathcal{A}}$, \begin{align*} & 2\cdot{\text{KL}}(\bar{f}_{1}(\cdot,Z),\bar{f}_{2}(\cdot,Z))\\ & =\log\frac{|M|}{|\tilde{M}|}-2n+\mathrm{trace}(M^{-1}\tilde{M})\\ & \leq nc_{1}^{2}c_{2}^{2}k^{2}+nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n} \frac{(c_{1}^{2}(1-S_{1})+c_{2}^{2})\sigma_{i}}{1+c_{1}^{2}S_{1}\sigma_{i} }-2n+n\mathrm{trace}(\Sigma)\\ & \qquad+\Vert g\Vert_{2}^{2}\mathrm{trace}(Z^{\top}Z)-\mathrm{trace} \left[ \left( \tilde{g}^{\top}\Sigma\tilde{g}Z^{\top}Z+c_{1} ^{4}Z^{\top}ZZ^{\top}Z\right) \left( I+\Vert\tilde{g}\Vert_{2} ^{2}Z^{\top}Z\right) ^{-1}\right] \\ & =nc_{1}^{2}c_{2}^{2}k^{2}+nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n} \frac{(c_{1}^{2}(1-S_{1})+c_{2}^{2})\sigma_{i}}{1+c_{1}^{2}S_{1}\sigma_{i}}\\ & \qquad+c_{1}^{2}\sum_{i=1}^{n}\sigma_{i}-\sum_{i=1}^{n}\frac{(c_{1} ^{2}+c_{2}^{2}+2c_{1}^{2}c_{2}^{2}k)\sigma_{i}+c_{1}^{4}\sigma_{i}^{2} }{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}\\ & =nc_{1}^{2}c_{2}^{2}k^{2}+nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n} \frac{(c_{1}^{2}(1-S_{1})+c_{2}^{2})\sigma_{i}}{1+c_{1}^{2}S_{1}\sigma_{i}}\\ & \qquad+\sum_{i=1}^{n}\left( \frac{c_{1}^{2}\sigma_{i}[1+(c_{1}^{2} +c_{2}^{2})\sigma_{i}]-(c_{1}^{2}+c_{2}^{2}+2c_{1}^{2}c_{2}^{2}k)\sigma _{i}-c_{1}^{4}\sigma_{i}^{2}}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}\right) \\ & =nc_{1}^{2}c_{2}^{2}k^{2}+nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}\left( \frac{(c_{1}^{2}(1-S_{1})+c_{2}^{2})\sigma_{i}}{1+c_{1}^{2}S_{1}\sigma_{i} }+\frac{c_{2}^{2}c_{1}^{2}\sigma_{i}^{2}-(c_{2}^{2}+2c_{1}^{2}c_{2} ^{2}k)\sigma_{i}}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}\right) \\ & =nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}\left( \frac{(c_{1}^{2} (1-S_{1})+c_{2}^{2})\sigma_{i}}{1+c_{1}^{2}S_{1}\sigma_{i}}+\frac{c_{2} ^{2}c_{1}^{2}\sigma_{i}^{2}-(c_{2}^{2}+2c_{1}^{2}c_{2}^{2}k)\sigma_{i} }{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}+c_{1}^{2}c_{2}^{2}k^{2}\right) \\ & =nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}\left( \frac{(c_{1}^{2} (1-S_{1})+c_{2}^{2})\sigma_{i}}{1+c_{1}^{2}S_{1}\sigma_{i}}+\frac{c_{1} ^{2}c_{2}^{2}(\sigma_{i}-k)^{2}-c_{2}^{2}\sigma_{i}+c_{1}^{2}c_{2}^{2} k^{2}(c_{1}^{2}+c_{2}^{2})\sigma_{i}}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i} }\right) \\ & =nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}\left( \frac{c_{1}^{2}c_{2} ^{2}(\sigma_{i}-k)^{2}}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}\right) +\sum _{i=1}^{n}\left( \frac{c_{1}^{2}(1-S_{1})+c_{2}^{2}}{1+c_{1}^{2}S_{1} \sigma_{i}}+\frac{-c_{2}^{2}+c_{1}^{2}c_{2}^{2}k^{2}(c_{1}^{2}+c_{2}^{2} )}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}\right) \sigma_{i}\\ & \overset{{\text{(i)}}}{=}nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}\left( \frac{c_{1}^{2}c_{2}^{2}(\sigma_{i}-k)^{2}}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i} }\right) +\sum_{i=1}^{n}\left( \frac{-c_{1}^{4}c_{2}^{2}k^{2}S_{1}+c_{2} ^{2}}{1+c_{1}^{2}S_{1}\sigma_{i}}+\frac{-c_{2}^{2}+c_{1}^{2}c_{2}^{2} k^{2}(c_{1}^{2}+c_{2}^{2})}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}\right) \sigma_{i}\\ & =nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}\left( \frac{c_{1}^{2}c_{2} ^{2}(\sigma_{i}-k)^{2}}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}\right) +\sum _{i=1}^{n}\frac{-c_{1}^{4}c_{2}^{2}k^{2}S_{1}+c_{1}^{2}c_{2}^{2}k^{2} (c_{1}^{2}+c_{2}^{2})}{\left( 1+c_{1}^{2}S_{1}\sigma_{i}\right) \left( 1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}\right) }\sigma_{i}\\ & \qquad+\sum_{i=1}^{n}\sigma_{i}^{2}\frac{(-c_{1}^{4}c_{2}^{2}k^{2} S_{1}+c_{2}^{2})(c_{1}^{2}+c_{2}^{2})-c_{2}^{2}c_{1}^{2}S_{1}+c_{1}^{4} c_{2}^{2}k^{2}(c_{1}^{2}+c_{2}^{2})S_{1}}{\left( 1+c_{1}^{2}S_{1}\sigma _{i}\right) \left( 1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}\right) }\\ & =nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}\left( \frac{c_{1}^{2}c_{2} ^{2}(\sigma_{i}-k)^{2}}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}}\right) +\sum _{i=1}^{n}\frac{c_{1}^{4}c_{2}^{2}k^{2}(1-S_{1})+c_{1}^{2}c_{2}^{4}k^{2} }{\left( 1+c_{1}^{2}S_{1}\sigma_{i}\right) \left( 1+(c_{1}^{2}+c_{2} ^{2})\sigma_{i}\right) }\sigma_{i}\\ & \qquad+c_{2}^{2}\sum_{i=1}^{n}\sigma_{i}^{2}\frac{c_{2}^{2}+c_{1} ^{2}(1-S_{1})}{\left( 1+c_{1}^{2}S_{1}\sigma_{i}\right) \left( 1+(c_{1} ^{2}+c_{2}^{2})\sigma_{i}\right) } \end{align*} \begin{align*} & \overset{{\text{(ii)}}}{=}nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}\left( \frac{c_{1}^{2}c_{2}^{2}(\sigma_{i}-k)^{2}}{1+(c_{1}^{2}+c_{2}^{2})\sigma_{i} }\right) +\sum_{i=1}^{n}\frac{-c_{1}^{6}c_{2}^{4}k^{4}S_{1}+c_{1}^{2} c_{2}^{4}k^{2}}{\left( 1+c_{1}^{2}S_{1}\sigma_{i}\right) \left( 1+(c_{1}^{2}+c_{2}^{2})\sigma_{i}\right) }\sigma_{i}\\ & \qquad+c_{2}^{4}\sum_{i=1}^{n}\sigma_{i}^{2}\frac{1-c_{1}^{4}k^{2}S_{1} }{\left( 1+c_{1}^{2}S_{1}\sigma_{i}\right) \left( 1+(c_{1}^{2}+c_{2} ^{2})\sigma_{i}\right) }\\ & \leq nc_{1}^{4}c_{2}^{4}k^{4}+\sum_{i=1}^{n}c_{1}^{2}c_{2}^{2}(\sigma _{i}-k)^{2}+c_{1}^{2}c_{2}^{4}k^{2}\sum_{i=1}^{n}\sigma_{i}+c_{2}^{4} \sum_{i=1}^{n}\sigma_{i}^{2}\\ & \overset{{\text{(iii)}}}{\leq}nc_{1}^{4}c_{2}^{4}k^{4}+16c_{1}^{2}c_{2} ^{2}n^{2}k+c_{1}^{2}c_{2}^{4}k^{2}\cdot(n\cdot2k)+c_{2}^{4}\cdot(n\cdot 4k^{2})\\ & \overset{{\text{(iv)}}}{=}O\left( n^{[1/2-3(\xi_{1}+\xi_{2})]/(\xi_{1} +\xi_{2}+1/2)}\right) +16C_{1}^{2}C_{2}^{2}C_{0}^{-(2\xi_{1}+2\xi_{2}+1)}\\ & \qquad+O\left( n^{(1/2-\xi_{1}-3\xi_{2})/(\xi_{1}+\xi_{2}+1/2)}\right) +O\left( n^{(1/2+\xi_{1}-3\xi_{2})/(\xi_{1}+\xi_{2}+1/2)}\right) \\ & \overset{{\text{(v)}}}{=}16C_{1}^{2}C_{2}^{2}C_{0}^{-(2\xi_{1}+2\xi_{2} +1)}+o(1), \end{align*} where (i) and (ii) follow by $1-S_{1}=-c_{1}^{2}c_{2}^{2}k^{2}S_{1}$ (due to $S_{1}=(1-c_{1}^{2}c_{2}^{2}k^{2})^{-1}$), (iii) follows by the definition of ${\mathcal{A}}$, (iv) follows by $c_{1}=C_{1}k^{-\xi_{1}-1/2}$, $c_{2} =C_{2}k^{-\xi_{2}-1/2}$ and $k=C_{0}n^{1/(\xi_{1}+\xi_{2}+1/2)}$ and (vii) follows by $\xi_{2}\geq1/4>\xi_{1}$. Thus, we have proved that on the event ${\mathcal{A}}$, \[ {\text{KL}}(\bar{f}_{1}(\cdot,Z),\bar{f}_{2}(\cdot,Z))\leq8C_{1}^{2}C_{2}^{2}C_{0}^{-(2\xi _{1}+2\xi_{2}+1)}+o(1). \] \textbf{Step 4:} Deriving the final result. By first Pinsker's inequality (Lemma 2.5 of Tsybakov2009), we have that on the event ${\mathcal{A}}$, \begin{equation} {\text{TV}}(\bar{f}_{1}(\cdot,Z),\bar{f}_{2}(\cdot,Z))\leq\sqrt{{\text{KL}}(\bar{f}_{1}(\cdot,Z),\bar {f}_{2}(\cdot,Z))/2}\leq2C_{1}C_{2}C_{0}^{-(\xi_{1}+\xi_{2}+1/2)}+o(1). \end{equation} By Scheff\'e's theorem (Lemma 2.1 of Tsybakov2009), \begin{equation} {\text{TV}}(\bar{f}_{1}(\cdot,Z),\bar{f}_{2}(\cdot,Z))=\frac{1}{2}\int|\bar{f}_{1}(r,Z)-\bar {f}_{2}(r,Z)|dr. \end{equation} Since $Z_{i}$ is i.i.d $N(0,I_{k})$, we have that for $j\in\{1,2\}$, $\Phi_{j,\omega}(r,z)=f_{j,\omega}(r,z)\mu_{n}(z)$ with $r=(y,d)$, where $\mu_{n}(z)=\prod_{i=1}^{n}\mu(z_{i})$ with $z=(z_{1},...,z_{n})$. This means that \[ \bar{\phi}_{j}(r,z)=\int_{{\mathbb{R}}^{k}}\Phi_{j,\omega}(r,z)\mu (\omega)d\omega=\int_{{\mathbb{R}}^{k}}f_{j,\omega}(r,z)\mu_{n} (z)\mu(\omega)d\omega=\bar{f}_{j}(r,z)\mu_{n}(z). \] In the following argument, $E$ denotes expectation with respect to the randomness in $Z$ given by $Z_{i}\sim i.i.d.\ N(0,I_{k})$. Then \begin{align*} {\text{TV}}(\bar{\phi}_{1},\bar{\phi}_{2}) & =\frac{1}{2}\int\mu_{n} (z)|\bar{f}_{1}(r,z)-\bar{f}_{2}(r,z)|drdz\\ & =\frac{1}{2}\mathbb{E}\int|\bar{f}_{1}(r,Z)-\bar{f}_{2}(r,Z)|dr\\ & =\frac{1}{2}\mathbb{E}\int|\bar{f}_{1}(r,Z)-\bar{f}_{2}(r,Z)|dr\cdot{\mathbf{1} }\{Z\in{\mathcal{A}}\}+\frac{1}{2}\mathbb{E}\int|\bar{f}_{1}(r,Z)-\bar{f} _{2}(r,Z)|dr\cdot{\mathbf{1}}\{Z\notin{\mathcal{A}}\}\\ & \overset{{\text{(i)}}}{\leq}2C_{1}C_{2}C_{0}^{-(\xi_{1}+\xi_{2} +1/2)}+o(1)+\frac{1}{2}\mathbb{E}\int|\bar{f}_{1}(r,Z)-\bar{f}_{2}(r,Z)|dr\cdot {\mathbf{1}}\{Z\notin{\mathcal{A}}\}\\ & \overset{{\text{(ii)}}}{\leq}2C_{1}C_{2}C_{0}^{-(\xi_{1}+\xi_{2} +1/2)}+o(1)+P(Z\notin{\mathcal{A}}), \end{align*} where (i) follows by ((ref)) and ((ref)) and (ii) follows by the fact that $\int\bar{f}_{j}(r,Z)dr=1$ almost surely. Since $P({\mathcal{A}})=1-o(1)$, we have ${\text{TV}}(\bar{\phi}_{1},\bar{\phi} _{2})\leq2C_{1}C_{2}C_{0}^{-(\xi_{1}+\xi_{2}+1/2)}+o(1)$. \end{proof} \begin{lemma} Assume that $p\gtrsim n^{2}$. If $\xi _{1}+\xi_{2}<1/2$ and $\xi_{2}\geq1/4>\xi_{1}$, then $$\mathcal{R}^{\rm{Ordered}}_{\xi_{1},\xi_{2}}\gtrsim n^{-(\xi_{1}+\xi_{2})/(\xi_{1}+\xi _{2}+1/2)}.$$ \end{lemma} \begin{proof} [Proof of Lemma (ref)] Define $$k=\left\lfloor C_{0}n^{1/(\xi_{1}+\xi_{2} +1/2)}\right\rfloor, \ c_{1}=C_{1}k^{-\xi_{1}-1/2} \mbox{ and } c_{2}=C_{2} k^{-\xi_{2}-1/2},$$ where $C_{0},C_{1},C_{2}>0$ are constants to be determined. Since $\xi_{1}+\xi_{2}<1/2$, we have $k\gg n$. Since $p\gtrsim n^{2}$ and $k\asymp n^{1/(\xi_{1}+\xi_{2}+1/2)}\ll n^{2}$, we have $p\gg k$. In the following analysis, we set $\beta_{j}=\phi_{j}=0$ for $j>k$. This effectively reduces the dimension from $p$ to $k$ in our analysis. For any $\omega\in{\mathbb{R}}^{k}$, let $$\lambda_{\omega}=(c_{1}c_{2} k,c_{1}\omega,0,1-c_{1}^{2}c_{2}^{2}k^{2},1), \mbox{ and } \tilde{\lambda}_{\omega }=(0,c_{1}\omega,c_{2}\omega,1,1).$$ Notice that $1-c_{1}^{2}c_{2}^{2}k^{2}$ is well defined as the value of a variance (i.e., $1-c_{1}^{2}c_{2}^{2} k^{2}>0$) if we choose $C_{0},C_{1},C_{2}>0$ small enough; to see this notice that $c_{1}c_{2}k\asymp k^{-(\xi_{1}+\xi_{2})}=o(1)$. We consider two probability measures: $P_{1}=\int P_{\lambda_{\omega}} \mu(\omega)d\omega$ and $P_{2}=\int P_{\tilde{\lambda}_{\omega}}\mu (\omega)d\omega$, where $\mu$ is the density of $N(0,I_{k})$, i.e., $\mu(\omega)=(2\pi)^{-k/2} \operatorname{exp}(-\|\omega\|_{2}^{2}/2)$. Under $P_{\lambda_{\omega}}$, $Z_{i}\sim N(0,I_{k})$ and \[ \begin{pmatrix} Y_{i}\\ D_{i} \end{pmatrix} = \begin{pmatrix} -D_{i}c_{1}c_{2}k+c_{1}Z_{i}^{\top}\omega+\varepsilon_{i}\\ D_{i} \end{pmatrix} = \begin{pmatrix} c_{1}Z_{i}^{\top}\omega+c_{1}c_{2}ku_{i}+\varepsilon_{i}\\ u_{i} \end{pmatrix} \] with $\mathbb{E} u_{i}^{2}=1$ and $\mathbb{E} v_{i}^{2}=1-c_{1}^{2}c_{2}^{2}k^{2}$. Therefore, under $P_{\lambda_{\omega}}$, \[ \begin{pmatrix} Y_{i}\\ D_{i} \end{pmatrix} \mid Z\sim N\left( \begin{pmatrix} c_{1}Z_{i}^{\top}\omega\\ 0 \end{pmatrix} ,\ \Sigma\right) \qquad{\text{and}}\qquad\Sigma= \begin{pmatrix} 1 & c_{1}c_{2}k\\ c_{1}c_{2}k & 1 \end{pmatrix} . \] Similarly, we observe that under $P_{\tilde{\lambda}_{\omega}}$, $Z_{i}\sim N(0,I_{k})$ and \[ \begin{pmatrix} Y_{i}\\ D_{i} \end{pmatrix} = \begin{pmatrix} c_{1}Z_{i}^{\top}\omega+\varepsilon_{i}\\ c_{2}Z_{i}^{\top}\omega+u_{i} \end{pmatrix} , \] which means that \[ \begin{pmatrix} Y_{i}\\ D_{i} \end{pmatrix} \mid Z\sim N\left( \begin{pmatrix} c_{1}Z_{i}^{\top}\omega\\ c_{2}Z_{i}^{\top}\omega \end{pmatrix} ,\ I_{2}\right) . \] By Lemma (ref), we have ${\text{TV}}(P_{1},P_{2})\leq2C_{1}C_{2}C_{0}^{-(\xi _{1}+\xi_{2}+1/2)}+o(1)$. We can choose constants $C_{1},C_{2},C_{0}>0$ small enough such that $C_{1}C_{2}C_{0}^{-(\xi_{1}+\xi_{2}+1/2)}\leq\alpha/4$. Thus, \begin{equation} {\text{TV}}(P_{1},P_{2})\leq\alpha/2+o(1). \end{equation} Let $\psi$ be a test for \[ H_{0}:\ \lambda\in\Lambda_{0}\qquad{\text{versus}}\qquad H_{1}:\ \lambda\in \Lambda_{1}, \] where $$\Lambda_{1}=\{\lambda=(\beta,\phi,\pi,\sigma_{\varepsilon}^{2},\sigma_{u}^{2} )\in\Lambda^{\rm{Ordered}}(\xi_{1},\xi_{2}):\ |\beta|\geq C_{3}n^{-(\xi_{1}+\xi_{2})/(\xi _{1}+\xi_{2}+1/2)}\}$$ and $$\Lambda_{0}=\{\lambda=(\theta,\beta,\phi,\sigma_{\varepsilon} ^{2},\sigma_{u}^{2})\in\Lambda^{\rm{Ordered}}(\xi_{1},\xi_{2}):\ \theta=0\}$$ with $C_{3} =C_{1}C_{2}C_{0}^{-(\xi_{1}+\xi_{2})}/2$. Let $\eta(\lambda)=\mathbb{E}_{\lambda}(\psi)$ be the power function. By definition of $\psi$, we have $\sup_{\lambda\in \Lambda_{0}}\eta(\lambda)\leq\alpha$. By Lemma (ref), we can choose small enough constants $C_{1},C_{2}>0$ such that \begin{equation} \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\lambda_{\omega}\in\Lambda_{0} \}\mu(\omega)d\omega=1-o(1). \end{equation} and \begin{equation} \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda}_{\omega}\in\Lambda _{1}\}\mu(\omega)d\omega=1-o(1). \end{equation} Then \begin{align*} \left\vert \int_{{\mathbb{R}}^{k}}\eta(\tilde{\lambda}_{\omega})\mu (\omega)d\omega-\int_{{\mathbb{R}}^{k}}\eta(\lambda_{\omega})\mu (\omega)d\omega\right\vert & =\left\vert \int\left( \int\psi dP_{\lambda_{\omega}}\right) \mu(\omega)d\omega-\int\left( \int\psi dP_{\tilde{\lambda}_{\omega}}\right) \mu(\omega)d\omega\right\vert \\ & =\left\vert \int\psi dP_{1}-\int\psi dP_{2}\right\vert \overset{{\text{(i)} }}{\leq}2\cdot{\text{TV}}(P_{1},P_{2})\overset{{\text{(ii)}}}{\leq} \alpha+o_{P}(1), \end{align*} where (i) follows by $|\psi|\leq1$ and Scheff\'e's theorem (Lemma 2.1 of Tsybakov2009) and (ii) follows by ((ref)). Since the power function is bounded by 1, ((ref)) and ((ref)) imply \begin{align*} & \left\vert \int_{{\mathbb{R}}^{k}}\eta(\lambda_{\omega})\mu(\omega )d\omega-\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\lambda_{\omega}\in\Lambda _{0}\}\eta(\lambda_{\omega})\mu(\omega)d\omega\right\vert \\ & =\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\lambda_{\omega}\notin\Lambda _{0}\}\eta(\lambda_{\omega})\mu(\omega)d\omega\leq\int_{{\mathbb{R}}^{k} }{\mathbf{1}}\{\lambda_{\omega}\notin\Lambda_{0}\}\mu(\omega)d\omega=o(1) \end{align*} and similarly \[ \left\vert \int_{{\mathbb{R}}^{k}}\eta(\tilde{\lambda}_{\omega})\mu (\omega)d\omega-\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda} _{\omega}\in\Lambda_{1}\}\eta(\tilde{\lambda}_{\omega})\mu(\omega )d\omega\right\vert =o(1). \] By the above three displays, we have \[ \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda}_{\omega}\in\Lambda _{1}\}\eta(\tilde{\lambda}_{\omega})\mu(\omega)d\omega\leq\alpha +\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\lambda_{\omega}\in\Lambda_{0} \}\eta(\lambda_{\omega})\mu(\omega)d\omega+o(1)\overset{{\text{(i)}}}{\leq }2\alpha+o(1), \] where (i) follows by $\sup_{\lambda\in\Lambda_{0}}\eta(\lambda)\leq\alpha$. On the other hand, by ((ref)), \begin{align*} & \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda}_{\omega}\in\Lambda _{1}\}\eta(\tilde{\lambda}_{\omega})\mu(\omega)d\omega \\ &\geq\left( \operatorname{inf}_{\lambda\in\Lambda_{1}}\eta(\lambda)\right) \times \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda}_{\omega}\in\Lambda _{1}\}\mu(\omega)d\omega=\left( \operatorname{inf}_{\lambda\in\Lambda_{1} }\eta(\lambda)\right) \times(1-o(1)). \end{align*} Combining the above two displays, we have that \[ \operatorname{inf}_{\lambda\in\Lambda_{1}}\eta(\lambda)\leq\frac{2\alpha +o(1)}{1-o(1)}=2\alpha+o(1). \] Therefore, for testing $H_{0}:\ \tau(\lambda)=0$ versus $H_{1}:\ |\tau (\lambda)|\geq C_{3}n^{-(\xi_{1}+\xi_{2})/(\xi_{1}+\xi_{2}+1/2)}$, any test of size $\alpha$ can has the worst-case power bounded by $2\alpha+o(1)$. The desired result follows. \end{proof} \begin{lemma} \textit{ Let }$C_{0},C_{1},C_{2},\xi_{1},\xi_{2}>0$\textit{ be constants. Define }$k=\left\lfloor C_{0}n^{1/(\xi_{1}+\xi_{2} +1/2)}\right\rfloor $\textit{, }$c_{1}=C_{1}k^{-\xi_{1}-1/2}$\textit{ and }$c_{2}=C_{2}k^{-\xi_{2}-1/2}$\textit{, where }$\left\lfloor \cdot \right\rfloor $\textit{ denotes the integer part. For any }$\omega\in \mathbb{R}^{k} $\textit{, let }$\Phi_{1,\omega}$\textit{ denote the distribution of i.i.d }$\{(Y_{i},D_{i},Z_{i})\}_{i=1}^{n}$\textit{ given by }$Z_{i}\sim N(0,I_{k})$ \[ \begin{pmatrix} D_{i}\\ Y_{i} \end{pmatrix} \mid Z\sim N\left( \begin{pmatrix} c_{2}Z_{i}^{\top}\omega\\ 0 \end{pmatrix} ,\ \Sigma\right) \qquad{\text{and}}\qquad\Sigma= \begin{pmatrix} 1 & c_{1}c_{2}k\\ c_{1}c_{2}k & 1 \end{pmatrix} . \] \textit{Let }$\Phi_{2,\omega}$\textit{ denote the distribution of i.i.d }$\{(Y_{i},D_{i},Z_{i})\}_{i=1}^{n}$\textit{ given by }$Z_{i}\sim N(0,I_{k} )$, \[ \begin{pmatrix} D_{i}\\ Y_{i} \end{pmatrix} \mid Z\sim N\left( \begin{pmatrix} c_{2}Z_{i}^{\top}\omega\\ c_{1}Z_{i}^{\top}\omega \end{pmatrix} ,\ \begin{pmatrix} 1 & 0\\ 0 & 1 \end{pmatrix} \right) . \] \textit{Assume that }$\xi_{1}\geq1/4>\xi_{2}$\textit{ and }$\xi_{1}+\xi _{2}<1/2$\textit{. Let }$Z_{i}$\textit{ be i.i.d from }$N(0,I_{k})$\textit{. Then} \[ {\text{TV}}(\bar{\phi}_{1},\bar{\phi}_{2})\leq2C_{1}C_{2}C_{0}^{-(\xi_{1} +\xi_{2}+1/2)}+o(1), \] \textit{where }$TV$\textit{ denotes the total variation distance, }$\bar{\phi }_{1}=\int_{{\mathbb{R}}^{k}}\Phi_{1,\omega}\mu(\omega)d\omega$\textit{, }$\bar{\phi}_{2}=\int_{{\mathbb{R}}^{k}}\Phi_{2,\omega}\mu(\omega)d\omega $\textit{ and }$\mu$\textit{ is the density function of }$N(0,I_{k})$\textit{, i.e., }$\mu(\omega)=(2\pi)^{-k/2}\operatorname{exp}(-\Vert\omega\Vert _{2}^{2}/2)$. \end{lemma} \begin{proof} [Proof of Lemma (ref)] The proof is analogous to that of Lemma (ref). We simply swapped the role of $c_{1}$ and $c_{2}$ as well as the role of $Y_{i}$ and $D_{i}$. \end{proof} \begin{lemma} \textit{Assume that }$p\gtrsim n^{2}$\textit{. If }$\xi _{1}+\xi_{2}<1/2$\textit{ and }$\xi_{1}\geq1/4>\xi_{2}$\textit{, then } $$\mathcal{R}^{\rm{Ordered}}_{\xi_{1},\xi_{2}}\gtrsim n^{-(\xi_{1}+\xi_{2})/(\xi_{1}+\xi_{2}+1/2)}.$$ \end{lemma} \begin{proof} [Proof of Lemma (ref)] The argument is analogous to that of Lemma (ref) with $\xi_{1}$ and $\xi_{2}$ swapped. Define $$k=C_{0}n^{1/(\xi_{1}+\xi_{2}+1/2)}, \qquad c_{1} =C_{1}k^{-\xi_{1}-1/2}, \mbox{ and } c_{2}=C_{2}k^{-\xi_{2}-1/2},$$ where $C_{0} ,C_{1},C_{2}>0$ are constants to be determined. Since $\xi_{1}+\xi_{2}<1/2$, we have $k\gg n$. Since $p\gtrsim n^{2}$ and $k\asymp n^{1/(\xi_{1}+\xi _{2}+1/2)}\ll n^{2}$, we have $p\gg k$. In the following analysis, we set $\beta_{j}=\phi_{j}=0$ for $j>k$. This effectively reduces the dimension from $p$ to $k$ in our analysis. For any $\omega\in{\mathbb{R}}^{k}$, let $$\lambda_{\omega}=(c_{1} c_{2}k,-c_{1}c_{2}^{2}k\omega,c_{2}\omega,1-c_{1}^{2}c_{2}^{2}k^{2},1) \mbox{ and } \tilde{\lambda}_{\omega}=(0,c_{1}\omega,c_{2}\omega,1,1).$$ Notice that $1-c_{1}^{2}c_{2}^{2}k^{2}$ is well defined as the value of a variance (i.e., $1-c_{1}^{2}c_{2}^{2}k^{2}>0$) if we choose $C_{0},C_{1},C_{2}>0$ small enough; to see this notice that $c_{1}c_{2}k\asymp k^{-(\xi_{1}+\xi_{2} )}=o(1)$. We consider two probability measures: $$P_{1}=\int P_{\lambda_{\omega}} \mu(\omega)d\omega \mbox{ and } P_{2}=\int P_{\tilde{\lambda}_{\omega}}\mu (\omega)d\omega,$$ where $\mu$ is the density of $N(0,I_{k})$, i.e., $\mu(\omega)=(2\pi)^{-k/2} \operatorname{exp}(-\|\omega\|_{2}^{2}/2)$. Under $P_{\lambda_{\omega}}$, $Z_{i}\sim N(0,I_{k})$ and \[ \begin{pmatrix} D_{i}\\ Y_{i} \end{pmatrix} = \begin{pmatrix} D_{i}\\ D_{i}c_{1}c_{2}k-c_{1}c_{2}^{2}kZ_{i}^{\top}\omega+\varepsilon_{i} \end{pmatrix} = \begin{pmatrix} c_{2}Z_{i}^{\top}\omega+u_{i}\\ c_{1}c_{2}ku_{i}+\varepsilon_{i} \end{pmatrix} \] with $Eu_{i}^{2}=1$ and $Ev_{i}^{2}=1-c_{1}^{2}c_{2}^{2}k^{2}$. Therefore, under $P_{\lambda_{\omega}}$, \[ \begin{pmatrix} D_{i}\\ Y_{i} \end{pmatrix} \mid Z\sim N\left( \begin{pmatrix} c_{2}Z_{i}^{\top}\omega\\ 0 \end{pmatrix} ,\ \Sigma\right) \qquad{\text{and}}\qquad\Sigma= \begin{pmatrix} 1 & c_{1}c_{2}k\\ c_{1}c_{2}k & 1 \end{pmatrix} . \] Similarly, we observe that under $P_{\tilde{\lambda}_{\omega}}$, $Z_{i}\sim N(0,I_{k})$ and \[ \begin{pmatrix} D_{i}\\ Y_{i} \end{pmatrix} = \begin{pmatrix} c_{2}Z_{i}^{\top}\omega+u_{i}\\ c_{1}Z_{i}^{\top}\omega+\varepsilon_{i} \end{pmatrix} , \] which means that \[ \begin{pmatrix} D_{i}\\ Y_{i} \end{pmatrix} \mid Z\sim N\left( \begin{pmatrix} c_{2}Z_{i}^{\top}\omega\\ c_{1}Z_{i}^{\top}\omega \end{pmatrix} ,\ I_{2}\right) . \] By Lemma (ref), we have ${\text{TV}}(P_{1},P_{2})\leq2C_{1}C_{2}C_{0}^{-(\xi _{1}+\xi_{2}+1/2)}+o(1)$. We can choose constants $C_{1},C_{2},C_{0}>0$ small enough such that $C_{1}C_{2}C_{0}^{-(\xi_{1}+\xi_{2}+1/2)}\leq\alpha/4$. Thus, \begin{equation} {\text{TV}}(P_{1},P_{2})\leq\alpha/2+o(1). \end{equation} Let $\psi$ be a test for \[ H_{0}:\ \lambda\in\Lambda_{0}\qquad{\text{versus}}\qquad H_{1}:\ \lambda\in \Lambda_{1}, \] where $$\Lambda_{1}=\{\lambda=(\theta,\beta,\phi,\sigma_{\varepsilon}^{2},\sigma_{u}^{2} )\in\Lambda^{\rm{Ordered}}(\xi_{1},\xi_{2}):\ |\theta|\geq C_{3}n^{-(\xi_{1}+\xi_{2})/(\xi _{1}+\xi_{2}+1/2)}\}$$ and $$\Lambda_{0}=\{\lambda=(\theta,\beta,\phi,\sigma_{\varepsilon} ^{2},\sigma_{u}^{2})\in\Lambda^{\rm{Ordered}}(\xi_{1},\xi_{2}):\ \theta=0\}$$ with $C_{3} =C_{1}C_{2}C_{0}^{-(\xi_{1}+\xi_{2})}/2$. Let $\eta(\lambda)=\mathbb{E}_{\lambda}(\psi)$ be the power function. By definition of $\psi$, we have $\sup_{\lambda\in \Lambda_{0}}\eta(\lambda)\leq\alpha$. By Lemma (ref), we can choose small enough constants $C_{1},C_{2}>0$ such that \begin{equation} \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\lambda_{\omega}\in\Lambda_{0} \}\mu(\omega)d\omega=1-o(1). \end{equation} and \begin{equation} \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda}_{\omega}\in\Lambda _{1}\}\mu(\omega)d\omega=1-o(1). \end{equation} Then \begin{align*} \left\vert \int_{{\mathbb{R}}^{k}}\eta(\tilde{\lambda}_{\omega})\mu (\omega)d\omega-\int_{{\mathbb{R}}^{k}}\eta(\lambda_{\omega})\mu (\omega)d\omega\right\vert & =\left\vert \int\left( \int\psi dP_{\lambda_{\omega}}\right) \mu(\omega)d\omega-\int\left( \int\psi dP_{\tilde{\lambda}_{\omega}}\right) \mu(\omega)d\omega\right\vert \\ & =\left\vert \int\psi dP_{1}-\int\psi dP_{2}\right\vert \overset{{\text{(i)} }}{\leq}2\cdot{\text{TV}}(P_{1},P_{2})\overset{{\text{(ii)}}}{\leq} \alpha+o_{P}(1), \end{align*} where (i) follows by $|\psi|\leq1$ and Scheff\'e's theorem (Lemma 2.1 of Tsybakov2009) and (ii) follows by ((ref)). Since the power function is bounded by 1, ((ref)) and ((ref)) imply \begin{align*} & \left\vert \int_{{\mathbb{R}}^{k}}\eta(\lambda_{\omega})\mu(\omega )d\omega-\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\lambda_{\omega}\in\Lambda _{0}\}\eta(\lambda_{\omega})\mu(\omega)d\omega\right\vert \\ & =\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\lambda_{\omega}\notin\Lambda _{0}\}\eta(\lambda_{\omega})\mu(\omega)d\omega\leq\int_{{\mathbb{R}}^{k} }{\mathbf{1}}\{\lambda_{\omega}\notin\Lambda_{0}\}\mu(\omega)d\omega=o(1) \end{align*} and similarly \[ \left\vert \int_{{\mathbb{R}}^{k}}\eta(\tilde{\lambda}_{\omega})\mu (\omega)d\omega-\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda} _{\omega}\in\Lambda_{1}\}\eta(\tilde{\lambda}_{\omega})\mu(\omega )d\omega\right\vert =o(1). \] By the above three displays, we have \[ \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda}_{\omega}\in\Lambda _{1}\}\eta(\tilde{\lambda}_{\omega})\mu(\omega)d\omega\leq\alpha +\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\lambda_{\omega}\in\Lambda_{0} \}\eta(\lambda_{\omega})\mu(\omega)d\omega+o(1)\overset{{\text{(i)}}}{\leq }2\alpha+o(1), \] where (i) follows by $\sup_{\lambda\in\Lambda_{0}}\eta(\lambda)\leq\alpha$. On the other hand, by ((ref)), \begin{align*} &\int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda}_{\omega}\in\Lambda _{1}\}\eta(\tilde{\lambda}_{\omega})\mu(\omega)d\omega \\ &\geq\left( \operatorname{inf}_{\lambda\in\Lambda_{1}}\eta(\lambda)\right) \times \int_{{\mathbb{R}}^{k}}{\mathbf{1}}\{\tilde{\lambda}_{\omega}\in\Lambda _{1}\}\mu(\omega)d\omega=\left( \operatorname{inf}_{\lambda\in\Lambda_{1} }\eta(\lambda)\right) \times(1-o(1)). \end{align*} Combining the above two displays, we have that \[ \operatorname{inf}_{\lambda\in\Lambda_{1}}\eta(\lambda)\leq\frac{2\alpha +o(1)}{1-o(1)}=2\alpha+o(1). \] Therefore, for testing $H_{0}:\ \tau(\lambda)=0$ versus $H_{1}:\ |\tau (\lambda)|\geq C_{3}n^{-(\xi_{1}+\xi_{2})/(\xi_{1}+\xi_{2}+1/2)}$, any test of size $\alpha$ can has the worst-case power bounded by $2\alpha+o(1)$. The desired result follows. \end{proof} \begin{proof}[Proof of Theorem (ref)] We consider two cases: (1) $\max\{\xi_{1},\xi _{2}\}\geq1/4$ and (2) $\max\{\xi_{1},\xi_{2}\}<1/4$. We first consider the case of $\max\{\xi_{1},\xi_{2}\}\geq1/4$, which means that either $\xi_1\geq 1/4>\xi_2$ or $\xi_2\geq 1/4>\xi_1$ (due to $\xi_1+\xi_2<1/2$). Then by Lemmas (ref) and (ref), we have proved that $\mathcal{R}^{\rm{Ordered}}_{\xi_{1},\xi_{2}}\gtrsim n^{-(\xi_{1}+\xi_{2})/(\xi_{1}+\xi_{2}+1/2)} \gg n^{-1/2}$. Thus the result follows for the case of $\max\{\xi_{1},\xi_{2}\}\geq1/4$. Now we consider the case of $\max\{\xi_{1},\xi_{2}\}<1/4$. Without loss of generality, we assume $\xi_{1}\geq\xi_{2}$. Then we have $1/4>\xi_{1}\geq \xi_{2}$. Now we set $\tilde{\xi}_{1}=1/4$ and $\tilde{\xi}_{2}=\xi_{2}$. Thus, we have $\tilde{\xi}_{1}+\tilde{\xi}_{2}=1/4+\xi_{2}<1/4+1/4=1/2$ and $\max\{\tilde{\xi}_{1},\tilde{\xi}_{2}\}=1/4$. The results for the case of $\max\{\xi_{1},\xi_{2}\}\geq1/4$ implies $\mathcal{R}^{\rm{Ordered}}_{\tilde{\xi}_{1},\tilde{\xi}_{2}}\gtrsim n^{-(\tilde{\xi}_{1}+\tilde{\xi}_{2})/(\tilde{\xi }_{1}+\tilde{\xi}_{2}+1/2)}\gg n^{-1/2}$. By $\tilde{\xi}_{1}>\xi_{1}$, $\Lambda^{\rm{Ordered}}(\tilde{\xi}_{1},\tilde{\xi}_{2})\subset\Lambda^{\rm{Ordered}}(\xi_{1},\xi_{2})$, which means that $\mathcal{R}^{\rm{Ordered}}_{\xi_{1},\xi_{2}}\geq \mathcal{R}^{\rm{Ordered}}_{\tilde{\xi}_{1},\tilde{\xi}_{2}}$. Hence, $\mathcal{R}^{\rm{Ordered}}_{\xi_{1},\xi_{2}}\gg n^{-1/2}$. \end{proof} \section{Proof of Theorem (ref)} For the proof of Theorem (ref), recall $Z_{i}=(\psi_{1}(\tilde{Z}_{i}),...,\psi_{p}(\tilde{Z}_{i}))^{\top}\in\mathbb{R}^{p}$ and $X_{i}=(D_{i},Z_{i}^{\top})^{\top}\in\mathbb{R}^{p+1}$. Let $\tilde{X}_{i}=(D_{i},\tilde{Z}_{i}^{\top})^{\top}$. Define $\alpha_{0}(\tilde{X}_{i})=\sigma_{u}^{-2}D_{i}-\sigma_{u}^{-2}g(\tilde{Z}_{i})$, where $\sigma_{u}^{2}=\mathbb{E}(u_{i}^{2})$. Then \[ \mathbb{E}\alpha_{0}(\tilde{X}_{i})X_{i}=\begin{pmatrix}\mathbb{E}\alpha_{0}(\tilde{X}_{i})D_{i}\\ \mathbb{E}\alpha_{0}(\tilde{X}_{i})u_{i} \end{pmatrix}=e_{1}. \] Since $g\in\mathcal{M}_{C_{0},\xi_{2}}$, we have that $\alpha_{0}\in\mathcal{M}_{C,\xi_{2}}$ for some $C>0$. Define \begin{equation} \varepsilon_{n}=\sqrt{\log(p)/n} \qquad \mbox{and} \qquad s_{0}\geq C\varepsilon_{n}^{-2/(2\xi_{2}+1)}, \end{equation} and $\pi$ be coefficients of the least squares projection of $\alpha_{0}(\tilde{X}_{i})$ on $X_{i}$, satisfying \[ e_{1}-\Sigma\pi=\mathbb{E}[X_{i}\{\alpha_{0}(\tilde{X_{i}})-X_{i}^{\top}\pi\}]=0. \] Since $\alpha_{0}(\cdot)\in\mathcal{M}_{C,\xi_{2}}$ for some $C,\xi_{2}>0$, we have $\mathbb{E}(\alpha_{0}(\tilde{X}_{i})-X_{i}^{\top}\pi)^{2}\leq(Cp^{-\xi_{2}})^{2}$. Moreover, $\alpha_{0}(\cdot)\in\mathcal{M}_{C,\xi_{2}}$ also implies that $\mathbb{E}(\alpha_{0}(\tilde{X}_{i})-X_{i}^{\top}\tilde{\pi})^{2}\leq(Cs_{0}^{-\xi_{2}})^{2}$ for some $\tilde{\pi}\in\mathbb{R}^{p}$ such that for some $J_{0}\subset\{1,...,p\}$ with $|J_{0}|=s_{0}$ and $\tilde{\pi}_{j}=0$ for $j\notin J_{0}$. Therefore, $$\mathbb{E}(X_{i}^{\top}\pi-X_{i}^{\top}\tilde{\pi})^{2}\leq2(Cp^{-\xi_{2}})^{2}+2(Cs^{-\xi_{2}})^{2}\leq4(Cs^{-\xi_{2}})^{2}\lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}.$$ Thus, \begin{equation} (\pi-\tilde{\pi})^{\top}\Sigma(\pi-\tilde{\pi})\leq C\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}. \end{equation} Since the eigenvalues of $\Sigma$ are bounded, we have \[ \left\Vert \pi-\tilde{\pi}\right\Vert _{2}\leq C\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}. \] We define $\pi_{\ast}$ as \begin{equation} \pi_{\ast}\in\arg\min_{v}\;(\pi-v)^{\top}\Sigma(\pi-v)+2\varepsilon_{n}\sum_{j\in J_{0}^{c}}|v_{j}|\text{.} \end{equation} \textcolor{black}{Let $J$ be the vector of indices of nonzero elements of $\pi_{\ast}.$ Define $\gamma=(\theta,\beta^{\top})^{\top}$, where $\beta$ is defined as in the main text. We notice that \[ \mathbb{E}Y_{i}X_{i}=\mathbb{E}(X_{i}^{\top}\gamma+\eta_{f,i}+\varepsilon_{i})X_{i}=\Sigma\gamma+\mathbb{E}\eta_{f,i}X_{i}=\Sigma\gamma+\begin{pmatrix}\mathbb{E}\eta_{f,i}D_{i}\\ \mathbb{E}\eta_{f,i}Z_{i} \end{pmatrix}=\Sigma\gamma+\begin{pmatrix}\mathbb{E}\eta_{f,i}\eta_{g,i}\\ 0 \end{pmatrix}, \] where (i) follows by $\mathbb{E}\eta_{f,i}Z_{i}=0$ (due to the definition of $\beta$) and $\mathbb{E}\eta_{f,i}D_{i}=\mathbb{E}\eta_{f,i}u_{i}+\mathbb{E}\eta_{f,i}Z_{i}^{\top}\phi+\mathbb{E}\eta_{f,i}\eta_{g,i}=\mathbb{E}\eta_{f,i}\eta_{g,i}$ (due to the definition of $\beta$, which implies $\mathbb{E}\eta_{f,i}Z_{i}=0$). Then \begin{equation} \mathbb{E}Y_{i}X_{i}=\Sigma\gamma+e_{1}\varsigma, \end{equation} where $\varsigma=\mathbb{E}\eta_{f,i}\eta_{g,i}$ and $e_{1}\in\mathbb{R}^{p+1}$ is the first column of the $(p+1)\times(p+1)$ identity matrix. Since $p\gg\max\{n^{1/(2\xi_{1})},n^{1/(2\xi_{2})}\}$, we have \begin{equation} |\varsigma|\leq\sqrt{\mathbb{E}\eta_{f,i}^{2}}\cdot\sqrt{\mathbb{E}\eta_{g,i}^{2}}=\sqrt{O(p^{-2\xi_{1}})}\cdot\sqrt{O(p^{-2\xi_{2}})}=o(n^{-1/2})=o(\varepsilon_{n}). \end{equation} We will use the approximation $g(\tilde{Z}_{i})=Z_{i}^{\top}\phi+\eta_{g,i}$. } We first prove some preliminary results (Lemmas (ref)-(ref)) and then present the proof of Theorem (ref). \begin{lemma} Let Assumption (ref) hold. Then the following hold:\\ (1) $\|\Sigma(\pi_{*}-\pi)\|_{\infty}\leq\varepsilon_{n}$, $(\pi-\pi_{*})^{\top}\Sigma(\pi-\pi_{*})\lesssim\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}$ and $|J|\lesssim\varepsilon_{n}^{-2/(2\xi_{2}+1)}$.\\ (2) If $\xi_{2}>1/2$, then $\|\pi_{*}-\pi\|_{1}\lesssim\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}$. \end{lemma} \begin{proof}[Proof of Lemma (ref)] This follows by the same argument as in Lemmas (ref), (ref), (ref) and (ref). \end{proof} \begin{lemma} Let Assumption (ref) hold. Then with probability at least $1-o(1)$, we have $\|\hat{\Sigma}_{(1)}\pi_{*}-\Sigma\pi_{*}\|_{\infty}\leq C_{1}\varepsilon_{n}$, $\|\mathbb{E}_{n,2}Z_{i}Y_{i}-\mathbb{E}Z_{i}Y_{i}\|_{\infty}\leq C_{1}\varepsilon_{n}$, $\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}Z_{i}D_{i}\|_{\infty}\leq C_{1}\varepsilon_{n}$ and $\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}_{n,1}Z_{i}D_{i}\|_{\infty}\leq C_{1}\varepsilon_{n}$ for some constant $C_{1}>0$. \end{lemma} \begin{proof}[Proof of Lemma (ref)] Notice that \[ \sqrt{\mathbb{E}(X_{i}^{\top}\pi_{*})^{2}}=\|\Sigma^{1/2}\pi_{*}\|_{2}\leq\|\Sigma^{1/2}\pi\|_{2}+\|\Sigma^{1/2}(\pi-\pi_{*})\|_{2}. \] Since \begin{equation} (\pi-\pi_{*})^{\top}\Sigma(\pi-\pi_{*})\lesssim\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)} \end{equation} (due to Lemma (ref)), we have that $\|\Sigma^{1/2}(\pi-\pi_{*})\|_{2}\lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}=o(1)$ (by $\xi_{2}>0$). \textcolor{black}{By assumption, $\mathbb{E}|Y_{i}|^{2+\kappa}$ is bounded and entries of $Z_{i}$ are sub-Gaussian. Then Lemma (ref) yields the bound for $\|\mathbb{E}_{n,2}Z_{i}Y_{i}-\mathbb{E}Z_{i}Y_{i}\|_{\infty}$.} Similarly, since entries of $Z_{i}$ are sub-Gaussian and $D_{i}$ is sub-Gaussian, the same argument yields the bound for $\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}Z_{i}D_{i}\|_{\infty}$ and $\|\mathbb{E}_{n,1}Z_{i}D_{i}-\mathbb{E}_{n,1}Z_{i}D_{i}\|_{\infty}$. Hence, $\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}_{n,1}Z_{i}D_{i}\|_{\infty}\lesssim\varepsilon_{n}$ with probability $1-o(1)$. \textcolor{black}{In the case of sub-Gaussian $X_{i}$, $X_{i}^{\top}\pi_{*}$ is sub-Gaussian and thus has bounded moments. Therefore, Lemma (ref) implies that \[ \left\Vert \mathbb{E}_{n,1}X_{i}X_{i}^{\top}\pi_{*}-\mathbb{E} X_{i}X_{i}^{\top} \pi_{*}\right\Vert _{\infty}\lesssim\varepsilon_{n} \] with probability $1-o(1)$. It remains to show the above for the case of bounded $X_{i}$. } \textcolor{black}{Recall that $\alpha_{0}(\tilde{X}_{i})=\sigma_{u}^{-2}D_{i}-\sigma_{u}^{-2}g(\tilde{Z}_{i})=\sigma_{u}^{-2}u_{i}$. Let $\eta_{\alpha,i}=\alpha_{0}(\tilde{X}_{i})-X_{i}^{\top}\pi$. Then \[ X_{i}^{\top}\pi_{*}-\alpha_{0}(\tilde{X}_{i})=X_{i}^{\top}\pi-\alpha_{0}(\tilde{X}_{i})+X_{i}^{\top}(\pi_{*}-\pi)=-\eta_{\alpha,i}+X_{i}^{\top}(\pi_{*}-\pi). \]} \textcolor{black}{By ((ref)), we have that $\mathbb{E}(X_{i}^{\top}(\pi-\pi_{*}))^{2}=(\pi-\pi_{*})^{\top}\Sigma(\pi-\pi_{*})\lesssim\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}\ll1/\log(pn)$. We also know that $\mathbb{E}\eta_{\alpha,i}^{2} \lesssim p^{-\xi_{2}} \ll1/\log(pn)$ (due to $\alpha_{0}\in\mathcal{M}_{C,\xi_{2}}$ and $p\gg\max\{n^{1/(2\xi_{1})},n^{1/(2\xi_{2})}\}$). Then $\mathbb{E}(X_{i}^{\top}\pi_{*}-\alpha_{0}(\tilde{X}_{i}))^{2}\ll1/\log(pn)$. } By the boundedness of $\|X_{i}\|_{\infty}$ and Lemma (ref), \[ \left\Vert \mathbb{E}_{n,1}X_{i}(X_{i}^{\top}\pi_{*}-\alpha_{0}(\tilde{X}_{i}))-\mathbb{E}[X_{i}(X_{i}^{\top}\pi_{*}-\alpha_{0}(\tilde{X}_{i}))\right\Vert _{\infty}\lesssim\varepsilon_{n} \] with probability $1-o(1)$. By $\alpha_{0}(\tilde{X}_{i})=\sigma_{u}^{-2}u_{i}$ and the assumption of $\mathbb{E}|u_{i}|^{2+\kappa}$ being bounded, the boundedness of $\|X_{i}\|_{\infty}$, Lemma (ref) implies \[ \left\Vert \mathbb{E}_{n,1}X_{i}\alpha_{0}(\tilde{X}_{i})-\mathbb{E}X_{i}\alpha_{0}(\tilde{X}_{i})\right\Vert _{\infty}\lesssim\varepsilon_{n} \] with probability $1-o(1)$. Therefore, \[ \left\Vert \mathbb{E}_{n,1}X_{i}X_{i}^{\top}\pi_{*}-\mathbb{E}[X_{i}X_{i}^{\top}\pi_{*}\right\Vert _{\infty}\lesssim\varepsilon_{n} \] with probability $1-o(1)$. \end{proof} \begin{lemma} Let Assumption (ref) hold. Suppose that $\log p\ll n^{1/2-1/(4\max\{\xi_{1},\xi_{2}\})}$. Then $\hat{\pi}$ is well defined and \[ \|\hat{\pi}-\pi\|_{2}\lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)} \] with probability at least $1-o(1)$ and \[ (\hat{\pi}-\pi)^{\top}\hat{\Sigma}_{(1)}(\hat{\pi}-\pi)=O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}). \] Moreover, if $\xi_{2}>1/2$, then with probability at least $1-o(1)$, $\|\hat{\pi}-\pi\|_{1}\lesssim\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}$. \end{lemma} \begin{proof}[Proof of Lemma (ref)] Recall the estimator in ((ref)): \begin{align*} \hat{\pi}=\arg\min_{v\in\mathbb{R}^{p+1}} & \|v\|_{1}\qquad s.t.\qquad\|\hat{\Sigma}_{(1)}v-e_{1}\|_{\infty}\leq\lambda_{1} \end{align*} where $\lambda_{1}=\lambda_{0}\varepsilon_{n}$ for a large enough constant $\lambda_{0}>0$. Let $J$ be the vector of indices of nonzero elements of $\pi_{\ast}$ defined in ((ref)). By Lemma (ref), $|J|\lesssim\varepsilon_{n}^{-2/(2\xi_{2}+1)}\lesssim(n/\log p)^{1/(2\xi_{2}+1)}$. For constants $\kappa_{1},\kappa_{2}>0$ (to be chosen), define the event \[ \mathcal{A}=\left\{ \|\hat{\Sigma}_{(1)}\pi_{*}-\Sigma\pi_{*}\|_{\infty}\leq C_{1}\varepsilon_{n}\right\} \bigcap\left\{ \min_{|A|\leq \kappa_{1} (n/\log p)^{1/(2\xi_{2}+1)} } \min_{\|v_{A^{c}}\|_{1}\leq\|v_{A}\|_{1}}\frac{v^{\top}\hat{\Sigma}_{(1)}v}{\|v_{A}\|_{2}^{2}}\geq\kappa_{2}\right\} , \] where $C_{1}>0$ is a constant that makes the first part of $\mathcal{A}$ holds with probability $1-o(1)$ (this is possible due to Lemma (ref)). The rest of two proof consists of two steps. We first show $P(\mathcal{A})=1-o(1)$ and then derive the final result. \textbf{Step 1:} show that for any $\kappa_{1}>0$, there exists $\kappa_{2}>0$ such that $P(\mathcal{A})\geq1-o(1)$. We first show this in the case of bounded $X_{i}$. We apply Theorem 22 of rudelson2013reconstruction, which requires $n\gg|A|(\log p)\log^{3}(|A|\log p)$. By $|A| \lesssim(n/\log p)^{1/(2\xi_{2}+1)}$, we notice that $|A|(\log p)\log^{3}(|A|\log p)\lesssim(n/\log p)^{1/(2\xi_{2}+1)}(\log p)(\log n)^{3}$. Hence, we only need $\log p\ll n(\log n)^{-3(2\xi_{2}+1)/(2\xi_{2})}$, which holds due to $\log p\ll n^{1/2-1/(4\max\{\xi_{1},\xi_{2}\})}\ll n^{1/2}$. Therefore, for bounded $X_{i}$, Theorem 22 of rudelson2013reconstruction implies the second part of $\mathcal{A}$ has probability approaching one for some $\kappa_{2}>0$. For the case of sub-Gaussian $X_{i}$, we apply Theorem 16 of rudelson2013reconstruction, which requires $n\gg|A|\log(p/|A|)$. This holds due to $|A|\lesssim(n/\log p)^{1/(2\xi_{2}+1)}$ and $\log p \ll n^{1/2}$. Thus, $P(\mathcal{A})\geq1-o(1)$ for some $\kappa_{2}>0$. \textbf{Step 2:} show the final result. The rest of the argument applies on the event $\mathcal{A}$ with a large $\kappa_{1}$ and a corresponding $\kappa_{2}>$ such that $P(\mathcal{A})=1-o(1)$. Notice that $\Sigma\pi=e_{1}$. By Lemma (ref), we have that for a large constant $\lambda_{0}>0$ \[ \|\hat{\Sigma}_{(1)}\pi_{*}-e_{1}\|_{\infty}=\|\hat{\Sigma}_{(1)}\pi_{*}-\Sigma\pi\|_{\infty}\leq\|\hat{\Sigma}_{(1)}\pi_{*}-\Sigma\pi_{*}\|_{\infty}+\|\Sigma\pi_{*}-\Sigma\pi\|_{\infty}\leq\varepsilon_{n}+C_{1}\varepsilon_{n}\leq\lambda_{0}\varepsilon_{n}=\lambda_{1}. \] Hence, the constraint $\|\hat{\Sigma}_{(1)}v-e_{1}\|_{\infty}\leq\lambda_{1}$ is satisfied by $v=\pi_{*}$ so $\hat{\pi}$ is well defined and $\|\hat{\pi}\|_{1}\leq\|\pi_{*}\|_{1}$. Let $\Delta=\hat{\pi}-\pi_{*}$. Since $J={\rm supp}(\pi_{*})$, we have that $\|\pi_{*}+\Delta_{J}\|_{1}+\|\Delta_{J^{c}}\|_{1}\leq\|\pi_{*}\|_{1}$ and thus \begin{equation} \|\Delta_{J^{c}}\|_{1}\leq\|\Delta_{J}\|_{1}. \end{equation} Since $\|\hat{\Sigma}_{(1)}\pi_{*}-e_{1}\|_{\infty}\leq\lambda_{1}$ and $\|\hat{\Sigma}_{(1)}\hat{\phi}-e_{1}\|_{\infty}\leq\lambda_{1}$, we have that $\|\hat{\Sigma}_{(1)}\Delta\|_{\infty}\leq2\lambda_{1}$. Then \begin{equation} \Delta^{\top}\hat{\Sigma}_{(1)}\Delta\leq\|\Delta\|_{1}\|\hat{\Sigma}_{(1)}\Delta\|_{\infty}\overset{\text{(i)}}{\leq}2\sqrt{|J|}\|\Delta_{J}\|_{2}\cdot2\lambda_{1}, \end{equation} where (i) follows by $\|\Delta\|_{1}=\|\Delta_{J}\|_{1}+\|\Delta_{J^{c}}\|_{1}\leq2\|\Delta_{J}\|_{1}\leq2\sqrt{|J|}\|\Delta_{J}\|_{2}$ due to ((ref)). From the definition of $\mathcal{A}$, we have \[ \Delta^{\top}\hat{\Sigma}_{(1)}\Delta\geq\kappa_{2}\|\Delta_{J}\|_{2}^{2}. \] The above two displays imply \begin{equation} \|\Delta_{J}\|_{2}\leq4\kappa_{2}^{-1}\lambda_{1}\sqrt{|J|}. \end{equation} Let $B$ denote the indices corresponding to the largest (in magnitude) $|J|$ entries in $\Delta_{J^{c}}$. Then $|B\bigcup J|=2\cdot|J|$ and $\|\Delta_{(B\bigcup J)^{c}}\|_{1}\leq\|\Delta_{J^{c}}\|_{1}\leq\|\Delta_{J}\|_{1}\leq\|\Delta_{B\bigcup J}\|_{1}$. Thus, we can repeat the same argument as above and obtain \[ \|\Delta_{B\bigcup J}\|_{2}\leq4\kappa_{2}^{-1}\lambda_{1}\sqrt{|B\bigcup J|}=4\kappa_{2}^{-1}\lambda_{1}\sqrt{2|J|}. \] By Lemma 6.9 of buhlmann2011statistics, we have that \begin{equation} \|\Delta_{(B\bigcup J)^{c}}\|_{2}\leq|B\bigcup J|^{-1/2}\|\Delta\|_{1}. \end{equation} Since $\|\Delta\|_{1}=\|\Delta_{J}\|_{1}+\|\Delta_{J^{c}}\|_{1}\leq2\|\Delta_{J}\|_{1}$, ((ref)) implies that \begin{equation} \|\Delta\|_{1}\leq2\|\Delta_{J}\|_{1}\leq2|J|^{1/2}\|\Delta_{J}\|_{2}\leq8\kappa_{2}^{-1}\lambda_{1}|J|. \end{equation} The above two displays and $|B\bigcup J|=2\cdot|J|$ imply \[ \|\Delta_{(B\bigcup J)^{c}}\|_{2}\leq2|B\bigcup J|^{-1/2}\|\Delta_{J}\|_{1}\leq16|B\bigcup J|^{-1/2}\kappa_{2}^{-1}\lambda_{1}|J|=8\sqrt{2}\kappa_{2}^{-1}\lambda_{1}\sqrt{|J|}. \] Hence, ((ref)) and the above display imply \begin{equation} \|\Delta\|_{2}\leq\|\Delta_{B\bigcup J}\|_{2}+\|\Delta_{(B\bigcup J)^{c}}\|_{2}\leq4\kappa_{2}^{-1}\lambda_{1}\sqrt{2|J|}+8\sqrt{2}\kappa_{2}^{-1}\lambda_{1}\sqrt{|J|}=12\sqrt{2}\kappa_{2}^{-1}\lambda_{1}\sqrt{|J|}. \end{equation} Since $|J|\lesssim\varepsilon_{n}^{-2/(2\xi_{2}+1)}$ and $\lambda_{1}=\lambda_{0}\varepsilon_{n}$, the above display implies \[ \|\Delta\|_{2}\lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)} \] and ((ref)) implies \[ \|\Delta\|_{1}\lesssim\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}. \] If $\xi_{2}>1/2$, then $\|\hat{\pi}-\pi\|_{1}\leq\|\hat{\pi}-\pi_{*}\|_{1}+\|\pi_{*}-\pi\|_{1}=\|\Delta\|_{1}+\|\pi_{*}-\pi\|_{1}\lesssim\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}$ due to $\|\pi_{*}-\pi\|_{1}\lesssim\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}$ (Lemma (ref)). We now bound $\|\hat{\pi}-\pi\|_{2}$. Since we already showed that $\|\hat{\pi}-\pi_{*}\|_{2}=\|\Delta\|_{2}\lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}$, it remains to show that $\|\pi_{*}-\pi\|_{2}\lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}$. This is true because $(\pi-\pi_{*})^{\top}\Sigma(\pi-\pi_{*})\lesssim\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}$ (due to Lemma (ref)) and the eigenvalues of $\Sigma$ are bounded below by $\kappa_{3}>0$. Finally, we bound $(\hat{\pi}-\pi)^{\top}\hat{\Sigma}_{(1)}(\hat{\pi}-\pi)$. By ((ref)) and ((ref)), we have that \[ \Delta^{\top}\hat{\Sigma}_{(1)}\Delta\leq4\sqrt{|J|}\|\Delta_{J}\|_{2}\lambda\leq16\kappa_{2}^{-1}\lambda^{2}|J|\lesssim\varepsilon_{n}^{2}\cdot\varepsilon_{n}^{-2/(2\xi_{2}+1)}\lesssim\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)} \] and thus \begin{align*} & (\hat{\pi}-\pi)^{\top}\hat{\Sigma}_{(1)}(\hat{\pi}-\pi)\\ & =[\Delta+(\pi_{*}-\pi)]^{\top}\hat{\Sigma}_{(1)}[\Delta+(\pi_{*}-\pi)]\\ & \leq2\Delta^{\top}\hat{\Sigma}_{(1)}\Delta+2(\pi_{*}-\pi)^{\top}\hat{\Sigma}_{(1)}(\pi_{*}-\pi)\\ & =O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)})+2(\pi_{*}-\pi)^{\top}\hat{\Sigma}_{(1)}(\pi_{*}-\pi)\overset{\text{(i)}}{=}O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}), \end{align*} where (i) follows by \[ \mathbb{E}(\pi_{*}-\pi)^{\top}\hat{\Sigma}_{(1)}(\pi_{*}-\pi)=(\pi_{*}-\pi)^{\top}\Sigma(\pi_{*}-\pi)\lesssim\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}. \] The proof is complete. \end{proof} \begin{lemma} Recall $\hat{\phi}$ from ((ref)): \[ \hat{\phi}=\arg\min_{v\in\mathbb{R}^{p}}\|v\|_{1}\qquad s.t.\qquad\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}v-\mathbb{E}_{n,2}Z_{i}D_{i}\|_{\infty}\leq\lambda_{1}, \] where $\lambda_{1}=\lambda_{0}\varepsilon_{n}$. Let Assumption (ref) hold. Suppose that $\log p\ll n^{1/2-1/(4\max\{\xi_{1},\xi_{2}\})}$. Then for large enough $\lambda_{0}>0$, $\hat{\phi}$ is well defined and $\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\phi}-\phi)\|_{\infty}\leq2\lambda_{1}$ with probability at least $1-o(1)$ and \[ (\hat{\phi}-\phi)^{\top}\mathbb{E}_{n,1}[Z_{i}Z_{i}^{\top}](\hat{\phi}-\phi)=O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}). \] Moreover, if $\xi_{2}>1/2$, then $\|\hat{\phi}-\phi\|_{1}\lesssim\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}$ and $\|\hat{\phi}-\phi\|_{2}\lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}$ with probability at least $1-o(1)$. \end{lemma} \begin{proof}[Proof of Lemma (ref)] This follows by the same argument for Dantzig selector as the proof of Lemma (ref). We only need to verify \begin{equation} P\left(\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}_{n,2}Z_{i}D_{i}\|_{\infty}\leq\lambda\right)\geq1-o(1) \end{equation} and $\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}$ satisfies the restricted eigenvalue condition. Notice that \[ \hat{\Sigma}_{(1)}=\begin{pmatrix}\mathbb{E}_{n,1}D_{i}^{2} & \mathbb{E}_{n,1}D_{i}Z_{i}^{\top}\\ \mathbb{E}_{n,1}D_{i}Z_{i} & \mathbb{E}_{n,1}Z_{i}Z_{i}^{\top} \end{pmatrix}. \] In the proof of Lemma (ref), we show that $\hat{\Sigma}_{(1)}$ satisfies the restricted eigenvalue condition. This means that $\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}$ also satisfies the restricted eigenvalue condition. It remains to show ((ref)). We do so in two cases: bounded $X_{i}$ and sub-Gaussian $X_{i}$. \textbf{Case 1:} show ((ref)) for bounded $X_{i}$. By Lemma (ref), $P(\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}Z_{i}D_{i}\|_{\infty}\leq C_{1}\varepsilon_{n})\geq1-o(1)$. By the same argument as Lemma (ref), we have that $\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}Z_{i}Z_{i}^{\top}\phi\|_{\infty}\leq C_{2}\varepsilon_{n}$ with probability approaching one for some constant $C_{2}>0$. Hence, with probability approaching one, \begin{multline*} \|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}_{n,2}Z_{i}D_{i}\|_{\infty}\leq\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}Z_{i}Z_{i}^{\top}\phi\|_{\infty}+\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}Z_{i}D_{i}\|_{\infty}\\ +\|\mathbb{E}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}Z_{i}D_{i}\|_{\infty}\overset{\text{(i)}}{\leq}C_{1}\varepsilon_{n}+C_{2}\varepsilon_{n}, \end{multline*} where (i) follows by $\mathbb{E}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}Z_{i}D_{i}=\mathbb{E}Z_{i}[Z_{i}^{\top}\phi-g(\tilde{Z}_{i})]-\mathbb{E}Z_{i}u_{i}=0$ (due to $\mathbb{E}Z_{i}[Z_{i}^{\top}\phi-g(\tilde{Z}_{i})]$ by the definition of $\phi$). Since $\lambda_{1}=\lambda_{0}\varepsilon_{n}$ with a large constant $\lambda_{0}$, ((ref)) follows. \textcolor{black}{\textbf{Case 2:} show ((ref)) for sub-Gaussian $X_{i}$.} \textcolor{black}{Again, by $\mathbb{E}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}Z_{i}D_{i}=0$, we have \[ \|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}_{n,2}Z_{i}D_{i}\|_{\infty}\leq\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}Z_{i}Z_{i}^{\top}\phi\|_{\infty}+\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}Z_{i}D_{i}\|_{\infty}. \]} \textcolor{black}{By Lemma (ref), $P(\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}Z_{i}D_{i}\|_{\infty}\leq C_{1}\varepsilon_{n})\geq1-o(1)$. Since $Z_{i}$ is a sub-vector of $X_{i}$, which is sub-Gaussian, $Z_{i}^{\top}\phi$ is sub-Gaussian and hence has bounded third moments. By Lemma (ref), $\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\phi-\mathbb{E}Z_{i}Z_{i}^{\top}\phi\|_{\infty}\leq C\varepsilon_{n}$ with probability $1-o(1)$ for a large enough $C>0$. Hence, ((ref)) follows for a large enough constant $\lambda_{0}$. } \end{proof} \begin{lemma} \textcolor{black}{Recall $\hat{\gamma}=(\hat{\theta},\hat{\beta}^{\top})^{\top}$ from ((ref)): \begin{align*} \hat{\gamma}=(\hat{\theta},\hat{\beta}^{\top})^{\top}=\arg\min_{b\in\mathbb{R},g\in\mathbb{R}^{p}} & \|(b,g^{\top})^{\top}\|_{1}\\ s.t. & \|\mathbb{E}_{n,2}Z_{i}D_{i}b+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}g-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\leq\lambda_{1}\\ & |\mathbb{E}_{n,2}D_{i}^{2}b+\mathbb{E}_{n,2}D_{i}Z_{i}^{\top}g-\mathbb{E}_{n,2}D_{i}Y_{i}|\leq\lambda_{1}\\ & |\mathbb{E}_{n,2}D_{i}^{2}b+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}g-\mathbb{E}_{n,2}D_{i}Y_{i}|\leq\lambda_{2}, \end{align*} where $\lambda_{1}=\lambda_{0}\varepsilon_{n}$ and $\lambda_{2}=\lambda_{n}n^{-1/4}$ for a large $\lambda_{0}>0$. Let Assumption (ref) hold. Suppose that that $\max\{\xi_{1},\xi_{2}\}>1/2$ and $\log p\ll n^{1/2-1/(4\max\{\xi_{1},\xi_{2}\})}$. Then $\hat{\gamma}$ is well defined with probability at least $1-o(1)$, \[ \|\hat{\gamma}-\gamma\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{1}/(2\xi_{1}+1)}), \] \[ (\hat{\beta}-\beta)^{\top}\mathbb{E}_{n,1}[Z_{i}Z_{i}^{\top}](\hat{\beta}-\beta)=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)}) \] and $\|\mathbb{E}_{n,2}Z_{i}D_{i}(\hat{\theta}-\theta)+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)\|_{\infty}=O_{P}(\varepsilon_{n})$. Moreover, if $\xi_{1}>1/2$, then \[ \|\hat{\gamma}-\gamma\|_{1}=O_{P}\left(\varepsilon_{n}^{(2\xi_{1}-1)/(2\xi_{1}+1)}\right). \] Furthermore, if $\xi_{2}>1/2$, then $|\mathbb{E}_{n,2}D_{i}^{2}(\hat{\theta}-\theta)+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)|=O_{P}(n^{-1/4})$. } \end{lemma} \begin{proof}[Proof of Lemma (ref)] By the same argument as in Lemma (ref), we can show that there exists $\gamma_{*}$ with $J_{*}={\rm supp}(\gamma_{*})$ such that (1) $\|\Sigma(\gamma_{*}-\gamma)\|_{\infty}\leq2\varepsilon_{n}$, $(\gamma-\gamma_{*})^{\top}\Sigma(\gamma-\gamma_{*})\lesssim\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)}$ and $|J_{*}|\lesssim\varepsilon_{n}^{-2/(2\xi_{1}+1)}$ and (2) if $\xi_{1}>1/2$, $\|\gamma_{*}-\gamma\|_{1}\lesssim\varepsilon_{n}^{(2\xi_{1}-1)/(2\xi_{1}+1)}$. Notice that the only difference is that now we have $\mathbb{E}Y_{i}X_{i}=\Sigma\gamma+e_{1}\varsigma$ from ((ref)), rather than $\mathbb{E}Y_{i}X_{i}=\Sigma\gamma$; however, since $|\varsigma|\ll\varepsilon_{n}$ (due to ((ref))), all the calculations involved in Lemma (ref) hold, potentially at the cost of larger constants. Partition $\gamma_{*}=(\theta_{*},\beta_{*}^{\top})^{\top}$. Notice that $|J_{*}|\lesssim\varepsilon_{n}^{-2/(2\xi_{1}+1)}\ll\varepsilon_{n}^{-2}$ (due to $\xi_{1}>0$). For constants $\kappa_{1},\kappa_{2}>0$ (to be chosen), we define the event $\mathcal{A}=\mathcal{A}_{1}\bigcap\mathcal{A}_{2}$, where \[ \mathcal{A}_{1}=\left\{ \text{Constraints of }\hat{\gamma}\ \text{are satisfied with}\ \gamma_{*}\right\} \] and \[ \mathcal{A}_{2}=\left\{ \min_{|A|\leq \kappa_{1} (n/\log p)^{1/(2\xi_{2}+1)} } \min_{\|v_{A^{c}}\|_{1}\leq\|v_{A}\|_{1}} \frac{v^{\top}\tilde{\Sigma}v}{\|v_{A}\|_{2}^{2}}\geq\kappa_{2} \right\} \quad{\rm with}\quad\tilde{\Sigma}=\begin{pmatrix}\mathbb{E}_{n,2}D_{i}^{2} & \mathbb{E}_{n,2}D_{i}Z_{i}^{\top}\\ \mathbb{E}_{n,2}Z_{i}D_{i} & \mathbb{E}_{n,1}Z_{i}Z_{i}^{\top} \end{pmatrix}. \] Notice that \begin{align*} \hat{\gamma}=(\hat{\theta},\hat{\beta}^{\top})^{\top}=\arg\min_{b\in\mathbb{R},v\in\mathbb{R}^{p}} & \|(b,v^{\top})^{\top}\|_{1}\\ s.t. & \left\Vert \tilde{\Sigma}\begin{pmatrix}b\\ v \end{pmatrix}\right\Vert _{\infty}\leq\lambda_{1}\\ & |\mathbb{E}_{n,2}D_{i}^{2}b+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}v-\mathbb{E}_{n,2}D_{i}Y_{i}|\leq\lambda_{2}. \end{align*} The rest of the proof proceeds in four steps. We first show two preliminary results in two steps and then present the proof of the final bounds in additional steps. \textcolor{black}{\textbf{Step 1:} show that $P(\mathcal{A}_{1})\geq1-o(1)$.} \textcolor{black}{We can follow the same argument as in Lemma (ref) and show $P(\|\mathbb{E}_{n,1}Z_{i}X_{i}^{\top}\gamma_{*}-\mathbb{E}Z_{i}X_{i}^{\top}\gamma_{*}\|_{\infty}\leq C_{2}\varepsilon_{n})=1-o(1)$ for some $C_{2}>0$. Then with probability approaching one, \begin{align*} & \|\mathbb{E}_{n,2}Z_{i}D_{i}\theta_{*}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\\ & =\|\mathbb{E}_{n,2}Z_{i}D_{i}\theta_{*}-\mathbb{E}_{n,1}Z_{i}D_{i}\theta_{*}+\mathbb{E}_{n,1}Z_{i}D_{i}\theta_{*}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\\ & =\|\mathbb{E}_{n,2}Z_{i}D_{i}\theta_{*}-\mathbb{E}_{n,1}Z_{i}D_{i}\theta_{*}+\mathbb{E}_{n,1}Z_{i}X_{i}^{\top}\gamma_{*}-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\\ & \leq|\theta_{*}|\cdot\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}_{n,1}Z_{i}D_{i}\|_{\infty}+\|\mathbb{E}_{n,1}Z_{i}X_{i}^{\top}\gamma_{*}-\mathbb{E}Z_{i}X_{i}^{\top}\gamma_{*}\|_{\infty}\\ & \quad+\|\mathbb{E}_{n,2}Z_{i}Y_{i}-\mathbb{E}Z_{i}Y_{i}\|_{\infty}+\|\mathbb{E}Z_{i}X_{i}^{\top}\gamma_{*}-\mathbb{E}Z_{i}Y_{i}\|_{\infty}\\ & \overset{\text{(i)}}{\leq}O(1)\cdot C_{1}\varepsilon_{n}+C_{2}\varepsilon_{n}+C_{1}\varepsilon_{n}+\|\mathbb{E}Z_{i}X_{i}^{\top}\gamma_{*}-\mathbb{E}Z_{i}Y_{i}\|_{\infty}\\ & \overset{\text{(ii)}}{=}O(1)\cdot C_{1}\varepsilon_{n}+C_{2}\varepsilon_{n}+C_{1}\varepsilon_{n}+\|\mathbb{E}Z_{i}X_{i}^{\top}\gamma_{*}-\mathbb{E}Z_{i}X_{i}^{\top}\gamma\|_{\infty}\\ & =O(1)\cdot C_{1}\varepsilon_{n}+C_{2}\varepsilon_{n}+C_{1}\varepsilon_{n}+\max_{2\leq j\leq p+1}\left|\mathbb{E}X_{i,j}X_{i}^{\top}\gamma_{*}-\mathbb{E}X_{i,j}X_{i}^{\top}\gamma\right|\\ & \leq O(1)\cdot C_{1}\varepsilon_{n}+C_{2}\varepsilon_{n}+C_{1}\varepsilon_{n}+\|\Sigma(\gamma_{*}-\gamma)\|_{\infty}\\ & \overset{\text{(iii)}}{\leq}O(1)\cdot C_{1}\varepsilon_{n}+C_{2}\varepsilon_{n}+C_{1}\varepsilon_{n}+2\varepsilon_{n}, \end{align*} where (i) follows by $\|\mathbb{E}_{n,1}Z_{i}X_{i}^{\top}\gamma_{*}-\mathbb{E}Z_{i}X_{i}^{\top}\gamma_{*}\|_{\infty}\leq C_{2}\varepsilon_{n}$, Lemma (ref) and $|\theta_{*}|\leq\|\gamma_{*}\|_{2}\lesssim1$, (ii) follows by $Y_{i}=\varepsilon_{i}+X_{i}^{\top}\gamma+\eta_{f,i}$ and $\mathbb{E} Z_{i} \eta_{f,i}=0$ (due to the definition of $\beta$) and (iii) follows by $\|\Sigma(\gamma_{*}-\gamma)\|_{\infty}\leq2\varepsilon_{n}$. Since $\lambda_{1}=\lambda_{0}\varepsilon_{n}$ with a large constant $\lambda_{0}>0$, we have that \[ P\left(\|\mathbb{E}_{n,2}Z_{i}D_{i}\theta_{*}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\leq\lambda_{1}\right)\geq1-o(1). \]} \textcolor{black}{Moreover, we have \begin{align*} & \left|\mathbb{E}_{n,2}D_{i}^{2}\theta_{*}+\mathbb{E}_{n,2}D_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}D_{i}Y_{i}\right|\\ & =\left|\mathbb{E}_{n,2}D_{i}(D_{i}\theta_{*}+Z_{i}^{\top}\beta_{*}-Y_{i})\right|\\ & =\left|\mathbb{E}_{n,2}D_{i}(X_{i}^{\top}\gamma_{*}-X_{i}^{\top}\gamma-\varepsilon_{i}-\eta_{f,i})\right|\\ & \overset{\text{(i)}}{\leq}\left|\mathbb{E}_{n,2}D_{i}X_{i}^{\top}(\gamma_{*}-\gamma)\right|+O_{P}(n^{-1/2})\\ & \leq\left|\mathbb{E}D_{i}X_{i}^{\top}(\gamma_{*}-\gamma)\right|+\left|\left(\mathbb{E}_{n,2}D_{i}X_{i}-\mathbb{E}D_{i}X_{i}\right)^{\top}(\gamma_{*}-\gamma)\right|+O_{P}(n^{-1/2})\\ & \overset{\text{(ii)}}{\leq}\left|\mathbb{E}D_{i}X_{i}^{\top}(\gamma_{*}-\gamma)\right|+o_{P}(n^{-1/2})+O_{P}(n^{-1/2})\\ & \overset{\text{(iii)}}{\leq}\|\Sigma(\gamma_{*}-\gamma)\|_{\infty}+o_{P}(n^{-1/2})+O_{P}(n^{-1/2})\overset{\text{(iv)}}{\leq}2\varepsilon_{n}+o_{P}(n^{-1/2})+O_{P}(n^{-1/2}), \end{align*} where (i) follows by $$\mathbb{E}\left(\mathbb{E}_{n,2}D_{i}\varepsilon_{i}\right)^{2}=n_{2}^{-2}\sum_{i\in I_{2}}\mathbb{E}D_{i}^{2}\varepsilon_{i}^{2}\lesssim n_{2}^{-2}\sum_{i\in I_{2}}\mathbb{E}\varepsilon_{i}^{2}\lesssim n^{-1}$$ and $\left(\mathbb{E}_{n,2}D_{i}\eta_{f,i}\right)^{2}\leq[\mathbb{E}_{n,2}D_{i}^{2}]\cdot[\mathbb{E}_{n,2}\eta_{f,i}^{2}]=O_{P}(1)\cdot O_{P}(p^{-2\xi_{1}})$ together with $p\gg n^{1/(2\xi_{1})}$, (ii) follows by \begin{eqnarray*} &&\mathbb{E}[\left(\mathbb{E}_{n,2}D_{i}X_{i}-\mathbb{E}D_{i}X_{i}\right)^{\top}(\gamma_{*}-\gamma)]^{2} \\ &=&n_{2}^{-1}\mathbb{E}[\left(D_{i}X_{i}-\mathbb{E}D_{i}X_{i}\right)^{\top}(\gamma_{*}-\gamma)]^{2} \\ &\leq &n_{2}^{-1}\mathbb{E}[D_{i}X_{i}^{\top}(\gamma_{*}-\gamma)]^{2}\leq n_{2}^{-1} \mathbb{E} (D_{i}^2) \mathbb{E}[X_{i}^{\top}(\gamma_{*}-\gamma)]^{2}\ll n^{-1} \end{eqnarray*} due to $\mathbb{E}[X_{i}^{\top}(\gamma_{*}-\gamma)]^{2}=(\gamma-\gamma_{*})^{\top}\Sigma(\gamma-\gamma_{*})\lesssim\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)}=o(1)$, (iii) follows by the fact that $\mathbb{E}D_{i}X_{i}^{\top}(\gamma_{*}-\gamma)$ is the first entry of $\Sigma(\gamma_{*}-\gamma)$ and (iv) follows by $\|\Sigma(\gamma_{*}-\gamma)\|_{\infty}\leq2\varepsilon_{n}$. Notice that $\lambda_{1}=\lambda_{0}\varepsilon_{n}$ with a large enough $\lambda_{0}$ and $\varepsilon_{n}\gg n^{-1/2}$. Thus, \begin{equation} P\left(\left|\mathbb{E}_{n,2}D_{i}^{2}\theta_{*}+\mathbb{E}_{n,2}D_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}D_{i}Y_{i}\right|\leq\lambda_{1}\right)\geq1-o(1). \end{equation}} \textcolor{black}{Finally, we need to verify that if $\max\{\xi_{1},\xi_{2}\}>1/2$, then \begin{equation} P\left(\left|\mathbb{E}_{n,2}D_{i}^{2}\theta_{*}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}D_{i}Y_{i}\right|\leq\lambda_{2}\right)\geq1-o(1). \end{equation}} \textcolor{black}{Notice that $\mathbb{E}_{n,2}D_{i}^{2}\theta_{*}=\mathbb{E}D_{i}^{2}\theta_{*}+O_{P}(n^{-1/2})$ and $\mathbb{E}_{n,2}D_{i}Y_{i}=\mathbb{E}D_{i}Y_{i}+O_{P}(n^{-1/2})$ are bounded since $D_{i}$ is sub-Gaussian and $\mathbb{E}Y_{i}^{2}$ is bounded by assumption. Moreover, notice that \begin{align*} \mathbb{E}(D_{i}Z_{i}^{\top}\beta_{*})^{2}&=\mathbb{E}(u_{i}Z_{i}^{\top}\beta_{*}+\phi^{\top}Z_{i}Z_{i}^{\top}\beta_{*}+\eta_{g,i}Z_{i}^{\top}\beta_{*})^{2} \\ &=\mathbb{E}(u_{i}Z_{i}^{\top}\beta_{*})^{2}+\mathbb{E}(\phi^{\top}Z_{i}Z_{i}^{\top}\beta_{*})^{2}+\mathbb{E}(\eta_{g,i}Z_{i}^{\top}\beta_{*})^{2} \end{align*} due to $\mathbb{E}(u_{i}\mid Z_{i})=0$ and $\mathbb{E}\eta_{g,i}Z_{i}=0$ (from the definition of $\eta_{g,i}$). For bounded $X_{i}$, since $\mathbb{E}(X_{i}^{\top}\gamma_{*})^{2}$ has been shown to be bounded and $|D_{i}|\leq\|X_{i}\|_{\infty}$ is bounded, so is $\mathbb{E}(D_{i}Z_{i}^{\top}\beta_{*})^{2}\lesssim\mathbb{E}(Z_{i}^{\top}\beta_{*})^{2}=\mathbb{E}(X_{i}^{\top}\gamma_{*}-D_{i}\theta_{*})^{2}\leq2\mathbb{E}(X_{i}^{\top}\gamma_{*})^{2}+2\theta_{*}^{2}\mathbb{E}D_{i}^{2}$. Thus, $\mathbb{E}(\phi^{\top}Z_{i}Z_{i}^{\top}\beta_{*})^{2}$ is bounded. For sub-Gaussian $X_{i}$, both $Z_{i}^{\top}\phi$ and $Z_{i}^{\top}\beta_{*}$ are sub-Gaussian, which means that $\mathbb{E}(\phi^{\top}Z_{i}Z_{i}^{\top}\beta_{*})^{2}$ is bounded. Hence, \begin{align*} \mathbb{E}\left(\phi^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\right)^{2} & =n_{1}^{-2}\sum_{i\in I_{1}}\mathbb{E}[\phi^{\top}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}\phi^{\top}Z_{i}Z_{i}^{\top}\beta_{*}]^{2}\\ & \leq n_{1}^{-2}\sum_{i\in I_{1}}\mathbb{E}[\phi^{\top}Z_{i}Z_{i}^{\top}\beta_{*}]^{2}\lesssim n_{1}^{-1}\mathbb{E}(Z_{i}^{\top}\beta_{*})^{2}\lesssim n^{-1}. \end{align*}} \textcolor{black}{Therefore, \begin{align*} & \left|\mathbb{E}_{n,2}D_{i}^{2}\theta_{*}+\phi^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}D_{i}Y_{i}\right|\\ & \leq\left|\mathbb{E}_{n,2}D_{i}^{2}-\mathbb{E}D_{i}^{2}\right|\cdot|\theta_{*}|+\left|\phi^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\right|\\ & \quad+|\mathbb{E}_{n,2}D_{i}Y_{i}-\mathbb{E}D_{i}Y_{i}|+\left|\mathbb{E}D_{i}^{2}\theta_{*}+\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}D_{i}Y_{i}\right|\\ & =O_{P}(n^{-1/2})+\left|\mathbb{E} D_{i}^{2}\theta_{*}+\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}D_{i}Y_{i}\right|\\ & =O_{P}(n^{-1/2})+\left|\mathbb{E}D_{i}^{2}\theta_{*}+\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}D_{i}(D_{i}\theta+Z_{i}^{\top}\beta+\eta_{f,i}+\varepsilon_{i})\right|\\ & \leq O_{P}(n^{-1/2})+\left|\mathbb{E}D_{i}^{2}(\theta_{*}-\theta)+\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right| +\left|\mathbb{E} D_i \eta_{f,i} \right| \\ & \overset{\text{(i)}}{=} O_{P}(n^{-1/2})+\left|\mathbb{E}D_{i}^{2}(\theta_{*}-\theta)+\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right| +\left|\mathbb{E} \eta_{g,i} \eta_{f,i} \right| \\ & \overset{\text{(ii)}}{=}O_{P}(n^{-1/2})+\left|\mathbb{E}D_{i}X_{i}^{\top}(\gamma_{*}-\gamma)\right| +\left|\mathbb{E} \eta_{g,i} \eta_{f,i} \right|\\ & \overset{\text{(iii)}}{=}O_{P}(n^{-1/2})+O\left(\sqrt{n^{-1}\log p}\right) +\left|\mathbb{E} \eta_{g,i} \eta_{f,i} \right| \overset{\text{(iv)}}{=}o_{P}(n^{-1/4}), \end{align*} where (i) follows by $D_{i}=u_{i}+Z_{i}^{\top}\phi+\eta_{g,i}$ and $\mathbb{E} Z_{i}\eta_{f,i}=0$ (due to the definition of $\beta$), (ii) follows by $\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}=\mathbb{E}D_{i}Z_{i}^{\top}$, (iii) follows by the fact that $\mathbb{E}D_{i}X_{i}^{\top}(\gamma_{*}-\gamma)$ is the first entry of $\Sigma(\gamma_{*}-\gamma)$ and $\|\Sigma(\gamma_{*}-\gamma)\|_{\infty}\leq2\varepsilon_{n}\asymp\sqrt{n^{-1}\log p}$ and (iv) follows by $\log p\ll\sqrt{n}$, $|\mathbb{E} \eta_{g,i} \eta_{f,i}|^2 \leq \mathbb{E} \eta_{f,i}^2 \cdot \mathbb{E} \eta_{g,i}^2 \lesssim p^{-2\xi_{1}} p^{-2\xi_{2}}$ and $p\gg n^{1/(2\xi_{1})}$. Thus, to show ((ref)), it suffices to verify that if $\max\{\xi_{1},\xi_{2}\}>1/2$, then \begin{equation} \left|(\hat{\phi}-\phi)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}\right|=o_{P}(n^{-1/4}). \end{equation}} \textcolor{black}{We verify ((ref)) in two cases: (a) $\xi_{1}>1/2$ and (b) $\xi_{2}>1/2$.} \textcolor{black}{In Case (a), we have that $\log p\ll n^{1/2-1/(4\xi_{1})}$ and \begin{multline*} \left|(\hat{\phi}-\phi)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}\right|\leq\|\beta_{*}\|_{1}\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\phi}-\phi)\|_{\infty}\\ \leq\|\gamma_{*}\|_{1}\cdot\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\phi}-\phi)\|_{\infty}\overset{\text{(i)}}{=}O_{P}\left(\varepsilon_{n}^{-1/(2\xi_{1}+1)}\cdot\varepsilon_{n}\right)\overset{\text{(ii)}}{=}o_{P}(n^{-1/4}) \end{multline*} where (i) follows by $\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\phi}-\phi)\|_{\infty}\leq2\lambda_{1}\lesssim\varepsilon_{n}$ (Lemma (ref)) and $\|\gamma_{*}\|_{1}\leq\sqrt{|J_{*}|}\cdot\|\gamma_{*}\|_{2}\lesssim\sqrt{|J_{*}|}\lesssim\varepsilon_{n}^{-1/(2\xi_{1}+1)}$ due to $\|\gamma_{*}\|_{2}\lesssim1$, $J_{*}={\rm supp}(\gamma_{*})$ and $|J_{*}|\lesssim\varepsilon_{n}^{-2/(2\xi_{1}+1)}$ (shown above) and (ii) follows by $\log p\ll n^{1/2-1/(4\xi_{1})}$.} \textcolor{black}{In Case (b), we have that $\log p\ll n^{1/2-1/(4\xi_{2})}$, $\|\hat{\phi}-\phi\|_{1}\lesssim\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}$ and $\|\hat{\phi}-\phi\|_{2}\lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}$ with probability $1-o(1)$ (due to Lemma (ref)). Then with probability $1-o(1)$, \begin{align} & \left|(\hat{\phi}-\phi)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}\right|\nonumber \\ & \leq\left|(\hat{\phi}-\phi)^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\right|+\left|(\hat{\phi}-\phi)^{\top}(\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}-\mathbb{E}Z_{i}Z_{i}^{\top})\beta_{*}\right|\nonumber \\ & \leq\|\hat{\phi}-\phi\|_{2}\cdot\|\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\|_{2}+\|\hat{\phi}-\phi\|_{1}\cdot\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\|_{\infty}\nonumber \\ & \lesssim\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}\cdot\|\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\|_{2}+\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}\cdot\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\|_{\infty}\nonumber \\ & \overset{\text{(i)}}{\lesssim}\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}\cdot1+\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}\cdot\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\|_{\infty}\nonumber \\ & \overset{\text{(ii)}}{\lesssim}\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}\cdot1+\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}\cdot\varepsilon_{n}\overset{\text{(iii)}}{\ll}n^{-1/4}, \end{align} where (i) follows by $\|\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\|_{2}\leq\lambda_{\max}(\mathbb{E}Z_{i}Z_{i}^{\top})\cdot\|\beta_{*}\|_{2}\leq\lambda_{\max}(\Sigma)\cdot\|\gamma_{*}\|_{2}\lesssim1$ (due to $\|\gamma_{*}\|_{2}\lesssim1$ as shown above) and (ii) follows by $\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}Z_{i}Z_{i}^{\top}\beta_{*}\|_{\infty}\lesssim\varepsilon_{n}$; to see this, follow the same argument in Lemma (ref). Moreover, (iii) in the above display follows by $\log p\ll n^{1/2-1/(4\xi_{2})}$. Therefore, we have verified ((ref)). Hence, we have proved that $\gamma_{*}$ satisfies all the constraints with probability at least $1-o(1)$.} \textcolor{black}{\textbf{Step 2:} show that for any $\kappa_{1}>0$, there exists $\kappa_{2}>0$ such that $P(\mathcal{A}_{2})\geq1-o(1)$.} \textcolor{black}{Notice that \[ \tilde{\Sigma}-\hat{\Sigma}_{(1)}=\begin{pmatrix}\mathbb{E}_{n,2}D_{i}^{2}-\mathbb{E}_{n,1}D_{i}^{2} & \mathbb{E}_{n,2}D_{i}Z_{i}^{\top}-\mathbb{E}_{n,1}D_{i}Z_{i}^{\top}\\ \mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}_{n,1}Z_{i}D_{i} & 0 \end{pmatrix}. \] In Lemma (ref), we have proved that for any $\kappa_{1}$, there exists $\kappa_{2}>0$ such that $$ P \left( \min_{|A|\leq \kappa_{1} (n/\log p)^{1/(2\xi_{2}+1)} } \min_{\|v_{A^{c}}\|_{1}\leq\|v_{A}\|_{1}}\frac{v^{\top}\hat{\Sigma}_{(1)}v}{\|v_{A}\|_{2}^{2}}\geq 2 \kappa_{2} \right)\geq 1-o(1). $$ Therefore, we only need to show that \begin{equation} P\left(\max_{|A|\leq \kappa_{1} (n/\log p)^{1/(2\xi_{2}+1)} } \max_{\|v_{A^{c}}\|_{1}\leq\|v_{A}\|_{1}}\frac{\left|v^{\top}\left(\tilde{\Sigma}-\hat{\Sigma}_{(1)}\right)v\right|}{\|v_{A}\|_{2}^{2}}\leq\kappa_{2} \right)\geq1-o(1). \end{equation}} \textcolor{black}{Fix a set $A$. Notice that if $v=(v_{1},v_{-1}^{\top})^{\top}$ with $v_{1}=0$, then $v^{\top}\left(\tilde{\Sigma}-\hat{\Sigma}_{(1)}\right)v=0$. Hence, we only need to consider $v=(v_{1},v_{-1}^{\top})^{\top}$ with $v_{1}\neq0$. Without loss of generality, we can normalize $v=(1,v_{-1}^{\top})^{\top}$ since the ratio $\left|v^{\top}\left(\tilde{\Sigma}-\hat{\Sigma}_{(1)}\right)v\right|/\|v_{A}\|_{2}^{2}$ depends only on $v_{-1}/v_{1}$. Notice that for $v=(1,v_{-1}^{\top})^{\top}$, \[ \left|v^{\top}\left(\tilde{\Sigma}-\hat{\Sigma}_{(1)}\right)v\right|=\left|\left(\mathbb{E}_{n,2}D_{i}^{2}-\mathbb{E}_{n,1}D_{i}^{2}\right)+2v_{-1}^{\top}\left(\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}_{n,1}Z_{i}D_{i}\right)\right|\leq\tau_{1}+\|v_{-1}\|_{1}\tau_{2}, \] where $\tau_{1}=|\mathbb{E}_{n,2}D_{i}^{2}-\mathbb{E}_{n,1}D_{i}^{2}|$ and $\tau_{2}=\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}_{n,1}Z_{i}D_{i}\|_{\infty}$.} \textcolor{black}{We discuss two cases: (a) $1\in A$ and (b) $1\notin A$.} \textcolor{black}{In Case (a), we have \begin{multline*} \frac{\left|v^{\top}\left(\tilde{\Sigma}-\hat{\Sigma}_{(1)}\right)v\right|}{\|v_{A}\|_{2}^{2}}\leq\frac{\tau_{1}+\|v_{-1}\|_{1}\tau_{2}}{\|v_{A}\|_{2}^{2}}\overset{\text{(i)}}{\leq}\frac{\tau_{1}+2\|v_{A}\|_{1}\tau_{2}}{\|v_{A}\|_{2}^{2}}\\ \leq\frac{\tau_{1}+2\sqrt{|A|}\cdot\|v_{A}\|_{2}\tau_{2}}{\|v_{A}\|_{2}^{2}}=\frac{\tau_{1}}{\|v_{A}\|_{2}^{2}}+\frac{2\sqrt{|A|}\tau_{2}}{\|v_{A}\|_{2}}\overset{\text{(ii)}}{\leq}\tau_{1}+2\sqrt{|A|}\tau_{2}, \end{multline*} where (i) follows by $\|v_{-1}\|_{1}\leq\|v\|_{1}=\|v_{A}\|_{1}+\|v_{A^{c}}\|_{1}\leq2\|v_{A}\|_{1}$ due to $\|v_{A^{c}}\|_{1}\leq\|v_{A}\|_{1}$ and (ii) follows by $\|v_{A}\|_{2}\geq|v_{1}|=1$ due to $v_{1}=1$ and $1\in A$.} \textcolor{black}{In Case (b), we have that $1\in A^{c}$ and thus $\|v_{A^{c}}\|_{1}\geq|v_{1}|=1$, which means that $\|v_{A}\|_{1}\geq\|v_{A^{c}}\|_{1}=1$. Thus, \begin{multline*} \frac{\left|v^{\top}\left(\tilde{\Sigma}-\hat{\Sigma}_{(1)}\right)v\right|}{\|v_{A}\|_{2}^{2}}\leq\frac{\tau_{1}+\|v_{-1}\|_{1}\tau_{2}}{\|v_{A}\|_{2}^{2}}\leq\frac{\tau_{1}+2\|v_{A}\|_{1}\tau_{2}}{\|v_{A}\|_{2}^{2}}\\ \leq\frac{\tau_{1}+2\sqrt{|A|}\cdot\|v_{A}\|_{2}\tau_{2}}{\|v_{A}\|_{2}^{2}}=\frac{\tau_{1}}{\|v_{A}\|_{2}^{2}}+\frac{2\sqrt{|A|}\tau_{2}}{\|v_{A}\|_{2}}\leq\tau_{1}+2\sqrt{|A|}\tau_{2}. \end{multline*}} \textcolor{black}{Therefore, we have proved that \begin{equation} \max_{\|v_{A^{c}}\|_{1}\leq\|v_{A}\|_{1}}\frac{\left|v^{\top}\left(\tilde{\Sigma}-\hat{\Sigma}_{(1)}\right)v\right|}{\|v_{A}\|_{2}^{2}}\leq\tau_{1}+2\sqrt{|A|}\tau_{2}. \end{equation}} \textcolor{black}{Notice that $\tau_{1}=o_{P}(1)$, $\tau_{2}=O_{P}(\sqrt{n^{-1}\log p})=O_{P}(\varepsilon_{n})$ (due to Lemma (ref)) and $|A|\lesssim \varepsilon_{n}^{-2/(2\xi_{2}+1)}$ The above display is $o_{P}(1)+o_{P}(\varepsilon_{n}^{-1/(2\xi_{2}+1)}\cdot\varepsilon_{n})=o_{P}(1)$. Thus, ((ref)) follows by ((ref)).} \textcolor{black}{\textbf{Step 3:} show the bounds for $\|\hat{\gamma}-\gamma\|_{2}$, $\|\hat{\gamma}-\gamma\|_{1}$ and $(\hat{\beta}-\beta)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)$.} \textcolor{black}{In the previous two steps, we have verified that for any $\kappa_{1}>0$, there exists $\kappa_{2}>0$ such that $P(\mathcal{A})\geq1-o(1)$. On the event $\mathcal{A}$, $\hat{\gamma}$ is well defined. The same argument as in Lemma (ref) yields the bound for $\|\hat{\gamma}-\gamma\|_{2}$ and $\|\hat{\gamma}-\gamma\|_{1}$.} \textcolor{black}{We now show the bound for $(\hat{\beta}-\beta)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)$. Let $\Delta:=\hat{\gamma}-\gamma_{*}$ be partitioned as $\Delta=(\Delta_{\theta},\Delta_{\beta}^{\top})^{\top}$. From the same argument as ((ref)) and ((ref)) in Lemma (ref), we can derive that on the event $\mathcal{A}$, $\|\Delta\|_{1}\leq8\kappa_{2}^{-1}\lambda_{1}|J|$ and $\|\Delta\|_{2}\leq12\sqrt{2}\kappa_{2}^{-1}\lambda_{1}\sqrt{|J|}$ with $|J|\lesssim\varepsilon_{n}^{-2/(2\xi_{1}+1)}$ and $\lambda_{1}\asymp\varepsilon_{n}$. Hence, $\|\Delta\|_{1}=O_{P}(\varepsilon_{n}^{(2\xi_{1}-1)/(2\xi_{1}+1)})$ and $\|\Delta\|_{2}=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})$.} \textcolor{black}{On the event $\mathcal{A}$, both $\gamma_{*}$ and $\hat{\gamma}$ satisfy the constraints of $\hat{\gamma}$, which means that \[ \begin{cases} \|\mathbb{E}_{n,2}Z_{i}D_{i}\theta_{*}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\leq\lambda_{1}\\ \|\mathbb{E}_{n,2}Z_{i}D_{i}\hat{\theta}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\hat{\beta}-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\leq\lambda_{1} \end{cases} \] and \[ \begin{cases} |\mathbb{E}_{n,2}D_{i}^{2}\theta_{*}+\mathbb{E}_{n,2}D_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}D_{i}Y_{i}|\leq\lambda_{1}\\ |\mathbb{E}_{n,2}D_{i}^{2}\hat{\theta}+\mathbb{E}_{n,2}D_{i}Z_{i}^{\top}\hat{\beta}-\mathbb{E}_{n,2}D_{i}Y_{i}|\leq\lambda_{1}. \end{cases} \]} \textcolor{black}{Therefore, by the triangular inequality, we have that on the event $\mathcal{A}$, \begin{equation} \|\mathbb{E}_{n,2}Z_{i}D_{i}\Delta_{\theta}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\Delta_{\beta}\|_{\infty}\leq2\lambda_{1} \end{equation} and \begin{equation} |\mathbb{E}_{n,2}D_{i}^{2}\Delta_{\theta}+\mathbb{E}_{n,2}D_{i}Z_{i}^{\top}\Delta_{\beta}|\leq2\lambda_{1}. \end{equation}} \textcolor{black}{By ((ref)) and Hölder's inequality, we have that $|\Delta_{\beta}^{\top}\mathbb{E}_{n,2}Z_{i}D_{i}\Delta_{\theta}+\Delta_{\beta}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\Delta_{\beta}|\leq2\lambda_{1}\|\Delta_{\beta}\|_{1}$, which means that \[ \Delta_{\beta}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\Delta_{\beta}\leq2\lambda_{1}\|\Delta_{\beta}\|_{1}+|\Delta_{\beta}^{\top}\mathbb{E}_{n,2}Z_{i}D_{i}|\cdot|\Delta_{\theta}|. \]} \textcolor{black}{By ((ref)), we have $|\mathbb{E}_{n,2}D_{i}Z_{i}^{\top}\Delta_{\beta}|\leq\mathbb{E}_{n,2}D_{i}^{2}\cdot|\Delta_{\theta}|+2\lambda_{1}$. Hence, \begin{align*} \Delta_{\beta}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\Delta_{\beta} & \leq2\lambda_{1}\|\Delta_{\beta}\|_{1}+2\lambda_{1}\cdot|\Delta_{\theta}|+\mathbb{E}_{n,2}D_{i}^{2}\cdot\Delta_{\theta}^{2}\\ & \overset{\text{(i)}}{\leq}4\lambda_{1}\|\Delta\|_{1}+\mathbb{E}_{n,2}D_{i}^{2}\cdot\|\Delta\|_{2}^{2}\overset{\text{(ii)}}{=}O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)}), \end{align*} where (i) follows by $\|\Delta_{\beta}\|_{1}\leq\|\Delta\|_{1}$, $|\Delta_{\theta}|\leq\|\Delta\|_{1}$ and $\Delta_{\theta}^{2}\leq\|\Delta\|_{2}^{2}$ and (ii) follows by the sub-Gaussianity of $D_{i}$, $\lambda_{1}=O(\varepsilon_{n})$, $\|\Delta\|_{1}=O_{P}(\varepsilon_{n}^{(2\xi_{1}-1)/(2\xi_{1}+1)})$ and $\|\Delta\|_{2}=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})$. Now we notice that \begin{align*} & (\hat{\beta}-\beta)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)\\ & =[\Delta_{\beta}+(\beta_{*}-\beta)]^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}[\Delta_{\beta}+(\beta_{*}-\beta)]\\ & \leq2\Delta_{\beta}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\Delta_{\beta}+2(\beta_{*}-\beta)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\\ & \overset{\text{(i)}}{=}O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})+2(\beta_{*}-\beta)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\\ & \overset{\text{(ii)}}{=}O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)}), \end{align*} where (i) follows by $\Delta_{\beta}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\Delta_{\beta}=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})$ and (ii) follows by \begin{multline*} \mathbb{E}(\beta_{*}-\beta)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)=(\beta_{*}-\beta)^{\top}(\mathbb{E}Z_{i}Z_{i}^{\top})(\beta_{*}-\beta)\leq\lambda_{\max}(\mathbb{E}Z_{i}Z_{i}^{\top})\cdot\|\beta_{*}-\beta\|_{2}^{2}\\ \leq\lambda_{\max}(\Sigma)\cdot\|\gamma_{*}-\gamma\|_{2}^{2}\leq\frac{\lambda_{\max}(\Sigma)}{\lambda_{\min}(\Sigma)}(\gamma_{*}-\gamma)^{\top}\Sigma(\gamma_{*}-\gamma)\lesssim\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)}. \end{multline*}} \textcolor{black}{\textbf{Step 4:} show the bounds for $\|\mathbb{E}_{n,2}Z_{i}D_{i}(\hat{\theta}-\theta)+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)\|_{\infty}$ and $|\mathbb{E}_{n,2}D_{i}^{2}\delta_{\theta}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}|$.} \textcolor{black}{To bound $\|\mathbb{E}_{n,2}Z_{i}D_{i}(\hat{\theta}-\theta)+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)\|_{\infty}$, we observe that on the event $\mathcal{A}$, \begin{multline} \|\mathbb{E}_{n,2}Z_{i}D_{i}(\hat{\theta}-\theta_{*})+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta_{*})\|_{\infty}\leq\|\mathbb{E}_{n,2}Z_{i}D_{i}\hat{\theta}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\hat{\beta}-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\\ +\|\mathbb{E}_{n,2}Z_{i}D_{i}\theta_{*}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}Z_{i}Y_{i}\|_{\infty}\leq2\lambda_{1}. \end{multline}} \textcolor{black}{We notice that $\mathbb{E}Z_{i}D_{i}(\theta_{*}-\theta)+\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)=\mathbb{E}Z_{i}X_{i}^{\top}(\gamma_{*}-\gamma)$ are the last $p$ elements of $\Sigma(\gamma_{*}-\gamma)$, which means that \[ \|\mathbb{E}Z_{i}D_{i}(\theta_{*}-\theta)+\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\|_{\infty}\leq\|\Sigma(\gamma_{*}-\gamma)\|_{\infty}\leq\varepsilon_{n}. \]} \textcolor{black}{By Lemma (ref), $\|\mathbb{E}_{n,2}Z_{i}D_{i}-\mathbb{E}Z_{i}D_{i}\|_{\infty}=O_{P}(\varepsilon_{n})$. Notice that $\|\beta_{*}-\beta\|_{2}\leq\|\gamma_{*}-\gamma\|_{2}\leq\sqrt{(\gamma-\gamma_{*})^{\top}\Sigma(\gamma-\gamma_{*})/\lambda_{\min}(\Sigma)}\lesssim1$. Thus, the same argument as in Lemma (ref) yields $\|\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)-\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\|_{\infty}=O_{P}(\varepsilon_{n})$. Therefore, \begin{multline*} \|\mathbb{E}_{n,2}Z_{i}D_{i}(\theta_{*}-\theta)+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\|_{\infty}\\ \leq\|\mathbb{E}Z_{i}D_{i}(\theta_{*}-\theta)+\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\|_{\infty}+O_{P}(\varepsilon_{n})=O_{P}(\varepsilon_{n}). \end{multline*}} \textcolor{black}{The above display and ((ref)) imply $\|\mathbb{E}_{n,2}Z_{i}D_{i}(\hat{\theta}-\theta)+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)\|_{\infty}=O_{P}(\varepsilon_{n})$.} \textcolor{black}{To bound $|\mathbb{E}_{n,2}D_{i}^{2}\delta_{\theta}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}|$ for $\xi_{2}>1/2$, we similarly observe that on the event $\mathcal{A}$, the triangular inequality yields \begin{multline} |\mathbb{E}_{n,2}D_{i}^{2}(\hat{\theta}-\theta_{*})+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta_{*})|\leq|\mathbb{E}_{n,2}D_{i}^{2}\hat{\theta}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\hat{\beta}-\mathbb{E}_{n,2}D_{i}Y_{i}|\\ +|\mathbb{E}_{n,2}D_{i}^{2}\theta_{*}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\beta_{*}-\mathbb{E}_{n,2}D_{i}Y_{i}|\leq2n^{-1/4}. \end{multline}} \textcolor{black}{We notice that $\mathbb{E}D_{i}^{2}(\theta_{*}-\theta)+\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)=\mathbb{E}D_{i}^{2}(\theta_{*}-\theta)+\mathbb{E}D_{i}Z_{i}^{\top}(\beta_{*}-\beta)=\mathbb{E}D_{i}X_{i}^{\top}(\gamma_{*}-\gamma)$ is the first element of $\Sigma(\gamma_{*}-\gamma)$, which means that \[ |\mathbb{E}D_{i}^{2}(\theta_{*}-\theta)+\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)|\leq\|\Sigma(\gamma_{*}-\gamma)\|_{\infty}\leq\varepsilon_{n}. \]} \textcolor{black}{We notice that \begin{align*} & \left|\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)-\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right|\\ & \leq\left|(\hat{\phi}-\phi)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right|+\left|\phi^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)-\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right|\\ & \overset{\text{(i)}}{=}o_{P}(n^{-1/4})+\left|\phi^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)-\phi^{\top}\mathbb{E}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right|\\ & \overset{\text{(ii)}}{=}o_{P}(n^{-1/4})+\left|\mathbb{E}_{n,1}(D_{i}-u_{i})Z_{i}^{\top}(\beta_{*}-\beta)-\mathbb{E}D_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right|\\ & \leq o_{P}(n^{-1/4})+\left|\mathbb{E}_{n,1}u_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right|+\left|\mathbb{E}_{n,1}D_{i}Z_{i}^{\top}(\beta_{*}-\beta)-\mathbb{E}D_{i}Z_{i}^{\top}(\beta_{*}-\beta)\right|\\ & \overset{\text{(iii)}}{\leq}o_{P}(n^{-1/4})+O_{P}(n^{-1/2}), \end{align*} where (i) follows by the same argument as in ((ref)) with $\beta_{*}$ replaced by $\beta_{*}-\beta$ for $\xi_{2}>1/2$, (ii) follows by $D_{i}=\phi^{\top}Z_{i}+\eta_{g,i}+u_{i}$ and $\mathbb{E}Z_{i}\eta_{g,i}=0$, and (iii) follows by Markov's inequality and $$\mathbb{E}[D_{i}Z_{i}^{\top}(\beta_{*}-\beta)]^{2}\leq \mathbb{E}[D_{i}^2] \cdot \mathbb{E}[Z_{i}^{\top}(\beta_{*}-\beta)]^{2}\lesssim1$$ and $$\mathbb{E}[u_{i}Z_{i}^{\top}(\beta_{*}-\beta)]^{2}\lesssim\mathbb{E}[Z_{i}^{\top}(\beta_{*}-\beta)]^{2}\lesssim1$$ (due to $\mathbb{E}(u_{i}^{2}\mid Z_{i})\lesssim1$); to see the last step, we simply notice that $n_{1}^{-1}\sum_{i\in I_{1}}Q_{i}=O_{P}(n^{-1/2})$ for independent variables $Q_{i}$ if $\mathbb{E}(Q_{i})=0$ and $\mathbb{E}(Q_{i}^{2})\lesssim1$.} \textcolor{black}{Also notice that $|\mathbb{E}_{n,2}D_{i}^{2}(\theta_{*}-\theta)-\mathbb{E}D_{i}^{2}(\theta_{*}-\theta)|=O_{P}(n^{-1/2})$ (due to the law of large numbers). This and the above two displays imply \[ |\mathbb{E}_{n,2}D_{i}^{2}(\theta_{*}-\theta)+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\beta_{*}-\beta)|=O_{P}(\varepsilon_{n})+o_{P}(n^{-1/4}). \] } \textcolor{black}{The above display and ((ref)) imply $|\mathbb{E}_{n,2}D_{i}^{2}(\hat{\theta}-\theta)+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)|=O_{P}(\varepsilon_{n})+O_{P}(n^{-1/4})=O_{P}(n^{-1/4})$. The proof is complete. } \end{proof} \begin{proof}[Proof of Theorem (ref)] Let $\delta_{\gamma}=\hat{\gamma}-\gamma$ and $\delta_{\pi}=\hat{\pi}-\pi$. We partition $\delta_{\gamma}=(\delta_{\theta},\delta_{\beta}^{\top})^{\top}$ and $\delta_{\pi}=(\delta_{\pi_{1}},\delta_{\pi_{-1}}^{\top})^{\top}$ such that the first component is a scalar and the second component is an element in $\mathbb{R}^{p}$. \textcolor{black}{ Let $\eta_{f,i}=f(\tilde{Z}_{i})-Z_{i}^{\top}\beta$ and $\eta_{\alpha,i}=\alpha_{0}(\tilde{X}_{i})-X_{i}^{\top}\pi$. We show that if $\max\{\xi_{1},\xi_{2}\}>1/2$, then \begin{equation} \tilde{\theta}_{1}-\theta=\pi^{\top}\mathbb{E}_{n,1}X_{i}\varepsilon_{i}+o_{P}(n^{-1/2}). \end{equation} Notice that this would mean $\tilde{\theta}_{1}-\theta=\sigma_{u}^{-2}\mathbb{E}_{n,1}u_{i}\varepsilon_{i}+o_{P}(n^{-1/2})$ because $\mathbb{E} (\pi^{\top}X_{i}-\alpha_{0}(\tilde{X}_{i}))^{2} \lesssim p^{-2\xi_{2}} \ll n^{-1}$ (due to $\alpha_{0}\in \mathcal{M}_{C,\xi_{2}}$ and $p \gg n^{1/(2\xi_{2}}$).} An analogous argument (with $I_{1}$ and $I_{2}$ swapped) yields $$\tilde{\theta}_{2}-\theta=\pi^{\top}\mathbb{E}_{n,2}X_{i}\varepsilon_{i}+o_{P}(n^{-1/2})=\sigma_{u}^{-2}\mathbb{E}_{n,2}u_{i}\varepsilon_{i}+o_{P}(n^{-1/2}).$$ Since $\hat{\theta}=\frac{n_{1}}{n}\tilde{\theta}_{1}+\frac{n_{2}}{n}\tilde{\theta}_{2}$, we have that $\sqrt{n}(\tilde{\theta}-\theta)=n^{-1/2}\sum_{i=1}^{n}\pi^{\top}X_{i}\varepsilon_{i}+o_{P}(1)$. Notice that $$\mathbb{E}(\pi^{\top}X_{i}\varepsilon_{i})^{2}=\mathbb{E}[\mathbb{E}(\varepsilon_{i}^{2}\mid X_{i})\cdot(X_{i}^{\top}\pi)^{2}]\lesssim\mathbb{E}[(X_{i}^{\top}\pi)^{2}]=\pi^{\top}\Sigma\pi \lesssim \| \pi\|_{2}^{2}\lesssim1.$$ We have $\mathbb{E}(n^{-1/2}\sum_{i=1}^{n}\pi^{\top}X_{i}\varepsilon_{i})^{2}\lesssim1$, which means that $\sqrt{n}(\tilde{\theta}-\theta)=O_{P}(1)$. Since either $\xi_{1}\geq\xi_{2}$ or $\xi_{1}<\xi_{2}$, we show ((ref)) in these two cases in two steps. \textbf{Step 1:} show ((ref)) assuming $\xi_{1}\geq\xi_{2}$. Since $\max\{\xi_{1},\xi_{2}\}>1/2$ , $\xi_{1}\geq\xi_{2}$ implies $\xi_{1}>1/2$. The assumption of $\log p\ll n^{1/2-1/(4\max\{\xi_{1},\xi_{2}\})}$ and $\xi_{1}\ge\xi_{2}$ imply $\log p\ll n^{1/2-1/(4\xi_{1})}$. Notice that \begin{align} \tilde{\theta}_{1}-\theta & =e_{1}^{\top}\hat{\gamma}-e_{1}^{\top}\gamma+\hat{\pi}^{\top}\mathbb{E}_{n,1}X_{i}Y_{i}-\hat{\pi}^{\top}\hat{\Sigma}_{(1)}\hat{\gamma}\nonumber \\ & =e_{1}^{\top}\delta_{\gamma}+\hat{\pi}^{\top}\mathbb{E}_{n,1}X_{i}Y_{i}-\hat{\pi}^{\top}\hat{\Sigma}\gamma-\hat{\pi}^{\top}\hat{\Sigma}_{(1)}\delta_{\gamma}\nonumber \\ & =(e_{1}^{\top}-\hat{\pi}^{\top}\hat{\Sigma}_{(1)})\delta_{\gamma}+\hat{\pi}^{\top}\mathbb{E}_{n,1}X_{i}(X_{i}^{\top}\gamma+\varepsilon_{i}+\eta_{f,i})-\hat{\pi}^{\top}\hat{\Sigma}_{(1)}\gamma\nonumber \\ & =(e_{1}^{\top}-\hat{\pi}^{\top}\hat{\Sigma}_{(1)})\delta_{\gamma}+\pi^{\top}\mathbb{E}_{n,1}X_{i}\varepsilon_{i}+(\hat{\pi}-\pi)^{\top}\mathbb{E}_{n,1}X_{i}\varepsilon_{i}+\hat{\pi}^{\top}\mathbb{E}_{n,1}X_{i}\eta_{f,i}. \end{align} We notice that \begin{align} \mathbb{E}\left([(\hat{\pi}-\pi)^{\top}\mathbb{E}_{n,1}X_{i}\varepsilon_{i}]^{2}\mid\{X_{i}\}_{i\in I_{1}}\right) & \overset{\text{(i)}}{=}n_{1}^{-2}\sum_{i\in I_{1}}[X_{i}^{\top}(\hat{\pi}-\pi)]^{2}\mathbb{E}(\varepsilon_{i}^{2}\mid\{X_{i}\}_{i\in I_{1}})\nonumber \\ & \overset{\text{(ii)}}{\lesssim }n_{1}^{-2}\sum_{i\in I_{1}}[X_{i}^{\top}(\hat{\pi}-\pi)]^{2}\nonumber \\ & =n_{1}^{-1}(\hat{\pi}-\pi)^{\top}\hat{\Sigma}_{(1)}(\hat{\pi}-\pi)\nonumber \\ & \overset{\text{(iii)}}{=}n_{1}^{-1}O_{P}\left(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}\right)=o_{P}(n^{-1}), \end{align} where (i) follows by the fact that $\hat{\pi}$ only depends on $\{X_{i}\}_{i\in I_{1}}$ and $\mathbb{E}(\varepsilon_{i}\mid\{X_{i}\}_{i\in I_{1}})=0$, (ii) follows by the assumption of $\mathbb{E}(\varepsilon_{i}^{2}\mid X_{i})$ being bounded and (iii) follows by $(\hat{\pi}-\pi)^{\top}\hat{\Sigma}_{(1)}(\hat{\pi}-\pi)=O_{P}\left(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}\right)$ from Lemma (ref). By Lemma (ref), with probability approaching one, $\hat{\pi}$ is well defined and thus $\|\hat{\Sigma}_{(1)}\hat{\pi}-e_{1}\|_{\infty}\leq\lambda_{1}\lesssim\varepsilon_{n}$. By Lemma (ref) and $\xi_{1}>1/2$, $\|\delta_{\gamma}\|_{1}=O_{P}\left(\varepsilon_{n}^{(2\xi_{1}-1)/(2\xi_{1}+1)}\right)$. Hence, \[ \left|(e_{1}^{\top}-\hat{\pi}^{\top}\hat{\Sigma}_{(1)})\delta_{\gamma}\right|\leq\|\hat{\Sigma}_{(1)}\hat{\pi}-e_{1}\|_{\infty}\|\delta_{\gamma}\|_{1}=O_{P}(\varepsilon_{n})\cdot O_{P}\left(\varepsilon_{n}^{(2\xi_{1}-1)/(2\xi_{1}+1)}\right)\overset{\text{(i)}}{=}o_{P}(n^{-1/2}), \] where (i) follows by $\xi_{1}>1/2$ and $\log p\ll n^{1/2-1/(4\xi_{1})}$. Moreover, by $\mathbb{E}\eta_{f,i}^{2}\lesssim p^{-2\xi_{1}}$, $\mathbb{E}_{n,1}(X_{i}^{\top}\hat{\pi})^{2}=O_{P}(1)$ and $p\gg n^{1/(2\min\{\xi_{1},\xi_{2}\})}\geq n^{1/(2\xi_{1})}$, we have \[ \left|\hat{\pi}^{\top}\mathbb{E}_{n,1}X_{i}\eta_{f,i}\right|\leq\sqrt{\mathbb{E}_{n,1}(X_{i}^{\top}\hat{\pi})^{2}}\cdot\sqrt{\mathbb{E}_{n,1}\eta_{f,i}^{2}}=\sqrt{O_{P}(1)}\cdot\sqrt{O_{P}(p^{-2\xi_{1}})}=o_{P}(n^{-1/2}). \] The above four displays imply ((ref)). \textcolor{black}{\textbf{Step 2:} show ((ref)) assuming $\xi_{1}<\xi_{2}$.} \textcolor{black}{Since $\max\{\xi_{1},\xi_{2}\}>1/2$ , $\xi_{2}>\xi_{1}$ implies $\xi_{2}>1/2$. The assumption of $\log p\ll n^{1/2-1/(4\max\{\xi_{1},\xi_{2}\})}$ and $\xi_{2}>\xi_{1}$ imply $\log p\ll n^{1/2-1/(4\xi_{2})}$. Notice that ((ref)) and ((ref)) still hold and thus \begin{align} \tilde{\theta}_{1}-\theta & =(e_{1}^{\top}-\hat{\pi}^{\top}\hat{\Sigma}_{(1)})\delta_{\gamma}+\pi^{\top}\mathbb{E}_{n,1}X_{i}\varepsilon_{i}+o_{P}(n^{-1/2})\nonumber \\ & =(e_{1}^{\top}-\pi^{\top}\hat{\Sigma}_{(1)})\delta_{\gamma}-\delta_{\pi}^{\top}\hat{\Sigma}_{(1)}\delta_{\gamma}+\pi^{\top}\mathbb{E}_{n,1}X_{i}\varepsilon_{i}+o_{P}(n^{-1/2}). \end{align}} \textcolor{black}{Then we observe that \begin{align} (e_{1}^{\top}-\pi^{\top}\hat{\Sigma}_{(1)})\delta_{\gamma} & =\left[1-(\pi_{1}\mathbb{E}_{n,1}D_{i}^{2}+\mathbb{E}_{n,1}\pi_{-1}^{\top}Z_{i}D_{i})\right]\delta_{\theta}-\pi_{1}\mathbb{E}_{n,1}u_{i}Z_{i}^{\top}\delta_{\beta}\nonumber \\ & \overset{\text{(i)}}{=}\left[1-\mathbb{E}_{n,1}\pi^{\top}X_{i}D_{i}\right]\delta_{\theta}+O_{P}\left(n^{-1/2}\mathbb{E}_{n,1}(Z_{i}^{\top}\delta_{\beta})^{2}\right)\nonumber \\ & \overset{\text{(ii)}}{=}\left[1-\mathbb{E}_{n,1}\pi^{\top}X_{i}D_{i}\right]\delta_{\theta}+o_{P}(n^{-1/2})\nonumber \\ & \overset{\text{(iii)}}{=}O_{P}(n^{-1/2})|\delta_{\theta}|+o_{P}(n^{-1/2}), \end{align} where (i) follows by the fact that $\delta_{\beta}$ only depends on $\{Z_{i}\}_{i\in I_{1}},\{(Z_{i},Y_{i},D_{i})\}_{i\in I_{2}}$ and \begin{eqnarray*} && \mathbb{E}[(\mathbb{E}_{n,1}u_{i}Z_{i}^{\top}\delta_{\beta})^{2}\mid\{Z_{i}\}_{i\in I_{1}},\{(Z_{i},Y_{i},D_{i})\}_{i\in I_{2}}] \\ &=&n_{1}^{-2}\sum_{i\in I_{1}}\mathbb{E}(u_{i}^{2}\mid Z_{i})(Z_{i}^{\top}\delta_{\beta})^{2}\lesssim n_{1}^{-2}\sum_{i\in I_{1}}(Z_{i}^{\top}\delta_{\beta})^{2}\lesssim n^{-1}\mathbb{E}_{n,1}(Z_{i}^{\top}\delta_{\beta})^{2} \end{eqnarray*} (due to $\mathbb{E}(u_{i}\mid Z_{i})=0$), (ii) follows by $\mathbb{E}_{n,1}(Z_{i}^{\top}\delta_{\beta})^{2}=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})=o_{P}(1)$ (due to Lemma (ref)) and (iii) follows by $1-\mathbb{E}_{n,1}\pi^{\top}X_{i}D_{i}=O_{P}(n^{-1/2})$; to see the last step, notice that $\mathbb{E}\pi^{\top}X_{i}D_{i}=1$ (due to $e_{1}=\Sigma \pi$) and we will show $\mathbb{E}\left(\pi^{\top}X_{i}D_{i}\right)^{2} \lesssim 1$. For sub-Gaussian $X_{i}$, both $\pi^{\top}X_{i}$ and $D_{i}$ are sub-Gaussian, which means that $\mathbb{E}\left(\pi^{\top}X_{i}D_{i}\right)^{2} \lesssim 1$; for bounded $X_{i}$, we have that $\mathbb{E}\left(\pi^{\top}X_{i}D_{i}\right)^{2} \lesssim \mathbb{E}\left(\pi^{\top}X_{i}\right)^{2} =\pi^{\top}\Sigma \pi \lesssim 1 $.} \textcolor{black}{Since $|\delta_{\theta}|\leq\|\delta_{\gamma}\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})=o_{P}(1)$ due to Lemma (ref), ((ref)) implies that \begin{equation} (e_{1}^{\top}-\pi^{\top}\hat{\Sigma}_{(1)})\delta_{\gamma}=O_{P}(n^{-1/2})|\delta_{\theta}|+o_{P}(n^{-1/2})=o_{P}(n^{-1/2}). \end{equation}} \textcolor{black}{We decompose \begin{equation} \delta_{\pi}^{\top}\hat{\Sigma}_{(1)}\delta_{\gamma}=\underset{T_{1}}{\underbrace{\delta_{\pi_{1}}\left(\mathbb{E}_{n,1}D_{i}^{2}\delta_{\theta}+\mathbb{E}_{n,1}D_{i}Z_{i}^{\top}\delta_{\beta}\right)}}+\underset{T_{2}}{\underbrace{\delta_{\pi_{-1}}^{\top}\left(\mathbb{E}_{n,1}D_{i}Z_{i}\delta_{\theta}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\right)}}. \end{equation}} \textcolor{black}{By Lemma (ref), $\|\mathbb{E}_{n,2}Z_{i}D_{i}\delta_{\theta}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\|_{\infty}=O_{P}(\varepsilon_{n})$, which means that \begin{align} T_{2} & =\delta_{\pi_{-1}}^{\top}\left(\mathbb{E}_{n,2}Z_{i}D_{i}\delta_{\theta}+\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\right)+\delta_{\pi_{-1}}^{\top}\left(\mathbb{E}_{n,1}Z_{i}D_{i}-\mathbb{E}_{n,2}Z_{i}D_{i}\right)\delta_{\theta}\nonumber \\ & \overset{\text{(i)}}{=}O_{P}\left(\varepsilon_{n}\|\delta_{\pi_{-1}}\|_{1}\right)+\delta_{\pi_{-1}}^{\top}\left(\mathbb{E}_{n,1}Z_{i}D_{i}-\mathbb{E}_{n,2}Z_{i}D_{i}\right)\delta_{\theta}\nonumber \\ & \overset{\text{(ii)}}{=}O_{P}\left(\varepsilon_{n}\|\delta_{\pi_{-1}}\|_{1}\right)+O_{P}\left(\|\delta_{\pi_{-1}}\|_{1}\cdot\varepsilon_{n}\cdot|\delta_{\theta}|\right)\overset{\text{(iii)}}{=}o_{P}(n^{-1/2}), \end{align} where (i) follows by Hölder's inequality, (ii) follows by Hölder's inequality and $\|\mathbb{E}_{n,1}Z_{i}D_{i}-\mathbb{E}_{n,2}Z_{i}D_{i}\|_{\infty}=O(\varepsilon_{n})$ (due to Lemma (ref)) and (iii) follows by $$|\delta_{\theta}|\leq\|\delta_{\gamma}\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{1}/(2\xi_{1}+1)})=o_{P}(1)$$ (due to $\xi_{1}>0$), $\xi_{2}>1/2$, $\|\delta_{\pi_{-1}}\|_{1}\leq\|\delta_{\pi}\|_{1}=O_{P}(\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)})$ (due to Lemma (ref)) and $\log p\ll n^{1/2-1/(4\xi_{2})}$.} \textcolor{black}{For $T_{1}$, we observe \begin{align*} T_{1} & =\delta_{\pi_{1}}\left(\mathbb{E}_{n,1}D_{i}^{2}\delta_{\theta}+\mathbb{E}_{n,1}D_{i}Z_{i}^{\top}\delta_{\beta}\right)\\ & =\delta_{\pi_{1}}\left(\mathbb{E}_{n,1}D_{i}^{2}\delta_{\theta}+\phi^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\right)+\delta_{\pi_{1}}\mathbb{E}_{n,1}u_{i}Z_{i}^{\top}\delta_{\beta} + \delta_{\pi_{1}}\mathbb{E}_{n,1}\eta_{g,i}Z_{i}^{\top}\delta_{\beta} \\ & \overset{\text{(i)}}{=}\delta_{\pi_{1}}\left(\mathbb{E}_{n,1}D_{i}^{2}\delta_{\theta}+\phi^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\right)+o_{P}(n^{-1/2})\\ & =\delta_{\pi_{1}}\left(\mathbb{E}_{n,2}D_{i}^{2}\delta_{\theta}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\right)+\delta_{\pi_{1}}\left(\mathbb{E}_{n,1}D_{i}^{2}-\mathbb{E}_{n,2}D_{i}^{2}\right)\delta_{\theta}\\ & \qquad+\delta_{\pi_{1}}(\phi-\hat{\phi})^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}+o_{P}(n^{-1/2})\\ & \overset{\text{(ii)}}{=}\delta_{\pi_{1}}\left(\mathbb{E}_{n,2}D_{i}^{2}\delta_{\theta}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\right)+o_{P}(n^{-1/2})+\delta_{\pi_{1}}(\phi-\hat{\phi})^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}+o_{P}(n^{-1/2}) \end{align*} where (i) follows by $\mathbb{E}_{n,1}u_{i}Z_{i}^{\top}\delta_{\beta}=o_{P}(n^{-1/2})$ as shown above in ((ref)) and $\mathbb{E}_{n,1}\eta_{g,i}^2 =O_{P}(p^{-2\xi_2}) =o_{P}(n^{-1}) $ (due to $p\gg n^{1/(2\xi_2)}$) and (ii) follows by $\mathbb{E}_{n,1}D_{i}^{2}-\mathbb{E}_{n,2}D_{i}^{2}=O_{P}(n^{-1/2})$ (due to the sub-Gaussianity of $D_{i}$). Notice that \begin{multline*} \left|\delta_{\pi_{1}}(\phi-\hat{\phi})^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\right|\leq|\delta_{\pi_{1}}|\cdot\sqrt{\mathbb{E}_{n,1}(Z_{i}^{\top}(\hat{\phi}-\phi))^{2}}\times\sqrt{\mathbb{E}_{n,1}(Z_{i}^{\top}\delta_{\beta})^{2}}\\ \overset{\text{(i)}}{\leq}O_{P}(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})\cdot O_{P}\left(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}\right)\cdot o_{P}(1)\overset{\text{(ii)}}{=}o_{P}(n^{-1/2}), \end{multline*} where (i) follows by $|\delta_{\pi_{1}}|\leq\|\delta_{\pi}\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})$ (due to Lemma (ref)), $(\hat{\phi}-\phi)^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\phi}-\phi)=O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)})$ (due to Lemma (ref)) and $\mathbb{E}_{n,1}(Z_{i}^{\top}\delta_{\beta})^{2}=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})=o_{P}(1)$ (due to Lemma (ref)) and (ii) follows by $\log p\ll n^{1/2-1/(4\xi_{2})}$.} \textcolor{black}{We also notice that since $\xi_{2}>1/2$, we have \begin{align*} \left|\delta_{\pi_{1}}\left(\mathbb{E}_{n,2}D_{i}^{2}\delta_{\theta}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}\right)\right| & \leq|\delta_{\pi}|\cdot|\mathbb{E}_{n,2}D_{i}^{2}\delta_{\theta}+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}\delta_{\beta}|\\ & \overset{\text{(i)}}{=}O_{P}(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})\cdot O_{P}(n^{-1/4})\overset{\text{(ii)}}{=}o_{P}(n^{-1/2}), \end{align*} where (i) follows by $|\mathbb{E}_{n,2}D_{i}^{2}(\hat{\theta}-\theta)+\hat{\phi}^{\top}\mathbb{E}_{n,1}Z_{i}Z_{i}^{\top}(\hat{\beta}-\beta)|=O_{P}(n^{-1/4})$ (due to Lemma (ref)) and $|\delta_{\pi_{1}}|\leq\|\delta_{\pi}\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})$ (due to Lemma (ref)) and (ii) follows by $\log p\ll n^{1/2-1/(4\xi_{2})}$.} \textcolor{black}{The above three displays imply $T_{1}=o_{P}(n^{-1/2})$, which together with ((ref)) and ((ref)) yields $\delta_{\pi}^{\top}\hat{\Sigma}_{(1)}\delta_{\gamma}=o_{P}(n^{-1/2})$. Then the desired result follows by ((ref)) and ((ref)). } \end{proof} \section{Proofs for Section (ref)} For the proof of results in Section (ref), again define $\varepsilon_{n}=\sqrt{\log(p)/n}$ and let $s_{0}\geq C\varepsilon_{n}^{-2/(2\xi_{2}+1)}$, and $\pi$ be coefficients of the least squares projection of $\alpha_0(X)$ on $\psi(X)$, satisfying \[ M-\Sigma\pi=\mathbb{E}[\psi(X)\{\alpha_0(X)-\psi(X)^{\top}\pi\}]=0. \] Suppose that $\alpha_0(\cdot)\in\mathcal{M}_{C,\xi_2}$ for some $C,\xi_2>0$. Then $\mathbb{E}(\alpha_0(X)-\psi(X)^{\top}\pi)^2\leq (C p^{-\xi_2})^2$. Moreover, $\alpha_0(\cdot)\in\mathcal{M}_{C,\xi_2}$ also implies that $\mathbb{E}(\alpha_0(X)-\psi(X)^{\top}\tilde{\pi})^2\leq (C s_{0}^{-\xi_2})^2$ for some $\tilde{\pi}\in\mathbb{R}^{p}$ such that for some $J_{0}\subset\{1,...,p\}$ with $|J_0|=s_0$ and $\tilde{\pi}_j=0$ for $j\notin J_0$. Therefore, $ \mathbb{E}(\psi(X)^{\top}\pi-\psi(X)^{\top}\tilde{\pi})^2\leq 2 (C p^{-\xi_2})^2+2 (C s^{-\xi_2})^2\leq 4(C s^{-\xi_2})^2 \lesssim \varepsilon_n^{2\xi_2/(2\xi_2+1)} $. Thus, \begin{equation} (\pi-\tilde{\pi})^{\top}\Sigma (\pi-\tilde{\pi}) \leq C \varepsilon_n^{4\xi_2/(2\xi_2+1)}. \end{equation} Since the eigenvalues of $\Sigma$ are bounded, we have \[ \left\Vert \pi-\tilde{\pi}\right\Vert _{2} \leq C\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}. \] We define $\pi_{\ast}$ as \begin{equation} \pi_{\ast}\in\arg\min_{v}\;(\pi-v)^{\top}\Sigma(\pi-v)+2\varepsilon_{n} \sum_{j\in J_{0}^{c}}|v_{j}|\text{.} \end{equation} Let $J$ be the vector of indices of nonzero elements of $\pi_{\ast}.$ \begin{lemma} If $\alpha_0(\cdot)\in \mathcal{M}_{C,\xi_2}$, then $\Vert\Sigma(\pi_{\ast}-\pi)\Vert_{\infty}\leq C\varepsilon_{n}.$ \end{lemma} \begin{proof} [Proof of Lemma (ref)] Let $e_{j}\in\mathbb{R}^{p}$ denote the $j$-th column of $I_{p}$. The first-order condition for $\pi^{\ast}$ implies that for $j\in J_{0}$, we have $e_{j}^{\top}\Sigma(\pi_{\ast}-\pi)=0$; for $j\in J_{0}^{c} $, we have that $e_{j}^{\top}\Sigma(\pi_{\ast}-\pi)+\varepsilon_{n}z_{j}=0$, where $z_{j}=\text{sign}(\pi_{\ast,j})$ if $\pi_{\ast,j}\neq0$ and $z_{j} \in\lbrack-1,1]$ if $\pi_{\ast,j}=0$. Therefore, for any $j$, we have that $|e_{j}^{\top}\Sigma(\pi_{\ast}-\pi)|\leq\varepsilon_{n}$. Hence, $\Vert\Sigma(\pi_{\ast}-\pi)\Vert_{\infty}\leq\varepsilon_{n}$. \end{proof} \begin{lemma}Let Assumption (ref) hold. If $\alpha_0(\cdot)\in \mathcal{M}_{C,\xi_2}$, then $$(\pi-\pi_{\ast})^{\top}\Sigma(\pi-\pi_{\ast})\leq C\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}.$$ \end{lemma} \begin{proof}[Proof of Lemma (ref)] By the definition of $\pi_{\ast}$, we have that by the bounded eigenvalues of $\Sigma$, \begin{align*} (\pi-\pi_{\ast})^{\top}\Sigma(\pi-\pi_{\ast})+\varepsilon_{n}\sum_{j\in J_{0}^{c}}|\pi_{\ast,j}| & \leq(\pi-\tilde{\pi})^{\top}\Sigma(\pi -\tilde{\pi})+\varepsilon_{n}\sum_{j\in J_{0}^{c}}|\tilde{\pi}_{j} |=(\pi-\tilde{\pi})^{\top}\Sigma(\pi-\tilde{\pi})\\ & \leq C\left\Vert \pi-\tilde{\pi}\right\Vert _{2}^{2}\leq C\varepsilon _{n}^{4\xi_{2}/(2\xi_{2}+1)}. \end{align*} \end{proof} \begin{lemma} Let Assumption (ref) hold. If $\alpha_0(\cdot)\in \mathcal{M}_{C,\xi_2}$, then $\left\vert J\right\vert \leq C\varepsilon_{n} ^{-2/(2\xi_{2}+1)}$. \end{lemma} \begin{proof}[Proof of Lemma (ref)] For all $j\in J\backslash J_{0}$ the first order conditions to Equation ((ref)) imply $|e_{j}^{\top}\Sigma(\pi_{\ast}-\pi )|=\varepsilon_{n}$. Therefore, It follows that \[ \sum_{j\in J\backslash J_{0}}\left( e_{j}^{\top}\Sigma(\pi_{\ast} -\pi)\right) ^{2}=\frac{1}{4}\varepsilon_{n}^{2}|J\backslash J_{0}|\text{.} \] In addition, \begin{align*} \sum_{j\in J\backslash J_{0}}\left( e_{j}^{\top}\Sigma(\pi_{\ast} -\pi)\right) ^{2} & \leq\sum_{j=1}^{p}\left( e_{j}^{\top}\Sigma (\pi_{\ast}-\pi)\right) ^{2}=(\pi_{\ast}-\pi)^{\top}\Sigma\left( \sum_{j=1}^{p}e_{j}e_{j}^{\top}\right) \Sigma(\pi_{\ast}-\pi)\\ & =(\pi_{\ast}-\pi)^{\top}\Sigma^{2}(\pi_{\ast}-\pi)\leq\lambda_{\max }(\Sigma)\{(\pi-\pi_{\ast})^{\top}\Sigma(\pi-\pi_{\ast})\}\leq C\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}\text{,} \end{align*} where the last inequality follows by Lemma (ref) and $\lambda_{\max}(\Sigma)\leq C.$ Combining the above two displays, we obtain \[ \frac{1}{4}\varepsilon_{n}^{2}|J\backslash J_{0}|\leq C\varepsilon_{n} ^{4\xi_{2}/(2\xi_{2}+1)}\text{.} \] Dividing through by $\varepsilon_{n}^{2}$ gives $|J\backslash J_{0}|\leq C\varepsilon_{n}^{-2/(2\xi_{2}+1)}$. Thus by $s_{0}\leq C\varepsilon _{n}^{-2/(2\xi_{2}+1)},$ \[ |J|=|J_{0}|+|J\backslash J_{0}|=s_{0}+|J\backslash J_{0}|\leq s_{0} +C\varepsilon_{n}^{-2/(2\xi_{2}+1)}\leq C\varepsilon_{n}^{-2/(2\xi_{2} +1)}. \] \end{proof} \begin{lemma} Let Assumption (ref) hold. If $\alpha_0(\cdot)\in \mathcal{M}_{C,\xi_2}$ with $\xi_{2}>1/2$, then $\Vert\pi_{\ast} -\pi\Vert_{1}\lesssim\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}$. \end{lemma} \begin{proof} [Proof of Lemma (ref)] By Lemma (ref) and $\xi_{2}>1/2$, we have that \[ \left\Vert \pi_{J_{0}^{c}}\right\Vert _{1}\leq C s_{0}^{1/2-\xi_{2}}\leq C\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}. \] Let $J_{1}=J\cup J_{0}$ note that $J\subset J_{1}$ and $J_{0}\subset J_{1}$ imply $J_{1}^{c}\subset J^{c}$ and $J_{1}^{c}\subset J_{0}^{c},$ so that \[ \left\Vert (\pi_{\ast})_{J_{1}^{c}}-\pi_{J_{1}^{c}}\right\Vert _{1}=\left\Vert \pi_{J_{1}^{c}}\right\Vert _{1}\leq\left\Vert \pi_{J_{0}^{c}}\right\Vert _{1}. \] Also, by Lemma (ref), \[ \left\vert J_{1}\right\vert \leq\left\vert J\right\vert +\left\vert J_{0}\right\vert \leq C\varepsilon_{n}^{-2/(2\xi_{2}+1)}+s_{0}\leq C\varepsilon_{n}^{-2/(2\xi_{2}+1)} \] Therefore we have \begin{align*} \Vert\pi_{\ast}-\pi\Vert_{1} & =\left\Vert (\pi_{\ast})_{J_{1}}-\pi_{J_{1} }\right\Vert _{1}+\left\Vert (\pi_{\ast})_{J_{1}^{c}}-\pi_{J_{1}^{c} }\right\Vert _{1}\leq\left\Vert (\pi_{\ast})_{J_{1}}-\pi_{J_{1}}\right\Vert _{1}+\left\Vert \pi_{J_{0}^{c}}\right\Vert _{1}\\ & \leq\sqrt{|J_{1}|}\left\Vert (\pi_{\ast})_{J_{1}}-\pi_{J_{1}}\right\Vert _{2}+C\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}\\ & \leq C\varepsilon_{n}^{-1/(2\xi_{2}+1)}\Vert\pi_{\ast}-\pi\Vert _{2}+C\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}\leq C\varepsilon_{n} ^{(2\xi_{2}-1)/(2\xi_{2}+1)}. \end{align*} \end{proof} \begin{lemma} Let Assumption (ref) hold. If $\alpha_0(\cdot)\in \mathcal{M}_{C,\xi_2}$, then $P\left(\Vert\hat{\Sigma}\pi_{\ast}-\Sigma\pi_{\ast}\Vert_{\infty } > C\varepsilon_n\right)=o_{P}(1)$ and $P\left(\Vert\hat{\Sigma}\pi-\Sigma\pi\Vert_{\infty} >C\varepsilon_n \right) =o_{P}(1)$, where $C>0$ is a constant. \end{lemma} \begin{proof}[Proof of Lemma (ref)] \textcolor{black}{The results follow by the same argument as in the proof of Lemma (ref). For sub-Gaussian $\psi(X_{i})$, we apply Lemma (ref). For bounded $\psi(X_{i})$, we apply Lemma (ref) for $\psi(X_{i})\alpha_{0}(X_{i})$ and Lemma (ref) for $\psi(X_{i})(\alpha_{0}(X_{i})-\psi(X_{i})^{\top}\pi_{\ast})$ and $\psi(X_{i})(\alpha_{0}(X_{i})-\psi(X_{i})^{\top}\pi)$. We omit the details for brevity. } \end{proof} Next let \[ \hat{\pi}=\arg\min_{\pi}\{-2\hat{M}^{\top}\pi+\pi^{\top}\hat{\Sigma} \pi+2r\left\Vert \pi\right\Vert _{1}\}, \] for $\hat{M}$ to be specified later in this appendix. Recall that $r=\kappa \varepsilon_n$. Let $\Delta=\hat{\pi}-\pi^{\ast}$. \begin{lemma}Let Assumption (ref) hold. Suppose that $\alpha_0(\cdot)\in \mathcal{M}_{C,\xi_2}$. Suppose that $\kappa>0$ is large enough. If $P\left(\left\Vert \hat{M}-M\right\Vert _{\infty }\leq C_0 \varepsilon_{n} \right)\geq 1-o(1)$, then for any $\hat{J}$ such that $(\pi^{\ast})_{\hat{J}^{c}}=0,$ with probability approaching one, \[ \Delta^{\top}\hat{\Sigma}\Delta\leq3r\Vert\Delta\Vert_{1}\text{\ and \ } \Vert\Delta_{\hat{J}^{c}}\Vert_{1}\leq3\Vert\Delta_{\hat{J}}\Vert_{1}. \] \end{lemma} \begin{proof}[Proof of Lemma (ref)] By the definition of the estimator, we have \[ \hat{\pi}^{\top}\hat{\Sigma}\hat{\pi}-2\hat{M}^{\top}\hat{\pi}+2r\Vert \hat{\pi}\Vert_{1}\leq\pi_{\ast}^{\top}\hat{\Sigma}\pi_{\ast}-2\hat {M}^{\top}\pi_{\ast}+4r\Vert\pi_{\ast}\Vert_{1}\text{.} \] Plugging $\hat{\pi}=\pi_{\ast}+\Delta$ into the above equation and rearranging the terms gives \begin{equation} \Delta^{\top}\hat{\Sigma}\Delta+2r\Vert\pi_{\ast}+\Delta\Vert_{1}\leq 2r\Vert\pi_{\ast}\Vert_{1}+2(\hat{M}-\hat{\Sigma}\pi_{\ast})^{\top}\Delta. \end{equation} By assumption, $P(\mathcal{A}_1)\geq 1-o(1)$, where $\mathcal{A}_1=\{\Vert\hat{M}-M\Vert_{\infty}\leq C_0 \varepsilon_n\}$. By Lemma (ref), $P(\mathcal{A}_2)\geq 1-o(1)$, where $\mathcal{A}_2=\{\Vert\hat{\Sigma}\pi_{\ast}-\Sigma\pi_{\ast}\Vert_{\infty} \leq C_1\varepsilon_{n}\}$. Then by Lemma (ref), $M=\Sigma\pi,$ and the triangle inequality, on the event $\mathcal{A}_1\bigcap\mathcal{A}_2$, \begin{align*} \Vert\hat{M}-\hat{\Sigma}\pi_{\ast}\Vert_{\infty} & \leq\Vert\hat{\Sigma} \pi_{\ast}-\Sigma\pi_{\ast}\Vert_{\infty}+\Vert\hat{M}-M\Vert_{\infty }+\left\Vert M-\Sigma\pi_{\ast}\right\Vert _{\infty}\\ & \leq (C_0+C_1)\varepsilon_{n}+\Vert M-\Sigma\pi\Vert_{\infty}+\Vert\Sigma (\pi_{\ast}-\pi)\Vert_{\infty}\leq (C_0+C_1+C)\varepsilon_n , \end{align*} Therefore, by H\"older's inequality we have $\left\vert (\hat{M}-\hat{\Sigma }\pi_{\ast})^{\top}\Delta\right\vert \leq\Vert\hat{M}-\hat{\Sigma}\pi_{\ast }\Vert_{\infty}\Vert\Delta\Vert_{1},$ so that on the event $\mathcal{A}_1\bigcap\mathcal{A}_2$, \begin{equation} \Delta^{\top}\hat{\Sigma}\Delta+2r\Vert\pi_{\ast}+\Delta\Vert_{1}\leq 2r\Vert\pi_{\ast}\Vert_{1}+C_2\varepsilon_{n}\Vert\Delta\Vert_{1}, \nonumber \end{equation} where $C_2=2(C_0+C_1+C)$. By $r=\kappa\varepsilon_n$ and $\kappa>C_2$, it follows that with probability approaching one \begin{equation} \Delta^{\top}\hat{\Sigma}\Delta+2r\Vert\pi_{\ast}+\Delta\Vert_{1}\leq 2r\Vert\pi_{\ast}\Vert_{1}+r\Vert\Delta\Vert_{1}. \end{equation} The triangle inequality implies $\Vert\pi_{\ast}\Vert_{1}=\Vert\pi_{\ast }+\Delta-\Delta\Vert_{1}\leq\Vert\pi_{\ast}+\Delta\Vert_{1}+\Vert\Delta \Vert_{1}$ so subtracting $2r\Vert\pi_{\ast}+\Delta\Vert_{1}$ from both sides gives the first conclusion. Next, since $\Delta^{\top}\hat{\Sigma}\Delta\geq0$ it also follows from Equation ((ref)) that $2r\Vert\pi_{\ast}+\Delta\Vert_{1}\leq 2r\Vert\pi_{\ast}\Vert_{1}+r\Vert\Delta\Vert_{1}$ with probability approaching one, so dividing through by $r$ gives \[ 2\Vert\pi_{\ast}+\Delta\Vert_{1}\leq2\Vert\pi_{\ast}\Vert_{1}+\Vert\Delta \Vert_{1}\text{.} \] It follows by $(\pi_{\ast})_{\hat{J}^{c}}=0$ that $\Vert\pi_{\ast}+\Delta \Vert_{1}=\Vert(\pi_{\ast})_{\hat{J}}+\Delta_{\hat{J}}\Vert_{1}+\Vert \Delta_{\hat{J}^{c}}\Vert_{1}\geq \|(\pi_{*})_{\hat{J}}\|_{1}-\|\Delta_{\hat{J}}\|_{1}+\|\Delta_{\hat{J}^{c}}\|_{1} $ and $\Vert\pi_{\ast}\Vert_{1}=\Vert(\pi_{\ast })_{\hat{J}}\Vert_{1}$. Substituting in the previous display then gives $$ 2\|(\pi_{*})_{\hat{J}}\|_{1}-2\|\Delta_{\hat{J}}\|_{1}+2\|\Delta_{\hat{J}^{c}}\|_{1}\leq 2\|(\pi_{*})_{\hat{J}}\|_{1}+ \|\Delta_{\hat{J}}\|_{1}+\|\Delta_{\hat{J}^{c}}\|_{1}. $$ The second conclusion follows. \end{proof} \begin{lemma}Let Assumption (ref) hold. Suppose that $\kappa>0$ is large enough and $n\gg (\log p)^{3+3/(2\xi_2)}$. If $P\left(\left\Vert \hat{M}-M\right\Vert _{\infty }\leq C_0 \varepsilon_{n} \right)\geq 1-o(1)$, then $$\Delta^{\top}\hat{\Sigma}\Delta=O_{P}( \varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}), \quad \left\Vert \Delta\right\Vert _{1}=O_{P}(\varepsilon_{n}^{(2\xi_{2}-1)/(2\xi_{2}+1)}) \mbox{ and } \left\Vert \Delta\right\Vert _{2}=O_{P}(\varepsilon _{n}^{2\xi_{2}/(2\xi_{2}+1)}).$$ \end{lemma} \begin{proof}[Proof of Lemma (ref)] We apply the argument in Step 1 of the proof of Lemma (ref) and obtain that for any $C_{0}>0$, there exists $C_{1}>0$ such that \begin{equation} P\left(\min_{|S|\leq C_{0} (n/\log p)^{1/(2\xi_{2}+1)}}\min_{\|v_{S^{c}}\|_{1}\leq3\|v_{S}\|_{1}}\frac{v'\hat{\Sigma}v}{\|v_{S}\|_{2}^{2}}\geq C_{1}\right)\geq1-o(1), \end{equation} By Lemma (ref), $|J| \lesssim \varepsilon_n^{-2/(2\xi_2+1)} \lesssim (n/\log p)^{1/(2\xi_{2}+1)}$. For $\hat{J}=J$ it follows from the above display and Lemma (ref) that with high probability \begin{align*} \left\Vert \Delta_{J}\right\Vert _{2}^{2} & \leq C\Delta^{\top}\hat {\Sigma}\Delta\leq Cr\left\Vert \Delta\right\Vert _{1}\leq Cr\left\Vert \Delta\right\Vert _{1}=Cr(\left\Vert \Delta_{J}\right\Vert _{1}+\left\Vert \Delta_{J}^{c}\right\Vert _{1})\leq Cr\left\Vert \Delta_{J}\right\Vert _{1}\\ & \leq Cr\sqrt{\left\vert J\right\vert }\left\Vert \Delta_{J}\right\Vert _{2}\leq Cr\varepsilon_{n}^{-1/(2\xi_{2}+1)}\left\Vert \Delta_{J}\right\Vert _{2}=C\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}\left\Vert \Delta_{J}\right\Vert _{2}. \end{align*} Dividing through by $\left\Vert \Delta_{J}\right\Vert _{2}$ then gives \[ \left\Vert \Delta_{J}\right\Vert _{2}\leq C\varepsilon _{n}^{2\xi_{2}/(2\xi_{2}+1)}. \] Plugging this back in the final expression in the previous inequality gives the first conclusion. For the second conclusion note that by Lemma (ref), \begin{align*} \left\Vert \Delta\right\Vert _{1} & =\left\Vert \Delta_{J^{c}}\right\Vert _{1}+\left\Vert \Delta_{J}\right\Vert _{1}\leq4\left\Vert \Delta _{J}\right\Vert _{1}\leq4\sqrt{\left\vert J\right\vert }\left\Vert \Delta _{J}\right\Vert _{2}\\ & \leq C\varepsilon_{n}^{-1/(2\xi_{2}+1)}\varepsilon _{n}^{2\xi_{2}/(2\xi_{2}+1)}=C\varepsilon_{n}^{(2\xi _{2}-1)/(2\xi_{2}+1)}. \end{align*} For the third conclusion let $N$ denote the indices corresponding to the largest $\left\vert J\right\vert $ entries in $\Delta_{J^{c}}$, so that $N\subset J^{c}$, $|N|=\left\vert J\right\vert $ and $|\Delta_{j}|\geq |\Delta_{k}|$ for any $j\in J^{c}\cap N$ and $k\in J^{c}\backslash N$. By Lemma (ref) for $\hat{J}=J\cup N$ it follows exactly as in second previous display that \[ \left\Vert \Delta_{\hat{J}}\right\Vert _{2}\leq C\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}. \] By Lemma 6.9 of buhlmann2011statistics and Lemma (ref), \[ \Vert\Delta_{\hat{J}^{c}}\Vert_{2}\leq(\left\vert J\right\vert )^{-1/2} \Vert\Delta_{\hat{J}^{c}}\Vert_{1}\leq(\left\vert J\right\vert )^{-1/2} 3\Vert\Delta_{\hat{J}}\Vert_{1}\leq3(\left\vert J\right\vert )^{-1/2} \sqrt{\left\vert J\right\vert }\Vert\Delta_{J}\Vert_{2}\leq C \varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)}. \] Therefore, by the triangle inequality, \[ \Vert\Delta\Vert_{2}\leq\Vert\Delta_{\hat{J}}\Vert_{2}+\Vert\Delta_{\hat {J}^{c}}\Vert_{2}\leq C \varepsilon_{n}^{2\xi_{2}/(2\xi _{2}+1)}, \] giving the third conclusion. \end{proof} \begin{lemma} Let Assumption (ref) hold. Suppose that $\alpha_0(\cdot)\in \mathcal{M}_{C,\xi_2}$. If $P\left(\left\Vert \hat{M}-M\right\Vert _{\infty }\leq C_0 \varepsilon_{n} \right)\geq 1-o(1)$, then $P\left(\left\Vert \hat{\Sigma}(\hat{\pi}-\pi)\right\Vert _{\infty}\leq C_1 \varepsilon_n\right)\geq 1-o(1)$ for some $C_1>0$. \end{lemma} \begin{proof}[Proof of Lemma (ref)] The Lasso first order conditions imply $\left\Vert \hat {\Sigma}\hat{\pi}-\hat{M}\right\Vert _{\infty}\leq r.$ By Lemma (ref), $\left\Vert \hat{\Sigma}\pi-\Sigma\pi\right\Vert _{\infty}\leq C\varepsilon_{n}$ with high probability. Then by the triangle inequality, with high probability \begin{align*} \left\Vert \hat{\Sigma}(\hat{\pi}-\pi)\right\Vert _{\infty} & \leq\left\Vert \hat{\Sigma}\hat{\pi}-\hat{M}\right\Vert _{\infty}+\left\Vert \hat {M}-M\right\Vert _{\infty}+\left\Vert M-\Sigma\pi\right\Vert _{\infty }+\left\Vert \left( \Sigma-\hat{\Sigma}\right) ^{\top}\pi\right\Vert _{\infty}\\ & \leq r + C_0\varepsilon_n +0 +C\varepsilon_n . \end{align*} \end{proof} \begin{lemma} Let Assumption (ref) hold. If $P\left(\left\Vert \hat{M}_1-M\right\Vert _{\infty }\leq C_0 \varepsilon_{n} \right)\geq 1-o(1)$, then $$(\hat{\pi }_{1}-\pi)^{\top}\hat{\Sigma}_1(\hat{\pi}_{1}-\pi)=O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)})=o_{P}(1).$$ \end{lemma} \begin{proof}[Proof of Lemma (ref)] By Lemma (ref), $(\pi-\pi_{*})^{\top}\Sigma(\pi-\pi_{*})\lesssim\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)} $. Since $\mathbb{E}(\pi-\pi_{*})^{\top}\hat{\Sigma}_{1}(\pi-\pi_{*})=(\pi-\pi_{*})^{\top}\Sigma(\pi-\pi_{*})$, Markov's inequality implies that $$ (\pi-\pi_{*})^{\top}\hat{\Sigma}_{1}(\pi-\pi_{*})=O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}). $$ By Lemma (ref) (applied to $\hat{\Sigma}_1$ and $\hat{\pi}_1$), $$(\hat{\pi}_{1}-\pi_{*})^{\top}\hat{\Sigma}_{1}(\hat{\pi}_{1}-\pi_{*})=O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}).$$ The result follows by the above two displays. \end{proof} \begin{comment} Consider $\tilde{\pi}$ from Lemma (ref) and let $\pi_{\ast}$ be as defined in Equation ((ref)) for $J_{0}=\left\{ j:\tilde{\pi}_{j} \neq0\right\} $. By the definition of $\pi_{\ast}$, we have \begin{align} (\pi-\pi_{\ast})^{\top}\Sigma(\pi-\pi_{\ast})+\varepsilon_{n}\sum_{j\in J_{0}^{c}}|\pi_{\ast,j}| & \leq(\pi-\tilde{\pi})^{\top}\Sigma(\pi -\tilde{\pi})+\varepsilon_{n}\sum_{j\in J_{0}^{c}}|\tilde{\pi}_{j}| \nonumber \\ &= (\pi-\tilde{\pi})^{\top}\Sigma(\pi-\tilde{\pi}) =o(1). \end{align} Let $J$ be the vector of indices of nonzero elements of $\pi_{\ast}.$ For all $j\in J\backslash J_{0}$ the first order conditions to Equation ((ref)) imply $|e_{j}^{\top}\Sigma(\pi_{\ast}-\pi)|=\varepsilon_{n}$. Therefore, it follows that \[ \sum_{j\in J\backslash J_{0}}\left( e_{j}^{\top}\Sigma(\pi_{\ast} -\pi)\right) ^{2}=\varepsilon_{n}^{2}|J\backslash J_{0}|\text{.} \] In addition \begin{align*} \sum_{j\in J\backslash J_{0}}\left( e_{j}^{\top}\Sigma(\pi_{\ast} -\pi)\right) ^{2} & \leq\sum_{j=1}^{p}\left( e_{j}^{\top}\Sigma(\pi _{\ast}-\pi)\right) ^{2}=(\pi_{\ast}-\pi)^{\top}\Sigma\left( \sum _{j=1}^{p}e_{j}e_{j}^{\top}\right) \Sigma(\pi_{\ast}-\pi)\\ & =(\pi_{\ast}-\pi)^{\top}\Sigma^{2}(\pi_{\ast}-\pi)\leq\lambda_{\max }(\Sigma)\{(\pi-\pi_{\ast})^{\top}\Sigma(\pi-\pi_{\ast})\}\leq o(1). \end{align*} Then from the previous equation $\varepsilon_{n}^{2}|J\backslash J_{0}|\leq C\delta_{n}^{2}$ implying $|J\backslash J_{0}|\leq C\delta_{n}^{2} \varepsilon_{n}^{-2}.$ By Lemma (ref) we also have $|J_{0}|\leq C\delta_{n} ^{2}\varepsilon_{n}^{-2},$ so by summing \[ \left\vert J\right\vert \leq C\delta_{n}^{2}\varepsilon_{n}^{-2}. \] For convenience we drop the $1$ subscript on $\hat{\pi}_{1}$ and $\hat{\Sigma }_{1}$ for the rest of the proof. Then by Lemma (ref), we have \[ \Delta^{\top}\hat{\Sigma}\Delta\leq3r\Vert\Delta\Vert_{1}=3r(\Vert \Delta_{J^{c}}\Vert_{1}+\Vert\Delta_{J}\Vert_{1})\leq12r\Vert\Delta_{J} \Vert_{1}\leq Cr\sqrt{\left\vert J\right\vert }\left\Vert \Delta _{J}\right\Vert _{2}\leq Cr\varepsilon_{n}^{-1}\delta_{n}\left\Vert \Delta _{J}\right\Vert _{2}, \] By Assumption (ref), $\left\Vert \Delta_{J}\right\Vert _{2}^{2}\leq C\Delta ^{\top}\hat{\Sigma}\Delta$ with probability approaching one. Then dividing through by $\left\Vert \Delta_{J}\right\Vert _{2}$ gives $\left\Vert \Delta_{J}\right\Vert _{2}\leq Cr\varepsilon_{n}^{-1}\delta_{n}$ with probability approaching one. Plugging this inequality in the previous Equation gives \[ \Delta^{\top}\hat{\Sigma}\Delta\leq C(r\varepsilon_{n}^{-1}\delta_{n})^{2} \] with probability approaching one. Therefore $\Delta^{\top}\hat{\Sigma} \Delta=O_{P}((r\varepsilon_{n}^{-1}\delta_{n})^{2}).$ Also, by Assumption (ref) $\delta_{n}\leq Cr\varepsilon_{n}^{-1}\delta_{n}$ for large enough $n,$ so by the Markov inequality and Equation ((ref)), \begin{align*} (\pi_{\ast}-\pi)^{\top}\hat{\Sigma}(\pi_{\ast}-\pi) & =O_{P}(\mathbb{E}[(\pi_{\ast }-\pi)^{\top}\hat{\Sigma}(\pi_{\ast}-\pi)])=O_{P}((\pi_{\ast}-\pi)^{\top }\Sigma(\pi_{\ast}-\pi))\\ & =O_{P}(\delta_{n}^{2})=O_{P}((r\varepsilon_{n}^{-1}\delta_{n})^{2}). \end{align*} Then by the Cauchy-Schwartz and triangle inequalities, \[ (\hat{\pi}-\pi)^{\top}\hat{\Sigma}(\hat{\pi}-\pi)\leq C\Delta^{\top} \hat{\Sigma}\Delta+C(\pi_{\ast}-\pi)^{\top}\hat{\Sigma}(\pi_{\ast}-\pi)\leq O_{P}((r\varepsilon_{n}^{-1}\delta_{n})^{2}). \] \end{comment} \begin{comment} \begin{proof}[Proof of Lemma (ref)] By $\alpha_0(X)\in\mathcal{B}$ there exists a sequence $\alpha_0_{k}(x)$ of finite dimensional linear combinations of $(\psi_{1}(x),\psi_{2}(x),...)$ such that $\left\Vert \alpha_0_{k}-\bar{\alpha }\right\Vert _{2}\rightarrow0.$ Since each $\alpha_0_{k}(x)$ is a finite dimensional linear combination there exists $p_{k}$ such that for $b^{k}(x)=(\psi_{1}(x),...,\psi_{p_{k}}(x))^{\top}$ we have $\alpha_0 _{k}(x)=b^{k}(x)^{\top}\gamma_{k}$ for some $\gamma_{k}.$ Let $\alpha _{k}(X)$ be the least square projection of $\alpha_0(X)$ on $b^{k}(X).$ Then by $\left\Vert \alpha_0-\alpha_{k}\right\Vert _{2}\leq$ $\left\Vert \alpha_0-\alpha_0_{k}\right\Vert _{2}$ it follows that \[ \left\Vert \alpha_0-\alpha_{k}\right\Vert _{2}\rightarrow0. \] Let $\delta_{k}=\left\Vert \alpha_0-\alpha_{k}\right\Vert _{2}.$ By $n/\log(p)\rightarrow\infty$ and $p\rightarrow\infty$ we can choose $k_{n}$ so that $p_{k_{n}}\leq p$ and \[ p_{k_{n}}\leq\delta_{k_{n}}^{2}\frac{n}{\log(p)}. \] Let $\delta_{n}=\delta_{k_{n}}$ and $\tilde{\pi}=(\pi_{k_{n}}^{\top },0^{\top})^{\top}$ where $\pi_{k_{n}}^{\top}$ are the coefficients for $\alpha_{k_{n}}.$ Then $\left\Vert \tilde{\pi}\right\Vert _{0}\leq p_{k_{n} }\leq\delta_{n}^{2}n/\log(p)$ and $\left\Vert \alpha_{n}-\bar{\alpha }\right\Vert_{2}\leq\left\Vert \alpha_{k_{n}}-\alpha_0\right\Vert_{2}$ so that \begin{align*} (\pi-\tilde{\pi})^{\top}\Sigma(\pi-\tilde{\pi}) & =\left\Vert \alpha _{n}-\alpha_{k_{n}}\right\Vert ^{2}_{2}=\left\Vert \alpha_{n}-\alpha_0 -(\alpha_{k_{n}}-\alpha_0)\right\Vert ^{2}_{2}\\ & \leq2\left\Vert \alpha_{n}-\alpha_0\right\Vert ^{2}_{2}+2\left\Vert \alpha_{k_{n}}-\alpha_0\right\Vert ^{2}_{2}\leq4\delta_{k_{n}}^{2}=4\delta _{n}^{2}. \end{align*} \end{proof} \end{comment} \begin{comment} \begin{lemma} If Assumptions (ref) and (ref)-(ref) are satisfied, then \[ \left\Vert \hat{\rho}-\rho_{n}\right\Vert _{2}=O_{P}(\varepsilon_{n}^{2\xi_{1}/(2\xi_{1}+1)}) \] and in addition if $\xi_{1}>1/2$ \textit{and }$\left\vert \gamma _{1}\right\vert \leq C$ for all $p$, then $\sup_{x\in X}\left\vert \hat{\rho}(x)\right\vert =O_{P}(1).$ \end{lemma} \begin{proof} Let $\rho^{\ast}(x)=\psi(x)^{\top}\gamma_{\ast}$, where $\gamma_{\ast}$ is defined analogously to $\pi_{\ast}$. By Lemma (ref), Assumptions (ref) and (ref) imply that $P(\|\hat{\mu}_{1}-\mu\|_{\infty}\geq C_{0}\varepsilon_{n})=o(1)$ for some constant $C_{0}>0$. By the bounded eigenvalues of $\Sigma$ bounded and Lemma (ref) (applied to $\hat{\gamma}$ and $\gamma_*$), \[ \left\Vert \hat{\rho}-\rho^{\ast}\right\Vert _{2}^{2}=(\hat{\gamma}-\gamma_{*})^{\top} \Sigma(\hat{\gamma}-\gamma_{*}) \leq C\left\Vert \hat{\gamma}-\gamma_{*}\right\Vert ^{2}_{2}=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)}). \] Also, by the Lemma (ref) (applied to $\gamma$ and $\gamma_*$), \[ \left\Vert \rho^{\ast}-\rho_{n}\right\Vert _{2}^{2}=(\gamma^{\ast} -\gamma)^{\top}\Sigma(\gamma^{\ast}-\gamma)=O(\varepsilon_{n}^{4\xi _{1}/(2\xi_{1}+1)}). \] The first conclusion then follows by the triangle inequality. Also, by Assumption (ref) and the proof of Lemma (ref), \[ \left\Vert \gamma\right\Vert _{1}\leq\left\vert \gamma_{1}\right\vert +C\leq C, \] so by the triangle inequality, Assumption (ref), and Lemmas (ref) and (ref), \[ \sup_{x\in X}\left\vert \hat{\rho}(x)\right\vert \leq C\left\Vert \hat{\gamma }\right\Vert _{1}\sup_{x\in X,1\leq j\leq p}\left\vert \psi_{j}(x)\right\vert \leq C(\left\Vert \hat{\gamma}-\gamma\right\Vert _{1}+\left\Vert \gamma\right\Vert _{1})=O_{P}(1). \] \end{proof} \end{comment} Before proving Theorem (ref) we first give an intermediate result that takes care of important remainders in a stochastic expansion of $\hat{\theta}$ when $\xi_{1}>1/2.$ Recall that $n_{\ell}$ denotes the number of elements of $I_{\ell}$ and define \begin{align*} \theta_{n} & :=\gamma^{\top}\Sigma\pi=\mu^{\top}\pi,\text\rho _{n}(x):=\psi(x)^{\top}\gamma,\text\alpha_{n}(x):=\psi(x)^{\top}\pi.\\ \hat{U} & :=\hat{\mu}_{1}-\hat{\Sigma}_{1}\gamma,\text\hat{R}:=\hat{M} _{1}-\hat{\Sigma}_{1}\pi,\\ \varUpsilon_{n}(w) & :=m(w,\rho_{n})-\theta_{n}+\alpha_{n}(x)[y-\rho_{n}(x)]. \end{align*} \begin{lemma} Let Assumptions (ref) and (ref)(i) hold. Then \[ \hat{\theta}_{1}=\theta_{n}+\frac{1}{n_{1}}\sum_{i\in I_{1}}\varUpsilon_{n} (W_{i})+o_{P}(n^{-1/2})=\theta_{n}+O_{P}(n^{-1/2}). \] \end{lemma} \begin{proof} [Proof of Lemma (ref)] By adding and subtracting terms and by linearity of $m(w,\rho)$ in $\rho,$ \begin{align} & \hat{\theta}_{1}-\theta_{n}=\frac{1}{n_{1}}\sum_{i\in I_{1}}\left\{ m(W_{i},\hat{\rho}_{2})+\hat{\alpha}_{1}(X_{i})[Y_{i}-\hat{\rho}_{2} (X_{i})]-\theta_{n}\right\} \\ & =\hat{M}_{1}^{\top}\hat{\gamma}_{2}+\hat{\pi}_{1}^{\top}(\hat{\mu} _{1}-\hat{\Sigma}_{1}\hat{\gamma}_{2})-\theta_{n}=\hat{\mu}_{1}^{\top} \pi-\theta_{n}+\hat{\mu}_{1}^{\top}(\hat{\pi}_{1}-\pi)+\hat{M}_{1}^{\top }\hat{\gamma}_{2}-\hat{\pi}_{1}^{\top}\hat{\Sigma}_{1}\hat{\gamma} _{2}\nonumber\\ & =\hat{\mu}_{1}^{\top}\pi-\theta_{n}+(\hat{U}+\hat{\Sigma}_{1} \gamma)^{\top}(\hat{\pi}_{1}-\pi)+\hat{M}_{1}^{\top}\hat{\gamma}_{2} -\hat{\pi}_{1}^{\top}\hat{\Sigma}_{1}\hat{\gamma}_{2}\nonumber\\ & =\hat{\mu}_{1}^{\top}\pi-\theta_{n}+\hat{U}^{\top}(\hat{\pi}_{1} -\pi)+\gamma^{\top}\hat{\Sigma}_{1}(\hat{\pi}_{1}-\pi)+\hat{M}_{1}^{\top }\hat{\gamma}_{2}-\pi^{\top}\hat{\Sigma}_{1}\hat{\gamma}_{2}-(\hat{\pi} _{1}-\pi)^{\top}\hat{\Sigma}_{1}\hat{\gamma}_{2}\nonumber\\ & =\hat{\mu}_{1}^{\top}\pi-\theta_{n}+\hat{U}^{\top}(\hat{\pi}_{1} -\pi)+(\gamma-\hat{\gamma}_{2})^{\top}\hat{\Sigma}_{1}(\hat{\pi}_{1} -\pi)+\hat{R}^{\top}\hat{\gamma}_{2}\nonumber\\ & =\hat{M}_{1}^{\top}\gamma+\pi^{\top}(\hat{\mu}_{1}-\hat{\Sigma} _{1}\gamma)-\theta_{n}+\hat{R}^{\top}(\hat{\gamma}_{2}-\gamma)+\hat {U}^{\top}(\hat{\pi}_{1}-\pi)+(\gamma-\hat{\gamma}_{2})^{\top}\hat{\Sigma }_{1}(\hat{\pi}_{1}-\pi)\nonumber\\ & =\frac{1}{n}\sum_{i=1}^{n}\varUpsilon_{n}(W_{i})+T_{2}+T_{3}+T_{1}\text{,} \nonumber \end{align} where $$ T_{1} =(\gamma-\hat{\gamma}_{2})^{\top}\hat{\Sigma}_{1}(\hat{\pi} _{1}-\pi),\text{ }T_{2}=\hat{R}^{\top}(\hat{\gamma}_{2}-\gamma),\text{ }T_{3}=\hat{U}^{\top}(\hat{\pi}_{1}-\pi). $$ Since $\hat{M}_{1}=\mathbb{E}_{n,1} m(W_{i},\psi)$ and entries of $m(W_{i},\psi)$ are sub-Gaussian, the exponential-tail of sub-Gaussian variables and union bound yield $P(\Vert\hat{M}_{1}-M\Vert_{\infty } > C_{0}\varepsilon_n)=o_{P}(1)$ for some constant $C_{0}>0$. By Assumption (ref)(i) and Lemmas (ref) and (ref) (applied to $\hat{\gamma},$ $\gamma_{\ast},$ $\gamma$ in place of $\hat{\pi},$ $\pi_{\ast}$, and $\pi$) and the triangle inequality, $\left\Vert \hat{\gamma }_{2}-\gamma\right\Vert _{1}=O_{P}(\varepsilon_{n} ^{(2\xi_1-1)/(2\xi_1+1)}).$ Then by Lemma (ref) and H\"older's inequality, $$ \left\vert T_{1}\right\vert \leq\left\Vert \hat{\Sigma}_{1}(\hat{\pi} _{1}-\pi)\right\Vert _{\infty}\left\Vert \hat{\gamma}_{2}-\gamma\right\Vert _{1}=O_{P}(\varepsilon_n)O_{P}(\varepsilon_{n}^{(2\xi_1-1)/(2\xi_1+1)}) =o_{P}(n^{-1/2}), $$ where the last step follows by $n\gg (\log p)^{(2\xi_1)/(\xi_1-1/2)} $. Also, by Lemma (ref), \begin{align} \left\Vert \hat{R}\right\Vert _{\infty} & \leq\left\Vert \hat{M} _{1}-M\right\Vert _{\infty}+\left\Vert (\Sigma-\hat{\Sigma}_{1})\pi\right\Vert _{\infty}+\left\Vert M-\Sigma\pi\right\Vert _{\infty}\\ & =O_{P}(\varepsilon_{n})+O_{P}(\varepsilon_{n})+0=O_{P}(\varepsilon _{n}).\nonumber \end{align} Therefore it follows that $$ \left\vert T_{2}\right\vert \leq\left\Vert \hat{R}\right\Vert _{\infty }\left\Vert \hat{\gamma}_2-\gamma\right\Vert _{1}=O_{P}(\varepsilon_{n} )O_{P}(\varepsilon_{n}^{(2\xi_1-1)/(2\xi_1+1)})=o_{P}(n^{-1/2}), $$ where the last step follows by $n\gg (\log p)^{(2\xi_1)/(\xi_1-1/2)} $. Next, note that for $\varepsilon_{i}=Y_{i}-\rho_{0}(X_{i})$ and $d_{i} =\rho_{0}(X_{i})-\rho_{n}(X_{i}),$ \[ T_{3}=T_{31}+T_{32},\text{ }T_{31}=\frac{1}{n_{1}}\sum_{i\in I_{1}} \varepsilon_{i}\{\psi(X_{i})^{\top}(\hat{\pi}_{1}-\pi)\},\text{ }T_{32} =\frac{1}{n_{1}}\sum_{i\in I_{1}}d_{i}\{\psi(X_{i})^{\top}(\hat{\pi}_{1} -\pi)\}. \] For $\tilde{X}=\{X_{i}|i\in I_{1}\}$ we have $\mathbb{E}[\varepsilon_{i}|\tilde{X}]=0$, so that \[ \mathbb{E}[T_{31}|\tilde{X}]=\frac{1}{n_{1}}\sum_{i\in I_{1}}\mathbb{E}[\varepsilon_{i} |\tilde{X}]\{\psi(X_{i})^{\top}(\hat{\pi}_{1}-\pi)\}=0. \] Also, by Lemma (ref) and the assumption of bounded $Var(Y\mid X)$, we have \[ Var(T_{31}|\tilde{X})=\frac{1}{n_{1}^{2}}\sum_{i\in I_{1}}Var(Y_{i} |X_{i})\{\psi(X_{i})^{\top}(\hat{\pi}_{1}-\pi)\}^{2}\leq\frac{C}{n_{1}} (\hat{\pi}_{1}-\pi)^{\top}\hat{\Sigma}_{1}(\hat{\pi}_{1}-\pi)=o_{P} (1/n_{1}), \] so that $T_{31}=o_{P}(n^{-1/2}).$ Furthermore by the Cauchy-Schwartz and Markov's inequalities and Lemma (ref), $$ \left\vert T_{32}\right\vert \leq\sqrt{\frac{1}{n_{1}}\sum_{i\in I_{1}}d_{i}^{2}}\sqrt{(\hat{\pi}_{1}-\pi)^{\top}\hat{\Sigma}_{1}(\hat{\pi} _{1}-\pi)} =O_{P}(\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2})\cdot o_P(1)=o_{P}(n^{-1/2}), $$ where the last step follows by $\|\rho_0-\rho_n\|_2 \lesssim p^{-\xi_1}$ (due to $\rho_{0}\in \mathcal{M}_{C,\xi_{1}}$) and $p\gg n^{1/(2\xi_1)}$. The conclusion then follows by the triangle inequality. \end{proof} We also give a result corresponding to Lemma (ref) for the case where $\xi _{1}\xi_{2}>1/4.$ \begin{lemma} Let Assumptions (ref) and (ref)(ii) hold. Then \[ \hat{\theta}_{1}=\theta_{n}+\frac{1}{n_{1}}\sum_{i\in I_{1}}\varUpsilon_{n} (W_{i})+o_{P}(n^{-1/2})=\theta_{n}+O_{P}(n^{-1/2}). \] \end{lemma} \begin{proof} [Proof of Lemma (ref)] We consider the same remainder decomposition as the proof of Lemma (ref) with $T_{1}=(\gamma-\hat{\gamma}_{2})^{\top}\hat{\Sigma}_{1}(\hat{\pi}_{1}-\pi),$ $T_{2}=\hat{R}^{\top}(\hat{\gamma}_{2}-\gamma),$ $T_{3}=\hat{U}^{\top }(\hat{\pi}_{1}-\pi).$ Note that \begin{align*} T_{1} & =\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\rho}_{2}(X_{i})-\rho _{n}(X_{i})][\hat{\alpha}_{1}(X_{i})-\alpha_{n}(X_{i})],\\ T_{2} & =\frac{1}{n_{1}}\sum_{i\in I_{1}}[m(W_{i},\hat{\rho}_{2}-\rho _{n})+\alpha_{n}(X_{i})\{\hat{\rho}_{2}(X_{i})-\rho_{n}(X_{i})\}],\\ T_{3} & =\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}(X_{i})-\alpha _{n}(X_{i})]\{Y_{i}-\rho_{n}(X_{i})\}. \end{align*} We consider first $T_{1}$. By Lemmas (ref) and (ref) and the Markov inequality \begin{align*} & \frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{1}(X_{i})-\alpha_{n} (X_{i})]^{2}\\ & =(\hat{\pi}_{1}-\pi)^{\top}\hat{\Sigma}_{1}(\hat{\pi}_{1}-\pi)\leq 2(\hat{\pi}_{1}-\pi_*)^{\top}\hat{\Sigma}_{1}(\hat{\pi}_{1}-\pi_*)+2(\pi-\pi_{\ast})^{\top}\hat{\Sigma }_{1}({\pi}-\pi_{\ast})\\ & =O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2} +1)})+O_{P}(\mathbb{E}[(\pi-\pi_{\ast})^{\top}\hat{\Sigma}_{1}({\pi}-\pi_{\ast })])\\ & =O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2} +1)})+O_{P}((\pi-\pi_{\ast})^{\top}\Sigma({\pi}-\pi_{\ast} ))=O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)}). \end{align*} Similarly, \begin{align*} &\mathbb{E}[\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\gamma}_{2}'\psi(X_{i})-\gamma' \psi(X_{i})]^{2}|\{W_{i}\}_{i\in I_{2}}] \\ & =(\hat{\gamma}_{2}-\gamma)^{\top}\Sigma (\hat{\gamma}_{2}-\gamma)\leq 2(\hat{\gamma}_{2}-\gamma_*)^{\top}\Sigma(\hat{\gamma}_{2}-\gamma_*) +2(\gamma-\gamma_{\ast})^{\top}\Sigma(\gamma-\gamma_{\ast})\\ & \lesssim \left\Vert \hat{\gamma}_{2}-\gamma_*\right\Vert _{2}^{2}+\varepsilon_{n}^{4\xi _{1}/(2\xi_{1}+1)}=O_{P}(\varepsilon_{n}^{4\xi _{1}/(2\xi_{1}+1)}). \end{align*} It then follows by the conditional Markov inequality that \[ \frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\rho}_{2}(X_{i})-\rho_n(X_{i})]^{2}=\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\gamma}_{2}'\psi(X_{i})-\gamma'\psi(X_{i})]^{2}=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi _{1}+1)}). \] Then by the Cauchy-Schwartz inequality, \begin{align*} \left\vert T_{1}\right\vert & \leq\{\frac{1}{n_{1}}\sum_{i\in I_{1}} [\hat{\rho}_{2}(X_{i})-\rho_{n}(X_{i})]^{2}\}^{1/2}\{\frac{1}{n_{1}} \sum_{i\in I_{1}}[\hat{\alpha}_{1}(X_{i})-\alpha_{n}(X_{i})]^{2}\}^{1/2}\\ & =O_{P}(\varepsilon_{n}^{2\xi_{1}/(2\xi_{1} +1)+2\xi_{2}/(2\xi_{2}+1)})\overset{\text{(i)}}{=}o_{P}(1/\sqrt{n}), \end{align*} where (i) follows by the assumption of $n^{\xi_1\xi_2-1/4}\gg (\log p)^{2\xi_1\xi_2+(\xi_1+\xi_2)/2}$. Also \begin{align*} T_{2} & =T_{21}+T_{22},\textT_{21}=\frac{1}{n_{1}}\sum_{i\in I_{1} }[m(W_{i},\hat{\rho}_{2}-\rho_{n})+\alpha_0(X_{i})\{\hat{\rho}_{2} (X_{i})-\rho_{n}(X_{i})\}],\\ T_{22} & =\frac{1}{n_{1}}\sum_{i\in I_{1}}[\{\alpha_{n}(X_{i})-\alpha_0(X_{i})\}\{\hat{\rho}_{2}(X_{i})-\rho_{n}(X_{i})\}. \end{align*} Note that for $i\in I_1$, \[ \mathbb{E}[m(W_{i},\hat{\rho}_{2}-\rho_{n})+\alpha_0(X)\{\hat{\rho}_{2}(X)-\rho _{n}(X)\}|\{W_{i}\}_{i\in I_{2}}]=0. \] \begin{comment} Let $\delta_n=\hat{\rho}_{2}-\rho_{n}$. By Assumption (ref), $$ \mathbb{E}[|m(W_{i},\delta_n)|^{1+c_1}|\{W_{i}\}_{i\in I_{2}}] \leq C\left\Vert \delta_n \right\Vert _{2}^{c_2}=o_{P}(1). $$ Thus, $\mathbb{E}[|m(W_{i},\delta_n)||\{W_{i}\}_{i\in I_{2}}] =O_P( \|\delta_n\|_{2}^{c_2/(1+c_1)})$. By Rosenthal's inequality (Lemma 9.1 in Chapter 3 of gut2013probability), it follows that $$ \left|\sum_{i\in I_{1}}\left[m(W_{i},\delta_n)- \mathbb{E}(m(W_{i},\delta_n)\mid \{W_i\}_{i\in I_2}) \right]\right|^{1+c_1} =O_P\left(n \left\Vert \delta_n \right\Vert _{2}^{c_2} + n^{1+c_1} \|\delta_n\|_{2}^{c_2} \right). $$ Therefore, $$ \frac{1}{n_{1}}\sum_{i\in I_{1}}\left[m(W_{i},\delta_n)- \mathbb{E}(m(W_{i},\delta_n)\mid \{W_i\}_{i\in I_2}) \right]=O_P(\|\delta_n\|_{2}^{c_2/(1+c_1)}). $$ \end{comment} Let $\Delta_{2,\gamma}=\hat{\gamma}_2-\gamma$. Then $\hat{\rho}_{2}(X_{i})-\rho_{n}(X_{i})=\psi(X_{i})^{\top} \Delta_{2,\gamma}$. Notice that $\|\Delta_{\gamma,2}\|_{2}\leq\|\hat{\gamma}_{2}-\gamma_{*}\|_{2}+\|\gamma_{*}-\gamma\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{1}/(2\xi_{1}+1)})=o_{P}(1)$. By the assumption of bounded eigenvalues of $\mathbb{E}[\alpha_0(X)^{2} \psi(X)\psi(X)^{\top}]$ and $\mathbb{E} [m(W,\psi)m(W,\psi)^{\top}]$, we have that for $i\in I_1$, $$ \mathbb{E}[|m(W_{i},\hat{\rho}_{2}-\rho_{n})|^{2}|\{W_{i}\}_{i\in I_{2}}] =\Delta_{2,\gamma}'\mathbb{E}[m(W,\psi)m(W,\psi)^{\top}]\Delta_{2,\gamma} \lesssim\|\Delta_{2,\gamma}\|_{2}^{2}=o_{P}(1) $$ and $$ \mathbb{E}[\alpha_{0}(X_{i})^{2}\{\hat{\rho}_{2}(X_{i})-\rho_{n}(X_{i})\}^{2}|\{W_{i}\}_{i\in I_{2}}]=\Delta_{2,\gamma}'\mathbb{E}[\alpha_{0}(X)^2\psi(X)\psi(X)^{\top}]\Delta_{2,\gamma} \lesssim\|\Delta_{2,\gamma}\|_{2}^{2}=o_{P}(1). $$ Therefore it follows by independent observations that $\mathbb{E}[T_{21}^{2} |\{W_{i}\}_{i\in I_{2}}]=o_{P}(n^{-1}),$ so by the conditional Markov inequality $T_{21}=o_{P}(n^{-1/2}).$ Similarly to the result for $T_{1}$, we observe that $\alpha_0\in \mathcal{M}_{C,\xi_2} $ implies that $\mathbb{E}[(\alpha_0(X)-\alpha_n(X))^2]\lesssim p^{-2\xi_2} =o(n^{-1})$ due to $p\gg n^{1/(2\xi_2)}$. Then $\frac{1}{n_{1}}\sum_{i\in I_{1}} [\alpha_0(X_{i})-\alpha_{n}(X_{i})]^{2}=o_P(n^{-1/2})$ and thus \begin{align*} \left\vert T_{22}\right\vert & \leq\{\frac{1}{n_{1}}\sum_{i\in I_{1}} [\alpha_0(X_{i})-\alpha_{n}(X_{i})]^{2}\}^{1/2}\{\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\rho}_{2}(X_{i})-\rho_{n}(X_{i})]^{2}\}^{1/2}\\ & =o_{P}(n^{-1/2})O_{P} (\varepsilon_{n}^{2\xi_{1}/(2\xi_{1}+1)})=o_{P}(n^{-1/2}). \end{align*} Then by the triangle inequality $T_{2}=o_{P}(n^{-1/2})$. It also follows similarly to the proof of Lemma (ref) that $T_3=o_P(n^{-1/2})$. The conclusion then follows by the triangle inequality. \end{proof} \begin{comment} Similar to the proof of Lemma (ref), note that for $\varepsilon_{i}=Y_{i}-\rho_{0}(X_{i})$ and $d_{i} =\rho_{0}(X_{i})-\rho_{n}(X_{i})$, we have $T_{3}=T_{31}+T_{32}$, where \[ T_{31}=\frac{1}{n_{1}}\sum_{i\in I_{1}} \varepsilon_{i}\{\psi(X_{i})^{\top}(\hat{\pi}_{1}-\pi)\}\text{ and }T_{32} =\frac{1}{n_{1}}\sum_{i\in I_{1}}d_{i}\{\psi(X_{i})^{\top}(\hat{\pi}_{1} -\pi)\}. \] The same argument as in the proof of Lemma (ref) yields $T_{31}=o_P(n^{-1/2})$. Furthermore by the Cauchy-Schwartz and Markov's inequalities and Lemma (ref), $$ \left\vert T_{32}\right\vert \leq\sqrt{\frac{1}{n_{1}}\sum_{i\in I_{1}}d_{i}^{2}}\sqrt{(\hat{\pi}_{1}-\pi)^{\top}\hat{\Sigma}_{1}(\hat{\pi} _{1}-\pi)} =O_{P}(\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2})\cdot o_P(1)=o_{P}(n^{-1/2}), $$ where the last step follows by $\|\rho_0-\rho_n\|_2 \lesssim p^{-\xi_1}$ (Assumption (ref)) and $p\gg n^{1/(2\xi_1)}$. $T_{3}=T_{31}+T_{32}$ for the same $T_{31}$ and $T_{32}$ given in Lemma (ref). Then $T_{31}=o_{P}(n^{-1/2})$ follows as in the proof of Lemma (ref) and by the Cauchy-Schwartz and Markov inequalities, Lemma (ref), and Assumption (ref), \begin{align*} \left\vert T_{32}\right\vert & \leq\sqrt{\frac{1}{n_{1}}\sum_{i\in I_{1} }d_{i}^{2}}\sqrt{(\hat{\pi}_{1}-\pi)^{\top}\hat{\Sigma}_{1}(\hat{\pi} _{1}-\pi)}=O_{P}(\sqrt{\mathbb{E}[\{d_{i}^{2}\}]})O_{P}((r/\varepsilon_{n} )\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})\\ & =O_{P}(\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2})O_{P}((r/\varepsilon _{n})\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})=o_{P}(n^{-1/2}). \end{align*} \end{comment} \begin{proof}[Proof of Theorem (ref)] By Lemma (ref) or Lemma (ref) (and the same results for $I_{2}$), we have $$ \hat{\theta} =\frac{n_{1}}{n}\hat{\theta}_{1}+\frac{n_{2}}{n}\hat{\theta }_{2}=\sum_{\ell=1}^{2}\frac{n_{\ell}}{n}\{\theta_{n}+\frac{1}{n_{\ell}} \sum_{i\in I_{\ell}}\varUpsilon_{n}(W_{i})+o_{P}(n^{-1/2})\}\\ =\theta_{n}+\frac{1}{n}\sum_{i=1}^{n}\varUpsilon_{n}(W_{i})+o_{P}(n^{-1/2} ). $$ Notice that $\theta_{n}=\rho'\Sigma\gamma=\mathbb{E}\alpha_{n}(X)\rho_{n}(X)=\mathbb{E}\alpha_{0}(X)\rho_{n}(X)$ since $\mathbb{E} \psi(X)(\alpha_{n}(X)-\alpha_{0}(X))=0$. By $\rho_{0}\in\mathcal{M}_{C,\xi_{1}}$, we have that $\mathbb{E}(\rho_{0}(X)-\rho_{n}(X))^{2}\lesssim p^{-2\xi_{1}}$ and thus \begin{align*} \left|\theta_{0}-\theta_{n}\right| & =\left|\mathbb{E}\alpha_{0}(X)\rho_{0}(X)-\mathbb{E}\alpha_{0}(X)\rho_{n}(X)\right|\\ & =\left|\mathbb{E}\alpha_{0}(X)(\rho_{0}(X)-\rho_{n}(X))\right|\\ & \leq\sqrt{\mathbb{E}\alpha_{0}(X)^{2}}\cdot\sqrt{\mathbb{E}(\rho_{0}(X)-\rho_{n}(X))^{2}}\lesssim p^{-\xi_{1}}\overset{\text{(i)}}{=}o(n^{-1/2}), \end{align*} where (i) follows by $p\gg n^{1/(2\xi_{1})}$. Hence, \[ \hat{\theta}-\theta_{0}=n^{-1}\sum_{i=1}^{n}\varUpsilon_{n}(W_{i})+o_{P}(n^{-1/2}). \] Define $\tilde{\varUpsilon}_{n}(w)=m(w,\rho_{n})-\theta_{n}+\alpha_{n}(x)[y-\rho_{0}(x)]$. Then by $p\gg n^{1/(2\xi_{1})}$, \begin{align*} \mathbb{E}\left|n^{-1}\sum_{i=1}^{n}[\varUpsilon_{n}(W_{i})-\tilde{\varUpsilon}_{n}(W_{i})]\right| & \leq \mathbb{E} |\varUpsilon_{n}(W_{i})-\tilde{\varUpsilon}_{n}(W_{i})|\\ & =\mathbb{E} |\alpha_{n}(X)[\rho_{n}(X)-\rho_{0}(X)]|\\ & \leq\sqrt{\mathbb{E}\alpha_{n}(X)^{2}}\cdot\sqrt{\mathbb{E}(\rho_{0}(X)-\rho_{n}(X))^{2}}\\ & \overset{\text{(i)}}{\lesssim}p^{-\xi_{1}}=o(n^{-1/2}), \end{align*} where (i) follows by $\sqrt{\mathbb{E}\alpha_{n}(X)^{2}}\leq\sqrt{\mathbb{E}\alpha_{0}(X)^{2}}+\sqrt{\mathbb{E}(\alpha_{n}(X)-\alpha_{0}(X))^{2}}=O(1)+O(p^{-\xi_{2}})$ (due to $\alpha_{0}\in\mathcal{M}_{C,\xi_{2}}$) and $\mathbb{E}(\rho_{0}(X)-\rho_{n}(X))^{2}\lesssim p^{-2\xi_{1}}$. Therefore, \[ \hat{\theta}-\theta_{0}=n^{-1}\sum_{i=1}^{n}\tilde{\varUpsilon}_{n}(W_{i})+o_{P}(n^{-1/2}). \] Recall that $\theta_{n}=\mathbb{E}\alpha_{0}(X)\rho_{n}(X)$ (shown above). Thus, \[ \mathbb{E}\tilde{\varUpsilon}_{n}(W)=\mathbb{E} m(W,\rho_{n})-\theta_{n}=\mathbb{E}\alpha_{0}(X)\rho_{n}(X)-\theta_{n}=0. \] We notice that \begin{align*} \mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2} & =\mathbb{E}[m(W,\rho_{n})-\theta_{n}]^{2}+\mathbb{E}\alpha_{n}(X)^{2}[Y-\rho_{0}(X)]^{2}\\ & =\mathbb{E} m(W,\rho_{n})^{2}-\theta_{n}^{2}+\mathbb{E}\alpha_{n}(X)^{2}[Y-\rho_{0}(X)]^{2}\\ & \leq \mathbb{E} m(W,\rho_{n})^{2}+\mathbb{E}\alpha_{n}(X)^{2}[Y-\rho_{0}(X)]^{2}\\ & \overset{\text{(i)}}{=}\mathbb{E} m(W,\rho_{n})^{2}+O(1)\mathbb{E}\alpha_{n}(X)^{2}=\mathbb{E} m(W,\rho_{n})^{2}+O(1), \end{align*} where (i) follows by the assumption that $\mathbb{E} |Y-\rho_{0}(X)|^{2+c}$ is bounded. Notice that $\mathbb{E}\rho_{0}(X)^{2}=\mathbb{E}\rho_{n}(X)^{2}+\mathbb{E}(\rho_{0}(X)-\rho_{n}(X))^{2}$ and $\mathbb{E}\rho_{0}(X)^{2}\leq EY^{2}$ is bounded. Thus, $\mathbb{E}\rho_{n}(X)^{2}=O(1)$. Since $\rho_{n}(X)=\psi(X)^{\top}\gamma$, we have that $\gamma^{\top}\Sigma\gamma=\mathbb{E}\rho_{n}(X)^{2}=O(1)$. By the bounded eigenvalue assumption of $\Sigma^{-1}$, we have $\|\gamma\|_{2}=O(1)$. Therefore, by Assumption (ref), \[ \mathbb{E} m(W,\rho_{n})^{2}=\gamma^{\top}\mathbb{E}[m(W,\psi)m(W,\psi)^{\top}]\gamma\lesssim\|\gamma\|_{2}^{2}=O(1). \] The above displays imply $\mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2}=O(1)$. By $\mathbb{E}\tilde{\psi}(W)=0$, we have $n^{-1}\sum_{i=1}^{n}\tilde{\varUpsilon}_{n}(W_{i})=O_{P}(n^{-1/2})$. Therefore, $\hat{\theta}-\theta_{0}=O_{P}(n^{-1/2})$. \end{proof} \begin{proof}[Proof of Theorem (ref)] \textcolor{black}{By Assumption (ref), \begin{align*} \mathbb{E} |\tilde{\varUpsilon}_{n}(W_{i})|^{2+c} & \lesssim \mathbb{E} |m(W,\rho_{n})|^{2+c}+\mathbb{E} |\alpha_{0}(X)|^{2+c}|Y-\rho_{0}(X)|^{2+c}\\ & \overset{\text{(i)}}{\lesssim}\mathbb{E} |m(W,\rho_{n})|^{2+c}+\mathbb{E} |\alpha_{0}(X)|^{2+c}\lesssim1, \end{align*} where (i) follows by Assumption (ref). Without loss of generality, we can assume that $c\in(0,1)$; if $c\geq1$, we can reduce $c$ and Assumption (ref) still holds. By Theorem (ref), $\mathbb{E}\tilde{\varUpsilon}_{n}(W)=0$. Notice that $\mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2}\geq \mathbb{E}\alpha_{0}(X)^{2}(Y-\rho_{0}(X))^{2}$, which is assumed to be bounded below by a positive constant. By a generalized Berry-Esseen bound (Theorem 6.3 in Chapter 7 of gut2013probability), it follows that \[ \sup_{x\in\mathbb{R}}\left|P\left(\frac{n^{-1/2}\sum_{i=1}^{n}\tilde{\varUpsilon}_{n}(W_{i})}{\sqrt{\mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2}}}\leq x\right)-\Phi(x)\right|\lesssim\frac{n\mathbb{E} |\tilde{\varUpsilon}_{n}(W)|^{2+c}}{n^{1+c/2}}\lesssim n^{-c/2}, \] where $\Phi(\cdot)$ is the cdf of the standard normal distribution $N(0,1)$. The above display implies that \[ \frac{n^{-1/2}\sum_{i=1}^{n}\tilde{\varUpsilon}_{n}(W_{i})}{\sqrt{\mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2}}}\overset{d}{\rightarrow}N(0,1). \] Notice that $\mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2}=\mathbb{E}[m(W_{i},\rho_{n})-\theta_{n}]^{2}+\mathbb{E}\alpha_{n}(X_{i})^{2}[Y_{i}-\rho_{0}(X_{i})]^{2}$. Define $\bar{V}=(n_{1}/n)\bar{V}_{1}+(n_{2}/n)\bar{V}_{2}$, where $\bar{V}_{\ell}=\frac{1}{n_{\ell}}\sum_{i\in I_{\ell}}[(m(W_{i},\rho_{n})-\theta_{n})^{2}+\alpha_{n}(X_{i})^{2}[Y_{i}-\rho_{0}(X_{i})]^{2}]$ for $\ell\in\{1,2\}$. Since $\mathbb{E} |\tilde{\varUpsilon}_{n}(W_{i})|^{2+c}\lesssim1$, the law of large numbers (e.g., Corollary 8.2 in Chapter 3 of gut2013probability) implies that $\bar{V}=\mathbb{E}\tilde{\varUpsilon}_{n}(W)^{2}+o_{P}(1)$. To show that $\hat{V}=n^{-1}\sum_{i=1}^{n}\tilde{\varUpsilon}_{n}(W_{i})^{2}+o_{P}(1)$, we need to show $\hat{V}_{\ell}-\bar{V}_{\ell}=o_{P}(1)$ for $\ell\in\{1,2\}$. We do so for $\ell=1$; the argument for $\ell=2$ is analogous. Notice that \begin{align} \hat{V}_{1}-\bar{V}_{1} & =\frac{1}{n_{1}}\sum_{i\in I_{1}}\left[\left(m(W_{i},\hat{\rho}_{2})-\hat{\theta}\right)^{2}-\left(m(W_{i},\rho_{n})-\theta_{n}\right)^{2}\right]\nonumber \\ & \quad+\frac{1}{n_{1}}\sum_{i\in I_{1}}\left[\hat{\alpha}_{2}(X_{i})^{2}[Y_{i}-\hat{\rho}_{2}(X_{i})]^{2}-\alpha_{n}(X_{i})^{2}[Y_{i}-\rho_{0}(X_{i})]^{2}\right]\nonumber \\ & =T_{1}+T_{2}+T_{3}+T_{4}+T_{5}, \end{align} where \begin{align*} T_{1} & =\frac{1}{n_{1}}\sum_{i\in I_{1}}\left[\left(m(W_{i},\hat{\rho}_{2})-\hat{\theta}\right)^{2}-\left(m(W_{i},\rho_{n})-\theta_{n}\right)^{2}\right]\\ T_{2} & =\frac{1}{n_{1}}\sum_{i\in I_{1}}\left(\hat{\alpha}_{2}(X_{i})^{2}-\alpha_{n}(X_{i})^{2}\right)[Y_{i}-\rho_{0}(X_{i})]^{2}\\ T_{3} & =\frac{2}{n_{1}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}(Y_{i}-\rho_{0}(X_{i}))(\rho_{0}(X_{i})-\rho_{n}(X_{i}))\\ T_{4} & =\frac{2}{n_{1}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}(Y_{i}-\rho_{0}(X_{i}))(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))\\ T_{5} & =\frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}(\rho_{0}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}. \end{align*}} \textcolor{black}{For $\ell\in\{1,2\}$, define $\delta_{\alpha,\ell}(X)=\hat{\alpha}_{\ell}(X)-\alpha_{n}(X)$ and $\delta_{\rho,\ell}(X)=\hat{\rho}_{\ell}(X)-\rho_{n}(X)$. We bound $T_{1}$,...,$T_{5}$ in five steps. } \textcolor{black}{\textbf{Step 1:} show $T_{1}=o_{P}(1)$.} \textcolor{black}{Let $\Delta_{\gamma,2}=\hat{\gamma}_{2}-\gamma$. Since $\Delta_{\gamma,2}$ is independent of $\{W_{i}\}_{i\in I_{1}}$, by $\delta_{\rho,2}(x)=\psi(x)^{\top}(\hat{\gamma}_{2}-\gamma)$ and Assumption (ref), it follows that $$ \mathbb{E}\left[\frac{1}{n_{1}}\sum_{i\in I_{1}}m(W_{i},\delta_{\rho,2})^{2}\mid\{W_{i}\}_{i\in I_{2}}\right]=\Delta_{\gamma,2}^{\top}\mathbb{E}\left[m(W,\psi)(W,\psi)^{\top}\right]\Delta_{\gamma,2} \lesssim\|\Delta_{\gamma,2}\|_{2}^{2}\overset{\text{(i)}}{=}o_{P}(1), $$ where (i) follows by $\|\Delta_{\gamma,2}\|_{2}=o_{P}(1)$ as argued in the proof of Lemma (ref). Thus, $\frac{1}{n_{1}}\sum_{i\in I_{1}}m(W_{i},\delta_{\rho,2})^{2}=o_{P}(1)$. Since $\hat{\theta}-\theta_{n}=o_{P}(1)$ (shown in the proof of Theorem (ref)), we have that $\frac{1}{n_{1}}\sum_{i\in I_{1}}[m(W_{i},\delta_{\rho,2})+\theta_{n}-\hat{\theta}]^{2}=o_{P}(1)$. Therefore, \begin{multline*} T_{1}=\frac{1}{n_{1}}\sum_{i\in I_{1}}\left(m(W_{i},\delta_{\rho,2})+\theta_{n}-\hat{\theta}\right)^{2}\\ +\frac{2}{n_{1}}\sum_{i\in I_{1}}\left(m(W_{i},\rho_{n})-\theta_{n}\right)\left(m(W_{i},\delta_{\rho,2})+\theta_{n}-\hat{\theta}\right)=o_{P}(1). \end{multline*}} \textcolor{black}{\textbf{Step 2:} show $T_{2}=o_{P}(1)$.} \textcolor{black}{Since $\mathbb{E}([Y_{i}-\rho_{0}(X_{i})]^{2}\mid X_{i})$ is bounded (due to the assumption of bounded $Var(Y\mid X)$), we have \begin{align*} & \mathbb{E}\left[\frac{1}{n_{1}}\sum_{i\in I_{1}}\left|\left(\hat{\alpha}_{2}(X_{i})^{2}-\alpha_{n}(X_{i})^{2}\right)[Y_{i}-\rho_{0}(X_{i})]^{2}\right|\mid\{W_{i}\}_{i\in I_{2}}\right]\\ & \lesssim\frac{1}{n_{1}}\sum_{i\in I_{1}}\left|\hat{\alpha}_{2}(X_{i})^{2}-\alpha_{n}(X_{i})^{2}\right|\\ & =\frac{1}{n_{1}}\sum_{i\in I_{1}}\left|(\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i}))(\hat{\alpha}_{2}(X_{i})+\alpha_{n}(X_{i}))\right|\\ & \leq\sqrt{\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i})]^{2}}\cdot\sqrt{\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})+\alpha_{n}(X_{i})]^{2}}. \end{align*}} \textcolor{black}{Let $\Delta_{\pi,2}=\hat{\pi}_{2}-\pi$ and note $\hat{\alpha}_{2}(X)-\alpha_{n}(X)=\psi(X)^{\top}\Delta_{\pi,2}$. Notice that \begin{equation} \mathbb{E}\left(\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i})]^{2}\mid\{W_{i}\}_{i\in I_{2}}\right)\lesssim\Delta_{\pi,2}^{\top}\Sigma\Delta_{\pi,2}\overset{\text{(i)}}{=}o_{P}(1), \end{equation} where (i) follows by $\|\Delta_{\pi,2}\|_{2}\leq\|\hat{\pi}_{2}-\pi_{*}\|_{2}+\|\pi_{*}-\pi\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})=o_{P}(1)$; Lemma (ref) gives $\|\hat{\pi}_{2}-\pi_{*}\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})$ and the proof of Lemma (ref) gives $(\pi_{*}-\pi)^{\top}\Sigma(\pi_{*}-\pi)=O(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})$, which implies $\|\pi_{*}-\pi\|_{2}=O(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})$. Therefore, the above two displays imply \[ |T_{2}|\leq\frac{1}{n_{1}}\sum_{i\in I_{1}}\left|\left(\hat{\alpha}_{2}(X_{i})^{2}-\alpha_{n}(X_{i})^{2}\right)[Y_{i}-\rho_{0}(X_{i})]^{2}\right|=o_{P}(1)\sqrt{\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})+\alpha_{n}(X_{i})]^{2}}. \]} \textcolor{black}{Since $[\hat{\alpha}_{2}(x)+\alpha_{n}(x)]^{2}=[\hat{\alpha}_{2}(x)-\alpha_{n}(x)+2\alpha_{n}(x)]^{2}\leq2[\hat{\alpha}_{2}(x)-\alpha_{n}(x)]^{2}+8\alpha_{n}(x)^{2}$ and $\alpha_{n}(x)=[\alpha_{n}(x)-\alpha_{0}(x)+\alpha_{0}(x)]^{2}\leq2[\alpha_{n}(x)-\alpha_{0}(x)]^{2}+2\alpha_{0}(x)^{2}$, we have \begin{eqnarray*} &&\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})+\alpha_{n}(X_{i})]^{2} \\ &\leq&\frac{2}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i})]^{2}+\frac{16}{n_{1}}\sum_{i\in I_{1}}[\alpha_{n}(X_{i})-\alpha_{0}(X_{i})]^{2} +\frac{16}{n_{1}}\sum_{i\in I_{1}}\alpha_{0}(X_{i})^{2}\overset{\text{(i)}}{=}O_{P}(1), \end{eqnarray*} where (i) follows by $\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i})]^{2}=o_{P}(1)$, $\mathbb{E}\frac{1}{n_{1}}\sum_{i\in I_{1}}[\alpha_{n}(X_{i})-\alpha_{0}(X_{i})]^{2}=\mathbb{E}[\alpha_{n}(X)-\alpha_{0}(X)]^{2}\lesssim p^{-2\xi_{2}}=o(1)$ and $\mathbb{E}\frac{1}{n_{1}}\sum_{i\in I_{1}}\alpha_{0}(X_{i})^{2}=\mathbb{E}\alpha_{0}(X)^{2}=O(1)$. The above two displays imply $T_{2}=o_{P}(1)$. } \textcolor{black}{\textbf{Step 3:} show $T_{3}=o_{P}(1)$.} \textcolor{black}{Let $v_{i}=Y_{i}-\mathbb{E}(Y_{i}\mid X_{i})=Y_{i}-\rho_{0}(X_{i})$. Notice that \begin{align} \mathbb{E}\left(T_{3}^{2}\mid\{X_{i}\}_{i\in I_{1}},\{W_{i}\}_{i\in I_{2}}\right) & =\frac{4}{n_{1}^{2}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{4}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}\mathbb{E}\left[v_{i}^{2}\mid\{X_{i}\}_{i\in I_{1}}\right]\nonumber \\ & \overset{\text{(i)}}{\lesssim}\frac{4}{n_{1}^{2}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{4}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}, \end{align} where (i) follows by Assumption (ref). Notice that \[ \mathbb{E}\left(n_{1}^{-1}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}\mid\{W_{i}\}_{i\in I_{2}}\right)=\hat{\pi}_{2}^{\top}\Sigma\hat{\pi}_{2}\overset{\text{(i)}}{=}O_{P}(1), \] where (i) follows by $\|\hat{\pi}_{2}-\pi\|_{2}=\|\Delta_{\pi,2}\|_{2}=o_{P}(1)$ (shown in Step 2) and $\|\pi\|_{2}=O(1)$. Thus, $n_{1}^{-1}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}=O_{P}(1)$, which means that $\max_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}\leq\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}=O_{P}(n)$. Hence, \begin{align} & \mathbb{E}\left(T_{3}^{2}\mid\{X_{i}\}_{i\in I_{1}},\{W_{i}\}_{i\in I_{2}}\right)\nonumber \\ & \lesssim\frac{4}{n_{1}^{2}}\left(\sum_{i\in I_{1}}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}\right)\cdot\left(\max_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{4}\right)\nonumber \\ & =\frac{4}{n_{1}^{2}}\left(\sum_{i\in I_{1}}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}\right)\cdot O_{P}(n^{2})\nonumber \\ & =O_{P}(1)\cdot\sum_{i\in I_{1}}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}\overset{\text{(i)}}{=}O_{P}(np^{-2\xi_{1}})\overset{\text{(ii)}}{=}o_{P}(1), \end{align} where (i) follows by $\mathbb{E}\sum_{i\in I_{1}}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}=n_{1}\mathbb{E}(\rho_{0}(X)-\rho_{n}(X))^{2}\lesssim np^{-2\xi_{1}}$ and (ii) follows by $p\gg n^{1/(2\xi_{1})}$. Therefore, $T_{3}=o_{P}(1)$. } \textcolor{black}{\textbf{Step 4:} show $T_{4}=o_{P}(1)$.} \textcolor{black}{Similar to ((ref)), we notice that \begin{equation} \mathbb{E}\left(T_{4}^{2}\mid\{X_{i}\}_{i\in I_{1}},\{W_{i}\}_{i\in I_{2}}\right)\lesssim\frac{4}{n_{1}^{2}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{4}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}. \end{equation}} \textcolor{black}{Suppose that Assumption (ref)(i) holds. Notice that $\rho_{n}(x)-\hat{\rho}_{2}(x)=\psi(x)^{\top}(\gamma-\hat{\gamma}_{2})$. As argued in the proof of Lemma (ref), $\|\hat{\gamma}_{2}-\gamma\|_{1}=O_{P}(\varepsilon_{n}^{(2\xi_{1}-1)/(2\xi_{1}+1)})$. Since $\xi_{1}>1/2$, we have $\|\hat{\gamma}_{2}-\gamma\|_{1}=o_{P}(1)$. Due to the boundedness of $\|\psi(x)\|_{\infty}$, we have that $\max_{i\in I_{1}}|\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i})|\leq\|\hat{\gamma}_{2}-\gamma\|_{1}\cdot\max_{i\in I_{1}}\|\psi(X_{i})\|_{\infty}=o_{P}(1)$. Thus, by ((ref)), we have \[ \mathbb{E}\left(T_{4}^{2}\mid\{X_{i}\}_{i\in I_{1}},\{W_{i}\}_{i\in I_{2}}\right)=o_{P}(1)\cdot\frac{4}{n_{1}^{2}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{4}\overset{\text{(i)}}{\lesssim}o_{P}(1)\cdot\frac{4}{n_{1}^{2}}\cdot O_{P}(n^{2})=o_{P}(1), \] where (i) follows by $\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{4}\leq\left(\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}\right)$ and $\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}=O_{P}(n)$ (shown in Step 3). Thus, $T_{4}=o_{P}(1)$ under Assumption (ref). } \textcolor{black}{Suppose that Assumption (ref)(ii) holds. Similar to ((ref)), we have \begin{eqnarray} T_{4}^{2}&=&O_{P}(1)\cdot\frac{4}{n_{1}^{2}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{4}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2} \\ &\overset{\text{(i)}}{=}&O_{P}(1)\cdot\frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}, \nonumber \end{eqnarray} where (i) follows by $\max_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}=O_{P}(n)$ (shown in Step 3). } \textcolor{black}{Moreover, by $$\hat{\rho}_{2}(x)-\rho_{n}(x)=\psi(x)^{\top}\Delta_{\gamma,2},$$ we have $\frac{1}{n_{1}}\sum_{i\in I_{1}}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}=\Delta_{\gamma,2}^{\top}\hat{\Sigma}_{2}\Delta_{\gamma_{2}}$. Since $$\mathbb{E}[\Delta_{\gamma,2}^{\top}\hat{\Sigma}_{2}\Delta_{\gamma_{2}}\mid\{W_{i}\}_{i\in I_{2}}]=\Delta_{\gamma,2}^{\top}\Sigma\Delta_{\gamma,2}\lesssim\|\Delta_{\gamma,2}\|_{2}^{2}$$ and $\|\Delta_{\gamma,2}\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{1}/(2\xi_{1}+1)})$ (shown in Step 1), we have that $\frac{1}{n_{1}}\sum_{i\in I_{1}}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}=O_{P}(\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})$. Similarly, $\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i})]^{2}=O_{P}(\|\Delta_{\pi,2}\|_{2}^{2})$. By $\|\Delta_{\pi,2}\|_{2}=O_{P}(\varepsilon_{n}^{2\xi_{2}/(2\xi_{2}+1)})$ (shown in Step 2), we have $\frac{1}{n_{1}}\sum_{i\in I_{1}}[\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i})]^{2}=O_{P}(\varepsilon_{n}^{4\xi_{2}/(2\xi_{2}+1)})$. Hence, \begin{align} & \frac{1}{n_{1}}\sum_{i\in I_{1}}(\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i}))^{2}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}\nonumber \\ & \leq\frac{1}{n_{1}}\left(\sum_{i\in I_{1}}|\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i})|\cdot|\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i})|\right)^{2}\nonumber \\ & \overset{\text{(i)}}{\leq}\frac{1}{n_{1}}\left(\sum_{i\in I_{1}}|\hat{\alpha}_{2}(X_{i})-\alpha_{n}(X_{i})|^{2}\right)\cdot\left(\sum_{i\in I_{1}}|\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i})|^{2}\right)\nonumber \\ & =O(n^{-1})\cdot O_{P}\left(n^{2}\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)+4\xi_{2}/(2\xi_{2}+1)}\right)\overset{\text{(ii)}}{=}o_{P}(1), \end{align} where (i) follows by Cauchy-Schwarz inequality and (ii) follows by $n^{\xi_{1}\xi_{2}-1/4}\gg(\log p)^{2\xi_{1}\xi_{2}+(\xi_{1}+\xi_{2})/2}$ and $\xi_{1}\xi_{2}>1/4$ (due to Assumption (ref)(ii)). We also notice that \begin{align} & \frac{1}{n_{1}}\sum_{i\in I_{1}}(\alpha_{n}(X_{i})-\alpha_{0}(X))^{2}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}\nonumber \\ & \leq\left(\max_{i\in I_{1}}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}\right)\cdot\frac{1}{n_{1}}\sum_{i\in I_{1}}(\alpha_{n}(X_{i})-\alpha_{0}(X))^{2}\nonumber \\ & \overset{\text{(i)}}{\leq}O_{P}(n\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})\cdot O_{P}(p^{-2\xi_{2}})\leq O_{P}(n)\cdot O_{P}(p^{-2\xi_{2}})\overset{\text{(ii)}}{=}o_{P}(1), \end{align} where (i) follows by $\max_{i\in I_{1}}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}\leq\sum_{i\in I_{1}}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}=O_{P}(n\varepsilon_{n}^{4\xi_{1}/(2\xi_{1}+1)})$ (shown above) and $\mathbb{E}\frac{1}{n_{1}}\sum_{i\in I_{1}}(\alpha_{n}(X_{i})-\alpha_{0}(X))^{2}=\mathbb{E}(\alpha_{n}(X)-\alpha_{0}(X))^{2}=O(p^{-2\xi_{2}})$ (due to $\alpha_{0}\in\mathcal{M}_{C,\xi_{2}}$) and (ii) follows by $p\gg n^{1/(2\xi_{2})}$ (due to Assumption (ref)(ii)). } \textcolor{black}{Since $\hat{\rho}_{2}(x)-\rho_{n}(x)=\psi(x)^{\top}\Delta_{\gamma,2}$ and $\Delta_{\gamma,2}$ is independent of $\{X_{i}\}_{i\in I_{1}}$, we have \begin{align*} & \mathbb{E}\left[\frac{1}{n_{1}}\sum_{i\in I_{1}}\alpha_{0}(X)^{2}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}\mid\{W_{i}\}_{i\in I_{2}}\right]\\ & =\Delta_{\gamma,2}\mathbb{E}[\alpha_{0}(X)^{2}\psi(X)\psi(X)^{\top}]\Delta_{\gamma,2}\overset{\text{(i)}}{\lesssim}\|\Delta_{\gamma,2}\|_{2}^{2}=o_{P}(1), \end{align*} where (i) follows by the boundedness of the eigenvalues of $\mathbb{E}[\alpha_{0}(X)^{2}\psi(X)\psi(X)^{\top}]$ (due to Assumption (ref)). Thus, $\frac{1}{n_{1}}\sum_{i\in I_{1}}\alpha_{0}(X)^{2}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}=o_{P}(1)$. This, ((ref)) and ((ref)), displays imply \begin{equation} \frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}=o_{P}(1). \end{equation}} \textcolor{black}{By ((ref)), $T_{4}^{2}=o_{P}(1)$ under Assumption (ref)(ii).} \textcolor{black}{\textbf{Step 5:} show $T_{5}=o_{P}(1)$.} \textcolor{black}{Notice that \begin{align*} & \frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}\\ & \leq\left(\max_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}\right)\cdot\left(\frac{1}{n_{1}}\sum_{i\in I_{1}}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}\right)\\ & \overset{\text{(i)}}{\leq}O_{P}(n)\cdot\left(\frac{1}{n_{1}}\sum_{i\in I_{1}}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}\right)\overset{\text{(ii)}}{=}O_{P}(n)\cdot O_{P}(p^{-2\xi_{1}})\overset{\text{(iii)}}{=}o_{P}(1), \end{align*} where (i) follows by $\max_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}=O_{P}(n)$ (shown in Step 3), (ii) follows by $\mathbb{E}\frac{1}{n_{1}}\sum_{i\in I_{1}}(\rho_{0}(X_{i})-\rho_{n}(X_{i}))^{2}=\mathbb{E}(\rho_{0}(X)-\rho_{n}(X))^{2}=O(p^{-2\xi_{1}})$ (due to $\rho_{0}\in\mathcal{M}_{C,\xi_{1}}$) and (iii) follows by $p\gg n^{1/(2\xi_{1})}$. Therefore, by the definition of $T_{5}$, it remains to verify that \begin{equation} \frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}(\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i}))^{2}=o_{P}(1). \end{equation}} \textcolor{black}{In Step 4 (after ((ref))), we already proved that $\max_{i\in I_{1}}|\rho_{n}(X_{i})-\hat{\rho}_{2}(X_{i})|=o_{P}(1)$ under Assumption (ref)(i). Since $\frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\alpha}_{2}(X_{i})^{2}=O_{P}(1)$ (shown in Step 3), we obtain ((ref)) under Assumption (ref)(i).} \textcolor{black}{We proved ((ref)) in Step 4 under Assumption (ref)(ii). This is the same as ((ref)). The proof is complete. } \end{proof} \begin{comment} \begin{lemma} If $\mathbb{E}[\{Y-\rho_{0}(X)\}^{2}|X]$ and $\mathbb{E}[\alpha_0(X)^{2}]$ are bounded, $\mathbb{E}[|m(W,\rho)|]\leq C\left\Vert \rho\right\Vert _{2}$ and $\left\Vert \rho_{n}-\rho_{0}\right\Vert _{2}=o(1/\sqrt{n})$, then \[ \frac{1}{n}\sum_{i=1}^{n}[\theta_{n}+\varUpsilon_{n}(W_{i})-\theta_{0}-\psi_{0} (W_{i})]=o_{P}(n^{-1/2}). \] \end{lemma} \begin{proof} Note that \begin{align*} \theta_{n}+\varUpsilon_{n}(w)-\theta_{0}-\psi_{0}(w) & =T_{1n}(w)+T_{2n} (w)+T_{3n}(w),\\ T_{1n}(w) & :=m(w,\rho_{n}-\rho_{0})-\alpha_0(x)[\rho_{n}(x)-\rho _{0}(x)],\text\\ T_{2n}(w) & :=[\alpha_{n}(x)-\alpha_0(x)][y-\rho_{0}(x)],\\ T_{3n}(w) & :=-[\rho_{n}(x)-\rho_{0}(x)][\alpha_{n}(x)-\alpha_0(x)]. \end{align*} Let $T_{j}:=\sum_{i}T_{jn}(W_{i})/\sqrt{n},$ $(j=1,2,3).$ Note that $\mathbb{E}[T_{1n}(W)]=0.$ By the triangle and Cauchy-Schwartz inequalities \begin{align*} \mathbb{E}[\left\vert T_{1n}(W)\right\vert ] & \leq \mathbb{E}[\left\vert m(w,\rho_{n} -\rho_{0})\right\vert ]+\mathbb{E}[\left\vert \alpha_0(x)[\rho_{n}(x)-\rho _{0}(x)]\right\vert ]\\ & \leq C\left\Vert \rho_{n}-\rho_{0}\right\Vert _{2}=o(1/\sqrt{n}) \end{align*} so $\mathbb{E}[\left\vert T_{1}\right\vert ]\leq\sqrt{n}\mathbb{E}[\left\vert T_{1n} (W)\right\vert ]=o(1),$ so $T_{1}=o_{P}(1)$ follows by the Markov inequality. Next, note that by $\alpha_0,\alpha_{n}\in\Gamma$ it follows that $\mathbb{E}[T_{2n}(W)]=0$. Also, by $\mathbb{E}[\{Y-\rho_{0}(X)\}^{2}|X]$ bounded, \[ \mathbb{E}[T_{2n}(W)^{2}]=\mathbb{E}[\{\alpha_{n}(X)-\alpha_0(X)\}^{2}\{Y-\rho_{0} (X)\}^{2}]\leq C\left\Vert \alpha_{n}-\alpha_0\right\Vert _{2} ^{2}\rightarrow0. \] Then $\mathbb{E}[T_{2}^{2}]\rightarrow0$ so $T_{2}=o_{P}(1)$ follows by the Markov inequality. Also, note that by the Cauchy-Schwartz inequality and either Assumption (ref) or Assumption (ref), \[ \mathbb{E}[|T_{3n}(W)|]\leq\left\Vert \alpha_{n}-\alpha_0\right\Vert _{2}\left\Vert \rho_{n}-\rho_{0}\right\Vert _{2}=o(1/\sqrt{n}). \] Then by the triangle inequality $\mathbb{E}[\left\vert T_{3}\right\vert ]\leq\sqrt {n}\mathbb{E}[|T_{3n}(W)|]\rightarrow0,$ so $T_{2}=o_{P}(1)$ follows by the Markov inequality. The conclusion then follows from the triangle inequality. \end{proof} \begin{proof}[Old Proof of Theorem (ref)] By Lemma (ref) or Lemma (ref), the same results for $I_{2},$ and Lemma (ref) the triangle inequality gives \begin{align*} \hat{\theta} & =\frac{n_{1}}{n}\hat{\theta}_{1}+\frac{n_{2}}{n}\hat{\theta }_{2}=\sum_{\ell=1}^{2}\frac{n_{\ell}}{n}\{\theta_{n}+\frac{1}{n_{\ell}} \sum_{i\in I_{\ell}}\varUpsilon_{n}(W_{i})+o_{P}(n^{-1/2})\}\\ & =\theta_{n}+\frac{1}{n}\sum_{i=1}^{n}\varUpsilon_{n}(W_{i})+o_{P}(n^{-1/2} )=\theta_{0}+\frac{1}{n}\sum_{i=1}^{n}\psi_{0}(W_{i})+o_{P}(n^{-1/2}). \end{align*} Note also that by Assumption (ref), $\mathbb{E}[m(W,\rho_{0})^{2}]<\infty$ and \[ \mathbb{E}[\alpha_0(X)^{2}\{Y-\rho_{0}(X)\}^{2}]=\mathbb{E}[\alpha_0(X)^{2}Var(Y|X)]\leq C\mathbb{E}[\alpha_0(X)^{2}]<\infty, \] so that $\mathbb{E}[\psi_{0}(W)^{2}]<\infty.$ Then the first conclusion follows by the the Lindeberg-Levy central limit theorem. Let $\psi_{i}=m(W_{i},\rho_{0})+\alpha_0(X_{i})(Y_{i}-\rho_{0} (X_{i}))-\theta_{0},$ by Khintchines law of large numbers $\sum_{i}\psi _{i}^{2}/n=V+o_{P}(1).$ Let $\hat{\delta}_{i}:=\hat{\psi}_{i}-\psi_{i}.$ By the triangle and Cauchy-Scwartz inequalities, \begin{align*} \left\vert \hat{V}-\frac{1}{n}\sum_{i}\psi_{i}^{2}\right\vert & \leq\frac {1}{n}\sum_{i}\left\vert \hat{\psi}_{i}^{2}-\psi_{i}^{2}\right\vert \leq \frac{1}{n}\sum_{i}(\hat{\delta}_{i}^{2}+2\left\vert \psi_{i}\right\vert \left\vert \hat{\delta}_{i}\right\vert )\\ & \leq\frac{1}{n}\sum_{i}\hat{\delta}_{i}^{2}+2(\frac{1}{n}\sum_{i}\left\vert \psi_{i}\right\vert ^{2})^{1/2}(\frac{1}{n}\sum_{i}\hat{\delta}_{i}^{2} )^{1/2}. \end{align*} By existence of $V$ and the Markov inequality $\sum_{i}\left\vert \psi _{i}\right\vert ^{2}/n=O_{P}(1),$ so the second conclusion will follow from $\sum_{i}\hat{\delta}_{i}^{2}/n=o_{P}(1).$ Let $\varepsilon_{i}=Y_{i}-\rho _{0}(X_{i})$. Note that for $i\in I_{1},$ \begin{align*} \hat{\delta}_{i} & =\hat{\delta}_{i1}+\hat{\delta}_{i2}+\hat{\delta} _{i3}+\hat{\theta}-\theta_{0},\text\hat{\delta}_{i1}:=m(W_{i},\hat{\rho }_{2}-\rho_{0}),\text\\ \hat{\delta}_{i2} & :=\tilde{\alpha}_{1}(X_{i})[\rho_{0}(X_{i})-\hat{\rho }_{2}(X_{i})]\text{, }\hat{\delta}_{i3}:=\{\tilde{\alpha}_{1}(X_{i} )-\alpha_0(X_{i})\}\varepsilon_{i}. \end{align*} If Assumption (ref) iii) a) or Assumption (ref) is satisfied then \[ \mathbb{E}[\frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\delta}_{i1}^{2}|\{W_{i}\}_{i\in I_{2} }]=\int[m(w,\hat{\rho}_{2}-\rho_{0})]^{2}F_{W}(dw)\leq C\left\Vert \hat{\rho }_{2}-\rho_{0}\right\Vert _{2}^{2}\overset{p}{\rightarrow}0. \] If Assumption (ref) iii) b) is satisfied then $\sup_{x\in X}\left\vert \rho _{0}(x)\right\vert \leq C$ and by Lemma (ref) $\sup_{x\in X}\left\vert \hat {\rho}(x)\right\vert =O_{P}(1).$ Therefore by Lemma (ref), and \begin{align*} \mathbb{E}[\frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\delta}_{i1}^{2}|\{W_{i}\}_{i\in I_{2} }] & =\int[m(w,\hat{\rho}_{2}-\rho_{0})]^{2}F_{W}(dw)\\ & \leq\sup_{X}\left\vert \hat{\rho}_{2}(x)-\rho_{0}(x)\right\vert \int\left\vert m(w,\hat{\rho}_{2}-\rho_{0})\right\vert a(w)F_{W}(dw)\\ & \leq O_{P}(1)\left\Vert \hat{\rho}-\rho_{0}\right\Vert _{2}\leq O_{P}(1)(\left\Vert \hat{\rho}-\rho_{n}\right\Vert _{2}+\left\Vert \rho _{n}-\rho_{0}\right\Vert_{2} )\overset{p}{\rightarrow}0. \end{align*} In either case the conditional Markov inequality implies $\sum_{i\in I_{1} }\hat{\delta}_{i1}^{2}/n=o_{P}(1).$ Similarly, by $\left\vert \tilde{\alpha }_{1}(X_{i})\right\vert \leq\bar{\tau}_{n}$, \[ \mathbb{E}[\frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\delta}_{i2}^{2}|\{W\}_{i\in I_{2} }]\leq\bar{\tau}_{n}^{2}\int[\hat{\rho}_{2}(x)-\rho_{0}(x)]^{2}F_{W} (dw)\leq\bar{\tau}_{n}^{2}\left\Vert \hat{\rho}_{2}-\rho_{0}\right\Vert _{2}^{2}=o_{P}(1), \] so that $\sum_{i\in I_{1}}\hat{\delta}_{i2}^{2}/n=o_{P}(1)$ follows by the conditional Markov inequality. In addition, since $\tilde{\alpha}(X_{i})$ depends only on $\{X_{i}\}_{i\in I_{1}},$ \[ \mathbb{E}[\frac{1}{n_{1}}\sum_{i\in I_{1}}\hat{\delta}_{i3}^{2}|\{X_{i}\}_{i\in I_{1} }]=\frac{1}{n_{1}}\sum_{i\in I_{1}}(\tilde{\alpha}_{1}(X_{i})-\bar{\alpha }(X_{i}))^{2}Var(Y_{i}|X_{i})\leq C\frac{1}{n_{1}}\sum_{i\in I_{1}} (\tilde{\alpha}_{1}(X_{i})-\alpha_0(X_{i}))^{2}. \] \newline By $\left\vert \tau_{n}(\tilde{a})-\tau_{n}(a)\right\vert \leq\left\vert \tilde{a}-a\right\vert $ for any scalars $\tilde{a}$ and $a$ and $\mathbb{E}[1(\left\vert \alpha_0(X_{i})\right\vert \geq\bar{\tau}_{n} )\alpha_0(X_{i})^{2}]\rightarrow0,$ by $\bar{\tau}_{n}\rightarrow \infty,$ the Cauchy Schwartz and Markov inequalities give \begin{align*} \frac{1}{n_{1}}\sum_{i\in I_{1}}(\tilde{\alpha}_{1}(X_{i})-\alpha_0 (X_{i}))^{2} & \leq\frac{C}{n_{1}}\sum_{i\in I_{1}}(\hat{\alpha}(X_{i} )-\alpha_0(X_{i}))^{2}+\frac{C}{n_{1}}\sum_{i\in I_{1}}1(\left\vert \alpha_0(X_{i})\right\vert \geq\bar{\tau}_{n})\alpha_0(X_{i})^{2}\\ & \leq C(\hat{\pi}_{1}-\pi)^{\top}\hat{\Sigma}_{1}(\hat{\pi}_{1}-\pi )+\frac{C}{n_{1}}\sum_{i\in I_{1}}\{\alpha_{n}(X_{i})-\alpha_{0}(X_{i} )\}^{2}+o_{P}(1)\\ & =o_{P}(1). \end{align*} Then by the conditional Markov inequality we have $\sum_{i\in I_{1}} \hat{\delta}_{i3}^{2}/n_{1}=o_{P}(1).$ Applying the same arguments for $i\in I_{2}$ also gives $\sum_{i\in I_{2}}\hat{\delta}_{ik}^{2}/n_{2}=o_{P}(1),$ $k=1,2,3.$ Then combining these results it follows that $\sum_{i=1}^{n} \hat{\delta}_{ik}^{2}/n=o_{P}(1),$ $k=1,2,3,$ so the second conclusion then follows by $\hat{\theta}=\theta_{0}+o_{P}(1)$ and the triangle and Cauchy-Schwartz inequalities. \end{proof} \end{comment} \begin{comment} \begin{proof}[Proof of Corollary (ref)] For Example (ref), we let $m(W,\rho)=\rho(\bar {d},Z)-\rho(d,Z)]/(\bar{d}-d)$, $U=D-\bar{D}$ for $\bar{D}$ the least squares projection of $D$ on the mean square closure of finite linear combinations of the covariates $\{Z_{1},Z_{2},...\},$ and $\bar{\alpha }(X)=U/\mathbb{E}[U^{2}]$ as at the beginning of Section (ref). It was shown there that $\mathbb{E}[m(W,\rho)]=\mathbb{E}[\alpha_0(X)\rho(X)]$ for all $\rho\in\mathcal{B}$. Also $m(W,\psi_{j})=\tilde{m}_{0}(W)\tilde{m}_{j}(W)$ for $\tilde{m}_{0}(W):=1$ and $\tilde{m}_{j}(W):=\psi_{j}(1,Z)-\psi_{j}(0,Z),$ with Assumption (ref) implying \[ \mathbb{E}[\tilde{m}_{0}(W)^{2}]<\infty,\text{ }\left\vert \tilde{m}_{j}(W)\right\vert =\left\vert \psi_{j}(1,Z)-\psi_{j}(0,Z)\right\vert \leq C. \] Therefore Assumption (ref) is satisfied. Let $\mathcal{B}_{Z}$ denote the mean square closure of finite linear combinations of $\{Z_{1},Z_{2},...\}.$ It follows by standard calculations for partially linear projections that any $\rho(X)\in\mathcal{B}$ has the form $\theta D+h(Z),$ $h\in\mathcal{B}_{Z}.$ Therefore by $\mathbb{E}[U(h+\bar{D})]=0,$ \[ \left\Vert \rho\right\Vert _{2}^{2}=\mathbb{E}[(\theta U+h+\bar{D}))^{2}=\theta ^{2}\mathbb{E}[U^{2}]+\mathbb{E}[\{h+\bar{D}\}^{2}]\geq\theta^{2}\mathbb{E}[U^{2}]. \] Therefore \[ \mathbb{E}[m(W,\rho)^{2}]=\theta^{2}\leq C\left\Vert \rho\right\Vert _{2}^{2},\text{ }C=1/\mathbb{E}[U^{2}]. \] Also $U$ is bounded by assumption, so that $\alpha_0(X)$ is bounded. Therefore the conditions of Assumption (ref) a) are Assumption (ref) iii) and iv)\ are satisfied. The conclusions then follow by Theorem (ref). \end{proof} \begin{proof}[Proof of Corollary (ref)] In Example (ref), let $\alpha_{0}(X):=S(D)\omega (D)f(D|Z)^{-1}.$ Note that by the hypotheses, \[ \mathbb{E}[\alpha_{0}(X)^{2}]\leq \mathbb{E}[\{1+S(D)^{4}\}\omega(D)^{2}f(D|Z)^{-2}]<\infty, \] so that $\mathbb{E}[\alpha_0(X)^{2}]<\infty$ for $\alpha_0(X)=\mathrm{proj}(\alpha _{0}(X)|\mathcal{B)}.$ For $\rho\in\mathcal{B}$ it follows by dividing and multiplying by $f(D|Z)$ that \[ \mathbb{E}[m(W,\rho)]=\int S(u)\rho(u,z)\omega(du)F_{Z}(dz)=\mathbb{E}[\alpha_{0}(X)\rho (X)]=\mathbb{E}[\alpha_0(X)\rho(X)]. \] Also $m(W,\psi_{j})=\tilde{m}_{0}(W)\tilde{m}_{j}(W)$ for $\tilde{m} _{0}(W):=S(U)$ and $\tilde{m}_{j}(W):=\psi_{j}(U,Z),$ with Assumption (ref) implying \[ \mathbb{E}[\tilde{m}_{0}(W)^{2}]=\mathbb{E}[S(U)^{2}]=\int S(u)^{2}\omega(u)du<\infty,\text{ }\left\vert \tilde{m}_{j}(W)\right\vert =\left\vert \psi_{j}(U,Z)\right\vert \leq C. \] Therefore Assumption (ref) is satisfied. Also, for $a(W)=S(U),$ $\left\vert m(W,\rho)\right\vert \leq a(W)\sup_{x\in X}\left\vert \rho(x)\right\vert ,$ and \begin{align*} \mathbb{E}[\{1+a(W)\}\left\vert m(W,\rho)\right\vert ] & =\mathbb{E}[\{1+S(U)\}\left\vert S(U)\right\vert |\rho(U,Z)|]=\mathbb{E}[\{1+S(D)\}\left\vert S(D)\right\vert \omega(D)f(D|Z)^{-1}|\rho(X)|],\\ & \leq\left\{ \mathbb{E}[\{1+S(D)\}^{2}\left\vert S(D)\right\vert ^{2}\omega (D)^{2}f(D|Z)^{-2}]\right\} ^{1/2}\left\Vert \rho\right\Vert _{2}\leq C\left\Vert \rho\right\Vert _{2}. \end{align*} Also, by $S(D)^{2}\omega(D)f(D|Z)^{-1}\leq C,$ \[ \mathbb{E}[m(W,\rho)^{2}]=\mathbb{E}[S(U)^{2}\rho(U,X)^{2}]=\mathbb{E}[S(D)^{2}\omega(D)f(D|Z)^{-1} \rho(X)^{2}]\leq C\left\Vert \rho\right\Vert _{2}^{2}. \] Therefore the inequalities in Assumption (ref) iii) a) and b) are satisfied, so the conclusions follow by Theorem (ref). \end{proof} \begin{proof}[Proof of Corollary (ref)] In Example (ref), let $\alpha_{0}(X):=\pi _{0}(Z)^{-1}D-\{1-\pi_{0}(Z)\}^{-1}(1-D).$ Note that by assumption, \begin{align*} \mathbb{E}[\alpha_{0}(X)^{2}] & =\mathbb{E}[\pi_{0}(Z)^{-2}D^{2}]+\mathbb{E}[\{1-\pi_{0}(Z)\}^{-2} (1-D)^{2}]\\ & =\mathbb{E}[\pi_{0}(Z)^{-1}+\{1-\pi_{0}(Z)\}^{-1}]=\mathbb{E}[\pi_{0}(Z)^{-1}\{1-\pi _{0}(Z)\}^{-1}]<\infty, \end{align*} so that $\mathbb{E}[\alpha_0(X)^{2}]<\infty$ for $\alpha_0(X)=\mathrm{proj}(\alpha _{0}(X)|\mathcal{B)}.$ For any $\rho\in\mathcal{B}$ it follows by a standard calculation that \[ \mathbb{E}[m(W,\rho)]=\mathbb{E}[\alpha_{0}(X)\rho(X)]=\mathbb{E}[\alpha_0(X)\rho(X)]. \] Also $m(W,\psi_{j})=\tilde{m}_{0}(W)\tilde{m}_{j}(W)$ for $\tilde{m}_{0}(W):=1$ and $\tilde{m}_{j}(W):=\psi_{j}(1,Z)-\psi_{j}(0,Z),$ with Assumption (ref) implying \[ \mathbb{E}[\tilde{m}_{0}(W)^{2}]<\infty,\text{ }\left\vert \tilde{m}_{j}(W)\right\vert =\left\vert \psi_{j}(1,Z)-\psi_{j}(0,Z)\right\vert \leq C. \] Therefore Assumption (ref) is satisfied. Also, for $a(W)=2,$ $\left\vert m(W,\rho)\right\vert \leq a(W)\sup_{x\in X}\left\vert \rho(x)\right\vert ,$ and \begin{align*} \mathbb{E}[\{1+a(W)\}\left\vert m(W,\rho)\right\vert ] & =3\mathbb{E}[|\rho(1,Z)-\rho (0,Z)|]\leq3\mathbb{E}[|\rho(1,Z)|]+3\mathbb{E}[|\rho(0,Z)|]\\ & =3\mathbb{E}[\{\frac{D}{\pi_{0}(Z)}+\frac{1-D}{1-\pi_{0}(Z)}\}|\rho(X)|]\leq \sqrt{\mathbb{E}[\alpha_{0}(X)^{2}]}\left\Vert \rho\right\Vert _{2}\leq C\left\Vert \rho\right\Vert _{2}. \end{align*} Thus all the conditions of Assumption (ref) b) are satisfied. Also, under Assumption (ref) with $\pi_{0}(Z)$ bounded away from zero and one, \begin{align*} \mathbb{E}[m(W,\rho)^{2}] & =\mathbb{E}[|\rho(1,Z)-\rho(0,Z)|^{2}]\leq \mathbb{E}[\rho(1,Z)^{2} +\rho(0,Z)^{2}]\\ & =\mathbb{E}[\{\frac{D}{\pi_{0}(Z)}+\frac{1-D}{\pi_{0}(Z)}\}\rho(X)^{2} ]\leq\left\Vert \rho\right\Vert _{2}^{2}, \end{align*} and all the conditions of Assumption (ref) are satisfied. The conclusions then follow by Theorem (ref). \end{proof} \begin{proof}[Proof of Theorem (ref)] Note that $\hat{\theta}$ solves \[ \sum_{\ell=1}^{2}\sum_{i\in I_{\ell}}\{D_{i}-\hat{\eta}_{\ell}(Z_{i} )\}[Y_{i}-\hat{\zeta}_{\ell}(Z_{i})-\theta\{D_{i}-\hat{\eta}_{\ell} (Z_{i})\}]=0. \] We will proceed by showing that \begin{align} & \frac{1}{\sqrt{n}}\sum_{\ell=1}^{2}\sum_{i\in I_{\ell}}\{D_{i}-\hat{\eta }_{\ell}(Z_{i})\}[Y_{i}-\hat{\zeta}_{\ell}(Z_{i})-\theta_{0}\{D_{i}-\hat{\eta }_{\ell}(Z_{i})\}]\overset{d}{\rightarrow}N(0,\Omega),\\ & \frac{1}{n}\sum_{\ell=1}^{2}\sum_{i\in I_{\ell}}\{D_{i}-\hat{\eta}_{\ell }(Z_{i})\}^{2}\overset{p}{\rightarrow}Q.\nonumber \end{align} The first conclusion then follows by standard arguments (e.g. Newey and McFadden, 1994). To show the first result in Equation ((ref)) note that it suffices to show \begin{align*} \frac{1}{\sqrt{n}}\sum_{i\in I_{\ell}}\{D_{i}-\hat{\eta}_{\ell}(Z_{i} )\}[Y_{i}-\hat{\zeta}_{\ell}(Z_{i})] & =\frac{1}{\sqrt{n}}\sum_{i\in I_{\ell }}\{D_{i}-\eta_{0}(Z_{i})\}[Y_{i}-\zeta_{0}(Z_{i})]+o_{P}(1),\\ \frac{1}{\sqrt{n}}\sum_{i\in I_{\ell}}\{D_{i}-\hat{\eta}_{\ell}(Z_{i})\}^{2} & =\frac{1}{\sqrt{n}}\sum_{i\in I_{\ell}}\{D_{i}-\eta_{0}(Z_{i})\}^{2} +o_{P}(1),\text{ \thinspace}(\ell=1,2),\\ \eta_{0}(Z) & =\mathbb{E}[D|Z],\text\zeta_{0}(Z)=\mathrm{proj}(Y|Z_{1},Z_{2},...). \end{align*} Note also that \begin{align*} T & :=\frac{1}{\sqrt{n}}\sum_{i\in I_{\ell}}\left\{ \{D_{i}-\hat{\eta} _{\ell}(Z_{i})\}[Y_{i}-\hat{\zeta}_{\ell}(Z_{i})]-\{D_{i}-\eta_{n} (Z_{i})\}[Y_{i}-\zeta_{n}(Z_{i})]\right\} \\ & =\frac{-1}{\sqrt{n}}\sum_{i\in I_{\ell}}\left\{ Y_{i}\hat{\eta}_{\ell }(Z_{i})+\hat{\zeta}_{\ell}(Z_{i})[D_{i}-\hat{\eta}_{\ell}(Z_{i})]-\{Y_{i} \eta_{n}(Z_{i})+\zeta_{n}(Z_{i})[D_{i}-\eta_{n}(Z_{i})]\}\right\} . \end{align*} Now we follow the proof of Lemma (ref), for $x=z,$ $\rho(x)=\eta(z),$ $m(w,\rho)=y\eta(z),$ $\eta_{n}(Z)=\mathrm{proj}(D|Z_{1},...,Z_{n})$, and $\zeta _{n}(Z)=\mathrm{proj}(Y|Z_{1},...,Z_{n})$, with $\Sigma=\mathbb{E}[ZZ^{\top}],$ $\mathbb{E}[ZD]-\Sigma \gamma=0$, $\mathbb{E}[ZY]-\Sigma\pi=0.$ \ In this setting $T=T_{1}+T_{2}+T_{3}$ as in the proof of Lemma (ref), with $T_{1}=o_{P}(n^{-1/2})$ and $T_{2}=o_{P} (n^{-1/2})$ as shown there. Also \[ \hat{U}=\frac{1}{n_{\ell}}\sum_{i\in I_{\ell}}Z_{i}[D_{i}-\eta_{n}(Z_{i})]. \] Then for $u_{i}=D_{i}-\eta_{0}(Z_{i})$ and $d_{i}=\eta_{0}(Z_{i})-\eta _{n}(Z_{i}),$ \[ T_{3}=T_{31}+T_{32},\text{ }T_{31}=\frac{1}{n_{\ell}}\sum_{i\in I_{\ell}} u_{i}\{\hat{\zeta}_{\ell}(Z_{i})-\zeta_{n}(Z_{i})\},\text{ }T_{32}=\frac {1}{n_{\ell}}\sum_{i\in I_{\ell}}d_{i}\{\hat{\zeta}_{\ell}(Z_{i})-\zeta _{n}(Z_{i})\}. \] Then for $\tilde{W}$ the observations not in $I_{\ell}$ (that are independent of observations in $I_{\ell}$) it follows that $\mathbb{E}[u_{i}|Z,\tilde{W}]=0$. Then by $\hat{\zeta}_{\ell}(Z_{i})$ depending only on $Z$ and $\tilde{W}$ and $Var(D|Z)\leq C,$ \begin{align*} \mathbb{E}[T_{31}|Z,\tilde{W}] & =\frac{1}{n_{\ell}}\sum_{i\in I_{\ell}} \mathbb{E}[u_{i}|Z,\tilde{W}]\{\hat{\zeta}_{\ell}(Z_{i})-\zeta_{n}(Z_{i})\}=0,\\ Var(T_{31}|Z,\tilde{W}) & =\frac{1}{n_{\ell}^{2}}\sum_{i\in I_{\ell} }Var(D_{i}|Z_{i1},Z_{i2},...)\{\hat{\zeta}_{\ell}(Z_{i})-\zeta_{n} (Z_{i})\}^{2}\leq C\frac{1}{n_{\ell}^{2}}\sum_{i\in I_{\ell}}\{\hat{\zeta }_{\ell}(Z_{i})-\zeta_{n}(Z_{i})\}^{2}\\ & =C(\tilde{\pi}_{\ell}-\pi)^{\top}\tilde{\Sigma}_{\ell}(\tilde{\pi}_{\ell }-\pi)/n=o_{P}(n^{-1}), \end{align*} where the last equality follows as in Lemma (ref). It follows by the conditional Markov inequality that $T_{31}=o_{P}(n^{-1/2}).$ Also note that by the triangle and Markov inequalities and Lemma (ref), \begin{align*} \left\vert T_{32}\right\vert & \leq\left( \frac{1}{n_{\ell}}\sum_{i\in I_{\ell}}d_{i}^{2}\right) ^{1/2}\left( \frac{1}{n_{\ell}}\sum_{i\in I_{\ell }}\{\hat{\zeta}_{\ell}(Z_{i})-\zeta_{n}(Z_{i})\}^{2}\right) ^{1/2}\\ & =O_{P}(\left\Vert \eta_{n}-\eta_{0}\right\Vert _{2}r\varepsilon_{n} ^{-1}\delta_{n})=o_{P}(n^{-1/2}). \end{align*} Then $T=o_{P}(n^{-1/2})$ follows by the triangle inequality. It then follows from Lemma (ref) and $\mathbb{E}[Y^{2}|Z_{1},Z_{2},...]$ and $\mathrm{proj}(Y|Z_{1},Z_{2},...)$ bounded that \[ \frac{1}{\sqrt{n}}\sum_{i\in I_{\ell}}\left\{ \{D_{i}-\eta_{n}(Z_{i} )\}[Y_{i}-\zeta_{n}(Z_{i})]-\{D_{i}-\eta_{0}(Z_{i})\}[Y_{i}-\zeta_{0} (Z_{i})]\right\} =o_{P}(1). \] The triangle inequality then gives \[ \frac{1}{\sqrt{n}}\sum_{i\in I_{\ell}}\left\{ \{D_{i}-\hat{\eta}_{\ell} (Z_{i})\}[Y_{i}-\hat{\zeta}_{\ell}(Z_{i})]-\{D_{i}-\eta_{0}(Z_{i} )\}[Y_{i}-\zeta_{0}(Z_{i})]\right\} =o_{P}(1). \] It follows analogously that \begin{equation} \frac{1}{\sqrt{n}}\sum_{i\in I_{\ell}}\{D_{i}-\hat{\eta}_{\ell}(Z_{i} )\}^{2}-\{D_{i}-\eta_{0}(Z_{i})\}^{2}=o_{P}(1). \end{equation} Also, the central limit theorem gives \[ \frac{1}{\sqrt{n}}\sum_{i=1}^{n}\{D_{i}-\eta(Z_{i})\}[Y_{i}-\zeta_{0} (Z_{i})-\theta_{0}\{D_{i}-\eta_{0}(Z_{i})\}]\overset{d}{\rightarrow }N(0,\Omega) \] The first result in Equation ((ref)) then follows by the triangle inequality and the Slutzky Lemma. The second result follows by Khintchine's law of large numbers and the triangle inequality. \end{proof} \end{comment} \section{Some General Lemmas} \begin{lemma} For any $a\in \mathbb{R}^{p}$ such that $\left\Vert a-b_{s}\right\Vert _{2}\leq Cs^{-r} $ for any $s\geq0$, where $C,r>0$\ are constants and $b_{s}=\arg\min_{\left\Vert v\right\Vert _{0}\leq s}\left\Vert a-v\right\Vert _{2}.$ If $r>1/2$ and $s\geq2$, then $\left\Vert a-b_{s}\right\Vert _{1}\leq Ds^{1/2-r},$ where $D>0$ is a constant depending only on $C$ and $r.$ \end{lemma} \begin{proof}[Proof of Lemma (ref)] Without loss of generality, we assume that $|a_{1}|\geq |a_{2}|\geq\cdots\geq|a_{p}|\geq0$. Then clearly, $a-b_{k}=(0,0,\ldots ,0,a_{k+1},a_{k+2},\ldots,a_{p})^\top$ for $k\geq0$. By assumption, we have that for any $k\geq0$, \begin{equation} \sum_{j=k+1}^{p}a_{j}^{2}\leq C^{2}k^{-2r}. \end{equation} Let $g\in\mathbb{N}$ be defined as $2^{g}<p/s\leq2^{g+1}$. With a slight abuse of notation, we extend $a$ to be a $2^{g+1}s$-dimensional vector with $a_{j}=0$ for $j>p$. Then we have that \begin{eqnarray*} \sum_{j=s+1}^{p}a_{j}&=&\sum_{m=0}^{g}\sum_{j=2^{m}s+1}^{2^{m+1}s}a_{j}\leq \sum_{m=0}^{g}\sqrt{2^{m}s\sum_{j=2^{m}s+1}^{2^{m+1}s}a_{j}^{2}}\\ &\leq&\sum _{m=0}^{g}\sqrt{2^{m}s\sum_{j=2^{m}s+1}^{p}a_{j}^{2}}\\ &\overset{\text{(i)}}{\leq}& \sum_{m=0}^{g}\sqrt{2^{m}sC^{2}(2^{m}s)^{-2r} }=Cs^{1/2-r}\sum_{m=0}^{g}\left( 2^{1/2-r}\right) ^{m} \\ &<&Cs^{1/2-r}\sum _{m=0}^{\infty}\left( 2^{1/2-r}\right) ^{m}\overset{\text{(ii)}}{=}C\frac {1}{1-2^{1/2-r}}s^{1/2-r}\text{,} \end{eqnarray*} where (i) follows by ((ref)) applied to $k=2^{m}s$ and (ii) follows by the fact that $2^{1/2-r}<1$ (since $r>1/2$). \end{proof} Let $X_{i}=(X_{i,1},...,X_{i,p})^{\top}\in{\mathbb{R}}^{p}$ and $X_{i,0}$ be a scalar, with all random variables allowed to depend on $n.$ \begin{lemma} \textcolor{black}{Let $\{(X_{i},X_{i,0})\}_{i=1}^{n}$ be independent observations. Assume that there exist constants $\kappa,C_{1},C_{2}>0$ such that the sub-Gaussian norm of $X_{i,j}$ is bounded by $C_{1}$ for any $j\in\{1,...,p\}$ and $\\mathbb{E} \left[\,|X_{i,0}|^{2+\kappa}\,\right]\leq C_{2}$. Suppose that $p\lesssim\exp(n^{\zeta})$ for some $\zeta<\kappa/(2\kappa+4)$. Let $D_{i}=X_{i}X_{i,0}-\mathbb{E}[X_{i}X_{i,0}]$ and $\bar{D}=\sum_{i=1}^{n}D_{i}/n.$ Then, there exists a constant $\bar{C}>0$ (depending only on $C_{1}$, $C_{2}$, $\zeta$, and $\kappa$) such that \[ P\left(\|\bar{D}\|_{\infty}>\bar{C}\sqrt{\log(p)/n}\right)=o(1). \]} \end{lemma} \begin{proof}[Proof of Lemma (ref)] \textcolor{black}{We prove this result using symmetrization. Let $\varepsilon_{1},...,\varepsilon_{n}$ be i.i.d Rademacher random variables independent of $X_{i}$ for for all observations, i.e., $P(\varepsilon_{i}=1)=P(\varepsilon_{i}=-1)=1/2$. Define the symmetrized quantity $W_{*,j}=\sum_{i=1}^{n}D_{i,j}\varepsilon_{i}$.} \textcolor{black}{Since the sub-Gaussian norm of $X_{i,j}$ is bounded by a constant uniformly in $j$, $\max_{1\leq j\leq p}\mathbb{E}|X_{i,j}|^{2+4/\kappa}$ is also bounded by a constant. Then \begin{multline*} \max_{1\leq j\leq p}\mathbb{E}[D_{i,j}^{2}]\leq\max_{1\leq j\leq p}\mathbb{E}[X_{i,j}^{2}X_{i,0}^{2}]\leq\max_{1\leq j\leq p}\left(\mathbb{E}|X_{i,j}|^{2\cdot(2\kappa^{-1}+1)}\right)^{\kappa/(2+\kappa)}\cdot\left(\mathbb{E}|X_{i,0}|^{2\cdot(1+\kappa/2)}\right)^{2/(2+\kappa)}\\ =\max_{1\leq j\leq p}\left(\mathbb{E}|X_{i,j}|^{4\kappa^{-1}+2}\right)^{\kappa/(2+\kappa)}\cdot\left(\mathbb{E}|X_{i,0}|^{2+\kappa/}\right)^{2/(2+\kappa)}\leq C. \end{multline*}} \textcolor{black}{Since $\varepsilon_{i}$ is a Rademacher variable, by Hoeffding's lemma, for any $t\in{\mathbb{R}}$, $\mathbb{E}\operatorname{exp}(t\varepsilon_{i})\leq\operatorname{exp}(t^{2}/2)$. Since $\{\varepsilon_{i}\}_{i=1}^{n}$ is independent of $X$ we have \begin{multline*} \mathbb{E}\left[\operatorname{exp}(tW_{\ast,j})|X\right]=\mathbb{E}\left[\prod_{i=1}^{n}\operatorname{exp}[tD_{i,j}\varepsilon_{i}]\mid X\right]=\prod_{i=1}^{n}\mathbb{E}\left[\operatorname{exp}[tD_{i,j}\varepsilon_{i}]\mid X\right]\\ \leq\prod_{i=1}^{n}\operatorname{exp}\left(t^{2}D_{i,j}^{2}/2\right)=\operatorname{exp}\left(t^{2}\sum_{i=1}^{n}D_{i,j}^{2}/2\right). \end{multline*} Similarly, apply the same argument to $-W_{\ast,j}$ to obtain $\mathbb{E}\left[\operatorname{exp}(-tW_{\ast,j})\mid X\right]\leq\operatorname{exp}\left(t^{2}\sum_{i=1}^{n}D_{i,j}^{2}/2\right).$ Since $\operatorname{exp}(t|W_{\ast,j}|)\leq\operatorname{exp}(tW_{\ast,j})+\operatorname{exp}(-tW_{\ast,j})$, we have \[ \mathbb{E}\left[\operatorname{exp}(t|W_{\ast,j}|)\mid X\right]\leq2\operatorname{exp}\left(t^{2}\sum_{i=1}^{n}D_{i,j}^{2}/2\right). \]} \textcolor{black}{Next let $z>0$ be a non-random quantity to be chosen later and $\Vert W_{\ast}\Vert_{\infty}=\max_{1\leq j\leq p}|W_{\ast,j}|$. By Lemma 2.3.7 of van1996weak, we have \begin{equation} \left(1-\beta_{n}(z)\right)P\left(\max_{1\leq j\leq p}\left\vert \sum_{i=1}^{n}D_{i,j}\right\vert >z\right)\leq2P\left(\Vert W_{\ast}\Vert_{\infty}>z/4\right), \end{equation} where \[ \beta_{n}(z)=1-4z^{-2}n\max_{1\leq j\leq p}\mathbb{E}[D_{i,j}^{2}]\geq1-Cz^{-2}n. \] The rest of the proof proceeds in two steps. We first bound $\max_{1\leq j\leq p}\sum_{i=1}^{n}D_{i,j}^{2}$ and then derive the final result.} \textcolor{black}{\textbf{Step 1:} bound $\max_{1\leq j\leq p}\sum_{i=1}^{n}D_{i,j}^{2}$.} \textcolor{black}{Notice that $D_{i,j}^{2}\leq2X_{i,j}^{2}X_{i,0}^{2}+2(\mathbb{E}[X_{i,j}X_{i,0}])^{2}$. Since $|\mathbb{E}[X_{i,j}X_{i,0}]|\leq\sqrt{\mathbb{E}[X_{i,j}^{2}X_{i,0}^{2}]}$, which has been shown to be bounded, it remains to bound $\max_{1\leq j\leq p}\sum_{i=1}^{n}X_{i,j}^{2}X_{i,0}^{2}$. Notice that \begin{multline} \max_{1\leq j\leq p}n^{-1}\sum_{i=1}^{n}X_{i,j}^{2}X_{i,0}^{2}\leq\max_{1\leq j\leq p}\left(n^{-1}\sum_{i=1}^{n}(X_{i,j}^{2})^{(\kappa+4)/\kappa}\right)^{\frac{\kappa}{4+\kappa}}\left(n^{-1}\sum_{i=1}^{n}(X_{i,0}^{2})^{1+\kappa/4}\right)^{\frac{4}{4+\kappa}}\\ =\left(\max_{1\leq j\leq p}n^{-1}\sum_{i=1}^{n}|X_{i,j}|^{2+8/\kappa}\right)^{\frac{\kappa}{4+\kappa}}\left(n^{-1}\sum_{i=1}^{n}|X_{i,0}|^{2+\kappa/2}\right)^{\frac{4}{4+\kappa}}. \end{multline}} \textcolor{black}{Let $$S_{n}=\sum_{i=1}^{n}[|X_{i,0}|^{2+\kappa/2}-\mathbb{E}|X_{i,0}|^{2+\kappa/2}].$$ Let $\alpha=(2+\kappa)/(2+\kappa/2)=2(\kappa+2)/(\kappa+4)>1$. Notice that $\mathbb{E}\left||X_{i,0}|^{2+\kappa/2}-\mathbb{E}|X_{i,0}|^{2+\kappa/2}\right|^{\alpha}$ is bounded since $\mathbb{E}|X_{i,0}|^{(2+\kappa/2)\alpha}=\mathbb{E}|X_{i,0}|^{2+\kappa}$ is bounded. By Rosenthal's inequality (Theorem 9.1 in Chapter 3.9 of gut2013probability), $$\mathbb{E}|S_{n}|^{\alpha}\leq C_{1}(n+n^{\alpha/2})$$ for some constant $C_{1}>0$. We choose $C_{1}$ large enough such that $\mathbb{E}|X_{i,0}|^{2+\kappa/2}\leq C_{1}$. Since $\alpha>1$, it follows that \begin{equation} P\left(n^{-1}\sum_{i=1}^{n}|X_{i,0}|^{2+\kappa/2}>2C_{1}\right)\leq P\left(n^{-1}S_{n}>C_{1}\right)\leq\frac{\mathbb{E}|S_{n}|^{\alpha}}{(nC_{1})^{\alpha}}\leq\frac{C_{1}(n+n^{\alpha/2})}{(nC_{1})^{\alpha}}=o(1). \end{equation}} \textcolor{black}{On the other hand, let $\beta=2+8/\kappa$ and choose $C_{2}>0$ to be a bound on the sub-Gaussian norm of $X_{i,j}$. Then for any $t\geq1$, \begin{equation} \mathbb{E}|X_{i,j}|^{t}\leq C_{3}^{t}t^{t/2}, \end{equation} where $C_{3}>0$ is a constant depending only on $C_{2}$. Let $Z_{i,j}=|X_{i,j}|^{\beta}-\mathbb{E}|X_{i,j}|^{\beta}$. Then for $m\geq1$, \begin{multline*} \mathbb{E}|Z_{i,j}|^{m}\overset{\text{(i)}}{\leq}2^{m-1}\left[\mathbb{E}|X_{i,j}|^{\beta m}+\left(\mathbb{E}|X_{i,j}|^{\beta}\right)^{m}\right]\\ \leq2^{m-1}\left[\mathbb{E}|X_{i,j}|^{\beta m}+\mathbb{E}|X_{i,j}|^{\beta m}\right]=2^{m}\mathbb{E}|X_{i,j}|^{\beta m}\overset{\text{(ii)}}{\leq}2^{m}C_{3}^{\beta m}(\beta m)^{\beta m/2}, \end{multline*} where (i) follows by the elementary inequality that $|a+b|^{m}\leq2^{m-1}(|a|^{m}+|b|^{m})$ for any $a,b\in\mathbb{R}$ and $m\geq1$ and (ii) follows by ((ref)).} \textcolor{black}{We apply Theorem 2 of nagaev1978some (with $c=t=r$ in their theorem). Then for any $r\geq2$, \begin{multline*} \mathbb{E}\left|\sum_{i=1}^{n}Z_{i,j}\right|^{r}\leq r^{r}\left(\sum_{i=1}^{n}\mathbb{E}|Z_{i,j}|^{r}\right)+r^{r/2+1}\exp(r)\left(\sum_{i=1}^{n}\mathbb{E}Z_{i,j}^{2}\right)^{r/2}\int_{0}^{1}x^{r/2-1}(1-x)^{r/2-1}dx\\ \leq r^{r}\left(\sum_{i=1}^{n}\mathbb{E}|Z_{i,j}|^{r}\right)+r^{r/2+1}\exp(r)\left(\sum_{i=1}^{n}\mathbb{E}Z_{i,j}^{2}\right)^{r/2}. \end{multline*} } \textcolor{black}{ We will choose $r$ later. By ((ref)), we also have \[ \sum_{i=1}^{n}\mathbb{E}Z_{i,j}^{2}\leq n2^{2}C_{3}^{2\beta}(2\beta)^{\beta}=C_{4}^{2}n \] and \[ \sum_{i=1}^{n}\mathbb{E}|Z_{i,j}|^{r}\leq2^{r}C_{3}^{\beta r}(\beta r)^{\beta r/2}n=C_{5}r^{\beta r/2}n, \] where $C_{4}=\sqrt{2^{2}C_{3}^{2\beta}(2\beta)^{\beta}}$ and $C_{5}=2C_{3}^{\beta}\beta^{\beta/2}$. The above three displays imply \[ \mathbb{E}\left|\sum_{i=1}^{n}Z_{i,j}\right|^{r}\leq C_{5}r^{(\beta/2+1)r}n+r^{r/2+1}\exp(r)C_{4}^{r}n^{r/2}. \]} \textcolor{black}{Since $p\lesssim\exp(n^{\zeta})$, we have $\log p\leq C_{6}+n^{\zeta}$ for some $C_{6}>0$. Therefore, by the union bound, we have \begin{align*} P\left(n^{-1}\max_{1\leq j\leq p}\left|\sum_{i=1}^{n}Z_{i,j}\right|\geq1\right) & =P\left(\max_{1\leq j\leq p}\left|\sum_{i=1}^{n}Z_{i,j}\right|^{r}\geq n^{r}\right)\\ & \leq\frac{\mathbb{E}\left(\max_{1\leq j\leq p}\left|\sum_{i=1}^{n}Z_{i,j}\right|^{r}\right)}{n^{r}}\\ & \leq n^{-r}\sum_{j=1}^{p}\mathbb{E}\left|\sum_{i=1}^{n}Z_{i,j}\right|^{r}\\ & \leq n^{-r}p\left[C_{5}r^{(\beta/2+1)r}n+r^{r/2+1}\exp(r)C_{4}^{r}n^{r/2}\right]\\ & =C_{5}\exp\left\{ \log p+(\beta/2+1)r\log r-(r-1)\log n\right\} \\ & \quad+\exp\left\{ \log p+(r/2+1)\log r+r(1+\log C_{4})-(r/2)\log n\right\} \\ & \leq C_{5}\exp\left\{ C_{6}+n^{\zeta}+(\beta/2+1)r\log r-(r-1)\log n\right\} \\ & \quad+\exp\left\{ C_{6}+n^{\zeta}+(r/2+1)\log r+r(1+\log C_{4})-(r/2)\log n\right\} \end{align*}} \textcolor{black}{Let $\theta=\beta/2+1=(2\kappa+4)/\kappa$. By the assumption of $\zeta<\kappa/(2\kappa+4)$, we have $\zeta<1/\theta$. Now choose $\sigma\in(\zeta,1/\theta)$ and set $r=n^{\sigma}$. Notice that $\sigma<1$. Then \[ n^{\zeta}+(\beta/2+1)r\log r-(r-1)\log n=n^{\zeta}+\sigma\theta n^{\sigma}\log n-(n^{\sigma}-1)\log n\rightarrow-\infty \] and \begin{multline*} n^{\zeta}+(r/2+1)\log r+r(1+\log C_{4})-(r/2)\log n\\ =n^{\zeta}+\sigma(n^{\sigma}/2+1)\log n+n^{\sigma}(1+\log C_{4})-(n^{\sigma}/2)\log n\rightarrow-\infty. \end{multline*}} \textcolor{black}{The above three displays imply $P\left(n^{-1}\max_{1\leq j\leq p}\left|\sum_{i=1}^{n}Z_{i,j}\right|\geq1\right)=o(1)$. By ((ref)), we have $\max_{1\leq j\leq p}n^{-1}\sum_{i=1}^{n}\mathbb{E}|X_{i,j}|^{\beta}\leq C_{3}$. Then the above implies \begin{multline*} P\left(n^{-1}\max_{1\leq j\leq p}\sum_{i=1}^{n}|X_{i,j}|^{\beta}\geq1+C_{3}\right)\\ \leq P\left(n^{-1}\max_{1\leq j\leq p}\left|\sum_{i=1}^{n}Z_{i,j}\right|+\max_{1\leq j\leq p}n^{-1}\sum_{i=1}^{n}\mathbb{E}|X_{i,j}|^{\beta}\geq1+C_{3}\right)=o(1). \end{multline*}} \textcolor{black}{We now combine the above with ((ref)) and ((ref)), obtaining \[ P\left(\max_{1\leq j\leq p}n^{-1}\sum_{i=1}^{n}D_{i,j}^{2}>2\tilde{M}\right)=P\left(\max_{1\leq j\leq p}n^{-1}\sum_{i=1}^{n}X_{i,j}^{2}X_{i,0}^{2}>2\tilde{M}\right)=o(1) \] for some constant $\tilde{M}>0$. Therefore, $P({\mathcal{A})\rightarrow}0$, where ${\mathcal{A}}=\left\{ \max_{1\leq j\leq p}n^{-1}\sum_{i=1}^{n}D_{i,j}^{2}>2\tilde{M}\right\} $.} \textcolor{black}{\textbf{Step 2:} derive the final result.} \textcolor{black}{On the event ${\mathcal{A}}^{c}$ we have that \begin{multline} \mathbb{E}\left[\operatorname{exp}\left(t\Vert W_{\ast}\Vert_{\infty}\right)\mid X\right]=\mathbb{E}\left[\operatorname{exp}\left(t\max_{1\leq j\leq p}\left\vert W_{\ast,j}\right\vert \right)\mid X\right]\leq\sum_{j=1}^{p}\mathbb{E}\left(\operatorname{exp}(t|W_{\ast,j}|)\mid X\right)\\ \leq2\sum_{j=1}^{p}\operatorname{exp}\left(t^{2}\sum_{i=1}^{n}D_{i,j}^{2}/2\right)\leq2p\operatorname{exp}\left(\tilde{M}t^{2}n\right), \end{multline} Let $t>0$ be a non-random quantity to be chosen. By the Markov inequality we have \begin{align*} P\left(\Vert W_{\ast}\Vert_{\infty}>z/4\mid X\right){\mathbf{1}}_{{\mathcal{A}}^{c}} & \leq P\left(\operatorname{exp}(t\Vert W_{\ast}\Vert_{\infty})>\operatorname{exp}(tz/4)\mid X\right){\mathbf{1}}_{{\mathcal{A}}^{c}}\\ & \leq\operatorname{exp}(-tz/4)\mathbb{E}\left[\operatorname{exp}(t\Vert W_{\ast}\Vert_{\infty})\mid X\right]\cdot{\mathbf{1}}_{{\mathcal{A}}^{c}}\\ & \leq\operatorname{exp}(-tz/4)\cdot2p\operatorname{exp}\left(\tilde{M}t^{2}n\right)=\operatorname{exp}\left(-\frac{1}{4}tz+\log(2p)+\tilde{M}_{n}t^{2}n\right). \end{align*} Now choose $t=z/[8n\tilde{M}]$ to obtain \[ P\left(\Vert W_{\ast}\Vert_{\infty}>z/4\mid X\right){\mathbf{1}}_{{\mathcal{A}}^{c}}\leq\operatorname{exp}\left(-z^{2}/(nK)+\log(2p)\right), \] where $K=64\tilde{M}$. Therefore we have \begin{align*} P\left(\Vert W_{\ast}\Vert_{\infty}>z/4\mid X\right) & =P\left(\Vert W_{\ast}\Vert_{\infty}>z/4\mid X\right)({\mathbf{1}}_{{\mathcal{A}}^{c}}+{\mathbf{1}}_{{\mathcal{A}}})\\ & \leq\operatorname{exp}\left(-z^{2}/(nK)+\log(2p)\right)+{\mathbf{1}}_{{\mathcal{A}}}. \end{align*} Then by iterated expectations and $\mathbb{E}[{\mathbf{1}}_{{\mathcal{A}}}]=P({\mathcal{A})\rightarrow}0,$ \[ P\left(\Vert W_{\ast}\Vert_{\infty}>z/4\right)\leq\operatorname{exp}\left(-z^{2}/(nK)+\log(2p)\right)+o(1). \]} \textcolor{black}{Finally choose $z=2\sqrt{Kn\log(2p)}$ and note that \[ \beta_{n}(z):=1-Cz^{-2}n=1-\frac{C}{4K\log(2p)}\rightarrow1. \] Define $B_{n}:=\sqrt{n/\log(p)}\left\Vert \bar{D}\right\Vert _{\infty}.$ Then by ((ref)) we have for large enough $n$ \begin{align*} P(B_{n} & \geq2\sqrt{K\frac{\log(2p)}{\log(p)}})=P(\left\Vert n\bar{D}\right\Vert _{\infty}>z)\leq CP\left(\Vert W_{\ast}\Vert_{\infty}>\frac{z}{4}\right)\\ & \leq\operatorname{exp}\left(-z^{2}/(nK_{n})+\log(2p)\right)+o(1)=\operatorname{exp}\left(-4\log(2p)+\log(2p)\right)+o(1)\rightarrow0. \end{align*} Note that $\log(2p)/\log(p)=1+2/\log(p)$ is bounded by 2 for large $p$. We have that $P(B_{n}>2\sqrt{2K})=o(1)$. } \end{proof} \begin{lemma} \textcolor{black}{Let $\{(X_{i},X_{i,0})\}_{i=1}^{n}$ be independent observations. Assume that there exist a constant $\kappa>0$ such that $X_{i,j}$ is bounded by $\kappa$ for any $(i,j)\in\{1,...,n\}\times\{1,...,p\}$. Suppose that $\mathbb{E}X_{i,0}^{2}=o(1/\log(pn))$. Let $$D_{i}=X_{i}X_{i,0}-\mathbb{E}[X_{i}X_{i,0}]$$ and $\bar{D}=\sum_{i=1}^{n}D_{i}/n.$ Then, there exists a constant $\bar{C}>0$ (depending only on $\kappa$) such that \[ P\left(\|\bar{D}\|_{\infty}>\bar{C}\sqrt{\log(p)/n}\right)=o(1). \]} \end{lemma} \begin{proof}[Proof of Lemma (ref)] \textcolor{black}{We follow the same argument and same notations as the proof of Lemma (ref) and obtain \[ \mathbb{E}\left[\operatorname{exp}(t|W_{\ast,j}|)\mid X\right]\leq2\operatorname{exp}\left(t^{2}\sum_{i=1}^{n}D_{i,j}^{2}/2\right) \] and \[ \left(1-\beta_{n}(z)\right)P\left(\max_{1\leq j\leq p}\left\vert \sum_{i=1}^{n}D_{i,j}\right\vert >z\right)\leq2P\left(\Vert W_{\ast}\Vert_{\infty}>z/4\right), \] where \[ \beta_{n}(z)=1-4z^{-2}n\max_{1\leq j\leq p}\mathbb{E}[D_{i,j}^{2}]. \]} \textcolor{black}{By the sub-Gaussian assumption and the union bound, $\max_{1\leq i\leq n,1\leq j\leq p}|X_{i,j}|=O_{P}(\sqrt{\log(pn)})$. Thus, \[ \max_{1\leq j\leq p}n^{-1}\sum_{i=1}^{n}X_{i,j}^{2}X_{i,0}^{2}\leq\left(\max_{1\leq i\leq n,1\leq j\leq p}X_{i,j}^{2}\right)\cdot n^{-1}\sum_{i=1}^{n}X_{i,0}^{2}=O_{P}(\log(pn))\cdot o_{P}(1/\log(pn))=o_{P}(1). \] This completes Step 1 in the proof of Lemma (ref). We also have \[ \max_{1\leq j\leq p}\mathbb{E}[D_{i,j}^{2}]\leq\max_{1\leq j\leq p}\mathbb{E}[X_{i,j}^{2}X_{i,0}^{2}]\lesssim\mathbb{E}X_{i,0}^{2}=o(1). \] The rest of the argument follows the same in proof of Lemma (ref). } \end{proof} \begin{comment} \section{An estimator without cross-fitting} It may also be of interest that without cross-fitting the automatic debiased machine learner attains root-n consistency under the minimal approximate sparsity condition $\xi_{1}>1/2$ for the regression. The estimator without cross fitting is \[ \hat\theta =\frac{1}{n}\sum_{i=1}^{n}\{m(W_{i},\hat{\rho} )+\hat{\alpha}(X_{i})[Y_{i}-\hat{\rho}(X_{i})], \hat{V}=\frac{1}{n}\sum_{i=1}^{n}\hat{\psi}_{i}^{2},\text{ }\hat{\psi} _{i}=m(W_{i},\hat{\rho})+\tilde{\alpha}(X_{i})[Y_{i}-\hat{\rho} (X_{i})]-\hat{\theta}, \] where $\hat\rho(x)$ and $\hat\alpha_0(x)$ are obtained as in equations (4.1) and (4.6) respectively, with averages over over all $n$ observations rather than a subset of observations, and $\tilde\alpha(x)=\tau_{n}(\hat\alpha(x))$. The following result gives the properties of this estimator. \begin{corollary} If Assumptions (ref)-(ref), then for $V=\mathbb{E}[\psi_{0}(W)^{2}],$ \[ \sqrt{n}(\hat{\theta}-\theta_{0})=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi _{0}(W_{i})+o_{P}(1)\overset{d}{\rightarrow}N(0,V). \] If in addition $\bar{\tau}_{n}\rightarrow\infty$ and $\bar{\tau}_{n}[(r/\varepsilon_{n})r^{2\xi_{1}/(2\xi_{1}+1)}+\left\Vert \rho_{0}-\rho_{n}\right\Vert _{2} ]\rightarrow0$, then $\hat {V}\overset{p}{\rightarrow}V.$ \end{corollary} \textcolor{PantoneGreen}{Yinchu: Do we keep this Section? This still has trimming for the variance estimation. The proof also needs changes.} \textcolor{pantoneRed}{Jelena: I don't know - is it better then Auto DML papers? If so, then yes} This result shows that without cross-fitting the automatic debiased machine learner is root-n consistent and asymptotically normal with consistent asymptotic variance estimator when $\xi_{1}>1/2.$ \begin{proof}[Proof of Corollary (ref)] The proof of the first conclusion follows exactly as for the proof of Theorem (ref) for the case where $\xi_{1}=1/2,$ upon noting that the proof of Lemma (ref) and (ref) do not use cross fitting. In contrast, the proof of the second conclusion proceeds in a different way. By Assumption (ref) and H\"older's inequality, \[ \left\vert m(W_{i},\hat{\rho}-\rho_{n})\right\vert =\left\vert m(W_{i} ,b)^{\top}(\hat{\gamma}-\gamma)\right\vert \leq\left\Vert m(W_{i} ,b)\right\Vert _{\infty}\left\Vert \hat{\gamma}-\gamma\right\Vert _{1}\leq C\left\vert \tilde{m}_{0}(W_{i})\right\vert \left\Vert \hat{\gamma} -\gamma\right\Vert _{1}. \] Also follows similarly to the proof of Theorem (ref) that \[ \left\Vert \hat{\gamma}-\gamma\right\Vert _{1}=O_{P}(\left( \frac {r}{\varepsilon_{n}}\right) (\varepsilon_{n})^{(2\xi_{1}-1)/(2\xi_{2} +1)})=o_{P}(1). \] Then by \thinspace$\mathbb{E}[\tilde{m}_{0}(W)^{2}]<\infty$ and the Markov inequality, \[ \frac{1}{n}\sum_{i}m(W_{i},\hat{\rho}-\rho_{n})^{2}\leq C\left( \frac{1} {n}\sum_{i}\tilde{m}_{0}(W_{i})^{2}\right) \left\Vert \hat{\gamma} -\gamma\right\Vert _{1}^{2}=O_{P}(1)o_{P}(1)=o_{P}(1). \] If Assumption (ref) iii) a) is satisfied then by the Markov inequality \[ \frac{1}{n}\sum_{i}m(W_{i},\rho_{n}-\rho_{0})^{2}=O_{P}(\mathbb{E}[m(W,\rho_{n} -\rho_{0})^{2}])=O_{P}(\left\Vert \rho_{n}-\rho_{0}\right\Vert _{2}^{2} )=o_{P}(1). \] If Assumption (ref) iii) b) is satisfied then by $\rho_{n}(x)$ and $\rho_{0}(x)$ bounded and the Markov inequality \[ \frac{1}{n}\sum_{i}m(W_{i},\rho_{n}-\rho_{0})^{2}\leq C\frac{1}{n}\sum _{i}a(W_{i})\left\vert m(W_{i},\rho_{n}-\rho_{0})\right\vert =O_{P}(\left\Vert \rho_{n}-\rho_{0}\right\Vert )=o_{P}(1). \] Then for $\hat{\delta}_{i1}=m(W_{i},\hat{\rho}-\rho_{0})$ as in the proof of Theorem (ref), the triangle and Cauchy Schwartz inequalities give \[ \frac{1}{n}\sum_{i}\hat{\delta}_{i1}^{2}\leq C\frac{1}{n}\sum_{i} \{m(W_{i},\hat{\rho}-\rho_{n})^{2}+m(W_{i},\rho_{n}-\rho_{0})^{2}\}=o_{P}(1). \] Also, by $\left\Vert \psi(x)\right\Vert _{\infty}\leq C$ and H\"older's inequality we have \[ [\hat{\rho}(X_{i})-\rho_{n}(X_{i})]^{2}\leq C\left\Vert \hat{\gamma }-\gamma\right\Vert _{1}^{2}. \] Then for $\hat{\delta}_{i2}=\tilde{\alpha }(X_{i})[\hat{\rho}(X_{i})-\rho_{0}(X_{i})]$ we have \begin{align*} \frac{1}{n}\sum_{i}\hat{\delta}_{i2}^{2} & \leq\bar{\tau}_{n}^{2}\frac{1} {n}\sum_{i}[\hat{\rho}(X_{i})-\rho_{0}(X_{i})]^{2}\\ & \leq\bar{\tau}_{n}^{2} \frac{C}{n}\sum_{i}\{[\hat{\rho}(X_{i})-\rho_{n}(X_{i})]^{2}+[\rho_{n} (X_{i})-\rho_{0}(X_{i})]^{2}\}\\ & \leq C\bar{\tau}_{n}^{2}(\left\Vert \hat{\gamma}-\gamma\right\Vert _{1} ^{2}+\left\Vert \rho_{n}-\rho_{0}\right\Vert _{2}^{2})=o_{P}(1). \end{align*} Furthermore, for $\hat{\delta}_{i3}=\{\tilde{\alpha}(X_{i})-\alpha (X_{i})\}[Y_{i}-\rho_{0}(X_{i})]$ it follows as in the proof of Theorem (ref) that $\sum_{i}\hat{\delta}_{i3}^{2}/n=o_{P}(1).$ The conclusion then follows as in proof of Theorem (ref). \end{proof} \end{comment}