Some New Asymptotic Theory for Least Squares Series: Pointwise and Uniform Results

Some New Asymptotic Theory for Least Squares Series: Pointwise and Uniform Results

\address[Alexandre Belloni]{Fuqua School of Business, Duke University, United States} \address[Victor Chernozhukov]{Department of Economics, MIT, United States} \address[Denis Chetverikov]{Department of Economics, UCLA, United States}\email{[email removed] } \address[Kengo Kato]{Graduate School of Economics, The University of Tokyo, Japan}

Introduction

Series estimators have been playing a central role in various fields. In econometric applications it is common that the exact form of a conditional expectation is unknown and having a flexible functional form can lead to improvements over a pre-specified functional form, especially if it nests some successful parametric economic models. Series estimation offers exactly that by approximating the unknown function based on $k$ basis functions, where $k$ is allowed to grow with the sample size $n$ to balance the trade off between variance and bias. Moreover, the series modelling allows for convenient nesting of some theory-based models, by simply using corresponding terms as the first $k_0 \leq k$ basis functions. For instance, our series could contain linear and quadratic functions to nest the canonical Mincer equations in the context of wage equation modelling or the canonical translog demand and production functions in the context of demand and supply modelling.

Several asymptotic properties of series estimators have been investigated in the literature. The focus has been on convergence rates and asymptotic normality results vandeGeer1990,Andrews1991,EastwoodGallant1991,GallantSouza1991,Newey1997,vandeGeer2002,Huang2003b,Chen2006,CF2013.

This work revisits the topic by making use of new critical ingredients:

To the best of our knowledge, our results are the first applications of the first ingredient to statistical estimation problems. After the use in this work, some recent working papers are also using related matrix inequalities and extending some results in different directions, e.g. Chen and Christensen (2013) allows $\beta$-mixing dependence, and Hansen (2014) handles unbounded regressors and also characterizes a trade-off between the number of finite moments and the allowable rate of expansion of the number of series terms. Regarding the second ingredient, it has already been used by Huang2003 but for splines only. All of these ingredients are critical for generating sharp results.

This approach allows us to contribute to the series literature in several directions. First, we weaken considerably the condition on the number $k$ of approximating functions used in series estimation from the typical $k^2/n \to 0$ Newey1997 to $$ k/n \to 0 \text{ (up to logs)} $$ for bounded or local bases which was previously available only for spline series Huang2003, Stone1994, and recently established for local polynomial partition series CF2013. An example of a bounded basis is Fourier series; examples of local bases are spline, wavelet, and local polynomial partition series. To be more specific, for such bases we require $k\log k/n\to 0$. Note that the last condition is similar to the condition on the bandwidth value required for local polynomial (kernel) regression estimators ($h^{-d}\log (1/h)/n\to 0$ where $h=1/k^{1/d}$ is the bandwidth value). Second, under the same weak conditions we derive $L^{2}$ rates and pointwise central limit theorems results when the approximation error vanishes. Under a misspecified model, i.e. when the approximation error does not vanish, analogous results are also shown. Third, under stronger conditions we derive uniform rates that hold if the approximation error vanishes or not. An important contribution here is that we show that the series estimator achieves the optimal uniform rate of convergence under quite general conditions. Previously, the same result was shown only for local polynomial partition series estimator CF2013. In addition, we derive a functional central limit theorem. By the functional central limit theorem we mean here that the entire estimate of the nonparametric function is uniformly close to a Gaussian process that can change with $n$. That is, we derive the strong approximation for the entire estimate of the nonparametric function.

Perhaps the most important contribution of the paper is a set of completely new results that provide estimation and inference methods for the entire linear functionals $\theta(\cdot)$ of the conditional mean function $g:\mathcal{X}\to {\Bbb{R}}$. Examples of linear functionals $\theta(\cdot)$ of interest include

where $\partial_j g(x)$ denotes the partial derivative of $g(x)$ with respect to $j$th component of $x$, $x^s$ is a subvector of $x$, and the measure $\mu$ entering the definitions above is taken as known; the result can be extended to include estimated measures. We derive uniform (in $x$) rates of convergence, large sample distributional approximations, and inference methods for the functions above based on the Gaussian approximation. To the best of our knowledge all these results are new, especially the distributional and inferential results. For example, using these results we can now perform inference on the entire partial derivative function. The only other reference that provides analogous results but for quantile series estimator is BCF2011. Before doing uniform analysis, we also update the pointwise results of Newey1997 to weaker, more general conditions.

Notation. In what follows, all parameter values are indexed by the sample size $n$, but we omit the index whenever this does not cause confusion. We use the notation $(a)_+ = \max\{a,0\}$, $a \vee b = \max\{ a, b\}$ and $a \wedge b = \min\{ a , b \}$. The $\ell_2$-norm of a vector $v$ is denoted by $\|v\|$, while for a matrix $Q$ the operator norm is denoted by $\|Q\|$. We also use standard notation in the empirical process literature, $$ \mathbb{E}_n[f] = \mathbb{E}_n[f(w_i)] = \frac{1}{n}\sum_{i=1}^n f(w_i)\text{ and }\mathbb{G}_n[f]=\mathbb{G}_n[f(w_i)]= \frac{1}{\sqrt{n}}\sum_{i=1}^n (f(w_i) - E[f(w_i)]) $$ and we use the notation $a \lesssim b$ to denote $a \leqslant c b$ for some constant $c>0$ that does not depend on $n$; and $a\lesssim_P b$ to denote $a=O_P(b)$. Moreover, for two random variables $X, Y$ we say that $X=_dY$ if they have the same probability distribution. Finally, $S^{k-1}$ denotes the space of vectors $\alpha$ in $\mathbb{R}^k$ with unit Euclidean norm: $\|\alpha\|=1$.

Set-Up

Throughout the paper, we consider a sequence of models, indexed by the sample size $n$,

where $y_i$ is a response variable, $x_i$ a vector of covariates (basic regressors), $\epsilon_i$ noise, and $x\mapsto g(x) = E[y_i|x_i =x]$ a regression (conditional mean) function; that is, we consider a triangular array of models with $y_i=y_{i,n}$, $x_i=x_{i,n}$, $\epsilon_i=\epsilon_{i,n}$, and $g=g_n$. We assume that $g\in\mathcal{G}$ where $\mathcal{G}$ is some class of functions. Since we consider a sequence of models indexed by $n$, we allow the function class $\mathcal{G}=\mathcal{G}_n$, where the regression function $g$ belongs to, to depend on $n$ as well. In addition, we allow $\mathcal{X}=\mathcal{X}_n$ to depend on $n$ but we assume for the sake of simplicity that the diameter of $\mathcal{X}$ is bounded from above uniformly over $n$ (dropping the uniform boundedness condition is possible at the expense of more technicalities; for example, without uniform boundedness condition, we would have an additional term $\log \text{diam}(\mathcal{X})$ in ((ref)) and ((ref)) of Lemma (ref)). We denote $\sigma_i^2=E[\epsilon_i^2|x_i]$, $\bar{\sigma}^2:=\sup_{x\in\mathcal{X}}E[\epsilon_i^2|x_i=x]$, and $\underline{\sigma}^2:=\inf_{x\in\mathcal{X}}E[\epsilon_i^2|x_i=x]$. For notational convenience, we omit indexing by $n$ where it does not lead to confusion.

Condition A.1 (Sample) For each $n$, random vectors $(y_i, x_i')'$, $i=1,\ldots,n,$ are i.i.d. and satisfy ((ref)).

We approximate the function $x\mapsto g(x)$ by linear forms $x \mapsto p(x)' b$, where $$ x \mapsto p(x) := (p_{1}(x),\ldots,p_k(x))' $$ is a vector of approximating functions that can change with $n$; in particular, $k$ may increase with $n$. We denote the regressors as $$ p_i := p(x_i):= (p_{1}(x_i),\ldots,p_k(x_i))'. $$ The next assumption imposes regularity conditions on the regressors.

Condition A.2 (Eigenvalues) Uniformly over all $n$, eigenvalues of $Q:= E[p_i p_i']$ are bounded above and away from zero.

Condition A.2 imposes the restriction that $p_1(x_i),\dots,p_k(x_i)$ are not too co-linear. Given this assumption, it is without loss of generality to impose the following normalization:

Normalization. To simplify notation, we normalize $Q = I$, but we shall treat $Q$ as unknown, that is we deal with random design.

The following proposition establishes a simple sufficient condition for A.2 based on orthonormal bases with respect to some measure.

It is well known that the least squares parameter $\beta$ is defined by $$ \beta := \arg \min_{b \in \Bbb{R}^k} E\left[(y_i - p_i'b)^2\right], $$ which by ((ref)) also implies that $\beta=\beta_g$ where $\beta_g$ is defined by

We call $x \mapsto g(x)$ the target function and $x\mapsto g_k(x) = p(x)'\beta$ the surrogate function. In this setting, the surrogate function provides the best linear approximation to the target function.

For all $x\in\mathcal{X}$, let

denote the approximation error at the point $x$, and let $$ r_i := r(x_i)= g(x_i) - p(x_i)'\beta_g $$ denote the approximation error for the observation $i$. Using this notation, we obtain a many regressors model $$ y_i = p_i'\beta + u_i, \ \ E [u_i x_i] =0, \ \ u_i:= r_i+\epsilon_i. $$ The least squares estimator of $\beta$ is

where $\widehat Q:=\mathbb{E}_n[p_i p_i']$. The least squares estimator $\widehat{\beta}$ induces the estimator $\widehat g(x) := p(x)'\widehat \beta$ for the target function $g(x)$. Then it follows from ((ref)) that we can decompose the error in estimating the target function as $$ \widehat g(x) - g(x) = p(x)'(\widehat \beta - \beta) - r(x), $$ where the first term on the right-hand side is the estimation error and the second term is the approximation error.

We are also interested in various linear functionals $\theta$ of the conditional mean function. As discussed in the introduction, examples include the partial derivative function, the average partial derivative function, and the conditional average partial derivative. Importantly, in each example above we could be interested in estimating $\theta=\theta(w)$ simultaneously for many values $w\in\mathcal{I}$. By the linearity of the series approximations, the above parameters can be seen as linear functions of the least squares coefficients $\beta$ up to an approximation error, that is

where $\ell_\theta(w)' \beta$ is the series approximation, with $\ell_\theta(w)$ denoting the $k$-vector of loadings on the coefficients, and $r_\theta(w)$ is the remainder term, which corresponds to the approximation error. Indeed, the decomposition ((ref)) arises from the application of different linear operators $\mathcal{A}$ to the decomposition $g(\cdot) = p(\cdot)'\beta + r(\cdot)$ and evaluating the resulting functions at $w$:

Examples of the operator $\mathcal{A}$ corresponding to the cases enumerated in the introduction are given by, respectively,

where $x^s$ is a subvector of $x$. For notational convenience, we use the formulation ((ref)) in the analysis, instead of the motivational formulation ((ref)).

We shall provide the inference tools that will be valid for inference on the series approximation $$ \ell_\theta(w)' \beta, \ \ w \in \mathcal{I}. $$ If the approximation error $r_\theta(w), \ w \in \mathcal{I},$ is small enough as compared to the estimation error, these tools will also be valid for inference on the functional of interest $$ \theta(w), \ \ w \in \mathcal{I}. $$ In this case, the series approximation $\ell_\theta(w)$ is an important intermediary target, whereas the functional $\theta(w)$ is the ultimate target. The inference will be based on the plug-in estimator $\widehat \theta(w) := \ell_\theta(w)' \widehat \beta$ of the the series approximation $\ell_\theta(w)' \beta$ and hence of the final target $\theta(w)$.

Approximation Properties of Least Squares

Next we consider approximation properties of the least squares estimator. Not surprisingly, approximation properties must rely on the particular choice of approximating functions. At this point it is instructive to consider particular examples of relevant bases used in the literature. For each example, we state a bound on the following quantity: $$ \xi_k:= \sup_{x \in \mathcal{X}} \| p(x)\|. $$ This quantity will play a key role in our analysis.\footnote{{ Most results extend directly to the case that $\xi_k\geq \max_{i\leq n} \| p(x_i)\|$ holds with probability $1-o(1)$. We refer to Hansen2014 for recent results that explicit allows for unbounded regressors which required extending the concentration inequalities for matrices.}} Excellent reviews of approximating properties of different series can also be found in huang1998 and Chen2006, where additional references are provided.

Each basis described in Examples (ref)-(ref) has different approximation properties which also depend on the particular class of functions $\mathcal{G}$. The following assumption captures the essence of this dependence into two quantities.

Condition A.3 (Approximation) For each $n$ and $k$, there are finite constants $c_k$ and $\ell_k$ such that for each $f \in \mathcal{G}$, $$ \|r_f\|_{F,2} := \sqrt{{

r_f^2(x) d F(x)}} \leq c_k \ \ \ {\em and} \ \ \ \|r_f\|_{F,\infty} := \sup_{x \in \mathcal{X}} |r_f(x)| \leq \ell_k c_k. $$

Here $r_f$ is defined by ((ref)) and ((ref)) with $g$ replaced by $f$. We call $\ell_k$ the Lebesgue factor because of its relation to the Lebesgue constant defined in Section (ref) below. Together $c_k$ and $\ell_k$ characterize the approximation properties of the underlying class of functions under $L^2(\mathcal{X},F)$ and uniform distances. Note that constants $c_k=c_k(\mathcal{G})$ and $\ell_k=\ell_k(\mathcal{G})$ are allowed to depend $n$ but we omit indexing by $n$ for simplicity of notation. Next we discuss primitive bounds on $c_k$ and $\ell_k$.

Bounds on $c_k$

In what follows, we call the case where $c_k \to 0$ as $k \to \infty$ the correctly specified case. In particular, if the series are formed from bases that span $\mathcal{G}$, then $c_k \to 0$ as $k \to \infty$. However, if series are formed from bases that do not span $\mathcal{G}$, then $c_k \not\to 0$ as $k \to \infty$. We call any case where $c_k \not \to 0$ the incorrectly specified (misspecified) case.

To give an example of the misspecified case, suppose that $d=2$, so that $x=(x_1,x_2)'$ and $g(x)=g(x_1,x_2)$. Further, suppose that the researcher mistakenly assumes that $g(x)$ is additively separable in $x_1$ and $x_2$: $g(x_1,x_2)=g_1(x_1)+g(x_2)$. Given this assumption, the researcher forms the vector of approximating functions $p(x_1,x_2)$ such that each component of this vector depends either on $x_1$ or $x_2$ but not on both; see Newey1997 and NPV99 for the description of nonparametric series estimators of separately additive models. Then note that if the true function $g(x_1,x_2)$ is not separately additive, linear combinations $p(x_1,x_2)'b$ will not be able to accurately approximate $g(x_1,x_2)$ for any $b$, so that $c_k$ does not converge to zero as $k\to\infty$. Since analysis of misspecified models plays an important role in econometrics, we include results both for correctly and incorrectly specified models.

To provide a bound on $c_k$, note that for any $f\in\mathcal{G}$, $$ \inf_b \| f - p'b\|_{F,2}\leq \inf_{b} \| f - p'b\|_{F, \infty}, $$ so that it suffices to set $c_k$ such that $c_k\geq\sup_{f\in\mathcal{G}}\inf_{b} \| f - p'b\|_{F, \infty}$. Next, the bounds for $\inf_{b} \| f - p'b\|_{F, \infty}$ are readily available from the Approximation Theory; see DeVoreLorentz1993. A typical example is based on the concept of $s$-smooth classes, namely H\"{o}lder classes of smoothness order $s$, $\Sigma_s(\mathcal{X})$. For $s\in(0,1]$, the H\"{o}lder class of smoothness order $s$, $\Sigma_s(\mathcal{X})$, is defined as the set of all functions $f:\mathcal{X}\to\mathbb{R}$ such that for $C>0$, $$ |f(x)-f(\widetilde x)|\leq C\Big(\sum_{j=1}^d(x_j-\widetilde{x}_j)^2\Big)^{s/2} $$ for all $x=(x_1,\dots,x_d)'$ and $\widetilde x=(\widetilde{x}_1,\dots,\widetilde{x}_d)'$ in $\mathcal{X}$. The smallest $C$ satisfying this inequality defines a norm of $f$ in $\Sigma_s(\mathcal{X})$, which we denote by $\|f\|_s$. For $s>1$, $\Sigma_s(\mathcal{X})$ can be defined as follows. For a $d$-tuple $\alpha=(\alpha_1,\dots,\alpha_d)$ of nonnegative integers, let $$ D^\alpha=\partial_{x_1}^{\alpha_1}\dots\partial_{x_d}^{\alpha_d}. $$ Let $[s]$ denote the largest integer strictly smaller than $s$. Then $\Sigma_s(\mathcal{X})$ is defined as the set of all functions $f:\mathcal{X}\to\mathbb{R}$ such that $f$ is $[s]$ times continuously differentiable and for some $C>0$, $$ |D^{\alpha}f(x)-D^{\alpha}f(\widetilde{x})|\leq C\Big(\sum_{j=1}^d(x_j-\widetilde{x}_j)^2\Big)^{(s-[s])/2}\text{ and }|D^\beta f(x)|\leq C $$ for all $x=(x_1,\dots,x_d)'$ and $\widetilde{x}=(\widetilde{x}_1,\dots,\widetilde{x}_d)'$ in $\mathcal{X}$ and for all $d$-tuples $\alpha=(\alpha_1,\dots,\alpha_d)$ and $\beta=(\beta_1,\dots,\beta_d)$ of nonnegative integers satisfying $\alpha_1+\dots+\alpha_d=[s]$ and $\beta_1+\dots+\beta_d\leq[s]$. Again, the smallest $C$ satisfying these inequalities defines a norm of $f$ in $\Sigma_s(\mathcal{X})$, which we denote $\|f\|_s$.

If $\mathcal{G}$ is a set of functions $f$ in $\Sigma_s(\mathcal{X})$ such that $\|f\|_s$ is bounded from above uniformly over all $f\in\mathcal{G}$ (that is, $\mathcal{G}$ is contained in a ball in $\Sigma_s(\mathcal{X})$ of finite radius), then we can take

for the polynomial series and $$ c_k \lesssim k^{-(s\wedge s_0)/d} $$ for spline, CDV wavelet, and local polynomial partition series of order $s_0$. If in addition we assume that each element of $\mathcal{G}$ can be extended to a periodic function, then ((ref)) also holds for the Fourier series. See, for example, Newey1997 and Chen2006 for references.

Bounds on $\ell_k$

We say that a least squares approximation by a particular series for the function class $\mathcal{G}$ is co-minimal if the Lebesgue factor $\ell_k$ is small in the sense of being a slowly varying function in $k$. A simple bound on $\ell_k$, which is independent of $\mathcal{G}$, is established in the following proposition:

The proof of this proposition is based on the ideas of Newey1997 and is provided in the Appendix. The advantage of the bound established in this proposition is that it is universally applicable. It is, however, not sharp in many cases because $\xi_k$ satisfies $$ \xi_k^2\geq E[\|p(x_i)\|^2]=E[p(x_i)'p(x_i)]=k $$ so that $\xi_k \gtrsim \sqrt{k}$ in all cases. Much sharper bounds follow from Approximation Theory for some important cases. To apply these bounds, define the Lebesgue constant: $$ \widetilde{\ell}_k:=\sup \left ( \frac{\|p'\beta_f\|_{F,\infty}}{\|f\|_{F,\infty} }: \|f\|_{F,\infty} \neq 0, f \in \bar{\mathcal{G}} \right), $$ where $\bar{\mathcal{G}} = \mathcal{G} + \{p'b: b \in \Bbb{R}^k\} = \{ f + p'b: f \in \mathcal{G}, b \in \Bbb{R}^k\}$. The following proposition provides a bound on $\ell_k$ in terms of $\widetilde{\ell}_k$:

Note that in all examples above, we provided $c_k$ such that $c_k\geq\sup_{f\in\mathcal{G}}\inf_b\|f-p'b\|_{F,\infty}$, and so the results of Propositions (ref) and (ref) apply in our examples. We now provide bounds on $\widetilde{\ell}_k$.

The following example illustrate the performance of the series estimator using different bases for a real data set.

Limit Theory

$L^{2}$ Limit Theory

After we have established the set-up, we proceed to derive our results. We start with a result on the $L^{2}$ rate of convergence. Recall that $\bar{\sigma}^2=\sup_{x\in\mathcal{X}}E[\epsilon_i^2|x_i=x]$. In the theorem below, we assume that $\overline{\sigma}^2\lesssim 1$. This is a mild regularity condition.

Pointwise Limit Theory

Next we focus on pointwise limit theory (some authors refer to pointwise limit theory as local asymptotics; see Huang2003b). That is, we study asymptotic behavior of $\sqrt{n}\alpha'(\widehat{\beta}-\beta)$ and $\sqrt{n}(\widehat{g}(x)-g(x))$ for particular $\alpha\in S^{k-1}$ and $x\in\mathcal{X}$. Here $S^{k-1}$ denotes the space of vectors $\alpha$ in $\mathbb{R}^k$ with unit Euclidean norm: $\|\alpha\|=1$. Note that both $\alpha$ and $x$ implicitly depend on $n$. As we will show, pointwise results can be achieved under weak conditions similar to those we required in Theorem (ref). The following lemma plays a key role in our asymptotic pointwise normality result.

With the help of Lemma (ref), we derive our asymptotic pointwise normality result. We will use the following additional notation: $$\widetilde \Omega:= Q^{-1} E [ (\epsilon_i + r_i)^2 p_i p_i' ]Q^{-1}\text{ and }\Omega_0 := Q^{-1}E [\epsilon_i^2 p_i p_i' ]Q^{-1}. $$ In the theorem below, we will impose the condition that $\sup_{x \in \mathcal{X}} E\left[\epsilon_i^2 1\{ |\epsilon_i| > M \}|x_i =x\right] \to 0$ as $M\to \infty$ uniformly over $n$. This is a mild uniform integrability condition. Specifically, it holds if for some $m>2$, $\sup_{x \in \mathcal{X}} E[|\epsilon_{i}|^{m} | x_{i} = x] \lesssim 1$. In addition, we will impose the condition that $1\lesssim \underline{\sigma}^2$. This condition is used to properly normalize the estimator.

Uniform Limit Theory

Finally, we turn to a uniform limit theory. Not surprising, stronger conditions are required for our results to hold when compared to the pointwise case. Let $m>2$. We will need the following assumption on the tails of the regression errors.

Condition A.4 (Disturbances) Regression errors satisfy $\sup_{x \in \mathcal{X}} E[|\epsilon_{i}|^{m} | x_{i} = x] \lesssim 1$.

It will be convenient to denote $\alpha(x):=p(x)/\|p(x)\|$ in this subsection. Moreover, denote $$ \xi_k^L:=\sup_{x,x'\in\mathcal{X}: \, x\neq x'}\frac{\|\alpha(x)-\alpha(x')\|}{\|x - x' \|} $$ We will also need the following assumption on the basis functions to hold with the same $m>2$ as that in Condition A.4.

Condition A.5 (Basis) Basis functions are such that (i) $\xi_{k}^{2m/(m-2)} \log k/n \lesssim 1$, (ii) $\log \xi^L_k\lesssim \log k$, and (iii) $\log \xi_k\lesssim \log k$.

The following lemma provides uniform linearization of the series estimator and plays a key role in our derivation of the uniform rate of convergence.

The following theorem provides uniform rate of convergence of the series estimator.

After establishing the auxiliary results on the uniform rate of convergence, we present two results on inference based on the series estimator. The first result on inference is concerned with the strong approximation of a series process by a Gaussian process and is a (relatively) minor extension of the result obtained by CLR2008. The extension is undertaken to allow for a non-vanishing specification error to cover misspecified models. In particular, we make a distinction between $\widetilde \Omega= Q^{-1} E [ (\epsilon_i + r_i)^2 p_i p_i' ]Q^{-1},$ and $\Omega_0 = Q^{-1}E [\epsilon_i^2 p_i p_i' ]Q^{-1}$ which are potentially asymptotically different if $\bar{R}_{2n} \not\to_P 0$. To state the result, let $a_n$ be some sequence of positive numbers satisfying $a_n \to \infty$.

Our result on the strong approximation by a Gaussian process plays an important role in our second result on inference that is concerned with the weighted bootstrap. Consider a set of weights $h_1,\ldots,h_n$ that are i.i.d. draws from the standard exponential distribution and are independent of the data. For each draw of such weights, define the weighted bootstrap draw of the least squares estimator as a solution to the least squares problem weighted by $h_1,\ldots,h_n$, namely

For all $x\in\mathcal{X}$, denote $\widehat{g}^b(x)=p(x)'\widehat{\beta}^b$. The following theorem establishes a new result that states that the weighted bootstrap distribution is valid for approximating the distribution of the series process.

We close this section by establishing sufficient conditions for consistent estimation of $\Omega$. Recall that $Q=E[p_ip_i']=I$. In addition, denote $\Sigma = E[(\epsilon_i+r_i)^2p_ip_i']$, $\widehat Q = \mathbb{E}_n[p_ip_i']$, and $\widehat \Sigma = \mathbb{E}_n[\widehat \epsilon_i^2 p_ip_i']$ where $\widehat \epsilon_i = y_i-p_i'\widehat \beta$, and let $v_n =(E[\max_{1\leq i \leq n} | \epsilon_i|^2])^{1/2}$.

Rates and Inference on Linear Functionals

In this section, we derive rates and inference results for linear functionals $\theta(w), w\in \mathcal{I}$ of the conditional expectation function such as its derivative, average derivative, or conditional average derivative. To a large extent, with the exception of Theorem (ref), the results presented in this section can be considered as an extension of results presented in Section (ref), and so similar comments can be applied as those given in Section (ref). Theorem (ref) deals with construction of uniform confidence bands for linear functionals under weak conditions and is a new result.

By the linearity of the series approximations, the linear functionals can be seen as linear functions of the least squares coefficients $\beta$ up to an approximation error, that is $$ \theta(w) = \ell_\theta(w)' \beta + r_\theta(w), \ \ w \in \mathcal{I}, $$ where $\ell_\theta(w)' \beta$ is the series approximation, with $\ell_\theta(w)$ denoting the $k$-vector of loadings on the coefficients, and $r_\theta(w)$ is the remainder term, which corresponds to the approximation error. Throughout this section, we assume that $\mathcal{I}$ is a subset of some Euclidean space $\mathbb{R}^l$ equipped with its usual norm $\|\cdot\|$. We allow $\mathcal{I}=\mathcal{I}_n$ to depend on $n$ but for simplicity, we assume that the diameter of $\mathcal{I}$ is bounded from above uniformly over $n$. Results allowing for the case where $\mathcal{I}$ is expanding as $n$ grows can be covered as well with slightly more technicalities.

In order to perform inference, we construct estimators of $ \sigma_\theta^2(w) = \ell_\theta(w)'\Omega\ell_\theta(w)/n$, the variance of the associated linear functionals, as

In what follows, it will be convenient to have the following result on consistency of $\widehat{\sigma}_\theta(w)$:

By Lemma (ref), under our conditions, ((ref)) is uniformly consistent for $\sigma_\theta^2(w)$ in the sense that $\widehat \sigma_\theta^2(w)/ \sigma_\theta^2(w) = 1 + o_P(1)$ uniformly over $w\in \mathcal{I}$.

Pointwise Limit Theory for Linear Functionals

We now present a result on pointwise rate of convergence for linear functionals. The rate we derive is $\|\ell_\theta(w)\|/\sqrt{n}$. Some examples with explicit bounds on $\|\ell_\theta(w)\|$ are given below.

To perform pointwise inference, we consider the t-statistic: $$ t(w) = \frac{ \widehat \theta(w) - \theta(w) }{ \widehat \sigma_\theta(w)}. $$ We can carry out standard inference based on this statistic because of the following theorem.

The same comments apply here as those given in Section (ref) for pointwise results on estimating the function $g$ itself.

Uniform Limit Theory for Linear Functionals

In obtaining uniform rates of convergence and inference results for linear functionals, we will denote $$ \xi_{k,\theta}:=\sup_{w\in\mathcal{I}}\|\ell_{\theta}(w)\|\,\,\text{ and }\,\,\xi_{k,\theta}^L:=\sup_{w,w'\in\mathcal{I}: \, w\neq w'}\frac{\|\ell_{\theta}(w)-\ell_{\theta}(w')\|}{\|w - w' \|}. $$

The value of $\xi_{k,\theta}$ depends on the choice of the basis for the series estimator and on the linear functional. Newey1997 and Chen2006 provide several examples. In the case of splines with $\mathcal{X}=[0,1]^d$, it has been established that $\xi_k \lesssim \sqrt{k}$ and $\sup_{x\in\mathcal{X}}\|\partial_j^mp(x)\| \lesssim k^{1/2 + m}$; see, for example, Newey1997. With this basis we have for

where in the last example it is assumed that ${\rm supp}(\mu) \subset {\rm int}\mathcal{X}$, $x_1$ is continuously distributed with the density bounded below from zero on ${\rm supp}(\mu)$, and $x \mapsto \partial_l \mu(x)$ is continuous on ${\rm supp}(\mu)$ with $|\partial_l \mu(x)| \lesssim 1$ uniformly in $x \in {\rm supp}(\mu)$ for all $l=1,\dots,k$.

We will impose the following regularity condition on the loadings on the coefficients $\ell_\theta(w)$:

Condition A.6 (Loadings) Loadings on the coefficients satisfy (i) $\sup_{w\in\mathcal{I}}1/\|\ell_{\theta}(w)\|\lesssim 1$ and (ii) $\log \xi^L_{k,\theta}\lesssim \log k$.

The first part of this condition implies that the linear functional is normalized appropriately. The second part is a very mild restriction on the rate of the growth of the Lipschitz coefficient of the map $w \mapsto \theta(w)$.

Under Conditions A.1-A.6, results presented in Lemma (ref) on uniform linearization can be extended to cover general linear functionals considered here:

From Lemma (ref), we can derive the following theorem on uniform rate of convergence for linear functionals.

Theorem (ref) establishes uniform rates that are up to $\sqrt{\log k}$ factor agree with the pointwise rates. The requirement (ii) on the approximation error can be seen as an undersmoothing condition as discussed in Comment (ref).

Next, we consider the problem of uniform inference for linear functionals based on the series estimator. We base our inference on the t-statistic process:

We present two results for inference on linear functionals. The first result is an extension of Theorem (ref) on strong approximations to cover the case of linear functionals. As we discussed in Comment (ref), in order to perform uniform in $w\in\mathcal{I}$ inference on $\theta(w)$, we would like to approximate the distribution of the whole process ((ref)). However, one can show that this process typically does not have a limit distribution in $\ell^\infty(\mathcal{I})$. Yet, we can construct a Gaussian process that would be close to the process ((ref)) for all $w\in\mathcal{I}$ simultaneously with a high probability. Specifically, we will approximate the $t$-statistic process by the following Gaussian coupling:

where $\mathcal{N}_k$ denotes a vector of $k$ i.i.d. $N(0,1)$ random variables.

As in the case of inference on the function $g(x)$, we could also consider the use of the weighted bootstrap method to obtain a result analogous to that in Theorem (ref). For brevity of the paper, however, we do not consider weighted bootstrap method here.

The second result on inference for linear functionals is new and concerns with the problem of constructing uniform confidence bands for the linear functional $\theta(w)$. Specifically, we are interested in the confidence bands of the form

where $c_n(1-\alpha)$ is chosen so that $\theta(w)\in[\dot{\iota}(w),\ddot{\iota}(w)]$ for all $w\in\mathcal{I}$ with the prescribed probability $1-\alpha$ where $\alpha\in(0,1)$ is a user-specified level. For this purpose, we would like to set $c_n(1-\alpha)$ as the $(1-\alpha)$-quantile of $\sup_{w\in\mathcal{I}}|t(w)|$. However, this choice is infeasible because the exact distribution of $\sup_{w\in\mathcal{I}}|t(w)|$ is unknown. Instead, Theorem (ref) suggests that we can set $c_n(1-\alpha)$ as the $(1-\alpha)$-quantile of $\sup_{w\in\mathcal{I}}|t^*(w)|$ or, if $\Omega$ is unknown and has to be estimated, that we can set

where $$ \widehat{t}_n^*(w):=\frac{\ell(w)^\prime\widehat{\Omega}^{1/2}\mathcal{N}_k/\sqrt{n}}{\widehat{\sigma}_\theta(w)}, \, w\in\mathcal{I} $$ and $\mathcal{N}_k\sim N(0,I_k)$. Note that $c_n(1-\alpha)$ defined in ((ref)) can be approximated numerically by simulation. Yet, conditions of Theorem (ref) are rather strong. Fortunately, CCK2012 noticed that when we are only interested in the supremum of the process and do not need the process itself, sufficient conditions for the strong approximation can be much weaker. Specifically, we have the following theorem, which is an application of a general result obtained in CCK2012:

Construction of uniform confidence bands also critically relies on the following anti-concentration lemma due to CCK2012b (Corollary 2.1):

From Theorem (ref) and Lemma (ref), we can now derive the following result on uniform validity of confidence bands in ((ref)):

Tools: Maximal Inequalities for Matrices and Empirical Processes

In this section we collect the main technical tools that our analysis rely upon, namely Khinchin Inequalities for Matrices and Data Dependent Maximal Inequalities.

Khinchin Inequalities for Matrices

For $p\geq 1$, consider the Schatten norm $S_p$ on symmetric $k\times k$ matrices $Q$ defined by $$ \|Q\|_{S_p} = \left( \sum_{j=1}^k | \lambda_j(Q) |^p \right)^{1/p} $$ where $\lambda_1(Q),\dots,\lambda_k(Q)$ is the system of eigenvalues of $Q$. The case $p=\infty$ recovers the operator norm $\|\cdot \|$ and $p=2$ the Frobenius norm. It is obvious that for any $p\geq 1$ $$ \|Q\| \leq \|Q\|_{S_p} \leq k^{1/p} \|Q\|. $$ Therefore, setting $p=\log k$ and observing that $k^{1/\log k}=e$ for any $k\geq 1$, we get the relation:

This version of the Khinchin inequality is proven in Section 3 of Rudelson1999. We also provide some details of the proof in the Appendix. The notable feature of this inequality is the $\sqrt{\log k }$ factor instead of the $\sqrt{k}$ factor expected from the conventional maximal inequalities based on entropy. This inequality due to LustPicardPisier generalizes the Khinchin inequality for vectors. A version of this inequality was derived by GuedonRudelson using generalized entropy (majorizing measure) arguments. This is a striking example where the use of generalized entropy yields drastic improvements over the use of entropy. Prior to this, Talagrand1996 provided ellipsoidal examples where the difference between the two approaches was even more extreme.

LLN for Matrices

The following lemma is a variant of a fundamental result obtained by Rudelson1999.

For completeness, we provide the proof of this lemma in the Appendix; see also Tropp12 for a nice exposition of this result as well as many others concerning with maximal and deviation inequalities for matrices.

Maximal Inequalities

Consider a measurable space $(S, \mathcal{S})$, and a suitably measurable class of functions $\mathcal{F}$ mapping $S$ to $\Bbb{R}$, equipped with a measurable envelope function $F(z) \geq \sup_{f \in \mathcal{F}} |f(z)|$. (By “suitably measurable" we mean the condition given in Section 2.3.1 of vdV-W; pointwise measurablity and Suslin measurability are sufficient.) The covering number $N(\mathcal{F},L^{2}(Q),\varepsilon)$ is the minimal number of $L^{2}(Q)$-balls of radius $\varepsilon$ needed to cover $\mathcal{F}$. The covering number relative to the envelope function is given by

The entropy is the logarithm of the covering number.

We rely on the following result.

The proof is based on a truncation argument and maximal inequalities for uniformly bounded classes of functions developed in GK06. We recall its version.

Acknowledgements.

This paper was presented and first circulated in a series of lectures given by Victor Chernozhukov at “Stats in the Ch\^{a}teau" Statistics Summer School on “Inverse Problems and High-Dimensional Statistics" in 2009 near Paris. Participants, especially Xiaohong Chen, and one of several referees made numerous helpful suggestions. We also thank Bruce Hansen for extremely useful comments.