Local Regression Distribution Estimators

Local Regression Distribution EstimatorsSupplemental Appendix

\thispagestyle{empty}

\onehalfspacing \pagestyle{plain}

Setup

Suppose $x_1,x_2,\cdots,x_n$ is a random sample from a univariate distribution with cumulative distribution function $F(\cdot)$. Also assume the distribution function admits a (sufficiently accurate) linear-in-parameters local approximation near an evaluation point $\mathsf{x}$:

where $R(\cdot)$ is a known basis function. The parameter $\theta(\mathsf{x})$ can be estimated by the following local $L^2$ method:

where $K(\cdot)$ is a kernel function, $\mathcal{X}$ is the support of $F(\cdot)$, and $G(\cdot)$ is a known weighting function to be specified later. The local $L^2$ estimator (ref) is closely related to another estimator, which is constructed by local regression:

The local regression estimator can be equivalently expressed as $\hat\theta_{\hat{F}}$, meaning that it can be viewed as a special case of the local $L^2$ estimator, with $G(\cdot)$ in (ref) replaced by the empirical distribution function $\hat{F}(\cdot)$.

For future reference, we first discuss some of the notation we use in the main paper and this Supplemental Appendix (SA). For a function $g(\cdot)$, we denote its $j$-th derivative as $g^{(j)}(\cdot)$. For simplicity, we also use the “dot” notation to denote the first derivative: $\dot{g}(\cdot) = g^{(1)}(\cdot)$. Assume $g(\cdot)$ is suitably smooth on $[\mathsf{x}-\delta,\mathsf{x})\cup(\mathsf{x},\mathsf{x}+\delta]$ for some $\delta>0$, but not necessarily continuous or differentiable at $\mathsf{x}$. If $g(\cdot)$ and its one-sided derivatives can be continuously extended to $\mathsf{x}$ from the two sides, we adopt the following special notation:

With $j=0$, the above is simply $g_u = \mathds{1}(u<0)g(\mathsf{x}-)+\mathds{1}(u\geq 0)g(\mathsf{x}+)$. Also for $j=1$, we use $\dot{g}_u = g^{(1)}_u$. Convergence in probability and in distribution are denoted by $\overset{\mathbb{P}}{\to}$ and $\rightsquigarrow$, respectively, and limits are taken with respect to the sample size $n$ going to infinity unless otherwise specified. We use $|\cdot|$ to denote the Euclidean norm.

The following matrices will feature in asymptotic expansions of our estimators:

and

Now we list the main assumptions.

Pointwise Distribution Theory

We discuss pointwise (i.e., for a fixed evaluation point $\mathsf{x}\in\mathcal{X}$) large-sample properties of the local $L^2$ estimator (ref), and that of the local regression estimator (ref). For ease of exposition, we suppress the dependence on the evaluation point $\mathsf{x}$ whenever possible.

Local $L^2$ Distribution Estimation

With simple algebra, the local $L^2$ estimator in (ref) takes the following form

We can further simplify the above. First note that the “denominator” can be rewritten as

The same technique can be applied to the “numerator”, which leads to

The above provides a further expansion of the local $L^2$ estimator into a term that contributes as bias, and another term that contributes asymptotically to the variance.

The large-sample properties of the local $L^2$ estimator (ref) are as follows.

For valid inference, one needs to construct standard errors. To start, note that $\Gamma_{h}$ is known, and hence we only need to estimate $\Sigma_{h}$. Consider the following:

where $\hat{F}(\cdot)$ is the empirical distribution function. The following theorem shows that standard errors constructed using $\hat{\Sigma}_h$ are consistent.

Local Regression Distribution Estimation

The local regression distribution estimator (ref) can be understood as a special case of the local $L^2$ estimator by setting $G=\hat{F}$ (i.e., using the empirical distribution as the design). However, the empirical measure $\hat{F}$ is not smooth, so that large-sample properties of the local regression estimator cannot be deduced directly from Theorem (ref). In this subsection, we will show that estimates obtained by the two approaches, (ref) and (ref), are asymptotically equivalent under suitable regularity conditions. To be precise, we establish the equivalence of the local regression distribution estimator, $\hat\theta$, and the (infeasible) local $L^2$ distribution estimator, $\hat{\theta}_F$ (i.e., using $F$ as the design weighting in (ref)). As before, we suppress the dependence on the evaluation point $\mathsf{x}$.

First, the local regression estimator can be written as

where

and $\Gamma_h$ is defined as before with $G=F$.

To proceed, we further expand as follows

The last two terms correspond to the leave-in bias and the approximation bias, respectively. We further decompose the first term with conditional expectation:

The following theorem studies the large-sample properties of each term in the above decomposition, and shows that the local regression distribution estimator is asymptotically equivalent to the local $L^2$ estimator by setting $G=F$, and hence it is asymptotically normally distributed.

We now discuss how to construct standard errors in the local regression framework. Note that $\Gamma_{h}$ can be estimated by $\hat{\Gamma}_h$, whose properties have already been studied in Theorem (ref)(i). To estimate $\Sigma_{h}$, we propose the following

where $\hat{F}(\cdot)$ is the empirical distribution function. The following theorem shows that standard errors constructed using $\hat{\Sigma}_h$ are consistent.

Efficiency

For ease of presentation, we focus on the (infeasible) local $L^2$ distribution estimator $\hat\theta_F$,

but all the results in this section are applicable to the local regression distribution estimator $\hat\theta$, as we showed earlier that it is asymptotically equivalent to $\hat\theta_F$. In addition, we consider a specific basis:

where $P(u)$ is a polynomial basis of order $p$:

and $Q(u)$ is a scalar function, and hence is a “redundant regressor.” Without $Q(\cdot)$, the above reduces to the local polynomial estimator of Cattaneo-Jansson-Ma_2020_JASA. See Section (ref) and Example (ref) for an introduction.

We consider additional regressors because they may help improve efficiency (i.e., reduce the asymptotic variance). Following Assumption (ref), we assume there exists a scalar $\upsilon_h$ (depending on $h$) such that $\upsilon_hQ(u)=Q(u/h)$. Therefore, $\Upsilon_h$ is a diagonal matrix containing $1,h^{-1},h^{-2},\cdots,h^{-p},\upsilon_h$. As we consider a (local) polynomial basis, it is natural to impose smoothness assumptions on $F(\cdot)$. In particular,

Under the above assumption, the approximation error satisfies $\varrho(h)=O(h^{p+1})$, and the parameter $\theta$ can be partitioned into the following:

We first state a corollary, which specializes Theorem (ref) to the polynomial basis (ref).

Effect of Orthogonalization

To start, consider the following (sequentially) orthogonalized basis:

where

The above transformation can be represented by the following:

where $\Lambda_h$ is a nonsingular upper triangular matrix. (Note that the matrix $\Lambda_h$ depends on the bandwidth only because we would like to handle both interior and boundary evaluation points. If, for example, we fix the evaluation point to be in the interior of the support of the data, then $\Lambda_h$ is a fixed matrix and no longer depends on $h$. Alternatively, one could also use the notation “$\Lambda_\mathsf{x}$” to denote such dependence.) Now consider the following orthogonalized local polynomial $L^2$ estimator

To discuss its properties, we partition the estimator and the target parameter as

where $\hat\theta_{1,F}^\perp$ is the first element of $\hat\theta_F^\perp$ and $\hat\theta_{Q,F}^\perp$ is the last element of $\hat\theta_F^\perp$. Similarly, we can partition the target parameter,

so that $\theta^\perp_{1}$ is the first element of $\Lambda_h^{-1}\theta$ and $\theta^\perp_{Q}$ is the last element of $\Lambda_h^{-1}\theta$. As $\theta_{Q}=0$, simple least squares algebra implies

Note that, in general, $\theta^\perp_{1}\neq \theta_{1}$, meaning that after orthogonalization, the intercept of the local polynomial estimator no longer estimates the distribution function $F(\mathsf{x})$.

The following corollary gives the large-sample properties of the orthogonalized local polynomial estimator, excluding the intercept.

Optimal $Q$

Now we discuss the optimal choice of $Q$, which minimizes the asymptotic variance of the minimum distance estimator. Recall from the main paper that, with orthogonalized basis, the minimum distance estimator of ${f}^{(\ell)}(\mathsf{x})$, for $0\leq \ell\leq p-1$, has an asymptotic variance

where $e_\ell$ is the $(\ell+1)$-th standard basis vector. In subsequent analysis, we drop the multiplicative factor $f(\mathsf{x})$.

Let $p_\ell(u)$ be defined as

then the objective is to maximize

Alternatively, we would like to solve (recall that $Q(u)$ is a scaler function)

To proceed, define the following transformation for a function $g(\cdot)$:

This transformation satisfies two important properties, which are summarized in the following lemma.

With the previous lemma, we can rewrite the maximization problem as

Before closing this section, we make several remarks on the variance bound derived in Theorem (ref), as well as to what extent it is achievable.

Uniform Distribution Theory

We establish distribution approximation for $\{\hat\theta_G(\mathsf{x}),\mathsf{x}\in\mathcal{I}\}$ and $\{\hat\theta(\mathsf{x}),\mathsf{x}\in\mathcal{I}\}$, which can be viewed as processes indexed by the evaluation point $\mathsf{x}$ in some set $\mathcal{I}\subseteq \mathcal{X}$. Recall the definition of $\Gamma_{h,\mathsf{x}}$ and $\Sigma_{h,\mathsf{x}}$ from Section (ref), and we define $\Omega_{h,\mathsf{x}} = \Gamma_{h,\mathsf{x}}^{-1}\Sigma_{h,\mathsf{x}}\Gamma_{h,\mathsf{x}}^{-1}$.

We first study the following (infeasible) centered and Studentized process:

where we consider linear combinations through a (known) vector $c_{h,\mathsf{x}}$, which can depend on the sample size through the bandwidth $h$, and can depend on the evaluation point. Again, we use the subscript $G$ to denote the local $L^2$ approach with $G$ being the design distribution. To economize notation, let

then we can conveniently rewrite (ref) as

and hence the centered and Studentized process $\mathfrak{T}_G(\cdot)$ takes a kernel form. The difference compared to standard kernel density estimators, however, is that the (equivalent) kernel in our case changes with the evaluation point, which is why our estimator is able to adapt to boundary points automatically. From the pointwise distribution theory developed in Section (ref), the process $\mathfrak{T}_G(\mathsf{x})$ has variance

We can also compute the covariance as

where $\Omega_{h,\mathsf{x},\mathsf{y}} = \Gamma_{h,\mathsf{x}}^{-1}\Sigma_{h,\mathsf{x},\mathsf{y}}\Gamma_{h,\mathsf{y}}^{-1}$, and

Of course one can further expand the above, but this is unnecessary for our purpose.

For future reference, let

We also make some additional assumptions.

For any $h>0$ (and fixed $n$), we can define a centered Gaussian process, $\{\mathfrak{B}_G(\mathsf{x}):\mathsf{x}\in\mathcal{I}\}$, which has the same variance-covariance structure as the process $\mathfrak{T}_G(\cdot)$. The following lemma shows that it is possible to construct such a process, and that $\mathfrak{T}_G(\cdot)$ and $\mathfrak{B}_G(\cdot)$ are “close in distribution.”

Next we consider the continuity property of the implied (equivalent) kernel of the process $\mathfrak{T}_G(\cdot)$, which will help control the complexity of the Gaussian process $\mathfrak{B}_G(\cdot)$. To be precise, define the pseudo-metric $\sigma_G(\mathsf{x},\mathsf{y})$ as

we would like to provide an upper bound of $\sigma_G(\mathsf{x},\mathsf{y})$ in terms of $|\mathsf{x}-\mathsf{y}|$ (at least for all $\mathsf{x}$ and $\mathsf{y}$ such that $|\mathsf{x}-\mathsf{y}|$ is small enough).

Local $L^2$ Distribution Estimation

We first discuss the covariance estimator. For the local $L^2$ distribution estimator, let $\hat{\Omega}_{h,\mathsf{x},\mathsf{y}} = \Gamma_{h,\mathsf{x}}^{-1}\hat\Sigma_{h,\mathsf{x},\mathsf{y}}\Gamma_{h,\mathsf{y}}^{-1}$ with $\hat\Sigma_{h,\mathsf{x},\mathsf{y}}$ given by

The next lemma characterizes the convergence rate of $\hat{\Omega}_{h,\mathsf{x},\mathsf{y}}$.

We now consider the estimator $c^\prime_{h,\mathsf{x}}\hat\theta_G(\mathsf{x})$. From (ref) and (ref), one has

In the following lemma, we analyze the two terms in the above decomposition.

Now we state the main result on uniform distributional approximation.

The following theorem shows that a feasible approximation to the process $\mathfrak{B}_G(\cdot)$ can be achieved by simulating a Gaussian process with covariance estimated from the data. In the following, we use $\mathbb{P}^\star$, $\mathbb{E}^\star$ and $\mathbb{C}\mathrm{ov}^\star$ to denote the probability, expectation and covariance operator conditioning on the data $X_n=(x_1,x_2,\dots,x_n)'$.

Local Regression Distribution Estimation

Now we consider the local regression estimator $\{\hat\theta(\mathsf{x}),\mathsf{x}\in\mathcal{I}\}$. As before, we first discuss the construction of the covariance $\Omega_{h,\mathsf{x},\mathsf{y}}$. Let $\hat{\Omega}_{h,\mathsf{x},\mathsf{y}} = \hat{\Gamma}_{h,\mathsf{x}}^{-1}\hat{\Sigma}_{h,\mathsf{x},\mathsf{y}}\hat{\Gamma}_{h,\mathsf{y}}^{-1}$. Construction of $\hat{\Gamma}_{h,\mathsf{x}}$ is given in Section (ref). The following lemma shows that $\hat{\Gamma}_{h,\mathsf{x}}$ is uniformly consistent.

Construction of $\hat{\Sigma}_{h,\mathsf{x},\mathsf{y}}$ also mimics that in Section (ref). To be precise, we let

where $\hat{F}(\cdot)$ remains to be the empirical distribution function. The following result justifies consistency of $\hat{\Omega}_{h,\mathsf{x},\mathsf{y}}$.

The following is an expansion of $T(\cdot)$.

Finally we have the following result on uniform distributional approximation for the local regression distribution estimator, as well as a feasible approximation by simulating from a Gaussian process with estimated covariance.

{

Proofs

Proof of Theorem (ref)

Part (i)

The bias term can be bounded by

Part (ii)

The variance can be found as

To establish asymptotic normality, we verify the Lyapunov condition with a fourth moment calculation. Take $c$ to be a nonzero vector of conformable dimension, and we employ the Cramer-Wold device:

If $c^\prime \Sigma_h c$ is bounded away from zero as the bandwidth decreases, the above will have order $n^{-1}$, as $K(\cdot)$ is bounded and compactly supported and $R(\cdot)$ is locally bounded. Therefore, the Lyapunov condition holds in this case. The more challenging case is when $c^\prime \Sigma_h c$ is of order $h$. In this case, it implies

Now consider the fourth moment. The leading term is

meaning that for the Lyapunov condition to hold, we need the requirement that $nh\to \infty$.

Part (iii)

This follows immediately from Part (i) and (ii).

Proof of Theorem (ref)

To study the property of $\hat\Sigma_{h}$, we make the following decomposition:

First, it is obvious that term (II) is of order $O_{\mathbb{P}}(1/n)$. Term (I) requires more delicate analysis. Let $c$ be a vector of unit length and suitable dimension, and define

Then

which implies that

With the same argument used in the proof of Theorem (ref), one can show that

which implies

Proof of Theorem (ref)

Part (i)

For the “denominator,” its variance is bounded by

Therefore, under the assumption that $h\to 0$ and $nh\to \infty$, we have

which further implies that

Part (ii)

The order of the leave-in bias is clearly $1/n$. For the approximation bias (ref), we obtained its mean in the proof of Theorem (ref) by setting $G=F$, which has an order of $\varrho(h)$. The approximation bias has a variance of order

Therefore,

provided that $nh\to \infty$.

Part (iii)

We compute the variance of the U-statistic (ref). For simplicity, define

which satisfies $\mathbb{E}[u_{ij}]=\mathbb{E}[u_{ij}|x_i]=\mathbb{E}[u_{ij}|x_j]=0$. Therefore

meaning that

Part (iv)

This follows immediately from Part (i)--(iii) and Theorem (ref).

Proof of Theorem (ref)

We first decompose $\hat{\Sigma}_{h}$ into two terms,

where we use $R_i = R(x_i-\mathsf{x})$ and $W_i = K((x_i-\mathsf{x})/h)/h$ to conserve space.

(II) satisfies

It is obvious that

As for the second part, we have

which holds as long as $nh\to \infty$. Then it further implies that

To analyze (I), we further expand this term into “diagonal” and “non-diagonal” sums:

By calculating the expectation of the absolute value of the summands above, it is straightforward to show

Therefore, we have

To proceed, define

Then we can further decompose (I.1) into

We have already analyzed (I.1.1) in Theorem (ref), which suggests

Now we study (I.1.2), which satisfies

With variance calculation, it is easy to see that

Therefore we have

since (I.1.2.1) and (I.1.2.2) are transpose of each other. To close the proof, we calculate the variance of the last term in the above.

The expectation is further given by (note that $i$, $j$ and $i'$ are assumed to be distinct indices)

If $c^\prime \Sigma_{h} c = O(1)$, then the above will have order $h$, which means

If $c^\prime \Sigma_{h} c = O(h)$, however, $\mathbb{E}\left[ c^\prime u_{ij}\bar{u}_i^\prime c c^\prime u_{i'j}\bar{u}_{i'}^\prime c \right]$ will be $O(1)$, which will imply that

As a result, we have

Now consider

From the analysis of $\hat\Sigma_h$, we have

For the second term, we have

The third term has order

Proof of Corollary (ref)

This follows directly from Theorem (ref).

Proof of Corollary (ref)

To understand (ref), note that

which means $\hat\theta_{F}^\perp = \Lambda_h^{-1}\hat\theta_{F}$. Then we have (up to an approximation bias term)

We first discuss the transformed parameter vector $\Lambda_h^{-1}\theta_0$. By construction, the matrix $\Lambda_h$ takes the following form:

where $c_{i,j}$ are some constants (possibly depending on $h$). Therefore, the above matrix differs from the identity matrix only in its first row and in the last column. This observation also holds for $\Lambda_h^{-1}$. Since the last component of $\theta_0$ is zero (because the extra regressor $Q_h(\cdot)$ is redundant), we conclude that, except for the first element, $\Lambda_h\theta$ and $\theta$ are identical. More specifically, let $I_{-1}$ be the identity matrix excluding the first row:

which is used to extract all elements of a vector except for the first one, then by Theorem (ref),

where $\theta^\perp_{P,F}$ contains the second to the $p+1$-th element of $\theta^\perp_{F}$, and $\theta^\perp_{Q,F}$ is the last element.

Now we discuss the covariance matrix in the above display. Due to orthogonalization, $\Gamma_h^\perp$ is block diagonal. To be precise,

Finally, using the structure of $\Lambda_h$ and $\Upsilon_h$, we have

The form of $\Sigma_h^\perp$ is quite involved, but with some algebra, and using the fact that the basis $R(\cdot)$ (or $R^\perp(\cdot)$) includes a constant and polynomials, one can show the following:

where $\Upsilon_{-1, h}$, $\Gamma_{-1,h}^\perp$ and $\Sigma_{-1,h}^\perp$ are obtained by excluding the first row and the first column of $\Upsilon_{h}$, $\Gamma_{h}^\perp$ and $\Sigma_{h}^\perp$, respectively:

and

Proof of Lemma (ref)

Part (i)

To start,

where to show the last equality, we used the fact that $v_1\leq \frac{\overline{x}-\mathsf{x}}{h}\wedge 1$ and $v_2\leq \frac{\overline{x}-\mathsf{x}}{h}\wedge 1$ for the outer double integral.

Part (ii)

For this part,

Again, to show the last equality, we used the fact that $v\leq \frac{\overline{x}-\mathsf{x}}{h}\wedge 1$ for the outer integral.

Proof of Theorem (ref)

To find a bound of the maximization problem, we note that for any $c\in\mathbb{R}^{p-1}$, one has

due to the constraint. Therefore, an upper bound of the objective function is (due to the Cauchy-Schwartz inequality)

which is minimized by setting

As a result, an upper bound of (ref) is

We may further simplify the above. First,

Second, note that

As a result, an upper bound of (ref) is

Additional Preliminary Lemmas

\noindentProof. Let $A_\varepsilon$ be an $\varepsilon$-covering of $A$, then

Next we apply the union bound and Bernstein's inequality:

Now take $u$ sufficiently large, then the above is further bounded by:

which tends to zero if $\log N(\varepsilon, A, |\cdot|)\to\infty$ and

in which case we have

We can apply the same technique to the other term, and obtain

where $\gamma_{\varepsilon}$ is any sequence satisfying

\qed

Proof of Theorem (ref)

To bound the distance between the two processes, $\tilde{\mathfrak{T}}_G(\cdot)$ and $\mathfrak{B}_G(\cdot)$, we employ the proof strategy of Gine-Koltchinskii-Sakhanenko_2004_PTRF. Recall that $F$ denotes the distribution of $x_i$, and we define

Take $v<v'$ in $[0,1]$, we have

Therefore, the function $\mathcal{K}_{h,\mathsf{x}}\circ F^{-1}(\cdot)$ has a total variation bounded by

It is well-known that functions of bounded variation can be approximated (pointwise) by convex combination of indicator functions of half intervals. To be more precise,

Following (2.3) and (2.4) of Gine-Koltchinskii-Sakhanenko_2004_PTRF, we have

where $C_5$ is some universal constant.

Proof of Lemma (ref)

Take $|\mathsf{x}-\mathsf{y}|\leq \varepsilon$ to be some small number, then

For term (I), its variance (replace the placeholder $x$ by $x_i$) is

Term (II) has variance

where the order $\frac{\varepsilon}{h}\wedge 1$ comes from the difference $\Gamma_{h,\mathsf{x}}^{-1} - \Gamma_{h,\mathsf{y}}^{-1}$.

Next for term (III), we have

The first term has bound

The third term has bound

Finally, the second term can be bounded as

Overall, we have that

Given our assumptions on the basis function and on the kernel function, it is obvious that term (IV) has variance

The bound on $\mathbb{E}[\sup_{\mathsf{x}\in\mathcal{I}}|\mathfrak{B}_G(\mathsf{x})|]$ can be found by standard entropy calculation, and the bound on $\mathbb{E}[\sup_{\mathsf{x}\in\mathcal{I}}|\mathfrak{T}_G(\mathsf{x})|]$ is obtained by the following fact

and that

which follows from Theorem (ref) and our assumption that $\log n/(nh)\to 0$.

Proof of Lemma (ref)

We adopt the following decomposition (the integration is always on $\frac{\mathcal{X}-\mathsf{y}}{h}\times\frac{\mathcal{X}-\mathsf{x}}{h}$, unless otherwise specified):

By the uniform convergence of the empirical distribution function, we have that

From the definition of $\Sigma_{h,\mathsf{x},\mathsf{y}}$, we know that

As (I) is a sum of bounded terms, we can apply Lemma (ref) and easily show that

Proof of Lemma (ref)

We rewrite (ref) as

where the final bound holds uniformly for $\mathsf{x}\in\mathcal{I}$.

Next, we expand term (ref) as

Term (I) can be easily bounded by

Proof of Theorem (ref)

The claim follows from Theorem (ref) and previous lemmas.

Proof of Theorem (ref)

Let $\mathcal{I}_\varepsilon$ be an $\varepsilon$-covering (with respect to the Euclidean metric) of $\mathcal{I}$, and assume $\varepsilon\leq h$. Then the process ${\mathfrak{B}}_G(\cdot)$ can be decomposed into:

where $\Pi_{\mathcal{I}_\varepsilon}:\mathcal{I}\to \mathcal{I}_\varepsilon$ is a mapping satisfying:

We first study the properties of ${\mathfrak{B}}_G(\mathsf{x}) - {\mathfrak{B}}_G(\Pi_{\mathcal{I}_\varepsilon}(\mathsf{x}))$. With standard entropy calculation, one has:

where

for some $C>0$ that does not depend on $\varepsilon$ and $h$, and $N(\lambda, \mathcal{I}, \sigma_G)$ is the covering number of $\mathcal{I}$ measured by the pseudo metric $\sigma_G(\cdot,\cdot)$, which satisfies

Therefore, we have

A similar bound holds for the process $\hat{\mathfrak{B}}_G(\cdot)$ due to the uniform consistency of the covariance estimator.

Now consider the discretized version of ${\mathfrak{B}}_G(\cdot)$ and $\hat{\mathfrak{B}}_G(\cdot)$. By applying Lemmas (ref) and (ref), we directly obtain the following bound:

As $\varepsilon$ appears in (I) polynomially but only logarithmically in (II), it is possible to choose $\varepsilon$ sufficiently small so that the discretization error becomes negligible. Therefore,

Proof of Lemma (ref)

We apply Lemma (ref). For simplicity, assume $R(\cdot)$ is scalar, and let

Then it is easy to see that

Let $|\mathsf{x} - \mathsf{y}|\leq \varepsilon\leq h$, we also have

where $M$ is some constant that does not depend on $n$, $h$ or $\varepsilon$. Then it is easy to see that

Now take $\varepsilon = \sqrt{h\log n/n}$, then $\log N(\varepsilon, \mathcal{I}, |\cdot|)=O(\log n)$. Lemma (ref) implies that

Proof of Lemma (ref)

Let $R_i(\mathsf{x}) = R(x_i-\mathsf{x})$ and $W_i(\mathsf{x}) = K((x_i-\mathsf{x})/h)/h$, then we split $\hat{\Sigma}_{h,\mathsf{x},\mathsf{y}}$ into two terms,

(II) satisfies

It is obvious that

As for the second part, one can employ the same technique used to prove Lemma (ref) and show that

implying that

For (I), we first define

and

Then

Term (I.1) has been analyzed in Lemma (ref), which satisfies

Term (I.2) has expansion:

By the same technique of Lemma (ref), one can show that

We need a further decomposition to make (I.2.1) a degenerate U-statistic:

(I.2.1) has zero mean. By discretizing $\mathcal{I}$ and apply Bernstein's inequality, one can show that the (I.2.1.1) has order $O_{\mathbb{P}}\left(\sqrt{\log n/n}\right)$.

For (I.2.1.2), we first discretize $\mathcal{I}$ and then apply a Bernstein-type inequality (Lemma (ref)) for degenerate U-statistics, which gives an order

Overall, we have

and the same bound applies to (I.3).

For (I.4), one can show that

which means

under our assumption that $\log n/(nh^2)\to 0$.

As a result, we have

Now take $c$ to be a generic vector. Then we have

From the analysis of $\hat\Sigma_{h,\mathsf{x},\mathsf{y}}$, we have

For the second term, we have

The same bound holds for the third term.

Proof of Lemma (ref)

We decompose (ref) as

As both $\hat\Gamma_{h,\mathsf{x}}$ and $c_{h,\mathsf{x}}^\prime\Upsilon_{h}\hat{\Omega}_{h,\mathsf{x}}\Upsilon_{h}c_{h,\mathsf{x}}$ are uniformly consistent, term (I) has order

For (II), we can employ the same technique used to prove Lemma (ref) and show that

where the leading order in the above represents the mean of ${R((x_i-\mathsf{x})/h)[1-F(x_i)]\frac{1}{h}K(\frac{x_i-\mathsf{x}}{h})}$.

Next, term (ref) is bounded by

Employing the same argument used to prove Lemma (ref), we have

To bound term (II), recall that $K(\cdot)$ is supported on $[-1,1]$, meaning that

Term (II.2) has the bound $\sup_{\mathsf{x}\in\mathcal{I}}\varrho(h,\mathsf{x})$. Term (II.1) can be bounded by mean and variance calculations and adopting the proof of Lemma (ref), which leads to

To show the last conclusion, define the following:

then $n^{-2}\sum_{i,j=1, i\neq j}^n u_{ij}(\mathsf{x})$ is a degenerate U-statistic. We rewrite (ref) as

As before, we have

Now we consider (II). Let $\mathcal{I}_\varepsilon$ be an $\frac{\varepsilon}{2}$-covering of $\mathcal{I}$, we have

We rely on the concentration inequality in Lemma (ref) for degenerate second order U-statistics. By our assumptions, $A$ can be chosen to be $C_1h^{-1}$ where $C_1$ is some constant that is independent of $\mathsf{x}$. Similarly, $B$ can be chosen to be $C_2\sqrt{n}h^{-1}$ for some constant $C_2$ which is independent of $\mathsf{x}$, and $D$ can be chosen as $C_3nh^{-1/2}$ for some $C_3$ independent of $\mathsf{x}$. Therefore, by setting $\eta=K\log n/\sqrt{n^2h}$ for some large constant $K$, we have

As $\varepsilon$ is at most polynomial in $n$, the above tends to zero for all $K$ large enough, which implies

With tedious but still straightforward calculations, it can be shown that

and to match the rates, let $\varepsilon = h\log n/\sqrt{n^2h}$.

Proof of Lemma (ref)

The proof resembles that of of Lemma (ref).

Proof of Theorem (ref)

The proof resembles that of Theorem (ref).

Proof of Theorem (ref)

The proof resembles that of Theorem (ref).

}

\singlespacing