Improved Density and Distribution Function Estimation

Improved Density and Distribution Function Estimation

}

\numberwithin{remark}{section} \numberwithin{equation}{section} \numberwithin{assumption}{section} \numberwithin{theorem}{section}

Introduction

In many statistical and economic applications, additional distributional information about the data observation $d_{z}$-vector $z$ may be available in the form of moment restrictions on its distribution. These constraints may arise from a particular economic or physical law, e.g., chen1997, be implied by estimating equations, qin1994, or correspond to known population moments of another observable random vector correlated with $z$, e.g., in survey samples with auxiliary population information available from census data, e.g., chen1993 and qin1994. The primary purpose of the paper is to explore the advantages of this additional information for the estimation of the density and distribution function of a scalar residual-like function of $z$ which may depend on unknown parameters.

To this end, let $g(z,\beta)$ denote a $d_{g}$-vector of known functions of the data observation $d_{z}$-vector $z\in\mathcal{Z}$ and the $d_{\beta}$-vector $\beta\in\mathcal{B}$ of parameters where the sample space $\mathcal{Z}\subseteq\mathbbm{R}^{d_{z}}$ and parameter space $\mathcal{B}\subset\mathbbm{R}^{d_{\beta}}$ with $d_{\beta}\leq d_{g}$. The moment indicator vector $g(z,\beta)$ will form the basis for inference in the following discussion and analysis. In particular, it is assumed that the true value $\beta_{0}$ taken by $\beta$ uniquely satisfies the population unconditional moment equality condition

where $\operatorname{E}[\cdot]$ denotes expectation taken with respect to the true population probability law of $z$. The true parameter value $\beta_{0}$ is generally unknown, but can also be fully or partially known in particular applications.

Models specified in the form of unconditional moment restrictions (ref) convey partial information about the distribution $F^{z}$ of $z$ and are ubiquitous in economics; see, e.g., the monographs hall2005 and matyas1999. Many other commonly used models lead to estimators that can be reformulated as solutions to a set of moment restrictions. Clearly, models given by conditional moment restrictions imply (ref). Traditionally, such models are estimated by the generalised method of moments (GMM). However, the performance of GMM estimators and associated test statistics is often poor in finite samples, which has lead to the development of a number of (information-theoretic) alternatives to GMM.

This paper focuses on the class of generalised (G) empirical likelihood (EL) estimators, which has attractive large sample properties; see, e.g., newey2004, smith1997,smith2011, and parente2014 for a recent review. Special cases of GEL include EL, owen1988,owen1990, qin1994, exponentially tilting (ET), corcoran1998, kitamura1997, imbens1998, and continuous-updating (GMM) estimators (CUE), hansen1996; see also Euclidean EL, antoine2007. Of these estimators, EL has the attractive property of being Bartlett-correctable; see chen2007.

When the parameter vector $\beta_{0}$ is overidentified by the moment restriction (ref), i.e., $d_{\beta}<d_{g}$, these constraints generally carry useful additional information about $F^{z}$. Given a random sample $z_{i}$, $i=1,\ldots,n$, of observations on $z$, such information is captured by the associated (G)EL implied probabilities $\pi_{i}$, $i=1,\ldots,n$, which enable a nonparametric description of $F^{z}$ satisfying the moment condition (ref) given by the estimator $F_{\pi}^{z}(z)=\sum_{i=1}^{n}\pi_{i}\mathbbm{1}\{z_{i}\leq z\}$, where $\mathbbm{1}\{\cdot\}$ denotes the indicator function, back1993, qin1994. In the absence of the moment information (ref) or when $\beta_{0}$ is just identified, $d_{\beta}=d_{g}$, $F_{\pi}^{z}(z)$ reduces to the empirical distribution function (EDF) $F_{n}^{z}(z)=n^{-1}\sum_{i=1}^{n}\mathbbm{1}\{z_{i}\leq z\}$. In general, if $d_{\beta}<d_{g}$, $F_{\pi}^{z}(z)$ is a more efficient estimator of $F^{z}$ than the EDF $F_{n}^{z}(z)$ reflecting the value of the overidentifying information in (ref). This observation suggests therefore that estimation of the functionals of $F^{z}$, $T(F^{z})$, by $T(F_{\pi}^{z})$ rather than $T(F_{n}^{z})$ will be similarly more efficient. Indeed this is the case when estimating expectations of certain known functions of $z$, see brown1998. A similar advantage is apparent for EL estimation of quantile functions with known $\beta_{0}$, e.g., chen1993 and zhang1995, general EL-based quantile estimation, yuan2014, and EL-based kernel estimation of a univariate density function, e.g., chen1997 and zhang1998.

The concern of this paper is with efficient kernel estimation of the probability density (p.d.f.) and distribution (c.d.f.) functions of a scalar-valued function $u(z,\beta_{0})$ of the data observation $z$ with either known or unknown parameter vector $\beta_{0}$. The former case, when $\beta_{0}$ is known, is the classical situation briefly mentioned above. The central case of interest, when $\beta_{0}$ is unknown, is estimation of the p.d.f and c.d.f. of an error term based on the estimated residuals. Such estimates are routinely computed by practitioners and are used for both visual diagnostics, e.g., potentially revealing omitted structure such as multimodality or other features of interest, and formal diagnostic tests, e.g., goodness-of-fit and tests of parametric assumptions on the error distribution. The importance of obtaining residual density estimates with good (higher order) properties can hardly be understated. Yet, as discussed below, simply applying standard kernel estimators with default bandwidths to estimated residuals may result in an inconsistent p.d.f. or c.d.f. estimators as further conditions on the kernel function and bandwidth are generally required. Similar conclusions have been reached elsewhere in related literature on residual density estimation in nonparametric regression and other settings; see, e.g., ahmad1992, cheng2004, kiwitt2008, gyorfi2012 and the discussion and references in bott2013.

When $\beta_{0}$ is known, kernel density and distribution function estimators exploiting the (G)EL implied probabilities instead of the uniform EDF $n^{-1}$ weights achieve a reduction of higher order variance due to the systematic use of the extra moment information in (ref). The efficiency gains are first order asymptotically in the c.d.f. case and second order for p.d.f. estimation. In contradistinction, for residual p.d.f. and c.d.f. estimation, such gains will not always be realised. One can, however, expect efficiency gains from the knowledge that the mean of residuals is zero.

The outline of the paper is as follows. Section (ref) briefly describes (G)EL estimation and the associated (G)EL implied probabilities. The main results concerning p.d.f. and c.d.f. estimators are given in Sections (ref) and (ref) for both known and unknown $\beta_{0}$ cases. The finite sample performance of the proposed estimators is evaluated via a simulation study reported in Section (ref). Section (ref) concludes. Supplement Supplement (ref): Proofs and (ref): Examples in the Supplementary Information respectively details some additional assumptions for and the proofs of the results in the main text and analyses a number of examples to illustrate the the properties of the estimators developed in the paper.

Generalised Empirical Likelihood

The GEL class of estimators for $\beta_{0}$ is defined in terms of a real valued scalar carrier function $\rho:\mathcal{V}\mapsto\mathbbm{R}$ that is concave on an open interval $\mathcal{V}$ containing zero with derivatives $\rho^{(j)}(v)=\mathrm{d}^{j}\rho(v)/\mathrm{d}v^{j}$ and $\rho_{j}=\rho^{(j)}(0)$, $j=1,2,\ldots$, normalized without loss of generality such that $\rho_{1}=\rho_{2}=-1$. The special cases $\rho(v)=\ln(1-v)$ for $\mathcal{V}=(-\infty, 1)$, $\rho(v)=-\exp(v)$ and $\rho(v)=-v^{2}/2-v$ correspond to EL, ET and CUE respectively and are all members of the cressie1984 family where $\rho(v)=-(1+\gamma v)^{(\gamma +1)/\gamma}/(\gamma+1)$.

Given a random sample $z_{i}$, $i=1,\ldots,n$, of size $n$ of observations on the $d_{z}$-dimensional vector $z$, let $g_{i}(\beta)=g(z_{i},\beta)$, $g_{i}=g_{i}(\beta_{0})$, and $G_{i}(\beta)=\partial g(z_{i},\beta)/\partial\beta^{\top}$, $G_{i}=G_{i}(\beta_{0})$, $i=1,\ldots,n$. Also let $\Lambda _{n}(\beta)=\{\lambda : \lambda^{\top}g_{i}(\beta)\in\mathcal{V}, i=1,\ldots,n\}$. The GEL criterion $P_{n}(\beta,\lambda)$ is defined by $P_{n}(\beta,\lambda)=n^{-1}\sum_{i=1}^{n}\rho(\lambda^{\top}g_{i}(\beta))-\rho(0)$, with $\lambda$ a $d_{g}$-vector of auxiliary parameters, each element of which corresponding to an element of the moment function vector $g(z,\beta)$; for members of the cressie1984 family of power divergence criteria $\lambda$ is the Lagrange multiplier vector associated with imposition of the moment restriction (ref). The GEL estimator $\hat{\beta}$ is the solution to the saddle point problem

If Supplement (ref): Assumptions (ref) and (ref) are satisfied, in particular, the population Jacobian $G=\operatorname{E}[\partial g(z,\beta_{0})/\partial\beta^{\top}]$ and variance $\Omega =\operatorname{E}[g(z,\beta_{0})g(z,\beta_{0})^{\top}]$ matrices are full column rank and positive definite respectively, then all GEL estimators share the same first order large sample properties, see, e.g., newey2004, i.e., $n^{1/2}(\hat{\beta} -\beta _{0})\xrightarrow{d}N(0,\Sigma)$, achieving the semiparametric efficiency lower bound $\Sigma =(G^{\top}\Omega^{-1}G)^{-1}$, chamberlain1987. Furthermore, if the additional Supplement (ref): Assumption (ref) is imposed, defining $H=\Sigma G^{\top}\Omega^{-1}$ and $P=\Omega^{-1}-\Omega^{-1}G(G^{\top}\Omega^{-1}G)^{-1}G^{\top}\Omega^{-1}$, the second order bias of $\hat{\beta}$ is $\operatorname{E}[\hat{\beta}]-\beta _{0}=n^{-1}H\zeta_{\lambda}+\mathit{O}_{}(n^{-2})$, where

with $c_{\rho}=1+\rho_{3}/2$ and $a$ a $d_{g}$-vector with elements $a^{j}=\mathop{\mathrm{tr}}(\Sigma\operatorname{E}[\partial^{2}g^{j}(z,\beta_{0})/\partial\beta\partial\beta^{\top}])$, $j=1,\ldots,d_{g}$; see newey2004.

For given $\beta$, the auxiliary parameter estimator is defined by $\lambda(\beta)=\operatorname*{argmax}_{\lambda\in\Lambda_{n}(\beta)}P_{n}(\beta,\lambda)$. Whenever the constraint in $\lambda\in\Lambda_{n}(\beta)$ is not binding, $\lambda(\beta)$ solves the first-order conditions \\ $n^{-1}\sum_{i=1}^{n}\rho^{(1)}(\lambda(\beta)^{\top}g_{i}(\beta))g_{i}(\beta)=0$. The GEL implied probabilities are then

The sample moment constraint $\sum_{i=1}^{n}\pi_{i}(\beta)g_{i}(\beta)=0$ holds whenever the first order conditions for $\lambda (\beta)$ hold. In what follows, $\hat{\pi}_{i}=\pi_{i}(\hat{\beta})$, $i=1,\ldots,n$, corresponds to the solution $\hat{\lambda}=\lambda(\hat{\beta})$, and, if $\beta_{0}$ is known, $\tilde{\pi}_{i}=\pi_{i}(\beta_{0})$, $i=1,\ldots,n$, with auxiliary parameter estimator $\tilde{\lambda}=\lambda(\beta_{0})$. The generic notation $\pi_{i}$, $i=1,\ldots,n$, is used whenever the distinction is unnecessary.

GEL-Based Density Estimation

Suppose the p.d.f. $f(\cdot)$ of the scalar random variable $u=u(z,\beta _{0})$ is of interest, where the scalar function $u:\mathcal{Z}\times\mathcal{B}\mapsto\mathcal{U}\subseteq\mathbbm{R}$ is known up to the parameter vector $\beta_{0}$.

Let $\mathcal{N}$ denote an open neighbourhood of $\beta_{0}$.

Known \texorpdfstring{$\beta_{0}$}{beta}

Suppose that $u_{i}=u(z_{i},\beta _{0})$, $i=1,\ldots,n$, are observed. Then the classical kernel density estimator for the p.d.f. $f$ of $u=u(z,\beta_{0})$ can be employed; viz.

where $k_{b}(x)=k(x/b)/b$, $k(\cdot)$ is a kernel function and $b=b_{n}>0$ is a bandwidth sequence; see rosenblatt1956 and parzen1962. The estimator $\tilde{f}$ (ref) will serve as a benchmark for later comparisons.

The properties of $\tilde{f}$ are well known and can be formally established under different combinations of smoothness and integrability conditions on the kernel $k$ and density $f$; see, e.g., rao1983. A standard set of such conditions is given in Assumption (ref) below. If $k$ is square integrable, but not absolutely integrable, as is the case for the sinc kernel, conditions such as those in tsybakov2009 can be imposed.

Let $R(k)=\int_{-\infty}^{\infty}k(x)^{2}\mathrm{d}x$ for any square integrable function $k$; the limits of integration are omitted whenever there is little scope for confusion. Also let $f^{(j)}(u)=\mathrm{d}^{j}f(u)/\mathrm{d}u^{j}$ for any $j$th order differentiable function $f$.

Let $1\leq r<\infty$. Suppose that Assumptions (ref)(a)(ii), (ref)(b) with $s=2r+2$, (ref)(c) together with $\mu_{2r+2}(k)<\infty$ and $\int x^{2}k(x)^{2}\mathrm{d}x<\infty$ hold. Then

Hence,

The mean integrated squared error (MISE), $\operatorname{MISE}[\tilde{f}]=\operatorname{E}[\int(\tilde{f}(u)-f(u))^{2}\mathrm{d}u]$, is a commonly used global measure of performance. The optimal bandwidth is then defined as that value of $b>0$ minimising MISE, or an approximation thereof. In particular, the asymptotically optimal bandwidth is defined as the value $b^{\ast}$ minimising the two leading terms in the expansion

i.e., $b^{\ast}=cn^{-1/(4r+1)}$ where $c=[(2r)!^{2}R(k)/(4r\mu_{2r}(k)^{2}R(f^{(2r)})]^{1/(4r+1)}$. The asymptotically optimal MISE is thereby

The GEL-based kernel density estimator incorporates the information embedded in the moment restriction (ref) replacing the sample EDF weights $n^{-1}$ in the construction of $\tilde{f}(u)$ (ref) by the implied probabilities $\tilde{\pi}_{i}$, $i=1,\ldots,n$; viz.

To describe the properties of GEL-based kernel density estimator $\tilde{f}_{\rho }(u)$ (ref), the shorthand notation, e.g., $\operatorname{E}[g_{i}|u]=\operatorname{E}[g(z,\beta_{0})|\{z: u(z,\beta_{0})=u\}]$, for conditional expectations given $u$ is adopted.

Thus, the estimators $\tilde{f}$ and $\tilde{f}_{\rho}$ are asymptotically first-order equivalent, and the asymptotically optimal bandwidth for $\tilde{f}_{\rho}$ is identical to that of $\tilde{f}$, i.e., $b^{\ast}$.

Whenever $c_{\rho}=0$, as is the case for (G)EL with $\rho_{3}=-2$ , e.g., EL, the $n^{-1}$ bias term in (ref) vanishes. In general, provided the bandwidth does not go to zero faster than $n^{-1/(2r)}$, and certainly when $b=b^{\ast}\sim n^{-1/(4r+1)}$, this bias term is at most third order. Its contribution to MISE is via the integrated squared bias (ISB)

with the $\mathit{O}_{}(n^{-1}b^{2r})$ term generally non-zero and either positive or negative. With the asymptotically optimal bandwidth, $n^{-1}(b^{\ast})^{2r}\sim n^{-3/2+1/(8r+2)}$, which approaches $n^{-3/2}$ arbitrarily closely as $r$ increases, whereas the leading terms in $\operatorname{MISE}[\tilde{f}(\cdot;b^{\ast})]$ becomes arbitrarily close to $n^{-1}$.

As long as $\operatorname{E}[g_{i}|u]\neq 0$, the GEL-based estimator $\tilde{f}_{\rho}$ enjoys a second-order reduction in variance due to the $n^{-1}$ term in (ref), which does not depend on the choice of GEL carrier function $\rho(\cdot)$. Hence

While this reduction is negligible asymptotically, the leading term in $\operatorname{MISE}[\tilde{f}]$ approaches zero only a little more slowly than $n^{-1}$. Hence the effect could be substantial in small samples.

Unknown \texorpdfstring{$\beta_{0}$}{beta 0}

Suppose now that $\beta_{0}$ is unknown. Then, after substitution of the estimators $\hat{u}_{i}=u(z_{i},\hat{\beta})$ for $u_{i}$, $i=1,\ldots,n$, in $\tilde{f}$ and $\tilde{f}_{\rho}$ in (ref) and (ref), the analogous estimators of $f(u)$ are

respectively. Because $u_{i}$, $i=1,\ldots,n$, are not directly observable, the behaviour of the estimation error $\hat{u}_{i}-u_{i}$, $i=1,\ldots,n$, needs to be constrained with additional restrictions imposed on $k$ and $b$. Assumption (ref) gives a set of mild sufficient conditions, see, e.g., vanryzin1969 and ahmad1992; similar conditions have also been considered in, e.g., cheng2005 and kiwitt2008.

The uniform $\alpha$-H\"{o}lder condition Assumption (ref)(b) on $u(z,\beta)$, also known as a Lipschitz condition of order $\alpha$, is an appropriate way to quantify the `degree of continuity' of $u(z,\beta)$; see zygmund2003. Many kernels used in practice are Lipschitz continuous, and hence satisfy Assumption (ref)(a) with $\tau=1$. For example, a kernel that satisfies Assumption (ref)(a) for any $0<\tau\leq\gamma$ but not for $\gamma<\tau\leq1$ is $k(x) = (1+\gamma)(1-\lvertx\rvert)^{\gamma}/2$ if $\lvertx\rvert\leq 1$ and $0$ otherwise, yielding the Bartlett (triangular) kernel if $\gamma=1$. Assumption (ref)(c) is important as it prevents the bandwidth from being too small. Intuitively, if $b$ is very small, the kernel $k_{b}(u-\hat{u}_{i})$ is very narrowly centered around the incorrect value $\hat{u}_{i}$ potentially excluding the true value $u_{i}$; see, e.g., silverman1986 for a generic illustration. Assumption (ref)(c) requires $nb^{4}\to\infty$ regardless of the values of $\tau$ and $\alpha$ and $b=n^{-1/4}$ is the fastest rate achievable when $\alpha=\tau=1$. Note that the optimal bandwidth $b^{\ast}$ is excluded if $[\alpha(4r+1)-2]\tau <2$.

Under these conditions, Theorem (ref) establishes that the differences between the kernel density estimators $\hat{f}$ (ref) and $\hat{f}_{\rho }$ (ref) and their counterparts $\tilde{f}$ (ref) and $\tilde{f}_{\rho}$ (ref) based on observable $u_{i}$, $i=1,\ldots,n$, are negligible asymptotically.

To obtain higher order expansions for the mean and variance of $\hat{f}(u)$ (ref) and $\hat{f}_{\rho}(u)$ (ref) requires a further strengthening of the assumptions. Let $\nabla u(z,\beta)$ and $\nabla^{2}u(z,\beta)$ denote respectively the $d_{\beta}$-vector and $d_{\beta}\times d_{\beta}$ matrix of the first and second derivatives of $u(z,\beta)$ with respect to $\beta$. Also let $\nabla u_{i}=\nabla u(z_{i},\beta _{0})$ and $\nabla^{2}u_{i}=\nabla^{2}u(z_{i},\beta _{0})$.

Assumption (ref)(a)(b) implies Assumption (ref)(a)(b) holds with $\alpha=\tau=1$ with the requirement in Assumption (ref)(c) rendered as $n^{1/2}b^{2}\to\infty$. Note that Assumption (ref)(a) also implies Assumption (ref)(a)(i). Assumption (ref)(d) imposes additional smoothness and integrability conditions on $f$ and $u(z,\beta)$. Assumption (ref)(c) is much stronger than Assumption (ref)(c) requiring $nb^{8}\to\infty$ regardless of the values of $\tau$ and $\alpha$ thereby prohibiting the asymptotically optimal bandwidth $b^{\ast}$ when $k$ is a second order kernel. For $r\geq2$, $b^{\ast}$ is permissible as long as $\tau>6/(4r-1)$ and $\alpha>5/(8r+2)$. Note that, if $\alpha>5/16$, $n^{\tau/2}b^{3+\tau}\to\infty$ implies $n^{\alpha/2}b^{5/4}\to\infty$.

The extra bias term $\delta (u)$ (ref) for $\hat{f}_{\rho}$ and those terms appearing in $\operatorname{Var}[\hat{f}(u)]$ (ref) primarily arise due to the substitution of $\hat{u}_{i}$ for $u_{i}$, $i=1,\ldots,n$. Supplement (ref): Examples (ref) and (ref) examine these terms in more detail for regression on a constant and (G)EL with a constant and zero mean condition respectively. Here, although $\int [\mathrm{d}\{\operatorname{E}[\nabla u_{i}|u] f(u)\}/\mathrm{d}u]^{\top}\Sigma[\mathrm{d}\{\operatorname{E}[\nabla u_{i}|u] f(u)\}/\mathrm{d}u]\mathrm{d}u$ is non-negative, the term $\int [\mathrm{d}\{\operatorname{E}[\nabla u_{i}|u] f(u)\}/\mathrm{d}u]^{\top}H\operatorname{E}[g_{i}|u]f(u) \mathrm{d}u$ can be negative, as can be the ISB term due to the additional $\delta(u)$ (ref).

Bias Correction

While the contribution from the $n^{-1}$ bias terms to MISE is of a lower order than the contribution from the variance terms, the effect of bias can be substantial in small and moderate samples, potentially offsetting any reduction in variance. The direction of the bias cannot of course be known a priori. Hence it may be advisable to bias-correct the density estimates by estimating and subtracting the $n^{-1}$ bias term.

To be more specific, the bias-corrected estimates are defined as

where $\hat{\delta}(u)$ and $\hat{\delta}_{\rho}(u)$ are suitable (asymptotically) unbiased estimators of $\delta (u)$ (ref) and $\delta_{\rho }(u)$ (ref). The implied probabilities $\hat{\pi}_{i}$, $i=1,\ldots,n$, can be used to obtain efficient estimators of the component quantities entering $\delta(u)$ and $\delta_{\rho }(u)$ with the modifications described in glad2003 applied to ensure that the bias-corrected estimate is a density.

GEL-Based Distribution Function Estimation

The results for distribution function estimation parallel those given in Section (ref) for density estimation but can be shown to hold under much weaker conditions, and so are given here separately.

Known \texorpdfstring{$\beta_{0}$}{beta}

When $u_{i}$, $i=1,\ldots,n$, are observed, the c.d.f. $F$ of $u(z,\beta_{0})$ can be estimated by

with $K(u)=\int_{-\infty}^{u}k(x)\mathrm{d}x$; see nadaraya1964b and watson1964. The kernel distribution function estimator (ref) can be obtained by integrating (ref) or motivated as a smoothed version of the EDF.

Assumption (ref)(a)(i) is sufficient for $\widetilde{F}$ to be an asymptotically unbiased and consistent estimator of $F$ at all continuity points of $F$ if $b\to0$ as $n\to\infty$. In addition, if $F$ is continuous then $\widetilde{F}$ converges to $F$ uniformly with probability $1$ (w.p.$1$.); see yamato1973. If $k$ satisfies Assumption (ref)(a)(ii) with $\mu_{2r+2}(k)<\infty$ for some $r\geq1$, $f$ satisfies Assumption (ref)(b) with $s=2r+1$, and $b\to0$ as $n\to\infty$ (Assumption (ref)(c) is not required here), then

where $\psi(k) = 2\int xK(x)k(x)\mathrm{d}x$. Hence

where $V_{F}={\begingroup\textstyle\int\endgroup} F(u)(1- F(u))\mathrm{d}u$.

Provided $\psi(k)>0$, the asymptotically optimal bandwidth minimising the leading terms in (ref) is $b^{\ast}=\varsigma n^{-1/(4r-1)}$, where $\varsigma=[(2r)!^{2}\psi(k)/(4r\mu_{2r}(k)^{2}R(f^{(2r-1)}))]^{1/(4r-1)}$, and the asymptotically optimal MISE is

The modified GEL kernel distribution function estimator corresponding to $\tilde{f}_{\rho}$ (ref) which incorporates the information embedded in the moment restrictions (ref) is

These results are qualitatively similar to Theorem (ref), the important difference being that the reduction in variance is now first-order asymptotically, whereas the contribution from the $n^{-1}$ bias term in (ref) to MISE is of order $n^{-1}b^{2r}$. Ceteris paribus, the asymptotically optimal c.d.f. bandwidth converges to zero at a faster rate than that for density estimation. Hence the additional bias effect can be expected to be of less importance.

Unknown \texorpdfstring{$\beta_{0}$}{beta}

When $\beta_{0}$ is unknown, the analogues of $\widetilde{F}$ and $\widetilde{F}_{\rho}$ are

respectively.

Similar to Theorem (ref), Theorem (ref) establishes that the differences between $\widehat{F}$ (ref) and $\widehat{F}_{\rho}$ (ref) and their counterparts based on observable $u_{i}$, $i=1,\ldots,n$, are negligible asymptotically. No additional requirements are placed on $k$ beyond the standard conditions in (ref)(a)(i) and the restriction on the bandwidth is thus weaker than Assumption (ref)(c).

Higher order expansions similar to those in Theorem (ref) may be obtained under the following conditions.

Simulation Evidence

Preliminaries

Consider the inverse hyperbolic sine (IHS) transformation model

here $\beta = (\delta,\gamma,\theta)^{\top}$ and $z=(y,x)^{\top}$. The IHS transformation has been proposed in johnson1949 as an alternative to the Box-Cox power transform, $(y^{\lambda}-1)/\lambda$, $y\geq 0$, and developed in burbidge1988 and mackinnon1990; see also, e.g., ramirez1994, brown2015 and the references therein for recent applications in statistics and econometrics, and tsai2017 for comparisons with other transformations. When $\theta =0$, the IHS transform is defined as the limiting value, $\lim_{\theta\to0}\operatorname{arsinh}(\theta y)/\theta=y$, which corresponds to the Box-Cox transform with $\lambda=1$; when $\theta\neq0$, the shapes of the IHS transforms are similar to those of the Box-Cox with $\lambda<1$. The advantage of the IHS transform is that it is a smooth function of $y\in\mathbbm{R}$ and $\theta\in\mathbbm{R}$ with values at $\theta=0$ defined as the corresponding limits.

The infeasible optimal instruments in the IHS transformation model (ref) are

see robinson1991. The last element of $S(x;\beta_{0})$, $s_{3}(x;\beta_{0})$, depends on the conditional distribution of $u$ given $x$, and, in general, there is little reason to argue for a particular scalar function of $x$ as a good approximant. For example, if $u|x\sim N(0,\sigma ^{2})$, based on $\tanh (x)\simeq 2\Phi ((\pi /2)^{1/2}x)-1$ twice, $s_{3}(x;\beta_{0})$ is approximately $\tanh\left(\theta_{0}(\delta_{0}+\gamma_{0}x)/(\pi\theta_{0}^{2}\sigma^{2}/2+1)^{1/2}\right)/\theta_{0}^{2} -(\delta_{0}+\gamma_{0}x)/\theta_{0}$ which suggests the use of odd degree polynomials in $x$ as instruments; other and better approximations are of course available.

In all cases the true parameters are $\delta_{0}=1$, $\gamma_{0}=2$ and $\theta_{0}=0.08$ which yield a signal-to-noise ratio of $\gamma_{0}^{2}/(1+\gamma_{0}^{2})=4/5=0.8$ somewhat more stringent than that of $16/17=0.941$ in robinson1991.

Design

Given the uncertainty concerning the conditional distribution $u|x$ the approach adopted here is to simply compare estimators based on moment conditions $\operatorname{E}[g(z,\beta_{0})]=0$ (ref) where

for $d_{g}=3$ (exactly identified), $4$ and $5$ (over-identified).

Three data generating processes for $(x,u)$ are considered.

\noindentScenario 1. $x$ and $u$ are distributed as independent standard normal $N(0,1)$; cf. robinson1991.

\noindentScenarios 2 and 3. $x$ and $u$ have joint density $f_{ux}(u,x)=xf_{NM}(ux)f_{x}(x)$ where $x$ is a generalised gamma random variable, stacy1962, with parameters $p=2$, $d=\nu$ and $a=(2/\nu)^{1/2}$ for some $\nu>4$ and $f_{NM}$ is the normal mixture density with $m$ components, viz. $f_{NM}(w) = \sum_{j=1}^{m}\omega_{j}\phi_{\sigma_{j}}(w-\mu_{j})$, $-\infty<\mu _{j}<\infty$, $\sigma_{j}>0$, $j=1,\ldots,m$, $\sum_{j=1}^{m}\omega_{j}=1$, and $\sum_{j=1}^{m}\omega_{j}\mu_{j}=0$, i.e., $\operatorname{E}[w]=0$. Here $\phi(x)$ denotes the standard normal p.d.f. and $\phi_{\sigma}(x)=\phi(x/\sigma)/\sigma$. The joint density $f_{ux}$ is the density of $u=w/x$ and $x$ where $w$ and $x$ are independent. The conditional density of $u$ given $x$ is $f_{u|x}(u|x) = xf_{NM}(ux) = \sum_{j=1}^{m}\omega_{j}\phi_{\sigma_{j}/x}(u-\mu_{j}/x)$. Hence, $\operatorname{E}[u|x]=0$ and $\operatorname{E}[u_{i}^{2}|x] = \sum_{j=1}^{m}\omega_{j}(\sigma_{j}^{2}+\mu_{j}^{2})/x^{2}$. The marginal density of $u$ is a mixture of noncentral $t$ densities $f_{u}(u) = \sum_{j=1}^{m}\omega_{j}t_{\nu}(u/\sigma_{j};\: \mu_{i}/\sigma_{i})/\sigma_{j}$ where $t_{\nu}(\cdot;\eta)$ is the density of a noncentral $t$-distributed random variable with $\nu$ degrees of freedom and noncentrality parameter $\eta$ allowing a wide variety of shapes for $f_{u}$ by varying the mixture $f_{NM}$. The skewed unimodal and bimodal densities shown in Figure (ref) describe the NM densities for Scenarios 2 and 3 respectively, i.e., the mixture densities marron1992 centered to have zero mean.

Kernel Functions and Bandwidths

Fourth order Gaussian-based kernels, $k(x)=(3-x^{2})\phi(x)/2$ and $K(x)=\Phi(x)+x\phi (x)/2$, $\Phi(x)=\int_{-\infty}^{x}\phi(u)\mathrm{d}u$, are employed; see wand1990 and oryshchenko2017 respectively. Thus the choices of the asymptotically optimal bandwidths $(27/4\sqrt{\pi})^{1/9}R(f^{(4)})^{-1/9}n^{-1/9}$ and $(7/2\sqrt{\pi })^{1/7}R(f^{(3)})^{-1/7}n^{-1/7}$ for p.d.f. and c.d.f. estimation respectively are permitted, thereby satisfying Assumptions (ref)(c) and (ref)(b). The practical issue of estimating the derivatives of $f$ required for the computation of $R(f^{(j)})$, $j=3,4$, is ignored and the respective true values used. For the standard normal distribution these are $R(\phi^{(3)})=15/(16\sqrt{\pi})$ and $R(\phi^{(4)})=105/(32\sqrt{\pi})$; for the mixture distributions, approximate values are shown in Figure (ref).

Results

The study compares the performance of GEL-based kernel density p.d.f. and c.d.f. estimators. The GEL parameter estimators are CUE, EL and ET, the most notable special cases of the GEL family. For each estimator the mean and variance were computed on a grid $1000$ of points between $-5$ and $5$ and are reported as the integrated squared bias and integrated variance relative to those of the corresponding infeasible estimator based on the true $u$, i.e., $\tilde{f}$ and $\widetilde{F}$.

Tables (ref), (ref) and (ref) report results for Scenarios 1, 2 and 3 respectively. The ISB, IVar and MISE (all $\times10^5$) for the infeasible $\tilde{f}$ and $\widetilde{F}$ are presented. Rows ISB, IVar, and MISE are the ISB, IVar, and MISE of $\hat{f}$, $\hat{f}_{\rho}$ ($\widehat{F}$, $\widehat{F}_{\rho}$) relative to the infeasible $\tilde{f}$ ($\widetilde{F}$), respectively; row `vs $d_{g}=3$' is the MISE of $\hat{f}$, $\hat{f}_{\rho}$ ($\hat{F}$, $\widehat{F}_{\rho}$) relative to the corresponding value for $d_{g}=3$; row `w. vs unw.' is the MISE of $\hat{f}_{\rho}$ ($\widehat{F}_{\rho}$) relative to $\hat{f}$ ($\widehat{F}$). Rows MISE, `vs $d_{g}=3$', and `w. vs unw.' examine the significance of the paired $t$-statistics in a two-sided test for equality of the respective ISE means, e.g., $\int (\hat{f}(u)-f(u))^{2}\mathrm{d}u$; the symbol \dag\ indicates that the $p$-value is between $0.01$ and $0.05$ whereas \ddag\ that it is less than $0.01$ and in all other cases the $p$-value is greater than $0.05$. Values of relative MISE less than $1$ are emphasised in bold.

Sample sizes $n=100$, $500$, $1000$, and $2000$ are examined.

All computations were carried out in MATLAB; the relevant code and additional results, including the properties of GEL estimators, are available from the first named author upon request. All results are based on $10,000$ random draws.

{5pt}

{6pt}

Scenario 1

The first $\sim b$ term in eq. (ref) is approximately $-0.321n^{-1/9}$, which for $n=100$, $500$, $1000$, and $2000$ is approximately $-0.192$, $-0.161$, $-0.149$, and $-0.138$ respectively. The second $\sim b$ term is approximately $0.04728n^{-1/9}$ for $d_{g}=4$ and $0.04708n^{-1/9}$ for $d_{g}=5$, which offsets the reduction in variance slightly. The predicted relative IVar of $\hat{f}$ and $\hat{f}_{\rho}$ up to order $\mathit{o}_{}(b)$ is thus $0.836$, $0.863$, $0.873$ and $0.882$ for $n=100$, $500$, $1000$, and $2000$ respectively and is identical within three digit precision for $d_{g}=4$ and $5$.

The results reported in Table (ref) confirm these predictions. In fact, the reduction in variance is even larger than expected in small and medium samples due to the $\mathit{o}_{}(b)$ effects. Furthermore, estimators $\hat{f}$ and $\hat{f}_{\rho}$ have smaller ISB relative to $\tilde{f}$. A comparison of $\hat{f}$ and $\hat{f}_{\rho}$ between $d_{g}=3$ (just-identified) and $d_{g}=4,5$ (over-identified) for moderate and larger sample sizes emphasises further the contribution of additional moment information. Hence $\hat{f}$ and $\hat{f}_{\rho}$ enjoy a reduction in MISE of as much as $21\%$ for $n=100$ and $10\%$ for $n=2000$ relative to $\tilde{f}$. The benefits are even more pronounced for c.d.f. estimation, where the reduction in MISE can be as much as $56\%$ for $n=100$ and around $53\%$ in moderate samples. There are also small but statistically significant benefits to re-weighting which are mostly due to the smaller biases of $\hat{f}_{\rho}$ and $\widehat{F} _{\rho}$ relative to $\hat{f}$ and $\widehat{F}$ at moderate and larger sample sizes. There is some deterioration in ISB, IVar and, thus, MISE with increases in $d_{g}$ which can be contributed to the increased importance of outliers.

Finally, while in moderate and large samples the performances of CUE, EL, and ET are virtually identical, in small samples ET can be unstable with larger $d_{g}$.

Scenarios 2 and 3

Scenarios 2 and 3 with densities of $(x,u)$ which are heavy-tailed and also, e.g., skewed and bimodal, illustrate the many difficulties for both GEL estimation and kernel p.d.f. and c.d.f. estimation which are absent in the relatively benign Scenario 1.

The performance of CUE in small samples is generally worse than that of EL and ET. It ranks last by MSE in both scenarios with $n=100$ and $500$, except Scenario 3 with $n=100$ where ET underperforms. In a number of cases increasing with $d_{g}$ the optimisation routine for ET failed. Somewhat surprisingly, although it is known to be sensitive to outliers, EL appears to deliver good results in the simulation experiments. It ranks first by MSE in Scenario 3 with $d_{g}=5$ and alternates with ET otherwise. These differences become very small with $n=1,000$ and greater.

The conclusion about the inferior performance of CUE in small samples holds true for CUE-based kernel density p.d.f. and c.d.f. estimators as well; see Tables (ref) and (ref), in particular, the ISBs of $\hat{f}$ and $\hat{f}_{\rho}$ with $d_{g}=4,5$ in Table (ref). However, the ranking of EL and ET-based kernel density p.d.f. and c.d.f. estimators by MISE does not always correspond to the ranking of the underlying EL and ET estimators of $\beta_{0}$ by MSE. In particular, the sensitivity of EL to outliers adversely affects the estimators $\hat{f}_{\rho}$ and $\widehat{F}_{\rho}$ via the implied probabilities in Scenario 3 with $n=500$ and greater; see Table (ref). ET and CUE perform better in those cases.

Unlike Scenario 1, in Scenario 3 none of the feasible kernel density estimators have smaller MISE than their infeasible counterparts for the sample sizes considered. In Scenario 2, with less complicated distributional features, these estimators do achieve a reduction in MISE with $d_{g}=4,5$. The same is true for the feasible kernel c.d.f. estimators in Scenario 2 with $d_{g}=3,4,5$, and more often than not in Scenario 3 as well, with the few exceptions mentioned above. Importantly, it is generally beneficial to increase the number of moment conditions beyond those necessary to identify the parameters except when stability of GEL estimators of $\beta_{0}$ is likely to deteriorate.

Finally, the benefits of re-weighting are present, but not universal, and as expected, are quite small; cf. Supplement (ref): Example (ref).

Summary and Conclusions

Large sample results and simulation evidence reported in this paper suggest that it is generally sensible to apply either the standard or re-weighted kernel estimators to estimate the p.d.f. or c.d.f. of a scalar residual $u(z,\beta_{0})$ in a variety of situations, provided error associated with the estimation of $\beta_{0}$ satisfies some mild regularity conditions and care is taken to ensure the bandwidth is not too small. If the assumptions on $u(z,\beta)$ prove difficult to verify in practice, using fourth or higher order kernels and the corresponding asymptotically optimal bandwidths will generally assist with ensuring the appropriate regularity conditions hold.

Incorporating information from overidentifying moment conditions by re-weighting the estimators using GEL implied probabilities offers efficiency gains which are realised in regular situations. However, if the model is highly nonlinear and the distribution of the data is heavy-tailed or contaminated with outliers, the methods proposed in this paper, including GEL, should be applied with some caution in very small samples. Robustified hybrid estimators such as the exponentially tilted empirical likelihood, see, e.g., schennach2007, may prove useful in these circumstances.

While the results in this paper were presented only for the scalar-valued $u(z,\beta)$, generalisations to the vector case are relatively straightforward provided an analogue of the bijection Assumption (ref) holds.

An issue for future research to usefully address is the construction of tests for overidentifying moment conditions or parametric restrictions based on the differences between the kernel p.d.f. estimators $\hat{f}_{\rho}$ and $\hat{f}$ or $\tilde{f}_{\rho}$ and $\tilde{f}$ for known $\beta_{0}$. Test statistics of the Bickel-Rosenblatt type based on the integrated squared difference $\int(\hat{f}_{\rho}(u)-\hat{f}(u))^{2}\mathrm{d}u$, bickel1973, fan1994,fan1998, or the integrated absolute difference, cao2005, would be of interest. Alternatively, Kolmogorov-Smirnov or Cram\'{e}r-von Mises-type tests could be constructed based on the differences between kernel c.d.f. estimators.

\phantomsection \addcontentsline{toc}{section}{References} {0pt plus 0.3ex}

\setcounter{page}{1} \setcounter{section}{0}

}

\phantomsection \addcontentsline{toc}{section}{Supplement A: Proofs} \textcolor{white}{

\protected@write \@auxout {\string \newlabel {Supp:Proofs}{{A}{[B.\arabic{page}]}{A}{Supp:Proofs}} } \hypertarget{Supp:Proofs}{A}

}

Throughout the Appendix, $0<C<\infty$ and $0\leq \omega\leq1$ will denote generic constants that may be different in different uses. CS, T, and H refer to the Cauchy-Schwarz, triangle, and H\"{o}lder inequalities, respectively with LIE and WLLN the law of iterated expectations and Khintchine's i.i.d. weak law of large numbers. MVT is the mean value theorem.

In addition, $\mathop{\mathrm{int}}(\cdot)$ denotes the interior of $\cdot$, w.p.(a.)1 with probability (approaching) 1, and $\mathcal{N}$ is an open neighbourhood of $\beta_{0}$.

GEL Stochastic Expansions

The following identification and regularity conditions are imposed.

Assumption (ref) is newey2004 and is sufficient for the consistency of $\hat{\beta}$. Moreover, $\hat{\lambda}=\operatorname*{argmax}_{\lambda\in\Lambda_{n}(\hat{\beta})}P_{n}(\hat{\beta},\lambda)$ exists w.p.a.1 and $\hat{\lambda}=\mathit{O}_{p}(n^{-1/2})$; see newey2004.

Assumption (ref) is newey2004. If Assumptions (ref) and (ref) hold then $n^{1/2}((\hat{\beta}-\beta_{0})^{\top},\:\hat{\lambda}^{\top})^{\top}\xrightarrow{d}N\left(0,\mathop{\mathrm{diag}}(\Sigma,P)\right)$; see newey2004.

Let $\nabla^{2}g(z,\beta)$ denote a vector of all distinct second order partial derivatives with respect to $\beta$.

Cf. newey2004.

Write $\tilde{g}=n^{-1}\sum_{i=1}^{n}g_{i}$, $\widetilde{G}=n^{-1}\sum_{i=1}^{n}G_{i}-G$, and $\widetilde{\Omega}=n^{-1}\sum_{i=1}^{n}g_{i}g_{i}^{\top}-\Omega$. Also let $g_{i}^{j}=\partial g(z_{i},\beta_{0})/\partial\beta_{j}$ and $G_{i}^{j}=\partial^{2}g(z_{i},\beta_{0})/\partial\beta_{j}\partial\beta^{\top}$, $j=1,\ldots,d_{\beta}$. From the proof of Theorem 3.4 in newey2004, GEL estimators satisfy the following stochastic expansion

where

Preliminary Lemmas

\noindentProof. Let $\hat{v}_{i}=\hat{\lambda}^{\top}g(z_{i},\hat{\beta})$. A third order Taylor expansion of $\rho^{(1)}(\hat{v}_{i})$ around $0$ yields

noting $\lvert\hat{v}_{i}\rvert\xrightarrow{p}0$ uniformly $i=1,\ldots,n$ by newey2004. A Taylor expansion from eq. (ref) of $g(z_{i},\hat{\beta})$ about $\beta_{0}$ yields $g(z_{i},\hat{\beta})=g_{i}-G_{i}H\tilde{g}+\mathit{o}_{p}(n^{-1/2})$ uniformly $i=1,\ldots,n$ by owen1990. Hence, substituting, using eq. (ref),

From a similar expansion, using $n^{-1}\sum_{i=1}^{n}g(z_{i},\hat{\beta})=\Omega P\tilde{g}+\mathit{O}_{p}(n^{-1})$, eq. (ref), and $P\Omega P=P$,

Hence, $[{\begingroup\textstyle\sum\endgroup}_{i=1}^{n}\rho^{(1)}(\hat{v}_{i})]^{-1} = -n^{-1}[1+c_{\rho}\tilde{g}^{\top}P\tilde{g}+\mathit{O}_{p}(n^{-3/2})]$ and

uniformly $i=1,\ldots,n$. $\blacksquare$

Let $a(z)$ denote a real scalar function of $z$ such that $\operatorname{E}[a(z)^{2}]<\infty$. Write $a_{i}=a(z_{i})$, $i=1,\ldots,n$.

\noindentProof. The first result follows from the expansion for $\hat{\pi}_{i}$ in Lemma (ref). In particular, noting $\operatorname{E}[g_{i}]=0$ and $\operatorname{E}[a_{i}\mathit{o}_{p}(n^{-1})]=\mathit{o}_{}(n^{-1})$ by uniformity of $\mathit{o}_{p}(n^{-1})$, then, by independence,

uniformly $i=1,\ldots,n$, using $\operatorname{E}[\tilde{\zeta}]=(0^{\top},n^{-1}\zeta_{\lambda}^{\top})^{\top}$, $P\Omega P=P$, $H\Omega P=0$, and $\mathop{\mathrm{tr}}(\Omega P)=d_{g}-d_{\beta}$. Eqs. (ref) and (ref) follow by a similar argument.

Finally note that $\hat{a}-\bar{a}=n^{-1}\sum_{i=1}^{n}(n\hat{\pi}_{i}-1)a_{i}$. Hence, $\operatorname{Var}[\hat{a}] = \operatorname{Var}[\bar{a}]+\operatorname{Var}[\hat{a}-\bar{a}]+2\operatorname{Cov}[\hat{a}-\bar{a},\bar{a}]$. Now, from above, $\operatorname{E}[\hat{a}-\bar{a}]=\mathit{O}_{}(n^{-1})$. Hence,

Also,

Repeated use is made of the following lemma; see bochner1955 and parzen1962. See also pagan1999.

Proofs of Theorems

\noindentProof of Theorem (ref). Write $\tilde{f}_{\rho}(u) = \tilde{f}(u) + n^{-1}\sum_{i=1}^{n}(n\tilde{\pi}_{i}-1)k_{b}(u-u_{i})$. By Corollary (ref) and owen1990, $\max_{1\leq i\leq n}\lvertn\tilde{\pi}_{i}-1\rvert=\mathit{o}_{p}(1)$. By Lemma (ref), $\operatorname{E}[\lvertk_{b}(u-u_{i})\rvert]<\infty$ whenever $\lvertf(u)\rvert<\infty$ which holds a.e. Thus, $k_{b}(u-u_{i})$, $i=1,\ldots,n$, satisfies the conditions for WLLN. Hence, the first conclusion follows.

From Assumption (ref)(a)(i), $b\operatorname{E}[k_{b}(u-u_{i})^{2}]<\infty$ a.e. By CS, invoking Assumptions (ref)(e) and (ref)(a), $\operatorname{E}[\lvertg_{i}b^{1/2}k_{b}(u-u_{i})\rvert]<\infty$ and $\operatorname{E}[\lvertg_{i}^{\top}\Omega^{-1}g_{i}b^{1/2}k_{b}(u-u_{i})\rvert]<\infty$. Hence, by Corollary (ref), setting $a_{i}=b^{1/2}k_{b}(u-u_{i})$,

Under Assumption (ref)(a)(i), $\operatorname{E}[k_{b}(u-u_{i})]=f(u)+\mathit{o}_{}(1)$. Invoking Assumption (ref) and the change of variables $z\mapsto(u,v^{\top})^{\top}$, then, by LIE and Lemma (ref), $\operatorname{E}[g_{i}k_{b}(u-u_{i})] = \int \operatorname{E}[g_{i}|t]f(t)k_{b}(u-t)\mathrm{d}t = \operatorname{E}[g_{i}|u]f(u) + \mathit{o}_{}(1)$. Similarly, $\operatorname{E}[g_{i}^{\top}\Omega^{-1}g_{i}k_{b}(u-u_{i})] = \operatorname{E}[g_{i}^{\top}\Omega^{-1}g_{i}|u]f(u)+\mathit{o}_{}(1)$. The final result is a direct consequence of Lemma (ref) and the same argument. $\blacksquare$

Set

Note $\hat{f}(u) = \tilde{f}(u) + \hat{\delta}_{1}(u)$ and $\hat{f}_{\rho}(u) = \hat{f}(u) +\hat{\delta}_{2}(u)+\hat{\delta}_{3}(u)$.

\noindentProof of Theorem (ref). Under Assumptions (ref) and (ref), $\hat{\beta}\in\mathcal{N}$ w.p.a.1 and $n^{1/2}(\hat{\beta}-\beta_{0})=\mathit{O}_{p}(1)$. First, by Assumption (ref)(a,b), from eq. (ref),

since $n^{-1}\sum_{i=1}^{n}d(z_{i})^{\tau}=\mathit{O}_{p}(1)$ by WLLN and $n^{\alpha\tau/2}b^{1+\tau}\to\infty$ from Assumption (ref)(c). Next, $\max_{1\leq i\leq n}\lvertn\hat{\pi}_{i}-1\rvert=\mathit{o}_{p}(1)$ by Lemma (ref) and owen1990, from eq. (ref),

Hence, the first conclusion follows. The final result follows from eq. (ref), by noting that also

by WLLN since $\operatorname{E}[\lvertk_{b}(u-u_{i})\rvert]<\infty$ a.e. by Lemma (ref). $\blacksquare$

\noindentProof of Theorem (ref). Preliminaries. From a second order Taylor expansion around $\beta_{0}$,

where $k_{b}^{(j)}(x)=k^{(j)}(x/b)/b^{j+1}$, $j=1,2$, and $\bar{u}_{i}=u(z_{i},\bar{\beta})$, $i=1,\ldots,n$, with $\bar{\beta}$ on the line segment joining $\hat{\beta}$ and $\beta_{0}$; and $\nabla\bar{u}_{i}$ and $\nabla^{2}\bar{u}_{i}$, $i=1,\ldots,n$, are defined analogously. Note that $\lVert\bar{\beta}-\beta_{0}\rVert\leq \lVert\hat{\beta}-\beta_{0}\rVert=\mathit{O}_{p}(n^{-1/2})$. Assumption (ref)(b) and twice differentiability of $u(z,\beta)$ for $\beta\in\mathcal{N}$ implies there exist $d_{0}(z)\geq0$ with $\operatorname{E}[d_{0}(z)^{4}]<\infty$ and $d_{1}(z)\geq0$ with $\operatorname{E}[d_{1}(z)^{4}]<\infty$ such that $\lvertu(z,\beta)-u(z,\beta_{0})\rvert\leq d_{0}(z)\lVert\beta-\beta_{0}\rVert$ and $\lVert\nabla u(z,\beta)-\nabla u(z,\beta_{0})\rVert\leq d_{1}(z)\lVert\beta-\beta_{0}\rVert$ for all $z$ and $\beta\in\mathcal{N}$. Thus, by T, $\lVert\nabla \bar{u}_{i}\nabla^{\top} \bar{u}_{i}-\nabla u_{i}\nabla^{\top}u_{i}\rVert \leq 2d_{1}(z_{i})\lVert\nabla u_{i}\rVert\lVert\hat{\beta}-\beta_{0}\rVert +d_{1}(z_{i})^{2}\lVert\hat{\beta}-\beta_{0}\rVert^{2}$. By owen1990, $\max_{1\leq i\leq n}d_{1}(z_{i})^{2}=\mathit{o}_{p}(n^{1/2})$ and $\max_{1\leq i\leq n}d_{1}(z_{i})\lVert\nabla u_{i}\rVert=\mathit{o}_{p}(n^{1/2})$. Hence, $\lVert\nabla \bar{u}_{i}\nabla^{\top} \bar{u}_{i}-\nabla u_{i}\nabla^{\top}u_{i}\rVert \leq n^{-1/2}[d_{1}(z_{i})\lVert\nabla u_{i}\rVert+\mathit{o}_{p}(1)]\mathit{O}_{p}(1)$. By CS, from Assumption (ref)(b), for $0<\tau\leq1$, $\operatorname{E}[d_{0}(z_{i})^{\tau}\lVert\nabla u_{i}\rVert^{2}]<\infty$ and $\operatorname{E}[d_{1}(z_{i})^{2}\lVert\nabla u_{i}\rVert^{2}]<\infty$. Thus, $\operatorname{E}[\lvertb^{-1/2}k^{(2)}((u-u_{i})/b)\rvertd_{1}(z_{i})\lVert\nabla u_{i}\rVert]<\infty$ since $\operatorname{E}[b^{-1}k^{(2)}((u-u_{i})/b)^{2}]<\infty$ also by CS and using Lemma (ref). Hence, by T, and noting $n^{\tau/2}b^{3+\tau}\to\infty$, $0<\tau\leq1$, from Assumption (ref)(c),

Assumption (ref)(a) implies $k^{(1)}$ is Lipschitz, and hence, invoking Assumption (ref)(b), for all mean values $\bar{\beta}$ between $\hat{\beta}$ and $\beta_{0}$, $\lvertk_{b}^{(1)}(u-\bar{u}_{i}) -k_{b}^{(1)}(u-u_{i})\rvert\leq b^{-3}Cd_{0}(z_{i})\lVert\hat{\beta}-\beta_{0}\rVert$ w.p.a.1. By Assumption (ref)(a) and Lemma (ref), $\operatorname{E}[b^{-1}\vert k^{(1)}((u-u_{i})/b)\vert^{4/3}]<\infty$ a.e., and as $\operatorname{E}[d(z_{i})^{4}]<\infty$, $\operatorname{E}[\lvertb^{-3/4}k^{(1)}((u-u_{i})/b)\rvertd(z_{i})]<\infty$ using the H\"{o}lder inequality with exponents $4/3$ and $4$. Therefore, by the same argument as above,

Using expansion eq. (ref) and Lemma (ref) eq. (ref), from eq. (ref),

from eq. (ref),

and, from eq. (ref),

\noindentExpectation. Since $H\Omega H^{\top}=\Sigma$, from eq. (ref),

Assumption (ref)(a) states $\lim_{\lvertx\rvert\to\infty}\lvertx^{2}k^{(1)}(x)\rvert=0$ and implies that $\int k^{(1)}(x)\mathrm{d}x=0$, $\int xk^{(1)}(x)\mathrm{d}x=-1$, and $xk^{(1)}(x)$ satisfies the hypotheses of Lemma (ref), i.e., it is bounded and absolutely integrable. Thus, invoking Assumption (ref)(d), by MVT and Lemma (ref),

Similarly, $\operatorname{E}[k_{b}^{(1)}(u-u_{i})\nabla^{\top}u_{i}Hg_{i}] = \mathrm{d}\{\operatorname{E}[\nabla^{\top}u_{i}Hg_{i}|u]f(u)\}/\mathrm{d}u+ \mathit{o}_{}(1)$ and $\operatorname{E}[k_{b}^{(1)}(u-u_{i})\nabla^{2}u_{i}] = \mathrm{d}\{\operatorname{E}[\nabla^{2}u_{i}|u]f(u)\}/\mathrm{d}u+ \mathit{o}_{}(1)$. Furthermore, Assumption (ref)(a) also implies that $\int k^{(2)}(x)\mathrm{d}x=0$, $\int xk^{(2)}(x)\mathrm{d}x=0$, $\int x^{2}k^{(2)}(x)\mathrm{d}x=2$, and $x^{2}k^{(2)}(x)$ satisfies the hypotheses of Lemma (ref). Thus, by a second order Taylor expansion and a similar argument to eq. (ref),

Since $H\Omega P=0$, from eq. (ref), $\operatorname{E}[\hat{\delta}_{2}(u)]=\mathit{o}_{}(n^{-1})$. By Lemma (ref) eq. (ref) and the same argument used in the proof of Theorem (ref), $\operatorname{E}[\hat{\delta}_{3}(u)] = n^{-1}\{ -c_{\rho}\operatorname{E}[g_{i}^{\top}Pg_{i}|u] +\operatorname{E}[g_{i}|u]^{\top}P\zeta_{\lambda} + c_{\rho}(d_{g}-d_{\beta})\}f(u) +\mathit{o}_{}(n^{-1})$.

\noindentVariance. Since $\operatorname{E}[\hat{\delta}_{1}(u)]=\mathit{O}_{}(n^{-1})$, from eq. (ref),

Similarly, noting $\operatorname{E}[\hat{\delta}_{2}(u)]=\mathit{o}_{}(n^{-1})$, from Lemma (ref), it is straightforward to verify that $\operatorname{Var}[\hat{\delta}_{2}(u)]=\mathit{o}_{}(n^{-1})$. Furthermore, also using Lemma (ref), as $\operatorname{E}[\hat{\delta}_{3}(u)]=\mathit{O}_{}(n^{-1})$ and $\operatorname{E}[k_{b}(u-u_{i})g_{i}]=\operatorname{E}[g_{i}|u]f(u)$, $\operatorname{Var}(\hat{\delta}_{3}(u)) = n^{-1}\operatorname{E}[g_{i}|u]^{\top}P\operatorname{E}[g_{i}|u]f(u)^{2}+\mathit{o}_{}(n^{-1})$. It is straightforward to verify that $\operatorname{Cov}[\hat{\delta}_{1},\hat{\delta}_{2}]=\mathit{o}_{}(n^{-1})$, recalling $H\Omega P=0$,

$\operatorname{Cov}[\hat{\delta}_{2}(u),\hat{\delta}_{3}(u)]=\mathit{o}_{}(n^{-1})$, $\operatorname{Cov}[\hat{\delta}_{2}(u),\tilde{f}(u)]=\mathit{o}_{}(n^{-1})$, noting again $H\Omega P=0$, and finally,

Combining these results gives eqs. (ref)--(ref). $\blacksquare$

\noindentProof of Theorem (ref). Since $\lim_{x\to-\infty}K(x)=0$ and $\lim_{x\to\infty}K(x)=1$, $2\int K(x)k(x)\mathrm{d}x=1$, and $\int\lvertK(x)k(x)\rvert\mathrm{d}x<\infty$, $\operatorname{E}[K((u-u_{i})/b)] = F(u)+\int k(t)[F(u-bt)-F(u)]\mathrm{d}t$ and $\operatorname{E}[K((u-u_{i})/b)^{2}] = F(u)+2\int K(t)k(t)[F(u-bt)-F(u)]\mathrm{d}t$. $F$ as a c.d.f. is bounded and hence $\operatorname{E}[K((u-u_{i})/b)^{2}]<\infty$, and $\operatorname{E}[K((u-u_{i})/b)] = F(u)+\mathit{o}_{}(1)$ and $\operatorname{E}[K((u-u_{i})/b)^{2}] = F(u)+\mathit{o}_{}(1)$ as $b\to0$ and at all points of continuity of $F$. Therefore, cf. the proof of Theorem (ref), $\lvert\widetilde{F}_{\rho}(u)-\widetilde{F}(u)\rvert=\mathit{o}_{p}(1)$.

Equation (ref) follows by Corollary (ref) with $a_{i}=K((u-u_{i})/b)$, $i=1,\ldots,n$. Assumptions (ref)(a)(i) and $\lim_{\lvertx\rvert\to\infty}\lvertx^{2}k(x)\rvert=0$ imply that $xk(x)$ satisfies conditions of Lemma (ref). Since $\int xk(x)=0$ and $\operatorname{E}[\lvert\operatorname{E}[g_{i}|u]\rvert]<\infty$, integration by parts and an application of MVT give

Similarly, $\operatorname{E}[g_{i}^{\top}\Omega^{-1}g_{i}K((u-u_{i})/b)] =\int_{-\infty}^{u}\operatorname{E}[g_{i}^{\top}\Omega^{-1}g_{i}|t]\mathrm{d}F(t)$. Eq. (ref) follows by Corollary (ref) and eq. (ref). $\blacksquare$

Set

Note $\widetilde{F}(u) = \widehat{F}(u)+ \widehat{\Delta}_{1}(u)$ and $\widehat{F}_{\rho}(u) = \widehat{F}(u) +\widehat{\Delta}_{2}(u)+\widehat{\Delta}_{3}(u)$.

\noindentProof of Theorem (ref). Since $k$ is bounded, $K$ is Lipschitz continuous and, by the proof of Theorem (ref), $\operatorname{E}[\lvertK((u-u_i)/b)\rvert]<\infty$ for all $u$. Then, as in the proof of Theorem (ref), invoking Assumptions (ref)--(ref) and (ref)(b), from eqs. (ref)--(ref),

\noindentProof of Theorem (ref). Preliminaries. From a second order Taylor expansion around $\beta_{0}$,

where $\bar{u}_{i}=u(z_{i},\bar{\beta})$, $i=1,\ldots,n$, with $\bar{\beta}$ on the line segment joining $\hat{\beta}$ and $\beta_{0}$; $\nabla\bar{u}_{i}$ and $\nabla^{2}\bar{u}_{i}$, $i=1,\ldots,n$, are defined analogously. By the same argument as in the proof of Theorem (ref), noting that Assumption (ref)(a) implies $k$ is Lipschitz and $nb^{6}\to\infty$ as $n^{\tau/2}b^{2+\tau}\to\infty$, invoking Assumption (ref)(b),

and

Therefore, using expansion eq. (ref) and Lemma (ref),

\noindentExpectation. Similarly to the proof of Theorem (ref), from eq. (ref),

Assumption (ref)(a) implies $k(x)$ satisfies the hypotheses of of Lemma (ref). Hence $\operatorname{E}[k_{b}(u-u_{i})\nabla u_{i}] = \operatorname{E}[\nabla u_{i}|u]f(u)+\mathit{o}_{}(1)$, $\operatorname{E}[k_{b}(u-u_{i})\nabla^{\top} u_{i}Hg_{i}] = \operatorname{E}[\nabla^{\top}u_{i}Hg_{i}|u]f(u)+\mathit{o}_{}(1)$, and $\operatorname{E}[k_{b}(u-u_{i})\nabla^{2}u_{i}] = \operatorname{E}[\nabla^{2}u_{i}|u]f(u)+\mathit{o}_{}(1)$. Assumption (ref)(a) also implies $xk^{(1)}(x)$ satisfies the hypotheses of Lemma (ref). Hence, by MVT as in eq. (ref), $\operatorname{E}[k_{b}^{(1)}(u-u_{i})\nabla u_{i}\nabla^{\top}u_{i}] = \mathrm{d}\{\operatorname{E}[\nabla u_{i}\nabla^{\top}u_{i}|u]f(u)\}/\mathrm{d}u+\mathit{o}_{}(1)$. Therefore, $\operatorname{E}[\widehat{\Delta}_{1}(u)]=n^{-1}\Delta(u)+\mathit{o}_{}(n^{-1})$ as required.

Likewise, as in the proof of Theorem (ref), from eq. (ref), $\operatorname{E}[\widehat{\Delta}_{2}(u)]=\mathit{o}_{}(n^{-1})$. Finally, by Lemma (ref) and proof of Theorem (ref), $\operatorname{E}[\widehat{\Delta}_{3}(u)] = n^{-1}\Delta_{\rho}(u) +\mathit{o}_{}(n^{-1})$.

\noindentVariance. Using expansions eqs. (ref)--(ref) for $\widehat{\Delta}_{j}(u)$, $j=1,2,3$, $\operatorname{Cov}[\widehat{\Delta}_{1}(u),\widehat{\Delta}_{2}(u)]$, \\ $\operatorname{Cov}[\widehat{\Delta}_{1}(u),\widehat{\Delta}_{3}(u)]$, $\operatorname{Cov}[\widehat{\Delta}_{2}(u),\widehat{\Delta}_{3}(u)]$, $\operatorname{Cov}[\widetilde{F}(u),\widehat{\Delta}_{2}(u)]$, and $\operatorname{Var}[\widehat{\Delta}_{2}(u)]$ are all $\mathit{O}_{}(n^{-2})$. Also,

Eqs. (ref) and (ref) then follow immediately using eq. (ref) and $\operatorname{E}[k_{b}(u-u_{i})\nabla u_{i}] = \operatorname{E}[\nabla u_{i}|u]f(u)+\mathit{o}_{}(1)$. If $\mathrm{d}\{\operatorname{E}[\nabla u_{i}|u]f(u)\}/\mathrm{d}u$ is absolutely integrable, then, using Lemma (ref), $\operatorname{E}[k_{b}(u-u_{i})\nabla u_{i}] = \operatorname{E}[\nabla u_{i}|u]f(u) - b\int(\mathrm{d}\{\operatorname{E}[\nabla u_{i}|u-\omega bt]f(u-\omega bt)\}/\mathrm{d}u)tk(t)\mathrm{d}t =\operatorname{E}[\nabla u_{i}|u]f(u)+\mathit{o}_{}(b)$. $\blacksquare$

\setcounter{page}{1} \setcounter{section}{0}

}

\phantomsection \addcontentsline{toc}{section}{Supplement B: Examples} \textcolor{white}{

\protected@write \@auxout {\string \newlabel {Supp:Examples}{{B}{[B.\arabic{page}]}{B}{Supp:Examples}} } \hypertarget{Supp:Examples}{B}

}

\phantomsection \addcontentsline{toc}{section}{Example (ref): \texorpdfstring{$u$}{u} is not a function of \texorpdfstring{$\beta$}{beta}}

\phantomsection \addcontentsline{toc}{section}{Example (ref): Regression on a constant}

\phantomsection \addcontentsline{toc}{section}{Example (ref): GEL with a constant and zero mean restriction}

\phantomsection \addcontentsline{toc}{section}{Example (ref): Linear regression model with \texorpdfstring{$\operatorname{E}[u|x]=0$}{E[u|x]=0} but dependent \texorpdfstring{$u$}{u} and \texorpdfstring{$x$}{x}}