Honest Confidence Sets in Nonparametric IV Regression and Other Ill-Posed Models

Honest Confidence Sets in Nonparametric IV Regression and Other Ill-Posed Models

Introduction

This paper develops honest and uniform confidence sets for the structural function $\varphi$ in a generic class of ill-posed models treated with the Tikhonov regularization. The leading example is a nonparametric instrumental variable regression (NPIV) studied in florens2003inverse, newey2003instrumental, darolles2011nonparametric, hall2005nonparametric, and blundell2007semi. The NPIV model is

where $Z\in\ensuremath{\mathbf{R}}^p$ is a vector of explanatory variables, $W\in\ensuremath{\mathbf{R}}^q$ is a vector of instruments, and $\varphi$ is the function of interest. The NPIV model is ill-posed in the sense that the map from the distribution of the data to the function $\varphi$ is not continuous. As a result, we need to introduce some amount of regularization that would smooth out discontinuities and yield a consistent estimator.

In empirical studies, the function $\varphi$ represents a structural economic relation, such as an Engel curve, cost function, or demand curve. It is not enough to estimate this function to infer possible economic effects. We can only infer the range of possible economic effects from confidence sets. This paper is the first to provide inferential methods for Tikhonov-regularized estimators. I focus on uniform inference, which amounts to constructing a set containing the entire function $\varphi$ with a high probability. Uniform inference allows one to assess global features of the estimated function and to quantify the range of possible economic effects compatible with the data. Global features may include the evidence for non-linearities, the amount of endogeneity bias compared to the local polynomial estimator, monotonicity, concavity/convexity, or other shape properties. In contrast, pointwise confidence intervals only contain the value $\varphi(z_0)$ at some particular point $z_0$ with high probability and do not provide a valid inference for the entire function $\varphi$. Another feature of confidence sets constructed in this paper is honesty in the sense of li1989honest. Honesty means that coverage properties have uniform validity over a large class of specified models.\footnote{See Eq. (ref) in Section (ref) for a formal definition.} Honesty is desirable since the underlying model is never known and coverage properties of dishonest sets may vary from one model to another.

Building uniform confidence sets for a function requires approximating the distribution of the supremum of a certain stochastic process. We show that for a broad class of ill-posed models treated with Tikhonov regularization, the stochastic process driving the distribution of the estimator does not converge weakly in the space of functions. As a result, it is not possible to build the uniform confidence set by relying on the uniform central limit theorem which calls for alternative approaches to inference.

To construct confidence sets with good coverage properties, we rely on two different approaches. The first inferential method developed in this paper relies on estimates of tail probabilities with a suitable concentration inequality. A simple illustration of this approach is to build a uniform confidence band for the empirical distribution function using the Dvoretzky-Kiefer-Wolfowitz inequality. Given the empirical distribution function $F_n(x)=\frac{1}{n}\sum_{i=1}^n\ensuremath{\mathds{1}}_{\{X_i\leq x\}}$ based on the i.i.d. sample $(X_i)_{i=1}^n$, the Dvoretzky-Kiefer-Wolfowitz tells us that for any finite sample size $n$ the probability of $F_n$ deviating from the true distribution function $F$ in the supremum norm declines at the exponential rate

Setting $x=\sqrt{\frac{\log (2/\gamma)}{2n}}$ and $\gamma\in(0,1)$, the inequality becomes

This allows us to build a uniform confidence band for $F(x)$ with a guaranteed coverage probability $1-\gamma$ taking $F_n(x)\pm\sqrt{\frac{\log (2/\gamma)}{2n}}$. In this simple example, there is no coverage error (the coverage is at least $1-\gamma$ for any finite sample size $n$), and the diameter of the set shrinks at the rate $1/\sqrt{n}$. This inferential approach is different from the alternative asymptotic approach based on the Donsker central limit theorem, which also leads to confidence sets with diameters shrinking at the rate $1/\sqrt{n}$, but having the coverage level $1-\gamma-o(1)$. The coverage error disappears only as sample size goes to infinity. The empirical distribution function is an unbiased estimator, while the majority of estimators of functions are biased. As a result, confidence sets for such estimators, including the ill-posed models considered in this paper, usually have coverage errors.

In this paper, we rely on a concentration inequality valid for more complex statistics than $\|F_n - F\|_\infty$, see boucheron2013concentration. In particular, we exploit a data-driven concentration inequality for the supremum of the variance of the estimator to approximate quantiles of the unknown distribution. This approach does not rely on the existence of the supremum of the Gaussian process approximating the supremum of the variance of the estimator. As a result, it is valid for a broad class of data-generating processes and is especially useful in settings where all other approaches to inference fail.

For confidence sets based on the concentration inequality, we characterize non-asymptotic rates of coverage errors and expected diameters explicitly. It is especially important to know both rates, since consistency alone may not be very informative for inference. For instance, the coverage error may decrease at a rate slower than the shrinking of the confidence set, requiring larger sample sizes to achieve good coverage. We show that the bias of the estimator, not the noise coming from the estimation of the operator, drives the coverage errors of our confidence sets. To the best of our knowledge, convergence rates for coverage errors have not been derived for the NPIV model or other ill-posed models considered in this paper.

As an alternative, we also study a more traditional bootstrap inference that relies on a non-trivial application of Gaussian and bootstrap coupling inequalities developed in a seminal series of papers, chernozhukov2014gaussian, chernozhukov2016empirical. To the best of our knowledge, our application of these coupling results to the nonparametric IV model and Tikhonov estimators is new.

Although the Tikhonov-regularized NPIV is the leading example, the inferential methods developed in this paper are valid for other ill-posed models, including functional regression models and the density deconvolution model. To the best of our knowledge, despite the extensive existing literature on $L_2$ results for the functional regression models, no uniform convergence rates or uniform inferential methods are currently available. Moreover, Tikhonov regularization plays a prominent role in the statistical learning theory. The inferential results obtained in this paper can also be applied in that setting.

\paragraph*{Contribution and related literature.} This paper is the first to develop honest uniform inferential methods in a general and unifying framework, encompassing different ill-posed models treated with Tikhonov regularization. This paper is also the first to provide uniform inference for several ill-posed inverse problems based on the concentration inequality and to provide the non-asymptotic analysis, including convergence rates of coverage errors. Uniform confidence sets are available only for sieve-type estimators of the NPIV model and the density deconvolution model. The literature is also silent about coverage errors of confidence sets. Moreover, it is not known what sorts of restrictions are needed to construct honest confidence sets for ill-posed inverse problems. In this paper, we show that the class of models should involve both the class of functions and the class of operators, which is different from classical direct nonparametric problems where we need only to restrict the function. horowitz2012uniform develop uniform confidence bands for sieve NPIV estimator by first constructing pointwise confidence intervals at a finite grid of points and then letting the number of grid points grow at a certain speed to achieve uniform coverage. chen2018optimal develop inferential methods for the sieve NPIV estimator without relying on discrete approximations. They focus on uniform inference for a collection of linear and nonlinear functionals using Yurinskii's coupling and obtain uniform confidence bands in the particular case of point evaluation functionals, see also belloni2015some for the conditional mean function and tao2014inference for general conditional moment restriction models. lounici2011global obtain uniform confidence sets for the wavelet deconvolution estimator by relying on Bousquet's version of Talagrand's concentration inequality.

kato2016uniform consider a more general case, where the density of the noise is estimated from an auxiliary sample. This paper studies the estimator based on Fourier inversion and builds on the coupling inequalities developed in chernozhukov2014gaussian and chernozhukov2016empirical.

We show that the confidence sets developed in this paper may enjoy polynomial convergence rates of coverage errors in mildly ill-posed, and some severely ill-posed, cases. An appealing feature of Tikhonov's and more general spectral regularizations is that the estimator is always well-defined in finite samples and converges to the best approximation to the structural parameter under various identification failures, see florens2011identification and babiiflorens2016b.

The theoretical validity of a uniform confidence set relies on uniform convergence rates of the corresponding estimator. gagliardini2012tikhonov obtain uniform convergence rates for the Tikhonov-regularized minimum distance estimator relying on the Sobolev embedding. chen2018optimal show that the sieve nonparametric IV estimator can attain optimal asymptotic uniform convergence rates under a somewhat different set of assumptions. babiiflorens2016b derive non-asymptotic $L_2$ and uniform risk bounds for more general spectrally regularized estimators not tied to a particular estimator of the conditional mean function and show that it is possible to attain polynomial convergence rates in some severely ill-posed cases.\footnote{The minimax-optimal uniform convergence rates in the severely ill-posed case are logarithmically slow, see chen2018optimal. However, in the more restricted smoothness class, when the estimated function can be described by the finite number of generalized Fourier coefficients, or more generally if its smoothness matches the ill-posedness of the operator, one can achieve faster polynomial uniform convergence rates.}

Some pointwise inferential results are available for spectral cut-off estimators when the ill-posed operator is known and is not estimated from the data, see carrasco2011spectral, gagliardini2012nonparametric, gautier2013nonparametric, and florenshorowitzkeielgom2016. At the same time, chen2015sieve provide pointwise inference and bootstrap confidence bands for conditional moment restriction models treated with a sieve approach and nesting the NPIV model as a special case. cardot2007clt provide results on the asymptotic normality of linear functionals, which solves the prediction problem for functional regression models. Lastly, carrasco2013asymptotic study asymptotic normality of inner products for Tikhonov estimators.

The structure of the paper is as follows. We introduce the notation in the remaining part of this section. Section (ref) describes the problem of constructing honest uniform confidence sets and introduces two inferential approaches in the special case of the nonparametric IV model. Under a general set of assumptions that are verified later on in each particular application, we establish convergence rates for coverage errors and diameters of constructed sets, uniform over a general set of models. Section (ref) extends these results to different functional regression models, and to the density deconvolution. In Section (ref) we show how to implement confidence sets in practice for the NPIV estimator and explore their finite-sample properties with Monte Carlo experiments. Section (ref) considers the empirical application to Engel curves, and Section (ref) concludes.

\paragraph{Notation.} Let $L_2[0,1]^p$ denote the space of functions on some compact set $[0,1]^p\subset \ensuremath{\mathbf{R}}^p$, square integrable with respect to the Lebesgue measure $\lambda$. For $\varphi\in L_2[0,1]^p$, let $\|.\|$ denote the usual $L_2$ norm derived from the inner product $\langle .,.\rangle$. Let $C[a,b]^p$ denote the space of continuous functions endowed with supremum norm $\|.\|_\infty$. For some positive real number $\beta$, let $C^\beta_M[a,b]^p$ denote the class of $\beta$-H\"{o}lder functions on $(a,b)^p$ with $0<M<\infty$

where $k=(k_1,\dots,k_p)\in\ensuremath{\mathbf{N}}^p$ is a multi-index, $|k|=\sum_{j=1}^pk_j$, $\varphi^{(k)}(z)=\frac{\partial^{|k|}\varphi(z)}{\partial z_1^{k_1}\dots\partial z_p^{k_p}}$, and $\lfloor \beta\rfloor$ is the largest integer strictly smaller than $\beta$. For $A\subset\ensuremath{\mathbf{R}}$, let $BV(A)$ be a set of functions of bounded variation on $A$. Let $\ensuremath{\mathcal{L}}_{2}$, $\ensuremath{\mathcal{L}}_{2,\infty}$ and $\ensuremath{\mathcal{L}}_{\infty}$ be spaces of bounded linear operators from $L_2$ to $L_2$, from $L_2$ to $C$, and from $C$ to $C$. Sets on which functions are defined should be clear from the context. Spaces $\ensuremath{\mathcal{L}}_2,\ensuremath{\mathcal{L}}_{2,\infty}$, and $\ensuremath{\mathcal{L}}_{\infty}$ are endowed with standard operator norms, denoted by $\|K\|=\sup_{\|\varphi\|\leq 1}\|K\varphi\|$, $\|K\|_{2,\infty} = \sup_{\|\varphi\|\leq1}\|K\varphi\|_\infty$, and $\|K\|_\infty=\sup_{\|\varphi\|_\infty\leq1}\|K\varphi\|_\infty$. For $K\in\ensuremath{\mathcal{L}}_{2}$, let $K^*$ denote its Hilbert adjoint operator. Let $\ensuremath{\mathcal{R}}(T)$ and $\ensuremath{\mathcal{D}}(T)$ be the range and the domain of the operator $T$. For two real numbers $a$ and $b$, I denote $a\wedge b = \min\{a,b\}$ and $a\vee b = \max\{a,b\}$.

Honest uniform confidence sets

We focus on uniform confidence sets, honest to some class of models $\ensuremath{\mathcal{P}}$. The class of $\ensuremath{\mathcal{P}}$ consists of probability distributions of the data satisfying certain restrictions. For a given level $\gamma\in(0,1)$, the honest $1-\gamma$ uniform confidence set, denoted $C_{n,1-\gamma}=\left\{C_{n,1-\gamma}(z)=\left[C_l(z), C_u(z)\right],\;z\in[0,1]^p\right\}$, satisfies the following coverage

for some sequence $\delta_n\to 0$. Honesty is desirable for theoretical and practical reasons. On the theoretical side, it is well-known that focusing on a fixed model leads to inconsistent concepts of optimality for non-parametric models; see, e.g., tsybakov2009introduction, p.16-19. Honesty also ensures that for a sufficiently large sample size $n$, independent from the distribution of the data, the coverage level of the set will be close to $1-\gamma$, no matter what model in $\ensuremath{\mathcal{P}}$ nature gives us. Honesty is desirable in practice since the underlying probability distribution is unknown. In contrast, a dishonest set requires a weaker condition

and the sample size $n$ needed to achieve the coverage close to $1-\gamma$ will depend on the distribution of the data.

The second requirement for the confidence set is that its expected diameter under the supremum norm, denoted $|C_{n,1-\gamma}|_\infty=\|C_u-C_l\|_\infty$, shrinks at some rate $\rho_n\to0$

We will also consider a variation of this requirement when the diameter is bounded in probability only.

Nonparametric estimation usually involves a bias-variance trade-off for the risk of the estimator. This trade-off translates into a trade-off between the rate $\delta_n$ at which coverage errors tend to zero and the rate $\rho_n$ at which the expected diameter tends to zero. Roughly speaking, we show that the coverage error is driven by the bias of the estimator, while the variance determines its diameter. It is vital to know both rates and requiring only the limiting coverage $1-\gamma$ in the Eq. (ref) can be misleading, as the coverage error of the set may tend to zero arbitrarily slow. In the following sections, we show that the confidence sets based on the concentration inequality allow us to characterize coverage errors explicitly.

It is also worth mentioning that among confidence sets based on different $L_p,p\in[1,\infty]$ norms, only uniform confidence sets have appealing visualization and are easy to implement numerically. For example, if $\varphi$ is the function on the real line, the confidence set becomes a band on the plane, which contains the whole graph of $\varphi$ with high probability.

Nonparametric IV

The model

The nonparametric IV model

leads to the functional equation

which is an example of an ill-posed inverse problem. Even if we knew the distribution of the data $(Y,Z,W)$, obtaining $\varphi$ requires inverting the conditional expectation operator, which is typically not continuous. florens2003inverse and darolles2011nonparametric introduce Tikhonov regularization, see tikhonov1963solution, to smooth out discontinuities of inversion.

Multiplying Eq. (ref) by the marginal density function $f_W$

Following darolles2011nonparametric we focus on the kernel estimator, but other choices are possible. For simplicity, we use the product kernel and equal bandwidth parameters $h_n\to 0$ as $n\to\infty$ for all coordinates

The Tikhonov-regularized estimator solves the penalized least-squares problem

which admits a closed-form solution

where $\hat T^*$ is the adjoint operator of $\hat T$. The adjoint operator is a solution to $\langle \hat T\varphi,\psi \rangle = \langle \varphi,\hat T^*\psi\rangle$ and is computed by Fubini's theorem

Ill-posedness and regularization bias

We first introduce the class of functions and operators for which we wish to obtain the honest coverage.

Source conditions are at the heart of spectral regularization theory, and describe the regularity of the problem by restricting how ill-posed the operator $T$ is, compared to the smoothness of the parameter of interest $\varphi$ (see also chen2011rate for its comparability with other assumptions used in the literature). The present source condition is different from the one used to characterize $L_2$ rates, where we would require $\varphi = (T^*T)^\beta\psi$ only, see carrasco2007linear.

We show in the next section that under this source condition, our confidence sets enjoy polynomial convergence rates of coverage errors. In the severely ill-posed case, when singular values of the operator $T$ decay to zero exponentially fast, the source condition in Assumption (ref) requires that the function $\varphi$ is well-approximated by a small number of generalized Fourier coefficients with respect to the SVD basis of $T$. It is a remarkable fact that we can still achieve polynomial convergence rates in this case, since many functions encountered in practice have rapidly declining generalized Fourier coefficients.

Assumption (ref) allows us to control the regularization bias. Note that in the Proposition that follows, the bound is uniform over the source set, and this will be needed to establish honesty of our confidence sets.

In terms of the bias, the regularization takes advantage of the smoothness up to $\beta=1$. This effect is somewhat reminiscent of the saturation of convergence of the kernel density estimator and can be avoided using the iterated or the extrapolated Tikhonov regularization similarly to using higher or infinite order kernels for the kernel density estimator, see darolles2011nonparametric and carrasco2007linear for more discussion.

Honest confidence sets

Put

where $\hat X_{ni}(w) = \frac{1}{\sqrt{h_n^q}}\hat U_iK_w\left(h_n^{-1}(W_i - w)\right)$, $\hat U_i = Y_i - \hat\varphi(Z_i)$, and $\eta_i$ are i.i.d. Rademacher random variables. Let $c_{1-\gamma}^*$ be $1-\gamma$ quantile of

conditionally on the data $\ensuremath{\mathcal{D}} = (Y_i,Z_i,W_i)_{i=1}^n$ and $\varepsilon_i$ are i.i.d. N(0,1).

The class of models $\ensuremath{\mathcal{P}}$ is restricted by the following assumptions.

Most of these assumptions impose mild smoothness, boundedness, or moment conditions which are plausible in empirical applications. Conditions imposed on the kernel functions are extremely mild and are verified by all kernel functions commonly used in practice. Similar assumptions are usually made in other settings for uniform nonparametric estimation and inference. Assumption (ref) imposes mild regularity conditions on the data generating process. (i) is the standard sampling assumption and may be relaxed to time series data under some weak dependence conditions, e.g., see babii2016b; (ii) may rule out common elements between $Z$ and $W$, see discussion in Section (ref) how to deal with common elements; (iii) allows for boundary-corrected kernels, see darolles2011nonparametric\footnote{Otherwise, to avoid problems at end-points, we will make a restriction to the interior of $[0,1]^{p+q}$, and all results should be read as uniform over the interior of this set.} for further discussion. Assumption (v) is a completeness condition which may be questionable in empirical applications. Note, however, that in many cases the completeness condition can be relaxed, and it is possible to have valid inferences for various deviations from the completeness condition, see babiiflorens2016b for more details.

Assumption (ref) (vi) involves several conditions on tuning parameters and deserves special attention. It imposes implicitly that different components of the model have sufficient regularity with respect to the dimensions $p$ and $q$. To illustrate that these conditions are not contradictory, suppose that $h_n\sim n^{-c_1}$ and $\alpha_n\sim n^{-c_2}$ for some $c_1,c_2>0$ and that $\beta\geq 1$. Then Assumption (ref) reduces to the following 6 conditions: (1) $c_2>(p\vee q/2)c_1$, (2) $4c_2+qc_1>1$, (3) $2c_2+2qc_1<1$, (4) $c_1>1/t$, (5) $c_1(s-q/2)>c_2$, and (6) $c_2+c_1(2s+q)>1$. Conditions (1)-(3) define the set of feasible value for $c_1$ and $c_2$ depending on the dimension parameters $p$ and $q$. Then (4) and (5)-(6) simply require that smoothness parameters $t$ and respectively $s$ are sufficiently large. For instance, when $p=q=1$, we can take, $h_n\sim n^{-1/5}$ and $\alpha_n\sim n^{-1/4}$, and need $t>5$ and $s>7/4$.

The bootstrap and the concentration inequality-based confidence sets are described as

with

where the operator norm can be computed using Lemma (ref) as

The next result provides theoretical justification for our honest uniform confidence sets.

The proof of this result can be found in the Appendix and is a consequence of more general Theorems (ref) and (ref). In particular, Theorem (ref) derives coverage errors for concentration inequality-based confidence sets, while Theorem (ref) establishes the asymptotic validity of bootstrap confidence sets for a general class of ill-posed inverse models. Theorem (ref) is based on the Gaussian and bootstrap coupling inequalities as well as the anti-concentration inequality of chernozhukov2014gaussian and chernozhukov2016empirical. Note that concentration-based confidence sets allow us to characterize explicitly the coverage error, which has not been known for confidence sets in the nonparametric IV model. Inspection of the proof of Theorem (ref) reveals that bootstrap confidence sets have additional coverage errors due to the Gaussian and bootstrap approximations.

To illustrate that conditions needed for $\delta_n\to 0$ are not contradictory, suppose that $h_n\sim n^{-c_1}$ and $\alpha_n\sim n^{-c_2}$ for some $c_1,c_2>0$ and that $\beta\geq 1$. Then we need (1) $c_2>pc_1$, (2) $4c_2+qc_1>1$, (3) $2c_2+qc_1<1$, (4) $c_1>1/t$, (5) $c_1s>c_2$, and (6) $c_2+c_1(2s+q)>1$. Note that the set of feasible values of $c_1,c_2$ is larger than that allowed by Assumption (ref) (vi).

The following theorem describes the expected size of our confidence sets.

Note that we need $\alpha_n^2nh_n^{q}\to \infty$ to ensure that the concentration-based confidence set shrinks and a slightly stronger condition $\alpha_n^2nh_n^{2q}\to \infty$ to ensure that the bootstrap confidence set shrinks, which is also needed to ensure that the coverage error tends to zero.

Functional regressions and density deconvolution

Functional regressions

In the functional linear regression, the real dependent variable $Y$ is explained by the continuous-time stochastic process $Z(t),t\in[0,1]^p$. The distinctive feature of this class of models is that it allows handling high-dimensional data without relying on the sparsity assumption, which may be restrictive in this setting. There is vast literature on functional regression models in statistics, see cardot2003testing, hall2007methodology, and references therein. There is also some growing literature in econometrics, see florens2015instrumental, benatia2017functional, and babii2016b. However, all these papers focus on the estimation in the $L_2$ norm and to the best of my knowledge, there are no currently available uniform inferential methods for functional regression models.

The functional IV regression model is described as

The slope parameter $\varphi$ measures the strength of the impact of the process $Z$ at different points $t\in[0,1]^p$. If $W=Z$, we obtain the classical functional linear regression model without endogeneity, see, for instance, hall2007methodology, while in the IV case of florens2015instrumental, $W$ is some functional instrumental variable, uncorrelated with the error term.

The moment restriction leads to the ill-posed equation

The slope parameter $\varphi$ is identified when the covariance operator is 1-1, which generalizes the non-singularity condition for the covariance matrix in the finite-dimensional linear regression model.

A variation of this model is studied in babii2016b, where the identification is achieved with real-valued instrumental variables through the conditional moment restriction $\ensuremath{\mathds{E}}[U|W]=0$. The identifying restriction is the linear completeness condition. Conditional mean-independence leads to the following ill-posed equation

which for appropriate families of functions $\Psi(s,.)$ can equivalently be written as

In what follows we denote by $W(s)$ either the regressor $Z(s)$, some functional IV $W(s)$, or the function $\Psi(s,W)$ of some real IV $W$. This notation allows encompassing all variations of the functional regression discussed above.

Unlike the NPIV, which requires a non-parametric estimation of the joint density function, all components of functional regression models are estimated at the parametric rate (in the $L_2$ norm) using sample analogs to population moments

Operator $T$ can be estimated as

The adjoint operator $\hat T^*$ can be obtained using Fubini's theorem

Put

where $\hat X_{ni}(s) = \hat U_iW_i(s)$, $\hat U_i = Y_i - \langle Z_i,\varphi\rangle$, and $\eta_i$ are i.i.d. Rademacher random variables.

Let $c_{1-\gamma}^*$ be $1-\gamma$ quantile of

and $\varepsilon_i\sim_{i.i.d.}N(0,1)$.

The following assumption restricts the class of models $\ensuremath{\mathcal{P}}$.

Note that the functional linear regressions admit a simpler characterization and involve even less stringent assumptions than the nonparametric IV model.

The bootstrap and the concentration inequality-based confidence sets are

and

with

where the operator norm can be computed using Lemma (ref) as

The next result provides theoretical justification for confidence sets based on the concentration inequality and bootstrap confidence sets.

Note that for concentration inequality-based confidence sets the coverage error is effectively driven by the bias induced by the regularization and kernel smoothing. For bootstrap confidence sets we have an additional coverage error due to the Gaussian and bootstrap approximations and need additional condition $nh_n^2/\log^7n\to \infty$. The following result describes the expected size of various confidence sets.

To ensure that confidence sets shrink we need $\alpha_n^2n \to \infty$. In both cases, sets shrink at the same speed.

Density deconvolution

Often economic data are not measured precisely. Density deconvolution allows estimating the density of unobserved data from the data measured with errors. Density deconvolution is encountered in a variety of econometric applications, e.g. to the earning dynamics in bonhomme2010generalized, to panel data in evdokimov2010identification, or to the instrumental regression in adusumilli2018nonparametric. In the simplest example of this model, see carrasco2011spectral, we have some noisy scalar observations $Y$, of the latent variable $Z$, contaminated by measurement errors $U$

Distributions of both $Z$ and $U$ are assumed to be absolutely continuous with respect to the Lebesgue measure with densities $\varphi$ and $f$. The density of measurement errors $f$ is assumed to be known. The goal is to recover the density of the latent variable $Z$ from observing contaminated i.i.d. sample $(Y_i)_{i=1}^n$.

Independence and additivity of the noise imply that the density function $r$ of $Y$ satisfies the following convolution equation

where the operator $T:L_2\to L_2$. For simplicity of presentation and tractability of our results, let us assume that all densities are continuous, bounded and compactly supported, with support contained inside the interval $[0,1] \subset \ensuremath{\mathbf{R}}$. This assumption is not very restrictive, since most of the economic variables, e.g., reported earning or costs are bounded.

The adjoint operator to $T$ can be computed using Fubini's theorem

We estimate the density of $Y$ from the sample $(Y_i)_{i=1}^n$ using the kernel density estimator

Put

where $\hat X_{ni}(y) = \frac{1}{\sqrt{h_n}}K\left(\frac{Y_i - y}{h_n}\right) - \frac{1}{n\sqrt{h_n}}\sum_{i=1}^nK\left(\frac{Y_i - y}{h_n}\right)$ and $\eta_i$ are i.i.d. Rademacher random variables.

The class of models $\ensuremath{\mathcal{P}}$ is restricted by the following assumptions.

Note that for this model, operators $T$ and $T^*$ are known, which simplifies the analysis. Let $c_{1-\gamma}^*$ be $1-\gamma$ quantile of

and $\varepsilon_i\sim_{i.i.d.}N(0,1)$.

The bootstrap and the concentration inequality-based confidence sets are described as

and

with

where the operator norm can be computed using Lemma (ref) as

The next result provides theoretical justification for confidence sets based on the concentration inequality and bootstrap confidence sets.

Note that for concentration inequality-based confidence sets the coverage error is effectively driven by the bias induced by the regularization and kernel smoothing. For bootstrap confidence sets we have an additional coverage error due to the Gaussian and bootstrap approximations and need additional condition $nh_n^2/\log^7n\to \infty$. The following result describes the expected size of various confidence sets.

To ensure that a concentration inequality-based confidence set shrinks, we need $\alpha_n^2h_nn \to \infty$ while for the bootstrap confidence set we need the slightly stronger requirement $\alpha_n^2h_n^2n\to \infty$.

Implementation and Monte Carlo experiments

This section reports results of Monte Carlo experiments for our confidence sets. We focus on the NPIV estimator. Samples of size $n\in\{1000,5000\}$ are generated as follows

where $\sigma_z = \sigma_w = 0.3$, $\sigma_u = \sqrt{0.03}, \sigma_{zu} = 0.04$, and $\rho=0.3$. To be consistent with the theory, we keep only observations inside a sufficiently large compact set. The strength of the instrument as measured by the correlation coefficient $\rho$ is calibrated to our empirical application.

Since the joint density of $(Z,W)$ is bivariate normal, this choice of functional forms resembles up to a constant the convolution of Gaussian densities. Therefore, with appropriate modifications, we can easily adapt the simulation design not only to functional regression but also to the deconvolution model. Since the underlying integral equation would be the same in all cases, to economize on space, we do not report MC experiments for other models. We also remark that the normal density leads to rapidly declining singular values. This design corresponds to the most difficult severely ill-posed setting. We consider $5000$ replications of each experiment.

We consider confidence sets described in Eq. (ref), where the operator norm $\|\hat T^*\|_{2,\infty}$ is computed using the mixed norm of the kernel function of $\hat T^*$ by Lemma (ref). The estimate of the density function $f_{ZW}$ is obtained using kernel smoothing. For simplicity of implementation, we do not optimize the performance with higher-order boundary kernels and take the product of second-order Epanechnikov kernels.

The estimator is discretized using a simple Riemann sum on the grid of $100$ equidistant points. This suffices to ensure that the numerical errors are negligible compared to the statistical noise in our setting. A higher number of grid points or cubature can give better approximation if needed. The discretized estimator has closed-form expression, and its computation boils down to solving a system of linear equations

where $\mathbf{I}$ is the $T\times T$ identity matrix, $\mathbf{r} = \left(\frac{1}{n}\sum_{i=1}^nY_ih_n^{-1}K\left(h_n^{-1}(W_i - w)\right)\right)_{1\leq j\leq T}$, $\mathbf{K}=\boldsymbol{f}\Delta$ with $\boldsymbol{f} = (\hat f_{ZW}(z_k,w_j))_{1\leq j,k\leq T}$, and $\Delta$ is the grid step. Therefore, the estimator is straightforward to implement and fast to compute.

\paragraph{Data-driven choice of tuning parameters.} As discussed in Section (ref), for confidence sets, the bias-variance trade-off for the risk of the estimator reduces to the trade-off between coverage errors and the diameter of confidence sets. For larger values of tuning parameters, the band becomes too narrow, and the bias starts to dominate, reducing uniform coverage. On the other side, even though smaller values of tuning parameters reduce bias and the estimator becomes closer to its population value, they also increase the size of the variance and lead to wider bands. The optimal choice of tuning parameters balances the two.

Existing data-driven rules for Tikhonov regularization aim to obtain an accurate estimator, see feve2010practice or centorrino2014data, and do not take into account the trade-off between the coverage and the diameter. Note that the risk-optimal estimation conflicts with optimal inference. In particular, our theory requires undersmoothing in order to obtain valid inferences. We suggest the following procedure inspired by bissantz2007non:

Like bissantz2007non we find that $J=20$ and $\tau = 2$ work well in practice. We also find that $(\kappa_a,\kappa_h) = (3.5,1.5)$ is suitable for concentration-based confidence sets and $(\kappa_a,\kappa_h) = (1,1.5)$ works well for bootstrap confidence sets. Note that bissantz2007non have a single tuning parameter, and they find that the factor $1.5$ is suitable, which is what we find for the bandwidth parameter.

\paragraph{Simulation results.} In Figure (ref) we plot estimates with a $95\%$ confidence band based on bootstrap approximation, averaged over $5000$ Monte Carlo experiments. Figure (ref) represents the same plot for confidence bands based on concentration inequality. We also report MC coverage probabilities $\hat\gamma$. In our Monte Carlo experiments, bootstrap approximation leads to slightly better performance. Resulting confidence bands are narrower, and the estimator is centered more closely to the population value of the parameter of interest. Nonetheless, confidence sets based on the concentration inequality are also informative about global shape properties of the estimated function. For example, Figure (ref) (b) gives an excellent confidence band with a reasonably small amount of bias.

\paragraph{Exogenous covariates.} In the presence of additional exogenous covariates, one can consider several avenues. As an example, suppose that one is interested in estimating the demand function, so $Z=(Z_1,Z_2)$ with $Z_1$ denoting the endogenous price and $Z_2$ denoting exogenous covariates, such as income and other control variables. One way to accommodate the exogenous covariates is to estimate the semiparametric partially linear specification allowing for non-linearities in prices, see yatchew2001household and florens2015instrumental.

Alternatively, one may want to keep the fully non-parametric specification. In this case, the model $Y=\varphi(Z_1,Z_2)+U,\ensuremath{\mathds{E}}[U|Z_2,W]=0$ leads to the following family of ill-posed integral equations

For any fixed value of $z_2$, we are back to our setting, and our uniform inferential methods would allow us to construct uniform confidence sets for $z_1\mapsto \varphi(z_1,z_2)$ for any fixed value of $z_2$. Uniform inference for the demand function for different income levels might be sufficient to reveal the amount of non-linearities and/or endogeneity. Constructing the joint uniform confidence set for the bivariate function $\varphi$ and the theoretical analysis of such sets is beyond the scope of the present paper.

Engel curves in the United States

In this section we estimate Engel curves using the NPIV approach and construct uniform confidence sets using the bootstrap. Engel curves describe how the demand for a commodity changes while the household's budget increases. Estimation of Engel curves is fundamental for the analysis of consumer behavior and has implications in different fields of empirical research. Interesting applications include the measurement of welfare losses associated with tax distortions in banks1997quadratic, estimation of growth and inflation in nakamura2014chinese, or estimation of income inequality across countries in almaas2012international.

Previously blundell2007semi estimated a shape-invariant system of Engel curves for food, alcohol, and fuel on UK data with a sieve approach. For simplicity, we focus on the non-parametric specification of the Engel curve.

Our dataset is drawn from the 2015 US Consumer Expenditure Survey data, and we estimate Engel curves for a set of goods, including food, tobacco, alcohol, gas and oil, and health. In our subsample, we have married couples with positive income during the past 12 months, including households with and without children. The dependent variable is the share of expenditures on the particular commodity. The log of total expenditures on non-durable goods is used as an independent variable. As in blundell2007semi we instrument the log of total expenditures with income before tax. However, in contrast to blundell2007semi, we also estimate Engel curves for tobacco and health care.

We plot NPIV estimates with a $95\%$ uniform confidence bands in Fig. (ref). We also plot the local linear estimator, which does not correct for the endogeneity bias. Estimates in Figure (ref) illustrate that there is significant endogeneity bias in the Engel curves. Indeed, the local polynomial estimator is often outside the confidence band. For instance, the local polynomial estimator largely overestimates the shape of the Engel curve for higher budgets. In most of the cases the estimated Engel curves exhibit more curvature than suggested by the local polynomial estimator. While it may look like Engel curves slightly increase at the beginning, this observation is not statistically significant and most likely can be attributed to boundary effects.

Conclusions

This paper studies uniform inferential methods in ill-posed models treated with Tikhonov regularization. Building uniform confidence sets in this setting is a difficult problem and requires approximating the distribution of the supremum of a complex empirical process. We show that it is not possible to establish functional convergence of this process for a very general class of ill-posed models known in econometrics and statistics. Nonetheless, we demonstrate that it is possible to obtain uniform inference relying on alternative methods.

We develop two approaches to uniform inference that lead to honest confidence sets. Honest confidence sets are of practical interest for several reasons. They ensure the existence of a certain sample size after which the coverage level will be not significantly smaller than the nominal coverage level, regardless of how complex the estimated function is within a given smoothness class. Moreover, it is widely recognized in non-parametric statistics, results for a fixed model can lead to inconsistent notions of optimality. Honest confidence sets, on the other hand, have the uniform validity.

The first approach developed in this paper relies on data-driven concentration inequality and allows for non-asymptotic characterization of coverage errors and expected diameters. These rates are new and have not been previously discussed in the literature. We also develop the asymptotic bootstrap approach to inference relying on the coupling inequalities of chernozhukov2016empirical and show that the resulting confidence sets have desirable theoretical properties.

Both methods demonstrate good performance in Monte Carlo experiments, and it seems that bootstrap confidence sets are slightly more narrow for a comparable coverage level. On the other hand, concentration-based confidence sets are computationally less demanding and still reveal lots of information about the global shape properties of functional parameters. Lastly, unlike bootstrap and/or Gaussian approximation-based inference, concentration inequalities do not impose any restrictions on indexing classes of empirical processes. Therefore, this approach can be easily adapted to other complex nonparametric estimators.

\setcounter{page}{1} \setcounter{section}{0} \setcounter{equation}{0} \setcounter{table}{0} \setcounter{figure}{0}

Concentration of the supremum of empirical processes

Let $(X_i)_{i\in\ensuremath{\mathbf{N}}}$ be a sequence of i.i.d. random variables taking values in some measure space $(S,\ensuremath{\mathcal{S}})$, and let $\ensuremath{\mathcal{H}}$ be a countable class of real functions defined on $S$. Consider the empirical process

and denote the symmetrized process by

where $(\eta_i)_{i\in\ensuremath{\mathbf{N}}}$ is a sequence of i.i.d. Rademacher random variables, independent from $(X_i)_{i\in\ensuremath{\mathbf{N}}}$. We use $\|.\|_\ensuremath{\mathcal{H}}$ to denote the supremum norm over the class of functions $\ensuremath{\mathcal{H}}$.

The second inequality in Eq. (ref), is a symmetrization inequality, while the first is the desymmetrization inequality. These inequalities allow us to establish uniform limit theorems and play an essential role in the empirical process theory, see van2000weak. We refer to koltchinskii2006local, p.7, and references therein for a proof of this modification of the symmetrization inequality.

Our construction of confidence sets relies on the concentration inequality for functions of bounded difference, also known as McDiarmid's inequality, see the excellent treatment of concentration inequalities in boucheron2013concentration. Such concentration inequalities tell us that the probability that the supremum of the empirical process $\|\nu_n\|_\ensuremath{\mathcal{H}}$ deviates from its expected value $\ensuremath{\mathds{E}}\|\nu_n\|_\ensuremath{\mathcal{H}}$ declines exponentially fast. While this expected value is unknown in practice, it can be estimated using the multiplier bootstrap with Rademacher multipliers. The symmetrization inequality allows us then to compare the expected value of this estimate $\|\nu_n^\eta\|_\ensuremath{\mathcal{H}}$ to the original expected value, which leads to the data-driven concentration inequality. This insightful idea comes from the statistical learning literature (see, e.g., koltchinskii2001rademacher).

The next proposition states the precise version of the data-driven concentration inequality used in the present paper.

See koltchinskii2011oracle, Theorem 4.6, for the proof of this result.

General ill-posed inverse problems

Impossibility of weak convergence

We first discuss the impossibility of using the uniform central limit theorem to obtain the distribution of the estimator.

To fix notation through the rest of the paper, let $P$ be a probability measure to any of the ill-posed models introduced in the previous section. The model is described by the function equation $r=T\varphi$, where $\varphi:[0,1]^p\to\ensuremath{\mathbf{R}}$ is an infinite dimensional parameter of interest. We aim to construct a random set $C_{n,1-\gamma}$ containing the function $\varphi$ with probability of at least $1-\gamma$ for $\gamma\in(0,1)$ and such that its diameter shrinks as the sample size increases at a specific rate. We focus on confidence sets for the Tikhonov-regularized estimator defined as follows

where $\hat T,\hat T^*$, and $\hat r$ are appropriate estimators\footnote{If some operators are known, which is the case in the density deconvolution model, we replace estimators by known quantities.} and $\alpha_n$ is some positive sequence converging to zero as $n\to\infty$. This estimator belongs to the general family of spectral regularization schemes, see carrasco2007linear.

For models considered in this paper, the dominating stochastic component of the Tikhonov-regularized estimator is driven by a sequence of i.i.d. centered random functions

Assume that $T$ is an integral operator with continuous kernel function, so that $T,T^*$, and $T^*T$ map to the space of continuous functions. Thus, we can also think of the operator $T$ as acting between $(C,\|.\|_\infty)$ spaces. By Lemma (ref) in the Appendix C, the operator $(\alpha_n I + T^*T)$ is invertible between spaces of continuous functions. Thus, trajectories of the process in Eq. (ref) belong to the space $(C,\|.\|_\infty)$.

To build uniform confidence sets, we need to approximate the distribution of the supremum of this process. The most straightforward route to achieve this would be to establish its weak convergence to some Gaussian process and then to rely on quantiles of Gaussian suprema to build uniform confidence sets. Unfortunately, the process in Eq. (ref) can not converge weakly as a random element in $(C,\|.\|_\infty)$ space. Indeed, we introduced regularization to smooth out discontinuities of the operator inversion. However, in the limit, as the regularization parameter tends to zero, we get back a discontinuous/unbounded operator. This discontinuity in the limit is a precise reason why the functional convergence fails.

More precisely, we can see that depending on the direction $\delta\in L_2$, the speed of convergence of inner products $\langle\hat\varphi - \varphi,\delta\rangle$ differs, see carrasco2007linear, which makes it impossible to converge weakly in $L_2$. In Proposition (ref) we show formally that the weak convergence in $(C,\|.\|_\infty)$ is impossible generalizing the result of cardot2007clt who focus on the weak convergence in $L_2$ in the special case of the functional linear regression without instrumental variables. Note that our result holds for the arbitrary ill-posed inverse model, including the nonparametric IV, functional IV regressions, and the density deconvolution. Note also that our result holds regardless of what type of the estimator is used in practice, be it kernel or series estimators in the nonparametric IV model. In particular, it is worth comparing it to the impossibility of weak convergence of the kernel density estimator, where inner products do converge at the same speed, but the only compatible limiting process is a white noise process, as in ruymgaart1998note.

Proposition (ref) tells us that it is not possible to rely on the asymptotic approximation with the conventional central limit theorem. As a result, we need to rely on alternative approaches to inference.

Bootstrap confidence sets

Consider the empirical process

indexed by some pointwise measurable pre-Gaussian class of functions $\ensuremath{\mathcal{G}}_n\subset L_2(P)$, where $L_2(P)$ is the set of square-integrable functions with respect to $P$ and $P_n$ is the empirical measure. We will always assume that the data are i.i.d. In our setting the empirical process $V_{n}$ can also be considered as a stochastic process

indexed by $w\in[0,1]^q$. Let

be the supremum of the empirical process and let

be the supremum of the tight centered Gaussian process with the same covariance structure as $V_{n}$, i.e., for all $g_1,g_2\in\ensuremath{\mathcal{G}}_n$

We shall note that we will work only with processes having continuous trajectories so that all suprema can be restricted to countable sets of rational numbers, and whence are measurable.

Note that by the linearity

so we can always drop the absolute value from the supremum extending the supremum to the enlarged class of functions. A similar observation is valid for the supremum of the Gaussian process dudley2014uniform, Theorem 2.1. We will typically work with suprema of absolute value and will use the above facts repeatedly.

Let $G_{n}$ be the envelope of the class $\ensuremath{\mathcal{G}}_n$, which we assume to be square-integrable and let $N(\ensuremath{\mathcal{G}}_n,\|.\|_{Q,2},\epsilon)$ be the $\epsilon$-covering of $(\ensuremath{\mathcal{G}}_n,\|.\|_{Q,2})$, with $\|g\|_{Q,q} = (Q|g|^q)^{1/q}$ for some probability measure $Q$. We assume that $\ensuremath{\mathcal{G}}_n$ is a VC-type class of functions.

For an i.i.d. sequence of Gaussian random variables $(\varepsilon_i)_{i=1}^n$, consider the supremum of the bootstrapped process

The process $X_{ni}$ is typically unobserved. Let $\hat X_{ni}$ be its empirical counterpart estimated from the data and let $c_{1-\gamma}^*$ be $1-\gamma$ conditional on the data $\ensuremath{\mathcal{D}}_n$ quantile of the supremum of the bootstrapped process

where $(\varepsilon_i)_{i=1}^n$ are i.i.d. $N(0,1)$ random variables independent from the sample.

Assumptions (ref) and (ref) consist of several technical conditions needed for the Gaussian and bootstrap approximation of chernozhukov2014gaussian.

Assumption (C3) is asymptotic linearization of residuals in the model. In some cases this expansion is exact and $R_{1n}$ is trivially zero. (C1) controls the estimation error in the operator, while (C2) controls the error in the bootstrap process.

We focus on bootstrap confidence sets described as

with $q_n^* = \frac{c_{1-\gamma}^*\|\hat T^*\|_{2,\infty} + \log^{-1}n}{\alpha_nn^{1/2}}u_n$.

In the following theorem, we show that our bootstrap confidence sets are honest to $\ensuremath{\mathcal{P}}$ and characterize coverage errors explicitly. This result will be specialized to each particular model in the following sections.

To characterize expected diameters of bootstrap confidence sets we need additional assumptions.

Concentration-based confidence sets

The confidence sets based on the concentration inequality are described as

where $\hat q_n = 2\left\|\hat \nu_{n}^\eta\right\|_\infty + \frac{3\|\hat T^*\|_{2,\infty}G\sqrt{2\log(1/\gamma)} + \log^{-1} n}{\alpha_nn^{1/2}}u_n$ and

denotes the estimate of the supremum of the symmetrized process. The operator norm $\|\hat T^*\|_{2,\infty}$ can be computed by Lemma (ref), while $G$ is selected so that $\|X_{ni}\|\leq G$.

For concentration inequality we need a somewhat different version of Assumption (ref).

The next result describes the convergence rate of the expected diameter of the confidence set based on the concentration inequality.

NPIV, functional regressions, and density deconvolution

NPIV model

Functional regressions

Density deconvolution

Technical lemmas

We first characterize the bias of the regularized estimator uniformly over our class of models.

We also need the following inequality known in the theory of numerical ill-posed inverse problems.

For the next result, put

For the following lemma denote