The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
54,025 characters
Adaptive estimation for some nonparametric instrumental variable models
\allowdisplaybreaks
\title{Adaptive estimation for some nonparametric instrumental variable models}
\author{Fabian Dunker\footnote{\noindent School of Mathematics and Statistics, University of Canterbury, Private Bag 4800, Christchurch 8140, New Zealand, \textit{Email:} [email removed]}\\
University of Canterbury}
\date{}
\maketitle
\parindent0pt\textbf{Abstract:}
The problem of endogeneity in statistics and econometrics is often handled by introducing instrumental variables (IV) which fulfill the mean independence assumption, i.e. the unobservable is mean independent of the instruments. When full independence of IV's and the unobservable is assumed, nonparametric IV regression models and nonparametric demand models lead to nonlinear integral equations with unknown integral kernels. We prove convergence rates for the mean integrated square error of the iteratively regularized Newton method applied to these problems. Compared to related results we derive stronger convergence results that rely on weaker nonlinearity restrictions. We demonstrate in numerical simulations for a nonparametric IV regression that the method produces better results than the standard model.
\vskip .2in
\noindent {\sl MSC: AMS 2010 subject classification.} primary 62G08, secondary 62G20\\
\noindent {\sl Keywords and phrases:} Nonparametric regression, instrumental variables, nonlinear inverse problems, regularization
\onehalfspacing
\parindent8pt
\section{Introduction}
Dependence of an unobservable error term and covariates is a frequent problem in statistical and econometrical modeling known as endogeneity. An efficient way to deal with
endogeneity is to use instrumental variables (IV) in the estimation. These are additional variables which can assumed to be independent or mean independent of the unobservable. In the context of nonparametric estimation the IV approach usually leads to ill-posed problems with an unknown operator that needs to be estimated. The solution $\varphi$ of the nonparametric IV problem can be characterized by a possibly nonlinear operator equation
\begin{align}\label{eq:opeq}
\mathcal{F} (\varphi) = \psi.
\end{align}
In some regression models $\psi = 0$, in others $\psi$ is a function that has to be estimated from observations by some estimator $\widehat{\psi}$. The operator $\mathcal{F}: \mathbb{X} \to \mathbb{Y}$
is an integral operator between some Banach or Hilbert spaces $\mathbb{X}$ and $\mathbb{Y}$ which is unknown in applications. Only an estimator $\widehat{\mathcal{F}}$ is available.
The inverse of the operators $\mathcal{F}$ or $\widehat{\mathcal{F}}$ is usually not continuous. Even with an arbitrarily small variance in $\widehat\psi$ and $\widehat\mathcal{F}$ we usually
have $\mathop{\rm {\mathbb V}ar}\nolimits(\|\widehat{\mathcal{F}}^{-1}\widehat\psi\|_\mathbb{X}) = \infty$ and the straightforward estimator $\widehat\varphi = \widehat{\mathcal{F}}^{-1}\widehat\psi$ is typically inconsistent.
We discuss specific examples for nonparametric IV models and the related operators together with the respective literature in Section \ref{sec:application}.
In this paper we describe and analyze a consistent estimator for this type of problem, when $\mathcal{F}$ is an operator between Hilbert spaces. The estimator is based on the iteratively regularized Gau{\ss}-Newton method (IRGNM) with iterated Tikhonov regularization defined below in \eqref{eq:it_tik}. Details about the method will be given in Section \ref{sec:estimation}.
This method was suggested by \cite{Baku:92}. Important monographs on this topic are \cite{BK:04b} and \cite{KNS:08}. These contributions consider only problems with known operators and deterministic right hand side in equation \eqref{eq:opeq}. The use of IRGNM for nonparametric IV problems was proposed and analyzed by \cite{DFHJM:14}. They derived rates for convergence in probability with a priori parameter choice using variational methods.
The novelty of this paper is that we prove significantly faster convergence rates for the mean integrated squared error (MISE) rather than convergence in probability under a different set of assumptions. In addition, we propose adaptive estimation with Lepski\u\i's principle and prove rates for this case.
Furthermore, we assume a significantly weaker nonlinearity condition for the operator $\mathcal{F}$ which has a clear interpretation and is reasonable for most applications while the nonlinearity condition in \cite{DFHJM:14} is difficult to interpret and to check. We also prove faster rates of convergence when the regression function is smooth enough. Our proofs do not use variational methods. Instead we rely on spectral methods as in \cite{BauHohMun:09}. We also use a modification of Hoeffding's inequality from \cite{McD:89}.
The paper is organized as follows. We discuss in Section \ref{sec:application} some IV models which fit into the framework of this paper and explain the estimator. The estimator is introduced in Section \ref{sec:estimation}.
Section \ref{sec:convergence} contains convergence rate theorems. Finally, we present some numerical simulations in Section \ref{sec:numerics}. All proofs are in the Appendix.
\section{Nonparametric instrumental variable models}\label{sec:application}
In our general framework a function $\varphi^{\dagger}$ is characterized by the possibly nonlinear operator equation
\begin{align}\label{eq:opeq0}
\mathcal{F}(\varphi^{\dagger}) = 0,
\end{align}
i.e. $\varphi^{\dagger}$ is the true solution. Here $\mathcal{F} :B_{2R}(\varphi_0) \subseteq \mathbb{X} \rightarrow \mathbb{Y}$ is an operator between Hilbert spaces with norms $\|\cdot\|_\mathbb{X}$ and $\|\cdot\|_\mathbb{Y}$ respectively. Tyical examples for $\mathbb{X}$ and $\mathbb{Y}$ are $L^2$ and $L^2$ based Sobolev spaces $H^i$ for $i=1,2,\ldots$. A ball $B_{2R}(\varphi_0)$ with radius $2R$ around an initial guess $\varphi_0$ is contained in the domain of $\mathcal{F}$. In practice, large values of $R$ are possible. The operator equation is allowed to be ill-posed, i.e. $\mathcal{F}^{-1}$ may not be continuous. Furthermore, the operator $\mathcal{F}$ is not known in applications. Only a series of estimators $\widehat{\mathcal{F}}_n :B_{2R}(\varphi_0) \subseteq \mathbb{X} \rightarrow \mathbb{Y}$ are available where $n$ denotes the sample size. We assume that $\varphi^{\dagger}$ is a unique solution to \eqref{eq:opeq0} in $B_{2R}$, i.e. the problem is locally identified. In the following we discuss econometric examples for this setup.
\subsection{Nonparametric IV regression}
\paragraph{Mean independence}
The simplest nonparametric IV regression model has a separable error term and a mean independence condition
\begin{align}\label{eq:iv_reg_mean_independence}
Y = \varphi(X) + U \qquad \mbox{with } \mathbb{E}[U|Z] = 0.
\end{align}
Here and in all following models $Y$ and $U$ are univariate random variables, while $X$ and $Z$ can be multivariate and their dimensions do not have to coincide. The regressor $X$, the instrument $Z$ and the response $Y$ are observed, while the error term $U$ is unobservable.
This model was proposed by \cite{NewPow:03} and \cite{florens:03}. It was further studied and applied in \cite{HalHor:05}, \cite{BluCheKri:07}, \cite{CheRei:11}, \cite{DFFR:11} \cite{FJV:11}, \cite{Hor:11}, \cite{JVV:11}, \cite{GS:12_reg}, \cite{ChePou:12}, \cite{Horowitz14}, \cite{ChenChrist:15}, \cite{BreJoh:15}, \cite{ChenChrist:18}, as well as \cite{Babii:20} among others. For an overview see \cite{Hor:14_survey}.
We can write \eqref{eq:iv_reg_mean_independence} equivalently as $\mathbb{E}[\varphi(X)|Z] = \mathbb{E}[Y|Z]$ and if the conditional densities $f_{X|Z}$ and $f_{Y|Z}$ exist, as
\begin{align}\label{eq:iv_reg_lin_op}
\int f_{X|Z}(x|z) \varphi(x) dx = \int f_{Y|Z}(y|z)dy \qquad \mbox{for all } z \in \mathrm{supp}\,(Z).
\end{align}
We define the linear integral operator $(\mathcal{F}_{ce}\varphi)(z) := \int f_{X|Z}(x|z) \varphi(x) dx$
with integral kernel $f_{X|Z}(x|z)$ and the function
$\psi(z) := \int y f_{Y|Z}(y|z)dy$. Model \eqref{eq:iv_reg_mean_independence} can be given in operator form $(\mathcal{F}_{ce}\varphi)(z) = \psi(z)$.
The integral kernel $f_{X|Z}$ and thereby $\mathcal{F}_{ce}$ as well as the function $\psi$ are unknown and have to be estimated from a sample of $Y,X,Z$. An Density estimators $\widehat f_{X|Z}$ and $\widehat f_{Y|Z}$ give estimators $\widehat\mathcal{F}_{ce}$ and $\widehat \psi$ in a natural way.
While the main focus of this paper is on nonlinear operator equations,
we use model \eqref{eq:iv_reg_mean_independence}
as a benchmark for the IRGNM applied to model
\eqref{eq:iv_reg_full_independence} below.
The model identifies the regression function
$\varphi$ if and only if $\mathcal{F}_{ce}$ is injective. This property is called completeness, see \cite{Hault:11}, \cite{HF:15}, \cite{Andrews:17}, and \cite{BF:20}.
\paragraph{Full independence}\label{sec:full_ind}
In many applications the error term can be assumed to be independent of the instrument. Hence, mean independence of the instrument can be replace by full independence as proposed in \cite{DFHJM:14}
\begin{align}\label{eq:iv_reg_full_independence}
Y = \varphi(X) + U \qquad \mbox{with } U \protect\mathpalette{\protect\independenT}{\perp} Z \mbox{ and } \mathbb{E}[U] = 0.
\end{align}
Since the new assumptions $U \protect\mathpalette{\protect\independenT}{\perp} Z$ and $\mathbb{E}[U] = 0$ imply $\mathbb{E}[U|Z] = 0$ but not vice versa model \eqref{eq:iv_reg_full_independence} makes stronger assumptions than model \eqref{eq:iv_reg_mean_independence}.
Consequently, whenever \eqref{eq:iv_reg_mean_independence} identifies
the solution so does \eqref{eq:iv_reg_full_independence}. Furthermore, there are cases in which \eqref{eq:iv_reg_full_independence} can identify a solution, while
\eqref{eq:iv_reg_mean_independence} fails. This is for example the case with discrete instruments and continuous regressors as discussed in
\cite{DFHJM:14}, \cite{Torgo}, \cite{HF:15}, \cite{CFF:19}, and \cite{Loh:19}.
We can translate model \eqref{eq:iv_reg_full_independence} into an operator equation by defining the operator
\begin{align}\label{eq:nonlininstreg:opeq1}
(\widetilde\mathcal{F}_{ind}(\varphi))(u,z) := \left(
\begin{array}{c}
\mathbb{P}[Y - \varphi(X) \le u] - \mathbb{P}[Y - \varphi(X) \le u| Z=z] \\
\mathbb{E}[Y - \varphi(X)]
\end{array}
\right).
\end{align}
When $Y,X,Z$ have a joint density $f_{YXZ}$, taking the derivative with respect to $u$ yields the alternative operator
\begin{equation}\label{eq:nonlininstreg:opeq2}
(\mathcal{F}_{ind}(\varphi))(u,z) := \left(\begin{array}{c}
\int f_{YXZ}(u+\varphi(x),x,z) - f_{YX}(u+\varphi(x),x) f_Z(z)\,dx\\
\int \varphi(x)f_X(x)\,dx-\int y f_Y(y)\,dy
\end{array}\right).
\end{equation}
Model \eqref{eq:iv_reg_full_independence} is equivalent to the operator equations $\widetilde\mathcal{F}_{ind}(\varphi) = 0$ or $\mathcal{F}_{ind}(\varphi) = 0$.
Note that the operators are nonlinear due to the first line of \eqref{eq:nonlininstreg:opeq1} or \eqref{eq:nonlininstreg:opeq2}. Furthermore, the operators are not known and have to be estimated. A density estimator
$\widehat f_{YXZ}$ gives a straight forward estimator $\widehat \mathcal{F}_{ind}$.
For any $\varphi$ that sets the first line of the operator $\mathcal{F}_{ind}$ to $0$ also $c+ \varphi$ with $c\in\mathbb{R}$ sets it to $0$. In addition, for any $\varphi$, the second line of $\mathcal{F}_{ind}$ is set to $0$ by $\varphi -\mathbb{E}[Y-\varphi(X)]$. Hence, for any solution $\varphi$ of the first line of the operator we have $\mathcal{F}_{ind}\big(\varphi-\mathbb{E}[Y-\varphi(X)]\big)=0$. The nonlinear inverse problem is to find a $\varphi$ that solves the first line of $\mathcal{F}_{ind}$. The second line is a parametric problem that can be estimated with the parametric rate.
When we discuss this example below we will only consider the first line of the operator as this is dominating the convergence rate.
Let us denote the integral kernel of the first line of the operator \eqref{eq:nonlininstreg:opeq2} and its estimator by
$k_{ind}(y,x,z) := f_{YXZ}(y,x,z) - f_{YX}(y,x) f_Z(z)$ and $\widehat k_{ind}(y,x,z) := \widehat f_{YXZ}(y,x,z) - \widehat f_{YX}(y,x) \widehat f_Z(z)$
respectively. Then the first component of the operator reads $(\mathcal{F}_{ind}(\varphi))(u,z) = \int k_{ind}(u+\varphi(x),x,z)dx$.
\subsection{Quantile regression and non-separable models}
\paragraph{Nonparametric IV quantile regression}
Another model that leads to a different nonlinear operator equation
is nonparametric IV quantile regression proposed by \cite{HorLee:07}. For $q \in [0,1]$ the $q$-th quantile regression function $\varphi_q$ is characterized by
\begin{align}\label{eq:iv_quant_reg}
Y= \varphi_q(X) +U \qquad \mathbb{P}(U\leq 0|Z=z) = q \qquad \mbox{for all }z.
\end{align}
If the joint density $f_{YXZ}$ exists, the model is equivalent to an operator equation $\mathcal{F}_q(\varphi_q)=0$ with
\begin{equation}\label{eq:iv_quant_reg_op}
(\mathcal{F}_q(\varphi))(z) := \int F_{YXZ}(\varphi(x),x,z)\,dx- q f_Z(z)
\end{equation}
where $F_{YXZ}(y,x,z):=\int_{-\infty}^y f_{YXZ}(\tilde{y},x,z)\,d\tilde{y}$. Different estimation procedures for this model were proposed and analyzed in
\cite{HorLee:07}, \cite{ChePou:12}, \cite{GS:12_quant} \cite{DFHJM:14}, and \cite{Breunig:15}. Local identification properties of this and related models are discussed in \cite{CCLN:14}.
We can write $(\mathcal{F}_q(\varphi))(z) = \int k_q(\varphi(x),x,z)dx$ with integral kernel
\[
k_q(y,x,z) := F_{YXZ}(y,x,z) - q f_{XZ}(x,z).
\]
Replacing $q f_Z(z)$ by $\int q f_{XZ}(x,z)dx$ is impractical in applications but makes it easier to discuss properties of \eqref{eq:iv_quant_reg_op} in this paper. $\mathcal{F}_q$ and $k_q$ are unknown and have to be estimated. If we plug-in a density estimator $\widehat f_{YXZ}$, we get straight forward estimators $\widehat k_q$ and $\widehat{\mathcal{F}}_q$.
\paragraph{Non-separable model}
A related example that falls in our framework is nonparametric IV regression with unseparable error, wich was proposed in
\cite{CheImbNew:07}. See also \cite{CheHan:05}. The model is
\begin{align}\label{eq:iv_nonsep_reg}
\begin{split}
Y=\phi(X,U) \qquad \mbox{with } &U \protect\mathpalette{\protect\independenT}{\perp} Z \mbox{ and}\\
&\phi(x,u) \mbox{ strictly monotonic increasing in } u.
\end{split}
\end{align}
It was pointed out in \cite{HorLee:07}, and \cite{CheImbNew:07} that this model is already
contained in model \eqref{eq:iv_quant_reg}. Let $F_U$ be the cumulative distribution function of $U$. Normalize $\widetilde U:= F_U(U)$ and
$\widetilde\phi(x, \tilde u) := \phi(x,F_U^{-1}(\tilde u))$. Then $\widetilde U$ is uniformly distributed on $[0,1]$. The value of $\widetilde U$ corresponds to a quantile in model
\eqref{eq:iv_quant_reg}. This reduces \eqref{eq:iv_nonsep_reg} to model \eqref{eq:iv_quant_reg} with $\varphi_q(x) = \widetilde\phi(x,q)$.
\paragraph{Further examples}
We briefly comment on further econometric models that fall into the framwork of this paper. A problem that has a similar mathematical structure as IV regression appears in some nonparametric demand models for differentiated products. It was considered with mean independence assumption similar to \eqref{eq:iv_reg_mean_independence} in \cite{BH:11}, \cite{BH:14} and with full independence similar to \eqref{eq:iv_reg_full_independence} in \cite{DHK:14}. Some models for games of incomplete information lead to a nonlinear inverse problem with deterministic operator, see for example \cite{FS:10}. Nonlinear inverse problems with deterministic operators also occur in functional linear quantile regression (without instrumental variables) as in \cite{Kato:12}. The estimator in this paper can be applied to these type of problems. However, the error analysis would be different since there is no randomness in the operator.
Also related are nonparametric ARCH($\infty$) models which can be treated as linear inverse problem, see \cite{LM:05}. Further linear inverse problems in econometrics are discussed in \cite{CFR:07}.
\section{Estimation}\label{sec:estimation}
\subsection{The estimator}
Remember that $\varphi^{\dagger}$ denotes the true solution and let $\varphi_0$ be an initial guess.
Our method is based on linearizing $\mathcal{F}$ which motivates the following assumption.
\begin{assumption}\label{ass:dif}
\begin{enumerate}
\item $\|\varphi^{\dagger} - \varphi_0\|_\mathbb{X} < R$
\item $\mathcal{F}$ and all $\widehat{\mathcal{F}}_n$ are Fr\'echet differentiable on $B_{2R}(\varphi_0)$ with Fr\'echet derivatives $\mathcal{F}'$ and $\widehat{\mathcal{F}}_n'$ respectively.
\end{enumerate}
\end{assumption}
The iteratively regularized Gau\ss-Newton method with iterated Tikhonov regularization consists of two nested iterations. The outer iteration is a Newton method. It starts at $\varphi_0$ and produces in the $j$-th step the estimate $\widehat{\varphi}_{j+1}$. In the $j$-th step the operator is linearized as $\widehat{\mathcal{F}}(\varphi) \approx \widehat{\mathcal{F}}_n'[\widehat{\varphi}_j](\varphi - \widehat{\varphi}_j) + \widehat{\mathcal{F}}_n(\widehat{\varphi}_j)$. A regular Newton method would invert the linear operator $\widehat{\mathcal{F}}_n'[\widehat{\varphi}_j]$ to compute the next step. Due to the ill-posedness, this would be unstable and we use a regularized inverse instead. The regularized inverse is computed by $m$-times iterated Tikhonov regularization which is the inner iteration of the method. In the following scheme the Newton iteration is indexed by $j$ and the Tikhonov iteration by $i$, and $\alpha_j > 0$ is a regularization parameter
\begin{equation}\label{eq:it_tik}
\begin{alignedat}{1}
&\text{for } j = 0 \text{ to } J\\
&\quad \overline\varphi_{j+1,0} :=\, \widehat{\varphi}_j\\
&\quad\text{for } i = 0 \text{ to } m-1\\
&\quad\quad\overline\varphi_{j+1,i+1} := \mathop{\mathrm{argmin}}\limits_{\varphi \in \mathbb{X}} \left(\|\widehat{\mathcal{F}}_n'[\widehat{\varphi}_j](\varphi - \widehat{\varphi}_j) + \widehat{\mathcal{F}}_n(\widehat{\varphi}_j)\|_{\mathbb{Y}}^2 + \alpha_j \|\varphi - \overline\varphi_{j+1,i}\|_\mathbb{X}^2 \right)\\
&\quad \text{end}\\
&\quad\widehat \varphi_{j+1} :=\, \overline\varphi_{j+1,m}\\
&\quad\mbox{stop if } \|\widehat{\varphi}_j-\varphi_0 \|_\mathbb{X} > 2R \mbox{ and set } \widehat{\varphi}_{j+1} = \varphi_0\\
&\text{end}.
\end{alignedat}
\end{equation}
As usual for Newton methods, convergence can fail if the initial guess $\widehat\varphi_0$ is too far from the true solution $\varphi^\dag$. In practice and in simulations the method proves to be quite robust to the choice of $\widehat\varphi_0$. If no a priori information about $\varphi^\dag$ is available, $\widehat{\varphi}_0 = 0$ is usually a good choice.
With a small $\alpha_j$ the method has a large variance due to the ill-posedness of $\mathcal{F}$. While a larger $\alpha_j$ controls the variance but adds some bias. We choose $\alpha_0$ large enough to stabilize the problem and let $\alpha_j$ decay in every Newton step by
\begin{align}\label{eq:alpha}
\alpha_{j+1} = q_\alpha \alpha_j \quad \mbox{with some fixed} \quad 0<q_\alpha<1
\end{align}
to reduce the bias. A second parameter that has to be chosen is the number of inner iterations $m$. A large $m$ is of advantage for very smooth $\varphi^{\dagger}$. We will address the choice of $\alpha_0$ and $m$ in Section \ref{sec:spec_source} and Assumption \ref{ass:saturation}. The Newton iteration needs to be stopped at an appropriate iteration step. The size of the regularization parameter is linked to the number of steps. Hence, the number of steps corresponds to a bias variance trade-off. We will investigate parameter choice with a priori knowledge in Section \ref{sec:rates} and fully data driven in Section \ref{sec:Lepski}.
We introduce the following notations for shorter formulas
\[
T_\dag := \mathcal{F}'[\varphi^{\dagger}] \qquad \widehat T_{n,j} := \widehat{\mathcal{F}}_n'[\widehat{\varphi}_j] \qquad \widehat T_{n \dag} := \widehat{\mathcal{F}}'_n[\varphi^{\dagger}].
\]
An alternative formulation of the method can be obtained by using the functional calculus. Let $\widehat T_{n,j}^*$ denote the adjoint operator of $\widehat T_{n,j}$
and set
\begin{equation}\label{eq:g_alpha}
g_\alpha(\lambda) := \frac {(\lambda + \alpha)^m - \alpha^m}{\lambda(\lambda + \alpha)^m}~.
\end{equation}
Then
\eqref{eq:it_tik} is equivalent to
\begin{equation}\label{eq:irgnm_spektral}
\begin{split}
&\text{for } j = 0 \text{ to } J\\
&\quad\widehat{\varphi}_{j+1} = \varphi_0 + g_{\alpha_j}(\widehat T_{n,j}^* \widehat T_{n,j})\widehat T_{n,j}^* \left(\widehat T_{n,j}(\widehat{\varphi}_j-\varphi_0) - \widehat{\mathcal{F}}_n(\widehat{\varphi}_j) \right)\\
&\quad\mbox{stop if } \|\widehat{\varphi}_j-\varphi_0 \|_\mathbb{X} > 2R \mbox{ and set } \widehat{\varphi}_{j+1} = \varphi_0\\
&\text{end}.
\end{split}
\end{equation}
\begin{example}[Fr\'echet differentiability]\label{ex:frechet}
Assumption \ref{ass:dif} is usually fulfilled in our examples. The operators are well defined and Fr\'echet differentiable on the whole space under mild integrability conditions on the joint density $f_{YXZ}$. The Fr\'echet derivative of the operator in \eqref{eq:nonlininstreg:opeq2} exists when $f_{YXZ}$ is partially differentiable in the first variable. The operator in \eqref{eq:iv_quant_reg_op} is differentiable without further assumptions.
\begin{align*}
(\mathcal{F}'_{ind}[\varphi]\psi)(u,z) &= \left(\begin{array}{c}
\int \big[ \frac{\partial}{\partial y}f_{YXZ}(u+\varphi(x),x,z) - \frac{\partial}{\partial y}f_{YX}(u+\varphi(x),x) f_Z(z)\big] \psi(x)\,dx\\
\int \psi(x) f_X(x)\,dx
\end{array}\right),\\
(\mathcal{F}'_q[\varphi](\psi))(z) &= \int f_{YXZ}(\varphi(x),x,z)\psi(x)\,dx.
\end{align*}
Note that the derivatives are linear integral operators with kernel $\frac{\partial}{\partial y}k(\varphi(x),x,z)$.
\end{example}
\section{Convergence Rates}\label{sec:convergence}
The convergence theory is presented in four steps. We start by introducing assumptions for the general operator equation \eqref{eq:opeq0} as well as for the IV regression models \eqref{eq:nonlininstreg:opeq2} and \eqref{eq:iv_quant_reg_op}. Afterwards, we state convergence rate result for the MISE with a priori choice of the stopping parameter $j$. Then we compare the result to \cite{HorLee:07}. The last step is a theorem with data driven choice of $j$ by Lepski\u\i's principle.
\subsection{Assumptions}
\subsubsection{Smoothness condition}\label{sec:spec_source}
As usual for nonparametric methods a smoothness assumption has to be imposed on the true solution $\varphi^{\dagger}$ to get convergence rates. In our setup with an ill-posed operator equation \eqref{eq:opeq} it is necessary to link the smoothness of $\varphi^{\dagger}$ to the smoothing properties of the operator $\mathcal{F}$. An efficient and popular way to formulate this is a source condition. The following definition uses the functional calculus.
\begin{definition}\label{def:spec_sc}
Let $\Lambda : [0 , \infty) \; \to [0 , \infty)$ be continuous, strictly increasing with $\Lambda(0) = 0$. A representation of the initial error as
\begin{equation}\label{eq:spec_sc}
\varphi_0 - \varphi^{\dagger} = \Lambda(T^*_\dag T_\dag)\omega\;, \qquad \omega \in \mathbb{X} \text{ with } \rho:=\|\omega\|_\mathbb{X}
\end{equation}
is called a spectral source condition and $\Lambda$ is called an index function.
\end{definition}
When $T_\dag$ is a linear integral operator with kernel $\frac{\partial}{\partial y}k(\varphi^{\dagger}(x),x,z)$ as in Example \ref{ex:frechet}, this definition can be interpreted in the following way. We assume for simplicity that $T_\dag$ is compact which is for example the case if $\frac{\partial}{\partial y}k(\varphi^{\dagger}(x),x,z)$ is continuous. It was shown in \cite{reade841} and \cite{reade842} that the singular values of such an operator decay at least polynomially if $\frac{\partial}{\partial y}k(\varphi^{\dagger}(x),x,z)$ belongs to a Sobolev space, and exponentially if $\frac{\partial}{\partial y}k(\varphi^{\dagger}(x),x,z)$ is analytic.
Let $(\sigma_t,\, u_t,\, v_t)$ be a singular system for $T_\dag$. The source condition \eqref{eq:spec_sc} implies for $e_0 = \varphi_0 - \varphi^{\dagger}$
\[
\omega = \sum_{t\in \mathbb{N}}\frac{\langle e_0,\, v_t \rangle}{\Lambda(\sigma_t^2)} u_t \in \mathbb{X} \qquad \mbox{and thereby} \qquad
\sum_{t=1}^\infty \left(\frac{\langle e_0,\, v_t \rangle}{\Lambda(\sigma_t^2)}\right)^2 < \infty.
\]
Hence, a $\omega$ fulfilling \eqref{eq:spec_sc} only exists if $\Lambda$ compensates the decay of the singular values in a way that $\langle e_0,\, v_t \rangle\Lambda(\sigma_t^2)^{-1}$ is square summable. The decay of singular values describes the smoothing properties of the $T_\dag$ with respect to the singular vectors. While the decay of $\langle e_0,\, v_t \rangle$ describes the smoothness of $e_0$ with respect to the singular vectors. Thus, the rate of decay for $\Lambda(x)$ when $x \searrow 0$ compares these two degrees of smoothness. For the examples above the source condition compares the smoothness of $f_{YXZ}$ with the smoothness of the regression function $\varphi^{\dagger}$.
When $\sigma_t$ and $\langle e_0,\, v_t \rangle$ both decay polynomially or both decay exponentially, i.e. $\sigma_t \lesssim \exp(-c_\sigma t)$ and $\langle e_0,\, v_t \rangle \lesssim \exp(-c_{e_0} t)$ with some constants $c_\sigma$ and $c_{e_0}$, the source condition is fulfilled with $\Lambda(x) = x^\mu$. Where $\mu >0$ is a sufficiently small constant. A source condition with polynomial $\Lambda$ is called a H\"older source condition, which is a concept that goes back to \cite{lavrentev62} and \cite{morozov68}.
For exponential decay of $\sigma_t$ but only polynomial decay of $\langle e_0,\, v_t \rangle$ the source condition holds when the operator is rescaled to $\|T_\dag\| < 1$ and $\Lambda(x) = (-\ln(x))^{-p}$ with some $0<p$.
This choice of $\Lambda$ was proposed by \cite{mair94} and \cite{hohage:97} and
is called logarithmic source condition.
Despite the word ``condition'' in the name ``source condition'' it is rather a relation that selects an index function. Corollary 2 in \cite{mh08} shows that for any compact injective operator $T_\dag$ and any $e_0$ exists an index function $\Lambda$ such that a source condition is fulfilled.
In this paper we focus on H\"older source conditions with $\mu > 1/2$. Notice that this implies $e_0 \in \text{Range}(\mathcal{F}'[\varphi^{\dagger}]^*)$. The case of $\mu \le 1/2$ and logarithmic source conditions was analyzed in \cite{DFHJM:14}. We make the formal assumption:
\begin{assumption}\label{ass:source}
The true solution $\varphi^{\dagger}$ fulfills a source condition \eqref{eq:spec_sc} with sufficiently small $\rho$ and with an index function that satisfies $\Lambda(x) = \mathcal{O}(x^\mu)$ for $x \searrow 0$ with $\mu > 1/2$.
\end{assumption}
\begin{example}
For nonparametric IV regression with full independence \eqref{eq:nonlininstreg:opeq2} Assumption \ref{ass:source} implies that $\varphi_0-\varphi^{\dagger}$ is in the same smoothness class as
$\frac{\partial}{\partial y}f_{YXZ}(u+\varphi(x),x,z) - \frac{\partial}{\partial y}f_{YX}(u+\varphi(x),x) f_Z(z).$
For nonparametric IV quantile regression \eqref{eq:iv_quant_reg_op} Assumption \ref{ass:source} implies that $\varphi_0-\varphi^{\dagger}$ is in the same smoothness class as
$f_{YXZ}(\varphi^{\dagger}(x),x,z).$
The smoother $\varphi_0-\varphi^{\dagger}$ the larger is $\mu$.
\end{example}
Closely related to the smoothness of the true solution is the choice of the parameters $\alpha_0$ and $m$ for the IRGNM. In the following assumption $\|T\|_{\mathcal{L}(\mathbb{X},\mathbb{X})} := \sup_{\varphi} \{\|T\varphi\|_\mathbb{X}~| ~\|\varphi\|_\mathbb{X}=1\}$ denotes the usual operator norm for linear operators.
\begin{assumption}\label{ass:saturation}
\begin{enumerate}
\item The number of iterations of the Tikhonov regularization $m$ is larger or equal to $\mu$ in the source conditions $m\ge \mu$, i.e. $\Lambda(x)^{-1}x^{m} = \mathcal{O}(1)$ for $x \searrow 0$.
\item The initial regularization parameter $\alpha_0$ is large enough such that $\alpha_0 \geq \|\widehat T_{n \dag}^* \widehat T_{n \dag}\|_{\mathcal{L}(\mathbb{X},\mathbb{X})}/(1-q_\alpha)$.
\end{enumerate}
\end{assumption}
Both parameters need to be large enough but there is not much harm in choosing them larger than necessary. Any $\alpha_0$ and $m$ fulfilling Assumption \ref{ass:saturation} will lead to comparable estimates. However, increasing $\alpha_0$ will lead to a few more Newton steps. The lower bound of $\alpha_0$ depends on the derivative of the estimated operator and is thereby random but not unknown.
As usual for nonparametric methods the rate of convergence increases if the true solution $\varphi^{\dagger}$ is smoother, i.e. if $\mu$ is larger. But this increase is only realized if $m \ge \mu$. However, $m$ does not act as a regularization parameter.
Since the inner iteration is numerically cheap it is save to chose a larger value for $m$ without having a significant disadvantage.
\subsubsection{Nonlinearity restriction}
The non-linearity of $\mathcal{F}$ needs to be restricted for the algorithm to work. We use a Lipschitz condition on the derivative for this purpose.
\begin{assumption}\label{ass:lipschitz}
There exists $L > 0$ such that
\begin{equation}\label{eq:lipschitz}
\|\widehat{\mathcal{F}}_n'[\xi_1] - \widehat{\mathcal{F}}_n'[\xi_2]\|_{\mathcal{L}(\mathbb{X},\mathbb{Y})} \leq L \|\xi_1 - \xi_2\|_\mathbb{X}
\end{equation}
almost surely for all $\xi_1 , \xi_2 \in B_R(\varphi^{\dagger})$ and large $n$.
\end{assumption}
The special structure of the of $\mathcal{F}_{ind}$ and $\mathcal{F}_q$ allows us to replace Assumption \ref{ass:lipschitz} for the IV regression examples by the following alternative. Lemma \ref{lem:nonlin_restrict} in the appendix shows that Assumption \ref{ass:lipschitz_alt} implies Assumption \ref{ass:lipschitz}.
\begin{assumption}\label{ass:lipschitz_alt}
For the operators \eqref{eq:nonlininstreg:opeq2} and \eqref{eq:iv_quant_reg_op} the integral kernels $k_{ind}
$, $k_q$, and their estimates are twice differentiable with respect to $y$ with bounded derivative and the support of the instrument has finite measure $\mu(\mathrm{supp}\,(Z)) < \infty$. Furthermore, the integral kernels are estimated by an estimator which is strongly consistent for the second derivative. There exists $L>0$ such that
\[
\mu(\mathrm{supp}\,(Z))\sup_{y, z, w} \left| \frac{\partial^2}{\partial y^2} k(y, z, w) \right| < L
\]
with $k = k_{ind}$ or $k=k_q$ respectively.
\end{assumption}
Common nonparametric density estimators are strongly consistent. Assumption \ref{ass:lipschitz_alt} implies for the operators \eqref{eq:nonlininstreg:opeq2} and \eqref{eq:iv_quant_reg_op}
\[
\sup_{y, x, z} \left| \frac{\partial^2}{\partial y^2} f_{YXZ}(y, x, z) \right| < \infty \qquad \mbox{or} \qquad \sup_{y, x, z} \left| \frac{\partial}{\partial y} f_{YXZ}(y, x, z) \right| < \infty
\]
respectively.
\subsubsection{Concentration inequalities}
The estimation error in the operator and its derivative needs to be bounded by exponential inequalities.
\begin{assumption}\label{ass:concentration}
There are constants $c_1, c_2, c_3, c_4 \ge 0$ such that for all $n \in \mathbb{N}$ and all $\tau \ge 0$
\begin{align}\label{eq:concentration_noi}
&\mathbb{P} \left\{ \left|\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_\mathbb{Y} - \mathbb{E}\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_\mathbb{Y} \right| \ge \sqrt{\tau \mathop{\rm {\mathbb V}ar}\nolimits \left(\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_\mathbb{Y}\right)} \right\} \leq c_1 e^{-c_2\tau}\quad \mbox{and}\\
\begin{split}\label{eq:concentration_der}
&\mathbb{P} \Bigg\{\left|\|\widehat T_{n \dag} - T_\dag\|_D^{1+\mu} - \mathbb{E}\left(\|\widehat T_{n \dag} - T_\dag\|_D^{1+\mu}\right) \right| \geq\\
& \hspace{5.3cm}\sqrt{ \tau \mathop{\rm {\mathbb V}ar}\nolimits\left(\|\widehat T_{n \dag} - T_\dag\|_D^{1+\mu}\right)} \Bigg\} \leq c_3 e^{-c_4\tau}.
\end{split}
\end{align}
Where $\|\cdot\|_D$ is the operator norm $\|\cdot\|_{\mathcal{L}(\mathbb{X},\mathbb{Y})}$ or some norm that dominates the operator norm.
\end{assumption}
The following lemma shows that Assumption \ref{ass:concentration} holds for the IV regression applications \eqref{eq:nonlininstreg:opeq2} and \eqref{eq:iv_quant_reg_op} under mild conditions when $\mathbb{Y}$ is a $L^2$ space and $\|\cdot\|_D$ is the Hilbert-Schmidt norm. The Hilbert-Schmidt norm bounds the operator norm from above and is denoted by $\|\cdot\|_{HS}$. For linear integral operators it coincides with the $L^2$ norm of the integral kernel.
\begin{lemma}\label{lem:concentration}
Consider the operators \eqref{eq:nonlininstreg:opeq2} and \eqref{eq:iv_quant_reg_op} as maps into $L^2(U,Z)$ or $L^2(Z)$ respectively. Assume that $f_{YXZ}$ is estimated by a kernel density estimator with a product kernel composed of a one-dimensional kernel $K_Y$ and two multivariate kernels $K_X$ and $K_Z$ corresponding to the dimensions $\dim(X)=d_X$ and $\dim(Z)=d_Z$ with joint bandwidth $h$. Assume for \eqref{eq:nonlininstreg:opeq2} that $n^{-1}h^{-d_z-1} = O(1)$, and $n^{-1-2\mu}h^{-(1+\mu)(d_x+d_z+3)} = O(1)$. Assume for \eqref{eq:iv_quant_reg_op} that $n^{-1}h^{-d_z} = O(1)$, and $n^{-1-2\mu}h^{-(1+\mu)(d_x+d_z+2)} = O(1)$. Then constants $c_2,c_4>0$ exist such that for all $\tau \ge 0$
\begin{align}\label{eq:concentration_op}
\mathbb{P} \left\{ \left| \|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_{L^2} - \mathbb{E}\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_{L^2} \right| \ge \sqrt{\tau \mathop{\rm {\mathbb V}ar}\nolimits \left(\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_{L^2}\right)}
\right\} \leq 2 e^{-c_2\tau}
\end{align}
and
\begin{align}\label{eq:concentration_de}
\mathbb{P} \Bigg\{\left|\|\widehat T_{n \dag} - T_\dag\|_{HS}^{1+\mu} - \mathbb{E}\left(\|\widehat T_{n \dag} - T_\dag\|_{HS}^{1+\mu}\right) \right| \geq \sqrt{ \tau \mathop{\rm {\mathbb V}ar}\nolimits\left(\|\widehat T_{n \dag} - T_\dag\|_{HS}^{1+\mu}\right)} \Bigg\} \leq 2 e^{-c_4\tau}.
\end{align}
\end{lemma}
\subsection{Convergence rates with a priori parameter choice}\label{sec:rates}
Our first convergence rate theorem assumes that $\mu$ in Assumption \ref{ass:source} is known, i.e. the smoothness of the true solution is known. Adaptive estimation will be discussed in the next section.
\begin{theorem}\label{theo:spec_conv}
Let Assumptions \ref{ass:dif}, \ref{ass:source}, \ref{ass:saturation}, \ref{ass:lipschitz}, and \ref{ass:concentration} hold. Define the stopping index as
\[
J:= \mathop{\mathrm{argmin}}\limits_{j\in\mathbb{N}} \left(\alpha_j^\mu + \alpha_j^{-1/2} \mathbb{E}\big[\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_\mathbb{Y}^2\big] \right)
\]
and set
\begin{align}\label{eq:J*}
J^*:=\begin{cases}
J \hspace{1cm} \text{ if } \widehat{\varphi}_j \in B_{2R}(\varphi_0) \text{ for } j=1,\ldots,J\\
0 \hspace{1cm} \text{ else}.
\end{cases}
\end{align}
Then,
\[
\mathbb{E}\left[\|\widehat{\varphi}_{J^*} -\varphi^{\dagger}\|_\mathbb{X}^2\right] = \mathcal{O} \left(\left(\mathbb{E}\big[\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_\mathbb{Y}^2\big]\right)^{\frac{2\mu}{2\mu + 1}} + \mathbb{E}\big[\|\widehat T_{n \dag} - T_\dag\|_{\mathcal{L}(\mathbb{X},\mathbb{Y})}^{2+2\mu}\big] \right).
\]
\end{theorem}
In the special cases of the IV regression examples in $L^2$ spaces, the convergence rate can be given more explicitly.
\begin{corollary}\label{cor:rates_npiv}
Let $\mathbb{X}$ be an $L^2$ space and let Assumptions \ref{ass:dif}, \ref{ass:source}, \ref{ass:saturation}, \ref{ass:lipschitz_alt} and the conditions of Lemma \ref{lem:concentration} hold. Assume that in the case of operator \eqref{eq:nonlininstreg:opeq2} the density $f_{YXZ}$ and that in case of operator \eqref{eq:iv_quant_reg_op} the function $F_{YXZ}$ is $r$ times differentiable and is estimated with a kernel estimator where the kernel is of order at least $r$. If $J^*$ is chosen as in Theorem \ref{theo:spec_conv}, then
\[
\mathbb{E}\big[\|\widehat{\varphi}_{J^*} -\varphi^{\dagger}\|_{L^2}^2\big] = \mathcal{O} \left((n^{-1} h^{-(d_Z+1)})^{\frac{2\mu}{2\mu + 1}} + n^{-1-\mu} h^{-((1+2\mu)(d_X+d_Z+2)+1)} + h^{\frac{4\mu r}{2\mu + 1}} \right).
\]
\end{corollary}
\subsection{Comparison to an alternative quantile regression estimator}
We can compare our result to the rates for nonparametric quantile regression in \cite{HorLee:07}. They use nonlinear Tikhonov regularization for the operator $\mathcal{F}_q$ and proved optimal rates under assumptions which are more restrictive than ours. In contrast to our rates, their rates do not depend on the derivative $\widehat T_{n \dag}$. The main challenge for nonlinear Tikhonov regularization
\[
\widehat{\varphi} = \mathop{\mathrm{argmin}}_\varphi\|\mathcal{F}_q \varphi \|_\mathbb{Y}^2 + \alpha\|\varphi\|_\mathbb{X}^2
\]
is to find the minimizer of the nonlinear functional $\|\mathcal{F}_q \varphi \|_\mathbb{Y}^2 + \alpha\|\varphi\|_\mathbb{X}^2$ which usually has multiple local minima. In \cite{HorLee:07} it is assumed that this minimum is known exactly which is unrealistic in practice. A convergence analysis which takes the performance of a minimization algorithm into account would typically lead to a different rate which also depends on some derivative depending on the particular minimization algorithm. The IRGNM does not have this problem since we only have to solve a linear least squares problem in every Newton step. For a fair comparison of the convergence rates, we will assume that the term $\left(\mathbb{E}\big[\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_\mathbb{Y}^2\big]\right)^{\frac{2\mu}{2\mu + 1}}$ dominates our rate. For sake of simplicity we assume $d_z=1$. Hence, our rate in the case of $\mathbb{X}=L^2$ is
\[
\mathbb{E}\big[\|\widehat{\varphi}_{J^*} -\varphi^{\dagger}\|_{L^2}^2\big] = \mathcal{O} \left((n^{-1} h^{-2} + h^{2r} )^{\frac{2\mu}{2\mu + 1}}\right),
\]
while the rate in \cite{HorLee:07} is $\mathcal{O}\left(n^{-(2\beta-1)/(2\beta+a)}\right)$ with $a$ and $\beta$ defined an their paper as $\alpha$ and $\beta$. \cite{HorLee:07} restricts the bandwidth choice to
\[
h = C_h n^{-\gamma} \quad \text{ with }\quad \frac{2\beta+a -1)}{2r(2\beta+a} <\gamma< \frac{a}{2(2\beta+a)}.
\]
Under their assumptions on $a$ and $\beta$, the density estimator achieves the rate
\[
n^{-1} h^{-2} + h^{2r} = \mathcal{O}\left(n^{-\frac{2\beta +a-1}{2\beta-1}}\right).
\]
Note that this will only coincide with the optimal rate $n^{-\frac{2r}{2r+2}}$ in special cases. Futhermore, in their notation the source condition in our Assumption \ref{ass:source} holds for any $\mu < (\beta-\frac{1}{2})/a$. Hence, $2\mu/(2\mu+1) < (2\beta-1)/(2\beta +a -1)$, where $\mu$ can be chosen such that the left hand side is arbitrarily close to the right hand side. Therefore, under their assumptions our rate is arbitrarily close to
\[
\left(n^{-\frac{2\beta +a-1}{2\beta-1}}\right)^\frac{2\beta-1}{2\beta +a -1} = n^{-\frac{2\beta-1}{2\beta+a}},
\]
which is also the optimal rate in Theorem 2 and 3 in \cite{HorLee:07}.
Our assumptions are less restrictive which leads to faster rates of convergence in many cases compared to \cite{HorLee:07}. Firstly, we have no restrictions on $h$ which means we can chose the optimal bandwidth $h = \mathcal{O}\left(n^{-\frac{1}{2r+2}}\right)$ and achieve
\[
\mathbb{E}\big[\|\widehat{\varphi}_{J^*} -\varphi^{\dagger}\|_{L^2}^2\big] = \mathcal{O} \left(n^{-\frac{2r}{2r+2}\frac{2\mu}{2\mu + 1}}\right)
\]
This has also the advantage that we can choose $h$ by standard data driven bandwidth selectors like cross-validation. It is pointed out in \cite{HorLee:07} that a bandwidth selector for their assumptions does not yet exist.
Secondly, we have no upper bound on $\mu$ while their method requires $\mu \le 1$. Hence, our rate can be significantly better for smooth $\varphi^{\dagger}$. This is not a contradiction to their optimality result, as their result only holds under their more restrictive assumptions.
\subsection{Optimality}
There is no uniform optimality theory for nonlinear inverse problems. Our setup is too general to settle optimality for the rates above.
However, this can be achieved in special cases with further assumptions. We briefly discuss optimality for some special cases in this section. First of all, the rate in \eqref{eq:ese_rate} is known to be optimal in the case of a linear operator $\mathcal{F}$ with separable noise. See for example \cite{Tautenhahn:98}.
For the nonparametric instrumental quantile regression, optimal minimax rates for convergence in probability were derived in \cite{HorLee:07}. If in Corollary \ref{cor:rates_mise} the influence of $\delta^{noi}_n$ and $\sigma^{noi}_n$ dominates $\delta^{der}_n$ and $\sigma^{der}_n$, the same rate is proved for the risk of the IRGNM. This result for the risk is even stronger than convergence in probability. Nevertheless, when $\delta^{der}_n$ and $\sigma^{der}_n$ dominate the convergence, the rate in Corollary \ref{cor:rates_mise} can be slower than the one in \cite{HorLee:07}. As discussed in Section \ref{sec:estimation} the analysis of the nonlinear Tikhonov regularization does not consider the influence of an optimization algorithm that solves the minimization problem in \eqref{eq:nonlin-tyk}. If the properties of such an algorithm were added to the convergence analysis, the derivative of $\mathcal{F}_q$ will likely show up in the convergence rate in a similar way as in Corollary \ref{cor:rates_mise}.
Furthermore, we like to compare our results for the nonparametric instrumental regression with full independence to those with mean independents. When the influence of $\delta^{noi}_n$ and $\sigma^{noi}_n$ dominates $\delta^{der}_n$ and $\sigma^{der}_n$ and a source condition with the same $\mu$ holds for both models, the rates coincide. These rates are known to be optimal for the nonparametric instrumental regression with mean independents as shown in \cite{HalHor:05} and \cite{CheRei:11}. However, the source condition for both models are linked only implicitly. The $\mu$ and thereby the rates can differ in actual examples. The advantage of model \eqref{eq:iv_reg_full_independence} together with the IRGNM is not to improve upon the rates, but to be applicable in cases where model \eqref{eq:iv_reg_mean_independence} fails to identify the solution.
\fi
\subsection{Convergence rates for adaptive estimation }\label{sec:Lepski}
The parameter choice \eqref{eq:J*} in Theorem \ref{theo:spec_conv} and Corollary \ref{cor:rates_npiv} depends on the unknown $\mu$
, which is unfeasible in practice. We present in this section convergence rates for a data driven choice of the stopping parameter $J$ by Lepski{\u\i}'s principle. This is a popular parameter choice rule in the context of statistical inverse problems, see \cite{Tsybakov:00}, \cite{BauHoh:05}, \cite{Mathe:06}, \cite{BauHohMun:09}, and \cite{HW:16}. In the context of nonparametric IV, Lepski{\u\i}'s principle was used for adaptive estimation in \cite{ChenChrist:15}.
The following theorem gives convergence rates of the MISE with a Lepski{\u\i} type parameter choice. We lose a logarithmic factor compared to Theorem \ref{theo:spec_conv}. The constant $C_d$ and $\gamma_{nl}$ used in the theorem are specified in the appendix in formula \eqref{eq:der_noi_est} and in Lemma \ref{lem:nonlin} respectively.
\begin{theorem}\label{theo:lepskii_risk}
Let the assumptions of Theorem \ref{theo:spec_conv} hold.
For all sequences $\delta^{noi}_n$, $\sigma^{noi}_n$, $\delta^{der}_n$, and $\sigma^{der}_n$ such that
\begin{align*}
&\delta^{noi}_n \geq \mathbb{E}(\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_\mathbb{Y}), &&(\sigma^{noi}_n)^2 \geq \mathop{\rm {\mathbb V}ar}\nolimits(\|\widehat{\mathcal{F}}_n(\varphi^{\dagger})\|_\mathbb{Y}),\\
&\delta^{der}_n \geq \mathbb{E}(\|\widehat T_{n \dag} - T_\dag\|_{\mathcal{L}(\mathbb{X},\mathbb{Y})}^{1+\mu}), &&(\sigma^{der}_n)^2 \geq \mathop{\rm {\mathbb V}ar}\nolimits(\|\widehat T_{n \dag} - T_\dag\|_{\mathcal{L}(\mathbb{X},\mathbb{Y})}^{1+\mu})
\end{align*}
set
\[
\widetilde\Phi^{noi}_n(j) := \sqrt{\frac{m}{\alpha_j}} \left(\delta^{noi}_n + \ln((\sigma^{noi}_n)^{-2}) \sigma^{noi}_n \right) + C_d \rho (\delta^{der}_n + \ln((\sigma^{der}_n)^{-2}) \sigma^{der}_n)
\]
and define the Lepski{\u\i} stopping parameter by
\[
J_{Lep}:= \min \left\{j \leq J_{max} \Big| \|\widehat{\varphi}_i - \widehat{\varphi}_j\|_\mathbb{X} \leq 4(1+\gamma_{nl}) \widetilde\Phi^{noi}_n(j) \quad \text{for all } i = 1,\, \ldots,\, J_{max} \right\}
\]
and the stopping parameter by
\[
J^*:=\begin{cases}
J_{Lep} &\text{if } \widehat{\varphi}_j \in B_{2R}(\varphi_0) \text{ for } j=1,\ldots,J_{max}\\
0 \hspace{1cm} &\text{else}.
\end{cases}
\]
Then
\begin{align*}
E\big[\|&\widehat{\varphi}_{J^*} -\varphi^{\dagger}\|_\mathbb{X}^2\big]\\
& = \mathcal{O} \left(\left[(\delta^{noi}_n)^2 + \ln\big((\sigma^{noi}_n)^{-1}\big)(\sigma^{noi}_n)^2 \right]^{\frac{2\mu}{2\mu + 1}} + (\delta^{der}_n)^2 + \ln\big((\sigma^{der}_n)^{-1}\big)(\sigma^{der}_n)^2 \right).
\end{align*}
\end{theorem}
\section{Numerical examples}\label{sec:numerics}
We evaluate the small sample behavior of the estimator based on the IRGNM with simulated data. As a test problem we use a nonparametric IV regression consistent with models \eqref{eq:iv_reg_mean_independence} and \eqref{eq:iv_reg_full_independence}, with univariate covariate $X$ and instrument $Z$. This setup allows to compare our estimator with an estimator which solves \eqref{eq:iv_reg_lin_op} with iterated Tikhonov regularization.
\subsection{Implementation}
The test problem described in the next section is solved on the domain
\[
\mathrm{supp}\,(Y) \times \mathrm{supp}\,(X) \times \mathrm{supp}\,(Z) = [-1/2 , 1/2] \times [0 , 1] \times [0 , 1]
\]
discretized by an equidistant grid with $100 \times 100 \times 100$ nodes. The joint density is estimated on this grid by a standard adaptive density estimator. Trying different density estimators, we found that both method are quite robust with respect to the density estimate. They tolerate some undersmoothing of the density as long as the stopping index $J$ and the regularization parameter $\alpha$ are chosen properly. In the simulations below the same density estimate is used for both the IRGNM and the iterated Tikhonov regularization which allows for a fair comparison of the methods. The initial guess for both methods is the constant function with the value $\mathbb{E}[Y]$, and the penalty functional for both methods is the squared $H^1$ norm.
The Fr\'{e}chet derivative is implemented as in Example \ref{ex:frechet}. The partial derivative of the density and the derivatives for the $H^1$ norm are computed by the central differencing scheme. Operators and norms are evaluated using numerical integration. We tried rectangle rule, trapezoid rule and Simpson's rule but could not find a significant difference in the output of the estimator.
The least squares problems in each step of the iterated Tikhonov regularization and in the inner iteration of the IRGNM are computed by QR decomposition. Note that only one QR decomposition is needed in every Newton step. We tried different numbers of iterations $m$ in the inner iteration of the IRGNM and the iterated Tikhonov regularization for the test example below. No significant difference in the results was observed, which indicates that $\mu$ is not large.
The regularization parameters for the example below are $\alpha_0 = 1$ and $\alpha_{n+1} = 0.9 \alpha_n$. Lepski{\u\i}'s principle is used to find the stopping parameter of the Newton iteration. For the alternative estimator using model \eqref{eq:iv_reg_mean_independence} the regularization parameter $\alpha$ has to be chosen instead, which is done by Lepski{\u\i}'s principle as well. The iterated Tikhonov regularization is computed for a large number of different $\alpha$. Then one of these approximation is chosen by Lepski{\u\i}'s principle.
Hence, both methods are fully data driven.
\subsection{Simulations}
The regressor of the test example is generated by some function $g$ and a random variable $V$ such that $X = g(Z) + V$ and $V \protect\mathpalette{\protect\independenT}{\perp} Z$.
In addition, an exact solution $\varphi^{\dagger}$ and an error term $U$
depending on $V$ but not on $Z$ are chosen. Then $Y$ is defined as $Y := \varphi^{\dagger}(X) + U$.
With this construction both models \eqref{eq:iv_reg_mean_independence} and \eqref{eq:iv_reg_full_independence} identify the true solution. The functions and probability densities that were used for the test example are
\begin{align*}
\varphi^{\dagger}(x) &= \frac{1}{6}\sin(2\pi(x+0,25)),\\[6pt]
f_Z(z) &= \frac{9}{7}\sqrt{z} + \frac{1}{7} \qquad \text{on the interval }[0,\, 1],\\[6pt]
g(z) &= 0,8z+0,1,\\[6pt]
f_V(v) &= \frac{1}{0,08\sqrt{2\pi}} \exp\left(-\frac{1}{2}\left(\frac{v}{0,08}\right)^2\right), \mbox{ and}\\[6pt]
f_{U}(y,v) &= \frac{1}{0,07\sqrt{2\pi}} \exp\left(-\frac{1}{2}\left(\frac{y-2v}{0,07}\right)^2\right).
\end{align*}
The densities of $V$ and $U$
are constructed with Gaussians in a way that the expectation of $U$
depends on $v$.
Figure \ref{fig:reg_without_W1d} shows the exact solution (blue) compared to the solution a nonparametric regression without instrumental variables would yield asymptotically (green).
\begin{center}
\includegraphics[width=0.5\textwidth]{bsp3reg2.jpg}
\captionof{figure}{\label{fig:reg_without_W1d}Necessity of the instrument: A standard nonparametric regression would asymptotically yield the green curve which is considerably different from the true curve $\varphi^\dag$ (blue).}
\end{center}
Both methods were tested on samples of $500$ and $1000$ observations. For each of the two sample sizes 1000 samples were generated and the joint density $f_{YXZ}$ was estimated by a kernel method.
Figures \ref{fig:hist500} and \ref{fig:hist1000} show histograms for the $L^2$ error of the reconstructions for both methods and different sample sizes. The values are normed by the initial error, so that on this scale the initial error becomes $1$.
\begin{center}
\includegraphics[width=0.47\textwidth]{nonlin500_1lepskii_scaled.jpg}\qquad\includegraphics[width=0.47\textwidth]{lin500_1lepskii_scaled.jpg}
\captionof{figure}{\label{fig:hist500}$L^2$ error for the sample size $n=500$. Left panel: IRGNM with the assumption $U \protect\mathpalette{\protect\independenT}{\perp} Z$. Right panel: iterated Tikhonov regularization with the assumption $\mathbb{E}[U|Z] = 0$}
\end{center}
In Figure \ref{fig:hist500} we compare the errors of both methods for the sample size $n=500$. Both methods produce acceptable results. The variance as well as the number of outliers observed for the method with independent instrument are significantly smaller than the variance or number of outliers of the method with the conditional mean assumption. The latter method produces a considerable number of outliers with the same or even larger errors than the initial guess. This cannot be observed for the IRGNM. In addition, the mean error of the IRGNM is smaller.
\begin{center}
\includegraphics[width=0.47\textwidth]{nonlin1000_2lepskii_scaled.jpg}\qquad\includegraphics[width=0.47\textwidth]{lin1000_2lepskii.jpg}
\captionof{figure}{\label{fig:hist1000}$L^2$ error for the sample size $n=1000$. Left panel: IRGNM with the assumption $U \protect\mathpalette{\protect\independenT}{\perp} Z$. Right panel: iterated Tikhonov regularization with the assumption $\mathbb{E}[U|Z] = 0$}
\end{center}
Similar histograms for sample size $n=1000$ are displayed in Figures \ref{fig:hist1000}. Both methods perform well. The advantages of the IRGNM with less outliers, smaller variance and smaller mean error can be observed again. The following table provides the mean and some quantiles of the errors normed by the initial error.
\begin{small}
\begin{center}
\begin{tabular}{|l||l|ll|l|l|l|}
\hline
sample size and method & mean & quantiles\hspace{-7pt} & $q = 0.25$ & $q = 0.5$ & $q = 0.75$ & $q = 0.9$\\ \hline
$n = 500$, $\;\,U \protect\mathpalette{\protect\independenT}{\perp} W$ & 0.2535 & & 0.2012 & 0.2398 & 0.2940 & 0.3495\\ \hline
$n = 500$, $\;\,\mathbb{E}[U|W] = 0$ & 0.4042 & & 0.2738 & 0.3437 & 0.4475 & 0.6407\\ \hline
$n = 1000$, $U \protect\mathpalette{\protect\independenT}{\perp} W$ & 0.2152 & & 0.1780 & 0.2064 & 0.2439 & 0.2868\\ \hline
$n = 1000$, $\mathbb{E}[U|W] = 0$ & 0.3067 & & 0.2339 & 0.2846 & 0.3482 & 0.4325\\ \hline
\end{tabular}
\end{center}
\end{small}
We close this section with examples of median reconstructions for both sample sizes displayed in Figure \ref{fig:example}. They illustrate the advantage of the regression model with independent instruments solved with the IRGNM.
\begin{center}
\includegraphics[width=0.47\textwidth]{ex500.png}\qquad\includegraphics[width=0.47\textwidth]{ex1000.png}
\captionof{figure}{\label{fig:example}Examples for reconstructions with sample size $n=500$ (left) and $n=1000$ (right). The blue line shows the exact solution, the red curve the reconstruction with the conditional mean assumption and the green curve the reconstruction with independent instrument.}
\end{center}
These results suggest that both methods give consistent estimators for the nonparametric instrumental regression with clear advantages for the regression model with independent instruments \eqref{eq:iv_reg_full_independence} and the IRGNM.
\section*{Acknowledgment}
The author would like to thank Thorsten Hohage and Johannes Schmidt-Hieber for interesting and fruitful discussions on this topic. He also would like to thank two anonymous referees for valuable comments that improved the paper.
\bibliography{lit}
\bibliographystyle{apalike}