Debiased Ill-Posed Regression

Debiased Ill-Posed Regression

Introduction

In many applications in statistics, particularly in causal inference, given a random sample drawn from the unknown distribution of a random vector $V,$ we are interested in estimating a function $h$ satisfying a conditional moment restriction of the form

where $V_{h},V_{q}\subseteq V$, and $g_{0}$ and $g_{1}$ are known functions. If $V_{h}=V_{q}$, Equation (ref) can be reformulated as a standard regression problem. Yet, we consider the more challenging case where $V_{h}\neq V_{q}$. In this case, Equation (ref) is an ill-posed integral equation kress2013linear. We refer to the task of estimating a function $h$ that solves this equation as ill-posed regression. Two of the most prominent applications of ill-posed regression in the filed of causal inference are found in instrumental variable (IV) and in proximal causal inference methodologies. In both cases, the goal is to perform causal inference in settings where unobserved confounders in the treatment-outcome relationship cannot be ruled out.

IV methods require the existence of an auxiliary variable, called an instrumental variable, which only affects the outcome variable through the treatment variable and does not share an unobserved confounder with the outcome variable. A particularly popular IV-based model is the nonparametric IV (NPIV) model newey2003instrumental, hall2005nonparametric, darolles2011nonparametric, florens2011identification,chen2012estimation. This model formalizes the aforementioned IV conditions as follows. Let $W\in \mathcal{W}$ and $Y\in \mathcal{Y}$ be the treatment and the outcome variables, respectively. The NPIV model assumes that the outcome variable $Y$ is generated from a structural function of the treatment variable, $h_{0}(W)$, with additive noise. That is, the outcome is generated as \[ Y=h_{0}(W)+\epsilon , \] where $\epsilon$ is a mean-zero error term. Interest is in estimation of the unknown function $h_{0}$ based on random sample from the target population, when $W$ is not assumed to be exogenous, and hence $\mathbb{E}[\epsilon \mid W]$ is not assumed to be equal to zero. Therefore, $\mathbb{E}[Y\mid W=\cdot ]$ is not necessarily equal to the structural function $h_{0}\left( \cdot \right) $. Instead, in the NPIV model, we assume existence of an instrumental variable $Z\in \mathcal{Z}$, for which we have $\mathbb{E}[\epsilon \mid Z]=0$. That is, $h_{0}$ satisfies the conditional moment restriction

Estimation of $h_{0}$ is a special case of the introduced ill-posed regression with $V_{q}=\{Z\}$, $V_{h}=\{W\}$, $g_{1}(V)=1$, and $g_{0}(V)=Y$. In words, this conditional moment restriction requires that after projecting onto the space of functions of $Z$, random variables $Y$ and $h_{0}(W)$ have the same projection. This requirement implies that the IV affects the outcome variable only through the treatment variable.

Proximal causal inference methods miao2018identifying, tchetgen2020introduction, cui2023semiparametric allow for the presence of unobserved confounders but require access to two proxy variables of the confounder, which satisfy certain statistical conditions. Consider, for example, the goal of estimating the mean of an outcome of interest in the counterfactual world in which all subjects in the population receive the same treatment, based on a random sample of $\left( A,X,Y,Z,W\right) $, where $A=1$ if the subject received the treatment of interest and $A=0$ otherwise, $X$ is a vector of observed baseline covariates, $Z$ and $W$ are the treatment and outcome proxy variables (defined in Section (ref)), and $Y$ is the outcome variable. Under the proximal causal inference assumptions (detailed in Section (ref)), the counterfactual mean under treatment $A=a$, for $a\in\{0,1\}$, is equal to $\mathbb{E}\left[ h_{0}\left( W,X\right) \right] $ where $h_{0}$ solves the conditional moment equation \[ \mathbb{E}[I(A=a)h(W,X)-I(A=a)Y\mid Z,A,X]=0. \] Estimation of $h_{0}$ is another special case of ill-posed regression with $V_{q}=\{Z,A,X\}$, $V_{h}=\{W,X\}$, $g_{1}(V)=I(A=a)$, and $g_{0}(V)=I(A=a)Y$. Under slightly different assumptions, cui2023semiparametric, showed that the counterfactual mean is equal to the mean of $YI(A=a)q_{0}\left(Z,X\right) $, where $q_{0}(Z,X)$ solves the conditional moment equations \[ \mathbb{E}\left[I(A=a)q(Z,A,X)-1\mid W,X\right]=0, \] which is another special case of ill-posed regression with $V_{q}=\{W,X\}$, $V_{h}=\{Z,A,X\}$, $g_{1}(V)=I(A=a)$, and $g_{0}(V)=1$.

The preceding examples illustrate the important role of ill-posed regression in causal inference, and motivates the development of solutions for this task. For a function $h$, define $\mathbb{E}[\mathbb{E}[g_{1}(V)h(V_{h})-g_{0}(V)\mid V_{q}]^{2}]$ as the projected error of $h$. Because the solution $h_{0}$ to Equation (ref) minimizes the projected error over all functions $h$, a common approach in the literature for ill-posed regression is to estimate $h_{0}$ with a minimizer of an estimate of (a regularized version of) the projected error. Therefore, the more precise the estimator of the projected error is, the better the quality of the estimated structural function, and hence, improving the estimator should be considered. Formalizing this idea is one of the main goals in this work.

In this paper, we propose a novel strategy for ill-posed regression which is based on minimizing a debiased estimator of the projection error. We perform the debiasing step using techniques from the modern semiparametric theory, specifically, by leveraging the influence function of the projected error. The technique of influence function-based debiasing has recently gained significant attention in the literature of causal inference and missing data laan2003unified, van2011targeted, chernozhukov2018double, starting from the seminal work of robins1994estimation. For the instrumental variable setting, compared to existing work, our proposed method requires the estimation of an extra nuisance function, which is used for debiasing the original estimator of the projected error. We demonstrate that the proposed estimator will have a second-order bias with respect to the nuisance functions and possesses a desirable robustness property with respect to misspecification of one of its nuisance functions. As the first main result of this work, we provide finite-sample bounds on the convergence rate of our proposed estimator. The proposed estimator involves a hyper-parameter, for which the optimal value depends on certain functionals of the underlying unknown data generating process. Hence, we further propose a hyper-parameter tuning approach based on cross-validation. While there has been work in the literature studying whether or not hyper-parameter tuning has a detrimental effect on the convergence rate in standard non-parametric regression settings (e.g., vaart2006oracle, van2003unified), to the best of our knowledge, the impact of hyper-parameter tuning on ill-posed regression is not well-understood. As our second main result, we derive finite-sample convergence bounds on the error of the cross-validated estimator. Our bounds point to the potential rate loss due to not knowing the optimal value of the hyper-parameter, and emphasize the importance of using the debiasing strategy in this setting.

In a recent work, ghassami2022minimax considered a class of pathwise differentiable parameters $\psi _{0}$ with influence function of the form \[ IF_{\psi_{0}}(V)=s_{1}(V)q_{0}(V_{q})h_{0}(V_{h})+s_{2}(V)q_{0}(V_{q})+s_{3}(V)h_{0}(V_{h})+s_{4}(V)-\psi _{0}, \] where $s_{1}$, $s_{2}$, $s_{3}$, and $s_{4}$ are known functions, i.e., they do not depend on the data generating law, and $h_{0}$ and $q_{0}$ are unknown functions. This is a generalization of the Robins' parameter class robins2008higher to the case where $V_{h}\neq V_{q}$. It is shown in ghassami2022minimax that the unknown functions $h_{0}$ and $q_{0}$ solve equations of the form (ref) for suitable choices of $g_{0}$ and $g_{1}.$ As will be explained in Section (ref) and discussed in ghassami2022minimax, the aforementioned counterfactual mean under the proximal causal inference assumptions is a parameter with influence function of the form as in the last display. The authors of ghassami2022minimax,kallus2021causal proposed an adversarial learning approach for estimating $h_{0}$ and $q_{0}$, and studied the asymptotic properties of the debiased (influence function-based) estimator for $\psi _{0}$. Here, we apply our ill-posed regression strategy to the setting of ghassami2022minimax, and extend the debiasing point of view by replacing the adversarial approach to the estimation of the nuisance functions $h_{0}$ and $q_{0}$ with estimators that minimize a debiased (influence function-based) estimator of the projected error. Hence, we propose an estimation strategy with two layers of debiasing. We demonstrate the gained benefit from this extra added debiasing layer in terms of the conditions needed for obtaining a consistent and asymptotically normal estimator for $\psi _{0}$. Importantly, we show that without using the debiasing strategy, achieving asymptotic normality may not be feasible.

The rest of the paper is organized as follows. In Section (ref), we introduce the model that we study in this work. We present our debiased ill-posed regression approach in Section (ref), and analyze its bias structure. We present the finite-sample convergence rate of our proposed estimator in Section (ref). We describe the hyper-parameter tuning approach and its convergence analysis in Section (ref). We study the application of our proposed ill-posed regression approach in the parameter class of ghassami2022minimax and illustrate it with the estimation of the counterfactual mean under the proximal causal inference framework in Section (ref). Our concluding remarks are presented in Section (ref).

The Integral Equation

Given a law $P$ on the sample space of $V,$ we let $P_{V_{h}}$ and $P_{V_{q}}$ be the corresponding marginal laws on the sample spaces of $V_{h}\subset V$ and $V_{q}\subset V$, respectively. We let $L^{2}(P_{V_{h}})$ (resp. $L^{2}(P_{V_{q}})$) be the set of measurable functions on the range of $V_{h}$ (resp. $V_{q}$) with finite second moment under $P_{V_{h}}$ (resp. $P_{V_{q}}$). We denote the law of $V$ by $P_{0}$. Let $T:L^{2}(P_{0,V_{h}})\rightarrow L^{2}(P_{0,V_{q}})$ be the linear, bounded, operator $(Th)(V_{q})=\mathbb{E}[g_{1}(V)h(V_{h})\mid V_{q}]$, where $g_{1}$ is a known function and throughout, $\mathbb{E}$ stands for expectation under $P_{0}$. Moreover, let $r_{0}\in L^{2}(P_{0,V_{q}})$ be defined as $r_{0}(V_{q})=\mathbb{E}[g_{0}(V)\mid V_{q}]$, where $g_{0}$ is a known function. The conditional moment restriction in Equation (ref) can be written using the operator notation as

Throughout we assume that a solution to this equation exists, i.e., we make the following assumption.

When the operator $T$ is compact (see, e.g., kress2013linear), a situation that can only arise when $V_{h}$ and $V_{q}$ do not share common components, a sufficient condition for $r_{0}\in \mathcal{R}(T)$ is provided by the celebrated Picard's theorem, which we review next. To proceed, we remind the reader of an important property of compact linear operators. For any compact linear operator $T:\mathcal{E}\rightarrow \mathcal{E}'$ between Hilbert spaces $\mathcal{E}$ and $\mathcal{E}'$, let $(\sigma _{i})_{i=1}^{\infty }$ denote the sequence of, decreasing to 0, square roots of the eigenvalues of the self-adjoint compact operator $T^{\ast }T,$ where $T^{\ast }$ denotes the adjoint of $T$. Then, there exist orthonormal sequences $(\varphi _{i})_{i=1}^{\infty }$ and $(\psi_{i})_{i=1}^{\infty }$ in $\mathcal{E}$ and $\mathcal{E}'$, respectively, such that for all $i$: \[ T\varphi _{i}=\sigma _{i}\psi _{i},~~~~~T^{\ast }\psi _{i}=\sigma _{i}\varphi _{i}. \] The sequence $(\sigma _{i},\varphi _{i},\psi _{i})_{i=1}^{\infty }$ is called a singular system of $T$.

See kress2013linear for a proof.

Throughout, we do not assume that the operator $T$ is injective, i.e., we do not assume that Equation (ref) has a unique solution. Rather, we focus our interest on the estimation of the solution $h_{0}$ with minimum norm, which we refer to as the $L^{2}$-minimal solution, i.e., satisfying \[ h_{0}=\operatorname*{arg\,min}_{h\in L^{2}(P_{0,V_{h}}):Th=r_{0}}\Vert h\Vert_{2}, \] where in the display above and throughout the rest of the exposition, $\Vert\cdot \Vert _{2}$ is the $L^{2}\left( P_{0}\right) $ norm. For instance, in the NPIV model, this is tantamount to assuming that the underlying structural function is $L^{2}$-minimal. The minimum norm solution always exists and it is unique. It is equal to the residual from the $L^{2}(P_{0,V_h})$-projection of any solution $h$ of (ref) onto the null space $\mathcal{N}(T)$ of the operator $T.$ In fact, when $T$ is compact, the solution will be that in Equation (ref).

A well-known challenge in solving integral equations of the form (ref) is that in many settings of interest, e.g., when $T$ is compact, the $L^{2}$-minimal solution $h_{0}$ is not a continuous function of $r_{0}$, which renders the problem ill-posed. Specifically, it follows from Equation (ref) that a small deviation $\delta \psi _{i}$ from $r_{0}$ can be significantly amplified, as it corresponds to the solution $h_{0}+\left(\delta /\sigma _{i}\right) \varphi _{i}$, and the absolute value of $\delta/\sigma _{i}$ tends to infinity as $i$ increases because $\sigma _{i}$ tends to 0. This lack of continuity implies that consistent estimation of $h_{0}$ is not feasible without employing some regularization technique. Several different types of regularization techniques exist and are well-studied in the literature. These include spectral cutoff, Tikhonov, iterative Tikhonov and Landweber. We refer the readers to engl1996regularization for a detailed discussion of regularization for inverse problems. In this work, we use iterative Tikhonov regularization, described in Section (ref).

When employing regularization techniques, regularity conditions are required for attaining a convergence rate on the estimation error. In standard non-parametric regression problems, regularity conditions in the form of assumptions on smoothness or complexity are imposed on the non-parametric regression function of interest. In contrast, in inverse problems, regularity conditions are required for both the $L^2$-minimal solution $h_{0}$, as well as the operator $T$. One often invoked regularity condition in the literature is the so-called $\beta$-source condition tautenhahn1996error, carrasco2007linear, darolles2011nonparametric, bennett2023source, which relates the smoothness of the target parameter to the smoothing effect of the operator.

For compact operators, the $\beta$-source condition requires that if the operator is highly smoothing, the target parameter must be sufficiently smooth to ensure that the summation $\sum_{i=1}^{\infty }\langle h_{0},\varphi _{i}\rangle ^{2}/\sigma _{i}^{\beta }$ is finite. The $\beta$-source condition can be leveraged to control the regularization bias as demonstrated in Theorems (ref) and (ref) below.

Debiased Ill-Posed Regression

For a function $h:\mathcal{V}_h\rightarrow\mathbb{R}$, define its (root mean squared) projected error as

Moreover, define the (root mean squared) source error of $h$ as \[ {\mathbb{E}\left[\left\{h(V_h)-h_0(V_h)\right\}^2\right]}^{1/2}=\|h-h_0\|_2. \] Note that the conditional moment restriction in Equation (ref) (and (ref)) can be equivalently written as $\|T(h-h_0)\|_2=0$.

Based on Proposition (ref), we have \[ h_0 \in\operatorname*{arg\,min}_{h\in L^2(P_{0,V_{h}})}\psi(h), \] where, for any function $h\in L^2(P_{0,V_{h}})$, we define \[ \psi(h):=\mathbb{E}\left[\left\{\mathbb{E}\left[g_1(V)h(V_h)\mid V_q\right]-g_0(V)\right\}^2\right]= \mathbb{E}\left[\left\{(Th)(V_q)-g_0(V)\right\}^2\right]=\|Th-g_0\|_2^2. \] This observation as well as the minimality assumption suggests the following estimator for $h_0$: \[ \hat h_\lambda^\text{B}=\operatorname*{arg\,min}_{h\in\mathcal{H}}\mathbb{E}_n\left[\left\{(\hat Th)(V_q)-g_0(V)\right\}^2\right]+\lambda\mathbb{E}_n\left[h^2(V_h)\right], \] where $\hat T$ is an estimator of the operator $T$, $\mathbb{E}_n$ is the empirical mean operator, $\mathcal{H}\subset L^2(P_{0,V_{h}})$ is a user specified function class, and $\lambda$ is the regularization parameter that can depend on the sample size $n$. $\hat h_\lambda^\text{B}$ is referred to as the Tikhonov regularized estimator. The regularization term ensures converging towards the $L^2$-minimal solution, yet introduces regularization bias, which depends on the value of the hyper parameter $\lambda$. This Tikhonov regularized estimation is in fact the approach taken by hartford2017deep and later by li2024regularized to estimate the structural equation in the NPIV framework. Finite sample analysis of the estimator $\hat h_\lambda^\text{B}$ for the NPIV setting is provided in li2024regularized.

We notice that in the approach above, first, the nuisance parameter $T$ is estimated with $\hat T$, then $\hat T$ is used for estimating $\psi(h)$, which in turn is used for estimating $h_0$. Hence, a concern with this estimation approach is that it can be sensitive to the misspecification and/or slow convergence rate of the estimator $\hat T$ of $T$. Therefore, it is desired to reduce the sensitivity of the estimator of $\psi(h)$ to that of $T$. To this end, we propose to augment the objective function with a debiasing term. This can be achieved by leveraging the influence function of $\psi(h)$ which we derive below.

Based on Theorem (ref), for a given function $h\in L^2(P_{0,V_{h}})$, we propose the following estimator for $\psi(h)$:

where $\hat T\in\mathcal{T}$ and $\hat r\in\mathcal{R}$ are estimators for $T$ and $r_0$, respectively, which are obtained from an independent sample.

We have the following result regarding the structure of the bias of $\hat\psi(h)$.

Proposition (ref) demonstrates that the bias of the estimator $\hat\psi(h)$ is of second order. It also demonstrates that the estimator is robust to the misspecification of the nuisance function $r_0$, in the sense that it is (asymptotically) unbiased as long as the nuisance parameter $T$ is (asymptotically) correctly specified, even if the nuisance parameter $r_0$ is misspecified.

Equipped with the influence function-based estimator $\hat\psi(h)$, we propose to estimate the function $h_0$ using the following iterative Tikhanov regularized estimator.

In our analyses in the following sections, we use $t=2$ and $\hat h^{\text{IF}}_{\lambda,0}=0$. Therefore, our estimator will be $\hat h^{\text{IF}}_{\lambda,2}$. However, any initial function or number of iterations $t$ can be used. The drawback of using a large $t$ is that it leads to an exponentially large constant in our expression for convergence rate upper bound (see the proof of Theorem (ref)); hence we avoid such choices. The discussion of selecting the hyper-parameter $\lambda$ is postponed to Subsection (ref).

We require the following realizability and boundedness conditions.

Convergence Rate

In this subsection, we present our results regarding the convergence rate of the projected error and the source error of the estimator $\hat h^{\text{IF}}_{\lambda,2}$. We use the statistical learning-theoretic local complexity measures of localized Rademacher complexity and the related concept of critical radius for characterizing the complexity of the function space and convergence rate of excess risk wainwright2019high. Before presenting our result, we briefly review these notions.

wainwright2019high also provided the empirical counterparts of the localized Rademacher complexity and critical radius, which can be used to estimate the corresponding true values.

In the sequel, the notation $X_1\lesssim X_2$ means that there exists a constant $c$ such that $X_1\le c X_2$ ($X_1\gtrsim X_2$ is defined similarly), and the notation $X_1\asymp X_2$ means that there exist constants $c_1$ and $c_2$ such that $c_1X_1\le X_2\le c_2X_1$. Moreover, for a function class $\mathcal{F}$, we define $\mathcal{F}-\mathcal{F}:=\{f_1-f_2:f_1,f_2\in\mathcal{F}\}$, and denote the star hull of $\mathcal{F}$ by $star(\mathcal{F})$ which is defined as $\{\alpha f: f\in \mathcal{F}, \alpha\in[0,1]\}$.

We are now ready to present our results regarding the convergence rate of the projected and source errors of estimator $\hat h^{\text{IF}}_{\lambda,2}$. Our first result provides a bound on the projected error. We will use the following function classes in this result: $\mathcal{G}_1:=\{\hat Th:h\in\mathcal{H}\}$, and $\mathcal{G}_2:=\{\tilde Th_0:\tilde T\in\mathcal{T}\}$. Our first result does not utilize the $\beta$-source condition and only focuses on projected error.

Next, we show that by additionally requiring Assumption (ref) (the $\beta$-source condition), we can obtain an alternative bounds on the projected error, and we can also bound the source error. We will use the function $h_{\lambda,t}^*$ defined below in our result.

with $h^*_{\lambda,0}=0$. We present two versions of this result in Theorems (ref) and (ref) below. While one can simply take the minimum of the bounds in these theorems, we state them separately to maintain clarity in the exposition.

Several remarks are in order.

Therefore, in the worst case, we are back to the result of li2024regularized.

Hyper-Parameter Selection

Although Theorems (ref) and (ref) provide bounds on the source error, the value of the hyper parameter $\lambda^*$ in those theorems depends on the parameter $\beta$, an upper bound on the critical radii of certain function classes, and convergence rates of $\hat T$ and $\hat r$. Most of these quantities are usually unknown. In this subsection, we address hyper-parameter selection for the debiased ill-posed regression and demonstrate the resulting potential rate loss due to not knowing $\lambda^*$. Our results can be considered as generalization of existing results in the literature on rate analysis for hyper-parameter tuning (e.g., vaart2006oracle, van2003unified) to the setting of ill-posed regression.

Let $\mathcal{C}:=\{\hat h_1,...,\hat h_M\}$ be our candidate set containing $M$ candidate estimators for the function $h_0$. For any given function $h$, recall the definition

Suppose candidates in $\mathcal{C}$ and nuisance components $\hat T$ and $\hat r$ are learned on separate independent data folds. We propose to select an estimator candidate $\hat h$ on an independent validation dataset (the empirical expectation above is with respect to the validation data) as follows:

Compared to the estimator in (ref), here we do not have the regularization term and the optimization is over $\mathcal{C}$ as opposed to $\mathcal{H}$.

Our goal is to analyze the convergence rate of the source error and the projected error of $\hat h$ in (ref). We start with the analysis of the projected error of $\hat h$.

In the context of hyper-parameter selection for the estimator in (ref), suppose we have $M$ candidates $\lambda_1<\lambda_2<\cdots<\lambda_M$ for the hyper-parameter in the optimization problem (ref). Let $\hat h_i$ denote the estimator corresponding to $\lambda_i$, for $i\in\{1,...,M\}$; let $\mathcal{C}:=\{\hat h_1,...,\hat h_M\}$ be the candidate set. We again consider candidates in $\mathcal{C}$ and nuisance components $\hat T$ and $\hat r$ are learned on separate independent data folds, and we select an estimator candidate on an independent validation dataset based on the following approach. Let $(b_n)_{n=1}^\infty$ and $(B_n)_{n=1}^\infty$ be positive decreasing sequences converging to zero, approximating $\delta_n^{1-\varepsilon}$ and $\delta_n^{1/3}$, respectively, where $\delta_n$ is defined in Theorem (ref), and $0<\varepsilon\ll1$. We choose $M=n$, and \[ \lambda_i=b_n+\frac{i}{n}B_n ,\qquad i\in\{1,...,n\}, \] where $n$ is the sample size for both training and validation datasets. We assume that the debiasing succeed in the sense that $\delta^2_n=\Delta_n$ and $\delta^2_{M,n}=\Delta_{M,n}$, where $\delta_n$, $\delta_{M,n}$, $\Delta_n$, and $\Delta_{M,n}$ were defined in the statement of Theorems (ref) and (ref). We also consider $\Delta_{M,n}\lesssim\delta_n$. In this case, our choices for $(b_n)_{n=1}^\infty$ and $(B_n)_{n=1}^\infty$ guarantee that our upper bounds in Theorems (ref) and (ref) converge to zero. We only utilize knowledge of $\delta_n$. However, if one has knowledge of other components involved in $\Delta_n$ and $\Delta_{M,n}$, it can be incorporated to relax the aforementioned assumption.

We have the following corollary of Theorems (ref)-(ref).

Several remarks are in order.

For analyzing the source error, we require the following additional assumption.

Assumption (ref) is reminiscent of those used in analyses based on Hilbert scales (e.g., chen2011rate, florens2011identification). Roughly speaking, Assumption (ref) aims to characterize the smoothness and the convergence (if the projected error converges) of the error functions with one constant $\alpha$, hence the name $\alpha$-error condition. The assumption requires a certain level of quality on the candidates in $\mathcal{C}$ (not on all functions in the function class $\mathcal{H}$) and is essentially that the smoothness and convergence is so that the same value of $\alpha$ works for all sample sizes. In general, one can find a value $\alpha$ only if the sequence $(\mu_i^{(n)})_{i,n=1}^\infty$ has an overall decreasing trend and converges to zero (see Remark (ref) for a special case where $(\mu_i^{(n)})_{i,n=1}^\infty$ need not converge to zero in $n$). In such general cases, if convergence of error function is too slow (captured by $\mu_i^{(n)}\rightarrow 0$ too slowly as $n\rightarrow\infty$), or if the smoothness is not sufficient (captured by $\mu_i^{(n)}\rightarrow 0$ too slowly as $i\rightarrow\infty$), then we will not be able to find a finite constant $\alpha$.

In the following, we present our result regarding the convergence rate of the source error of the estimator $\hat h$ in (ref).

Linear Functionals of Ill-Posed Regression Functions

In a recent work, ghassami2022minimax considered a parameter class of regular parameters $\psi_0$ with influence function of the form \[ IF_{\psi_0}(V)=s_1(V)q_0(V_q)h_0(V_h)+s_2(V)q_0(V_q)+s_3(V)h_0(V_h)+s_4(V)-\psi_0, \] where $s_1$, $s_2$, $s_3$, and $s_4$ are known functions. This is a generalization of the Robins' parameter class robins2008higher to the case where $V_h\neq V_q$. ghassami2022minimax showed that the nuisance functions $h_0$ and $q_0$ are solutions to conditional moment restrictions of the form (ref). Specifically, $h_0$ solves \[ \mathbb{E}\left[s_1(V)h_0(V_h)+s_2(V)\mid V_q\right]=0, \] and $q_0$ solves \[ \mathbb{E}\left[s_1(V)q_0(V_q)+s_3(V)\mid V_h\right]=0. \] The authors used an adversarial learning approach to design estimators $\hat h$ and $\hat q$ for $h_0$ and $q_0$, respectively. These estimators are in-turn used in a debiased (influence function-based) estimator of $\psi_0$ of the form

Here, we apply our ill-posed regression strategy to estimate $h_0$ and $q_0$, and extend the debiasing point of view by also using a debiased (influence function-based) estimator for the nuisance functions. Hence, we propose an estimation strategy for $\psi_0$ with two layers of debiasing.

It can be shown that the estimator $\hat\psi$ has the so-called mixed-bias property rotnitzky2021characterization, which implies that the bias in estimating $\psi_0$ is connected to a product of the biases in estimating $h_0$ and $q_0$. Formally, the mixed-bias property states that the bias of the estimator $\hat\psi$ satisfies \[ \psi_0-\mathbb{E}[\hat\psi] =\mathbb{E}[s_1(V)(\hat q(V_q)-q_0(V_q))(\hat h(V_h)-h_0(V_h))]. \] This mixed-bias property results in the estimator to be doubly robust, meaning that it is (asymptotically) unbiased as long as either $\hat h$ or $\hat q$ is (asymptotically) correctly specified. ghassami2022minimax used the mixed-bias property to study the convergence rate of $\hat\psi$ and showed that, under certain mild regularity conditions and consistency of the estimators $\hat h$ and $\hat q$, the estimator $\hat\psi$ is $\sqrt{n}$-consistent and asymptotically normal if

That is, the rate requirement is on the product of the projected error of one of the nuisance functions and the source error of the other nuisance function.

Let $T_h:L^2(P_{0,V_h})\rightarrow L^2(P_{0,V_q})$ be the linear, bounded, operator $(T_hh)(V_q)=\mathbb{E}[s_1(V)h(V_h)\mid V_q]$, and let $r_h\in L^2(P_{0,V_q})$ be defined as $r_h(V_q)=\mathbb{E}[s_2(V)\mid V_q]$. Moreover, let $T_q:L^2(P_{0,V_q})\rightarrow L^2(P_{0,V_h})$ be the linear, bounded, operator $(T_qq)(V_h)=\mathbb{E}[s_1(V)q(V_q)\mid V_h]$, and let $r_q\in L^2(P_{0,V_h})$ be defined as $r_q(V_h)=\mathbb{E}[s_3(V)\mid V_h]$. We have the following corollary regarding the root-$n$ consistency of the estimator $\hat\psi$.

Application to Proximal Causal Inference

Proximal causal inference framework is a recently proposed framework for assessing the causal effect of a treatment variable on an outcome variable of interest miao2018identifying, tchetgen2020introduction. This framework allows for the presence of an unobserved confounder $U$ in the system, yet, requires having access to two proxy variables of $U$. Let $A\in\{0,1\}$ be a binary treatment variable, $Y\in\mathcal{Y}$ be the outcome variable of interest, and $X\in\mathcal{X}$ be the observed covariates. We use the potential outcomes framework neyman1923application,rubin1974estimating and denote the potential outcome variable of $Y$, had the treatment variables been set to value $A=a$ (possibly contrary to the fact) by $Y^{(a)}$, where $a\in\{0,1\}$. We assume the standard consistency and positivity assumptions which require that $Y^{(A)}=Y$, and $0<p(A=1\mid X,U)<1$, respectively. Our causal parameter is the average treatment effect (ATE), defined as $\mathbb{E}[Y^{(1)}-Y^{(0)}]$.

The most common approach for identification of ATE is to assume conditional exchangebility hernan2020causal, which requires independence of $Y^{(a)}$ from $A$, for $a\in\{0,1\}$ conditioned on $X$, denoted as $Y^{(a)}\perp\mkern-9.5mu\perp A\mid X$. Proximal causal inference framework relaxes this assumption to a version called latent conditional exchangeability.

Instead, this framework requires having access to two proxy variables $Z$ and $W$ of the unobserved confounder, which satisfy the following assumption.

Variables $Z$ and $W$ satisfying Assumption (ref) are called the treatment proxy variable and the outcome proxy variable, respectively.

We focus on the potential outcome mean $\mathbb{E}[Y^{(a)}]$, where $a\in\{0,1\}$. We require the following conditions on the data generating distribution.

We denote the $L^2$-minimal solutions to integral equations in Parts $(a)$ and $(c)$ of Assumption (ref) by $h_0$ and $q_0$, respectively.

miao2018identifying showed that under Assumptions (ref), (ref), (ref)$(a)$, and (ref)$(b)$, the potential outcome mean is identified as $\mathbb{E}[Y^{(a)}]=\mathbb{E}[h(W,X)]$. Later, cui2023semiparametric showed that alternatively, under Assumptions (ref), (ref), (ref)$(c)$, and (ref)$(d)$, the potential outcome mean is identified as $\mathbb{E}[Y^{(a)}]=\mathbb{E}[I(A=a)Yq(W,A,X)]$. Based on these two results, cui2023semiparametric obtained the influence function of the parameter \[ \psi_0=\mathbb{E}[h(W,X)], \] and showed that under the above assumptions, the parameter $\mathbb{E}[Y^{(a)}]$ can be identified by the following functional which incorporates the information of both $h$ and $q$. \[ \mathbb{E}[Y^{(a)}]=\mathbb{E}\left[-I(A=a)h(W,X)q(W,A,X)+I(A=a)Yq(W,A,X)+h(W,X)\right]. \] Based on this representation, the following estimator can be used for the parameter $\psi_0$. \[ \hat\psi=\mathbb{E}_n\left[-I(A=a)\hat h(W,X)\hat q(W,A,X)+I(A=a)Y\hat q(W,A,X)+\hat h(W,X)\right]. \] where $\hat h$ and $\hat q$ are estimators of $h_0$ and $q_0$, respectively. Note that this is a special case of the estimator (ref) with $V=\{W,Z,A,X,Y\}$, $V_h=\{W,X\}$, $V_q=\{Z,A,X\}$, $s_1(V)=-I(A=a)$, $s_2(V)=I(A=a)Y$, $s_3(V)=1$, and $s_4(V)=0$. Therefore, our results from the previous sections can be directly applied to the proximal causal inference setting.

Conclusion

We studied the problem of estimating the $L^2$-minimal solution to a conditional moment equation under smoothness conditions on the operator and the solution. This takes the form of an integral equation, a type of inverse problem that is generally ill-posed. Instances of this problem have been extensively discussed in the literature, especially in the nonparametric instrumental variable framework, where a standard approach is to find the minimizer of a regularized version of the projected mean squared error. However, this approach can be sensitive to misspecification or slow convergence rate of the involved nuisance functions, especially, the estimator of the operator of the integral equation. To address this, we proposed a debiased estimation strategy based on the influence function of a modification of the projected error. We demonstrated that this approach leads to robustness against slow convergence rate of the operator estimator, and obtained finite-sample convergence rate of our proposed estimator using modern statistical learning-theoretic tools to characterize function space complexity. Additionally, we proposed a hyper-parameter tuning strategy and studied the loss in the convergence rate caused by hyper-parameter tuning compared to the case that the optimal order of the hyper-parameter is assumed to be known. We further studies the application of our approach to the estimation of regular parameters in a specific parameter class, which are linear functionals of the solutions to the conditional moment restrictions. For this class, we established new sufficient conditions for achieving root-$n$ consistency of a debiased influence function-based estimator. This yields an estimation strategy for this setting that incorporates two layers of debiasing.

Acknowledgements

Andrea Rotnitzky's work has been supported by the National Heart, Lung, and Blood Institute grant R01-HL137808, and by the National Institute of Allergy and Infectious Diseases grants UM1-AI068635 and R37-AI029168.

\appendices

Proofs

Proof of Proposition (ref)

The claim follows from observing that

Proof of Theorem (ref)

We use the notation $\partial_tg(t)$ to denote $\frac{\partial g(t)}{\partial t}\big|_{t=0}$. For parameter $\psi(h)$, let ${\psi(h)}_t$ be the parameter of interest under a regular parametric sub-model indexed by $t$, that includes the ground-truth model at $t=0$. We obtain an influence function of $\psi(h)$ as follows. We need to find a random variable $M$ with mean zero, that satisfies \[ \partial_t{\psi(h)}_t=E(MS(V)), \] where $S(V)=\partial_t\log p_t(V)$.

Let $G_0=g_0(V)$, and $G_1=g_1(V)$. Note that

For (J1), we have

For (J2), we have

Note that \[ \mathbb{E}[2\{\mathbb{E}[G_1h(V_h)\mid V_q]-\mathbb{E}[G_0\mid V_q]\}\{G_1h(V_h)-\mathbb{E}[G_1h(V_h)\mid V_q]\}S(V_q)]=0. \] Therefore, \[ \text{(J2)}=\mathbb{E}[2\{\mathbb{E}[G_1h(V_h)\mid V_q]-\mathbb{E}[G_0\mid V_q]\}\{G_1h(V_h)-\mathbb{E}[G_1h(V_h)\mid V_q]\}S(V)]. \]

Combining the obtained expression for (J1) and (J2), we have

which concludes the desired result.

Proof of Proposition (ref)

\begingroup \allowdisplaybreaks

\endgroup

Proof of Theorem (ref)

We first note that \begingroup \allowdisplaybreaks

\endgroup where we used the fact that for any $h$, \begingroup \allowdisplaybreaks

\endgroup

By Proposition (ref), we have

Similarly,

Therefore,

where,

and

Regarding $(\text{J}1)$, We define the loss function

Then, we have \begingroup \allowdisplaybreaks

\endgroup where the last inequality follows from the definition of $\hat h_{\lambda,t}^\text{IF}$ and that $h_0$ is assumed to belong to $\mathcal{H}$.

Note that \begingroup \allowdisplaybreaks

\endgroup with probability at least $1-5\zeta$, where the last inequality is due to Lemma (ref).

Therefore, with probability at least $1-5\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup Note that \begingroup \allowdisplaybreaks

\endgroup Therefore, with probability at least $1-5\zeta$, we have

Combining (ref), (ref), and (ref), we conclude that, with probability at least $1-5\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup Using the weighted AM-GM inequality, \[ ab\le \frac{a^2}{2w}+\frac{wb^2}{2}, \] we have \begingroup \allowdisplaybreaks

\endgroup Therefore, there exists positive constants $c_1$ and $c_2$, with $c_1<1$, such that, with probability at least $1-5\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup which implies that \begingroup \allowdisplaybreaks

\endgroup or equivalently, \begingroup \allowdisplaybreaks

\endgroup with probability at least $1-5\zeta$.

Proof of Theorem (ref)

Recall the definition of the regularized population-level objective function $h^*_{\lambda,t}$ in Display (ref). By triangle inequality, we have

Here, the second term in the upper bound characterizes the regularization bias. By Lemma (ref), under Assumption (ref), this term can be bounded as

Hence, it suffices to bound $\|\hat h_{\lambda,t}^\text{IF}-h^*_{\lambda,t}\|_2^2$.

Define \[ L(\tau):= \left\|T\left(h^*_{\lambda,t}+\tau(\hat h_{\lambda,t}^\text{IF}-h^*_{\lambda,t})\right)-g_0(V)\right\|_2^2 +\lambda\left\|h^*_{\lambda,t}+\tau(\hat h_{\lambda,t}^\text{IF}-h^*_{\lambda,t})-h^*_{\lambda,t-1}\right\|_2^2. \] Similar to bennett2023source, we start by leveraging the convexity of $L$. By the definition of $h^*_{\lambda,t}$, $L(\tau)$ is minimized at $\tau=0$, and hence, $\frac{d}{d\tau}L(\tau)|_{\tau=0}=0$. Therefore, by Taylor expansion, we have

Therefore,

By Proposition (ref), we have

Similarly,

Therefore,

where,

and

Regarding $(\text{J}1)$, We define the loss function

Then, we have \begingroup \allowdisplaybreaks

\endgroup where the inequality follows from the definition of $\hat h_{\lambda,t}^\text{IF}$.

We note that by Assumption (ref), $\ell$ is Lipschitz in its argument $h$. Specifically, we have \begingroup \allowdisplaybreaks

\endgroup where in the last step, we used the fact that $T$ is a contraction.

Therefore, by Lemma (ref), with probability at least $1-\zeta$, we have

Moreover, we note that \begingroup \allowdisplaybreaks

\endgroup

This together with Displays (ref) imply that, with probability at least $1-\zeta$, we have

Regarding $(\text{J}2)$, we have \begingroup \allowdisplaybreaks

\endgroup where we used Assumption (ref) to obtain the last expression.

Combining the bounds on $(\text{J}1)$ and $(\text{J}2)$, we obtain

Using the weighted AM-GM inequality, \[ ab\le \frac{a^2}{2w}+\frac{wb^2}{2}, \] we have

for some positive constants $c_1$, $c_2$, and $c_3$, where the weight in the weighted AM-GM inequality is chosen such that $c_2<1$. Therefore,

Regarding bounding the source error, Equation (ref) implies

Recall that $\hat h_{\lambda,0}^\text{IF}=h^*_{\lambda,0}=0$. Therefore, for $t=1$, we have

Therefore,

Equations (ref), (ref), and (ref) conclude that

Regarding bounding the projected error, by triangle inequality, we have

Here, the second term in the upper bound characterizes the regularization bias. By Lemma (ref) for projected error, under Assumption (ref), this term can be bounded as

Hence, it suffices to bound $\|T(\hat h_{\lambda,t}^\text{IF}-h^*_{\lambda,t})\|_2^2$.

Using Equation (ref), we have

where we used Display (ref). Equations (ref), (ref), and (ref) conclude that

Proof of Theorem (ref)

With an approach similar to that in the proof of Theorem (ref), we have

where, \begingroup \allowdisplaybreaks

\endgroup and

With an approach similar to that in the proof of Theorem (ref), we have \begingroup \allowdisplaybreaks

\endgroup with probability at least $1-5\zeta$, where the last inequality is due to Lemma (ref).

Therefore, with probability at least $1-5\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup Note that \begingroup \allowdisplaybreaks

\endgroup Similarly,

Therefore, with probability at least $1-5\zeta$, we have

Combining (ref) and (ref), we conclude that, with probability at least $1-5\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup Using the weighted AM-GM inequality, \[ ab\le \frac{a^2}{2w}+\frac{wb^2}{2}, \] we have \begingroup \allowdisplaybreaks

\endgroup Therefore, there exists positive constants $c_1$ and $c_2$, with $c_1<1$, such that, with probability at least $1-5\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup which implies that, with probability at least $1-5\zeta$, \begingroup \allowdisplaybreaks

\endgroup By Lemma (ref) for projected error, under Assumption (ref), we have

Therefore, with probability at least $1-5\zeta$, \begingroup \allowdisplaybreaks

\endgroup Next, we note that by the AM-GM inequality, \begingroup \allowdisplaybreaks

\endgroup Therefore, with a similar argument as before, there exists a choice of $w$ leading to \begingroup \allowdisplaybreaks

\endgroup Recall that $\hat h_{\lambda,0}^\text{IF}=h^*_{\lambda,0}=0$. Therefore, for $t=1$, we have

Therefore, for $\lambda<1$, with probability at least $1-5\zeta$,

This implies that, with probability at least $1-5\zeta$,

Adding the regularization bias similar to the proof of Theorem (ref) concludes the desired result.

Proof of Theorem (ref)

For any given function $h$ and nuisance parameter pair $\tilde\eta=(\tilde T,\tilde r)$, we define

Note that for $\eta_0=(T,r_0)$, i.e., the true values of the nuisance components, we have

Moreover, our previously defined estimator $\hat\psi(h)$ can be written as

Based on this notation, we have \[ h_0\in\operatorname*{arg\,min}_{h\in L^2(P_{0,V_h})}\mathbb{E}[\ell(h,\eta_0)], \] and \[ \hat h=\operatorname*{arg\,min}_{h\in\mathcal{C}}\mathbb{E}_n[\ell(h,\hat\eta)]. \] We first target bounding $\mathbb{E}[\ell(\hat h,\eta_0)]-\mathbb{E}[\ell(h^*,\eta_0)]$.

Define $L(h,\eta):=\ell(h,\eta)-\ell(h_0,\eta)$. Note that $\mathbb{E}[L(h,\eta_0)]\ge 0$, for all $h\in L^2(P_{0,V_h})$, and $\mathbb{E}[\ell(\hat h,\eta_0)]-\mathbb{E}[\ell(h^*,\eta_0)]=\mathbb{E}[L(\hat h,\eta_0)]-\mathbb{E}[L(h^*,\eta_0)]$. Therefore, we focus on the right hand side. This quantity can be decomposed as \begingroup \allowdisplaybreaks

\endgroup where the last inequality holds because $\mathbb{E}_n[L(\hat h,\hat\eta)]-\mathbb{E}_n[L(h^*,\hat\eta)]\le0$ as $h^*\in\mathcal{C}$. Following the ideas in vaart2006oracle,van2003unified, for any $\delta>0$, the last expression is equal to \begingroup \allowdisplaybreaks

\endgroup where the last inequality holds due to the same reason as above.

We write the last expression as \[ (T1)+(1+\delta)\{(T2)+(T3)\}+2\delta\mathbb{E}[L(h^*,\eta_0)], \] where

We start with $(T2)$. We first note that \begingroup \allowdisplaybreaks

\endgroup Therefore, \begingroup \allowdisplaybreaks

\endgroup We conclude that \[ (T2)\lesssim\max_{h\in\mathcal{C}}\| h-h^*\|_2\|T-\hat T\|^2+\max_{h\in\mathcal{C}}\| h-h^*\|_2\|T-\hat T\|\|r_0-\hat r\|_2. \]

Moving on to $(T3)$, we first note that \begingroup \allowdisplaybreaks

\endgroup Define $Z_{h,i}:=\ell_i(h^*,\hat\eta)-\ell_i(h^*,\eta_0)-\ell_i( h,\hat\eta)+\ell_i( h,\eta_0)-\mathbb{E}[\ell(h^*,\hat\eta)-\ell(h^*,\eta_0)-\ell( h,\hat\eta)+\ell( h,\eta_0)]$, which is a mean-zero random variable. In addition, define \begingroup \allowdisplaybreaks

\endgroup Our goal is to find a bound on $T_n$, satisfied with high probability, which will be a bound on $(T3)$, with high probability.

Since $h$'s and nuisance functions are estimated on separate data folds, $\sum_{i=1}^nZ_{h,i}$ is a sum of i.i.d. mean-zero random variables. Moreover, we have $\|Z_{h,1}\|_\infty<C$ for some constant $C$, and also, we have \begingroup \allowdisplaybreaks

\endgroup Note that we have \begingroup \allowdisplaybreaks

\endgroup We address each line in the last expression separately. The first line can be written as \begingroup \allowdisplaybreaks

\endgroup The second line can be written as \begingroup \allowdisplaybreaks

\endgroup The third line can be written as \begingroup \allowdisplaybreaks

\endgroup The fourth line can be written as \begingroup \allowdisplaybreaks

\endgroup Finally, the fifth line can be written as \begingroup \allowdisplaybreaks

\endgroup Therefore, we have \begingroup \allowdisplaybreaks

\endgroup Hence, we have an upper bound $\sigma^2=c(\max_{h\in\mathcal{C}}\|h-h^*\|_2^2\|T-\hat T\|^2+\max_{h\in\mathcal{C}}\| h-h^*\|^2_2\|r_0-\hat r\|^2_2)$ on $var(Z_{h,1})$, for some constant $c$.

Therefore, by Lemma (ref), for all $x>0$, we have \begingroup \allowdisplaybreaks

\endgroup Therefore, \begingroup \allowdisplaybreaks

\endgroup But $x>0$, hence $x=Cb+ \sqrt{C^2b^2+2bn\sigma^2}$. This implies that with probability at least $1-\zeta$, \[ T_n\le\frac{x}{n}=\frac{1}{n}\{Cb+ \sqrt{C^2b^2+2bn\sigma^2}\}. \]

Recall that $\sqrt{a+b}\le\sqrt{a}+\sqrt{b}$, for $a,b\ge0$. Therefore, we have \begingroup \allowdisplaybreaks

\endgroup

We conclude that with probability at least $1-\zeta$, \[ (T3)\lesssim\frac{\log (2M/\zeta)}{n}+ \sqrt{\log(2M/\zeta)}\frac{1}{\sqrt{n}}\left\{\max_{h\in\mathcal{C}}\|h-h^*\|_2\|T-\hat T\|+\max_{h\in\mathcal{C}}\| h-h^*\|_2\|r_0-\hat r\|_2\right\}. \]

Moving on to $(T1)$, we first note that \begingroup \allowdisplaybreaks

\endgroup Define $Z_{h,i}:=L_i( h,\eta_0)-\mathbb{E}[L( h,\eta_0)]$, which is a mean-zero random variable. In addition, define \begingroup \allowdisplaybreaks

\endgroup Our goal is to find a bound on $T_n$, satisfied with high probability, which then implies that $(T3)\le2(\text{bound on }T_n)$ with high probability.

Since $h$'s and nuisance functions are estimated on separate data folds, $\sum_{i=1}^nZ_{h,i}$ is a sum of i.i.d. mean-zero random variables. Moreover, we have $\|Z_{h,1}\|_\infty<C$ for some constant $C$, and also, we have \begingroup \allowdisplaybreaks

\endgroup

Note that we have \begingroup \allowdisplaybreaks

\endgroup Hence, we have an upper bound $\sigma^2=c\|T(h-h_0)\|_2^2$ on $var(Z_{h,1})$, for some constant $c$.

Therefore, by Lemma (ref), for all $x>0$, we have \begingroup \allowdisplaybreaks

\endgroup

Therefore, \begingroup \allowdisplaybreaks

\endgroup Suppose the maximum above is achieved at $\tilde h$. As noted in the appendix of vaart2006oracle,

where $y=\frac{1}{1+\delta}x$, and $\lambda=\frac{\delta}{1+\delta}\mathbb{E}[L(\tilde h,\eta_0)]$. Therefore, \begingroup \allowdisplaybreaks

\endgroup

With $\zeta=2M\exp\left\{\frac{-1}{2}\frac{y\lambda+\lambda^2}{2\frac{\sigma^2}{n}}\right\}$, we have $x=\frac{1}{n}\frac{1}{1+\delta}\frac{\sigma^2}{\lambda}4\log\frac{2M}{\zeta}-\lambda$, and with $\zeta=2M\exp\left\{\frac{-1}{2}\frac{y+\lambda}{2\frac{C}{n}}\right\}$, we have $x=\frac{1}{n}\frac{1}{1+\delta}C4\log\frac{2M}{\zeta}-\lambda$. Therefore, with probability at least $1-\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup where $\lambda=\frac{\delta}{1+\delta}\mathbb{E}[L(\tilde h,\eta_0)]$.

Suppose we can show $\frac{\sigma^2}{\lambda}\le C'$, for some constant $C'$. Then we can conclude that with probability at least $1-2\zeta$, we have \[ (T1)\lesssim\frac{\log(2M/\zeta)}{n}. \] Therefore, it remains to prove the boundedness of $\frac{\sigma^2}{\lambda}$.

Note that

Set $\delta=1$. Regarding $\mathbb{E}[L(\tilde h,\eta_0)]$, note that \begingroup \allowdisplaybreaks

\endgroup

Therefore, \[ \frac{\sigma^2}{\lambda}\lesssim 2\frac{\|T(\tilde h-h_0)\|_2^2}{\|T(\tilde h-h_0)\|_2^2}=2. \]

\\

From the obtained bounds for $(T1)$, $(T2)$, and $(T3)$, we conclude that with probability at least $1-3\zeta$, \begingroup \allowdisplaybreaks

\endgroup

As we saw in Display (ref), $\|T(h-h_0)\|_2^2=\mathbb{E}[L(h,\eta_0)]$. Therefore, using inequality (ref), with probability at least $1-3\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup which implies that

Proof of Corollary (ref)

Based on the result of Theorem (ref), it suffices to bound $\big\|T(h^*-h_0)\big\|_2^2$.

Let $\lambda^*_1$ be a choice of the hyper-parameter stated in Theorem (ref), and let $\lambda_\mathcal{C}^*$ be the smallest element in $\{\lambda_i\}_{i=1}^n$ which is larger than or equal to $\lambda^*_1$. That is, \[ \lambda^*_1\le\lambda_\mathcal{C}^*\le\lambda^*_1+B_n/n. \] Moreover, let $h^*_1:=\hat h_{\lambda^*_\mathcal{C},2}^\text{IF}$, that is, $h^*_1$ corresponds to hyper parameter $\lambda_\mathcal{C}^*$.

Hence, using Theorem (ref), with probability at least $1-2\zeta$, we have \begingroup \allowdisplaybreaks

\endgroup

Let $\lambda^*_2$ be a choice of the hyper-parameter among the candidates satisfying $\lambda^*_2\lesssim\Delta_{M,n}^{\frac{1}{\min\{2,\beta+1\}}}$. Let $h^*_2$ be the corresponding candidate function.

Hence, using Theorem (ref), with probability at least $1-5\zeta$, we have

Therefore, (ref) and (ref) conclude that

Proof of Theorem (ref)

Regarding the source error, using Jensen's inequality similar to the approach of florens2011identification, we have \begingroup \allowdisplaybreaks

\endgroup where $(a)$ and $(b)$ follow from Parts 1 and 2 of Assumption (ref), respectively. Hence, we conclude that \[ \|\hat h-h_0\|_2^{2}\lesssim\|T(\hat h-h_0)\|_2^{\frac{2}{1+\alpha}}. \]

Auxiliary Lemmas

See carrasco2007linear for a proof for the case of compact operators, and engl1996regularization for a proof for the general case. See also bennett2023source.