Towards a General Large Sample Theory for Regularized Estimators

Towards a General Large Sample Theory for Regularized Estimators

\startcontents[section1] \printcontents[section1]{1}{

Table of Contents

}

{2.5pt} {2.5pt}

Introduction

It was noted as early as Stein Stein1956 that in many complex models, the parameter mapping, $\psi$, that links the probability distribution generating the data, $P$, to some parameter space may be ill-behaved or even ill-defined when evaluated at the empirical distribution. The widespread solution in these cases is to regularize the problem. Regularization procedures are ubiquitous in statistics and elsewhere, examples of these include kernel-based estimators; series-based estimators and penalization-based estimators among many others.\footnote{Examples of regularizations are so ubiquitous that providing a thorough review is outside the scope of the paper; see e.g. BickelLi06, buhlmann2011statistics, HardleLinton1994 and Chen2007 for excellent reviews of several regularization methods.} Even though there has been an enormous amount of work in statistics and other sciences studying the properties of these procedures, they are viewed, by and large, as separate and unrelated. In particular, results like consistency or large sample distribution theory, when they exists, they have only been derived in a case-by-case basis; to our knowledge, there is no general theory or systematic approach. The goal of this paper is to fill this gap by providing the basis for an unifying large sample theory for regularized estimators that will allow us to make systematic progress in studying their large sample properties.

Our point of departure is the general conceptual framework put forward by Bickel and Li (BickelLi06), wherein the authors propose a general definition of regularization. According to their framework, a regularization can be viewed sequence of parameter mappings, $(\psi_{k})_{k=1}^{\infty}$ that replaces the original parameter mapping, $\psi$, each element is well-behaved, and its limit coincides with the original mapping. The index of this sequence (denoted by $k$) represents what is often referred as the tuning (or regularization) parameter; e.g. it is the (inverse of the) bandwidth for kernels, the number of terms in a series expansion, or the (inverse of the) scale parameter in penalizations. While Bickel and Li's framework encompasses many examples and applications, it is unclear what type of asymptotic properties can be obtained in such a general framework. We provide two set of general theorems under intuitive conditions that establish large sample properties for regularized estimators. One set of results establishes consistency and rate of convergence, and a data-driven method for choosing the tuning parameter that achieves these rates. Another set of results provide foundations for large sample distribution theory by deriving a generalization of the classical asymptotic linearity property.

Our approach to obtain consistency and convergence rate results is akin to the one used in the standard large sample theory for “plug-in" estimators, in the sense that it relies on continuity of the mapping used for estimation (see Wolfowitz1957, DonohoLiu1991). The key difference is that in plug-in estimation this mapping is $\psi$, but for regularized estimators the natural mapping is the (sequence of) regularized parameter mappings, $(\psi_{k})_{k=1}^{\infty}$; this difference --- in particular, the fact that we have a sequence of mappings --- introduces nuances that are not present in the standard “plug-in" estimation case. We show that the key component of the convergence rate is the modulus of continuity of the regularized mapping, which, typically, will deteriorate as one moves further into the sequence of regularized mappings, thus yielding a generalized version of the well-known “noise-bias" trade-off. While this result, by itself, does not constitute a big leap from Bickel and Li's framework, we use the underlying insights to propose a data-driven method to choose the tuning parameter that under some conditions yields convergence rates proportional to the “oracle" ones, i.e., those implied by the choice that balances the “noise-bias" trade-off. This method is an extension of the Lepski method as presented in PereverzevSchock2006 for ill-posed inverse problems.\footnote{Similar versions has been used in several particular applications. Closest to our examples are the work: by Pouzo2017 for regularized M-estimators; by ChenChristensen2015 in non-parametric IV regressions; by GineNickl2008 for estimation of the integrated square density; by gaillac2019adaptive in a random coefficient model; by lepski1997 for estimation of a function at a point.}

Our second set of results are concerned with obtaining a type of asymptotic linear representation for regularized estimators. The property of asymptotic linearity is well-known in the literature and is the cornerstone of large sample distribution theory. This property states that the estimator, once centered at the true parameter, is equal to a sample average of a mean zero function --- referred as the influence function --- plus an asymptotically negligible term.

In parametric models, asymptotic linearity is typically satisfied by commonly used estimators like the “plug-in" estimator. In more complex settings such semi-/non-parametric models, however, this is not longer true. In such cases, there are no estimators satisfying this property, because, for instance, the efficiency bound of the parameter of interest is infinite, or more generally, the parameter is not root-n estimable. For these situations, asymptotic representations analogous to asymptotic linearity have been obtained in specific examples for specific regularizations, but, to our knowledge, there is no general approach. This is specially problematic as there is no systematic method for properly standardizing the estimator in situations where the parameter is not root-n estimable.\footnote{For density and regression estimation problems there is a large literature, especially for particular functionals like evaluation at a point; e.g. see EggermontLariccia2001 Vol I and II for references and results. In more general contexts such as M-estimation and GMM-based models, to our knowledge, the literature is much more sparse with only a few papers allowing for slower than root-n parameters in particular settings. Closest to ours are the papers by ChenLiao2014, chen2014sieve in the context of M-estimation models with series/sieve-based estimators; Newey1994 in a two-stage moment model using kernel-based estimators; ChenPouzo2015 in conditional moment models with sieve-based estimators; cattaneo2013optimal in partitioning estimators of the conditional expectation function and its derivatives.} Our goal is to propose a systematic approach by considering a generalization of asymptotic linearity that relaxes certain features of the standard property but still provides a useful asymptotic characterization of the estimator. This property, which is already present in many examples and we refer to as Generalized Asymptotic Linearity (GAL for short), relaxes the standard one in two dimensions: It allows for the location term to be different from the true parameter, and it allows for this centering and the influence function to vary with the sample size. Each of these relaxations attempts to capture different nuances that already exists in the many scattered examples in the literature. Our results, which we now describe, will shed more light on the role and necessity of each.

We provide sufficient conditions for regularized estimators to satisfy GAL. Analogously to the theory of asymptotic linearity for plug-in estimators, our results rely on a notion of differentiability, but contrary to plug-in estimators, it relies on differentiability of each element in the sequence of regularized mappings, $(\psi_{k})_{k=1}^{\infty}$ not on differentiability of the original mapping $\psi$.

As a consequence of this approach, GAL for regularized estimators exhibits two simplified features. First, the location term is given by $\psi_{k}(P)$ which can be interpreted as a psuedo-true parameter. The second simplified feature concerns the influence function and its dependence on the sample size. As in the location term, the dependence on the sample size of the influence function arises only through the dependence of the tuning parameter, $k$, on the sample size. Thus, the relevant object is a sequence of influence functions, each related to the derivative of the elements in $(\psi_{k})_{k=1}^{\infty}$. We view this quantity as the natural departure from the traditional influence function as it is the sequence of regularized mappings, $(\psi_{k})_{k=1}^{\infty}$, an not the original mapping, $\psi$, the one used for constructing the estimator. This last feature allows us to propose a natural and systematic way of standardizing the estimator regardless of whether root-n consistency holds. To explain this, we first note that in situations where asymptotic linearity holds, the proper standardization is given by square root of the sample size divided by the standard error of the value of the influence function. Under GAL the standarization of the regularized estimators turns out to be analogous except that in this case the influence function is indexed by the tuning parameter which at the same time depends on the sample size. Whether the standardization is root-n or slower depends on the behavior of the standard error of the value influence function as we move further into the sequence of regularized mappings (i.e., as $k$ diverges).

Throughout the paper we present examples not to break new ground but to illustrate our assumptions and results, and their scope.

Notation. The term “wpa1-$P$" is short for with probability approaching 1 under $P$, so for a generic sequence of IID random variables $(Z_{n})_{n}$ with $Z_{n} \sim P$, the phrase “$Z_{n} \in A$ wpa1-$P$" formally means $P(Z_{n} \notin A) = o(1)$. For any random variables $(X,Y)$ we use $p_{X}$ and $p_{XY}$ to denote the pdf (w.r.t. Lebesgue) corresponding to $X$ and $X,Y$ resp. For any linear normed spaces $(A,||.||_{A})$ and $(B,||.||_{B})$, let $A^{\ast}$ be the dual of $A$, and for any continuous, homogeneous of degree 1 function $f : (A,||.||_{A}) \mapsto (B,||.||_{B})$, $||f||_{\ast} = \sup_{a \in A \colon ||a||_{A} \ne 1} ||f(a)||_{B}$. For a Euclidean set $S$, we use $L^{p}(S)$ to denotes the set of $L^{p}$ functions with respect to Lebesgue. For any other measure $\mu$, we use $L^{p}(S,\mu)$ or $L^{p}(\mu)$. The norm $||.||$ denotes the Euclidean norm and when applied to matrices it corresponds to the operator norm. For any matrix $A$, let $e_{min}(A)$ denote the minimal eigenvalue. The symbol $\precsim$ denotes less or equal up to universal constants; $\succsim$ is defined analogously.

Setup

Let $\mathbb{Z} \subseteq \mathbb{R}^{d}$ and let $\boldsymbol{z} \equiv (z_{1},z_{2},...) \in \mathbb{Z}^{\infty}$ denote a sequence of IID data drawn from some $P \in \mathcal{P}(\mathbb{Z}) \subset ca(\mathbb{Z})$, where $\mathcal{P}(\mathbb{Z})$ is the set of Borel probability measures over $\mathbb{Z}$ and $ca(\mathbb{Z})$ is the space of signed Borel measures of finite variation. For each $P \in \mathcal{P}(\mathbb{Z})$, let $\mathbf{P}$ be the induced probability over $\mathbb{Z}^{\infty}$. A model is defined as a subset of $\mathcal{P}(\mathbb{Z})$; and it will typically be denoted as $\mathcal{M}$.

A parameter on model $\mathcal{M}$ is a mapping $\psi : \mathcal{M} \rightarrow \Theta$ with $(\Theta,||.||_{\Theta})$ being a normed space.\footnote{If the mapping does not point-identified an element of $\Theta$, i.e., $\psi$ is one-to-many, our results go through with minimal changes that account for the fact that $\psi(P)$ is a set in $\Theta$.}

For the results in this paper, we need to endow $\mathcal{M}$ with some topology. For the results in Section (ref) it suffices to work with a distance, $d$, under which the empirical distribution (defined below) converges to $P$. For the results in Section (ref) and beyond, however, it is convenient to have more structure on the distance function, and thus, we work with a distance of the form

where $\mathcal{S}$ is some class of Borel measurable and uniformly bounded functions (bounded by one). For instance, the total variation norm can be viewed as taking $\mathcal{S}$ as the class of indicator functions over Borel sets, and its denoted directly as $||.||_{TV}$; the weak topology over $\mathcal{P}(\mathbb{Z})$ is metricized by taking $\mathcal{S}=LB$ --- the space of bounded Lipschitz functions --- and its norm is denoted directly as $||.||_{LB}$; see VdV-W1996 for a more thorough discussion.

Regularization

Let $\mathcal{D} \subseteq \mathcal{P}(\mathbb{Z})$ be the set of all discretely supported probability distributions. Let $P_{n} \in \mathcal{D}$ be the empirical distribution, where $P_{n}(A) = n^{-1} \sum_{i=1}^{n} 1\{ Z_{i} \in A \}$ for any $A \subseteq \mathbb{Z}$. As illustrated by our examples, in many situations --- especially in non-/semi-parametric models --- the parameter mapping might be either ill-defined (e.g., if $P_{n} \notin \mathcal{M}$) or ill-behaved when evaluated at the empirical distribution $P_{n}$, so it has to be regularized.

The following definition of regularization is based on the first part of the definition in BickelLi06 p. 7. To state it, we define a tuning set as any subset of $\mathbb{R}_{+}$ that is unbounded from above, and the approximation error function as $k \mapsto B_{k}(P) \equiv ||\psi_{k}(P) - \psi(P)||_{\Theta}$.

Condition 1 ensures that $\psi_{k}(P_{n})$ and $\psi_{k}(P)$ are well-defined and that they are singletons for all $k \in \mathbb{K}$. Condition 2 ensures that, in the limit, the regularization approximates the original parameter mapping; the limit is warranted as the tuning set $\mathbb{K}$ is unbounded from above. In many applications the tuning set is given by $\mathbb{N}$ but there are applications such as kernel-based estimators, where it is more natural to use a (uncountable) subset of $\mathbb{R}_{+}$.

For each $k \in \mathbb{K}$, the implied estimator is given by $\psi_{k}(P_{n})$ which --- like the “plug-in" estimator --- is permutation invariant. While, this restriction still encompasses a wide array of commonly used methods, it does rule out some estimation methods, notably those that rely on non-trivial sample-splitting procedures. We briefly discuss how to extend our framework to these cases in Appendix (ref).

Conditions 1 and 2 are not enough to obtain “nice" asymptotic properties of the regularized estimator such as consistency and asymptotic normality. In analogy to the standard asymptotic theory for “plug-in" estimators, these properties will be obtained by essentially imposing different degrees of smoothness on the regularization.

Examples

The following examples complement those in BickelLi06 to illustrate that the Definition (ref) encompasses a wide array of commonly used methods.

The next is not an example but rather a commonly used estimation technique that also fits in our framework.

As these examples, those below and those in BickelLi06 illustrate, the formulation of regularization is very flexible. Indeed, Definition (ref) is sufficiently mild that one may wonder whether any results can be obtained at the present level of generality. In what follows, we establish some useful asymptotic properties by imposing additional smoothness restrictions on the regularization.

Consistency for Regularized Estimators

We say a regularization is consistent if for each $k \in \mathbb{K}$, $\psi_{k}(P_{n}) = \psi_{k}(P) + o_{P}(1)$. Using Definition (ref) it is straightforward to show that for consistent regularizations there exists a tuning parameter sequence $(k_{n})_{n}$ such that $\psi_{k_{n}}(P_{n}) = \psi(P) + o_{P}(1)$. This claim, however, is silent about how to choose the tuning parameter sequence and what is the corresponding rate. This feature does not seem to be a shortcoming of the claim, but rather a manifestation of the fact that the consistency requirement is very mild. In other words, it would appear that some strengthening of this requirement is needed if we want obtain stronger conclusions. In this section and the next one, our goal is to give general but “easy-to-interpret" conditions that will enable us to obtain convergence rates and a data-driven choice of tuning parameter sequence that achieves such rates, thus providing a general guideline for establishing asymptotic concentration results for regularized estimators.

In order to do this, we take as given that the empirical distribution converges to the truth at rate, $(r_{n})_{n}$ --- a diverging positive real-valued sequence --- under a distance $d$, i.e., $d(P_{n},P) = O_{P}(r^{-1}_{n})$. Such results can typically be obtained from empirical process theory and are not the focus of this paper. Under this condition, from the classical results of Wolfowitz1957, continuity of the regularization (with respect to $d$) presents itself as a natural condition to establish consistency of the regularization. To define the notion of continuity formally we say a function $f : \mathbb{R}_{+} \rightarrow \mathbb{R}_{+}$ is a modulus of continuity if $f$ is continuous, non-decreasing and such that $f(t)=0$ iff $t=0$.

The definition is equivalent to the standard “$\delta/\epsilon$"-definition of continuity because the modulus of continuity of $\psi_{k}$, $\delta_{k}$, can converge to $0$ arbitrary slowly. Admittedly, this condition is not necessary to obtain our results as it requires continuity to hold for any deviation of $P$, but the notion of consistent regularization requires continuity only along those deviations taken by $P_{n}$ for all $n$ large enough. However, we still view this condition as a reasonable “initial step" to obtain consistency results. Moreover, the definition does not impose any uniform bounds on $\delta_{k}$ across different $k \in \mathbb{K}$. While such restriction would simplify the proofs considerably, it is too strong for many applications. Recall that the regularization is introduced precisely due to the poor behavior of $\psi$ at $P$.

As the lemma (ref) in Appendix (ref) shows, when the regularization is continuous, the “sampling error" term, $||\psi_{k}(P_{n}) - \psi_{k}(P)||_{\Theta}$ is of order $\delta_{k}(r^{-1}_{n})$ in probability. From this result and the fact that $\limsup_{n \rightarrow \infty} \delta_{k}(r^{-1}_{n}) = 0$ for every $k \in \mathbb{K}$, a simple diagonalization argument establishes existence of a tuning parameter sequence, $(k_{n})_{n}$ for which $(\psi_{k_{n}}(P_{n}))_{n \in \mathbb{N}}$ is consistent. The next theorem formalizes this claim.

Although constructive, the consistency result outlined above suffers from the potential problem that the associated tuning parameter sequence makes no attempt to ensure that the magnitude of the estimation error is in some sense minimal. Indeed, because the tuning parameter sequence has been designed to satisfy only the minimal requirement that $B_{k_{n}}(P) + \delta_{k_{n}}(d(P_{n},P)) = o_{P}(1)$; the only property that can be claimed on the part of the regularized estimator is consistency. The goal of the next section is to improve on this as well as to construct a data-driven choice of tuning parameter.

Data-driven Choice of Tuning Parameter

Let $\mathbb{K}_{n} $ be the (user-specified) set over which the tuning parameter is chosen; it is assumed to be a finite subset of $\mathbb{K}$.\footnote{In Appendix (ref) we extend the main theorem of this section to the case where $\mathbb{K}_{n}$ is any closed set of $\mathbb{K}$, not necessarily finite.} Also, in this section, we strengthen the consistency of $P_{n}$ to $d(P_{n},P) = o_{P}(r^{-1}_{n})$ (remark (ref) below discusses the reason behind this choice).

By Theorem (ref) and the triangle inequality it is easy to see that the distance between the regularized estimator, $\psi_{k_{n}}(P_{n})$, and the true parameter is bounded by the sum of two terms: the “sampling error", $\delta_{k_{n}}(r^{-1}_{n})$ and the “approximation error", $B_{k_{n}}(P)$, which generalizes the well-known “noise-bias" trade-off present in many applications. Thus, this observation suggests the following criterion to construct tuning sequence $(k_{n})_{n}$ that yields a consistent estimators:

which minimizes the trade-off between the approximation and the sampling errors. This choice represents commonly used heuristics and it is a good prescription to obtain approximately optimal rate of convergences (BirgeMassart1998). However, often times it is unfeasible, since it relies on knowledge of the approximation error, which is typically unknown because it depends on features of the unknown $P$.

It is thus desirable to construct a choice of tuning parameter that sidestep this issue while still providing similar rates of convergence. We propose an adaptation of the Lepski method that provides a data-driven choice that satisfies these properties, without requiring knowledge of $B_{k}(P)$. Due to the nature of the Lepski method, in order to establish the desired results we need monotonicity of the sampling and approximation errors as functions of the tuning parameter. Since these functions may not be monotonic, we replace them by monotonic majorants. Formally, let $k \mapsto \bar{B}_{k}(P)$ be a non-increasing function from $\mathbb{R}_{+}$ to itself such that $\bar{B}_{k}(P) \geq ||\psi_{k}(P) - \psi(P)||_{\Theta}$ for all $k \geq 0$, $\lim_{k \rightarrow \infty} \bar{B}_{k}(P) = 0$, and, for each $n \in \mathbb{N}$, let $ k \mapsto \bar{\delta}_{k}(r^{-1}_{n})$ be a non-decreasing function from $\mathbb{R}_{+}$ to itself such that $\bar{\delta}_{k}(r^{-1}_{n}) \geq \delta_{k}(r^{-1}_{n})$.

For each $n \in \mathbb{N}$, given $\mathbb{K}_{n}$ and a non-decreasing function $\upsilon_{n} : \mathbb{K}_{n} \rightarrow \mathbb{R}_{+}$ such that $k \mapsto \upsilon_{n}(k) = 4\bar{\delta}_{k}(r^{-1}_{n})$, the Lepski choice is given by

where

The following theorem is the main result of this section.

The following corollary is a direct consequence of Theorem (ref) and its proof is omitted.

Theorem (ref) and its corollary show that our data-driven choice of tuning parameter achieves the same rate as the one corresponding to the “infeasible" choice, provided the monotonicity conditions hold. Hence, this result, Theorem (ref) and Theorem (ref) offer a general road-map for establishing consistency and convergence rates of regularized estimators based on continuity of the regularization.\footnote{Without further restriction on $\mathbb{K}_{n}$ there is no guarantee that $\inf_{k \in \mathbb{K}_{n}} \{ \delta_{k}(r^{-1}_{n}) + B_{k}(P) \}$ is of the same magnitude as $\inf_{k \in \mathbb{K}} \{ \delta_{k}(r^{-1}_{n}) + B_{k}(P) \}$. In Appendix (ref), Proposition (ref) gives conditions on $\mathbb{K}_{n}$ that guarantee this result.}

Examples

The following examples illustrate how to apply our results to existing applications and in the process establish a new result. Example (ref) considers the case of bootstrapping the mean of a distribution when it is known to be non-negative, and it is based on andrews2000inconsistency. In this paper, the author showed inconsistency of the bootstrap and proposed several consistent alternatives; we take one --- the “k-out-of-n" bootstrap (BickelFreedman1981) --- and illustrate how our methods can be used to derive the rate of convergence of this procedure and to choose the tuning parameter $k$ that achieves this rate. To our knowledge this last result is novel.\footnote{BickelLi06 and bickel2008choice perform a similar exercise but for a different case: estimation of largest order statistic.} Example (ref) provides primitive conditions for establishing continuity in M-estimation problems.

Asymptotic Representations for Regularized Estimators

The goal of this section is to provide “easy-to-interpret" sufficient conditions to obtain a generalized asymptotic linear (GAL) representation for regularized estimators, assuming that $P_{n}$ converges to $P$ in some sense. Throughout this section we assume $\Theta \subseteq \mathbb{R}$ to simplify the exposition; the results can be easily be extended to vector-valued parameters.\footnote{In other cases where the parameter of interest is infinite-dimensional GAL is too weak and a stronger notion is needed; we refer the reader to a previous version of this paper jansson2017general for this case.}

We say a regularization, $\boldsymbol{\psi}$, is asymptotically linear if there exists $\boldsymbol{\nu} \equiv (\nu_{k})_{k\in \mathbb{K}}$ such that, for all $k \in \mathbb{K}$, $\nu_{k} \in L^{2}_{0}(P) \equiv \{ f\in L^{2}(P)\setminus\{0 \} \colon E_{P}[f(Z)] = 0 \}$ and

In this case, we say $\boldsymbol{\nu}$ is the influence of the regularization. It is straightforward to show that for such regularizations there exists a sequence $(k_{n})_{n}$ for which the following property is satisfied:

If a regularization satisfies GAL($\boldsymbol{k}$) then, in order to study its asymptotic behavior, it suffices to study the behavior of $ n^{-1/2} \sum_{i=1}^{n} \frac{\nu_{\boldsymbol{k}(n)}(Z_{i})}{||\nu_{\boldsymbol{k}(n)}||_{L^{2}(P)}}$. Moreover, this property suggests a systematic way for how to scale the estimator: Using by $\sqrt{n}/||\nu_{\boldsymbol{k}(n)}||_{L^{2}(P)}$ as opposed to just $\sqrt{n}$. This insight is particularly useful in situations where root-n estimation is not possible. Thus, this property can be viewed as extending the standard asymptotic linearity one for root-n estimable parameters to a larger class of problems.

Representations akin to GAL are already present in many examples; our contribution, as we view it, is to put these insights on a common framework so they can be applied more generally and to provide primitive properties on the structure of the regularization that guarantee asymptotic linearity --- and consequently, guarantee GAL. Regarding this last point, it is well known that in cases where the “plug-in" method is used, differentiabilty is the natural property, the definition of asymptotic linear regularization suggests differentiability of $\psi_{k}$ for each $k \in \mathbb{K}$ as reasonable starting point. Before presenting the definition of differentiable regularization we present a classical example that illustrates GAL, the influence and the scaling when the parameter is not root-n estimable

We now present the definition of differentiable regularization. For this, let $\mathcal{T}_{P} \equiv \{ a \mu \colon a\geq 0~and~ \mu \in \mathcal{D} - \{P\}\}$ and let $\tau$ be any locally convex topology over $ca(\mathbb{Z})$ dominated by $||.||_{TV}$. \footnote{Since we are working with measures, and not probabilities, it is convenient to allow for (non-metrizable) topologies. Locally convex topology means that it is constructed in terms of a family of semi-norms; dominated by $||.||_{TV}$ means that for any semi-norm, $\rho$, $\rho(Q) =O( ||Q||_{TV})$ for all $Q \in ca(\mathbb{Z})$.}

The following is the main result of this section.

It is easy to check that the influence of the regularization implied by the theorem is given by the sequence of $L^{2}_{0}(P)$ mappings, $(\varphi_{k}(P))_{k \in \mathbb{N}}$ where

While the theorem shows existence of a sequence of tuning parameters for which generalized asymptotic linearity holds, it is silent about how to construct such sequence; we discuss this in Section (ref).

Examples

We now present a series of examples. The goal here is not to break new ground but to use classical examples to illustrate the conditions and components of Theorem (ref).

The following example is the celebrated integrated square density, for which BickelRitov1990 showed that even though the efficiency bound is finite, no estimator converges at root-n rate; thereby illustrating that in some circumstances studying the local shape of $\psi$ can be quite misleading. Our approach does not suffer from this criticism since it directly captures the (local) behavior of the estimator at hand. It is also general enough to encompass many of the proposed estimators in the literature, including “leave-one-out" types.

Next, we consider the NPIV example. It is not hard to see that the influence of $\boldsymbol{\gamma}$ will be given by $z \mapsto \int D \psi_{k}(P)^{\ast}[\pi](z) - E_{P}[D \psi_{k}(P)^{\ast}[\pi](Z)]$ provided $D \psi_{k}(P) : \mathcal{T}_{P}^{\ast} \rightarrow L^{2}([0,1])$ and its adjoint $D \psi_{k}(P)^{\ast} : L^{2}([0,1]) \rightarrow \mathcal{T}_{P}^{\ast}$ exists ($\mathcal{T}_{P}^{\ast}$ is the dual of $\mathcal{T}_{P}$). For sieve-based and penalization-based regularization schemes, we characterize $D \psi_{k}(P)^{\ast}$ and show how its standard deviation can be used to appropriately scale the estimator to obtain a generalized asymptotic linear representation regardless of whether the parameter is root-n estimable or not. This last result, illustrates how our method can be used to generalize the approach proposed in ChenPouzo2015 to general regularizations. As a by-product, we extend the results in AckerbergChenHahnLiao2014 and link the influence function of the sieve-based regularization to simpler, fully parametric, misspecified GMM models.

Data-driven Choice of Tuning Parameter and Undersmoothing

Theorem (ref) implies existence of a $n \mapsto k(n)$ such that\footnote{The display hold provided $\liminf_{n \rightarrow \infty} ||\varphi_{k(n)}(P)||_{L^{2}(P)} >0 $ For the applications we have in mind, this restriction is natural and non-binding. Our results are not designed for cases where $\lim_{k \rightarrow \infty} ||\varphi_{k}(P)||_{L^{2}(P)} = 0$; this case can be handled separately --- and rather easily --- since both the approximation error and the rate of $k \mapsto \eta_{k}(P_{n}-P)$ decrease as $k$ increases.}

I.e., the asymptotic behavior of the regularized estimator --- once scaled and centered --- is characterized by a term due to the approximation error and a stochastic term. For obtaining asymptotic distributions, it is common practice to try to find sequences $(k(n))_{n}$ satisfying Theorem (ref) for which the approximation term in expression ((ref)) vanishes. Unfortunately it is known that such sequences do not always exists at this level of generality; e.g. BickelRitov88 and HallMarron87. In view of this remark it is natural to seek choices of tuning parameter that make the terms in the RHS of expression ((ref)) as “small as possible". Such choices will guarantee that GAL and the asymptotic negligibility of the approximation error both hold when possible, and otherwise, will at least yield good rates of convergence for $\psi_{k}(P_{n}) - \psi(P)$.

The result in this section shows that the data-driven way of choosing tuning parameters described in Section (ref) satisfies this property. For each $n \in \mathbb{N}$, the data-driven choice of tuning parameter is of the form $ \tilde{k}_{n} = \arg\min\{ k \colon k \in \mathcal{L}_{n}(\Lambda) \}$ for a suitable chosen function $ k \mapsto \Lambda(k)$. In section (ref), the relevant function was $k \mapsto 4 \bar{\delta}_{k}(r^{-1}_{n})$; in this case, however, the structure of the problem is different. In particular, in addition to the reminder term $(\eta_{k})_{k}$ implied by differentiability and the scaled approximation error, there is the additional term given by $n^{-1/2} \sum_{i=1}^{n} \frac{ \varphi_{k(n)}(P)(Z_{i}) }{|| \varphi_{k(n)}(P) ||_{L^{2}(P)} }$. The following assumption introduces the quantities to construct $(\Lambda_{k})_{k}$. For each $n$, let $\mathbb{K}_{n}$ be the grid defined as in Section (ref).

As the proof of Lemma (ref) in Appendix (ref) suggests, the sequence that defines our tuning parameter, for each $n \in \mathbb{N}$, is given by $k \mapsto \Lambda(k) \equiv 4 (\bar{\delta}_{1,k}(n) + \bar{\delta}_{2,k}(n))$.

The rate in the proposition is --- up to $C^{2}_{n}$ factor --- the minimum value of the sum of two terms: $\sqrt{n} \frac{\bar{\delta}_{1,k}(n) + \bar{\delta}_{2,k}(n)}{||\varphi_{k}(P)||_{L^{2}(P)}}$, that controls the reminder term of GAL and another one, $\sqrt{n} \frac{B_{k}(P)}{||\varphi_{k}(P)||_{L^{2}(P)}} $, that controls the approximation error term. Therefore, if there exists a choice of tuning parameter for which both these terms are asymptotically negligible, our result implies that

That is, the asymptotic distribution of $ \sqrt{n} \frac{ \psi_{\tilde{k}_{n}}(P_{n}) - \psi(P)}{|| \varphi_{\tilde{k}_{n}}(P)||_{L^{2}(P)} }$ is given that of $n^{-1/2} \sum_{i=1}^{n} \frac{ \varphi_{\tilde{k}_{n}}(P)(Z_{i})}{|| \varphi_{\tilde{k}_{n}}(P)||_{L^{2}(P)} }$. On the other hand, if no such sequence exists, the proposition readily implies a rate of convergence of the form $\left| \frac{ \psi_{\tilde{k}_{n}}(P_{n}) - \psi(P) }{|| \varphi_{\tilde{k}_{n}}(P)||_{L^{2}(P)}} \right| = O_{P} \left( n^{-1/2} + C^{2}_{n} \inf_{k \in \mathbb{K}_{n} } \left\{ \frac{\bar{\delta}_{1,k}(n) + \bar{\delta}_{2,k}(n) + \bar{B}_{k}(P) }{|| \varphi_{k}(P)||_{L^{2}(P)}} \right\} \right)$.

The sequence $(C_{n})_{n}$ quantifies the discrepancy of $k \mapsto || \varphi_{k}(P)||_{L^{2}(P)}$ within the grid. In cases where Assumption (ref)(ii) holds for all $k'$ and $k$ in $\mathbb{K}_{n}$, it readily follows that $C_{n} = 1 + |\mathbb{K}_{n}|^{-1} \sup_{k \in \mathbb{K}_{n}} \bar{\delta}_{2,k}(n)/||\varphi_{k}(P)||_{L^{2}(P)}$.

In order to shed more light on these expressions and the assumptions, we applied our results to the estimation of the integrated square PDF (example (ref)). In this setting, GineNickl2008 already provide a data-driven method to choose the bandwidth which is akin to ours. In fact, our method can be viewed as generalization of theirs to general regularizations, and the example illustrates that, at least in their setup, we do not have to pay an extra price for the added generality.

Conclusion

We propose an unifying framework to study the large sample properties of regularized estimators that extends the scope of the existing large sample theory for “plug-in" estimators to a large class containing regularized estimators. Our results suggest that the large sample theory for regularized estimators does not constitute a large departure from the existing large sample theory for “plug-in" estimators, in the sense that both are based on local properties of the mappings used for constructing the estimator. This last observation indicates that other large sample results developed for “plug-in" estimators can also be extended to the more general setting of regularized estimators; e.g., estimation of the asymptotic variance of the estimator and, more generally, inference procedure like the bootstrap. We view this as a potentially worthwhile avenue for future research.