The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
117,383 characters
De-Biased Machine Learning of Global and Local Parameters Using Regularized Riesz Representers
\begin{abstract}
We provide adaptive inference methods, based on $\ell_1$ regularization, for regular (semi-parametric) and non-regular (nonparametric) linear functionals of the conditional expectation function. Examples of regular functionals include average treatment effects, policy effects, and derivatives. Examples of non-regular functionals include average treatment effects, policy effects, and derivatives conditional on a covariate subvector fixed at a point. We construct a Neyman orthogonal equation for the target parameter that is approximately invariant to small perturbations of the nuisance parameters. To achieve this property, we include the Riesz representer for the functional as an additional nuisance parameter. Our analysis yields weak ``double sparsity robustness'': either the approximation to the regression or the approximation to the representer can be ``completely dense'' as long as the other is sufficiently ``sparse''. Our main results are non-asymptotic and imply asymptotic uniform validity over large classes of models, translating into honest confidence bands for both global and local parameters.
\keywords{Neyman orthogonality, Gaussian approximation, sparsity}
\end{abstract}
\section{Introduction}\label{sec:intro}
Many statistical objects of interest can be expressed as a linear functional of a regression function (or projection, more generally). Examples include global parameters: average treatment effects, policy effects from changing the distribution of or transporting regressors, and average directional derivatives, as well as their local versions defined by taking averages over regions of shrinking volume. This variety of important examples motivates the problem of learning linear functionals of regressions. Global parameters are typically regular (estimable at $1/\sqrt{n}$ rate), and local parameters are non-regular (estimable at slower than $1/\sqrt{n}$ rates). Global parameters can also be non-regular under weak identification (for example, in average treatment effects, when propensity scores accumulate mass near zero or one, along a given sequence of models).
Often the regression is high dimensional, depending on many variables such as covariates in a treatment effect model. Plugging a machine learner into a functional of interest can give a badly biased estimator. To avoid such bias, we use debiased/``double'' machine learning (DML) based on Neyman orthogonal scores that have zero derivative with respect to each first step learner (e.g.,
\cite{neyman1959,belloni2014pivotal,belloni2015uniform,chernozhukov2016locally,chernozhukov2018double,foster2019orthogonal}). Note that the word ``double'' emphasizes the connection to double robustness, a property which orthogonal scores have in this case. Such scores are constructed by adding a bias correction term: the average product of the regression residual with a learner of the functional's Riesz representer (RR). This construction builds upon and is directly inspired by \cite{newey1994asymptotic} where such scores arise in the computation of the semi-parametric efficiency bound for regular functionals.
We also remove overfitting bias (high entropy bias) by using cross-fitting, an efficient form of sample splitting, where we average over data observations different than those used by the nonparametric learners. See, e.g, \cite{schick1986asymptotically} for early use and \cite{chernozhukov2018double} for more recent use in the context of debiased
machine learning.
Using closed-form solutions for Riesz representers in several examples, \cite{chernozhukov2016locally,chernozhukov2018double} defined DML estimators in high dimensional settings and established their good properties. In comparison, the new approach proposed in this paper has the following advantages and some limitations:
\begin{enumerate}
\item We provide a novel algorithm based on $\ell_1$ regularization to \textit{automatically estimate} the Riesz representer from the empirical analog of equations that implicitly characterize it.
\item Even when a closed-form solution for the Riesz representer is available, the method avoids estimating each of its components. For example, the method avoids explicit density derivative estimation for the average derivative, and it avoids inverting estimated propensity scores for average treatment effects.
\item The adaptive inference theory covers both regular objects (estimable at the $1/\sqrt{n}$ rate) and nonregular ones (with rates $L/\sqrt{n}$, where
$L \to \infty$ is the operator norm of the linear functional).
\item As far as we know, the adaptive inference theory given here is the first non-asymptotic Gaussian approximation analysis of de-biased machine learning.
\item Our approach remains interpretable under misspecification, estimating a linear functional of the projection rather than regression. (This point is made explicit in Section~\ref{sec:estimation}).
\item We provide a non-asymptotic analysis when using the $\ell_1$-penalized method to learn the regression, and an asymptotic analysis when using other modern machine learning estimators to learn the regression.
\item The current analysis focuses on linear functionals. In follow-up work, \citet*{chernozhukov2018learning} extend the approach to nonlinear functionals through a linearization.
\end{enumerate}
This paper is a revised version of \citet*{chernozhukov2018robins} that gave an algorithm based on $\ell_1$ regularization for automatically estimating the Riesz representer. This version is distinguished from \citet*{chernozhukov2018robins}, \cite{chernozhukov2018double}, \cite{chernozhukov2016locally}, and \citet*{chernozhukov2018learning} in covering local objects that are estimated at a rate slower than $1/\sqrt{n}$. Providing debiased machine learning for such local objects is an important contribution of this paper.
Sections~\ref{sec:target} and~\ref{sec:applications} present the main ideas for a general audience. In Section~\ref{sec:target}, we define global, local, and perfectly localized linear functionals of the regression, and provide orthogonal representations for these functionals. In Section~\ref{sec:applications}, we present two empirical examples: local and global average treatment effects, and local and global average derivatives.
Sections~\ref{sec:estimation} and~\ref{sec:general} are theoretical. In Section~\ref{sec:estimation}, we provide estimation theory, demonstrating concentration and approximate Gaussianity of the DML estimator with regression and Riesz representer estimated via regularized moment conditions. We provide rates of convergence for estimating the Riesz representer, giving both fast rates under approximate sparsity and slow rates under the dense model. In Section~\ref{sec:general}, we demonstrate asymptotic consistency and Gaussianity of the DML estimator with regression estimated via general machine learning.
The supplement provides supporting material. In Section~\ref{sec:related}, we give a detailed account of how our work relates to previous and contemporary work. In Section~\ref{sec:preliminary}, we review prelimaries of functional analysis. In Section~\ref{sec:examples}, we analyze the structure of the leading examples, providing bounds on operator norm, variance of the score, and kurtosis. Finally, we provide proofs for each section.
\section{Overview of target functionals, orthogonal representation, estimation, and inference}\label{sec:target}
\subsection{Target functionals}
We consider a random element $W$ with distribution $P$ taking values $w$ in its support $\mathcal{W}$. Denote the $L^q(P)$ norm of a measurable function $f: \mathcal{W} \to \Bbb{R}$ and also the $L^q(P)$ norm of random variable $f(W)$ by $\| f \|_{P,q} = \| f(W)\|_{P,q}$. For a differentiable map $x \mapsto g(x)$, from $\mathbb{R}^d$ to $\mathbb{R}^k$, we use $\partial_{x'} g$ to abbreviate the partial derivatives $(\partial/\partial x') g(x)$, and we use $\partial_{x'} g(x_0)$ to mean $\partial_{x'} g (x) \mid_{x = x_0}$, etc. We use $x'$ to denote the transpose of a column vector $x$.
Let $(Y,X)$ denote a random sub-vector of $W$ taking
values in their support sets, $y \in \mathcal{Y} \subset \Bbb{R}$ and $x \in \mathcal{X} \subset \Bbb{R}^{d_x}$, where $d_x=\infty$ is allowed. Let $F$ denote the law of $X$. We define $$x \mapsto \gamma_0^\star(x) := {\mathrm{E}} [Y\mid X=x],$$ as the unknown regression function of $Y$ on $X$. We consider the convex parameter space $\Gamma_0$ for $\gamma_0^\star$ with
elements $\gamma$. (Later, in the theoretical sections, we generalize and replace the regression function by a projection).
Our goal is to construct high-quality inference methods for real-valued linear functionals of $\gamma_0^\star$. To present examples below we need to endow $\gamma_0^\star$ with a causal interpretation, which requires us to assume that it is a structural function, invariant to the changes in the distribution of $X$ under policies described below. This property is not guaranteed for an arbitrary regression problem. For the reader who is unfamiliar with these concepts, we note that a simple sufficient condition for invariance is follows: given a stochastic process $x \mapsto Y(x)$, called potential outcomes or structural function, vector $X$ is generated to follow distribution $F$ independently of $x \mapsto Y(x)$ and $Y$ is generated as $Y = Y(X)$. In this case we have $\gamma_0^\star(x) = {\mathrm{E}} Y(x)$ for any $F$. This condition is conventionally called exogeneity in econometrics and random assignment in statistics. The measurability requirement here is that $(x, \omega)\mapsto Y(x,\omega)$ is a measurable map. We refer to \cite{imbens2015causal}, \cite{hernan2019causal}, and \cite{peters2017elements} for the relevant formalizations that enable causal interpretation.
\begin{example}[Average treatment effect]\label{ex:ate} Let
$X = (D,Z)$ and $\gamma_0^\star(X) = \gamma_0^\star(D,Z)$, where $D \in \{0,1\}$ is the indicator of the receipt of the treatment.
Define $$
\theta^\star_0 = \int (\gamma_0^\star(1,z) - \gamma_0^\star(0,z)) \ell(x) d F(x),$$
where $x \mapsto \ell(x)$ is a weighting function. This statistical parameter is a weighted average treatment effect under the standard conditional exogeneity assumption, which guarantees that $\gamma_0^\star$ is invariant to changes in the
distributions of $D$ conditional on $Z$. The assumption requires $D$ to be independent of the potential outcome process $d \mapsto Y(d,Z)$ and outcome to be generated as $Y= Y(D,Z)$, so that $\gamma_0^\star(d,z) = {\mathrm{E}} [Y(d,Z) \mid Z=z]$. Here $\gamma_0^\star$ is invariant to changes in the conditional distributions of $D$, but not to the changes in the distribution of $Z$.
\end{example}
Here and below, a weighting function is a measurable function $x \mapsto \ell(x) $ such that $\int \ell dF = 1$
and $\int \ell^2 dF < \infty$. In this example, setting
\begin{itemize}
\item $\ell(x) = 1$ gives average treatment effect (ATE) in the entire population,
\item $\ell(x)= 1(d=1)/P(D=1)$ gives the ATE for the \textit{treated} population,
\item $\ell(x) = 1( z \in N)/P(Z \in N)$ the ATE
conditional on $Z$ in the neighborhood $N$,
\end{itemize}
and so on. We can model small neighborhoods $N$ as shrinking in volume with the sample size. The local weighting
and kernel weighting discussed below are applicable to all key examples. Moreover, they are combinable with other weighting functions so that, for example, we can target inference on local average treatment effects for the treated.
\begin{example}[Policy effect from changing distribution of $X$]\label{ex:policy1} The average causal effect of the policy that shifts the distribution of covariates from $F_0$ to $F_1$ with the support contained in $\mathcal{X}$, when $\gamma_0^\star$ is invariant over $\{F, F_0, F_1\}$, for the weighting function $x \mapsto \ell(x)$, is given by:
$$
\theta^\star_0 = \int \gamma_0^\star(x) \ell(x) d G( x); \quad G(x) = F_1( x) - F_0(x).
$$
Exogeneity is a sufficient condition for the stated invariance of $\gamma_0^\star$.
\end{example}
\begin{example}[Policy effect from transporting $X$]\label{ex:policy2} A weighted average effect of changing covariates $X$ according to a transport map $X \mapsto T(X)$, where $T$ is deterministic measurable map from $\mathcal{X}$ to $\mathcal{X}$, with the weighting function $x \mapsto \ell(x)$, is given by:
$$
\theta^\star_0 = \int[ \gamma_0^\star(T(x)) - \gamma_0^\star(x) ]\ell(x) d F(x).
$$
This has a causal interpretation if the policy induces the \textit{equivariant} change in the regression function, namely the outcome $\tilde Y$ under the policy obeys
${\mathrm{E}} [\tilde Y |X] = \gamma_0^\star(T(X))$. Exogeneity is a sufficient condition. \end{example}
\begin{example}[Average directional derivative]\label{ex:deriv} In the same settings as the previous example, a weighted average derivative of a continuously differentiable $\gamma_0$ with respect to component vector $d$ in the direction $d \mapsto t(x)$ and weighed by $x \mapsto \ell(x)$ is the linear functional of the form:
$$
\theta^\star_0 = \int \ell(x) t(x) '\partial_d \gamma_0^\star(d,z) dF(x).
$$
In causal analysis, $\theta^\star_0$ is an approximation to $1/r$ times the average causal effect of the policy that shifts
the distribution of covariates via the map $X=(D,Z) \mapsto T(X) = (D + r t(X), Z)$ for small $r$, weighted by $\ell(X)$.
Here we require that $(d,x) \mapsto \partial_d \gamma_0^\star(x)$ exists and is continuous on $\mathcal{X}$.
\end{example}
In this example, consider the case when $X=(D,Z)$ consists of continuous treatment variable $D$ and
covariates $Z$. Further suppose $\ell(x)=\ell(d)$ and $t(x)=1$. Then the parameter of interest
is $\theta_{0}^{\ast}={\mathrm{E}}[\ell(D)T(D)],$ where $T(d)={\mathrm{E}}[\partial_d\gamma_{0}^{\ast
}(D,Z)|D=d].$ When $Y=Y(D)$ for a potential outcome process $Y(d)$
that is independent of treatment $D$ conditional on the covariates $Z$ and
differentiable in $d,$ it was shown by \cite{altonji2005cross} and \cite{florens2008identification} that $T(d)={\mathrm{E}}[\partial_d Y(D)|D=d],$ which is an average
treatment effect on the treated. Thus $\theta_{0}^{\ast}$ is a weighted
average of the effect of treatment on the treated and would be equal to $T(d)$
for the perfectly localized $\ell(d)=1(D=d)/f_{D}(d),$ where $f_{D}(d)$ is the
pdf of $D.$ Also for $\ell(d)\equiv1$, \cite{imbens2009identification} showed that
$\theta_{0}^{\ast}={\mathrm{E}}[T(D)]={\mathrm{E}}[\partial_d Y(D)]$, which is an average
treatment effect. See also \cite{rothenhausler2019incremental}.
In Example~\ref{ex:deriv}, we consider the case where the variable of differentiation is also the variable of localization. As explained above, this case corresponds to effects of continuous treatments, and it turns out to require extra care in Section~\ref{sec:examples}. The other possible case is where the variable of differentiation is different than the variable of localization. Such a case turns out to be simpler and is handled by similar arguments as Examples~\ref{ex:ate},~\ref{ex:policy1}, and~\ref{ex:policy2} in Section~\ref{sec:examples}.
All of these statistical parameters play an important role in causal inference, counterfactual decompositions, and predictive analyses. Introduction of the weighting function $\ell(X)$ allows us to study subgroup effects and local effects, and these will be covered by our non-asymptotic results and asymptotic results. All of the above examples can be viewed as real-valued linear functionals of the regression function.
\begin{definition}[Target parameter] Our target is the real-valued linear functional of $\gamma_0^\star$:
\begin{equation} \label{eq: def theta}
\theta^\star_0= \theta(\gamma_0^\star), \text{
where } \gamma \mapsto \theta (\gamma) := {\mathrm{E}} m(W, \gamma),\end{equation}
$\gamma \mapsto m(w, \gamma)$ is a linear operator for each $w \in \mathcal{W}$, defined on $\Gamma= \mathrm{span}(\Gamma_0)$, and the map $w \mapsto m(w, \gamma)$ is measurable with finite second moment under $P$ for each $\gamma \in \Gamma$.
\end{definition}
The linear operator $\gamma \mapsto \theta(\gamma)$ has the following generating function $m$ in these examples:\begin{enumerate}
\item[2.1] $m(w, \gamma) = (\gamma(1,z) - \gamma(0,z)) \ell(x) $;
\item[2.2] $m (w, \gamma) = m(\gamma) = \int \gamma(x) \ell(x) d G(x); \ \ G(x) = F_1(x) - F_0(x)$;
\item[2.3] $m(w, \gamma) = \ell(x)(\gamma(T(x)) - \gamma(x))$;
\item[2.4] $m(w, \gamma) = \ell(x) t(x)'\partial_d \gamma(x)$.
\end{enumerate}
In these examples, we can recognize the dependency on the weighting function by writing
$
m(w, \gamma; \ell).
$
In Examples~\ref{ex:ate},~\ref{ex:policy2}, and~\ref{ex:deriv} we can decompose
$m(w, \gamma;\ell) = m_0(w, \gamma)\ell(x).$
Estimation of some parameters of the form in Definition 2.1 is very straightforward, such as ${\mathrm{E}}[w(X)\gamma_0(X)]$ for a known function $w(x)$. These can be estimated as the sample mean of $w(X)Y$. Such simple estimation is not possible for the causal, counterfactual parameters in Examples~\ref{ex:ate},~\ref{ex:policy1},~\ref{ex:policy2}, and~\ref{ex:deriv}. The approach of this paper provides estimators for these counterfactual parameters and can be used for many others.
Our local functionals are defined by using the weight function that localizes the functionals around value $d_0$ of a low-dimensional vector component $D$. Here $D$ is a $p_1$-dimensional component of vector $X$.
We consider the weighting function
\begin{equation}
\ell_h(D) = \frac{1}{h^{p_1}} K\left( \frac{d_0-D}{h}\right)/\omega, \quad \omega = {\mathrm{E}} \left[ \frac{1}{h^{p_1}} K\left( \frac{d_0-D}{h}\right) \right], \quad h \in \Bbb{R}_+,
\end{equation}
where $K: \Bbb{R}^{p_1} \to \Bbb{R}$ is a kernel function of order $\mathsf{o}$ such that $\int K = 1$ and
$$\int (\otimes^m u) K(u) du = 0, \quad \text{ for } m=1,..., \mathsf{o}-1,$$ with its support contained in the cube $[-1,1]^{p_1}$. The simplest example is the box kernel with $K(u) = \times_{j=1}^{p_1}1(-1 < u_j < 1)/2$,
which is of order $\mathsf{o} = 2$.
\begin{remark} For the case of Example 1, the localization is understood to be with respect to any continuous component $\tilde D$ of covariate vector $Z$, but of course not the binary treatment $D$.
\end{remark}
To present the main results in the most clear way, we assume that $\ell_h$ is known,
i.e. $\omega$ is known. Our main results also hold for one sided kernels. We leave to future work the application of this theory to settings with one sided limits, e.g. regression discontinuity design.
\begin{definition}[Local and localized functionals]
We consider the local functional
$$
\theta(\gamma_0^\star; \ell_h) := {\mathrm{E}} m(W, \gamma_0^\star; \ell_h),
$$
as well as the (perfectly) localized functional
$$
\theta(\gamma_0^\star; \ell_0) := \lim_{h \to 0} \theta(\gamma_0^\star; \ell_h).
$$
\end{definition}
The difficulty in targeting
localized functionals is that they are not pathwise differentiable. A key quantity in the analysis is the operator norm (the modulus of continuity) of $\gamma \mapsto \theta (\gamma)$ on $\Gamma$, defined as
\begin{equation} L:= \sup_{\gamma \in \Gamma\setminus\{0\}} |\theta(\gamma)|/\| \gamma\|_{P,2}.
\end{equation}
We consider $L=\infty$ as non-regular cases, e.g. perfectly-localized functionals. We also consider cases
where $L \to \infty$ as $n \to \infty$ as non-regular. Indeed, the latter case arises from approximating the functional
with $L=\infty$ by functionals where $L \to \infty$, e.g. local functionals with $h \to 0$.
The $L=\infty$ case also arises in triangular array asymptotics where $P$ changes with $n$.
The asymptotic thought experiment, where $L \to \infty$, approximates
non-asymptotic cases where $L$ is high. We emphasize that we derive both non-asymptotic
results and their asymptotic corollaries (which lead to simplified statements conveying key qualitative features of
non-asymptotic results).
\subsection{Building an orthogonal representation of the target functional}
Equation~(\ref{eq: def theta}) can be thought of as a direct formulation of the target parameter. Next we introduce a dual formulation and finally an orthogonal formulation. Towards this end, we define the Riesz representer $\alpha_0$.
\begin{definition}[Linear and minimal linear representer] A linear representer (also called a Riesz representer) for the linear functional $\gamma \mapsto \theta(\gamma)$ is $\alpha_0 \in L^2(F)$ such that
\begin{equation}\label{eq: Riesz}
\theta(\gamma) ={\mathrm{E}} \gamma(X) \alpha_0(X), \text{ for all } \gamma \in \Gamma.
\end{equation}
If $\alpha_0 \in \bar \Gamma := \text{closure}(\Gamma)$ in $L^2(F)$, we call it the minimal representer and denote it by $\alpha^\star_0$; if not, we call it a representer. Any representer can be reduced to the minimal representer by projecting it onto $\bar \Gamma$.
\end{definition}
A minimal linear representer exists if and only if $L< \infty$, as a consequence of the Riesz--Frechet theorem; see Lemma~\ref{lemma: ERR} below. Therefore, when $L< \infty$, we define the following dual linear representation for the target parameter\begin{equation}\label{RRR}
\theta^\star_0 = \theta(\alpha_0^\star); \ \ \theta(\alpha) := {\mathrm{E}} [\alpha(X) Y]. \end{equation}
To motivate the upcoming orthogonal representation, we note that either the direct or the dual identification strategies can be used for direct plug-in estimation, but this does not give good estimators, as explained in the following technical remark.
\begin{remark}[Non-orthogonality of direct and dual formulations]Even if we knew expectation operator ${\mathrm{E}}$ and use $\theta(\hat \gamma)$ or $\theta(\hat \alpha)$ as the estimator for $\theta^\star_0$, this estimator would
have high biases. Indeed, neither $\gamma \mapsto \theta(\gamma)$ nor $\alpha \mapsto \theta(\alpha)$ are
orthogonal to local perturbations $h \in \Gamma$ of $\gamma_0^\star$ or $\bar h \in \Gamma$ of $\alpha_0^\star$, namely $$
\partial_t \theta(\gamma_0^\star + t h) \Big |_{t =0} = {\mathrm{E}} m (W, h) \neq 0, \ \
\partial_t \theta(\alpha_0^\star + t \bar h ) \Big |_{t =0} ={\mathrm{E}} \gamma_0^\star (X) \bar h(X) \neq 0.
$$
Consequently, the quantities ${\mathrm{E}} m (W, \hat \gamma - \gamma_0^\star)$ and $ {\mathrm{E}} \gamma_0^\star (\hat \alpha - \alpha_0^\star)$
are first order biases for $\theta(\hat \gamma)$ and $\theta(\hat \alpha)$. The regularized estimators $\hat \gamma$ or $\hat \alpha$ exploit structure of $\gamma_0^\star$
and $\alpha_0^\star$ to estimate them well in high-dimensional problems, but they exhibit biases that vanish at rates slower than $1/\sqrt{n}$, which makes $\theta(\hat \gamma)$ and $\theta(\hat \alpha)$ converge at the same slow rate.
\end{remark}
Therefore we proceed to construct another representation for $\theta^\star_0 $ that has the required Neyman orthogonality structure.
\begin{definition}[Orthogonal representation for the target functional]
We have \begin{equation}\label{DR}
\theta^\star_0 = \theta(\alpha_0^\star,\gamma_0^\star); \ \ \theta(\alpha, \gamma) := {\mathrm{E}} [m(W, \gamma) + \alpha(X) (Y - \gamma(X))],
\end{equation}
where $(\alpha, \gamma)$ are the nuisance parameters with the true value $(\alpha_0^\star,\gamma_0^\star)$.
\end{definition}
Unlike the direct or dual representations for the functional, this representation is Neyman orthogonal to perturbations $(\bar h, h)$ $\in \Gamma^2$ of $(\alpha_0^\star,\gamma_0^\star)$ such that
\begin{equation}\label{eq:invariance}
\frac{\partial}{\partial t} \theta(\alpha_0^\star + t \bar h, \gamma_0^\star + t h) \Big |_{t=0} = {\mathrm{E}} m(W, h) - {\mathrm{E}} \alpha_0^\star(X) h(X) + {\mathrm{E}} [(Y - \gamma_0^\star(X)) \bar h(X)] = 0.
\end{equation}
In fact, a stronger property holds
\begin{equation}
\theta(\alpha, \gamma) - \theta(\alpha^\star_0, \gamma^\star_0) = - \int (\gamma - \gamma^\star_0) ( \alpha - \alpha_0^\star) dF,
\end{equation}
which implies (\ref{eq:invariance}) as well as double robustness.
The Neyman orthogonality property states that the representation of the target parameter $\theta_0$ in terms
of the nuisance parameters $(\alpha, \gamma)$ is invariant to the local perturbations of the values of the nuisance parameter.
This property makes the orthogonal representation an excellent basis for constructing high quality point and interval estimators of $\theta^\star_0$ in modern high-dimensional settings when we will be plugging-in biased estimators in lieu of $\gamma_0^\star$ and $\alpha_0^\star$, where the bias occurs because of the regularization (see, e.g., \cite{chernozhukov2016locally,chernozhukov2018double}).
Both $\gamma_0^*$ and $\alpha_0^*$ are identified, $\gamma_0^*$ as ${\mathrm{E}}[Y|X]$ and $\alpha_0^*$ by virtue of the consistent estimator we give in Section~\ref{sec:preview}. Identification allows us to use the orthogonal representation to estimate target parameters.
\subsection{The case of finite-dimensional linear regression}\label{sec:linear} It is instructive to consider the case of linear finite-dimensional regression. Consider $x \mapsto b(x) = \{ b_j(x)\}_{j=1}^p$ as a $p$-dimensional dictionary of basis
functions with $b_j \in L^2(F)$ for each $j=1,...,p$. The regression function is assumed to obey the linear functional form $\gamma^\star_0 = b'\beta_0$ for some $\beta_0$. Also define
$$
G= {\mathrm{E}} b(X)b(X)', \quad M = {\mathrm{E}} m(W, b).
$$
First, observe that for $\gamma = b'\beta$,
$$\theta (\gamma) = {\mathrm{E}} m(W, b'\beta) = {\mathrm{E}} m(W, b)' \beta = M \beta.$$ For instance, in Examples~\ref{ex:ate},~\ref{ex:policy1},~\ref{ex:policy2}, and~\ref{ex:deriv}:
\begin{enumerate}
\item[] 2.1 $M = {\mathrm{E}} (b(1,Z) - b(0,Z))\ell(X)$\quad
2.2 $M= \int b \ell (d F_1- dF_0)$,
\item[] 2.3 $M= {\mathrm{E}} (b(T(X)) - b(X) )\ell(X)$\quad
2.4 $M = {\mathrm{E}} \partial_d b(D,Z) t(X) \ell(X)$.
\end{enumerate}
Second, we make a guess that the linear representer $\alpha^\star_0$ to be of the form
$
\alpha^\star_0(x)= b(x)' \rho_0,
$
for $\rho_0$ defined below. We can define the parameters
$\beta_0$ and $\rho_0$ as any minimal $\ell_1$-norm solutions to the system of equations:
\begin{equation}\label{eq: linear}
\min \|\beta\|_1 + \| \rho\|_1: \quad G \beta = {\mathrm{E}} Y b(X) , \quad G \rho = M.
\end{equation}
In particular, if $G$ is full rank, the solutions are
$\beta_0= G^{-1} {\mathrm{E}} b(X) Y$
and $\rho_0 = G^{-1} M$.
We now verify the representation property for our guess:
$$
{\mathrm{E}} \gamma(X) \alpha^\star_0(X) = {\mathrm{E}} \beta' b(X)b(X)'\rho_0 = \beta' G \rho_0 = \beta'M = \theta(\gamma),
$$
for all $\beta$'s and hence all $\gamma$'s. The operator norm of $\theta (\gamma) = M'\beta$ is given by
$$
L= \sup_{\beta \in \Bbb{R}^p\setminus \{0\}}\frac{ | M'\beta| }{\sqrt{\beta'G\beta}} =
\sup_{\beta \in \Bbb{R}^p\setminus \{0\}} \frac{|\beta'G\rho_0|}{\sqrt{\beta'G\beta}} = \sqrt{\rho_0' G \rho_0} < \infty.
$$
We conclude that direct, dual, and orthogonal representations are given by
$$
\theta(\gamma) = M'\beta; \quad \theta(\alpha) = \rho' {\mathrm{E}} b(X)Y ; \quad \theta(\gamma, \alpha) =
M'\beta + \rho' {\mathrm{E}} b(X)Y - \rho' G\beta,
$$
where $\beta$ is $\gamma$'s parameter and $\rho$ is $\alpha$'s parameter. These representations appear to be both novel
and useful.
\subsection{The case of infinite-dimensional regression} In the infinite-dimensional case, we can employ the Riesz--Frechet representation theorem and Hahn--Banach extension theorem to establish existence of the linear Riesz representer.
\begin{lemma}[Extended Riesz representation]\label{lemma: ERR} (i) If $L< \infty$, there exists a unique minimal
representer $\alpha_0^\star \in \bar \Gamma$ and $L= \| \alpha^{\star }_0\|_{P,2}$. (ii) If there exists a linear representer $\alpha_0$ on $\Gamma$ with $\|\alpha^{}_0\|_{P,2} < \infty$, then $L = \|\alpha^{\star }_0\|_{P,2} \leq \|\alpha^{}_0\|_{P,2} < \infty$, where
$\alpha^{\star}_0$, obtained by projecting $\alpha_0$ onto $\bar \Gamma$, is the unique minimal representer. In both cases
$\gamma \mapsto \theta(\gamma)$ can be extended to $\bar \Gamma$ or to the entire $L^2(F)$ with the modulus of continuity $L$.
\end{lemma}
The first part of the lemma shows (implicit) existence of a linear representer when $L < \infty$. Our estimation results will rely \textit{only on the existence} of minimal representers. In some cases, however, we may utilize the closed-form solutions for linear representers (see, e.g., Section~\ref{sec:examples} for the key examples), to improve the basis functions for estimating the minimal representers. There is also an efficiency reason to work with minimal representers rather than any linear representer, as highlighted in Section~\ref{sec:estimation} analyzing semi-parametric efficiency.
\subsection{Informal preview of estimation and inference results}\label{sec:preview}
Our estimation and inference will exploit empirical analogs of both
the orthogonal representation of the parameter (\ref{DR}) and the equation defining the RR property (\ref{eq: Riesz}).
To approximate the regression function and the RR, we consider the $p$-vector of dictionary functions $b$, where the dimension $p$ of the dictionary can be large, potentially much larger than $n$. We approximate $\alpha^\star_0$ by a linear form $b'\rho_0$, and we approximate
$\gamma^\star_0$ by a linear form $b'\beta_0$, and estimate the parameters using the algorithms below.
\begin{enumerate}
\item Let $(W_i)_{i=1}^n= (Y_i, X_i)_{i=1}^n$ denote i.i.d. copies of data vector $W$. We use cross-fitting to avoid biases from overfitting that can arise in high-dimensional settings. To this end, let $(I_{1},..., I_K)$ be a partition of the
observation index set $\{1,...,n\}$ into $K$ distinct subsets of about equal
size. Let $\mathbb{E}_A f = \mathbb{E}_A f(W)$ denote the empirical average of $f(W)$ over $i \in A \subset \{1,..., n\}$: $\mathbb{E}_{A} f := \mathbb{E}_{A} f(W) = | A | ^{-1} \sum_{i \in A} f(W_i).$
\item For each block $k=1,...,K$, we obtain generalized Dantzig selector (GDS) estimates $\hat \alpha_k= b'\hat \rho_k$ and $\hat \gamma_k= b'\hat\beta_k$, where
\begin{equation}
\begin{array}{l} \hat \rho_k = \arg \min_{\rho \in \Bbb{R}^p} \| \rho\|_1 : \| \hat{D}^{-1}\left\{\mathbb{E}_{I^c_k} m(W, b) - \mathbb{E}_{I^c_k} b(X) b(X)'\rho\right\} \|_\infty \leq \lambda_\rho,\\
\hat \beta_k = \arg \min_{\beta \in \Bbb{R}^p} \| \beta \|_1 : \| \hat{D}^{-1}\left\{\mathbb{E}_{I^c_k} (Y- b(X)'\beta) b(X))\right\}\|_\infty \leq \lambda_\beta,
\end{array}
\end{equation}
where $I^c_k = \{1,...,n\}\setminus I_k$ is the set of observation indices leaving $I_k$ out, $\lambda$'s are tuning parameters, and $\hat{D}$ is a scaling detailed in Section~\ref{sec:diff}. Typically $\lambda$'s scale like $\sqrt{ \log (p \vee n)/n}$; Section~\ref{sec:applications} provides concrete choices.
\item The DML estimator is an average of estimated orthogonal representations over $k$:
\begin{equation}
\hat{\theta}=\frac{1}{n}\sum_{k=1}^{K}\sum_{i\in I_{k}}\{m(W_{i}
,\hat{\gamma}_{k})+\hat{\alpha}_{k}(X_{i})[Y_{i}-\hat{\gamma}_{k
}(X_{i})]\}. \label{Estimator}
\end{equation}
The estimator of its asymptotic variance is
\begin{equation}
\hat{\sigma}^2=\frac{1}{n}\sum_{k=1}^{K}\sum_{i\in I_{k}}\{m(W_{i}
,\hat{\gamma}_{k})+\hat{\alpha}_{k}(X_{i})[Y_{i}-\hat{\gamma}_{k
}(X_{i})]-\hat{\theta}\}^2. \label{Estimator_var}
\end{equation}
\end{enumerate}
We remark that the RR estimator in step 2 is of Dantzig selector type, but is not exactly the Dantzig selector, requiring some new analysis. We use the GDS rather than series or spline estimation to accommodate high dimensional specifications for the regression and Riesz representer.
The dictionary $b(x)$ is very important for the GDS estimator. This dictionary should be chosen so that linear combinations of $b(x)$ can approximate in mean square any element of $\Gamma$. For example if $\Gamma$ is the set of linear combinations of an infinite sequence of regressors, as for a high dimensional regression, then $b(x)$ could be chosen as the first $p$ elements of that sequence. Also $p$ can be chosen flexibly, because $p$ will be allowed to grow faster than the sample size, as specified in the asymptotic theory to follow. In practice multiple choices of $p$ could be tried.
Next, we state the key concentration and approximate Gaussianity results informally. Key quantities in the analysis are the ``true'' score and its moments:
$$
\psi_{0}^\star(W) := \theta_0^\star-m(W, \gamma_0^\star)-\alpha^\star _0(X)(Y- \gamma_0^\star(X)),\ \ \sigma^2 := {\mathrm{E}} \psi^2_{0}(W), \ \ \kappa^3 := {\mathrm{E}} |\psi^3_{0}(W)|.
$$
We establish conditions under which
\begin{equation}\label{eq:SC-1}
\| \hat \gamma_k - \gamma_0^\star\|_{P,2} + \| \hat \alpha_k - \alpha_0^\star\|_{P,2}/\sigma \to 0,\quad
\sqrt{n}\int (\hat \gamma_k - \gamma^\star_0) (\hat \alpha_k - \alpha^\star_0) dF/\sigma \to 0.
\end{equation}
These include a bound on the $\ell_1$ norm of coefficients and that either the regression function or
the RR is approximately sparse with the effective dimension $s$ less than $\sqrt{n}$. This allows either nuisance parameter to be completely dense.
Given that (\ref{eq:SC-1}) holds, we establish that the resulting de-biased (or ``double'') machine learning (DML) estimator $\hat \theta$ is \textit{adaptive}, namely it is approximated up to the error $o(\sigma/\sqrt{n})$ by the oracle estimator
$$
\bar \theta := \theta^\star_0 - n^{-1} \sum_{i =1}^n \psi_0(W_i),
$$
where the \textit{oracle} knows the scores $\psi_0$. Hence the approximate deviation of $\hat \theta$ from $\theta_0^\star$ is determined by $\|\psi_{0}\|_{P,2}/\sqrt{n}$, which is the standard deviation of the oracle estimator.
Consequently, $\hat \theta$ concentrates in a $\sigma/\sqrt{n}$ neighborhood of the target with deviations controlled by the normal laws,
$$
\sup_{t \in \Bbb{R} }\Big |{\mathrm{P}} ( \sqrt{n}(\hat \theta - \theta^\star_0)/ \sigma \leq t) - {\mathrm{P}}( N(0,1) \leq t) \Big | \leq A (\kappa/\sigma)^3/\sqrt{n} + \mathrm{error}_n \to 0,
$$
where the $\mathrm{error}_n$ bound is non-asymptotic and tends to zero as $n \to \infty$. Of course, $\sigma/ \sqrt{n} \to 0$ is required
for concentration. The non-asymptotic bound automatically implies the uniform validity of results over large classes of probability laws $P$ for $W$.
There are two cases to consider:
\begin{enumerate}
\item\textsc{Regular case}: \textit{the parameters $\sigma$, $\kappa/\sigma$, and $L$ are bounded, leading to $1/\sqrt{n}$ concentration, adaptation, and Gaussian approximation.}
\item \textsc{Non-regular case}, \textit{the parameters $\sigma, \kappa/\sigma$, and $L$ diverge, so that we need
$$
\sigma/\sqrt{n} \to 0, L/\sqrt{n} \to 0, (\kappa/\sigma)/\sqrt{n} \to 0,
$$
for $\sigma/\sqrt{n}$ concentration, adaptation, and Gaussian approximation.}
\end{enumerate}
As we show in Section~\ref{sec:examples}, in the case of local functionals, the latter condition can be more succinctly stated as
$$
(\kappa/\sigma) \lesssim \sigma \asymp L , \quad L/\sqrt{n} \to 0.
$$
Finally, we establish that we can \textit{transfer} learning and inference guarantees for local functionals to
those for the \textit{(perfectly) localized} functionals if the localization bias is sufficiently small, namely
$$
\sqrt{n} (\theta(\gamma_0^\star; \ell_h) - \theta(\gamma_0^\star; \ell_0)/\sigma \to 0.
$$
We think it is remarkable that a single inference theory covers both regular and non-regular cases,
and provides uniform validity over large classes of $P$.
\section{Applications}\label{sec:applications}
\subsection{Global and local effects of 401(k) eligibility on net financial assets}
First, we use our method to answer a question in household finance: what is the average treatment effect of 401(k) eligibility on net financial assets (over a horizon of about two years)? We follow the identification strategy of \cite{poterba1994} and \cite{poterba1995}, who assume selection on observables. The authors assume that when 401(k) was introduced, workers ignored whether a given job offered 401(k) and instead made employment decisions based on income and other observable job characteristics; after conditioning on income and job characteristics, 401(k) eligibility was exogenous at the time. This empirical question corresponds to Example~\ref{ex:ate}.
We use data from the 1991 US Survey of Income and Program Participation (\cite{chernozhukov2018double_data}), using sample selection and variable construction as in \cite{abadie2003semiparametric} and \cite{chernozhukov2004effects}. The outcome $Y$ is net financial assets defined as the sum of IRA balances, 401(k) balances, checking accounts, US saving bonds, other interest-earning accounts, stocks, mutual funds, and other interest-earning assets minus non-mortgage debt. The treatment $D$ is an indicator of eligibility to enroll in a 401(k) plan. The raw covariates $X$ are age, income, years of education, family size, marital status, two-earner status, benefit pension status, IRA participation, and home-ownership. We impose common support of the propensity score for the treated and untreated groups based on these covariates, yielding $n=9869$ observations. We consider the fully-interacted specification $b(D,X)$ of \cite{chernozhukov2018double} with $p=277$ including polynomials of continuous covariates, interactions among all covariates, and interactions between covariates and treatment status.
Tables~\ref{table_ATE} and~\ref{table_ATE2} summarize results for the entire population and for each quintile of the income distribution. We use $K=5$ folds in cross-fitting. To estimate the RR, we use the generalized Dantzig selector (GDS) procedure introduced in the present work. To estimate the regression, we use GDS, Lasso, random forest, or neural network. GDS is implemented using the tuning procedure described in Section~\ref{sec:diff}. Lasso is implemented using the tuning procedure described in \citet*{chernozhukov2018learning}. Random forest and neural network are implemented with the same settings as \cite{chernozhukov2018double}, i.e. with 1000 trees or a single hidden layer of eight neurons, respectively. We find ATE of $7608$ $(1395)$ using GDS for both the RR and the regression. This ATE estimate is stable across different choices of regression estimator. We find that localized ATE is not statistically significant for the second quintile, and it is statistically significant, positive, and strongly heterogeneous for the other quintiles. Interpreting the relatively high effect of 401(k) eligibility for the first quintile is a question for future research.
For comparison, \cite{chernozhukov2018double} report ATE of $7170$ $(1398)$ by DML, which estimates the RR by estimating the propensity score and plugging it into the RR functional form. Though these two estimators are asymptotically equivalent under correct specification, our estimator avoids the estimated propensity score in the denominator which could cause numerical instability. The ATE results are broadly consistent with \cite{poterba1994} and \cite{poterba1995}, who use a simpler specification motivated by economic reasoning. The localized ATE estimates by income quintile group appear to be new empirical results and are of interest in their own right. In Section~\ref{sec:diff} we report analogous estimates without debiasing. Without debiasing, the GDS and Lasso estimates of ATE are attenuated due to regularization. The bias is smaller for the random forest and neural network estimates of ATE.
\begin{table}[h]
\centering
\caption{Average treatment effect of 401(k) eligibility on net financial assets. Localized average
treatment effects are reported by income quintile groups. The regression is estimated by GDS or Lasso. Standard errors are reported in parentheses.}
\begin{tabular}{@{}ccccccc@{}}
\hline
Income quintile & N treated & N untreated & \multicolumn{2}{c}{GDS} &\multicolumn{2}{c}{Lasso} \\
\hline
All & 3682 & 6187 & 7607.95 & (1394.92)& 7733.31 & (1416.46)\\
1 & 272 & 1702 & 4500.33 & (924.12)&4477.43 & (920.31)\\
2 & 527 & 1447 & 1051.60 & (1501.03)& 1119.06 & (1500.78)\\
3 & 755 & 1219 & 5204.93 & (1199.87)&4919.65 & (1200.10)\\
4 & 962 & 1012 & 9515.58 & (2141.92) & 8837.39 & (2150.58)\\
5 & 1166 & 807 & 19354.00 & (7934.70)& 14138.37 & (8310.59)\\
\hline
\end{tabular}
\label{table_ATE}
\end{table}
\begin{table}[h]
\centering
\caption{Average treatment effect of 401(k) eligibility on net financial assets. Localized average
treatment effects are reported by income quintile groups. The regression is estimated by random forest or neural network. Standard errors are reported in parentheses.}
\begin{tabular}{@{}ccccccc@{}}
\hline
Income quintile & N treated & N untreated & \multicolumn{2}{c}{Random forest}&\multicolumn{2}{c}{Neural network} \\
\hline
All & 3682 & 6187 & 8638.15 & (1621.78)& 7364.66 & (1844.39)\\
1 & 272 & 1702 & 4874.49 & (937.86)&4664.61 & (1309.59)\\
2 & 527 & 1447 & 1957.72 & (1738.61)& 1635.69 & (1603.19)\\
3 & 755 & 1219 & 3973.11 & (1474.72)&5106.03 & (1287.70)\\
4 & 962 & 1012 & 10056.79 & (2375.44) & 9529.03 & (2205.61)\\
5 & 1166 & 807 & 21168.13 & (8015.79)& 20138.57 & (7506.92)\\
\hline
\end{tabular}
\label{table_ATE2}
\end{table}
\subsection{Global and local price elasticity of gasoline demand}
Second, we use our method to estimate the average price elasticity of household gasoline demand: the percentage change in demand due to a unit percentage change in price. This parameter is critical for assessing the welfare consequences of tax changes, and it has been studied in \cite{hausman1995nonparametric,schmalensee1999household,yatchew2001household,blundell2012measuring}. Formally, the parameter of interest is the average derivative of log demand with respect to log price holding income and demographic characteristics fixed. The exact version of this empirical question corresponds to Example~\ref{ex:deriv}. The approximate version of this empirical question corresponds to Example~\ref{ex:policy2}.
We use data from the 1994-1996 Canadian National Private Vehicle Use Survey (\cite{semenova2020debiased_data}), using sample selection and variable construction as in \cite{yatchew2001household} and \cite{belloni2019conditional}. The outcome $Y$ is log gasoline consumption. The variable $D$ with respect to which we differentiate is log price per liter. The raw covariates $X$ are log age, log income, and log distance as well as geographical, time, and household composition dummies. In total we have $n=5001$ observations. We consider the specification $b(D,X)$ previously considered by \cite{semenova2020debiased} augmented with additional interactions. The \cite{semenova2020debiased} specification includes polynomials of continuous covariates, and interactions of log price (and its square) with time and household composition dummies. We further include interactions of log price (and its square) with log age, log age squared, log income, and long income squared to allow for heterogeneity. Altogether, $p=99$.
Table~\ref{table_APE} summarizes results for the entire population and for each quintile of the income distribution. We use $K=5$ folds in cross-fitting. To estimate the RR, we use the generalized Dantzig selector (GDS) procedure introduced in the present work. To estimate the regression, we use GDS, Lasso, random forest, or neural network. Again, GDS is implemented using the tuning procedure described in Section~\ref{sec:diff}, Lasso is implemented using the tuning procedure described in \citet*{chernozhukov2018learning}, and random forest and neural network are implemented with the same settings as \cite{chernozhukov2018double}. We find average price elasticity of $-0.28$ $(0.06)$ using GDS for both the RR and the regression. Lasso gives similar results. Note that random forest is not differentiable, and the derivative of a neural network may be difficult to extract from a black-box implementation. When using these estimators, we implement a partial difference approximation of the derivative, detailed in Section~\ref{sec:diff}. We conjecture that this approximation explains why the results using random forest appear attenuated and why the results using neural network appear positive or statistically insignificant. Using GDS, we find that localized average price elasticity is statistically significant and negative in each income quintile, with substantial heterogeneity.
\begin{table}[h]
\centering
\caption{Estimated average derivative (price elasticity) of gasoline demand. Localized average derivatives are
reported by income quintile groups. The regression is estimated by GDS, Lasso, random forest, or neural network. Standard errors are reported in parentheses.}
\begin{tabular}{@{}ccccccccccc@{}}
\hline
Income quintile & N& \multicolumn{2}{c}{GDS} &\multicolumn{2}{c}{Lasso}&\multicolumn{2}{c}{Random forest} & \multicolumn{2}{c}{Neural network}\\
\hline
All & 5001 & -0.28 & (0.06)&-0.16 & (0.05) & -0.01 & (0.06) &0.15 & (0.05)\\
1 & 1001& -0.84 & (0.13)&-0.44 & (0.12)&-0.37 & (0.14) & 0.06 & (0.12)\\
2 & 1000& -0.36 & (0.12)&-0.27 & (0.11)&-0.13 & (0.13) & 0.42 & (0.12)\\
3 & 1000 & -1.40 & (0.15)&-0.91 & (0.13)&-0.60 & (0.13) &-0.28 & (0.13)\\
4 & 1000 & -1.06 & (0.14)&-0.79 & (0.14) &-0.32 & (0.15) &0.13 & (0.14)\\
5 & 1000& -0.11 & (0.14)&-0.03 & (0.11)&0.16 & (0.12) &0.58 & (0.10)\\
\hline
\end{tabular}
\label{table_APE}
\end{table}
For comparison, OLS regression of log consumption on log price, log age, log income, and log distance as well as geographical, time, and household composition dummies yields an estimate of $0.14$ $(0.06)$. The linear specification leads to a positive elasticity estimate, contradicting economic intuition (since it says there would be more gasoline consumption when prices are higher). Our localized average price elasticity results using GDS are broadly consistent with \cite{semenova2020debiased}, who more explicitly consider the relationship between average price elasticity and income. In Section~\ref{sec:diff} we report analogous estimates without debiasing. Without debiasing, the GDS and Lasso estimates of quintile elasticities are attenuated due to regularization. The bias is smaller for the random forest and neural network estimates of quintile elasticities.
\section{Estimation and inference for high dimensional approximately linear models}\label{sec:estimation}
\subsection{ Best linear approximations for the regression function and the Riesz representer}
To approximate the regression function, we consider the $p$-vector of dictionary functions $$x \mapsto b(x) = (b_j(x))_{j=1}^p, \quad b_j \in L^2(F).$$
The dimension $p$ of the dictionary can be large, potentially much larger than $n$. Let $\Gamma_b$ be the linear subspace
of $L^2(F)$ generated by $b$. We assume that as $n \to \infty$ we have that $p \to \infty$ and
$
\Gamma_b \to \bar \Gamma:= \text{closure}(\Gamma),
$
where $\bar \Gamma$ is a linear subspace of $L^2(F)$ with the basis functions $\{\tilde b_j\}_{j=1}^{\infty}$. Here convergence means that any convergent sequence in $\Gamma_b$ has its limit
in $\bar \Gamma$ and for each $\gamma \in \bar \Gamma$ we have a sequence in $\Gamma_b$ converging to it, with respect to the $L^2(F)$ norm. Note that this setup allows the dictionary $b= b_n$ to change with $n$, as for example with $b$-splines.
Here we define $\gamma^\star_0$ as a projection of $Y$ onto $\bar \Gamma$, i.e. $\gamma^\star_0$ is the projection of $Y$ on the infinite set of variables $\{\tilde b_j(X)\}_{j=1}^{\infty}$. This setup is slightly more general than in the introduction, where
$\gamma^\star_0$ was the conditional expectation function. Of course, if the latter is an element of $\bar \Gamma$, it automatically coincides with $\gamma^\star_0$.
We approximate $\gamma_0^\star$ by the finite-dimensional best linear predictor (BLP) $\gamma_0$ via
$$
\gamma^\star_0 = \gamma_0 + r_\gamma := b'\beta_0 + r_\gamma: {\mathrm{E}} [ b(X) r_\gamma(X) ] =0,
$$
where $r_\gamma$ is the approximation error, and $ \gamma_0 := b'\beta_0$ is the best linear predictor of $Y$ and best linear approximation to $\gamma^\star_0$. We define $\beta_0$ as a minimal $\ell_1$-norm solution to the system of equations
$$
\min \|\beta\|_1 : {\mathrm{E}} [ b(X) ( \gamma^\star_0(X) - b(X)'\beta)] = 0,
$$
when $G = {\mathrm{E}} b(X)b(X)'$ is not full rank.
Similarly, we approximate the Riesz representer $\alpha^\star_0$, which exists by Lemma~\ref{lemma: ERR} whenever $L<\infty$, via the best linear approximation $\alpha_0$:
$$
\alpha^\star_0 = \alpha_0 + r_\alpha = b'\rho_0 + r_\alpha: {\mathrm{E}} [r_\alpha (X) b(X) ] = 0.
$$
We define $\rho_0$ as a minimal $\ell_1$-norm solution to the system of equations
$$
\min \| \rho \|_1 : {\mathrm{E}} [(\alpha^\star_0(X) - b(X)'\rho)b(X)] = 0.
$$
Using that ${\mathrm{E}} \alpha_0^\star(X) b(X) = {\mathrm{E}} m(W, b)$, we note that
\begin{equation}\label{eq:RRP}
0={\mathrm{E}} [r_\alpha (X) b(X) ] = {\mathrm{E}} ((\alpha^\star_0(X) - b(X)'\rho_0)b(X)) =
{\mathrm{E}} m(W, b) - {\mathrm{E}} \alpha_0(X) b(X).
\end{equation}
Hence $\alpha_0$ is the Riesz representer for ${\mathrm{E}} m(W,\gamma)$ for each $\gamma \in \Gamma_b$.
Here we can interpret $\Gamma_b$ as the collection of test functions on which the representation property (\ref{eq:RRP}) holds.
\begin{definition}[Penultimate and ultimate target parameters] Our penultimate target is
the linear functional applied to the BLP $\gamma_0$:
$$
\theta_0 := {\mathrm{E}} [m(W, \gamma_0)] = {\mathrm{E}} [\alpha_0(X) \gamma_0(X)] = {\mathrm{E}} [m(W, \gamma_0) + \alpha_0(X) ( Y - \gamma_0(X)) ].$$
Our ultimate target is the linear functional applied to $\gamma_0^\star$
$$
\theta^\star_0 := {\mathrm{E}} [m(W, \gamma^\star_0)] ={\mathrm{E}} [\alpha^\star_0(X) \gamma^\star_0(X)]
= {\mathrm{E}} [m(W, \gamma^\star_0) + \alpha^\star_0(X) ( Y - \gamma^\star_0(X))].$$
\end{definition}
If the approximation errors are such that
\begin{equation} \label{eq:specified}
(\sqrt{n}/\sigma )\int r_\alpha r_\gamma dF \to 0
\end{equation}
our inference will target the ultimate parameter. In the non-regular setup, the second order error condition $\int r_{\alpha}r_{\gamma}dF\leq \sigma/\sqrt{n}$ is weaker than what is usually required for pathwise differentiable functionals (since $\sigma\rightarrow\infty$ is the non-regular case); there is a lower bar for oracle rates in non-regular problems. This phenomenon was also noted by \cite{foster2019orthogonal} and \cite{kennedy2020optimal}. Otherwise our inference will target
an interpretable penultimate parameter. We shall formally refer to the latter case as the \textit{misspecified case}.
\begin{lemma}[Basic properties of the score]\label{lemma:score} Our DML estimator of $\theta_0$ will be based on the following score function:
$$
\psi (W, \theta; \beta, \rho) = \theta - m(W, b)'\beta - \rho' b(X) (Y - b(X)'\beta),
$$
which has the following properties:
$$
\partial_{\beta} \psi (W, \theta; \beta, \rho) = - m(W, b) + {\rho}' b(X) b(X)', \quad \partial_{\rho}\psi(W, \theta; \beta, \rho) = -b(X) (Y - b(X)'\beta),
$$$$
\partial^2_{\beta \beta'} \psi (W, \theta; \beta, \rho) = \partial^2_{\rho \rho'}\psi(W, \theta; \beta, \rho) =0,
\quad \partial^2_{\beta \rho'} \psi (W, \theta; \beta, \rho) = b(X) b(X)'.
$$
This score function is Neyman orthogonal at $(\beta_0, \rho_0)$:
$$
{\mathrm{E}} [\partial_{\beta} \psi (W, \theta ; \beta, \rho_0)] = -{\mathrm{E}} [ m(W, b)] + G\rho_0 = 0,$$$$
{\mathrm{E}} [\partial_{\rho} \psi (W, \theta; \beta_0, \rho)]= {\mathrm{E}}[ -b(X) (Y - b(X)'\beta_0)] = - {\mathrm{E}} [ b(X) \gamma_0(X) ]+ G\beta_0 = 0.
$$
\end{lemma}
The second claim of the lemma is immediate from the definition of $(\beta_0, \rho_0)$ and the first follows from elementary calculations. The orthogonality property above says that the score function is invariant to small perturbations of the nuisance parameters $\rho$ and $\beta$ around their ``true values''
$\rho_0$ and $\beta_0$. This invariance property plays a crucial role in removing the impact of biased estimation of nuisance parameters $\rho_0$ and $\beta_0$ on the estimation of the main parameters $\theta_0$.
\subsection{Estimators}
Estimation will be carried out using the following Dantzig selector-type estimators (\cite{candes2007dantzig}). In a follow-up work, \citet*{chernozhukov2018learning} consider Lasso-type estimators.
\begin{definition}[Generalized Dantzig selector estimator] Consider a parameter $t \in T \subset \Bbb{R}^p$, where $T$ is a convex set. Consider the moment functions $t \mapsto g(t) $ and the estimated moment functions $t \mapsto \hat g(t) $,
mapping $\Bbb{R}^p$ to $\Bbb{R}^p$: $$
g(t) = G t - M; \quad \hat g(t) = \hat G t - \hat M,
$$
where $G$ and $\hat G$ are $p$ by $p$ non-negative-definite matrices and $M$ and $\hat M$
are $p$-vectors. Define $t_0$ as a minimal $\ell_1$-norm solution to $g(t) =0$ and assume $t_0 \in T$. Define the GDS estimator $\hat t$ by solving
$$
\hat t \in \arg \min \| t\|_1: \| \hat g(t) \|_\infty \leq \lambda, \quad t \in T$$
where $\lambda $ is chosen such that
$ \| \hat g(t_0) - g(t_0)\|_\infty \leq \lambda,$
with probability at least $1-\epsilon$.
\end{definition}
Here we record the possibility of convex restrictions on the parameter space by placing $t$ in a convex parameter space $T$. If parameter restrictions are correct, then this can potentially improve theoretical guarantees by weakening the requirements on $G$
and other primitives.
\begin{definition}[GDS for BLP: Dantzig selector]\label{D.1} Given a diagonal positive-definite
normalization matrix $D_{\beta}$, define $\hat \beta_A = D_\beta \hat t$, where $\hat t$ is the GDS estimator for $t_0=D_{\beta}^{-1}\beta_0$ with
$$
\quad G = {\mathrm{E}} b(X) b(X)', \hat G= \mathbb{E}_A b(X) b(X)', M = D_{\beta}^{-1}{\mathrm{E}} Y b(X), \hat M = D_{\beta}^{-1}\mathbb{E}_A Y b(X); T_\beta \subset \Bbb{R}^p.
$$
\end{definition}
In this setting, our estimator specializes to the original Dantzig selector. In practice, we use $T_\beta = \Bbb{R}^p$, although when we are interested in average derivative functionals, it is theoretically
helpful to impose the convex restrictions of the sort $T = \{ t \in \Bbb{R}^p: \sup_{x \in \mathcal{X}} | \partial_d b(x)'t | \leq B\}$, where $B$ is some a priori known upper bound on the derivative. Ideally, $D_{\beta}$ is chosen such that $\mathrm{diag}( Var( D^{-1}_{\beta} (\hat G \beta_0 - \hat M) ) = I$. Our practical algorithm given in Section~\ref{sec:diff} estimates $D_\beta$ from the data.
\begin{definition}[GDS for Riesz representer]\label{D.2} Given a diagonal positive-definite
normalization matrix $D_{\rho}$, define $\hat \rho_A = D_{\rho}\hat t$, where $\hat t$ is the GDS estimator
of the parameter $t_0=D_{\rho}^{-1}\rho_0$ with
$$
\quad G = {\mathrm{E}} b(X) b(X)', \hat G = \mathbb{E}_A b(X) b(X)', M = D_{\rho}^{-1}{\mathrm{E}} m(W, b), \hat M = D_{\rho}^{-1}\mathbb{E}_A m(W, b); T_\rho \subset \Bbb{R}^p.
$$
\end{definition}
In this setting, our estimator is a generalization of the original Dantzig selector. In practice, we are using $T_\rho =\Bbb{R}^p$, even though it is possible to exploit some structured restrictions
on the problem motivated by the nature of the universal Riesz representers. Ideally, $D_{\rho}$ is chosen such that $\mathrm{diag}( Var( D^{-1}_{\rho} (\hat G \rho_0 - \hat M) ) = I$. Our practical algorithm given in Section~\ref{sec:diff} estimates $D_\rho$ from the data.
We now define the DML estimator with Riesz Representers, which makes use of cross-fitting.
\begin{definition}[DML with RR] Consider the partition of $\{1,...,n\}$ into $K \geq 2$ blocks $(I_k)_{k=1}^K$, with $m= \lfloor n/K\rfloor$
observations in $I_k$, for $k< K$ and $\lceil n/K \rceil $ remaining in $I_K$. For each $k =1,...,K$, let
$ \hat \beta_{k}$ and $\hat \rho_{k}$ denote GDS estimators obtained using data $(W_i)_{i \in I^c_k}$, where $I^c_k= \{1,...,n\} \setminus I_k$, and let estimator $\hat \theta_{k}$ be defined as
$$
\hat{\theta}_k= \mathbb{E}_{I_k} [m(W, b)'\hat \beta_k + \hat \rho_{k}' b(X) (Y - b(X)'\hat \beta_k)].$$
Define the DML estimator $\hat \theta$ as
the average:
$$
\hat \theta = \sum_{k=1}^{K} \hat \theta_{k} w_k; \quad w_k = \frac{\lfloor n/K \rfloor}{n} \text{ if } k < K, \quad w_K =
\frac{\lceil n/K\rceil }n .
$$
\end{definition}
\subsection{Properties of DML: Main result}\label{sec:DML}
We provide a single non-asymptotic result that allows us to cover both global and local functionals, implying uniformly
valid rates of concentration and normal approximations over large sets of $P$.
Consider the oracle estimator based upon the true score functions:
$$
\bar \theta := \theta_0 - n^{-1} \sum_{i =1}^n \psi_0(W_i), \quad \psi_0(W) :=\psi(W, \theta_0; \beta_0, \rho_0).
$$
We seek to establish minimal conditions under which the DML estimator approximates the oracle estimator, and is approximately
normal with distribution
$$
N(0, \sigma^2/n), \quad \sigma := \| \psi_0\|_{P,2}.
$$
For regular functionals $\sigma$ is bounded, giving $1/\sqrt{n}$ concentration around $\theta_0$, and for non-regular functionals $\sigma \propto L \to \infty$ requring $L/\sqrt{n} \to 0$ to get concentration. Our normal approximation is accurate if kurtosis of $\psi_0$ does not grow too fast:
$$
(\kappa/\sigma)^3 /\sqrt{n} \text{ is small}, \ \ \kappa:= \| \psi_0\|_{P,3}.
$$
In the regular case $(\kappa/\sigma)^3$ is bounded, but for the non-regular cases it can scale as fast as $L$, again requiring $L/\sqrt{n} \to 0$.
Fix all of these sequences and the constants.
Define the guarantee set:
$$
\mathsf{S} = \left\{ \begin{array}{c}(u, v) \in \Bbb{R}^{2p}: \sqrt{ u'Gu} \leq r_1, \sqrt{ v'Gv} \leq \sigma r_2, \ \
|u'Gv| \leq \sigma r_3, \beta_0 + u \in T_\beta, \rho_0 + v \in T_{\rho} \end{array} \right \},
$$
We will take $u=\hat{\beta}_k-\beta_0$ and $v=\hat{\rho}_k-\rho_0$. As such, $r_1$ measures the non-asymptotic mean square rate for the BLP; $r_2$ measures the non-asymptotic mean square rate for the RR; and $r_3$ measures how the estimation errors interact. Note the presence of $\sigma$ acting on $r_2$ and $r_3$, which accommodates non-regular functionals. We will instantiate $(r_1,r_2,r_3)$ as fast and slow rates by analyzing the GDS estimator, in Theorem~\ref{theorem: rates1} below.
Next, define
$\mu$ to be the smallest modulus of continuity such that on $(u,v) \in \mathsf{S} $
$$
\sqrt{Var} ( (-m(W, b) + {\rho_0}' b(X) b(X)) 'u) \leq \mu \sigma \| b'u\|_{P,2},
\sqrt{Var} ( (Y - b(X)'\beta_0) b(X) 'v) \leq \mu \| b'v\|_{P,2}, \\
$$$$
\sqrt{Var} (u 'b(X) b(X)' v) \leq \mu ( \| b'u\|_{P,2}+\| b'v\|_{P,2}).
$$
In typical applications, the modulus of continuity $\mu$ is bounded.
Indeed, if elements of the dictionary are bounded with probability one, $ \| b(X) \|_\infty \leq C$, then we can select $\mu = C B$ for many functionals of interest, so the assumption is plausible. If $b(X) = X$ are sub-Gaussian, then this assumption is also easily satisfied; however, this case is not of central interest to us. See \cite{chernozhukov2021simple} for a more general discussion.
Consider $P$ that satisfies the following conditions.
\begin{itemize}
\item[R$(\delta)$]\textit{With probability $1-\varepsilon$, the estimation errors $\{(\hat \beta_k- \beta_0, \hat \rho_k- \rho_0)\}_{k=1}^{K}$ take values in $\mathsf{S}^K$},
\textit{with quality of the guarantee obeying} $$ \sigma^{-1} ( \sqrt{m}\sigma r_3 + \mu r_1 (1+ \sigma) + \mu \sigma r_2 ) \leq \delta.$$
\end{itemize}
$R(\delta)$ is a requirement on how the sequences $(r_1,r_2,r_3)$ evolve relative to $(\sigma,\mu,m)$. We will formally verify $R(\delta)$ for the approximately sparse setting, in Corollary~\ref{Cor:SUFF} below. $R(\delta)$ is the key condition for our main result, Theorem~\ref{theorem: DML}.
\begin{theorem}[Adaptive estimation and approximate Gaussian inference]\label{theorem: DML} Suppose $K$ divides $n$ for simplicity. Under condition R$(\delta)$, we have the adaptivity property, namely the difference
between the DML and the oracle estimator is small: for any $\Delta \in (0,1)$,
$$
|\sqrt{n}(\hat \theta - \bar \theta)/\sigma| \leq \sqrt{K} 4 \delta/\Delta
$$
with probability at least $1- \varepsilon- \Delta^2$.
As a consequence, $\hat \theta$ concentrates in a $\sigma/\sqrt{n}$ neighborhood of $\theta_0$, with deviations approximately distributed according to the Gaussian law $\Phi(z)= {\mathrm{P}}(N(0,1) \leq z)$:
\begin{eqnarray*}
& & \sup_{z \in \Bbb{R}}\Big | {\mathrm{P}} ( \sigma^{-1} \sqrt{n} (\hat \theta_0 - \theta_0) \leq z) - \Phi(z) \Big | \leq A(\kappa/\sigma)^3 n^{-1/2}+ \sqrt{K} 2 \delta/ \Delta + \varepsilon+ \Delta^2,
\end{eqnarray*}
where $A < 1/2$ is the sharpest absolute constant in the Berry--Esseen bound.
\end{theorem}
The conclusions of this result are distinguished from those of \cite{chernozhukov2018double} and \citet*{chernozhukov2018learning} in applying to local, nonparametric objects, in providing finite sample bounds, and in being uniform over the parameter space. The conclusions are similar to this previous work in relying on a rate condition that is the product of rates of estimation for two distinct functions, here the regression and the Riesz representer.
The constants can be chosen to yield an asymptotic result.
\begin{corollary}[Uniform asymptotic adaptivity and Gaussianity]\label{corollary:uniform} Let $\mathcal{P}_n$ be any nondecreasing set of probability laws $P$ that obey condition $R(\delta_n)$ where $\delta_n \to 0$ is a given sequence. Then the DML estimator $\hat \theta$ is uniformly asymptotically equivalent to the oracle estimator $\bar \theta$,
that is $$|\sqrt{n}(\hat \theta - \bar \theta)/\sigma| = O_P (\delta_n)$$ uniformly in $P \in \mathcal{P}_n$ as $n \to \infty$.
In addition, if for each $P \in \mathcal{P}_n$ the kurtosis of $\psi_0$ does not grow too fast, namely:
$$
(\kappa/\sigma)^3/\sqrt{n} \leq \delta_n,
$$
we have that $\sqrt{n} (\hat \theta - \theta_0)/\sigma$ is asymptotically Gaussian uniformly in $P \in \mathcal{P}_n$:
$$
\lim_{n \to \infty} \sup_{P \in \mathcal{P}_n} \sup_{z \in \Bbb{R}}\Big | {\mathrm{P}}_P (\sqrt{n}(\hat \theta_0 - \theta_0)/\sigma \leq z) - \Phi(z) \Big | = 0.
$$
\end{corollary}
Hence the DML estimator of the linear functionals of the BLP function $\gamma_0$ enjoys good properties under the stated regularity conditions. This result does not distinguish between inference on global functionals from inference on local functionals, as long as the latter are not perfectly localized. We state a separate result for perfectly localized functionals below.
\begin{corollary}[Inference on the ultimate parameter $\theta_0^\star$] Suppose
that, in addition to conditions of Corollary~\ref{corollary:uniform}, $P$ satisfies the small approximation error condition:
\begin{equation}\label{eq: smallbias}
(\sqrt{n}/\sigma) | \theta_0 - \theta^\star_0| = (\sqrt{n}/\sigma) \left |\int r_\alpha r_\gamma dF \right | \leq \delta.
\end{equation}
Then conclusions of Theorem~\ref{theorem: DML} hold with $\theta_0^\star$ replacing $\theta_0$, with
$\sqrt{K} 4 \delta/\Delta$ increased by $\delta$, and the same probability. Conclusions of Corollary~\ref{corollary:uniform} continue to hold
with $\theta_0^\star$ replacing $\theta_0$ for a class of probability laws $\mathcal{P}_n$, provided each $P \in \mathcal{P}_n$ satisfies the conditions of Corollary~\ref{corollary:uniform} and (\ref{eq: smallbias}) for the given $\delta= \delta_n \to 0$.
\end{corollary}
The approximation bias for the ultimate target can be plausibly small due to the fact that many rich function classes admit regularized linear approximations with respect to conventional dictionaries $b$. For instance, \cite{tsybakov2012introduction} and \cite{belloni2014pivotal} show small approximation bias using Fourier bases as dictionaries, and using Sobolev and rearranged Sobolev balls, respectively, as the function classes.
\begin{corollary}[Inference on the perfectly localized parameter] Suppose
that, in addition to conditions of Corollary~\ref{corollary:uniform}, $P$ satisfies the small approximation error condition:
\begin{equation}\label{eq: PLC1}
\sqrt{n}| \theta_0 (\gamma_0; \ell_h) - \theta_0(\gamma_0^\star; \ell_h) |/\sigma = \sqrt{n}\left |\int r_\alpha r_\gamma dF \right |/\sigma \leq \delta,
\end{equation}
and the localization bias is small:
\begin{equation}\label{eq: PLC2}
\sqrt{n}|\theta_0(\gamma_0^\star; \ell_h) -\theta_0(\gamma_0^\star; \ell_0) |/\sigma \leq \delta,
\end{equation}
Then conclusions of Theorem~\ref{theorem: DML} hold with $\theta_0(\gamma_0^\star; \ell_0)$ replacing $\theta_0$, with
$\sqrt{K} 4 \delta/\Delta$ increased by $2\delta$, and the same probability. Conclusions of Corollary~\ref{corollary:uniform} continue to hold
with $\theta^\star_0(\gamma^\star; \ell_0)$ replacing $\theta_0=\theta_0 (\gamma_0; \ell_h)$ for a class of probability laws $\mathcal{P}_n$, provided each $P \in \mathcal{P}_n$ satisfies the conditions of Corollary~\ref{corollary:uniform}
and (\ref{eq: PLC1})-(\ref{eq: PLC2}) for the given $\delta= \delta_n \to 0$.
\end{corollary}
\subsection{Semi-parametric efficiency}
Below we use concepts from semi-parametric efficiency, as presented in
\cite{bickel1993efficient} and
\cite{vaart}; we do not recall them here for brevity.
The DML estimator $\hat \theta$ will be asymptotically efficient for estimating
$\theta^\star_0$, defined as a functional of $\gamma^\star_{0}$, the projection
of $Y$ on $\bar{\Gamma}$. The distribution of a data observation is unrestricted in
this case, so that there will only be one influence function for each
functional of interest, and the estimator is asymptotically linear with that
influence function. The standard semiparametric efficiency results then imply
that our estimator will have the smallest asymptotic concentration among
estimators that are locally regular; see \cite{bickel1993efficient} and
\cite{vaart}.
Our formal result stated below only implies efficiency for the regular case, where
the operator norm of the function $L$ is bounded, holding $P$ fixed. We expect
that a similar result continues to hold with $L \to \infty$, by developing an appropriate
formalization that handles $P$ changing with $n$ and rules out super-efficiency phenomena. However,
this formalization requires a separate major development, which we leave to future research.
In what follows, the notation $\gamma^\star_{0,P}$ emphasizes the dependence of the projection $\gamma^\star_0$ on $P$.
\begin{theorem}[Efficiency]\label{theorem:eff} Let $
\psi_{0}^\star(W) := \theta_0^\star- m(W, \gamma_0^\star)-\alpha^\star _0(X)(Y- \gamma_0^\star(X)).
$
Suppose that ${\mathrm{E}}[Y^{2}]<\infty,$ ${\mathrm{E}}[\psi_0(W)^{2}]<\infty,$ and $m(W,\gamma)$ is mean square continuous in $\gamma$ under $P$. Then $\theta_{0,P} := \int m(w, \gamma^\star_{0,P}) dP(w)$
is differentiable at $P$, in the sense that
$$
\lim_{\tau \searrow 0 } \frac{\theta_{0,P_\tau} - \theta_{0,P}}{\tau} = {\mathrm{E}}_{P} \delta(W) \psi_0(W),
$$
where $\psi_0$ is called the influence function and is unique, and the directional perturbation $P_{\tau}$ is defined
as $dP_{\tau} = dP[1+ \tau \delta]$, where the direction $\delta$ is any element of the tangent set $\{\delta \text{ measurable } : \mathcal{W} \to \Bbb{R}: \int \delta dP =0, \|\delta\|_\infty< M\}$ for each $0<M<\infty$. Consequently, the asymptotic variance of every regular sequence of estimators is bounded below by $\| \psi_0\|_{P,2}.$
Further, since the tangent set is a convex cone, other conclusions of the Theorem~25.20 and Theorem~25.21 of \cite{vaart} also hold, namely the convolution and the minimax characterization of the efficiency.
\end{theorem}
\subsection{Properties of GDS estimators}
Our goal is to verify that the guarantee $R(\delta)$ holds. In particular we have to analyze $(r_1,r_2,r_3)$ by bounding the population prediction norm
$
v \mapsto \sqrt{v' G v}.
$
This is a more nuanced problem than bounding the empirical prediction norm $v \mapsto \sqrt{v' \hat G v}$, which has been
accomplished in a variety of prior analyses done on Dantzig-type and Lasso-type estimators.
We begin with the following condition, which only controls the max of error rates and controls the $\ell_1$ norm of true parameters:
\begin{itemize}
\item[MD] \textit{We have that $t_0 \in T$ and $\|t_0\|_1 \leq B$, where $B \geq 1$, and the empirical moments obey the following bounds
with probability at least $1 - \varepsilon$, for $\bar \lambda \geq \lambda$ $$
\| \hat G - G\|_\infty \leq \bar \lambda, \ \| \hat G t_0 - \hat M\|_{\infty}\leq \lambda.
$$}
\end{itemize}
The bounds on $\ell_1$ norm of coefficients are naturally motivated, for example, by working in Sobolev or rearranged Sobolev
spaces (see, \cite{tsybakov2012introduction} and \cite{belloni2014pivotal}, respectively). Rearranged Sobolev spaces allow the first $p$ regression coefficients in the series expansion to be arbitrarily rearranged, allowing a much greater degree of oscillatory behaviors than in the original Sobolev spaces. The complexity of these function classes are also different.
Sobolev spaces are Donsker sets under sufficient smoothness, whereas rearranged Sobolev spaces have the covering
entropy bounded below by $\log p$ and are not Donsker if $p \to \infty$.
At the core of this approach is the restricted set
$$
S (t_0, \nu) := \{ \delta: \| G \delta \|_\infty \leq \nu , \| t_0+ \delta\|_1 \leq \|t_0\|_1, t + \delta \in T\},
$$
where $\nu$ is the noise level. As demonstrated in the proof, the GDS estimator belongs to this set with high probability $1- \epsilon$ for
the noise level $\nu= 4B \bar \lambda,$
where $\lambda$ is the penalty level of GDS ($\nu$ scales like $\sqrt{ \log (p \vee n) }/\sqrt{n}$ in our problems).
\begin{definition}[Effective dimension]
Define the effective dimension of $t_0$ at the noise level $\nu>0$ as:
$$
s(t_0) := s(t_0; \nu) := \sup_{\delta \in S (t_0, \nu) } | \delta'G \delta|/\nu^2.
$$
\end{definition}
The effective dimension is defined in terms of the population (rather than sample) covariance matrix $G$, which makes it easy to verify regularity conditions. Note that if $G=I$ and $\|t_0\|_0 =s$, then $s(t_0) \leq s.$
More generally, $s(t_0)$ measures the effective difficulty of estimating $t_0$ in the prediction norm, created by design $G$ and the structure of $t_0$.
The condition imposes no conditions on the restricted or sparse eigenvalues of $G$. For example,
take $G = 11'$, a rank 1 matrix, and suppose $\|t_0\|_0 = 1$. Then $s(t_0) \leq 1$ holds in this case, giving
useful and intuitive performance bounds, while the standard restricted eigenvalues and cone invertibility factors are all zero
in this case, yielding no bounds on the performance in the population prediction norm. This type of example illustrates
the possibility of accommodation of overcomplete (multiple or amalgamated) dictionaries in $b$, whose use in conjunction with $\ell_1-$ penalization has been advocated by \cite{donoho2005stable}. Of course, the bounds
on effective dimension follow from the bounds on cone-invertibility factors and restricted eigenvalues.
Given a vector $\delta \in \Bbb{R}^p$, let $\delta_A$ denote a vector with
the $j$-th component set to $\delta_j$ if $j \in A$ and $0$ if $j \not \in A$.
\begin{lemma}[Bound on effective dimension in approximately sparse model]\label{lemma: boundS} Suppose that
$t_0$ is approximately sparse, namely $$|t_{0}|_j^* \leq A j^{-a} \quad j =1,...,p,$$
for some finite positive constants $A$ and $a>1$, where $(|t_{0}|_j^*)_{j=1}^{p}$ is the non-increasing rearrangement of
$(|t_{0j}|)_{j=1}^p$. Let $ t_0^{\mathcal{M}} := t_0 (1 ( |t_0| > \nu) := (t_{0j} 1(| t_{0j}| >\nu ))_{j=1}^p$ denote the vector with components smaller than $\nu$
trimmed to 0. Then
$$
s(t_0; \nu) \leq s\times \left(k^{-1} \vee \frac{6a}{a-1}\right ), \quad \| t_0^{\mathcal{M}}\|_0 \leq s:= (A/\nu)^{1/a},
$$
$k$ is the cone invertibility factor: $$
k := \inf \frac{ |\mathcal{M}| \|G\delta\|_\infty}{\|\delta\|_1}: \delta \neq 0, \ \ \|\delta_{\mathcal{M}^c}\|_1 \leq 2 \|\delta_{\mathcal{M}}\|_1,$$ $\mathcal{M} = \text{support} ( t_0^{\mathcal{M}})$, $\mathcal{M}^c = \{1,...,p\} \setminus \mathcal{M}$, and $|\mathcal{M}|\leq s$. \end{lemma}
The cone invertibility factor is a generalization of the restricted eigenvalue condition of \cite{bickel2009simultaneous}, proposed by \cite{ye2010rate}. The concept of the effective dimension does not split $t_0$ into a sparse component and a small dense component, as is done in the now standard analysis of $\ell_1$-regularized estimators of approximately sparse $t_0$. The effective dimension is simply stated in terms of $t_0$ alone.
Since approximate sparsity is a simple condition that implies a bound on effective dimension, we pause and interpret approximate sparsity in the context of a motivating example from causal inference. In particular, we revisit ATE (Example~\ref{ex:ate}). For simplicity, consider the global parameter and assume that the function ${\mathrm{E}}[Y|D,Z]$ is an element of $\bar{\Gamma}$, so that $\gamma_0^{\star}(D,Z)={\mathrm{E}}[Y|D,Z]$ and $\alpha_0^{\star}(D,Z)=D/\pi_0^{\star}(Z)-(1-D)/(1-\pi_0^{\star}(Z))$ where $\pi_0^{\star}(Z)={\mathrm{E}}[D|Z]$ is the propensity score. Consider the dictionary $b(d,z)=(dq(z)',(1-d)q(z)')'$ where $\{q_j(z)\}_{j=1}^{p/2}$ are the initial $p/2$ elements of a sequence of basis functions that approximates the functions ${\mathrm{E}}[Y|1,Z]$, ${\mathrm{E}}[Y|0,Z]$, $1/\pi_0^{\star}(Z)$, and $1/(1-\pi_0^{\star}(Z))$.
Suppose the minimal $\ell_1$-norm mean square projections of ${\mathrm{E}}[Y|1,Z]$ and ${\mathrm{E}}[Y|0,Z]$ onto $\{q_j(z)\}_{j=1}^{p/2}$ are approximately sparse after rescaling appropriately by $D^{-1}_{\beta}$. (Note that if ${\mathrm{E}}[Y|1,Z]$ and ${\mathrm{E}}[Y|0,Z]$ are already approximately sparse then so are their projections.) It follows that the minimal $\ell_1$-norm mean square projection of $\gamma_0^{\star}$ is approximately sparse and $s_{\beta}:=s(D^{-1}_\beta \beta_0;\nu)$ is small.
Suppose instead that the minimal $\ell_1$-norm mean square projections of $1/\pi_0^{\star}(Z)$ and $1/(1-\pi_0^{\star}(Z))$ onto $\{q_j(z)\}_{j=1}^{p/2}$ are approximately sparse after rescaling appropriately by $D^{-1}_{\rho}$. (Note that if $1/\pi_0^{\star}(Z)$ and $1/(1-\pi_0^{\star}(Z))$ are already approximately sparse then so are their projections.) It follows that the minimal $\ell_1$-norm mean square projection of $\alpha_0^{\star}$ is approximately sparse and $s_{\rho}:=s(D^{-1}_\rho \rho_0;\nu)$ is small.
\begin{lemma}[Non-asymptotic bound for GDS in population prediction norm]\label{lemma:RMD} Suppose that MD holds.
Then with probability $1- 2\varepsilon$ the estimator $\hat t$ exists and obeys:
$$
(\hat t-t_0)'G(\hat t- t_0) \leq ( s(t_0; \nu) \nu^2) \wedge (2B \nu).
$$
\end{lemma}
The bound is a minimum of what is called the ``fast rate bound'' and the ``slow rate'' bound. This result tightens the result in \cite{chatterjee2013assumptionless} who established a ``slow rate'' bound (in the context of Lasso) that applies under no assumptions on $G$. If the effective dimension is not too big, as in the examples above, the ``fast rate'' $s(t_0;\nu) \nu^2$ provides a tighter bound under weak assumptions on $G$. It is important to emphasize that the result is stated in terms
of the population prediction norm rather than the empirical norm.
We now apply this result to GDS estimators of the Riesz representer and the BLP.
We impose the following conditions. Let $\Bbb{G}_A$ denote the empirical process over $f \in \mathcal{F}: \mathcal{W} \to \Bbb{R}^p$ and $ i \in A$, namely $$\mathbb{G}_A f := \mathbb{G}_A f (W) := | I | ^{-1/2} \sum_{ i \in A} (f(W_i) - P f), \quad Pf := P f(W) := \int f(w) d P(w).$$
The following is a sufficient condition that will deliver the guarantee $R(\delta)$ for $\delta \to 0$. Let $\tilde \ell$ denote
a positive constant (that increases to $\infty$ as $n \to \infty$ in the asymptotic results).
\begin{itemize}
\item[SC] \textit{(a) The $\ell_1$ norms of coefficients are bounded as $\|D^{-1}_\rho \rho_0 \|_1 \leq B$ and $\|D^{-1}_\beta \beta_0\|_1 \leq B$, for $B \geq 1$, and the scaling matrices obey $ \|D_{\rho} v\| \leq \mu_D \sigma \| v\| $ for $ D^{-1}_\rho v \in S ( D^{-1}_\rho \rho_0, \nu)$
and $ \|D_{\beta} u\| \leq \mu_D \| u\| $ for $D^{-1}_\beta u \in S (D^{-1}_\beta \beta_0, \nu)$ for $\nu = 4 B \tilde \ell/\sqrt{n}$. (b) Given a random subset $A$ of $\{1,...,n\}$ of size $m \geq n - \lfloor n/K \rfloor $, dictionary $b$ obeys with probability at least $1-\epsilon$,}
$
\|\Bbb{G}_{A} b b'\|_\infty \leq \tilde \ell.
$ \textit{ (c) The penalty levels $\lambda_{\rho}$ and $\lambda_{\beta}$
are chosen
such that with probability at least $1- \epsilon$,
$
\| D_{\beta}^{-1} (\Bbb{G}_{A} b b' \beta_0 - \Bbb{G}_{A} Y b(X))\|_\infty/\sqrt{m} \leq \lambda_\rho,$ $
\| D_{\rho}^{-1} (\Bbb{G}_{A} b b' \beta_0 - \Bbb{G}_{A} m(W, b))\|_\infty/\sqrt{m} \leq \lambda_\beta,
$
and are not overly large, $ \lambda_{\beta} \vee \lambda_{\rho} \leq \tilde \ell/\sqrt{m}$.}
\end{itemize}
SC(a) records a restriction on the $\ell_1$ norm of $\beta_0$ and $\rho_0$. For instance,
in Examples~\ref{ex:ate},~\ref{ex:policy1}, and~\ref{ex:policy2}, $D_\rho \asymp \sigma I \asymp L I$, which requires the $\ell_1$-norm of $\rho_0$ to increase at most at
the speed $L\asymp \sigma$.
SC(b) is a weak assumption: the bound $\bar \lambda $ and the penalty level $ \lambda$ can be chosen proportionally to $\sqrt{ \log (p \vee n) }/\sqrt{n}$, that is $$\tilde \ell \asymp \sqrt{\log (p \vee n)}$$ using self-normalized moderate deviation bounds (\cite{jing2003self,belloni2014pivotal}) or high-dimensional central limit theorems (\cite{chernozhukov2017central}), under mild moment conditions, without requiring sub-Gaussianity. For instance, \cite{belloni2014pivotal} employ these tools to show that, for the bounded design case $\| b\|_\infty \leq C$, $\lambda$ can be chosen as in the Gaussian error case, provided that errors follow $t(2+\delta)$ distribution (having above $2$ bounded moments), and get the error bounds similar to the Gaussian case. Here we state a general condition as our working assumption, instead of focusing on more specific condition that get us Gaussian-type conclusions.
\begin{theorem}[GDS for BLP and RR]\label{theorem: rates1} Suppose SC holds. Then with probability at least $1- K 4\epsilon$,
we have that $
u = \hat \beta_A - \beta_0$ and $v = \hat \rho_A - \rho_0
$ obey, for some absolute constant $C$,
$$
u'G u \leq r_1^2, \ \ v'Gv \leq \sigma^2 r^2_2 , \ \ |u'Gv| \leq \sigma r_3 ,
$$$$
r_1^2 = C \mu_D^ 2 (B^2 \tilde \ell^2 s_\beta/ n) \wedge ( B^2 \tilde \ell /\sqrt{n} ), \quad r_2^2 = C \mu_D^ 2 ( B^2 \tilde \ell^2 s_{\rho} / n) \wedge ( B^2 \tilde \ell /\sqrt{n}), \quad r_3 = r_1 r_2,
$$
where $s_\beta$ and $ s_\rho$ are the effective dimensions for parameters $D^{-1}_\beta \beta_0$ and $D^{-1}_\rho \rho_0$ for the noise level $\nu= 4 B \tilde \ell/\sqrt{n}$
\end{theorem}
In other words, we have instantiated $(r_1,r_2,r_3)$ for approximately sparse models in the guarantee set $\mathsf{S}$. We have the following corollary, which verifies $R(\delta)$ for approximately sparse models and hence provides sufficient conditions for Theorem~\ref{theorem: DML}.
\begin{corollary}[Sufficient condition for $R(\delta)$]\label{Cor:SUFF} Suppose SC holds. The guarantee $R(\delta)$ holds with $\varepsilon = 1 - K 4 \epsilon$, provided
$$
\text{either } C s_\beta \leq \sqrt{n}\delta^2/(\tilde \ell^3 \mu^2 \mu^2_D) \text{ or } C s_\rho \leq \sqrt{n} \delta^2/(\tilde \ell^3 \mu^2 \mu^2_D),
$$
for some large enough constant $C$ that only depends on $B$ and $K$.
\end{corollary}
\begin{remark}[Sharpness of conditions: Double sparsity robustness] This gives sufficient conditions such that (ignoring
slowly growing term $\tilde \ell$) the condition $R(o(1))$ holds if
$$
\text{ either $s_\beta \ll \sqrt{n} \ \text{ or } s_\rho \ll \sqrt{n}$,}
$$
where $s_\beta$ and $s_\rho$ are measures of the effective dimensions of parameters $D_\beta^{-1} \beta_0$ and $D_{\rho}^{-1}\rho_0$. In well-behaved exactly sparse models, these effective dimensions are proportional to the sparsity indices divided by restricted eigenvalues. The latter possibility allows one of the parameter values to be ``dense'', having unbounded effective dimension, in which case this parameter can be estimated at some ``slow'' rate $n^{-1/4}$.
These types of conditions appear to be rather sharp, matching similar conditions used in \cite{javanmard2018debiasing} in the case of inference on a single coefficient in Gaussian exactly sparse linear regression models.
\end{remark}
\section{Estimation and inference using general regression learners}\label{sec:general}
In this section we generalize the previous analysis to allow for any
regression learner $\hat{\gamma}$ of ${\mathrm{E}}[Y|X]$ to be used in the construction
of the estimator. As we have done in preceding sections we continue to include
local functionals in our analysis, so that the results apply to nonregular
objects as well as regular ones that can be estimated $\sqrt{n}$-consistently.
Compared to the global case, the local case may have smaller regularization and model selection biases relative to the variance. Nonetheless, bias correction is important for inference in theory and in practice. Theoretically, the local case begins to resemble the global case as the number of dimensions being integrated increases. Empirically, we provide local estimates without bias correction in Section~\ref{sec:diff}. The differences can be substantial.
The only conditions we will impose on the regression learner are certain $L^{2}$
convergence properties that we will specify in this section. These properties
will allow for a wide variety of learners, including GDS, Lasso, neural
nets, boosting, and others. Thus we provide estimators of local functions that
can be constructed using many regression learners.
We continue to consider estimators that use cross-fitting and have the form
$$
\hat{\theta}=\frac{1}{n}\sum_{k=1}^{K}\sum_{i\in I_{k}}\left\{ m(W_{i}
,\hat{\gamma}_{k})+\hat{\alpha}_{k}(X_{i})[Y_{i}-\hat{\gamma}_{k}
(X_{i})]\right\} .
$$
where $\hat{\gamma}_{k}$ denotes the regression learner computed from
observations not in $I_{k}$ and $\hat{\alpha}_{k}(x)=b(x)^{\prime}\hat{\rho
}_{k}$ is the GDS learner of the Riesz representer described in previous sections.
To allow for as many regression learners as possible under as weak conditions
as possible we focus on asymptotic analysis in this section. The fundamental
property we will require of $\hat{\gamma}$ is that it have some mean square
convergence rate as an estimator of the true conditional mean $\gamma
_{0}^{\star}.$ Specifically we require that there is $r_{1}^{\star}$ converging
to zero such that for each $k,$
\begin{equation}
\left\Vert \hat{\gamma}_{k}-\gamma_{0}^{\star}\right\Vert _{P,2}=O_{p}
(r_{1}^{\star}).\label{L2rate}
\end{equation}
For purposes of formulating regularity conditions it is also useful to work
with $\alpha_{0}^{\star}$ rather than $\alpha_{0}$. We will also require that
\begin{equation}
\left\Vert \alpha_{0}^{\star}(\hat{\gamma}_{k}-\gamma_{0}^{\star})\right\Vert
_{P,2}=o_{p}(\sigma),\text{ }\left\Vert m(\cdot,\hat{\gamma}_{k}-\gamma
_{0}^{\star})\right\Vert _{P,2}=o_{p}(\sigma).\label{L2cons}
\end{equation}
In the regular case these conditions generally follow from the mean square
consistency of $\hat{\gamma}_{k}$ under boundedness of $\alpha_{0}^{\star}.$ In
nonregular cases they may impose additional conditions. For example, under the
conditions of Lemma~\ref{lemma: lstr13} it will be sufficient for these conditions to hold that
$$
h^{-p_{1}/2}\left\Vert \hat{\gamma}_{k}-\gamma_{0}^{\star}\right\Vert
_{P,2}\to_{{\mathrm{P}}}0.
$$
This condition will hold as long as $h$ grows slowly enough relative to the
mean-square convergence rate of each $\hat{\gamma}_{k}.$
Recall from Theorem~\ref{theorem: rates1} that $r_{2}$ is the convergence rate of $\sigma
^{-1}\left\Vert \hat{\alpha}_{k}-\alpha_{0}\right\Vert _{P,2}$. Let
$r_{2}^{\star}=r_{2}+\sigma^{-1}\left\Vert \alpha_{0}-\alpha_{0}^{\star
}\right\Vert _{P,2}$ and
$$
\psi_{0}^{\star}(W)=\theta_{0}-m(W,\gamma_{0}^{\star})-\alpha_{0}^{\star
}(X)[Y-\gamma_{0}^{\star}(X)].
$$
\begin{theorem}[Asymptotic Gaussian inference with general regression learner]\label{theorem: asymptotic}
Suppose $Var(Y|X)$ is bounded; $r_{1}^{\star}\rightarrow0$ and $r_{2}^{\star}\rightarrow
0$; equations (\ref{L2rate}) and (\ref{L2cons}) are satisfied; and $\sqrt{n}r_{1}^{\star}r_{2}^{\star}\rightarrow0$. Then
$$
\hat{\theta}=\theta_{0}-\frac{1}{n}\sum_{i=1}^{n}\psi_{0}^{\star}(W_{i}
)+o_{p}\left(\frac{\sigma}{\sqrt{n}}\right)
,\quad \text{hence}\quad
\frac{\sqrt{n}}{\sigma}(\hat{\theta}-\theta_{0})\overset{d}{\rightarrow}\mathcal{N}(0,1).
$$
\end{theorem}
This result shows that asymptotic linearity of the estimator $\hat{\theta}$
will result if $r_{2}^{\star}\rightarrow0$ fast enough relative to
$r_{1}^{\star}$. Asymptotic linearity implies asymptotic Gaussian inference by standard central limit theorem arguments. As in regular doubly robust estimation problems it allows for
a tradeoff between the speed of convergence $r_{2}^{\star}$ of the Riesz
representer and $r_{1}^{\star}$ of the regression. It only requires a mean
square convergence rate for the regression learner $\hat{\gamma}_{k}$ and so
allows for a wide variety of first step machine learning estimators.
We could also formulate a non-asymptotic analog to this asymptotic result. This
would depend on the availability of non-asymptotic results for the learner
$\hat{\gamma}_{k}$. To the best of our knowledge such results are not
available for many learners, such as neural nets and random forests. To allow
the results of this section to include as many first steps as possible we
focus here on the asymptotic result and reserve the non-asymptotic result to
future work.
\section*{Acknowledgements}
The National Science Foundation provided partial financial support via
grants 1559172 and 1757140. Rahul Singh thanks the Jerry Hausman Dissertation Fellowship.
\begin{thebibliography}{}
\bibitem[\protect\citeauthoryear{Abadie}{Abadie}{2003}]{abadie2003semiparametric}
Abadie, A. (2003).
\newblock Semiparametric instrumental variable estimation of treatment response
models.
\newblock {\em Journal of Econometrics\/}~{\em 113\/}(2), 231--263.
\bibitem[\protect\citeauthoryear{Altonji and Matzkin}{Altonji and
Matzkin}{2005}]{altonji2005cross}
Altonji, J.~G. and R.~L. Matzkin (2005).
\newblock Cross section and panel data estimators for nonseparable models with
endogenous regressors.
\newblock {\em Econometrica\/}~{\em 73\/}(4), 1053--1102.
\bibitem[\protect\citeauthoryear{Athey, Imbens, and Wager}{Athey
et~al.}{2018}]{athey2018approximate}
Athey, S., G.~W. Imbens, and S.~Wager (2018).
\newblock Approximate residual balancing: Debiased inference of average
treatment effects in high dimensions.
\newblock {\em Journal of the Royal Statistical Society: Series B (Statistical
Methodology)\/}~{\em 80\/}(4), 597--623.
\bibitem[\protect\citeauthoryear{Athey, Tibshirani, and Wager}{Athey
et~al.}{2019}]{athey2019generalized}
Athey, S., J.~Tibshirani, and S.~Wager (2019).
\newblock Generalized random forests.
\newblock {\em The Annals of Statistics\/}~{\em 47\/}(2), 1148--1178.
\bibitem[\protect\citeauthoryear{Belloni, Chen, Chernozhukov, and
Hansen}{Belloni et~al.}{2012}]{belloni2012sparse}
Belloni, A., D.~Chen, V.~Chernozhukov, and C.~Hansen (2012).
\newblock Sparse models and methods for optimal instruments with an application
to eminent domain.
\newblock {\em Econometrica\/}~{\em 80\/}(6), 2369--2429.
\bibitem[\protect\citeauthoryear{Belloni, Chernozhukov, Chetverikov, and
Fern{\'a}ndez-Val}{Belloni et~al.}{2019}]{belloni2019conditional}
Belloni, A., V.~Chernozhukov, D.~Chetverikov, and I.~Fern{\'a}ndez-Val (2019).
\newblock Conditional quantile processes based on series or many regressors.
\newblock {\em Journal of Econometrics\/}~{\em 213\/}(1), 4--29.
\bibitem[\protect\citeauthoryear{Belloni, Chernozhukov, and Hansen}{Belloni
et~al.}{2011}]{belloni2011inference}
Belloni, A., V.~Chernozhukov, and C.~Hansen (2011).
\newblock Inference for high-dimensional sparse econometric models.
\newblock {\em arXiv:1201.0220\/}.
\bibitem[\protect\citeauthoryear{Belloni, Chernozhukov, and Hansen}{Belloni
et~al.}{2014}]{belloni2014inference}
Belloni, A., V.~Chernozhukov, and C.~Hansen (2014).
\newblock Inference on treatment effects after selection among high-dimensional
controls.
\newblock {\em The Review of Economic Studies\/}~{\em 81\/}(2), 608--650.
\bibitem[\protect\citeauthoryear{Belloni, Chernozhukov, and Kato}{Belloni
et~al.}{2015}]{belloni2015uniform}
Belloni, A., V.~Chernozhukov, and K.~Kato (2015).
\newblock Uniform post-selection inference for least absolute deviation
regression and other {Z}-estimation problems.
\newblock {\em Biometrika\/}~{\em 102\/}(1), 77--94.
\bibitem[\protect\citeauthoryear{Belloni, Chernozhukov, and Wang}{Belloni
et~al.}{2014}]{belloni2014pivotal}
Belloni, A., V.~Chernozhukov, and L.~Wang (2014).
\newblock Pivotal estimation via square-root lasso in nonparametric regression.
\newblock {\em The Annals of Statistics\/}~{\em 42\/}(2), 757--788.
\bibitem[\protect\citeauthoryear{Bickel, Klaassen, Ritov, and Wellner}{Bickel
et~al.}{1993}]{bickel1993efficient}
Bickel, P.~J., C.~A. Klaassen, Y.~Ritov, and J.~A. Wellner (1993).
\newblock {\em Efficient and Adaptive Estimation for Semiparametric Models},
Volume~4.
\newblock Johns Hopkins University Press.
\bibitem[\protect\citeauthoryear{Bickel and Ritov}{Bickel and
Ritov}{1988}]{bickel1988estimating}
Bickel, P.~J. and Y.~Ritov (1988).
\newblock Estimating integrated squared density derivatives: Sharp best order
of convergence estimates.
\newblock {\em Sankhy{\=a}: The Indian Journal of Statistics, Series A\/},
381--393.
\bibitem[\protect\citeauthoryear{Bickel, Ritov, and Tsybakov}{Bickel
et~al.}{2009}]{bickel2009simultaneous}
Bickel, P.~J., Y.~Ritov, and A.~B. Tsybakov (2009).
\newblock Simultaneous analysis of {L}asso and {D}antzig selector.
\newblock {\em The Annals of Statistics\/}~{\em 37\/}(4), 1705--1732.
\bibitem[\protect\citeauthoryear{Blundell, Horowitz, and Parey}{Blundell
et~al.}{2012}]{blundell2012measuring}
Blundell, R., J.~L. Horowitz, and M.~Parey (2012).
\newblock Measuring the price responsiveness of gasoline demand: Economic shape
restrictions and nonparametric demand estimation.
\newblock {\em Quantitative Economics\/}~{\em 3\/}(1), 29--51.
\bibitem[\protect\citeauthoryear{Bradic and Kolar}{Bradic and
Kolar}{2017}]{bradic2017uniform}
Bradic, J. and M.~Kolar (2017).
\newblock Uniform inference for high-dimensional quantile regression: Linear
functionals and regression rank scores.
\newblock {\em arXiv:1702.06209\/}.
\bibitem[\protect\citeauthoryear{Cai and Guo}{Cai and
Guo}{2017}]{cai2017confidence}
Cai, T.~T. and Z.~Guo (2017).
\newblock Confidence intervals for high-dimensional linear regression: Minimax
rates and adaptivity.
\newblock {\em The Annals of Statistics\/}~{\em 45\/}(2), 615--646.
\bibitem[\protect\citeauthoryear{Candes and Tao}{Candes and
Tao}{2007}]{candes2007dantzig}
Candes, E. and T.~Tao (2007).
\newblock The {D}antzig selector: Statistical estimation when $p$ is much
larger than $n$.
\newblock {\em The Annals of Statistics\/}~{\em 35\/}(6), 2313--2351.
\bibitem[\protect\citeauthoryear{Chatterjee}{Chatterjee}{2013}]{chatterjee2013assumptionless}
Chatterjee, S. (2013).
\newblock Assumptionless consistency of the lasso.
\newblock {\em arXiv:1303.5817\/}.
\bibitem[\protect\citeauthoryear{Chernozhukov, Chetverikov, Demirer, Duflo,
Hansen, Newey, and Robins}{Chernozhukov
et~al.}{2018a}]{chernozhukov2018double}
Chernozhukov, V., D.~Chetverikov, M.~Demirer, E.~Duflo, C.~Hansen, W.~Newey,
and J.~Robins (2018a).
\newblock Double/debiased machine learning for treatment and structural
parameters.
\newblock {\em The Econometrics Journal\/}~{\em 21\/}(1), C1--C68.
\bibitem[\protect\citeauthoryear{Chernozhukov, Chetverikov, Demirer, Duflo,
Hansen, Newey, and Robins}{Chernozhukov
et~al.}{2018b}]{chernozhukov2018double_data}
Chernozhukov, V., D.~Chetverikov, M.~Demirer, E.~Duflo, C.~Hansen, W.~Newey,
and J.~Robins (2018b).
\newblock Double/debiased machine learning for treatment and structural
parameters: Replication package.
\newblock {\em The Econometrics Journal\/}~{\em 21\/}(1), Data deposited at
\url{https://doi.org/10.1111/ectj.12097}.
\bibitem[\protect\citeauthoryear{Chernozhukov, Chetverikov, and
Kato}{Chernozhukov et~al.}{2013}]{chernozhukov2013gaussian}
Chernozhukov, V., D.~Chetverikov, and K.~Kato (2013).
\newblock Gaussian approximations and multiplier bootstrap for maxima of sums
of high-dimensional random vectors.
\newblock {\em The Annals of Statistics\/}~{\em 41\/}(6), 2786--2819.
\bibitem[\protect\citeauthoryear{Chernozhukov, Chetverikov, and
Kato}{Chernozhukov et~al.}{2017}]{chernozhukov2017central}
Chernozhukov, V., D.~Chetverikov, and K.~Kato (2017).
\newblock Central limit theorems and bootstrap in high dimensions.
\newblock {\em The Annals of Probability\/}~{\em 45\/}(4), 2309--2352.
\bibitem[\protect\citeauthoryear{Chernozhukov, Escanciano, Ichimura, Newey, and
Robins}{Chernozhukov et~al.}{2016}]{chernozhukov2016locally}
Chernozhukov, V., J.~C. Escanciano, H.~Ichimura, W.~K. Newey, and J.~M. Robins
(2016).
\newblock Locally robust semiparametric estimation.
\newblock {\em arXiv:1608.00033\/}.
\bibitem[\protect\citeauthoryear{Chernozhukov and Hansen}{Chernozhukov and
Hansen}{2004}]{chernozhukov2004effects}
Chernozhukov, V. and C.~Hansen (2004).
\newblock The effects of 401(k) participation on the wealth distribution: An
instrumental quantile regression analysis.
\newblock {\em Review of Economics and Statistics\/}~{\em 86\/}(3), 735--751.
\bibitem[\protect\citeauthoryear{Chernozhukov, Hansen, and
Spindler}{Chernozhukov et~al.}{2015}]{chernozhukov2015valid}
Chernozhukov, V., C.~Hansen, and M.~Spindler (2015).
\newblock Valid post-selection and post-regularization inference: An
elementary, general approach.
\newblock {\em Annual Review of Economics\/}~{\em 7\/}(1), 649--688.
\bibitem[\protect\citeauthoryear{Chernozhukov, Newey, and Robins}{Chernozhukov
et~al.}{2018}]{chernozhukov2018robins}
Chernozhukov, V., W.~K. Newey, and J.~Robins (2018).
\newblock Double/de-biased machine learning using regularized {R}iesz
representers.
\newblock Technical report, cemmap working paper, No. CWP15/18.
\bibitem[\protect\citeauthoryear{Chernozhukov, Newey, and Singh}{Chernozhukov
et~al.}{2018}]{chernozhukov2018learning}
Chernozhukov, V., W.~K. Newey, and R.~Singh (2018).
\newblock Learning {L}2 continuous regression functionals via regularized
{R}iesz representers.
\newblock {\em arXiv:1809.05224\/}.
\bibitem[\protect\citeauthoryear{Chernozhukov, Newey, and Singh}{Chernozhukov
et~al.}{2021}]{chernozhukov2021simple}
Chernozhukov, V., W.~K. Newey, and R.~Singh (2021).
\newblock A simple and general debiased machine learning theorem with finite
sample guarantees.
\newblock {\em arXiv:2105.15197\/}.
\bibitem[\protect\citeauthoryear{Colangelo and Lee}{Colangelo and
Lee}{2020}]{colangelo2020double}
Colangelo, K. and Y.-Y. Lee (2020).
\newblock Double debiased machine learning nonparametric inference with
continuous treatments.
\newblock {\em arXiv:2004.03036\/}.
\bibitem[\protect\citeauthoryear{D{\'\i}az and van~der Laan}{D{\'\i}az and
van~der Laan}{2013}]{diaz2013targeted}
D{\'\i}az, I. and M.~J. van~der Laan (2013).
\newblock Targeted data adaptive estimation of the causal dose--response curve.
\newblock {\em Journal of Causal Inference\/}~{\em 1\/}(2), 171--192.
\bibitem[\protect\citeauthoryear{Donoho, Elad, and Temlyakov}{Donoho
et~al.}{2005}]{donoho2005stable}
Donoho, D.~L., M.~Elad, and V.~N. Temlyakov (2005).
\newblock Stable recovery of sparse overcomplete representations in the
presence of noise.
\newblock {\em IEEE Transactions on Information Theory\/}~{\em 52\/}(1), 6--18.
\bibitem[\protect\citeauthoryear{Fan, Hsu, Lieli, and Zhang}{Fan
et~al.}{2019}]{lieli}
Fan, Q., Y.-C. Hsu, R.~P. Lieli, and Y.~Zhang (2019).
\newblock Estimation of conditional average treatment effects with
high-dimensional data.
\newblock {\em arXiv:1908.02399\/}.
\bibitem[\protect\citeauthoryear{Florens, Heckman, Meghir, and
Vytlacil}{Florens et~al.}{2008}]{florens2008identification}
Florens, J.-P., J.~J. Heckman, C.~Meghir, and E.~Vytlacil (2008).
\newblock Identification of treatment effects using control functions in models
with continuous, endogenous treatment and heterogeneous effects.
\newblock {\em Econometrica\/}~{\em 76\/}(5), 1191--1206.
\bibitem[\protect\citeauthoryear{Foster and Syrgkanis}{Foster and
Syrgkanis}{2019}]{foster2019orthogonal}
Foster, D.~J. and V.~Syrgkanis (2019).
\newblock Orthogonal statistical learning.
\newblock {\em arXiv:1901.09036\/}.
\bibitem[\protect\citeauthoryear{Galvao and Wang}{Galvao and
Wang}{2015}]{galvao2015uniformly}
Galvao, A.~F. and L.~Wang (2015).
\newblock Uniformly semiparametric efficient estimation of treatment effects
with a continuous treatment.
\newblock {\em Journal of the American Statistical Association\/}~{\em
110\/}(512), 1528--1542.
\bibitem[\protect\citeauthoryear{Genovese and Wasserman}{Genovese and
Wasserman}{2008}]{genovese2008adaptive}
Genovese, C. and L.~Wasserman (2008).
\newblock Adaptive confidence bands.
\newblock {\em The Annals of Statistics\/}~{\em 36\/}(2), 875--905.
\bibitem[\protect\citeauthoryear{Guo and Zhang}{Guo and
Zhang}{2019}]{guo:zhang}
Guo, Z. and C.-H. Zhang (2019).
\newblock Local inference in additive models with decorrelated local linear
estimator.
\newblock {\em arXiv:1907.12732\/}.
\bibitem[\protect\citeauthoryear{Hasminskii and Ibragimov}{Hasminskii and
Ibragimov}{1979}]{hasminskii1979nonparametric}
Hasminskii, R.~Z. and I.~A. Ibragimov (1979).
\newblock On the nonparametric estimation of functionals.
\newblock In {\em Proceedings of the Second Prague Symposium on Asymptotic
Statistics}.
\bibitem[\protect\citeauthoryear{Hausman and Newey}{Hausman and
Newey}{1995}]{hausman1995nonparametric}
Hausman, J.~A. and W.~K. Newey (1995).
\newblock Nonparametric estimation of exact consumers surplus and deadweight
loss.
\newblock {\em Econometrica\/}, 1445--1476.
\bibitem[\protect\citeauthoryear{Hernan and Robins}{Hernan and
Robins}{2019}]{hernan2019causal}
Hernan, M.~A. and J.~M. Robins (2019).
\newblock {\em Causal Inference}.
\newblock CRC.
\bibitem[\protect\citeauthoryear{Hirshberg and Wager}{Hirshberg and
Wager}{2017}]{hirshberg2017augmented}
Hirshberg, D.~A. and S.~Wager (2017).
\newblock Balancing out regression error: Efficient treatment effect estimation
without smooth propensities.
\newblock {\em arXiv:1712.00038v1\/}.
\bibitem[\protect\citeauthoryear{Hirshberg and Wager}{Hirshberg and
Wager}{2018}]{hirshberg2018debiased}
Hirshberg, D.~A. and S.~Wager (2018).
\newblock Debiased inference of average partial effects in single-index models.
\newblock {\em arXiv:1811.02547\/}.
\bibitem[\protect\citeauthoryear{Hirshberg and Wager}{Hirshberg and
Wager}{2019}]{hirshberg2019augmented}
Hirshberg, D.~A. and S.~Wager (2019).
\newblock Augmented minimax linear estimation.
\newblock {\em arXiv:1712.00038v5\/}.
\bibitem[\protect\citeauthoryear{Imbens and Newey}{Imbens and
Newey}{2009}]{imbens2009identification}
Imbens, G.~W. and W.~K. Newey (2009).
\newblock Identification and estimation of triangular simultaneous equations
models without additivity.
\newblock {\em Econometrica\/}~{\em 77\/}(5), 1481--1512.
\bibitem[\protect\citeauthoryear{Imbens and Rubin}{Imbens and
Rubin}{2015}]{imbens2015causal}
Imbens, G.~W. and D.~B. Rubin (2015).
\newblock {\em Causal Inference in Statistics, Social, and Biomedical
Sciences}.
\newblock Cambridge University Press.
\bibitem[\protect\citeauthoryear{Jankova and Van De~Geer}{Jankova and Van
De~Geer}{2015}]{jankova2015confidence}
Jankova, J. and S.~Van De~Geer (2015).
\newblock Confidence intervals for high-dimensional inverse covariance
estimation.
\newblock {\em Electronic Journal of Statistics\/}~{\em 9\/}(1), 1205--1229.
\bibitem[\protect\citeauthoryear{Jankova and Van De~Geer}{Jankova and Van
De~Geer}{2016}]{jankova2016confidence}
Jankova, J. and S.~Van De~Geer (2016).
\newblock Confidence regions for high-dimensional generalized linear models
under sparsity.
\newblock {\em arXiv:1610.01353\/}.
\bibitem[\protect\citeauthoryear{Jankova and Van De~Geer}{Jankova and Van
De~Geer}{2018}]{jankova2018semiparametric}
Jankova, J. and S.~Van De~Geer (2018).
\newblock Semiparametric efficiency bounds for high-dimensional models.
\newblock {\em The Annals of Statistics\/}~{\em 46\/}(5), 2336--2359.
\bibitem[\protect\citeauthoryear{Javanmard and Montanari}{Javanmard and
Montanari}{2014a}]{javanmard2014confidence}
Javanmard, A. and A.~Montanari (2014a).
\newblock Confidence intervals and hypothesis testing for high-dimensional
regression.
\newblock {\em The Journal of Machine Learning Research\/}~{\em 15\/}(1),
2869--2909.
\bibitem[\protect\citeauthoryear{Javanmard and Montanari}{Javanmard and
Montanari}{2014b}]{javanmard2014hypothesis}
Javanmard, A. and A.~Montanari (2014b).
\newblock Hypothesis testing in high-dimensional regression under the
{G}aussian random design model: Asymptotic theory.
\newblock {\em IEEE Transactions on Information Theory\/}~{\em 60\/}(10),
6522--6554.
\bibitem[\protect\citeauthoryear{Javanmard and Montanari}{Javanmard and
Montanari}{2018}]{javanmard2018debiasing}
Javanmard, A. and A.~Montanari (2018).
\newblock Debiasing the lasso: Optimal sample size for {G}aussian designs.
\newblock {\em The Annals of Statistics\/}~{\em 46\/}(6A), 2593--2622.
\bibitem[\protect\citeauthoryear{Jing, Shao, and Wang}{Jing
et~al.}{2003}]{jing2003self}
Jing, B.-Y., Q.-M. Shao, and Q.~Wang (2003).
\newblock Self-normalized {C}ram{\'e}r-type large deviations for independent
random variables.
\newblock {\em The Annals of Probability\/}~{\em 31\/}(4), 2167--2215.
\bibitem[\protect\citeauthoryear{Kallus and Zhou}{Kallus and
Zhou}{2018}]{kallus2018policy}
Kallus, N. and A.~Zhou (2018).
\newblock Policy evaluation and optimization with continuous treatments.
\newblock In {\em International Conference on Artificial Intelligence and
Statistics}, pp.\ 1243--1251.
\bibitem[\protect\citeauthoryear{Kennedy}{Kennedy}{2020}]{kennedy2020optimal}
Kennedy, E.~H. (2020).
\newblock Optimal doubly robust estimation of heterogeneous causal effects.
\newblock {\em arXiv:2004.14497\/}.
\bibitem[\protect\citeauthoryear{Kennedy, Ma, McHugh, and Small}{Kennedy
et~al.}{2017}]{kennedy2017nonparametric}
Kennedy, E.~H., Z.~Ma, M.~D. McHugh, and D.~S. Small (2017).
\newblock Nonparametric methods for doubly robust estimation of continuous
treatment effects.
\newblock {\em Journal of the Royal Statistical Society: Series B, Statistical
Methodology\/}~{\em 79\/}(4), 1229.
\bibitem[\protect\citeauthoryear{Lee, Okui, and Whang}{Lee
et~al.}{2017}]{lee2017doubly}
Lee, S., R.~Okui, and Y.-J. Whang (2017).
\newblock Doubly robust uniform confidence band for the conditional average
treatment effect function.
\newblock {\em Journal of Applied Econometrics\/}~{\em 32\/}(7), 1207--1225.
\bibitem[\protect\citeauthoryear{Luedtke and Van Der~Laan}{Luedtke and Van
Der~Laan}{2016}]{luedtke2016statistical}
Luedtke, A.~R. and M.~J. Van Der~Laan (2016).
\newblock Statistical inference for the mean outcome under a possibly
non-unique optimal treatment strategy.
\newblock {\em Annals of Statistics\/}~{\em 44\/}(2), 713.
\bibitem[\protect\citeauthoryear{Newey}{Newey}{1994a}]{newey1994asymptotic}
Newey, W.~K. (1994a).
\newblock The asymptotic variance of semiparametric estimators.
\newblock {\em Econometrica\/}, 1349--1382.
\bibitem[\protect\citeauthoryear{Newey}{Newey}{1994b}]{newey1994kernel}
Newey, W.~K. (1994b).
\newblock Kernel estimation of partial means and a general variance estimator.
\newblock {\em Econometric Theory\/}~{\em 10\/}(2), 1--21.
\bibitem[\protect\citeauthoryear{Newey, Hsieh, and Robins}{Newey
et~al.}{1998}]{newey1998undersmoothing}
Newey, W.~K., F.~Hsieh, and J.~M. Robins (1998).
\newblock Undersmoothing and bias corrected functional estimation.
\newblock Technical report, MIT Department of Economics.
\bibitem[\protect\citeauthoryear{Newey, Hsieh, and Robins}{Newey
et~al.}{2004}]{newey2004twicing}
Newey, W.~K., F.~Hsieh, and J.~M. Robins (2004).
\newblock Twicing kernels and a small bias property of semiparametric
estimators.
\newblock {\em Econometrica\/}~{\em 72\/}(3), 947--962.
\bibitem[\protect\citeauthoryear{Newey and Robins}{Newey and
Robins}{2018}]{newey2018cross}
Newey, W.~K. and J.~R. Robins (2018).
\newblock Cross-fitting and fast remainder rates for semiparametric estimation.
\newblock {\em arXiv:1801.09138\/}.
\bibitem[\protect\citeauthoryear{Neykov, Ning, Liu, and Liu}{Neykov
et~al.}{2018}]{neykov2018unified}
Neykov, M., Y.~Ning, J.~S. Liu, and H.~Liu (2018).
\newblock A unified theory of confidence regions and testing for
high-dimensional estimating equations.
\newblock {\em Statistical Science\/}~{\em 33\/}(3), 427--443.
\bibitem[\protect\citeauthoryear{Neyman}{Neyman}{1959}]{neyman1959}
Neyman, J. (1959).
\newblock Optimal asymptotic tests of composite statistical hypotheses.
\newblock In {\em Probability and Statistics}, pp.\ 416–444. Wiley.
\bibitem[\protect\citeauthoryear{Nie and Wager}{Nie and
Wager}{2017}]{nie2017quasi}
Nie, X. and S.~Wager (2017).
\newblock Quasi-oracle estimation of heterogeneous treatment effects.
\newblock {\em arXiv:1712.04912\/}.
\bibitem[\protect\citeauthoryear{Ning and Liu}{Ning and
Liu}{2017}]{ning2017general}
Ning, Y. and H.~Liu (2017).
\newblock A general theory of hypothesis tests and confidence regions for
sparse high dimensional models.
\newblock {\em The Annals of Statistics\/}~{\em 45\/}(1), 158--195.
\bibitem[\protect\citeauthoryear{Peters, Janzing, and Sch{\"o}lkopf}{Peters
et~al.}{2017}]{peters2017elements}
Peters, J., D.~Janzing, and B.~Sch{\"o}lkopf (2017).
\newblock {\em Elements of Causal Inference: Foundations and Learning
Algorithms}.
\newblock MIT press.
\bibitem[\protect\citeauthoryear{Poterba and Venti}{Poterba and
Venti}{1994}]{poterba1994}
Poterba, J.~M. and S.~F. Venti (1994).
\newblock 401(k) plans and tax-deferred saving.
\newblock In {\em Studies in the Economics of Aging}, pp.\ 105--142.
University of Chicago Press.
\bibitem[\protect\citeauthoryear{Poterba, Venti, and Wise}{Poterba
et~al.}{1995}]{poterba1995}
Poterba, J.~M., S.~F. Venti, and D.~A. Wise (1995).
\newblock Do 401(k) contributions crowd out other personal saving?
\newblock {\em Journal of Public Economics\/}~{\em 58\/}(1), 1--32.
\bibitem[\protect\citeauthoryear{Ren, Sun, Zhang, and Zhou}{Ren
et~al.}{2015}]{ren2015asymptotic}
Ren, Z., T.~Sun, C.-H. Zhang, and H.~H. Zhou (2015).
\newblock Asymptotic normality and optimalities in estimation of large
{G}aussian graphical models.
\newblock {\em The Annals of Statistics\/}~{\em 43\/}(3), 991--1026.
\bibitem[\protect\citeauthoryear{Robins, Sued, Lei-Gomez, and Rotnitzky}{Robins
et~al.}{2007}]{robins2007comment}
Robins, J., M.~Sued, Q.~Lei-Gomez, and A.~Rotnitzky (2007).
\newblock Comment on "performance of double-robust estimators when inverse
probability weights are highly variable".
\newblock {\em Statistical Science\/}~{\em 22\/}(4), 544--559.
\bibitem[\protect\citeauthoryear{Robins and Rotnitzky}{Robins and
Rotnitzky}{1995}]{robins1995semiparametric}
Robins, J.~M. and A.~Rotnitzky (1995).
\newblock Semiparametric efficiency in multivariate regression models with
missing data.
\newblock {\em Journal of the American Statistical Association\/}~{\em
90\/}(429), 122--129.
\bibitem[\protect\citeauthoryear{Robins and Rotnitzky}{Robins and
Rotnitzky}{2001}]{robins2001comment}
Robins, J.~M. and A.~Rotnitzky (2001).
\newblock Comment on “inference for semiparametric models: Some questions and
an answer”.
\newblock {\em Statistica Sinica\/}~{\em 11\/}(4), 920--936.
\bibitem[\protect\citeauthoryear{Robins, Rotnitzky, and Zhao}{Robins
et~al.}{1995}]{robins1995analysis}
Robins, J.~M., A.~Rotnitzky, and L.~P. Zhao (1995).
\newblock Analysis of semiparametric regression models for repeated outcomes in
the presence of missing data.
\newblock {\em Journal of the American Statistical Association\/}~{\em
90\/}(429), 106--121.
\bibitem[\protect\citeauthoryear{Rothenh{\"a}usler and Yu}{Rothenh{\"a}usler
and Yu}{2019}]{rothenhausler2019incremental}
Rothenh{\"a}usler, D. and B.~Yu (2019).
\newblock Incremental causal effects.
\newblock {\em arXiv:1907.13258\/}.
\bibitem[\protect\citeauthoryear{Rubin and van~der Laan}{Rubin and van~der
Laan}{2005}]{rubin2005general}
Rubin, D. and M.~J. van~der Laan (2005).
\newblock A general imputation methodology for nonparametric regression with
censored data.
\newblock Technical report, UC Berkeley Division of Biostatistics.
\bibitem[\protect\citeauthoryear{Rubin and van~der Laan}{Rubin and van~der
Laan}{2006}]{rubin2006extending}
Rubin, D. and M.~J. van~der Laan (2006).
\newblock Extending marginal structural models through local, penalized, and
additive learning.
\newblock Technical report, UC Berkeley Division of Biostatistics.
\bibitem[\protect\citeauthoryear{Schick}{Schick}{1986}]{schick1986asymptotically}
Schick, A. (1986).
\newblock On asymptotically efficient estimation in semiparametric models.
\newblock {\em The Annals of Statistics\/}~{\em 14\/}(3), 1139--1151.
\bibitem[\protect\citeauthoryear{Schmalensee and Stoker}{Schmalensee and
Stoker}{1999}]{schmalensee1999household}
Schmalensee, R. and T.~M. Stoker (1999).
\newblock Household gasoline demand in the {U}nited {S}tates.
\newblock {\em Econometrica\/}~{\em 67\/}(3), 645--662.
\bibitem[\protect\citeauthoryear{Semenova and Chernozhukov}{Semenova and
Chernozhukov}{2021a}]{semenova2020debiased}
Semenova, V. and V.~Chernozhukov (2021a).
\newblock Debiased machine learning of conditional average treatment effects
and other causal functions.
\newblock {\em The Econometrics Journal\/}~{\em 24\/}(2), 264--289.
\bibitem[\protect\citeauthoryear{Semenova and Chernozhukov}{Semenova and
Chernozhukov}{2021b}]{semenova2020debiased_data}
Semenova, V. and V.~Chernozhukov (2021b).
\newblock Debiased machine learning of conditional average treatment effects
and other causal functions: Replication package.
\newblock {\em The Econometrics Journal\/}~{\em 24\/}(2), Data deposited at
\url{https://doi.org/10.1093/ectj/utaa027}.
\bibitem[\protect\citeauthoryear{Shevtsova}{Shevtsova}{2011}]{shevtsova2011absolute}
Shevtsova, I. (2011).
\newblock On the absolute constants in the {B}erry-{E}sseen type inequalities
for identically distributed summands.
\newblock {\em arXiv:1111.6554\/}.
\bibitem[\protect\citeauthoryear{Toth and van~der Laan}{Toth and van~der
Laan}{2016}]{toth2016tmle}
Toth, B. and M.~van~der Laan (2016).
\newblock {TMLE} for marginal structural models based on an instrument.
\newblock Technical report, UC Berkeley Division of Biostatistics.
\bibitem[\protect\citeauthoryear{Tsybakov}{Tsybakov}{2012}]{tsybakov2012introduction}
Tsybakov, A.~B. (2012).
\newblock {\em Introduction to Nonparametric Estimation}.
\newblock Springer.
\bibitem[\protect\citeauthoryear{Van~de Geer, B{\"u}hlmann, Ritov, and
Dezeure}{Van~de Geer et~al.}{2014}]{van2014asymptotically}
Van~de Geer, S., P.~B{\"u}hlmann, Y.~Ritov, and R.~Dezeure (2014).
\newblock On asymptotically optimal confidence regions and tests for
high-dimensional models.
\newblock {\em The Annals of Statistics\/}~{\em 42\/}(3), 1166--1202.
\bibitem[\protect\citeauthoryear{Van Der~Laan and Dudoit}{Van Der~Laan and
Dudoit}{2003}]{van2003unified}
Van Der~Laan, M.~J. and S.~Dudoit (2003).
\newblock Unified cross-validation methodology for selection among estimators
and a general cross-validated adaptive epsilon-net estimator: Finite sample
oracle inequalities and examples.
\newblock Technical report, UC Berkeley Division of Biostatistics.
\bibitem[\protect\citeauthoryear{van~der Laan and Luedtke}{van~der Laan and
Luedtke}{2014}]{van2014targeted}
van~der Laan, M.~J. and A.~R. Luedtke (2014).
\newblock Targeted learning of an optimal dynamic treatment, and statistical
inference for its mean outcome.
\newblock Technical report, UC Berkeley Division of Biostatistics.
\bibitem[\protect\citeauthoryear{Van~der Laan and Rose}{Van~der Laan and
Rose}{2011}]{van2011targeted}
Van~der Laan, M.~J. and S.~Rose (2011).
\newblock {\em Targeted Learning: Causal Inference for Observational and
Experimental Data}.
\newblock Springer Science \& Business Media.
\bibitem[\protect\citeauthoryear{Van Der~Laan and Rubin}{Van Der~Laan and
Rubin}{2006}]{van2006targeted}
Van Der~Laan, M.~J. and D.~Rubin (2006).
\newblock Targeted maximum likelihood learning.
\newblock {\em The International Journal of Biostatistics\/}~{\em 2\/}(1).
\bibitem[\protect\citeauthoryear{Van Der~Vaart}{Van
Der~Vaart}{1991}]{van1991differentiable}
Van Der~Vaart, A. (1991).
\newblock On differentiable functionals.
\newblock {\em The Annals of Statistics\/}~{\em 19\/}(1), 178--204.
\bibitem[\protect\citeauthoryear{Van~der Vaart}{Van~der Vaart}{2000}]{vaart}
Van~der Vaart, A.~W. (2000).
\newblock {\em Asymptotic Statistics}, Volume~3.
\newblock Cambridge University Press.
\bibitem[\protect\citeauthoryear{Van Der~Vaart and Wellner}{Van Der~Vaart and
Wellner}{1996}]{van1996weak}
Van Der~Vaart, A.~W. and J.~A. Wellner (1996).
\newblock Weak convergence.
\newblock In {\em Weak Convergence and Empirical Processes}, pp.\ 16--28.
Springer.
\bibitem[\protect\citeauthoryear{Wang, Rotnitzky, and Lin}{Wang
et~al.}{2010}]{wang2010nonparametric}
Wang, L., A.~Rotnitzky, and X.~Lin (2010).
\newblock Nonparametric regression with missing outcomes using weighted kernel
estimating equations.
\newblock {\em Journal of the American Statistical Association\/}~{\em
105\/}(491), 1135--1146.
\bibitem[\protect\citeauthoryear{Yatchew and No}{Yatchew and
No}{2001}]{yatchew2001household}
Yatchew, A. and J.~A. No (2001).
\newblock Household gasoline demand in {C}anada.
\newblock {\em Econometrica\/}~{\em 69\/}(6), 1697--1709.
\bibitem[\protect\citeauthoryear{Ye and Zhang}{Ye and Zhang}{2010}]{ye2010rate}
Ye, F. and C.-H. Zhang (2010).
\newblock Rate minimaxity of the lasso and {D}antzig selector for the $\ell_q$
loss in $\ell_r$ balls.
\newblock {\em Journal of Machine Learning Research\/}~{\em 11\/}(Dec),
3519--3540.
\bibitem[\protect\citeauthoryear{Zhang and Zhang}{Zhang and
Zhang}{2014}]{zhang2014confidence}
Zhang, C.-H. and S.~S. Zhang (2014).
\newblock Confidence intervals for low dimensional parameters in high
dimensional linear models.
\newblock {\em Journal of the Royal Statistical Society: Series B (Statistical
Methodology)\/}~{\em 76\/}(1), 217--242.
\bibitem[\protect\citeauthoryear{Zhu and Bradic}{Zhu and
Bradic}{2017}]{zhu2017breaking}
Zhu, Y. and J.~Bradic (2017).
\newblock Breaking the curse of dimensionality in regression.
\newblock {\em arXiv:1708.00430.\/}.
\bibitem[\protect\citeauthoryear{Zhu and Bradic}{Zhu and
Bradic}{2018}]{zhu2018linear}
Zhu, Y. and J.~Bradic (2018).
\newblock Linear hypothesis testing in dense high-dimensional linear models.
\newblock {\em Journal of the American Statistical Association\/}~{\em
113\/}(524), 1583--1600.
\bibitem[\protect\citeauthoryear{Zimmert and Lechner}{Zimmert and
Lechner}{2019}]{lechner}
Zimmert, M. and M.~Lechner (2019).
\newblock Nonparametric estimation of causal heterogeneity under
high-dimensional confounding.
\newblock {\em arXiv:1908.08779\/}.
\end{thebibliography}