Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
Rendered from LaTeX for readability, not typeset faithfully. Citation keys are highlighted; maths is left as source; figures, tables and equation environments are summarised rather than reproduced; unrecognised commands are greyed out so nothing is silently dropped. Email addresses are removed.
It's Hard to Be Normal: The Impact of Noise on Structure-agnostic Estimation
\doparttoc
\faketableofcontents
abstractStructure-agnostic causal inference studies how well one can estimate a treatment effect given black-box machine learning estimates of nuisance functions (like the impact of confounders on treatment and outcomes).
Here, we find that the answer depends in a surprising way on the distribution of the treatment noise. Focusing on the partially linear model of robinson1988root, we first show that the widely adopted double machine learning (DML) estimator is minimax rate-optimal for Gaussian treatment noise, resolving an open problem of mackey2018orthogonal. Meanwhile, for independent non-Gaussian treatment noise, we show that DML is always suboptimal by constructing new practical procedures with higher-order robustness to nuisance errors. These ACE procedures use structure-agnostic cumulant estimators to achieve $r$-th order insensitivity to nuisance errors whenever the $(r+1)$-st treatment cumulant is non-zero. We complement these core results with novel minimax guarantees for binary treatments in the partially linear model. Finally, using synthetic demand estimation experiments, we demonstrate the practical benefits of our higher-order robust estimators.
Introduction
Modern machine learning (ML) offers a rich toolbox of flexible methods for modeling complex, high‑dimensional functions --- ranging from regularized linear regression belloni2014pivotal,zou2005regularization and random forests breiman2001random,biau2008consistency,syrgkanis2020estimation to neural networks schmidt2020nonparametric,farrell2021deep and hybrid combinations of these techniques dvzeroski2004combining,chernozhukov2022automatic. For statisticians and econometricians, it is natural to ask whether ML can improve both the accuracy and the robustness of estimating a target parameter of interest. Recently, balakrishnan2023fundamental introduced the paradigm of structure‑agnostic estimation (SAE).
SAE enables parameter inference by directly plugging in black-box ML‑based estimates of the so‑called nuisance functions ---
the other regression components that impact our observations while not being of primary inferential interest.
This framework characterizes the best possible target estimation accuracy in terms of the error of these nuisance estimates. By contrast, the classical semiparametric approach derives optimal error rates under explicit structural assumptions on the nuisance --- such as smoothness or sparsity --- which must be exploited by the estimator and can render it fragile when those assumptions fail stone1982optimal,chen1999improved,belloni2011inference.
In the field of causal inference, a typical goal is to estimate the impact of a treatment on an observed outcome in the presence of confounders $X$ that impact both treatment $T$ and outcome $Y$ in largely unknown ways.
When all confounders are observed, many causal estimands admit expressions in terms of regression functions that are themselves estimable from data. Consequently, causal parameter estimation falls squarely within the regime where ML‑based nuisance estimates may provide benefits.
Recently, double/debiased machine learning (DML) chernozhukov2018double,chernozhukov2022automatic
was proposed as an efficient way to estimate causal parameters from black-box nuisance estimates. Specifically, given $n$ independent and identically distributed (i.i.d.) observations and nuisance estimates with mean squared error ${\epsilon}$, these methods achieve ${\mathcal{O}}({\epsilon}^2+n^{-1/2})$ error rates, improving over the ${\mathcal{O}}({\epsilon}+n^{-1/2})$ rate achieved by naively plugging in the nuisance estimates. In particular, the estimates are $\sqrt{n}$-consistent even when the nuisance estimates converge as slowly as $n^{-1/4}$. Moreover, this rate is structure-agnostic, and recent works balakrishnan2023fundamental,jin2024structure proved that for several important causal parameters, this rate is in fact minimax rate-optimal in the SAE framework. In other words,
one cannot achieve smaller estimation estimation rates without strong prior knowledge on the underlying structure of the nuisance. These optimality results provide a strong justification for the popularity of doubly robust learning methods in practice.
A curious exception to this rule can be found in the work of mackey2018orthogonal. There, the authors study the popular partially linear model of robinson1988root:
equation[equation omitted — 286 chars of source]
Here, the observation $Z=(X,T,Y)$ comprises covariates $X \in {\mathcal{X}}$, a scalar treatment $T \in \mathbb{R}$, and a scalar outcome $Y \in \mathbb{R}$. The primary objective is the estimation of the parameter $\theta_0$, which represents the average causal effect of $T$ on $Y$. Within this framework, the DML estimator for $\theta_0$ is derived from the Neyman-orthogonal moment condition and typically takes the form:
equation[equation omitted — 227 chars of source]
where $\hat{g}$ and $\hat{q}$ are machine learning estimators for the nuisance functions $g_0(X) = \mathbb{E}[T \mid X]$ and $q_0(X) = \mathbb{E}[Y \mid X]$, respectively chernozhukov2018double. Common practice involves sample splitting, where one portion of the data is used to train $\hat{g}$ and $\hat{q}$ and another (independent) portion is used to compute $\hat{\theta}_{\mathrm{DML}}$ via (ref).
However, mackey2018orthogonal showed that for the model (ref), one can design structure-agnostic estimators that are more efficient than DML if the treatment noise $\eta$ is non-Gaussian and independent of $X$.\footnote{The structure-agnostic rates are not explicitly stated in mackey2018orthogonal but can be straightforwardly derived from their analysis.} bonhomme2024neyman consider parameter estimation problems in a conditional likelihood setting and show that arbitrarily higher-order orthogonal estimators can be constructed if the conditional likelihood function is known up to the nuisance parameters. These works highlight a gap in our understanding of how noise impacts optimal SAE rates, and this is the main question that we will address in this paper.
Given access to black-box nuisance estimates, we make the following contributions:
itemize[leftmargin=2em]
• On the one hand, we prove in (ref) the existence of Gaussian treatment barrier: for Gaussian treatment, no estimators can achieve a better rate than DML, even when the variance of the treatment is completely known. This implies that leveraging distributional information of the treatment noise as in mackey2018orthogonal cannot yield a better estimate and that their restrictions to the non-Gaussian setting is not an algorithmic issue.
• On the other hand, for non-Gaussian treatment, we propose a general procedure to construct higher-order orthogonal moment functions and provide structure-agnostic guarantees in (ref). Then, for treatment noise independent of $X$, we derive in (ref) a new agnostic cumulant-based estimator (ACE) that achieves $r$-th order insensitivity to nuisance estimate error ${\epsilon}$ and ${\mathcal{O}}({\epsilon}^r+n^{-1/2})$ error rates for treatment effect estimation whenever the $(r+1)$-st cumulant of $\eta$ is non-zero. To the best of our knowledge, this is the first structure-agnostic estimator that achieves arbitrarily high-order robustness.
• We complement these findings with additional contributions relevant to this setting. Specifically, we show in (ref) that DML is minimax optimal for (ref) with binary treatment, a case not covered in existing lower bounds that do not assume a partially linear outcome model. We also derive new lower and upper bounds for structure-agnostic moment and cumulant estimation in a standard non-parametric model in (ref) that might be of independent interest. Finally, in (ref), we conduct a synthetic demand estimation experiment, highlighting the benefits of the ACE estimator compared with existing approaches.
\paragraph{Notation}
We introduce the shorthand $[m] \triangleq\{1,\dots,m\}$ for each $m\in\mathbb{N}$.
For a function $g$ and distribution $P$ on a domain ${\mathcal{Z}}$, we let $\left\| g \right\|_{P,s}\triangleq \left\| g \right\|_{L^s(P)} \triangleq (\int |g(z)|^sdP(z))^{1/s}$ with $s\geq 1$ represent the $L^s(P)$ norm.
For a vector $\epsilon\in\mathbb{R}^l$, we define $\left\| {\epsilon} \right\|_{\infty} \triangleq \max_{1\leq i\leq l}|{\epsilon}_i|$. For two vectors $\alpha=(\alpha_1,\cdots,\alpha_l),\beta=(\beta_1,\cdots,\beta_l)\in\mathbb{R}^l$, we write $\alpha\leq\beta$ if $\alpha_i\leq\beta_i,1\leq i\leq l$.
Structure-agnostic estimation and minimax error
To evaluate and compare method quality in this work, we adopt the minimax structure-agnostic framework of balakrishnan2023fundamental.
Notably, structure-agnostic analyses make no explicit assumptions about nuisance smoothness, sparsity, or structure and instead simply assume access to black-box nuisance estimates with certain unobserved error levels.
We first define the class of all data generating distributions $P$ on our data domain ${\mathcal{Z}}$ and assign to each a target parameter $\theta_0(P)$ and a vector-valued nuisance function $h: {\mathcal{Z}}\mapsto \mathbb{R}^\ell$.
definition[Data generating distributions, target parameters, and nuisance functions]
Throughout, we let ${\mathcal{P}}$ be the set of candidate data generating distributions on the finite dimensional-domain ${\mathcal{Z}}$, let ${\mathcal{H}}\subseteq\left(\mathbb{R}^\ell\right)^{{\mathcal{Z}}}$ be the set of relevant vector-valued nuisance functions, and let $\Phi$ be a deterministic mapping from ${\mathcal{P}}$ to ${\mathcal{H}}$. Then, for any $P\in{\mathcal{P}}$, we say that $\theta_0(P)\in\mathbb{R}$ is the target parameter and $h=\Phi(P)\in{\mathcal{H}}$ is the nuisance function corresponding to the distribution $P$.
We will sometimes abuse notation and choose ${\mathcal{H}}$ to be a subset of ${\mathcal{H}}\subseteq\left(\mathbb{R}^\ell\right)^{{\mathcal{X}}}$ where ${\mathcal{X}}$ is the domain of the covariate component $X$ of $Z$.
Departing from prior work balakrishnan2023fundamental,jin2024structure, we leave the exact choice of nuisance mapping $\Phi$ unspecified in (ref). This allows us to study how the choice of nuisance functions affects the estimation error of the target parameter of interest.
Next, we introduce the ground-truth uncertainty sets associated with any nuisance estimate $\hat{h}$ and any target error level $\epsilon$. For any vector-valued function $h^{\star}:{\mathcal{X}}\mapsto\mathbb{R}^\ell$, distribution $P\in{\mathcal{P}}$, and $\epsilon\in\mathbb{R}_+^\ell$, we define ${\mathcal{B}}_{P,s}(h^{\star},\epsilon):=\big\{h_i\in L^s(P): \|h_i-h_i^{\star}\|_{P,s}\leq \epsilon_i, \forall i\in [\ell]\}$.
definition[Uncertainty sets]
For any nuisance estimate $\hat{h}\in{\mathcal{H}}$, error level vector ${\epsilon}\in\mathbb{R}_{+}^\ell$, and power $s\geq 1$, we define the uncertainty set ${\mathcal{P}}_{s,{\epsilon}}(\hat{h};\Phi)$ as the set of all $P\in{\mathcal{P}}_0$ satisfying
\begin{equation}
\|\hat{h}_i-h_i\|_{P,s} \leq {\epsilon}_{i},\quadfor \quad h = \Phi(P) \quadand each\quad i\in \{1,2,\cdots,\ell\},
\end{equation}
or, equivalently, $\Phi(P) \in {\mathcal{B}}_{P,s}(\hat{h}, {\epsilon})$.
For convenience, we will omit the dependency of ${\mathcal{P}}_{s,{\epsilon}}(\hat{h};\Phi)$ on ${\mathcal{P}}$, which will always be clear from context.
We will sometimes write ${\mathcal{P}}_{s,{\epsilon}}(\hat{h})$ when the choice of $\Phi$ is obvious.
Finally, for a given estimator $\hat{\theta}$ of a target parameter $\theta_0(P)$, we define the worst-case error over an uncertainty set.
definition[Minimax estimation error]
For any set of distributions ${\mathcal{P}}$ over ${\mathcal{Z}}$, we define the worst-case $(1-\gamma)$-quantile error of an estimator $\hat{\theta}:{\mathcal{Z}}^{\otimes n}\mapsto\mathbb{R}$ as
$\mathfrak{R}_{n,1-\gamma}(\hat{\theta}; {\mathcal{P}}) \triangleq \sup_{P\in{\mathcal{P}}}{\mathcal{Q}}_{P,1-\gamma}(|\hat{\theta} - \theta_0(P)|)$, where
${\mathcal{Q}}_{P,1-\gamma}(|\hat{\theta} - \theta_0(P)|)$ is a $(1-\gamma)$-quantile of $|\hat{\theta}(Z_1, \dots, Z_n)- \theta_0(P)|$ when $Z_i\overset{\textrm{\tiny\textrm{i.i.d.}\xspace}}{\sim} P$.
We further define the minimax estimation error of ${\mathcal{P}}$ as $\mathfrak{M}_{n,1-\gamma}({\mathcal{P}}) \triangleq \inf_{\hat{\theta}:{\mathcal{Z}}^{\otimes n}\mapsto\mathbb{R}} \mathfrak{R}_{1-\gamma}(\hat{\theta}; {\mathcal{P}}).$
Structure-agnostic lower bounds
In this section, we establish structure-agnostic lower bounds for treatment effect estimation in the partially linear model (ref).
Optimality of DML for binary treatment
We begin by establishing the minimax rate-optimality of DML when the treatment $T$ is binary.
Previous works balakrishnan2023fundamental,jin2024structure have established structure-agnostic lower bounds of a similar form. However, balakrishnan2023fundamental consider the estimation of the expected conditional covariance defined as $\mathbb{E}[\mathrm{Cov}(T,Y\mid X)]$, while jin2024structure consider the estimation of the average treatment effect with a different set of nuisance functions. Furthermore, neither work constrains the form of $\mathbb{E}[Y\mid T,X]$, while we assume a partially linear structure for the outcome model. Our result implies that even with this additional assumption, it is still impossible to improve over DML. For convenience, we introduce the following definitions as the “default" choice for the class of data generating distributions and nuisance functions:
definition[Set of feasible distributions]
We define ${\mathcal{P}}^{\star}$ as the set of all distributions of $(X,T,Y)$ generated by (ref). Moreover, we define the following subsets of ${\mathcal{P}}^{\star}$ as follows:
\begin{itemize}
• for any constants $C_{\uptheta},C_{\mathsf{T}},C_{\mathsf{Y}} \in[0,+\infty]$, we use ${\mathcal{P}}_{r}(C_{\uptheta},C_{\mathsf{T}},C_{\mathsf{Y}})$ to denote all distributions $P\in{\mathcal{P}}^{\star}$ that satisfy $|\theta_0|\leq C_{\uptheta}, \mathbb{E}[|T|^r]^{1/r}\leq C_{\mathsf{T}}$, and $\mathbb{E}[|Y|^r]^{1/r}\leq C_{\mathsf{Y}}$.
• for any constants $C_{\uptheta},C_{\mathsf{g}},C_{\mathsf{q}}, \psi_{\upxi}, \psi_{\upeta} \in (0,+\infty]$, we use ${\mathcal{P}}^{\star}(C_{\uptheta},C_{\mathsf{g}},C_{\mathsf{q}};\psi_{\upxi}, \psi_{\upeta})$ to denote the set of all distributions in $P\in {\mathcal{P}}^{\star}$ that satisfy
\begin{enumerate}[leftmargin=0.5em]
• $|\theta_0|\leq C_{\uptheta}, |g_0(X)|\leq C_{\mathsf{g}}$, $|q_0(X)|\leq C_{\mathsf{q}}$ a.s. for $(g_0,q_0)(X)=(\mathbb{E}[T\mid X],\mathbb{E}[Y\mid X])$
• $\xi\mid X$ and $\eta\mid X$ are $\psi_{\upxi}$ and $\psi_{\upeta}$-sub-Gaussian a.s..
\end{enumerate}
• for any constants $C_{\uptheta},C_{\mathsf{T}},C_{\mathsf{Y}} \in[0,+\infty]$, we use ${\mathcal{P}}_{\mathsf{b}}(C_{\uptheta},C_{\mathsf{T}},C_{\mathsf{Y}})$ to denote all distributions $P\in{\mathcal{P}}^{\star}$ that satisfy $|\theta_0|\leq C_{\uptheta}, |T|\leq C_{\mathsf{T}}$, and $|Y|\leq C_{\mathsf{Y}}$.
\end{itemize}
Finally, we define $\Phi^{\star}$ as a mapping from $P\in{\mathcal{P}}^{\star}$ to the “default" nuisance functions $h_0=(g_0,q_0)$.
We are interested in the minimax structure-agnostic estimation error induced by ${\mathcal{P}} = {\mathcal{P}}_{s,{\epsilon}}(\hat{h},\Phi)$ for some $s\geq 1$, in the sense of (ref), with ${\mathcal{P}}$ being chosen as a set of distributions that satisfies certain mild regularity conditions, such as the ones introduced in (ref). In other words, given black-box nuisance estimates of $h_0$ with $L^s(P)$ error rates, we would like to derive the optimal worst-case estimation error for the treatment effect $\theta_0$.
Our main result in this section is a lower bound for estimating $\theta_0$ when $T$ is a Bernoulli random variable. Our lower bound, (ref), is established under the additional assumption that $X$ has a uniform distribution on ${\mathcal{X}}=[0,1]^K$; as a consequence, the minimax error rate of DML cannot be improved even when the marginal distribution of $X$ is known.
theorem[Structure-agnostic lower bound for binary treatment]
Fix any $C_{\uptheta}>0$, $c_q,\delta\in(0,\frac{1}{4})$, and $K\in\mathbb{N}_+$, and let
\begin{align}
{\mathcal{P}} = \{P\in{\mathcal{P}}_{\mathsf{b}}(C_{\uptheta},1,1): T\in\{0,1\}, \mathrm{Var}_P(T\mid X)\geq \delta \ \ and\ \ X\sim Unif([0,1]^K)\}.
\end{align}
If $\left\| {\epsilon} \right\|_{\infty}\leq \delta/2$, then for any estimates $\hat{h}=(\hat{g},\hat{q})$ with $c_q \leq \hat{q}(X)\leq 1-c_q$ and $\hat{g}(X)(1-\hat{g}(X))\geq A^{-1}\delta, a.s.$, we have
\begin{equation}
\mathfrak{M}_{n,1-\gamma}\left({\mathcal{P}}_{2,{\epsilon}}(\hat{h},\Phi^{\star})\right) \geq c_{\gamma} \Big[ A^{-1} \delta^{-1}(c_q{\epsilon}_1^2+{\epsilon}_1{\epsilon}_2) + \delta^{-1/2} n^{-1/2} \Big],
\end{equation}
for any $\gamma\in(1/2,1)$, where $c_\gamma$ is a universal constant that only depends on $\gamma$.
When $\left\| {\epsilon} \right\|_{\infty}\leq \delta/2$, this matches the upper bound achieved by DML up to constant factors, stated in (ref) for completeness. The proof of (ref) can be found in (ref).
The Gaussian treatment barrier
In the previous section, we established the rate-optimality of DML for binary treatment. Is it possible to improve over DML if we make different distributional assumptions? In the literature, it is not uncommon to model the treatment assignment rule using a Gaussian distribution, i.e., $\eta\mid X \sim {\mathcal{N}}(0,\sigma(X)^2)$ for some function $\sigma(\cdot)$ imai2004causal,zhao2020propensity. However, in mackey2018orthogonal it is shown that Gaussianality of the noise variable $\eta$ is in fact a barrier for one to construct second-order orthogonal moments, thereby preventing them from deriving better error rates than DML by leveraging distributional information of $\eta$. However, it is unclear whether this is an issue specific to their approach or if there indeed exists a fundamental, non-algorithmic barrier for Gaussian treatment.
In this section, we resolve this open question and show that the latter is true: if the treatment noise is Gaussian, then DML is already minimax rate-optimal even when $\eta$ is independent of $X$ and one has exact knowledge of its distribution. Our lower bound, stated next, is proved in (ref).
theorem[The Gaussian treatment barrier]
Let $\sigma, C_{\uptheta},C_{\mathsf{g}},C_{\mathsf{q}} > 0$ be known constants and
${\mathcal{P}} = \{P\in{\mathcal{P}}_{\mathsf{b}}(C_{\uptheta},\infty,C_{\mathsf{q}}): \eta \mid X \sim{\mathcal{N}}(0,\sigma^2) \text{ and } |g_0(X)|\leq C_{\mathsf{g}} \text{ a.s.}\}$. If ${\epsilon}_1{\epsilon}_2=o(\log^{-1/2} n)$, then for any estimates $\hat{h}=(\hat{g},\hat{q})$ satisfying $|\hat{g}|\leq C_{\mathsf{g}}, |\hat{q}|\leq C_{\mathsf{q}}$ and any $1 \leq s \leq +\infty$, we have
\begin{align}
\mathfrak{M}_{n,1-\gamma}\left({\mathcal{P}}_{s,{\epsilon}}(\hat{h},\Phi^{\star})\right) \geq c_{\gamma}\big(\sigma^{-2}{\epsilon}_1{\epsilon}_2\,\big({\textstyle\frac{\log(1/{\epsilon}_1)}{\log n}}\big)^3+\sigma^{-1}n^{-1/2}\big)
\end{align}
for any $\gamma\in(1/2,1)$, where $c_{\gamma}>0$ is a constant that only depends on $\gamma$.
The assumption that $|\hat{g}|\leq C_{\mathsf{g}}, |\hat{q}|\leq C_{\mathsf{q}}$ is natural, since for any $P_0\in{\mathcal{P}}_0$, the ground-truth nuisance functions $g_0(X)=\mathbb{E}[T\mid X]$ and $q_0(X)=\mathbb{E}[Y\mid X]$ must satisfy $|g_0|\leq C_{\mathsf{g}}$ and $|q_0|\leq C_{\mathsf{q}}$ a.s. according to our assumption. Moreover, the lower bound does not depend on the value of $s$, meaning that no improvement is possible even if the nuisance estimates have small $L^{\infty}$-error.
Under the same assumptions, one can show that DML attains the minimax error rate up to the factor $\big({\textstyle\frac{\log(1/{\epsilon}_1)}{\log n}}\big)^3$ and matches the minimax error rate whenever ${\epsilon}_1 = {\mathcal{O}}(n^{-c})$ for some positive constant $c$.
For completeness, we include the details in (ref).
Notably, compared with the lower bound in (ref), the term $\Theta(\epsilon_1^2)$ disappears here since the variance $\sigma^2$ is assumed to be known. (ref) establishes the existence of a method-agnostic Gaussian treatment barrier as suggested (but not proved) in mackey2018orthogonal.
The proof of (ref) in (ref) is based on a constrained risk inequality for testing composite hypothesis developed in cai2011testing combined with novel constructions of the fuzzy hypotheses using moment matching techniques. For Gaussian treatment, we show that such hypotheses can be constructed in a way such that their induced target parameters $\theta_0$ are well-separated by leveraging a recursive property of Hermite polynomials.
Structure-agnostic upper bounds
(ref) established the rate optimality of DML for binary and Gaussian treatments but left open the possibility of improvement for other treatment distributions.
To exploit this opportunity, we will first introduce a new procedure that yields fast estimation rates whenever $\eta \mid X$ is non-Gaussian and the cumulants of $\eta \mid X$ are estimated accurately and next, in (ref), show that cumulant estimation is easy when $\eta$ is independent of $X$.
Our new procedure is based on the method of moments. Specifically, we will identify a moment function $m$ satisfying $\mathbb{E}_P[m(Z,\theta,h(X)) \mid X]=0, a.s.$ for all $P$ in some specified distribution set ${\mathcal{P}}$, where $\theta=\Phi(P)$ is the ground-truth parameter of interest and $h(\cdot)$ is some vector-valued nuisance functions that we need to estimate from data. Moreover, we require that $\theta=\Phi(P)$ is the unique solution to the moment equation. We proceed by plug in estimates of the nuisance $\hat{h}$ derived from a sample ${\mathcal{D}}_0$, and select $\hat{\theta}$ satisfying the empirical moment equation $\sum_{i\in{\mathcal{D}}} m(Z,\theta,\hat{h}(X)) = 0$ on a separate sample ${\mathcal{D}}$. This procedure is widely adopted in the development of DML-type methods chernozhukov2018double, and leads to efficient estimates as long as the moment function $m$ is Neyman-orthogonal, meaning that it is insensitive, under expectation, to nuisance estimation errors. The precise definition will be presented in (ref). The novelty of our construction lies in a specific recursive procedure that generates moment functions with arbitrarily high levels of insensitivity to nuisance estimation errors.
Consider the model (ref) and let $J_1(w,x)$ be any function of $w\in\mathbb{R}$ and $x\in{\mathcal{X}}$ satisfying
equation[equation omitted — 190 chars of source]
where each $\mathbb{E}[a_{i1}(X)^2] < \infty$ and each $\rho_{i1}$ is continuous.
Without loss of generality, we assume that $\rho_{11}(w)\equiv 1$ (as one can otherwise introduce a dummy summand into the expression (ref) with $a_{11}(x)\equiv 0$). Let $\left\{J_r(w,x)\right\}_{r=2}^{\infty}$ be a series of functions defined by
equation[equation omitted — 223 chars of source]
The following lemma, proved in (ref), derives the general form of $J_r$ for each $k\in\mathbb{Z}_+$.
lemma[Explicit formula for $J_r$]
If $M_r=M_1+r-1$, then
\begin{align}
\notag
J_r(w,x) = \sum_{i=1}^{M_r} a_{ir}(x)\rho_{ir}(w)\ \ for\ \ &a_{ir}(x) = \left\{
\begin{aligned}
a_{i-1,r-1}(x) &\quad if 1 < i \leq M_r \\
-\mathbb{E}[I_{r}(\eta,X)\mid X=x] &\quad if i=1,
\end{aligned} \right. \\
\rho_{1r}(w) = 1, \ \ and\ \ &\rho_{ir}(w) = \textstyle\int_0^{w} \rho_{i-1,r-1}(w')d w', \quad 2\leq i\leq M_r.
\end{align}
In particular, for all $r\geq 2$ we have $\rho_{2r}(w) = w$.
We would ideally use the moment function
equation[equation omitted — 148 chars of source]
to estimate $\theta_0$. However, by (ref), $J_r$ depends on the unknown data generating distribution via the functions $a_{ir}(\cdot)$.
Fortunately, our next theorem, proved in (ref), shows that an estimated moment function (ref) based on an estimate of $J_r$ yields improved treatment effect estimation rates
whenever $\theta_0$ is identifiable and $J_r$ is estimated sufficiently well.
theorem[Structure-agnostic error from estimated moments]
Consider the datasets $({\mathcal{D}}_1, {\mathcal{D}}_2) = (\{Z_i\}_{i=1}^{n/2},\{Z_i\}_{i=n/2+1}^{n})$ with each $Z_i \subseteq{\mathcal{Z}}$.
Define the estimated moment function
\begin{equation}
\begin{aligned}
\hat{m}_{r}\left(Z,\theta,h(X);{\mathcal{D}}_1\right) = \big[ Y-q(X)-\theta(T-g(X)) \big] \hat{J}_r(T-g(X),X;{\mathcal{D}}_1),
\end{aligned}
\end{equation}
where $h=\big(g,q\big)$ and $\hat{J}_r\big(w,x;{\mathcal{D}}_1\big)\triangleq\sum_{i=1}^{M_r}\hat{a}_{ir}(x;{\mathcal{D}}_1)\rho_{ir}(w)$ for $\hat{a}_{ir}:{\mathcal{X}}\times{\mathcal{Z}}^{n/2}\mapsto\mathbb{R}$ and $\rho_{ir}(\cdot)$ recursively defined via (ref).
Fix any $C_{\uptheta},C_{\mathsf{g}},C_{\mathsf{q}};\psi_{\upxi}, \psi_{\upeta} > 0$ , $\gamma\in(0,1)$, $s \geq r+1$, and $\Delta\in\mathbb{R}_+^2$, and let ${\mathcal{P}}\subseteq{\mathcal{P}}^{\star}(C_{\uptheta},C_{\mathsf{g}},C_{\mathsf{q}};\psi_{\upxi}, \psi_{\upeta})$ contain all distributions $P$ satisfying, with probability $1-\gamma/2$ over ${\mathcal{D}}_1\overset{\text{i.i.d.}}{\sim} P$, the following four conditions simultaneously:
\begin{align}
\textstyle
\bigl|\mathbb{E}_{Z\sim P}\bigl[\nabla_{\theta}\hat{m}_r(Z,\theta_0,h_0(X);{\mathcal{D}}_1 \mid {\mathcal{D}}_1)\bigr]\bigr|
&\geq \delta_{\mathsf{id}}
(identifiability)
\\
\sup_{h\in{\mathcal{B}}_{P,s}(h_0,\Delta)}\mathrm{Var}_{Z\sim P}\bigl(\hat{m}_r(Z,\theta_0,h(X);{\mathcal{D}}_1)\mid {\mathcal{D}}_1\bigr) &\leq V_{\mathsf{m}}
(finite variance)
\\
\max_{\beta\in\{r,r+1\}}\sup_{h\in{\mathcal{B}}_{P,s}(h_0,\Delta)} \big\|\hat{J}_r^{(\beta)}(T-g(X),X;{\mathcal{D}}_1)\mid {\mathcal{D}}_1\big\|_{L^{s}(P)} &\leq \Lambda_r (finite derivatives) \\
\sup_{x\in{\mathcal{X}}}\sup_{0\leq j\leq r}\big|\mathbb{E}\big[ \hat{J}_r^{(j)}(\eta,X;{\mathcal{D}}_1) \mid X=x, {\mathcal{D}}_1 \big]\big| &\leq {\epsilon}^{(j)} (near orthogonality)
\end{align}
where $\hat{J}_r^{(j)}(w,x)\triangleq\frac{\textup{d}^j}{\textup{d} w^j}\hat{J}_r(w,x)$.
Let $\hat{h}=(\hat{g},\hat{q})$ be a possibly random function independent of ${\mathcal{D}}$ and $\hat{\theta}$ be the solution of $\frac{1}{n}\sum_{i=n/2+1}^n \hat{m}_r\big(Z_i,\theta,\hat{h};{\mathcal{D}}_1\big) = 0$.
Then there exists a constant $C_{\gamma}>0$ that only depends on $\gamma$, such that for all ${\epsilon} \leq \Delta$,
\begin{equation}
\begin{aligned}
\mathfrak{R}_{n,1-\gamma} (\hat{\theta};{\mathcal{P}}_{s,{\epsilon}}(\hat{h},\Phi^{\star}))
&\leq C_{\gamma} \delta_{\mathsf{id}}^{-1}
\big[\sqrt{{\textstyle\frac{V_{\mathsf{m}}}{n}}} + \textstyle\sum_{j=0}^{r-1} {\textstyle\frac{1}{j!}}\max\{{\epsilon}^{(j+1)},{\epsilon}^{(j)}\}\big({\epsilon}_1^{j+1}+{\epsilon}_1^{j}{\epsilon}_2\big) \\
&\quad + {\textstyle\frac{1}{(r+1)!}} \Bigl(\big(4(\psi_{\upxi}+C_{\uptheta}\psi_{\upeta})\sqrt{s} + C_{\uptheta}\big){\epsilon}_1^{r+1}
+r{\epsilon}_1^r{\epsilon}_2\Bigr)\Lambda_r\big],
\end{aligned}
\end{equation}
In particular, if ${\epsilon}^{(j)} = {\mathcal{O}}(\max\{{\epsilon}_1,{\epsilon}_2\}^{r-j})$, then the treatment effect error rate (ref) has $r$-th order dependencies on the nuisance errors ${\epsilon}_i,i=1,2$ in place of the slower second-order dependencies of DML (see, e.g., (ref)). One caveat of applying this bound is that $\delta_{\mathrm{id}}, V_{\mathsf{m}}$, and $\lambda^{\star}$ all depend on the order of orthogonality $r$ and need to be computed in a case-by-case manner. In (ref), we will construct an explicit moment estimator that satisfies ${\epsilon}^{(j)} = {\mathcal{O}}(\max\{{\epsilon}_1,{\epsilon}_2\}^{r-j})$ and make the dependency on $r$ explicit.
The identifiability assumption (ref) is crucial and is why the construction of (ref) does not work for Gaussian treatments. Indeed, as we show in (ref), if the treatment is Gaussian and (ref) holds with ${\epsilon}^{(j)}\to 0$, then $\delta_{\mathsf{id}}\to 0$, so that (ref) cannot yield $\sqrt{n}$-consistency.
A natural choice for $J_1(w,x)$ satisfying (ref) is $J_1(w,x)\equiv w$, which corresponds to selecting $a_{11}\equiv 0, \rho_{11}\equiv 1, a_{21}\equiv 1,$ and $\rho_{21}(w) = w$.
In this case, (ref) implies that each $J_r(w,x)$ is the following $r$-th order polynomial of $w$:
equation[equation omitted — 223 chars of source]
Here, $\Pi_m$ denotes the set of all partitions of $[m]$ \footnote{For example, $\Pi_3 = \big\{\{\{1,2,3\}\},\ \{\{1,2\},\{3\}\},\ \{\{1,3\},\{2\}\},\ \{\{2,3\},\{1\}\},\ \{\{1\},\{2\},\{3\}\}\big\}$.} and $\kappa_i(x)$ is the $i$-th cumulant of $\eta\mid X=x$. In particular, if $\mu_i(x)$ denotes the $i$-th moment of $\eta\mid X=x$, then $J_2(w,x) = \frac{1}{2}(w^2 - \mu_2(x))$ and $J_3(w,x) = \frac{1}{6}(w^3 - 3\mu_2(x)w - \mu_3(x))$.
In (ref), we will show how to estimate these cumulant-based $J_r$ effectively whenever $\eta$ is non-Gaussian and independent of $X$.
Agnostic cumulant-based estimation (ACE)
In this section, we apply the general guarantee in (ref) to derive structure-agnostic estimators with better rates than DML when $\eta$ is non-Gaussian and independent of $X$.
Structure-agnostic cumulant estimation
The moment function induced by the $J_k(w,x)$ defined in (ref) requires estimating the cumulants $\kappa_i$ of the noise variable $\eta$ in the treatment regression model $T=g_0(X)+\eta$. In this subsection, we propose efficient structure-agnostic cumulant estimators assuming that $\eta$ is independent of $X$. We will also see in (ref) that our approach has potential benefits even when this assumption fails.
Our main result, stated below, indicates that an $r$-th order error rate can be attained for estimating the $r$-th cumulant of $\eta$.
theorem[Efficient cumulant estimator for noise with finite moments]
Let $C_{\mathsf{T}}>0$ be a constant, $r \geq 2$ be a positive integer, and ${\mathcal{P}}$ be the set of all distributions of $(X,T)$ generated from $T=g_0(X)+\eta$, such that $\mathbb{E}[|T|^r\mid X]^{1/r} \leq C_{\mathsf{T}}$ holds a.s.. The target parameter $\theta_0(P):{\mathcal{P}}\mapsto\mathbb{R}$ is the $r$-th order cumulant of $\eta=T-g_0(X)$ under $P$. Let $\Phi$ map $P\in{\mathcal{P}}$ to the nuisance function $g_0$. Let $\hat{g}:{\mathcal{X}}\mapsto\mathbb{R}$ be a nuisance estimate that satisfies $|\hat{g}(X)|\leq C_{\mathsf{g}}$ and $\hat{\kappa}_r: \{(X_i,T_i)\}_{i=1}^n\mapsto\mathbb{R}$ be the $r$-th order cumulant of the empirical residual distribution $P_n=\frac{1}{n}\sum_{i=1}^n\delta_{T_i-\hat{g}(X_i)}$ where $\delta_z$ is the Dirac measure at $z$. Then for any $\gamma \in (0,1)$ and $s \geq r$, $\hat\theta=\hat{\kappa}_r$ satisfies $\mathfrak{R}_{n,1-\gamma}\left(\hat\theta; {\mathcal{P}}_{s,{\epsilon}}(\hat{g},\Phi)\right) \leq C_{\gamma,r} n^{-1/2} + (2r{\epsilon})^r$ where $C_{\gamma,r}=10 r^{1/2}\gamma^{-1/2}(2C_{\mathsf{T}})^r(r-1)!$.
Our next theorem derives a refined bound for sub-Gaussian treatment noise.
theorem[Efficient cumulant estimator for sub-Gaussian noise]
Let $C_{\mathsf{g}}, \psi_{\upeta}>0$ be a constant and ${\mathcal{P}}$ be the set of all distributions of $(X,T)$ generated from $T=g_0(X)+\eta$, such that $|g_0(X)|\leq C_{\mathsf{g}}, a.s.$ and $\eta$ is mean-zero, independent of $X$ and $\psi_{\upeta}$-sub-Gaussian. The target parameter $\theta_0(P):{\mathcal{P}}\mapsto\mathbb{R}$ is the $r$-th order cumulant of $\eta=T-g_0(X)$ under $P$. Let $\Phi$ map $P\in{\mathcal{P}}$ to the nuisance function $g$. Let $\hat{g}:{\mathcal{X}}\mapsto\mathbb{R}$ be a nuisance estimate that satisfies $|\hat{g}(X)|\leq C_{\mathsf{g}}$ and $\hat{\kappa}_r: \{(X_i,T_i)\}_{i=1}^n\mapsto\mathbb{R}$ be the $r$-th order cumulant of the empirical residual distribution $P_n=\frac{1}{n}\sum_{i=1}^n\delta_{T_i-\hat{g}(X_i)}$ where $\delta_z$ is the Dirac measure at $z$. Then for any $\gamma \in (0,1)$ and $s \geq r$, $\hat\theta=\hat{\kappa}_r$ satisfies $\mathfrak{R}_{n,1-\gamma}\left(\hat\theta; {\mathcal{P}}_{s,{\epsilon}}(\hat{g},\Phi)\right) \leq C_{\gamma,r} n^{-1/2} + (2r{\epsilon})^r$ where $C_{\gamma,r}=3\big[12r(C_{\mathsf g}+\psi_{\upeta})\big]^r l^{1/2}\gamma^{-1/2}$.
remark[Comparing the bounds in (ref) and (ref)]
Under the assumptions in (ref), $C_{\mathsf{T}}$ in (ref) would be ${\mathcal{O}}(C_{\mathsf{g}}+\sqrt{r}\psi_{\upeta})$, so that (ref) implies a bound which scales as $(cr)^{3r/2}n^{-1/2}+(2r{\epsilon})^r$, while the bound in (ref) scales as $(c'r)^{r}n^{-1/2}+(2r{\epsilon})^r$, which is strictly tighter.
remark[Relaxing the independent noise assumption]
While (ref) assumes that the noise variable $\eta$ is independent of $X$, the estimator $\hat{\kappa}_r$
can also be of value when this assumption does not hold. In (ref), we consider $r=3$ and derive guarantees for this estimator when $\eta$ is “nearly” independent of $X$.
The fast rates in (ref) hold specifically for estimating the cumulants. When the target estimand is instead a moment of $\eta$, our structure-agnostic lower bound in (ref) requires $\Omega(n^{-1/2}+{\epsilon}^3)$ minimax error for $r=3$ and $\Omega(n^{-1/2}+{\epsilon}^2)$ minimax error for $r\neq 3$.
Notably, this cubic-quadratic bottleneck implies that, unlike the cumulant-based approach espoused here, the moment-based approach of mackey2018orthogonal cannot attain arbitrarily high‐order error rates.
The proofs of (ref) can be found in (ref). To the best of our knowledge, these results are novel and may be of independent interest. Next, we will apply these result to construct more efficient structure-agnostic estimators of treatment effects.
Fast rates with independent treatment noise
In this subsection, we introduce agnostic cumulant-based estimation (ACE), a novel treatment effect estimator that leverages the efficient cumulant estimators of (ref).
Throughout, we let $\hat{\kappa}_i$ be the empirical cumulant estimate defined in (ref)
and define
equation[equation omitted — 272 chars of source]
$\hat{J}_r(\cdot)$ can be viewed as an estimate of the cumulant-based function $J_r(\cdot)$ (ref) when $X$ is independent of $\eta$. The key observation is that this $\hat{J}_r(\cdot)$ satisfies (ref) with ${\epsilon}^{(j)}={\mathcal{O}}({\epsilon}_1^j)$, which follows from the key lemma stated below.
lemma[Key lemma; higher-order insensitivity condition]
For all $k\in[r]$, $\mathbb{E}\left[\hat{J}_r^{(k)}(T-g_0(X))\right] = \frac{1}{(r-k)!} \sum_{\pi\in\Pi_{r-k}} \prod_{B\in\pi} \left(\kappa_{|B|}-\hat{\kappa}_{|B|}\right).$
Notably, each term in the RHS in (ref) is a product of cumulant estimation errors. Recall in (ref) we show that the estimation error of $\hat{\kappa}_r$ is ${\mathcal{O}}({\epsilon}_1^r)$ when $\left\| \hat{g} - g \right\|_{P,s} \leq {\epsilon}_1$, so that $\prod_{B\in\pi} \left(\kappa_{|B|}-\hat{\kappa}_{|B|}\right) = {\mathcal{O}}\left(\prod_{B\in\pi}{\epsilon}^{|B|}\right) = {\mathcal{O}}({\epsilon}_1^{r-k})$. We additionally bound the coefficient hidden in the ${\mathcal{O}}(\cdot)$ in (ref).
In view of this favorable property, we propose our estimation algorithm, ACE, in (ref).
algorithm[algorithm omitted — 913 chars of source]
The next two theorems, proved in (ref) and (ref) respectively, show that ACE can achieve higher-order error rates for treatment effect estimation when $\eta$ is non-Gaussian.
theorem[ACE estimation error]
Let $r\in\mathbb{Z}_+$ and $\delta_{\mathsf{id}}, C_{\uptheta}, C_{\mathsf{T}}, C_{\mathsf{Y}} > 0$ be constants and ${\mathcal{P}}$ be the set of all distributions in ${\mathcal{P}}_{2r+2}(C_{\uptheta}, C_{\mathsf{T}}, C_{\mathsf{Y}})$ with $\eta$ independent of $X$ and $|\kappa_{r+1}|\geq \delta_{\mathsf{id},r}$.
Then, for any $\gamma\in(1/2,1)$, there exists $C_{\gamma}>0$ such that for all ${\epsilon}_1,{\epsilon}_2>0$, if
\begin{equation}
\textstyle
r \leq \min\left\{ \frac{b_1}{a_1}, \frac{b_2}{a_2\log(a_2b_2)}\right\}
\end{equation}
where $b_1=\log(\gamma n/100)$, $b_2=50\min\{1,C_{\uptheta}\}\delta_{\mathsf{id},r}\max\{{\epsilon}_1,{\epsilon}_2,(\gamma n)^{-1/2}C_{\mathsf{Y}}\}^{-1}$, $a_1 = 2\log(C_{\mathsf{T}}{\epsilon}_1^{-1}/2)$, and $a_2 = C_{\mathsf{T}}$, then then the $r$-th order ACE estimator $\hat\theta$ satisfies
\begin{equation}
\begin{aligned}
& \mathfrak{R}_{n,1-\gamma} (\hat{\theta};{\mathcal{P}}_{r,{\epsilon}}(\hat{h},\Phi^{\star})) \leq C_{\gamma} r!4^r \delta_{\mathsf{id},r}^{-1}\Big[ {\epsilon}_1^{r}{\epsilon}_2 + C_{\uptheta}{\epsilon}_1^{r+1} + 64C_{\mathsf{T}}^r\big(r^2C_{\mathsf{T}}+C_{\mathsf{Y}}\big) (\gamma n)^{-1/2} \Big].
\end{aligned}
\end{equation}
remark[Power of non-Gaussianity]
When $\eta$ is Gaussian, its cumulant $\kappa_{r+1}=0$ for all $r$, violating the assumption that $|\kappa_{r+1}|\geq\delta_{\mathsf{id},r}$ in (ref). Conversely, for non-Gaussian $\eta$, this condition is always satisfied for some $r$ by Levy’s Inversion
Formula durrett2019probability, allowing us to obtain higher-order error rates.
Notably, the constant $C_{\mathsf{T}}$ in (ref) may itself grow with $r$. For example, if $\eta = T - g_0(X)$ is sub-Gaussian, we can have $C_{\mathsf{T}} = \Theta(\sqrt{r})$. The theorem below makes this dependence explicit and delivers an even sharper bound in the sub-Gaussian regime.
theorem[ACE estimation error: sub-Gaussian noise]
Let $\delta_{\mathsf{id}}, C_{\uptheta}, C_{\mathsf{g}}, C_{\mathsf{q}}, \psi_{\upeta}, \psi_{\upxi} > 0$ and $r\in\mathbb{Z}_+$ be constants and ${\mathcal{P}}$ be the set of all distributions in ${\mathcal{P}}^*(C_{\uptheta}, C_{\mathsf{g}}, C_{\mathsf{q}}; \psi_{\upxi}, \psi_{\upeta})$ with $\eta$ independent of $X$ and $|\kappa_{r+1}|\geq \delta_{\mathsf{id},r}$.
Then, for any $\gamma\in(1/2,1)$, there exists $C_{\gamma}>0$ such that $\forall{\epsilon}_1,{\epsilon}_2>0$, if
\begin{equation}
\textstyle
r \leq \min\left\{ \frac{b_1}{a_1}-\frac{1}{a_1}\log \frac{b_1}{a_1}, \frac{b_2}{a_2\log(a_2b_2)}\right\}
\end{equation}
where $b_1=\log(\gamma n/9)$, $b_2=200 \min\{1,C_{\uptheta}\}\delta_{\mathsf{id},r}\max\left\{{\epsilon}_1,{\epsilon}_2,(\gamma n)^{-1/2}(\psi_{\upxi}+C_{\uptheta}\psi_{\upeta})\right\}^{-1}$,
$a_1=2\log(6(C_{\mathsf{g}}+\psi_{\upeta}){\epsilon}_1^{-1})$, and $a_2 = 4(C_{\mathsf{g}}+\psi_{\upeta})$ then the $r$-th order ACE estimator $
\hat\theta$ satisfies
\begin{equation}
\begin{aligned}
&\quad \mathfrak{R}_{n,1-\gamma} (\hat{\theta};{\mathcal{P}}_{r,{\epsilon}}(\hat{h},\Phi^{\star})) \\
&\leq C_{\gamma} r!16^r \delta_{\mathsf{id},r}^{-1}\Big[ {\epsilon}_1^{r}{\epsilon}_2 + C_{\uptheta}{\epsilon}_1^{r+1} + 64(C_{\mathsf{g}}+\psi_{\upeta})^r\big(r^2(C_{\mathsf{g}}+\psi_{\upeta})+\psi_{\upxi}+C_{\uptheta}\psi_{\upeta}\big) (\gamma n)^{-1/2} \Big].
\end{aligned}
\end{equation}
remark[Scale of the leading coefficient under uniform noise]
As shown in (ref), the estimation error of $r$-th order ACE estimator depends not only on $r, {\epsilon}_1,{\epsilon}_2,n$, but also on $\kappa_{r+1}$. This is intuitive as $\kappa_{r+1}$ is a measure of non-Gaussianity. An estimate of $\kappa_{r+1}$ can also be used to estimate the variance of $\hat{\theta}$; see (ref) for more details. To understand the role of $\delta_{\mathsf{id},r}$ in the bound, consider the case when $\eta$ follows a uniform distribution on $[-1,1]$. Then
for any $m\in\mathbb{Z}_+$, we have $\kappa_{2m}\sim 4\sqrt{\frac{\pi}{m}}\left(\frac{m}{\pi e}\right)^{2m}$ binet1839memoire. Plugging into (ref), we have
\begin{equation}
\mathfrak{R}_{n,1-\gamma} (\hat{\theta};{\mathcal{P}}_{s,{\epsilon}}(\hat{h})) \leq 4r (4\pi e)^r \delta_{\mathsf{id},r}^{-1}\Big[ {\epsilon}_1^{r}{\epsilon}_2 + C_{\uptheta}{\epsilon}_1^{r+1} + 8 (C_{\mathsf{Y}}+(C_{\uptheta}+1)C_{\mathsf{T}}) (4C_{\mathsf{T}})^{r} (\gamma n)^{-1/2} \Big].
\end{equation}
Hence the leading coefficient is only exponential in $r$, rather than super-exponential.
When $r= 1$, ACE is identical to DML. When $r=2,3$ it recovers the “second-order” orthogonal estimators proposed by mackey2018orthogonal. Interestingly, for $r=3$, the rate given by (ref) is faster than that of mackey2018orthogonal, as the latter did not establish third-order orthogonality. When $r\geq 4$, to the best of our knowledge, ACE is novel, and we derive the explicit expressions for $r=3,4$ in (ref).
As a concrete instantiation of (ref), consider the setting of high-dimensional linear nuisance,
taligng_0(x)=\langle{\alpha_0}{,x}\rangle
\ \ and\ \
q_0(x)=\langle{\beta_0}{,x}\rangle
\ \ for\ \
\alpha_0,\beta_0 \in \mathbb{R}^p,\ \
s_1\triangleq\left\| \alpha_0 \right\|_0,
\ \ and\ \
s_2\triangleq\left\| \beta_0 \right\|_0,
where $(p,s_1,s_2)$ all potentially grow with $n$, and the nuisance functions are estimated using Lasso regression hastie2015statistical.
In this setting, DML is known to provide order $n^{-1/2}$ estimation error for $\theta_0$ whenever the maximum sparsity level $\max(s_1, s_2) = o(n^{1/2}/\log p)$ chernozhukov2018double.
Remarkably, as we prove in (ref), $r$-th order ACE provides the same guarantee when $\max(s_1, s_2) = o(n^{r/(r+1)}/\log p)$.
Numerical experiments
To evaluate the empirical effectiveness of our proposed estimators, we simulate a demand estimation scenario using purchase and pricing data. In this setting, $Y$ represents observed demand, the treatment $T$ corresponds to an observed product price, $g_0(X)$ denotes a baseline product price determined by covariates $X$ that influence pricing policy, and the treatment noise $\eta$ represents a random discount offered to customers for demand assessment. Notably, $\eta$ is typically discrete (and thus distinctly non-Gaussian) and independent of $X$.
figure[figure omitted — 2,220 chars of source]
We replicate the experimental framework of mackey2018orthogonal, where $X\sim{\mathcal{N}}(0,I)$, ${\epsilon}\sim U([-3,3])$, and $\eta$ follows a discrete distribution on $\{0.5, 0, -1.5, -3.5\}$ with probabilities $\{0.65, 0.2, 0.1, 0.05\}$, respectively. Each nuisance function is specified as a sparse linear function in $p=100$ dimensions with $s=40$ non-zero coefficients.
We examine the $r$-th order ACE estimator introduced in (ref) across different values of $r$. For $r=1,2$, this framework precisely recovers the first-order chernozhukov2018double and second-order mackey2018orthogonal orthogonal estimators. First-stage nuisance function estimates are obtained using Lasso regression tibshirani1996regression, following (ref). Complete Python code for replicating all experiments is available at \url{https://github.com/JikaiJin/ACE}.
In view of the high-probability bounds in (ref), we empirically assess ACE performance for orders $r\leq 5$ across varying sample sizes. A comparison of the total RMSE is provided in (ref), demonstrating that the fifth-order ACE estimator achieves optimal performance. We further decompose RMSE into bias and variance components. (ref) compares bias across different orders, with fifth-order ACE exhibiting the smallest bias. Moreover, (ref) shows that the first-order ACE estimator achieves the lowest standard deviation, followed by the fifth-order estimator. (ref) present the distribution of estimated values using first- and fifth-order ACE estimators. Both distributions are approximately Gaussian, with the first-order estimator exhibiting substantially larger bias. Based on (ref), the variance of $\hat\theta$ is bounded by $\delta_{\mathsf{id}}^{-1}(V_{\mathsf{m}}/n)^{1/2}$, where $\delta_{\mathsf{id}}$ provides a lower bound for $\kappa_{r+1}$ in the context of (ref). This enables us to construct a direct plug-in variance estimate ${\mathcal{E}}_{\mathsf{var}}$ as ${\mathcal{E}}_{\mathsf{var}} = \hat{\kappa}_r^{-1}\sqrt{\frac{V_{\mathsf{m}}}{n}} \ \text{ for }\ V_{\mathsf{m}} = \frac{1}{n}\sum_{i=1}^{n}\big[ (Y_i-\hat{q}(X_i))^2 +\hat{\theta}^2(T_i-\hat{g}(X_i))^2 \big] \hat{J}_r(T_i-\hat{g}(X_i))^2.$ Lastly, following (ref), we construct the approximate 95% confidence interval $[\hat{\vartheta}-1.96{\mathcal{E}}_{\mathsf{var}}^{1/2},\hat{\vartheta}+1.96{\mathcal{E}}_{\mathsf{var}}^{1/2}]$ for $\theta_0$. (ref) demonstrates that approximately 95% of independent experiments yield confidence intervals that contain the true parameter value, confirming the validity of our constructed intervals.
Conclusion and future directions
In this paper, we provide new insights into how distributional properties could change the statistical limit of structure-agnostic estimation. Focusing on a partial linear outcome model, we show that the Gaussianity of the treatment variable creates a fundamental barrier for improving over DML, while improvements upon DML is possible for non-Gaussian treatment.
Moving forward, it would be of interest to exploit distributional properties to design estimators more efficient than DML for heterogeneous treatment effects.
\opt{opt-neurips}{
NeurIPS Paper Checklist
enumerate• {\bf Claims}
• Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
• Answer: \answerYes.
• Justification: At the end of Section 1 we directly point to our main results.
• Guidelines: See the main contributions listed at the end of Section 1.
\begin{itemize}
• The answer NA means that the abstract and introduction do not include the claims made in the paper.
• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
\end{itemize}
• {\bf Limitations}
• Question: Does the paper discuss the limitations of the work performed by the authors?
• Answer: \answerYes
• Justification: The main limitation is the assumed independence of noise, and this is discussed in (ref).
• Guidelines:
\begin{itemize}
• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
• The authors are encouraged to create a separate "Limitations" section in their paper.
• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
\end{itemize}
• {\bf Theory assumptions and proofs}
• Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
• Answer: \answerYes
• Justification: We explicitly point to the proof after stating each theorem.
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not include theoretical results.
• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
• All assumptions should be clearly stated or referenced in the statement of any theorems.
• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
• Theorems and Lemmas that the proof relies upon should be properly referenced.
\end{itemize}
• {\bf Experimental result reproducibility}
• Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
• Answer: \answerYes
• Justification: Details are provided in (ref).
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not include experiments.
• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
\begin{enumerate}
• If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
• If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
• If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
• We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
\end{enumerate}
\end{itemize}
• {\bf Open access to data and code}
• Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
• Answer: \answerYes
• Justification: The open-source code is provided in the supplementary materials.
• Guidelines:
\begin{itemize}
• The answer NA means that paper does not include experiments requiring code.
• Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
• While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
\end{itemize}
• {\bf Experimental setting/details}
• Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
• Answer: \answerYes
• Justification: These are discussed in (ref) and (ref).
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not include experiments.
• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
• The full details can be provided either with the code, in appendix, or as supplemental material.
\end{itemize}
• {\bf Experiment statistical significance}
• Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
• Answer: \answerYes
• Justification: Error bars and histogram under multiple runs are provided in (ref).
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not include experiments.
• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
• The assumptions made should be given (e.g., Normally distributed errors).
• It should be clear whether the error bar is the standard deviation or the standard error of the mean.
• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
\end{itemize}
• {\bf Experiments compute resources}
• Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
• Answer: \answerYes
• Justification: The experiments are of small-scale and can be run on a laptop in a reasonable amount of time.
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not include experiments.
• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
\end{itemize}
• {\bf Code of ethics}
• Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
• Answer: \answerYes
• Justification: We have checked the code of ethics.
• Guidelines:
\begin{itemize}
• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
\end{itemize}
• {\bf Broader impacts}
• Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
• Answer: \answerNA
• Justification: This paper is mainly theoretical and does not have direct societal impact.
• Guidelines:
\begin{itemize}
• The answer NA means that there is no societal impact of the work performed.
• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
\end{itemize}
• {\bf Safeguards}
• Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
• Answer: \answerNA
• Justification: The paper poses no such risks.
• Guidelines:
\begin{itemize}
• The answer NA means that the paper poses no such risks.
• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
\end{itemize}
• {\bf Licenses for existing assets}
• Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
• Answer: \answerNA
• Justification: The paper does not use existing assets.
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not use existing assets.
• The authors should cite the original paper that produced the code package or dataset.
• The authors should state which version of the asset is used and, if possible, include a URL.
• The name of the license (e.g., CC-BY 4.0) should be included for each asset.
• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
• If this information is not available online, the authors are encouraged to reach out to the asset's creators.
\end{itemize}
• {\bf New assets}
• Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
• Answer: \answerNA
• Justification: The paper does not release new assets.
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not release new assets.
• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
• The paper should discuss whether and how consent was obtained from people whose asset is used.
• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
\end{itemize}
• {\bf Crowdsourcing and research with human subjects}
• Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
• Answer: \answerNA
• Justification: The paper does not involve crowdsourcing nor research with human subjects.
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
\end{itemize}
• {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
• Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
• Answer: \answerNA
• Justification: The paper does not involve crowdsourcing nor research with human subjects.
• Guidelines:
\begin{itemize}
• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
\end{itemize}
• {\bf Declaration of LLM usage}
• Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
• Answer: \answerNA
• Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
• Guidelines:
\begin{itemize}
• The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
• Please refer to our LLM policy (\url{https://neurips.cc/Conferences/2025/LLM}) for what should or should not be described.
\end{itemize}
}