The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
283,915 characters
Simultaneous Inference for Local Structural Parameters with Random Forests$^*$
\begin{titlepage}
\begin{spacing}{1}
\title{\textbf{\Large Simultaneous Inference for\vspace{0.5em}\\ Local Structural Parameters with Random Forests$^*$}}
\author{
\begin{tabular}[t]{c@{\extracolsep{4em}}c}
\large{David M. Ritzwoller} & \large{Vasilis Syrgkanis}\vspace{-0.7em}\\ \vspace{-1em}
\small{Stanford University} & \small{Stanford University} \\ \vspace{-0.7em}
\end{tabular}
\\
}
\date{
\today\\ $^*$Email: [email removed], [email removed]. We thank Jiafeng Chen, Victor Chernozhukov, John Duchi, Matthew Gentzkow, Han Hong, Guido Imbens, Lihua Lei, Joseph Romano, Brad Ross, and Jann Spiess for helpful comments and conversations. Ritzwoller gratefully acknowledges support from the National Science Foundation under the Graduate Research Fellowship. Computational support was provided by the Data, Analytics, and Research Computing (DARC) group at the Stanford Graduate School of Business (RRI:SCR\_022938).}
\begin{abstract}
\smalltonormalsize{We construct simultaneous confidence intervals for solutions to conditional moment equations. The intervals are built around a class of nonparametric regression algorithms based on subsampled kernels. This class encompasses various forms of subsampled random forest regression, including Generalized Random Forests \citep{athey2019generalized}. Although simultaneous validity is often desirable in practice---for example, for fine-grained characterization of treatment effect heterogeneity---only confidence intervals that confer pointwise guarantees were previously available. Our work closes this gap. As a by-product, we obtain several new order-explicit results on the concentration and normal approximation of high-dimensional $U$-statistics.}
\\
\\
\textbf{Keywords:} Random Forest Regression, Half-Sample Bootstrap, $U$-statistics
\\
\textbf{JEL:} C01, C14, C12
\end{abstract}
\end{spacing}
\end{titlepage}
\maketitle
\thispagestyle{empty}
\setcounter{page}{1}
\begin{spacing}{1.3}
\section{Introduction}
Consider an independent and identically distributed sample $\mathbf{D}_n = (D_i)_{i=1}^n$. The scalar parameter $\theta_0(x)$ is defined as the unique solution to the conditional moment equation
\begin{equation}\label{eq: estimating equation}
M(x; \theta, g_0) = \mathbb{E}\left[m(D_i;\theta,g_0) \mid X_i = x\right] = 0
\end{equation}
in $\theta$. Here, $X_i$ is a sub-vector of the observation $D_i$, $g_0$ is an unknown nuisance parameter, identified via an auxiliary statistical problem, and $m(\cdot;\theta,g)$ is a known moment function. We propose a method for constructing simultaneous confidence intervals for the parameter vector $\theta_0(\bm{x}^{(d)})=(\theta_0(x^{(j)}))_{j=1}^d$, where $\bm{x}^{(d)}=(x^{(j)})_{j=1}^d$ is a specified $d$-vector in the domain of $X_i$.\footnote{The leading examples for choices of the query-vector $\bm{x}^{(d)}$ are cases where $\bm{x}^{(d)}$ is taken to be the observed values of the covariates $X_1,\ldots,X_n$ or where $\bm{x}^{(d)}$ gives a fine grid over the domain of $X_i$.} Many problems encountered in applied economics and causal inference can be formulated as instances of this setting, including nonparametric regression, quantile regression, and estimation of conditional average treatment effects.
To fix concepts, consider \cite{banerjee2015multifaceted}, who study the effects of a poverty alleviation program implemented in Ghana.\footnote{\cite{banerjee2015multifaceted} study data collected from several similar graduation programs. We focus on the data from their evaluation of the program implemented in Ghana. \cref{app: simulation} gives further details.} For each individual in their sample, they observe the data $D_i = (Y_i, W_i, Z_i)$, where $Y_i$ is a measurement of total assets taken two years after the implementation of the program, $W_i$ is an indicator denoting assignment to the program, and $Z_i$ is a vector of covariates. A broad aim of the study is to determine the conditions under which recipients of aid experience lasting improvements in welfare. One quantity that can inform this determination is the conditional average treatment effect (CATE)
\begin{equation}\label{eq: CATE}
\theta_0(x) = \mathbb{E}_P \left[ Y_i(1) - Y_i(0) \mid X_i = x\right]~,
\end{equation}
where $Y_i(1)$ and $Y_i(0)$ are the potential outcomes generated by the intervention $W_i$ and $X_i$ is some chosen subvector of $Z_i$. Many modern approaches to estimating CATEs are premised on the observation that $\theta_0(x)$ is the solution to the conditional moment equation
\begin{equation}\label{eq: CATE moment}
M(x; \theta, g_0) = \mathbb{E}\left[(\mu_1(Z_i) - \mu_0(Z_i)) + \beta(W_i, Z_i) (Y_i - \mu_{W_i}(Z_i)) - \theta \mid X_i = x\right]=0~,
\end{equation}
where the nuisance parameter $g_0$ collects the conditional outcome regression and Horvitz-Thompson weight
\begin{equation}\label{eq: CATE nuisance}
\mu_{w}(z) = \mathbb{E}_P\left[Y_i \mid W_i = w, Z_i = z\right]\quad\text{and}\quad \beta(w, z) = \frac{w}{\pi(z)} - \frac{1- w}{1-\pi(z)}~,
\end{equation}
for the propensity score $\pi(z) = P\{W_i = 1 \mid Z_i = z\}$. See \cite{semenova2021debiased}, \cite{foster2023orthogonal}, and \cite{kennedy2023towards}, for further discussion.
Often, estimates of solutions to conditional moment equations of the form \eqref{eq: estimating equation} or \eqref{eq: CATE moment} are obtained by solving the empirical conditional moment equation
\begin{equation}\label{eq: subsampled kernel}
M_{n}(x; \theta,\hat{g}_n, \mathbf{D}_n) = \sum_{i=1}^n K(x, X_i) m(D_i; \theta, \hat{g}_n) = 0
\end{equation}
in $\theta$, where $\hat{g}_n$ is some first-stage estimator of the nuisance parameter $g_0$ and $K(x,x^\prime)$ is some, potentially random and data-dependent, kernel function measuring the distance between $x$ and $x^\prime$. Popular examples of data-dependent kernel estimators include subsampled nearest-neighbor regression \citep{fix1989discriminatory,khosravi2019non,demirkaya2024optimal} and random forest regression \citep{breiman2001random,wager2018estimation}. Solutions to conditional moment equations \eqref{eq: subsampled kernel} constructed with subsampled random forest regression are referred to as Orthogonal Random Forests (ORF) \citep{oprescu2019orthogonal}. ORFs generalize Generalized Random Forests (GRF) \citep{athey2019generalized} to incorporate nuisance parameters estimated via auxiliary statistical problems.
\cref{fig: cate} displays CATE estimates for the experiment studied in \cite{banerjee2015multifaceted}, where the chosen conditioning covariates $X_i$ are pretreatment measurements of monthly consumption and total assets. The nuisance parameter estimate $\hat{g}_n$ and the kernel $K(x,x^\prime)$ are constructed with the implementation of random forest regression made available through the ``GRF'' R package \citep{athey2019generalized}. The graduation program appears to be most effective for individuals with a high level of baseline consumption and a low level of baseline assets.\footnote{The quartiles of baseline log consumption are 3.33, 3.76, and 4.20. The quartiles of baseline assets are -0.45, -0.71, and 0.03. Panel A of \cref{fig: density} displays a scatter plot of baseline log consumption and assets.} These results are suggestive of a poverty trap: individuals with an opportunity to increase their assets are able to do so only if they have a high level of baseline consumption \citep{balboni2022people,kraay2014poverty}.
\begin{figure}[t]
\begin{centering}
\caption{CATE Estimates}
\label{fig: cate}
\medskip{}
\begin{tabular}{c}
\includegraphics[scale=0.4]{plots/intro_estimate}\tabularnewline
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig: cate} displays a heat map giving CATE estimates for the intervention studied in \cite{banerjee2015multifaceted} on post-treatment assets. The color of each rectangle indicates the estimate of the CATE queried at the rectangle's central point. The horizontal and vertical axes display the baseline monthly consumption, normalized to dollars and measured in logs base 10, and the baseline value of an index for total assets. CATE estimates are obtained by solving the empirical moment equation \eqref{eq: subsampled kernel} for each value $x$ on an evenly spaced grid on both axes. See \cref{app: simulation} for further details.}{\footnotesize\par}
\end{figure}
We contribute a method for assessing the statistical significance of estimates typified by \cref{fig: cate}. In particular, we propose a computationally simple procedure for constructing simultaneous upper and lower confidence bounds for solutions to conditional moment equations \eqref{eq: estimating equation} centered around estimators of the form \eqref{eq: subsampled kernel}. Formally, we construct a family of random intervals
\begin{equation} \label{eq: example intervals}
\hat{\mathcal{C}}(\bm{x}^{(d)})
=
\left\{\hat{\mathcal{C}}(x^{(j)}) = [\hat{c}_{L}(x^{(j)}), \hat{c}_{U}(x^{(j)})] \right\}_{j=1}^d~,
\end{equation}
on the basis of the observed data, such that
\begin{equation} \label{eq: asymptotic validity}
\sup_{P\in\mathbf{P}} \big\vert P\left\{ \theta_0(\bm{x}^{(d)}) \in \hat{\mathcal{C}}(\bm{x}^{(d)}) \right\} - (1-\alpha) \big\vert \leq r_{n,d}
\end{equation}
for some sequence $r_{n,d}$, where $\mathbf{P}$ is some statistical family that contains the distribution $P$ of the data $D_i$. We say that a region \eqref{eq: example intervals} satisfying \eqref{eq: asymptotic validity} is uniformly asymptotically valid at the rate $r_{n,d}$. Here, uniformity operates over both the $d$-dimensional query-vector $\bm{x}^{(d)}$ and the statistical family $\mathbf{P}$. The main theoretical contribution of this paper is the construction of confidence regions whose error rate $r_{n,d}$ converges to zero in asymptotic regimes where the number of points $d$ in the query-vector $\bm{x}^{(d)}$ may increase much more quickly than the sample size $n$.
Many leading applications of conditional moment regression in applied economics necessitate simultaneous coverage guarantees. For example, researchers are often interested in characterizing statistically significant dimensions of treatment effect heterogeneity or in other forms of post-hoc exploratory statistical analysis. By contrast, only confidence intervals with pointwise guarantees, i.e., coverage guarantees that are valid at a single pre-specified point $x$, are available. See, for example, methods developed by \cite{sexton2009standard,wager2014confidence,mentch2016quantifying,wager2018estimation,athey2019generalized,oprescu2019orthogonal,khosravi2019non}, and \cite{demirkaya2024optimal}. Moreover, prior approaches for simultaneous confidence intervals, that have the potential to be relevant to random forest methods \citep{song2019approximating}, are not applicable, as they require assumptions that do not hold in hyperparameter regimes that ensure consistency of these methods. Our work bridges this gap between theory and practice and enables the use of random forest approaches, when simultaneous coverage guarantees are required.
We begin, in \cref{sec: construction}, by defining the proposed confidence region and illustrating its application to the \cite{banerjee2015multifaceted} experiment. Our construction can be seen as an instance of subsampling \citep{politis2012subsampling,politis1994large}, although our formal analysis is more directly connected to the exchangeably weighted bootstrap \citep{praestgaard1993exchangeably,chernozhuokov2022improved}, and, in particular, the half-sample bootstrap \citep{chung2013exact,yadlowsky2023evaluating}.
In \cref{sec: subsampled kernel regression}, we bound the accuracy of the proposed confidence region, under two substantive restrictions. First, we require that the estimator \eqref{eq: subsampled kernel} be based on a kernel constructed with subsampling. Second, we require that the moment function \eqref{eq: estimating equation} satisfy a Neyman orthogonality condition, in the sense of \cite{chernozhukov2018double}. This restriction mitigates the error induced by estimation of nuisance parameters. Our analysis builds on \cite{oprescu2019orthogonal}, who study pointwise inference for subsampled random forest estimators of solutions to Neyman orthogonal conditional moment equations. See also \cite{khosravi2019non} for a pointwise analysis of generic subsampled kernel regression estimators. We conclude \cref{sec: subsampled kernel regression} by documenting that the proposed confidence region is accurate and informative at empirically relevant sample sizes with a simulation calibrated to the \cite{banerjee2015multifaceted} data.
As a by-product of this analysis, we give several new results on the concentration and normal approximation of large order, high-dimensional, $U$-statistics. In particular, we give a concentration inequality and central limit theorem for high-dimensional $U$-statistics with explicit order-dependence. These bounds are applicable to non-degenerate $U$-statistics whose order $b$ satisfies $b = o(n)$, up to a dimension dependent logarithmic factor. This generality represents a substantial improvement over existing results \citep{song2019approximating,minsker2023u}, that apply to the regime $b = o(n^{1/3})$, and is essential for our application.\footnote{In particular, the subsampled kernel regression estimators considered in this paper require $n^{1 - c/p} \lesssim b$ for consistency, where $p$ is the dimension of $Z_i$.} Our results hinge on a new concentration inequality for the difference between a $U$-statistic and its H\'{a}jek projection \citep{hajek1968asymptotic}, enabled by a symmetrization inequality due to \cite{sherman1994maximal}. We collect these results in \cref{sec: u-stat}. \cref{sec: conclusion} concludes.
\subsection{Related Literature} There is an extensive literature on estimation of solutions to conditional moment equations. See, for example, \cite{Newey1993efficient}, \cite{ai2003efficient}, \cite{chen2012estimation}, and \cite{chernozhukov2023constrained}. \cite{chen2018optimal} and \cite{christensen2024adaptive} propose related approaches for constructing uniform confidence bands for parameters identified by conditional moments, emphasizing achieving minimax rates in H\"{o}lder classes by building confidence regions around carefully constructed sieve estimators with Lepski's method \citep{chernozhukov2014anti}. By contrast, our aim is to provide a simple procedure for constructing simultaneous confidence intervals around estimators whose precise structure may be unknown to the user.
We contribute to a large literature on the role of Neyman orthogonality in estimation of solutions to moment equations with nuisance parameters. \cite{chernozhukov2018double}, \cite{chernozhukov2022locally}, and \cite{ichimura2022influence} provide extensive discussion and guidance on the derivation of orthogonal moments. \cite{semenova2021debiased}, \cite{foster2023orthogonal}, and \cite{kennedy2023towards} apply aspects of this analysis to conditional moment estimation. Our results are comparable to the generic bounds on the accuracy of Gaussian multiplier bootstrap confidence regions for nonparametric regression and $Z$-estimation given in \cite{chernozhukov2014anti} and \cite{belloni2018uniformly}, respectively. We generalize these results, in the sense that we treat inference for conditional $Z$-estimators whose score function is potentially unknown to the researcher.
Our formal analysis builds on a groundbreaking sequence of papers on central limit theorems for maxima of sums initiated by \cite{chernozhukov2013gaussian}. Extensions and refinements of these results are given in \cite{chernozhukov2017central} and \cite{chernozhuokov2022improved}. The asymptotic analysis of $U$-statistics has a long and involved history. We provide a more detailed literature review in \cref{sec: u-stat}.
Finally, we contribute to a large literature on the statistical analysis of random forest regression. A wide variety of consistency results are given in, e.g., \cite{buhlmann2002analyzing}, \cite{lin2006random}, \cite{biau2008consistency}, \cite{mentch2014ensemble}, \cite{scornet2015consistency}, and \cite{cattaneo2024pointwise}. High dimensional consistency results are given in \cite{syrgkanis2020estimation}, \cite{chi2022asymptotic}, and \cite{huo2023adaptation}.
\subsection{Notation}The quantities $c$ and $C$ denote universal positive constants, whose values are allowed to depend only on the family of distributions $\mathbf{P}$. For two real-valued functions $f$ and $g$ on a domain $\mathcal{X}$, we say $g(x) \lesssim f(x)$ if $g(x)\leq C f(x)$ for each $x$ in $\mathcal{X}$. For a functional $F$ on $\mathcal{F}$, we use the notation
\[
\partial_f F(f)[h] = \frac{\text{d}}{\text{d}t}F(f+th)\big\vert_{t=0}
\quad\text{and}\quad
\partial_{f,f} F(f)[h]=\frac{\text{d}^2}{\text{d}t^2}F(f+th)\big\vert_{t=0}
\]
to denote first and second order directional derivatives, respectively. For any function $f(x)$ and vector $\bm{x}^{(d)}=(x^{(j)})_{j=1}^d$, we let $f(\bm{x}^{(d)})$ denote the vector $(f(x^{(1)}), \cdots, f(x^{(d)}))$. The set $\mathcal{S}_{n,b}$ collects all of the subsets of $[n]=\{1,\ldots,n\}$ of size $b$ and the quantity $D_{\mathsf{s}}$ denotes the subset of the observed data $\mathbf{D}_n$ with indices in the set $\mathsf{s}$.
\section{Simultaneous Confidence Intervals\label{sec: construction}}
We build simultaneous confidence intervals around solutions to the empirical moment equation
\begin{equation}\label{eq: empirical moment}
M_{n}(x; \theta,\hat{g}_n, \mathbf{D}_n) = \sum_{i=1}^n K(x, X_i) m(D_i; \theta, \hat{g}_n) = 0
\end{equation}
in the variable $\theta$, evaluated at each $x^{(j)}$ in $\bm{x}^{(d)}$.\footnote{For the sake of simplicity, we consider only the case that the moment $m(\cdot ;\theta,g_0)$ and the parameter $\theta$ are scalar-valued. The vector-valued case is relevant, e.g., when estimating CATEs with multiple treatment variables. Our results generalize to the vector-valued case at the cost of additional notation.} Let $\hat{\theta}_n(\bm{x}^{(d)})$ denote the vector of solutions to \eqref{eq: empirical moment} evaluated at each $x^{(j)}$ in $\bm{x}^{(d)}$. Let $\mathsf{h}$ denote a random element of $\mathcal{S}_{n,n/2}$, i.e., a random half-sample of $[n]$ and let $\hat{\theta}_\mathsf{h}(\bm{x}^{(d)})$ denote the vector of the solutions to \eqref{eq: empirical moment}, with the data $\mathbf{D}_n$ replaced by the half-sample $D_\mathsf{h}$. Our proposal is premised on approximating the distribution of the root
\begin{equation}
R_n(\bm{x}^{(d)})
= \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)})\label{eq: root}
\end{equation}
with the conditional distribution of the half-sample bootstrap root
\begin{equation}
R^*_n(\bm{x}^{(d)}) = \hat{\theta}_\mathsf{h}(\bm{x}^{(d)}) - \hat{\theta}_n(\bm{x}^{(d)})~,\label{eq: bootstrap root}
\end{equation}
The nuisance parameter estimator $\hat{g}_n$ does not need to be re-estimated when computing \eqref{eq: bootstrap root}. A version of the half-sample bootstrap is implemented by default in the GRF R package \citep{athey2019generalized}.\footnote{The version of the half-sample bootstrap considered in \cite{athey2019generalized} is based on combining estimates of the variances of components of a linearization of the moment $M(x;\theta,g)$ with a Delta method type argument. By contrast, the bootstrap root \eqref{eq: bootstrap root} is agnostic to the structure of the conditional moment under consideration.} The half-sample bootstrap is an instance of subsampling \citep{politis1994large,politis2012subsampling}.
Let $\hat{\lambda}^2_{n,j}$ denote the variance of $\sqrt{n}R^*_{n}(x^{(j)})$, conditioned on the data $\mathbf{D}_n$, and let $\hat{\mathsf{cv}}_{n}(\alpha)$ denote the $1-\alpha$ quantile of the distribution of the studentized statistic
\begin{equation}\label{eq: critical value quantity}
\hat{S}^*_{n}(\bm{x}^{(d)}) = \sqrt{n} \left\| \hat{\Lambda}_n^{-1/2}R^*_n(\bm{x}^{(d)}) \right\|_\infty~,
\end{equation}
again conditioned on the data $\mathbf{D}_n$. Here, $\hat{\Lambda}_n$ denotes the diagonal matrix with elements $\hat{\lambda}^2_{n,j}$.\footnote{The quantities $\hat{\lambda}^2_{n,j}$ and $\mathsf{cv}_{n}(\alpha)$ are easily approximated by resampling the bootstrap root \eqref{eq: bootstrap root}. To simplify exposition, we omit explicit consideration of residual randomness induced by this approximation.} We consider confidence intervals with the following structure.
\begin{defn}\label{def: uniform ci} Define the intervals
\begin{equation}
\hat{\mathcal{C}}(x^{(j)}) = \hat{\theta}_n(x^{(j)}) \pm n^{-1/2}\hat{\lambda}_{n,j} \hat{\mathsf{cv}}_{n}(\alpha)\quad\text{for each}\quad j \text{ in }[d]~.
\end{equation}
The level-$\alpha$ half-sample confidence region for $\theta_0(\bm{x}^{(d)})$ is given by $\hat{\mathcal{C}}(\bm{x}^{(d)})$.
\end{defn}
\noindent Confidence intervals with the same structure, based on different choices of bootstrap root, are studied in, e.g., \cite{chernozhukov2014anti} and \cite{belloni2018uniformly}. The essential feature of the bootstrap root \eqref{eq: bootstrap root} is that it can be computed without knowing anything about the structure of the estimator $\hat{\theta}_n(\bm{x}^{(d)})$. In particular, approaches based on the Rademacher or Gaussian multiplier bootstrap rely on knowledge of a linear approximation to $\hat{\theta}_n(\bm{x}^{(d)})$.
To gain intuition, suppose that the estimator $\hat{\theta}_n(\bm{x}^{(d)})$ satisfies a linear representation
\begin{equation}\label{eq: exact linear}
\hat{\theta}_n(\bm{x}^{(d)}) = \frac{1}{n} \sum_{i=1}^n \bar{u}(\bm{x}^{(d)}, D_i)
\end{equation}
for some function $\bar{u}(\cdot,\cdot)$. Let $V_1,\ldots,V_n$ denote a collection of random variables, where $V_i$ takes the value $1$ if the index $i$ is an element of the random set $\mathsf{h}$ used to define the half-sample bootstrap root \eqref{eq: bootstrap root}, and takes the value $-1$ otherwise. Observe that
\begin{flalign}
R^*_n(\bm{x}^{(d)}) = \hat{\theta}_\mathsf{h}(\bm{x}^{(d)}) - \hat{\theta}_n(\bm{x}^{(d)})
&= \frac{2}{n} \sum_{i=1}^n \mathbb{I}\{ i \in \mathsf{h}\} \bar{u}(\bm{x}^{(d)}, D_i) - \frac{1}{n} \sum_{i=1}^n \bar{u}(\bm{x}^{(d)}, D_i)\nonumber\\
&= \frac{1}{n}\sum_{i=1}^n V_i \left(\bar{u}(\bm{x}^{(d)}, D_i) - \theta_0(\bm{x}^{(d)})\right)~.
\label{eq: half as weight}
\end{flalign}
The weights $V_i$ are exchangeable Rademacher random variables, i.e., they are uniformly distributed on $\{1,-1\}$. If the weights were fully independent, the representation \eqref{eq: half as weight} reduces to the more familiar Rademacher bootstrap.\footnote{The representation \eqref{eq: half as weight} is due to \cite{yadlowsky2023evaluating}, who draw on a similar observation made in the context of two-sample testing in \cite{chung2013exact}. In \cref{app: bin boot}, we consider a variant of the bootstrap root \eqref{eq: bootstrap root} based on re-estimating and re-scaling $\hat{\theta}_n(\bm{x}^{(d)})$ on a subsample of a random size $\mathsf{Bin}(n,1/2)$. For this construction, the equivalent objects to the weights $V_i$ are fully independent. That is, this bootstrap root is equivalent to the Rademacher bootstrap root for linear statistics. We show that the resulting confidence regions obtain the same error rates on coverage accuracy.}
In practice, many widely applied estimators are not perfectly linearly decomposable. Often, however, estimators do satisfy an approximate linear decomposition, in the sense that the equality \eqref{eq: exact linear} holds with a remainder term of order, say, $o_p(n^{-\gamma})$ for some positive constant $\gamma$. As we will see, estimators constructed with subsampled kernels are approximately linear around some unknown function $\bar{u}(\cdot,\cdot)$. If an estimator $\hat{\theta}_n(\bm{x}^{(d)})$ is approximately linear, then the subsampled estimate $\hat{\theta}_{\mathsf{h}}(\bm{x}^{(d)})$ is \emph{immediately} also approximately linear.\footnote{Approximate linearity does not immediately imply a representation analogous to \eqref{eq: half as weight} for a root constructed with an empirical bootstrap, as approximate linearity would not necessarily hold under the empirical distribution.} Thus, for approximately linear statistics, the representation \eqref{eq: half as weight} continues to hold, now with a remainder term of order $o_p(n^{-\gamma})$. As a consequence, the validity of the confidence region formulated in \cref{def: uniform ci}, for approximately linear estimators, follows by deriving central limit theorems for the statistics \eqref{eq: exact linear} and \eqref{eq: half as weight} and applying an appropriate generalization of Slutsky's Theorem.
\subsection{Application}
We now return to the data studied in \cite{banerjee2015multifaceted}. \cref{fig: half} displays upper and lower half-sample confidence bounds for the CATE \eqref{eq: CATE} on post-treatment assets. The null hypothesis that the CATE is equal to zero is only rejected for individuals with low baseline assets. The lower bounds are meaningfully larger than zero only for individuals with low baseline assets and high baseline consumption. That is, the graduation program has a positive impact on individuals who do not have many assets to begin with, but who do have access to a stable source of consumption. On the other hand, the confidence regions contain zero for individuals who have either low baseline consumption or high baseline assets.
\begin{figure}[t]
\begin{centering}
\caption{Half-Sample Confidence Region}
\label{fig: half}
\medskip{}
\begin{tabular}{c}
\textit{Panel A: Upper Bound}\tabularnewline
\includegraphics[scale=0.40]{plots/half_upper}\tabularnewline
\textit{Panel B: Lower Bound}\tabularnewline
\includegraphics[scale=0.40]{plots/half_lower}\tabularnewline
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig: half} displays heat maps giving half-sample upper and lower confidence bounds for the CATE of the intervention studied in \cite{banerjee2015multifaceted} on post-treatment total assets. The confidence bounds are constructed at level $\alpha = 0.1$. The upper and lower bounds are displayed with different color palettes to emphasize the use of different scales. A contour line has been superimposed over the lower bound to demarcate where the bound crosses zero. The axes and estimator are the same as in \cref{fig: cate}.}{\footnotesize\par}
\end{figure}
It is illustrative to contrast the estimates and confidence bounds displayed in \cref{fig: cate,fig: half} with a more frequently encountered method for assessing treatment effect heterogeneity---interacted linear regression. \cref{tab: regression} reports estimates and standard errors associated with several linear regression specifications, constructed with the same data. The first column reports the coefficient from a regression of post-treatment assets on a treatment indicator. Consistent with results reported in \cite{banerjee2015multifaceted}, the average effect of the program is positive and statistically significant. The second and third specifications interact the treatment indicator with pre-treatment assets and consumption, respectively. In both cases, the estimate of the coefficient on the interaction is statistically insignificant. The fourth specification interacts both pre-treatment assets and consumption with a treatment indicator. Here, all coefficients lose statistical significance.
The implicit view of much of applied economics appears to be that the flexibility afforded by nonparametric methods is not worth sacrificing the statistical precision of more parsimonious, linear, alternatives.\footnote{We conduct a small survey of papers published in the \textit{American Economic Review} in the first six months of 2023. Of 38 empirical papers, 30 assess treatment effect heterogeneity in some way. As best as we can tell, only two papers display nonparametric estimates of effect heterogeneity. By contrast, 10 display the results of an interacted linear regression typified by \cref{tab: regression}. The rest are either structural papers, or only display coefficients on interactions of binary covariates. See \cref{app: survey} for further details.} The exercise here suggests otherwise. The linearity imposed by interacted linear regression masks the structure in the effect heterogeneity recovered by random forest regression. The resulting bias is so substantial that statistical significance is lost. The half-sample confidence regions developed in this paper enable the recovery of statistically significant measurements of effect heterogeneity. The remainder of the paper is devoted to developing theoretical guarantees on the accuracy of confidence bounds typified by \cref{fig: half}.
\begin{table}[t]
\begin{centering}
\caption{Interacted Linear Regression\label{tab: regression}}
{\small
\begin{tabular}{lcccc}
\toprule
& \multicolumn{4}{c}{Dependent Variable: Post-Treatment Assets}\tabularnewline
\midrule
& (1) & (2) & (3) & (4)\tabularnewline
\midrule
\midrule
\multirow{1}{*}{Treatment} & 0.218 (0.031) & 0.194 (0.032) & 0.102 (0.172) & 0.111 (0.201)\tabularnewline
\multirow{1}{*}{Assets} & & 0.770 (0.051) & & 1.229 (0.288)\tabularnewline
\multirow{1}{*}{Consumption} & & & 0.06 (0.022) & -0.026 (0.026)\tabularnewline
\multirow{1}{*}{Assets $\times$ Consumption} & & & & -0.119 (0.076)\tabularnewline
\multirow{1}{*}{Treatment $\times$ Assets} & & 0.047 (0.093) & & -0.065 (0.552)\tabularnewline
\multirow{1}{*}{Treatment $\times$ Consumption} & & & 0.031 (0.045) & 0.021 (0.050)\tabularnewline
\multirow{1}{*}{Treatment $\times$ Assets $\times$ Consumption} & & & & 0.027 (0.141)\tabularnewline
\midrule
\midrule
Observations: & 2,438 & 2,438 & 2,438 & 2,438\tabularnewline
\bottomrule
\end{tabular}}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{tab: regression} reports estimates of the coefficients of four linear regression specifications, constructed with the \cite{banerjee2015multifaceted} data. Robust standard errors are displayed in parentheses.}{\footnotesize\par}
\end{table}
\section{Coverage Accuracy\label{sec: subsampled kernel regression}}
We establish a bound on the accuracy of the nominal coverage probability for the confidence region introduced in \cref{def: uniform ci}. We begin in \cref{sec: sub kernel def} by discussing subsampled kernel regression and introducing several quantities that take a prominent role in our analysis. Our results apply to conditional moments that satisfy a Neyman orthogonality condition, in addition to several simple regularity conditions. We overview these restrictions in \cref{sec: nuisance and moment}. Our main result is stated in \cref{sec: kernel coverage}. The results of a simulation calibrated to the \cite{banerjee2015multifaceted} data are reported in \cref{sec: simulation}. Throughout, we assume that the data $Z_i$ and $X_i$ take values in the spaces $\mathcal{Z}$ and $\mathcal{X}$, respectively.
\subsection{Subsampled Kernel Regression\label{sec: sub kernel def}} Subsampled kernel regression is a broad class of algorithms for solving regression problems of the form \eqref{eq: subsampled kernel}, based on constructing a data-driven kernel function $K(x,x^\prime)$ with subsampling. Formally, fix some positive integer $r$ and let $(\mathsf{s}_q)_{q=1}^r$ collect a sequence of subsets of $[n]$ drawn independently and uniformly from $\mathcal{S}_{n,b}$. Let $\xi$ denote some auxiliary source of randomness and let $(\xi_{\mathsf{s}_q})_{q=1}^r$ collect a set of independent random variables with the same distribution as $\xi$. We study conditional empirical moment estimators of the form \eqref{eq: subsampled kernel}, where the kernel function $K(x,x^\prime)$ admits the decomposition
\begin{equation}\label{eq: subsampled kernel def}
K(x, X_i) = \sum_{q=1}^r \mathbb{I}\{i \in \mathsf{s}_q\} \kappa(x, X_i, D_{\mathsf{s}_q}, \xi_{\mathsf{s}_q})
\end{equation}
for some known kernel $\kappa(\cdot,\cdot,D_{\mathsf{s}}, \xi_{\mathsf{s}})$. Several widely applied instances of subsampled kernels are as follows.\footnote{The half-sample bootstrap is particularly computationally efficient for estimators constructed with subsampling, as the estimator and the half-sampled estimator can be constructed using the same collection of subsamples. See Section 4.1 of \cite{athey2019generalized} for further discussion.}
\begin{ex}\label{ex: knn}
Subsampled nearest-neighbors regression is a simple example of a kernel with the structure \eqref{eq: subsampled kernel def} \citep{fix1989discriminatory}. Here, the kernel $\kappa(x,X_{i^\prime},D_{\mathsf{s}}, \xi_{\mathsf{s}})$ is non-zero if and only if $X_{i^\prime}$ is one of the $k$ closest points to $x$ among the points in the subsample $D_{\mathsf{s}}$.
\end{ex}
\begin{ex}\label{ex: random forest}
Random forest regression, introduced by \cite{breiman2001random}, is another example of a kernel with the structure \eqref{eq: subsampled kernel def}. In this case, each pair $(D_{\mathsf{s}}, \xi_{\mathsf{s}})$ generates some partition of $\mathcal{X}$. The kernel $\kappa(x,x^\prime,D_{\mathsf{s}}, \xi_{\mathsf{s}})$ is non-zero if and only if $x$ and $x^\prime$ are in the same element of the partition generated by $(D_{\mathsf{s}}, \xi_{\mathsf{s}})$. Often, such partitions are constructed with recursive algorithms, e.g., the ``CART'' algorithm of \cite{breiman2017classification}.
\end{ex}
We impose the following restrictions on the kernels under consideration.
\begin{assumption}[Honesty and Positive Symmetry]\label{assu: kernel restriction}$\text{ }$\\
\noindent(i) The kernel $\kappa(\cdot,\cdot,D_{\mathsf{s}},\xi)$ is Honest in the sense that
\begin{equation}
\kappa(x,X_i,D_{\mathsf{s}},\xi_{\mathsf{s}}) \ci m(D_i; \theta, g) \mid X_i, D_{\mathsf{s}_{-i}}~,
\end{equation}
where $\ci$ denotes conditional independence and $\mathsf{s}_{-i}$ denotes the set $\mathsf{s}\setminus\{i\}$.
\noindent (ii) The kernel $\kappa(\cdot,\cdot,D_{\mathsf{s}},\xi)$ is positive and satisfies the restriction $\sum_{i\in{s}}\kappa(\cdot, X_i, D_{\mathsf{s}},\xi_{\mathsf{s}}) = 1$ almost surely. Moreover, the conditional expectation $\mathbb{E}\left[\kappa(\cdot, X_i, D_{\mathsf{s}},\xi_{\mathsf{s}})\mid D_{\mathsf{s}}\right]$ is invariant to permutations of the data $D_{\mathsf{s}}$.
\end{assumption}
\noindent The ``Honesty'' condition stipulated in Part (i) of \cref{assu: kernel restriction} imposes the restriction that any part of the data $D_i$ that can affect the value of the moment $m(D_i; \theta, g)$ cannot affect the value of the kernel $\kappa(x,X_i,D_{\mathsf{s}},\xi_{\mathsf{s}})$. This condition was introduced in \cite{athey2016recursive}. Honesty is often achieved through kernel construction schemes based on sample-splitting; see \cite{wager2018estimation} and \cite{athey2019generalized} for further discussion. Part (ii) of \cref{assu: kernel restriction} imposes several weak regularity conditions.
The following two quantities restrict the ``size'' and ``variability'' of the chosen kernel. Define the generic norm $\|\cdot\|_\infty$ on $\mathcal{X}$.
\begin{defn}[Shrinkage and Incrementality]\label{def: shrinkage and incrementality} Let $\mathsf{s}$ be an arbitrary element of $\mathcal{S}_{n,b}$ and let $l$ be any element of $\mathsf{s}$.\\
(i) We say that the kernel $\kappa(\cdot,\cdot,D_{\mathsf{s}},\xi_{\mathsf{s}})$ has a uniform shrinkage rate $\varepsilon_{b}$ if
\begin{equation}
\sup_{P \in \mathbf{P}} \sup_{j \in [d]} \mathbb{E}\left[ \max\left\{ \|X_i - x^{(j)} \|_\infty : \kappa(x^{(j)},X_i,D_{\mathsf{s}},\xi_{\mathsf{s}}) > 0 \right\} \right] \leq \varepsilon_{b}~.
\end{equation}
\noindent (ii) We say that the kernel $\kappa(\cdot,\cdot,D_{\mathsf{s}},\xi_{\mathsf{s}})$ is uniformly incremental if
\begin{equation}\label{eq: incrementality def}
\inf_{P \in \mathbf{P}} \sup_{j \in [d]} \Var\left( \mathbb{E}\left[ \sum_{i\in\mathsf{s}}
\kappa(x^{(j)}, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) m(D_i; \theta, g) \mid D_l = D \right] \right) \gtrsim b^{-1}
\end{equation}
where $D$ is an independent random variable with distribution $P$.
\end{defn}
\noindent The shrinkage rate of a kernel $\kappa(\cdot,\cdot,D_{\mathsf{s}},\xi_{\mathsf{s}})$ is analogous to the bandwidth of a classical, deterministic, kernel. The incrementality restriction ensures that the chosen kernel is not overly dependent on a single data point. Both notions were introduced by \cite{wager2018estimation} and have been characterized explicitly for various widely applied subsampled kernel estimators.\footnote{The terminology ``shrinkage'' was introduced in \cite{oprescu2019orthogonal}, and is not intended to connote (explicit) regularization.}
\begin{ex}[continues=ex: knn]
In the case of honest subsampled $k$-NN regression, \cite{khosravi2019non} show that $\varepsilon_{b} \lesssim b^{-1/p}$, where $p$ is the ``intrinsic dimension" of the measure of the covariates $X_i$. Roughly speaking, a distribution has an intrinsic dimension of $p$ if it is (locally) well approximated by a measure supported on a subspace of $\mathcal{X}$ of dimension $p$. See e.g., \cite{kpotufe2011k} for further discussion. In turn, \cite{khosravi2019non} and \cite{peng2022rates} show that the kernels associated with both honest and non-honest variants of $k$-NN regression are incremental, up to logarithmic factors that depend on the dimension of the covariates.
\end{ex}
\begin{ex}[continues=ex: random forest]
Analogously, for honest random forest regression, under suitable regularity conditions, \cite{wager2018estimation} establish that $\varepsilon_{b} \lesssim b^{-c/p}$, where $p$ is the dimension of the domain of $X_i$. See e.g., \cite{wager2018estimation} and \cite{oprescu2019orthogonal} for further discussion. Bounds adaptive to the intrinsic dimension of the measure of $X_i$ are given in \cite{huo2023adaptation} under further restrictions. \cite{wager2018estimation} and \cite{peng2022rates} give simple conditions under which the kernel associated with subsampled, honest, random forest regression is uniformly incremental, again up to dimension dependent logarithmic factors.
\end{ex}
\subsection{Moment Restrictions\label{sec: nuisance and moment}}
The uniform confidence region introduced in \cref{def: uniform ci} makes no explicit correction for bias. In other words, the confidence regions that we consider are reliant on the use of estimators $\hat{\theta}_n(\bm{x}^{(d)})$ whose bias is of a smaller stochastic order than the sampling variance. To this end, we emphasize the use of conditional moments $M(\cdot;\theta_0, g_0)$, that satisfy a local Neyman orthogonality condition \citep{chernozhukov2018double}.
To introduce Neyman orthogonality, we impose further structure on the nuisance parameter $g_0$. The nuisance parameter $g_0$ is a collection of $h$ real-valued functions $g_0 = (g_0^{(k)})_{k=1}^h$, each having domain $\mathcal{Z}$. Define the norm
\begin{equation}\label{eq: nuisance norm}
\| g - g_0 \|_{2,\infty} = \sup_{k\in [h]} \sup_{j\in [d]} \left(\mathbb{E} \left[ (g^{(k)}(Z_i) - g_0^{(k)}(Z_i))^2 \mid X_i = x^{(j)} \right] \right)^{1/2}
\end{equation}
for any $g = (g^{(k)})_{k=1}^h$. The nuisance parameter $g_0$ takes values in the space $\mathcal{G}$.
\begin{defn}[Local Neyman Orthogonality]\label{def: orthogonal}
We say that a conditional moment $M(\cdot;\theta_0, g_0)$ is uniformly locally Neyman orthogonal if
\begin{equation}
\partial_g M(x^{(j)}; \theta_0, g_0)[g-g_0] = 0
\end{equation}
for all $P$ in $\mathbf{P}$ and $x^{(j)}$ in $\bm{x}^{(d)}$.
\end{defn}
\noindent The use of Neyman orthogonal moments ensures that the bias induced by the estimation of the nuisance parameter $g_0$ with $\hat{g}_n$ is small \citep{chernozhukov2018double}.
In addition to Neyman orthogonality, we require several smoothness restrictions on the function $m(\cdot;\theta, g)$. In the main text, to ease exposition, we impose the following linearity and boundedness restriction.
\begin{assumption}[Moment Linearity and Boundedness]\label{assu: moment linearity}
The moment function $m(\cdot;\theta, g)$ satisfies the linear representation
\begin{equation}
m(D_i;\theta, g) = m^{(1)}(D_i;\theta, g)\cdot \theta + m^{(2)}(D_i; g)
\end{equation}
for some known functions $m^{(1)}(\cdot;\theta, g)$ and $m^{(2)}(\cdot; g)$. Moreover, the absolute value of the function $m(\cdot;\theta, g)$ is bounded by the constant $(\theta+1)\phi$ for some $\phi \geq 1$ almost surely.
\end{assumption}
\noindent The linearity restriction entailed in \cref{assu: moment linearity} is inessential and is imposed for the sake of simplicity.\footnote{In \cref{sec: generic}, we show that moment linearity can be replaced by the high-level assumption that $\hat{\theta}_n(\bm{x}^{(d)})$ is consistent for $\theta_0(\bm{x}^{(d)})$. This state of affairs is standard in $M$-estimation problems \citep[see e.g., ][]{newey1994large}. As our running examples use linear moments, and sufficient conditions for the consistency of $\hat{\theta}_n(\bm{x}^{(d)})$ have been established \citep[see e.g., Assumption 4.1 and Theorem 4.3 of ][]{oprescu2019orthogonal}, we omit a detailed consideration of nonlinear moments.} The boundedness restriction can be easily weakened to a slightly more involved assumption stated in terms of sub-exponential norms. Again, we impose boundedness to ease exposition.
We maintain the following additional mild smoothness restrictions.
\begin{assumption}[Moment Smoothness]\label{assu: moment smoothness}$\text{ }$\\
\noindent \textbf{(i)} The moment $M(\cdot; \theta, g_0)$ is uniformly second order smooth, in the sense that
\begin{equation}\label{eq: orth smooth}
\sup_{P\in\mathbf{P}} \sup_{j\in[d]} \big\vert \partial_{g,g} M(x^{(j)}; \theta_0, g_0)[g-g_0]\big\vert \lesssim \|g - g_0\|^2_{2,\infty}
\end{equation}
for each $g$ in $\mathcal{G}$.
\noindent \textbf{(ii)} The variogram
\begin{equation*}
V(x ; g) = \mathbb{E}\left[ (m(D_i ;\theta_0(x), g) - m(D_i ;\theta_0(x), g_0))^2 \mid X = x\right]
\end{equation*}
is uniformly Lipschitz in its first component, in the sense that
\begin{equation}
\sup_{P\in\mathbf{P}} \sup_{g\in\mathcal{G}} \big\vert V(x ; g) - V(x^\prime ; g) \big\vert \lesssim \|x - x^\prime\|_\infty
\end{equation}
holds for all $x$ and $x^\prime$ in $\mathcal{X}$. Moreover, the variogram satisfies the mean-squared continuity condition
\begin{equation}
\sup_{P\in\mathbf{P}} \sup_{j\in[d]} \vert V(x^{(j)}, g) - V(x^{(j)}, g^\prime)\vert \lesssim \|g - g^\prime\|^2_{2,\infty}
\end{equation}
for each $g$ and $g^\prime$ in $\mathcal{G}$.
\noindent \textbf{(iii)} Define the moments
\begin{align}
M^{(1)}(x;\theta,g) &= \mathbb{E}\left[ m^{(1)}(D_i;\theta,g) \mid X_i = x\right] \quad\text{and}\\
M^{(2)}(x;g) &= \mathbb{E}\left[ m^{(2)}(D_i;g) \mid X_i = x\right]~,
\end{align}
associated with the functions $m^{(1)}(\cdot;\theta, g)$ and $m^{(2)}(\cdot; g)$ introduced in \cref{assu: moment linearity}. Both moments are
uniformly Lipschitz in their first component. That is, it holds that
\begin{equation}
\sup_{P\in\mathbf{P}} \sup_{g\in\mathcal{G}} \big\vert M^{(1)}(x ;\theta, g) - M^{(1)}(x^\prime ;\theta, g) \big\vert \lesssim \|x - x^\prime\|_\infty
\end{equation}
for all $x$, $x^\prime$, and $\theta$, and analogously for $M^{(2)}(x;g)$. Moreover, the first moment is uniformly Lipschitz in its second component and bounded from below in the sense that
\begin{align}
\sup_{P\in\mathbf{P}} \sup_{j\in[d]} \big\vert M^{(1)}(x^{(j)}; \theta_0,g) - M^{(1)}(x^{(j)}; \theta_0,g_0)\big\vert & \lesssim \|g - g_0\|_{2,\infty}\quad\text{and}\label{eq: Lipschitz Jacobian main}\\
\inf_{P\in\mathbf{P}} \inf_{j\in[d]} \big\vert M^{(1)}(x^{(j)};\theta_0,g)\big\vert & \geq c\label{eq: well-posedness main}
\end{align}
for each $g$ and $\theta$ and some positive constant $c$.
\end{assumption}
Neyman orthogonal moments satisfying the smoothness restrictions specified in \cref{assu: moment linearity} and \cref{assu: moment smoothness} are available for many widely considered statistical problems. For example, the moment \eqref{eq: CATE moment} is the unique Neyman orthogonal identifying moment for the parameter \eqref{eq: CATE} \citep[see e.g.,][]{hahn1998role, chernozhukov2018double}.\footnote{For estimation of CATEs, Part (i) of \cref{assu: moment smoothness} is implied by the more refined bound
\begin{equation}
\sup_{P\in\mathbf{P}} \sup_{j\in[d]} \big\vert \partial_{g,g} M(x^{(j)}; \theta_0, g_0)[g-g_0]\big\vert \lesssim \|\mu - \mu_0\|_{2,\infty} \|\beta - \beta_0\|_{2,\infty}~,
\end{equation}
where $\mu$ and $\beta$ are defined in \eqref{eq: CATE nuisance}. This structure yields the celebrated ``Double Robustness'' result for estimation of average treatment effects and conditional average treatment effects; see \cite{chernozhukov2018double} and \cite{kennedy2023towards} for further discussion. We impose the more general condition \eqref{eq: orth smooth}, as this bound exhibits many of the same features and will hold for a wider variety of problems.} Additional examples of problems where smooth Neyman orthogonal identifying moments are available include estimation of partially linear regression and partially linear instrumental variable regression \citep{chernozhukov2018double}, local average treatment effects \citep{tan2006regression,frolich2007nonparametric}, dynamic treatment effects \citep{lewis2021double}, and long-term treatment effects identified by surrogate outcomes \citep{athey2020estimating,chen2023semiparametric}.
\subsection{Coverage Accuracy\label{sec: kernel coverage}}
The following theorem gives a bound on the error in the nominal coverage probability of the confidence regions introduced in \cref{def: uniform ci}. We emphasize that the result is applicable to asymptotic regimes where the dimension of the query-vector $\bm{x}^{(d)}$ can be exponentially larger than the sample size $n$.
\begin{theorem}\label{eq: asymptotic validity subsampled kernel}
Suppose that the kernel $\kappa(\cdot,\cdot,D_{\mathsf{s}},\xi_{\mathsf{s}})$ satisfies \cref{assu: kernel restriction}, has uniform shrinkage rate $\varepsilon_{b}$, and is uniformly incremental. Additionally, suppose that the Neyman orthogonal moment function $M(\cdot;\theta_0, g_0)$ satisfies \cref{assu: moment linearity} and \cref{assu: moment smoothness}. Moreover, suppose that quantity $\|\theta_0(\bm{x}^{(d)})\|_\infty$ is uniformly bounded as $P$ varies over $\mathbf{P}$ and that $r$ has been chosen to satisfy $n \leq b \sqrt{r}$. If the nuisance parameter estimator $\hat{g}_n$ is statistically independent of the data $\mathbf{D}_n$ and satisfies the probability bound
\begin{align}
\sup_{P\in\mathbf{P}}
P\left\{ \|\hat{g}_n - g_0 \|^2_{2,\infty} \geq \sqrt{\frac{b}{n}} \delta^2_{n,g} \right\} \lesssim \frac{1}{n}\label{eq: g rate reduce}
\end{align}
for some sequence $\delta_{n,g}$, then the confidence region formulated in \cref{def: uniform ci} satisfies the bound
\begin{align}
& \sup_{P\in{\mathbf{P}}}\big\vert P\left\{ \theta_0(\bm{x}^{(d)}) \in \hat{\mathcal{C}}(\bm{x}^{(d)}) \right\} - (1 - \alpha) \big\vert \nonumber \\
& \quad \quad
\lesssim \left(\frac{b \phi^4 \log^5(dn)}{n}\right)^{1/4} + \left(\delta^2_{n,g} + \sqrt{\frac{n}{b}}\varepsilon_b\right)\sqrt{\log(d)}~,\label{eq: kernel coverage}
\end{align}
for all $b$ larger than $C\log(n)$.\footnote{The statistical family $\mathbf{P}$ is defined implicitly by the omitted constants in the uniform bounds stated in \cref{def: shrinkage and incrementality} and \cref{assu: moment smoothness}, in addition to the restriction that $\|\theta_0(\bm{x}^{(d)})\|_\infty$ is bounded as $P$ varies over $\mathbf{P}$. }
\end{theorem}
\begin{remark}
To expedite exposition, in stating \cref{eq: asymptotic validity subsampled kernel}, we have assumed that the nuisance parameter estimator $\hat{g}_n$ is computed on a separate sample, independent of the data $\mathbf{D}_n$. This can be achieved by randomly splitting the available data into two subsamples. The first subsample can be used to construct the nuisance parameter estimate $\hat{g}_n$. The second subsample can then be used to construct the confidence region $\hat{\mathcal{C}}(\bm{x}^{(d)})$. There are practical issues with this strategy. First, the region $\hat{\mathcal{C}}(\bm{x}^{(d)})$ might be sensitive to the choice of the split of the data \citep{ritzwoller2023reproducible}. Second, by splitting the data, researchers incur a potentially meaningful loss in statistical precision. In \cref{app: stability} we give additional conditions under which the nuisance parameter estimator $\hat{g}_n$ can be computed using the same data used to construct the region $\hat{\mathcal{C}}(\bm{x}^{(d)})$.\footnote{Our analysis builds on a similar result given in \cite{chen2022debiased}, who give a pointwise analysis of unconditional moment estimators.} If these conditions are not plausible for a given application, in \cref{app: cross}, we detail a modified version of the confidence region given in \cref{def: uniform ci} that is constructed with cross-splitting, and will be less sensitive to residual randomness or subject to losses in statistical power.
\end{remark}
\begin{remark}
\cref{eq: asymptotic validity subsampled kernel} follows from an application of a more general result stated in \cref{sec: generic}. This result applies to conditional moment estimators that are not necessarily constructed with subsampled kernels. Rather, the result holds under the high-level assumption that the estimator $\hat{\theta}_n(\bm{x}^{(d)})$ is approximately linear, with a sufficiently small remainder term, and has sufficiently small bias and stochastic equicontinuity. An outline of the argument used to verify these conditions for subsampled kernel regression is given in \cref{app: kernel verify outline}. The detailed proof is given in \cref{app: kernel verify pf}.
Some aspects of this argument are standard. Others are new. First, through a standard series of expansions \citep[see e.g.,][]{chernozhukov2018double, oprescu2019orthogonal}, we show that the root $R_n(\bm{x}^{(d)})$ can be approximated by the statistic
\begin{align}
\bar{U}_{n,b}(\bm{x}^{(d)}) & = \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \mathbb{E}\left[u(\bm{x}^{(d)}; D_{\mathsf{s}}, \xi_{\mathsf{s}}, \theta_0, g_0)\mid D_{\mathsf{s}}\right]~,\quad\text{where}\label{eq: U expo}\\
u(x; D_\mathsf{s}, \xi_{\mathsf{s}}, \theta, g)
&= - M^{(1)}(x;g_0)^{-1}\sum_{i\in\mathsf{s}}
\big( \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) m(D_i; \theta, g) \\
& \quad\quad\quad\quad\quad\quad\quad\quad\quad- \mathbb{E}\big[ \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) m(D_i; \theta, g)\big]\big)~,\nonumber
\end{align}
with a remainder term given by the second term in \eqref{eq: kernel coverage}. The quantity \eqref{eq: U expo} can be recognized as a complete, deterministic, $U$-statistic of order $b$.\footnote{The incrementality condition specified in \cref{def: shrinkage and incrementality} ensures that this $U$-statistic is non-degenerate.} Second, we apply a new result that demonstrates that $U$-statistics of order $b$ are approximately linear with a remainder term of order approximately $(b/n)^{b/2}$, up to dimension dependent logarithmic factors. This is a dramatic improvement over analogous results given in \cite{song2019approximating} and \cite{minsker2023u}, whose remainder terms exhibit polynomial decay as $b$ and $n$ grow and only apply to regime $b = o(n^{1/3})$. This result has other applications and is presented in detail in \cref{sec: u-stat}. Written differently, we establish that the root $R_n(\bm{x}^{(d)})$ satisfies a linear representation of the form \eqref{eq: exact linear}, up to a small remainder term. It immediately follows that the bootstrap root $R^*_n(\bm{x}^{(d)})$ satisfies the linear representation \eqref{eq: half as weight}, up to a small remainder term. We conclude by applying suitable central limit theorems \citep{chernozhuokov2022improved} to the linear terms \eqref{eq: exact linear} and \eqref{eq: half as weight}.
\end{remark}
\begin{remark}
The bound \eqref{eq: kernel coverage} can be interpreted as a bias-variance decomposition. Or, perhaps more literally, a bias-normal-approximation decomposition. The first term in \eqref{eq: kernel coverage} results from a Berry-Esseen type bound on the accuracy of a normal approximation to (the maximum of) \eqref{eq: U expo}. The term involving the kernel shrinkage $\varepsilon_b$ is a remnant of a bound on the supremum of the bias of the estimator $\hat{\theta}_n(\bm{x}^{(d)})$.
\end{remark}
\cref{eq: asymptotic validity subsampled kernel} states a bound on coverage error in terms of two generic sequences: $\delta_{n,g}$, expressing the rate of convergence of the nuisance parameter estimator $\hat{g}_n$, and $\varepsilon_b$, measuring the effective bandwidth of the kernel. The bound \eqref{eq: kernel coverage} exhibits an interesting tradeoff between these objects and the choice of subsample size $b$. To see this, suppose that
\begin{equation}
b = n^{\gamma_b},\quad \varepsilon_b \lesssim b^{-\gamma_\varepsilon},\quad\text{and}\quad \|\hat{g}_n - g_0\|_{2,\infty} \lesssim n^{-\gamma_g}~,
\end{equation}
with probability greater than $1-1/n$, for some constants $\gamma_b$, $\gamma_\varepsilon$, and $\gamma_g$ between $0$ and $1$. In this case, ignoring logarithmic factors and other constants, the bound \eqref{eq: kernel coverage} can be re-expressed as
\begin{equation}
n^{\frac{\gamma_b - 1}{4}} + n^{\frac{1-\gamma_b-4\gamma_g}{2}} + n^{\frac{1-\gamma_b(1+2\gamma_\varepsilon)}{2} }~.
\end{equation}
In other words, the confidence region defined in \cref{def: uniform ci} is consistent if
\begin{equation}\label{eq: consistence conditions}
1 \leq \gamma_b+ 4\gamma_g \quad\text{and}\quad 1 \leq \gamma_b(1+2\gamma_\varepsilon)~,
\end{equation}
respectively. That is, we are able to accommodate larger values of the shrinkage rate $\varepsilon_b$ and nuisance parameter estimation error $\delta_{n,g}$ if the subsample size $b$ is larger, relative to the sample size $n$. However, as the subsample size $b$ increases, the normal approximation error (i.e., the first term in \eqref{eq: kernel coverage}) increases.
Recall from \cref{sec: sub kernel def} that, for many popular, honest, subsampled kernel estimators, the shrinkage rate $\varepsilon_b$ satisfies a bound $\varepsilon_b \lesssim b^{-c/p}$ for some small constant $c$ and some integer $p$ measuring the (potentially, intrinsic) dimension of the covariates $X_i$. Thus, in order to ensure that the second inequality in consistency condition \eqref{eq: consistence conditions} is satisfied, it is essential to accommodate subsample sizes $\gamma_b$ close to one, i.e., the regime $b = o(n)$. This is enabled by the general results on the asymptotic linearity of $U$-statistics given in \cref{sec: u-stat}.
When the subsample size scaling factor $\gamma_b$ is close to one, the restriction imposed by the consistency condition \eqref{eq: consistence conditions} on the rate of convergence of the nuisance parameter estimator $\hat{g}_n$ is less stringent. That is, if $\gamma_b = 1- c/p$, then we need that $\gamma_g \geq c/4p$. On the other hand, if $\gamma_b$ is close to zero, i.e., if $b$ is very small, then the first condition in \eqref{eq: consistence conditions} reduces to the familiar condition that nuisance parameters can be estimated at the rate $n^{-1/4}$ in root mean squared error \citep{chernozhukov2018double}. If the nuisance parameter estimator $\hat{g}_n$ is itself estimated with random forest regression, \cite{syrgkanis2020estimation}, \cite{chi2022asymptotic}, and \cite{huo2023adaptation}, among others, give conditions under which sufficient rates of convergence can be achieved.
\subsection{Performance\label{sec: simulation}}
We now measure the performance of the confidence region formulated in \cref{def: uniform ci}. We apply a method for simulation design proposed by \cite{athey2021using}. In particular, we calibrate a simulation to the \cite{banerjee2015multifaceted} data using a Generative Adversarial Network (GAN) \citep{goodfellow2014generative}. Further details on this calibration are given in \cref{app: simulation}. In effect, we construct a data generating process that approximates the \cite{banerjee2015multifaceted} data, where we know the true value of the CATE $\theta_0(x)$ queried at each value $x$ used to construct the grids displayed in \cref{fig: cate,fig: half}.
In this section, we consider the setting where the nuisance parameter estimator $\hat{g}_n$ is computed using the same data used to construct the region $\hat{\mathcal{C}}(\bm{x}^{(d)})$.\footnote{Recall that a theoretical justification for this approach is given in \cref{app: stability}.} Analogous results for an estimator constructed with cross-splitting are given in \cref{app: cross}. The results reported there are very similar.
Measurements of performance are taken as two parameters vary. First, we consider several values of the sample size $n$. In particular, we consider settings with $n = h\cdot n_0$, for $h$ in $\{1, 2.5, 5, 7.5\}$, where $n_0$ is the sample size of the \cite{banerjee2015multifaceted} data. Second, we vary the proportion $b/n$. We consider three regimes: $b/n = 0.05$, $b/n = (2/(h+1))0.05$, and $b/n=(1/h)0.05$. Observe that $b$ increases in proportion to $n$ in the first regime and that $b$ is constant as $n$ varies in the third regime. The second regime resides between these two extremes.
\begin{figure}[t]
\begin{centering}
\caption{Performance}
\label{fig: performance}
\medskip{}
\begin{tabular}{c}
\includegraphics[scale=0.38]{plots/coverage}
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig: performance} displays several measurements of the performance of the confidence intervals formulated in \cref{def: uniform ci} in a simulation calibrated to the \cite{banerjee2015multifaceted} data. The nominal level is $\alpha = 0.1$; a horizontal dotted line is displayed at the nominal coverage $1-\alpha$ in the first panel. The confidence bounds considered are constructed analogously to the confidence bounds displayed in \cref{fig: half}. The $x$-axis of each panel is the sample multiplier $h$. The color of each measurement varies with the choice of $b/n$. Further details on the design and implementation of this simulation are given in \cref{app: simulation}.}{\footnotesize\par}
\end{figure}
\cref{fig: performance} displays measurements of the coverage and width of the confidence region formulated in \cref{def: uniform ci}, in addition to measurements of the bias of the estimator $\hat{\theta}_n(\bm{x}^{(d)})$. The first row displays measurements of the coverage of the confidence region, the coverage of the lower bound (i.e., Panel B of \cref{fig: half}), and the coverage of the upper bound (i.e., Panel A of \cref{fig: half}). The nominal level is $\alpha = 0.1$. Throughout, the confidence region is somewhat conservative.
The second row of \cref{fig: performance} illustrates a bias-variance trade-off with the subsample size $b$. The first panel displays measurements of the average width of the confidence region. Here, the average is taken over both simulation draws and the query-vector $\mathbf{x}^{(d)}$. The width of the confidence region is increasing in the proportion $b/n$ and is essentially constant if $b/n$ is constant as $n$ increases. By contrast, the second two panels display the maximum and average bias of the estimator $\hat{\theta}_n(\bm{x}^{(d)})$, again taken over the query-vector $\mathbf{x}^{(d)}$. The bias is decreasing in the proportion $b/n$ and is essentially constant if $b$ is constant as $n$ varies.
\section{General Results for High-Dimensional $U$-Statistics\label{sec: u-stat}}
An essential step in the proof of \cref{eq: asymptotic validity subsampled kernel} follows from a new order-explicit bound on the remainder in a linear approximation to a high-dimensional $U$-statistic. In this section, we present this result and state several corollaries. In particular, we give new order-explicit results on the concentration and normal approximation of high-dimensional $U$-statistics. That is, we consider the asymptotic behavior of the $b$ order $U$-statistic
\begin{equation}\label{eq: gener determ u stat}
\bar{U}_{n,b}(\bm{x}^{(d)}) = \frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}} u(\bm{x}^{(d)}; D_\mathsf{s})~,
\end{equation}
where the vector
$
u(\bm{x}^{(d)}; \cdot)
$
collects the deterministic, symmetric, real-valued kernel function
$
u(x; \cdot),
$
evaluated at the $d$-vector of points $\bm{x}^{(d)} = (x^{(j)})_{j=1}^d$ in the space $\mathcal{X}$. We assume that each component of the kernel function $u(\bm{x}^{(d)}; D_\mathsf{s})$ has mean zero.
\subsection{Context\label{sec: lit review}}
The asymptotic analysis of $U$-statistics was initiated by \cite{hoeffding1948class}, who established a central limit theorem in the regime where the order $b$ is fixed and the sample size $n$ is increasing. The Hoeffding central limit theorem has been extended only recently to the regime where the order $b$ increases with the sample size $n$. \cite{diciccio2022clt} give a result with this flavor in the regime where $b = o(n^{1/2})$. \cite{wager2018estimation}, \cite{peng2022rates}, and \cite{minsker2023u} strengthen this result to the regime where $b=o(n)$. We state and prove this more general result for the sake of completeness, and because its main ideas will serve as useful touch points in the more involved analysis to follow. Define the kernel variance $\nu^2_j = \Var(u(x^{(j)}; D_{[b]}))$, the H\'{a}jek projection
\begin{equation}\label{eq: app hajek}
u^{(1)}(x^{(j)}; D) = \mathbb{E}\left[ u(x^{(j)}; D_{[b]}) \mid D_1 = D\right]~,
\end{equation}
and the H\'{a}jek projection variance $\sigma^2_{b,j} = \Var(u^{(1)}(x^{(j)}; D_i))$.
\begin{theorem}\label{thm: Hoef clt}
For any sequence of kernel orders $b = b_n$, where $\nu^2_j/n\sigma^2_{b,j} \to 0$ as $n\to \infty$, we have that
\begin{equation}
\sqrt{\frac{n}{\sigma^2_{b,j} b^2}} \frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}} u(x^{(j)}; D_\mathsf{s}) \overset{d}{\to} \mathsf{N}(0,1)~,
\end{equation}
as $n\to\infty$, where $\overset{d}{\to}$ denotes convergence in distribution.
\end{theorem}
\noindent The proof is given in \cref{eq: proof of hoef clt}. As a by-product of the proof, we show that $b \sigma^2_{b,j} \leq \nu^2_j$. Thus, the normalization $\nu^2_j/n\sigma^2_{b,j} \to 0$ implies that $b = o(n)$.
Large deviation bounds for high-dimensional $U$-statistics, i.e., $U$-statistics with vector-valued kernels, were not given until \cite{hoeffding1963probability}. This result is now more standard; a modern version is stated as follows. A proof is given in \cite{song2019approximating}. The norm $\|\cdot\|_{\psi_1}$ denotes the $\psi_1$-Orlicz norm.\footnote{Random variables are sub-exponential if and only if they have a finite $\psi_1$-Orlicz norm \citep[Section 2.7, ][]{vershynin2018high}.}
\begin{lemma}[Lemma A.5, \cite{song2019approximating}]\label{lem: Q tail}If the bound $\|u(x^{(j)};D_{\mathsf{s}})\|_{\psi_1} \leq \phi$
is satisfied for each $j$ in $[d]$, then
\begin{equation}
\|\bar{U}_{n,b}(\bm{x}^{(d)}) \|_\infty \lesssim
\sqrt{\frac{b\bar{\nu}^2 \log(dn)}{n}}
+ \frac{b\phi \log^2(dn)}{n}
\end{equation}
with probability greater than $1 - C/n$, where $\bar{\nu}^2 = \max_{j\in[d]} \nu^2_j$.
\end{lemma}
\noindent Again, \cref{lem: Q tail} demonstrates that $\bar{U}_{n,b}(\bm{x}^{(d)})$ concentrates in the regime that $b = o(n)$, up to a logarithmic factor that depends on the dimension $d$. Here, however, concentration is expressed in terms of the quantity $n^{-1} b\bar{\nu}^2$, rather than the more appropriate, and potentially substantively smaller, normalizing quantity $n^{-1} b^2\sigma^2_{b,j}$ used in \cref{thm: Hoef clt}. In part motivated by this incongruity, \cite{arcones1993limit}, \cite{arcones1995bernstein}, \cite{gine2000exponential}, establish a series of refined large deviation bounds for high-dimensional $U$-statistics that use the appropriate normalizing factor (among many other related results).\footnote{\cite{peel2010empirical} give empirical Bernstein versions of the \cite{arcones1993limit} deviation bounds. Again, here, the constants used to express these bounds depend exponentially on the order $b$.} See \cite{de2012decoupling} for a textbook treatment. However, the constants used to express these bounds depend implicitly on the order $b$, and so are not applicable to asymptotic regimes where $b$ may be growing with the sample size $n$.
More recently, \cite{chen2018gaussian}, \cite{chen2019randomized}, and \cite{song2019approximating} have studied central limit theorems for high-dimensional $U$-statistics. Of these papers, only \cite{song2019approximating} gives results with explicit dependence on the order $b$. Their results are only applicable to the regime where $b = o(n^{1/3})$. \cite{minsker2023u} gives a large deviation bound with the correct normalizing factor and explicit order dependence, but this result is again only applicable to the regime $b = o(n^{1/3})$. Our main application to inference for subsampled kernel regression requires $n^{1 - c/p} \lesssim b$, where $p$ is the dimension of the covariate vector $Z_i$. Thus, results applicable only to the regime $b = o(n^{1/3})$ are insufficient.
\subsection{Concentration of the H\'{a}jek Residual\label{sec: app projection}}
We obtain a large deviation bound on the difference
\begin{equation}\label{eq: display app Hajek resid}
\bar{U}_{n,b}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n u^{(1)}(\bm{x}^{(d)}; D_i)~.
\end{equation}
We refer to the quantity \eqref{eq: display app Hajek resid} as the H\'{a}jek residual \citep{hajek1968asymptotic}. This bound is used in the proof of \cref{eq: asymptotic validity subsampled kernel} and implies a new large deviation bound and central limit theorem for high-dimensional $U$-statistics, stated in the following subsection. The proof is given in \cref{app: u stat}.
\begin{theorem}
\label{lem: projection maximal}
If the kernel $u(x^{(j)}; D_{\mathsf{s}})$ satisfies the bound $\|u(x^{(j)};D_{\mathsf{s}})\|_{\psi_1} \leq \phi$ for each $j$ in $[d]$, then
\begin{align}
& \Big\| \bar{U}_{n,b}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n u^{(1)}(\bm{x}^{(d)}; D_i) \Big\|_{\infty} \lesssim \xi_{n,b}~, \quad\text{where}\label{eq: Hajek bound}\\
&\quad\quad\quad\xi_{n,b} =\left(\frac{C b\log(dn)}{n}\right)^{b/2}\left(\bar{\nu} + b \sqrt{\frac{b}{n}} \phi\log^2(dn)\right)\nonumber~,
\end{align}
with probability greater than $1-C/n$, where $\bar{\nu}^2 = \max_{j\in[d]} \nu^2_j$.
\end{theorem}
\begin{remark}
Roughly speaking, the bound \eqref{eq: Hajek bound} follows by first demonstrating that the H\'{a}jek residual can be expressed as a degenerate $U$-statistic of order $b$. This allows us to derive Hoffman-J\o{}rgensen type bounds on higher moments of \eqref{eq: display app Hajek resid} with a symmetrization argument. Here, we make essential use of a symmetrization inequality for completely degenerate kernels, with explicit dependence on the order, due to \cite{sherman1994maximal}. This symmetrization inequality was also used in \cite{song2019approximating} and \cite{minsker2023u}. These papers apply the \cite{sherman1994maximal} symmetrization inequality separately to terms in a Hoeffding expansion of the $U$-statistic $\bar{U}_{n,b}(\bm{x}^{(d)})$. Aggregating the resulting bounds necessitates the restriction to the regime $b = o(n^{1/3})$. By contrast, \emph{we avoid this restriction by applying the symmetrization inequality once, to the H\'{a}jek residual itself.}
\end{remark}
The bound \eqref{eq: Hajek bound} indicates that the H\'{a}jek residual is of stochastic order $b (b\log(dn)/n)^{b/2}$. That is, roughly speaking, the H\'{a}jek residual concentrates like $(b/n)^{b/2}$ for sufficiently large $b$. Recall that ${n \choose b}$ is proportional to $(b/n)^b$. So, \cref{lem: projection maximal} can be interpreted as showing that complete $U$-statistics, i.e., sums of approximately $(b/n)^b$ dependent random variables, concentrate like sums of $(b/n)^b$ independent random variables around their H\'{a}jek projections. Substantively, if $b = n^{1-\delta}$ for some $\delta >0$, then the bound \eqref{eq: Hajek bound} shrinks \emph{exponentially} as $n$ increases. Exponential convergence, of this form, is a dramatic improvement over existing bounds \citep{song2019approximating,minsker2023u}, which decay polynomially as $n$ increases, and only in the regime $b = o(n^{1/3})$.
In our view, this result hints at an explanation for the widespread success of subsampling as a primitive operation in machine learning workflows and in statistical inference. Subsampled statistics, of a large order, are \emph{essentially} linear. Belaboring the point, aggregation of a potentially highly non-linear statistic $u(x; D_{[b]})$---a decision tree, say---through subsampling is a linearizing operation. Once linearized, the regular behavior---concentration, normality---of independent sums holds.
\subsection{Concentration and Normal Approximation}
We now state a large deviation bound and central limit theorem for the high-dimensional $U$-statistic \eqref{eq: gener determ u stat}. Both results are corollaries of \cref{lem: projection maximal}, apply to the regime $b=o(n)$, and depend on the correct normalizing factor. Proofs are given in \cref{sec: app cor p1,sec: app cor p2}.
\begin{cor} \label{cor: general u stat asymptotic}
Suppose that the conditions of \cref{lem: projection maximal} continue to hold. Let $Z$ denote a centered Gaussian random vector with the same covariance matrix as the observation $u^{(1)}(\bm{x}^{(d)}, D_i)$ and let $\mathcal{R}$ denote the set of hyper-rectangles in $\mathbb{R}^d$. There exist some $C_1<1$ such that if $b\log(dn) / n < C_1$ and $b^{-C_2} \leq \underline{\sigma}^2_b$ for some $C_2 > 0$, then:
\noindent\textbf{(i)} The inequality
\begin{equation}\label{eq: general u stat moment}
\sqrt{\frac{n}{b^2}} \| \Sigma^{-1/2} \bar{U}_{n,b}(\bm{x}^{(d)}) \|_\infty \lesssim \log^{1/2}(dn) + \frac{\phi\log^2(dn)}{\underline{\sigma}_b n^{1/2}}
\end{equation}
holds with probability greater than $1-Cn^{-1}$ for all $b \gtrsim C_2\log(n)$.
\noindent \textbf{(ii)} The inequality
\begin{align}\label{eq: general u stat clt}
\sup_{\mathsf{R} \in \mathcal{R}} \bigg\vert
P\left\{\sqrt{\frac{n}{b^2}}\Sigma^{-1/2} \bar{U}_{n,b}(\bm{x}^{(d)}) \in \mathsf{R}\right\} - P\left\{ \Sigma^{-1/2}Z \in \mathsf{R}\right\}\bigg\vert \lesssim \left(\frac{ \phi^2 \log^5(dn)}{\underline{\sigma}^2_b n}\right)^{1/4}
\end{align}
holds for all $b \gtrsim C_2 \log(n)$.
\end{cor}
\begin{remark}
The restriction that $b \gtrsim C \log(n)$ can be dropped along any sequence with $b/n\to0$. That is, analogous results will hold for sufficiently large $n$ and any $b=b_n$ such that $b_n / n \to 0$ as $n\to\infty$.
\end{remark}
\begin{remark}
\cite{de2012decoupling} state a result analogous to Part (i) of \cref{cor: general u stat asymptotic}, in the sense that the $U$-statistic $\bar{U}_{n,b}(\bm{x}^{(d)})$ is normalized by the correct quantity $n^{1/2}b^{-1}\Sigma^{-1/2}$. The constants used in their bound depend implicitly on $b$. Part (i) of \cref{cor: general u stat asymptotic} improves substantially on \cref{lem: Q tail} in contexts where the H\'{a}jek projection variances $\sigma_{b,j}^2$ are smaller than $b^{-1}$.\footnote{Section 4 of \cite{song2019approximating} gives several examples of statistics where the H\'{a}jek projection variance $\sigma_{b,j}^2$ is smaller than $b^{-1}$. Under the restriction $\sigma_{b,j}^2\gtrsim b^{-1}$, imposed in the application to subsampled kernel regression considered in \cref{sec: subsampled kernel regression}, Part (i) of \cref{cor: general u stat asymptotic} only improves on \cref{lem: Q tail} by a constant factor.} Part (ii) of \cref{cor: general u stat asymptotic} gives a high-dimensional generalization of \cref{thm: Hoef clt}. An analogous half-sample bootstrap central limit theorem follows from arguments very similar to parts of the proof of \cref{eq: asymptotic validity subsampled kernel}.
\end{remark}
\section{Conclusion\label{sec: conclusion}}
We build simultaneous confidence intervals for solutions to conditional moment equations. The confidence intervals are built around an estimator based on subsampled kernel regression. As a running example, we consider the construction of confidence regions for conditional average treatment effects around an estimator based on random forest regression. Empirically, we document that the proposed confidence region is able to recover treatment effect heterogeneity undetected by interacted linear regression. Theoretically, we establish a bound on coverage accuracy that illustrates a bias-variance tradeoff in the user-chosen subsample size. In order to do this, we obtain several new, best-available results on the asymptotics of high-dimensional $U$-statistics.
We give conditions sufficient for the asymptotic validity of the proposed confidence intervals. However, the confidence intervals are not necessarily optimal, in any particular sense, under the maintained assumptions. It is likely to be the case that an optimal confidence region would need to incorporate a bias estimate and procedure for choosing tuning parameters to balance bias and variance (see e.g., \cite{chernozhukov2014anti} and \cite{christensen2024adaptive} for instantiations of these ideas). Adapting this approach to subsampled moment regression is an interesting direction for further research.
\newpage
\end{spacing}
\begin{spacing}{1.2}
\bibliographystyle{apalike}
\bibliography{references.bib}
\end{spacing}
\newpage
\begin{appendix}
\begin{center}
\large{\it Supplemental Appendix to:}
\vskip0.2cm
\begin{spacing}{1}
\Large{\textbf{Simultaneous Inference for\vspace{0.5em}\\ Local Structural Parameters with Random Forests\protect\daggerfootnote{\textit{Date}: \today}}}\\\vspace{1em}
\begin{tabular}[t]{c@{\extracolsep{4em}}c}
\large{David M. Ritzwoller} & \large{Vasilis Syrgkanis}\vspace{-0.7em}\\ \vspace{-1em}
\small{Stanford University} & \small{Stanford University} \\ \vspace{-0.7em}
\end{tabular}
\end{spacing}
\end{center}
\begin{spacing}{1.13}
\DoToC
\end{spacing}
\thispagestyle{empty}
\setcounter{page}{0}
\setcounter{figure}{0}
\newpage
\begin{spacing}{1.4}
\normalsize
\section{An Abstract Bound on Coverage Error\label{sec: generic}}
In this appendix, we give an abstract bound on the accuracy of the nominal coverage probability for the confidence region introduced in \cref{def: uniform ci}. We make no use of the kernel structure expressed in \eqref{eq: subsampled kernel}. The proof is given in \cref{app: sub pf of generic decomposition}. We require several mild smoothness restrictions on the moment function $M(\cdot;\theta, g)$. In contrast to the set of assumptions specified in \cref{sec: nuisance and moment}, we do not require moment linearity. Instead, we impose the following generalization of Part (iii) of \cref{assu: moment smoothness}.
\begin{assumption}[Moment Restrictions]\label{assu: app a moment function} The moment function $M(x;\theta_0, g_0)$ is twice continuously differentiable in its second argument. Let
\begin{equation}\label{eq: 1 and 2 moments}
M^{(1)}(x;\theta,g) = \frac{\partial}{\partial \theta^\prime} M(x;\theta^\prime, g_0) \vert_{\theta^\prime = \theta}
\quad\text{and}\quad
H(x;\theta,g) = \frac{\partial^2}{\partial^2 \theta^\prime} M(x;\theta^\prime, g_0) \vert_{\theta^\prime = \theta}
\end{equation}
denote the Jacobian and Hessian of $M(\cdot;\theta_0, g_0)$ in $\theta$, respectively. The Jacobian $M^{(1)}(x;\theta,g)$ is uniformly Lipschitz in its second argument and bounded from below in the sense that
\begin{align}
\sup_{P\in\mathbf{P}} \sup_{j\in[d]} \big\vert M^{(1)}(x^{(j)};\theta,g) - M^{(1)}(x^{(j)};\theta,g_0)\big\vert & \lesssim \|g - g_0\|_{2,\infty}\quad\text{and}\label{eq: Lipschitz Jacobian}\\
\inf_{P\in\mathbf{P}} \inf_{j\in[d]} \big\vert M^{(1)}(x^{(j)};\theta,g)\big\vert & \geq c\label{eq: well-posedness}
\end{align}
for each $g$ and $\theta$ and some positive constant $c$. The Hessian $H(x;\theta,g)$ is uniformly bounded as $x$, $\theta$, and $g$ vary over their respective domains.
\end{assumption}
\noindent Moreover, we impose an analogous restriction on the centered empirical moment
\begin{equation}\label{eq: centered}
\bar{M}_n(x;\theta,g) = M_n(x;\theta,g,\mathbf{D}_n) - \mathbb{E}\left[ M_n(x;\theta,g,\mathbf{D}_n) \right]~,
\end{equation}
where we have made the dependence on $\mathbf{D}_n$ implicit to ease notation.
\begin{assumption}[Empirical Smoothness]\label{assu: empirical smoothness}
The centered empirical moment \eqref{eq: centered} is twice continuously differentiable in its second argument. Let
\begin{align}
\bar{H}_n(x;\theta, g) &= \frac{\partial^2}{\partial^2 \theta^\prime} \bar{M}_n(x;\theta^\prime, g_0) \vert_{\theta^\prime = \theta}
\end{align}
denote the Hessian of $\bar{M}_n(\cdot;\theta, g)$ in $\theta$, respectively. The Hessian $H_n(x;\theta,g)$ is uniformly bounded almost surely as $x$, $\theta$, and $g$ vary over their respective domains.
\end{assumption}
Next, we impose a set of high-level restrictions on the structure of the empirical conditional moment \eqref{eq: empirical moment}. At times we refer to the normalized statistic
\begin{equation}\label{eq: U def}
U_{n}(x) = -(M^{(1)}(x;g_0))^{-1}\bar{M}_{n}(x;\theta_0,g_0)~.
\end{equation}
First, we impose a condition that ensures that \eqref{eq: U def} is approximately linear.
\begin{assumption}[Approximate Linearity]\label{assu: linearity}
There exists a function $\bar{u}(\cdot,\cdot)$, a constant $\varphi\geq 1$, and real-valued sequences $\delta_{n,u}$ and $\rho_{n,u}$ such that $\mathbb{E}\left[\bar{u}(x^{(j)},D_i)\right] = 0$,
\begin{equation}\label{eq: Orlicz bound}
\|\bar{u}(x^{(j)},D_i)\|_{\psi_1} \leq \varphi~,
\quad\text{and}\quad
\mathbb{E}\left[\bar{u}^4(x^{(j)},D_i)\right] \leq \Var(\bar{u}(x^{(j)},D_i)) \varphi^2
\end{equation}
hold for all $j$ in $[d]$ and $P$ in $\mathbf{P}$. Moreover, if $\lambda^2_{j}$ denotes $\Var(\bar{u}(x^{(j)},D_i))$ and $\underline{\lambda}^2 = \min_{j\in[d]} \lambda^2_{j}$, then
\begin{equation}\label{eq: linearity bound}
\sup_{P\in\mathbf{P}}
P\left\{ \sqrt{\frac{n}{\underline{\lambda}^2}} \big\| U_n(\bm{x}^{(d)}) - \frac{1}{n}\sum_{i=1}^n \bar{u}(\bm{x}^{(d)}, D_i) \big\|_\infty \geq \delta_{n,u} \right\} \leq \rho_{n,u}~.
\end{equation}
\end{assumption}
\noindent Second, we impose several restrictions relating to the estimators $\hat{\theta}_n(\bm{x}^{(d)})$ and $\hat{g}_n$. We measure the error in the estimator of $\hat{\theta}_n(\bm{x}^{(d)})$ in terms of the norm
\begin{equation}\label{eq: theta norm}
\|\hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty = \sup_{j\in[d]} | \hat{\theta}_n(x^{(j)}) - \theta_0(x^{(j)}) |~.
\end{equation}
A closely related collection of conditions is stated in \cite{chernozhukov2018double}.
\begin{assumption}[Bias, Consistency, and Stochastic Equicontinuity]\label{assu: bias}Recall the definition of the object $\underline{\lambda}^2$ introduced in \cref{assu: linearity}. Let
\begin{align}
\bar{M}^{(1)}_n(x;\theta, g) & = \frac{\partial}{\partial \theta^\prime} \bar{M}_n(x;\theta^\prime, g_0) \vert_{\theta^\prime = \theta}
\end{align}
denote the Jacobian of the centered empirical moment $\bar{M}_n(x; \theta, g)$.\\
\textbf{(i)} Define the quantity
\begin{equation}
\mathsf{Bias}_n(x;\theta, g) = M(x;\theta,g)- \mathbb{E}\left[M_{n}(x;\theta,g,\bm{D}_n)\right]~.
\end{equation}
There exists a sequence $\delta_{n,B}$ such that
\begin{equation}
\sup_{P\in\mathbf{P}} \sup_{g\in\mathcal{G}}
\sqrt{\frac{n}{\underline{\lambda}^2}} \| \mathsf{Bias}_n(\bm{x}^{(d)};\theta(\bm{x}^{(d)}), g) \|_\infty \lesssim (1 + \|\theta(\bm{x}^{(d)})\|_\infty) \delta_{n,B}
\end{equation}
uniformly over any vector $\theta(\bm{x}^{(d)}) = \{\theta(x^{(j)})\}_{j=1}^d$.
\noindent \textbf{(ii)} There exist sequences $\delta_{n,m}$, $\delta_{n,g}$, $\delta_{n,\theta}$, $\rho_{n,m}$, $\rho_{n,g}$, and $\rho_{n,\theta}$ such that
\begin{align}
\sup_{P\in\mathbf{P}}
P\left\{
\sqrt{\frac{n}{\underline{\lambda}^2}} \big\|
\bar{M}^{(1)}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), g_0) \big\|^2_\infty
\geq
\delta_{n,m} \right\} & \leq \rho_{n,m}~,\label{eq: concentration}\\
\sup_{P\in\mathbf{P}}
P\left\{ \sqrt{\frac{n}{\underline{\lambda}^2}} \|\hat{g}_n - g_0 \|_{2,\infty}^2 \geq \delta^2_{n,g} \right\} \leq \rho_{n,g}~,
\quad\text{and}\label{eq: g rate}\\
\sup_{P\in\mathbf{P}}
P\left\{ \sqrt{\frac{n}{\underline{\lambda}^2}} \|\hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty^2 \geq \delta^2_{n,\theta} \right\} \leq \rho_{n,\theta}~.\label{eq: theta rate}
\end{align}
\noindent \textbf{(iii)} There exist sequences $\delta_{n,S}$, $\delta_{n,J}$, $\rho_{n,S}$, and $\rho_{n,J}$ such that, uniformly over $P$ in $\mathbf{P}$,
\begin{align}
\sqrt{\frac{n}{\underline{\lambda}^2}} \big\|
\bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), \hat{g}_n)
-
\bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), g_0) \big\|_\infty
\geq
\delta_{n,S}
\end{align}
with probability less than $\rho_{n,S}$ and
\begin{align}
\sqrt{\frac{n}{\underline{\lambda}^2}} \big\|
\bar{M}^{(1)}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), \hat{g}_n)
-
\bar{M}^{(1)}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), g_0) \big\|_\infty
\geq
\delta_{n,J}~,\label{eq: Jacobian se}
\end{align}
with probability less than $\rho_{n,J}$, respectively.
\end{assumption}
The following theorem gives a non-asymptotic bound on the error in the nominal coverage probability of the confidence regions introduced in \cref{def: uniform ci}.
\begin{theorem}\label{thm: generic decomposition} Collect the error sequences
\begin{align*}
\delta_n
&= \delta^2_{n,g} + \delta^2_{n,\theta} + \delta^2_{n,m}
+ \delta_{n,B} + \delta_{n,S} + \delta_{n,u}
+ \underline{\lambda}^{1/2}n^{-1/4} \delta_{n,\theta}\left(\delta_{n,B} + \delta_{n,J}\right)
\end{align*}
and $\rho_n= \rho_{n,m} + \rho_{n,g} + \rho_{n,\theta} + \rho_{n,S} + \rho_{n,J} + \rho_{n,u}$ and assume that $\delta_{\varepsilon n} \leq C_\varepsilon \delta_{n}$ and $\rho_{\varepsilon n} \leq C_\varepsilon \rho_{n}$ for any $0<\varepsilon<1$. Suppose that the Neyman orthogonal moment function $M(x;\theta_0, g_0)$ satisfies \cref{assu: app a moment function} and Part (i) of \cref{assu: moment smoothness} and that the centered empirical moment function $\bar{M}_n(x;\theta_0, g_0)$ satisfies \cref{assu: empirical smoothness}. If \cref{assu: linearity} and \cref{assu: bias} hold, then the confidence region defined in \cref{def: uniform ci} satisfies
\begin{equation} \label{eq: asymptotic validity statement}
\sup_{P\in\mathbf{P}}
\big\vert P\left\{ \theta_0(\bm{x}^{(d)}) \in \hat{\mathcal{C}}(\bm{x}^{(d)}) \right\} - (1 - \alpha)\big\vert
\lesssim \left( \frac{\varphi^2\log^{5}\left(dn\right)}{\underline{\lambda}^{2}n}\right)^{1/4} + \delta_n\sqrt{\log{d}}+ \rho_n~.
\end{equation}
\end{theorem}
\section{Outline of Proof of \cref{eq: asymptotic validity subsampled kernel}\label{app: kernel verify outline}}
The result follows from an application of \cref{thm: generic decomposition}. We outline the proof in this Appendix. The detailed proof is given in \cref{app: kernel verify pf}. First observe that the simplifying \cref{assu: moment linearity} implies that \cref{assu: empirical smoothness} holds with
\begin{equation}
\bar{M}_n^{(1)}(x; \theta,g) = \sum_{i=1}^n \left( K(x,X_i)m^{(1)}(D_i; \theta, g) - \mathbb{E}\left[K(x,X_i) m^{(1)}(D_i; \theta, g)\right]\right)\label{eq: m bar b app}
\end{equation}
and $\bar{H}_n(x; \theta,g) = 0$, respectively.
The main challenge in proving \cref{thm: generic decomposition} is to quantify the various generic sequences defined in \cref{assu: linearity} and \cref{assu: bias}.
To quantify the generic sequences $\delta_n^{u}$ and $\rho_n^{u}$ defined in \cref{assu: linearity}, we consider the quantity
\begin{align}
U_{n}(x) & = - \frac{1}{r} \sum_{q=1}^r u(x; D_{\mathsf{s}_q}, \xi_{\mathsf{s}}, \theta_0, g_0)~,\quad\text{where}\label{eq: u b app}\\
u(x; D_\mathsf{s}, \xi_{\mathsf{s}}, \theta, g)
&= M^{(1)}(x;g_0)^{-1}\sum_{i\in\mathsf{s}}
\big( \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) m(D_i; \theta, g)
\nonumber\\
&\quad\quad\quad\quad\quad\quad\quad\quad- \mathbb{E}\big[ \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) m(D_i; \theta, g)\big]\big)~.\nonumber
\end{align}
In particular, we apply \cref{lem: projection maximal} to show that the, suitably scaled, difference between \eqref{eq: u b app} and its H\'{a}jek projection is small with high probability. To quantify the various sequences introduced in \cref{assu: bias}, we apply arguments similar to those used in \cite{wager2018estimation} and \cite{oprescu2019orthogonal}. The novelty is that each bound is uniform over the query-vector $\bm{x}^{(d)}$.
\section{Proof of \cref{lem: projection maximal}\label{app: u stat}}
We are interested in studying the quantity
\begin{equation}\label{eq: Hajek residual}
\bar{U}_{n,b}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i)~.
\end{equation}
Observe that
\begin{align}
\frac{b}{n} \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i)
& = {n-1 \choose b-1} {n \choose b}^{-1} \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i) \nonumber \\
& = \frac{1}{N_b} \sum_{s\in\mathcal{S}_{n,b}} \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i) \mathbb{I}\{ i \in \mathsf{s}\}
\end{align}
and that consequently the difference \eqref{eq: Hajek residual} can be written
\begin{align}
\frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
\left( \tilde{u}(\bm{x}^{(d)}; D_{\mathsf{s}})
- \sum_{i\in\mathsf{s}} \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i)\right)~.\label{eq: as u stat}
\end{align}
In other words, the difference \eqref{eq: Hajek residual} can be re-expressed as a scaled, complete, $U$-statistic of order $b$ with the kernel function
\begin{equation}\label{eq: Hajek residual kernel}
h(\bm{x}^{(d)}; D_{\mathsf{s}}) = \tilde{u}(\bm{x}^{(d)}; D_{\mathsf{s}}) - \sum_{i\in\mathsf{s}} \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i)~.
\end{equation}
Moreover, the kernel function \eqref{eq: Hajek residual kernel} is completely degenerate, in the standard sense that
\begin{equation}
\mathbb{E}\left[h(\bm{x}^{(d)}; D_{\mathsf{s}}) \mid i \in \mathsf{s}, D_i\right] = 0
\end{equation}
almost surely.
To give a high-probability bound on the difference \eqref{eq: as u stat}, we construct a bound on the higher-order moment
\begin{equation*}
\mathbb{E}\left[ \bigg\vert \frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
h(x; D_{\mathsf{s}}) \bigg\vert^q \right]
\end{equation*}
for an arbitrary $q\geq2$, where $x$ is an arbitrary element of the query-vector $\bm{x}^{(d)}$. We express this problem more tractably through a symmetrization argument. In particular, we apply the following symmetrization inequality for completely degenerate functions, due to \cite{sherman1994maximal}. See Theorem 5.2 of \cite{song2019approximating} for an expedited proof.
\begin{lemma}[\cite{sherman1994maximal}]\label{lem: degenerate symmetrization}
Let $Z_1,\ldots,Z_n$ denote a collection of independent and identically distributed real-valued random variables. Consider a real-valued symmetric kernel function $f$ of order $b$ that satisfies
\begin{equation}
\mathbb{E}\left[ f(Z_1,\ldots,Z_b) \mid Z_1 \right] = 0
\end{equation}
almost surely. Let $V_1,\ldots,V_n$ denote an independent collection of Rademacher random variables. If $\Phi(\cdot)$ is any convex function on $[0,\infty)$, then the symmetrization inequality
\begin{equation}
\mathbb{E} \left[ \Phi\left( \bigg\vert \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} f(Z_{\mathsf{s}}) \bigg\vert \right) \right]
\leq
\mathbb{E} \left[ \Phi\left( \bigg\vert 2^b \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} V_{\mathsf{s}} f(Z_{\mathsf{s}}) \bigg\vert \right) \right]
\end{equation}
holds, where $V_{\mathsf{s}} = \Pi_{i\in\mathsf{s}} V_i$ for each subset $\mathsf{s}$ in $[n]$.
\end{lemma}
\noindent We note that \cref{lem: degenerate symmetrization} was also used in \cite{song2019approximating} and \cite{minsker2023u}. We differ in the object that we are applying \cref{lem: degenerate symmetrization} to. Both \cite{song2019approximating} and \cite{minsker2023u}, decompose the $U$-statistic $\bar{U}_{n,b}(\bm{x}^{(d)})$ into a collection of degenerate $U$-statistics with a Hoeffding expansion, and apply \cref{lem: degenerate symmetrization} to each component of this decomposition. Our main insight is that \cref{lem: degenerate symmetrization} can be applied to the $U$-statistic \eqref{eq: as u stat} directly, as the kernel function \eqref{eq: Hajek residual kernel} is completely degenerate.
As in \cite{song2019approximating} and \cite{minsker2023u}, the application of \cref{lem: degenerate symmetrization} is facilitated by the following moment bound for higher moments of Rademacher chaos, often referred to as the Bonami inequality.
\begin{lemma}[Theorem 3.2.2, \cite{de2012decoupling}]\label{lem: chaos moment bound} Fix a collection of real-valued quantities $\{ z_\mathsf{s} : \mathsf{s} \in \mathcal{S}_{n,b} \}$ and let $V_1,\ldots,V_n$ denote an independent collection of Rademacher random variables. Consider the homogeneous Rademacher chaos of order $b$, given by
\begin{equation}
Z_{b} = \sum_{\mathsf{s} \in \mathcal{S}_{n,b}} V_\mathsf{s} z_{\mathsf{s}}~,
\end{equation}
where $V_{\mathsf{s}} = \Pi_{i\in\mathsf{s}} V_i$ for each subset $\mathsf{s}$ in $[n]$. The moment inequality
\begin{equation}
\mathbb{E} \left[ \vert Z_b \vert^q \right] \leq q^{bq/2} (\Delta_b)^{q/2}~,
\quad\text{where}\quad
\Delta_b = \sum_{s\in\mathcal{S}_{n,b}} (z_{\mathsf{s}})^2~,
\end{equation}
holds for every $q>2$.
\end{lemma}
\noindent \cref{lem: chaos moment bound} implies that
\begin{align}
\mathbb{E}\left[ \bigg\vert \frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
V_\mathsf{s} h(x; D_{\mathsf{s}}) \bigg\vert^q \mid \mathbf{D}_n\right]
\leq
q^{bq/2}
\left(
\frac{1}{N_b}
\sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
{n\choose b}^{-1} \left(h(x; D_{\mathsf{s}})\right)^2
\right)^{q/2}~.
\end{align}
Consequently, \cref{lem: degenerate symmetrization} implies that
\begin{align}
\mathbb{E}\left[
\bigg\vert \frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
h(x; D_{\mathsf{s}}) \bigg\vert^q \right]
& \leq 2^{bq} \mathbb{E} \left[ \mathbb{E}\left[
\bigg\vert \frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
V_\mathsf{s} h(x; D_{\mathsf{s}}) \bigg\vert^q
\mid \mathsf{D}_n \right] \right] \nonumber\\
& \leq 2^{bq} q^{bq/2} \mathbb{E} \left[
\left(
\frac{1}{N_b}
\sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
{n\choose b}^{-1} \left(h(x; D_{\mathsf{s}})\right)^2
\right)^{q/2} \right]~.\label{eq: after symm}
\end{align}
To simplify the expression \eqref{eq: after symm}, we apply the following representation of complete $U$-statistics, due to \cite{hoeffding1948class}. To express this result, we require some additional notation. Let $\mathcal{P}_n$ denote the set of permutations of $[n]$, treating each permutation $\pi$ in $\mathcal{P}_n$ as a bijection from $[n]$ to $[n]$. For each permutation $\pi$, define the set
\begin{equation}
\mathsf{s}_{\pi,l} = \{ \pi((l-1)b),\ldots, \pi(lb) \}~.
\end{equation}
Observe that if $n$ is divisible by $b$, the collection $\mathsf{s}_{\pi,1},\ldots,\mathsf{s}_{\pi,n/b}$ is a mutually exclusive partition of the set $[n]$ for each permutation $\pi$.
\begin{lemma}[\cite{hoeffding1948class}]\label{lem: Hoef rep}
The complete $U$-statistic of order $b$ with kernel function $u(\cdot)$ admits the alternative representations
\begin{equation*}
\frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}} u(D_\mathsf{s})
= \frac{1}{n!} \sum_{\pi \in \mathcal{P}} \bigg\lfloor \frac{b}{n} \bigg\rfloor \sum_{l = 1}^{\lfloor n/b \rfloor} u(D_{\mathsf{s}_{\pi,l}})~,
\end{equation*}
where $\lfloor x \rfloor$ denotes the largest integer smaller than or equal to $x$.
\end{lemma}
\noindent
In particular, by \cref{lem: Hoef rep}, Jensen's inequality, and the bound
\[
\left(\frac{n}{b}\right)^b \leq {n \choose b}
\]
we have that
\begin{align}
& 2^{bq} q^{bq/2} \mathbb{E} \left[
\left(
\frac{1}{N_b}
\sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
{n\choose b}^{-1} \left(h(x; D_{\mathsf{s}})\right)^2
\right)^{q/2}
\right] \nonumber \\
& \quad = 2^{bq} q^{bq/2} \mathbb{E} \left[
\left(
\frac{1}{n!}
\sum_{\pi \in \mathcal{P}_n}
\bigg\lfloor \frac{n}{b}\bigg\rfloor^{-1}
\sum_{l=1}^{\lfloor n/b\rfloor}
{n\choose b}^{-1} \left(h(x; D_{\mathsf{s_{\pi,l}}})\right)^2
\right)^{q/2}
\right] \nonumber \\
& \quad \leq 2^{bq} q^{bq/2} \mathbb{E} \left[
\left(
\bigg\lfloor \frac{n}{b}\bigg\rfloor^{-1} \sum_{l=1}^{\lfloor n/b \rfloor}
\left(\frac{b}{n}\right)^{b} \left(h(x; D_{\mathsf{s_{\pi,l}}})\right)^2
\right)^{q/2}
\right]~, \label{eq: pre rosenthal}
\end{align}
where $\pi$ is an arbitrary element of $\mathcal{P}_n$. We note that the summands in \eqref{eq: pre rosenthal} are now independent and identically distributed.
To bound the expectation \eqref{eq: pre rosenthal}, we apply the following version of Rosenthal's inequality for non-negative random variables.
\begin{lemma}[Theorem 15.10, \cite{boucheron2013concentration}]\label{lem: Rosenthal}
Let $Z_1,\ldots,Z_n$ denote a collection of independent real-valued and non-negative random variables. For all $q\geq 1$, the moment inequality
\[
\mathbb{E}\left[ \big\vert \sum_{i=1}^n Z_i\big\vert^q \right]^{1/q}
\lesssim
\mathbb{E}\left[ \sum_{i=1}^n Z_i \right]
+ q \mathbb{E}\left[ \max_{i \in [n]} \vert Z_i\vert^q \right]^{1/q}
\]
holds.
\end{lemma}
\noindent In particular, we find that
\begin{align}
& 2^{bq} q^{bq/2} \mathbb{E} \left[
\left(
\bigg\lfloor \frac{n}{b}\bigg\rfloor^{-1} \sum_{l=1}^{\lfloor n/b \rfloor}
\left(\frac{b}{n}\right)^{b} \left(h(x; D_{\mathsf{s_{\pi,l}}})\right)^2
\right)^{q/2}
\right] \nonumber \\
& \quad \leq C^{bq} q^{bq/2}
\Bigg(
\mathbb{E} \left[
\bigg\lfloor \frac{n}{b}\bigg\rfloor^{-1} \sum_{l=1}^{\lfloor n/b \rfloor}
\left(\frac{b}{n}\right)^{b} \left(h(x; D_{\mathsf{s_{\pi,l}}})\right)^2
\right]
\nonumber\\
&\quad \quad\quad \quad\quad \quad\quad
+\left(\frac{q}{2}\right) \mathbb{E} \left[
\max_{l \in [n/b]}
\left(
\left(\frac{b}{n}\right)^{b+1} \left(h(x; D_{\mathsf{s_{\pi,l}}})\right)^2
\right)^{q/2}
\right]^{2/q}
\Bigg)^{q/2} \nonumber \\
& \quad \lesssim C^{bq} q^{bq/2} \left(\frac{b}{n}\right)^{bq/2}
\Bigg(
\mathbb{E} \left[
\left(h(x; D_{\mathsf{s}})\right)^2
\right]^{q/2}\nonumber\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad
+q^{q/2} \left(\frac{b}{n}\right)^{q/2}
\mathbb{E} \left[
\max_{l \in [n/b]}
\left( h(x; D_{\mathsf{s_{\pi,l}}}) \right)^q
\right]
\Bigg)~.\label{eq: square and q moment}
\end{align}
where the first inequality follows from \cref{lem: Rosenthal} and the second inequality follows from the inequality $(x+y)^n \leq 2^n(x^n + y^ n)$.
It remains to bound the two moments in \eqref{eq: square and q moment}. To bound the variance term, we re-express the symmetric statistic $\tilde{u}(x; D_{\mathsf{s_{\pi,l}}})$ in terms of its Hoeffding expansion. That is, we apply the following result from \cite{efron1981jackknife}).
\begin{lemma}\label{lem: Hoeffding expansion}
Let $Z_1,\ldots,Z_b$ be a collection of $b$ independent and identically distributed real-valued random variables and $f:\mathbb{R}^b \to \mathbb{R}$ denote some symmetric function satisfying $\Var(f(Z_{[b]}))<\infty$. There exist functions $f_1,\ldots,f_b$ such that
\begin{equation}\label{eq: Hoeffding expansion}
f(Z_1,\ldots,Z_b)
= \mathbb{E}[f(Z_1,\ldots,Z_b)] + \sum_{l = 1}^b \sum_{\mathsf{s}\in\mathcal{S}_{b,l}} f_l(Z_{\mathsf{s}})~,
\end{equation}
and all $2^{b} - 1$ random terms on the right-hand side of \eqref{eq: Hoeffding expansion} are mean-zero and uncorrelated. Moreover, the function $f_1(\cdot)$ is given by the H\'{a}jek projection
$f_1(z) = \mathbb{E}[f(Z_1,\ldots,Z_n)\mid Z_1 = z]$.
\end{lemma}
\noindent In particular, \cref{lem: Hoeffding expansion} implies that there exist functions $\tilde{u}^{(1)},\ldots,\tilde{u}^{(b)}$ such that
\begin{equation}\label{eq: apply hoeff to u}
\tilde{u}(x; D_{\mathsf{s_{[b]}}}) = \sum_{l=1}^b \sum_{s \in \mathsf{s}_{b,l}} \tilde{u}^{(l)}(x; D_\mathsf{s})~,
\end{equation}
where all $2^{b} - 1$ random terms on the right-hand side of \eqref{eq: apply hoeff to u} are mean-zero and uncorrelated. Consequently, we find that
\begin{align}
\Var\left( h(x^{(j)}; D_{\mathsf{s_{[b]}}}) \right)
& = \Var\left( \tilde{u}(x; D_{\mathsf{s_{[b]}}}) - \sum_{i=1}^b \tilde{u}^{(1)}(x; D_{\mathsf{s_{[b]}}})\right) \nonumber \\
&= \Var\left( \sum_{l=2}^b \sum_{s \in \mathsf{s}_{b,l}} \tilde{u}^{(l)}(x; D_\mathsf{s}) \right) \nonumber\\
&\leq \Var\left( \sum_{l=1}^b \sum_{s \in \mathsf{s}_{b,l}} \tilde{u}^{(l)}(x; D_\mathsf{s}) \right) = \Var(\tilde{u}(x; D_{\mathsf{s_{[b]}}})) = \bar{\nu}^2\label{eq: phi variance bound}
\end{align}
To bound the higher order moment in \eqref{eq: square and q moment}, we apply the following standard maximal inequality, which follows from a simple application of e.g., Lemma 5.5, \cite{song2019approximating}.
\begin{lemma}\label{lem: moment maximal}
Let $Z_1, \ldots, Z_k$ denote a collection of centered real-valued random variables. If $\|Z_j\|_{\psi_1} \leq \varphi$ for all $j$ in $[k]$, then
\begin{equation}
\mathbb{E}\left[ \max_{j \in [k]} \vert Z_j \vert^q \right] \lesssim (2q\varphi \log(2k))^q~.
\end{equation}
\end{lemma}
\noindent The bound $\| h(x; D_{\mathsf{s_{[b]}}}) \|_{\psi_1} \leq (b+1)\phi$, follows from the triangle inequality and the fact that $\|\tilde{u}^{(1)}(x; D_i))\|_{\psi_1} \leq \phi$ by Jensen's inequality. Thus, \cref{lem: moment maximal} implies that
\begin{equation}
\mathbb{E} \left[
\max_{l \in [n/b]}
\left( h(x; D_{\mathsf{s_{\pi,l}}}) \right)^q
\right] \lesssim \left(4 q b \phi \log(2n)\right)^q~. \label{eq: phi q bound}
\end{equation}
Putting the pieces together, the bounds \eqref{eq: phi variance bound} and \eqref{eq: phi q bound} imply that
\begin{align}
&\mathbb{E}\left[ \bigg\vert \frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
h(x; D_{\mathsf{s}}) \bigg\vert^q \right] \nonumber \\
& \quad\leq C^{bq} q^{bq/2} \left(\frac{b}{n}\right)^{bq/2}
\Bigg(
\mathbb{E} \left[
\left(h(x; D_{\mathsf{s}})\right)^2
\right]^{q/2}
+q^{q/2} \left(\frac{b}{n}\right)^{q/2}
\mathbb{E} \left[
\max_{l \in [n/b]}
\left( h(x; D_{\mathsf{s_{\pi,l}}}) \right)^q
\right]
\Bigg) \nonumber \\
& \quad \leq
C^{bq} q^{bq/2} \left(\frac{b}{n}\right)^{bq/2}
\Bigg(\bar{\nu}^{q}
+q^{3q/2} b^q \left(\frac{b}{n}\right)^{q/2}
\phi^q\log^q(n)
\Bigg) \nonumber \\
& \quad
=
\left(C^{b} q^{b/2} \left(\frac{b}{n}\right)^{b/2}
\left(\bar{\nu}
+q^{3/2} b \left(\frac{b}{n}\right)^{1/2}
\phi\log(n)
\right)\right)^q~,
\end{align}
Hence, an application of Markov's inequality and a union bound implies that
\begin{align}
& P\bigg\{
\bigg\| \left(\bar{U}_{n}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i)\right) \bigg\|_\infty \nonumber\\
& \quad\quad
\geq
\frac{1}{e} \left(C q\frac{b}{n}\right)^{b/2}
\left(
\bar{\nu}
+q^{3/2} b \left(\frac{b}{n}\right)^{1/2} \phi\log(n)
\right)
\bigg\} \nonumber\\
& \quad \quad \leq
\frac{d \exp(-q) \mathbb{E}\left[ \bigg\vert \frac{1}{N_b} \sum_{\mathsf{s} \in \mathcal{S}_{n,b}}
h(x^{(j)}; D_{\mathsf{s}}) \bigg\vert^q \right]}
{C^{bq} q^{qb/2} \left(\frac{b}{n}\right)^{qb/2}
\left(\bar{\nu}
+q^{3/2} b \left(\frac{b}{n}\right)^{1/2} \phi\log(n)
\right)^q}
\leq d \exp(-q)~.
\end{align}
Through the choice $q = \log(dn)$, we find that
\begin{align}
& \bigg\| \left(\bar{U}_{n}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i)\right) \bigg\|_\infty \nonumber \\
& \quad \quad \leq
\left(\frac{C b\log(dn)}{n}\right)^{b/2}
\left(\bar{\nu} + b \left(\frac{b}{n}\right)^{1/2} \phi\log^2(dn)\right)
\end{align}
with probability greater than $1 - 1/n$, as required.\hfill\qed
\section{Proof of \cref{thm: generic decomposition}\label{app: sub pf of generic decomposition}}
Proofs for the Lemmas stated in this appendix are given in \cref{app: generic decomposition pf lemma}. Throughout, without loss, we will assume that there exists some constant $c$ such that
\begin{equation}
\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n} + \delta_n \sqrt{\log(dn)} < c < 1~,\label{eq: norm without loss}
\end{equation}
as otherwise the desired bound \eqref{eq: asymptotic validity statement} is vacuous. It will suffice to show that
\begin{flalign}
& \sup_{z\in\mathbb{R}} \big\vert P\left\{\sqrt{n} \| \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)}) \|_\infty < z \right\}
-
P\left\{\sqrt{n} \| \hat{\Lambda}_n^{-1/2}R^*_{n}(\bm{x}^{(d)}) \|_\infty < z \mid \mathbf{D}_n \right\} \big\vert \nonumber\\
& \quad\quad\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n\sqrt{\log{dn}} + \rho_n \label{eq: diff in cdfs}
\end{flalign}
with probability greater than $1-C(n^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) + \rho_n)$. To see this, let $\mathsf{cv}(\gamma)$ denote the $1-\gamma$ quantile of $\sqrt{n} \| \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)}) \|_\infty$ for each $\gamma$ in $(0,1)$ and fix
\begin{equation}
\beta_{n,d} = \left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4} + \delta_n\sqrt{\log{dn}} + \rho_n ~.
\end{equation}
Observe that \eqref{eq: diff in cdfs} implies that
\begin{align}
& P\left\{ \sqrt{n} \| \hat{\Lambda}_n^{-1/2}R^*_{n}(\bm{x}^{(d)}) \|_\infty < \mathsf{cv}(\alpha-\beta_{n,d}) \mid \mathbf{D}_n \right\} \nonumber \\
& \geq
P\left\{ \sqrt{n} \| \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)}) \|_\infty < \mathsf{cv}(\alpha-\beta_{n,d}) \right\} - \beta_{n,d} \geq 1-\alpha\label{eq: cv perturb lower}
\end{align}
and
\begin{align}
& P\left\{ \sqrt{n} \| \hat{\Lambda}_n^{-1/2}R^*_{n}(\bm{x}^{(d)}) \|_\infty < \mathsf{cv}(\alpha+\beta_{n,d}) \mid \mathbf{D}_n \right\} \nonumber \\
& \leq
P\left\{ \sqrt{n} \| \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)}) \|_\infty < \mathsf{cv}(\alpha+\beta_{n,d}) \right\} + \beta_{n,d} \leq 1-\alpha\label{eq: cv perturb upper}
\end{align}
each with probability greater than $1-C(n^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) + \rho_n)$. Thus, recalling that $\hat{\mathsf{cv}}(\alpha)$ denotes the $1-\alpha$ quantile of $\sqrt{n} \| \hat{\Lambda}_n^{-1/2}R^*_{n}(\bm{x}^{(d)}) \|_\infty$ conditioned on the data $\mathbf{D}_n$, the inequalities \eqref{eq: cv perturb lower} and \eqref{eq: cv perturb upper} imply that
\begin{align}
& P\left\{\mathsf{cv}(\alpha+\beta_{n,d}) < \hat{\mathsf{cv}}(\alpha) < \mathsf{cv}(\alpha-\beta_{n,d})\right\} \gtrsim 1- \left(\frac{\varphi^2\log^{3}(dn)}{\underline{\lambda}^2n}\right)^{1/2} - \rho_n \geq 1- \beta_{n,d}~,\label{eq: cv sandwich}
\end{align}
where the second inequality follows from the fact that the normalization \eqref{eq: norm without loss} implies that
\begin{equation}
\left(\frac{\varphi^2\log^{3}(dn)}{\underline{\lambda}^2n}\right)^{1/2}
= \left(\frac{\varphi^2\log(dn)}{\underline{\lambda}^2n}\right)^{1/4} \left(\frac{\varphi^2\log^{5}(dn)}{\underline{\lambda}^2n}\right)^{1/4}
\lesssim
\left(\frac{\varphi^2\log^{5}(dn)}{\underline{\lambda}^2n}\right)^{1/4}~.
\end{equation}
Consequently, as we can write
\begin{align}
P\left\{ \bm{\theta}_0(\bm{x}^{(d)}) \in \hat{\mathcal{C}}(\bm{x}^{(d)}, \mathbf{D}_n) \right\}
=
P\left\{ \sqrt{n} \| \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)}) \|_\infty \leq \hat{\mathsf{cv}}(\alpha) \right\}
\end{align}
by definition, the inequality \eqref{eq: cv sandwich} implies that
\begin{align}
& P\left\{ \bm{\theta}_0(\bm{x}^{(d)}) \in \hat{\mathcal{C}}(\bm{x}^{(d)}, \mathbf{D}_n) \right\} \nonumber \\
& \leq
P\left\{ \sqrt{n} \| \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)}) \|_\infty \leq \mathsf{cv}(\alpha-\beta_{n,d}) \right\} + \beta_{n,d}
\lesssim 1- \alpha + \beta_{n,d}
\end{align}
and
\begin{align}
& P\left\{ \bm{\theta}_0(\bm{x}^{(d)}) \in \hat{\mathcal{C}}(\bm{x}^{(d)}, \mathbf{D}_n) \right\} \nonumber \\
& \geq
P\left\{ \sqrt{n} \| \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)}) \|_\infty \geq \mathsf{cv}(\alpha-\beta_{n,d}) \right\} -\beta_{n,d}
\gtrsim 1- \alpha - \beta_{n,d}
\end{align}
respectively, as required.
Hence, the remainder of the proof is devoted to establishing the probability bound \eqref{eq: diff in cdfs}. To do this, we require some additional notation. Let $\mathcal{R}$ denote the set of hyper-rectangles in $\mathbb{R}^d$. Let $Z$ denote a centered Gaussian random vector with covariance matrix $\Var(\bar{u}(\bm{x}^{(d)}, D_i))$. Let $\Lambda$ be the diagonal matrix with components $\lambda_{j}^2 = \Var(\bar{u}(x^{(j)}, D_i))$. To verify \eqref{eq: diff in cdfs} it will suffice to establish that
\begin{flalign}
&\sup_{\mathsf{R} \in \mathcal{R}} \big\vert P\left\{\sqrt{n} \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)})\in\mathsf{R}\right\}
-P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}\right\} \big\vert \nonumber \\
&\quad\quad \lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n\sqrt{\log{dn}} + \rho_n \label{eq: A abs to prove}
\end{flalign}
and that
\begin{flalign}
&\sup_{\mathsf{R} \in \mathcal{R}} \big\vert P\left\{\sqrt{n} \hat{\Lambda}_n^{-1/2}R^*_{n}(\bm{x}^{(d)})\in\mathsf{R} \mid \mathbf{D}_n \right\}
-P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}\right\} \big\vert \nonumber \\
&\quad\quad \lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n\sqrt{\log{dn}} \label{eq: A* abs to prove}
\end{flalign}
with probability greater than $1-Cn^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) - \rho_n$. That is, the probability bound \eqref{eq: diff in cdfs} follows from \eqref{eq: A abs to prove} and \eqref{eq: A* abs to prove} by considering hyper-rectangles of the form $\mathsf{R} = [-\infty \bm{1}_{d}, z \bm{1}_{d} ]$ and applying the triangle inequality.
For the sake of exposition, we provide the details for the proofs of the upper bounds encoded in the absolute inequalities \eqref{eq: A abs to prove} and \eqref{eq: A* abs to prove}, respectively. In particular, fix a rectangle $\mathsf{R}=[a_{l},a_{u}]$ in $\mathcal{R}$,
where $a_{l}$ and $a_{u}$ are vectors in $\mathbb{R}^{d}$ with $a_{l}\leq a_{u}$, interpreted componentwise, and define the enlarged rectangle $\mathsf{R}_{t}=[a_{l}-t\bm{1}_{d},a_{u}+t\bm{1}_{d}]$ for each $t>0$. We obtain upper bounds
\begin{flalign}
& P\left\{\sqrt{n} \hat{\Lambda}_n^{-1/2}R_{n}(\bm{x}^{(d)})\in\mathsf{R}\right\}
-P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}\right\} \nonumber\\
& \quad\quad\leq
\big\vert P\left\{ \sqrt{n} \Lambda^{-1/2}R_{n}(\bm{x}^{(d)})\in\mathsf{R}_{t}\right\}
-P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}_{t}\right\} \big\vert\label{eq: m estimator clt to bound}\\
& \quad\quad\quad
+\big\vert P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}_{t}\right\}
-P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}\right\} \big\vert\label{eq: gaussian diff 1}\\
& \quad\quad\quad
+P\left\{ \sqrt{n} \|(\Lambda^{-1/2}-\hat{\Lambda}_{n}^{-1/2})R_{n}(\bm{x}^{(d)})\|_{\infty}\geq t\right\} ,\label{eq: variance estimation to bound}
\end{flalign}
and similarly
\begin{flalign}
& P\left\{ \sqrt{n} \hat{\Lambda}_n^{-1/2}R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\}
- P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}\right\} \nonumber\\
& \quad\quad\leq\vert
P\left\{ \sqrt{n} \Lambda^{-1/2}R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}_{t}\mid\mathbf{D}_{n}\right\}
- P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}_{t}\right\} \vert\label{eq: boot m estimator to bound}\\
& \quad\quad\quad+\vert
P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}_{t}\right\}
- P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}\right\} \vert\label{eq: gaussian diff 2}\\
& \quad\quad\quad
+P\left\{ \sqrt{n} \|(\Lambda^{-1/2}-\hat{\Lambda}_{n}^{-1/2})R_{n}^{*}(\bm{x}^{(d)})\|_{\infty}\geq t\mid\mathbf{D}_{n}\right\}~,\label{eq: variance boot estimation to bound}
\end{flalign}
respectively. Analogous lower bounds will follow from the same argument, with the enlarged rectangle $\mathsf{R}_{t}$ replaced with the shrunken rectangle $[a_{l}+t\bm{1}_{d},a_{u}-t\bm{1}_{d}]$. It thereby remains to obtain suitable bounds for each term, \eqref{eq: m estimator clt to bound} through \eqref{eq: variance boot estimation to bound}.
To bound the Gaussian approximation errors \eqref{eq: m estimator clt to bound}
and \eqref{eq: boot m estimator to bound}, we apply the following
Theorem, which establishes a generic quantitative central limit for the statistic $R_{n}$ in addition to a generic quantitative conditional central limit theorem for the half-sample bootstrap.
\begin{theorem}
\label{thm: generic m estimation unstudentized}Suppose that the moment function $M(x;\theta_0, g_0)$ satisfies \cref{assu: app a moment function} and Part (i) of \cref{assu: moment smoothness} and that \cref{assu: linearity,assu: bias} hold.
\noindent
\textbf{(i)} The inequality
\begin{flalign}
\sup_{R\in\mathcal{R}}\sup_{P\in\mathbf{P}}
\Big\vert P\left\{ \sqrt{n} R_{n}(\bm{x}^{(d)})\in\mathsf{R}\right\} -P\left\{ Z\in\mathsf{R}\right\} \Big\vert
& \lesssim \frac{\varphi^{1/2}}{\underline{\lambda}^{1/2}} \left(\frac{\log^{5}(dn)}{n}\right)^{1/4} + \delta_n\sqrt{\log{d}} + \rho_n\label{eq: generic estimator clt}
\end{flalign}
holds.
\noindent \textbf{(ii)} If the bootstrap root is constructed with the Half-Sample bootstrap, then the inequality
\begin{flalign}
\sup_{R\in\mathcal{R}}\sup_{P\in\mathbf{P}}
\Big\vert P\left\{ \sqrt{n} R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\} -P\left\{ Z\in\mathsf{R}\right\} \Big\vert
& \lesssim \frac{\varphi^{1/2}}{\underline{\lambda}^{1/2}}\left(\frac{\log^{5}\left(dn\right)}{n}\right)^{1/4}+ \delta_n\sqrt{\log{d}} \label{eq: generic bootstrap clt}
\end{flalign}
holds with probability greater than $1-C(n^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) + \rho_n)$.
\end{theorem}
\noindent In particular, \cref{thm: generic m estimation unstudentized} implies that
\begin{align}
& \big\vert P\left\{ \sqrt{n} \Lambda^{-1/2}R_{n}(\bm{x}^{(d)})\in\mathsf{R}_{t}\right\}
-P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}_{t}\right\} \big\vert \nonumber \\
&=
\big\vert P\left\{ \sqrt{n} R_{n}(\bm{x}^{(d)})\in \Lambda^{1/2} \mathsf{R}_{t}\right\}
-P\left\{ Z\in \Lambda^{1/2} \mathsf{R}_{t}\right\} \big\vert \nonumber \\
&\quad\quad \lesssim \frac{\varphi^{1/2}}{\underline{\lambda}^{1/2}} \left(\frac{\log^{5}(dn)}{n}\right)^{1/4} + \delta_n\sqrt{\log{d}} + \rho_n \label{eq: generic estimator clt apply }
\end{align}
and analogously
\begin{align}
& \vert P\left\{ \sqrt{n} \Lambda^{-1/2}R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}_{t}\mid\mathbf{D}_{n}\right\}
- P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}_{t}\right\} \vert \nonumber \\
& =
\big\vert P\left\{ \sqrt{n} R^*_{n}(\bm{x}^{(d)})\in \Lambda^{1/2} \mathsf{R}_{t}\mid \mathbf{D}_{n}\right\}
-P\left\{ Z\in \Lambda^{1/2} \mathsf{R}_{t}\right\} \big\vert \nonumber \\
& \quad\quad \lesssim \frac{\varphi^{1/2}}{\underline{\lambda}^{1/2}} \left(\frac{\log^{5}(dn)}{n}\right)^{1/4} + \delta_n\sqrt{\log{d}} + \rho_n \label{eq: generic bootstrap clt apply }
\end{align}
with probability greater than $1-C(n^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) + \rho_n)$, as $\Lambda^{1/2} \mathsf{R}_{t}$ is a hyper-rectangle.
To bound the differences in the Gaussian probabilities (\ref{eq: gaussian diff 1})
and (\ref{eq: gaussian diff 2}), we apply the following anti-concentration
inequality, stated in \citet{chernozhukov2017detailed} and often referred
to as Nazarov's inequality.
\begin{lemma}[{\citealp[Theorem 1, ][]{chernozhukov2017detailed}}]
\label{lem: Nazarov}Let $Z=(Z_{j})_{j=1}^{d}$ be a centered Gaussian
random vector in $\mathbb{R}^{d}$ such that $\mathbb{E}[Z_{j}^{2}]\geq c$
for all $j$ in $[d]$ and some constant $c$. For every $z\in\mathbb{R}^{d}$
and $t>0$, the inequality
\[
P\left\{ Z\leq z+t\right\} -P\left\{ Z\leq z\right\} \lesssim\frac{t}{c}\sqrt{\log d}
\]
holds.
\end{lemma}
\noindent In particular, we have that
\begin{equation}
\vert P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}_{t}\right\} -P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}\right\} \vert\lesssim t\sqrt{\log d}\label{eq: nazarov apply app a}
\end{equation}
for all $t>0$.
Finally, we bound the terms (\ref{eq: variance estimation to bound}) and (\ref{eq: variance boot estimation to bound}) resulting from variance estimation. Observe that
\begin{align}
\|(\Lambda^{-1/2} - \hat{\Lambda}_n^{-1/2}) R_n(\bm{x}^{(d)})\|_{\infty}
&= \|(I - \hat{\Lambda}_n^{-1/2}\Lambda^{1/2}) \Lambda^{-1/2} R_n(\bm{x}^{(d)})\|_{\infty} \nonumber\\
&\leq \sup_{j\in[d]} |\hat{\lambda}_{n,j}/\lambda_j - 1|\cdot \|\Lambda^{-1/2}R_n(\bm{x}^{(d)})\|_{\infty} \label{eq: var perturb in two}
\end{align}
and analogously
\begin{align}
\|(\Lambda^{-1/2} - \hat{\Lambda}_n^{-1/2}) R^*_n(\bm{x}^{(d)})\|_{\infty}
\leq \sup_{j\in[d]} |\hat{\lambda}_{n,j}/\lambda_j - 1|\cdot \|\Lambda^{-1/2}R^*_n(\bm{x}^{(d)})\|_{\infty}~.\label{eq: var boot perturb in two}
\end{align}
We give probability bounds for the terms on the right-hand-sides of \eqref{eq: var perturb in two} and \eqref{eq: var boot perturb in two}. Observe that the Borell-TIS inequality \citep[e.g., Theorem 2.1.1 of][]{adler2009random} implies that
\[
P\left\{ \|\Lambda^{-1/2}Z\|_{\infty}\geq C \sqrt{\log{dn}}\right\} \le n^{-1}~.
\]
Thus, \cref{thm: generic m estimation unstudentized} implies that
\begin{flalign}
P\left\{ \|\Lambda^{-1/2}R_{n}(\bm{x}^{(dn)})\|_{\infty}\geq C \sqrt{\log{dn}}\right\}
& \lesssim \left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4} + \delta_n\sqrt{\log{d}} + \rho_n \label{eq: BTIS estimator}
\end{flalign}
and that
\begin{flalign}
P\left\{ \|\Lambda^{-1/2}R^*_{n}(\bm{x}^{(d)})\|_{\infty}\geq C \sqrt{\log{dn}}\mid\mathbf{D}_{n}\right\}
& \lesssim \left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4} + \delta_n\sqrt{\log{d}}\label{eq: BTIS boot}
\end{flalign}
with probability greater than $1-C(n^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) + \rho_n)$. In turn, to bound the discrepancy between the bootstrap variance estimate $\hat{\lambda}_{n,j}$ and $\lambda_j$, we apply the following Lemma.
\begin{lemma}\label{lem: variance accuracy}Suppose that \cref{assu: linearity} holds. If the bootstrap root is constructed with the Half-Sample bootstrap, then
\begin{equation}
P\left\{\sup_{j\in[d]} \bigg\vert \frac{\hat{\lambda}_{n,j}}{\lambda_{j}} - 1\bigg\vert \geq C \frac{\varphi^2}{\underline{\lambda}^2 n} \log^2(dn) + C\frac{1}{n}\delta_n^2 \right\} \lesssim 1-C(\rho_n+n^{-1})~.
\end{equation}
\end{lemma}
\noindent Thus, by combining the bounds \eqref{eq: var perturb in two} and \eqref{eq: BTIS estimator} with Lemma \eqref{lem: variance accuracy}, we find that
\begin{flalign}
&P\left\{ \|(\Lambda^{-1/2}-\hat{\Lambda}_{n}^{-1/2})R_{n}(\bm{x}^{(d)})\|_{\infty}
\gtrsim \left(\frac{\varphi^2}{\underline{\lambda}^2 n}\log^2(dn) + \frac{1}{n}\delta_n^2\right)\sqrt{\log{dn}}\right\}\nonumber\\
& \quad\quad\quad\quad\quad\quad\quad\quad\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n\sqrt{\log{d}} + \rho_n~.\label{eq: var bound estimator state}
\end{flalign}
Similarly, by combining the bounds \eqref{eq: var boot perturb in two} and \eqref{eq: BTIS boot} with Lemma \eqref{lem: variance accuracy}, we find that
\begin{flalign}
&P\left\{ \|(\Lambda^{-1/2}-\hat{\Lambda}_{n}^{-1/2})R_{n}^{*}(\bm{x}^{(d)})\|_{\infty}
\gtrsim\left(\frac{\varphi^2}{\underline{\lambda}^2 n}\log^2(dn) + \frac{1}{n}\delta_n^2\right)\sqrt{\log{dn}}\mid\mathbf{D}_{n}\right\} \nonumber \\
& \quad\quad\quad\quad\quad\quad\quad\quad\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n\sqrt{\log{d}}\label{eq: var bound boot state}
\end{flalign}
with probability greater than $1-C(n^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) + \rho_n)$.
To put the pieces together, recall the upper bounds \eqref{eq: m estimator clt to bound} and \eqref{eq: boot m estimator to bound}. By setting
\[
t =\left(\frac{\varphi^2}{\underline{\lambda}^2 n}\log^2(dn) + \frac{1}{n}\delta_n^2\right)\sqrt{\log{dn}}~,
\]
the bounds \eqref{eq: generic estimator clt apply }, \eqref{eq: nazarov apply app a}, and \eqref{eq: var bound estimator state} imply that
\begin{flalign}
&
P\left\{ \hat{\Lambda}_{n}^{-1/2}R_{n}(\bm{x}^{(d)})\in\mathsf{R}\right\}
-P\left\{ Z\in\mathsf{R}\right\} \nonumber\\
& \quad\quad\quad\quad\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n \sqrt{\log(dn)}
+ \left(\frac{\varphi^2\log^2(dn)}{\underline{\lambda}^2 n} + \frac{1}{n}\delta_n^2\right)\log(dn)
+ \rho_n \nonumber \\
&\quad\quad\quad\quad\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n \sqrt{\log(dn)} + \rho_n \label{eq: final generic clt}
\end{flalign}
where the second inequality follows from the normalization \eqref{eq: norm without loss}. Similarly, the bounds \eqref{eq: generic bootstrap clt apply }, \eqref{eq: nazarov apply app a}, and \eqref{eq: var bound boot state} imply that
\begin{flalign}
&
P\left\{ \hat{\Lambda}_{n}^{-1/2}R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\}
-P\left\{ Z\in\mathsf{R}\right\} \nonumber\\
& \quad\quad\quad\quad\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+\left(\frac{\varphi^2\log^2(dn)}{\underline{\lambda}^2 n} + \frac{1}{n}\delta_n^2\right)\log(dn) \nonumber \\
& \quad\quad\quad\quad\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n \sqrt{\log(dn)} + \rho_n\label{eq: final generic boot clt}
\end{flalign}
with probability greater than $1-C(n^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) + \rho_n)$. Hence, the upper bounds \eqref{eq: final generic clt} and \eqref{eq: final generic boot clt} verify the upper bounds encoded in \eqref{eq: A abs to prove} and \eqref{eq: A* abs to prove}, respectively. Again, the corresponding lower bounds follow from an analogous argument.
\hfill\qed
\section{Proof of \cref{eq: asymptotic validity subsampled kernel}\label{app: kernel verify pf}}
Proofs for Lemmas stated in this appendix are given in \cref{app: kernel verify pf lemma}. The result follows from an application of \cref{thm: generic decomposition}. We begin by verifying the requisite assumptions. \cref{thm: generic decomposition} requires that the moment function $M(\cdot;\theta_0,g_0)$ is Neyman Orthogonal and satisfies \cref{assu: app a moment function}. Neyman Orthogonality is also imposed in the conditions of \cref{eq: asymptotic validity subsampled kernel}, and so is satisfied. \cref{assu: app a moment function} requires that the moment function $M(x;\theta_0,g_0)$ is twice continuously differentiable and has a Jacobian function
\begin{equation}
M^{(1)}(x; \theta, g)
= \frac{\partial}{\partial \theta^\prime} M(\cdot;\theta_0,g_0) \mid_{\theta^\prime = \theta}
\end{equation}
that satisfies the bounds \eqref{eq: Lipschitz Jacobian} and \eqref{eq: well-posedness}. Twice continuous differentiability
follows immediately from \cref{assu: moment linearity}, which states that the moment function $m(\cdot;\theta,g)$ satisfies the linear representation
\begin{equation}\label{eq: linear in online app}
m(D_i;\theta, g) = m^{(1)}(D_i;g)\cdot \theta + m^{(2)}(D_i;g)~,
\end{equation}
with bounded functions $m^{(1)}(\cdot;g)$ and $m^{(2)}(\cdot;g)$. In particular, the representation \eqref{eq: linear in online app} implies that the Jacobian can be written
\begin{equation}
M^{(1)}(x; \theta, g)
= \mathbb{E}[ m^{(1)}(D_i;g)]
\end{equation}
and that the Hessian can be written
\begin{equation}
H(x; \theta, g)
= \frac{\partial^2}{\partial^2 \theta^\prime} M(\cdot;\theta_0,g_0) \mid_{\theta^\prime = \theta} = 0~.
\end{equation}
Here, our ability to exchange integration and differentiation follows from boundedness of $m(\cdot;\theta, g)$. Consequently, the bounds \eqref{eq: Lipschitz Jacobian} and \eqref{eq: well-posedness} follow from the bounds \eqref{eq: Lipschitz Jacobian main} and \eqref{eq: well-posedness main} stated as Part (iii) of \cref{assu: moment smoothness}.
\cref{thm: generic decomposition} also requires that the centered empirical moment $\bar{M}_n(x; \theta, g)$ satisfy \cref{assu: empirical smoothness}. \cref{assu: empirical smoothness} imposes the restriction that $\bar{M}_n(x; \theta, g)$ is twice continuously differentiable and that its Hessian
\begin{align}
\bar{H}_n(x;\theta, g)&= \frac{\partial^2}{\partial^2 \theta^\prime} \bar{M}_n(x;\theta^\prime, g_0) \vert_{\theta^\prime = \theta}
\end{align}
is uniformly bounded almost surely as $x$, $\theta$, and $g$ vary over their respective domains. Recall that, in our case, we have that
\begin{equation}
\bar{M}_n(x; \theta,g) = \sum_{i=1}^n \left( K(x,X_i)m(D_i; \theta, g) - \mathbb{E}\left[K(x,X_i) m(D_i; \theta, g)\right]\right)~.
\end{equation}
Twice continuously differentiability again follows from \cref{assu: moment linearity}, i.e., the linear representation \eqref{eq: linear in online app}. In turn, we have that $\bar{H}_n(x;\theta, g) = 0$ almost surely, verifying the uniform boundedness condition. Moreover, for future reference, the empirical Jacobian is given by
\begin{equation}
\bar{M}_n^{(1)}(x; g) = \sum_{i=1}^n \left( K(x,X_i)m^{(1)}(D_i; g) - \mathbb{E}\left[K(x,X_i) m^{(1)}(D_i; g)\right]\right)
\end{equation}
again by \cref{assu: moment linearity}.
We now quantify the generic sequences $\delta_n^{u}$ and $\rho_n^{u}$ defined in \cref{assu: linearity}. Recall the normalized statistic $U_n(x)$ defined in \eqref{eq: U def}. By the definition \eqref{eq: subsampled kernel def}, this quantity can be written
\begin{align}
U_{n}(x) & = - \frac{1}{r} \sum_{q=1}^r u(x; D_{\mathsf{s}_q}, \xi_{\mathsf{s}}, \theta_0, g_0)~,\quad\text{where}\nonumber\\
u(x; D_\mathsf{s}, \xi_{\mathsf{s}}, \theta, g)
&= M^{(1)}(x;g_0)^{-1}\sum_{i\in\mathsf{s}}
\big( \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) m(D_i; \theta, g)- \mathbb{E}\big[ \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) m(D_i; \theta, g)\big]\big)~.\nonumber
\end{align}
Define the de-randomized kernel function and H\'{a}jek projection
\begin{align}
\tilde{u}(x; D) &= \mathbb{E}\left[u(x; D_{\mathsf{s}}, \xi_{\mathsf{s}}, \theta_0(x), g_0) \mid D_\mathsf{s} = D \right]\quad\text{and}\label{eq: derandom kernel}\\
\tilde{u}^{(1)}(x; D) &= \mathbb{E}\left[u(x; D_{\mathsf{s}}, \xi_{\mathsf{s}}, \theta_0(x), g_0) \mid i\in\mathsf{s}, D_i = D \right]~,\label{eq: hajek}
\end{align}
respectively, in addition to the quantities
\begin{equation}
\sigma^2_{b,j} = \Var(\tilde{u}(x^{(j)}, D_i))\quad\text{and}\quad \underline{\sigma}^2_b = \min_{j \in [d]} \sigma^2_{b,j}~.
\end{equation}
The result below follows from an application of \cref{lem: projection maximal}.
\begin{lemma}[Asymptotic Linearity]\label{lem: u stat linearity} Suppose that the de-randomized kernel function \eqref{eq: derandom kernel} is bounded by $\phi \geq 1$ almost surely and is invariant to permutations of its second argument. Suppose that $b$ and $r$ are chosen to satisfy $n\leq \sqrt{r} b$. There exists some constant $C_1 < 1$ such that if
\begin{equation}\label{eq: asymp lin conditions}
\frac{(1+\|\theta_0(\bm{x}^{(d)})\|_\infty) \phi b\log(dn)}{n} \leq C_1 \quad\text{and}\quad b^{-C_2} \leq \underline{\sigma}^2_b
\end{equation}
for some positive constant $C_2$, then
\begin{align}\label{eq: asymp lin statement}
\sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}}
\Big\| U_{n}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i) \Big\|_{\infty}
\lesssim \left( \frac{(1+\|\theta_0(\bm{x}^{(d)})\|_\infty)^2 \phi^2 \log(dn)}{n \underline{\sigma}^2_b} \right)^{1/2}
\end{align}
with probability greater than $1 - C/n$ for all $b$ larger than $(7 + C_2) \log(n)$.
\end{lemma}
\noindent In particular, observe that \cref{assu: moment linearity} implies that the de-randomized kernel function \eqref{eq: derandom kernel} is bounded by $\phi$, up to a constant that depends only on $\mathbf{P}$. Permutation invariance again follows by assumption. We may assume that the first condition in \eqref{eq: asymp lin conditions} holds, as otherwise the desired bound is vacuously true. The second condition in \eqref{eq: asymp lin conditions} follows, with $C_2 = 1$, from incrementality. Consequently, by incrementality, the fact that $\|\theta_0(\bm{x}^{(d)})\|_\infty$ is bounded as $P$ varies over $\mathbf{P}$, and the assumption that $b$ greater than $C\log(n)$, we can choose
\begin{equation}
\delta_{n,u} = C \left( \frac{ b \phi^2 \log(dn)}{n} \right)^{1/2}
\end{equation}
and $\rho_{n,u}= Cn^{-1}$, respectively.
Next, we quantify the sequences introduced in \cref{assu: bias}. The Lemma below follows from arguments similar to those used in \cite{wager2018estimation} and \cite{oprescu2019orthogonal}.
\begin{lemma}[Bias, Consistency, and Stochastic Equicontinuity] \label{lem: kernel bcs}
Suppose that the conditions of \cref{eq: asymptotic validity subsampled kernel} are satisfied. Moreover, suppose that there exists some constant $c<1$ such that the inequality
\begin{equation}\label{eq: normalization in online}
\frac{\phi^2 (1 + \| \theta(\bm{x}^{(d)})\|_\infty)^2 b\log^2(dn)}{n} < c
\end{equation}
holds.
\noindent \textbf{(i)} The bound
\begin{align}
& \| \mathsf{Bias}_n(\bm{x}^{(d)};\theta(\bm{x}^{(d)}), \hat{g}_n) \|_\infty \lesssim (1 + \| \theta(\bm{x}^{(d)})\|_\infty) \varepsilon_{b}\label{eq: bias eq}
\end{align}
holds.
\noindent \textbf{(ii)} The bound
\begin{align}
&P \left\{ \| \bar{M}^{(1)}_n(\bm{x}^{(d)}; g_0)\|_\infty \geq C \left(\frac{b \phi^2 \log^2(dn)}{n}\right)^{1/2} \right\} \lesssim \frac{1}{n}\label{eq: concentration eq}
\end{align}
holds for each $b\geq2$.
\noindent \textbf{(iii)} The bounds
\begin{align}
&P\left\{
\big\|
\bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), \hat{g}_n)
-
\bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), g_0) \big\|_\infty
\geq C \tau_{n,S} \right\} \lesssim \frac{1}{n}~\quad\text{and}\\
&P\left\{
\big\|
\bar{M}^{(1)}_n(\bm{x}^{(d)}; \hat{g}_n)
-
\bar{M}^{(1)}_n(\bm{x}^{(d)}; g_0) \big\|_\infty
\geq C \tau_{n,S} \right\} \lesssim \frac{1}{n}~,\quad\text{where}\\
&\tau_{n,S} = \sqrt{\frac{b\log(dn)}{n}} \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} + \varepsilon_b
+ \frac{b}{n}(1 + \|\theta_0(\bm{x}^{(d)})\|_\infty) \phi \log^2(dn)~.\nonumber
\end{align}
hold for each $b\geq2$.
\noindent \textbf{(iv)} If, additionally, there exists some constant $c < 1$ such that
\begin{equation}
\varepsilon_{b}\leq c
\quad\text{and}\quad
\left(\frac{b}{n}\right)^{1/4} \delta_{n,g} \leq c~,\label{eq: part iv normalizations}
\end{equation}
then the bound
\begin{align}
& P\left\{
\|\hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)})\|_\infty
\geq C \tau_{n,\theta}
\right\} \lesssim \frac{1}{n}~,\quad\text{where}\nonumber\\
&\tau_{n,\theta} = \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} + (1 + \| \theta_0(\bm{x}^{(d)})\|_\infty) \varepsilon_{b} + \sqrt{\frac{b}{n}}\log(dn)(1 + \|\theta_0(\bm{x}^{(d)})\|_\infty)\phi\nonumber~,
\end{align}
holds for each $b\geq2$.
\end{lemma}
\noindent Observe that the choice
\[
\bar{u}(x, D_i) = b\cdot \tilde{u}(x, D_i)~,
\]
suggested by \cref{lem: u stat linearity} implies that
\begin{equation}\label{eq: b bounds lambda sq}
b \gtrsim \underline{\lambda}^2 = b^2 \underline{\sigma}^2_b \gtrsim b~,
\end{equation}
where the first inequality is established in the Proof of \cref{thm: Hoef clt} and the second inequality follows from incrementality. We may assume that there exists some small constant $c$ such that
\begin{equation}\label{eq: final normalizations}
\varepsilon_n \leq c,\quad \delta_{n,g} \leq c,\quad\text{and}\quad \frac{b \phi^4\log^5(dn)}{n}\leq c~,
\end{equation}
as otherwise the bound is vacuously true. Consequently, all of the conditions of \cref{lem: kernel bcs} are satisfied. Thus, \cref{lem: kernel bcs} indicates that we can set the various sequences defined in \cref{assu: bias} as
\begin{align}
\delta_{n,b}
& = C \sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}} \varepsilon_{b}
\lesssim \sqrt{\frac{n}{b}} \varepsilon_{b} ~,\\
\delta_{n,m}
& = C \sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}} \frac{b \phi^2 \log^2(dn)}{n}
\lesssim \sqrt{\frac{b}{n}}\phi^2 \log^2(dn) \\
\delta_{n,\theta}
& = C \left(\frac{n}{b^2 \underline{\sigma}^2_b}\right)^{1/4}
\left(\left(\frac{b}{n}\right)^{1/4} \delta_{n,g} + \varepsilon_{b} + \sqrt{\frac{b}{n}} \log(dn)\phi\right)~,\nonumber \\
& \lesssim \delta_{n,g} + \left(\frac{n}{b}\right)^{1/4} \varepsilon_{b} + \left(\frac{b}{n}\right)^{1/4} \log(dn)\phi~, \quad\text{and}\\
\delta_{n,S} = \delta_{n,J} &
=
C \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} \log^{1/2}(dn) + \varepsilon_{b}
+ C \sqrt{\frac{b}{n}}\log^2(dn)\phi ~.
\end{align}
with
\begin{equation}
\rho_{n,m} = \rho_{n,g} = \rho_{n,\theta} = \rho_{n,S} = \rho_{n,J} = C n^{-1}
\end{equation}
respectively, where we have repeatedly used the fact that $\|\theta_0(\bm{x}^{(d)})\|_\infty$ is bounded as $P$ varies over $\mathbf{P}$.
With these results in place, we apply \cref{thm: generic decomposition}. We begin by giving a suitable upper bound for the
sequence
\begin{equation}
\delta_n = \delta^2_{n,g} + \delta^2_{n,\theta} + \delta^2_{n,m}
+ \delta_{n,B} + \delta_{n,S} + \delta_{n,u}
+ \underline{\lambda}^{1/2}n^{-1/4} \delta_{n,\theta}\left(\delta_{n,B} + \delta_{n,J}\right)
\end{equation}
introduced in the statement of \cref{thm: generic decomposition}. Observe that
\begin{align}
\frac{\underline{\lambda}^{1/2}}{n^{1/4}}(\delta_{n,B} + \delta_{n,J}) \delta_{n,\theta}
\lesssim
\frac{b^{1/4}}{n^{1/4}}(\delta_{n,B} + \delta_{n,J}) \delta_{n,\theta}
\lesssim
\delta_{n,B} + \delta_{n,J}~,
\end{align}
as otherwise the desired bound would be vacuous, where the first equality follows from \eqref{eq: b bounds lambda sq}. Consequently, we find that, if $\delta_{n,J} = \delta_{n,S}$, then
\begin{equation}
\delta_n \lesssim \delta^2_{n,g} + \delta^2_{n,\theta} + \delta^2_{n,m} + \delta_{n,B} + \delta_{n,S} + \delta_{n,u}~.
\end{equation}
Plugging in the choices specified above gives
\begin{align}
\delta_{n} & \lesssim \delta^2_{n,g} + \sqrt{\frac{n}{b}} \varepsilon_{b}
+ \sqrt{\frac{b}{n}} \log^{1/2}(dn) \phi + \frac{b}{n} \log^{4}(dn) \phi^4 \label{eq: first delta express}\\
& + \delta^2_{n,g} + \sqrt{\frac{n}{b}} \varepsilon^2_{b} + \sqrt{\frac{b}{n}} \log^2(dn)\phi^2 \nonumber \\
& + \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} \log^{1/2}(dn) + \varepsilon_{b} + \sqrt{\frac{b}{n}}\log^2(dn)\phi
\nonumber
\end{align}
where the first line collects the terms associated with $\delta^2_{n,g}$, $\delta_{n,B}$, $\delta_{n,u}$, and $\delta^2_{n,m}$, the second line collects the term associated with $\delta^2_{n,\theta}$, and the final line collects the term associated with $\delta_{n,S}$. Immediately, we can simplify the bound \eqref{eq: first delta express} to get
\begin{align}
\delta_{n} & \lesssim \delta^2_{n,g} + \sqrt{\frac{n}{b}} \varepsilon_{b}
+ \sqrt{\frac{ b}{n} \log^{4}(dn) \phi^4}
+ \frac{b}{n} \log^{4}(dn) \phi^4 \label{eq: second delta express}\\
& + \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} \log^{1/2}(dn) \nonumber \\
& \lesssim \delta^2_{n,g} + \sqrt{\frac{n}{b}} \varepsilon_{b} + \sqrt{\frac{ b}{n} \log^{4}(dn) \phi^4}
+ \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} \log^{1/2}(dn)~, \nonumber
\end{align}
where, to get the second inequality, we apply the normalization \eqref{eq: final normalizations}. Now, observe that
\begin{align}
\left(\frac{b}{n}\right)^{1/4} \delta_{n,g}\log^{1/2}(dn) & \lesssim \left(\frac{b \log^{2}(dn)}{n}\right)^{1/2} + \delta^2_{n,g}~.
\end{align}
Consequently, we find that
\begin{align}\label{eq: delta final}
\delta_{n} \lesssim \delta^2_{n,g} + \sqrt{\frac{n}{b}} \varepsilon_{b} + \sqrt{\frac{b}{n} \log^{4}(dn) \phi^4}
\end{align}
and that $\rho_{n} = Cn^{-1}$.
Hence, the inequality \eqref{eq: delta final}, the Orlicz-norm bound
\begin{equation*}
\| \bar{u}(x^{(j)}, D) \|_{\psi_1} \leq b\cdot (1 + |\theta(x^{(j)})|) \phi \lesssim b \phi~,
\end{equation*}
and \cref{thm: generic decomposition} imply that
\begin{align}
& \sup_{P\in\mathbf{P}}
\big\vert P\left\{ \theta_0(\bm{x}^{(d)}) \in \hat{\mathcal{C}}(\bm{x}^{(d)}) \right\} - (1 - \alpha)\big\vert \nonumber \\
& \lesssim \left( \frac{b \phi^2 \log^{5}\left(dn\right)}{n}\right)^{1/4}
+ \left(\delta^2_{n,g} + \sqrt{\frac{n}{b}} \varepsilon_{b}\right)\sqrt{\log{d}}
+ \frac{\sqrt{\log(d)}}{n} + \sqrt{\frac{b}{n} \log^{5}(dn) \phi^4} \nonumber\\
& \lesssim \left( \frac{b \phi^4 \log^{5}\left(dn\right)}{n}\right)^{1/4}
+ \left(\delta^2_{n,g} + \sqrt{\frac{n}{b}} \varepsilon_{b}\right)\sqrt{\log{d}} ~,\label{eq: asymptotic validity statement apply}
\end{align}
as required. \hfill\qed
\section{Additional Results and Discussion\label{app: additional}}
\subsection{A Small Survey of Heterogeneity Assessment\label{app: survey}} We conduct a small-scale survey of treatment effect heterogeneity estimation in applied economics. We review the 45 papers published in the \textit{American Economic Review} between January and June of 2023. We consider only the main text of each article.
First, we categorize each paper according to whether it was empirical. Of the empirical papers, we determine whether any of the figures or tables display estimates of treatment effect heterogeneity. (We exclude intertemporal effect heterogeneity, e.g., event-studies). We then categorize each of the papers that display estimates of treatment effect heterogeneity according to whether their report is nonparametric, based on interacted linear regression, based on the interaction of treatment with binary covariates, or involves a structural model.
We categorize 38 papers as empirical. Of these, 30 report treatment effect heterogeneity. Two papers report treatment effect heterogeneity nonparametrically. Nine papers use interacted linear regression. Seven papers use structural modeling. Twelve papers report coefficient on interactions of binary covariates. Many of these papers discretize a continuous covariate into a binary covariate, e.g., age into indicators for age above and below 50.
\subsection{Binomial-Sample Bootstrap\label{app: bin boot}}
Recall that the half-sample bootstrap root is given by
\begin{equation}
R^*_n(\mathbf{x}^{(d)}) = \hat{\theta}_\mathsf{h}(\mathbf{x}^{(d)}) - \hat{\theta}_n(\mathbf{x}^{(d)})
\end{equation}
where $\mathsf{h}$ denotes a random element of $\mathcal{S}_{n,n/2}$ and $\hat{\theta}_n(\mathbf{x}^{(d)})$ denotes a version of the estimator $\hat{\theta}_n(\mathbf{x}^{(d)})$ evaluated on the data $D_\mathsf{h}$. Our theoretical analysis of the half-sample bootstrap is based on the observation that if the estimator $\hat{\theta}_n(\mathbf{x}^{(d)})$ admits a linear representation
\begin{equation}\label{eq: app linear rep}
\hat{\theta}_n(\mathbf{x}^{(d)}) = \frac{1}{n} \sum_{i=1}^n \bar{u}(\mathbf{x}^{(d)}; D_i)
\end{equation}
for some function $\bar{u}(\cdot;\cdot)$, then the root $R^*_n(\mathbf{x}^{(d)})$ admits the representation
\begin{equation}
R^*_n(\mathbf{x}^{(d)}) = \frac{1}{n} \sum_{i = 1}^n V_i (\bar{u}(\mathbf{x}^{(d)}, D_i) - \theta_0(\mathbf{x}^{(d)}))
\end{equation}
where $V_i$ is equal to 1 if $i$ is in $\mathsf{h}$ and is equal to $-1$ otherwise. The weights $V_1, \ldots, V_n$ are exchangeable Rademacher random variables.
In this appendix, we discuss an alternative subsampling procedure that induces analogous weights that are fully independent. That is, for statistics that admit the linear representation \eqref{eq: app linear rep}, the sampling procedure considered here is equivalent to the Rademacher bootstrap. We refer to this procedure as the ``Binomial-Sample" bootstrap. The Binomial-Sample bootstrap root is given by
\begin{equation}
R^*_n(\mathbf{x}^{(d)}) = \frac{2Q_n}{n}( \hat{\theta}_\mathsf{s}(\mathbf{x}^{(d)}) - \hat{\theta}_n(\mathbf{x}^{(d)}))~,
\end{equation}
where $Q_n$ is an independent random variable with a $\mathsf{Bin}(n,1/2)$ distribution and $\mathsf{s}$ denotes a random element of $\mathcal{S}_{n,Q_n}$. That is, $\mathsf{s}$ is a random set in $[n]$ of cardinality $Q_n$. If the estimator $\hat{\theta}_n(\mathbf{x}^{(d)})$ admits a linear representation \eqref{eq: app linear rep}, then the root $R^*_n(\mathbf{x}^{(d)})$ admits the representation
\begin{equation}
R^*_n(\mathbf{x}^{(d)}) = \frac{1}{n} \sum_{i = 1}^n \tilde{V}_i (\bar{u}(\mathbf{x}^{(d)}, D_i) - \theta_0(\mathbf{x}^{(d)}))
\end{equation}
where $\tilde{V}_i$ is equal to 1 if $i$ is in $\mathsf{s}$ and is equal to $-1$ otherwise. The weights $\tilde{V}_1,\ldots,\tilde{V}_n$ are fully independent.
A bound exactly analogous to \cref{eq: asymptotic validity subsampled kernel} holds if the confidence region formulated in \cref{def: uniform ci} is constructed with the Binomial-Sample bootstrap. This follows immediately from the following Theorem, which gives results analogous to \cref{thm: generic m estimation unstudentized}, Part (ii), and \cref{lem: variance accuracy}, stated in the proof of \cref{thm: generic decomposition}. Proofs are given in \cref{app: bin boot proofs}.
\begin{theorem}\label{thm: binomial sample bootstrap}
Suppose that the moment function $M(x;\theta_0, g_0)$ satisfies \cref{assu: app a moment function} and Part (i) of \cref{assu: moment smoothness} and that \cref{assu: linearity,assu: bias} hold.
\noindent \textbf{(i)} If the bootstrap root is constructed with the Binomial-Sample bootstrap, then the inequality
\begin{flalign}
\sup_{R\in\mathcal{R}}\sup_{P\in\mathbf{P}}
\Big\vert P\left\{ \sqrt{n} R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\} -P\left\{ Z_n\in\mathsf{R}\right\} \Big\vert
& \lesssim \frac{\varphi^{1/2}}{\underline{\lambda}^{1/2}}\left(\frac{\log^{5}\left(dn\right)}{n}\right)^{1/4}+ \delta_n\sqrt{\log{d}} \label{eq: generic bootstrap clt}
\end{flalign}
holds with probability greater than $1-Cn^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) - \rho_n$.
\noindent \textbf{(ii)} Moreover, in this case, we have that
\begin{equation}
P\left\{\sup_{j\in[d]} \bigg\vert \frac{\hat{\lambda}^2_{n,j}}{\lambda^2_{j}} - 1\bigg\vert \geq C \frac{\varphi^2}{\underline{\lambda}^2 n} \log^2(dn) + C\frac{1}{n}\delta_n^2 \right\} \lesssim 1-C(\rho_n+n^{-1})~.
\end{equation}
\end{theorem}
\begin{remark}
\noindent\cref{fig: bin} displays upper and lower confidence bounds for the CATE \eqref{eq: CATE} on post-treatment assets. These bounds are built with the confidence region formulated in \cref{def: uniform ci}, implemented with the binomial-sample bootstrap. The qualitative and quantitative features of this figure are very similar to the features of \cref{fig: half}.\hfill$\blacksquare$
\end{remark}
\subsection{Stochastic Equicontinuity without Sample Splitting\label{app: stability}}
In the main text, we assume that the nuisance parameter estimator $\hat{g}_n$ is computed on a separate sample, independent of the data $\mathbf{D}_n$. In this section, we give additional conditions under which the nuisance parameter estimator $\hat{g}_n$ can be computed on the same data used to construct the estimator $\hat{\theta}_n(\bm{x}^{(d)})$. Our analysis builds on a similar result given in \cite{chen2022debiased}, who give a pointwise analysis of unconditional moment estimators. By contrast, we give a high-dimensional analysis of conditional moment estimators constructed with subsampled kernels.
In the proof of \cref{eq: asymptotic validity subsampled kernel}, the statistical independence of the estimator $\hat{g}_n$ and the data $\mathbf{D}_n$ is only needed in the proof of \cref{lem: kernel bcs}, Part (iii). Thus, our objective is to prove an analogous result, without imposing this restriction. In particular, consider the functions
\begin{flalign*}
a(x;D_{i},D_{\mathsf{s}},g) & =\kappa(x,X_{i};D_{\mathsf{s}})m(D_{i},g)\quad\text{and}\quad A(x;g)=\mathbb{E}\left[\kappa(x,X_{i};D_{\mathsf{s}})m(D_{i},g)\right],
\end{flalign*}
where $\kappa(\cdot,\cdot;D_{\mathsf{s}})$ is a kernel function and
$m(D_{i},g)$ is a moment function. We are interested in the complete,
deterministic, $U$-statistic
\begin{flalign}
& \frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}F(\bm{x}^{(d)};D_{\mathsf{s}},\hat{g}_{n})\label{eq: u stat}
\end{flalign}
where the kernel function $F(x;D_{\mathsf{s}},g)$ is given by
\begin{flalign*}
F(x;D_{\mathsf{s}},g) & =\sum_{i\in\mathsf{s}}f(x;D_{i},D_{\mathsf{s}},g),\\
f(x;D_{i},D_{\mathsf{s}},g) & =a(x;D_{i},D_{\mathsf{s}},g)-a(x;D_{i},D_{\mathsf{s}},g_{0})-\left(A(x;g)-\mathbb{E}\left[A(x;g_{0})\right]\right),
\end{flalign*}
and $\hat{g}_{n}$ is some estimator of the nuisance parameter $g_{0}$
computed with the data $\mathbf{D}_{n}$. That is, for the sake of
simplicity, we have restricted our attention to complete, deterministic
subsampled kernel estimators. Incomplete, random, subsampled
kernel estimators can be accommodated through the same methods applied
repeatedly throughout the arguments supporting \cref{eq: asymptotic validity subsampled kernel}.
To state our additional conditions, we require some additional notation. Let $\mathbf{D}_{n}^{\prime}=(D_{i}^{\prime})_{i=1}^{n}$ denote an
independent copy of $\mathbf{D}_{n}$. Let $\hat{g}_{n}^{(-\mathsf{s})}$
denote a version of the estimator $\hat{g}_{n}$ formed with all observations
in $\mathbf{D}_{n}$, except that the observations $D_{\mathsf{s}}$
are replaced by the observations $D_{\mathsf{s}}^{\prime}$. Define the norm
\[
\|g-\tilde{g}\|_{2q,\infty}=\max_{k\in[h]}\max_{j\in[d]}\left(\mathbb{E}\left[\left(g^{(k)}(Z_{i})-\tilde{g}^{(k)}(Z_{i})\right)^{2q}\mid X_{i}=x^{(j)}\right]\right)^{1/2q}
\]
for any nuisance parameters $g=(g^{(k)})_{k=1}^{h}$ and $\tilde{g}=(\tilde{g}^{(k)})_{k=1}^{h}$
in the space $\mathcal{G}$ and any positive integer $q>0$.
First, we require that moment $m(D_{i},g)$ satisfies a pair of smoothness conditions.
\begin{assumption}
\label{assu: mean-square-continuity}Define the higher-order
variogram
\[
V^{(q)}(x;g,g^{\prime})=\mathbb{E}\left[\left(m(D_{i},g)-m(D_{i},g^{\prime})\right)^{2q}\mid X_{i}=x\right]
\]
for each $g$ and $g^{\prime}$ in $\mathcal{G}$. For each integer
$q>0$, the Lipschitz condition
\begin{equation}
\vert V^{(q)}(x;g,g^{\prime})-V^{(q)}(x^{\prime};g,g^{\prime})\vert\lesssim\|x-x^{\prime}\|_{\infty}^{q}\label{eq: Lipschitz variogram}
\end{equation}
holds for each $x$ and $x^{\prime}$ in $\mathcal{X}$ and the mean-square
continuity condition
\begin{equation}
\sup_{j\in[d]}V^{(q)}(x^{(j)};g,g^{\prime})\lesssim\|g-g^{\prime}\|_{2q,\infty}^{2q}\label{eq: mean square continuity}
\end{equation}
holds for each $g$ and $g^{\prime}$ in $\mathcal{G}$.
\end{assumption}
\noindent
\cref{assu: mean-square-continuity} is a higher-order analogue to Part (ii) of \cref{assu: moment smoothness}.
Second, we impose the following higher-order moment stability conditions.
\begin{assumption}
\label{assu: stability}There exists some increasing sequence
$\gamma(q)$, which may depend on the parameters $n$, $b$, and $d$,
such that the $L_{q}$-norm stability bounds
\begin{flalign}
\|m(D_{1},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\|_{q} & \lesssim q\gamma(q)\frac{b}{n}\quad\text{and}\label{eq: stability bound D_1}\\
\|m(D_{1}^{\prime\prime},\hat{g}_{n})-m(D_{1}^{\prime\prime},\hat{g}_{n}^{(-[b])})\|_{q} & \lesssim q\gamma(q)\frac{b}{n}\label{eq: stability bound D}
\end{flalign}
hold, where $D_{1}^{\prime\prime}$ is an independent copy of $D_{1}$.
\end{assumption}
\noindent
\cref{assu: stability} quantifies the sensitivity of the moment $m(D_{1},\hat{g}_{n})$ to re-sampling $b$ elements of the data $\mathbf{D}_n$. Analogous conditions are studied in \cite{abou2019exponential} and \cite{chen2022debiased}. We discuss moments and estimators that satisfy \cref{assu: stability} in \cref{sec: verify}.
The following Theorem gives a suitable large deviation bound for the U-statistic \eqref{eq: u stat}.
\begin{theorem}
\label{lem: stability}Suppose that the kernel $\kappa(x,X_{i};D_{\mathsf{s}})$ satisfies \cref{assu: kernel restriction} and has shrinkage rate $\varepsilon_{b}$, that the moment function $m(D_{i},g)$ satisfies \cref{assu: mean-square-continuity}, and that the nuisance parameter estimator satisfies the high-order moment bound
\begin{equation}
\mathbb{E}\left[\|\hat{g}_{n}-g_{0}\|_{q,\infty}^{q}\right]\lesssim\left(q\left(\frac{b}{n}\right)^{1/4}\delta_{n,g}\right)^{q}\label{eq: rate condition se}
\end{equation}
for some sequence $\delta_{n,g}$. If \cref{assu: stability} holds, then
\begin{flalign}
\bigg\|\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}F(\bm{x}^{(d)};D_{\mathsf{s}},\hat{g}_{n})\bigg\|_{\infty}
& \lesssim\sqrt{\frac{b}{n}}\left(\left(\frac{b}{n}\right)^{1/4}\delta_{n,g}
+\gamma_{n,b}^{\star}\right)\log(dn) +\varepsilon_{b} \label{eq: stoch eq rate}\\
& \quad+\frac{b}{n}\gamma_{n,b}^{\star}\log^2(dn)\nonumber
\end{flalign}
with probability greater than $1-Cn^{-1}$, where $\gamma_{n,b}^{\star}=\gamma(2\lceil\log(dn)\rceil)b^{1/\log(dn)}$.
\end{theorem}
\begin{remark}
Compare the rate \eqref{eq: stoch eq rate} to the rate given in \cref{lem: kernel bcs}, Part (iii). The rates match, up to the additional term $\gamma_{n,b}^{\star}$. Thus, under the conditions of \cref{lem: stability}, if the quantity $\gamma_{n,b}^{\star}$ decreases to zero as $n$ and $b$ increase, a result analogous to \cref{eq: asymptotic validity subsampled kernel} will hold. We discuss conditions under which $\gamma_{n,b}^{\star}$ decreases in the \cref{sec: verify}.\hfill{}$\blacksquare$
\end{remark}
\begin{remark}
\cref{lem: stability} follows from an argument similar to the argument developed in \cite{chen2022debiased}. There is one important difference. \cite{chen2022debiased} apply a ``double-centering trick,'' due to \cite{kumar2013near}, to get a variance bound. To obtain a high-dimensional bound, with a logarithmic dependence on the dimension $d$, we replace this step with an application of a generalized Efron-Stein inequality, due to \cite{boucheron2005moment}.\hfill{}$\blacksquare$
\end{remark}
\subsubsection{\label{sec: verify}Verifying \cref{assu: stability}}
In this section, we study conditions under which \cref{assu: stability} is satisfied. First, we impose an additional smoothness condition on the
moment function $m(D_{i},g)$. Throughout, we write the nuisance parameter estimator $\hat{g}_{n}$ as $\hat{g}_{n}=(\hat{g}_{k,n})_{k=1}^{h}$.
\begin{assumption}
\label{assu: finite moment bound}The inequalities
\begin{equation}
\|m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\|_{2q}\lesssim\max_{k\in[h]}\mathbb{E}\left[\sup_{z\in\mathcal{Z}}\left(\hat{g}_{k,n}(z)-\hat{g}_{k,n}^{(-[b])}(z)\right)^{2q}\right]^{1/2q}\label{eq: same finte approx}
\end{equation}
and
\begin{equation}
\|m(D_{i}^{\prime\prime},\hat{g}_{n})-m(D_{i}^{\prime\prime},\hat{g}_{n}^{(-[b])})\|_{2q}\lesssim\max_{k\in[h]}\mathbb{E}\left[\mathbb{E}\left[\left(\hat{g}_{k,n}(Z_{i}^{\prime})-\hat{g}_{k,n}^{(-[b])}(Z_{i}^{\prime})\right)^{2q}\mid D_{i}^{\prime}\right]\right]^{1/2q}\label{eq: independent approx}
\end{equation}
hold for any positive integer $q$.
\end{assumption}
\noindent The bound (\ref{eq: same finte approx})
stipulates that the higher-order moments of $m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})$
are smaller than the supremum of the higher order-moments of $\hat{g}_{k,n}(z)-\hat{g}_{k,n}^{(-[b])}(z)$
over the space $\mathcal{Z}$. The bound (\ref{eq: independent approx})
is weaker, as we only need to control the distance $\hat{g}_{k,n}(z)-\hat{g}_{k,n}^{(-[b])}(z)$
at $z=Z_{i}^{\prime}$.
Bounds of the form specified by \cref{assu: finite moment bound} are satisfied by many standard choices of moment function. For example, consider the moment function
\begin{equation}
m(D_{i},g)=\mu(Z_{i},1)-\mu(Z_{i},0)+\beta(W_{i},Z_{i})(Y_{i}-\mu(Z_{i},W_{i}))\label{eq: aipw se}
\end{equation}
where the nuisance function $g$ collects the moment parameters $g=(\mu,\beta)$. Observe that \eqref{eq: aipw se} is analogous to the moment function \eqref{eq: CATE moment} used as a running example in the main text.
\begin{lemma}
\label{lem: continuity}If the quantities
\begin{equation}
\vert\hat{\beta}_{n}(W_{i},Z_{i})\vert\quad\text{and}\quad\vert Y_{i}-\hat{\mu}_{n}(Z_{i},W_{i})\vert\label{eq: bounded quantities}
\end{equation}
are bounded almost surely, then it holds that
\begin{flalign}
& \|m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\|_{2p}\nonumber \\
& \lesssim\max\bigg\{\max_{w\in\left\{ 0,1\right\} }\left\{ \|\hat{\mu}_{n}(w,Z_{i})-\hat{\mu}_{n}^{(-[b])}(w,Z_{i})\|_{2p}\right\} ,\label{eq: orlicz continuity}\\
& \quad\quad\quad\quad\quad\quad\|\hat{\mu}_{n}(W_{i},Z_{i})-\hat{\mu}_{n}^{(-[b])}(w,Z_{i})\|_{2p},\hat{\beta}_{n}(W_{i},Z_{i})-\hat{\beta}_{n}^{(-[b])}(W_{i},Z_{i})\|_{2p}\bigg\},\quad\text{and}\nonumber \\
& \|m(D_{i}^{\prime\prime},\hat{g}_{n})-m(D_{i}^{\prime\prime},\hat{g}_{n}^{(-i)})\|_{2p}\nonumber \\
& \lesssim\max\bigg\{\max_{w\in\left\{ 0,1\right\} }\left\{ \|\hat{\mu}_{n}(w,Z_{i}^{\prime\prime})-\hat{\mu}_{n}^{(-[b])}(w,Z_{i}^{\prime\prime})\|_{2p}\right\} ,\label{eq: orlicz continuity Z}\\
& \quad\quad\quad\quad\quad\quad\|\hat{\mu}_{n}(W_{i},Z_{i}^{\prime\prime})-\hat{\mu}_{n}^{(-[b])}(w,Z_{i}^{\prime\prime})\|_{2p},\hat{\beta}_{n}(W_{i},Z_{i}^{\prime\prime})-\hat{\beta}_{n}^{(-[b])}(W_{i},Z_{i}^{\prime\prime})\|_{2p}\bigg\},\nonumber
\end{flalign}
respectively, where $D_{i}^{\prime\prime}$ is an independent copy
of $D_{i}$ and $Z_{i}^{\prime\prime}$ is an independent copy of
$Z_{i}$.
\end{lemma}
Now, we consider the higher-order stability of the nuisance parameter estimator $\hat{g}_{n}(z)$. For the sake of simplicity, we assume that $\hat{g}_{n}(z)$ is scalar valued and takes the form of a complete, deterministic, $U$-statistic
\begin{equation}\label{eq: g hat as u stat}
\hat{g}_{n}(z)=\frac{1}{N_{b^{\prime}}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b^{\prime}}}u_{z}(D_{\mathsf{s}}),
\end{equation}
where $u_{z}(D_{\mathsf{s}})$ is some deterministic kernel function
of order $b^{\prime}$. Again, let $\hat{g}^{(-[b])}_{n}(z)$ be constructed analogously to $\hat{g}_{n}(z)$ using the $\mathbf{D}_{n}$, but replacing $D_{[b]}$ with an independent copy
$D_{[b]}^{\prime}$.
\begin{lemma}
\label{lem: kernel stability}Suppose that the kernel
function $u_{z}(D_{\mathsf{s}})$ satisfies the bound $\vert u_{z}(D_{\mathsf{s}})\vert\leq\phi$
almost surely for each $z$ in $\mathcal{Z}$. If the estimator $\hat{g}_{n}(z)$ is given by \eqref{eq: g hat as u stat}, then it holds that
\begin{flalign}
\|\hat{g}_{n}(z)-\hat{g}^{(-[b])}_{n}(z)\|_{q} & \lesssim\frac{\sqrt{b^{\prime}b}}{n}q\phi+q^{3/2}\phi^{2}n\left(C\frac{b^{\prime}q}{n}\right)^{b^{\prime}/2}\log(n).\label{eq: kernel stability state}
\end{flalign}
for each $z$ in $\mathcal{Z}$.
\end{lemma}
\begin{remark}
Under \cref{assu: finite moment bound}, \cref{lem: kernel stability} implies that
\begin{equation}
\|m(D_{1}^{\prime\prime},\hat{g}_{n})-m(D_{1}^{\prime\prime},\hat{g}_{n}^{(-[b])})\|_{q}\lesssim\frac{\sqrt{b^{\prime}b}}{n}q\phi+q^{3/2}\phi^{2}n\left(C\frac{b^{\prime}q}{n}\right)^{b^{\prime}/2}\log(n).\label{eq: moment bound apply kernel stability}
\end{equation}
In terms of the notation introduced in \cref{assu: stability}, the bound \eqref{eq: kernel stability state} implies that
\[
\gamma(q)=\sqrt{\frac{b^{\prime}}{b}}\phi+\frac{q^{1/2}n^{2}\phi^{2}}{b}\left(C\frac{b^{\prime}q}{n}\right)^{b^{\prime}/2}\log(n).
\]
At the choice $q=\log(dn)$, invoked in \cref{lem: stability}, and assuming that $b^{\prime}=o(n)$ by at least a logarithmic factor, we have that
\[
\gamma(\log(dn))\lesssim\frac{b^{\prime}}{b}\phi.
\]
Consequently, \cref{lem: kernel stability} suggests that the bound \eqref{eq: stoch eq rate} is $o_p(\sqrt{b/n})$ if $b^{\prime}=o(b)$.\hfill{}$\blacksquare$
\end{remark}
\begin{remark}
\cref{assu: stability} entails two stability bounds, \eqref{eq: stability bound D_1} and \eqref{eq: stability bound D}. Under \cref{assu: finite moment bound}, \cref{lem: kernel stability} quantifies the sequence $\gamma(q)$ introduced on the right-hand side of \eqref{eq: stability bound D_1}. By the discussion above, if $b^{\prime}=o(b)$, this sequence is sufficiently small for a result analogous to \cref{lem: kernel bcs}, Part (iii) to hold. Handling the bound \eqref{eq: stability bound D} requires developing a more refined argument. In particular, under \cref{assu: finite moment bound}, to bound the right-hand-side of \eqref{eq: stability bound D_1}, we must account for the supremum over the space $\mathcal{Z}$ within the expectation. Such a generalization should be reasonably straightforward, potentially through a chaining argument. Roughly speaking, we should expect the supremum to contribute a term like $\log(p)$ to the stability, where $p$ is the dimension of $\mathcal{Z}$. In this case, in settings where $p$ is not high-dimensional, this would not generate any issues. This should be contrasted with analogous supremums taken over the parameter space of a decision tree (which are generated if stochastic equicontinuity is controlled with a union bound). Here the dimension of the parameter space is roughly $2^{\text{Depth of Tree}}$, which may large enough to make a material difference. However, in order to operationalize this intuition, we would need to place further restrictions sufficient for the smoothness of the H\'{a}jek projection of $\hat{g}_n(z)$. We leave this for future work. \hfill{}$\blacksquare$
\end{remark}
\subsection{Cross-Fitting\label{app: cross}}
\cref{eq: asymptotic validity subsampled kernel} holds under the assumption that the nuisance parameter estimator $\hat{g}_n$ is computed on a data set that is statistically independent of the data used to construct the confidence region $\hat{\mathcal{C}}(\bm{x}^{(d)})$. This condition can be achieved by splitting the data into two parts, at the cost of reducing statistical precision and introducing randomness independent of the observed data. In \cref{app: stability}, we introduce further restrictions that allow the nuisance parameter estimator $\hat{g}_n$ and the confidence region $\hat{\mathcal{C}}(\bm{x}^{(d)})$ to be computed using the same data. However, in practice, these conditions are more stringent, and it may be unclear whether they are satisfied for a given nuisance parameter estimator $\hat{g}_n$.
In this section, we detail an alternative procedure for constructing a confidence region similar to the confidence region specified in \cref{def: uniform ci}, whose validity will hold under the same conditions imposed in \cref{eq: asymptotic validity subsampled kernel}.
The procedure is based on a cross-fit estimator. To introduce this estimator, we require some additional notation. Let $\mathcal{R}_{n,k}$ denote the set of partitions of $[n]$ into $k$ equally sized and mutually exclusive subsets. That is, for each $\mathsf{r} = (\mathsf{s}_1,\ldots,\mathsf{s}_k)$ in $\mathcal{R}_{n,k}$, the sets $\mathsf{s}_1,\ldots,\mathsf{s}_k$ are mutually exclusive, have union equal to $[n]$, and are each of size $n/k$. Throughout, we set $q=n/k$ and let $\tilde{\mathsf{s}}$ denote the complement of the set $\mathsf{s}$ in $[n]$. Finally, for each subset $\mathsf{s}$ of $[n]$, let $\hat{g}_{\mathsf{s}}$ denote a version of the nuisance parameter estimator $\hat{g}_n$ computed with the data $D_{\mathsf{s}}$.
The cross-fit estimator is computed as follows. For each subset $\mathsf{s}$ in $\mathcal{S}_{n,q}$, let the estimator $\hat{\theta}_\mathsf{s}(\bm{x}^{(d)})$ be constructed by solving the empirical conditional moment equality \eqref{eq: subsampled kernel} using the data $D_{\mathsf{s}}$ and the nuisance parameter estimator $\hat{g}_{\tilde{\mathsf{s}}}$. That is, the nuisance parameter estimator is computed on the sample of data whose indices are not in $\mathsf{s}$. The $k$-fold cross-fit estimator is given by
\begin{equation}\label{eq: k fold cross fit estimator}
\hat{\theta}_{\mathsf{r}}(\bm{x}^{(d)}) = \frac{1}{k} \sum_{\mathsf{s} \in \mathsf{r}} \hat{\theta}_\mathsf{s}(\bm{x}^{(d)})~,
\end{equation}
where $\mathsf{r}$ denotes a random element of $\mathcal{R}_{n,k}$.
It is somewhat unclear how to implement the half-sample bootstrap with the estimator $\hat{\theta}_\mathsf{r}(\bm{x}^{(d)})$. We propose the following computationally efficient variant. Let $\mathcal{H}(\mathsf{s})$ denote the set of half-samples of the set $\mathsf{s}$, i.e., the set of subsets of $\mathsf{s}$ that contain exactly half of its elements. We say that the collection $\mathsf{H} = (\mathsf{h}_l)_{l=1}^k$ is a half-sample of the partition $\mathsf{r} = (\mathsf{s}_l)_{l=1}^k$ if each $\mathsf{h}_l$ is an element of $\mathcal{H}(\mathsf{s}_l)$, i.e., if each $\mathsf{h}_l$ is a half-sample of $\mathsf{s}_l$. For each $l$ in $[k]$, let the estimator $\hat{\theta}_\mathsf{\mathsf{h}_l}(\bm{x}^{(d)})$ be constructed by solving the empirical conditional moment equality \eqref{eq: subsampled kernel} using the data $D_{\mathsf{h}_l}$ and the nuisance parameter estimator $\hat{g}_{\tilde{\mathsf{s}}_l}$, i.e., using the same nuisance parameter estimates used to construct \eqref{eq: k fold cross fit estimator}. The half-sample $k$-fold cross-split estimator is given by
\begin{equation}\label{eq: k fold half-split estimator}
\hat{\theta}_{\mathsf{H}}(\bm{x}^{(d)}) = \frac{1}{k} \sum_{\mathsf{h} \in \mathsf{H}} \hat{\theta}_\mathsf{h}(\bm{x}^{(d)})~.
\end{equation}
In this way, once the $k$-fold cross-fit estimator \eqref{eq: k fold cross fit estimator} has been computed, the nuisance parameter estimates $\hat{g}_{\tilde{\mathsf{s}}_1}, \ldots, \hat{g}_{\tilde{\mathsf{s}}_k}$ do not need to be recomputed to construct the half-split $k$-fold cross-split estimator \eqref{eq: k fold half-split estimator}. Simultaneous confidence intervals can then be constructed analogously to the intervals introduced in \cref{def: uniform ci} by using the half-sample $k$-fold cross-split bootstrap root
\begin{equation} \label{eq: half-sample k-fold root}
R^*_n(\bm{x}^{(d)} = \hat{\theta}_{\mathsf{H}}(\bm{x}^{(d)}) - \hat{\theta}_{\mathsf{r}}(\bm{x}^{(d)})
\end{equation}
in the place of the half-sample bootstrap root \eqref{eq: bootstrap root}. In other words, when implementing the half-sample bootstrap based on an estimator constructed with $k$-fold cross-fitting, one can avoid recomputing nuisance parameters if the half-samples are ``stratified'' across the $k$-folds. The error bound given in \cref{eq: asymptotic validity subsampled kernel} will generalize to the confidence region based on the bootstrap root \eqref{eq: half-sample k-fold root} through a
straightforward argument, so long as $k$ is bounded.
\cref{fig: cross cate} and \cref{fig: half cross} gives versions of \cref{fig: cate} and \cref{fig: half}, constructed using the $2$-fold cross-split estimator \eqref{eq: k fold cross fit estimator} and the $2$-fold cross-split bootstrap root \eqref{eq: half-sample k-fold root}. The estimates and confidence bounds are quantitatively and qualitatively very similar to the estimates and confidence bounds reported in the main text.
\cref{fig: cross performance} displays results for the simulation presented in \cref{sec: simulation}, analogous to \cref{fig: performance}, for the $2$-fold cross-split estimator \eqref{eq: k fold cross fit estimator} and the $2$-fold cross-split bootstrap root \eqref{eq: half-sample k-fold root}. The confidence region is very slightly anti-conservative, so long as $b/n$ is decreasing as $n$ increases. The measurements of bias and variance exhibit patterns very similar to the patterns displayed in \eqref{fig: performance}.
\subsection{Proof of \cref{thm: Hoef clt}\label{eq: proof of hoef clt}}
To ease notation, we drop the dependence on $x^{(j)}$. The argument will follow by first re-expressing the $U$-statistic of interest in terms of its Hoeffding expansion, stated in the proof of \cref{lem: projection maximal}. In particular, \cref{lem: Hoeffding expansion} implies that there exist functions $u^{(1)}(\cdot),\ldots,u^{(b)}(\cdot)$ such that
\begin{equation}\label{eq: gen hoef applied to u}
u(D_{[b]}) = \sum_{l = 1}^b \sum_{\mathsf{s}\in\mathcal{S}_{b,l}} u^{(l)}(D_{\mathsf{s}})
\end{equation}
and that all $2^{b} - 1$ terms on the right-hand side of \eqref{eq: gen hoef applied to u} are mean-zero and uncorrelated. Thus, we have that
\begin{equation}
\Var(u(D_{[b]})) = \sum_{l=1}^b {b \choose l} \Var(u^{(l)}(D_{[l]}))
\end{equation}
and that thereby
\begin{equation}\label{eq: hoef var inequality}
{b \choose l} \Var(u^{(l)}(D_{[l]})) \leq \Var(u(D_\mathsf{[b]}))~.
\end{equation}
Moreover, again by \eqref{eq: gen hoef applied to u}, we can write
\begin{align}
\frac{1}{N_b} \sum_{\mathsf{s} \in \mathsf{S}_{n,b}} u(D_\mathsf{s})
& = {n \choose b}^{-1} \sum_{\mathsf{s} \in \mathsf{S}_{n,b}} \sum_{l = 1}^b \sum_{\mathsf{r}\in\mathcal{S}_{l}(\mathsf{s})} u^{(l)}(D_{\mathsf{r}}) \\
& = \sum_{l = 1}^b {n \choose b}^{-1} {n - l \choose b - l} \sum_{r \in \mathcal{S}_{n,l}} u^{(l)}(D_{\mathsf{r}}) \nonumber\\
& = \sum_{l = 1}^b {b \choose l} {n \choose l}^{-1} \sum_{\mathsf{r}\in\mathcal{S}_{n,l}} u^{(l)}(D_{\mathsf{r}})~,\nonumber
\end{align}
where $\mathcal{S}_{l}(\mathsf{s})$ denotes the set of $\mathsf{s}$ of cardinality $l$.
Thereby, by the mutual orthogonality of the terms in the Hoeffding expansion \eqref{eq: gen hoef applied to u}, it holds that
\begin{align}
\Var\left(\frac{1}{N_b} \sum_{\mathsf{s} \in \mathsf{S}_{n,b}} u(D_\mathsf{s}) - \frac{b}{n} \sum_{i=1}^n u^{(1)}(D_i) \right)
& = \Var\left(\sum_{l = 2}^b {b \choose l} {n \choose l}^{-1} \sum_{\mathsf{r}\in\mathcal{S}_{n,l}} u^{(l)}(D_{\mathsf{r}}) \right)\nonumber\\
& = \sum_{l=2}^b {b \choose l}^{2} {n \choose l}^{-2} \sum_{\mathsf{r}\in\mathcal{S}_{n,l}} \Var(u^{(l)}(D_{\mathsf{r}}))\nonumber\\
& = \sum_{l=2}^b {b \choose l}^2 {n \choose l}^{-1} \Var(u^{(l)}(D_{[l]}))~. \label{eq: hajek variance equality}
\end{align}
Consequently, we have that
\begin{align}
\Var\left(\frac{1}{N_b} \sum_{\mathsf{s} \in \mathsf{S}_{n,b}} u(D_\mathsf{s}) - \frac{b}{n} \sum_{i=1}^n u^{(1)}(D_i) \right)\nonumber
& = \sum_{l=2}^b {b \choose l}^2 {n \choose l}^{-1} \Var(u^{(l)}(D_{[l]}))\nonumber\\
& \leq \Var(u(D_{[b]})) \sum_{l=2}^b {b \choose l} {n \choose l}^{-1} \nonumber\\
& \leq \Var(u(D_{[b]})) \sum_{l=2}^b \left(\frac{b}{n}\right)^l \nonumber\\
& \leq \Var(u(D_{[b]})) \left(\frac{b}{n}\right)^2\left(1 - \frac{b}{n}\right)^{-1} ~, \label{eq: sum bound}
\end{align}
where the first inequality follows from the inequality \eqref{eq: hoef var inequality} and the third equality follows from the geometric series formula. Now, consider the decomposition
\begin{align}
\sqrt{\frac{n}{\sigma^2_{b,j} b^2}} \left(\frac{1}{N_b} \sum_{\mathsf{s} \in \mathsf{S}_{n,b}} u(D_\mathsf{s})\right)
&= \sqrt{\frac{1}{\sigma^2_{b,j} n}} \sum_{i=1}^n u^{(1)}(D_\mathsf{s}) \label{eq: term to keep}
\\
&- \sqrt{\frac{n}{\sigma^2_{b,j} b^2}} \left(\frac{1}{N_b} \sum_{\mathsf{s} \in \mathsf{S}_{n,b}} u(D_\mathsf{s}) - \frac{b}{n} \sum_{i=1}^n u^{(1)}(D_i) \right)~. \label{eq: app term to cheb}
\end{align}
The normalization $\nu^2_j/n\sigma^2_{b,j} \to 0$, the bound \eqref{eq: sum bound}, and Chebychev's inequality imply that the term \eqref{eq: app term to cheb} is $o_p(1)$ as $n\to\infty$. The result then follows by applying the central limit theorem to the term \eqref{eq: term to keep}.\hfill\qed
\subsection{Proof of \cref{cor: general u stat asymptotic}, Part (i)\label{sec: app cor p1}}
Observe that
\begin{align}
\sqrt{\frac{n}{b^2}} \Sigma^{-1/2} \bar{U}_{n,b}(\bm{x}^{(d)})
& = \sqrt{\frac{1}{n}} \sum_{i=1}^n \Sigma^{-1/2} u^{(1)}(\bm{x}^{(d)}; D_i)\label{eq: cor decompositon}\\
& + \sqrt{\frac{n}{b^2}} \Sigma^{-1/2} \left(\bar{U}_{n,b}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n u^{(1)}(\bm{x}^{(d)}; D_i)\right)~.\label{eq: scaled resid}
\end{align}
\cref{lem: orlicz large deviation} implies that
\begin{align}
\Big\| \sqrt{\frac{1}{n}} \sum_{i=1}^{n} \Sigma^{-1/2} \tilde{u}^{(1)}(\bm{x}^{(d)}, D_i) \Big\|_{\infty} \lesssim \log^{1/2}(dn) + \frac{\phi\log^2(dn)}{\underline{\sigma}_b n^{1/2}}
\end{align}
with probability greater than $1-C/n$. We show in the proof of \cref{lem: u stat linearity} that there exists a constant $C_1$ such that if $b\log(dn)/n <C_1 < 1$, then \cref{lem: projection maximal} implies that \eqref{eq: scaled resid} is smaller than $\phi n^{-1}$, with probability greater than $1-Cn^{-1}$, so long as $b\gtrsim C_2 \log(n)$.
The result then follows by aggregating these two bounds.\hfill\qed
\subsection{Proof of \cref{cor: general u stat asymptotic}, Part (ii)\label{sec: app cor p2}}
Fix a rectangle $\mathsf{R} = [a_l, a_u]$ in $\mathcal{R}$, where $a_l$ and $a_u$ are vectors in $\mathbb{R}^d$ with $a_l\leq a_u$, interpreted componentwise. Define the enlarged rectangle $\mathsf{R}_t=[a_l-t\bm{1}_d,a_u+t\bm{1}_d]$ for each $t>0$. Define the normalized quantity
\begin{equation}
\hat{u}^{(1)}(\bm{x}^{(d)}; D_i) = \Sigma^{-1/2}\tilde{u}^{(1)}(\bm{x}^{(d)}; D_i)~.
\end{equation}
For reasons of space, we give the details of the proof of the upper bound encoded in \eqref{eq: general u stat clt}. The lower bound will follow from an analogous argument, and we note the differences where they occur.
Observe that the decomposition \eqref{eq: cor decompositon} implies the upper bound
\begin{align}
& P\left\{\sqrt{\frac{n}{b^2}}\Sigma^{-1/2} \bar{U}_{n,b}(\bm{x}^{(d)}) \in \mathsf{R}\right\} - P\left\{ \Sigma^{-1/2}Z \in \mathsf{R}\right\} \label{eq: general to upper bound}\\
& \leq \bigg\vert P\left\{\frac{1}{\sqrt{n}} \sum_{i=1}^n \hat{u}^{(1)}(\bm{x}^{(d)}; D_i) \in \mathsf{R}_t\right\} - P\left\{ \Sigma^{-1/2}Z \in \mathsf{R}_t\right\}\bigg\vert\label{eq: general iid}\\
& + \bigg\vert P\left\{ \Sigma^{-1/2}Z \in \mathsf{R}_t\right\} - P\left\{ \Sigma^{-1/2}Z \in \mathsf{R}\right\} \bigg\vert\label{eq: general comparions}\\
& + \bigg\vert P\left\{ \big \|\sqrt{\frac{n}{b^2}}\Sigma^{-1} \bar{U}_{n,b}(\bm{x}^{(d)})- \frac{1}{\sqrt{n}} \sum_{i=1}^n \hat{u}^{(1)}(\bm{x}^{(d)}; D_i)\big \|_\infty> t\right\} \bigg\vert~.\label{eq: general hajek}
\end{align}
A matching lower bound follows from an analogous argument, by considering the shrunken rectangle $[a_l+t\bm{1}_d,a_u-t\bm{1}_d]$.
We bound the normal approximation term \eqref{eq: general iid} through the application of \cref{lem: baseline clt}. In particular, observe that
\begin{equation}
\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\left(\hat{u}^{(1)}(x^{(j)}; D_{i})\right)^2\right]=1~,
\quad\
\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\left(\hat{u}^{(1)}(x^{(j)},D_{i})\right)^4\right]
\leq \left(\phi/\underline{\sigma}_b\right)^{2}
\label{eq: standardized}
\end{equation}
and $\| \hat{u}^{(1)}(x^{(j)}, D_{i}) \|_{\psi_1} \leq (\phi/\underline{\sigma}_b)$. Consequently, as
\begin{equation}
\Var(\hat{u}^{(1)}(x^{(j)}, D_{i}))=\Sigma^{-1/2}\Var\left(Z\right)\Sigma^{-1/2},
\end{equation}
Lemma \ref{lem: baseline clt} implies that
\begin{equation}
\bigg\vert P\left\{\frac{1}{\sqrt{n}} \sum_{i=1}^n \hat{u}^{(1)}(\bm{x}^{(d)}; D_i) \in \mathsf{R}_t\right\} - P\left\{ \Sigma^{-1/2}Z \in \mathsf{R}_t\right\}\bigg\vert
\lesssim \left(\frac{\phi^{2}\log^{5}(dn)}{\underline{\sigma}_b^{2}n}\right)^{1/4}~.\label{eq: baseline clt applied}
\end{equation}
Now, to bound the difference \eqref{eq: general comparions}, \cref{lem: Nazarov} gives
\begin{equation}
\Big\vert P\left\{ \Sigma^{-1/2}Z \in \mathsf{R}_t\right\} - P\left\{ \Sigma^{-1/2}Z \in \mathsf{R}\right\} \Big\vert \lesssim t \sqrt{\log(d)}~.
\end{equation}
Finally, as in the previous Part of the Lemma, recall that there exists a constant $C_1$ such that if $b\log(dn)/n <C_1 < 1$, then \cref{lem: projection maximal} implies that \eqref{eq: scaled resid} is smaller than $\phi n^{-1}$, with probability greater than $1-Cn^{-1}$, so long as $b\gtrsim C_2 \log(n)$. Thus, the proof is complete by setting $t = \phi n^{-1}$. \hfill\qed
\section{Details Concerning Data and Simulations\label{app: simulation}}
In this appendix, we document our treatment of the \cite{banerjee2015multifaceted} data. \cref{sec:data} details our acquisition and cleaning of these data. In \cref{sec: param}, we give further details concerning the construction of \cref{fig: cate} and \cref{fig: half}. \cref{app: calibration} discusses our simulation calibration.
\subsection{Data\label{sec:data}} The data from \cite{banerjee2015multifaceted} were acquired from \url{https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/NHIXNT} on
September 10, 2021. The data from the graduation program implemented in Pakistan are considered in \cite{chen2023semiparametric} and \cite{ritzwoller2023reproducible}. Here, we consider the data from the graduation program implemented in Ghana, as it has a larger sample size.
The data record measurements of many pre-treatment and post-treatment outcomes for 2,606 individuals in the northern region of Ghana. Baseline survey measurements were made prior to the allocation of treatment. A multifaceted treatment was randomly allocated to 632 of the individuals. \cite{banerjee2015multifaceted} consider data from two endline surveys, made two years and three years after the initial asset transfer, respectively. For the purpose of this paper, we consider only data from the baseline survey, records of the treatment allocation, and measurements from the first endline survey. We omit data from 164 attrited individuals, none of whom were assigned to the treatment.
The covariate vector $Z_i$ is composed of measurements of 16 pre-treatment outcomes. Five of these outcomes are associated with consumption: total monthly consumption and total monthly consumption on food, non-food, and durable commodities. Each consumption variable is measured in 2014 US dollars. We transform total monthly consumption to logs, base 10. Three of the variables are associated with assets, each of which is an index constructed from survey data measuring total assets, total productive assets, and total household assets. We transform the total assets measurement to logs, base 10. Five of the outcomes are associated with food security. These consist of four binary variables indicating different aspects of food security, e.g. did a child skip a meal, in addition to an index aggregating these measurements.\footnote{There are $2$ individuals with missing values for the food security index. We impute these values with the median values of the food security index.} The final four variables are associated with finance and income: the total amount of outstanding loans, the total amount of savings, income from agriculture, and total income from business. The covariate vector $X_i$ collects the total monthly consumption and assets for each individual. The outcome $Y_i$ measures the total assets two years after the initial asset transfer. Again, we transform these measurements to logs, base 10.
\subsection{Parameter Choices\label{sec: param}}
In \cref{fig: cate} and \cref{fig: half}, we set the subsample proportion $b/n$ equal to 0.05. In constructing the nuisance parameter estimate, we use set $b/n$ = 0.025. We use $r=200$ bootstrap replicates throughout. We use 20,000 trees to construct \cref{fig: cate} and 2,000 trees in each bootstrap replicate to construct \cref{fig: half} and throughout the simulation.
\subsection{Simulation Calibration\label{app: calibration}} We calibrate a simulation to the \cite{banerjee2015multifaceted} data using a collection of Generalized Adversarial Networks (GAN) \citep{goodfellow2014generative}. This approach to simulation design was proposed by \cite{athey2021using}. Roughly speaking, a GAN is a pair of neural networks. The objective of the first network, the generator, is to generate data that looks like the \cite{banerjee2015multifaceted} data. The objective of the second network, the discriminator, is to discriminate between the real \cite{banerjee2015multifaceted} data and the data generated by the generator. These networks compete iteratively until convergence. The idea is that, after convergence, the generator is a good proxy for the true process that generated the \cite{banerjee2015multifaceted} data. We use the ``WGAN'' package associated with \cite{athey2021using}.
To calibrate our simulation, we estimate three GANs. The first GAN estimates the distribution of the covariates $X_i$, i.e., baseline consumption and baseline total assets. The second GAN estimates the distribution of $Z_i$ conditional on $X_i$. Recall that $Z_i$ collects all baseline covariates, other than the covariates in $X_i$. The third GAN estimates the distribution of $Y_i$ conditioned on $Z_i$, $X_i$, and $W_i$. To generate an observation $D_i$, we generate $X_i$, generate $Z_i$ conditioned on $X_i$, and generate the potential outcomes $Y_i(1)$ and $Y_i(0)$ from the estimated distributions of $Y_i$ conditioned on $Z_i$, $X_i$, and $W_i = 1$ and $W_i = 0$, respectively. The treatment indicator is $W_i$ is drawn i.i.d., Bernoulli with the observed probability in the \cite{banerjee2015multifaceted} data and we set $Y_i = Y_i(W_i)$. In this way, we know the true treatment effect $Y_i(1) - Y_i(0)$ for each unit in our simulation. We draw 10 million observations $D_i$ with this process. In the simulation, datasets of various sizes are sampled from this collection.
We use a related procedure to determine the true CATE $\theta_0(x)$ at each value $x$ in the query-vector $\mathbf{x}^{(d)}$ (i.e., the centers of each of the rectangles displayed in \cref{fig: cate}). Specifically, for each $x$ in $\mathbf{x}^{(d)}$, we draw 100,000 observations from the distribution of $Z_i$ conditioned on $X_i = x$. We then draw observations $Y_i(1)$ and $Y_i(0)$ for each of these replicates, and compute the average of the true treatment effects $Y_i(1) - Y_i(0)$. \cref{fig: simulation cate} displays these pseudo-true values of the CATE $\theta_0(x)$. Our simulation design captures many of the same features of the data recovered by GRF, but gives a somewhat smoother picture of the CATE.
\cref{fig:validation} displays a scatterplot comparing the moments of the data from the \cite{banerjee2015multifaceted} data to the data generated by our calibrated simulation. The distributions match quite closely. \cref{fig: density} compares a scatter plot of the observed values of baseline consumption and baseline assets in the \cite{banerjee2015multifaceted} data with a heat-map of the distribution of these covariates in our simulation. The limits of the horizontal and vertical axes in this Figure match \cref{fig: cate,fig: half} displayed in the main text. Some observations fall outside of the limits of this figure. The quartiles of baseline log consumption are 3.33, 3.76, and 4.20. The quartiles of baseline assets are -0.45, -0.71, and 0.03.
\section{Proofs for Lemmas Supporting the Proof of \cref{thm: generic decomposition}\label{app: generic decomposition pf lemma}}
\subsection{Proof of \cref{thm: generic m estimation unstudentized}, Part (i)}
Throughout, we take $\theta_0(x)=0$ for all $x$ without loss of generality. We begin by giving a high-probability bound on the difference
\begin{equation}
\| R_n(\bm{x}^{(d)}) -U(\bm{x}^{(d)}) \|_\infty
\end{equation}
which will be needed at a later point in the proof. To this end, let $x$ be any component of the vector $\bm{x}^{(d)}$. By a Taylor expansion about $\theta_0(x)$, we have that
\begin{align}
M(x; \hat{\theta}_n(x), \hat{g}_n) - M(x ;\theta_0(x), \hat{g}_n)
& = (\hat{\theta}_n(x) - \theta_0(x)) M^{(1)}(x; \theta_0(x), \hat{g}_n) \nonumber \\
& + (\hat{\theta}_n(x) - \theta_0(x))^2 H(x; \tilde{\theta}_0(x), \hat{g}_n) \label{eq: taylor expand moment}
\end{align}
for some $\tilde{\theta}_0(x)$ between $\hat{\theta}_n(x)$ and $\theta_0(x)$. Moreover, we can write
\begin{align}
&(\hat{\theta}_n(x) - \theta_0(x)) M^{(1)}(x; \theta_0(x), \hat{g}_n)\nonumber \\
& = (\hat{\theta}_n(x) - \theta_0(x)) M^{(1)}(x; \theta_0(x), g_0)\nonumber \\
& + (\hat{\theta}_n(x) - \theta_0(x)) \left(M^{(1)}(x; \theta_0(x), \hat{g}_n) - M^{(1)}(x; \theta_0(x), g_0) \right) \label{eq: M 1 in g}
\end{align}
and
\begin{align}
M(x;\hat{\theta}_n(x),\hat{g}_n) - M(x;\theta_0(x),\hat{g}_n)
& = (M(x;\hat{\theta}_n(x),\hat{g}_n) - M_n(x;\hat{\theta}_n(x),\hat{g}_n,\mathbf{D}_n)) \nonumber\\
& \quad+ (M(x;\theta_0(x),g_0) - M(x;\theta_0(x),\hat{g}_n)) \nonumber\\
& = \left(M(x;\hat{\theta}_n(x),\hat{g}_n) - \mathbb{E}\left[M_n(x;\hat{\theta}_n(x),\hat{g}_n,\mathbf{D}_n)\right]\right) \nonumber\\
& \quad+ \left(\mathbb{E}\left[M_n(x;\hat{\theta}_n(x),\hat{g}_n,\mathbf{D}_n)\right] - M_n(x;\hat{\theta}_n(x),\hat{g}_n,\mathbf{D}_n)\right) \nonumber\\
& \quad+ (M(x;\theta_0(x),g_0) - M(x;\theta_0(x),\hat{g}_n))~.\label{eq: use minimization}
\end{align}
Thus, by the identity
\begin{align}
\bar{M}_n(x;\hat{\theta}_n(x),\hat{g}_n) &= \bar{M}_n(x;\hat{\theta}_n(x),\hat{g}_n) - \bar{M}_n(x;\theta_0(x),\hat{g}_n) \nonumber\\
& + \bar{M}_n(x;\theta_0(x),\hat{g}_n) - \bar{M}_n(x;\theta_0(x),g_0) + \bar{M}_n(x;\theta_0(x),g_0)~,
\end{align}
the equalities \eqref{eq: taylor expand moment}, \eqref{eq: M 1 in g}, and \eqref{eq: use minimization} imply that
\begin{align}
&M^{(1)}(x;\theta_0(x),g_0) (\hat{\theta}_n(x) - \theta_0(x))\nonumber\\
& = M^{(1)}(x;\theta_0(x),\hat{g}_n) (\hat{\theta}_n(x) - \theta_0(x)) \nonumber\\
& - \left(M^{(1)}(x;\theta_0(x),\hat{g}_n) - M^{(1)}(x;\theta_0,g_0)\right) (\hat{\theta}_n(x) - \theta_0(x)) \nonumber\\
& = -\bar{M}_n(x;\theta_0(x),g_0) \label{eq: basic decomposition}\\
& \quad+ \mathsf{Bias}(x;\hat{\theta}_n(x),\hat{g}_n) + \mathsf{Nuis}(x;\theta_0(x),\hat{g}_n) \label{eq: bias and nuis}\\
& \quad+ \mathsf{Stoch}^{(1)}(x; \hat{\theta}_n(x),\hat{g}_n) + \mathsf{Stoch}^{(2)}(x; \hat{g}_n) \label{eq: stoch terms}\\
& \quad- (\hat{\theta}_n(x) - \theta_0(x))^2 H(x; \tilde{\theta}_0(x), \hat{g}_n) \label{eq: theta squared}\\
& \quad- (\hat{\theta}_n(x) - \theta_0(x)) \left(M^{(1)}(x; \theta_0(x), \hat{g}_n) - M^{(1)}(x; \theta_0(x), g_0) \right)\label{eq: theta g term}~
\end{align}
where
\begin{align}
\mathsf{Bias}(x;\hat{\theta}_n(x),\hat{g}_n) &= M(x;\hat{\theta}_n(x),\hat{g}_n) - \mathbb{E}\left[M_n(x;\hat{\theta}_n(x),\hat{g}_n)\right]~,\label{eq: bias def}\\
\mathsf{Nuis}(x;\theta_0(x),\hat{g}_n) &= M(x;\theta_0(x),g_0) - M(x;\theta_0(x),\hat{g}_n)~,\label{eq: nuis def}\\
\mathsf{Stoch}^{(1)}(x; \hat{\theta}_n(x),\hat{g}_n) &= \bar{M}_n(x;\hat{\theta}_n(x),\hat{g}_n) - \bar{M}_n(x;\theta_0(x),\hat{g}_n)~,\quad\text{and}\label{eq: stoch 1 def}\\
\mathsf{Stoch}^{(2)}(x; \hat{g}_n)&= M(x;\theta_0(x),g_0) - M(x;\theta_0(x),\hat{g}_n)~,\label{eq: stoch 2 def}
\end{align}
respectively.
We now give bounds for the terms \eqref{eq: bias and nuis}, \eqref{eq: stoch terms}, \eqref{eq: theta squared}, and \eqref{eq: theta g term}. To handle \eqref{eq: bias and nuis}, observe that Parts (i) and (ii) of \cref{assu: bias} imply that
\begin{equation}
\sqrt{ \frac{n}{ \underline{\lambda}^2 } } \vert \mathsf{Bias}(\bm{x}^{(d)};\hat{\theta}_n(x),\hat{g}_n)\vert
\lesssim (1 + \|\hat{\theta}_n(\bm{x}^{(d)})\|_\infty ) \delta_{n,B}
\lesssim \delta_{n,B}\left(1 + \frac{\underline{\lambda}^{1/2}}{n^{1/4}} \delta_{n,\theta}\right)\label{eq: proof bias bound A}
\end{equation}
with probability greater than $1-\rho_{n,\theta}$. Moreover, a Taylor expansion, second-order smoothness, i.e., \cref{assu: app a moment function}, and \cref{assu: bias}, Part (ii), give that
\begin{align}
\sqrt{ \frac{n}{ \underline{\lambda}^2 } } \mathsf{Nuis}(\bm{x}^{(d)};\theta_0(x),\hat{g}_n) & = \sqrt{ \frac{n}{ \underline{\lambda}^2 } } \partial_g M(\bm{x}^{(d)}; \theta_0(x), g_0)[g-g_0] \nonumber\\
&+ \sqrt{ \frac{n}{ \underline{\lambda}^2 } } \partial_{g,g} M(\bm{x}^{(d)}; \theta_0(x), g_0)[g-g_0] \lesssim \sqrt{ \frac{n}{ \underline{\lambda}^2 } } \|g - g_0\|_{2,\infty}^2 \lesssim \delta^2_{n,g}~.\label{eq: neyman orth bound}
\end{align}
with probability greater than $1-\rho_{n,g}$.
Next, we handle the term \eqref{eq: stoch terms}. By \cref{assu: empirical smoothness}, a Taylor expansion gives
\begin{align}
\mathsf{Stoch}^{(1)}(x; \hat{\theta}_n(x),\hat{g}_n) & =
(\hat{\theta}_n(x) - \theta_0(x)) \bar{M}_n^{(1)}(x; \theta_0(x), \hat{g}_n) \nonumber \\
& + (\hat{\theta}_n(x) - \theta_0(x))^2 \bar{H}_n(x; \tilde{\theta}_0(x), \hat{g}_n) \label{eq: taylor expand empirical moment}
\end{align}
for some, potentially different, $\tilde{\theta}_0(x)$ between $\hat{\theta}_n(x)$ and $\theta_0(x)$. To bound this term, observe that
\begin{align}
&\vert \bar{M}_n^{(1)}(x; \theta_0(x), \hat{g}_n) \vert \leq
\vert \bar{M}_n^{(1)}(x; \theta_0(x), g_0)\vert
+\vert \bar{M}_n^{(1)}(x; \theta_0(x), \hat{g}_n) - \bar{M}_n^{(1)}(x; \theta_0(x), g_0)\vert~.
\end{align}
Hence, \cref{assu: empirical smoothness} and \cref{assu: bias}, Parts (ii) and (iii) imply that
\begin{equation}\label{eq: Stoch 1 bound}
\sqrt{ \frac{n}{ \underline{\lambda}^2 } }
\| \mathsf{Stoch}^{(1)}(\bm{x}^{(d)}; \hat{\theta}_n(\bm{x}^{(d)}),\hat{g}_n)\|_\infty
\leq
\delta_{n,m}\delta_{n,\theta} + \frac{\underline{\lambda}^{1/2} }{ n^{1/4}} \delta_{n,\theta} \delta_{n,J} + \delta^2_{n,\theta}
\end{equation}
with probability greater than $1 - \rho_{n,m} - \rho_{n,\theta} - \rho_{n,J}$. Moreover, we have that
\begin{equation}\label{eq: Stoch 2 bound}
\sqrt{ \frac{n}{ \underline{\lambda}^2 } } \| \mathsf{Stoch}^{(2)}(\bm{x}^{(d)}; \hat{g}_n)\|_\infty \leq \delta_{n,S}
\end{equation}
with probability greater than $1-\rho_{n,S}$.
Finally, we handle the terms \eqref{eq: theta squared} and \eqref{eq: theta g term}. \cref{assu: bias}, Part (ii), and \cref{assu: app a moment function} imply that
\begin{align}
\sqrt{ \frac{n}{ \underline{\lambda}^2 } } \|(\hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}))^2 H(\bm{x}^{(d)} ; \tilde{\theta}_0(\bm{x}^{(d)}), \hat{g}_n)\|_\infty
&\lesssim \delta^2_{n,\theta} \quad\text{and}\label{eq: theta 2 hessian bound}\\
\sqrt{ \frac{n}{ \underline{\lambda}^2 } } \|(\hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)})) (M^{(1)}(\bm{x}^{(d)}; \theta_0(x), \hat{g}_n) - M^{(1)}(\bm{x}^{(d)}; \theta_0(x), g_0)) \|_\infty
&\lesssim \delta_{n,\theta} \delta_{n,g}\label{eq: theta jac bound}
\end{align}
with probabilities greater than $1-\rho_{n,\theta}$ and $1-\rho_{n,g}$, respectively. Putting the pieces together, the decomposition \eqref{eq: basic decomposition} and the lower-boundedness of the Jacobian $M^{(1)}(\cdot;\theta,g)$ imply that
\begin{align}\label{eq: reduce to u stat}
&\sqrt{ \frac{n}{ \underline{\lambda}^2 } } \| R_n(\bm{x}^{(d)}) -U(\bm{x}^{(d)}) \|_\infty \\
& \lesssim \delta^2_{n,g} + \delta^2_{n,\theta} + \delta_{n,B} + \delta_{n,S}+ \delta_{n,\theta}\left(\delta_{n,m} + \delta_{n,g} + \underline{\lambda}^{1/2}n^{-1/4} \left(\delta_{n,B} + \delta_{n,J}\right)\right)\nonumber\\
& \lesssim \delta^2_{n,g} + \delta^2_{n,\theta} + \delta^2_{n,m}
+ \delta_{n,B} + \delta_{n,S}
+ \underline{\lambda}^{1/2}n^{-1/4} \delta_{n,\theta}\left(\delta_{n,B} + \delta_{n,J}\right)\nonumber
\end{align}
with probability greater than $1-\rho_{n,m} + \rho_{n,g} + \rho_{n,\theta} + \rho_{n,S} + \rho_{n,J}$, by the bounds \eqref{eq: proof bias bound A}, \eqref{eq: neyman orth bound}, \eqref{eq: Stoch 1 bound}, \eqref{eq: Stoch 2 bound}, \eqref{eq: theta 2 hessian bound}, and \eqref{eq: theta jac bound}.
With this in place, we turn to the proof of the normal approximation to $R_{n}(\bm{x}^{(d)})$ on hyper-rectangles. Fix a rectangle $\mathsf{R}=[a_{l},a_{u}]$ in $\mathcal{R}$. For the sake of exposition, we give the details of the proof of the upper bound
\begin{align}
P\left\{\sqrt{n} R_{n}(\bm{x}^{(d)})\in\mathsf{R}\right\} -P\left\{Z\in\mathsf{R} \right\}
\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n\sqrt{\log(d)} + \rho_n~.
\end{align}
The matching lower bound will follow from a very similar argument. We comment on the differences where they occur. Consider the decomposition
\begin{flalign}
R_n(x) & = \left(\frac{1}{n}\sum_{i=1}^n \bar{u}(x, D_i) - U_n(x)\right) - \frac{1}{n}\sum_{i=1}^n \bar{u}(x, D_i) + \Delta_n(x),\quad\text{where}\label{eq: R decomp}\\
\Delta_n(x) & = R_n(x) -U(x)\label{eq: Delta def}~,
\end{flalign}
and we recall that the function $\bar{u}(\cdot,\cdot)$ is defined in \cref{assu: linearity}. Define the normalized functions
\begin{equation*}
\hat{u}(\bm{x}^{(d)}, D_i) = \Lambda^{-1/2} \bar{u}(\bm{x}^{(d)}, D_i)\quad\text{and}\quad\hat{U}(\bm{x}^{(d)}) = \Lambda^{-1/2} U(\bm{x}^{(d)})
\end{equation*}
and the analogously normalized rectangle $\tilde{\mathsf{R}} = [\Lambda^{-1/2}a_l, \Lambda^{-1/2}a_u]$ and the enlarged rectangle $\tilde{\mathsf{R}}_t = [\Lambda^{-1/2}a_l - \mathbf{1}_dt, \Lambda^{-1/2}a_u + \mathbf{1}_dt]$. Observe that the decomposition \eqref{eq: R decomp} yields the upper bound
\begin{flalign}
& P\left\{\sqrt{n} R_{n}(\bm{x}^{(d)})\in\mathsf{R}\right\} -P\left\{Z\in\mathsf{R} \right\} \nonumber \\
& = P\left\{\sqrt{n} \Lambda^{-1/2} R_{n}(\bm{x}^{(d)})\in\tilde{\mathsf{R}}\right\} -P\left\{\Lambda^{-1/2}Z\in\tilde{\mathsf{R}} \right\} \nonumber \\
& \quad \leq\Big\vert P\left\{ \frac{1}{\sqrt{n}}\sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i) \in \tilde{\mathsf{R}}_t \right\}
-P\left\{\Lambda^{-1/2}Z\in \tilde{\mathsf{R}}_t \right\} \Big\vert \label{eq: linear clt}\\
& \quad\quad +\Big\vert P\left\{\Lambda^{-1/2}Z\in\tilde{\mathsf{R}_{t}}\right\}
-P\left\{\Lambda^{-1/2}Z\in\tilde{\mathsf{R}}\right\} \Big\vert\label{eq: normal diff expansion} \\
& \quad\quad +P\left\{\sqrt{n}\|\hat{U}_n(\bm{x}^{(d)}) - \frac{1}{n} \sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i) \|_\infty \ge \frac{1}{2}t \right\} \label{eq: term linear}\\
& \quad\quad +P\left\{\sqrt{n} \| \Lambda^{-1/2} \Delta_n(\bm{x}^{(d)})\|_\infty \ge \frac{1}{2}t \right\} \label{eq: term delta}
\end{flalign}
for each $t>0$. An analogous lower bound follows by considering a shrunken rectangle $[\Lambda^{-1/2}a_l + \mathbf{1}_dt, \Lambda^{-1/2}a_u - \mathbf{1}_dt]$. We proceed by providing appropriate bounds for the terms \eqref{eq: linear clt} through \eqref{eq: term delta}.
We begin by bounding the normal approximation term \eqref{eq: linear clt} through the application of the following quantitative central limit theorem,
stated as Theorem 2.1 of \citet{chernozhuokov2022improved}.
\begin{lemma}[{\citealp[Theorem 2.1, ][]{chernozhuokov2022improved}}]
\label{lem: baseline clt}Let $X_{1},\ldots,X_{n}$ be a collection
of independent, centered, random vectors in $\mathbb{R}^{d}$ and
let $Z$ be a centered Gaussian random vector with covariance matrix
\begin{equation}
\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[X_{i}X_{i}^{\top}\right].
\end{equation}
If there exist absolute constants $c$, $C_1$, and $\varphi$ such that the bounds
\begin{equation}
\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[X_{i,j}^{2}\right]\geq c~,\quad
\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[X_{i,j}^{4}\right]\leq C_1 \varphi^{2}~,
\quad\text{and}\quad
\|X_{i,j} \|_{\psi_1} \leq \varphi \label{eq: general clt moment bounds}
\end{equation}
hold, then the inequality
\begin{equation}
\sup_{\mathsf{R}\in\mathcal{R}}
\Big\vert P\left\{ \frac{1}{\sqrt{n}}\sum_{i=1}^{n}X_{i}\in \mathsf{R}\right\}
-P\left\{ \sqrt{n}Z\in \mathsf{R}\right\} \Big\vert\leq C_2 \left(\frac{\varphi^{2}\log^{5}(dn)}{n}\right)^{1/4}
\end{equation}
holds for some constant $C_2$ that depends only on $c$ and $C_1$.
\end{lemma}
\noindent In particular, observe that
\begin{equation}
\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\hat{u}^{2}(x^{(j)}, D_{i})\right]=1\label{eq: standardized}
\end{equation}
by definition and that
\begin{equation}
\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\hat{u}^{4}(x^{(j)},D_{i})\right]
\leq \left(\varphi/\underline{\lambda}\right)^{2}\label{eq: fourth bound}
\end{equation}
by \cref{assu: linearity}. Similarly we have that
\begin{equation}
\| \hat{u}\left(x^{j}), D_{i}\right) \|_{\psi_1}
\leq (\varphi /\underline{\lambda})
\end{equation}
by \cref{assu: linearity}. Consequently, as
\begin{equation}
\Var(\hat{u}(x^{(j)}, D_{i}))=\Lambda^{-1/2}\Var\left(Z\right)\Lambda^{-1/2},
\end{equation}
by definition, Lemma \ref{lem: baseline clt} implies that the bound
\begin{equation}
\Big\vert P\left\{ \frac{1}{\sqrt{n}}\sum_{i=1}^{n}\hat{u}\left(D_{i}\right)\in \mathsf{\tilde{R}}_t\right\}
-P\left\{ \Lambda^{-1/2}Z\in \mathsf{\tilde{R}}_t\right\} \Big\vert
\lesssim \left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}~.\label{eq: baseline clt applied}
\end{equation}
holds.
In turn, to bound the term \eqref{eq: normal diff expansion}, we have that
\begin{equation}\label{eq: nazarov basic decompositon}
\Big\vert P\left\{\Lambda^{-1/2}Z\in\mathsf{R}_{t}\right\}
-P\left\{\Lambda^{-1/2}Z\in\mathsf{R}\right\} \Big\vert \lesssim t \sqrt{\log d}~.
\end{equation}
by \cref{lem: Nazarov}. Moreover, to bound the term \eqref{eq: term linear}, recall that
\begin{equation}\label{eq: linearity basic decompositon}
P\left\{\sqrt{n}\|\hat{U}_n(\bm{x}^{(d)}) - \frac{1}{n} \sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i) \|_\infty \ge \delta_{n,u} \right\}
\leq \rho_{n,u}
\end{equation}
by \cref{assu: linearity}.
Thus, by choosing $t=C\delta_n$, the bound \eqref{eq: reduce to u stat} implies that the sum of term \eqref{eq: term linear} and term \eqref{eq: term delta} is
upper bounded by $C\rho_{n}$. Hence, by plugging this choice of $t$ into \eqref{eq: nazarov basic decompositon} and \eqref{eq: linearity basic decompositon}, we can conclude that
\begin{equation}
\Big\vert P\left\{\sqrt{n} R_{n}(\bm{x}^{(d)})\in\mathsf{R}\right\} -P\left\{ Z\in\mathsf{R}\right\} \Big\vert
\lesssim \left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^{2}n}\right)^{1/4}
+ \delta_n\sqrt{\log(d)} + \rho_n~,
\end{equation}
as required.\hfill\qed
\subsection{Proof of \cref{thm: generic m estimation unstudentized}, Part (ii)}
The result follows from an argument whose structure is similar to the proof of \cref{thm: generic m estimation unstudentized}, Part (i). Again, we take $\theta_0(x)=0$ for all $x$, without loss of generality. We are interested in studying the discrepancy
\begin{flalign*}
R_{n}^{*}(x) & =\hat{\theta}_\mathsf{h}(x)-\hat{\theta}_n(x) = (\hat{\theta}_\mathsf{h}(x)-\theta_{0}(x))-R_{n}(x).
\end{flalign*}
In terms of the decomposition \eqref{eq: R decomp}, we can write
\begin{flalign} \label{eq: re R decomp}
R_n(x) = \left(\frac{1}{n}\sum_{i=1}^n \bar{u}(x, D_i) - U_n(x)\right) - \frac{1}{n}\sum_{i=1}^n \bar{u}(x, D_i) + \Delta_n(x)~,
\end{flalign}
where $\Delta_n(x)$ is defined in \eqref{eq: Delta def}. On the other hand, as $\hat{\theta}_\mathsf{h}(x)$ is constructed with a random half-sample $\mathsf{h}$ of the data $\mathbf{D}_{n}$,
we have that
\begin{flalign} \label{eq: R* decomp boot}
\hat{\theta}_\mathsf{h}(x)-\theta_{0}(x)
= \left(\frac{2}{n}\sum_{i\in\mathsf{h}}^n \bar{u}(x, D_i) - U_\mathsf{h}(x)\right) - \frac{2}{n}\sum_{i\in\mathsf{h}}^n \bar{u}(x, D_i) + \Delta_\mathsf{h}(x)~,
\end{flalign}
where $U_\mathsf{h}(x)$ and $\Delta_\mathsf{h}(x)$ are constructed with the half-sample $\mathsf{h}$.
The proof of \cref{thm: generic m estimation unstudentized}, Part (i), worked by giving a high probability bound for the first and third term in \eqref{eq: re R decomp} and showing that the the second term satisfies a central limit theorem. Here, as we are interested in giving a bound conditioned on the data $\mathbf{D}_n$, we show that the difference between the second terms in \eqref{eq: re R decomp} and \eqref{eq: R* decomp boot} satisfies a central limit theorem on the event that the first and third terms in \eqref{eq: re R decomp} and \eqref{eq: R* decomp boot} satisfy a specified bound, which we show holds with high probability. In particular, let $\mathcal{F}_n(t)$ and $\mathcal{F}_\mathsf{h}(t)$ denote the events that
\begin{equation}\label{eq: F event}
\sqrt{\frac{n}{\underline{\lambda}^2}} \| \Delta_n(\bm{x}^{(d)})\|_\infty \leq t/4
\quad\text{and}\quad
\sqrt{\frac{n}{\underline{\lambda}^2}} \| \Delta_\mathsf{h}(\bm{x}^{(d)})\|_\infty \leq t/4~,
\end{equation}
respectively. Similarly, let $\mathcal{H}_n(t)$ and $\mathcal{H}_\mathsf{h}(t)$ denote the events that
\begin{flalign}\label{eq: H event}
&\sqrt{n}\| \frac{1}{n}\sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i) - \hat{U}_n(\bm{x}^{(d)}) \|_\infty \leq t/4
\quad\text{and}\\
&\sqrt{n}\| \frac{2}{n}\sum_{i\in\mathsf{h}}^n \hat{u}(\bm{x}^{(d)}, D_i) - \hat{U}_\mathsf{h}(\bm{x}^{(d)}) \|_\infty \leq t/4\nonumber
\end{flalign}
respectively, where $\hat{U}_\mathsf{h}(\bm{x}^{(d)})$ is defined analogously to $\hat{U}_n(\bm{x}^{(d)})$. Define the event $\mathcal{E}_n(t) = \mathcal{F}_n(t) \cap \mathcal{F}_\mathsf{h}(t) \cap \mathcal{H}_n(t) \cap \mathcal{H}_\mathsf{h}(t)$. Fix a hyper-rectangle $\mathsf{R}$ in $\mathcal{R}$. As before, we prove only the requisite upper bound. The lower bound follows from an analogous argument. Recall the definitions of the transformed rectangle $\tilde{\mathsf{R}}$ and the enlarged transformed rectangle $\tilde{\mathsf{R}}_t$. On the event $\mathcal{E}_n(t) $, we have
\begin{flalign}
& P\left\{\sqrt{n} R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\} - P\left\{ Z\in\mathsf{R}\right\} \nonumber \\
& = P\left\{\sqrt{n} \Lambda^{-1/2} R_{n}^{*}(\bm{x}^{(d)})\in\tilde{\mathsf{R}}\mid\mathbf{D}_{n}\right\} - P\left\{\Lambda^{-1/2} Z\in\tilde{\mathsf{R}}\right\} \nonumber \\
& \quad\leq P\left\{ \frac{2}{\sqrt{n}}\sum_{i\in\mathsf{h}}^n \hat{u}(\bm{x}^{(d)}, D_i) - \frac{1}{\sqrt{n}}\sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i)\in\tilde{\mathsf{R}}_{t} \mid\mathbf{D}_{n}\right\}
-P\left\{ \Lambda^{-1/2}Z\in\tilde{\mathsf{R}}_{t}\right\} \nonumber \\
& \quad+\vert P\left\{ \Lambda^{-1/2}Z\in\tilde{\mathsf{R}}\right\} -P\left\{ \Lambda^{-1/2}Z\in\tilde{\mathsf{R}}_t\right\} \vert\nonumber
\end{flalign}
for each $t>0$. As the data $D_\mathsf{h}$ are drawn independently and identically with distribution $P$ in $\mathbf{P}$ and we have assumed that $\delta_{n/2}\lesssim \delta_n$ and $\rho_{n/2}\lesssim \rho_n$, by setting $t = C\delta_n$, \cref{assu: linearity} and the bound \eqref{eq: reduce to u stat} imply that the event $\mathcal{E}_n(t)$ occurs with probability greater than $1-C\rho_n$. Thus, \cref{lem: Nazarov} implies that
\begin{flalign}
& P\left\{\sqrt{n} R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\} - P\left\{ Z\in\mathsf{R}\right\} \label{eq: up to u stat}\\
& \quad
\leq
P\left\{ \frac{2}{\sqrt{n}}\sum_{i\in\mathsf{h}}^n \hat{u}(\bm{x}^{(d)}, D_i)
- \frac{1}{\sqrt{n}}\sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i)\in\tilde{\mathsf{R}}_{t} \mid\mathbf{D}_{n}\right\}
-P\left\{ \Lambda^{1/2}Z\in\tilde{\mathsf{R}}_{t}\right\} \label{eq: to u stat CLT}\\
& \quad\quad+ \delta_n \sqrt{\log(d)} \nonumber
\end{flalign}
with probability greater than $1-C\rho_n$. Hence, it suffices to bound the term \eqref{eq: to u stat CLT}.
To this end, we apply a coupling argument introduced in \cite{yadlowsky2023evaluating}, which is similar to a Poissonization technique studied in \cite{praestgaard1993exchangeably} (see also Section 3.6.2 of \cite{van1996weak}). In particular, let $V_{i}$ be a random variable taking the value $1$ when $i$
is an element of the subset $\mathsf{h}$ and taking the value $-1$ otherwise. Observe that
\begin{flalign}\label{eq: binomialization}
\frac{2}{n}\sum_{i\in\mathsf{h}} \hat{u}(\bm{x}^{(d)}, D_i) - \frac{1}{n}\sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i) = \frac{1}{n} \sum_{i=1}^n V_i \hat{u}(\bm{x}^{(d)}, D_i)~.
\end{flalign}
Let $\tilde{V}_{1},\ldots,\tilde{V}_{n}$ denote a collection of random
variables valued on $\{-1,1\}$. We define their joint distribution
as follows. Let $Q_{n}$ denote a random variable with distribution
$\mathsf{Bin}\left(n,1/2\right)$. If $Q_{n}\geq n/2$, then choose $Q_{n}-n/2$
indices $i$ in $[n]$ with $V_{i}=-1$ and set $\tilde{V}_{i}=1$.
If $Q_{n}<n/2$, then choose $n/2-Q_{n}$ indices with $V_{i}=1$
and set $\tilde{V}_{i}=-1$. Set $\tilde{V}_{i}=V_{i}$ for all other
units. Observe that the collection $\tilde{V}_{i}$ are independent
and identically distributed Rademacher random variables. With this in place,
we obtain the decomposition
\begin{equation}\label{eq: binom approx}
\frac{1}{n}\sum_{i=1}^{n}V_{i}\hat{u}(\bm{x}^{(d)}, D_i)
=\frac{1}{n}\sum_{i=1}^{n}\tilde{V}_{i} \hat{u}(\bm{x}^{(d)}, D_i)
+\frac{1}{n}\sum_{i=1}^{n}\left(V_{i}-\tilde{V}_{i}\right)\hat{u}(\bm{x}^{(d)}, D_i)~.
\end{equation}
Let $\mathcal{V}(t)$ denote the event that
\begin{equation}
\sqrt{n} \Big\| \frac{1}{n}\sum_{i=1}^{n}\left(V_{i}-\tilde{V}_{i}\right)\hat{u}(\bm{x}^{(d)}, D_i) \Big\|_\infty > t~.
\end{equation}
Fix any rectangle $\mathsf{R}^\prime = [a^\prime_l, a^\prime_u]$ and define the enlarged rectangle $\mathsf{R}^\prime_t = [a_l - t\mathbf{1}_d,a_u + t\mathbf{1}_d]$.
On the event $\mathcal{V}(t)$, the decomposition \eqref{eq: binom approx} implies that
\begin{flalign}
& P\left\{ \frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i}\hat{u}(\bm{x}^{(d)}, D_i) \in \mathsf{R}^\prime \mid \mathbf{D}_n \right\}
- P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}^\prime \right\} \nonumber\\
& \leq P\left\{\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\tilde{V}_{i} \hat{u}(\bm{x}^{(d)}, D_i) \in \tilde{\mathsf{R}}_t \mid \mathbf{D}_n \right\}
- P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}^\prime_t\right\} \label{eq: to mult clt}\\
& \quad\quad + \vert P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}^\prime_{t}\right\} -P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}^\prime\right\} \vert\label{eq: binom nazarov}
\end{flalign}
for all $t>0$. \cref{lem: Nazarov} implies that \eqref{eq: binom nazarov} is less than $t\sqrt{\log(d)}$. To handle the term \eqref{eq: to mult clt}, we apply the following quantitative central limit theorem, stated as Lemma 4.6 in \citet{chernozhuokov2022improved}.
\begin{lemma}[{\citealp[Lemma 4.6, ][]{chernozhuokov2022improved}}]
\label{lem: multiplier bootstrap clt}Consider the setting and assumptions
of \cref{lem: baseline clt}. Let $\bm{X}_{n}=(X_{1},\ldots,X_{n})$
collect the observed data and let $\tilde{V}_{1},\ldots,\tilde{V}_{n}$ be a collection
of independent Rademacher random variables. We have that
\begin{equation}
\sup_{\mathsf{R}\in\mathcal{R}}\Big\vert P\left\{ n^{-1/2}\sum_{i=1}^{n}X_{i}\in \mathsf{R}\right\}
-P\left\{ n^{-1/2}\sum_{i=1}^{n}\tilde{V}_{i}X_{i}\in R\mid\bm{X}_{n}\right\} \Big\vert
\leq C_2 \left(\frac{\varphi^{2}\log^{5}(dn)}{n}\right)^{1/4}.
\end{equation}
with probability greater than
\begin{equation}
1-C\frac{\varphi \log^{3/2}(dn)}{n^{1/2}}
\end{equation}
for some constant $C_2$ that depends only on the constants $C_2$ and $c$ defined in the statement of \cref{lem: baseline clt}.
\end{lemma}
\noindent Thus, on the event $\mathcal{V}(t)$, \cref{lem: baseline clt} and \cref{lem: multiplier bootstrap clt} imply that
\begin{flalign*}
P\left\{ \frac{1}{\sqrt{n}}\sum_{i=1}^{n}V_{i}\hat{u}(\bm{x}^{(d)}; D_i) \in \mathsf{R}^\prime \mid \mathbf{D}_n \right\}
- P\left\{ \Lambda^{-1/2} Z\in\mathsf{R}^\prime\right\}
\lesssim \left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^2 n}\right)^{1/4} + t\sqrt{\log(d)}~,
\end{flalign*}
with probability greater than $1-Cn^{-1/2}\underline{\lambda}^{-1}\varphi \log^{3/2}(dn)$. Hence, it suffices to give a high probability bound on $\mathcal{V}(t)$ for a suitable choice of $t$.
To this end, observe that
\begin{flalign}
G_{n} & =\Big\|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\left(V_{i}-\tilde{V}_{i}\right)\hat{u}(\bm{x}^{(d)}; D_i)\Big\|_{\infty}
\quad
\text{is equidistributed with}
\quad\\
&\quad \Big\|\frac{2}{\sqrt{n}}\sum_{i=1}^{\vert Q_{n} - n/2\vert}\hat{u}(\bm{x}^{(d)};D_{i})\Big\|_{\infty}~.
\end{flalign}
Consider the decomposition
\begin{flalign}
P\left\{ G_{n}\geq t \right\}
& \leq P\left\{ G_{n}\geq t,\vert Q_{n} - n/2 \vert \leq\delta\frac{n}{2}\right\} +P\left\{ \vert Q_{n} - n/2 \vert \geq\delta\frac{n}{2}\right\} \nonumber \\
& \leq P\left\{ \max_{1\leq k\leq\delta\frac{n}{2}}\Big\|\frac{2}{\sqrt{n}}\sum_{i=1}^{k}\hat{u}\left(D_{i}\right)\Big\|_{\infty}\geq t\right\}
+P\left\{ \vert Q_{n} - n/2\vert \geq\delta\frac{n}{2}\right\} ,\label{eq: set up for maximal}
\end{flalign}
for some $\delta>0$ to be chosen. Observe that
\begin{equation}
P\left\{ \vert Q_{n} - n/2\vert \geq\frac{\delta n}{2}\right\} \le2\exp\left(-\frac{\delta^{2}n}{6}\right)\label{eq: multiplicative chernoff}
\end{equation}
by the multiplicative Chernoff bound. To bound the first term in (\ref{eq: set up for maximal}), we combine two inequalities. The first inequality is the following standard Bernstein-type bound, stated, e.g., in \cite{song2019approximating}.
\begin{lemma}[Lemma A.2, \cite{song2019approximating}]
\label{lem: orlicz large deviation}Let $Z_{1},\ldots,Z_{n}$ be independent,
centered, random vectors in $\mathbb{R}^{d}$. Define the quantity
\begin{equation}
\sigma^2 = \max_{j\in[d]} \sum_{i=1}^n \mathbb{E}\left[Z^2_{i,j}\right]
\end{equation}
and assume that $\|Z_{ij}\|_{\psi_{1}}\leq \phi $ for all $i\in[n]$ and $j\in[d]$.
The inequality
\begin{flalign*}
& P\left\{ \Big\|\sum_{i=1}^{n}Z_{i}\Big\|_{\infty}
\geq C \left(\sigma \log^{1/2}(dg) + \phi \log(dn)\left(\log(dn) + \log(g)\right)\right) \right\}
\lesssim \frac{1}{g}
\end{flalign*}
holds for any constant $g>0$.
\end{lemma}
\noindent The second inequality is a L\'{e}vy type inequality for independent random vectors, due to \cite{montgomery1993comparison}. See Chapter 1 of \cite{de2012decoupling} for a textbook treatment.
\begin{lemma}[Theorem 1.1.5, \cite{de2012decoupling}] \label{lem: independent levy}
Let $Z_{1},\ldots,Z_{n}$ be independent random vectors in $\mathbb{R}^{d}$. There exists a universal constants $C_1$ and $C_2$ such that
\begin{equation}
P\left\{ \max_{1\leq k \leq n} \| \sum_{i=1}^k Z_i \|_\infty > t \right\} \leq C_1 P\left\{ \| \sum_{i=1}^n Z_i \|_\infty > \frac{t}{C_2} \right\}
\end{equation}
for all $t>0$.
\end{lemma}
\noindent
In particular, as
\begin{equation*}
\|\frac{2}{\sqrt{n}}\hat{u}(x^{(j)}; D_{i}) \|_{\psi_{1}}
\leq\frac{2}{\sqrt{n}}\frac{\varphi}{\underline{\lambda}}~,
\end{equation*}
and
\begin{equation}
\max_{j \in [d]} \sum_{k=1}^{\delta n / 2} \mathbb{E}\left[ \frac{2}{\sqrt{n}}\hat{u}(x^{(j)}; D_{i}) \right] = 2\delta~,
\end{equation}
\cref{lem: orlicz large deviation} and \cref{lem: independent levy} imply that
\begin{align}\label{eq: apply levy orlicz}
P\left\{ \max_{1\leq k\leq\delta\frac{n}{2}}\Big\|\frac{2}{\sqrt{n}}\sum_{i=1}^{k}\hat{u}\left(\bm{x}^{(d)};D_{i}\right)\Big\|_{\infty}
\geq C \left(\delta \log^{1/2}(dn) + \frac{2}{\sqrt{n}}\frac{\varphi}{\underline{\lambda}} \log^2(dn)\right) \right\} \lesssim \frac{1}{n}~.
\end{align}
Now, the choice
\begin{equation}
\delta=C \sqrt{\frac{\log n}{n}}\label{eq: delta choice}
\end{equation}
gives
\begin{equation}
P\left\{ Q_{n}\geq\frac{\delta n}{2}\right\} \lesssim \frac{1}{n}\label{eq: binomial choice}
\end{equation}
by (\ref{eq: multiplicative chernoff}). Plugging this choice into \eqref{eq: apply levy orlicz} yields
\begin{flalign}\label{eq: HJ choice}
P\left\{ \max_{1\leq k\leq\delta\frac{n}{2}}\Big\|\frac{2}{\sqrt{n}}\sum_{i=1}^{k}\hat{u}\left(\bm{x}^{(d)};D_{i}\right)\Big\|_{\infty}
\geq C \frac{1}{\sqrt{n}}\frac{\varphi}{\underline{\lambda}} \log^2(dn) \right\} \lesssim \frac{1}{n}~.
\end{flalign}
Hence, we find that the inequality
\begin{flalign*}
P\left\{ \Big\|\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\left(V_{i}-\tilde{V}_{i}\right)\hat{u}(\bm{x}^{(d)}, D_{i})\Big\|_{\infty}
\geq C
\left(\frac{\varphi^2\log^{4}\left(dn\right)}{\underline{\lambda}^2 n}\right)^{1/2} \right\} & \lesssim \frac{1}{n}
\end{flalign*}
holds for all $n$ sufficiently large.
Thus, by setting
\begin{equation*}
t = C \left( \frac{\varphi^2\log^{4}\left(dn\right)}{\underline{\lambda}^2 n}\right)^{1/2}~,
\end{equation*}
we find that
\begin{align}
& P\left\{ \frac{1}{n}\sum_{i=1}^{n}V_{i}\hat{u}(\bm{x}^{(d)}, D_i) \in \mathsf{R}^{\prime} \mid \mathbf{D}_n \right\}
- P\left\{ \Lambda^{-1/2} Z\in \mathsf{R}^\prime \right\} \nonumber\\
& \quad\quad\quad\quad
\lesssim
\left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^2 n}\right)^{1/4}
+ \sqrt{\log (d)}\left( \frac{\varphi^2 \log^{4}\left(dn\right)}{\underline{\lambda}^2 n}\right)^{1/2}\\
& \quad\quad\quad\quad
\lesssim
\left( \frac{\varphi^2 \log^{5}\left(dn\right)}{\underline{\lambda}^2 n}\right)^{1/4}~,\label{eq: binom end}
\end{align}
with probability greater than $1-C n^{-1/2}\underline{\lambda}^{-1}\varphi \log^{3/2}(dn)$. Putting the pieces together, the inequalities \eqref{eq: up to u stat} and \eqref{eq: binom end} imply that
\begin{flalign*}
& P\left\{\sqrt{n} R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\} - P\left\{ Z\in\mathsf{R}\right\}
\lesssim
\left( \frac{\varphi^2 \log^{5}\left(dn\right)}{\underline{\lambda}^2 n}\right)^{1/4}+ \delta_n\sqrt{\log(d)}
\end{flalign*}
with probability greater than $1-C(n^{-1/2}\varphi \underline{\lambda}^{-1} \log^{3/2}(dn) + \rho_n)$, as required.\hfill\qed
\subsection{Proof of \cref{lem: variance accuracy}}
Again, we take $\theta_0(x)=0$ for all $x$, without loss of generality. Recall from the proof of \cref{thm: generic m estimation unstudentized}, Part (ii), that
$V_{i}$ is a random variable taking the value $1$ when $i$
is an element of the subset $\mathsf{h}$, and taking the value $-1$ otherwise, and that
\begin{flalign}\label{eq: re binomialization}
\frac{2}{n}\sum_{i\in\mathsf{h}}^n \bar{u}(x, D_i) - \frac{1}{n}\sum_{i=1}^n \bar{u}(x, D_i) = \frac{1}{n} \sum_{i=1}^n V_i \bar{u}(x, D_i)~.
\end{flalign}
To ease notation, define the objects
\begin{align*}\label{eq: Q term def}
T_n(x) &= \frac{1}{n}\sum_{i=1}^n \bar{u}(x, D_i) - U_n(x)\quad\text{and}\quad
T_\mathsf{h}(x)= \frac{2}{n}\sum_{i\in\mathsf{h}} \bar{u}(x, D_i) - U_\mathsf{h}(x)~.
\end{align*}
We are interested in studying
\begin{align*}
\hat{\lambda}^2_{n,j}
& = n\mathbb{E}_{V}\left[\left(R^*_n(x^{(j)})\right)^2 \right]\\
& = n\mathbb{E}_{V}\left[\left(\frac{1}{n} \sum_{i=1}^n V_i \bar{u}(x^{(j)}, D_i) + T_\mathsf{h}(x^{(j)}) - T_n(x^{(j)}) + \Delta_\mathsf{h}(x^{(j)}) - \Delta_n(x^{(j)})\right)^2\right]~,
\end{align*}
where the notation $\mathbb{E}_{V}\left[\cdot\right]$ denotes that the expectation is evaluated only over the random variables $V_1,\ldots,V_n$ and we recall that the object $ \Delta_\mathsf{h}(x^{(j)})$ and $\Delta_n(x^{(j)})$ introduced in \eqref{eq: Delta def} and \eqref{eq: R* decomp boot}, respectively. On the event $\mathcal{E}(t)$, defined as the intersection of the events \eqref{eq: F event} and \eqref{eq: H event}, we have that
\begin{flalign*}
& \sup_{j\in[d]} \big\vert \hat{\lambda}^2_{n,j} - \bar{\lambda}^2_{n,j} \big\vert \leq t^2~,
\quad\text{where}\quad
\bar{\lambda}^2_{n,j} = n\mathbb{E}_V\left[\left(\frac{1}{n} \sum_{i=1}^n V_i \bar{u}(x^{(j)}, D_i)\right)^2\right]~.
\end{flalign*}
We can evaluate
\begin{flalign}
\bar{\lambda}^2_{n,j}
&= \frac{1}{n}\sum_{i=1}^n \bar{u}^2(x^{(j)}, D_i) + \frac{1}{n}\sum_{i=1}^n \sum_{i^\prime \neq i} \mathbb{E}\left[V_i V_{i^\prime}\right] \bar{u}(x^{(j)}, D_i) \bar{u}(x^{(j)}, D_{i^\prime})~.\nonumber
\end{flalign}
Observe that
\begin{flalign*}
\mathbb{E}\left[V_i V_{i^\prime}\right]
&= \frac{1}{2} \mathbb{E}\left[V_i \mid V_{i^\prime} = 1\right] - \frac{1}{2} \mathbb{E}\left[V_{i^\prime} \mid V_i = -1\right]\\
&= \frac{1}{2}\left(\frac{n/2-1}{n-1} - \frac{n/2}{n-1}\right) - \frac{1}{2} \left(\frac{n/2}{n-1} - \frac{n/2-1}{n-1}\right) = -\frac{1}{n-1}
\end{flalign*}
and thereby
\begin{equation}\label{eq: bar lambda de randomize}
\bar{\lambda}^2_{n,j}
= \frac{1}{n}\sum_{i=1}^n \bar{u}^2(x^{(j)}, D_i) - \frac{1}{n}\frac{1}{n-1} \sum_{i=1}^n \sum_{i^\prime \neq i} \bar{u}(x^{(j)}, D_i) \bar{u}(x^{(j)}, D_{i^\prime})~.
\end{equation}
Now, observe that the first term in \eqref{eq: bar lambda de randomize} satisfies
\begin{equation*}
\sup_{j\in[d]} \Big\vert \frac{1}{n}\sum_{i=1}^n \bar{u}^2(x^{(j)}, D_i) - \lambda^2_{j} \Big\vert \lesssim \frac{\varphi}{n}\log(dn)
\end{equation*}
with probability greater than $1-n^{-1}$, by Bernstein's inequality (see e.g., Theorem 2.8.1 of \cite{vershynin2018high}). To handle the second term in \eqref{eq: bar lambda de randomize}, we apply the following sub-exponential formulation of the \cite{hanson1971bound} exponential concentration inequality for quadratic forms, due to \cite{gotze2021concentration}.
\begin{lemma}[Proposition 1.1, \cite{gotze2021concentration}]\label{lem: Hanson Wright}
Let $X_1,\ldots,X_n$ be independent, centered, random variables satisfying $\mathbb{E}\left[X_i^2\right] = \sigma_{i}^2$ and $\|X_i\|_{\psi_1}\leq\varphi$. If $A=(a_{i,i^\prime})$ is any symmetric $n\times n$ matrix, then the inequality
\begin{equation}
P\left\{ \big\vert \sum_{i=1}^n \sum_{i^\prime=1}^n a_{i,i^\prime} X_i X_{i^\prime} - \sum_{i=1}^n \sigma_i^2 a_{i,i} \big\vert \geq t \right\}
\leq
2 \exp\left( \frac{1}{C} \min \left(\frac{t^2}{\varphi^4 \|A\|^2_{F}}, \frac{t^{1/2}}{\varphi \|A\|_{\mathsf{op}}^{1/2} }\right) \right)
\end{equation}
holds for any $t\geq0$, where $\| \cdot \|_F$ and $\| \cdot \|_{\mathsf{op}}$ denote the Frobenius and $\ell_2$ operator norms, respectively.
\end{lemma}
\noindent In particular, if $A$ denotes the $n \times n$ matrix with zeroes on the diagonal and $(n(n-1))^{-1}$ in every other entry, then
\begin{align*}
\|A\|^2_{F} & = \frac{1}{n}\frac{1}{n-1}\quad\text{and}\quad
\|A\|^{1/2}_{\mathsf{op}} = \frac{1}{n}
\end{align*}
and so \cref{lem: Hanson Wright} implies that
\begin{equation*}
\sup_{j\in[d]} \Big\vert \frac{1}{n}\frac{1}{n-1} \sum_{i=1}^n \sum_{i^\prime\neq i} \bar{u}(x^{(j)}, D_i) \bar{u}(x^{(j)}, D_{i^\prime})\Big\vert
\lesssim \frac{\varphi^2}{n} \log^2(dn)
\end{equation*}
with probability greater than $1-n^{-1}$. Thus, on the event $\mathcal{E}_n(t)$, we have that
\begin{equation*}\label{eq: variance on E}
\sup_{j\in[d]} \big\vert \hat{\lambda}^2_{n,j} - \lambda^2_{j} \big\vert \lesssim \frac{\varphi^2}{n}\log^2(dn) + t^2
\end{equation*}
with probability greater than $1-Cn^{-1}$, as $\varphi \geq 1$. By setting
\[
t=C \sqrt{\frac{\underline{\lambda}^2}{n}} \delta_n~,
\]
\cref{assu: linearity} and the bound \eqref{eq: reduce to u stat} imply that the event $\mathcal{E}_n(t)$ occurs with probability greater than $1-\rho_n$. Thus, we can conclude that
\begin{equation*}
\sup_{j\in[d]} \big\vert \hat{\lambda}_{n,j}/\lambda_{n,j} - 1\big\vert \lesssim \sup_{j\in[d]} \big\vert \hat{\lambda}^2_{n,j}/\lambda^2_{n,j} - 1\big\vert \lesssim \frac{\varphi^2}{\underline{\lambda}^2n}\log^2(dn) + \frac{1}{n}\delta_n^2
\end{equation*}
with probability greater than $1-C(\rho_n+n^{-1})$, as required.\hfill\qed
\section{Proofs for Lemmas Supporting the Proof of \cref{eq: asymptotic validity subsampled kernel}\label{app: kernel verify pf lemma}}
\subsection{Proof of \cref{lem: u stat linearity}} To ease notation, we define the parameter
\[
\phi(\theta_0) = (1+\|\theta_0(\bm{x}^{(d)})\|)\phi
\]
and drop dependence on $\theta(x)$ and $g$ when writing $u(x; D_{\mathsf{s}},\xi_{\mathsf{s}}, \theta(x), g)$, as these values will be taken to be $\theta_0(x)$ and $g_0$ throughout. We are interested in studying the discrepancy
\begin{equation}
U_n(\bm{x}^{(d)}) - \frac{b}{n}\sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i)~.
\end{equation}
Define the quantities
\begin{flalign}
\hat{U}_{n}(\bm{x}^{(d)})
& =\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}
u(\bm{x}^{(d)}; D_{\mathsf{s}},\xi_{\mathsf{s}})
\quad\text{and}\quad
\bar{U}_{n}(\bm{x}^{(d)}) = \frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}
\tilde{u}(\bm{x}^{(d)}; D_\mathsf{s})\label{eq: complete derandom}
\end{flalign}
for some collection of independent random variables $\bm{\xi} = (\xi_{\mathsf{s}})_{\mathsf{s}\in\mathcal{S}_{n,b}}$
having the same distribution as $\xi$. The statistics $\hat{U}_{n}(\bm{x}^{(d)})$
and $\bar{U}_{n}(\bm{x}^{(d)})$ are the complete, randomized and de-randomized,
$U$-statistics associated with the randomized $d$-dimensional kernel $u(\bm{x}^{(d)};\cdot,\cdot)$,
respectively. Consider the decomposition
\begin{align}
& \sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}} U_n(\bm{x}^{(d)}) - \frac{b}{n}\sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i) \nonumber\\
& \quad\quad= \sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}} (U_{n}(\bm{x}^{(d)})-\hat{U}_{n}(\bm{x}^{(d)}))
+\sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}} (\hat{U}_{n}(\bm{x}^{(d)})-\bar{U}_{n}(\bm{x}^{(d)})) \nonumber \\
& \quad\quad\quad+\sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}} (\bar{U}_{n}(\bm{x}^{(d)})-\frac{b}{n}\sum_{i=1}^{n}\tilde{u}^{(1)}(\bm{x}^{(d)}; D_{i}))~.\label{eq: W tilde decomposition}
\end{align}
A high-probability bound for the third term in \eqref{eq: W tilde decomposition} is obtained by applying \cref{lem: projection maximal}, stated in \cref{sec: app projection}. In particular, \cref{lem: projection maximal} implies that
\begin{align}
\sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}}
& \Big\| \bar{U}_{n}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i) \Big\|_{\infty} \nonumber\\
& \quad\quad \lesssim
\left(\frac{C^\star b\log(dn)}{n}\right)^{b/2}
\left(
\left(\frac{n}{b^2 \underline{\sigma}^2_{b}}\right)^{1/2}
+\left(\frac{b\log^4(dn)}{\underline{\sigma}^2_{b}}\right)^{1/2}
\right) \phi(\theta_0)\label{eq: apply Hajek in main result}
\end{align}
with probability greater than $1-Cn^{-1}$, for some constant $C^\star > 1$. By choosing $C_1 = (C^\star)^{-1} (C^{\star\star})<1$ for some $(C^{\star\star})< 1$, the conditions
\[
\frac{b\log(dn)}{n} \leq C_1 < 1 \quad\text{and}\quad b^{-C_2} \leq \underline{\sigma}^2_b
\]
imply that
\begin{align}
& \left(\frac{C^\star b\log(dn)}{n}\right)^{b/2}
\left(
\left(\frac{n}{b^2 \underline{\sigma}^2_{b}}\right)^{1/2}
+\left(\frac{b\log^4(dn)}{\underline{\sigma}^2_{b}}\right)^{1/2}
\right) \phi(\theta_0)\nonumber \\
& \quad\quad \lesssim
\left(C{\star\star}\right)^{b/2}
\left(
\left(n b^{C_2 - 2}\right)^{1/2} +\left(b^{1+C_2}n^4\right)^{1/2}
\right) \phi(\theta_0)\nonumber \\
& \quad\quad \lesssim
\left(C^{\star\star}\right)^{b/2} n^{(5+C_2)/2} \phi(\theta_0)~.\label{eq: plug in normalization}
\end{align}
Observe that the conditions
\begin{equation}
q \log(C^{\star\star}) + \log(n) \leq 0\quad\text{and}\quad (C^{\star\star})^q \leq n^{-1}
\end{equation}
are equivalent. Consequently, if
\begin{equation} \label{eq: b condition pf}
b \geq (7 + C_2) \frac{-\log(n)}{\log(C^{\star\star})}
\end{equation}
then
\begin{align}
(C^{\star\star})^{b/2} &
\lesssim \left((C^{\star\star})^{\frac{-\log(n)}{\log(C^{\star\star})}}\right)^{(7 + C_2)/2}
\lesssim n^{-(7 + C_2)/2}~.
\end{align}
Hence, if \eqref{eq: b condition pf} holds, then the bounds \eqref{eq: apply Hajek in main result} and \eqref{eq: plug in normalization} imply that
\begin{align}
\sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}}
& \Big\| \bar{U}_{n}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i) \Big\|_{\infty} \lesssim n^{-1} \phi(\theta_0)\label{eq: applied bound on Hajek}
\end{align}
holds with probability greater than $1-Cn^{-1}$.
Bounds for the first two terms in \eqref{eq: W tilde decomposition} are obtained through the application of the following Lemma.
\begin{lemma}\label{eq: apply bernstein to randomness}
Let $\mathbf{D}_n = (D_i)_{i=1}^n$ and $\bm{\xi} = (\xi_{\mathsf{s}})_{\mathsf{s}\in\mathcal{S}_{n,b}}$ denote two independent collections of independent and identically distributed random variables. Consider the function $u(D_{\mathsf{s}}, \xi_{\mathsf{s}})$ valued in $\mathbb{R}^d$. Assume that the absolute value of each component of $u(D_{\mathsf{s}}, \xi_{\mathsf{s}})$ is bounded by the constant $\phi\geq1$ almost surely.
\noindent \textbf{(i)} If there exists some constant $c < 1 $ such that the bound
\begin{equation}
\frac{\phi b\log^{1/2}(dn)}{n} < c
\end{equation}
holds, then the bound
\begin{equation}
\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}
\left(u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}})-\mathbb{E}\left[u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}}) \mid \mathbf{D}_{\mathsf{s}_q} \right]\right)
\lesssim
\frac{ b \phi \log^{1/2}(dn)}{n}\label{eq: just data lemma state}
\end{equation}
holds with probability greater than $1-n^{-1}$ for all $b>2$.
\noindent \textbf{(ii)} Moreover, if, in addition, the sets $(\mathsf{s}_q)_{q=1}^r$ are drawn independently and identically from $\mathcal{S}_{n,b}$ and $n\leq \sqrt{r} b$, then the bound
\begin{equation}
\frac{1}{r}\sum_{q=1}^{r}
\left(u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}})-\mathbb{E}\left[u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}}) \mid \mathbf{D}_{n}, \bm{\xi} \right]\right)
\lesssim
\frac{ b \phi \log^{1/2}(dn)}{n}\label{eq: random lemma state}
\end{equation}
holds with probability greater than $1-n^{-1}$.
\end{lemma}
\noindent In particular, observe that
\[
\hat{U}_{n,b}
=\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}u(D_{\mathsf{s}},\xi_{\mathsf{s}})
=\mathbb{E}\left[u(D_{\mathsf{s}_{g}},\xi_{\mathsf{s}_{g}})\mid\mathbf{D}_{n},\bm{\xi}\right]
\]
and that therefore we can write
\[
U_{n}-\hat{U}_{n}
=\frac{1}{r}\sum_{q=1}^{r}Z_{q},
\quad\text{with}\quad
Z_{q}=u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}})-\mathbb{E}\left[u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}})\mid\mathbf{D}_{n},\bm{\xi}\right]~.
\]
Consequently, by the normalization $\phi(\theta_0) b\log(dn)/n \leq C_1 <1$, Part (ii) of \cref{eq: apply bernstein to randomness} implies that
\begin{align}
\sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}} (U_{n}(\bm{x}^{(d)})-\hat{U}_{n}(\bm{x}^{(d)}))
& \lesssim
\sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}}
\frac{ b \phi(\theta_0) \log^{1/2}(dn)}{n}
= \left(\frac{\phi(\theta_0)^2 \log(dn)}{\underline{\sigma}^2_b n}\right)^{1/2} \label{eq: data and rand term linear}
\end{align}
with probability greater than $1 - n^{-1}$. In turn, we can similarly write
\[
\hat{U}_{n}-\bar{U}_{n}
=\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}Z_{\mathsf{s}}
\quad\text{with}\quad
Z_{\mathsf{s}}
=u(D_{\mathsf{s}},\xi_{\mathsf{s}})
-\mathbb{E}\left[u(D_{\mathsf{s}},\xi_{\mathsf{s}})\mid D_{\mathsf{s}}
\right]~.
\]
Thus, again by the normalization $\phi(\theta_0) b\log(dn)/n \leq C_1 <1$, Part (i) of \cref{eq: apply bernstein to randomness} implies that
\begin{align}
\sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}} (\hat{U}_{n}(\bm{x}^{(d)})-\bar{U}_{n}(\bm{x}^{(d)}))
& \lesssim
\left(\frac{\phi(\theta_0)^2 \log(dn)}{\underline{\sigma}^2_b n}\right)^{1/2} \label{eq: just data term linear}
\end{align}
with probability greater than $1 - n^{-1}$. Putting the pieces together, the bounds \eqref{eq: applied bound on Hajek}, \eqref{eq: data and rand term linear}, and \eqref{eq: just data term linear} imply that
\begin{align}
& \sqrt{\frac{n}{b^2 \underline{\sigma}^2_b}}
\Big\| \bar{U}_{n}(\bm{x}^{(d)}) - \frac{b}{n} \sum_{i=1}^n \tilde{u}^{(1)}(\bm{x}^{(d)}; D_i) \Big\|_{\infty} \nonumber\\
& \lesssim \frac{\phi(\theta_0)}{n} + \left(\frac{\phi(\theta_0)^2 \log(dn)}{\underline{\sigma}^2_b n}\right)^{1/2}
\lesssim \left( \frac{\phi(\theta_0)^2\log(dn)}{n \underline{\sigma}^2_b} \right)^{1/2}~,\label{eq: apply three bounds in asymp lin}
\end{align}
with probability greater than $1-C/n$, where the second inequality follows from the fact that $\underline{\sigma}^2_b \lesssim b^{-1}$, which is established in the proof of \cref{thm: Hoef clt}.\hfill\qed
\subsection{Proof of \cref{lem: kernel bcs}, Part (i)} Observe that
\begin{align*}
\mathbb{E}\left[M_n(x; \theta, g) \right]
&= \mathbb{E}\left[\sum_{i=1}^n K(x, X_i) m(D_i; \theta, g) \right]\\
&= \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \sum_{i\in \mathsf{s}} \mathbb{E}\left[\kappa(x, X_i, \mathsf{s}, \xi) m(D_i; \theta, g) \right]\\
&= \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \sum_{i\in \mathsf{s}}
\mathbb{E}\left[ \mathbb{E}\left[\kappa(x, X_i, \mathsf{s}, \xi) \mid X_i, D_{\mathsf{s}_{-i}}\right] \mathbb{E}\left[ m(D_i; \theta, g)\mid X_i, D_{\mathsf{s}_{-i}}\right] \right]\tag{Honesty}\\
&= \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \sum_{i\in \mathsf{s}}
\mathbb{E}\left[ \kappa(x, X_i, \mathsf{s}, \xi) \mathbb{E}\left[ m(D_i; \theta, g)\mid X_i\right] \right]\\
&= \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \sum_{i\in \mathsf{s}}
\mathbb{E}\left[ \kappa(x, X_i, \mathsf{s}, \xi) M(X_i; \theta, g) \right]~.
\end{align*}
Therefore, the normalization
\begin{equation}\label{eq: proof normalization}
\sum_{i \in \mathsf{s}} \kappa(x, X_i, \mathsf{s}, \xi) = 1
\end{equation}
implies that
\begin{equation*}
\mathsf{Bias}_n(x; \theta, g)
= \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \sum_{i\in \mathsf{s}}
\mathbb{E}\left[ \kappa(x, X_i, \mathsf{s}, \xi) \left(M(X_i; \theta, g) - m(x; \theta, g)\right)\right]~.
\end{equation*}
By the boundedness part of \cref{assu: moment linearity} and Part (iii) of \cref{assu: moment smoothness}, we find that
\begin{equation}\label{eq: proof bias bound}
\mathsf{Bias}_n(x; \theta, g) \lesssim (1 + \vert\theta\vert) \mathbb{E} \left[ \kappa(x, X_i, \mathsf{s}, \xi) \|X_i - x\|_\infty \right] \leq (1 + \vert\theta\vert) \varepsilon_{b}~,
\end{equation}
where final inequality follows from the definition of the shrinkage rate $\varepsilon_n$ and the normalization \eqref{eq: proof normalization}.
\subsection{Proof of \cref{lem: kernel bcs}, Part (ii)}
Define the function
\begin{equation*}
J(x; D_\mathsf{s},\xi_\mathsf{s}) = \sum_{i\in \mathsf{s}} \left(\kappa(x,X_i,\mathsf{s},\xi_\mathsf{s})m^{(1)}(D_i; g_0) - \mathbb{E}\left[\kappa(x,X_i \mathsf{s},\xi_\mathsf{s}) m^{(1)}(D_i; g_0)\right]\right)
\end{equation*}
and observe that
\begin{equation}
\bar{M}_n^{(1)}(x; g_0) = \frac{1}{r} \sum_{q = 1}^n J(x; D_{\mathsf{s}_q},\xi_{\mathsf{s}_q})~.
\end{equation}
Consider the decomposition
\begin{equation}
\bar{M}_n^{(1)}(x,g_0) = \tilde{A}(x) + \hat{A}(x) + \bar{A}(x) ~,
\end{equation}
where
\begin{align}
\tilde{A}(x) & = \frac{1}{r} \sum_{q = 1}^n \left(J(x; D_{\mathsf{s}_q},\xi_{\mathsf{s}_q}) - \mathbb{E}\left[J(x; D_{\mathsf{s}_q},\xi_{\mathsf{s}_q}) \mid \mathbf{D}_n, \bm{\xi}\right]\right)~,\label{eq: tilde A}\\
\hat{A}(x) & = \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \left(J(x; D_{\mathsf{s}},\xi_\mathsf{s}) - \mathbb{E}\left[J(x; D_{\mathsf{s}},\xi_{\mathsf{s}} \mid D_\mathsf{s}\right]\right)~,\quad\text{and}\label{eq: hat A}\\
\bar{A}(x) & = \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \mathbb{E}\left[J(x; D_{\mathsf{s}},\xi_{\mathsf{s}}) \mid D_\mathsf{s}\right]~,\label{eq: bar A}
\end{align}
respectively. We again apply \cref{eq: apply bernstein to randomness} to bound \eqref{eq: tilde A} and \eqref{eq: hat A}. In particular, by \cref{assu: moment linearity}, the normalization \eqref{eq: normalization in online}, and the restriction $n\leq b \sqrt{r}$, \cref{eq: apply bernstein to randomness} implies that
\begin{align}
P\left\{ \| \tilde{A}(\bm{x}^{(d)}) \|_\infty \geq C \frac{b\phi \log^{1/2}(dn)}{n} \right\} & \lesssim \frac{1}{n}\quad\text{and}\label{eq: Q tilde bound bias}\\
P\left\{ \| \hat{A}(\bm{x}^{(d)}) \|_\infty \geq C\frac{b\phi \log^{1/2}(dn)}{n} \right\} & \lesssim \frac{1}{n}\label{eq: Q hat bound bias}
\end{align}
respectively. In turn, we apply \cref{lem: Q tail}, stated in \cref{sec: lit review}, to bound the term \eqref{eq: bar A}. In this case, by \cref{assu: moment linearity}, \cref{lem: Q tail} imply that
\begin{align}
\| \bar{A}_n(\bm{x}^{(d)}) \|_\infty
& \lesssim \left(\frac{b \phi^2 \log(dn)}{n}\right)^{1/2}
+ \left(\frac{b \phi \log^{2}(dn)}{n} \right)^{1/2}
\left(\frac{b \phi \log^{2}(dn)}{n}\right)^{1/2}\nonumber\\
& \lesssim \left(\frac{b \phi^2 \log^2(dn)}{n}\right)^{1/2} \label{eq: Q bar bound bias}
\end{align}
with probability greater than $1-Cn^{-1}$, where the second inequality follows from the normalization \eqref{eq: normalization in online}. Thus, the bounds \eqref{eq: Q tilde bound bias}, \eqref{eq: Q hat bound bias} and \eqref{eq: Q bar bound bias} imply that
\begin{equation}\label{eq: M 1 concentrate}
P\left\{ \| \bar{M}_n^{(1)}(\bm{x}^{(d)};g_0) \|_\infty \geq C \left(\frac{b \phi^2 \log^2(dn)}{n}\right)^{1/2} \right\} \lesssim \frac{1}{n}~,
\end{equation}
as required.\hfill\qed
\subsection{Proof of \cref{lem: kernel bcs}, Part (iii)}
We give the details of the proof of the stated probability bound on the discrepancy
\begin{equation}
\big\|
\bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), \hat{g}_n)
-
\bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), g_0) \big\|_\infty
\end{equation}
only. The argument giving the analogous bound associated with the term $\bar{M}^{(1)}_n(\bm{x}^{(d)}; g)$ is identical. Define the functions
\begin{align}
W(x; D_{\mathsf{s}}, \xi_{\mathsf{s}}, g)
& = \sum_{i\in\mathsf{s}}
f(x, X_i; D_{\mathsf{s}}, \xi_{\mathsf{s}}, g) \nonumber
\quad\text{and}\quad
\bar{W}(x; D_{\mathsf{s}}, g) = \mathbb{E}_{\xi_{\mathsf{s}}}\left[ f(x; D_{\mathsf{s}}, \xi_{\mathsf{s}}, g)\right]~,
\end{align}
where
\begin{align}
f(x, X_i; D_{\mathsf{s}}, \xi_{\mathsf{s}}, g) & = \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) \left( m(D_i ; \theta_0, g) - m(D_i ; \theta_0, g_0) \right)\nonumber \\
& \quad\quad\quad
- \mathbb{E}[\kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) \left( m(D_i ; \theta_0, g) - m(D_i ; \theta_0, g_0) \right) ]
\end{align}
and the notation $\mathbb{E}_{A}[\cdot]$ indicates that we are evaluating the expectation over the randomness in $A$. Define the quantities
\begin{align}
\tilde{W}_n(x; g) &= \frac{1}{r}\sum_{q=1}^r \left(W(x; D_{\mathsf{s}_q}, \xi_{\mathsf{s}_q}, g) - \mathbb{E}_\mathsf{s}\left[W(x; D_{\mathsf{s}}, \xi_{\mathsf{s}}), g\right]\right)~,\label{eq: tilde G}\\
\hat{W}_n(x; g) &= \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \left(W(x; D_{\mathsf{s}}, \xi_{\mathsf{s}}, g) - \bar{W}(x; D_{\mathsf{s}}, g) \right)~,\quad\text{and}\label{eq: hat G}\\
\bar{W}_n(x; g) &= \frac{1}{N_b} \sum_{\mathsf{s}\in\mathcal{S}_{n,b}} \bar{W}(x; D_{\mathsf{s}}, g)~.\label{eq: bar G}
\end{align}
and consider the decomposition
\begin{align}
\bar{M}_n(x;\theta_0(x), g) - \bar{M}_n(x;\theta_0(x), g_0) = \tilde{W}_n(x, g) + \hat{W}_n(x, g) + \bar{W}_n(x, g)~.\label{eq: G decomp}
\end{align}
We again apply \cref{eq: apply bernstein to randomness} to bound \eqref{eq: tilde G} and \eqref{eq: hat G}. In particular, by \cref{assu: moment linearity}, the normalization \eqref{eq: normalization in online}, and the restriction $n\leq b \sqrt{r}$, \cref{eq: apply bernstein to randomness} imply that
\begin{align}
P\left\{ \| \tilde{W}(\bm{x}^{(d)}; g) \|_\infty \geq C \frac{b(1 + \| \theta_0(\bm{x}^{(d)}) \|_\infty) \phi \log^{1/2}(dn)}{n} \right\} & \lesssim \frac{1}{n}\quad\text{and}\label{eq: W tilde bound bias}\\
P\left\{ \| \hat{W}(\bm{x}^{(d)}; g) \|_\infty \geq C\frac{b(1 + \| \theta_0(\bm{x}^{(d)}) \|_\infty) \phi \log^{1/2}(dn)}{n} \right\} & \lesssim \frac{1}{n}\label{eq: W hat bound bias}
\end{align}
respectively.
To bound the term \eqref{eq: bar G}, we apply \cref{lem: Q tail}. In particular, observe that
\begin{equation*}
\mathbb{E}[\bar{W}(\bm{x}^{(d)}; D_{\mathsf{s}}, g)] = 0
\quad\text{and}\quad
\|\bar{W}(\bm{x}^{(d)}; D_{\mathsf{s}}, g)\|_{\psi_1}
\lesssim (1 + \|\theta(\bm{x}^{(d)})\|_\infty) \phi
\end{equation*}
by the boundedness part of \cref{assu: moment linearity}. In turn, observe that
\begin{align*}
\mathbb{E}\left[ \left( \bar{W}(x; D_{\mathsf{s}}, g) \right)^2 \right]
& \leq \mathbb{E} \left[ \sum_{i\in\mathsf{s}} \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) \mathbb{E}\left[\left( m(D_i ; \theta_0, g) - m(D_i ; \theta_0, g_0) \right)^2\mid X_i \right] \right]\\
& \lesssim \mathbb{E} \left[ \sum_{i\in\mathsf{s}} \kappa(x, X_i, D_{\mathsf{s}}, \xi_{\mathsf{s}}) V(x, g) \right] + \varepsilon_{b}\\
& \lesssim \|g - g_0 \|_\infty^2+ \varepsilon_{b}
\end{align*}
where the first inequality follows from Honesty and Jensen's inequality, the second inequality follows from \cref{assu: moment smoothness}, Part (ii), and the definition of the shrinkage rate $\varepsilon_{b}$, and the third inequality follows from \cref{assu: moment smoothness}, Part (ii), and the normalization that $\sum_{i\in\mathsf{s}} \kappa(x,X_i,D_\mathsf{s},\xi) = 1$ almost surely. Thus, we obtain
\begin{align}
&P\left\{ \| \bar{W}_n(\bm{x}^{(d)}; g) \| \geq C \xi^\prime_{n}(g) \right\} \lesssim \frac{1}{n}~,\quad\text{where}\label{eq: G bar bound}\\
& \xi^\prime_{n}(g) = \sqrt{\frac{b (\|g - g_0 \|_{2,\infty}^2+ \varepsilon_{b})\log(dn)}{n}} + \frac{b(1 + \|\theta_0(\bm{x}^{(d)})\|_\infty) \phi \log^2(dn)}{n}~.\nonumber
\end{align}
Consequently, as $\hat{g}_n$ is statistically independent of $\mathbf{D}_n$, the decomposition \eqref{eq: G decomp} and the bounds \eqref{eq: W tilde bound bias}, \eqref{eq: W hat bound bias}, and \eqref{eq: G bar bound}, imply that
\begin{align}
\| \bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), \hat{g}_n) - \bar{M}_n(x;\theta_0(\bm{x}^{(d)}), g_0)
\|_\infty \lesssim \xi^\prime_{n}(\hat{g}_n)~,\label{eq: se general g bar bound}
\end{align}
with probability greater than $1-Cn^{-1}$. Now, observe that
\begin{align}
\sqrt{\frac{b (\|\hat{g}_n - g_0 \|_{2,\infty}^2+ \varepsilon_{n})\log(dn)}{n}}
& \lesssim \sqrt{\frac{b \log(dn)}{n}}\|\hat{g}_n - g_0 \|_{2,\infty} + \sqrt{\frac{b\log(dn)}{n}} \varepsilon^{1/2}_{b} \nonumber \\
& \lesssim \sqrt{\frac{b\log(dn)}{n}} \left(\left(\frac{b}{n}\right)^{1/4} \delta_{n,g} + \varepsilon^{1/2}_{b}\right) \label{eq: rate condition apply}
\end{align}
with probability greater than $1-n^{-1}$. Putting the pieces together, the bounds \eqref{eq: se general g bar bound} and \eqref{eq: rate condition apply} imply that
\begin{align}
&\big\|
\bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), \hat{g}_n)
-
\bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}), g_0) \big\|_\infty \nonumber \\
&\lesssim
\sqrt{\frac{b\log(dn)}{n}} \left(\left(\frac{b}{n}\right)^{1/4} \delta_{n,g} + \varepsilon^{1/2}_{b}\right)
+ \frac{b}{n}(1 + \|\theta_0(\bm{x}^{(d)})\|_\infty) \phi \log^2(dn) \nonumber \\
&\lesssim
\sqrt{\frac{b\log(dn)}{n}} \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} + \varepsilon_b
+ \frac{b}{n}(1 + \|\theta_0(\bm{x}^{(d)})\|_\infty) \phi \log^2(dn)
\end{align}
with probability greater than $1-n^{-1}$, as required.\hfill$\blacksquare$
\subsection{Proof of \cref{lem: kernel bcs}, Part (iv)}
Recall the decomposition
\begin{align}
&M^{(1)}(x;\theta_0(x),g_0) (\hat{\theta}_n(x) - \theta_0(x))\label{eq: basic decomp consist}\\
& = -\bar{M}_n(x;\theta_0(x),g_0) \label{eq: base u stat}\\
& \quad+ \mathsf{Bias}(x;\hat{\theta}_n(x),\hat{g}_n) + \mathsf{Nuis}(x;\theta_0(x),\hat{g}_n) \label{eq: bias and nuis consist}\\
& \quad+ \mathsf{Stoch}^{(1)}(x; \hat{\theta}_n(x),\hat{g}_n) + \mathsf{Stoch}^{(2)}(x; \hat{g}_n) \label{eq: stoch terms consist}\\
& \quad- (\hat{\theta}_n(x) - \theta_0(x)) \left(M^{(1)}(x; \theta_0(x), \hat{g}_n) - M^{(1)}(x; \theta_0(x), g_0) \right)\label{eq: theta g term consistency}~
\end{align}
stated as \eqref{eq: theta g term} in the Proof of \cref{thm: generic m estimation unstudentized}, Part (i), where we note that the various terms appearing in \eqref{eq: basic decomp consist} are defined in \eqref{eq: bias def} through \eqref{eq: stoch 2 def}. Here, we have used the fact that the Hessian $H(x; \theta, g) = 0$ almost surely, by \cref{assu: moment linearity}.
First, observe that, by an argument identical to the argument used to establish Part (ii) of this Lemma, i.e., \eqref{eq: M 1 concentrate}, we have that
\begin{equation}\label{eq: M concentrate consist}
\| \bar{M}_n(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}),g_0) \|_\infty \lesssim \left(\frac{b \phi^2(1 + \| \theta(\bm{x}^{(d)})\|_\infty)^2 \log^2(dn)}{n}\right)^{1/2}
\end{equation}
with probability greater than $1-Cn^{-1}$. Moreover, as before, we have that
\begin{align}
\bigg\|(\hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)})) \left(M^{(1)}(\bm{x}^{(d)}; \theta_0(\bm{x}^{(d)}), \hat{g}_n) -
M^{(1)}(\bm{x}^{(d)}; \theta_0(\bm{x}^{(d)}), g_0) \right)\bigg\|_\infty \nonumber \\
\lesssim \| \hat{g}_n - g_0 \|_{2,\infty} \| \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty \nonumber \\
\lesssim \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} \| \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty \label{eq: product term consist}
\end{align}
with probability greater than $1-n^{-1}$, where the first inequality follows from Part (iii) of \cref{assu: moment smoothness} and the second inequality follows from the rate condition \eqref{eq: g rate reduce}.
In turn, observe that Part (i) of this Lemma gives that
\begin{align}
\| \mathsf{Bias}_n(\bm{x}^{(d)};\theta(\bm{x}^{(d)}), \hat{g}_n) \|_\infty \lesssim (1 + \| \theta(\bm{x}^{(d)})\|_\infty) \varepsilon_{b}\label{eq: bias consist}
\end{align}
and that Neyman orthogonality give that
\begin{align}
\| \mathsf{Nuis}(\bm{x}^{(d)};\theta_0(\bm{x}^{(d)}),\hat{g}_n)\|_\infty \lesssim \| \hat{g}_n - g_0 \|^2_{2,\infty} \lesssim \sqrt{\frac{b}{n}} \delta^2_{n,g}\label{eq: nuisance consist}
\end{align}
with probability greater than $1-n^{-1}$ via the expansion \eqref{eq: neyman orth bound} and the the rate condition \eqref{eq: g rate reduce}.
Moreover, as before, by \cref{assu: empirical smoothness}, a Taylor expansion gives
\begin{align}
\vert \mathsf{Stoch}^{(1)}(x; \hat{\theta}_n(x),\hat{g}_n) \vert
& = \vert (\hat{\theta}_n(x) - \theta_0(x)) \bar{M}_n^{(1)}(x; \theta_0(x), \hat{g}_n) \vert \nonumber\\
& = \vert \hat{\theta}_n(x) - \theta_0(x) \vert \bar{M}_n^{(1)}(x; \theta_0(x), g_0)\vert \nonumber\\
& + \vert \hat{\theta}_n(x) - \theta_0(x) \vert \vert \bar{M}_n^{(1)}(x; \theta_0(x), \hat{g}_n) - \bar{M}_n^{(1)}(x; \theta_0(x), g_0)\vert
\end{align}
where we have used the fact that the empirical Hessian $\bar{H}_n(x; \tilde{\theta}_0(x), \hat{g}_n) = 0$ almost surely, by \cref{assu: moment linearity}. Consequently, we find that
\begin{align}
& \| \mathsf{Stoch}^{(1)}(\bm{x}^{(d)}; \hat{\theta}_n(\bm{x}^{(d)}),\hat{g}_n) \|_\infty \nonumber\\
& \lesssim
\| \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty
\| \bar{M}^{(1)}_n(\bm{x}^{(d)}; g_0)\|_\infty \nonumber\\
& \lesssim \| \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty
\big\|
\bar{M}^{(1)}_n(\bm{x}^{(d)}; \hat{g}_n)
-
\bar{M}^{(1)}_n(\bm{x}^{(d)}; g_0) \big\|_\infty \nonumber\\
& \lesssim \| \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty
\Bigg(\left(\frac{b \phi^2 \log^2(dn)}{n}\right)^{1/2} \nonumber\\
& \quad\quad+ \sqrt{\frac{b\log(dn)}{n}} \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} + \varepsilon_{n} + \frac{b\log^2(dn)}{n}(1 + \|\theta_0(\bm{x}^{(d)})\|_\infty) \phi \Bigg) \nonumber \\
& \lesssim \| \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty
\left( \left(\frac{b \phi^2 (1 + \|\theta_0(\bm{x}^{(d)}\|_\infty)^2 \log^2(dn)}{n}\right)^{1/2}
+ \varepsilon_b\right) \label{eq: stoch 1 bound consist}
\end{align}
with probability greater than $1-C/n$, where the second to last inequality follows from Parts (ii) and (iii) of this Lemma and the final inequality follows from the normalizations \eqref{eq: normalization in online} and \eqref{eq: part iv normalizations}. Similarly, Part (ii) of this Lemma gives that
\begin{align}
\| \mathsf{Stoch}^{(2)}(\bm{x}^{(d)}; \hat{g}_n) \|_\infty & \lesssim \sqrt{\frac{b\log(dn)}{n}} \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} + \varepsilon_{b} + \frac{b\log^2(dn)}{n}(1 + \|\theta_0(\bm{x}^{(d)})\|_\infty) \phi \nonumber \\
&\lesssim \left(\left(\frac{b \phi^2 (1 + \|\theta_0(\bm{x}^{(d)}\|_\infty)^2 \log^2(dn)}{n}\right)^{1/2}+ \varepsilon_{b}\right) \label{eq: se consist}
\end{align}
with probability greater than $1-n^{-1}$, where the second inequality follows from the normalizations \eqref{eq: normalization in online} and \eqref{eq: part iv normalizations}.
Consequently, plugging the bounds \eqref{eq: M concentrate consist}, \eqref{eq: product term consist}, \eqref{eq: bias consist}, \eqref{eq: nuisance consist}, \eqref{eq: stoch 1 bound consist}, and \eqref{eq: se consist} into the decomposition \eqref{eq: basic decomp consist}, we find that
\begin{align}
\|\hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)})\|_{\infty}
& \lesssim
\sqrt{\frac{b \log^2(dn)}{n}} (1 + \| \theta(\bm{x}^{(d)})\|_\infty)\phi \label{eq: linear to consistency bound}\\
& + \left(\frac{b}{n}\right)^{1/4} \delta_{n,g} \| \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty \\
&+ (1 + \| \theta_0(\bm{x}^{(d)})\|_\infty) \varepsilon_{b} + \sqrt{\frac{b}{n}} \delta^2_{n,g} \\
&+ \| \hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)}) \|_\infty
\sqrt{\frac{b \log^2(dn)}{n}} (1 + \| \theta(\bm{x}^{(d)})\|_\infty)\phi \\
& + \sqrt{\frac{b \log^2(dn)}{n}} (1 + \| \theta(\bm{x}^{(d)})\|_\infty)\phi
\end{align}
with probability greater than $1-Cn^{-1}$, where we have used the fact that $M^{(1)}(x;\theta_0(x),g_0)$ is bounded from below by Part (iii) of \cref{assu: moment smoothness}. Consequently, we find that
\begin{align}
\|\hat{\theta}_n(\bm{x}^{(d)}) - \theta_0(\bm{x}^{(d)})\|_{\infty}
& \lesssim
\sqrt{\frac{b \log^2(dn)}{n}} (1 + \| \theta(\bm{x}^{(d)})\|_\infty)\phi \\
& + (1 + \| \theta_0(\bm{x}^{(d)})\|_\infty) \varepsilon_{b} + \sqrt{\frac{b}{n}} \delta^2_{n,g}~,
\end{align}
with probability greater than $1-Cn^{-1}$, as required.\hfill\qed
\subsection{Proofs of \cref{eq: apply bernstein to randomness}}
We begin by considering the quantity
\begin{equation}
\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}} Z_{\mathsf{s}}
\quad\text{where}\quad
Z_{\mathsf{s}} = u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}})-\mathbb{E}\left[u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}}) \mid D_{\mathsf{s}} \right]~.
\end{equation}
Conditioned on the data $\mathbf{D}_n$, the observations $Z_{\mathsf{s}}$,
$\mathsf{s}\in\mathcal{S}_{n,b}$, are centered and mutually independent. Moreover, each component of $Z_{\mathsf{s}}$ is bounded by $2\phi$ almost surely. Thus, \cref{lem: orlicz large deviation} implies that
\begin{align}
\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}} Z_{\mathsf{s}}
& \lesssim
\frac{\phi \log^{1/2}(dn)}{N_b^{1/2}} + \frac{\phi ^2\log(dn)}{N_b} \label{eq: just data term}
\end{align}
with probability greater than $1 - n^{-1}$. As $(n/b)^b \lesssim N_b$, for each $b$, we have that
\begin{align}
& \frac{\phi \log^{1/2}(dn)}{N_b^{1/2}} + \frac{\phi ^2\log(dn)}{N_b} \nonumber \\
& \quad \lesssim \frac{b \phi \log^{1/2}(dn)}{n} + \frac{b\phi \log^{1/2}(dn)}{n}\frac{b \phi \log^{1/2}(dn)}{n} \nonumber \\
& \quad \lesssim \frac{b \phi \log^{1/2}(dn)}{n} \label{eq: apply normalization N_b}
\end{align}
for each $b\geq 2$, where in the second inequality we have used the normalization $b \phi \log^{1/2}(dn) / n <c$. The desired inequality \eqref{eq: just data lemma state} follows from \eqref{eq: just data term} and \eqref{eq: apply normalization N_b}.
Next, we consider the quantity
\[
\frac{1}{r}\sum_{q=1}^{r}Z_{q},
\quad\text{with}\quad
Z_{q}=u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}})-\mathbb{E}\left[u(D_{\mathsf{s}_{q}},\xi_{\mathsf{s}_{q}})\mid\mathbf{D}_{n},\bm{\xi}\right]~.
\]
Conditioned on the data $\mathbf{D}_n$ and the residual randomness $\bm{\xi}$, the observations $Z_{q}$, $q\in[r]$, are centered and mutually independent. Moreover, each component of $Z_{q}$ is bounded by $\phi$ almost surely. Consequently, \cref{lem: orlicz large deviation} implies that
\begin{align}
\frac{1}{r}\sum_{q=1}^{r}Z_{q}
& \lesssim
\frac{\phi(\theta_0) \log^{1/2}(dn)}{r^{1/2}} + \frac{\phi(\theta_0) ^2\log(dn)}{r} \label{eq: data and rand term}
\end{align}
with probability greater than $1 - n^{-1}$. Now, the restriction $n\leq \sqrt{r} b$ implies that
\begin{align}
& \frac{\phi \log^{1/2}(dn)}{r^{1/2}} + \frac{\phi ^2\log(dn)}{r} \nonumber \\
& \quad \lesssim \frac{b \phi \log^{1/2}(dn)}{n} + \frac{b\phi \log^{1/2}(dn)}{n}\frac{b \phi \log^{1/2}(dn)}{n} \nonumber \\
& \quad \lesssim \frac{b \phi \log^{1/2}(dn)}{n}~, \label{eq: apply normalization r}
\end{align}
where in the second inequality we have used the normalization $b \phi \log^{1/2}(dn) / n <c$. The desired inequality \eqref{eq: random lemma state} follows from \eqref{eq: data and rand term} and \eqref{eq: apply normalization r}.\hfill\qed
\section{Proofs and Figures for \cref{app: additional}\label{app: bin boot proofs}}
\subsection{Proof of \cref{thm: binomial sample bootstrap}, Part (i)} The result follows from an argument very similar to the proof of \cref{thm: generic m estimation unstudentized}, Part (ii). Again, we take $\theta_0(x)=0$ for all $x$, without loss of generality. Here, we are interested in studying the discrepancy
\begin{flalign*}
R_{n}^{*}(x) & = \left(\frac{2Q_n}{n}\right)\hat{\theta}_\mathsf{s}(x)-\hat{\theta}_n(x) = \left(\frac{2Q_n}{n}\right) \left(\hat{\theta}_\mathsf{s}(x)-\theta_{0}(x)\right)-R_{n}(x)~.
\end{flalign*}
Analogously to \eqref{eq: R* decomp boot}, we can write
\begin{flalign} \label{eq: R* decomp}
\left(\frac{2Q_n}{n}\right) \left(\hat{\theta}_\mathsf{s}(x)-\theta_{0}(x)\right)
& = \frac{2Q_n}{n} \left(\frac{1}{Q_n}\sum_{i\in\mathsf{s}} \bar{u}(x, D_i) - U_\mathsf{s}(x)\right) \\
& \quad- \frac{2}{n}\sum_{i\in\mathsf{s}} \bar{u}(x, D_i) + \frac{2Q_n}{n} \Delta_\mathsf{s}(x)~,
\end{flalign}
where $U_\mathsf{s}(x)$ is analogous to $U_n(x)$, but is constructed with the data with indices in the subsample $\mathsf{s}$, and
\begin{flalign}
\Delta_\mathsf{s}(x) & = R_{n}^{*}(x) -U_\mathsf{s}(x)~.\noindent
\end{flalign}
Let $\mathcal{Q}_n(t_0)$ denote the event that
\begin{equation}
\bigg\vert \frac{2Q_n}{n} - 1\bigg\vert \leq t_0~.
\end{equation}
Similarly, let $\mathcal{F}^\prime_n(t)$ and $\mathcal{H}^\prime_n(t)$ denote the events that
\begin{flalign}
\sqrt{n}\| \Lambda^{-1/2} \Delta_\mathsf{s}(\bm{x}^{(d)})\|_\infty & \leq t/4,\quad\text{and}\\
\sqrt{n}\| \frac{2}{n}\sum_{i\in\mathsf{s}}^n \hat{u}(\bm{x}^{(d)}, D_i) - \hat{U}_\mathsf{s}(\bm{x}^{(d)}) \|_\infty &\leq t/4~,
\end{flalign}
respectively, where $\hat{U}_\mathsf{s}(\bm{x}^{(d)}) $ is again defined analogously to $\hat{U}_n(\bm{x}^{(d)}) $. Define the event $\mathcal{E}^\prime_n(t,t_0) = \mathcal{F}_n(t) \cap \mathcal{F}_\mathsf{h}(t) \cap \mathcal{H}_n(t) \cap \mathcal{H}_\mathsf{h}(t)\cap \mathcal{Q}_n(t_0)$. Fix a hyper-rectangle $\mathsf{R}$ in $\mathcal{D}$, and again recall the normalized and enlarged hyper-rectangles $\tilde{\mathsf{R}}$ and $\tilde{\mathsf{R}}_t$. On the event $\mathcal{E}^\prime_n(t,t_0)$, we have
\begin{flalign}
& \vert P\left\{\sqrt{n} R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\} - P\left\{ Z\in\mathsf{R}\right\} \vert\nonumber \\
& = \vert P\left\{\sqrt{n} \Lambda^{-1/2} R_{n}^{*}(\bm{x}^{(d)})\in\tilde{\mathsf{R}} \mid\mathbf{D}_{n}\right\} - P\left\{ \Lambda^{-1/2} Z\in\tilde{\mathsf{R}}\right\} \vert\nonumber \\
& \quad\leq\vert P\left\{ \frac{2}{\sqrt{n}}\sum_{i\in\mathsf{s}}^n \hat{u}(\bm{x}^{(d)}, D_i) - \frac{1}{\sqrt{n}}\sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i)\in\tilde{\mathsf{R}}_{t(1+t_0)} \mid\mathbf{D}_{n}\right\}
-P\left\{ \Lambda^{1/2} Z\in\tilde{\mathsf{R}}_{t(1+t_0)}\right\} \vert\nonumber \\
& \quad+\vert P\left\{ \Lambda{Z}\in\tilde{\mathsf{R}}_{t(1+t_0)}\right\} -P\left\{ \Lambda^{1/2} Z\in\tilde{\mathsf{R}}\right\} \vert\nonumber
\end{flalign}
for each $t>0$. Let $\tilde{V}_i$ be a random variable taking the value $1$ with $i$ is an element of the subset $\mathsf{s}$ and taking the value $-1$ otherwise. Observe that
\begin{flalign}\label{eq: rademacherization}
\frac{2}{n}\sum_{i\in\mathsf{s}} \hat{u}(\bm{x}^{(d)}, D_i) - \frac{1}{n}\sum_{i=1}^n \hat{u}(\bm{x}^{(d)}, D_i) = \frac{1}{n} \sum_{i=1}^n \tilde{V}_i \hat{u}(\bm{x}^{(d)}, D_i)~.
\end{flalign}
and that the weights $\tilde{V}_i$ are independent and identically distributed Rademacher random variables. Thus, on the event $\mathcal{E}^\prime_n(t,t_0)$, we have
\begin{align}
& \vert P\left\{\sqrt{n} R_{n}^{*}(\bm{x}^{(d)})\in\mathsf{R}\mid\mathbf{D}_{n}\right\} - P\left\{ Z\in\mathsf{R}\right\} \vert\nonumber \\
& \quad\lesssim \left(\frac{\varphi^{2}\log^{5}(dn)}{\underline{\lambda}^2 n}\right)^{1/4} + t(1+t_0)\sqrt{\log (d)}\label{eq: binom pre plug}
\end{align}
with probability greater than $1-Cn^{-1/2}\underline{\lambda}^{-1}\varphi \log^{3/2}(dn)$ by \cref{lem: Nazarov,lem: multiplier bootstrap clt}. Hence, it suffice to give a high probability bound on $\mathcal{E}^\prime_n(t,t_0)$ for suitable choices of $t$ and $t_0$.
To this end, observe that a multiplicative Chernoff bound implies that
\begin{equation}
\bigg\vert \frac{2Q_n}{n} - 1\bigg\vert \lesssim \sqrt{\frac{\log(n)}{n}}
\end{equation}
with probability greater than $1-n^{-1}$. Thus, as the data $D_\mathsf{s}$ are drawn independently and identically with distribution $P$ in $\mathbf{P}$ and we have assumed that $\delta_{\varepsilon n} \lesssim \delta_n$ and $\rho_{\varepsilon n} \lesssim \rho_n$ for any fixed $0<\varepsilon<1$, by setting $t = \delta_n$ and $t_0 = (\log(n)/n)^{1/2}$, \cref{assu: linearity} and the bound \eqref{eq: reduce to u stat} imply that the event $\mathcal{E}^\prime_n(t,t_0)$ occurs with probability greater than $1-C(\rho_n + n^{-1})$, as required. \hfill\qed
\subsection{Proof of \cref{thm: binomial sample bootstrap}, Part (ii)}
Again, we take $\theta_0(x)=0$ for all $x$, without loss of generality. Recall from the proof of \cref{thm: generic m estimation unstudentized}, Part (iii), that
$\tilde{V}_{i}$ is a random variable taking the value $1$ when $i$
is an element of the subset $\mathsf{s}$, and taking the value $-1$ otherwise, and that
\begin{flalign}\label{eq: re rademacher}
\frac{2}{n}\sum_{i\in\mathsf{s}} \bar{u}(\bm{x}^{(d)}, D_i) - \frac{1}{n}\sum_{i=1}^n \bar{u}(\bm{x}^{(d)}, D_i) = \frac{1}{n} \sum_{i=1}^n \tilde{V}_i \bar{u}(\bm{x}^{(d)}, D_i)~.
\end{flalign}
Define the object
\begin{align*}\label{eq: binom T term def}
T_\mathsf{s}(x)= \frac{1}{Q_n}\sum_{i\in\mathsf{h}} \bar{u}_n(x, D_i) - U_\mathsf{s}(x)~.
\end{align*}
We are interested in studying
\begin{align*}
\hat{\lambda}^2_{n,j}
& = n\mathbb{E}_{\tilde{V}}\left[\left(R^*_n(x^{(j)})\right)^2 \right]\\
& = n\mathbb{E}_{\tilde{V}}\left[\left(\frac{1}{n} \sum_{i=1}^n \tilde{V}_i \bar{u}(x^{(j)}, D_i) + \frac{2Q_n}{n} T_\mathsf{s}(x^{(j)}) - T_n(x^{(j)}) + \frac{2Q_n}{n} \Delta_\mathsf{s}(x^{(j)}) - \Delta_n(x^{(j)})\right)^2\right]~,
\end{align*}
where the notation $\mathbb{E}_{\tilde{V}}\left[\cdot\right]$ denotes that the expectation is evaluated only over the random variables $\tilde{V}_1,\ldots,\tilde{V}_n$. On the event $\mathcal{E}_n^\prime(t_0, t)$, defined in the proof of \cref{thm: generic m estimation unstudentized}, Part (iii), we have that
\begin{flalign*}
& \sup_{j\in[d]} \big\vert \hat{\lambda}^2_{n,j} - \bar{\lambda}^2_{n,j} \big\vert \leq (t(1+ t_0))^2~,
\quad\text{where}\quad
\bar{\lambda}^2_{n,j} = n\mathbb{E}_V\left[\left(\frac{1}{n} \sum_{i=1}^n \tilde{V}_i \bar{u}(x^{(j)}, D_i)\right)^2\right]~.
\end{flalign*}
We can evaluate
\begin{flalign}
\bar{\lambda}^2_{n,j}
&= \frac{1}{n}\sum_{i=1}^n \bar{u}^2(x^{(j)}, D_i).\nonumber
\end{flalign}
as the weights $\tilde{V}_i$ are mutually independent.
Now, observe that
\begin{equation*}
\sup_{j\in[d]} \Big\vert \frac{1}{n}\sum_{i=1}^n \bar{u}^2(x^{(j)}, D_i) - \lambda^2_{j} \Big\vert \lesssim \frac{\varphi}{n}\log(dn)
\end{equation*}
with probability greater than $1-n^{-1}$ by Bernstein's inequality. Thus, on the event $\mathcal{E}_n^\prime(t_0, t)$, we find that
\begin{flalign*}
& \sup_{j\in[d]} \big\vert \hat{\lambda}^2_{n,j} - \lambda^2_{n,j} \big\vert \lesssim \frac{\varphi}{n}\log(dn) + (t(1+ t_0))^2~,
\end{flalign*}
with probability greater than $1-n^{-1}$. By setting
\[
t=C \sqrt{\frac{\underline{\lambda}^2}{n}} \delta_n~,
\]
and $t_0 = (\log(n)/n)^{1/2}$, \cref{assu: linearity} and the bound \eqref{eq: reduce to u stat} imply that the event $\mathcal{E}^\prime_n(t,t_0)$ occurs with probability greater than $1-C\rho_n-n^{-1}$. Consequently, we find that
\begin{flalign*}
& \sup_{j\in[d]} \big\vert \hat{\lambda}^2_{n,j} - \lambda^2_{n,j} \big\vert \lesssim \frac{\varphi}{n}\log(dn) + \frac{\underline{\lambda}^2}{n}\delta_n^2~,
\end{flalign*}
with probability greater than $1-C(\rho_n+n^{-1})$, as required.\hfill$\blacksquare$.
\subsection{Proof of \cref{lem: stability}}
Throughout, we let $x$ denote an arbitrary element of $\bm{x}^{(d)}$.
We are interested in giving a high probability bound for the quantity
\begin{flalign}
\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}F(\bm{x}^{(d)};D_{\mathsf{s}},\hat{g}_{n}) & =\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}\sum_{i\in\mathsf{s}}a(\bm{x}^{(d)};D_{i},D_{\mathsf{s}},g)-a(\bm{x}^{(d)};D_{i},D_{\mathsf{s}},g_{0})\nonumber \\
& \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-\left(A(\bm{x}^{(d)};g)-\mathbb{E}\left[A(\bm{x}^{(d)};g_{0})\right]\right).\label{eq: object of interest}
\end{flalign}
Building on an argument used in \citet{chen2022debiased}, we decompose
(\ref{eq: object of interest}) into three terms that will be easier
to handle in isolation. In particular, by \cref{lem: Hoef rep},
the quantity (\ref{eq: object of interest}) can be re-expressed as
\begin{flalign}
\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}F(\bm{x}^{(d)};D_{\mathsf{s}},\hat{g}_{n}) & =\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{1}(\bm{x}^{(d)};\mathsf{s}_{\pi,l})\label{eq: F_1}\\
& -\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{2}(\bm{x}^{(d)};\mathsf{s}_{\pi,l})\label{eq: F_2}\\
& +\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{3}(\bm{x}^{(d)};\mathsf{s}_{\pi,l})\label{eq: F_3}
\end{flalign}
where
\begin{flalign*}
F_{1}(x;\mathsf{s}) & =\sum_{i\in\mathsf{s}}a(x;D_{i},D_{\mathsf{s}},\hat{g}_{n})-a(x;D_{i},D_{\mathsf{s}},\hat{g}_{n}^{(-\mathsf{s})})\\
F_{2}(x;\mathsf{s}) & =\sum_{i\in\mathsf{s}}A(x;\hat{g}_{n})-A(x;\hat{g}_{n}^{(-\mathsf{s})}),\quad\text{and}\\
F_{3}(x;\mathsf{s}) & =\sum_{i\in\mathsf{s}}a(\bm{x}^{(d)};D_{i},D_{\mathsf{s}},\hat{g}_{n}^{(-\mathsf{s})})-a(\bm{x}^{(d)};D_{i},D_{\mathsf{s}},g_{0})\\
& \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-(A(\bm{x}^{(d)};\hat{g}_{n}^{(-\mathsf{s})})-A(\bm{x}^{(d)};g_{0})),
\end{flalign*}
respectively. We give suitable probability bounds for the terms in
(\ref{eq: F_1}) through (\ref{eq: F_3}).
We begin by considering the term (\ref{eq: F_1}). Fix any integer
$q\geq1$. It will suffice to show that
\begin{equation}
\Big\|\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{1}(x;\mathsf{s}_{\pi,l})\Big\|_{2q}\lesssim2qn^{-1}b\gamma(2q)b^{1/2q}.\label{eq: posited F 1 norm bound}
\end{equation}
In particular, if (\ref{eq: posited F 1 norm bound}) holds, then
we have that
\begin{flalign}
& P\left\{ \bigg\vert\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{1}(x;\mathsf{s}_{\pi,l})\bigg\vert>\frac{2}{\sqrt{e}}qn^{-1}b\gamma(2q)b^{1/2q}\right\} \nonumber \\
& \leq\frac{1}{\exp(q)}\frac{\mathbb{E}\left[\vert\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{1}(x;\mathsf{s}_{\pi,l})\vert^{2q}\right]}{\left(2qn^{-1}b\gamma(2q)b^{1/2q}\right)^{2q}}\lesssim\exp(-q),\label{eq: F 1 apply Markov}
\end{flalign}
by Markov's inequality. Thus, taking $q=\lceil\log(dn)\rceil$, by
combining the inequality (\ref{eq: F 1 apply Markov}) and a union
bound, we can conclude that
\begin{equation}
\bigg\|\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{1}(\bm{x}^{(d)};\mathsf{s}_{\pi,l})\bigg\|_{\infty}\lesssim\gamma_{n,b}^{\star}\frac{b\log(dn)}{n}\label{eq: F_1 high prob bound}
\end{equation}
with probability greater than $1-Cn^{-1}$, where we recall that $\gamma_{n,b}^{\star}=\gamma(2\lceil\log(dn)\rceil)b^{1/\log(dn)}.$
To this end, observe that
\begin{flalign}
\|\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{1}(x;\mathsf{s}_{\pi,l})\|_{2q} & \leq\|\sum_{i=1}^{b}\kappa(x,X_{i};D_{[b]})(m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])}))\|_{2q}\nonumber \\
& \leq\|\max_{i\in[b]}(m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])}))^{2}\|_{q}^{1/2},\label{eq: jensen and kernel}
\end{flalign}
where the first inequality follows from Jensen's inequality and the
second inequality follows from \cref{assu: kernel restriction}, Part
(ii). Lemma 2.2.2 of \citet{van1996weak}, implies that, if $A_{1},\ldots,A_{n}$
are any collection of real-valued random variables, then
\begin{equation}
\|\max_{i\in[n]}A_{i}\|_{q}\lesssim n^{1/q}\max_{i\in[n]}\|A_{i}\|_{q}.\label{eq: L_q maximal}
\end{equation}
Thus, we have that
\begin{alignat*}{1}
\|\max_{i\in[b]}(m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])}))^{2}\|_{q}^{1/2} & \leq b^{1/(2q)}\|m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\|_{2q}.\\
& \lesssim2qn^{-1}b\gamma(2q)b^{1/2q}
\end{alignat*}
where the second inequality follows from Assumption \ref{assu: stability}.
Hence, the condition (\ref{eq: posited F 1 norm bound}) holds, implying
the high probability bound (\ref{eq: F_1 high prob bound}).
We now turn to the term (\ref{eq: F_2}). Again, it will suffice to
show that
\begin{equation}
\Big\|\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{2}(x;\mathsf{s}_{\pi,l})\Big\|_{2q}\lesssim2qn^{-1}b\gamma(2q)b^{1/2q}\label{eq: posited F 2 norm bound}
\end{equation}
as this will imply that
\begin{equation}
\|\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{2}(\bm{x}^{(d)};\mathsf{s}_{\pi,l})\|_{\infty}\lesssim\gamma_{n,b}^{\star}\frac{b\log(dn)}{n}\label{eq: posited F 2 high prob bound}
\end{equation}
by the same steps used to show that (\ref{eq: posited F 1 norm bound})
implies (\ref{eq: F_1 high prob bound}). To this end, let $\mathbf{D}_{n}^{\prime\prime}=(D_{i}^{\prime\prime})_{i=1}^{n}$
be another independent copy of $\mathbf{D}_{n}$. Observe that
\begin{flalign}
\|\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{2}(x;\mathsf{s}_{\pi,l})\|_{2q} & \leq\|\sum_{i=1}^{b}A(x;\hat{g}_{n})-A(x;\hat{g}_{n}^{(-\mathsf{s})})\|_{2q}\nonumber \\
& \leq\|\sum_{i=1}^{b}\kappa(x,X_{i}^{\prime};D_{[b]}^{\prime})(m(D_{i}^{\prime},\hat{g}_{n})-m(D_{i}^{\prime},\hat{g}_{n}^{(-[b])}))\|_{2q}\nonumber \\
& \leq\|\max_{i\in[b]}(m(D_{i}^{\prime},\hat{g}_{n})-m(D_{i}^{\prime},\hat{g}_{n}^{(-[b])}))^{2}\|_{q}^{1/2},\nonumber \\
& \leq b^{1/(2q)}\|m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\|_{2q}.\nonumber \\
& \lesssim2qn^{-1}b\gamma(2q)b^{1/2q},
\end{flalign}
where the first two inequalities follow from Jensen's inequality,
the third inequality follows from \cref{assu: kernel restriction},
Part (ii), the fourth inequality follows from (\ref{eq: L_q maximal}),
and the final inequality follows from \cref{assu: stability}.
Hence, the condition (\ref{eq: posited F 2 norm bound}) holds, implying
the high probability bound (\ref{eq: posited F 2 high prob bound}).
Finally, we consider the term \ref{eq: F_3}. The argument here is
somewhat more involved. To simplify exposition, let $k=\lfloor n/b\rfloor$
and define the sets $\mathsf{s}_{l}=\left\{ (l-1)b+1,\ldots,bl\right\} $
for each $l$ in $1,\ldots,k$. It will suffice to show that
\begin{equation}
\|\frac{1}{k}\sum_{l=1}^{k}F_{3}(x;\mathsf{s}_{l})\|_{q}\lesssim\sqrt{\frac{1}{k}}\left(\left(\frac{1}{k}\right)^{1/4}\delta_{n,g}+\varepsilon_{b}^{1/2}+b^{1/q}\gamma(q)\right)q\label{eq: F_3 L_q to show}
\end{equation}
as Jensen's inequality implies that
\begin{equation}
\|\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{3}(x;\mathsf{s}_{\pi,l})\|_{q}\leq\|\frac{1}{k}\sum_{l=1}^{k}F_{3}(x;\mathsf{s}_{l})\|_{q}\label{eq: F 3 L_q Jensen}
\end{equation}
and the conditions (\ref{eq: F_3 L_q to show}) and (\ref{eq: F 3 L_q Jensen})
imply that
\begin{equation}
\|\frac{1}{n!}\sum_{\pi\in\mathcal{P}_{n}}\bigg\lfloor\frac{n}{b}\bigg\rfloor^{-1}\sum_{l=1}^{\lfloor n/b\rfloor}F_{3}(\bm{x}^{(d)};\mathsf{s}_{\pi,l})\|_{\infty}\lesssim\sqrt{\frac{b}{n}}\left(\left(\frac{b}{n}\right)^{1/4}\delta_{n,g}+\varepsilon_{b}^{1/2}+\gamma_{n,b}^{\star}\right)\log(dn)\label{eq: F_3 posited high probability bound}
\end{equation}
with probability greater than $1-Cn^{-1}$, by the same steps used
to show that (\ref{eq: posited F 1 norm bound}) implies (\ref{eq: F_1 high prob bound}).
To establish the bound (\ref{eq: F_3 L_q to show}), we apply the
following generalized Efron-Stein inequality, due to \citet{boucheron2005moment}.
See also Chapter 15.2 of \citet{boucheron2013concentration}.
\begin{lemma}[{\citealp[Theorem 2, ][]{boucheron2005moment}}]
\label{thm: generalized efron stein}Let $X=(X_{i})_{i=1}^{n}$ be
a sequence of independent random variables and let $X^{\prime}=(X_{i}^{\prime})_{i=1}^{n}$
denote an independent copy of $X$. Let $X^{(-i)}$ be constructed
by taking $X$ and replacing $X_{i}$ with $X_{i}^{\prime}$. Consider
the random variable $f(X)$, where $f(\cdot)$ is any real-valued
function. It holds that
\begin{flalign*}
\|f(X)-\mathbb{E}\left[f(X)\right]\|_{q} & \lesssim\sqrt{q}\|\sum_{i=1}^{n}(f(X)-f(X^{(-i)}))^{2}\|_{q/2}^{1/2}
\end{flalign*}
for any $q\geq2$.
\end{lemma}
\noindent To apply Theorem \ref{thm: generalized efron stein}, let
$\hat{g}_{n}^{(-\mathsf{s}_{l},-\mathsf{s}_{r})}$ denote a version
of the estimator $\hat{g}_{n}$ formed with all observations in $\mathbf{D}_{n}$,
except that the observations $D_{\mathsf{s}_{l}}$ and $D_{\mathsf{s}_{r}}$
are replaced by the observations $D_{\mathsf{s}_{l}}^{\prime}$ and
$D_{\mathsf{s}_{q}}^{\prime\prime}$, respectively. Additionally,
let $\mathbf{D}_{n}^{\prime\prime\prime}$ be an additional, independent,
copy of $\mathbf{D}_{n}$ and let $\tilde{g}_{n}^{(-\mathsf{s}_{l})}$
denote a version of the estimator $\hat{g}_{n}$ formed with all observations
in $\mathbf{D}_{n}$, except that the observations $D_{\mathsf{s}_{l}}$
are replaced by the observations $D_{\mathsf{s}_{l}}^{\prime\prime\prime}$.
Let the quantity $F_{3}^{(-r)}(x;\mathsf{s}_{l})$ be given by
\noindent
\begin{flalign*}
F_{3}^{(-r)}(x;\mathsf{s}_{l}) & =\sum_{i\in\mathsf{s}_{l}}a(x;D_{i},D_{\mathsf{s}_{l}},\hat{g}_{n}^{(-\mathsf{s}_{l},-\mathsf{\mathsf{s}}_{r})})-a(x;D_{i},D_{\mathsf{s}_{l}},g_{0})\\
& \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-(A(x;\hat{g}_{n}^{(-\mathsf{s}_{l},-\mathsf{\mathsf{s}}_{r})})-A(x;g_{0}))
\end{flalign*}
if $r\neq l$ and by
\begin{flalign*}
F_{3}^{(-l)}(x;\mathsf{s}_{l}) & =\sum_{i\in\mathsf{s}_{l}}a(x;D_{i}^{\prime\prime},D_{\mathsf{s}_{l}}^{\prime\prime},\tilde{g}_{n}^{(-\mathsf{s}_{l})})-a(x;D_{i}^{\prime\prime},D_{\mathsf{s}_{l}}^{\prime\prime},g_{0})\\
& \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-(A(x;\tilde{g}_{n}^{(-\mathsf{s}_{l})})-A(x;g_{0}))
\end{flalign*}
otherwise. Theorem \ref{thm: generalized efron stein} implies that
\begin{flalign}
& \|\frac{1}{k}\sum_{l=1}^{k}F_{3}(x;D_{\mathsf{s}_{l}})\|_{q}\nonumber \\
& \leq\sqrt{q}\bigg\|\sum_{l=1}^{k}\left(\frac{1}{k}\left(F_{3}(x;\mathsf{s}_{l})-F_{3}^{(-l)}(x;\mathsf{s}_{l})\right)+\frac{1}{k}\sum_{r\neq l}\left(F_{3}(x;\mathsf{s}_{r})-F_{3}^{(-l)}(x;\mathsf{s}_{r})\right)\right)^{2}\bigg\|_{q/2}^{1/2}~.\label{eq: apply generalized efron-stein}
\end{flalign}
\noindent We obtain a suitable bound for (\ref{eq: apply generalized efron-stein})
through the application of the following Lemma.
\begin{lemma}
\label{lem: V momen bound}Suppose that the kernel $\kappa(x,X_{i};D_{\mathsf{s}})$
satisfies \cref{assu: kernel restriction} and has shrinkage rate $\varepsilon_{b}$
and that the moment function $m(D_{i},g)$ satisfies \cref{assu: mean-square-continuity}.
Moreover, suppose that the nuisance parameter estimator satisfies
the bound \eqref{eq: rate condition se} for some sequence $\delta_{n,g}$.
If \cref{assu: stability} holds, then
\begin{flalign}
& \bigg\|\sum_{l=1}^{k}\left(\frac{1}{k}\left(F_{3}(x^{(j)};\mathsf{s}_{l})-F_{3}^{(-l)}(x^{(j)};\mathsf{s}_{l})\right)+\frac{1}{k}\sum_{r\neq l}\left(F_{3}(x^{(j)};\mathsf{s}_{r})-F_{3}^{(-l)}(x^{(j)};\mathsf{s}_{r})\right)\right)^{2}\bigg\|_{q/2}^{1/2}\nonumber \\
& \lesssim\sqrt{\frac{1}{k}}\left(\left(\frac{1}{k}\right)^{1/4}\delta_{n,g}+\varepsilon_{b}^{1/2}+b^{1/q}\gamma(q)\right)q\label{eq: V moment bound state}
\end{flalign}
for each $j$ in $[d]$ and each even integer $q$.
\end{lemma}
\noindent Consequently, the bound (\ref{eq: apply generalized efron-stein})
and Lemma \ref{lem: V momen bound} imply (\ref{eq: F_3 L_q to show}),
and thereby (\ref{eq: F_3 posited high probability bound}).
Putting the pieces together, by the decomposition (\ref{eq: F_1}),
the bounds (\ref{eq: F_1 high prob bound}), (\ref{eq: posited F 2 high prob bound}),
and (\ref{eq: F_3 posited high probability bound}) imply that
\begin{align}
\frac{1}{N_{b}}\sum_{\mathsf{s}\in\mathcal{S}_{n,b}}F(\bm{x}^{(d)};D_{\mathsf{s}},\hat{g}_{n})& \lesssim \sqrt{\frac{b}{n}}\left(\left(\frac{b}{n}\right)^{1/4}\delta_{n,g}+\varepsilon_{b}^{1/2}+\gamma_{n,b}^{\star}\right)\log(dn)+\frac{b}{n}\gamma_{n,b}^{\star}\log(dn) \nonumber \\
& \lesssim\sqrt{\frac{b}{n}}\left(\left(\frac{b}{n}\right)^{1/4}\delta_{n,g}+\gamma_{n,b}^{\star}\right)\log(dn)+ \varepsilon_{b} + \frac{b}{n}\gamma_{n,b}^{\star}\log^2(dn)
\end{align}
with probability greater than $1-Cn^{-1}$, as required.\hfill{}$\blacksquare$
\subsection{Proof of Lemma \ref{lem: V momen bound}}
Throughout, we let $x$ denote an arbitrary element of $\bm{x}^{(d)}$.
Observe that
\begin{flalign}
& \bigg\|\sum_{l=1}^{k}\left(\frac{1}{k}\left(F_{3}(x;\mathsf{s}_{l})-F_{3}^{(-l)}(x^{(j)};\mathsf{s}_{l})\right)+\frac{1}{k}\sum_{r\neq l}\left(F_{3}(x;\mathsf{s}_{r})-F_{3}^{(-l)}(x;\mathsf{s}_{r})\right)\right)^{2}\bigg\|_{q/2}\label{eq: V moment decomp}\\
& \leq\sum_{l=1}^{k}\bigg\|\left(\frac{1}{k}\left(F_{3}(x;\mathsf{s}_{l})-F_{3}^{(-l)}(x^{(j)};\mathsf{s}_{l})\right)+\frac{1}{k}\sum_{r\neq l}\left(F_{3}(x;\mathsf{s}_{r})-F_{3}^{(-l)}(x;\mathsf{s}_{r})\right)\right)^{2}\bigg\|_{q/2}\nonumber \\
& \lesssim\bigg\|\sum_{l=1}^{k}\frac{1}{k^{2}}\left(F_{3}(x;\mathsf{s}_{l})-F_{3}^{(-l)}(x^{(j)};\mathsf{s}_{l})\right)^{2}+\frac{1}{k}\sum_{l=1}^{k}\sum_{r\neq l}\left(F_{3}(x;\mathsf{s}_{r})-F_{3}^{(-l)}(x;\mathsf{s}_{r})\right)^{2}\|_{q/2}\nonumber \\
& \lesssim\frac{1}{k}\|F_{3}(x;\mathsf{s}_{l})\|_{q}^{2}\label{eq: p moment bound}\\
& +k\|F_{3}(x;\mathsf{s}_{r})-F_{3}^{(-l)}(x;\mathsf{s}_{r})\|_{q}^{2},\label{eq: p moment stability bound}
\end{flalign}
where the first inequality follows from the triangle inequality, the
second inequality follows from Cauchy-Schwarz and Jensen's inequality,
and the final inequality follows from the triangle inequality.
Consider the term (\ref{eq: p moment bound}). Observe that
\begin{flalign}
& \mathbb{E}\left[\vert F_{3}(x;D_{\mathsf{s}_{1}})\vert^{q}\right]\nonumber \\
& =\mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}a(x;D_{i},D_{\mathsf{s}},\hat{g}_{n}^{(-\mathsf{s}_{1})})-a(x;D_{i},D_{\mathsf{s}},g_{0})-(A(x;\hat{g}_{n}^{(-\mathsf{s}_{1})})-A(x;g_{0}))\bigg\vert^{q}\right]\nonumber \\
& \leq2^{q-1}\mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}a(x;D_{i},D_{\mathsf{s}},\hat{g}_{n}^{(-\mathsf{s}_{1})})-a(x;D_{i},D_{\mathsf{s}},g_{0})\bigg\vert^{q}\right]\label{eq: unconditional term a}\\
& +2^{q-1}\mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}A(x;\hat{g}_{n}^{(-\mathsf{s}_{1})})-A(x;g_{0})\bigg\vert^{q}\right]\label{eq: conditional term A}
\end{flalign}
by the Binomial Theorem and Hölder's inequality. We begin by bounding
the term (\ref{eq: unconditional term a}). Observe that
\begin{flalign}
& \mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}a(\bm{x}^{(d)};D_{i},D_{\mathsf{s}_{1}},\hat{g}_{n}^{(-\mathsf{s}_{1})})-a(\bm{x}^{(d)};D_{i},D_{\mathsf{s}_{1}},g_{0}))\bigg\vert^{q}\right]\nonumber \\
& =\mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}\kappa(x,X_{i};D_{\mathsf{s}_{1}})(m(D_{i},\hat{g}_{n}^{(-\mathsf{s}_{1})})-m(D_{i},g_{0}))\bigg\vert^{q}\right]\nonumber \\
& \leq\mathbb{E}\left[\sum_{i\in\mathsf{s}_{1}}\kappa(x,X_{i};D_{\mathsf{s}_{1}})\vert m(D_{i},\hat{g}_{n}^{(-\mathsf{s}_{1})})-m(D_{i},g_{0}))\vert^{q}\right]\nonumber \\
& \leq\mathbb{E}\left[\mathbb{E}\left[\sum_{i\in\mathsf{s}_{1}}\kappa(x,X_{i};D_{\mathsf{s}_{1}})\mathbb{E}\left[\vert m(D_{i},\hat{g}_{n}^{(-\mathsf{s}_{1})})-m(D_{i},g_{0}))\vert^{q}\mid X_{i},\hat{g}_{n}^{(-\mathsf{s}_{1})}\right]\mid\hat{g}_{n}^{(-\mathsf{s}_{1})}\right]\right]\nonumber \\
& \leq\mathbb{E}\left[\mathbb{E}\left[\sum_{i\in\mathsf{s}_{1}}\kappa(x,X_{i};D_{\mathsf{s}_{1}})\mathbb{E}\left[\vert m(D_{i},\hat{g}_{n}^{(-\mathsf{s}_{1})})-m(D_{i},g_{0}))\vert^{q}\mid X_{i}=x\right]\mid\hat{g}_{n}^{(-\mathsf{s}_{1})}\right]\right]+\varepsilon_{b}^{q/2}\nonumber \\
& \lesssim\mathbb{E}\left[\|\hat{g}_{n}^{(-\mathsf{s}_{1})}-g_{0}\|_{q,\infty}^{q}\right]+\varepsilon_{b}^{q/2},\label{eq: unconditional A label}
\end{flalign}
where the first inequality follows from Jensen's inequality, the second
inequality follows from Honesty, i.e., \cref{assu: kernel restriction},
Part (i), the third inequality follows from the definition of the
kernel shrinkage $\varepsilon_{b}$ and \cref{assu: mean-square-continuity},
and the fourth inequality follows from \cref{assu: mean-square-continuity}
and \cref{assu: kernel restriction}, Part (ii). To handle the second,
term (\ref{eq: conditional term A}), let $\mathbf{D}_{n}^{\prime\prime}=(D_{i}^{\prime\prime})_{i=1}^{n}$
be another independent copy of $\mathbf{D}_{n}$. We have that
\begin{flalign}
& \mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}A(x;\hat{g}_{n}^{(-\mathsf{s}_{1})})-A(x;\hat{g}_{0})\bigg\vert^{q}\right]\nonumber \\
& \leq\mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}a(\bm{x}^{(d)};D_{i}^{\prime\prime},D_{\mathsf{s}}^{\prime\prime},\hat{g}_{n}^{(-\mathsf{s}_{1})})-a(\bm{x}^{(d)};D_{i}^{\prime\prime},D_{\mathsf{s}}^{\prime\prime},g_{0})\bigg\vert^{q}\right]\nonumber \\
& \leq\mathbb{E}\left[\|\hat{g}_{n}^{(-\mathsf{s}_{1})}-g_{0}\|_{q,\infty}^{q}\right]+\varepsilon_{b}^{q/2},\label{eq: condition A label}
\end{flalign}
where the first inequality follows by Jensen's inequality and the
second inequality follows by the inequality (\ref{eq: unconditional A label}).
Hence, the bounds (\ref{eq: unconditional A label}) and (\ref{eq: condition A label})
imply that
\begin{flalign}
\|F_{3}(x;D_{\mathsf{s}_{q}})\|_{q}^{2} & =\mathbb{E}\left[\vert F_{3}(x;D_{\mathsf{s}_{1}})\vert^{q}\right]^{2/q},\nonumber \\
& \lesssim\left(q^{q}\left(\frac{b}{n}\right)^{\frac{q}{4}}\delta_{n,g}^{q}+\varepsilon_{b}^{q/2}\right)^{2/q}\nonumber \\
& \lesssim q^{2}\left(\left(\frac{b}{n}\right)^{1/2}\delta_{n,g}^{2}+\varepsilon_{b}\right)\label{eq: 2p bound no stable}
\end{flalign}
by the rate condition (\ref{eq: rate condition se}), where we have used
the fact that $\hat{g}_{n}^{(-\mathsf{s}_{1})}$ and $\hat{g}_{n}$
are identically distributed.
Next, consider the term (\ref{eq: p moment stability bound}). Observe that
\begin{flalign}
& \mathbb{E}\left[\bigg\vert F_{3}(x;D_{\mathsf{s}_{l}})-F_{3,-q}(x;D_{\mathsf{s}_{l}})\bigg\vert^{q}\right]\nonumber \\
& =\mathbb{E\Bigg[}\bigg\vert\sum_{i\in\mathsf{s}_{1}}a(x;D_{i},D_{\mathsf{s}_{1}},\hat{g}_{n}^{(-\mathsf{s}_{1})})-a(x;D_{i},D_{\mathsf{s}_{1}},\hat{g}_{n}^{(-\mathsf{s}_{1},-\mathsf{s}_{2})})\nonumber \\
& \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-(A(x;\hat{g}_{n}^{(-\mathsf{s}_{2})})-A(x;\hat{g}_{n}^{(-\mathsf{s}_{1},-\mathsf{s}_{2})}))\bigg\vert^{q}\Bigg]\nonumber \\
& \leq2^{q-1}\mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}a(x;D_{i},D_{\mathsf{s}_{1}},\hat{g}_{n}^{(-\mathsf{s}_{1})})-a(x;D_{i},D_{\mathsf{s}_{1}},\hat{g}_{n}^{(-\mathsf{s}_{1},-\mathsf{s}_{2})})\bigg\vert^{q}\right]\label{eq: stability term unconditional}\\
& +2^{q-1}\mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}A(x;\hat{g}_{n}^{(-\mathsf{s}_{2})})-A(x;\hat{g}_{n}^{(-\mathsf{s}_{1},-\mathsf{s}_{2})})\bigg\vert^{q}\right]\label{eq: stability term conditional}
\end{flalign}
by the Binomial Theorem and Hölder's inequality. To bound the quantity
(\ref{eq: stability term unconditional}), observe that
\begin{flalign}
& \mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}a(x;D_{i},D_{\mathsf{s}_{1}},\hat{g}_{n}^{(-\mathsf{s}_{1})})-a(x;D_{i},D_{\mathsf{s}_{1}},\hat{g}_{n}^{(-\mathsf{s}_{1},-\mathsf{s}_{2})})\bigg\vert^{q}\right]\nonumber \\
& \leq\mathbb{E}\left[\sum_{i\in\mathsf{s}_{1}}\kappa(x,X_{i};D_{\mathsf{s}_{1}})\vert m(D_{i},\hat{g}_{n}^{(-\mathsf{s}_{1})})-m(D_{i},\hat{g}_{n}^{(-\mathsf{s}_{1},-\mathsf{s}_{2})})\vert^{q}\right]\nonumber \\
& \leq\mathbb{E}\left[\max_{i\in[b]}\vert m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\vert^{q}\right].\nonumber \\
& \leq b\mathbb{E}\left[\vert m(D_{i},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\vert^{q}\right]\lesssim b\left(qn^{-1}b\gamma(q)\right)^{q},\label{eq: var bound stability term unconditional}
\end{flalign}
where the first inequality follows Jensen's inequality, the second
inequality follows from \ref{assu: kernel restriction}, Part (ii),
the third inequality follows from (\ref{eq: L_q maximal}), and the
final inequality follows from Assumption \ref{assu: stability}. Similarly,
to bound the quantity (\ref{eq: stability term conditional}), we
have that
\begin{flalign}
& \mathbb{E}\left[\bigg\vert\sum_{i\in\mathsf{s}_{1}}A(x;\hat{g}_{n}^{(-\mathsf{s}_{2})})-A(x;\hat{g}_{n}^{(-\mathsf{s}_{1},-\mathsf{s}_{2})})\bigg\vert^{q}\right]\nonumber \\
& \leq\mathbb{E}\left[\sum_{i\in\mathsf{s}_{1}}\kappa(x,X_{i};D_{\mathsf{s}_{1}}^{\prime\prime})\vert m(D_{i}^{\prime\prime},\hat{g}_{n}^{(-\mathsf{s}_{1})})-m(D_{i}^{\prime\prime},\hat{g}_{n}^{(-\mathsf{s}_{1},-\mathsf{s}_{2})})\vert^{q}\right]\nonumber \\
& \leq\mathbb{E}\left[\max_{i\in[b]}\vert m(D_{i}^{\prime\prime},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\vert^{q}\right].\nonumber \\
& \lesssim b\mathbb{E}\left[\vert m(D_{i}^{\prime\prime},\hat{g}_{n})-m(D_{i},\hat{g}_{n}^{(-[b])})\vert^{q}\right]\leq b\left(qn^{-1}b\gamma(q)\right)^{q},\label{eq: var bound stability term conditional}
\end{flalign}
where the first inequality follows from Jensen's inequality and the
other inequalities follow from the same steps used to obtain the bound
(\ref{eq: var bound stability term unconditional}). Consequently,
we have that
\begin{flalign}
\|F_{3}(x;\mathsf{s}_{r})-F_{3}^{(-l)}(x;\mathsf{s}_{r})\|_{q}^{2} & =\mathbb{E}\left[\left(F_{3}(x^{(j)};D_{\mathsf{s}_{l}})-F_{3,-q}(x;D_{\mathsf{s}_{l}})\right)^{q}\right]^{2/q}\nonumber \\
& \lesssim b^{2/q}\left(qn^{-1}b\gamma(q)\right)^{2},\label{eq: 2p bound stable}
\end{flalign}
as desired.
Putting the pieces together, we find that
\begin{flalign*}
& \bigg\|\sum_{l=1}^{k}\left(\frac{1}{k}\left(F_{3}(x;\mathsf{s}_{l})-F_{3}^{(-l)}(x^{(j)};\mathsf{s}_{l})\right)+\frac{1}{k}\sum_{r\neq l}\left(F_{3}(x;\mathsf{s}_{r})-F_{3}^{(-l)}(x;\mathsf{s}_{r})\right)\right)^{2}\bigg\|_{q/2}^{1/2}\\
& \lesssim\sqrt{\frac{1}{k}}\left(\left(\frac{1}{k}\right)^{1/4}\delta_{n,g}+\varepsilon_{b}^{1/2}+b^{1/q}\gamma(q)\right)q
\end{flalign*}
by applying the bounds (\ref{eq: 2p bound no stable}) and (\ref{eq: 2p bound stable})
to the decomposition (\ref{eq: V moment decomp}), as required.\hfill{}$\blacksquare$
\subsection{Proof of \cref{lem: continuity}}
Consider the decomposition
\begin{flalign*}
& m(D_{i},\hat{g}_{n}(Z_{i}))-m(D_{i},\hat{g}_{n}^{(-[b])}(Z_{i}))\\
& =\left(\hat{\mu}_{n}(Z_{i},1)-\hat{\mu}_{n}^{(-[b])}(Z_{i},1)\right)-\left(\hat{\mu}_{n}(Z_{i},0)-\hat{\mu}_{n}^{(-[b])}(Z_{i},0))\right)\\
& +\hat{\beta}_{n}^{(-[b])}(W_{i},Z_{i})(\hat{\mu}_{n}(Z_{i},W_{i})-\hat{\mu}_{n}^{(-[b])}(Z_{i},W_{i}))-\left(Y_{i}-\hat{\mu}_{n}(Z_{i},W_{i})\right)\left(\hat{\beta}_{n}(W_{i},Z_{i})-\hat{\beta}_{n}^{(-[b])}(W_{i},Z_{i})\right).
\end{flalign*}
We have that
\begin{flalign*}
& \|m(D_{i},\hat{g}_{n}(Z_{i}))-m(D_{i},\hat{g}_{n}^{(-[b])}(Z_{i}))\|_{2p}\\
& \leq\|\hat{\mu}_{n}(Z_{i},1)-\hat{\mu}_{n}^{(-[b])}(Z_{i},1)\|_{2p}+\|\hat{\mu}_{n}(Z_{i},0)-\hat{\mu}_{n}^{(-[b])}(Z_{i},0))\|_{2p}\\
& +\|\hat{\beta}_{n}^{(-[b])}(W_{i},Z_{i})(\hat{\mu}_{n}(Z_{i},W_{i})-\hat{\mu}_{n}^{(-[b])}(Z_{i},W_{i}))\|_{2p}\\
&+\|\left(Y_{i}-\hat{\mu}_{n}(Z_{i},W_{i})\right)\left(\hat{\beta}_{n}(W_{i},Z_{i})-\hat{\beta}_{n}^{(-[b])}(W_{i},Z_{i})\right)\|_{2p}\\
& \lesssim\|\hat{\mu}_{n}(Z_{i},1)-\hat{\mu}_{n}^{(-[b])}(Z_{i},1)\|_{2p}+\|\hat{\mu}_{n}(Z_{i},0)-\hat{\mu}_{n}^{(-[b])}(Z_{i},0))\|_{2p}\\
&+\|\hat{\mu}_{n}(Z_{i},W_{i})-\hat{\mu}_{n}^{(-[b])}(Z_{i},W_{i})\|_{2p}
+\|\hat{\beta}_{n}(W_{i},Z_{i})-\hat{\beta}_{n}^{(-[b])}(W_{i},Z_{i})\|_{2p}
\end{flalign*}
where the first inequality follows from the triangle inequality and
the second inequality follows from the condition \eqref{eq: bounded quantities}.
This verifies the condition (\ref{eq: orlicz continuity}). An identical
argument verifies the condition (\ref{eq: orlicz continuity Z}).\hfill{}$\blacksquare$
\subsection{Proof of \cref{lem: kernel stability}}
To ease notation, we omit dependence on the evaluation point $z$.
Recall the definition of the Hájek projection
\[
u^{(1)}(D)=\mathbb{E}\left[u(D_{[b]})\mid D_{1}=D\right]
\]
where $D$ is an independent copy of $D_{1}$. Let
\begin{flalign*}
\nu^{2} & =\Var(u(D_{[b]}))\quad\text{and}\quad\sigma_{b^{\prime}}^{2}=\Var(u^{(1)}(D))
\end{flalign*}
denote the kernel variance and Hájek projection variance, respectively.
Consider the decomposition
\begin{flalign}
\hat{g}_{n}(\mathbf{D}_{n})-\hat{g}_{n}(\mathbf{D}_{n}^{(-[b])}) & =\frac{b^{\prime}}{n}\sum_{i=1}^{b}\left(u^{(1)}(D_{i})-u^{(1)}(D_{i}^{\prime})\right)\label{eq: diff of Hajek}\\
& +\left(\hat{g}_{n}(\mathbf{D}_{n})-\frac{b^{\prime}}{n}\sum_{i=1}^{n}u^{(1)}(D_{i})\right)\label{eq: H proj resid one}\\
& +\left(\frac{b^{\prime}}{n}\sum_{i=1}^{b}u^{(1)}(D_{i}^{\prime})+\sum_{i=b+1}^{n}u^{(1)}(D_{i}^{\prime})-\hat{g}_{n}(\mathbf{D}_{n}^{(-[b])})\right).\label{eq: H proj resid 2}
\end{flalign}
The result is obtained by giving higher-order moment bounds for each
of the terms (\ref{eq: diff of Hajek}), (\ref{eq: H proj resid one}),
and (\ref{eq: H proj resid 2}).
We begin by considering the terms (\ref{eq: H proj resid one}), and
(\ref{eq: H proj resid 2}). Observe that the bound (C.19) gives
\begin{flalign}
\|\hat{g}_{n}(\mathbf{D}_{n})-\frac{b^{\prime}}{n}\sum_{i=1}^{n}u^{(1)}(D_{i})\|_{q}
& \leq\left(C\frac{b^{\prime}q}{n}\right)^{b^{\prime}/2}\left(1 -q^{3/2}\frac{b^{\prime3/2}}{n^{1/2}}\right)\phi\log(n)\label{eq: proj bound 1}\\
& \leq q^{3/2}\phi^{2}n\left(C\frac{b^{\prime}q}{n}\right)^{b^{\prime}/2}\log(n)\nonumber
\end{flalign}
and
\begin{flalign}
\|\frac{b^{\prime}}{n}\sum_{i=1}^{b}u^{(1)}(D_{i}^{\prime})+\sum_{i=b+1}^{n}u^{(1)}(D_{i}^{\prime})-\hat{g}_{n}(\mathbf{D}_{n}^{(-[b])}))\|_{q}
& \leq\left(C\frac{b^{\prime}q}{n}\right)^{b^{\prime}/2}\left(1+q^{3/2}\frac{b^{\prime3/2}}{n^{1/2}}\right)\phi\log(n)\label{eq: proj bound 2}\\
& \leq q^{3/2}\phi^{2}n\left(C\frac{b^{\prime}q}{n}\right)^{b^{\prime}/2}\log(n)\nonumber
\end{flalign}
respectively. To handle the term (\ref{eq: diff of Hajek}), we apply the following
Rosenthal type inequality.
\begin{lemma}[{\citealp[Theorem 9, ][]{boucheron2005moment}}]
\label{lem: Rosenthal centered}Let $X_{1},\ldots,X_{b}$ be independent
centered random variables. For any integer $q\geq2$, we have that
\[
\|\frac{1}{n}\sum X_{i}\|_{1}\lesssim\sqrt{\frac{\Var(X_{i})q}{b}}+\frac{q}{b}\|\max_{i\in[b]}X_{i}\|_{q}.
\]
\end{lemma}
\noindent In particular, Lemma \ref{lem: Rosenthal centered} implies that
\begin{flalign}
\|\frac{b^{\prime}}{n}\sum_{i=1}^{b}\left(u^{(1)}(D_{i})-u^{(1)}(D_{i}^{\prime})\right)\|_{q} & =\frac{b^{\prime}b}{n}\|\frac{1}{b}\sum_{i=1}^{b}\left(u^{(1)}(D_{i})-u^{(1)}(D_{i}^{\prime})\right)\|_{q}\nonumber \\
& \lesssim\frac{b^{\prime}b}{n}\left(\sqrt{\frac{\sigma_{b^{\prime}}^{2}q}{b}}+q\frac{\phi}{b}\right)\lesssim\frac{\sqrt{b^{\prime}b}}{n}q\phi,\label{eq: apply rosenthal stability}
\end{flalign}
where the second inequality follows from $\sigma_{b^{\prime}}^{2} \lesssim (b^\prime)^{-1}$, which was established in the proof of \cref{thm: Hoef clt}.
Putting the pieces together, we find that
\[
\|\hat{g}_{n}(\mathbf{D}_{n})-\hat{g}_{n}(\mathbf{D}_{n}^{(-[b])})\|_{p}\lesssim\frac{\sqrt{b^{\prime}b}}{n}q\phi+q^{3/2}\phi^{2}n\left(C\frac{b^{\prime}q}{n}\right)^{b^{\prime}/2}\log(n)
\]
by combining the bounds (\ref{eq: proj bound 1}), (\ref{eq: proj bound 2}),
and (\ref{eq: apply rosenthal stability}), as required.\hfill{}$\blacksquare$
\subsection{Figures Introduced in \cref{app: additional}}
In this appendix, we display various figures introduced in \cref{app: additional}. \cref{fig: bin} displays binomial sample bootstrap upper and lower confidence bounds for the CATE \eqref{eq: CATE} on post-treatment assets. \cref{fig: cross cate} and \cref{fig: half cross} gives versions of \cref{fig: cate} and \cref{fig: half}, constructed using the $2$-fold cross-split estimator \eqref{eq: k fold cross fit estimator} and the $2$-fold cross-split bootstrap root \eqref{eq: half-sample k-fold root}.
\cref{fig: cross performance} displays results for the simulation presented in \cref{sec: simulation}, analogous to \cref{fig: performance}, for the $2$-fold cross-split estimator \eqref{eq: k fold cross fit estimator} and the $2$-fold cross-split bootstrap root \eqref{eq: half-sample k-fold root}.
\cref{fig: simulation cate} displays the pseudo-true values of the CATE $\theta_0(x)$ used in the simulation. \cref{fig:validation} displays a scatterplot comparing the moments of the data from the \cite{banerjee2015multifaceted} data to the data generated by our calibrated simulation. \cref{fig: density} compares a scatter plot of the observed values of baseline consumption and baseline assets in the \cite{banerjee2015multifaceted} data with a heat-map of the distribution of these covariates in our simulation.
\clearpage
\begin{figure}
\begin{centering}
\caption{Binomial-Sample Lower Bound}
\label{fig: bin}
\medskip{}
\begin{tabular}{c}
\includegraphics[scale=0.4]{plots/bin_lower}\tabularnewline
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig: bin} displays heat maps giving binomial-sample lower confidence bound for the CATE of the intervention studied in \cite{banerjee2015multifaceted} on post-treatment total assets. The axes and estimator are the same as in \cref{fig: cate}.}{\footnotesize\par}
\end{figure}
\begin{figure}[t]
\begin{centering}
\caption{CATE Estimates, $2$-Fold Cross-Fitting}
\label{fig: cross cate}
\medskip{}
\begin{tabular}{c}
\includegraphics[scale=0.4]{plots/cross_estimate_2}\tabularnewline
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig: cate} displays a heat map giving CATE estimates for the intervention studied in \cite{banerjee2015multifaceted} on post-treatment assets, using the the $2$-fold cross-split estimator \eqref{eq: k fold cross fit estimator}. Other features of the display are analogous to \cref{fig: cate}.}{\footnotesize\par}
\end{figure}
\begin{figure}[t]
\begin{centering}
\caption{Performance, $2$-Fold Cross-Fitting}
\label{fig: cross performance}
\medskip{}
\begin{tabular}{c}
\includegraphics[scale=0.38]{plots/cross_coverage}
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig: performance} displays several measurements of the performance of the confidence intervals formulated in \cref{def: uniform ci}, constructed using using the $2$-fold cross-split estimator \eqref{eq: k fold cross fit estimator} and the $2$-fold cross-split bootstrap root \eqref{eq: half-sample k-fold root}, in a simulation calibrated to the \cite{banerjee2015multifaceted} data. Other features of the display are analogous to \cref{fig: performance}.}{\footnotesize\par}
\end{figure}
\begin{figure}[t]
\begin{centering}
\caption{Half-Sample Confidence Region Lower Bound, $2$-Fold Cross-Fitting}
\label{fig: half cross}
\medskip{}
\begin{tabular}{c}
\includegraphics[scale=0.40]{plots/cross_half_lower_2}\tabularnewline
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig: half} displays heat maps giving half-sample lower confidence bound for the CATE of the intervention studied in \cite{banerjee2015multifaceted} on post-treatment total assets. Other features of the display are analogous to \cref{fig: half}.}{\footnotesize\par}
\end{figure}
\begin{figure}
\begin{centering}
\caption{Calibrated CATEs}
\label{fig: simulation cate}
\medskip{}
\begin{tabular}{c}
\includegraphics[scale=0.4]{plots/simulation_cate}\tabularnewline
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig: simulation cate} displays the ``true'' value of CATE used in our calibrated simulation.}{\footnotesize\par}
\end{figure}
\begin{figure}
\caption{Validation}
\label{fig:validation}
\medskip{}
\begin{centering}
\begin{tabular}{c}
\includegraphics[scale=0.25]{plots/mean_verify}\tabularnewline
\includegraphics[scale=0.25]{plots/variance_verify}\tabularnewline
\includegraphics[scale=0.25]{plots/cross_correlation_verify}\tabularnewline
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: \cref{fig:validation} displays scatterplots comparing the moments of the data from \cite{banerjee2015multifaceted} to the GAN generated simulation data. Columns differentiate between different types of variables. Rows differentiate between different types of moments. The x-axis of each sub-panel measures the moments of the true data. The y-axis of each sub-panel measures the moments of the generated data. The x and y axes in the first two rows are displayed in log-scale. A forty-five degree line is displayed in all sub-panels. Blue and green dots denote moments conditioned on treatment being set to one and zero, respectively. Black dots denote unconditioned moments.}{\footnotesize\par}
\end{figure}
\begin{figure}[h]
\begin{centering}
\caption{Covariate Density}
\label{fig: density}
\medskip{}
\begin{tabular}{c}
\textit{Panel A: Observed Covariates}\tabularnewline
\includegraphics[scale=0.4]{plots/data}\tabularnewline
\textit{Panel B: Simulation Covariate Density}\tabularnewline
\includegraphics[scale=0.4]{plots/covariate_density}\tabularnewline
\end{tabular}
\par\end{centering}
\medskip{}
\justifying
{\footnotesize{}Notes: Panel A of \cref{fig: density} displays a scatter plot of the observed values of baseline consumption and baseline assets in the \cite{banerjee2015multifaceted} data. The horizontal and vertical axes display the baseline monthly consumption, normalized to 2014 dollars on a logarithmic scale base 10, and an index for baseline assets, respectively. Panel B displays a heat-map giving the density of the joint distribution of baseline consumption and baseline assets associated with our calibrated simulation.}{\footnotesize\par}
\end{figure}
\end{spacing}
\end{appendix}