Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
Rendered from LaTeX for readability, not typeset faithfully. Citation keys are highlighted; maths is left as source; figures, tables and equation environments are summarised rather than reproduced; unrecognised commands are greyed out so nothing is silently dropped. Email addresses are removed.
Generative modeling for the bootstrap
abstractGenerative modeling builds on and substantially advances the classical idea of simulating synthetic data from observed samples. This paper shows that this principle is not only natural but also theoretically well‐founded for bootstrap inference: it yields statistically valid confidence intervals that apply simultaneously to both regular and irregular estimators, including settings in which Efron’s bootstrap fails. In this sense, the generative modeling-based bootstrap can be viewed as a modern version of the smoothed bootstrap: it could mitigate the curse of dimensionality and remain effective in challenging regimes where estimators may lack root-$n$ consistency or a Gaussian limit.
{\bf Keywords:} GAN bootstrap, flow bootstrap, Wasserstein metric, M-estimator, isotonic regression.
Introduction
Simulating synthetic data from observed samples is by no means a new idea. In statistics, this principle has been proposed repeatedly in support of various data-analytic tasks. Its roots can be traced at least to the work of scott1954comparison and neyman1956distribution, who employed synthetic sampling for model checking and for detecting unsuspected patterns. Later, efron1979bootstrap introduced the {\it bootstrap}, which relies on repeated sampling from the empirical distribution function for statistical inference. rubin1993statistical and little1993statistical further explored the idea in the context of privacy protection, advocating the release of fully synthetic datasets and proposing to use {\it multiple imputation} rubin1987multiple, which generates new data by sampling from a Bayesian posterior distribution.
From a different corner of the scientific landscape, machine learning---originally centered on prediction breiman2001statistical---has undergone tremendous advances, particularly with the rise of deep learning. Against this backdrop, the seminal contributions of kingma2013auto and goodfellow2014generative, followed by chen2018neural, song2020score, and many others, sparked a new revolution. At the heart of this revolution, now known as {\it generative modeling}, lies the principle of “creating data from noise” song2020score: a perspective that differs in intriguing ways from the early ideas explored by Scott, Neyman, Efron, and Rubin.
Motivated by these subtle differences, as well as by the remarkable empirical success of generative modeling, this paper proposes to leverage the principle of generative modeling for bootstrap inference. In particular, we develop a new framework that embraces the perspective of “creating data from noise” and enables bootstrap procedures based on repeated sampling from a generative model learned from the observed data. Alternatively, this framework could be viewed as generalizing
itemize• the parametric bootstrap efron2012bayesian, which resamples from a learned parametric model;
• the smoothed bootstrap efron1979bootstrap,silverman1987bootstrap, which resamples from a nonparametric estimate of the data distribution.
In this sense, generative modeling-based bootstrap can be regarded as a modern version of the smoothed bootstrap: it approximates the distribution of an unknown statistical estimator by resampling from a flexible, nonparametric estimator of the underlying data distribution.
From a theoretical standpoint, under the proposed general framework we establish broad criteria ensuring that {\it any} data distribution estimator satisfying these conditions yields a consistent bootstrap method for both regular and irregular statistical procedures---the latter being settings in which Efron's bootstrap is known to fail kosorok2008bootstrapping,sen2010inconsistency,groeneboom2024confidence,lin2024failure,lin2026consistency. The resulting theory, presented in Theorems (ref) and (ref), may thus be viewed as a modern counterpart of bickel1981some, who already envisioned the possibility of resampling from a general estimator of the data distribution and developed the foundational theory more than forty years ago.
Specializing our framework to concrete generative modeling techniques, and building largely on the theoretical insights of biau2020some, shen2023asymptotic, and irons2022triangular, we identify conditions under which generative adversarial networks (GANs) and flow-based generative models naturally fit within our setup, thereby giving rise to GAN-based and flow-based bootstrap procedures. From this perspective, the flow bootstrap is particularly appealing: unlike GAN-based approaches, flow estimators are typically more regular and guaranteed to be nondegenerate. Consequently, they lead to consistent bootstrap procedures that apply to both regular and irregular estimators, whereas the GAN bootstrap generally lacks comparable consistency guarantees in the irregular setting. This provides an additional, statistical inferential, perspective favoring flow-based over GAN-based generative models kobyzev2020normalizing.
The rest of this paper is organized as follows. Section (ref) introduces the general framework and provides illustrative examples. Sections (ref) and (ref) develop the corresponding bootstrap consistency theory for regular M-estimators and for the isotonic regression estimator, the latter being a prominent example of an irregular estimator. Section (ref) presents conditions under which certain versions of GANs and flow-based models yield consistent bootstrap procedures. Simulation results are reported in Section (ref), and the proofs of the main theorems are collected in Section (ref).
Supporting lemmas are stated in Sections (ref) and (ref).
Generative modeling-based bootstrap
A general framework
Consider random vectors
\[
\mZ,\mZ_1,\mZ_2, \ldots \in \cZ \subset {\mathbb R}^p
\]
sampled independently from some unknown {\it data distribution} ${\mathrm P}_Z$ with an unknown support $\cZ$. In this paper, the {\it support} of the distribution ${\mathrm P}_Z$ of $\mZ$ refers to the smallest closed set $\cZ \subseteq {\mathbb R}^p$ such that ${\mathrm P}(\mZ \in \cZ) = 1$. A common statistical task is to estimate and infer an {\it estimand} $\theta_0 = \theta_0({\mathrm P}_Z)$ using an {\it estimator} $\hat\theta_n = \hat\theta_n(\mZ_1,\ldots,\mZ_n)$, which is a function of the data $\{\mZ_i : i \in [n] \}$ with size $n$ and $[n] := \{1,2,\ldots,n\}$.
Unlike estimation, inference requires a deeper understanding of the stochastic behavior of $\hat\theta_n$: in particular its (limiting) distribution. To approximate the distribution of $\hat\theta_n$, bootstrap methods are widely used and typically proceed in two steps:
itemize• Step 1: For each bootstrap iteration, resample $n$ {\it synthetic observations} $\tilde\mZ_1,\ldots,\tilde\mZ_n \in \tilde{\cZ}_n$ from a (random) distribution $\bP_{\tilde Z,n}$, with support $\tilde{\cZ}_n$, that is learned from the data and intended to approximate ${\mathrm P}_Z$.
• Step 2: Use the conditional distribution of $\hat\theta_n(\tilde\mZ_1,\ldots,\tilde\mZ_n)$ given the original sample to approximate the sampling distribution of $\hat\theta_n = \hat\theta_n(\mZ_1,\ldots,\mZ_n)$.
Different bootstrap procedures arise from different choices of $\bP_{\tilde Z,n}$. The choice $\bP_{\tilde Z,n} = \bP_n^Z$, the empirical measure of $\{\mZ_i\}_{i\in[n]}$, corresponds to the original proposal of efron1979bootstrap and remains the most widely used form of bootstrap.
Adopting the generative modeling philosophy, we introduce a new class of choices for $\bP_{\tilde Z,n}$ by incorporating additional randomness. Let
\[
\mU, \mU_1, \mU_2, \ldots ~~{\rm and}~~~ \tilde\mU_1, \tilde\mU_2, \ldots \in \cU \subset {\mathbb R}^p
\]
be random vectors sampled independently from a {\it known} distribution ${\mathrm P}_U$, with support $\cU$, and independent of the data. One may regard $\mU_i$'s and $\tilde\mU_i$'s as {\it noise} and ${\mathrm P}_U$ as the corresponding {\it noise distribution}. A broad class of generative models approximates the data distribution ${\mathrm P}_Z$ by learning a {\it generator}
align[align omitted — 66 chars of source]
from either the paired observations $\{(\mZ_i,\mU_i)\}_{i\in[n]}$ or from $\{\mZ_i\}_{i\in[n]}$ alone. The goal of this learning process is to ensure that the pushforward distribution $\hat\mG_n \# {\mathrm P}_U$ is close, in
some predefined metric, to the true data distribution ${\mathrm P}_Z$. The sample $\{\hat\mG_n(\tilde\mU_i)\}_{i\in[n]}$ then constitutes size-$n$ {\it synthetic data}, created from noise.
Because $\hat\mG_n \# {\mathrm P}_U$ is intended to approximate ${\mathrm P}_Z$, it is natural to introduce a new class of bootstrap procedures by setting
\[
\tilde\mZ_i = \hat\mG_n(\tilde\mU_i), \qquad i \in [n],
\]
and using the conditional distribution of
\[
\hat\theta_n\big(\tilde\mZ_1,\ldots,\tilde\mZ_n\big) \,|\, \{(\mZ_i,\mU_i)\}_{i\in[n]}
\]
to approximate the sampling distribution of $\hat\theta_n(\mZ_1,\ldots,\mZ_n)$. In this paper, we refer to such procedures as {\it generative modeling-based bootstraps}.
Examples
Different generative models correspond to different choices of $\hat\mG_n$ in (ref). To introduce the generative models of interest, we begin with some additional notation. For any vector, let $\dim(\cdot)$ denote its dimension, and let $\|\cdot\|_2$, and $\|\cdot\|_\infty$ denote its $\ell_2$, and $\ell_\infty$ norms, respectively. Whenever “$\leq$” is used to compare two vectors, the comparison is done componentwise. For any (not necessarily square) matrix $\mathbf{A}$, let $\|\mathbf{A}\|_{\mathrm{op}}$ denote its spectral norm, and $\|\mathbf{A}\|_{\max}$ denote the maximum absolute value among its entries. For a square matrix, let $\det(\cdot)$ denote its determinant. Throughout the manuscript, the symbols “$\vee$” and “$\wedge$” represent the maximum and minimum, respectively, of two quantities.
We first introduce the function class of {\it neural networks}.
definition[Neural networks]
A neural network function class, denoted by $\cF_\alpha( L, W, B, q_1, q_2)$, consists of all neural networks with {\it depth} $L$, {\it width bound} $W$, {\it magnitude bound} $B$, input dimension $q_1$, output dimension $q_2$, and activation function
\[
\alpha(\cdot): {\mathbb R} \rightarrow {\mathbb R}.
\]
A function $\mf \in \cF_\alpha(L, W, B, q_1, q_2)$ is a mapping $\mf:{\mathbb R}^{q_1} \to {\mathbb R}^{q_2}$ defined recursively by $\mf(\mx) = \mx^{(L)}$, where $\mx^{(0)} = \mx$ and
\[
\mx^{(\ell)} = \alpha \Big(\mathbf{A}^{(\ell)} \mx^{(\ell-1)} + \mb^{(\ell)}\Big), \quad \ell \in [L],
\]
with $\alpha(\cdot)$ applied componentwise and the matrices $\mathbf{A}^{(1)},\ldots,\mathbf{A}^{(L)}$ and vectors $\mb^{(1)},\ldots,\mb^{(L)}$ satisfying
\begin{align*}
\max_{\ell \in [L]} \|\mathbf{A}^{(\ell)}\|_{\mathrm{op}} \,\vee\, \max_{\ell \in [L]} \|\mb^{(\ell)}\|_{2} &\le B {\rm and} \max_{\ell \in [L]} \dim\big(\mb^{(\ell)}\big) \le W.
\end{align*}
With Definition (ref), we are ready to introduce the neural network-based (Wasserstein) GAN.
example[Wasserstein GAN-based generative models, arjovsky2017wasserstein]
Fix sequences of positive integers $L_n^{\rm gen}$,$ W_n^{\rm gen}$, $B_n^{\rm gen}$ and $L_n^{\rm disc}, W_n^{\rm disc}$ that may depend on the sample size $n$. Also, choose an activation function $\alpha(\cdot): {\mathbb R} \rightarrow {\mathbb R}$. Define the classes of {\it generator neural networks} and {\it discriminator neural networks} as
\[
\cG_n := \cF_\alpha(L_n^{\rm gen}, W_n^{\rm gen}, B_n^{\rm gen}, p, p),
\qquad
\cD_n := \cF_\alpha(L_n^{\rm disc}, W_n^{\rm disc}, 1, p, 1).
\]
A Wasserstein GAN (W-GAN) aims to minimize the loss function
\[
{\sf W}: \cG_n \times \cD_n \times {\mathbb R}^p \times {\mathbb R}^p \to {\mathbb R},
\qquad
{\sf W}(\mG, D, \mz, \bm{u}) := D(\mG(\bm{u})) - D(\mz),
\]
which is closely connected to the Wasserstein metric arjovsky2017wasserstein. To train a GAN generator $\hat\mG_n^{\rm GAN}$, alternating maximization/minimization is performed so that
\[
\hat D_n^{(k)}
\in
\arg\max_{D \in \cD_n}
\sum_{i=1}^n {\sf W}\big(\hat\mG_n^{(k-1)}, D, \mZ_i, \mU_i\big)
~~~
{\rm and}
~~~
\hat\mG_n^{(k)}
\in
\arg\min_{\mG \in \cG_n}
\sum_{i=1}^n {\sf W}\big(\mG, \hat D_n^{(k)}, \mZ_i, \mU_i\big).
\]
Both updates are implemented using stochastic gradient descent over the corresponding neural network parameters. The final generator $\hat\mG_n^{\rm GAN}$ (and discriminator $\hat D_n^{\rm GAN}$) is then taken as $\hat\mG_n^{(k)}$ (and $\hat D_n^{(k)}$) for some sufficiently large $k$.
Flow-based generative models provide attractive alternatives to GAN-based approaches, offering more tractable and stable distributions kobyzev2020normalizing. We illustrate this using the following {\it autoregressive flows} huang2018neural coupled with affine transformers dinh2016density, which are referred to as {\it affine autoregressive flows}, beginning with a definition of bijective monotone upper triangular functions.
definition[Bijective monotone upper triangular functions] A {\it bijective monotone upper triangular} function $\mF = (F_1,\ldots,F_p)^\top:{\mathbb R}^p \to {\mathbb R}^p$ is a function that satisfies (a) $\mF$ is bijective, (b) each $F_i$ is strictly increasing in each of its coordinates; and
(c) each $F_i$ depends only on the first $i$ coordinates of the input. That is, for any $\mx = (x_1, \ldots, x_p)^\top \in {\mathbb R}^p$,
\[
F_i(\mx) = F_i(x_1, \ldots, x_i).
\]
With Definition (ref), we are now ready to introduce the affine autoregressive flows and the corresponding generative models.
definition[Affine autoregressive flows]
For any positive integer $\nu$, a function class $\cF_\nu$ is called a class of affine autoregressive flows of depth $\nu$ if it can be expressed as
\[
\cF_\nu
=
\Bigl\{
\mF^\nu \circ \boldsymbol\Sigma_\nu \circ \cdots \circ \mF^1 \circ \boldsymbol\Sigma_1
:
\det(\boldsymbol\Sigma_j) \neq 0,\;
\mF^j \in \cT_\uparrow(p)
\Bigr\},
\]
where $\cT_\uparrow(p)$ denotes the set of all {\it bijective monotone upper triangular} functions with domain and range ${\mathbb R}^p$.
example[Affine autoregressive flow-based generative models]
Assume the known noise distribution ${\mathrm P}_U$ admits a Lebesgue density, $p_U$. For any $\mS \in \cF_\nu$, write
\[
\mS = \mF^\nu \circ \boldsymbol\Sigma_\nu \circ \cdots \circ \mF^1 \circ \boldsymbol\Sigma_1,
\quad
\mF^i = (F_1^i,\ldots,F_p^i)^\top,
\]
and define the objective function \(\Gamma : \cF_\nu \times {\mathbb R}^p \to {\mathbb R}\) as
$$
\Gamma(\mS,\mz)
= \log p_U(\mS(\mz)) + \sum_{i=1}^\nu \Big\{
\log \det(\boldsymbol\Sigma_i)
+ \sum_{j=1}^p \log\big( \sD_j F_j^i(\mx^{(i)}) \big)
\Big\},
$$
where $\mx^{(i)}
= \boldsymbol\Sigma_i \circ \cdots \circ \mF^1 \circ \boldsymbol\Sigma_1 (\mz)$ for $i = 1,\ldots,\nu$, and $\sD_j$ denotes the partial derivative with respect to the \(j\)-th coordinate for $j=1,\ldots, p$.
The change-of-variables formula gives
\[
\Gamma(\mS,\mz) = \log p_{S^{-1}(U)}(\mz),
\]
where \(p_{S^{-1}(U)}\) is the Lebesgue density of the transformed random variable \(\mS^{-1}(\mU)\).
In other words, the objective \(\Gamma(\mS,\mz)\) returns the log-density of \(\mS^{-1}(\mU)\) evaluated at the point \(\mz\).
Training a flow generator \(\hat\mG_n^{\rm flow}\) therefore reduces to the following (nonparametric) maximum likelihood estimation problem:
\begin{align}
\hat\mG_n^{\rm flow}
= (\hat\mS_n^{\rm flow})^{-1} with
\hat\mS_n^{\rm flow} \in \arg\max_{\mS \in \cF_\nu}
\sum_{i=1}^n \Gamma(\mS, \mZ_i).
\end{align}
In (ref), the optimization problem is often solved by considering a smaller function class than $\cF_\nu$, leading to Real NVP dinh2016density and many other popular normalizing flow models; cf. papamakarios2021normalizing. Next, thanks to the affine autoregressive structure, $\hat\mS_n^{\rm flow}$ can
be inverted analytically, leading to a computationally stable and tractable flow generator $\hat\mG_n^{\rm flow}$.
Discussion
We conclude this section with a brief discussion of the connections between the generative modeling-based bootstrap framework and the classical bootstrap literature, along with some related works. To begin with, we note that the framework introduced in Section (ref) also encompasses many classical bootstrap procedures. For example, let ${\mathrm P}_U$ denote the Lebesgue measure on $[0,1]^p$, and define
\[
\hat\mG_n(\bm{u})
= \sum_{i=1}^n \mZ_i \cdot
\mathds{1}\!\Big( \tfrac{i-1}{n} < u_1 \le \tfrac{i}{n} \Big),
\qquad
\text{for any } \bm{u} = (u_1,\ldots,u_p)^\top \in [0,1]^p,
\]
where $\mathds{1}(\cdot)$ denotes the indicator function. This construction immediately recovers Efron's original bootstrap. Such a connection is natural because, at least in one dimension, the quantile map constitutes the {\it optimal transport} from ${\mathrm P}_U$ to ${\mathrm P}_Z$ under any convex cost function panaretos2020invitation.
In a similar vein, the smoothed bootstrap introduced by efron1979bootstrap can also be accommodated within the framework of Section (ref). Specifically, given any {\it proper} estimator of the underlying data-generating distribution (e.g., the kernel density estimators), one may generate new samples by applying the Brenier map brenier1991polar to uniformly distributed noise via the optimal transport. From this perspective, the smoothed bootstrap may be interpreted as a generative modeling-based bootstrap procedure, even though Efron's original motivation was rooted in a rather different philosophical standpoint.
Overall, despite the natural appeal of such generative modeling-based bootstrap methods, it is somewhat striking that the literature along this direction remains relatively sparse. Three notable exceptions are haas2020statistical, which suggested using GANs to implement a version of the smoothed bootstrap; dahl2022time, which explored GAN-based bootstrap inference for time series; and athey2024using, which investigated the use of GANs for causal inference. In all cases, however, the scope is relatively specialized and the emphasis is predominantly empirical.
Theory for regular M-estimators
We illustrate the validity of generative modeling-based bootstrap by first considering one of the most prevalent classes of estimators: {\it M-estimators}.
Let
\[
{\sf L} : {\mathbb R}^q \times {\mathbb R}^p \to {\mathbb R}
\]
be a general objective function mapping a $q$-dimensional parameter
and a $p$-dimensional data point
to a real value.
For a parameter space \(\cK \subset {\mathbb R}^q\), define the population and empirical maximizers
\[
\meta_0 := \arg\max_{\meta \in \cK} {\mathrm E}\big[{\sf L}(\meta, \mZ)\big],
\qquad
\hat\meta_n \approx \arg\max_{\meta \in \cK} \sum_{i=1}^n {\sf L}(\meta, \mZ_i),
\]
where the uniqueness of the maximizers is assumed and “\(\approx\)” allows for numerical optimization error.
In the bootstrap analogue, define similarly
\[
\tilde\meta_0 \in \arg\max_{\meta \in \cK}
{\mathrm E}\!\Big[{\sf L}\big(\meta, \hat\mG_n(\mU)\big) \,\big|\, \mZ_1,\ldots,\mZ_n, \mU_1, ..., \mU_n\Big],
\qquad
\tilde\meta_n \approx \arg\max_{\meta \in \cK}
\sum_{i=1}^n {\sf L}\big(\meta, \hat\mG_n(\tilde\mU_i)\big).
\]
Bootstrap inference then proceeds by approximating the distribution of
\(
\hat\meta_n - \meta_0
\)
using the conditional distribution of
\(
\tilde\meta_n - \tilde\meta_0
\)
given the original data.
To establish consistency of generative modeling-based bootstrap procedures, we first lay out the required conditions on the data/noise space, the generator, and the M-estimators.
assumption[Data space, I]
Assume that:
\begin{enumerate}[label=(\alph*)]
• $\mZ, \mZ_1, \mZ_2, \ldots \in \cZ \subset {\mathbb R}^p$ are independently drawn from an unknown distribution ${\mathrm P}_Z$ that admits a continuous Lebesgue density $p_Z$ that has nonzero variance;
• the set $\cZ$ is convex and compact.
\end{enumerate}
assumption[Noise space, I]
Assume that $\mU, \mU_1, \mU_2, \ldots$ and $\tilde\mU_1, \tilde\mU_2, \ldots \in \cU$ are independently drawn from the known distribution ${\mathrm P}_U$, are independent of the data, and have nonzero variance.
assumption[Generator, I] Assume that:
\begin{enumerate}[label=(\alph*)]
• the generator $\hat\mG_n: \cU \rightarrow \tilde\cZ_n$, introduced in (ref), is a function of $\{(\mZ_i,\mU_i)\}_{i \in [n]}$;
• the random measure $\bP_{\tilde Z \mid \cO}$ has nonzero variance ${\mathrm P}_{\cO}$-almost surely, and satisfies
\[
{\sf W}_1\big(\bP_{\tilde Z \mid \cO},\, {\mathrm P}_Z\big) = o_{{\mathrm P}_{\cO}}(1),
\]
where ${\sf W}_1(\cdot,\cdot)$ denotes the Wasserstein-1 distance (in Euclidean metric space) and $\bP_{\tilde Z \mid \cO}$ is the conditional distribution of $\tilde \mZ = \hat\mG_n(\mU)$ given
\[
\cO := \sigma(\mZ_1,\mZ_2,\ldots,\mU_1,\mU_2,\ldots),
\]
the $\sigma$-field generated by data and noise, with the associated probability measure ${\mathrm P}_\cO$;
• for all $n$, $\tilde{\cZ}_n \subseteq \tilde{\cZ}$ ${\mathrm P}_{\cO}$-almost surely, where $\tilde{\cZ}$ is a nonrandom and compact subset of ${\mathbb R}^p$.
\end{enumerate}
assumption[Objective function]
For any $\mz \in \cZ \cup \tilde{\cZ}$ and $\meta \in \cK$, assume that:
\begin{enumerate}[label=(\alph*)]
• $\cK$ is convex and compact;
• the map $\boldsymbol\eta \mapsto {\sf L}(\boldsymbol\eta,\mz)$ is twice continuously differentiable on $\cK$;
• letting ${\rm D }^k_{\meta}$ be the $k$-th derivative with respect to $\meta$, the maps
\[
\mz \mapsto {\sf L}(\boldsymbol\eta, \mz),\qquad
\mz \mapsto {\rm D }_{\boldsymbol\eta}{\sf L}(\boldsymbol\eta, \mz),\qquad
\mz \mapsto {\rm D }_{\boldsymbol\eta}^2{\sf L}(\boldsymbol\eta, \mz)
\]
are continuous on $\cZ \cup \tilde \cZ$;
• Fisher's information matrix $- {\rm D}_{\boldsymbol\eta}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)]$ is invertible.
\end{enumerate}
assumption[M-estimator]
Assume that:
\begin{enumerate}[label=(\alph*)]
• $\boldsymbol\eta_0$ is an interior point of $\cK$;
• $\boldsymbol\eta_0$ uniquely maximizes ${\mathrm E}\Big[{\sf L}(\boldsymbol\eta,\mZ)\Big]$;
• the estimator $\hat\meta_n$ is consistent for $\meta_0$ in the sense that
$
\|\hat{\boldsymbol\eta}_n - \boldsymbol\eta_0\|_2
\xrightarrow[\;]{{\mathrm P}_{\cO}} 0;
$
• $\hat\meta_n$ is an approximate empirical maximizer in the sense that
$\hat\meta_n \in \cK$ and
\[
\frac1n \sum_{i=1}^n {\sf L}(\hat{\boldsymbol\eta}_n, \mZ_i) + o_{{\mathrm P}_{\cO}}(n^{-1})
\;\ge\;
\sup_{\boldsymbol \eta \in \cK} \frac1n \sum_{i=1}^n {\sf L}(\boldsymbol\eta, \mZ_i)
,
\]
allowing for numerical optimization error.
\end{enumerate}
assumption[Bootstrap M-estimator]
Assume that:
\begin{enumerate}[label=(\alph*)]
• The bootstrap estimator satisfies
$
\|\tilde{\boldsymbol\eta}_n - \tilde{\boldsymbol\eta}_0\|_2
\xrightarrow[\;]{{\mathrm P}_{\cO\tilde U}} 0,
$
where ${\mathrm P}_{\cO\tilde U}$ refers to the joint distribution of
$(\mZ_1,\mZ_2,\ldots)$ and $(\mU_1,\mU_2,\ldots,\tilde\mU_1,\tilde\mU_2,\ldots)$;
• $\tilde\meta_n$ is an approximate empirical maximizer in the bootstrap world,
in the sense that $\tilde\meta_n \in \cK$ and
\[
\frac1n \sum_{i=1}^n {\sf L}\big(\tilde{\boldsymbol\eta}_n,\, \hat\mG_n(\tilde\mU_i)\big) + o_{{\mathrm P}_{\cO\tilde U}}(n^{-1})
\;\ge\;
\sup_{\boldsymbol{\eta} \in \cK}\frac1n \sum_{i=1}^n {\sf L}\big({\boldsymbol\eta},\, \hat\mG_n(\tilde\mU_i)\big).
\]
\end{enumerate}
Assumption (ref) corresponds to the “classical conditions” described in, e.g., Vaart_1998. Assumption (ref) represents the standard identifiability and consistency condition for M-estimators, while Assumption (ref) serves as its bootstrap analogue. These assumptions hold automatically under Glivenko--Cantelli conditions for the loss function ${\sf L}$ over $\meta$; see Vaart_1998. Section (ref) will provide sufficient conditions under which the generative models discussed in Section (ref) satisfy Assumption (ref).
Although it is in principle possible to establish consistency of the generative modeling-based bootstrap under weaker smoothness conditions than, for instance, Assumption (ref), by appealing to more refined empirical process techniques van1996weak, we believe that doing so would add limited additional insight. The present theory already fulfills its intended purpose and underscores the main message: generative modeling-based bootstraps can consistently recover the distribution of regular M-estimators with appropriate theoretical guarantees. In this sense, they provide a viable alternative to existing bootstrap procedures.
In detail, with the above assumptions, the following theorem gives the bootstrap consistency of $\tilde{\boldsymbol\eta}_n$ to approximate the distribution of $\hat{\boldsymbol\eta}_n$.
theorem[Bootstrap consistency, regular M-estimators]
Under Assumptions (ref)-(ref), we have
$$
\sup_{\mt \in {\mathbb R}^q} \Big|\mathrm{P}\left(\sqrt{n} (\tilde{\boldsymbol\eta}_n - \tilde{\boldsymbol\eta}_0) \leq \mt| \mathcal{O}\right) - \mathrm{P}\left(\sqrt{n}(\hat{\boldsymbol\eta}_n - \boldsymbol\eta_0) \leq \mt\right)\Big| = o_{{\mathrm P}_{\cO}}(1).
$$
remarkFor reasons similar to those discussed prior to Theorem (ref), we do not attempt to extend Theorem (ref) to high-dimensional regimes in which the data dimension $p$ is large relative to the sample size $n$. Instead, this setting is examined empirically in Section (ref). The simulation results reported there provide encouraging evidence that generative modeling-based bootstrap methods implemented via GANs and normalizing flows can match the best performance of Efron's original bootstrap, while being substantially less affected by the curse of dimensionality than the smoothed bootstrap (based on kernel density estimators).
Theory for isotonic regression: an irregular estimator
Inference for irregular estimators---those that typically fail to achieve root-$n$ consistency and do not admit a Gaussian limit---has long been a central topic in mathematical statistics and econometrics. Prominent examples include shape-constrained inference groeneboom2014nonparametric and Manski-type estimators manski1981structural,cattaneo2020bootstrap.
Because the limiting distributions in such problems are often intricate, bootstrap-based methods are particularly attractive. However, it is now well understood that Efron’s original bootstrap is generally inconsistent in these settings. As a result, inference for irregular estimators is typically conducted using variants of the smoothed bootstrap kosorok2008bootstrapping,sen2010inconsistency,groeneboom2024confidence.
This section contributes to this literature by analyzing a canonical irregular estimator, the isotonic regression estimator, for which a comprehensive (smoothed) bootstrap consistency theory appears to remain unavailable. In addition, we investigate the use of generative modeling-based bootstrap methods as an alternative to the traditional smoothed bootstrap. Specifically, we establish general conditions under which the generative modeling-based bootstrap consistently approximates the sampling distribution of the estimator.
In detail, isotonic regression concerns pairs $\mZ_i = (X_i, Y_i) \in \cX \times \cY$, for $i \in [n]$, which are assumed to be independent and identically distributed (i.i.d.), with marginal distributions ${\mathrm P}_X$ and ${\mathrm P}_Y$, and corresponding supports $\cX$ and $\cY$. Assume the regression model
\[
Y_i = f_0(X_i) + \xi_i,
\]
where $\cX$ is {\it known}, $f_0 : \cX \to {\mathbb R}$ is an unknown, fixed, and nondecreasing function, the errors $\xi_i$ are i.i.d., independent of the $X_i$'s, and ${\mathrm E}[\xi_1]=0$. The isotonic regression estimator of $f_0$ is a {\it shape-constrained} least squares, given by
\[
\hat f_n
:=
\underset{f : \cX \to {\mathbb R} \text{ nondecreasing}}{\arg\min}
\sum_{i=1}^n (Y_i - f(X_i))^2.
\]
A common inferential goal is to construct confidence intervals for $f_0(x_0)$ based on $\hat f_n(x_0)$.
Let $\sigma^2 := {\mathrm E}[\xi_i^2]$. The following facts are well known.
enumerate[label=(\alph*)]
• {\bf (Cube-root asymptotics)}
Supposing $X_1$ is distributed uniformly on $[0,1]$, $x_0 \in (0,1)$, and $f_0$ and the errors obey mild regularity conditions, the estimator $\hat f_n(x_0)$ satisfies a cube-root rate of convergence to $f_0(x_0)$ and converges weakly to a Chernoff-type distribution brunk1969estimation,han2022berry. Specifically,
\begin{align}
\Big(\frac{n}{\sigma^2}\Big)^{1/3}
\big(\hat f_n(x_0) - f_0(x_0)\big)
\; converges weakly to \;
\Big(\frac{f_0'(x_0)}{2}\Big)^{1/3} \cdot {\mathbb D},
\end{align}
where $f_0'$ denotes the derivative of $f_0$, and ${\mathbb D}$ is the Chernoff distribution:
\[
{\mathbb D} = 2 \cdot \arg\max_{t \in {\mathbb R}} \Big\{{\mathbb B}(t) - t^2\Big\},
\]
with ${\mathbb B}$ denoting the two-sided Brownian motion.
• (Failure of the classical bootstrap)
Efron's original bootstrap fails for isotonic regression: the bootstrap limiting distribution of $\hat f_n(x_0)$ does not match the distribution in (ref). See, for example, groeneboom2024confidence and references therein.
We now propose a generative modeling-based bootstrap approach for statistical inference in isotonic regression.
To this end, consider the bootstrap data
\[
(\tilde X_i, \tilde Y_i)^\top := \hat\mG_n(\tilde\mU_i), \qquad i \in [n],
\]
with marginal distributions $\bP_{\tilde X \mid \cO}$ and $\bP_{\tilde Y \mid \cO}$, supports $\tilde \cX_n$ and $\tilde \cY_n$, respectively, together with the induced regression structure
align*[align* omitted — 228 chars of source]
Note that $\tilde f_0$ is not necessarily nondecreasing, and the bootstrap residuals $\{\tilde \xi_i\}_{i\in[n]}$ are not conditionally independent of the covariates $\{\tilde X_i\}_{i\in[n]}$ given $\cO$. Furthermore, the supports $\tilde\cX_n$ and $\tilde\cY_n$ may not necessarily equal $\cX$ and $\cY$.
The bootstrap isotonic regression estimator is defined as
align[align omitted — 209 chars of source]
where the indicator function in (ref) ensures that the optimization only involves those $\tilde X_i \in \cX$, on which the function $f_0$ is well defined.
Our goal is to show that, under suitably mild conditions, the conditional distribution of
$\tilde f_n(x_0)$ consistently approximates the sampling distribution of $\hat f_n(x_0)$.
This naturally requires additional regularity conditions on both the data-generating mechanism and the generator $\hat\mG_n$.
In what follows, $\sD_x$ and $\sD_x^2$ denote the first and second partial derivative, respectively, with respect to the first argument.
assumption[Data space, II]
Assume that:
\begin{enumerate}[label=(\alph*)]
• the regression function $f_0: \cX \rightarrow {\mathbb R}$ is twice continuously differentiable on $\cX$, with derivative uniformly bounded away from $0$;
• $(X_1,\xi_1),(X_2,\xi_2),\ldots$ are i.i.d., each $X_i$ is independent of $\xi_i$, and satisfy ${\mathrm E}[\xi_1]=0$, ${\mathrm{Var}}(X_1) >0 $ and $\sigma^2 > 0$;
• $\mZ_1$ admits a Lebesgue density $p_Z$ such that
$$
(x,y) \mapsto p_Z(x,y) \text{ is twice continuously differentiable on $\cX \times {\mathbb R}$,}
$$
• $X_1$ admits a Lebesgue density $p_X$ such that $p_X$ is uniformly bounded away from zero on $\cX$;
• $\cX, \cY$ are bounded, closed intervals satisfying $\cZ = \cX \times \cY$, and $x_0$ is an interior point of $\cX$.
\end{enumerate}
assumption[Generator, II]
Assume that, for all sufficiently large $n$, the conditional distribution of $\tilde \mZ = \hat\mG_n(\mU)$ given $\cO$ admits a Lebesgue density $\tilde p_n(\cdot) = \tilde p_n(\cdot \mid \cO)$ so that
\begin{enumerate}[label=(\alph*)]
• the map
$(x,y) \mapsto \tilde p_n(x, y)$ is always continuously differentiable and almost surely twice continuously differentiable on $\tilde\cX_n \times {\mathbb R} $;
• $\tilde X_1$ admits a Lebesgue density $\tilde p_X$ such that some universal constant $\tilde K > 0$ exists, for which
\[
\tilde K^{-1} \le \tilde p_X(x) \text{ for all } x \in \cX,
~~~{\rm and}~~~
|\tilde p_n(\mz)| \vee \|{{\rm D }} \tilde p_n(\mz)\|_2 \,\vee\, \|{{\rm D }}^2 \tilde p_n(\mz)\|_{\rm op} \le \tilde K
\text{ for all } \mz \in \tilde\cZ_n,
\]
almost surely;
• $\tilde \cX_n$ is an interval satisfying $\cX \subseteq \tilde\cX_n $ almost surely.
\end{enumerate}
Assumption (ref) resembles the classical conditions imposed in the smoothed bootstrap literature to ensure bootstrap consistency for irregular estimators; see, for example, sen2010inconsistency. Section (ref) will provide sufficient conditions under which a class of flow-based generative models satisfies Assumption (ref).
With the above assumptions, the following theorem establishes bootstrap consistency for the isotonic regression estimator.
theorem[Bootstrap consistency, isotonic regression] Assume Assumptions (ref), (ref), (ref), and (ref). We then have
$$
\sup_{t \in {\mathbb R}} \left|\mathrm{P}\left(n^{1/3}(\tilde f_n(x_0) - \tilde f_0(x_0)) \leq t \middle | \mathcal{O}\right) -\mathrm{P}\left(n^{1/3}(\hat f_n(x_0) - f_0(x_0)) \leq t \right)\right| = o_{{\mathrm P}_{\cO}}(1).
$$
GAN and flow bootstraps
This section gives sufficient conditions, under which the GAN- and flow-based generative models are able to satisfy the requirements in Theorems (ref) and (ref). To this end, we first regulate the noise distribution ${\mathrm P}_U$.
assumption[Noise space, II]
Assume that:
\begin{enumerate}[label=(\alph*)]
• the support of ${\mathrm P}_U$, $\cU$, is convex, compact, and contains $\mathbf{0}$, and that ${\mathrm P}_U$ admits a continuously differentiable Lebesgue density $p_U$ on $\cU$;
• there exists some constant $r_0 > 0$ such that $p_U$ is uniformly lower bounded away from $0$ on the set $\{\bm{u} \in {\mathbb R}^p : \|\bm{u} \|_2 \leq r_0\}.$
\end{enumerate}
The following slightly stronger condition is needed for Theorem (ref), particularly concerning Assumption $\ref{ass:generator}$.
assumption[Noise space, III]
Supposing that $\mU = (U_1, U_2)^\top \in {\mathbb R}^2$, assume that $p_U$ is twice-continuously differentiable on $\cU_1 \times {\mathbb R}$, where $\cU_1$ is the support of $U_1$.
W-GAN
This section demonstrates that suitably trained W-GANs in Example (ref) satisfy Assumption (ref). Our analysis builds upon the theoretical results of biau2020some on GANs and shen2023asymptotic on the asymptotic properties of neural networks.
assumption[W-GAN]
Assume that:
\begin{enumerate}[label=(\alph*)]
• the activation function $\alpha: {\mathbb R}\rightarrow {\mathbb R}$ is $1$-Lipschitz with $\alpha(0) = 0$, and the neural network parameters $L^{\rm gen}$ and $B^{\rm gen} $ are fixed positive constants;
• the W-GAN is well trained in the sense that
\[
\sup_{D \in {\rm Lip_1}(p,1)}\Big\{
\frac1n \sum_{i=1}^n
{\sf W}(\hat\mG_n^{\rm GAN}, D, \mZ_i, \mU_i)
-
\frac1n \sum_{i=1}^n
{\sf W}(\hat\mG_n^{\rm GAN}, \hat D_n^{\rm GAN}, \mZ_i, \mU_i)\Big\}
=
o_{{\mathrm P}_{\cO}}(1),
\]
where ${\rm Lip}_1(p,1)$ denotes the set of all $1$-Lipschitz functions from ${\mathbb R}^p$ to ${\mathbb R}$, and enjoys the universal approximation property in the sense that
\[
\frac1n \sum_{i=1}^n
{\sf W}(\hat\mG_n^{\rm GAN}, \hat D_n^{\rm GAN}, \mZ_i, \mU_i)
=
o_{{\mathrm P}_{\cO}}(1);
\]
• $\mathrm{Var}(\hat \mG_n^{\rm GAN}(\mU)\mid \cO) > 0$ holds ${\mathrm P}_{\cO}$-almost surely.
\end{enumerate}
theorem[GAN bootstrap]
Assume that Assumptions (ref), (ref), (ref), and (ref) hold.
Then Assumption (ref) is satisfied by the GAN generator.
Affine autoregressive flows
This section concerns affine autoregressive flows in Example (ref). To facilitate the analysis, we focus on the following subclass of $\cF_{\nu}$ that encourages more regularity and is encouraged by irons2022triangular:
\[
\cF_{\nu,K,M}
:=
\Big\{
\mF^\nu \circ \boldsymbol\Sigma_\nu \circ \cdots \circ \mF^1 \circ \boldsymbol\Sigma_1
\Big\}
\subset \cF_\nu,
\]
where, for each $i \in [\nu]$:
enumerate[label=(\alph*)]
• $\mF^i \in \cT_\uparrow(p)$, $\mF^i(\mathbf{0}) = \mathbf{0}$, and $\mF^i$ is three-times continuously differentiable on ${\mathbb R}^p$;
• $\boldsymbol{\Sigma}_{i}$ is symmetric, and satisfies $K^{-1} \le \lambda_{\rm min}(\boldsymbol\Sigma_i)$ and $\lambda_{\rm max}(\boldsymbol\Sigma_i)\ \le K,$ where $\lambda_{\rm min}(\boldsymbol\Sigma_i)$ and $\lambda_{\rm max}(\boldsymbol\Sigma_i)$ denote the smallest and largest eigenvalues of $\boldsymbol\Sigma_i$, respectively;
• the absolute values of all first-, second-, and third-order partial derivatives of $\mF^i$ are uniformly bounded above by $M$ and $M^{-1} \leq \inf_{\mz \in {\mathbb R}^p} \sD_j F^i_j(\mz)$ for $j\in[p]$.
Fixing $\nu, K, M >1$, we follow irons2022triangular and focus on the following more regular affine autoregressive flow as an alternative to $\hat\mS_n^{\rm flow}$ introduced in (ref):
\[
\hat\mG_n^{\rm rflow}
= (\hat\mS_n^{\rm rflow})^{-1}~~\text{ with }
\hat\mS_n^{\rm rflow} \in \argmax_{\mS \in \cF_{\nu,K,M}}
\sum_{i=1}^n \Gamma(\mS, \mZ_i).
\]
assumption[Affine autoregressive flow]
Assume that, for all sufficiently large $n$,
\begin{enumerate}[label=(\alph*)]
• the triangular flow $\hat\mG_n^{\rm rflow} = (\hat\mS_n^{\rm rflow})^{-1}$ is well trained in the sense that
\[
\frac{1}{n}\sum_{i=1}^n \Gamma(\hat\mS_n^{\rm rflow}, \mZ_i)
-
\frac{1}{n}\sum_{i=1}^n \log p_Z(\mZ_i)
=
o_{{\mathrm P}_{\cO}}(1);
\]
• for $r_0 > 0$ defined in Assumption (ref), we have
$$
\cZ \subseteq \Big\{\mz \in {\mathbb R}^p : \|\mz\|_2 \leq (KpM)^{-\nu} r_0\Big\} \subseteq \Big\{\mz \in {\mathbb R}^p : \|\mz\|_2 \leq r_0\Big\} \subseteq \cU.
$$
\end{enumerate}
theorem[Flow bootstrap]
Assume Assumptions (ref), (ref) and (ref), and ${\mathrm E}[|\log p_Z(\mZ)|] < \infty$.
\begin{enumerate}[label=(\alph*)]
• Regular estimators:
If Assumption (ref) holds, then Assumption (ref) is satisfied by the flow generator.
• Irregular estimators:
If Assumptions (ref) and (ref) hold,
then Assumptions (ref) and (ref) are satisfied by the flow generator.
\end{enumerate}
Comparing Theorems (ref) and (ref), we can see that the flow bootstrap has stronger theoretical guarantees for irregular estimators.
Simulation
Methods and implementation
This section complements the theoretical developments with illustrative empirical results. To this end, we compare four bootstrap procedures: Efron's original bootstrap, which resamples from the empirical measure; Efron's smoothed bootstrap, which resamples from a kernel density estimator of the underlying distribution; the GAN bootstrap (Example (ref)); and the flow bootstrap (Example (ref)).
To implement the smoothed bootstrap, we employ the tophat kernel and select the bandwidth according to Silverman's rule of thumb. The kernel density estimator is fitted using the implementation provided in scikit-learn scikit-learn.
To implement the GAN bootstrap, we specify both the generator and the discriminator as fully connected neural networks with fixed width $200$ and depth $6$ across all simulation settings. A dropout probability of $0.4$ is applied to all hidden layers (but not the input nor output layers) during training. The weight matrices in the generator are initialized with i.i.d.\ Gaussian entries with mean zero and variance $0.02$, and the bias vectors are initialized as $\mathbf{0}$. The same initialization scheme is adopted for the discriminator. Both networks are trained using full-batch ADAM with learning rate $0.0001$ and parameters $\beta_1 = 0.5$ and $\beta_2 = 0.9$. The training procedure follows Algorithm 1 of gulrajani2017improved, implemented as in cao_NF_imp. We train the generator for $2000$ steps; for each generator update, the discriminator is updated $5$ times. The gradient penalty coefficient used in discriminator training is set as $\lambda = 1$.
To implement the flow bootstrap, we adopt the GLOW architecture kingma2018glow using the implementation provided in duan_NF_imp. The flow model has depth $10$. The parameters in the ActNorm layers are initialized as $\mathbf{0}$. The neural networks used in the affine coupling layers are fully connected $\tanh$ networks with width $8$ and depth $3$, initialized using the default PyTorch initialization. We select the RealNVP option in the implementation. Each invertible convolution is initialized as a random orthogonal matrix. Training is performed using full-batch ADAM with learning rate $0.005$ and the default PyTorch values of $\beta_1$ and $\beta_2$. The flow model is trained for $1000$ steps.
All implementations are carried out in PyTorch paszke2019pytorch. The code to reproduce all simulation results is available at \url{https://github.com/leonkt/generative_modeling_for_bootstrap}.
Regular estimator: ordinary least squares
In our first simulation setting, we generate the following independent base random variables:
\[
S \sim \mathrm{Unif}[-4,4], \qquad
\epsilon_1, \ldots, \epsilon_{p-1} \sim \mathrm{Unif}[-0.5,0.5],
\qquad \text{and} \qquad
\epsilon_p \sim \mathrm{Unif}[-7,7].
\]
We then construct the predictor vector $\mX \in {\mathbb R}^{p-1}$ as follows. First, we independently sample
\[
X_j \sim \mathrm{Beta}(2,5),
\qquad \text{for } j \in [5].
\]
For the remaining coordinates, we set
\[
X_j
= \sin\Big\{ \frac{(j+1)S}{p} \Big\}
+ \cos\Big\{ \frac{(j+1)S}{p} \Big\}
+ \epsilon_j,
\qquad \text{for } j \in [p-1] \setminus [5].
\]
The response variable $Y$ is generated according to
\[
Y = \boldsymbol{\beta}_0^\top \mX + \epsilon_p,
\]
where the regression coefficient $\boldsymbol{\beta}_0 = (1,\ldots,1)^\top \in {\mathbb R}^{p-1}$ is the parameter of interest. The observed data vector is $\mZ = (\mX^\top, Y)^\top \in {\mathbb R}^{p}$. The estimator $\hat{\boldsymbol{\beta}}_n$ of $\boldsymbol{\beta}_0$ is the ordinary least squares (OLS) estimator. Without loss of generality, we do not include an intercept term in the OLS specification.
Table (ref) reports the empirical coverage probabilities for the OLS estimator. We vary the dimension $p \in \{24,50,100\}$ and the sample size $n \in \{500,1000,2000\}$. The empirical coverage probabilities are computed using elliptical confidence regions based on $500$ Monte Carlo replications.
More specifically, in each replication and under each bootstrap scheme, we generate $1000$ bootstrap samples and compute the corresponding least squares estimator, denoted by $\tilde{\boldsymbol{\beta}}_n$, for each resample. For the smoothed, GAN, and flow bootstraps, we additionally draw $50{,}000$ samples from the learned bootstrap distribution and compute the least squares estimator based on this large synthetic sample, denoted by $\tilde{\boldsymbol{\beta}}_0$. For the original bootstrap, the confidence ball is centered at $\hat{\boldsymbol{\beta}}_n$, which we set $\tilde{\boldsymbol{\beta}}_0$ to be.
More specifically, the empirical coverage is assessed by computing, in each Monte Carlo replication, the statistic
\[
n \Big\| \hat{\boldsymbol{\beta}}_n - \boldsymbol{\beta}_0 \Big\|_2^2,
\]
and comparing it with the empirical $(100 \cdot \alpha)\%$-quantile of
\[
n \Big\| \tilde{\boldsymbol{\beta}}_n - \tilde{\boldsymbol{\beta}}_0 \Big\|_2^2,
\]
approximated using the $1000$ bootstrap samples. We consider significance levels $\alpha \in \{0.90, 0.95\}$.
It can be readily observed that, in most cases, the smoothed bootstrap exhibits substantial distortion. In contrast, the original bootstrap, the GAN bootstrap, and the flow bootstrap perform markedly better. We emphasize that this setting is known to favor Efron's original bootstrap mammen1993bootstrap. Therefore, the fact that the GAN and flow bootstraps are able to match its performance is particularly revealing.
Isotonic Regression
In our second simulation setting, we generate bivariate data $\mZ = (X,Y)^\top$ such that
\[
X \sim \mathrm{Unif}[0,1],
\qquad
Y = X + \epsilon,
\qquad
\epsilon \sim \mathrm{Unif}[-0.01,0.01],
\]
where $X$ and $\epsilon$ are independent. In this setting, the true regression function is $f_0(x) = x$. We fix $x_0 = 0.5$ and evaluate the empirical coverage probabilities of the $100 \cdot \alpha\%$ confidence intervals for $f_0(x_0)$ constructed using the four bootstrap schemes.
Table (ref) reports the empirical coverage probabilities of the different bootstrap schemes for varying sample sizes. Coverage probabilities are computed over $500$ Monte Carlo replications.
In each replication and under each bootstrap scheme, we generate $1000$ bootstrap samples, retain those with covariates in $[0,1]$, and compute the isotonic regression estimator evaluated at $x_0$, denoted by $\tilde{f}_n(x_0)$, for each bootstrap sample. For the smoothed, GAN, and flow bootstraps, we additionally generate $50{,}000$ samples from the learned bootstrap distribution and define $\tilde{f}_0(x_0)$ as the local average
\[
\tilde{f}_0(x_0)
=
\frac{1}{\big| \{ 1 \le i \le 50000 : 0.4999 \le \tilde{X}_i \le 0.5001 \} \big|}
\sum_{i : 0.4999 \le \tilde{X}_i \le 0.5001} \tilde{Y}_i.
\]
For Efron's original bootstrap, we set $\tilde{f}_0(x_0) = \hat{f}_n(x_0)$, where $\hat{f}_n(x_0)$ is computed from the original data.
Empirical coverage is evaluated by computing, in each Monte Carlo replication, the statistic
\[
n^{1/3} \big( \hat{f}_n(x_0) - f_0(x_0) \big),
\]
and comparing it with the equal-tailed $(100 \cdot \alpha)\%$ confidence interval of
\[
n^{1/3} \big( \tilde{f}_n(x_0) - \tilde{f}_0(x_0) \big),
\]
approximated using the $1000$ bootstrap samples. We consider significance levels $\alpha \in \{0.90, 0.95\}$.
It can be seen that the original bootstrap fails in this setting, as expected. In contrast, the remaining three approaches—the smoothed, GAN, and flow bootstraps—all deliver satisfactory empirical coverages.
{
table[table omitted — 1,251 chars of source]
}
{
table[table omitted — 693 chars of source]
}
Proofs of main theorems
We start this section with an introduction to additional notation and conventions.
We use ${\mathrm P}$ as shorthand for the joint distribution of
$(\mZ, \mZ_1,\mZ_2,\ldots)$, $(\mU, \mU_1,\mU_2,\ldots)$,
and $(\tilde\mU_1,\tilde\mU_2,\ldots)$.
The bootstrap samples are denoted by $\tilde \mZ = \hat \mG_n(\mU)$ and $\tilde \mZ_{i} = \hat\mG_n(\tilde \mU_i)$, the latter of which will sometimes be written as $\tilde \mZ_{i,n}$ when we need to emphasize the dependence of the distribution on $n$.
We use $\bP_{\mid \cO}$ to denote the (regular) conditional probability of ${\mathrm P}$ given $\cO$.
Consider $\mV, \mW, \mV_1, \mV_2, ...$ to be some general random variables in ${\mathbb R}^r$. The distribution of $\mV$ under ${\mathrm P}$ is written as ${\mathrm P}_V$. Its conditional distribution under $\bP_{\mid \cO}$ will be written as $\bP_{V \mid \cO}$.
Take a non-random sequence of real numbers, $a_n > 0$, converging to $0$. We say $\mV_{n} = O(a_n)$ if $\limsup_{n\rightarrow \infty} \|\mV_n\|_2/a_n < \infty$ almost surely. We say $\mV_n = o(a_n)$ if $\lim_{n\rightarrow\infty} \|\mV_n\|_2/a_n \rightarrow 0$ almost surely.
We say $\mV_{n} = O_{{\mathrm P}}(a_n)$ if for every $\epsilon > 0$, there exists an $M_\epsilon > 0 $ such that $\limsup_{n\rightarrow\infty} {\mathrm P}(\|\mV_n\|_2/a_n \leq M_\epsilon) > 1-\epsilon$. We say $\mV_n = \Theta_{\mathrm P}(a_n)$ if $\mV_n = O_{\mathrm P}(a_n)$ and, for every $\epsilon > 0$, there exists an $m_\epsilon > 0$ such that $\limsup_{n\rightarrow\infty}{\mathrm P}(\|\mV_n\|_2/a_n \leq m_\epsilon) < \epsilon$ We say $\mV_n = o_{{\mathrm P}}(a_n)$ if for every $\epsilon > 0$, $\lim_{n\rightarrow\infty} {\mathrm P}(\|\mV_n\|_2/a_n > \epsilon) = 0$. When $a_n = 1$ for all $n=1,2,...$, an alternate notation to $\mV_n=o_{\mathrm P}(1)$ is $\mV_n \overset{{\mathrm P}}{\rightarrow} 0$.
The space of continuous, real-valued functions on some compact set $\cE \subseteq {\mathbb R}^r$ is denoted by $C(\cE)$. It is turned into a measurable space by endowing it with the supremum norm and giving it the Borel $\sigma$-algebra. In the following, consider $G_0, G_1, G_2, ...$ as a sequence of $C(\cE)$-valued random variables.
If $V$ is a real-valued random variable, then we define
$$
\|V\|_{\bP_{\mid \cO},\psi_2} = \inf\{C >0 : {\mathrm E}[\exp(V^2/C^2) \mid \cO] \leq 2 \text{ almost surely}\}.
$$
For a normed space $\cN$ with norm $\|\cdot \|$ and a positive number $\epsilon > 0$, we denote the closed $\epsilon$-ball centered at $x \in \cN$ as
$$
\cB(x, \epsilon, \|\cdot \|) = \Big\{m \in \cN : \|x_i - m\| \leq \epsilon\Big\}.
$$
Then, for any subset $\cS \subseteq \cN$, we denote its $\epsilon$-covering number by $N(\epsilon, \cS, \|\cdot\|)$, namely,
\[
N(\epsilon, \cS, \|\cdot\|):= \inf\Big\{N: \text{there exist some } \{x_i\}_{i=1}^N, x_i \in \cN \text{ such that } \cS \subseteq\bigcup_{i=1}^n \cB(x_i, \epsilon, \|\cdot\|)\Big\}.
\]
A sequence of random variables $\mV_1, \mV_2, ...$ is said to converge weakly to $\mV$ (resp. $G_1, G_2, ...$ converges weakly to $G_0$) if, for every bounded, continuous function $f : {\mathbb R}^r \rightarrow {\mathbb R}$, (resp. $f : C(\cE) \rightarrow {\mathbb R}$)
$$
{\mathrm E}[f(\mV_n)] - {\mathrm E}[f(\mV)] \rightarrow 0 \qquad \text{(resp. ${\mathrm E}[f(G_n)] - {\mathrm E}[f(G_0)] \rightarrow 0$})
$$
A sequence of random variables $\mV_1, \mV_2, ...$ is said to converge weakly to $\mV$ conditionally on $\cO$ (resp. $G_1, G_2,...$ converges weakly to $G_0$ conditionally on $\cO$) if, for every bounded continuous function $f : {\mathbb R}^r \rightarrow {\mathbb R}$, (resp. $f : C(\cE) \rightarrow {\mathbb R}$)
$$
{\mathrm E}[f(\mV_n) \mid \cO] - {\mathrm E}[f(\mV)\mid \cO] \rightarrow 0 \qquad \text{(resp. ${\mathrm E}[f(G_n) \mid \cO] - {\mathrm E}[f(G_0) \mid \cO] \rightarrow 0$})
$$
almost surely.
In the context of Section (ref), taking any $\ell, u > 0$, we define positive sequences $\ell_n = x_0 - \ell n^{-1/3}$ and $u_n = x_0 + un^{-1/3}$, and introduce the following notation for local averages:
align*[align* omitted — 1,036 chars of source]
In addition, for any $\ell > 0$ and $u \ge 0$, define
\[
G_{\ell,u}
:=
\frac{\sigma}{\sqrt{p_X(x_0)}}
\cdot
\frac{\bB(u) - \bB(-\ell)}{u+\ell}
+
\frac{f_0'(x_0)}{2}(u-\ell).
\]
Proof of Theorem (ref)
proofWe appeal to Lemma (ref) and thus, in the following, we will be explicit about the dependence of $\tilde{\boldsymbol{\eta}}_0$ on $n$, and denote it as $\tilde{\boldsymbol{\eta}}_{0,n}$. Take any subsequence $n_k$. It suffices to find a subsequence $n_{k_\ell}$ such that
$$
\sup_{\mt \in {\mathbb R}^q} \Big|\mathrm{P}\left(\sqrt{n_{k_\ell}} (\tilde{\boldsymbol\eta}_{n_{k_\ell}} - \tilde{\boldsymbol\eta}_{0,n_{k_\ell}}) \leq \mt| \mathcal{O}\right) - \mathrm{P}\left(\sqrt{n_{k_\ell}}(\hat{\boldsymbol\eta}_{n_{k_\ell}} - \boldsymbol\eta_0) \leq \mt\right)\Big| \rightarrow 0
$$
almost surely. By Polya's theorem (Lemma (ref)), it suffices to show that there is a subsequence $n_{k_\ell}$ and a Gaussian random variable $\mZ'$ such that $\sqrt{n_{k_\ell}} (\tilde{\boldsymbol\eta}_{n_{k_\ell}} - \tilde{\boldsymbol\eta}_{0,n_{k_\ell}})$ converges weakly to $\mZ'$ conditionally on $\cO$, and $\sqrt{n_{k_\ell}}(\hat{\boldsymbol\eta}_{n_{k_\ell}} - \boldsymbol\eta_0)$ converges weakly to $\mZ'$. Equivalently, by the Cramer-Wold device (Lemma (ref)), it suffices to show that, for any $\ma \in {\mathbb R}^q$, we have $\sqrt{n_{k_\ell}} \ma^\top(\tilde{\boldsymbol\eta}_{n_{k_\ell}} - \tilde{\boldsymbol\eta}_{0,n_{k_\ell}})$ converges weakly to $\ma^\top\mZ'$ conditionally on $\cO$ and $\sqrt{n_{k_\ell}}\ma^\top(\hat{\boldsymbol\eta}_{n_{k_\ell}} - \boldsymbol\eta_0)$ converges weakly to $\ma^\top\mZ'$.
\\
Step 1: Reduce to an appropriate, almost surely converging subsequence $n_{k_\ell}$.
Take any $\ma \in {\mathbb R}^q$.
By Lemma (ref), $ {\mathrm P}(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n}, \tilde\mZ_{1,n}) \mid \cO]) \text{ is invertible}) \rightarrow 1$, and
$$
\|\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n}, \tilde\mZ_{1,n}) \mid \cO]^{-1} - \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)]^{-1}\|_{\rm op} = o_{{\mathrm P}}(1) ~~~\text{and}~~~ \|\tilde{\boldsymbol\eta}_{0,n} - \boldsymbol{\eta}_0\|_2 = o_{{\mathrm P}}(1).
$$
Furthermore, $(\boldsymbol{\eta}, \mz) \mapsto \sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}, \mz)$ is continuous on $\cK \times (\cZ \cup \tilde\cZ)$ from Assumption (ref)(b,c). Therefore,
\begin{equation}
\sup_{\boldsymbol{\eta} \in \cK}\|{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}, \tilde\mZ_{1,n}) \mid \cO) - {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}, \mZ))\|_{\rm max} = o_{{\mathrm P}}(1)
\end{equation}
by Lemma (ref). Continuity of $\boldsymbol{\eta} \mapsto {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}, \mZ))$ on $\cK$, by the bounded convergence theorem, combined with (ref) implies
\begin{align*}
&\|{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_{0,n}, \tilde\mZ_{1,n}) \mid \cO) - {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}_0, \mZ))\|_{\rm max} \\
\leq& \|{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_{0,n}, \tilde\mZ_{1,n}) \mid \cO) - {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_{0,n}, \mZ))\|_{\rm max} + \|{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_{0,n}, \mZ)) - {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}({\boldsymbol{\eta}}_{0}, \mZ))\|_{\rm max} \%\tag{triangle inequality}\\
=& o_{{\mathrm P}}(1) + \|{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_{0,n}, \mZ)) - {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}({\boldsymbol{\eta}}_{0}, \mZ))\|_{\rm max}
\tag{Lemma (ref)}\\
=& o_{{\mathrm P}}(1). \tag{$\|\tilde{\boldsymbol\eta}_{0,n} - \boldsymbol{\eta}_0\|_2 = o_{{\mathrm P}}(1)$ and continuity of $\boldsymbol{\eta} \mapsto {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}, \mZ))$ on $\cK$}
\end{align*}
Appealing to Lemma (ref), we select a subsequence $n_{k_\ell}$ such that $\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO]^{-1}$ exists for each $n_{k_\ell}$, almost surely,
$$
\|{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO) - {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}_0, \mZ))\|_{\rm max} \rightarrow 0
$$
almost surely,
$$
\|\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO]^{-1} - \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)]^{-1}\|_{\rm op} \rightarrow 0, ~~~\text{and}~~~ \|\tilde{\boldsymbol\eta}_{0,n_{k_\ell}} - \boldsymbol{\eta}_0\|_2 \rightarrow 0
$$
almost surely. Applying Lemma (ref) again, we obtain
$$
\sqrt{n_{k_\ell}}\ma^\top(\hat{\boldsymbol\eta}_{n_{k_\ell}} - \boldsymbol\eta_0) = -\frac{1}{\sqrt{n_{k_\ell}}} \sum_{i=1}^{n_{k_\ell}} \ma^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)])^{-1} \sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ_i) + o_{{\mathrm P}}(1)
$$
and
$$
\sqrt{n_{k_\ell}}\ma^\top(\tilde{\boldsymbol\eta}_{n_{k_\ell}} -\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}) = -\frac{1}{\sqrt{n_{k_\ell}}} \sum_{i=1}^{n_{k_\ell}} \ma^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1}\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde \mZ_{i,n_{k_\ell}}) + o_{{\mathrm P}}(1).
$$
\\
Step 2: Apply the central limit theorem to the linear representation of $\sqrt{n_{k_\ell}}\ma^\top(\hat{\boldsymbol\eta}_{n_{k_\ell}} - \boldsymbol\eta_0)$.
The function $\mz \mapsto \sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mz)$ is bounded on $\cZ$, since $\mz \mapsto \sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mz)$ is continuous by Assumption (ref)(c) and $\cZ$ is compact by Assumption (ref)(b). The Cauchy-Schwarz inequality implies
$$
{\mathrm E}[\ma^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)])^{-1}\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ)\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ)^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)])^{-1} \ma] < \infty.
$$ Furthermore, Assumption (ref)(a) implies that
$$ \ma^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)])^{-1} \sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ_{n_{k_1}}), \ma^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)])^{-1} \sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ_{n_{k_2}}), ...
$$
are independent, identically distributed and square integrable.
Define $\mZ' \in {\mathbb R}^q$ as a Gaussian random variable with mean zero and variance matrix
$$
\boldsymbol{\Sigma} := \sD_{\boldsymbol{\eta}}^2{\mathrm E}[ {\sf L}(\boldsymbol{\eta}_0, \mZ)]^{-1}{\mathrm E}[\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ)\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ)^\top]\sD_{\boldsymbol{\eta}}^2{\mathrm E}[ {\sf L}(\boldsymbol{\eta}_0, \mZ)]^{-1}.
$$
The central limit theorem indicates that $\sqrt{n_{k_\ell}}\ma^\top(\hat{\boldsymbol\eta}_{n_{k_\ell}} - \boldsymbol\eta_0)$ converges weakly to a mean-zero Gaussian random variable with variance matrix
$$
\ma^\top{\mathrm E}[\sD_{\boldsymbol{\eta}}^2 {\sf L}(\boldsymbol{\eta}_0, \mZ)]^{-1}{\mathrm E}[\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ)\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ)^\top]{\mathrm E}[\sD_{\boldsymbol{\eta}}^2 {\sf L}(\boldsymbol{\eta}_0, \mZ)]^{-1}\ma = \ma^\top \boldsymbol{\Sigma}\ma.
$$
This is exactly the variance matrix of $\ma^\top \mZ'$, so we have shown that $\sqrt{n_{k_\ell}}\ma^\top(\hat{\boldsymbol\eta}_{n_{k_\ell}} - \boldsymbol\eta_0)$ converges weakly to $\ma^\top \mZ'$.
\\
Step 3: Apply the Lyapunov central limit theorem (Lemma (ref)) to the linear representation of $\sqrt{n_{k_\ell}}\ma^\top(\tilde{\boldsymbol\eta}_{n_{k_\ell}} -\tilde{\boldsymbol\eta}_{0,n_{k_\ell}})$.
By Assumptions (ref) and (ref)(a), the random variables $\{\tilde\mZ_{i,n}\}_{n\geq 1, 1\leq i \leq n}$ form a triangular array conditional on $\cO$, so as the random variables
$$
\Big\{\ma^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1}\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde \mZ_{i,n_{k_\ell}})\Big\}_{\ell \geq 1, 1 \leq i \leq n_{k_\ell}}.
$$
By the Cauchy-Schwarz inequality, for any $\check p \geq 1$,
\begin{align*}
{\mathrm E}\Bigg[\left|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1}\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}})^\top \ma \right|^{\check p}\mid \cO\Bigg] \leq \\ {\mathrm E}\Bigg[\left|\|\ma\|_2 \cdot \Big\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1} \Big\|_{\rm op} \cdot \|\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}})\|_2\right|^{\check p} \mid \cO \Bigg],
\end{align*}
which is further upper bounded by
\begin{equation}
\left(\|\ma\|_2 \cdot \Big\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1} \Big\|_{\rm op} \cdot \sup_{(\boldsymbol\eta, \mz) \in \cK \times \tilde\cZ}\|\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta, \mz)\|_2\right)^{\check p}.
\end{equation}
For notational concision, let
$$
\tilde V_{i, n_{k_\ell}}:= \ma^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1}\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde \mZ_{i,n_{k_\ell}}) ~~~~~~\text{and}~~~~~~ S_{n_{k_\ell}}^2 := \sum_{i=1}^{n_{k_\ell}} {\textrm{Var}}(\tilde V_{i, n_{k_\ell}} \mid \cO).
$$
When $\check p = 2$, (ref) verifies that
$$
{\mathrm E}\Big[\tilde V_{i, n_{k_\ell}}^2 | \cO\Big] < \infty
$$
almost surely. Also, ${\mathrm E}\Big[\tilde V_{i, n_{k_\ell}}^2 | \cO\Big] >0$ due to Assumption (ref)(b) that guarantees that $\tilde\mZ_{1}$ has nonzero variance, conditional on $\cO$, almost surely.
We then check that the Lyapunov condition
$$
\frac{1}{S_{n_{k_\ell}}^3} \sum_{i=1}^{n_{k_\ell}} {\mathrm E}\Big[|\tilde V_{i, n_{k_\ell}}|^3 \mid \cO\Big] \rightarrow 0
$$
holds almost surely, as $\ell\rightarrow \infty$. Since
$\{\tilde V_{i, n_{k_l}}\}_{i=1}^{n_{k_l}}$ are identically distributed, the equivalent condition to check is that
$$
\frac{n_{k_\ell} \cdot {\mathrm E}[|\tilde V_{1, n_{k_\ell}}|^3 \mid \cO]}{n_{k_\ell}^{3/2}{\textrm{Var}}(\tilde V_{1, n_{k_\ell}} \mid \cO)^{3/2}} \rightarrow 0
$$
holds almost surely, as $\ell \rightarrow \infty$. For all $\ell \geq 1$,
\begin{align*}
|\tilde V_{1, n_{k_l}}|^3 \leq \left(\|\ma\|_2 \cdot \Big\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1} \Big\|_{\rm op} \cdot \sup_{(\boldsymbol\eta, \mz) \in \cK \times \tilde\cZ}\|\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta, \mz)\|_2\right)^3
\end{align*}
by (ref) with $\check p=3$. Using the property of our subsequence, and the fact that $\|\mathbf{A}\|_{\rm op} \leq q\|\mathbf{A}\|_{\rm max}$ for matrices $\mathbf A \in {\mathbb R}^{q \times q}$,
$$
\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO]^{-1} \|_{\rm op} \rightarrow \|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol\eta_{0}, \mZ)]^{-1} \|_{\rm op} < \infty
$$
almost surely, by invertibility of the right-hand side by Assumption (ref)(d). Handling the denominator using Lemma (ref) and our choice of subsequence, ${\textrm{Var}}(\tilde V_{1, n_{k_\ell}} \mid \cO)$ converges to a nonzero constant almost surely.
Therefore, sending $\ell \rightarrow \infty$, we establish that
$$
\frac{n_{k_\ell} \cdot {\mathrm E}[|\tilde V_{1, n_{k_\ell}}|^3 \mid \cO]}{n_{k_\ell}^{3/2}{\textrm{Var}}(\tilde V_{1, n_{k_\ell}} \mid \cO)^{3/2}} \rightarrow 0
$$
almost surely. By the Lyapunov central limit theorem, Lemma (ref),
$\sum_{i=1}^{n_{k_\ell}} \tilde{V}_{i,n_{k_\ell}}/ S_{n_{k_\ell}}$ then converges weakly to a standard normal conditionally on $\cO$ .
\\
Step 4: Verify the limiting distributions are the same, and conclude.
Notice that
\begin{equation}
\frac{1}{n_{k_\ell}} S_{n_{k_\ell}}^2 = {Var}(\tilde V_{1, n_{k_\ell}} \mid \cO)
\end{equation}
by the identical distribution of $\{\tilde V_{i, n_{k_\ell}}\}_{i=1}^{n_{k_\ell}}$ conditional on $\cO$. The right-hand side of (ref) is, by definition,
$$
\ma^\top(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1}{\mathrm E}[\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde \mZ_{i,n_{k_\ell}}) \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde \mZ_{i,n_{k_\ell}})^\top \mid \cO] (\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO])^{-1}\ma.
$$
First,
\begin{align*}
\|\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO]^{-1} - \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)]^{-1}\|_{\rm max} &\leq \|\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO]^{-1} - \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)]^{-1}\|_{\rm op}\\
&\rightarrow 0
\end{align*}
almost surely, by the definition of $n_{k_\ell}$. In addition,
$$
\|{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO) - {\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}_0, \mZ))\|_{\rm max} \rightarrow 0
$$
almost surely by the definition of $n_{k_\ell}$. Since $\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}})$ has zero mean conditional on $\cO$, and $\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}_0, \mZ)$ is zero mean,
$$
{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde\mZ_{1,n_{k_\ell}}) \mid \cO) = {\mathrm E}[\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde \mZ_{i,n_{k_\ell}}) \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_{0,n_{k_\ell}}, \tilde \mZ_{i,n_{k_\ell}})^\top \mid \cO]
$$
and
$$
{\rm Var}(\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol{\eta}_0, \mZ)) = {\mathrm E}[\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ)\sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ)^\top].
$$
Hence,
$$
n_{k_\ell}^{-1/2}S_{n_{k_\ell}}/\sqrt{\ma^\top \boldsymbol\Sigma \ma} \rightarrow 1
$$
almost surely, so that Slutsky's theorem implies
$$
n_{k_\ell}^{-1/2}\sum_{i=1}^{n_{k_\ell}} \tilde V_{i, n_{k_\ell}}\Big/ \sqrt{\ma^\top \boldsymbol{\Sigma} \ma}
$$
converges weakly to a standard normal conditionally on $\cO$. Applying the continuous mapping theorem,
$n_{k_\ell}^{-1/2}\sum_{i=1}^{n_{k_\ell}} \tilde V_{i, n_{k_\ell}}
$
converges weakly to $\sqrt{\ma^\top \boldsymbol{\Sigma}\ma} $ times a standard normal, conditionally on $\cO$. This limit is equal in distribution to $\ma^\top \mZ'$. Since $\ma$ was arbitrary, by the Cramer-Wold device we conclude $\sqrt{n_{k_\ell}}(\tilde{\boldsymbol\eta}_{n_{k_\ell}} -\tilde{\boldsymbol\eta}_{0,n_{k_\ell}})$ converges to $\mZ'$ conditionally on $\cO$ .
By comparing the weak limits, Polya's theorem, and Lemma (ref), we establish
$$
\sup_{\mt \in {\mathbb R}^q} \Big|\mathrm{P}\left(\sqrt{n} (\tilde{\boldsymbol\eta}_{n} - \tilde{\boldsymbol\eta}_{0}) \leq \mt| \mathcal{O}\right) - \mathrm{P}\left(\sqrt{n}(\hat{\boldsymbol\eta}_{n} - \boldsymbol\eta_0) \leq \mt\right)\Big| = o_{{\mathrm P}}(1).
$$
This completes the proof.
Proof of Theorem (ref)
proofTake any $\epsilon > 0$. For the random variables $(\tilde L^* , \tilde U^*)$ defined in Lemma (ref), and $(L_G^*, U_G^*)$ defined in Lemma (ref), we choose a $K_\epsilon > 0$ large enough and $k_\epsilon >0$ small enough so that
$$
\limsup_{n\rightarrow\infty}\Big\{{\mathrm P}(\tilde L^* > K_\epsilon) \vee {\mathrm P}( \tilde L^* < k_\epsilon) \vee {\mathrm P}(L_G^* > K_\epsilon) \vee {\mathrm P}(L_G^* <k_\epsilon)\Big\} < \epsilon
$$
and
$$
\limsup_{n\rightarrow\infty}\Big\{{\mathrm P}(\tilde U^* > K_\epsilon) \vee {\mathrm P}( \tilde U^* < k_\epsilon ) \vee {\mathrm P}(U_G^* > K_\epsilon)\vee{\mathrm P}( U_G^* < k_\epsilon)\Big\} < \epsilon.
$$
By Lemma (ref),
$$
\sup_{t \in {\mathbb R}, k_\epsilon \leq u,\ell \leq K_\epsilon} \left|\mathrm{P}\left(n^{1/3}(\overline {\tilde Y}_{[\ell_n, u_n]} - \tilde f_0(x_0)) \leq t \middle | \mathcal{O}\right) - \mathrm{P}\left(G_{\ell, u}\leq t)\right)\right| = o_{{\mathrm P}}(1).
$$
In particular, we work on the event $ \{k_\epsilon \leq \tilde U^* , \tilde L^*, U_G^*, L_G^* \leq K_\epsilon\}$.We use the continuous mapping theorem on the space of random functions with bounded sample paths on $[k_\epsilon,K_\epsilon] \times [k_\epsilon, K_\epsilon]$, with the supremum norm. With respect to this metric, we use the continuous function $h \mapsto \sup_{k_\epsilon \leq \ell \leq K_\epsilon} \inf_{k_\epsilon \leq u \leq K_\epsilon}|h(\ell,u)|$ to obtain
$$
\sup_{t \in {\mathbb R}} \left|{\mathrm P}(\sup_{h_\epsilon < \ell \leq H_\epsilon} \inf_{h_\epsilon \leq u \leq H_\epsilon }n^{1/3}(\overline {\tilde Y}_{[x_0 - \ell n^{-1/3}, x_0 + u n^{-1/3}]} - \tilde f_0(x_0)) \leq t \mid \mathcal{O}) - {\mathrm P}(\sup_{k_\epsilon < \ell \leq K_\epsilon} \inf_{k_\epsilon \leq u \leq K_\epsilon }G_{\ell, u}\leq t)\right| = o_{{\mathrm P}}(1).
$$
By Lemma (ref), we can assume without loss of generality that the max-min formula holds so that
$$
\sup_{t \in {\mathbb R}} \left|\mathrm{P}\left(n^{1/3}(\tilde f_n(x_0) - \tilde f_0(x_0)) \leq t\middle | \mathcal{O}\right) - {\mathrm P}(\sup_{k_\epsilon < \ell \leq K_\epsilon} \inf_{k_\epsilon \leq u \leq K_\epsilon} G_{\ell, u} \leq t)\right| = o_{{\mathrm P}_{\cO}}(1).
$$
Since we are working on the event where $k_\epsilon \leq U_G^*, L_G^* \leq K_\epsilon$, we have, equivalently,
$$
\sup_{t \in {\mathbb R}} \left|\mathrm{P}\left(n^{1/3}(\tilde f_n(x_0) - \tilde f_0(x_0)) \leq t \middle | \mathcal{O}\right) - {\mathrm P}(\sup_{\ell > 0} \inf_{u \geq 0} G_{\ell, u} \leq t)\right| = o_{{\mathrm P}_{\cO}}(1).
$$
The theorem is then proven by Lemma (ref), then comparing the limits.
Proof of Theorem (ref)
proofFirst, Assumption (ref)(c) corresponds to the condition that $\bP_{\tilde Z \mid \cO}$ has nonzero variance almost surely. Second, by definition of $\hat \mG_n^{\rm GAN}$, Assumption (ref)(a) automatically holds. It remains to verify the remaining parts.
\\
Step 1: Show Assumption (ref)(c).
For a vector $\mx \in {\mathbb R}^p$ and a $1$-Lipschitz function $\alpha : {\mathbb R} \rightarrow {\mathbb R}$ applied componentwise,
\begin{align*}
\|\alpha(\mx) - \alpha(\mx)\|_2 &= \sqrt{(|\alpha(x_1) - \alpha(x_1')|)^2 + ... + (|\alpha(x_p) - \alpha(x_p')|)^2} \tag{definition of $\alpha$}\\
&\leq \sqrt{(|x_1 - x_1'|)^2 + ... + (|x_p-x_p'|)^2} \tag{$\alpha$ is 1-Lipschitz by Assumption (ref) and square root function is increasing}\\
&= \|\mx - \mx'\|_2.
\end{align*}\
Then, for $\mathbf{A} \in {\mathbb R}^{r \times p}, \mb \in {\mathbb R}^r$, we have
$$
\|\alpha(\mathbf{A} \mx + \mb) - \alpha(\mathbf{A} \mx + \mb)\|_2 \leq \|\mathbf{A}(\mx - \mx') \|_2 \leq \|\mathbf{A}\|_{\rm op} \cdot \|\mx - \mx'\|_2.
$$
Therefore, for $\mG \in \cF_\alpha(L^{\rm gen}, W_n^{\rm gen}, B^{\rm gen}, p, p)$,
$$
\|\mG(\mx) - \mG(\mx')\|_2 \leq \|\mx - \mx'\|_2 \cdot \prod_{i=1}^{L^{\rm gen}} \|\mathbf{A}^{(i)}\|_{\rm op} \leq (B^{\rm gen})^{L^{\rm gen}} \cdot \|\mx - \mx'\|_2.
$$
That is, the class of functions $\cF_\alpha(L^{\rm gen}, W_n^{\rm gen}, B^{\rm gen}, p, p)$ is uniformly $(B^{\rm gen})^{L^{\rm gen}}$-Lipschitz on ${\mathbb R}^p$. Then, $D \circ \mG$, where $D \in {\rm Lip}(p,1)$, $D(\mathbf{0}) = 0$, and $ \mG \in \cF_\alpha(L^{\rm gen}, W_n^{\rm gen}, B^{\rm gen}, p, p)$ is $(B^{\rm gen})^{L^{\rm gen}}$-Lipschitz on ${\mathbb R}^p$ too.
Next, for $\bm{u} \in \cU$, with $\alpha$ applied componentwise,
\begin{align*}
\|\alpha(\mathbf{A} \bm{u} + \mb)\|_2 &= \|\alpha(\mathbf{A} \bm{u} + \mb) - \alpha(\mathbf{0})\|_2 \tag{Assumption (ref)(a)}\\
&\leq \|\mathbf{A} \bm{u} + \mb\|_2 \\
&\leq \|\mathbf{A} \bm{u}\|_2 + \|\mb\|_2 \tag{triangle inequality}\\
&\leq B^{\rm gen} \cdot \|\bm{u}\|_2 + \|\mb\|_2 \tag{definition of operator norm and $B^{\rm gen}$}\\
&\leq B^{\rm gen} \cdot \|\bm{u}\|_2 + B^{\rm gen}. \tag{$\|\mb\|_2 \leq B^{\rm gen}$}
\end{align*}
Hence, using the recursive definition of $\mG$, and the fact that $\cU$ contains $\mathbf{0}$,
\begin{equation}
\|\mG(\bm{u})\|_2 \leq (B^{\rm gen})^{L^{\rm gen}} \cdot \|\bm{u}\|_2 + \sum_{i=1}^{L^{\rm gen}} (B^{\rm gen})^i.
\end{equation}
Since $\cU$ is compact by Assumption (ref), and $\mG$ is arbitrary, this shows that Assumption (ref)(c) holds, because $\hat \mG_n^{\rm GAN} \in \cF_\alpha(L^{\rm gen}, W_n^{\rm gen}, B^{\rm gen}, p, p).$
\\
Step 2: Show Assumption (ref)(b).
Next, take $\epsilon > 0$. We aim to show
\begin{align*}
{\mathrm P}\left({\sf W}_1(\bP_{\tilde Z \mid \cO}, {\mathrm P}_{Z}) \geq \epsilon \right) &= {\mathrm P}\left(\sup_{D \in {Lip}(p,1)}{\mathrm E}\Big[ D( \hat \mG_n^{\rm GAN} (\mU)) - D(\mZ) \mid \cO\Big] \geq \epsilon\right) \tag{Lemma (ref)}\\
&= {\mathrm P}\left(\sup_{D \in {Lip}(p,1), D(\mathbf{0}) = 0}{\mathrm E}\Big[ D( \hat \mG_n^{\rm GAN} ( \mU)) - D(\mZ) \mid \cO\Big]
\geq \epsilon\right)\\
&= {\mathrm P}\left(\sup_{D \in {Lip}(p,1), D(\mathbf{0}) = 0}{\mathrm E}[{\sf W}(\hat\mG_n^{\rm GAN}, D, \mZ, \mU) \mid \cO]
\geq \epsilon\right)
\end{align*}
goes to zero.
To do so, we first show
\begin{align}
\sup_{D \in {Lip}(p,1), D(\mathbf{0}) = 0} \Big\{\frac{1}{n} \sum_{i=1}^n {\sf W}(\hat\mG_n^{\rm GAN}, D, \mZ_i, \mU_i) - {\mathrm E}\Big[{\sf W}(\hat\mG_n^{\rm GAN}, D, \mZ, \mU) \mid \cO\Big]\Big\} = o_{{\mathrm P}}(1).
\end{align}
Since $D(\mathbf{0}) = 0$, for any $\bm{u} \in \cU$ and $\mz \in \cZ$,
$$
\|D\circ \hat\mG_n^{\rm GAN}(\bm{u})\|_2 \leq (B^{\rm gen})^{L^{\rm gen}} \cdot \|\bm{u}\|_2 + \sum_{i=1}^{L^{\rm gen}} (B^{\rm gen})^i
~~~\text{and}~~~
\|D(\mz)\|_2 \leq \|\mz\|_2,
$$
by (ref). Summing up our calculations, and using the notation of Lemma (ref), we obtain
$$
\Big\{D\circ \hat\mG_n^{\rm GAN} : D \in {\rm{Lip}}(p,1), D(\mathbf{0}) = 0 \Big\} \subseteq BL\left(\cU, (B^{\rm gen})^{L^{\rm gen}} + (B^{\rm gen})^{L^{\rm gen}} \cdot \sup_{\bm{u} \in \cU}\|\bm{u}\|_2 + \sum_{i=1}^{L^{\rm gen}} (B^{\rm gen})^i \right)
$$
and
$$
\Big\{D : D \in {\rm{Lip}}(p,1), D(\mathbf{0}) = 0 \Big\} \subseteq BL\left(\cZ, \sup_{\mz \in \cZ}\|\mz\|_2 + 1\right).
$$
Accordingly, Lemmas (ref) and (ref) imply (ref). Using the first condition in Assumption (ref)(b), it then suffices to show
$$
\frac1n \sum_{i=1}^n
{\sf W}(\hat\mG_n^{\rm GAN}, \hat D_n^{\rm GAN}, \mZ_i, \mU_i) = o_{{\mathrm P}}(1),
$$
which is exactly the second condition of Assumption (ref)(b). We thus conclude the proof.
Proof of Theorem (ref)
proofStep 1: Proof of Theorem (ref)(a).
Assumption (ref)(a) holds automatically by the definition of $\hat \mS_n^{\rm rflow}$. We then verify the remaining parts.
\noindentStep 1a: Show that $\tilde \cZ_n \subseteq \tilde \cZ$ for some nonrandom compact set $\tilde \cZ \subseteq {\mathbb R}^p$.
Take any $\mS = \mF^{\nu} \circ \boldsymbol{\Sigma}_\nu \circ ... \circ \mF^1 \circ \boldsymbol{\Sigma}_1 \in \cF_{\nu, K, M}$. Since $K >0$, part (a) of the definition of $\cT_\uparrow (p)$ and part (a) of the definition of $\cF_{\nu, K,M}$ indicate that $\mS$ is a continuously differentiable and bijective map from ${\mathbb R}^p$ to ${\mathbb R}^p$. Hence, $\mS^{-1}: {\mathbb R}^p \rightarrow {\mathbb R}^p$ exists. Since
\begin{align*}
\sD \mS(\mz) &= \sD(\mF^{\nu} \circ \boldsymbol{\Sigma}_\nu \circ ... \circ \mF^1 \circ \boldsymbol{\Sigma}_1)(\mz) \tag{definition of $S$}\\
&= \sD\mF^{\nu} (\boldsymbol{\Sigma}_\nu \circ ... \circ \mF^2 \circ \mF^1 \circ \boldsymbol{\Sigma}_1 (\mz)) \sD(\boldsymbol{\Sigma}_\nu \circ ... \circ \mF^1 \circ \boldsymbol{\Sigma}_1)(\mz) \tag{chain rule}\\
&= \sD\mF^{\nu} (\boldsymbol{\Sigma}_\nu \circ ... \circ \mF^2 \circ \mF^1 \circ \boldsymbol{\Sigma}_1 (\mz)) \boldsymbol{\Sigma}_\nu \sD(\mF^{\nu-1} \circ ... \circ \mF^1 \circ \boldsymbol{\Sigma}_1)(\mz) \tag{pull out constant matrix $\boldsymbol{\Sigma}_\nu$}
\end{align*}
for any $\mz \in {\mathbb R}^p$, we apply the chain rule $\nu-1$ more times in the same manner, to obtain
$$
\sD \mS(\mz) = \sD\mF^{\nu}(\mx^{\nu}) \boldsymbol{\Sigma}_\nu \, ... \, \sD \mF^1(\mx^1) \boldsymbol{\Sigma}_1 \qquad \text{with} \qquad \mx^{i} = (\boldsymbol{\Sigma}_i \circ ... \circ \mF^1 \circ \boldsymbol{\Sigma}_1)(\mz),
$$
which yields
$$
\det(\sD \mS(\mz)) = \prod_{i=1}^{\nu} \det(\sD\mF^{i}(\mx^{i})) \cdot \det(\boldsymbol{\Sigma}_i).
$$
Since $\mF^1, ..., \mF^\nu \in \cT_\uparrow (p),$ and the determinant of an upper triangular matrix is the product of its diagonal terms, we obtain
$$
\det(\sD \mF^i(\mz)) = \prod_{j=1}^p \sD_j F^i_j(\mz)
$$
so that
$$
\det(\sD \mS(\mz)) = \prod_{i=1}^{\nu} \prod_{j=1}^p \sD_j F^i_j(\mz) \cdot \det(\boldsymbol{\Sigma}_i).
$$
The definition of the class $\cF_{\nu, K, M}$ indicates that
\begin{equation}
(K^pM)^{-\nu} \leq \det(\sD \mS(\mz)) \leq (K^pM)^{\nu}.
\end{equation}
Since $\mz$ is arbitrary and applying Lemma (ref), $\mS^{-1}$ is also continuously differentiable. Since $\hat \mS_n^{\rm rflow} \in \cF_{\nu, K, M}$ , this argument shows $\hat \mG_n^{\rm rflow} = (\hat \mS_n^{\rm rflow})^{-1}$ is continuously differentiable on ${\mathbb R}^p$.
Moreover, by Lemma (ref), we have
$$
\sD(\mS^{-1})(\mz) = \sD\mS(\mS^{-1}(\mz))^{-1}.
$$
Accordingly, letting $\my = \mS^{-1}(\mz)$ and $\my^i = (\boldsymbol{\Sigma}_i \circ ... \circ \mF^1 \circ \boldsymbol{\Sigma}_1)(\my)$ for each $i \in [\nu]$, we obtain
\begin{align*}
\sD\mS(\my)^{-1} &=
(\sD\mF^{\nu}(\my^{\nu}) \boldsymbol{\Sigma}_\nu \, ... \, \sD \mF^1(\my^1) \boldsymbol{\Sigma}_1)^{-1} \tag{formula for $\sD \mS$}\\
&=\boldsymbol{\Sigma}_1^{-1} (\sD \mF^1(\my^1))^{-1} ... \boldsymbol{\Sigma}_\nu^{-1} (\sD\mF^{\nu}(\my^{\nu}))^{-1}. \tag{formula for inverse of product of invertible matrices}
\end{align*}
We thus reach that
\begin{align}
\|\sD\mS(\my)^{-1}\|_{\rm op} &\leq \|\boldsymbol{\Sigma}_1^{-1} (\sD \mF^1(\my^1))^{-1} ... \boldsymbol{\Sigma}_\nu^{-1} (\sD\mF^{\nu}(\my^{\nu}))^{-1}\|_{\rm op}\nonumber\\
&\leq \prod_{i=1}^\nu \|\boldsymbol{\Sigma}_i^{-1}\|_{\rm op} \cdot \|(\sD\mF^{i}(\my^{i}))^{-1}\|_{\rm op} \nonumber\tag{operator norm of product bound}\\
&= \prod_{i=1}^\nu \lambda_{\rm min}(\boldsymbol{\Sigma}_i)^{-1} \cdot \|(\sD\mF^{i}(\my^{i}))^{-1}\|_{\rm op} \nonumber\tag{$\boldsymbol{\Sigma}_i$ is symmetric and positive definite}\\
&\leq K^\nu \prod_{i=1}^\nu \|(\sD\mF^{i}(\my^{i}))^{-1}\|_{\rm op} \nonumber\tag{definition of $\cF_{\nu, K, M}$}\\
&\leq K^\nu \cdot \prod_{i=1}^\nu M\sqrt{p}\left(1+ M\|\sD\mF^{i}(\my^{i})\|_{\rm op}\right)^{p-1} \nonumber\tag{Lemma (ref)}\\
&\leq K^\nu \cdot \prod_{i=1}^\nu M\sqrt{p}\left(1+ M\sqrt{(pM)^2}\right)^{p-1}\nonumber\\
&= K^\nu \cdot \prod_{i=1}^\nu M\sqrt{p}\left(1+ M^2p\right)^{p-1},\nonumber
\end{align}
where the final inequality comes from the fact that
$$
\|\mathbf{A}\|_{\rm op} \leq \sqrt{\max_{j} \sum_{i\leq j} |A_{ij}| \cdot \max_{i'}\sum_{i' \geq j'} |A_{i'j'}|}
$$
and the fact that the magnitudes of all entries in $\sD\mF^{i}(\my^{i})$ are bounded by $M$ by definition of $\cF_{\nu, K, M}.$ Since $\mz$, and hence, $\my$ is arbitrary, we have
\begin{equation}
\sup_{\mz \in {\mathbb R}^p }\|\sD(\mS^{-1})(\mz)\|_{\rm op} \leq K^\nu \cdot \prod_{i=1}^\nu M\sqrt{p}\left(1+ M^2p\right)^{p-1}= K^\nu M^\nu p^{\nu/2}\left(1+ M^2p\right)^{\nu(p-1)}
\end{equation}
for any $\mS \in \cF_{\nu, K, M}$.
Since $\hat \mG_n^{\rm rflow}$ is continuously differentiable on ${\mathbb R}^p$, the mean value theorem implies
\begin{align}
\|\hat\mG_n^{\rm rflow}(\mz) - \hat\mG_n^{\rm rflow}(\mz')\|_2 &\leq \sup_{\mz”\in{\mathbb R}^p}\|\sD\hat\mG_n^{\rm rflow}(\mz”)\|_{\rm op}\cdot\| \mz - \mz'\|_2 \nonumber\\
&\leq K^\nu M^\nu p^{\nu/2}\left(1+ M^2p\right)^{\nu(p-1)}\cdot \|\mz - \mz'\|_2\nonumber\tag{$\hat\mS_{n}^{\rm rflow} \in \cF_{\nu, K,M}$ and (ref)}.
\end{align}
By setting $\mz' =\mathbf{0}$ and using the fact that each element of $\cF_{\nu, K,M}$ takes $\mathbf{0}$ to $\mathbf{0}$ by definition, the above inequality yields that
$$
\|\hat \mG_n^{\rm rflow}(\mz) - \hat \mG_n^{\rm rflow}(\mathbf{0})\|_2= \|\hat \mG_n^{\rm rflow}(\mz)\|_2 \leq K^\nu M^\nu p^{\nu/2}\left(1+ M^2p\right)^{\nu(p-1)} \cdot \|\mz\|_2.
$$
Thus, for any $\textit{\textbf u} \in \cU$, we obtain
$$
\|\hat \mG_n^{\rm rflow}(\bm{u})\|_2 \leq K^\nu M^\nu p^{\nu/2}\left(1+ M^2p\right)^{\nu(p-1)} \cdot \|\bm{u}\|_2 \leq K^\nu M^\nu p^{\nu/2}\left(1+ M^2p\right)^{\nu(p-1)} \cdot \sup_{\bm{u} \in \cU}\|\bm{u}\|_2,
$$
where Assumption (ref) implies the boundedness of $\cU$ so that the right-hand side is finite. Since the right-hand side is nonrandom and does not depend on $n$, we reach the conclusion of {\bf Step 1a}.
\\
Step 1b: Show that $\tilde \cZ_n \supseteq \cZ$ almost surely.
We have shown that each $\mS \in \cF_{\nu, K, M}$ is continuously differentiable on ${\mathbb R}^p$ and has an inverse that is also continuously differentiable on ${\mathbb R}^p$. Assumption (ref) guarantees that $p_U$ exists, so that applying Lemma (ref), the density of $\mS^{-1}(\mU)$ is
\begin{equation}
p_{S^{-1}(U)}(\mz) = p_U(\mS(\mz)) \cdot |\det(\sD\mS(\mz))|.
\end{equation}
(ref) shows that the determinant is always positive, so that the above expression is $0$ if and only if $p_U(\mS(\mz)) = 0$, which happens if and only if $\mS(\mz) \not\in \cU$.
Since $\mS$ is bijective, this shows that the support of $\mS^{-1}(\mU)$ is $\mS^{-1}(\cU)$, which denotes the image of the set $\cU$ under the mapping $\mS^{-1}$.
Next, by mean value theorem,
\begin{align}
\|\mS(\mz) - \mS(\mz')\|_2
&\leq \sup_{\mz” \in {\mathbb R}^p}\|\sD \mS(\mz”)\|_{\rm op} \cdot \|\mz - \mz'\|_2 \nonumber\\
&\leq \prod_{i=1}^\nu \|\boldsymbol{\Sigma}_i\|_{\rm op} \cdot \sup_{\mz” \in {\mathbb R}^p}\|\sD\mF^{i}(\mz”)\|_{\rm op}\cdot \|\mz - \mz'\|_2 \nonumber\tag{upper bound by product of operator norms and take supremum}\\
&= \prod_{i=1}^\nu \lambda_{\rm max}(\boldsymbol{\Sigma}_i) \cdot \sup_{\mz” \in {\mathbb R}^p}\|\sD\mF^{i}(\mz”)\|_{\rm op} \cdot \|\mz - \mz'\|_2 \nonumber\tag{$\boldsymbol{\Sigma}_i$ is symmetric and positive definite}\\
&\leq K^\nu \prod_{i=1}^\nu \sup_{\mz” \in {\mathbb R}^p}\|\sD\mF^{i}(\mz”)\|_{\rm op} \cdot \|\mz - \mz'\|_2 \nonumber \tag{definition of $\cF_{\nu,K,M}$}\\
&\leq (KpM)^{\nu} \cdot \|\mz - \mz'\|_2,
\end{align}
where
$$
\sup_{\mz'' \in {\mathbb R}^p}\|\sD\mF^{i}(\mz'')\|_{\rm op} \leq pM
$$
from the inequality
$$
\|\mathbf{A}\|_{\rm op} \leq \sqrt{\max_{j} \sum_{i\leq j} |A_{ij}| \cdot \max_{i'}\sum_{i' \geq j'} |A_{i'j'}|}
$$
and the fact that the absolute value all entries of $\sD\mF^{i}(\mz'')$ are upper bounded by $M$. Let $\mz' = \mathbf{0}$ and recall $\mS(\mathbf{0})=\mathbf{0}$. If $\mz \in \cZ$, then $\|\mz\|_2 \leq (KpM)^{-\nu}r_0$ for $r_0$ in Assumption (ref)(b). By (ref), $\|\mS(\mz)\|_2 \leq r_0$ implies
\begin{equation}
\mS(\cZ) \subseteq \Big\{\mx \in {\mathbb R}^p : \|\mx \|_2 \leq r_0\Big\}
\end{equation}
and, in particular, $\mS(\mz) \in \cU$ by Assumption (ref)(b). Since $\mS$ is invertible, $\mz \in \mS^{-1}(\cU)$. Since $\mS$ is arbitrary, we set $\mS = \hat \mS_{n}^{\rm rflow}$, so that $\mz \in \hat\mG_n^{\rm rflow}(\cU) = \tilde\cZ_n$. Finally, as $\mz$ was arbitrary, we have $\cZ \subseteq \tilde \cZ_n$ almost surely, for each $n$, completing the proof of {\bf Step 1b}.
\\
Step 1c: Show that $\bP_{\tilde Z \mid \cO}$ has nonzero variance, almost surely, and the Wasserstein distance between $\bP_{\tilde Z\mid \cO}$ and ${\mathrm P}_Z$ goes to $0$ in probability.
Since $\hat \mG_n^{\rm rflow}$ is an invertible map and $\cU$ has nonzero variance, by Assumption (ref), $\hat\mG_n^{\rm rflow}(\mU)$ cannot be constant almost surely, and, thus, must have nonzero variance. This confirms $\bP_{\tilde Z\mid \cO}$ has nonzero variance, almost surely.
By Step 1a and Step 1b, we have shown $\cZ \subseteq \tilde \cZ_n \subseteq \tilde \cZ$ almost surely, for all $n=1,2, ...$. Now applying Lemma (ref), we obtain
$$
{\sf W}_1(\bP_{\tilde Z \mid \cO}, {\mathrm P}_Z) \leq 2\cdot \sup_{\mz \in \tilde\cZ } \|\mz\|_2 \cdot \sqrt{\frac{1}{2} \cdot \mathrm{KL}({\mathrm P}_Z||\bP_{\tilde Z \mid \cO} )}
$$
almost surely. Due to the fact that $\tilde \cZ $ is bounded, it suffices to show
$$
\mathrm{KL}({\mathrm P}_Z||\bP_{\tilde Z\mid \cO} ) = o_{{\mathrm P}}(1).
$$
We rewrite the KL-divergence as
$$
\mathrm{KL}({\mathrm P}_Z || \bP_{\tilde Z\mid \cO}) = {\mathrm E}\Big[\log p_{Z}(\mZ)\Big] - {\mathrm E}\Big[\log \tilde p_n(\mZ) \mid \cO\Big].
$$
By assumption, ${\mathrm E}[|\log p_Z(\mZ)|] < \infty$.
Taking any $\mS \in \cF_{\nu, K,M}$, (ref) and (ref) combined give,
$$
\inf_{\mz \in \cZ}p_U(\mS(\mz)) \cdot (K^pM)^{-\nu} \leq p_{S^{-1}(U)}(\mz) \leq \sup_{\mz \in \cZ} p_U(\mS(\mz)) \cdot (K^pM)^{\nu}.
$$
We have already shown $\mS(\cZ) \subseteq \{\mz \in {\mathbb R}^p : \|\mz\|_2 \leq r_0\}$. By Assumption (ref), $\inf_{\mz \in \cZ}p_U(\mS(\mz)) \geq \inf_{\mz \in {\mathbb R}^p : \|
\mz \| \leq r_0}p_U(\mz) > 0$. Furthermore, Assumption (ref) implies continuity of $p_U$ on the compact set $\cU$. Assumption (ref)(b) guarantees $\sup_{\mz \in \cZ} p_U(\mS(\mz)) \leq \sup_{\bm{u} \in \cU} p_U(\bm{u}) < \infty$. Put together, we have ${\mathrm E}[|\log p_{S^{-1}(U)}(\mZ)|] < \infty$ and
\begin{equation}
\sup_{\mS \in \cF_{\nu, K,M}} \sup_{\mz \in \cZ}|\log p_{S^{-1}(U)}(\mz)| < \infty.
\end{equation}
Then, for any $\mz, \mz' \in \cZ$,
$$
\Big|\log p_{S^{-1}(U)}(\mz) - \log p_{S^{-1}(U)}(\mz')\Big| \leq \frac{ (K^pM)^{\nu}}{\inf_{\bm{u} : \|
\bm{u} \| \leq r_0}p_U(\bm{u})} \cdot |p_{S^{-1}(U)}(\mz) - p_{S^{-1}(U)}(\mz')|
$$
since $p_{S^{-1}(U)}$ is bounded below by $\inf_{\bm{u} : \|
\bm{u} \| \leq r_0}p_U(\bm{u}) \cdot (K^pM)^{-\nu}$. Accordingly, by the definition of $p_{S^{-1}(U)}$ and (ref),
\begin{align*}
\Big|p_{S^{-1}(U)}(\mz) - p_{S^{-1}(U)}(\mz')\Big| &\leq (K^pM)^{\nu} \cdot \Big|p_U(\mS(\mz)) - p_U(\mS(\mz'))\Big|\\ &+ \sup_{\bm{u} \in \cU} |p_u(\bm{u})| \cdot \Big|\det(\sD\mS(\mz)) - \det(\sD\mS(\mz')) \Big|.
\end{align*}
The mean value theorem and convexity of $\cU_1 \times {\mathbb R}$ combined then imply
$$
|p_U(\mS(\mz)) - p_U(\mS(\mz'))| \leq \sup_{\bm{u}\in \cU} \|\sD p_U(\bm{u})\|_{\rm op} \cdot \sup_{\mS \in \cF_{\nu, K, M}} \sup_{\mz'' \in {\mathbb R}^p} \|\sD \mS(\mz'')\|_{\rm op} \cdot \|\mz - \mz'\|_2.
$$
by the containment $\mS(\cZ) \subseteq \cU$ through (ref) and Assumption (ref)(b).
The continuity of $\sD p_U$ on $\cU$ from Assumption (ref) and the definition of $\cF_{\nu, K,M}$ means the right-hand side is finite. Furthermore, for some constant $L > 0$, depending on the dimension $p$, and parameters $\nu, M, K$, the Lipschitz condition
$$
\Big|\det(\sD\mS(\mz)) - \det(\sD\mS(\mz')) \Big| \leq L \cdot \|\mz - \mz'\|_2
$$
holds because the entries of $\sD\mS(\mz)$ are uniformly bounded in $\mz$ and $\mS$, and the determinant is a polynomial in the entries of its argument.
We have thus proven that $\{\log p_{S^{-1}(U)} : \mS \in \cF_{\nu, K,M}\}$ is uniformly bounded, and uniformly Lipschitz on $\cZ$. Hence, applying Lemma (ref) and then Lemma (ref), we obtain
$$
\sup_{\mS \in \cF_{\nu, K,M}} \left|\frac{1}{n} \sum_{i=1}^n \log p_{S^{-1}(U)}(\mZ_i) - {\mathrm E}[\log p_{S^{-1}(U)}(\mZ)]\right| = o_{{\mathrm P}}(1).
$$
Furthermore, we have assumed ${\mathrm E}[|\log p_Z(\mZ)|]< \infty $, so that the law of large numbers yields
$$
\frac{1}{n} \sum_{i=1}^n \log p_{Z}(\mZ_i) - {\mathrm E}[\log p_{Z}(\mZ)] = o_{{\mathrm P}}(1).
$$
Combining the above two displays, we obtain
$$
\sup_{\mS \in \cF_{\nu, K,M}} \left|\frac{1}{n} \sum_{i=1}^n \log p_Z(\mZ_i) - \log p_{S^{-1}(U)}(\mZ_i) - {\mathrm E}\Big[\log p_Z(\mZ) - \log p_{S^{-1}(U)}(\mZ)\Big]\right| = o_{{\mathrm P}}(1).
$$
Since $\hat\mS_n^{\rm rflow} \in \cF_{\nu, K,M}$, we also deduce
$$
\left|\frac{1}{n} \sum_{i=1}^n \log p_Z(\mZ_i) - \log \tilde p_{n}(\mZ_i) - {\mathrm E}\Big[\log p_Z(\mZ) - \log \tilde p_{n}(\mZ) \mid \cO\Big]\right| = o_{{\mathrm P}}(1).
$$
Combined with Assumption (ref)(a), we finish the proof of {\bf Step 1c}.
\\
\textbf{Step 2: Proof of Theorem (ref)(b).}
Assumption (ref) implies Assumption (ref), and thus Assumption (ref) holds under the conditions of Theorem (ref)(b). It remains to verify the remaining conditions.
\textbf{Step 2a: Show $\tilde \cX_n \supseteq \cX$ for all $n$, almost surely, and show $\tilde \cX_n$ is a closed interval.}
{\bf Step 1b} has shown that $\tilde \cZ_n \supseteq \cZ$ for all $n$, almost surely. To verify that $\tilde \cX_n$ is a closed interval, observe that $\hat \mG_n^{\rm rflow}$ is a continuous function and $\cU$ is a compact and connected set. Therefore, the image set, $\hat\mG_n^{\rm rflow}(\cU) = \tilde \cZ_n$, must be connected and compact. The map that projects onto the first component, $(x,y) \mapsto x$, is continuous. Therefore, $\tilde \cX_n$ must also be connected and compact. Since $\tilde \cX_n \subseteq {\mathbb R}$, it must be that $\tilde \cX_n$ is a closed interval.
\\
\textbf{Step 2b: Show that $\tilde p_n$ is always twice-continuously differentiable on $\tilde \cX_n \times {\mathbb R}$.}
It suffices to show this stronger statement that implies Assumption (ref)(a). By Lemma (ref),
$$
\tilde p_n(\mz) = p_U(\hat\mS_n^{\rm rflow}(\mz)) \cdot \det(\sD\hat\mS_n^{\rm rflow}(\mz))
$$
for $\mz \in {\mathbb R}^2$; note that due to (ref) there is no absolute value. Observe that
$$
\hat\mS_n^{\rm rflow}(\tilde\cX_n \times {\mathbb R}) \subseteq \cU_1 \times {\mathbb R}.
$$
Note first that Assumption (ref)(b) ensures that the function $p_U$ is twice continuously differentiable on $\hat\mS_n^{\rm rflow}(\tilde\cX_n \times {\mathbb R})$. Second, the function $\hat\mS_n^{\rm rflow} \in \cF_{\nu, K,M}$ is twice continuously differentiable on ${\mathbb R}^2$. Then, using the chain rule, we obtain that $p_U \circ \hat\mS_n^{\rm rflow}$ is continuously differentiable on $\tilde\cX_n \times {\mathbb R}$. Using the product and chain rules on the first derivative confirms that $p_U \circ \hat\mS_n^{\rm rflow} $ is twice continuously differentiable on $\tilde\cX_n \times {\mathbb R}$.
The determinant of a matrix is a polynomial in the entries of the matrix. As a result, $\det(\sD\hat\mS_n^{\rm rflow}(\mz))$, as a function of $\mz$, is twice-continuously differentiable on $\tilde \cX_n \times {\mathbb R}$ if the map $\sD\hat\mS_n^{\rm rflow}(\mz)$ is twice-continuously differentiable on $\tilde \cX_n \times {\mathbb R}$. Equivalently, this happens if $\hat\mS_n^{\rm rflow}(\mz)$ is three-times continuously differentiable on $\tilde \cX_n \times {\mathbb R}$. By the product rule, this yields the conclusion of {\bf Step 2b}.
\\
\textbf{Step 2c: Show that $\tilde p_X$ is lower bounded uniformly in $\cX$, and $\tilde p_n$, $\|\sD \tilde p_n\|_2$, and $\|\sD^2\tilde p_n\|_{\rm op}$ are upper bounded by a universal constant in $\tilde \cZ_n$.}
To show the lower bound, for each $x \in \cX$
\begin{align*}
\tilde p_{X}(x) &= \int_{\mathbb R} \tilde p_n(x,y) \, {\mathrm d} y \tag{marginal probability}\\
&= \int_{\mathbb R} p_U(\hat\mS_n^{\rm rflow}(x,y)) \cdot \det(\sD\hat\mS_n^{\rm rflow}(x,y)) \, {\mathrm d} y \tag{Lemma (ref)}\\
&\geq (K^p M)^{-\nu} \cdot \int_{{\mathbb R}} p_U(\hat\mS_n^{\rm rflow}(x,y)) \, {\mathrm d} y \tag{Equation (ref)}\\
&\geq (K^p M)^{-\nu} \cdot \int_{\{y \in \cY: \|\hat\mS_n^{\rm rflow}(x,y)\|_2 \leq r_0\}} p_U(\hat\mS_n^{\rm rflow}(x,y)) \, {\mathrm d} y \tag{restrict the integration set}\\
&\geq (K^p M)^{-\nu} \cdot \inf_{\bm{u} \in \cU} p_U(\bm{u}) \cdot \int_{\{y \in \cY: \|\hat\mS_n^{\rm rflow}(x,y)\|_2 \leq r_0\}} \, {\mathrm d} y \tag{ $p_U$ is uniformly lower bounded on $\{\mz \in {\mathbb R}^2 : \|\mz\|_2 \leq r_0\}$ by Assumption (ref)}\\
&= (K^p M)^{-\nu} \cdot \inf_{\bm{u} \in \cU} p_U(\bm{u}) \cdot \int_{\cY} \, {\mathrm d} y \tag{$(x,y) \in \cX \times \cY = \cZ$ from
Assumption (ref)(e), then applying Equation (ref)}\\
&> 0. \tag{ $\cY$ has positive measure from Assumption (ref)(e)}
\end{align*}
Taking the infimum with respect to $x\in \cX$, we have shown $\tilde p_X$ is uniformly lower bounded in $\cX$.
Next, we aim to show that $\tilde p_n$ is upper bounded in $\tilde \cZ_n$. By Lemma (ref),
\begin{align*}
\sup_{\mz \in \tilde \cZ_n} \tilde p_n(\mz) &= \sup_{\mz \in \tilde \cZ_n} p_U(\hat\mS_n^{\rm rflow}(\mz)) \cdot \det(\sD\hat\mS_n^{\rm rflow}(\mz)) \\
&= \sup_{\mz \in \tilde \cZ_n} p_U(\hat\mS_n^{\rm rflow}(\mz)) \cdot (K^pM)^{\nu} \tag{Equation (ref)}\&\leq \sup_{\bm{u} \in \cU} p_U(\bm{u}) \cdot (K^pM)^{\nu} \tag{for $\mz \in \tilde \cZ_n$, $\hat\mS_n^{\rm rflow}(\mz) \in \cU$}\\
&< \infty. \tag{Assumption (ref) ensures $p_U$ is continuous on the compact set $\cU$}
\end{align*}
For $\sD \tilde p_n$,
\begin{align*}
\|\sD \tilde p_n(\mz)\|_2 &= \|\sD p_U(\hat\mS_n^{\rm rflow}(\mz)) \cdot \det(\sD\hat\mS_n^{\rm rflow}(\mz))\|_2 + \|p_U(\hat\mS_n^{\rm rflow}(\mz)) \cdot \sD \det(\sD\hat\mS_n^{\rm rflow}(\mz))\|_2 \tag{product rule and triangle inequality}\\
&\leq \|\sD p_U(\hat\mS_n^{\rm rflow}(\mz))\|_2 \cdot (K^pM)^\nu + p_U(\hat\mS_n^{\rm flow}(\mz)) \cdot \|\sD \det(\sD\hat\mS_n^{\rm rflow}(\mz)) \|_2\tag{Equation (ref)}\\
&\leq \sup_{\bm{u} \in \cU}\|\sD p_U(\bm{u})\|_2 \cdot (K^pM)^\nu + \sup_{\bm{u} \in \cU}p_U(\bm{u}) \cdot \|\sD \det(\sD\hat\mS_n^{\rm rflow}(\mz))\|_2 \tag{for $\mz \in \tilde \cZ_n$, $\hat\mS_n^{\rm rflow}(\mz) \in \cU$}.
\end{align*}
We have $\sup_{\bm{u} \in \cU}\|\sD p_U(\bm{u})\|_2 < \infty$ and $\sup_{\bm{u} \in \cU}p_U(\bm{u}) < \infty$ because Assumption (ref) ensures $p_U$ and $\sD p_U$ are continuous on the compact set $\cU$. The function $\det(\sD\hat\mS_n^{\rm rflow}(\mz))$ is a polynomial in the first partial derivatives of $\hat \mS_{n}^{\rm rflow}$. By the chain rule, each component of $\|\sD \det(\sD\hat\mS_n^{\rm rflow}(\mz))\|_2$ is a polynomial in the first and second-order partial derivatives of $\hat \mS_n^{\rm rflow}$. By definition, $\hat \mS_n^{\rm rflow} \in \cF_{\nu, K,M}$, and has first and second order partial derivatives bounded by $M$ on ${\mathbb R}^p$. A uniform bound on $\|\sD \det(\sD\hat\mS_n^{\rm rflow}(\mz))\|_2$, in terms of $M$, holds as a result. The exact same reasoning leads to a uniform bound of $\|\sD^2 \det(\sD\hat\mS_n^{\rm rflow}(\mz))\|_{\rm op}$, since the definition of $\cF_{\nu, K,M}$ involves bounds on third partial derivatives over ${\mathbb R}^p$.
Applying the product rule again and using the twice-continuous differentiability of $p_U$ from Assumption (ref) to show $\sup_{\bm{u} \in \cU}\|\sD^2p_U(\bm{u})\|_{\rm op} < \infty$, we obtain that $\|\sD^2 \tilde p_n\|_{\rm op}$ is upper bounded by a universal constant, and thus complete the proof.
Supporting lemmas
In the proofs of lemmas in Sections (ref) and (ref), we use $C, C'$ to represent some generic constants whose values may change from statement to statement.
Supporting lemmas for Theorem (ref)
We demonstrate uniform convergence of expectations and variances of certain classes of functions.
lemmaAssume Assumptions (ref), (ref), and (ref). Then, for any function $\mg: \cK \times (\cZ \cup \tilde\cZ) \rightarrow {\mathbb R}^r$ continuous in both arguments, we have
\begin{align}
\sup_{\boldsymbol{\eta} \in \cK}\Big\|{\rm{Var}}(\mg(\boldsymbol{\eta}, \mZ_1)) - {\rm{Var}}(\mg(\boldsymbol{\eta}, \tilde \mZ_1) \mid \cO)\Big\|_{\rm{max}} = o_{{\mathrm P}_{\cO}}(1)
\end{align}
and
\begin{align}
\sup_{\boldsymbol{\eta} \in \cK}\Big\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)] - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO]\Big\|_{\infty} = o_{{\mathrm P}_{\cO}}(1).
\end{align}
proofWe first prove (ref). Note that $\mg(\boldsymbol{\eta}, \mZ_1)$ and $\mg(\boldsymbol{\eta}, \tilde \mZ_1)$ are almost surely bounded so that their conditional covariance matrices and expectations exist almost surely for every $\boldsymbol{\eta} \in \cK$.
Decomposing the covariance matrix and using the triangle inequality, we obtain
\begin{align*}
\Big\|{\rm{Var}}(\mg(\boldsymbol{\eta},\mZ_1)) -& {\rm{Var}}(\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO)\Big\|_{\rm{max}} \leq \Big\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)\mg(\boldsymbol{\eta},\mZ_1)^\top] - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1)\mg(\boldsymbol{\eta}, \tilde \mZ_1)^\top \mid \cO]\Big\|_{\rm{max}}\\
&+ \Big\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)]{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)]^\top - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO] {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO]^\top\Big\|_{\rm{max}}
\end{align*}
and taking a supremum over $\boldsymbol{\eta} \in \cK$ on both sides, it suffices to show that
$$
\sup_{\boldsymbol{\eta} \in \cK} \Big\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)]{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)]^\top - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO] {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO]^\top\Big\|_{\rm{max}} = o_{{\mathrm P}_{\cO}}(1)
$$
and
$$
\sup_{\boldsymbol{\eta} \in \cK} \Big\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)\mg(\boldsymbol{\eta},\mZ_1)^\top] - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1)\mg(\boldsymbol{\eta}, \tilde \mZ_1)^\top \mid \cO]\Big\|_{\rm{max}} = o_{{\mathrm P}_{\cO}}(1).
$$
As the proofs of these conclusions are alike, we only show the first equality.
We then have
\begin{align*}
&\sup_{\boldsymbol{\eta} \in \cK} \Big\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)]{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)]^\top - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO] {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO]^\top\Big\|_{\rm{max}}\\
\leq & \sup_{\boldsymbol{\eta} \in \cK}\Big\|{\mathrm E}[\mg(\boldsymbol{\eta}, \mZ_1)]{\mathrm E}[\mg(\boldsymbol{\eta}, \mZ_1)]^\top - {\mathrm E}[\mg(\boldsymbol{\eta}, \tilde \mZ_1) \mid \cO] {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO]^\top\Big\|_{\rm op}\\
\leq& \sup_{\boldsymbol{\eta} \in \cK}\|\big({\mathrm E}[\mg(\boldsymbol{\eta}, \mZ_1)] - {\mathrm E}[\mg(\boldsymbol{\eta}, \tilde \mZ_1) \mid \cO] \big) {\mathrm E}[\mg(\boldsymbol{\eta}, \mZ_1)]^\top + {\mathrm E}[\mg(\boldsymbol{\eta}, \tilde\mZ_1) \mid \cO ] \big({\mathrm E}[\mg(\boldsymbol{\eta}, \mZ_1) ] - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde\mZ_1) \mid \cO]\big)^\top\|_{\rm op}\\
\leq & \sup_{\boldsymbol{\eta} \in \cK}\|\big({\mathrm E}[\mg(\boldsymbol{\eta}, \mZ_1)] - {\mathrm E}[\mg(\boldsymbol{\eta}, \tilde \mZ_1) \mid \cO] \big) {\mathrm E}[\mg(\boldsymbol{\eta}, \mZ_1)]^\top\|_{\rm op}\\
&+ \sup_{\boldsymbol{\eta}' \in \cK}\|{\mathrm E}[\mg(\boldsymbol{\eta}', \tilde\mZ_1) \mid \cO ] \big({\mathrm E}[\mg(\boldsymbol{\eta}', \mZ_1) ] - {\mathrm E}[\mg(\boldsymbol{\eta}',\tilde\mZ_1) \mid \cO]\big)^\top\|_{\rm op}\\
\leq &\sup_{\boldsymbol{\eta} \in \cK}\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)] - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO] \|_2 \cdot \left(\sup_{\boldsymbol{\eta}' \in \cK}\|{\mathrm E}[\mg(\boldsymbol{\eta}',\mZ_1)]\|_2 + \sup_{\boldsymbol{\eta}' \in \cK} \|{\mathrm E}[\mg(\boldsymbol{\eta}',\tilde \mZ_1) \mid \cO] \|_2\right). \addtocounter{equation}{1}\tag{\theequation}
\end{align*}
As $\mg(\boldsymbol{\eta}, \mZ_1)$ and $\mg(\boldsymbol{\eta}, \tilde\mZ_1)$ are almost surely contained in the compact set $\mg(\cK \times(\tilde \cZ \cup \cZ))$, so that
$$
\sup_{\boldsymbol{\eta}' \in \cK}\|{\mathrm E}[\mg(\boldsymbol{\eta}',\mZ_1)]\|_2 + \sup_{\boldsymbol{\eta}' \in \cK} \|{\mathrm E}[\mg(\boldsymbol{\eta}',\tilde \mZ_1) \mid \cO] \|_2 \leq 2\sup_{(\boldsymbol{\eta}, \mz) \in \cK \times (\tilde\cZ \cup \cZ)} \|\mg(\boldsymbol{\eta}, \mz)\|_2 < \infty.
$$
This yields the following upper bound for (ref):
\begin{align}
2\sup_{(\boldsymbol{\eta}, \mz) \in \cK \times (\tilde\cZ \cup \cZ)} \|\mg(\boldsymbol{\eta}, \mz)\|_2 \cdot \sup_{\boldsymbol{\eta}\in \cK}\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)] - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO] \|_2.
\end{align}
Taking any joint distribution $\pi$ between $\bP_{\tilde Z \mid \cO}$ and ${\mathrm P}_{Z}$ and letting $(\tilde \mZ, \mZ) \sim \pi$, we have
\begin{equation}
\sup_{\boldsymbol{\eta} \in \cK}\|{\mathrm E}[\mg(\boldsymbol{\eta},\mZ_1)] - {\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO] \|_2 = \sup_{\boldsymbol{\eta} \in \cK}\|{\mathrm E}_\pi[\mg(\boldsymbol{\eta},\mZ) - \mg(\boldsymbol{\eta},\tilde\mZ)]\|_2
\end{equation}
since $\tilde\mZ \sim \bP_{\tilde Z\mid\cO}$ and $\mZ \sim {\mathrm P}_{Z}$ so that ${\mathrm E}[\mg(\boldsymbol{\eta}, \mZ_1)] = {\mathrm E}_\pi[\mg(\boldsymbol{\eta}, \mZ)]$ and ${\mathrm E}[\mg(\boldsymbol{\eta},\tilde \mZ_1) \mid \cO] = {\mathrm E}_\pi[\mg(\boldsymbol{\eta},\tilde\mZ)]$.
Take any $\epsilon > 0$. Since $\mg$ is continuous on a compact set $\cK \times (\cZ \cup \tilde\cZ)$, and, hence, uniformly continuous, we choose a $\delta >0 $ such that
\begin{equation}
\sup_{\boldsymbol{\eta} \in \cK}\sup_{\mz, \mz' \in \cZ \cup \tilde\cZ : \|\mz-\mz'\|\leq \delta} \|\mg(\boldsymbol{\eta}, \mz) - \mg(\boldsymbol{\eta}, \mz')\|_2 < \epsilon.
\end{equation}
Leveraging Jensen's inequality and (ref), we obtain
\begin{align*}
\sup_{\boldsymbol{\eta} \in \cK}\|{\mathrm E}_\pi[\mg(\boldsymbol{\eta},\mZ) - \mg(\boldsymbol{\eta},\tilde\mZ)]\|_2
&\leq \sup_{\boldsymbol{\eta} \in \cK}{\mathrm E}_\pi[\|\mg(\boldsymbol{\eta},\mZ) - \mg(\boldsymbol{\eta},\tilde\mZ)\|_2 \cdot \mathds{1}(\|\mZ - \tilde\mZ\|_2 > \delta)] + \epsilon\\
&\leq \frac{1}{\delta} \cdot 2\max_{(\boldsymbol{\eta},\mz) \in \cK \times (\cZ \cup \tilde\cZ)}\|\mg(\boldsymbol{\eta},\mz)\|_2 \cdot {\mathrm E}_\pi[\|\mZ - \tilde\mZ\|_2] + \epsilon \tag{Markov's inequality}.
\end{align*}
Putting together and taking an infimum over $\pi$, an upper bound for (ref) is then
$$
\frac{4}{\delta} \cdot \left(\max_{(\boldsymbol{\eta},\mz) \in \cK \times (\cZ \cup \tilde\cZ)}\|\mg(\boldsymbol{\eta},\mz)\|_2 \right)^2\cdot {\sf W}_1\big(\bP_{\tilde Z\mid \cO},\, {\mathrm P}_Z\big) + 2 \max_{(\boldsymbol{\eta},\mz) \in \cK \times (\cZ \cup \tilde\cZ)}\|\mg(\boldsymbol{\eta},\mz)\|_2\cdot \epsilon.
$$
By Assumption (ref)(b), ${\sf W}_1\big(\bP_{\tilde Z\mid \cO}, {\mathrm P}_Z\big) = o_{{\mathrm P}_{\cO}}(1)$, so taking $n\rightarrow \infty$ and then $\epsilon \rightarrow 0$, as $\epsilon$ was arbitrary, we obtain the desired conclusion.
Equation (ref) is established in an identical way, and we thus complete the whole proof.
lemma\begin{enumerate}[label=(\alph*)]
• Under Assumptions (ref), (ref) and (ref),
\[
\sqrt{n} (\hat{\boldsymbol\eta}_n - \boldsymbol\eta_0) = -\frac{1}{\sqrt{n}} \sum_{i=1}^n (\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)])^{-1} \sD_{\boldsymbol{\eta}}{\sf L}(\boldsymbol\eta_0, \mZ_i) + o_{{\mathrm P}_{\cO}}(1).
\]
• Assuming further Assumption (ref), (ref) and Assumption (ref), it holds true that
\begin{enumerate}[label=(\roman*)]
• $\|\tilde{\boldsymbol\eta}_0 - \boldsymbol\eta_0\|_2 = o_{{\mathrm P}_{\cO}}(1)$;
• ${\mathrm P}\Big(\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ) \mid \cO]) \text{ is invertible}\Big) \rightarrow 1$;
• $\|\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ) \mid \cO]^{-1} - \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)]^{-1}\|_{\rm op} = o_{{\mathrm P}}(1).$
• $\sqrt{n}(\tilde{\boldsymbol\eta}_n -\tilde{\boldsymbol\eta}_0) = -\frac{1}{\sqrt{n}} \sum_{i=1}^n (\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ) \mid \cO])^{-1}\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde \mZ_i) + o_{{\mathrm P}_{\cO\tilde\mU}}(1)$.
\end{enumerate}
\end{enumerate}
proofLemma (ref)(a) follows from Lemma 5.23 of Vaart_1998. It remains to prove Lemma (ref)(b).
\noindentStep 1: Show Lemma (ref)(b)(i).
Since $\boldsymbol{\eta}_0$ maximizes the function $\boldsymbol{\eta} \mapsto {\mathrm E}[{\sf L}({\boldsymbol{\eta}}, \mZ)]$ by Assumption (ref)(b), we obtain
$$
0 \leq {\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \mZ) ] -{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mZ)].
$$
Rewriting the right-hand side,
\begin{align*}
{\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \mZ) ] -{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mZ)] =& {\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \mZ) ] -{\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]\\
&+{\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] - {\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]\\
&+ {\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]-{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mZ)].
\end{align*}
The function ${\sf L}(\cdot,\cdot)$ is jointly continuous by Assumption (ref)(b) and (c). Also, $\boldsymbol{\eta}_0 \in \cK$ by Assumption (ref)(a). Additionally, $\tilde{\boldsymbol{\eta}}_0 \in \cK$ by definition. Applying Lemma (ref) then yields
$$
{\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \mZ) ] - {\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] + {\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]-{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mZ)] = o_{{\mathrm P}_{\cO}}(1).
$$
By definition, $\tilde{\boldsymbol{\eta}}_0$ is a maximizer of $\boldsymbol{\eta} \mapsto {\mathrm E}[{\sf L}({\boldsymbol{\eta}}_, \tilde\mZ) \mid \cO]$ so that
$$
{\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] - {\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]\leq 0
$$
almost surely.
Putting this all together,
$$
0 \leq {\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \mZ) ] -{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mZ)] \leq o_{{\mathrm P}_{\cO}}(1).
$$
Since $\boldsymbol{\eta}_0$ is the unique maximizer of $\boldsymbol{\eta} \mapsto {\mathrm E}[{\sf L}({\boldsymbol{\eta}}_, \mZ)]$ by Assumption (ref)(b), $\|\tilde{\boldsymbol\eta}_0 - \boldsymbol\eta_0\|_2 = o_{{\mathrm P}_{\cO}}(1)$ as desired.
\\
\noindentStep 2: Show Lemma (ref)(b)(ii) and Lemma (ref)(b)(iii).
By the mean value theorem and the bounded convergence theorem,
$$
\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] = {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]
$$
almost surely. Thus, it suffices to show the same conclusion for ${\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]$.
First, note $\boldsymbol{\eta} \mapsto {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}, \mZ)]$ is continuous by the mean value theorem. Applying Lemma (ref) to the function $\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}, \mz)$, we obtain
\begin{equation}
\sup_{\boldsymbol{\eta} \in \cK }\Big\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}, \tilde\mZ) \mid \cO] - {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}, \mZ)]\Big\|_{\rm max} = o_{{\mathrm P}}(1).
\end{equation}
Then, by Step 1 and (ref),
\begin{align*}
\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] - {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}_0, \mZ) ]\|_{\rm max} &\leq \|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] - {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mZ) ]\|_{\rm max} \\
&+\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mZ)] - {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}_0, \mZ) ]\|_{\rm max} \\
&= o_{{\mathrm P}}(1).
\end{align*}
Next, we note a consequence of the property of matrix inversion. Pick any $\epsilon' > 0$. Since the set of invertible matrices is an open set with respect to $\|\cdot \|_{\max }$, and matrix inversion is continuous, there is a fixed $\delta' > 0$ for which any matrix $\mathbf{M} \in {\mathbb R}^{q \times q}$ that satisfies $\|\mathbf{M} - {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}_0, \mZ) ]\|_{\rm max} < \delta'$ is invertible and also satisfies $\|\mathbf{M}^{-1} - {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}_0, \mZ) ]^{-1}\|_{\rm max} < \epsilon'$. The existence of ${\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}_0, \mZ) ]^{-1}$ is from
Assumption (ref)(d).
Then,
\begin{align*}
{\mathrm P}\left({\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]^{-1} exists, \|\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ) \mid \cO]^{-1} - \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)]^{-1}\|_{\rm max} < \epsilon' \right) \geq\\ {\mathrm P}\left(\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] - {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}_0, \mZ) ]\|_{\rm max} < \delta'\right)
\end{align*}
by the definition of $\delta'$.
Since $\delta'$ is fixed,
$$
{\mathrm P}\left(\|{\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] - {\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}_0, \mZ) ]\|_{\rm max} < \delta'\right) \rightarrow 1,
$$
which means
$$
{\mathrm P}\left({\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]^{-1} \text{ exists}, \|\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ) \mid \cO]^{-1} - \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\boldsymbol\eta_0, \mZ)]^{-1}\|_{\rm max} < \epsilon'\right) \rightarrow 1.
$$
\noindentStep 4: Show Lemma (ref)(b)(iv).
Since $\boldsymbol{\eta}_0$ is an interior point of $\cK$, we take $d > 0$ small enough so that $\cB({\boldsymbol{\eta}}_0, d, \|\cdot\|_2 )$ is contained in the interior of $\cK$. We have just shown $\|\tilde{\boldsymbol\eta}_0 - \boldsymbol\eta_0\|_2 = o_{{\mathrm P}_{\cO}}(1)$. By Assumption (ref)(a), $\|\tilde{\boldsymbol\eta}_n - \tilde{\boldsymbol\eta}_0\|_2 = o_{{\mathrm P}_{\cO\tilde U}}(1)$ so that we assume without loss of generality that $\tilde {\boldsymbol{\eta}}_n, \tilde{\boldsymbol{\eta}}_0$ are within this ball.
Due to the mean value theorem and the characterization of $\tilde {\boldsymbol{\eta}}_0$ as a maximizer, we interchange the derivative and expectation to obtain
\begin{equation}
\mathbf{0} = \sD_{\boldsymbol{\eta}} {\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ) \mid \cO] = {\mathrm E}[\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]
\end{equation}
almost surely.
We now verify the conditions (ref) and (ref) of Lemma (ref). An application of Taylor's theorem (Theorem 12.14 of apostol1974mathematical, e.g.) then gives, using (ref),
\begin{align*}
|{\mathrm E}[{\sf L}({\boldsymbol\eta}, \tilde\mZ) - {\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ)\mid \cO]| \leq \frac{q^2}{2} \cdot \max_{\mz \in \tilde\cZ}\max_{\boldsymbol{\eta'} \in \cK} \|\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}', \mz)\|_{\rm max} \cdot \|{\boldsymbol\eta} - \tilde{\boldsymbol\eta}_0\|_2^2
\end{align*}
for any $\boldsymbol{\eta}$ in the interior of $\cK$. Thus, for all $\delta < d/3$, by the definition of $d$ and the triangle inequality, $\cB(\tilde{\boldsymbol{\eta}}_0, \delta, \|\cdot\|_2 )$ is contained in the interior of $\cK$. As a result, we have
$$
\sup_{\|\boldsymbol{\eta} - \tilde{\boldsymbol{\eta}}_0 \|_2 < \delta} |{\mathrm E}[{\sf L}({\boldsymbol\eta}, \tilde\mZ) - {\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ)\mid \cO]|\leq \frac{q^2}{2} \cdot \max_{\mz \in \tilde\cZ}\max_{\boldsymbol{\eta'} \in \cK} \|\sD_{\boldsymbol{\eta}}^2{\sf L}(\boldsymbol{\eta}', \mz)\|_{\rm max} \cdot \delta^2
$$
and thus verify the condition (ref).
Condition (ref) is directly verified in Lemma (ref). Consequently, $\|\tilde{\boldsymbol\eta}_n - \tilde{\boldsymbol\eta}_0\|_2 = O_{{\mathrm P}_{\cO\tilde U }}(n^{-1/2})$.
Twice-differentiability of $\boldsymbol{\eta} \mapsto {\mathrm E}[{\sf L}(\boldsymbol{\eta}, \tilde\mZ) \mid \cO]$ almost surely, combined with
$\sD_{\boldsymbol{\eta}}{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ)\mid \cO] = \mathbf{0}$ almost surely, implies
$$
{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_n, \tilde\mZ) \mid \cO] - {\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO] - \frac{1}{2}(\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0)^\top\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO](\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0) = o_{{\mathrm P}_{\cO\tilde U }}(n^{-1}).
$$
Using Lemma (ref) on $\tilde{\boldsymbol{\eta}}_n$, and manipulating terms, we obtain then
\begin{equation}
\left(-\frac{1}{\sqrt{n}}\sum_{i=1}^n \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ_i})\right)^\top \sqrt{n}(\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0) - \frac{1}{2}(\sqrt{n}(\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0))^\top\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO](\sqrt{n}(\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0)) = \tilde\epsilon_n,
\end{equation}
where
$$
\tilde\epsilon_n := \sum_{i=1}^n {\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ_i}) - {\sf L}(\tilde{\boldsymbol{\eta}}_n, \tilde{\mZ_i}) + o_{{\mathrm P}}(1).
$$
Since ${\mathrm P}\left({\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]^{-1} \text{ exists}\right) \rightarrow 1$ by Step 2, it is no loss of generality to assume ${\mathrm E}[\sD_{\boldsymbol{\eta}}^2{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]^{-1}$ exists for the remainder of the proof.
Define
$$
\check{\boldsymbol{\eta}}_n = \tilde{\boldsymbol{\eta}}_0
-\frac{1}{n} \sum_{i=1}^n (\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde\mZ) \mid \cO])^{-1}\sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol\eta}_0, \tilde \mZ_i)
$$
and observe, by the Lyapunov central limit theorem, that $\check{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0 = O_{{\mathrm P}}(n^{-1/2})$. Using $\check{\boldsymbol{\eta}}_n$ in Lemma (ref), we can show
\begin{equation}
\left(-\frac{1}{\sqrt{n}}\sum_{i=1}^n \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ_i})\right)^\top \sqrt{n}(\check{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0) - \frac{1}{2}(\sqrt{n}(\check{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0))^\top\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO](\sqrt{n}(\check{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0)) = \check\epsilon_n,
\end{equation}
where
$$
\check\epsilon_n := \sum_{i=1}^n {\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ_i}) - {\sf L}(\check{\boldsymbol{\eta}}_n, \tilde{\mZ_i}) + o_{{\mathrm P}}(1).
$$
Expanding the definition of $\check{\boldsymbol{\eta}}_n$ and simplifying Equation (ref), we get
\begin{equation}
\frac{1}{2}\left(\frac{1}{\sqrt{n}}\sum_{i=1}^n \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ_i})\right)^\top \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ_1) \mid \cO]^{-1}\left(\frac{1}{\sqrt{n}}\sum_{i=1}^n \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ_i})\right) = \check \epsilon_n.
\end{equation}
Subtracting (ref) from (ref), then completing the square, we get
\begin{align*}
\frac{1}{2}\Big\|(-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO])^{1/2} \sqrt{n}(\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0) - \frac{1}{\sqrt{n}}\sum_{i=1}^n (-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO])^{-1/2} \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ}_i) \Big\|_2^2 =
\tilde \epsilon_n - \check \epsilon_n.
\end{align*}
The matrix $-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]$ is almost surely symmetric, as ${\sf L}$ is twice continuously differentiable and swapping second derivatives and expectations. In addition, $-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]$ is almost surely positive definite because $\tilde{\boldsymbol{\eta}}_0$ is a maximizer, as $\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]$ is negative definite, almost surely. Hence, the matrix square root exists for $-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]$ and its inverse.
Expanding the definition of $\tilde \epsilon_n$ and $\check \epsilon_n$,
\begin{align*}
\tilde \epsilon_n - \check \epsilon_n &= \sum_{i=1}^n {\sf L}(\check{\boldsymbol{\eta}}_n, \tilde{\mZ_i}) - {\sf L}(\tilde{\boldsymbol{\eta}}_n, \tilde{\mZ_i}) + o_{{\mathrm P}}(1)\\
&=\sum_{i=1}^n {\sf L}(\check{\boldsymbol{\eta}}_n, \tilde{\mZ_i}) - \sup_{\boldsymbol{\eta} \in \cK} \sum_{i=1}^n {\sf L}({\boldsymbol{\eta}}, \tilde{\mZ_i}) + \sup_{\boldsymbol{\eta} \in \cK} \sum_{i=1}^n {\sf L}({\boldsymbol{\eta}}, \tilde{\mZ_i})- {\sf L}(\tilde{\boldsymbol{\eta}}_n, \tilde{\mZ_i}) + o_{{\mathrm P}}(1)\\
&=\sum_{i=1}^n {\sf L}(\check{\boldsymbol{\eta}}_n, \tilde{\mZ_i}) - \sup_{\boldsymbol{\eta} \in \cK} \sum_{i=1}^n {\sf L}({\boldsymbol{\eta}}, \tilde{\mZ_i}) + o_{{\mathrm P}}(1). \tag{Assumption (ref)(b)}\\
&\leq o_{{\mathrm P}}(1).
\end{align*}
Therefore,
$$
(-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO])^{1/2} \sqrt{n}(\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0) - \frac{1}{\sqrt{n}}\sum_{i=1}^n (-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO])^{-1/2} \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ}_i) = o_{{\mathrm P}}(1).
$$
Factoring out $(-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO])^{1/2}$ and noting, from \textbf{Step 2},
$$
\|(-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO])^{1/2} - (-\sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}({\boldsymbol{\eta}}_0, \mZ) ])^{1/2}\|_{\rm op} = o_{{\mathrm P}}(1),
$$
we have
$$
\sqrt{n}(\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0) + \frac{1}{\sqrt{n}}\sum_{i=1}^n \sD_{\boldsymbol{\eta}}^2{\mathrm E}[{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde\mZ) \mid \cO]^{-1} \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \tilde{\mZ}_i) = o_{{\mathrm P}}(1)
$$
and thus complete the whole proof.
Supporting lemmas for Theorem (ref)
We first establish the uniform convergence of the regression functions and their first derivatives.
lemmaSuppose that Assumptions (ref), (ref), and (ref) hold. Then, for any bounded closed interval $\cC \subseteq {\mathbb R}$, it holds true that
\begin{align*}
&\sup_{\mz \in \cX \times \cC}\Big|\tilde p_n(\mz) - p_Z(\mz)\Big| = o_{{\mathrm P}}(1), \sup_{\mz \in \cX \times \cC}\Big\|\sD\tilde p_n(\mz) - \sD p_Z(\mz)\Big\|_2 = o_{{\mathrm P}}(1),\\
&\sup_{x \in \cX}\Big|\tilde p_X(x) - p_X(x)\Big| = o_{{\mathrm P}}(1), \sup_{x \in \cX}\Big|\tilde p_X'(x) - p_X'(x)\Big| = o_{{\mathrm P}}(1),\\
&\sup_{x \in \cX}\Big|\tilde f_0(x) - f_0(x)\Big| = o_{{\mathrm P}}(1), {\rm and} \sup_{x \in \cX}\Big|\tilde f_0'(x) - f_0'(x)\Big| = o_{{\mathrm P}}(1).
\end{align*}
proofWe denote $\tilde f_0$ as $\tilde f_{0,n}$, and $\tilde f'_0$ as $\tilde f'_{0,n}$ and analogously for the marginal densities in this proof to emphasize the approximating sequences' dependence on the sample size.
\\
{\bf Step 1.} We appeal to Lemma (ref): for each subsequence $n_k$, we aim to find a further subsequence $n_{k_{\ell_m}}$ such that
$$
\sup_{\mz \in \cX \times \cC}\Big| \tilde p_{n_{k_{\ell_m}}}(\mz) - p_Z(\mz)\Big| \rightarrow 0
$$
almost surely.
By Lemma (ref), $\tilde p_{n}$ is Lipschitz on $ \cX \times {\mathbb R}$, with probability $1$. Hence, almost surely, the sequence of functions $\tilde p_{n}$'s is uniformly Lipschitz on $ \cX \times \cC$ for all sufficiently large $n$. Also, they are uniformly bounded on $ \cX \times \cC$, by $\tilde K$ by Assumption (ref)(b).
Furthermore, $\tilde p_{n}$ is a $C( \cX\times \cC)$-valued random variable for sufficiently large $n$ due to Assumption (ref)(a,c). By Arzela-Ascoli theorem (see Theorem 11.28 of rudin1987real e.g.), there then exists a compact $\tilde \cC \subseteq C( \cX \times \cC)$ for which
$$
{\mathrm P}(\tilde p_{n} \in \tilde \cC) = 1
$$
for sufficiently large $n$, yielding tightness of the sequences. Take any subsequence $n_k$. Applying Lemma (ref), then Assumption (ref)(b) along with Lemma (ref), there exists a further subsequence $n_{k_\ell}$ for which
$$
\tilde p_{{n_{k_\ell}}} \text{ converges weakly to } \phi \qquad \text{and \qquad } {\sf W}_1(\bP_{\tilde Z_{1, n_{k_\ell}} \mid \cO} , {\mathrm P}_{Z}) \rightarrow 0 \text{ almost surely}
$$
for some $C( \cX \times \cC)$-valued random variable $\phi$. Note that $\phi$ is a density almost surely because $\tilde p_{n_{k_\ell}}$ are densities.
Our next goal is to identify the limit $\phi$ as the (deterministic) function $p_Z$. Choose any bounded, continuous function $f : {\mathbb R}^2 \rightarrow {\mathbb R}$. As $\mZ$ admits a Lebesgue density by Assumption (ref)(c), the function $\mz \mapsto f(\mz) \cdot \mathds{1}(\mz \in \cX \times \cC)$ is ${\mathrm P}_{Z}$-almost surely continuous. Lemma (ref) and ${\sf W}_1(\bP_{\tilde Z_{1, n_{k_\ell}} \mid \cO} , {\mathrm P}_{Z}) \rightarrow 0$ then implies, almost surely,
$$
\tilde \mZ_{n_{k_\ell}} \text{ converges weakly to } \mZ \text{ conditional on }\cO.
$$
The continuous mapping theorem then implies
$$
f(\tilde \mZ_{n_{k_\ell}}) \cdot \mathds{1}(\tilde \mZ_{n_{k_\ell}} \in \cX \times \cC) \text{ converges weakly to } f(\mZ) \cdot \mathds{1}(\mZ \in \cX \times \cC) \text{ conditional on }
\cO.
$$
The sequence $f(\tilde \mZ_{n_{k_\ell}}) \cdot \mathds{1}(\tilde \mZ_{n_{k_\ell}} \in \cX \times \cC)$ is uniformly bounded. Therefore, this sequence is uniformly integrable, so that
\begin{equation}
{\mathrm E}\Big[f(\tilde \mZ_{n_{k_\ell}}) \cdot \mathds{1}(\tilde \mZ_{n_{k_\ell}} \in \cX \times \cC) \mid \cO\Big] - {\mathrm E}\Big[f(\mZ) \cdot \mathds{1}(\mZ \in \cX \times \cC)\Big] \rightarrow 0.
\end{equation}
almost surely.
Define the function $g_f : C( \cX\times \cC) \rightarrow {\mathbb R}$ as
$$
g_f(h) := \int_{ \cX \times \cC} f(\mz) \cdot (h(\mz)-p_Z(\mz)) \, {\mathrm d} \mz,
$$
which, we note, is continuous. By the continuous mapping theorem,
$$
g_f(\tilde p_{n_{k_\ell}}) \text{ converges weakly to } g_f(\phi).
$$
By definition,
\begin{align*}
g_f(\tilde p_{n_{k_\ell}}) = &
{\mathrm E}\Big[f(\tilde \mZ_{n_{k_\ell}}) \cdot \mathds{1}(\tilde \mZ_{n_{k_\ell}} \in \cX \times \cC) \mid \cO\Big] - {\mathrm E}\Big[f(\mZ) \cdot \mathds{1}(\mZ \in \cX \times \cC)\Big]
\end{align*}
and we have shown $g_f(\tilde p_{n_{k_\ell}})$ converges to $0$ almost surely by (ref). Thus, $g_f(\phi) =0$ almost surely, yielding
$$
\int_{ \cX \times \cC} f(\mz) \cdot (\phi(\mz) - p_Z(\mz)) \, {\mathrm d} \mz = 0
$$
almost surely, for a fixed choice of $f$. Choosing the sequence of bounded continuous functions $f_1, f_2, ... : (\cX \times \cC) \rightarrow {\mathbb R}$ guaranteed by Lemma (ref), we have
$$
\int_{ \cX \times \cC} f_i(\mz) \cdot (\phi(\mz) - p_Z(\mz)) \, {\mathrm d} \mz = 0
$$
almost surely, for all $i=1,2,...$ simultaneously, as the countable intersection of almost sure events is also almost sure. Consequently, by Lemma (ref), the measures defined by $\phi$ and $p_Z$ are equal almost surely, which implies that the density functions $\phi$ and $p_Z$ are equal for Lebesgue almost all points $ \cX \times \cC$, almost surely. Since $\phi$ and $p_Z$ are continuous on $ \cX \times \cC$, we have equality of $\phi$ and $p_Z$ at all points $ \cX \times \cC$, almost surely. Lemma (ref) then yields
$$
\sup_{\mz \in \cX \times \cC }\Big|\tilde p_{n_{k_\ell}}(\mz) - p_Z(\mz)\Big| = o_{{\mathrm P}}(1).
$$
Finally, pick a further subsequence of $n_{k_\ell}$, $n_{k_{\ell_m}}$, so that Lemma (ref) deducing
$$
\sup_{\mz \in \cX \times \cC }\Big|\tilde p_{n_{k_{\ell_m}}}(\mz) - p_Z(\mz)\Big| \rightarrow 0
$$
almost surely, which yields the final result.
\\
{\bf Step 2.} Next, we show
$$
\sup_{\mz \in \cX \times \cC}\|\sD\tilde p_{n}(\mz) - \sD p_Z(\mz)\|_2 = o_{{\mathrm P}}(1).
$$
We show this holds componentwise and denote the partial derivative with respect to the first and second argument of $\tilde p_n$ as $\sD_x$ and $\sD_y$, respectively.
We again appeal to Lemma (ref). Lemma (ref) ensures that the functions $\sD_x \tilde p_{n}$'s, for all sufficiently large $n$, are uniformly bounded and uniformly equicontinuous on $ \cX \times \cC$. Accordingly, by Arzela-Ascoli theorem rudin1987real, there exists a compact $\tilde\cC'' \subseteq C( \cX \times \cC)$ for which
$$
{\mathrm P}(\sD_x \tilde p_{n} \in \tilde \cC'') = 1
$$
holds for all sufficiently large $n$, which implies tightness of the sequence. Take any subsequence $n_k$. Applying Lemma (ref), take a further subsequence $n_{k_\ell}$ for which
$$
\sD_x \tilde p_{ n_{k_\ell}} \text{ converges weakly to }\varphi ~~~~~~\text{and}~~~~~~
\sup_{\mz \in \cX \times \cC }|\tilde p_{n_{k_{\ell}}}(\mz) - p_Z(\mz)| \rightarrow 0 \text{ almost surely}
$$
for some $C( \cX \times \cC)$-valued random variable $\varphi$.
Our next goal is to identify the limit $\varphi$ as the (deterministic) function $\sD_x p_Z$. Fix a $\mz = (x,y) \in \cX \times \cC$, and let $h > 0$ so $x + h \in \cX$. The fundamental theorem of calculus ensures
$$
\tilde p_{n_{k_\ell}}(x + h, y) - \tilde p_{n_{k_\ell}}(x,y) = \int_{x}^{x+h} \sD_x \tilde p_{n_{k_\ell}} (t,y) \, {\mathrm d} t,
$$
while implies
$$
\frac{\tilde p_{n_{k_\ell}}(x + h,y) - \tilde p_{n_{k_\ell}}(x,y)}{h} = \frac{1}{h}\int_{x}^{x+h} \sD_x\tilde p_{n_{k_\ell}} (t,y) \, {\mathrm d} t.
$$
The left-hand side satisfies
$$
\lim_{\ell \rightarrow \infty}\frac{\tilde p_{n_{k_\ell}}(x + h,y) - \tilde p_{n_{k_\ell}}(x,y)}{h} = \frac{p_Z(x+h,y) - p_Z(x,y)}{h}
$$
almost surely, using uniform convergence almost surely by our choice of $n_{k_\ell}$. Since the function
$$
g \mapsto \frac{1}{h}\int_x^{x+h} g(t, y) \, {\mathrm d} t
$$
is a continuous map on $C( \cX \times \cC)$, the continuous mapping theorem ensures that
$$
\frac{1}{h}\int_x^{x+h} \sD_x \tilde p_{n_{k_\ell}}(t, y) \, {\mathrm d} t \text{ converges weakly to }\frac{1}{h} \int_x^{x+h} \varphi(t,y) \, {\mathrm d} t
$$
as $\ell \rightarrow \infty$.
Thus, for any fixed $y \in \cC$ and any fixed and sufficiently small $h > 0$,
$$
\frac{1}{h}\int_x^{x+h} \varphi(t,y) \, {\mathrm d} t = \frac{p_Z(x+h,y) - p_Z(x,y)}{h} = \frac{1}{h}\int_x^{x+h} \sD_xp_Z(t, y) \, {\mathrm d} t
$$
almost surely by the uniqueness of limits. Using the continuity of $\varphi$ and $\sD_xp_Z$, we obtain $\varphi(\mz) = \sD_xp_Z(\mz)$ for all $\mz \in \cX \times \cC$.
Since $\sD_x p_Z$ is deterministic, we apply Lemma (ref), yielding
$$
\sup_{\mz \in \cX \times \cC}\Big|\sD_x \tilde p_{ n_{k_\ell}}(\mz) - \sD_xp_Z(\mz)\Big| = o_{{\mathrm P}}(1).
$$
Lemma (ref) then implies the existence of a further subsequence $n_{k_{\ell_m}}$ with
$$
\sup_{\mz \in \cX \times \cC}\Big|\sD_x\tilde p_{ n_{k_{\ell_m}}}(\mz) - \sD_x p_Z(\mz)\Big| \rightarrow 0
$$
almost surely. The exact same argument, taking $\sD_y$ instead, yields the full conclusion.
\\
{\bf Step 3.} Assumption (ref)(c) and (ref)(e) implies the existence of a closed, bounded interval $\tilde \cY$ for which $\cY \cup \tilde \cY_n \subseteq \tilde \cY$ for all $n$, almost surely. {\bf Step 1} and {\bf Step 2} combined then yield
$$
\sup_{\mz \in \cX \times \tilde \cY}\Big|\tilde p_n(\mz) - p_Z(\mz)\Big| = o_{{\mathrm P}}(1) ~~~~~~\text{and}~~~~~~ \sup_{\mz \in \cX \times \tilde\cY}\Big\|\sD\tilde p_n(\mz) - \sD p_Z(\mz)\Big\|_2 = o_{{\mathrm P}}(1).
$$
The above is used to establish the remaining claims.
\\
{\bf Step 4.} Next, we show $\sup_{x \in \cX}|\tilde p_X(x) - p_X(x)| = o_{{\mathrm P}}(1)$.
Expanding,
\begin{align*}
\sup_{x \in \cX}\Big|\tilde p_X(x) - p_X(x)\Big|
&= \sup_{x \in \cX}\left|\int_{\tilde \cY} \tilde p_n(x, y) - p_Z(x,y) \, {\mathrm d} y\right|
\tag{definition of $\tilde \cY$}\\
&\leq \sup_{\mz \in \cX \times \tilde \cY}\left|\tilde p_n(\mz) - p_Z(\mz)\right| \cdot\int_{\tilde \cY} \, {\mathrm d} y\\
&= o_{{\mathrm P}}(1),
\end{align*}
where the last equality comes from the fact that $\tilde \cY$ is bounded, so its Lebesgue measure is finite.
\\
{\bf Step 5.} Next, we show $\sup_{x\in \cX} |\tilde f_0(x) - f_0(x)| = o_{{\mathrm P}}(1)$. Expanding the regression functions,
\begin{align*}
\sup_{x \in \cX}|\tilde f_0(x) - f_0(x)| &=\sup_{x \in \cX} \left| \int_{{\mathbb R}} y \cdot \left(\frac{\tilde p_n(x,y)}{\tilde p_X(x)} - \frac{p_Z(x,y)}{p_X(x)}\right) \, {\mathrm d} y\right|\\
&=\sup_{x \in \cX} \left| \int_{\tilde \cY} y \cdot \left(\frac{\tilde p_n(x,y)}{\tilde p_X(x)} - \frac{p_Z(x,y)}{p_X(x)}\right) \, {\mathrm d} y\right| \\
&\leq \sup_{(x',y') \in \cX \times \tilde \cY}\left|\frac{\tilde p_n(x',y')}{\tilde p_X(x')} - \frac{p_Z(x',y')}{p_X(x')}\right| \cdot \int_{\tilde\cY} |y| \, {\mathrm d} y \\
&= o_{{\mathrm P}}(1),
\end{align*}
almost surely.
Here the last equality comes from the fact that $\tilde \cY$ is bounded, and $p_X$ is uniformly bounded below on $ \cX$ so that the continuous mapping theorem applies.
By the analogous argument, by replacing $p_X$ and $f_0$ with $p_X'$ and $\tilde f_0'$, respectively, we have
$$
\sup_{x\in \cX} |\tilde p_X'(x) - p_X'(x)| = o_{{\mathrm P}}(1)~~~{\rm and}~~~
\sup_{x \in \cX}|\tilde f_0'(x) - f_0'(x)| = o_{{\mathrm P}}(1).
$$
This completes the whole proof.
Next, we calculate the bias of the local average of $\tilde f_0$.
lemmaSuppose that Assumptions (ref), (ref), (ref), and (ref) hold. Then, for any $k,H > 0$,
$$
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|n^{1/3}(\overline{\tilde {f}}_{[\ell_n, u_n]} - \tilde f_0(x_0)) - \frac{f_0'(x_0)}{2}(u-\ell) \right| = o_{{\mathrm P}}(1).
$$
proofWe take $n$ large enough so that $[x_0 - H n^{-1/3}, x_0 +H n^{-1/3}] \subseteq \cX$, which is possible since $x_0$ is set to be an interior point of $\cX$. Since $u,\ell \leq H$, we have $[x_0 - \ell n^{-1/3}, x_0 + u n^{-1/3}] \subseteq \cX$ for all $0 \leq u, \ell \leq H$ simultaneously. Without loss of generality, we assume $n$ is large enough that this holds for the remainder of the proof. As a result,
$$
\Big|\Big\{i : \tilde X_i \in [\ell_n, u_n] \cap \cX\Big\}\Big| = \Big|\Big\{i : \tilde X_i \in [\ell_n, u_n]\Big\}\Big| \quad \text{and} \quad \mathds{1}\Big(\tilde X_i \in [\ell_n, u_n] \cap \cX\Big) = \mathds{1}\Big(\tilde X_i \in [\ell_n, u_n]\Big).
$$
Rewriting the scaled bias,
\begin{equation}
n^{1/3}(\overline{\tilde f}_{[\ell_n,u_n]} - \tilde f_0(x_0)) = \frac{n \cdot n^{-1/3}}{|\{\tilde X_i : \ell_n \leq \tilde X_i \leq u_n\}|} \cdot \frac{1}{n^{1/3}}\sum_{i=1}^n ( \tilde f_0(\tilde X_i) - \tilde f_0(x_0)) \cdot \mathds{1}(\ell_n \leq \tilde X_i \leq u_n).
\end{equation}
A routine application of Lemma (ref) and Markov's inequality applied to the functions
$$
\Big\{f_{u,\ell}: x \mapsto n^{1/2}(\tilde f_0(x) - \tilde f_0(x_0)) \cdot\mathds{1}(-\ell n^{-1/3} \leq x - x_0\leq u n^{-1/3}) : (\ell,u) \in (0,H] \times [0,H]\Big\}
$$
demonstrates
\begin{equation}
\sup_{(\ell,u) \in (0,H] \times [0,H]} \left|\frac{1}{\sqrt{n}}\sum_{i=1}^n f_{u,\ell}(\tilde X_i) - {\mathrm E}[f_{u,\ell}(\tilde X_1) \mid \cO]\right| = O_{{\mathrm P}}(1) .
\end{equation}
By similar reasoning applied to indicator functions, we also obtain
$$
\sup_{(\ell,u) \in (0,H] \times [0,H]} \left|\frac{|\{\tilde X_i : \ell_n \leq \tilde X_i \leq u_n\}|}{ n \cdot n^{-1/3}} - \frac{{\mathrm P}(\ell_n \leq \tilde X_i \leq u_n \mid \cO)}{n^{-1/3} } \right|= o_{{\mathrm P}}(1).
$$
Next, for sufficiently large $n$, the mean value theorem combined with Lemma (ref) ensures
\begin{align*}
\sup_{(\ell,u) \in (0,H] \times [0,H]} \left| \frac{{\mathrm P}(\ell_n \leq \tilde X_i \leq u_n \mid \cO) }{ n^{-1/3}} - (u + \ell) \cdot \tilde p_{X}(x_0)\right| \leq 2H^2\cdot\sup_{t \in \cX} |\tilde p_X'(t)| \cdot n^{-1/3}.
\end{align*}
By the first conclusion of Lemma (ref), we have
\[
\sup_{(\ell,u) \in (0,H] \times [0,H]}\Big|(u+\ell) \cdot (\tilde p_X(x_0) - p_X(x_0))\Big| = o_{{\mathrm P}}(1),
\]
so that the continuous mapping theorem, combined with the fact that $(u+\ell)\cdot p_X(x_0)$ is bounded uniformly below from Assumption (ref)(d), yields
\begin{equation}
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|\frac{n \cdot n^{-1/3}}{ |\{\tilde X_i : \ell_n \leq \tilde X_i \leq u_n\}|} - \frac{1}{(u+\ell) \cdot p_X(x_0)}\right| = o_{{\mathrm P}}(1).
\end{equation}
Applying (ref) and (ref) to (ref) then yields
$$
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|n^{1/3}(\overline{\tilde f}_{[\ell_n,u_n]} - \tilde f_0(x_0)) - \frac{n^{2/3}}{(u+\ell) \cdot p_X(x_0)} \cdot {\mathrm E}\left[ (\tilde f_0(\tilde X_1) - \tilde f_0(x_0)) \cdot \mathds{I}(\ell_n \leq \tilde X_1 \leq u_n) \mid \mathcal{O}\right]\right| = o_{{\mathrm P}}(1).
$$
Next, we aim to show
\begin{equation}
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|\frac{n^{2/3}}{(u+\ell) \cdot p_X(x_0)} \cdot \left({\mathrm E}\left[ \left(\tilde f_0(\tilde X_1) - \tilde f_0(x_0) - \tilde f_0'(x_0) \cdot (\tilde X_1 - x_0)\right) \cdot \mathds{I}(\ell_n \leq \tilde X_1 \leq u_n) \mid \mathcal{O}\right]\right)cal{O}\right|}} = o_{{\mathrm P}}(1).
\end{equation}
Since $n$ is large enough so that $[x_0 - \ell n^{-1/3}, x_0 + un^{-1/3}] \subseteq \cX$, for all $(\ell,u) \in (0,H] \times [0,H]$, $\tilde f_0$ is twice continuously differentiable on $ \cX$, by Lemma (ref). Hence, Taylor's theorem with remainder yields an upper bound of (ref)
\begin{equation}
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|\frac{n^{2/3}}{(u+\ell) \cdot p_X(x_0)} \cdot \sup_{x \in \cX}\frac{\tilde f_0”(x)}{2}\cdot {\mathrm E}\left[ (\tilde X_1 - x_0)^2 \cdot \mathds{I}(\ell_n \leq \tilde X_1 \leq u_n) \mid \mathcal{O}\right]\right|.
\end{equation}
Evaluating the expectation in (ref), we deduce
\begin{align*}
{\mathrm E}\left[ (\tilde X_1 - x_0)^2 \cdot \mathds{I}(\ell_n \leq \tilde X_1 \leq u_n) \mid \mathcal{O}\right] &= \int_{\ell_n}^{u_n} (x - x_0)^2 \tilde p_X(x) \, {\mathrm d} x \tag{definition of expectation}\\
&\leq \sup_{t \in \cX} |\tilde p_X(t)| \int_{\ell_n}^{u_n} (x-x_0)^2 \, {\mathrm d} x \\
&= \frac{\sup_{t \in \cX} |\tilde p_X(t)|\cdot(u^3 - \ell^3)}{3n}.
\end{align*}
Thus, we obtain
$$
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|\frac{n^{2/3}}{(u+\ell) \cdot p_X(x_0)} \cdot \sup_{x \in \cX}\frac{\tilde f_0''(x)}{2} \cdot \frac{\sup_{t \in \cX} |\tilde p_X(t)|\cdot(u^3 - \ell^3)}{3n}\right| = o_{{\mathrm P}}(1),
$$
so that
\begin{equation}
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}}\left|n^{1/3}(\overline{\tilde f}_{[\ell_n,u_n]} - \tilde f_0(x_0)) - \frac{n^{2/3}\tilde f_0'(x_0) }{(u+\ell) \cdot p_X(x_0)} \cdot {\mathrm E}\Bigg[ (\tilde X_1 - x_0) \cdot \mathds{1}(\ell_n \leq \tilde X_1 \leq u_n) \mid \cO\Bigg]\right| = o_{{\mathrm P}}(1)
\end{equation}
Since Lemma (ref) implies that $\sup_{x\in \cX} |\tilde p_X(x) - p_X(x)| = o_{{\mathrm P}}(1)$, and $\sup_{x\in \cX} |\tilde f_0'(x) - f_0'(x)| = o_{{\mathrm P}}(1)$, calculating the integral in (ref) demonstrates
\begin{equation}
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}}\left|n^{1/3}(\overline{\tilde f}_{[\ell_n,u_n]} - \tilde f_0(x_0)) - \frac{n^{2/3}f_0'(x_0) }{(u+\ell) \cdot p_X(x_0)} \cdot {\mathrm E}\Bigg[ (X_1 - x_0) \cdot \mathds{1}(\ell_n \leq X_1 \leq u_n) \Bigg]\right| = o_{{\mathrm P}}(1).
\end{equation}
Finally, evaluating the expectation in (ref), we obtain
$$
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|n^{1/3}(\overline{\tilde {f}}_{[\ell_n, u_n]} - \tilde f_0(x_0)) - \frac{f_0'(x_0)}{2}(u-\ell) \right| = o_{{\mathrm P}}(1),
$$
which finishes the proof.
Next, we note a consequence of the central limit theorem and Lemma (ref).
lemmaSuppose that Assumptions (ref), (ref), (ref), and (ref) hold.
Then, for any $k, H >0$,
$$
\sup_{t \in {\mathbb R}} \sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k} }\left|\mathrm{P}\left(n^{1/3}(\overline {\tilde Y}_{[\ell_n, u_n]} - \tilde f_0(x_0)) \leq t \middle | \mathcal{O}\right) - \mathrm{P}\left(G_{\ell, u}\leq t)\right)\right| = o_{{\mathrm P}}(1).
$$
proofRewrite the local average as
\begin{equation}
n^{1/3}\Big(\overline {\tilde Y}_{[\ell_n, u_n]} - \tilde f_0(x_0)\Big) = n^{1/3}\Big(\overline {\tilde \xi}_{[\ell_n, u_n]} + \overline{\tilde f}_{[\ell_n,u_n]}- \tilde f_0(x_0)\Big).
\end{equation}
We assume without loss of generality that $n$ is large enough so that $[x_0 - H n^{-1/3}, x_0 + Hn^{-1/3}] \subseteq \cX$.
\\
{\bf Step 1.} For (ref), we first obtain
\begin{align}
n^{1/3}\overline {\tilde \xi}_{[\ell_n, u_n]}= &\frac{n^{1/3} \cdot \sqrt{n \cdot n^{-1/3}}} {|\{i: \ell_n \leq \tilde X_i\leq u_n\}|} \cdot \frac{\sum_{i=1}^n \{\tilde \xi_{i} \cdot\mathds{1}(\ell_n \leq \tilde X_i\leq u_n) - {\mathrm E}[\tilde\xi_{1} \cdot\mathds{1}(\ell_n \leq \tilde X_1\leq u_n) \mid \cO]\}}{\sqrt{n \cdot n^{-1/3}}} \nonumber\\
&+ \frac{n^{1/3}}{|\{i: \ell_n \leq \tilde X_i\leq u_n\}|} \cdot {\mathrm E}[\tilde\xi_{1} \cdot\mathds{1}(\ell_n \leq \tilde X_1\leq u_n) \mid \cO].
\end{align}
Supposing $[\ell_n, u_n] \subseteq \cX$, it holds true that
\begin{align}
\Big|\tilde\xi_{1} \cdot \mathds{1}(\ell_{n} \leq \tilde X_{1}\leq u_{n})\Big| &= \Big|(\tilde Y_{1} - \tilde f_{0}(\tilde X_{1})) \cdot \mathds{1}(\ell_{n} \leq \tilde X_{1}\leq u_{n})\Big| \nonumber \tag{definition of $\tilde \xi_{1}$}\\
&\leq (|\tilde Y_{1}| + |\tilde f_{0}(\tilde X_{1})|) \cdot \mathds{1}(\ell_{n} \leq \tilde X_{1}\leq u_{n}) \nonumber \tag{triangle inequality}\\
&\leq 2\cdot \sup_{\mz \in \tilde \cZ} \|\mz\|_2 \cdot \mathds{1}(\ell_{n} \leq \tilde X_{1}\leq u_{n}),
\end{align}
where the final bound comes from the fact that $\tilde f_0$ is defined as a conditional expectation of $\tilde Y_1$ and Assumption (ref)(c). Equation (ref) then ensures
$$
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}}\left|\frac{n^{1/3}}{|\{i: \ell_n \leq \tilde X_i\leq u_n\}|} \cdot {\mathrm E}[\tilde\xi_{1} \cdot\mathds{1}(\ell_n \leq \tilde X_1\leq u_n) \mid \cO] \right|= o_{{\mathrm P}}(1).
$$
\\
Step 2: Reduce to a subsequence.
Assumptions (ref)(c) and (ref)(e) ensure that there exists a closed, bounded interval $\tilde \cY$ for which $\cY \cup \tilde \cY_n \subseteq \tilde \cY$ for all $n$, almost surely. We then appeal to Lemma (ref). Take any subsequence $n_k$.
Lemma (ref) and Assumption (ref)(b) ensure the existence of a further subsequence $n_{k_m}$ such that
$$
\sup_{x \in \cX}\Big|\tilde f_{0,n_{k_m}}(x) - f_0(x)\Big|\rightarrow 0, \sup_{x \in \cX \times \tilde\cY}\Big|\tilde p_{n_{k_m}}(\mz) - p_Z(\mz)\Big|\rightarrow 0, \text{ and } {\sf W}_1(\bP_{\tilde Z_{1, n_{k_m}} \mid \cO} , {\mathrm P}_{Z}) \rightarrow 0
$$
almost surely. Furthermore, by (ref) and Lemma (ref), suppose that this subsequence also satisfies
$$
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}}\left|\frac{n_{k_m} \cdot n_{k_m}^{-1/3}}{ |\{i : \ell_{n_{k_m}} \leq \tilde X_{i,n_{k_m}} \leq u_{n_{k_m}}\}|} - \frac{1}{(u+\ell)\cdot p_X(x_0)}\right| \rightarrow 0
$$
and
$$
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}}\left|n^{1/3}(\overline{\tilde {f}}_{[\ell_{n_{k_m}}, u_{n_{k_m}}]} - \tilde f_0(x_0)) - \frac{f_0'(x_0)}{2}(u-\ell) \right| \rightarrow 0
$$
almost surely.
\\
Step 3: Check that the ratio of variances uniformly converges to $1$.
This step aims to show that
$$
\sup_{(\ell,u) \in (0,H] \times [0,H]}\left|\frac{{\textrm{Var}}(\tilde \xi_{1,n_{k_m}} \cdot\mathds{1}(\ell_{n_{k_m}} \leq \tilde X_{1,n_{k_m}}\leq u_{n_{k_m}}) \mid \cO)}{{\textrm{Var}}( \xi_{1} \cdot\mathds{1}(\ell_{n_{k_m}} \leq X_{1}\leq u_{n_{k_m}})) }-1\right| \rightarrow 0
$$
almost surely. Equivalently, we show
\begin{equation}
\sup_{(\ell,u) \in (0,H] \times [0,H]}\left|\frac{{Var}(\tilde \xi_{1,n_{k_m}} \cdot\mathds{1}(\ell_{n_{k_m}} \leq \tilde X_{1,n_{k_m}}\leq u_{n_{k_m}}) \mid \cO) - {Var}( \xi_{1} \cdot\mathds{1}(\ell_{n_{k_m}} \leq X_{1}\leq u_{n_{k_m}})) }{{Var}( \xi_{1} \cdot\mathds{1}(\ell_{n_{k_m}} \leq X_{1}\leq u_{n_{k_m}})) } \right| \rightarrow 0
\end{equation}
almost surely. Rewriting the denominator of (ref),
\begin{align}
{Var}( \xi_{1} \cdot\mathds{1}(\ell_{n_{k_m}} \leq X_{1}\leq u_{n_{k_m}})) &= {\mathrm E}[\xi_1^2 \cdot \mathds{1}(\ell_{n_{k_m}} \leq X_1 \leq u_{n_{k_m}})^2] - ({\mathrm E}[\xi_1 \cdot \mathds{1}(\ell_{n_{k_m}} \leq X_1 \leq u_{n_{k_m}})])^2 \notag\\
&= {\mathrm E}[\xi_1^2 \cdot \mathds{1}(\ell_{n_{k_m}} \leq X_1 \leq u_{n_{k_m}})^2] \nonumber\\
&={\mathrm E}[\xi_1^2 \cdot \mathds{1}(\ell_{n_{k_m}} \leq X_1 \leq u_{n_{k_m}})] \notag\\
&= \sigma^2 \cdot {\mathrm P}(\ell_{n_{k_m}} \leq X_1 \leq u_{n_{k_m}})\notag\\
&= \sigma^2 \int_{\ell_{n_{k_m}}}^{u_{n_{k_m}}} p_X(t) \, {\mathrm d} t \nonumber\\
&\geq \sigma^2 \cdot \inf_{t \in \cX} p_X(t) \cdot (u + \ell) \cdot n_{k_m}^{-1/3}.
\end{align}
Next, the numerator of (ref) is upper bounded by
\begin{align*}
\Big|{\mathrm E}\Big[\tilde \xi_{1,n_{k_m}}^2 \cdot \mathds{1}(\ell_{n_{k_m}} \leq \tilde X_{1,n_{k_m}}\leq u_{n_{k_m}}) - \xi_1^2 \cdot \mathds{1}(\ell_{n_{k_m}} \leq X_{1}\leq u_{n_{k_m}}) \mid \cO\Big]\Big| \\
+ \Big|\Big({\mathrm E}\Big[\tilde \xi_{1,n_{k_m}} \cdot \mathds{1}(\ell_{n_{k_m}} \leq \tilde X_{1,n_{k_m}}\leq u_{n_{k_m}}) \mid \cO\Big]\Big)^2\Big|.
\end{align*}
Similar arguments to (ref) give
\begin{equation}
\Big|\Big({\mathrm E}\Big[\tilde \xi_{1,n_{k_m}} \cdot \mathds{1}(\ell_{n_{k_m}} \leq \tilde X_{1,n_{k_m}}\leq u_{n_{k_m}}) \mid \cO\Big]\Big)^2\Big| = (u + \ell)^2 \cdot O_{{\mathrm P}}(n_{k_m}^{-2/3})
\end{equation}
and, by Lemma (ref), we have
\begin{equation}
\Big|{\mathrm E}\Big[\tilde \xi_{1,n_{k_m}}^2 \cdot \mathds{1}(\ell_{n_{k_m}} \leq \tilde X_{1,n_{k_m}}\leq u_{n_{k_m}}) - \xi_1^2 \cdot \mathds{1}(\ell_{n_{k_m}} \leq X_{1}\leq u_{n_{k_m}}) \mid \cO\Big]\Big| = (u + \ell) \cdot o_{{\mathrm P}}(n_{k_m}^{-1/3}).
\end{equation}
Combining (ref), (ref), and (ref) in (ref) then proves the claim.
\\
\textbf{Step 4: Obtain a uniform central limit theorem using Lemma (ref), and conclude the proof.}
Consider the following function class
$$
\Big\{h_{\ell,u}(x,\xi) = \xi \cdot \mathds{1}(-\ell \leq x \leq u):0 \leq \ell, u \leq K, \ell + u \geq k \Big\}
$$
coupled with the triangular array $
\{(n^{1/3}(\tilde X_{i,n} - x_0) , n^{1/6}\tilde\xi_{i,n})\}_{i \in [n]} $ and envelope function $(x, \xi) \mapsto \xi \cdot \mathds{1}(-K \leq x \leq K)$. Lemma (ref) combined with {\bf Step 3} demonstrates that the following term that comes from (ref),
$$
\frac{1}{\sqrt{{n_{k_m}}}}\sum_{i=1}^{n_{k_m}} {n_{k_m}^{1/6}} \cdot \Big(\tilde \xi_{i,n_{k_m}} \cdot\mathds{1}(\ell_{n_{k_m}} \leq \tilde X_{i,n_{k_m}}\leq u_{n_{k_m}}) - {\mathrm E}\Big[\tilde\xi_{1,n_{k_m}} \cdot\mathds{1}(\ell_{n_{k_m}} \leq \tilde X_{1,n_{k_m}}\leq u_{n_{k_m}}) \mid \cO\Big]\Big),
$$
converges weakly to some mean-zero Gaussian process uniformly in $u, \ell \in (0,H] \times [0,H]$ with $\ell + u \geq k$ with variance function $(\ell, u) \mapsto (\ell + u) \cdot \sigma^2 \cdot p_X(x_0)$, conditionally on $\cO$.
Note that, by our choice of the subsequence, it holds true that
$$
\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}}\left|\frac{n_{k_m} \cdot n_{k_m}^{-1/3}}{ |\{i : \ell_{n_{k_m}} \leq \tilde X_{i,n_{k_m}} \leq u_{n_{k_m}}\}|} - \frac{1}{(u+\ell) \cdot p_X(x_0)}\right| \rightarrow 0
$$
almost surely.
We then obtain
$$
{n_{k_m}}^{1/3}(\overline {\tilde Y}_{[\ell_{n_{k_m}}, u_{n_{k_m}}]} - \tilde f_{0,n_{k_m}}(x_0)) \text{ converges weakly}\text{ to } G_{\ell, u} \text{ conditional on }\cO
$$
uniformly in $(\ell,u) \in (0,H] \times [0,H]$, with $\ell + u \geq k$, which further implies
$$
\sup_{t \in {\mathbb R}} \sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|\mathrm{P}\left(n_{k_m}^{1/3}(\overline {\tilde Y}_{[\ell_{n_{k_m}}, u_{n_{k_m}}]} - \tilde f_{0,n_{k_m}}(x_0)) \leq t \middle | \mathcal{O}\right) - \mathrm{P}\left(G_{\ell, u}\leq t)\right)\right| \rightarrow 0
$$
almost surely. Since $n_{k}$ was an arbitrary subsequence, we conclude that
$$
\sup_{t \in {\mathbb R}}\sup_{\substack{(\ell,u) \in (0,H] \times [0,H]\\ \ell + u \geq k}} \left|\mathrm{P}\left(n^{1/3}(\overline {\tilde Y}_{[\ell_{n}, u_{n}]} - \tilde f_{0}(x_0)) \leq t \middle | \mathcal{O}\right) - \mathrm{P}\left(G_{\ell, u}\leq t)\right)\right| = o_{{\mathrm P}}(1),
$$
and thus finish the proof.
lemmaSuppose that Assumptions (ref), (ref), (ref), and (ref) hold. Then, there exist some random variables $(\tilde L^*, \tilde U^*)$ such that
$$
{\mathrm P}\left( \tilde f_n(x_0) = \overline{\tilde Y}_{[\tilde L^*_n, \tilde U^*_n]}\right) \rightarrow 1
$$
as $n\rightarrow \infty$, where $\tilde L^*_n = x_0 - \tilde L^* n^{-1/3}$ and $\tilde U^*_n = x_0 + \tilde U^* n^{-1/3}$. Furthermore, these random variables satisfy
\begin{enumerate}
• $\tilde L^*_n, \tilde U^*_n \in \cX$ always;
• ${\mathrm P}(|\{i \in [n] : \tilde X_i \in [\tilde L_n^*, \tilde U_n^*]\}| \neq 0) \rightarrow 1$;
• $\tilde U^* = \Theta_{\mathrm P}(1)$ and $\tilde L^* = \Theta_{\mathrm P}(1)$
\end{enumerate}
proofStep 1: Apply the max-min formula to obtain the representation of $\tilde f_n(x_0)$ and choose $\tilde L_n^*, \tilde U_n^*$ that satisfy the first two conditions.
By Assumption (ref)(e), $\cX$ takes the form $\cX = [a^*, b^*]$ for some $a^*, b^* \in {\mathbb R}$. The probability of the event
$$
\left\{\tilde f_n(x_0) = \max_{a \in [n] : \tilde X_a \leq x_0} \min_{b \in [n] : \tilde X_b\geq x_0} \overline{\tilde Y}_{[\tilde X_a, \tilde X_b]}\right\}
$$
tends to $1$ by Lemma (ref). A reparameterization yields
$$
\max_{\ell > 0 :a^* \leq \ell_n \leq \max_{i \in [n]: \tilde X_i \leq x_0} \tilde X_i} \min_{u \geq 0 : b^* \geq u_n\geq \min_{i \in [n]: \tilde X_i \geq x_0} \tilde X_i} \overline{\tilde{Y}}_{[\ell_n, u_n]}
$$
always, for $\ell_n = x_0 - \ell n^{-1/3}$ and $u_n = x_0 + un^{-1/3}$ . Indeed, when the set $\{i : \tilde X_i \in \cX\}$ is empty, both are defined by convention to be $0$.
When $\{i \in [n] : \tilde X_i \in \cX\}$ is nonempty, we then choose $\tilde L^*$ and $\tilde U^*$ to satisfy
$$
\overline{\tilde Y}_{[\tilde L_n^*, \tilde U_n^*]} = \max_{\ell > 0 : a^* \leq \ell_n \leq \max_{i \in [n]: \tilde X_i \leq x_0} \tilde X_i} \min_{u \geq 0 : b^* \geq u_n\geq \min_{i \in [n]: \tilde X_i \geq x_0} \tilde X_i} \overline{\tilde{Y}}_{[\ell_n, u_n]}
$$
for
\[
\tilde{L}_n^* = x_0 - \tilde L^* n^{-1/3}~~~ {\rm and}~~~ \tilde{U}_n^* = x_0 + \tilde U^* n^{-1/3}.
\]
When $\{i \in [n] : \tilde X_i \in \cX\}$ is empty, we define $\tilde L^*$ and $\tilde U^*$ to be arbitrary constants such that $\tilde L_n^*, \tilde U_n^* \in \cX$.
By construction, the first and second conditions in Lemma (ref) are then automatically satisfied.
\\
Step 2: Show that $n^{1/3}(\tilde f_n(x_0) - \tilde f_0(x_0)) = O_{{\mathrm P}}(1)$.
We have
\begin{align}
n^{1/3}(\tilde f_n(x_0) - \tilde f_0(x_0))
&= n^{1/3}(\overline{\tilde Y}_{[\tilde L_n^*, \tilde U_n^*]} - \tilde f_0(x_0))\nonumber \\
&\leq n^{1/3}(\overline{\tilde Y}_{[\tilde L_n^*, x_0 + n^{-1/3}]} - \tilde f_0(x_0)) \nonumber\\
&= n^{1/3}\Big(\overline{\tilde \xi}_{[\tilde L_n^*,x_0+ n^{-1/3}]} + \overline{\tilde f}_{[\tilde L_n^*,x_0+ n^{-1/3}]}- \tilde{f}_0(x_0)\Big)\nonumber \\
&\leq n^{1/3}\Big(\sup_{\ell \geq 0}|\overline{\tilde \xi}_{[x_0 - \ell n^{-1/3},x_0+ n^{-1/3}]}| + \overline{\tilde f}_{[\tilde L_n^*,x_0+ n^{-1/3}]}- \tilde{f}_0(x_0)\Big) \notag\\
&= O_{{\mathrm P}}(1) + n^{1/3}(\overline{\tilde f}_{[\tilde L_n^*,x_0+ n^{-1/3}]}- \tilde{f}_0(x_0)). \nonumber\tag{Lemma (ref)}
\end{align}
Assumption (ref)(a) ensures that $\inf_{x \in \cX} f'_0(x) > 0$ and Lemma (ref) ensures that $\sup_{x \in \cX} |\tilde f'_{0}(x) - f'_0(x)| =o_{\mathrm P}(1)$. For large enough $n$, $\tilde f_0$ is continuously differentiable almost surely on $\cX$ by Lemma (ref). Also, $\cX$ is a closed interval, so we obtain
$$
{\mathrm P}\Big(\tilde f_0 \text{ is strictly increasing on }\cX\Big) \rightarrow 1
$$
as $n\rightarrow \infty$.
Next, since $\tilde L^*_n \in \cX$,
\begin{align*}
n^{1/3}(\overline{\tilde f}_{[\tilde L_n^*, x_0 + n^{-1/3}]} - \tilde{f}_0(x_0)) &\leq n^{1/3}(\overline{\tilde f}_{[x_0, x_0 + n^{-1/3}]} -\tilde{f}_0(x_0))\\
&= O_{{\mathrm P}}(1). \tag{Lemma (ref)}
\end{align*}
A lower bound follows analogously, and we thus prove the claim.
\\
Step 3: Show that $\tilde U^*, \tilde L^* = O_{{\mathrm P}}(1).$
Take any $\epsilon > 0$. Denote the events
$$
\Omega_{\epsilon} := \left\{|n^{1/3}(\tilde f_n(x_0) - \tilde f_0(x_0))| > f_0'(x_0) \cdot (H_\epsilon - 1)/8\right\}
$$
and
$$
\check\Omega_\epsilon := \left\{-n^{1/3} \sup_{u \geq 0} |\overline{\tilde \xi}_{[x_0 - n^{-1/3}, x_0 + un^{-1/3}]}| < -f_0'(x_0) \cdot (H_\epsilon - 1)/8\right\},
$$
where $H_\epsilon > 0$ is chosen so that
$$
\limsup_{n\rightarrow \infty }\Big\{{\mathrm P}(\Omega_\epsilon) + {\mathrm P}(\check \Omega_\epsilon)\Big\} < \epsilon.
$$
We then have
\begin{align*}
{\mathrm P}(\tilde U^* \geq H_\epsilon) \leq {\mathrm P}\Big(\tilde U^* \geq H_\epsilon, \Omega_\epsilon^c, \check{\Omega}_\epsilon^c \Big) + {\mathrm P}(\Omega_\epsilon) + {\mathrm P}(\check \Omega_\epsilon).
\end{align*}
We now work on the intersection of events $\Omega_\epsilon^c\,$,$\, \check{\Omega}_\epsilon^c$, and $\tilde U^* \geq H_\epsilon$. In a calculation similar to Step 2,
\begin{align*}
n^{1/3}\Big(-\sup_{u \geq 0}|\overline{\tilde \xi}_{[x_0- n^{-1/3},x_0 + u n^{-1/3}]}| + \overline{\tilde f}_{[x_0 - n^{-1/3}, \tilde U_n^*]}- \tilde{f}_0(x_0)\Big) \leq n^{1/3}(\tilde f_n(x_0) - \tilde f_0(x_0)).
\end{align*}
On the events $\check{\Omega}_\epsilon^c$ and $\Omega_\epsilon^c$,
$$
-\frac{f_0'(x_0)}{8}(H_\epsilon - 1) + n^{1/3}(\overline{\tilde f}_{[x_0 - n^{-1/3}, \tilde U_n^*]}- \tilde{f}_0(x_0)) \leq \frac{f_0'(x_0)}{8}(H_\epsilon - 1).
$$
As we have assumed $\tilde f_0$ is strictly increasing in $\cX$, and $ \tilde U_n^* \in \cX$ without loss of generality, $x_0 + H_\epsilon n^{-1/3} \in \cX$ for large enough $n$. Then, the inequality
$$
-\frac{f_0'(x_0)}{8}(H_\epsilon - 1) + n^{1/3}(\overline{\tilde f}_{[x_0 - n^{-1/3}, x_0 + H_\epsilon n^{-1/3}]}- \tilde{f}_0(x_0)) \leq \frac{f_0'(x_0)}{8}(H_\epsilon - 1)
$$
holds since $\tilde U^* \geq H_\epsilon$. By Lemma (ref), we then obtain
$$
\left|n^{1/3}(\overline{\tilde f}_{[x_0 - n^{-1/3}, x_0 + H_\epsilon n^{-1/3}]}- \tilde{f}_0(x_0)) - \frac{f'_0(x_0)}{2} (H_\epsilon - 1)\right| = o_{{\mathrm P}}(1).
$$
Hence, $ {\mathrm P}(\tilde U^* \geq H_\epsilon, \Omega_\epsilon^c, \check{\Omega}_\epsilon^c ) \rightarrow 0$. By our choice of $H_\epsilon$,
$$
\limsup_{n\rightarrow \infty }{\mathrm P}(\tilde U^* \geq H_\epsilon) \leq \epsilon
$$
and since $\epsilon$ was arbitrary, we have shown $\tilde U^* = O_{{\mathrm P}}(1).$
Analogous arguments follow to show lower bounds so that $\tilde L^* = O_{\mathrm P}(1)$. Showing $\tilde U^*, \tilde L^* = \Theta_{\mathrm P}(1)$ is then done by combining the argmax continuous mapping theorem (see Lemma 3.2.2 of van1996weak, e.g.), Lemma (ref), and Lemma (ref).
We state a result that summarizes those proven in han2022berry, demonstrating the maximizer and minimizer of the limit process, $G_{\ell, u},$ exist almost surely and are tight.
lemma[Lemmas 4.4 and 4.5 of han2022berry]
Suppose that Assumption (ref) holds. Define the random variables $(L_G^*, U_G^*)$ as the solutions to
$$
G_{L_G^*, U_G^*} = \max_{\ell > 0} \min_{u \geq 0} G_{\ell, u}.
$$
Then, $L_G^* = \Theta_{\mathrm P}(1)$ and $U_G^* = \Theta_{\mathrm P}(1)$.
Finally, we state the main conclusion of han2022berry which summarizes the asymptotic theory for the the original estimator.
lemma[Theorem 2.2 of han2022berry]
Suppose Assumption (ref) holds. Then,
$$
\sup_{t \in {\mathbb R}} \left|\mathrm{P}\left(n^{1/3}(\hat f_n(x_0) - f_0(x_0)) \leq t \middle | \mathcal{O}\right) - {\mathrm P}(\sup_{\ell > 0} \inf_{u \geq 0} G_{\ell, u} \leq t)\right| = o_{{\mathrm P}_{\cO}}(1).
$$
Supporting lemmas for Theorem (ref)
We note a condition for a uniform law of large numbers to hold uniformly over a class of probability measures, which is important in the triangular array setting. This is Theorem 2.8.1 of van1996weak adapted to our setting.
lemma[Theorem 2.8.1 of van1996weak]
Let $\cH$ be a class of uniformly bounded measurable functions from some subset $\cV \subseteq {\mathbb R}^r$ to ${\mathbb R}$. Suppose, as $n\rightarrow\infty$,
$$
\frac{\log N(\epsilon, \cH, \|\cdot\|_{\cV})}{n} \rightarrow 0
\text{ for every $\epsilon > 0$}
$$
where $\|h\|_{\cV}$ denotes the infinity norm on $\cV$.
Then, for any $\epsilon > 0$, and for any collection of probability measures $\cQ$, on ${\mathbb R}^r$
$$
\sup_{{\mathrm Q} \in \cQ}{\mathrm P}\left( \sup_{h \in \cH} \left|\frac{1}{n }\sum_{i=1}^n h(\mV_i) - {\mathrm E}[h(\mV_1)]\right| > \epsilon\right) \rightarrow 0
$$
where $\mV_1, \mV_2, ..., \mV_n$ are independently distributed as ${\mathrm Q}$ inside the supremum.
Next is a covering number lemma for Lipschitz functions on a compact, convex domain.
lemma[Theorem 2.7.1 of van1996weak]
Suppose that $\cV$ is a bounded, convex subset of ${\mathbb R}^r$. Fix $R >0$. Then, for the norm $\|f\|_{\cV} = \sup_{\mv \in \cV} |f(\mv)|$ defined for real-valued functions on $\cV$, we have
$$
\log N(\epsilon, BL(\cV,R), \|\cdot\|_{\cV}) \leq C(r,R,\cV) \cdot \epsilon^{-r}.
$$
Here, $C(r,R,\cV)$ is a constant depending only on $(r, R, \cV)$, and $BL(\cV,R)$ denotes the set of bounded, real-valued Lipschitz functions $f$ on $\cV$ that satisfy
$$
\sup_{\mv \in \cV} |f(\mv)| + \sup_{\mv \neq \mv'} \frac{|f(\mv) - f(\mv')|}{\|\mv - \mv'\|_2} \leq R.
$$.
Supporting lemmas for Theorem (ref)
We first state a few analytical results to relate topological properties of a forward map to its inverse. The first is a direct consequence of the inverse function theorem, which we state without proof.
lemma[Theorem 19.24 of rudin1976principles]
Suppose $\mh : {\mathbb R}^r \rightarrow {\mathbb R}^r$ is a bijective, continuously differentiable map with $\det(\sD \mh(\mv)) \neq 0$ for all $\mv \in {\mathbb R}^r$. Then, the inverse map $\mh^{-1}:{\mathbb R}^r \rightarrow {\mathbb R}^r$ is also continuously differentiable with
$$
\sD(\mh^{-1})(\mv) = \sD\mh(\mh^{-1}(\mv))^{-1}.
$$
Next, we state a basic lemma for upper triangular matrices that bounds the operator norm of their inverse.
lemmaSuppose $\mathbf{A} \in {\mathbb R}^{r\times r}$ is an upper triangular matrix, with diagonal terms lower bounded by a constant $k > 0$. Then,
$$
\|\mathbf{A}^{-1}\|_{\rm op} \leq \frac{\sqrt{r}}{k}\left(1+ \frac{\|\mathbf{A}\|_{\rm op}}{k}\right)^{r-1}.
$$
proofSince $\mathbf{A}$ is upper triangular, the proof idea is to use backsubstitution and explicitly bound the terms of $\mathbf{A}^{-1}$. We omit the proof as it involves tedious algebraic details.
Next is a bound of the Wasserstein distance via the KL-divergence, provided the measures' supports are uniformly bounded.
lemmaSuppose that $\mV_1, \mV_2,...$ and $\mV$ are ${\mathbb R}^r$-valued random variables with supports $\cV_1, \cV_2,... $ and $\cV$ contained in a fixed and compact set $\tilde \cV \subseteq {\mathbb R}^r$. Also, suppose $ \cV_i \supseteq \cV$ for all $i=1,2,...$ Then,
$$
{\sf W}_1({\mathrm P}_{V_i}, {\mathrm P}_V) \leq 2\cdot \sup_{\mv \in \tilde\cV} \|\mv\|_2 \cdot \sqrt{\frac{1}{2} \cdot \mathrm{KL}({\mathrm P}_V||{\mathrm P}_{V_i} )} < \infty
$$
for all $i=1,2, ...$
proofThis follows by using that $\tilde \cV$ is bounded, then Pinsker's inequality.
We then note a dual representation of ${\sf W}_1$.
lemma[Remark 5.16 of villani2008optimal]
Suppose that $\mV$ and $\mW$ are ${\mathbb R}^r$-valued random variables with ${\mathrm E}[\|\mV\|_2] < \infty$ and ${\mathrm E}[\|\mW\|_2]< \infty$. Denoting ${\rm{Lip}}(r,1)$ as the collection of all real-valued, $1$-Lipschitz functions on ${\mathbb R}^r$, we then have
$$
{\sf W}_1({\mathrm P}_V, {\mathrm P}_W) = \sup_{D \in {\rm Lip}(r,1)}{\mathrm E}\Big[D(\mV)-D(\mW)\Big].
$$
Next is the change-of-variables formula for densities.
lemma[Corollary 4.7.4 of 10.1093/oso/9780198572237.003.0008]
Suppose that $\mV$ is an ${\mathbb R}^r$-valued random variable with Lebesgue density $p_V$. Take a continuously differentiable, bijective function with a continuously differentiable inverse $\mH^{-1}: {\mathbb R}^r \rightarrow {\mathbb R}^r$. The Lebesgue density of $\mH(\mV)$, denoted by $p_{H(V)}$, then exists and is given by
$$
p_{H(V)}(\mv) = p_V(\mH^{-1}(\mv)) \cdot |\det(\sD (\mH^{-1})(\mv))|
$$
for any $\mv \in {\mathbb R}^r$.
Auxiliary results
Auxiliary results for Theorem (ref)
We collect auxiliary technical lemmas in this section. We cite without proof the following as they apply directly to our situation.
lemma[Theorem 2.3.2 of Durrett2019]
Let $\mV_1, \mV_2,...$ be a sequence of ${\mathbb R}^r$-valued random variables. The sequence converges in probability to some random variable $\mV$ if and only if for each subsequence $n_k$ of $n$, there is a further subsequence $n_{k_\ell}$ that converges almost surely to $\mV$.
Since our notion of weak convergence conditional on $\cO$ is nonstandard, we state analogues of weak convergence results. Their proofs follow by applying the classical results to each realization of the sequence of conditional measures. First is Polya's theorem.
lemma[Problem 3.2.9 of Durrett2019]
Let $\mV, \mV_1, \mV_2,...$ be ${\mathbb R}^r$-valued random variables, with $\mV$ continuous. Then, $\mV_1, \mV_2, ...$ converges weakly to $\mV$ conditionally on $\cO$ if and only if
$$
\sup_{\mv \in {\mathbb R}^r} | {\mathrm P}(\mV_n \leq \mv \mid \cO) - {\mathrm P}(\mV \leq \mv)| \rightarrow 0
$$
almost surely.
Next is the Cramer-Wold device.
lemma[Theorem 3.10.6 of Durrett2019]
Let $\mV, \mV_1, \mV_2, ...$ be ${\mathbb R}^r$-valued random variables. The sequence $\mV_1, \mV_2, ...$ converges weakly to $\mV$ conditionally on $\cO$ if, for any $\ma \in {\mathbb R}^r$, $\ma^\top \mV$ converges weakly to $\ma^\top\mV$ conditionally on $\cO$.
Next is the Lyapunov central limit theorem.
lemma[Problem 3.4.12 of Durrett2019]
Let $\{V_{i,n}\}_{n \geq 1, 1 \leq i \leq n}$ be a triangular array, conditional on $\cO$, of real-valued random variables. Suppose $0 < {\mathrm E}[V_{i,n}^2 \mid \cO] < \infty$ almost surely, for all $n\geq 1$ and $1 \leq i \leq n$. Furthermore, define $s_n^2 = \sum_{i=1}^n \mathrm{Var}(V_{i,n} \mid \cO)$ and suppose that, for some $\delta > 0$, the Lyapunov condition
$$
\frac{1}{s_n^{2+\delta}} \sum_{i=1}^n {\mathrm E}[|V_{i,n} - {\mathrm E}[V_{i,n} \mid \cO]|^{2 + \delta} \mid \cO] \rightarrow 0 \text{ almost surely}
$$
is satisfied. We then have
$
\frac{\sum_{i=1}^n V_{i,n} - {\mathrm E}[V_{i,n} \mid \cO]}{s_n}$ converges weakly to the standard normal distribution conditionally on $\cO$.
Next, we have Dudley's entropy integral stated conditionally.
lemma[Corollary 2.2.9 of van1996weak]
Suppose $\{V(\mt) : \mt \in \cT\}$ is a collection of real-valued random variables indexed by a subset $\cT \subseteq {\mathbb R}^r$. Suppose
$$
\|V(\mt) - V(\mt')\|_{\bP_{\mid \cO}, \psi_2} \leq C \|\mt - \mt'\|_2
$$
almost surely, for all $\mt, \mt' \in \cT$ and some non-random constant $C > 0$. There then exists a non-random constant $C' > 0$ such that, for every $\delta > 0$,
\begin{align*}
{\mathrm E}\Bigg[\sup_{\mt,\mt \in \cT : \|\mt - \mt'\|_2 \leq \delta} |V(\mt) - V(\mt')| \mid \cO \Bigg] \leq C' \int_{0}^\delta \sqrt{\log N(\epsilon/2, \cT, \|\cdot \|_2)} \, {\mathrm d} \epsilon
\end{align*}
almost surely.
Next, we cite a covering number bound.
lemma[Problem 7, Chapter 2.1.1 of van1996weak]
Take $R > 0$ and $\mv \in {\mathbb R}^r$. Then, for all $\epsilon > 0$,
$$
N(\epsilon, \cB(\mv, R, \|\cdot\|_2), \|\cdot \|_2) \leq \left(
\frac{3R}{\epsilon}\right)^r.
$$
Next, we show that empirical processes over certain classes of functions indexed by $\cK$ satisfy an asymptotic equicontinuity condition.
lemmaSuppose Assumption (ref), (ref), and (ref) hold.
Let
$$
\tilde V_n(\boldsymbol{\eta}) := \frac{1}{\sqrt{n}} \sum_{i=1}^n {\sf L}(\boldsymbol{\eta}, \tilde\mZ_i)- {\mathrm E}[{\sf L}(\boldsymbol{\eta}, \tilde\mZ)\mid \cO].
$$
Also, denote, for $\boldsymbol{\eta} \in \cK$ and $\mz \in \tilde\cZ$,
$$
\check R(\boldsymbol{\eta}, \mz) = \|\boldsymbol{\eta} - \tilde{\boldsymbol{\eta}}_0\|_2^{-1} \cdot \left({\sf L}(\boldsymbol{\eta}, \mz) - {\sf L}(\tilde{\boldsymbol{\eta}}_0, \mz) - \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mz)^\top (\boldsymbol{\eta} - \tilde{\boldsymbol{\eta}}_0)\right),
$$
and define the process
$$
\check V_n(\boldsymbol{\eta}) := \frac{1}{\sqrt{n}}\sum_{i=1}^n \check R(\boldsymbol{\eta},\tilde\mZ_i)- {\mathrm E}\Big[\check R(\boldsymbol{\eta},\tilde\mZ)\mid \cO\Big].
$$
There then exist some constants $C, C'>0$ such that any $\delta > 0$ small enough, we have
$$
{\mathrm E}\Bigg[\sup_{\boldsymbol{\eta} \in \cK : \|\boldsymbol{\eta} - \tilde{\boldsymbol{\eta}}_0\|_2 < \delta} \left|\tilde V_n(\boldsymbol{\eta}) - \tilde V_n(\tilde{\boldsymbol{\eta}}_0)\right| \mid \cO \Bigg] \leq C\delta
~~~{\rm and}~~~
{\mathrm E}\Bigg[\sup_{\boldsymbol{\eta} \in \cK : \|\boldsymbol{\eta} - \tilde{\boldsymbol{\eta}}_0\|_2 < \delta} \left|\check V_n(\boldsymbol{\eta})\right| \mid \cO \Bigg] \leq C'\delta
$$
almost surely for all $n$.
proofThe proof of this theorem is a consequence of using Taylor's theorem to verify the hypothesis of Lemma (ref). We omit the details as it only involves some algebraic manipulation.
From Theorem 5.52 of Vaart_1998, in the context of Section (ref), we obtain the rate of convergence for the studied estimator sequence.
lemma[Theorem 5.52 of Vaart_1998]
Suppose Assumption (ref) and Assumption (ref) holds. Furthermore, assume that there exists some constant $C > 0$ such that, for every $n$ and for every sufficiently small $\delta > 0$,
\begin{equation}
\sup_{\|\boldsymbol{\eta} - \tilde{\boldsymbol{\eta}}_0 \|_2 < \delta} {\mathrm E}[{\sf L}(\boldsymbol{\eta}, \tilde \mZ) - {\sf L}(\tilde{\boldsymbol{
\eta
}}_0, \tilde\mZ) \mid \cO] \leq -C\delta^2
\end{equation}
and
\begin{equation}
{\mathrm E}\Bigg[\sup_{\|\boldsymbol{\eta} - \tilde{\boldsymbol{\eta}}_0 \|_2 < \delta} \left|\frac{1}{\sqrt{n}}
\sum_{i=1}^n{\sf L}(\boldsymbol{\eta}, \tilde\mZ_i) - {\sf L}(\tilde{\boldsymbol{\eta}}_{0}, \tilde\mZ_i) - {\mathrm E}[{\sf L}(\boldsymbol{\eta}, \tilde\mZ) - {\sf L}(\tilde{\boldsymbol{\eta}}_{0}, \tilde\mZ) \mid \cO]\right| \mid \cO \Bigg] \leq C\delta
\end{equation}
almost surely. We then have
$$
n^{1/2}(\tilde{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0) = O_{{\mathrm P}_{\cO\tilde U}}(1).
$$
The following lemma is the conditional triangular array analogue of Lemma 19.24 of Vaart_1998. As usual, we ignore measure-theoretic complications.
lemma[Lemma 19.24 of Vaart_1998]
Suppose $\{\mV_{i,n}\}_{1 \leq i \leq n, n\geq 1}$ forms a triangular array conditional on $\cO$ of ${\mathbb R}^r$-valued random variables, and $\mV_{1,n}, ..., \mV_{n,n}$ are identically distributed conditional on $\cO$. Furthermore, suppose that $\cH_n$'s are classes of functions from ${\mathbb R}^r$ to ${\mathbb R}$ such that
\begin{enumerate}[label=(\alph*)]
• for each $n$, $\cH_n$ always contains the function $h_0: {\mathbb R}^r \rightarrow {\mathbb R}$ with $h_0(\mv) = 0$ for any $\mv\in {\mathbb R}^r$;
• for each $n$ and all $h \in \cH_n$, we have ${\mathrm E}[h(\mV_{1,n})^2 \mid \cO] < \infty$ almost surely;
• letting $\rho_n: \cH_n \times \cH_n \rightarrow [0,\infty)$ be
$$
\rho_n(h_1,h_2) := \Big({\mathrm E}\Big[(h_1(\mV_{1,n}) - h_2(\mV_{1,n}))^2 \mid \cO\Big]\Big)^{1/2},
$$
the conditional asymptotic continuity condition at $h_0$,
$$
\lim_{\delta \rightarrow 0} \limsup_{n\rightarrow \infty} {\mathrm P}\left(\sup_{\rho_n(h,h_0) \leq \delta} \left|\frac{1}{\sqrt{n}} \sum_{i=1}^n h(\mV_{i,n}) - {\mathrm E}[h(\mV_{1,n}) \mid \cO]\right| > \epsilon \mid \cO\right) = 0,
$$
holds for any choice of $\epsilon > 0$.
\end{enumerate}
Finally, suppose $\hat H_n: {\mathbb R}^r \rightarrow {\mathbb R}$ satisfies that ${\mathrm P}(\hat H_n \not\in \cH_n) = o_{{\mathrm P}}(1)$ and ${\mathrm E}[\hat H_n(\mV_{1,n})^2 \mid \cO] = o_{{\mathrm P}}(1)$.
We then have
$$
\frac{1}{\sqrt{n}}\sum_{i=1}^n \hat{H}_n(\mV_{i,n}) - {\mathrm E}[\hat{H}_n(\mV_{1,n}) \mid \cO] = o_{{\mathrm P}}(1).
$$
The following is a consequence of Lemma (ref).
lemmaSuppose Assumptions (ref), (ref), (ref) hold and that $\tilde{\boldsymbol{\eta}}_0$ is in the interior of $\cK$ with probability tending to $1$. Let $\boldsymbol{\eta}_n$ be a sequence of random variables on the bootstrap space such that $\|\boldsymbol{\eta}_n - \tilde{\boldsymbol{\eta}}_0\| = o_{{\mathrm P}}(1)$, and define
$$
\check R({\boldsymbol{\eta}}_n, \mz) = \|{\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0\|_2^{-1} \cdot \left({\sf L}({\boldsymbol{\eta}}_n, \mz) - {\sf L}(\tilde{\boldsymbol{\eta}}_0, \mz) - \sD_{\boldsymbol{\eta}}{\sf L}(\tilde{\boldsymbol{\eta}}_0, \mz)^\top ({\boldsymbol{\eta}}_n - \tilde{\boldsymbol{\eta}}_0)\right).
$$
We then have
$$
\frac{1}{\sqrt{n}}\sum_{i=1}^{n} \check R({\boldsymbol{\eta}}_n, \tilde \mZ_{i}) - {\mathrm E}[\check R({\boldsymbol{\eta}}_n, \tilde \mZ_1) \mid \cO] = o_{{\mathrm P}}(1).
$$
proofThe proof of this lemma reduces to applying Lemma (ref). In turn, the non-trivial conditions of this lemma are verified by Lemma (ref), the twice-differentiability of ${\sf L}$, and the consistency of $\boldsymbol{\eta}_n$. We omit the details.
Auxiliary results for Theorem (ref)
We recount Prokhorov's theorem applied to $C(\cE)$, for $\cE \subseteq {\mathbb R}^r$ compact, which we recall is complete and separable with respect to the supremum norm.
lemma[Theorems 5.1 and 5.2 of billing]
Suppose that $G_1, G_2, ... $ is a tight sequence of $C(\cE)$-valued random variables. Then, for each subsequence $n_k$, there exists a further subsequence $n_{k_\ell}$ for which $G_{n_{k_1}}, G_{n_{k_2}}, ...$ converges weakly to some random variable $G_0 \in C(\cE)$.
The next is a basic relation between weak convergence to a deterministic limit, and convergence in probability explicitly applied to the space $C(\cE)$. This follows from the discussion around Equation (3.7) of billing.
lemma[Equation (3.7) of billing]
Suppose that $G_1, G_2, ...$ is a sequence of $C(\cE)$-valued random variables that converges weakly to a deterministic $G_0 \in C(\cE)$. Then,
$$
\sup_{\mx \in \cE}\Big|G_n(\mx) - G_0(\mx)\Big| = o_{{\mathrm P}}(1).
$$
The next lemma states that there is a countable collection of functions that one must integrate against to test equality of measures.
lemma[Problem 1.10 of billing]
Take two ${\mathbb R}^r$-valued random variables $\mV$ and $\mW$ with supports contained in a compact subset $\cE \subseteq {\mathbb R}^r$. There then exists a sequence of real-valued continuous functions on $\cE$, denoted $f_1, f_2, ... $, for which
$$
{\mathrm E}[f_i(\mV)] = {\mathrm E}[f_i(\mW)] \text{ for all }i=1,2,... \text{ if and only if } \mV \text{ and } \mW \text{ have equal distributions.}
$$
The following result connects weak convergence to convergence in Wasserstein distance.
lemma[Theorem 6.9 of villani2008optimal]
Take an integer $d \geq 1$. Suppose that $\mV, \mV_1, \mV_2$, ... are random variables in ${\mathbb R}^r$ with ${\mathrm E}[\|\mV\|_2^d] < \infty$ and ${\mathrm E}[\|\mV_i\|_2^d]< \infty$ for each $i=1,2,...$ Then
$$
{\sf W}_{d}({\mathrm P}_{V_i}, {\mathrm P}_{V}) \rightarrow 0
$$
if and only if $\mV_1, \mV_2, ...$ converges weakly to $\mV$ and
$$
{\mathrm E}[\|\mV_i\|_2^d] \rightarrow {\mathrm E}[\|\mV\|_2^d].
$$
Here, ${\sf W}_{d}(\cdot,\cdot)$ is the Wasserstein-$d$ metric using the Euclidean 2-norm.
We next present a few analytical facts about the joint distribution.
lemmaSuppose that Assumptions (ref) and (ref) hold and assume that $n$ is sufficiently large. Then, $\tilde p_n(x,y)$ and $\sD \tilde p_n(x,y)$ are $\tilde K$- Lipschitz on $ \cX \times {\mathbb R}$ almost surely.
proofThe conclusion follows from applying the mean value theorem to $\tilde p_n$ and $\sD \tilde p_n$ on the set $\cX \times {\mathbb R}$. Then, we observe $\|\sD\tilde p_n(\mz)\|_{2}$ and $\|\sD^2 \tilde p_n(\mz)\|_{\rm op}$ are zero outside $\tilde \cZ_n$, and bounded by $\tilde K$, by Assumption (ref)(b), on $\tilde \cZ_n$.
Next, we show that $\tilde f_0$ is twice-continuously differentiable on $\cX$ with a uniformly bounded second derivative.
lemmaSuppose that $n$ is large enough so that Assumptions (ref) and (ref) hold. Then, $\tilde f_0$ is twice-continuously differentiable on $\cX$. Furthermore, $\tilde f_0''$ is upper bounded by a universal constant in $\cX$.
proofWe provide a sketch of the proof, as it involves tedious algebraic details. For any $x \in \cX$, it is almost surely true that $\tilde p_X(x) \geq \tilde K$, so that we can rewrite the regression function as
$$
\tilde f_0(x) = \int_{\tilde \cY} y \cdot \frac{\tilde p_n(x,y)}{\tilde p_X(x)} \, {\mathrm d} y
$$
for some compact set $\tilde \cY \subseteq {\mathbb R}$. By the mean value theorem and the bounded convergence theorem, we can demonstrate the identities
$$
\tilde f_0'(x) = \int_{\tilde \cY} y \cdot \sD_x\left(\frac{\tilde p_n(x,y)}{\tilde p_X(x)}\right) \, {\mathrm d} y \qquad \text{and} \qquad
\tilde f_0''(x) = \int_{\tilde \cY} y \cdot \sD_x^2\left(\frac{\tilde p_n(x,y)}{\tilde p_X(x)}\right) \, {\mathrm d} y
$$
hold almost surely. Continuity and boundedness of the latter display by a universal constant follows due to the continuity and boundedness of the second derivatives of $\tilde p_n$ and $\tilde p_X$.
We cite an algebraic identity of robertson1988order.
lemma[Theorem 1.4.4 of robertson1988order]
Suppose that $m \geq 1$ is a positive integer, $\cV$ is a nondegenerate closed interval, and that $(v_1, w_1), ..., (v_m,w_m) \in \cV \times {\mathbb R}$ are fixed points. Define $\check f: \cV \rightarrow {\mathbb R}$ as the solution to
$$
\check f :=
\underset{f : \cV \to {\mathbb R} \text{ nondecreasing}}{\arg\min}
\sum_{i=1}^m \big(w_i - f(v_i)\big)^2.
$$
Take any point $v_0 \in \cV$. The following identity then holds true:
$$
\check f(v_0) = \max_{a \in [m] : v_a < v_0} \min_{b \in [m]: v_b \geq v_0}\frac{1}{|\{1 \leq i \leq m : v_i \in [v_a,v_b] \}|} \sum_{1 \leq i \leq m : v_i \in [v_a,v_b]} w_i.
$$
Lemma (ref) then implies that the max-min identity holds true for the bootstrap data with probability tending to $1$.
lemmaSuppose Assumptions (ref) and (ref) hold. Then, as $n\rightarrow\infty$,
$$
{\mathrm P}\Big(\Big|\Big\{i \in [n] : \tilde X_i \in \cX \cap [x_0, \infty)\Big\}\Big| \neq 0\Big) \wedge{\mathrm P}\Big(\Big|\Big\{i \in [n] : \tilde X_i \in \cX \cap (-\infty, x_0]\Big\}\Big| \neq 0\Big) \rightarrow 1.
$$
As a result, the probability of the event
$$
\left\{\tilde f_n(x_0) = \max_{a \in [n] : \tilde X_a \leq x_0} \min_{b \in [n] : \tilde X_b\geq x_0} \overline{\tilde Y}_{[\tilde X_a, \tilde X_b]}\right\}
$$
tends to $1$ as $n\rightarrow \infty$. That is, the bootstrap least-squares solution at $x_0$, $\tilde f_n(x_0)$, satisfies a max-min formula with high probability as $n\rightarrow\infty$.
proofTo demonstrate the first conclusion, recall that
$$
{\mathrm P}\Big(\tilde X_i \in \cX \cap (-\infty,x_0] \mid \cO\Big) \geq \int_{\cX\cap (-\infty,x_0]} \tilde K \, {\mathrm d} x > 0
$$
almost surely from Assumption (ref)(b). Since $\tilde X_i$'s are conditionally independent given $\cO$, we have
$$
{\mathrm P}\Big(\Big|\Big\{i \in [n] : \tilde X_i \in \cX \cap (-\infty,x_0]\Big\}\Big| = 0 \mid \cO\Big) \leq \left(1 - \int_{\cX \cap (-\infty, x_0]}\tilde K\,{\mathrm d} x\right)^n \rightarrow 0
$$
almost surely. The exact same reasoning for
$$
{\mathrm P}\Big(\Big|\Big\{i \in [n] : \tilde X_i \in \cX \cap [x_0, \infty)\Big\}\Big| = 0 \mid \cO\Big)
$$
combined with the bounded convergence theorem completes the proof of the first claim.
To show that the second claim is true, we first work on the intersection of the events $\{|\{i \in [n] : \tilde X_i \in \cX \cap (-\infty, x_0]\}| \neq 0\}$ and $\{|\{i \in [n] : \tilde X_i \in \cX \cap [x_0, \infty)\}| \neq 0\}$.
Since $\tilde f_n$ solves (ref),
Lemma (ref) yields
$$
\tilde f_n(x_0) = \max_{a \in [n] : \tilde X_a \leq x_0} \min_{b \in [n] : \tilde X_b\geq x_0} \frac{1}{|\{i : \tilde X_i \in \cX \cap [\tilde X_a,\tilde X_b]\}|} \sum_{1 \leq i \leq n : \tilde X_i \in \cX \cap [\tilde X_a,\tilde X_b]} \tilde Y_i.
$$
where $\cX = [a^*, b^*]$. This is exactly the claim of the lemma.
Next is a consequence of Lemma 4.3 from han2022berry.
lemma[Lemma 4.3 of han2022berry]
Suppose Assumptions (ref) and (ref) hold. Then,
$$
n^{1/3} \sup_{u \geq 0} \Big|\overline{\tilde \xi}_{[x_0-n^{-1/3}, x_0 + un^{-1/3}]}\Big| = O_{\mathrm P}(1) \quad \text{and} \quad n^{1/3} \sup_{\ell \geq 0} \Big|\overline{\tilde \xi}_{[x_0 - \ell n^{-1/3}, x_0 + n^{-1/3}]}\Big| = O_{\mathrm P}(1).
$$
The next lemma states conditions, under which a uniform central limit theorem holds conditional on $\cO$ in a triangular array setting. This is the combination of Lemmas 2.8.2 and 2.8.7 in van1996weak.
lemma[Lemmas 2.8.2 and 2.8.7 in van1996weak]
Suppose $\cH_{\cT} = \{h_{t} : \mt \in \cT\}$ is a collection of real-valued functions defined on ${\mathbb R}^r$ indexed by a subset $\cT \subseteq {\mathbb R}^r$. Suppose that $\cH_{\cT}$ contains $h_0$, the function that maps all points to zero. Additionally, suppose that there exists a non-negative function $H : {\mathbb R}^r \rightarrow {\mathbb R}$ such that
\begin{enumerate}[label=(\alph*)]
• $\sup_{h_t \in \cH_{\cT}} |h_t(\mv)| \leq H(\mv) \text{ for all }\mv \in {\mathbb R}^r;$
• $\sup_{n} {\mathrm E}[H(\mV_{1,n})^2 \mid \cO] < \infty$ almost surely;
• $\limsup_{n\rightarrow \infty} {\mathrm E}[H(\mV_{1,n})^2 \cdot \mathds{1}(H(\mV_{1,n}) \geq \epsilon n^{1/2}) \mid \cO]=0$ almost surely, for every $\epsilon > 0$,
\end{enumerate}
where $\{\mV_{i,n}\}_{1\leq i\leq n, n\geq 1}$ are ${\mathbb R}^r$-valued random variables forming a triangular array conditional on $\cO$ and $\mV_{1,n}, ..., \mV_{n,n}$ are identically distributed conditional on $\cO$. Let $\mV$ be another ${\mathbb R}^r$-valued random variable. Let
$$
\rho_{n}(h_t, h_{t'}) = {\mathrm E}\Big[\Big(h_{t}(\mV_{1,n}) - h_{t'}(\mV_{1,n})\Big)^2 \mid \cO \Big]^{1/2} \quad \text{and} \quad \rho_0(h_t, h_{t'}) = {\mathrm E}\Big[\Big(h_{t}(\mV) - h_{t'}(\mV)\Big)^2 \Big]^{1/2}.
$$
Suppose that
\begin{enumerate}[label=(\alph*)]
• as $n\rightarrow \infty$, $\sup_{\mt, \mt' \in \cT} |\rho_n(h_t, h_{t'}) - \rho_0(h_t, h_{t'})| \rightarrow 0
$ almost surely;
• uniformly in $n=1,2,...$, $\cH_{\cT}$ is totally bounded with respect to $\rho_n$ almost surely;
• for every $\epsilon > 0$,
$$
\lim_{\delta \rightarrow 0} \limsup_{n\rightarrow \infty} {\mathrm P}\left(\sup_{\rho_n(h_t, h_{t'}) < \delta} \left|\frac{1}{\sqrt{n}} \sum_{i=1}^n h_{t}(\mV_{i,n}) -h_{t'}(\mV_{i,n}) - {\mathrm E}[h_{t}(\mV_{i,n}) - h_{t'}(\mV_{i,n}) \mid \cO]\right| > \epsilon \mid \cO\right) = 0
$$
almost surely.
\end{enumerate}
Then,
$$
\sup_{\mt \in \cT, s \in {\mathbb R}} \left|{\mathrm P}\left(\frac{1}{\sqrt{n}}\sum_{i=1}^n h_{t}(\mV_{i,n}) - {\mathrm E}[h_t(\mV_{1,n})\mid \cO] \leq s \mid \cO\right)
- {\mathrm P}\left(S \leq s\right)\right| \rightarrow 0
$$
almost surely, where $S$ is a mean-zero normal random variable with variance ${\rm Var}(h_t(\mV))$.