Bootstrap consistency for general double/debiased machine learning estimators

abstractDouble/debiased machine learning (DML) provides a general framework for inference with high-dimensional or otherwise complex nuisance parameters by combining Neyman-orthogonal scores with cross-fitting, thereby circumventing classical Donsker-type conditions in many modern machine-learning settings. Despite its strong empirical performance, bootstrap inference for DML estimators has received little theoretical justification. This is particularly noteworthy since bootstrap methods are suggested and used for inference on DML estimators, even though bootstrap procedures can fail for estimators that are root-\(n\) consistent and asymptotically normal. This paper fills this gap by establishing bootstrap validity for DML estimators under general exchangeably weighted resampling schemes, with Efron’s bootstrap as a special case. Under exactly the same conditions required for the validity of DML itself, we prove that the bootstrap law converges conditionally weakly to the sampling law of the original estimator.

assumptionFor the triangular array of bootstrap weights $W$ of the law ${\mathrm P}_w$, it is assumed that the following holds true. \begin{enumerate} [itemsep=-.5ex, label=(\roman*)] • The random vector $W=(W_1, \ldots, W_n)^{\top}$ is exchangeable. • $W_i \ge 0$ for each $i \in \llbracket n\rrbracket$ and $\sum_{i=1}^n W_i = n$. • $\limsup_{n \to \infty} \int_0^{\infty} \sqrt{{\mathrm P}_W \left(W_1 \ge t\right)} {\mathrm d} t \le C$ for some constant $C< \infty$. • $\lim_{t_0 \to \infty} \limsup_{n \to \infty} \sup_{t \ge t_0} t^2 {\mathrm P}_W \left(W_1 \ge t\right) = 0$. • $\operatorname{plim}_{n \to \infty} n^{-1} \sum_{i=1}^n (W_i-1)^2 = c^2 >0$. \end{enumerate}

exampleThe resampling schemes satisfying Assumption (ref) include the following. \begin{enumerate} [itemsep=-.5ex, label=(\roman*)] • Efron’s bootstrap efron1979bootstrap: the weight vector is multinomially distributed such that \[ W \sim {\rm Mult}(n;(n^{-1}, \ldots, n^{-1})). \] In this case, $c^2=1$. • Normalized multiplier bootstrap: the weight vector is \[ W_i=Y_i\Big/\Big(\frac{1}{n}\sum_{i=1}^n Y_i\Big), ~~\text{for }i\in\llbracket n\rrbracket, \] where $\{Y_i\}_{i=1}^n$ are independent and identically distributed positive random variables satisfying \[ \int_0^{\infty} \sqrt{{\mathrm P}_Y \left(Y_1 \ge t\right)} {\mathrm d} t<\infty. \] In this case $c^2 = \Var[Y_1]/({\mathrm E}[Y_1])^2$. A special normalized multiplier bootstrap procedure is the Bayesian bootstrap rubin1981bayesian, obtained by taking $Y_1 \sim {\rm Exp}(1)$, for which $c^2=1$. More generally, $Y_1 \sim \Gamma(\alpha,\alpha)$ yields $c^2 = 1/\alpha$. • Double bootstrap beran1987prepivoting: let \[ M=(M_1,\ldots,M_n)^\top \sim {\rm Mult}(n;(n^{-1}, \ldots, n^{-1})); \] then conditioning on $M$, $W$ is sampled from \[ W|M \sim {\rm Mult}(n;(n^{-1}M_1, \ldots, n^{-1}M_n)). \] In this case $c^2=2$. \end{enumerate}

remarkIn the special case of Efron’s bootstrap, the weighted empirical measure \({\mathbb{P}_{n,k}^{\ast}}\) admits the following equivalent representation. First, draw bootstrap indices \(\{j_i\}_{i=1}^n\) independently from the uniform distribution on \(\llbracket n\rrbracket\). Then form the bootstrap sample \(\{X_i^* := X_{j_i}\}_{i=1}^n\). Under this representation, the fold-restricted bootstrap empirical measure can be written as \begin{align*} {\mathbb{P}_{n,k}^{\ast}}[f] = \frac{1}{m}\sum_{i=1}^n \ind(j_i \in I_k)\, f(X_i^*), \qquad for any measurable function f. \end{align*}

remarkFor simplicity of presentation, this paper focuses exclusively on bootstrap procedures based on a single round of resampling from the full dataset. In practice, however, it may be also natural to consider within-fold resampling. Specifically, for each \(k \in \llbracket K\rrbracket\), let \((W_{k,1},\ldots,W_{k,m})\) be bootstrap weights satisfying Assumption (ref), generated independently across folds. The corresponding weighted bootstrap empirical measure is then defined by \[ {\tilde{\mathbb{P}}_{n,k}^{\ast}}[f] = \frac{1}{m}\sum_{i\in I_k} W_{k,i} f(X_i), \qquad \text{for any measurable function } f. \] The associated bootstrapped DML estimator can be defined analogously. By inspecting the proofs below, it is clear that the bootstrap consistency result established in Theorem (ref) continues to hold for this within-fold bootstrap procedure as well. We omit the details for brevity.

assumptionFor \({\mathrm P} \in \cP_n\), the following conditions are assumed to hold: \begin{enumerate}[itemsep=-.5ex,label=(\roman*)] • \(\{X_i\}_{i=1}^n\) are independent draws from \({\mathrm P}\); • there exists a fixed integer \(K \ge 2\) such that \(n\) is divisible by \(K\) and \(m = n/K\); • the score function \(\psi(\cdot;\theta_0,\eta_0)\) satisfies the moment condition \[ {\mathrm P}\big[\psi(\cdot;\theta_0,\eta_0)\big] = 0; \] • the computational error \(\epsilon_n\) in (ref) and (ref) satisfies \[ \epsilon_n = o(\delta_n n^{-1/2}). \] \end{enumerate}

assumptionFor every \(n \ge 3\) and every \({\mathrm P} \in \mathcal{P}_n\), the following conditions hold true. \begin{enumerate}[itemsep=-.5ex,label=(\roman*)] • The true parameter \(\theta_0\) satisfies the moment restriction (ref), and the parameter space \(\Theta\) contains the ball centered at \(\theta_0\) with radius \(c_1 n^{-1/2}\log n\). • The mapping \[ (\theta,\eta) \mapsto {\mathrm P}\big[\psi(\cdot;\theta,\eta)\big] \] is twice continuously G\^ateaux differentiable on \(\Theta \times T\). • For every \(\theta \in \Theta\), the following local identification condition is satisfied: \[ 2\Big\lVert {\mathrm P}\big[\psi(\cdot;\theta,\eta_0)\big] \Big\rVert \ge \|J_0(\theta-\theta_0)\| \wedge c_0, \] where \begin{align*} J_0 := \left. \partial_{\theta^\top} \Big\{ {\mathrm P}\big[\psi(\cdot;\theta,\eta_0)\big] \Big\} \right|_{\theta=\theta_0}. \end{align*} In addition, all singular values of \(J_0\) lie in the interval \([c_0,c_1]\). • The score function \(\psi\) satisfies an approximate Neyman-orthogonality condition on the nuisance realization set \(\mathcal{T}_n \subset T\) with tolerance level \(\lambda_n = \delta_n n^{-1/2}\). Specifically, for every \(\eta \in \mathcal{T}_n\), \begin{align*} \Big\lVert \partial_\eta {\mathrm P}\big[\psi(\cdot;\theta_0,\eta_0)\big][\eta-\eta_0] \Big\rVert \le \lambda_n. \end{align*} \end{enumerate}

assumptionFor every \(n \ge 3\) and every \({\mathrm P} \in \mathcal{P}_n\), the following conditions are assumed to be satisfied. \begin{enumerate}[itemsep=-.5ex,label=(\roman*)] • Let \(I\) be a random subset of \(\llbracket n\rrbracket\) of cardinality \(n/K\). The nuisance estimator \[ \hat{\eta}_0 = \hat{\eta}_0\big(\{X_i\}_{i\in I^c}\big) \] belongs to the realization set \(\mathcal{T}_n\) with \({\mathrm P}\)-probability at least \(1-\Delta_n\), where \(\{\Delta_n\}_{n\ge 1}\) is a sequence of positive constants converging to zero. Moreover, \(\mathcal{T}_n\) contains \(\eta_0\) and satisfies the conditions stated below. • The parameter space \(\Theta\) is bounded. For each \(\eta \in \mathcal{T}_n\), define the class of score functions \[ \mathcal{F}_{1,\eta} := \Big\{ \psi_j(\cdot;\theta,\eta) : j=1,\ldots,d_\theta,\ \theta\in\Theta \Big\}. \] This class is assumed to be suitably measurable, and its uniform covering entropy satisfies \begin{align*} \sup_{{\mathrm Q}} \log N\Big( \epsilon \|F_{1,\eta}\|_{{\mathrm Q},2}, \mathcal{F}_{1,\eta}, \|\cdot\|_{{\mathrm Q},2} \Big) \le v \log(a/\epsilon), \qquad for all 0<\epsilon\le 1, \end{align*} where \(F_{1,\eta}\) is a measurable envelope of \(\mathcal{F}_{1,\eta}\) satisfying \[ \|F_{1,\eta}\|_{{\mathrm P},q}\le c_1. \] • The following rate conditions hold for the quantities \(r_n\), \(r_n'\), and \(\lambda_n'\): \begin{align*} r_n &:= \sup_{\eta\in\mathcal{T}_n,\ \theta\in\Theta} \Big\lVert {\mathrm P}\big[\psi(\cdot;\theta,\eta)-\psi(\cdot;\theta,\eta_0)\big] \Big\rVert \le \delta_n \tau_n, \\ r_n' &:= \sup_{\eta\in\mathcal{T}_n,\ \|\theta-\theta_0\|\le \tau_n} \Big( {\mathrm P} \big\| \psi(\cdot;\theta,\eta)-\psi(\cdot;\theta_0,\eta_0) \big\|^2 \Big)^{1/2}, \qquad r_n' \log^{1/2}(1/r_n') \le \delta_n, \\ \lambda_n' &:= \sup_{\substack{r\in(0,1),\ \eta\in\mathcal{T}_n,\\ \|\theta-\theta_0\|\le \tau_n}} \Big\lVert \partial_r^2 {\mathrm P}\Big[ \psi\big(\cdot;\theta_0+r(\theta-\theta_0),\,\eta_0+r(\eta-\eta_0)\big) \Big] \Big\rVert \le \delta_n n^{-1/2}. \end{align*} • The score has nondegenerate covariance at the truth, in the sense that all eigenvalues of \begin{align*} {\mathrm P}\Big[ \psi(\cdot;\theta_0,\eta_0)\psi(\cdot;\theta_0,\eta_0)^\top \Big] \end{align*} are bounded below by \(c_0\). \end{enumerate}

theorem[Theorem 3.3, chernozhukov2018double] Suppose Assumptions (ref)--(ref) hold. Then the DML estimator \(\check{\theta}_0\) lies in an \(n^{-1/2}\)-neighborhood of \(\theta_0\) and admits an asymptotically linear expansion. More precisely, \begin{align*} \sqrt{n}(\check{\theta}_0-\theta_0) = \sqrt{n}\,{\mathbb{P}_{n}} \bar{\psi}_0(\cdot) + O_{{\mathrm P}}(\rho_n) \end{align*} uniformly over \({\mathrm P} \in \cP_n\), where \[ {\mathbb{P}_{n}} := \frac{1}{n}\sum_{i=1}^n \delta_{X_i}, \qquad \bar{\psi}_0(\cdot) := -J_0^{-1}\psi(\cdot;\theta_0,\eta_0), \] and the remainder term satisfies \begin{align*} \rho_n &:= n^{-1/2+1/q}\log n + r_n' \log^{1/2}(1/r_n') + n^{1/2}\lambda_n + n^{1/2}\lambda_n' + n^{1/2}\epsilon_n\\ &\lesssim \delta_n = o(1). \end{align*} As a consequence, uniformly over \({\mathrm P} \in \cP_n\), \begin{align*} \sup_{t \in \bR^{d_\theta}} \Big| {\mathrm P}\big(\sqrt{n}(\check{\theta}_0-\theta_0)\le t\big) - {\mathrm P}\big(N(0,\Sigma^2)\le t\big) \Big| = o(1), \end{align*} where the inequality “\(\le\)” is understood componentwise, and \begin{align*} \Sigma^2 := J_0^{-1} {\mathrm E}_{{\mathrm P}}\Big[\psi(X;\theta_0,\eta_0)\psi(X;\theta_0,\eta_0)^\top\Big] (J_0^{-1})^\top. \end{align*}

theorem[Bootstrap distribution consistency for DML estimators] Suppose Assumptions (ref)--(ref) hold for the original DML estimator, and that the bootstrap weights satisfy Assumption (ref). Then the bootstrapped DML estimator satisfies \begin{align*} \sqrt{n}(\check{\theta}_0^*-\check{\theta}_0) = \bG_n^*[\bar{\psi}_0(\cdot)] + O_{{\mathrm P}_{XW}}^o(\rho_n^*) \addtocounter{equation}{1}\tag{\theequation} \end{align*} uniformly over \({\mathrm P}_X \in \mathcal{P}_n\), where \begin{align*} \rho_n^* &:= n^{-1/2+1/q}\log n + r_n' \log^{1/2}(1/r_n') + n^{1/2}\lambda_n + n^{1/2}\lambda_n' + n^{1/2}\epsilon_n + a_n^{(q-2)/(3q-2)}\log(1/a_n) \\ & = o(1). \end{align*} Consequently, we have, uniformly over \({\mathrm P}_X \in \cP_n\), \begin{align*} \sup_{t \in \bR^{d_\theta}} \Big| {\mathrm P}_{W|X}\Big(\sqrt{n}c^{-1}(\check{\theta}_0^*-\check{\theta}_0)\le t\Big) - {\mathrm P}\big(N(0,\Sigma^2)\le t\big) \Big| &= o_{{\mathrm P}_X}^o(1), \addtocounter{equation}{1}\tag{\theequation} \\ \sup_{t \in \bR^{d_\theta}} \Big| {\mathrm P}_{W|X}\Big(\sqrt{n}c^{-1}(\check{\theta}_0^*-\check{\theta}_0)\le t\Big) - {\mathrm P}\Big(\sqrt{n}(\check{\theta}_0-\theta_0)\le t\Big) \Big| &= o_{{\mathrm P}_X}^o(1), \addtocounter{equation}{1}\tag{\theequation} \end{align*} where \(c^2\) is the limit appearing in Assumption (ref)(v).

proposition[Distributional approximation rates for different bootstrap weights] Suppose the bootstrap weights \(W\) satisfy Assumption (ref). Then, for any \(x>0\) possibly depending on \(n\), the quantity \(a_n\) satisfies \begin{align} \frac{1}{\sqrt{n}} \le a_n \le \frac{x}{\sqrt{n}} + \frac{\sqrt{n}}{x} \Big[\sup_{t \ge x} t^2 {\mathrm P}(W_1 \ge t)\Big]. \end{align} In particular, \begin{enumerate}[itemsep=-.5ex, label=(\alph*)] • if \(W_1 \le B_n\) almost surely for some constant \(B_n\) possibly depending on \(n\), then $a_n \le B_n/\sqrt{n}$; • if \({\mathrm P}(W_1 \ge t) \lesssim \exp(-bt)\) for some \(b>0\), then $a_n \lesssim \log n/\sqrt{n}$; • if \({\mathrm P}(W_1 \ge t) \lesssim t^{-\beta}\) for some \(\beta>2\), then $a_n \lesssim n^{1/\beta - 1/2}$. \end{enumerate} In particular, for the bootstrap schemes in Example (ref), we have: \begin{enumerate}[itemsep=-.5ex, label=(\roman*)] • for Efron's bootstrap, \begin{align*} a_n = (1+o(1))\frac{1}{\sqrt{n}}\frac{\log n}{\log\log n}; \end{align*} • for normalized multiplier bootstrap with Gamma weights, including the Bayesian bootstrap, as well as for the double bootstrap, we have $a_n \lesssim \log n/\sqrt{n}$; • the optimal rate \(n^{-1/2}\) for \(a_n\) can be attained by some suitably constructed bootstrap weights. \end{enumerate}

proof[Proof of Theorem (ref)] We prove this theorem in six steps. For establishing (ref), by the definition of $\check{\theta}_0$ and $\check{\theta}_0^*$, we will show the general arguments for establishing \begin{align*} \sqrt{m} (\check{\theta}_{0,k}^* - \check{\theta}_{0,k}) = - J_0^{-1} {\mathbb{G}_{n,k}^{\ast}} \big[ \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] + O_{{\mathrm P}_{XW}}^o(\rho_n^*) \qquad for all k\in\llbracket K\rrbracket \end{align*} in Step I-II, and then use Steps III-V to derive the bounds used in Step I-II. We prove the distributional convergence results in (ref) in Step VI; and (ref) follows easily from Theorem (ref). For the most parts of our proof, we implicitly condition on $\{X_i\}_{i \in I_k^c}$ so that $\hat{\eta}_{0,k}$ can be treated as fixed. To verify the asserted uniformity, fix any sequence $\{{\mathrm Q}_n\}_{n\ge1}$ such that ${\mathrm Q}_n \in \cP_n$ for all $n$. For each $n$, we interpret the generic data-generating law ${\mathrm P}_X$ as ${\mathrm Q}_n$. All stochastic orders and all law-dependent objects (for example, $\theta_0,\eta_0,J_0,\Sigma^2,\bar\psi_0$) are understood as those associated with ${\mathrm Q}_n$. We suppress this dependence on $n$. Since the sequence $\{{\mathrm Q}_n\}_{n\ge1}$ is arbitrary, the row-wise statements proved below hold uniformly over ${\mathrm P}_X \in \cP_n$ by Lemma (ref). When subsequence arguments are used in Step VI, we view the resulting triangular array on a common row-wise product space. {\bf Step I.} First we derive the preliminary rate for the bootstrapped estimator $\check{\theta}_{0,k}^*$. That is, in this step we show that, with ${\mathrm P}_{XW}^o$-probability $1-o(1)$, one has \begin{align*} \big\lVert \check{\theta}_{0,k}^* - {\theta}_0 \big\rVert \le \tau_n. \addtocounter{equation}{1}\tag{\theequation} \end{align*} Observe that, by the approximate $\epsilon_n$-solution condition, we have \begin{align*} \big\lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \big\rVert \le \big\lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(\cdot; \theta_0, \hat{\eta}_{0,k})] \big\rVert + \epsilon_n, \epsilon_n = o(\delta_n n^{-1/2}). \end{align*} Applying the triangle inequality to the lefthand side, we have \begin{align*} \big\lVert {\mathrm P} [\psi(\cdot; \check{\theta}_{0,k}^*, {\eta}_{0})] \big\rVert &\le \big\lVert {\mathrm P} [\psi(\cdot; \check{\theta}_{0,k}^*, {\eta}_{0}) - \psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \big\rVert + \big\lVert ({\mathrm P}-{\mathbb{P}_{n,k}}) [\psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \big\rVert \\ & + \big\lVert ({\mathbb{P}_{n,k}} - {\mathbb{P}_{n,k}^{\ast}}) [\psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \big\rVert + \big\lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \big\rVert \end{align*} along with \begin{align*} \big\lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(\cdot; \theta_0, \hat{\eta}_{0,k})] \big\rVert &\le \big\lVert ({\mathbb{P}_{n,k}^{\ast}} - {\mathbb{P}_{n,k}}) [\psi(\cdot; {\theta}_{0}, \hat{\eta}_{0,k})] \big\rVert + \big\lVert ( {\mathbb{P}_{n,k}} - {\mathrm P}) [\psi(\cdot; {\theta}_{0}, \hat{\eta}_{0,k})] \big\rVert \\ & + \big\lVert {\mathrm P} [\psi(\cdot; {\theta}_{0}, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_{0}, {\eta}_{0})] \big\rVert + \big\lVert {\mathrm P} [\psi(\cdot; {\theta}_{0}, {\eta}_{0})] \big\rVert \end{align*} on the righthand side. Combine the terms and note the moment condition yields \begin{align*} \big\lVert {\mathrm P} [\psi(\cdot; \check{\theta}_{0,k}^*, {\eta}_{0})] \big\rVert &\le 2T_1 + 2T_2 + 2T_3 + \epsilon_n, \end{align*} where \begin{align*} T_1 &:= \sup_{\theta \in \Theta, \eta \in \mathcal{T}_n} \big\lVert {\mathrm P} [\psi(\cdot; {\theta}, {\eta}) - \psi(\cdot; {\theta}, {\eta}_{0})] \big\rVert, \\ T_2 &:= m^{-1/2} \sup_{\theta \in \Theta} \big\lVert {\mathbb{G}_{n,k}}[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \big\rVert, \\ T_3 &:= m^{-1/2} \sup_{\theta \in \Theta} \big\lVert {\mathbb{G}_{n,k}^{\ast}}[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \big\rVert. \end{align*} Here, by Assumption (ref)(iii), one has \[ T_1 = r_n \le \delta_n \tau_n = o(\tau_n). \] Moreover, conditioning on $\{X_i\}_{i \in I_k^c}$, the nuisance estimator $\hat{\eta}_{0,k}$ can be treated as fixed. Lemma (ref) then yields, with ${\mathrm P}_{X}$-probability $1-o(1)$, \begin{align*} T_2 \lesssim n^{-1/2}(1+n^{-1/2+1/q}\log n) \lesssim n^{-1/2} = o(\tau_n). \end{align*} In Step III below, we will show that \[ T_3 = o(\tau_n)~~ \text{ with ${\mathrm P}_{XW}^o$-probability goes to $1$.} \] Since $\epsilon_n = o(\delta_n n^{-1/2}) = o(\tau_n)$ as $n^{-1/2}\log n \le \tau_n$, by Assumption (ref)(iii) we conclude that, with ${\mathrm P}_{XW}^o$-probability $1-o(1)$, \begin{align*} \big\lVert J_0 (\check{\theta}_{0,k}^*- \theta_0) \big\rVert \wedge c_0 \le 2 \big\lVert {\mathrm P} [\psi_(\cdot; \check{\theta}_{0,k}^*, \eta_0)] \big\rVert = o(\tau_n). \end{align*} Since the singular values of $J_0$ is bounded away from zero, we then have proved (ref) with the same probability. Combined with the proof of Theorem (ref) in chernozhukov2018double, which gives a similar treatment to $\check{\theta}_{0,k}$, one has the following event holds in ${\mathrm P}_{XW}^o$-probability goes to $1$: \begin{align*} \Big\{ \hat{\eta}_{0,k} \in \cT_n, \big\lVert \check{\theta}_{0,k} - {\theta}_0 \big\rVert \vee \big\lVert \check{\theta}_{0,k}^* - {\theta}_0 \big\rVert \le \tau_n \Big\}; \end{align*} and we will work on this event in the subsequent steps. {\bf Step II.} Here we prove the bootstrap consistency by a linearization argument. First note that, for any $\theta \in \Theta$ and $\eta \in \cT_n$, by Taylor's expansion of the function $$r \mapsto {\mathrm P}\big[ \psi (\cdot; \theta_0 + r(\theta-\theta_0), \eta_0 + r(\eta- \eta_0) )\big],$$ one has \begin{align*} {\mathrm P}\big[ \psi (\cdot; \theta, \eta ) \big] &= J_0(\theta-\theta_0) + \partial_\eta {\mathrm P} \big[\psi(\cdot;\theta_0,\eta_0)\big][\eta- \eta_0] \\ & + \int_0^1 (1-r) \partial_r^2 {\mathrm P}\big[ \psi \big(\cdot; \theta_0 + r(\theta-\theta_0), \eta_0 + r(\eta- \eta_0) \big) \big] {\mathrm d} r \addtocounter{equation}{1}\tag{\theequation} \end{align*} as ${\mathrm P}[ \psi(\cdot; \theta_0 + r(\theta-\theta_0), \eta_0 + r(\eta- \eta_0) )]$ vanishes at $r=0$. Apply this equality to both $(\check{\theta}_{0,k}, \hat{\eta}_{0,k})$ and $(\check{\theta}_{0,k}^*, \hat{\eta}_{0,k})$, respectively, we note that by Neyman $\lambda_n$-orthogonality, \begin{align*} \big\lVert \partial_\eta {\mathrm P}[ \psi (\cdot; \theta_0, \eta_0)] [\hat{\eta}_{0,k} - \eta_0] \big\rVert \le \lambda_n. \end{align*} At the same time, by simple algebra we can write \begin{align*} & \sqrt{m} {\mathrm P}\big[ \psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k}) - \psi(\cdot; \check{\theta}_{0,k}, \hat{\eta}_{0,k}) \big] + {\mathbb{G}_{n,k}^{\ast}} \big[ \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] \\ &= - {\mathbb{G}_{n,k}^{\ast}} \big[ \psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] + {\mathbb{G}_{n,k}} \big[ \psi(\cdot; \check{\theta}_{0,k}, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] \\ & + {\mathbb{G}_{n,k}} \big[ \psi(\cdot; {\theta}_{0}, {\eta}_{0}) - \psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k}) \big] + \sqrt{m} {\mathbb{P}_{n,k}^{\ast}} \big[\psi(\cdot; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k}) \big] - \sqrt{m} {\mathbb{P}_{n,k}} \big[\psi(\cdot; \check{\theta}_{0,k}, \hat{\eta}_{0,k}) \big], \addtocounter{equation}{1}\tag{\theequation} \end{align*} and by the approximate $\epsilon_n$-solution condition, for the last two terms in (ref) one has \begin{align*} \lVert {\mathbb{P}_{n,k}} [\psi(X; \check{\theta}_{0,k}, \hat{\eta}_{0,k})] \rVert & \le \inf_{\theta \in \Theta} \lVert {\mathbb{P}_{n,k}} [\psi(X; \theta, \hat{\eta}_{0,k})] \rVert + \epsilon_n, \\ \lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(X; \check{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \rVert & \le \inf_{\theta \in \Theta} \lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(X; \theta, \hat{\eta}_{0,k})] \rVert + \epsilon_n. \end{align*} Combining (ref) and (ref) and arranging the terms, we then obtain, with probability $1-o(1)$, \begin{align*} \sqrt{m} \big\lVert J_0\big( \check{\theta}_{0,k}^* - \check{\theta}_{0,k} \big) + {\mathbb{G}_{n,k}^{\ast}} \big[ \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] \big\rVert \le T_4 + T_5 + 2T_6 + T_7 + 2T_8 + 2\sqrt{m}\lambda_n + 2\sqrt{m}\epsilon_n, \end{align*} where \begin{align*} T_4 &:= \inf_{\theta \in \Theta} \sqrt{m} \lVert {\mathbb{P}_{n,k}} [\psi(X; \theta, \hat{\eta}_{0,k})] \rVert,\\ T_5 &:= \inf_{\theta \in \Theta} \sqrt{m} \lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(X; \theta, \hat{\eta}_{0,k})] \rVert, \\ T_6 &:= \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \big\lVert {\mathbb{G}_{n,k}} \big[ \psi(\cdot; {\theta}, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] \big\rVert, \\ T_7 &:= \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \big\lVert {\mathbb{G}_{n,k}^{\ast}} \big[ \psi(\cdot; {\theta}, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] \big\rVert, \end{align*} and \begin{align*} T_8 &:= \sup_{\theta: \|\theta - \theta_0\| \le \tau_n} \sup_{\eta\in\mathcal{T}_n} \sqrt{m} \Big\lVert \int_0^1(1-r) \partial_r^2 {\mathrm P}\big[ \psi \big(\cdot; \theta_0 + r(\theta-\theta_0), \eta_0 + r(\eta- \eta_0) \big) \big] {\mathrm d} r \Big\rVert. \end{align*} In the following steps, we will establish bounds on the rates for each of these terms. In particular, in Step IV we will show that, with probability $1-o(1)$, \begin{align*} T_6 \le r_n' \log^{1/2}(1/r_n') + n^{1/q-1/2} \log n \addtocounter{equation}{1}\tag{\theequation} \end{align*} and \begin{align*} T_7 = O_{{\mathrm P}_{XW}}^o(r_n' \log^{1/2}(1/r_n') + a_n^{(q-2)/(3q-2)}\log(1/a_n)). \addtocounter{equation}{1}\tag{\theequation} \end{align*} Moreover, by Assumption (ref)(iii), we have \[ T_8 \le \sqrt{n}\lambda_n', \] and finally, we will show \begin{align*} T_4 &\le \sqrt{n} \lambda_n + T_8 + T_6, \addtocounter{equation}{1}\tag{\theequation} \\ T_5 &\le \sqrt{n} \lambda_n + T_8 + T_7 + T_6, \addtocounter{equation}{1}\tag{\theequation} \end{align*} with probability $1-o(1)$ in Step V. Combining (ref)-(ref), we then obtain \begin{align*} \sqrt{m} (\check{\theta}_{0,k}^* - \check{\theta}_{0,k}) = - J_0^{-1} {\mathbb{G}_{n,k}^{\ast}} \big[ \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] + O_{{\mathrm P}_{XW}}^o(\rho_n^*), \end{align*} where \begin{align*} \rho_n^* := n^{-1/2+1/q}\log n + r'_n \log^{1/2} (1/r'_n) + n^{1/2} (\lambda_n + \lambda'_n + \epsilon_n) + a_n^{(q-2)/(3q-2)}\log(1/a_n) = o(1). \end{align*} {\bf Step III.} Here we derive the rate for $T_3$. To prove \[ T_3 = o(\tau_n) ~~\text{ with ${\mathrm P}_{XW}^o$-probability $1-o(1)$,} \] it suffices to show \[ T_3 = o_{{\mathrm P}_{XW}}^o(\tau_n). \] We illustrate this by writing $T_3$ as the summation of a term of order $o(\tau_n)$ with high probability and a term of order $O_{{\mathrm P}_{XW}}(n^{-1/2})$. The order of $\tau_n$ then yields the claimed result. Notice that by Assumption (ref)(ii), for any measurable $f$, we can write \begin{align*} {\mathbb{G}_{n,k}^{\ast}}[f] &= \frac{1}{\sqrt{m}} \sum_{i\in I_k} (W_i-1)\delta_{X_i}[f] \\ &= \frac{1}{\sqrt{m}} \sum_{i\in I_k} (W_i-1)(\delta_{X_i} - {\mathrm P}_X)[f] + \frac{1}{\sqrt{m}} \sum_{i\in I_k} (W_i-1) {\mathrm P}_X[f] \\ &=: {\mathbb{G}_{n,k}^{\circ}}[f] + \Delta_{n,k}{\mathrm P}_X[f]; \end{align*} and correspondingly, we can write \begin{align*} T_3 \le m^{-1/2} \sup_{\theta \in \Theta} \big\lVert {\mathbb{G}_{n,k}^{\circ}}[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \big\rVert + m^{-1/2} \sup_{\theta \in \Theta} \big\lVert \Delta_{n,k}{\mathrm P}_X[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \big\rVert =: T_{3,c} + T_{3,r} .\addtocounter{equation}{1}\tag{\theequation} \end{align*} For the first term in (ref), we show \[ T_{3,c} = o(\tau_n)~~\text{ with ${\mathrm P}_{XW}^o$-probability $1-o(1)$.} \] Since by Markov inequality \begin{align*} {\mathrm P}_{XW}^o \big(T_{3,c} \le n^{-1/2} \log \log (n) \big) \ge 1 - \frac{\sqrt{n} {\mathrm E}_{XW}^o[T_{3,c}]}{ \log \log (n)}, \end{align*} and by definition of $\tau_n$ one has $T_{3,c} / \tau_n \le \log \log (n)/\log n = o(1)$ on the same event defined in the probability, it only suffices to show \begin{align*} \limsup_{n \to \infty} \sqrt{m} {\mathrm E}_{XW}^o [ T_{3,c} ] < \infty. \addtocounter{equation}{1}\tag{\theequation} \end{align*} Given an independent bootstrap weights copy $\bar{W}=(\bar W_i)_{i=1}^n$, since by Assumption (ref)(i)(ii) one has ${\mathrm E}_{\bar{W}}[\bar{W}_i]=1$ for all $i \in I_k$, we can then leverage the symmetrization argument to establish \begin{align*} \sqrt{m} {\mathrm E}_{XW}^o [ T_{3,c} ] &= {\mathrm E}_{XW}^o \Big[ \sup_{\theta \in \Theta} \big\lVert {\mathbb{G}_{n,k}^{\circ}}[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \big\rVert \Big] \\ &= {\mathrm E}_{XW}^o \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i\in I_k} (W_i-1)(\delta_{X_i} - {\mathrm P}_X)[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \Big\rVert \Big] \\ &= {\mathrm E}_{XW}^o \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i\in I_k} \big(W_i-{\mathrm E}_{\bar{W}}[\bar{W}_i]\big)(\delta_{X_i} - {\mathrm P}_X)[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \Big\rVert \Big] \\ & \le {\mathrm E}_{XW}^o {\mathrm E}_{\bar{W}}^o \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i\in I_k} \big(W_i-\bar{W}_i'\big)(\delta_{X_i} - {\mathrm P}_X)[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \Big\rVert \Big] \\ & \le {\mathrm E}_{XW}^o \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i\in I_k} W_i(\delta_{X_i} - {\mathrm P}_X)[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \Big\rVert \Big] \\ & + {\mathrm E}_{XW}^o {\mathrm E}_{\bar{W}}^o\Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i\in I_k} \bar{W}_i(\delta_{X_i} - {\mathrm P}_X)[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \Big\rVert \Big] \\ & = 2{\mathrm E}_{XW}^o \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i\in I_k} W_i(\delta_{X_i} - {\mathrm P}_X)[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \Big\rVert \Big] \\ &= 2{\mathrm E}_{XW}^o \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i\in I_k} W_i Z_i(\theta) \Big\rVert \Big], \end{align*} where we define \begin{align*} Z_i(\theta) := (\delta_{X_i} - {\mathrm P}_X)[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \end{align*} for $i \in I_k$. Re-indexing the samples in the fold, by Lemma (ref) we have, for any fixed $m_0 < \infty$, \begin{align*} \limsup_{n \to \infty} {\mathrm E}_{XW}^o \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i=1}^{m} W_i Z_i(\theta) \Big\rVert \Big] &\le m_0 \limsup_{n \to \infty} {\mathrm E}_{X}^o \Big[ \sup_{\theta \in \Theta} \big\lVert Z_1(\theta) \big\rVert \Big] \Big(\frac{{\mathrm E}_{W}\big[ \max_{1\le i \le m} W_i \big]}{\sqrt{m}}\Big) \\ & + \limsup_{n \to \infty} \Big\{ S_n \max_{m_0 \le i \le m} {\mathrm E}_{X}^o \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{i}} \sum_{j=m_0+1}^i Z_j(\theta) \Big\rVert \Big] \Big\}, \end{align*} where \begin{align*} S_n := \int_0^\infty \sqrt{{\mathrm P}_W(W_1\ge t)}{\mathrm d} t. \addtocounter{equation}{1}\tag{\theequation} \end{align*} To show (ref), it then only suffices to show \begin{align*} &\limsup_{n \to \infty} {\mathrm E}_{X}^o \Big[ \sup_{\theta \in \Theta} \big\lVert Z_1(\theta) \big\rVert \Big] < \infty, \addtocounter{equation}{1}\tag{\theequation} \\ &\limsup_{n \to \infty} a_n = \limsup_{n \to \infty}\Big(\frac{{\mathrm E}_{W}\big[ \max_{1\le i \le n} W_i \big]}{\sqrt{n}}\Big) = 0, \addtocounter{equation}{1}\tag{\theequation} \\ &\limsup_{n \to \infty} S_n < \infty, \addtocounter{equation}{1}\tag{\theequation} \\ &\limsup_{ n \to \infty} \max_{m_0 < i \le m} {\mathrm E}_{X} \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{i}} \sum_{j=m_0+1}^i Z_j(\theta) \Big\rVert \Big] < \infty \addtocounter{equation}{1}\tag{\theequation} . \end{align*} To establish (ref), by Assumption (ref)(ii), we can write \begin{align*} \limsup_{n \to \infty} {\mathrm E}_{X}^o \Big[ \sup_{\theta \in \Theta} \big\lVert Z_1(\theta) \big\rVert \Big] &= \limsup_{n \to \infty} {\mathrm E}_{X}^o \Big[ \sup_{\theta \in \Theta} \big\lVert (\delta_{X_1} - {\mathrm P}_X)[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \big\rVert \Big] \\ & \le \limsup_{n \to \infty} \Big\{ {\mathrm E}_{X}^o \Big[ \sup_{\theta \in \Theta} \big\lVert \psi(X_1; {\theta}, \hat{\eta}_{0,k}) \big\rVert \Big] + \sup_{\theta \in \Theta} \big\lVert {\mathrm P}_X [\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \big\rVert \Big\} \\ & \lesssim \limsup_{n \to \infty} 2 {\mathrm E}_{X} \big[F_{1,\hat{\eta}_{0,k}}(X_1)\big]\\ & \le 2c_1. \end{align*} Equation (ref) is a standard implication for exchangeable bootstrap weights under Assumptions (ref)(iii)(iv). To see this, either refer to our proof in Step VI or directly apply Lemma 4.7 in praestgaard1993exchangeably. Equation (ref) is a direct result of $S_n$ is uniformly bounded for all $n$ by Assumption (ref)(iii). To show (ref), we further re-index \[ s:=i-m_0 \text{ for } i>m_0, \] and therefore \begin{align*} \max_{m_0 < i \le m} {\mathrm E}_{X} \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{i}} \sum_{j=m_0+1}^i Z_j(\theta) \Big\rVert \Big] &= \max_{0 < s \le m-m_0} \sqrt{\frac{s}{s+m_0}} {\mathrm E}_{X} \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{s}} \sum_{j=1}^s Z_j(\theta) \Big\rVert \Big] \\ & \lesssim \max_{0 < s \le m-m_0} {\mathrm E}_{X} \Big[ \lVert \mathbb{G}_s \rVert_{\mathcal{F}_{1,\hat \eta_{0,k}}} \Big], \end{align*} where for each $s$, the re-indexed empirical process operator is defined as \begin{align*} \mathbb{G}_s := \frac{1}{\sqrt{s}} \sum_{j=1}^s (\delta_{X_j} - {\mathrm P}_X). \addtocounter{equation}{1}\tag{\theequation} \end{align*} Applying Lemma (ref) with sample size $s$ and noting Assumption (ref)(ii), we then obtain \begin{align*} \max_{m_0 < i \le m} {\mathrm E}_{X} \Big[ \sup_{\theta \in \Theta} \Big\lVert \frac{1}{\sqrt{i}} \sum_{j=m_0+1}^i Z_j(\theta) \Big\rVert \Big] \lesssim \max_{0 < s \le m-m_0} \big( 1+s^{-1/2+1/q} c_1 \big) \lesssim 1, \end{align*} which concludes (ref). For the second term in (ref), we aim to show \[ T_{3,r} = O_{{\mathrm P}_{XW}}(n^{-1/2}). \] To this end, note that \begin{align*} T_{3,r} = \sup_{\theta \in \Theta} \Big\lVert \frac{1}{m} \sum_{i\in I_k} (W_i-1) {\mathrm P}_X[\psi(\cdot; {\theta}, \hat{\eta}_{0,k})] \Big\rVert \lesssim \Big\lvert \frac{1}{m} \sum_{i\in I_k} (W_i-1) \Big\rvert \lVert F_{1,\hat \eta_{0,k}}\rVert_{{\mathrm P},q}. \end{align*} Then, from Assumption (ref)(ii), we only have to show $$\frac{1}{m}\sum_{i\in I_k} (W_i-1) = O_{{\mathrm P}}(n^{-1/2}).$$ Observe that the fact \begin{align*} 0 = \Var\Big[\sum_{i=1}^n W_i\Big] = n\Var[ W_i] + n(n-1) \Cov[W_1,W_2] \end{align*} yields \begin{align*} {\mathrm E}\Big[ \Big(\frac{1}{\sqrt{m} }\Delta_{n,k} \Big)^2 \Big] = \Var\Big[\frac{1}{m} \sum_{i\in I_k} (W_i-1)\Big] = \frac{n-m}{m(n-1)} \Var[W_1] = O(n^{-1}). \end{align*} To see the last equality, we prove that (iii)(iv) of Assumption (ref) implies the sequence $\{W_i\}_{i=1}^n$ is uniformly square-integrable: for any $\lambda$ sufficiently large such that $u^2 {\mathrm P}(W_{i} \ge u) \le \epsilon^2$ for large $n$ when $u \ge \lambda$, write \begin{align*} {\mathrm E}\big[W_{i}^2 \ind(W_{i} \ge \lambda)\big] &= 2 \int_0^\infty u {\mathrm P}(W_{i} \ge u \vee \lambda) {\mathrm d} u \\ &= \lambda^2 {\mathrm P}(W_{i} \ge \lambda) + 2 \int_{\lambda}^\infty u {\mathrm P}(W_{i} \ge u) {\mathrm d} u \\ &\le \epsilon^2 + 2 \Big[ \sup_{u \ge \lambda} u \sqrt{{\mathrm P}(W_{i} \ge u)} \Big]\int_{\lambda}^\infty \sqrt{{\mathrm P}(W_{i} \ge u)} {\mathrm d} u \\ &\le \epsilon^2 + 2C \epsilon. \end{align*} The arbitrariness of $\epsilon$ then yields uniform square-integrablility, that is, \begin{align*} \lim_{\lambda \to \infty} \limsup_{n \to \infty} {\mathrm E}\big[W_{i}^2 \ind(W_{i} \ge \lambda)\big] = 0, \addtocounter{equation}{1}\tag{\theequation} \end{align*} which implies the existence of $W_i$'s second moment and concludes $\Delta_{n,k} = O_{{\mathrm P}}(1)$. Together we have established the rate for $T_3$ as claimed. {\bf Step IV.} Here we first derive the bound (ref) for $T_6$. For any fixed $\eta \in \mathcal{T}_n$, define \begin{align*} \mathcal{F}_{2,\eta} := \big\{ \psi_j(\cdot; \theta, \eta) - \psi_j(\cdot; \theta_0, \eta_0): j \in \llbracket d_{\theta}\rrbracket, \|\theta - \theta_0\| \le \tau_n \big\}. \end{align*} It is then immediate that \[ T_6 \lesssim \lVert {\mathbb{G}_{n,k}} \rVert_{\mathcal{F}_{2,\hat\eta_{0,k}}}. \] Obviously, $F_{2,\eta} := F_{1,\eta} + F_{1,\eta_0}$ is an envelope function for $\mathcal{F}_{2,\eta}$ and satisfies the moment condition $$\lVert F_{2,\eta} \rVert_{{\mathrm P},q} \le \lVert F_{1,\eta} \rVert_{{\mathrm P},q} + \lVert F_{1,\eta_0} \rVert_{{\mathrm P},q} = 2c_1$$ by the triangle inequality. We can also find a constant $C'$ large enough so that $$\sup_{f \in \mathcal{F}_{2,\eta}} \| f \|_{{\mathrm P},2} \le C' r_n' \le \| F_{2,\eta} \|_{{\mathrm P},2}$$ by Assumption (ref)(i)(iii). Moreover, since $\mathcal{F}_{2,\eta} \subseteq \mathcal{F}_{1,\eta} - \mathcal{F}_{1,\eta_0}$, the proof of Theorem 3 in andrews1994empirical yields \begin{align*} & \sup_{\mathrm Q} \log N \big(\epsilon \|F_{2,\eta}\|_{{\mathrm Q},2}, \mathcal{F}_{2,\eta}, \| \cdot \|_{{\mathrm Q},2} \big) \le \sup_{\mathrm Q} \log N \big(\epsilon \|F_{2,\eta}\|_{{\mathrm Q},2}, \mathcal{F}_{1,\eta} - \mathcal{F}_{1,\eta_0}, \| \cdot \|_{{\mathrm Q},2} \big) \\ & \le \sup_{\mathrm Q} \log N \big((\epsilon/2) \|F_{1,\eta}\|_{{\mathrm Q},2}, \mathcal{F}_{1,\eta}, \| \cdot \|_{{\mathrm Q},2} \big) + \sup_{\mathrm Q} \log N \big((\epsilon/2) \|F_{1,\eta_0}\|_{{\mathrm Q},2}, \mathcal{F}_{1,\eta_0}, \| \cdot \|_{{\mathrm Q},2} \big), \end{align*} which is upper bounded by $2v \log(2a/\epsilon)$ for all $0<\epsilon \le 1$ by Assumption (ref)(ii). Therefore, applying Lemma (ref) conditioning on $\{X_i\}_{i \in I_k^c}$, we have, with ${\mathrm P}_{X}^o$-probability $1-o(1)$, \begin{align*} \sup_{f \in \mathcal{F}_{2,\eta}} \lvert {\mathbb{G}_{n,k}}(f) \rvert \lesssim r_n' \log^{1/2}(1/r_n') + n^{1/q-1/2} \log n. \end{align*} Replacing the $\eta$ with $\hat \eta_{0,k}$ concludes (ref). Next, we prove the probability bound (ref) for $T_7$. Similarly as in Step III, we can write \begin{align*} T_7 &\le \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \big\lVert {\mathbb{G}_{n,k}^{\circ}} \big[ \psi(\cdot; {\theta}, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] \big\rVert + \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \big\lVert {\mathrm P}\big[ \psi(\cdot; {\theta}, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_{0}, {\eta}_{0}) \big] \big\rVert \lvert \Delta_{n,k} \rvert \\ & =: T_{7,c} + T_{7,r}. \end{align*} For the second term above, note that Assumption (ref)(iii) and $\Delta_{n,k} = O_{{\mathrm P}}(1)$, which was shown in Step III, immediately gives \[ T_{7,r} = O_{{\mathrm P}_{XW}}(r_n'). \] It then suffices to show \begin{align*} {\mathrm E}_{XW}^o [ T_{7,c} ] \lesssim r_n' \log^{1/2}(1/r_n') + a_n^{(q-2)/(3q-2)}\log(1/a_n). \addtocounter{equation}{1}\tag{\theequation} \end{align*} With a similar symmetrization argument as in step III, for any fixed $\eta \in \cT_n$ we have \begin{align*} {\mathrm E}_{XW}^o \Big[ \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \big\lVert {\mathbb{G}_{n,k}^{\circ}}[\psi(\cdot; {\theta}, \eta) - \psi(\cdot; {\theta}_0, {\eta}_{0})] \big\rVert \Big] \le 2{\mathrm E}_{XW}^o \Big[ \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i=1}^{m} W_i Z_i'(\theta) \Big\rVert \Big], \addtocounter{equation}{1}\tag{\theequation} \end{align*} where we define \begin{align*} Z_i'(\theta) := \big( \delta_{X_i} - {\mathrm P}_X \big) \big[\psi(\cdot; {\theta}, \eta) - \psi(\cdot; {\theta}_0, {\eta}_{0}) \big]. \end{align*} Again, by the multiplier inequality, for any $m_0 < \infty$, we have \begin{align*} {\mathrm E}_{XW}^o \Big[ \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \Big\lVert \frac{1}{\sqrt{m}}\sum_{i=1}^{m} W_i Z_i'(\theta) \Big\rVert \Big] &\le m_0 {\mathrm E}_{X}^o \Big[ \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \big\lVert Z_1'(\theta) \big\rVert \Big] \Big(\frac{{\mathrm E}_{W}\big[ \max_{1\le i \le m} W_i \big]}{\sqrt{m}}\Big) \\ & + S_n \max_{m_0 \le i \le m} {\mathrm E}_{X}^o \Big[ \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \Big\lVert \frac{1}{\sqrt{i}} \sum_{j=m_0+1}^i Z_j'(\theta) \Big\rVert \Big] \addtocounter{equation}{1}\tag{\theequation} \end{align*} with $S_n$ identically defined as in (ref). Similarly as (ref), we have, uniformly for all $n$, \begin{align*} {\mathrm E}_{X}^o \Big[ \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \big\lVert Z_1'(\theta) \big\rVert \Big] \le 2 \sqrt{d_\theta} {\mathrm E}_{X}[F_{2,\eta}] \leq 4c_1\sqrt{d_\theta}. \end{align*} Recall $a_n = n^{-1/2}{\mathrm E}[\max_{1\le i \le n}W_i]$ converge to $0$ as $n$ goes to infinity, and by Assumption (ref)(iii) $S_n$ is uniformly bounded. Re-indexing $s:=i-m_0$ and invoking the definition of $\mathbb{G}_s$ in (ref), we bound the last term in (ref) by applying Lemma (ref) and a similar argument as we did for $T_6$: \begin{align*} & \max_{1 \le s \le m-m_0} \Big\{ \sqrt{\frac{s}{s+m_0}}{\mathrm E}_{X}^o \Big[ \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \Big\lVert \frac{1}{\sqrt{s}} \sum_{j=1}^s Z_j'(\theta) \Big\rVert \Big] \Big\} \lesssim \max_{1 \le s \le m-m_0} \Big\{ \sqrt{\frac{s}{s+m_0}}{\mathrm E}_{X}^o \Big[ \lVert \mathbb{G}_s \rVert_{\cF_{2,\eta}} \Big] \Big\} \\ & \lesssim \max_{1 \le s \le m-m_0} \sqrt{\frac{s}{s+m_0}} \big\{ r_n' \log^{1/2}(1/r_n') +s^{1/q-1/2} \log(s) \big\} \\ & \le r_n' \log^{1/2}(1/r_n') + \max_{1 \le s \le m-m_0} \sqrt{\frac{s}{s+m_0}} \big\{ s^{1/q-1/2} \log(s)\big\} \\ & \lesssim r_n' \log^{1/2}(1/r_n') + m_0^{1/q-1/2}\log(m_0). \addtocounter{equation}{1}\tag{\theequation} \end{align*} Combining (ref)-(ref), one has, for any $\eta \in \cT_n$, \begin{align*} {\mathrm E}_{XW}^o \Big[ \sup_{\theta: \lVert \theta - \theta_0 \rVert \leq \tau_n} \big\lVert {\mathbb{G}_{n,k}^{\circ}}[\psi(\cdot; {\theta}, \eta) - \psi(\cdot; {\theta}_0, {\eta}_{0})] \big\rVert \Big] \lesssim m_0 a_n + r_n' \log^{1/2}(1/r_n') + m_0^{1/q-1/2}\log(m_0). \end{align*} Setting $m_0(n) := \floor{a_n^{-1/(3/2-1/q)}}{}$ and letting $\eta = \hat \eta_{0,k}$ establishes (ref). {\bf Step V.} First, we derive the bound (ref) for $T_4$. Let $$\bar{\theta}_{0,k} := \theta_0 - J_0^{-1}{\mathbb{P}_{n,k}}[\psi(\cdot;\theta_0,\eta_0)].$$ Since the singular values of $J_0$ are bounded away from zero and, by Assumption (ref)(ii) with $\eta_0 \in \cT_n$ we have ${\mathrm E}[ \lVert {\mathbb{G}_{n,k}}[\psi(\cdot;\theta_0,\eta_0)]\rVert^2]\lesssim 1$, Markov's inequality yields \begin{align*} \lVert \bar{\theta}_{0,k} - \theta_0 \rVert \le m^{-1/2} \lVert J_0^{-1} \rVert \lVert {\mathbb{G}_{n,k}}[\psi(\cdot;\theta_0,\eta_0)] \rVert = O_{{\mathrm P}_X}^o(n^{-1/2}) = o_{{\mathrm P}_X}^o(n^{-1/2} \log n) = o_{{\mathrm P}_X}^o(\tau_n). \end{align*} Therefore, by definition of $\Theta$ in Assumption (ref)(i), one has \[ \bar{\theta}_{0,k} \in \Theta~~\text{ with ${\mathrm P}_{X}^o$-probability $1-o(1)$.} \] Then with the same probability, \begin{align*} T_4 = \inf_{\theta \in \Theta} \sqrt{m} \big\lVert {\mathbb{P}_{n,k}} [\psi(X; \theta, \hat{\eta}_{0,k})] \big\rVert \le \sqrt{m} \big\lVert {\mathbb{P}_{n,k}} [\psi(X; \bar{\theta}_{0,k}, \hat{\eta}_{0,k})] \big\rVert. \end{align*} The righthand side of the above inequality has the upper bound \begin{align*} \sqrt{m} \big\lVert {\mathbb{P}_{n,k}} [\psi(X; \bar{\theta}_{0,k}, \hat{\eta}_{0,k})] \big\rVert &\le \sqrt{m} \big\lVert {\mathbb{P}_{n,k}} [\psi(X; {\theta}_{0}, {\eta}_{0})] + {\mathrm P} [\psi(\cdot; \bar{\theta}_{0,k}, \hat{\eta}_{0,k})] \big\rVert \\ & + \big\lVert {\mathbb{G}_{n,k}} [ \psi(\cdot; \bar{\theta}_{0,k}, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_0, {\eta}_{0}) ] \big\rVert \\ & \le \sqrt{n} \lambda_n + T_8 + T_6, \end{align*} where the second inequality is given by (ref), the linearization of ${\mathrm P} [\psi(\cdot; \bar{\theta}_{0,k}, \hat{\eta}_{0,k})]$. Next, we derive the bound (ref) for $T_5$. Let \[ \bar{\theta}_{0,k}^* := \theta_0 - J_0^{-1} {\mathbb{P}_{n,k}^{\ast}}[\psi(\cdot;\theta_0,\eta_0)]. \] Notice that \begin{align*} \sqrt{m} {\mathbb{P}_{n,k}^{\ast}}[\psi(\cdot;\theta_0,\eta_0)] = {\mathbb{G}_{n,k}^{\ast}} [\psi(\cdot;\theta_0,\eta_0)] + {\mathbb{G}_{n,k}} [\psi(\cdot;\theta_0,\eta_0)] = O_{{\mathrm P}_{XW}}(1), \end{align*} as ${\mathbb{G}_{n,k}} [\psi(\cdot;\theta_0,\eta_0)] = O_{{\mathrm P}}(1)$ and we claim ${\mathbb{G}_{n,k}^{\ast}} [\psi(\cdot;\theta_0,\eta_0)] = O_{{\mathrm P}_{XW}}(1)$: for any $j \in \llbracket d_{\theta}\rrbracket$, the conditional second moment of the $j$-th component of ${\mathbb{G}_{n,k}^{\ast}} [\psi(\cdot;\theta_0,\eta_0)]$ can be written as \begin{align*} & {\mathrm E}_{W|X}\Big[\Big(\frac{1}{\sqrt{m}} \sum_{i \in I_k} (W_i-1) \psi_{j}(X_i;\theta_0, \eta_0)\Big)^2\Big] \\ & = \frac{1}{m} \Big\{ {\mathrm E}\big[(W_1-1)^2\big] \sum_{i \in I_k} \psi_{j}^2(X_i;\theta_0, \eta_0) + {\mathrm E}[(W_1-1)(W_2-1)] \sum_{i, \ell \in I_k, i \neq\ell} \psi_{j}(X_i;\theta_0, \eta_0) \psi_{j}(X_\ell;\theta_0, \eta_0) \Big\} \\ & = \frac{1}{m} {\mathrm E}\big[(W_1-1)^2\big] \Big\{ \sum_{i \in I_k} \psi_{j}^2(X_i;\theta_0, \eta_0) - \frac{1}{n-1} \sum_{i, \ell \in I_k, i \neq\ell} \psi_{j}(X_i;\theta_0, \eta_0) \psi_{j}(X_\ell;\theta_0, \eta_0) \Big\} \\ & \le \frac{1}{m} \Big(1+\frac{m-1}{n-1} \Big) {\mathrm E}\big[(W_1-1)^2\big] \Big( \sum_{i \in I_k} \psi_{j}^2(X_i;\theta_0, \eta_0) \Big)\\ & \le 2 {\mathrm E}\big[(W_1-1)^2\big] {\mathbb{P}_{n,k}} [\psi_{j}^2(\cdot;\theta_0, \eta_0)], \end{align*} where the second equality is a result of expanding $\Var[\sum_{i=1}^n(W_i-1)]=0$ and the first inequality uses the fact that for any $I_k$-indexed sequence of real numbers $\{b_i\}_{i \in I_k}$, one has the basic inequality $\lvert \sum_{i \neq \ell} b_i b_\ell \rvert \le (m-1) \sum_{i \in I_k}b_i^2$. Since ${\mathrm E}[(W_1-1)^2]$ is uniformly bounded by (ref), applying the law of total expectation and noting Assumption (ref)(ii) completes the proof for the claim. We then have \begin{align*} \lVert \bar{\theta}_{0,k}^* - \theta_0 \rVert \le m^{-1/2} \lVert J_0^{-1} \rVert \lVert m^{1/2} {\mathbb{P}_{n,k}^{\ast}}[\psi(\cdot;\theta_0,\eta_0)] \rVert = O_{{\mathrm P}_{XW}}(n^{-1/2}) = o_{{\mathrm P}_{XW}}(n^{-1/2} \log n) = o_{{\mathrm P}_{XW}}(\tau_n), \end{align*} and we have $\bar{\theta}_{0,k}^* \in \Theta$ with ${\mathrm P}_{XW}^o$-probability $1-o(1)$. With the same probability, \begin{align*} T_5 = \inf_{\theta \in \Theta} \sqrt{m} \lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(X; \theta, \hat{\eta}_{0,k})] \rVert \le \sqrt{m} \big\lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(X; \bar{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \big\rVert. \end{align*} By the triangle inequality, the righthand side is upper bounded by \begin{align*} \sqrt{m} \big\lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(X; \bar{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \big\rVert &\le \sqrt{m} \big\lVert {\mathbb{P}_{n,k}^{\ast}} [\psi(X; {\theta}_{0}, {\eta}_{0})] + {\mathrm P} [\psi(\cdot; \bar{\theta}_{0,k}^*, \hat{\eta}_{0,k})] \big\rVert \\ & + \big\lVert {\mathbb{G}_{n,k}^{\ast}} [ \psi(\cdot; \bar{\theta}_{0,k}^*, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_0, {\eta}_{0}) ] \big\rVert \\ & + \big\lVert {\mathbb{G}_{n,k}} [ \psi(\cdot; \bar{\theta}_{0,k}^*, \hat{\eta}_{0,k}) - \psi(\cdot; {\theta}_0, {\eta}_{0}) ] \big\rVert \\ & \le \sqrt{n} \lambda_n + T_8 + T_7 + T_6, \end{align*} as we can linearize ${\mathrm P} [\psi(\cdot; \bar{\theta}_{0,k}^*, \hat{\eta}_{0,k})]$ in the same way. {\bf Step VI.} In this part we will show that (ref) indeed implies (ref). Employing Cramér–Wold device, it is equivalent to showing that, for any $z \in \bR^{d_\theta}$, \begin{align*} \sup_{t \in \bR} \Big\lvert {\mathrm P}_{W|X}\big(c^{-1} \bG_{n}^* [z^\top \bar \psi_0 (\cdot)] \le t \big) - {\mathrm P}\big(N(0, z^\top \Sigma^2 z) \le t\big) \Big\rvert &= o_{{\mathrm P}_X}^o(1). \end{align*} To this end, we adapt the proof of mason1992rank to our setup. Define \begin{align*} H_n := \Big[\frac{1}{n}\sum_{i=1}^n(W_i-1)^2 \frac{1}{n}\sum_{i=1}^n \big(z^\top \bar \psi_0(X_i) - \bP_n [ z^\top \bar{\psi}_0(\cdot)] \big)^{2} \Big]^{-1/2} \bG_{n}^* [z^\top \bar \psi_0 (\cdot)]. \end{align*} Since one has, by Assumption (ref)(v) and the weak law of large numbers , \begin{align*} \frac{1}{n} \sum_{i=1}^n (W_i-1)^2 \stackrel{\sf p}{\longrightarrow} c^2 and \frac{1}{n} \sum_{i=1}^n \big(z^\top \bar \psi_0(X_i) - \bP_n [ z^\top \bar{\psi}_0(\cdot)] \big)^{2} = z^\top \Sigma^2 z + o_{{\mathrm P}}(1). \addtocounter{equation}{1}\tag{\theequation} \end{align*} To prove (ref), it then suffices to show \begin{align*} \sup_{t \in \bR} \Big\lvert {\mathrm P}_{W|X}\big(H_n \le t \big) - {\mathrm P}\big(N(0,1) \le t\big) \Big\rvert = o_{{\mathrm P}_X}(1). \addtocounter{equation}{1}\tag{\theequation} \end{align*} Let $R=(R_i)_{i=1}^n$ be a random permutation of $\llbracket n\rrbracket$, uniformly distributed over all $n!$ permutations and independent of $(X,W)$. Writing $\{W_{R_i}\}_{i=1}^n$ to be the permuted weights of $\{W_i\}_{i=1}^n$ according to $R$, we denote the permuted version of $H_n$ as \begin{align*} H_n^R &:= \Big[\sum_{i=1}^n(W_i-1)^2\sum_{i=1}^n \big(z^\top \bar \psi_0(X_i) - \bP_n [ z^\top \bar{\psi}_0(\cdot)] \big)^2 \Big]^{-1/2} \sqrt{n} \sum_{i=1}^n (W_{R_i}-1)z^\top \bar \psi_0(X_i). \end{align*} Since $W$ is exchangeable, $H_n$ and $H_n^R$ have the same distribution and it remains to show \begin{align*} \sup_{t \in \bR} \Big\lvert {\mathrm P}_{RW|X}\big(H_n^R \le t \big) - {\mathrm P}\big(N(0, 1) \le t\big) \Big\rvert = o_{{\mathrm P}_X}(1), \addtocounter{equation}{1}\tag{\theequation} \end{align*} or equivalently, any subsequence $\{n_k\}_{k \ge 0}$ contains a further subsequence $\{n_{k(\ell)}\}_{\ell \ge 0}$ such that \begin{align*} \sup_{t \in \bR} \Big\lvert {\mathrm P}_{RW|X}\big(H_{n_{k(\ell)}}^R \le t \big) - {\mathrm P}\big(N(0, 1) \le t\big) \Big\rvert \stackrel{\sf a.s.}{\longrightarrow} 0 \end{align*} almost surely as $\ell \to \infty$. For any $i,j \in \llbracket n\rrbracket$, we define \begin{align*} U_i^2 &:= \frac{(z^\top \bar \psi_0(X_i) - {\mathbb{P}_{n}} [z^\top \bar \psi_0(\cdot)] )^2}{\sum_{i=1}^n(z^\top \bar \psi_0(X_i) - {\mathbb{P}_{n}} [z^\top \bar \psi_0(\cdot)])^2}, \\ V_i^2 &:= \frac{(W_{i}-1)^2}{\sum_{i=1}^n(W_{i}-1)^2}, \\ \delta_{ij}^2 &:= \frac{n (z^\top \bar \psi_0(X_i) - {\mathbb{P}_{n}} [z^\top \bar \psi_0(\cdot)] )^2 (W_{j}-1)^2}{\sum_{i=1}^n(z^\top \bar \psi_0(X_i) - {\mathbb{P}_{n}} [z^\top \bar \psi_0(\cdot)] )^2 \sum_{j=1}^n(W_{j}-1)^2} =n U_i^2 V_j^2, \end{align*} and for any $\delta>0$, \begin{align*} d_n(\delta) := \frac{1}{n} \sum_{i=1}^n \sum_{j=1}^n \delta_{ij}^2 \ind\big(\delta_{ij}^2 > \delta \big). \end{align*} We then claim \begin{align*} \max_{i \in \llbracket n\rrbracket} U_i^2 = o_{{\mathrm P}_X}(1) and \max_{i \in \llbracket n\rrbracket} V_i^2 = o_{{\mathrm P}_W}(1). \addtocounter{equation}{1}\tag{\theequation} \end{align*} The first part in (ref) follows from ${\mathrm E}[\lvert z^\top \bar \psi_0(X_i)\rvert^q] < \infty$, which is implied by Assumption (ref)(iii) and Assumption (ref)(ii). We can see this claim by observing that, for any $\epsilon>0$, \begin{align*} {\mathrm P}\Big(\max_{i \in \llbracket n\rrbracket}\lvert z^\top \bar \psi_0(X_i) \rvert >\epsilon\sqrt n\Big) \le n{\mathrm P}\big(\lvert z^\top \bar \psi_0(X_1)\rvert>\epsilon\sqrt n\big) \le \epsilon^{-q} n^{1-q/2} {\mathrm E}\big[|z^\top \bar \psi_0(X_1)|^q\big], \end{align*} and the upper bound holds as \begin{align*} U_i^2= \frac{(z^\top\bar \psi_0(X_i) - {\mathbb{P}_{n}} [z^\top \bar \psi_0(\cdot)])^2}{n {\mathbb{P}_{n}} (z^\top \bar \psi_0(\cdot) - {\mathbb{P}_{n}} [z^\top \bar \psi_0(\cdot)])^2} \le \frac{\max_{i \in \llbracket n\rrbracket} 2 \lvert z^\top\bar \psi_0(X_i) \rvert^2}{n {\mathbb{P}_{n}} (z^\top \bar \psi_0(\cdot) - {\mathbb{P}_{n}} [z^\top \bar \psi_0(\cdot)])^2} \addtocounter{equation}{1}\tag{\theequation} \end{align*} with \[ {\mathbb{P}_{n}} (z^\top \bar \psi_0(\cdot) - {\mathbb{P}_{n}} [z^\top \bar \psi_0(\cdot)])^2 = z^\top\Sigma^2 z + o_{{\mathrm P}}(1). \] For establishing the second part in (ref), we first note the uniform square-integrablility of $W$ in (ref) and by Markov inequality, for any $\epsilon>0$, we can choose $n$ sufficiently large such that \begin{align*} \limsup_{n \to \infty} u^{2} {\mathrm P}\big( W_{i} >u \big) \le \limsup_{n \to \infty} {\mathrm E} \big[ W_{i}^2 \ind(W_{i} >u ) \big] \le \epsilon^2 \end{align*} for all $u \ge \epsilon \sqrt{m}$; by the first part of Proposition (ref), it then implies \begin{align*} \frac{1}{\sqrt{n}} {\mathrm E}\Big[ \max_{i \in \llbracket n\rrbracket} W_{i} \Big] \le \epsilon + \sqrt{n} \Big[ \sup_{ u \ge \epsilon \sqrt{n}} u^{2} {\mathrm P}(W_{i} >u ) \Big] \int_{\epsilon \sqrt{n}}^{\infty} u^{-2} {\mathrm d} u \le 2\epsilon. \end{align*} Following an analogous argument as (ref) above and applying Markov's inequality, one can show that the second part of (ref) is achieved. Next, we shall show that for all $\delta>0$, \[ d_n(\delta) = o_{{\mathrm P}_{XW}}(1). \] For any $\epsilon>0$, let \begin{align*} A_n := \Big\{\max_{i \in \llbracket n\rrbracket} U_i^2 < \epsilon \Big\} and B_n := \Big\{ \frac{1}{n}\sum_{i \in \llbracket n\rrbracket} (W_{i}-1)^2 > \frac{c^2}{2} \Big\} \end{align*} be the events with ${\mathrm P}_{XW}$-probability approaching one. It then follows that \begin{align*} \{\delta_{ij}^2 > \delta \} \cap A_n \cap B_n \subseteq \{n V_j^2 > \epsilon^{-1} \delta\} \cap B_n \subseteq \{(W_{j}-1)^2 > \epsilon^{-1} \delta c^2/2\}, \end{align*} which gives \begin{align*} \frac{1}{n} \sum_{i=1}^n \sum_{j=1}^n \delta_{ij}^2 \ind\big(\delta_{ij}^2 > \delta \big) \ind_{A_n\cap B_n} & \le \sum_{i=1}^n \sum_{j=1}^n U_i^2 V_j^2 \ind\big((W_{j}-1)^2 > \epsilon^{-1} \delta c^2 /2\big) \ind_{B_n} \\ &= \frac{{n}^{-1} \sum_{j=1}^n (W_{j}-1)^2 \ind\big((W_{j}-1)^2 > \epsilon^{-1} \delta c^2 /2\big) \ind_{B_n}}{{n}^{-1} \sum_{j=1}^n (W_{j}-1)^2 } \\ & \le \frac{2}{c^2 n} \sum_{j=1}^n (W_{j}-1)^2 \ind\big((W_{j}-1)^2 > \epsilon^{-1} \delta c^2 /2\big). \end{align*} The above further implies that, for any $\epsilon'>0$, one has by Markov inequality \begin{align*} & {\mathrm P}_{XW} \Big( \frac{1}{n} \sum_{i=1}^n \sum_{j=1}^n \delta_{ij}^2 \ind\big(\delta_{ij}^2 > \delta \big) > \epsilon' \Big) \\ & \le {\mathrm P}(A_n^c) + {\mathrm P}(B_n^c) + {\mathrm P} \Big( \frac{2}{c^2 n} \sum_{j=1}^n (W_{j}-1)^2 \ind\big((W_{j}-1)^2 > \epsilon^{-1} \delta c^2/2\big) > \epsilon' \Big) \\ & \le {\mathrm P}(A_n^c) + {\mathrm P}(B_n^c) + \frac{2}{c^2 \epsilon'} {\mathrm E} \big[ (W_{j}-1)^2 \ind\big((W_{j}-1)^2 > \epsilon^{-1} \delta c^2 /2\big) \big]. \end{align*} Due to the arbitrariness of $\epsilon$ and noting that (ref) gives $W_{j}-1$ is uniformly square-integrable, we conclude \begin{align*} \frac{1}{n} \sum_{i=1}^n \sum_{j=1}^n \delta_{ij}^2 \ind\big(\delta_{ij}^2 > \delta \big) = o_{{\mathrm P}_{XW}}(1). \addtocounter{equation}{1}\tag{\theequation} \end{align*} Now, combining (ref) and (ref), for any subsequence $\{n_k\}_{k \ge 1}$, we then have a further subsequence $\{n_{k(\ell)}\}_{\ell \ge 1}$ such that for all $\delta>0$, \begin{align*} \max_{1\le i \le n_{k(\ell)}} U_i^2 \stackrel{\sf a.s.}{\longrightarrow} 0 , \max_{1\le i \le n_{k(\ell)}} V_i^2 \stackrel{\sf a.s.}{\longrightarrow} 0 and d_{n_{k(\ell)}}(\delta) \stackrel{\sf a.s.}{\longrightarrow} 0, \addtocounter{equation}{1}\tag{\theequation} \end{align*} where the last claim follows from a diagonal argument and noting that $d_n(\delta)$ is decreasing in $\delta$. Conditioning on $(X,W)$ and applying Theorem 4.1 in hajek1961some, the linear rank statistics satisfies \begin{align*} \sup_{t \in \bR} \Big\lvert {\mathrm P}_{R| XW}\big(H_{n_{k(\ell)}}^{R} \le t \big) - {\mathrm P}\big(N(0,1) \le t\big) \Big\rvert \stackrel{\sf a.s.}{\longrightarrow} 0 \end{align*} as $\ell \to \infty$. A conditional bounded convergence theorem then yields \begin{align*} \sup_{t \in \bR} \Big\lvert {\mathrm P}_{RW|X}\big(H_{n_{k(\ell)}}^{R} \le t \big) - {\mathrm P}\big(N(0, 1) \le t\big) \Big\rvert \stackrel{\sf a.s.}{\longrightarrow} 0 \end{align*} as $\ell \to \infty$, which completes the proof of (ref). By Lemma (ref), this yields the row-wise version of (ref) along the arbitrary sequence $\{{\mathrm Q}_n\}_{n\ge1}$. Since the latter sequence was arbitrary, Lemma (ref) upgrades this row-wise convergence to the uniform statement in (ref).

proof[Proof of Proposition (ref)] We prove the proposition in four parts. {\bf Part I. Proving (ref).} For the lower bound, note $\max_{i \in \llbracket n\rrbracket}W_i \ge \frac{1}{n}\sum_{i=1}^nW_i = 1$. For the upper bound, write \begin{align*} \frac{1}{\sqrt{n}} {\mathrm E}\Big[ \max_{i \in \llbracket n\rrbracket} W_{i} \Big] & = \frac{1}{\sqrt{n}} \int_{0}^{x} {\mathrm P}\Big( \max_{i \in\llbracket n\rrbracket} W_{i} >t \Big) {\mathrm d} t + \frac{1}{\sqrt{n}} \int_{x}^{\infty} \sum_{i \in \llbracket n\rrbracket} {\mathrm P}( W_{i} >t ) {\mathrm d} t \\ & \le \frac{x}{\sqrt{n}} + \sqrt{n} \Big[ \sup_{ t \ge x} t^{2} {\mathrm P}(W_{i} >t ) \Big] \int_{x}^{\infty} t^{-2} {\mathrm d} t, \end{align*} and the corresponding rates in (a)-(c) can be readily obtained. {\bf Part II. Proving Claim (i).} Now we give the rate for Efron's bootstrap. First, we derive the upper bound for ${\mathrm E}[\max_{i\in\llbracket n\rrbracket}W_i]$. For any $\epsilon>0$ with $L_n := \log n / \log\log n$, define \[ m_+ := (1+\epsilon)L_n. \] We write \begin{align*} {\mathrm E}\Big[\max_{i \in \llbracket n\rrbracket}W_i\Big] = \sum_{m < m_+}{\mathrm P} \Big(\max_{i \in \llbracket n\rrbracket}W_i \geq m\Big) + \sum_{m \ge m_+} {\mathrm P} \Big(\max_{i \in \llbracket n\rrbracket}W_i \geq m\Big) \leq m_+ + \sum_{m \ge m_+} {\mathrm P} \Big(\max_{i \in \llbracket n\rrbracket}W_i \geq m\Big). \end{align*} By the arbitrariness of $\epsilon$, it only suffices to show the second term goes to $0$. By Chernoff bound, for any $m>0$ and setting $\lambda = \log(m)$ gives \begin{align*} {\mathrm P}(W_1 \ge m) \le \exp(-\lambda m)\Big(1-\frac{1}{n}+\frac{1}{n}\exp(\lambda)\Big)^n \le \exp\big\{\exp(\lambda)-1-\lambda m \big\} = e^{-1}\Big(\frac{e}{m}\Big)^m, \end{align*} which implies \begin{align*} {\mathrm P} \Big(\max_{i \in \llbracket n\rrbracket}W_i \geq m\Big) \le ne^{-1}\Big(\frac{e}{m}\Big)^m := B(m). \end{align*} Notice for large $m$, \begin{align*} \frac{B(m+1)}{B(m)} = \frac{e}{m+1}\Big(\frac{m}{m+1}\Big)^m \asymp \frac{1}{m}; \end{align*} then \begin{align*} \sum_{m \ge m_+} {\mathrm P} \Big(\max_{i \in \llbracket n\rrbracket}W_i \geq m\Big) \lesssim B(m_+) \sum_{j\geq0} \Big( \frac{1}{m_+} \Big)^j \lesssim B(m_+) \end{align*} and the desired result follows immediately from \[ \log(B(m_+)) = -\epsilon \log(n) + o(\log(n)) \to -\infty \] as $n$ goes to infinity. Next we derive the lower bound. With any $\epsilon>0$ and the same $L_n$ we write \[ m_-:= \floor{(1-\epsilon)L_n}{}. \] By Markov's inequality, \begin{align*} {\mathrm E}\Big[\max_{i \in \llbracket n\rrbracket}W_i\Big] \ge m_- {\mathrm P} \Big(\max_{i \in \llbracket n\rrbracket}W_i \geq m_-\Big) \ge m_- {\mathrm P}(N_{m_-}>0) \ge m_- \frac{{\mathrm E}[N_{m_-}]^2}{{\mathrm E}[N_{m_-}^2]}, \end{align*} where for any $m>0$, we define $N_m$ as the number of weights such that the weight takes the value $m$, that is, $ N_m := \sum_{i=1}^n \ind(W_i = m)$. Moreover, the last inequality above is obtained by applying the Paley–Zygmund inequality. Then by the arbitrariness of $\epsilon$, it only suffices to show \[ {\mathrm E}[N_{m_-}]^2/{\mathrm E}[N_{m_-}^2]= 1+o(1). \] For the first moment, we have when $m = o(n^{1/2})$, \begin{align*} {\mathrm E}[N_m] &= n{\mathrm P}(W_1=m) = n \binom{n}{m}n^{-m}\Big(1-\frac{1}{n} \Big)^{n-m} \\ & = n (1-o(1))\frac{1}{m!}(e^{-1}+o(1))(1+o(1))= n \frac{e^{-1}+o(1)}{m!}, \end{align*} where the first equality on the second line comes from \begin{align*} \frac{1}{m!} \ge \binom{n}{m}n^{-m} \ge \frac{1}{m!} \Big(1- \frac{m(m-1)}{2n}\Big) = \frac{1-o(1)}{m!} \end{align*} and \begin{align*} \log\Big\{\Big(1-\frac{1}{n}\Big)^{-m}\Big\} = -\frac{m}{n}+O\Big(\frac{1}{n^2}\Big) \to 0. \end{align*} Then by Stirling's formula, \begin{align*} \log({\mathrm E}[N_{m_-}]) = \log(n) - m_-\log(m_-) -m_- +O(\log(m_-)) = \epsilon\log(n) + o(\log(n)), \end{align*} which implies ${\mathrm E}[N_{m_-}] \to \infty$. For the second moment, for any $m=o(n^{1/2})$, with a similar analysis we have \begin{align*} {\mathrm E}[N_{m}^2] &= {\mathrm E}[N_{m}] + n(n-1){\mathrm P}(W_1=W_2=m) \\ &= {\mathrm E}[N_{m}] + n(n-1) \frac{n!}{m!m!(n-2m)!}n^{-2m}\Big(1-\frac{2}{n}\Big)^{n-2m}\\ &= {\mathrm E}[N_{m}] + n(n-1) \frac{e^{-2}+o(1)}{(m!)^2}. \end{align*} Comparing this with the order for ${\mathrm E}[N_{m_-}]$, we derive the desired result. {\bf Part III. Proving Claim (ii).} We first prove the rate for normalized multiplier bootstrap with exponential-tail weights. Denoting ${\mathrm E}[Y_1]$ as $\mu$, we observe that for any $s>0$, by Chernoff bound, one has \begin{align*} {\mathrm E}\Big[\max_{i \in \llbracket n\rrbracket}W_i\Big] \le \frac{2}{\mu} {\mathrm E}\Big[\max_{i \in \llbracket n\rrbracket}Y_i\Big] + n{\mathrm P}\Big(\frac{1}{n}\sum_{i=1}^n Y_i < \frac{\mu}{2}\Big) \le \frac{2}{\mu} {\mathrm E}\Big[\max_{i \in \llbracket n\rrbracket}Y_i\Big] + n e^{sn\mu/2}\big({\mathrm E}[e^{-sY_1}]\big)^n. \end{align*} Further note that, by sub-exponentiality, the second term above is exponentially small as \[ {\mathrm E}[e^{-sY_1}]<e^{-s\mu/2}~~ \text{ for sufficiently small $s$}. \] Additionally, the first term satisfies \begin{align*} {\mathrm E}\Big[\max_{i \in \llbracket n\rrbracket}Y_i\Big] \le x + n \int_{x}^\infty {\mathrm P}(Y_1 \ge t){\mathrm d} t \lesssim \log(n), \end{align*} which conclude the case for the normalized multiplier bootstrap. For the double bootstrap, note that, for any $\lambda>0$, \begin{align*} {\mathrm E}[e^{\lambda W_1} | M_1] = \Big(1+\frac{M_1}{n}(e^\lambda-1)\Big)^n \le \exp\{(e^\lambda-1)M_1\}, \end{align*} which implies \begin{align*} {\mathrm E}[e^{\lambda W_1}] \le {\mathrm E}[\exp\{(e^\lambda-1)M_1\}] \le \Big(1+\frac{e^{e^\lambda-1}-1}{n} \Big)^n \le \exp\{e^{e^\lambda-1}-1\}. \end{align*} By Chernoff bound, we know $W_1$ has a uniform exponential tail and the first part of the proof gives the claimed result. {\bf Part IV. Proving Claim (iii).} We prove the existence by considering the grouped delete-$h$ jackknife, which has been discussed in Example 3.6 of praestgaard1993exchangeably and satisfies Assumption (ref). In this bootstrap scheme, we set \[ \cS_{n,h_n}:=\Big\{S\subseteq \llbracket n\rrbracket: \lvert S\rvert=h_n\Big\} \] and let $S_n$ be drawn uniformly from $\cS_{n,h_n}$. For each $i \in \llbracket n\rrbracket$, the bootstrap weight component is defined as \[ W_i := \frac{n}{n-h_n}\ind(i\notin S_n). \] Note for this bootstrap procedure, if $h_n/n \to \gamma \in (0,1)$, then $c^2= \lim_{n\to \infty} \frac{h_n}{n-h_n} =\frac{\gamma}{1-\gamma}$ and thus \begin{align*} a_n = \frac{1}{\sqrt{n}}\frac{n}{n-h_n} \asymp n^{-1/2}. \end{align*} This completes the proof.

lemma[Multiplier inequality, wellner1996bootstrapping] Let $W = (W_1, \ldots, W_n)^{\top}$ be a non-negative exchangeable random vector on $(\Omega_W, \cA_W, {\mathrm P}_W)$ and, for every $n$, \begin{align*} S_n := \int_0^{\infty} \sqrt{{\mathrm P}_W \left(W_1 \ge t\right)} {\mathrm d} t < \infty. \end{align*} Let $\{Z_i\}_{i=1}^n$'s be i.i.d. $\ell^{\infty}(\mathcal{F}_n)$-valued random elements in $(\Omega_X^{\infty}, \cA_X^{\infty}, {\mathrm P}_X^{\infty})$. Write $\lVert \cdot \rVert_n = \sup_{f \in \mathcal{F}_n} \lvert Z_i(f) \rvert$ and assume $\{Z_i\}_{i=1}^n$'s are independent with $W$. Then for any $n_0, n$ such that $1 \le n_0 < n < \infty$, we have the following inequality: \begin{align*} {\mathrm E}_{XW}^o \Big[ \frac{1}{\sqrt{n}} \Big\lVert \sum_{i=1}^n W_iZ_i \Big\rVert_n \Big] &\le n_0 {\mathrm E}_{XW}^o \big[ \big\lVert Z_1 \big\rVert_n \big] \frac{{\mathrm E}_{W}\big[ \max_{1\le i \le n} W_i \big]}{\sqrt{n}} \\ & + S_n \max_{n_0 < i \le n} {\mathrm E}_{X}^o \Big[ \frac{1}{\sqrt{i}} \Big\lVert \sum_{j=n_0+1}^i Z_j \Big\rVert_n \Big]. \end{align*}

lemma[Maximal Inequality, chernozhukov2014gaussian] Let $\{X_i\}_{i=1}^n$ be i.i.d. random elements in $(\Omega_X, \cA_X, {\mathrm P}_X )$. Suppose that $F\geq \sup_{f \in \mathcal{F}}\lvert f\rvert$ is a measurable envelope for a suitably measurable function class $\mathcal{F}$ with $\lVert F\rVert_{{\mathrm P},q} < \infty$ for some $q \geq 2$. Let $M := \max_{1\le i\le n} F(X_i)$ and $\sigma^{2} > 0$ be any positive constant such that $\sup_{f \in \mathcal{F}} \lVert f \rVert_{{\mathrm P},2}^{2} \leq \sigma^{2} \leq \lVert F \rVert_{{\mathrm P},2}^{2}$. Suppose that there exist some constants $a \ge e$ and $v \ge 1$ such that \begin{align*} \log \sup_{{\mathrm Q}} N\big(\epsilon \| F \|_{{\mathrm Q},2}, \mathcal{F}, \| \cdot \|_{{\mathrm Q},2}\big) \leq v \log (a/\epsilon), \ 0 < \epsilon \leq 1. \end{align*} Then \begin{align*} {\mathrm E}_{\mathrm P} \big[ \lVert \bG_{n} \rVert_{\mathcal{F}} \big] \leq K \bigg\{ \sqrt{v\sigma^{2} \log \Big( \frac{a \lVert F \rVert_{{\mathrm P},2}}{\sigma} \Big) } + \frac{v\lVert M \rVert_{{\mathrm P}, 2}}{\sqrt{n}} \log \Big( \frac{a \lVert F \rVert_{{\mathrm P},2}}{\sigma} \Big) \bigg\}, \end{align*} where $K$ is an absolute constant. Moreover, for every $t \geq 1$, with probability $> 1-t^{-q/2}$, \begin{align*} \lVert \bG_{n} \rVert_{\mathcal{F}} \leq (1+\alpha) {\mathrm E}_{\mathrm P} \big[ \lVert \bG_{n} \rVert_{\mathcal{F}} \big] + K(q) \Big\{ (\sigma + n^{-1/2} \lVert M \rVert_{{\mathrm P},q}) \sqrt{t} + \alpha^{-1} n^{-1/2} \lVert M \rVert_{{\mathrm P},2}t \Big\}, \ \forall \alpha > 0, \end{align*} where $K(q) > 0$ is a constant depending only on $q$. In particular, setting $a \geq n$ and $t = \log n$, with probability $> 1- c(\log n)^{-1}$, \begin{align*} \lVert \bG_{n} \rVert_{\mathcal{F}} \leq K(q,c) \bigg\{ \sigma \sqrt{v \log \Big ( \frac{a \lVert F \rVert_{{\mathrm P},2}}{\sigma} \Big ) } + \frac{v \| M \|_{{\mathrm P},q} } {\sqrt{n}}\log \Big( \frac{a \lVert F \rVert_{{\mathrm P},2}}{\sigma} \Big) \bigg\}, \end{align*} where $ \lVert M \rVert_{{\mathrm P},q} \leq n^{1/q} \lVert F\rVert_{{\mathrm P},q}$ and $K(q,c) > 0$ is a constant depending only on $q$ and $c$.

lemma[Sequence characterization of uniform probabilistic statements] Let $\{\cP_n\}_{n\ge1}$ be arbitrary classes of probability measures, let $\{b_n\}_{n\ge1}$ be a sequence of positive deterministic numbers, and for each $n$ and each ${\mathrm P} \in \cP_n$, let $Z_{n,{\mathrm P}}$ be a random element defined on a probability space with probability law denoted by ${\mathrm P}_{{\mathrm P}}$. Then the following conditions are equivalent: \begin{enumerate}[itemsep=-.5ex, label=(\alph*)] • for every $\epsilon>0$, $\sup_{{\mathrm P}\in\cP_n}{\mathrm P}_{{\mathrm P}}\big(\lVert Z_{n,{\mathrm P}} \rVert>\epsilon b_n\big)\to 0$; • for every $\epsilon>0$ and every sequence $\{{\mathrm P}_n\}_{n\ge1}$ such that ${\mathrm P}_n\in\cP_n$ for all $n$, ${\mathrm P}_{{\mathrm P}_n}\big(\lVert Z_{n,{\mathrm P}_n} \rVert>\epsilon b_n\big)\to 0.$ \end{enumerate} Moreover, the following conditions are equivalent: \begin{enumerate}[itemsep=-.5ex, label=(\alph*)] • for every deterministic sequence $\ell_n\to\infty$, $\sup_{{\mathrm P}\in\cP_n}{\mathrm P}_{{\mathrm P}}\big(\lVert Z_{n,{\mathrm P}} \rVert>\ell_n b_n\big)\to 0$; • for every deterministic sequence $\ell_n\to\infty$ and every sequence $\{{\mathrm P}_n\}_{n\ge1}$ such that ${\mathrm P}_n\in\cP_n$ for all $n$, ${\mathrm P}_{{\mathrm P}_n}\big(\lVert Z_{n,{\mathrm P}_n}\rVert>\ell_n b_n\big)\to 0.$ \end{enumerate} The same equivalences remain valid when ${\mathrm P}_{{\mathrm P}}$ is replaced throughout by an outer probability measure.

proof{\bf Part I.} We first prove the sequence and uniform characterizations for convergence to zero in probability are equivalent. It is trivial that (a) implies (b). Conversely, suppose (a) fails, then there exist $\epsilon_0>0$, $\eta_0>0$, and a subsequence $\{n_k\}_{k\ge1}$ such that \begin{align*} \sup_{{\mathrm P}\in\cP_{n_k}}{\mathrm P}_{{\mathrm P}}\big(\lVert Z_{n_k,{\mathrm P}}\rVert>\epsilon_0 b_{n_k}\big)>\eta_0 \end{align*} for all $k$. Hence one can choose ${\mathrm P}_{n_k}\in\cP_{n_k}$ such that \begin{align*} {\mathrm P}_{{\mathrm P}_{n_k}}\big(\lVert Z_{n_k,{\mathrm P}_{n_k}}\rVert>\epsilon_0 b_{n_k}\big)>\eta_0/2 \end{align*} for all $k$, Extending $\{{\mathrm P}_{n_k}\}_{k\ge1}$ arbitrarily to a full sequence $\{{\mathrm P}_n\}_{n\ge1}$ with ${\mathrm P}_n\in\cP_n$ contradicts (b). {\bf Part II.} Similarly, to prove the sequence and uniform characterizations of boundedness in probability are equivalent, we note (a) implies (b) from \begin{align*} {\mathrm P}_{{\mathrm P}_n}\big(\|Z_{n,{\mathrm P}_n}\|>\ell_n b_n\big) \le \sup_{{\mathrm P}\in\cP_n}{\mathrm P}_{{\mathrm P}}\big(\|Z_{n,{\mathrm P}}\|>\ell_n b_n\big). \end{align*} Conversely, if (a) fails, there then exist a deterministic sequence $\ell_n\to\infty$, a constant $\eta_0>0$, and a subsequence $\{n_k\}_{k\ge1}$ such that \begin{align*} \sup_{{\mathrm P}\in\cP_{n_k}}{\mathrm P}_{{\mathrm P}}\big(\lVert Z_{n_k,{\mathrm P}}\rVert>\ell_{n_k} b_{n_k}\big)>\eta_0 \end{align*} for all $k$. Hence one can choose ${\mathrm P}_{n_k}\in\cP_{n_k}$ such that \begin{align*} {\mathrm P}_{{\mathrm P}_{n_k}}\big(\lVert Z_{n_k,{\mathrm P}_{n_k}}\rVert>\ell_{n_k} b_{n_k}\big)>\eta_0/2 \end{align*} for all $k$. Extending $\{{\mathrm P}_{n_k}\}_{k\ge1}$ arbitrarily to $\{{\mathrm P}_n\}_{n\ge1}$ with ${\mathrm P}_n\in\cP_n$ contradicts (b). {\bf Part III.} The last claim for outer probabilities follows by the same proof, with ${\mathrm P}_{{\mathrm P}}$ replaced throughout by the corresponding outer probability.

lemma[Transition of stochastic orders, modified from Lemma 3 in cheng2010bootstrap] For a random quantity $V_n$, we have \[ V_n = o_{{\mathrm P}_{XW}}^o(1) \text{ if and only if }V_n=o_{{\mathrm P}_W}^o(1) \text{ in ${\mathrm P}_{X}^o$-probability.} \] Moreover, when a random variable $Q_n$ on the product probability space has a bootstrap distribution that asymptotically imitates the unconditional distribution of $Q$, which has a continuous cumulative distribution function, that is, \begin{align*} \sup_{t \in \bR} \big\lvert {\mathrm P}_{W|X}\big(Q_n \le t \big) - {\mathrm P}(Q \le t) \big\rvert &= o_{{\mathrm P}_{X}}^o(1), \end{align*} we then also have the same claim holds true for $Q_n + V_n$, if $V_n=o_{{\mathrm P}_W}^o(1)$ in ${\mathrm P}_{X}^o$-probability: \begin{align*} \sup_{t \in \bR} \big\lvert {\mathrm P}_{W|X}\big(Q_n+V_n \le t \big) - {\mathrm P}(Q \le t) \big\rvert &= o_{{\mathrm P}_{X}}^o(1). \addtocounter{equation}{1}\tag{\theequation} \end{align*}

proof{\bf Part I.} We first prove \begin{align*} V_n = o_{{\mathrm P}_{XW}}^o(1) if and only if V_n=o_{{\mathrm P}_W}^o(1) in ${\mathrm P}_{X}^o$-probability. \end{align*} For all $\epsilon,\delta>0$, by Markov's inequality one has \begin{align*} {\mathrm P}_{X}^o( {\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) \ge \delta ) \le \frac{1}{\delta} {\mathrm E}_{X}^o\big[{\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) \big] = \frac{1}{\delta} {\mathrm E}_{X}^o\big[{\mathrm E}_{W|X}^o\{\ind(\lvert V_n \rvert \ge \epsilon)\} \big]. \end{align*} From Fubini's theorem for repeated outer expectations kosorok2008introduction, we can further bound the righthand side of the above by $${\mathrm E}_{XW}^o [\ind(\lvert V_n \rvert \ge \epsilon)] = {\mathrm P}_{XW}^o(\lvert V_n \rvert \ge \epsilon).$$ Therefore we conclude that $V_n = o_{{\mathrm P}_{XW}}^o(1)$ implies $V_n=o_{{\mathrm P}_W}^o(1)$ in ${\mathrm P}_{X}^o$-probability. The other direction follows from for any $\epsilon$ and an arbitrary $\eta$, \begin{align*} {\mathrm P}_{XW}^o(\lvert V_n \rvert \ge \epsilon) &= {\mathrm E}_{X}^o\big[{\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) \big] \addtocounter{equation}{1}\tag{\theequation} \\ &\le {\mathrm E}_{X}^o\big[{\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) \ind \big({\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) \ge \eta \big) \big] \\ & + {\mathrm E}_{X}^o\big[{\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) \ind \big({\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) < \eta \big) \big] \\ &= {\mathrm E}_{X}^o\big[ \ind \big({\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) \ge \eta \big) \big] + \eta \\ &\le {\mathrm P}_{X}^o\big( {\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) \ge \eta \big) + \eta. \end{align*} Specially, we note that any random element $V_n$ defined only on $( \Omega_X^\infty, \cA_X^\infty, {\mathrm P}_X^\infty )$ with stochastic order $o_{{\mathrm P}_X}^o(1)$ is also of of an order $o_{{\mathrm P}_W}^o(1)$ in ${\mathrm P}_X^o$-probability. As for any $\epsilon, \delta>0$, we have \begin{align*} {\mathrm P}_X^o \big( {\mathrm P}_{W|X}^o(\lvert V_n \rvert > \epsilon) > \delta \big) &\le \frac{1}{\delta} {\mathrm E}_X^o \big[ {\mathrm E}_{W|X}^o\{\ind(\lvert V_n \rvert > \epsilon) \}\big] = \frac{1}{\delta} {\mathrm E}_X^o \big[ \ind(\lvert V_n \rvert > \epsilon)\big] =\frac{1}{\delta} {\mathrm P}_X^o (\lvert V_n \rvert > \epsilon), \end{align*} by Markov's inequality and noting $V_n$ does not depend on the bootstrap weights. {\bf Part II.} Now we proceed to prove that $V_n$ does not have an impact on the bootstrap distribution of $Q_n$. For any $t \in \bR$ and $\epsilon>0$, on one hand, we have \begin{align*} {\mathrm P}_{W|X}^o(Q_n+V_n \le t) &= {\mathrm P}_{W|X}^o(Q_n+V_n \le t, \lvert V_n \rvert \ge \epsilon) + {\mathrm P}_{W|X}^o(Q_n+V_n \le t, \lvert V_n \rvert < \epsilon) \\ &\le {\mathrm P}_{W|X}^o(\lvert V_n \rvert \ge \epsilon) + {\mathrm P}_{W|X}^o(Q_n \le t + \epsilon). \end{align*} On the other hand, we have \begin{align*} {\mathrm P}_{W|X}^o(Q_n+V_n \le t) &= 1-{\mathrm P}_{W|X}^i(Q_n+V_n> t) \\ &= 1-{\mathrm P}_{W|X}^i(Q_n+V_n> t, \lvert V_n \rvert < \epsilon) -{\mathrm P}_{W|X}^i(Q_n+V_n> t, \lvert V_n \rvert \ge \epsilon) \\ & \ge 1-{\mathrm P}_{W|X}^i(Q_n> t-\epsilon) -{\mathrm P}_{W|X}^i( \lvert V_n \rvert \ge \epsilon), \end{align*} where ${\mathrm P}_{W|X}^i$ stands for the corresponding inner probability. Combining these two inequalities, we obtain that, for any $t \in \bR$ and $\epsilon>0$, \begin{align*} o_{{\mathrm P}_{X}}^o(1) + {\mathrm P}_{W|X}^o(Q_n \le t-\epsilon) \le {\mathrm P}_{W|X}^o(Q_n+V_n \le t) \le {\mathrm P}_{W|X}^o(Q_n \le t + \epsilon) + o_{{\mathrm P}_{X}}^o(1). \end{align*} Accordingly, at all $t$ such that $ {\mathrm P}(Q \le t)$ is continuous at $t$, we can take $\epsilon \downarrow 0$ and establish \[ {\mathrm P}_{W|X}^o(Q_n+V_n \le t) = {\mathrm P}(Q \le t) + o_{{\mathrm P}_{X}}^o(1). \] The uniform result in (ref) follows from Polya's theorem.