A maximal inequality for local empirical processes under weak dependence

Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.

14,000 characters · 4 sections · 18 citation commands

Rendered from LaTeX for readability, not typeset faithfully. Citation keys are highlighted; maths is left as source; figures, tables and equation environments are summarised rather than reproduced; unrecognised commands are greyed out so nothing is silently dropped. Email addresses are removed.

abstractWe introduce a maximal inequality for a local empirical process under strongly mixing data. Local empirical processes are defined as the (local) averages $\frac{1}{nh}\sum_{i=1}^n \mathbf{1}\{x - h \leq X_i \leq x+h\}f(Z_i)$, where $f$ belongs to a class of functions, $x \in \mathbb{R}$ and $h > 0$ is a bandwidth. Our nonasymptotic bounds control estimation error uniformly over the function class, evaluation point $x$ and bandwidth $h$. They are also general enough to accomodate function classes whose complexity increases with $n$. As an application, we apply our bounds to function classes that exhibit polynomial decay in their uniform covering numbers. When specialized to the problem of kernel density estimation, our bounds reveal that, under weak dependence with exponential decay, these estimators achieve the same (up to a logarithmic factor) sharp uniform-in-bandwidth rates derived in the iid setting by Einmahl2005.

Introduction

In this paper, we introduce a maximal inequality for a local empirical process under exponential decay of the sample $\alpha$-mixing coefficients. We provide a nonasymptotic bound on an Orlicz norm of the uniform estimation error of the local-average process $\frac{1}{nh}\sum_{i=1}^n \mathbf{1}\{x - h \leq X_i \leq x+h\}f(Z_i) $, where uniformity holds simultaneously over the evaluation point $x \in \mathbb{R}$, bandwidth $ h \in [a_n, b_n)$, and function $f \in \mathcal{F}$, with $a_n \leq b_n$ being positive constants and $\mathcal{F}$ a function class. The nonasymptotic nature of our results allows for function classes whose complexity, as measured by their uniform covering numbers, increases with $n$. The latter is especially useful for applications in high-dimensional statistics (see Belloni2017 for an example). We also discuss how to extend our results to multidimensional $x$ and subgeometric decay of mixing coefficients.

We then apply our results to functional classes $\mathcal{F}$ that exhibit polynomial decay in their uniform covering numbers. This class is particulary interesting, as polynomial decay in uniform entropy is ensured by a finite VC dimension vanderVaart1996. When specialized to the problem of kernel density estimation, our results show that, under exponential decay in the mixing coefficients, kernel estimators achieve the same, up to a logarithmic factor, sharp uniform-in-bandwidth rates obtained by Einmahl2005 in the iid setting.

To the best of our knowledge, the closest paper to ours is Escanciano2020, who provides uniform-in-bandwidth asymptotic rates for local empirical processes under stationarity, a $\beta$-mixing condition and polynomial decay in bracketing entropy. There are some important differences between his approach and ours, though. First, Escanciano2020's analysis is asymptotic, thus not applicable to function classes of increasing complexity. Second, while the author uses bracketing entropy to control complexity, our analysis relies on uniform covering numbers. Finally, his main result (Theorem 2.1) leads to a rate of $a_n^{-1/2}$ over the function classes encompassed by his setting. When compared to our application in (ref), we see that our resulting rates are logarithmic in $a_n$, thus offering a great improvement over his results in kernel-type problems.

The remainder of this paper is organized as follows. Section (ref) introduces our main result. Section (ref) applies it to function classes with uniform entropy displaying polynomial decay. Section (ref) concludes.

A maximal inequality for local empirical processes

We present our main result in the theorem below. In what follows, we define, for some $p \geq 0$, $\psi_p(x) \coloneqq \exp(-x^p) - 1$; and, for a random variable $Z$, the Orlicz norm with respect to $\psi_p$ is given by:

$$\lVert Z \rVert_{\psi_p} \coloneqq \inf \left\{C > 0: \mathbb{E}\left[\psi_p\left(\frac{|Z|}{C}\right)\right]<\infty\right\} \, .$$

Moreover, for a sequence $\{Y_i\}_{i=1}^n$ of random variables defined on a common probability space $(\Omega, \Sigma, \mathbb{P})$, we define the $\alpha$-mixing coefficients as, for each $k \in \mathbb{N}$.

$$\alpha(k) \coloneqq \sup_{t \in \mathbb{N}} \sup_{A \in \sigma(Y_1,\ldots, Y_t), B \in \sigma( Y_{t+k},Y_{t+k+1},\ldots)} |\mathbb{P}[A\cap B] - \mathbb{P}[A]\mathbb{P}[B]| \, .$$

theoremLet $\{(X_i, Z_i)\}_{i \in \mathbb{N}}$ be a sequence of random variables defined on a common probability space $(\Omega, \Sigma, \mathbb{P})$, with mixing coefficients satisfying, for some $c > 0$, $\alpha(i) \leq \exp(-2 c i)$ $\forall i \in \mathbb{N}$. Suppose that each $X_i$ is real valued, with common distribution fucntion $G$ that admits a bounded Lebesgue density $g$. Suppose that each $Z_i$ takes values on a measurable space $(\mathcal{Z}, \Sigma)$. Let $\mathcal{F}$ be a pointwise measurable class of real-valued functions with domain $\mathcal{Z}$. Define: $$S_n(x,f; h) \coloneqq \frac{1}{\sqrt{nh}}\sum_{i=1}^n \mathbf{1}\{ x - h \leq X_i \leq x+h\} f(Z_i) \, , \quad u \in [0,1], f \in \mathcal{F}, h \in [a_n,b_n) \, ,$$ where $0<a_n\leq b_n $ are positive constants, with $ \frac{1}{\lVert g\rVert_\infty n } \leq b_n < \frac{1}{\lVert g\rVert_\infty }$. Suppose that the class of functions $\mathcal{F}$ admits a measurable and bounded envelope $\kappa$. Assume that, for some $q \in (1,\infty]$, $\omega(\epsilon) \coloneqq \sup_{P \in \mathcal{P}(\mathcal{Z})}\mathcal{N}(2\epsilon \lVert \kappa \rVert_\infty,\mathcal{F}, \lVert \cdot \rVert_{q,P})$, is finite for every $\epsilon > 0$, where $\lVert f \rVert_{P,q} = (P|f|^q)^{1/q}$ and the supremum is taken over all probability measures on $(\mathcal{Z}, \Sigma)$. Then, for every $n \geq 2$: \begin{equation} \left\lVert \sup_{x \in \mathbb{R}, h \in [a_n, b_n), f \in \mathcal{F} }|S_n(x,f;h) - \mathbb{E}[S_n(x,f;h)]| \right\rVert_{\psi_1}\leq C\frac{\lVert \kappa \rVert_\infty}{\sqrt{na_n}} + \frac{D \lVert \kappa \rVert_\infty}{\sqrt{na_n}} \sum_{l=\lceil -\log_2(\lVert g\rVert_\infty b_n)\rceil}^{\lceil\log_2(n) \rceil} \inf_{\delta \geq 0} \psi_l(\delta) \, , \end{equation} where $C>0$ is an absolute consant; the constant $D>0$ depends solely on $c$; and: \begin{equation} \begin{aligned} \psi_l(\delta) = \left(\log(n)^2[(l+1)\lor \log(\omega(\delta))] + \sqrt{\frac{ln}{2^l} \lor 1}\sqrt{[(l+1)\lor \log(\omega(\delta))]}\right) + \\ n\delta \left[\left(n^{-1}\log(n)^2(l+1) + \sqrt{\frac{l}{2^l n} \lor \frac{1}{n^2}}\sqrt{(l+1)}\right)^\frac{{q-1}}{q} +\frac{1}{2^{\frac{(q-1)}{q}(l+1)}}\right] \, . \end{aligned} \end{equation}

The proof of Theorem (ref) is deferred to (ref). It relies on a chaining argument due to Rio2017 (Rio2017), which we couple with the Bernstein inequality of Merlevede2009 to help achieve control of the error of the localizing function $\mathbf{1}\{x \leq X \leq x + h \}$ uniformly over the bandwidth. Note that our bound is nonasymptotic, with the constant $D$ depending solely on the decay parameter $c$. Our bound also depends on minimising the functions $\psi_l(\delta)$. This minimisation involves finding a “small” $\delta$ that is able to ensure appropriate control of the uniform entropy $\omega(\delta)$. In the next section, we show that, for classes with polynomial decay in their uniform entropy, a useul upper bound to this infimum can be computed.

remark[Extension to nid $X$] It is possible to extend our results to the setting where the distribution of $X_i$ varies with $i$. In this case, one has to adapt Lemma (ref) in Appendix (ref) to allow for a different localization parameter $K_i$ for every $i$. Since this brings clutter to the proofs and does not reveal any new insights, for ease of exposition, we focus on the case where the $X_i$ are identically distributed. Observe that, in the statement of (ref), we do not require the $Z_i$ to be identically distributed.

remark[Extension to multidimensional $X$] It is possible to extend our results to the setting where $X$ is multidimensional by relying on Lemma 7.2 of Rio2017. This result decomposes the difference between a $d$-dimensional cdf at two points as a sum of $d$ differences of the $d$ marginal cdfs at different evaluation points. Using this result, we may extend Lemma (ref) in Appendix (ref) to the multivariate setting, and use it to prove a multivariate version of Theorem (ref).

remark[Extension to unbounded envelopes and subgeometric decay] The statement of (ref) requires the function class to display bounded envelope; and mixing coefficients to exhibit exponential (geometric) decay. These assumptions can be relaxed by replacing the Bernstein inequality of Merlevede2009 with a result due to Merlevede2010. In this case, a tradeoff between the tail behaviour of the envelope function and the decay of mixing coefficients emerges. Consequently, we achieve control over the uniform estimation error under a different Orlicz norm.

Application: function classes with polynomial decay

In this section, we apply our results to classes of functions that exhibit polynomial decay in their covering numbers. Specifically, in the setting of (ref), we take $q=2$ and consider a sequence of bounded function classes $\{\mathcal{F}_n\}_{n \in \mathbb{N}}$ with corresponding envelope $ \kappa_n$, whose uniform entropy numbers $\omega_n$ satisfy:

$$\omega_n(\delta) \leq C_n \left(\frac{1}{\delta}\right)^{v_n} \,, \quad \forall \delta \in (0,1],$$ for positive constants $C_n> 0$,$v_n > 0$, $n \in \mathbb{N}$. We also assume the envelopes to be uniformly bounded, with $\sup_{n \in \mathbb{N}} \lVert \kappa_n\rVert_\infty \leq \phi < \infty$. Suppose that $na_n \to \infty$. In this case, we may find a constant $G>0$ such that, for every $n$ above a certain threshold, $ \lceil - \log_2(\lVert g \rVert_\infty b_n )\rceil \leq l \leq \lceil \log_2(n)\rceil$; and $\delta < \exp\left(\frac{1}{v_n}(\log(C_n) - (l+1))\right)$:

equation[equation omitted — 254 chars of source]

Consider the choice:

$$\delta_l = \frac{C_n^{1/v_n}}{2^{((1/v_n))(l+1)}\sqrt{n} (\lVert g \rVert b_n)^{1/v_n} \log(n)^{3/2}}\, .$$

Assuming that:

$$\frac{\log(n)^3 \log \log n}{\sqrt{Ta_t}} \to 0 \, .$$

In this case, we obtain the rate:

equation[equation omitted — 328 chars of source]

Our rate explicitly accomodates for function classes of increasing complexity. If the sequence of functions has finite (albeit possibly increasing) VC dimension, then Theorem 2.6.7 of vanderVaart1996 shows that the $v_n$ and $C_n$ may be taken such that $C_n^{1/v_n}$ is bounded. In this case, the complexity of the function class only affects the rate through the $\sqrt{v_n}$ term, where $v_n$ scales linearly with the VC dimension.

Finally, we consider the problem of kernel density estimation. In this case, the function class is the same for every $n \in \mathbb{N}$. If the bandwidths are taken such that $a_n \asymp b_n$, then our rate simplifies to:

$$\sqrt{-\log(a_n)}\sqrt{\log(n) \lor -\log(a_n)} \, .$$

In contrast, Remark 2 in Einmahl2005 shows that, in the iid setting, one can achieve the rate:

$$\sqrt{\log \log (n) \lor - \log(a_n)} \, ,$$ which coincides with our rate except for logarithmic factors. Note that, if we consider, as it is usually done in practice, polynomial bandwidths, i.e $a_n = C n^{-\alpha}$ for $\alpha > 0$, then our rate simplifies to $\log(n)$, whereas Einmahl2005's collapses to $\sqrt{\log(n)}$.

Concluding remarks

In this paper, we introduced a maximal inequality for the uniform estimation error of a local empirical process under strongly mixing data, where uniformity holds simultaneously over the function class, bandwidth and evaluation point. Our nonasymptotic bounds accomodate function classes with increasing complexity, which is a useful feature for “high-dimensional” statistical analyses. As an application, we computed our bounds to function classes that exhibit polynomial decay in their uniform entropy. When specialized to the kernel density estimation problem, these results show that our bound leads to the same optimal rates derived by Einmahl2005 in the iid setting.

More generally, we view our results as a first step in the development of rigorous uniform inference tools in local estimation problems under weak dependence and data-driven bandwidths. Specifically, one may combine our results with couplings in the weakly dependent setting (e.g. Cattaneo2022, Cattaneo2022) to devise test statistics that control size uniformly over the evaluation point $x$. An example is the construction of uniform-in-$x$ confidence bands for local polynomial quantile regression estimators with time series data. We intend to study such procedures in future research.