Statistical Properties of Deep Neural Networks with Dependent Data

Statistical Properties of Deep Neural Networks with Dependent Data

{4pt} {4pt}

Introduction

Deep neural networks (DNNs) have proven useful in the analysis of time series in economics and finance (e.g., gu_empirical_2020; bucci_realized_2020; criado_ramon_electric_2022; lazcano_back_2024 ) and have become increasingly popular in empirical modeling (e.g., sadhwani_deep_2021; maliar_deep_2021; leippold_machine_2022; murray_charting_2024 ). However, the statistical properties of DNN estimators with dependent data are largely unknown, and existing results for general nonparametric estimators are often inapplicable to DNN estimators. As a result, empirical use of DNN estimators often lacks a theoretical foundation.

This paper aims to address this deficiency by first providing general results for nonparametric sieve estimators that offer a framework that is flexible enough for studying DNN estimators under dependent data. These results are then applied to both nonparametric regression and classification contexts, yielding theoretical properties for a class of DNN architectures commonly used in applications. Notably, brown_inference_2024 demonstrates the practical implications of these results in a partially linear regression model with dependent data by obtaining $\sqrt{n}$-asymptotic normality of the estimator for the finite dimensional parameter after first-stage DNN estimation of infinite dimensional parameters.

DNN estimators can be viewed as adaptive linear sieve estimators, where inputs are passed through hidden layers that `learn' basis functions from the data by optimizing over compositions of simpler functions. \footnote{ See Subsection (ref) for a description of DNN architectures. See chen_chapter_2007 for a treatment of sieve estimators, and farrell_deep_2021 for further discussion on framing DNN estimators in the context of more familiar nonparametric estimation procedures. } Some general conditions that are sufficient to obtain statistical properties of certain nonparametric estimators have been studied under independent and identically distributed data (i.i.d.) (e.g., shen_convergence_1994; chen_chapter_2007), and dependent data (wooldridge_white_1991; chen_sieve_1998; chen_optimal_2015). Different from the extant literature, I provide two results for general sieve estimators that are applicable to DNN estimators in settings with dependent data that take values on unbounded sets. Theorem (ref) provides rates of convergence in probability in a setting similar to that of wooldridge_white_1991, which differs from previous results, such as those in chen_sieve_1998 and chen_optimal_2015, by not requiring stationarity. Theorem (ref) extends farrell_deep_2021 beyond DNNs and \iid settings, to provide non-asymptotic probability bounds on both the theoretical and empirical $\Lp{2}$-errors of general sieve estimators under stationary $\beta$-mixing data. These results are well suited to the study of DNN estimators for two key reasons. First, they accommodate general sieve extremum estimation and are not restricted to series methods treated by chen_optimal_2015, as verifying basis function properties is impractical with DNNs' adaptive structure. Second, they avoid conditions on the sieve spaces, relying on entropy with bracketing or interpolation between $\Lp{\infty}$ and $\Lp{2}$ norms (e.g. chen_sieve_1998), which are not feasible for DNNs when network depth diverges with sample size.

Using these general results, I derive statistical properties for DNN estimators with architectures that reflect modern applications: (i) fully connected feedforward networks with continuous piece-wise linear activation functions; (ii) no parameter constraints; and (iii) depth and width that grow with sample size. \footnote{Fully connected feedforward neural networks with more than three hidden layers are often referred to as multilayer perceptrons in the DNN literature.} While early research focused on shallow, often single layer networks with smooth activation functions (e.g. white_artificial_1992; makovoz_uniform_1998; anthony_bartlett_neural_1999), modern applications favor deep networks with many hidden layers (szegedy_inception-v4_2016; schmidt_hieber_nonparametric_2020). To mitigate the increased computational demands of deep networks, modern implementations do not impose parameter constraints and often use non-smooth activation functions (e.g., glorot_deep_2011). Among DNN architectures, fully connected feedforward DNNs are standard in practice (almeida2020multilayer; criado_ramon_electric_2022), and are frequently applied in time-series settings (e.g. dudek_multilayer_2016; borghi_covid19_2021; alshafeey_evaluating_2021 ). Recently, the most popular activation function has been the rectified linear unit (ReLU), $\relu(x)=\max\{0,x\}$ (lecun_deep_2015), which will be the main focus of this paper's DNN results. \footnote{Compared to smooth activation functions, ReLU activation functions have also been shown to offer improved properties both empirically (e.g., sadhwani_deep_2021) and theoretically (e.g., glorot_deep_2011). } However, Subsection (ref) shows that my results apply to DNNs with any continuous piecewise-linear activation function, and discusses how similar results could be obtained for alternative DNN architectures, including those with sigmoid activation functions. \footnote{Recurrent neural networks are also an important class of DNNs for time series settings not considered in this paper. See Subsection (ref) for a brief discussion of these architectures.}

Two results are obtained for these DNN estimators in nonparametric regression settings with mixing processes and unbounded regressors. Theorem (ref) applies Theorem (ref), to obtain convergence rates for the $\Lp{2}$-error with non-stationary $\alpha$-mixing data, and Theorem (ref) applies Theorem (ref) to obtain error bounds with stationary $\beta$-mixing data. When the regressors are bounded, Theorem (ref) implies a convergence rate differing from the rate of that obtained by farrell_deep_2021 under \iid data by only a poly-logarithmic factor, making this result useful for inference in some semiparametric settings, as discussed below.

A third DNN result pertains to classification, one of the most common applications for neural networks. I apply Theorem (ref) to obtain convergence rates in logistic binomial autoregression models with covariates. A similar approach could also yield results for multinomial or non-logistic models using the ideas from farrell_deep_2021. Previous studies on DNN estimators in classification contexts have considered their empirical performance with dependent data (see fawaz_deep_2019 for a review) or their statistical properties in i.i.d. settings (e.g., kim_fast_2021; yara_nonparametric_2024; and citations therein). To the best of my knowledge, this work is the first to derive their statistical properties with dependent data.

Much of the recent theoretical work for DNN estimators focuses on estimating regression functions with additive or hierarchical structures under \iid data (see, kohler_nonparametric_2017; bauer_deep_2019; schmidt_hieber_nonparametric_2020; kohler_rate_2021) and under dependent data (see, kohler_rate_2021; kurisu_adaptive_2024). The compositional functional form of neural networks makes them well-suited for estimating these restricted function classes. These studies use these restrictions to obtain fast, near minimax convergence rates, in some cases surpassing traditional nonparametric estimators (schmidt_hieber_nonparametric_2020). While this literature offers possible theoretical insights into why DNNs have outperformed traditional estimators in some empirical work (e.g., gu_empirical_2020; bucci_realized_2020 ), my approach differs by establishing a more flexible framework for studying various DNN estimators in more general settings under time series data. I provide statistical properties for a common class of DNN architectures, considering both nonparametric regression and classification, under a Hölder smoothness condition, which allows for greater generality relative to these restricted classes. My work also adds generality by not placing bounds on the parameters, which can be critical for feasible implementation (see farrell_deep_2021 for discussion).

Recently, kurisu_adaptive_2024 provided closely related general results for DNN estimators and sparse-penalized adaptive DNN estimators for nonparametric regression under nonstationary $\beta$-mixing data. My work adds generality since they impose parameter constraints, focus only on regression settings, and impose structural assumptions on the regression function when applying their findings, although their general results do not explicitly require this restriction. While I do not address adaptive network architectures, empirical gains from sparsity penalties or other regularization techniques are often unclear (zhang_understanding_2017), and in many cases my general results could apply to similar adaptive DNNs in contexts beyond nonparametric regression, following the ideas discussed in Subsection (ref). Theorem (ref) of this paper also offers some added generality by allowing nonstationary $\alpha$-mixing data, rather than $\beta$-mixing. One advantage of kurisu_adaptive_2024 is that they offer some consideration of $\beta$-mixing coefficients with polynomial decay, whereas my DNN results require exponential decay.

The rest of the paper is organized as follows. Section (ref) considers a general nonparametric estimation setting and gives the main results for general sieve estimators. Section (ref) describes the class of DNN architectures considered in this paper and applies the results of Section (ref) to derive properties of DNN estimators in nonparametric regression and classification contexts. A discussion of extensions to alternative architectures is also provided. Section (ref) concludes and discusses avenues for future research. An appendix provides general measurability results for sieve estimation settings, technical proofs for all of the results presented in the paper, and a complete description of the adopted notation.

General sieve estimators

This section considers the problem of non-parametric sieve extremum estimation with dependent data. The two main results are given in Subsections (ref) and (ref).

Let $\probspace$ be a complete probability space, and $\{\Zt\}_{t\in\mathbb{N}}$ be a stochastic sequence on $\probspace$, with coordinates given by random vectors $\Zt:\Omega \to \R^{d_{\boldsymbol{W}}}\subseteq\R^{d_Z}$ for some $d_Z\in\N$ and each $t$. The parameter space, $\measurableftns$, is a space of functions with elements $f:\R^{d_{\boldsymbol{W}}}\to\R$ measurable-$\borel(\R^{d_{\boldsymbol{W}}})/\borel(\R)$. Let $\crit: \R^{d_{\boldsymbol{W}}} \times \R \to \R$ be the single observation criterion function which is assumed to be measurable-$\big(\borel(\R^{d_{\boldsymbol{W}}})\otimes\borel(\R)\big)/\borel(\R)$, and $\crit\big(\Zt(\cdot),f(\Zt(\cdot))\big) \in \Lp{1}\probspace$, for each $t$ and $f\in\measurableftns$. \footnote{ Requiring that $\crit$ be defined on $\R^{d_{\boldsymbol{W}}} \times \R$ instead of the subset $\R^{d_{\boldsymbol{W}}} \times \measurableftns(\R^{d_{\boldsymbol{W}}})$, where $\measurableftns(\R^{d_{\boldsymbol{W}}}) \coloneqq \cup_{\z\in\R^{d_{\boldsymbol{W}}}} \big\{ f(\z): f\in\measurableftns \big\}\subseteq\R, $ is only for notational convenience later on and is without loss of generality by stinchcombe_measurability_1992. } The empirical criterion function is $$ \Crit(f) = Q\big(\{\Zt\}_{t=1}^n,f\big) \coloneqq \sampavg \crit(\Zt,f(\Zt)), \qquad \text{for }\; n\in\N,\; f\in\measurableftns . $$ The true parameter $ \fO\in\measurableftns $ is defined by $ \E[\Crit(\fO)] \leq \E[\Crit(f)], $ for all $f\in\measurableftns$.

This setting covers a wide range of non/semi-parametric models. I give two examples that illustrate it's breadth. Example (ref) is the regression model from kurisu_adaptive_2024, and Example (ref) considers a classification problem in a logistic binomial autoregression model. Generalizations of these examples will be considered in Section (ref) as applications of this section's results. To consider these examples with the notation used above, note that when $\Zt=(\Yt,\Xt)$ any mapping $\Xt\mapsto f(\Xt)$ can trivially be defined on $\R^{d_{\boldsymbol{W}}}$ using the coordinate projection $\pi_{X}(\Zt)=\Xt$.

Sieve extremum estimation and measurability

The target function $\fO$ can be estimated using the method of sieves (grenander_abstract_1981 chen_chapter_2007). This approach covers a wide variety of nonparametric estimation methods. For a textbook treatment of sieve estimation see chen_chapter_2007.

Let $\{\mathcal{G}\}_{n\in\N}$ be a sequence of sieve spaces such that $\mathcal{G}\subseteq\measurableftns$, and $ \sup_{f\in\mathcal{G}}\norm{f}_\infty<{\infty}$. If $h_n:\Omega\to[0,\infty)$ is a random variable, such that $h_n = \op(1)$, then $\f\in\mathcal{G}$ is an approximate sieve estimator if

$h_n$ is often referred to as the “plug-in” error, and whenever feasible $h_n \coloneqq 0$. In this case $\f$ is referred to as an exact sieve estimator.

In general, the infimum over $\mathcal{G}$ in (ref), and the mapping $\omega\mapsto \f$ from $\Omega$ to $\mathcal{G}$, may not be measurable when $\mathcal{G}$ is uncountable. Definition (ref), and Propositions (ref) and (ref), provide easily verifiable conditions that assure measurability in many sieve estimation settings (see Section (ref) and Remark (ref) for example). Appendix (ref) includes the proofs and a discussion of similar results. The definition of pointwise-separability that follows has many similar forms, see e.g., van_der_vaart_wellner_book_1996.

{

} The next result uses Definition (ref), to provide conditions that ensure the measurability of infimum over $\mathcal{G}$. Clearly, Proposition (ref) will similarly apply to suprema, such as those in the proofs of this paper's results.

{

} { With some abuse of notation, if $\mathcal{G}$ is pointwise-separable, then $ \sup_{f\in\mathcal{G}}\Crit(f)$ is measurable, for any $\mathcal{G}\subseteq\mathcal{G}$. This follows by letting $$ \inf_{f\in\mathcal{G}} \delta_n\Big( \cdot\, , \big\{g_{a}(\boldsymbol{W}_t(\cdot))\big\}_{t=1}^n \Big) \coloneqq \inf_{f\in\mathcal{G}} \sampavg \crit\big(\Zt(\cdot),g_{a}(\Zt(\cdot))\big) :\Omega\to\overline{\R} , $$ in Proposition (ref), since $\crit$ is measurable-$\big(\borel(\R^{d_{\boldsymbol{W}}})\otimes\borel(\R)\big)/\borel(\R)$. \footnote{ To see this, note that, for any $t$, the mapping $(\omega,x)\mapsto(\Zt(\omega),x)$, from $\Omega\times\R $ to $ \boldsymbol{W}\times\R$, is measurable-$\big(\Zsig\otimes\borel(\R)\big)/\big(\borel(\R^{d_{\boldsymbol{W}}})\otimes\borel(\R)\big)$ (aliprantis_infinite_2006). Consequently, the function $(\omega,x)\mapsto \crit(\Zt(\omega),x)$, from $\Omega\times\R $ to $ \R$, is measurable-$\big(\Zsig\otimes\borel(\R)\big)/\borel(\R)$ since $\crit$ is measurable-$\big(\borel(\R^{d_{\boldsymbol{W}}})\otimes\borel(\R)\big)/\borel(\R)$. Then, by aliprantis_infinite_2006, the mapping $ (\omega,x_1,\ldots,x_n) \mapsto \delta_n(\omega,x_1,\ldots,x_n) \coloneqq \sampavg \crit(\Zt(\omega),x_t) , $ from $\Omega\times \R^n $ to $\R$, is measurable-$(\Zsig\otimes\borel(\R^n))/\borel(\R)$, and Proposition (ref) can be applied. } However, for $\f$ as in (ref), this does not ensure that the mapping $\omega\mapsto\f$ is measurable. This will be dealt with using outer integrals/probability in Subsection (ref) (defined therein). Everywhere else in this paper, the following proposition will be used, which adds structure to $\measurableftns$ and $\mathcal{G}$ to ensure the existence of a measurable mapping for $\f$. }

{

The conditions on $U_n$ in Proposition (ref) are somewhat stronger than those used in Proposition (ref) since they imply that $U_n$ is measurable-$(\Zsig\otimes\borel(\R^n))/\borel(\R)$ by using aliprantis_infinite_2006. Notably, $\Crit$ will satisfy this stronger condition for many common choices of $\crit$, such as least squares or logistic loss. }

Proposition (ref) implies that a mapping $\omega\mapsto\f$ from $\Omega\to\mathcal{G}$ exists and is measurable whenever $\crit(\Zt(\omega),\cdot):\R\to\R$ is continuous for all $\omega\in\Omega$, and $\measurableftns\subseteq \Lp{r}(\P_{\{\Zt\}_{t=1}^n})$ is a pointwise-separable class of functions such that $\{f(\z):f\in\mathcal{G}\}\subset\R$ is compact for each $\z\in\R^{d_{\boldsymbol{W}}}$. Proposition (ref) is closely related to wooldridge_white_1991, which is commonly used to ensure the existence and measurability of sieve estimators (e.g., chen_chapter_2007). Although wooldridge_white_1991 is applicable to metric spaces $(\mathcal{G},\rho)$ where $\rho$ may not be induced by an $\Lp{r}$-norm, they require that $(\mathcal{G},\rho)$ be a compact metric space. Therefore, Proposition (ref) adds generality by only requiring $\mathcal{G}\subset \Lp{r}(\P_{\{\Zt\}_{t=1}^n})$ such that $\{f(\z):f\in\mathcal{G}\}\subset\R$ is compact for all $\z\in\R^{d_{\boldsymbol{W}}}$, which does not imply that $(\mathcal{G},\norm{\cdot}_{\Lp{r}(\P_{\{\Zt\}_{t=1}^n})})$ is a compact metric space. This will be crucial when the sieve spaces are constructed using DNNs with unbounded weights as in Section (ref).

Convergence rates without stationarity

This subsection gives a convergence rate result for sieve estimators applicable to very general estimation settings where $\{\Zt\}_{t\in\N}$ is possibly non-stationary. The complexity of $\mathcal{G}$ will be controlled using a covering number as defined below. $(\mathbb{M},\norm{\cdot})$ denotes a semi-metric space, induced by the norm $\norm{\cdot}$.

For $\mathcal{G}\subset\measurableftns$ and a sample $\{\Zt\}_{t=1}^a$, note that $ \cover \big( \delta,\mathcal{G}|_{\{\boldsymbol{W}_t\}_{t=1}^a},\norm{\cdot}_{r,a} \big) $ depends on $\omega\in\Omega$, but $\cover^{(\infty)}_{r}(\delta,\mathcal{G},a)$ does not. Following the usual convention, if $\mathcal{G}=\emptyset$ then $\cover(\delta,\mathcal{G},\norm{\cdot})=1$, for any $\delta>0$.

Theorem (ref) is the first main result of this section. It provides a rate of convergence in probability using an approach similar to the consistency results in wooldridge_white_1991. For generality, the following will not impose the conditions of Proposition (ref) to ensure the measurability of $\f$. Instead, I will use outer probability, $\PrO$, as defined in van_der_vaart_wellner_book_1996, i.e., for an arbitrary set $B\subseteq\Omega$ $$ \PrO(B)= \inf\Big\{P(A):\; A\supseteq B,\; A\in\Zsig\Big\}. $$

\vx

Condition (ref) is a curvature condition on $\crit$ near $\fO$ that is standard in nonparametric estimation (see e.g., chen_sieve_1998; chen_chapter_2007; farrell_deep_2021). Condition (ref)(i) is a Lipschitz condition on $\crit$. In many estimation settings, these requirements are met by common choices of $\crit$, such as least squares or logistic regression (farrell_deep_2021). Conditions (ref)(ii) and (ref)(iii) are requirements on the tail behavior of $\mn$ and $\crit$, which are often satisfied when the extrema of $\{\Zt\}_{t=1}^n$, and $\mathcal{G}$, grow sufficiently slowly with $n$. In many cases (ref)(ii) will imply (ref)(iii). This will be demonstrated in Subsection (ref), with the use of Lemma (ref).

The regularity conditions imposed by (ref) are more general than those typically used in the nonparametric sieve estimation literature. For instance, farrell_deep_2021 requires that $\R^{d_{\boldsymbol{W}}}$ be compact, and $\crit$ satisfy a Lipschitz condition like (ref)(i), except $\mn$ must be a constant. Condition A.4 of chen_sieve_1998 requires that there exist $s\in(0,2)$, $\gamma>4$, and $C>0$ such that, for any $\delta>0$, $ \sup_{\{f\in\mathcal{G}:\metric(f,g)\leq\delta\}} | \crit\big(\z,f(\z)\big) - \crit\big(\z,h(\z)\big) | \leq \delta^s\,\mnZt, $ and $\sup_n\E[\mnZt^\gamma]<C$. In either case, these conditions imply (ref).

Letting $\metric$ vary with $n$ is often necessary in settings where $\{\Zt\}_{t\in\N}$ is non-stationary. For instance, when $\metric$ is the metric induced by $\norm{\cdot}_{\Lp{2}(\PrO_{\{\Zt\}_{t=1}^n})}$ condition (ref) is easily verified in many estimation settings (see Appendix (ref)). Note that condition (ref) will often not hold for fixed $\metric = \rho$, such as $ \metric= \rho= \norm{\cdot}_{\Lp{2}(\PrO_{\{\Zt\}_{t=1}^\infty})} $ for all $n,$ when $\{\Zt\}_{t\in\N}$ is non-stationary.

Condition (ref)(i) can often be met by using an exponential inequality to obtain $\Pboundl$, such as wooldridge_white_1991, or merlevede_bernstein_2009. These inequalities can be applied without requiring $\crit$ to be bounded, due to (ref)(i) and the inclusion of $\mathbbm{1}_{\{\mnZt < \Bound\}}$ in (ref)(i), provided that $\mathcal{G}$ satisfies a boundedness condition, such as $ \sup_{f\in\mathcal{G}}\norm{f}_{\infty}\lesssim \Bound. $

Nonasymptotic error bounds with stationarity and $\beta$-mixing

This section gives finite sample error bounds for sieve estimators in a setting where $\{\Zt\}_{t\in\N}$ is strictly stationary and $\beta$-mixing. When $\{\Zt\}_{t\in\N}$ is stationary let $\Pz=\P_{\Zt}$ for all $t\in\N$. The following definition for the $\beta$-mixing coefficient is from dehling_empirical_2002, and is equivalent to many standard definitions.

Conditions that ensure $\beta$-mixing properties of stochastic sequences have been heavily studied (e.g., doukhan_mixing_1994; bradley_basic_2005), and there are many examples of interesting processes that are $\beta$-mixing with exponentially decreasing coefficients, also referred to as geometric $\beta$-mixing. For instance, conditions to ensure stationarity and geometric $\beta$-mixing for ARMA processes are given by mokkadem_mixing_1988 and for GARCH processes by carrasco_mixing_2002. The following corollary provides an example for nonlinear ARCH processes, using chen_geometric_2000 with the discussion from dinh_tuan_mixing_1986.

This section's main result will use the pseudo-dimension to control the complexity of the sieve spaces $\mathcal{G}$. The following definition is from bartlett_nearly_tight_2019.

Pseudo-dimension, along with related complexity measures like Vapnik–Chervonenkis dimension, (see e.g. bartlett_nearly_tight_2019) is often described as `scale insensitive' because, unlike the covering number, it does not depend on a specific threshold $\delta>0$. Scale-insensitive complexity measures are particularly well-suited for function classes constructed with DNNs that have unbounded parameters, for which bartlett_nearly_tight_2019 provides nearly tight pseudo-dimension bounds. Pseudo-dimension can also be used to bound the covering number with anthony_bartlett_neural_1999 (also see Lemma (ref) herein).

Theorem (ref) extends farrell_deep_2021 to general nonparametric sieve estimators in settings with dependent data that may take values in unbounded sets. This extension relies on stationarity and $\beta$-mixing to apply a standard independent blocking technique, and an exponential inequality from merlevede_bernstein_2009. Following farrell_deep_2021, the proof of Theorem (ref) differs from classic sieve analysis (e.g., wooldridge_white_1991; van_der_vaart_wellner_book_1996; chen_sieve_1998) by using a localization approach with scale-insensitive measures of complexity to offer straightforward applicability to DNN estimators that have unbounded parameters (see farrell_deep_2021 for more discussion).

{

The existence of a measurable mapping $\omega\to\f$ is not an extra assumption, as it will follow from Proposition (ref) and the assumptions in the statement of Theorem (ref). Therefore it is not necessary to use outer probabilities/integrals when dealing with $\f$.

The requirement that $ n > \frac{\tilde{\epsilon}_n(\log n)(\log\log n)}{B} $ will not be binding for large $n$. This follows because (ref) and (ref) imply $ \lim_{n\to\infty} \frac{\tilde{\epsilon}_n(\log n)(\log\log n)}{B} =0 . $

Note that $n\geq16B^2/\log(n)$ and $n\geq 2\Pdim(\mathcal{G})$ for all $n$ sufficiently large is implied by (ref). Requiring that $\Cuib\geq 1$ and $\Cpcq\leq 1$ is also without loss of generality. To see this, if (ref)(ii), or (ref) hold with $\Cuib< 1$ or $\Cpcq> 1$, respectively, then they will also hold with the constant replaced by one. However, this requirement can be dropped if $n$ is large enough or ${(4\cdot288) \Cuib } /{\Cpcq} \geq \sqrt{38}$ (see first paragraph of Appendix (ref) for more detials).

Conditions (ref) and (ref)(i) are analogous to Conditions (ref) (ref)(i), and the discussion following Theorem (ref) still applies. Condition (ref)(ii) is related to (ref)(iii), but Condition (ref)(ii) is somewhat more general since the uniform integrability type requirement on $\crit$ can be replaced with $ \lim_{n\to\infty} B \mnZt\mathbbm{1}_{\{\mnZt>\CuibB\}} = 0 $.

Although Theorem (ref) does not explicitly require $ \lim_{n\to\infty} \Pr\big( \max_{t\in\{1,\ldots,n\}}\mnZt>\CuibB \big) = 0 $, like Condition (ref)(iii), this will be necessary for the error bounds to hold with probability approaching one. In light of this, the requirement that $\crit(\Zt(\omega),\cdot):\R\to\R$ be continuous for all $\omega\in\Omega$, imposes very little additional structure, since Condition (ref)(i) will imply this with probability approaching one whenever $ \lim_{n\to\infty} \Pr\big( \max_{t\in\{1,\ldots,n\}}\mnZt>\CuibB \big) = 0 . $

Theorem (ref) may also imply a rate of convergence by choosing $\delta=\delta_n$ and $n=a_n$ such that $\delta_n$ and $a_n$ go to infinity with $n$ at an appropriate rate. For this, note that for any $\{a_n\}_{n\in\N}$ such that $\lim_{n\to\infty}a_n/\log(n)=\infty$ then $\textstyle \log(n)\frac{n\,\beta(a_n)}{a_n}=0$. To see this, choose $b_n=a_n\Cbeta \log^{-1}(n)$, so by Condition (ref) $$ \frac{\log(n)n\,\beta(a_n)}{a_n} \lesssim \frac{\log(n)ne^{-\Cbeta a_n}}{a_n} = \frac{\log(n)n^{1-b_n}}{a_n} = \frac{\log(n)n^{1-\frac{a_n\Cbeta}{\log(n)} }}{a_n} . $$

}

DNN estimators

This section applies the results of Section (ref) to obtain theoretical properties for a class of DNN estimators commonly used in applications. The proofs for this section are included in Appendix (ref).

Throughout this section, $\{\Zt=(\Yt,\Xt^{\tr})^{\tr}\}_{t\in\mathbb{N}}$ will be a stochastic sequence on the complete probability space $\probspace$, such that, $\Yt\in\R$ and $\Xt\in\R^{d_{\boldsymbol{W}}}$ for each $t$. The object to be estimated is $\fO:\R^{d_{\boldsymbol{W}}}\to[-1,1]$, which satisfies the following H\"older smoothness assumption.

This type of smoothness assumption is standard in the nonparametric estimation literature (e.g. stone_optimal_1982; chen_sieve_1998; chen_chapter_2007; chen_optimal_2015; farrell_deep_2021), and yarotsky_error_2017 shows that DNNs approximate these functions well. Note that this assumption is weaker than the structural assumptions imposed in much of the DNN literature (e.g., kohler_nonparametric_2017; schmidt_hieber_nonparametric_2020; kohler_rate_2021 ).

For most of the results in this section, $\beta$-mixing will be stronger than necessary. In these cases, I use $\alpha$-mixing, as defined by dehling_empirical_2002.

Sufficient conditions for a process to be $\alpha$-mixing have been heavily studied (e.g. doukhan_mixing_1994; or bradley_basic_2005), and an example is included in Corollary (ref) of Subsection (ref). It is well known that $\alpha(j)\leq\beta(j)$, so any $\beta$-mixing process is also $\alpha$-mixing with a rate at least as fast. Consequently, Corollary (ref) can also be used to imply $\alpha$-mixing.

DNN sieve spaces $(\mathcal{G})$

The sieve spaces used here, $\mathcal{G}$, will be constructed using fully connected feedforward DNNs. For concreteness, I focus on DNNs equipped with the ReLU activation function $\relu(x) = \max\{0,x\}$. However, Corollary (ref) shows that all results in the following subsections apply for any continuous piecewise-linear activation function. Considerations for other feedforward architectures and activation functions are also discussed in Subsection (ref).

For $n\in\mathbb{N}$ the graph structure of the architecture $\mathcal{G}$ is characterized by its depth $\L\in\mathbb{N}$, and width vector $\Hb=[H_{n,0},H_{n,1},\ldots,H_{n,\L}] \in \N^{\L+1}$ where $H_{n,0}=d$ for all $n$. Each hidden layer, $l\in\{1,\ldots,\L\}$, is comprised of $H_{n,l}$ “hidden” computational units, referred to as nodes, and denoted as $\n_{l,h}$. The $d$ inputs $\x=[x_1,\ldots,x_d]'\in\R^{d_{\boldsymbol{W}}}$ are fed into each node in the first hidden layer $l=1$. Then each node in layer $l=1$ feeds into each node of the next layer $l=2$. This process repeats with each node in layer $l-1$ passing its output into each node in layer $l$ up to the last hidden layer $l=\L$. Finally, each node in the last hidden layer, $l=\L$, feeds into the output layer of the network, $l=\L+1$, which consists of only one computation unit. Note that nodes in layer $l$ receive inputs only from nodes in layer $l-1$, and none from layers $l-2$ or earlier. See Figure (ref) for an example of an $\mathcal{G}$ architecture.

Each node is a function taking values in $\R$ which depends on a real-valued vector of parameters $\gv_{l,h}$, and takes as arguments the outputs of all nodes in the previous layer, which ultimately depend on the original input $\x$. The parameters for each node consist of a scalar intercept term $\gamma_{l,h,0}$, \footnote{The machine learning literature often refers to $\gamma_{l,h,0}$ as the bias, or threshold. To avoid confusion with similarly named objects in the econometrics literature I refer to this as the intercept term.} and weights $\gamma_{l,h,1},\ldots,\gamma_{l,h,H_{n,l-1}}$ such that $$\gv_{l,h} = [\gamma_{l,h,0},\gamma_{l,h,1},\ldots,\gamma_{l,h,H_{n,l-1}}]^\tr \in \R^{H_{n,l-1}+1}.$$ The parameters for the entire network are collected together as

where $\W$ is the total number of parameters in the network. For a set of parameters, the output of node $h$, in layer $l$, is denoted as $\n_{l,h}(\x\;|\; \gV)$ and the single node in the last layer ($l=\L+1$) is denoted as $\n_{\L+1,1}(\x\;|\; \gV)$ where the dependence on $\gV$ will often be suppressed. Then each node can be written as

With this notation, $\n_{\L+1,1}: \R^{d_{\boldsymbol{W}}} \times \R^{\W} \rightarrow \R$. Then, define the architecture $\mathcal{G}$ as

A particular network $f\in \mathcal{G}$ is determined by $\gV$ and can be written as $f(\cdot) = \n_{\L+1,1}\left(\cdot \;|\; \gV\right):\R^{d_{\boldsymbol{W}}}\to[-B,B]$.

In what follows, the mappings $\omega\mapsto\f$ from $\Omega$ to $\mathcal{G}$ will be considered measurable. This will follow using Remark (ref) since Proposition (ref) will be applicable in the estimation settings considered throughout this section.

DNNs for nonparametric regression

I consider a general nonparametric regression estimation setting, that includes Example (ref) as a special case. Theorem (ref) will apply Theorem (ref) to show the convergence of DNN estimators in a very general setting without assuming stationarity. Theorem (ref) will apply Theorem (ref) to obtain non-asymptotic probability bounds on the empirical and theoretical error of DNN estimators with stationary $\beta$-mixing data.

The goal is to estimate the function $\fO=\E[\Yt|\Xt]$, with a DNN sieve estimator $\f$ as in (ref), using the least squares criterion $$ \crit(\Zt,f) \coloneqq (\Yt-f(\Xt))^2 , $$ and the DNN sieve spaces $ \mathcal{G} = \big\{ f(\pi_{X}(\cdot)) : f\in\mathcal{G} \big\} . $ The regressands $\{\Yt\}_{t\in\N}$ will be assumed to satisfy the following conditions.

This assumption will be the key to ensuring Conditions (ref)(ii)(iii) and (ref)(ii) hold in this setting. The two main results of this section will also specify some additional conditions to control the dependence of $\{\Zt\}_{t\in\N}$ and ensure $B$ grows sufficiently slowly. In both cases, these requirements will be quite general for nonparametric estimation as they will hold without $\Yt$ taking values on a compact set (farrell_deep_2021), and without $\E[\Yt^2|\{\Xt,\Y_{t-1}\}_{t=1}^n]$ being uniformly bounded or $\E[\Yt^{2+\delta}]<\infty$ for some $\delta>0$ (chen_optimal_2015).

Assumption (ref) can be verified using results from extreme value theory for a wide variety of $\{\Yt\}_{t\in\N}$ that are not uniformly bounded almost surely. The following proposition gives two examples that use leadbetter_extremes_1983, and leadbetter_extremes_1983. See Appendix (ref) for a proof showing these results can be applied.

The following lemma shows that Assumption (ref) implies a form of uniform integrability that will be used to verify (ref)(ii) and (ref)(iii) for this subsection's main results. See Appendix (ref) for the proof of Lemma (ref).

The first main result of this section is Theorem (ref), which applies Theorem (ref), to obtain a rate of convergence in probability for DNN estimators in general nonparametric regression settings with nonstationary $\alpha$-mixing data.

Theorem (ref) shows that DNN estimators are consistent in very general settings, however the convergence rate is strictly slower than $n^{-1/4}$. The next result uses Theorem (ref) to obtain nonasymptotic error bounds which can imply faster rates of convergence in probability when $\{\Zt\}_{t\in\N}$ is stationary and $\beta$-mixing.

This result can imply convergence rates as fast as $n^{ -\left(\frac{\smooth}{2(\smooth+d)}\right) } \log^6(n) $ when $\Yt$ is almost surely bounded and $h_n$ converges sufficiently quickly. In this case, the rate only differs by a factor of $\log^2(n)$ from the rate implied by farrell_deep_2021 for settings where $\{\Zt\}_{t\in\N}$ forms an \iid sequence and $\Yt$ takes values in a compact set. Notably, brown_inference_2024 demonstrates that the rates implied by Theorem (ref) can be sufficiently fast to obtain $\sqrt{n}$-asymptotic normality of a finite-dimensional parameter following first stage nonparametric estimation of conditional expectations with DNN estimators in a partially linear regression under stationary $\beta$-mixing data with unbounded regressands.

The convergence rates from Theorem (ref) and the rate implied by Theorem (ref) will not depend on $\alpha(j)$ and $\beta(j)$ respectively, provided they are geometrically mixing. Although, the probability bound from Theorem (ref) will depend on $\Cbeta, \Cbeta'$.

DNNs for binomial autoregressions with covariates

{ This section considers logistic binomial autoregression models, building upon Example (ref). The main result of this section is Theorem (ref), which demonstrates the applicability of Theorem (ref) for DNN estimators in categorical autoregressive settings. Such settings are of particular interest for DNNs since they are frequently employed for classification problems. Although this binomial logistic setting is a simple case of the classification problem, similar results for multinomial and non-logistic settings could be attained with the methods employed here. Also, see farrell_deep_2021 for an example of how the criterion functions for Poisson, Gamma, and multinomial logistic models can be shown to satisfy the requirements of Theorem (ref).

The setting will be a special case of the models considered in truquet_strong_2021. For all $t\in\N$, let $ \Zt\coloneqq(\Yt,\Xt) $ such that $\Yt\in\{0,1\}$ and $ \Xt=(\boldsymbol{V}_{t-1},\Y_{t-1},\ldots\Y_{t-r}) \in [0,1]^{d-r}\times\{0,1\}^r \subset \R^{d_{\boldsymbol{W}}} $ for some random vector of covariates $\boldsymbol{V}_t\in[0,1]^{d-r}$, where $d>r$. Suppose

where $\upsilon_t\in\R$ is some random noise, and the function $ s: [0,1]^{d-r} \times \{0,1\}^{r} \times \R \to \{0,1\} $ is measurable. The estimation target will be a scaled version of the function $ \textstyle \logp{ \frac {\E[\Yt|\Xt]} {1-\E[\Yt|\Xt]} } $ under the following logistic regression assumption.

It will be convenient to write the assumption this way since $\snorm{\fO}\leq 1$ under Assumption (ref). Indeed, Assumption (ref) is equivalent to the usual assumption $ \E[\Yt|\Xt] = e^{s_0}\big[1+e^{s_0}\big]^{-1} $ where it is also assumed that $\snorm{s_0}\leqB$, by setting $\fO=s_0/B$. In what follows, the particular value of $B$ will play no role in the convergence rate of the estimator. Also note that Assumption (ref) will require $\fO$ to be defined on $\R^{d_{\boldsymbol{W}}}$, which imposes some additional structure on $\fO$ since $ \Xt \in [0,1]^{d-r}\times\{0,1\}^r \subset \R^{d_{\boldsymbol{W}}} . $

The goal is to estimate $ \fO = B^{-1} \logp{ \frac {\E[\Yt|\Xt]} {1-\E[\Yt|\Xt]} } , $ using a DNN sieve estimator $\f$ as in (ref) where the criterion is $$ \crit(\Zt,f) \coloneqq -\Yt B f(\Xt) + \logp{1+e^{B f(\Xt)}} , $$ and the DNN sieve spaces are $ \mathcal{G} = \big\{ f(\pi_{X}(\cdot)) : f\in\mathcal{G} \big\} . $

To the best of my knowledge, this is the first result providing a convergence rate for DNN estimators in classification settings with dependent data. Theorem (ref) provides a convergence rate in settings with nonstationary $\alpha$-mixing data that is identical, up to a logarithmic factor, to the rate implied by farrell_deep_2021 under \iid data. In addition, this convergence rate is unaffected by the rate of decay of the $\alpha$-mixing coefficient, provided it is geometric mixing.

Theorem (ref) allows for very general forms of dependence, and includes many interesting examples. The following corollary provides two examples of settings in which Theorem (ref) can be applied without directly assuming the mixing condition for $\{\Zt\}_{t\in\N}$. Corollary (ref)(i) follows from truquet_strong_2021 and point 2 of the discussion following their result, since Assumption (ref) implies $ \P(\Yt=1|\Xt) \in \left[ e^{-B}/ \big(1+e^{-B}\big) \,,\; e^{B}/ \big(1+e^{B}\big) \right] . $ Corollary (ref)(ii) follows from truquet_strong_2021. \footnote{ To apply truquet_strong_2021 note that ergodicity of $\{\boldsymbol{V}_t\}$ is implied by stationarity and $\alpha$-mixing (see e.g. bradley_basic_2005 ). } Using these results, one can also obtain similar sufficient conditions for mixing properties of $\{\Zt\}_{t\in\N}$ in the multinomial case, $\Yt\in\{0,1,\ldots,N\}$. Let the mixing coefficient for $\{\boldsymbol{V}_t\}_{t=0}^\infty$ and $\{\Zt\}_{t\in\N}$ be denoted as $\alpha_{\boldsymbol{V}}$ and $\alpha_{\boldsymbol{W}}$ respectively.

Corollary (ref)(i) allows general distributions for $\upsilon_t$ but imposes a strict exogeneity assumption with respect to $\{\boldsymbol{V}_t\}_{t=0}^{\infty}$. Corollary (ref)(ii) requires $\{\upsilon_t\}_{t=0}^{\infty}$ to be \iid with a uniform distribution, and $\{\boldsymbol{V}_t\}$ to be mixing at a faster rate, but allows for some endogeneity, since $\boldsymbol{V}_t$ can depend on $\upsilon_t$, provided $\boldsymbol{V}_t$ is independent of $\upsilon_{t-j}$, for $j=1,\ldots,t$. These results are somewhat stronger than needed for Theorem (ref), since the stationarity of $\{\Zt\}_{t\in\N}$, or $\{\boldsymbol{V}_t\}_{t=0}^{\infty}$ is not required. truquet_strong_2021 suggests these results could generalize to the non-stationary case, but further work is needed to verify this. }

Extensions to alternative DNN architectures

While the networks considered in the previous sections are standard, similar results can be obtained for other architectures. The key to this will be obtaining results for the complexity--either covering number or pseudo-dimension--and approximation power of the DNN sieve spaces under consideration. This section demonstrates that this is possible and provides extensions of this paper's results to alternative feedforward DNN architectures.

The first result uses yarotsky_error_2017 to show that Theorems (ref), (ref), and (ref) are directly applicable to fully connected feedforward networks with any continuous piecewise-linear activation function, $\varphi$, that has a fixed number of breakpoints, $b\in\N$. One important example of this is the `leaky' ReLU (LReLU) activation function

for some small constant $c>0$ (often set to $c=0.01$) that is predetermined before optimizing over the parameters $\gV$. The LReLU is often used in practice to address the vanishing gradient problem that arises with the ReLU where certain computation units may never have non-zero outputs.

The proofs of Theorems (ref), (ref), and (ref) follow similarly when $\mathcal{G}$ is replaced with $\mathcal{N}_{n,\varphi}$. To see this, note that with yarotsky_error_2017 the approximation result of Lemma (ref) can be extended to DNNs with a continuous piecewise-linear activation function by only increasing $\H$ by a constant factor; and the complexity results of Lemmas (ref) and (ref) are applications of bartlett_nearly_tight_2019 and anthony_bartlett_neural_1999 which also apply to $\mathcal{N}_{n,\varphi}$.

The next result provides a framework to obtain theoretical properties of DNN estimators with a wide variety of activation functions and any feedforward graph structure, provided an approximation result like Lemma (ref) exists for the DNN under consideration. These DNNs will be denoted as $\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}$, which allow for any graph structure where units in layer $l\in\{1,\ldots,\L+1\}$ takes inputs from any of the units in layers $l'\in\{0,\ldots,l-1\}$. See Figure (ref) for an example of $\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}$'s graph structure.

The pseudo-dimension bounds follow from bartlett_nearly_tight_2019 for the piecewise-linear case, bartlett_nearly_tight_2019 for the piecewise-polynomial case, and anthony_bartlett_neural_1999 for the sigmoid case. The covering number bound uses anthony_bartlett_neural_1999. To apply Theorems (ref) or (ref) with Corollary (ref), an approximation result, like Lemma (ref), will be needed for $\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}$ to verify Conditions (ref) or (ref) respectively.

Summary and extensions

This paper addresses the lack of statistical foundation for empirical work using deep neural network (DNN) estimators under dependent data. By establishing general results for sieve estimators, I provide a flexible framework that applies to various DNN estimators in a wide range of dependent data settings. These results extend existing work to more complex and realistic scenarios, allowing for non-\iid data with very general forms of dependence and taking values in unbounded sets. I apply this framework to derive properties for DNN estimators in both nonparametric regression and classification contexts, focusing on architectures that reflect modern applications-- featuring ReLU activation functions, unbounded parameters, fully connected feedforward structures, and depth and width that grow with sample size. Notably, Corollary (ref) shows that these results also apply when the ReLU activation function is replaced with any continuous piecewise-linear activation function, such as the leaky ReLU. The practical relevance of my DNN results is demonstrated by brown_inference_2024 which considers a partially linear regression model with dependent data and unbounded regressands, and shows that the estimator for the finite-dimensional parameter, constructed using DNN-estimated nuisance components, achieves $\sqrt{n}$-consistency and asymptotic normality.

While this work only considers standard DNN architectures, Subsection (ref) demonstrates how the general sieve estimator results presented here offer a pathway for extending the analysis to more complex architectures. Perhaps the most important avenue for future research on DNN estimators under dependent data is recurrent neural networks (RNNs). RNNs are a class of DNN architectures that are specifically designed for time series settings, due to a recursive feedback loop that gives the network a form of `memory' for past events. While the empirical results from lazcano_back_2024 indicate that the DNN architectures studied here outperform RNNs in simpler time-series models, RNNs have demonstrated superior empirical performance in more complex settings, such as in the work by bucci_realized_2020 on forecasting stock market volatility. Thus far, very little work has been done on the theoretical properties of RNNs. While kohler_recurrent_2020 provides an initial study of a specific recurrent architecture in time-series nonparametric regression, much remains to be understood, particularly for more general recurrent architectures, and dependence settings. Following the ideas of Subsection (ref), the sieve estimator framework introduced in this paper could facilitate future research for RNNs once their approximation power and complexity are better understood.

Many other important aspects of DNNs are also not considered here, such as computational efficiency or potential gains from alternative architectures and regularization techniques. Another important class of DNNs not considered here are convolutional neural networks, which are standard in many important DNN applications, such as image recognition. The results given here could also be adapted for classes of functions beyond the standard H\"older smoothness condition, using approximation results such as Imaizumi_DNN_nonsmooth_2019. These considerations are left for future research.

\addappheadtotoc