The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
107,402 characters
Statistical Properties of Deep Neural Networks with Dependent Data
\setlength{\abovedisplayskip}{4pt}
\setlength{\belowdisplayskip}{4pt}
\maketitle
\begin{abstract}
\vspace{-.3cm}
\singlespacing
\noindent
This paper establishes statistical properties of deep neural network (DNN) estimators under dependent data.
Two general results for nonparametric sieve estimators directly applicable to DNN estimators are given.
The first establishes rates for convergence in probability
under nonstationary data.
The second provides non-asymptotic probability bounds on $\mathcal{L}^{2}$-errors under stationary $\beta$-mixing data.
I apply these results to DNN estimators in both regression and classification contexts imposing only a standard H\"older smoothness assumption.
The DNN architectures considered are common in applications, featuring fully connected feedforward networks
with any continuous piecewise linear
activation function, unbounded weights, and a width and depth that grows with sample size.
The framework provided also offers potential for research into other DNN architectures and time-series applications.
\\
\noindent\textbf{Keywords:} Sieve Extremum Estimates, Nonparametric Estimation, Deep Learning, Neural Networks, Rectified Linear Unit, Dependent Data.\\
\noindent\textbf{JEL codes:} C14, C32, C45.
\end{abstract}
\newpage
\section{Introduction}\label{sec:intro}
Deep neural networks (DNNs) have proven useful in the analysis of time series in economics and finance
(e.g.,
\citealp{gu_empirical_2020};
\citealp{bucci_realized_2020};
\citealp{criado_ramon_electric_2022};
\citealp{lazcano_back_2024}
)
and have become increasingly popular in empirical modeling
(e.g.,
\citealp{sadhwani_deep_2021};
\citealp{maliar_deep_2021};
\citealp{leippold_machine_2022};
\citealp{murray_charting_2024}
).
However, the statistical properties of DNN estimators with dependent data are largely unknown, and existing results for general nonparametric estimators are often inapplicable to DNN estimators.
As a result, empirical use of DNN estimators often lacks a theoretical foundation.
This paper aims to address this deficiency
by first providing general results for nonparametric sieve estimators that offer a framework that is flexible enough for studying DNN estimators under dependent data.
These results are then applied to both nonparametric regression and classification contexts, yielding theoretical properties for a class of DNN architectures commonly used in applications.
Notably, \citet{brown_inference_2024} demonstrates the practical implications of these results
in a partially linear regression model with dependent data
by obtaining $\sqrt{n}$-asymptotic normality of the estimator for
the finite dimensional parameter after first-stage DNN estimation of infinite dimensional parameters.
DNN estimators can be viewed as adaptive linear sieve estimators, where inputs are passed through hidden layers that
`learn'
basis functions
from the data
by optimizing over compositions of simpler functions.
\footnote{
See Subsection \ref{sec:DNN_sieve_spaces} for a description of DNN architectures.
See \cite{chen_chapter_2007} for a treatment of sieve estimators, and
\citet{farrell_deep_2021} for further discussion on framing DNN estimators in the context of more familiar nonparametric estimation procedures.
}
Some general conditions that are sufficient to obtain statistical properties of certain nonparametric estimators have been studied under independent and identically distributed data (i.i.d.)
(e.g., \citealp{shen_convergence_1994};
\citealp{chen_chapter_2007}),
and dependent data
(\citealp{wooldridge_white_1991};
\citealp{chen_sieve_1998};
\citealp{chen_optimal_2015}).
Different from the extant literature, I provide two results for general sieve estimators that are applicable to DNN estimators in settings with dependent data that take values on unbounded sets.
Theorem \ref{thrm:ROC_V1}
provides rates of convergence in probability in a setting similar to that of \citet{wooldridge_white_1991},
which differs from previous results, such as those in
\citet{chen_sieve_1998} and
\citet{chen_optimal_2015},
by not requiring stationarity.
Theorem \ref{thrm:ROC_V2} extends
\citet[Theorem 2]{farrell_deep_2021}
beyond DNNs and \iid settings,
to provide non-asymptotic probability bounds on both the theoretical and empirical $\Lp{2}$-errors of general sieve estimators under stationary $\beta$-mixing data.
These results are well suited to the study of DNN estimators for two key reasons.
First, they accommodate general sieve extremum estimation and are not restricted to series methods treated by \citet{chen_optimal_2015}, as verifying basis function properties is impractical with DNNs' adaptive structure.
Second, they avoid conditions on the sieve spaces, relying on
entropy with bracketing or interpolation between $\Lp{\infty}$ and $\Lp{2}$ norms
(e.g. \citealp[Conditions A.3 and A.4]{chen_sieve_1998}),
which are not feasible for DNNs when network depth diverges with sample size.
Using these general results,
I derive statistical properties
for DNN estimators with architectures that reflect modern applications:
(i)
fully connected feedforward networks with continuous piece-wise linear activation functions;
(ii)
no parameter constraints;
and
(iii)
depth and width that grow with sample size.
\footnote{Fully connected feedforward neural networks with more than three hidden layers are often referred to as multilayer perceptrons in the DNN literature.}
While early
research focused on shallow, often single layer networks
with smooth activation functions
(e.g. \citealp{white_artificial_1992}; \citealp{makovoz_uniform_1998}; \citealp{anthony_bartlett_neural_1999}),
modern applications favor deep networks with many hidden layers
(\citealp{szegedy_inception-v4_2016};
\citealp{schmidt_hieber_nonparametric_2020}).
To mitigate the increased computational demands of deep networks,
modern implementations
do not impose
parameter constraints
and
often use non-smooth activation functions (e.g., \citealp{glorot_deep_2011}).
Among DNN architectures,
fully connected feedforward DNNs
are standard in practice
(\citealp{almeida2020multilayer};
\citealp{criado_ramon_electric_2022}),
and are frequently applied in time-series settings
(e.g.
\citealp{dudek_multilayer_2016};
\citealp{borghi_covid19_2021};
\citealp{alshafeey_evaluating_2021}
).
Recently, the most popular activation function has been
the rectified linear unit (ReLU), $\relu(x)=\max\{0,x\}$ (\citealp{lecun_deep_2015}),
which will be the main focus of this paper's DNN results.
\footnote{Compared to smooth activation functions, ReLU activation functions have also been shown to offer improved properties both empirically (e.g., \citealp{sadhwani_deep_2021})
and
theoretically
(e.g., \citealp{glorot_deep_2011}).
}
However, Subsection \ref{sec:DNN_Alt_Arch} shows that my results apply to DNNs with any continuous piecewise-linear activation function, and discusses how similar results could be obtained for alternative DNN architectures, including those with sigmoid activation functions.
\footnote{Recurrent neural networks are also an important class of DNNs for time series settings not considered in this paper. See Subsection \ref{sec:Conc} for a brief discussion of these architectures.}
Two results are obtained for these DNN estimators in nonparametric regression settings with mixing processes and unbounded regressors.
Theorem \ref{thrm:DNN_ROC_V1} applies Theorem \ref{thrm:ROC_V1}, to obtain convergence rates for the $\Lp{2}$-error with non-stationary $\alpha$-mixing data, and
Theorem \ref{thrm:DNN_ROC_V2} applies Theorem \ref{thrm:ROC_V2} to obtain error bounds with stationary $\beta$-mixing data.
When the regressors are bounded, Theorem \ref{thrm:DNN_ROC_V2} implies a convergence rate differing from
the rate of
that obtained by \citet[Theorem 1]{farrell_deep_2021} under \iid data by only a
poly-logarithmic factor,
making this result useful for inference in some semiparametric settings, as discussed below.
A third DNN result pertains to classification, one of the most common applications for neural networks.
I apply Theorem \ref{thrm:ROC_V1} to obtain convergence rates
in logistic binomial autoregression models with covariates.
A similar approach could also yield results for multinomial or non-logistic models using the ideas from \citet[Lemma 9]{farrell_deep_2021}.
Previous studies on DNN estimators in classification contexts have considered their empirical performance with dependent data
(see \citealp{fawaz_deep_2019} for a review)
or their statistical properties in i.i.d. settings (e.g.,
\citealp{kim_fast_2021};
\citealp{yara_nonparametric_2024};
and citations therein).
To the best of my knowledge, this work is the first to derive their statistical properties
with dependent data.
Much of the recent theoretical work for DNN estimators focuses on estimating regression functions with additive or hierarchical structures
under \iid data
(see, \citealp{kohler_nonparametric_2017};
\citealp{bauer_deep_2019};
\citealp{schmidt_hieber_nonparametric_2020};
\citealp{kohler_rate_2021})
and under dependent data
(see, \citealp{kohler_rate_2021};
\citealp{kurisu_adaptive_2024}).
The compositional
functional form
of neural networks makes them well-suited for estimating these restricted function classes.
These studies use
these restrictions to obtain fast, near minimax convergence rates, in some cases surpassing traditional nonparametric estimators
(\citealp{schmidt_hieber_nonparametric_2020}).
While this literature offers possible theoretical insights into why DNNs have outperformed traditional estimators in some empirical work
(e.g.,
\citealp{gu_empirical_2020};
\citealp{bucci_realized_2020}
),
my approach differs by establishing a more flexible framework for studying various DNN estimators in more general settings under time series data.
I provide statistical properties for a common class of DNN architectures, considering both nonparametric regression and classification, under a
Hölder smoothness condition, which
allows for greater generality relative to
these restricted classes.
My work also adds generality by not placing bounds on the parameters, which can be critical for feasible implementation (see \citealp{farrell_deep_2021} for discussion).
Recently, \citet{kurisu_adaptive_2024} provided closely related general results for DNN estimators and sparse-penalized adaptive DNN estimators for nonparametric regression under nonstationary $\beta$-mixing data.
My work
adds generality
since they impose parameter constraints, focus only on regression settings, and
impose structural assumptions on the regression function when applying their findings, although their general results do not explicitly require this restriction.
While I do not address adaptive network architectures, empirical gains from sparsity penalties or other regularization techniques are often unclear (\citealp{zhang_understanding_2017}), and in many cases my general results could apply to similar adaptive DNNs in contexts beyond nonparametric regression, following the ideas discussed in Subsection \ref{sec:DNN_Alt_Arch}.
Theorem \ref{thrm:DNN_ROC_V1} of this paper also offers some added generality by allowing nonstationary $\alpha$-mixing data, rather than $\beta$-mixing.
One advantage of \citet{kurisu_adaptive_2024} is that they offer some consideration of
$\beta$-mixing coefficients with polynomial decay, whereas my DNN results
require
exponential decay.
The rest of the paper is organized as follows.
Section \ref{sec:gen_res}
considers a general nonparametric estimation setting and gives the main results for general sieve estimators.
Section \ref{sec:DNN}
describes the class of DNN architectures considered in this paper and applies the results of Section \ref{sec:gen_res} to derive properties of DNN estimators in nonparametric regression and classification contexts. A discussion of extensions to alternative architectures is also provided.
Section \ref{sec:Conc}
concludes and discusses avenues for future research.
An appendix provides general measurability results for sieve estimation settings, technical proofs for all of the results presented in the paper, and a complete description of the adopted notation.
\section{General sieve estimators}\label{sec:gen_res}
This section considers the problem of non-parametric sieve extremum estimation with dependent data.
The two main results
are
given in Subsections \ref{sec:ROC_NSD} and \ref{sec:EB_SD}.
Let $\probspace$ be a complete probability space,
and
$\{\Zt\}_{t\in\mathbb{N}}$
be a stochastic sequence on
$\probspace$,
with coordinates given by random vectors
$\Zt:\Omega \to \R^{d_{\boldsymbol{W}}}\subseteq\R^{d_Z}$
for some $d_Z\in\N$ and each $t$.
The parameter space, $\measurableftns$,
is a
space
of functions
with elements $f:\R^{d_{\boldsymbol{W}}}\to\R$
measurable-$\borel(\R^{d_{\boldsymbol{W}}})/\borel(\R)$.
Let
$\crit: \R^{d_{\boldsymbol{W}}} \times \R \to \R$
be the single observation criterion function
which is assumed to be
measurable-$\big(\borel(\R^{d_{\boldsymbol{W}}})\otimes\borel(\R)\big)/\borel(\R)$,
and
$\crit\big(\Zt(\cdot),f(\Zt(\cdot))\big) \in \Lp{1}\probspace$,
for each $t$ and $f\in\measurableftns$.
\footnote{
Requiring that $\crit$ be defined on
$\R^{d_{\boldsymbol{W}}} \times \R$
instead of the subset
$\R^{d_{\boldsymbol{W}}} \times \measurableftns(\R^{d_{\boldsymbol{W}}})$,
where
$\measurableftns(\R^{d_{\boldsymbol{W}}})
\coloneqq
\cup_{\z\in\R^{d_{\boldsymbol{W}}}}
\big\{
f(\z): f\in\measurableftns
\big\}\subseteq\R,
$
is only for notational convenience later on and
is without loss of generality by
\citet[Lemma 2.14]{stinchcombe_measurability_1992}.
}
The empirical criterion function is
$$
\Crit(f)
=
Q\big(\{\Zt\}_{t=1}^n,f\big)
\coloneqq
\sampavg \crit(\Zt,f(\Zt)),
\qquad
\text{for }\;
n\in\N,\;
f\in\measurableftns
.
$$
The true parameter
$
\fO\in\measurableftns
$
is defined by
$
\E[\Crit(\fO)] \leq \E[\Crit(f)],
$
for all $f\in\measurableftns$.
This setting covers a wide range of non/semi-parametric models.
I give two examples that illustrate it's breadth.
Example \ref{ex:reg} is the regression model from \cite{kurisu_adaptive_2024}, and
Example \ref{ex:log} considers a classification problem in a logistic binomial autoregression model.
Generalizations of these examples will be considered in Section \ref{sec:DNN} as applications of this section's
results.
To consider these examples with the notation used above,
note that when $\Zt=(\Yt,\Xt)$ any mapping
$\Xt\mapsto f(\Xt)$
can trivially be defined on $\R^{d_{\boldsymbol{W}}}$ using the coordinate projection $\pi_{X}(\Zt)=\Xt$.
\begin{example}\label{ex:reg}
\textbf{\textsc{(Nonparametric time-series regression)}}
For all $t\in\N$, let
$\Zt\coloneqq(\Yt,\Xt)\in
\R^{d_{\boldsymbol{W}}}
\subseteq
\R\times\R^{d}
$
for some $d\in\N$,
such that $\Yt\in\Lp{2}\probspace$ and
\begin{equation}\label{eq:ex1}
\begin{aligned}
\Yt
=
\fO(\Xt) + \eta(\Xt)\upsilon_t
\end{aligned}
\end{equation}
where $\eta\in\Lp{2}\big(\P_{\{\Xt\}_{t\in\N}}\big)$,
$\upsilon_t\in\Lp{2}\probspace$,
and
$\E[\upsilon_t|\Xt]=0$.
Then,
$\measurableftns=\Lp{2}\big(\P_{\{\Xt\}_{t\in\N}}\big),\;$
$
\fO
=
\E[\Yt|\Xt]
,\;
$
and
$
\crit(\Zt,f)
=
\big(\Yt-f(\Xt)\big)^2
.
$
This nonparametric location-scale model includes many popular models, such as a nonlinear
AR($p$)-ARCH($r$) model by letting
$1\leq p,r \leq d$
and
$\Xt=(Y_{t-1},\ldots,Y_{t-d})^\tr$,
given initial conditions
$Y_{0},\ldots,Y_{1-d}$.
See \cite{kurisu_adaptive_2024} for other special cases of \eqref{eq:ex1}.
\end{example}
\begin{example}\label{ex:log}
\textbf{\textsc{(Logistic autoregression)}}
For all $t\in\N$, let
$
\Zt\coloneqq(\Yt,\Xt)
$
such that
$\Yt\in\{0,1\}$
and
$
\Xt=(\boldsymbol{V}_{t-1},\Y_{t-1},\ldots\Y_{t-r})
$
for some random vector of covariates $\boldsymbol{V}_t\in\R^{d-r}$ for $d>r$.
Suppose for any $y\in\{0,1\}$, $t\in\N$,
$$
\P\Big(
\Yt=y
\,
\big|
\,
\{\boldsymbol{V}_t\}_{t=0}^{\infty},
\Y_{t-1}
,\Y_{t-2}
,\ldots
\Big)
=
\P\Big(
\Yt=y
\,
\big|
\,
\Xt
\Big)
,
$$
and
$
\E[\Yt|\Xt]
=
e^{\fO}\big[1+e^{\fO}\big]^{-1}
$
where $|\fO(\x)|<\infty$ for all $\x\in\R^d$.
Then,
$$
\fO
=
\logp{
\frac
{\E[\Yt|\Xt]}
{1-\E[\Yt|\Xt]}
}
,
\quad
\text{and}
\quad
\crit(\Zt,f(\Zt))
=
-\Yt f(\Xt)
+
\logp{1+e^{f(\Xt)}}
.
$$
Note that $\boldsymbol{V}_{t-2},\ldots\boldsymbol{V}_{t-r}$ can trivially be included by replacing
$\boldsymbol{V}_{t-1}$
with
$\boldsymbol{V}_{t-1}^*\coloneqq (\boldsymbol{V}_{t-1},\ldots\boldsymbol{V}_{t-r})$.
\end{example}
\subsection{Sieve extremum estimation and measurability}\label{sec:Sieve_Est_Meas}
The target function $\fO$ can be estimated using the method of sieves (\citealp{grenander_abstract_1981} \citealp{chen_chapter_2007}).
This approach covers a wide variety of nonparametric estimation methods.
For a textbook treatment of sieve estimation see \citet{chen_chapter_2007}.
Let
$\{\mathcal{G}\}_{n\in\N}$
be a sequence of sieve spaces such that
$\mathcal{G}\subseteq\measurableftns$, and
$
\sup_{f\in\mathcal{G}}\norm{f}_\infty<{\infty}$.
If
$h_n:\Omega\to[0,\infty)$
is a random variable, such that
$h_n = \op(1)$,
then
$\f\in\mathcal{G}$
is an \textit{approximate sieve estimator} if
\begin{equation}\label{eq:fhat}
\Crit(\f)
\leq
\inf_{f\in\mathcal{G}} \Crit(f) + h_n
.
\end{equation}
$h_n$ is often referred to as the ``plug-in'' error, and whenever feasible
$h_n \coloneqq 0$.
In this case
$\f$ is referred to as an \textit{exact sieve estimator}.
In general, the infimum over $\mathcal{G}$ in \eqref{eq:fhat}, and the mapping
$\omega\mapsto \f$ from $\Omega$ to $\mathcal{G}$,
may not be measurable when $\mathcal{G}$ is uncountable.
Definition \ref{def:point_sep}, and Propositions \ref{prop:meas_V2_sieve} and \ref{prop:meas_V3_sieve}, provide easily verifiable conditions that assure measurability in many sieve estimation settings
(see Section \ref{sec:DNN} and Remark \ref{rem:FMLP} for example).
Appendix \ref{sec:meas} includes the proofs and a discussion of similar results.
The definition of pointwise-separability that follows has many similar forms, see e.g., \citet[Example 2.3.4]{van_der_vaart_wellner_book_1996}.
{
\begin{definition}\label{def:point_sep}
\textbf{\textsc{(Pointwise-Separable)}}
Let $\mathcal{G}$ be a set of functions with elements
$g_{a}:\R^{d_{\boldsymbol{W}}} \to \R^{d_{\boldsymbol{W}}}$
that are measurable-$\borel(\R^{d_{\boldsymbol{W}}})/\borel(\R^{d_{\boldsymbol{W}}})$.
The set $\mathcal{G}$ is pointwise-separable if there is a countable subset
$\{g_{a}_j\}_{j\in\N}\subseteq\mathcal{G}$
where for every
$g_{a}\in\mathcal{G}$,
$\z\in\R^{d_{\boldsymbol{W}}}$, and
$\delta>0$,
there exists
$j=j(\z,\delta,g_{a}) \in \N$
such that
$|g_{a}_j(\z)-g_{a}(\z)|<\delta$.
\end{definition}
}
The next result uses Definition \ref{def:point_sep}, to provide conditions that ensure the measurability of infimum over $\mathcal{G}$.
Clearly, Proposition \ref{prop:meas_V2_sieve} will similarly apply to suprema, such as those in the proofs of this paper's results.
{
\begin{proposition}\label{prop:meas_V2_sieve}
Let $\mathcal{G}$ be a pointwise-separable class of functions.
Then,
for any $n\in\N$ and
$\delta_n:\Omega\times \R^n \to \R$
that is measurable-$(\Zsig\otimes\borel(\R^n))/\borel(\R)$,
the mappings
\begin{gather*}
\omega
\mapsto
\inf_{g_{a}\in \mathcal{G}}
\delta_n\Big(
\omega\,
,
\big\{g_{a}(\boldsymbol{W}_t(\omega))\big\}_{t=1}^n
\Big)
,
\;\;\;\;
\forall\, \mathcal{G}\subseteq \mathcal{G},
\;
\mathcal{G} \neq \emptyset,
\end{gather*}
from $\Omega$ to $\overline{\R}$,
are measurable-$\Zsig/\borel(\overline{\R})$.
\end{proposition}
}
{
With some abuse of notation, if $\mathcal{G}$ is pointwise-separable, then
$ \sup_{f\in\mathcal{G}}\Crit(f)$ is measurable, for any $\mathcal{G}\subseteq\mathcal{G}$.
This follows by
letting
$$
\inf_{f\in\mathcal{G}}
\delta_n\Big(
\cdot\,
,
\big\{g_{a}(\boldsymbol{W}_t(\cdot))\big\}_{t=1}^n
\Big)
\coloneqq
\inf_{f\in\mathcal{G}}
\sampavg \crit\big(\Zt(\cdot),g_{a}(\Zt(\cdot))\big)
:\Omega\to\overline{\R}
,
$$
in Proposition \ref{prop:meas_V2_sieve},
since $\crit$ is measurable-$\big(\borel(\R^{d_{\boldsymbol{W}}})\otimes\borel(\R)\big)/\borel(\R)$.
\footnote{
To see this, note that, for any $t$, the mapping
$(\omega,x)\mapsto(\Zt(\omega),x)$,
from
$\Omega\times\R $ to $ \boldsymbol{W}\times\R$,
is measurable-$\big(\Zsig\otimes\borel(\R)\big)/\big(\borel(\R^{d_{\boldsymbol{W}}})\otimes\borel(\R)\big)$
(\citealp[Lemma 4.49]{aliprantis_infinite_2006}).
Consequently, the function
$(\omega,x)\mapsto \crit(\Zt(\omega),x)$,
from
$\Omega\times\R $ to $ \R$,
is measurable-$\big(\Zsig\otimes\borel(\R)\big)/\borel(\R)$
since $\crit$ is measurable-$\big(\borel(\R^{d_{\boldsymbol{W}}})\otimes\borel(\R)\big)/\borel(\R)$.
Then, by \citet[Lemma 4.52]{aliprantis_infinite_2006}, the mapping
$
(\omega,x_1,\ldots,x_n)
\mapsto
\delta_n(\omega,x_1,\ldots,x_n)
\coloneqq
\sampavg
\crit(\Zt(\omega),x_t)
,
$
from $\Omega\times \R^n $ to $\R$,
is measurable-$(\Zsig\otimes\borel(\R^n))/\borel(\R)$, and
Proposition \ref{prop:meas_V2_sieve} can be applied.
}
However,
for $\f$ as in \eqref{eq:fhat}, this does not ensure that the mapping
$\omega\mapsto\f$
is measurable.
This will be dealt with using outer integrals/probability in Subsection \ref{sec:ROC_NSD} (defined therein).
Everywhere else in this paper, the following proposition will be used, which adds structure to $\measurableftns$ and $\mathcal{G}$ to ensure the existence of a measurable mapping for $\f$.
}
{
\begin{proposition}\label{prop:meas_V3_sieve}
Let $1\leq r <\infty$ and $n\in\N$.
Suppose
$\mathcal{G}\subset\Lp{r}(\P_{\{\Zt\}_{t=1}^n})$
is a pointwise-separable class of functions
such that
$\{g_{a}(\z):g_{a}\in\mathcal{G}\}\subset\R$
is compact for each $\z\in\R^{d_{\boldsymbol{W}}}$.
Let
$\delta_n:\Omega\times \R^n \to \R$
be
such that
for each $\x\in\R^n$ the function
$\delta_n(\cdot,\x):\Omega\to\R$
is measurable-$\Zsig/\borel(\R)$,
and
for each $\omega\in\Omega$ the function
$\delta_n(\omega,\cdot):\R^n\to\R$
is continuous.
Then, there exists a function
$s:\Omega\to\mathcal{G}$
such that for each $\omega\in\Omega$
$$
s(\omega)
\in
\bigg\{
g_{a}\in\mathcal{G}
:\;
\delta_n\Big(
\omega\,
,
\big\{g_{a}(\boldsymbol{W}_t(\omega))\big\}_{t=1}^n
\Big)
=
\inf_{g_{a}\in \mathcal{G}}
\delta_n\Big(
\omega\,
,
\big\{g_{a}(\boldsymbol{W}_t(\omega))\big\}_{t=1}^n
\Big)
\bigg\}
\neq
\emptyset
,
$$
and $s$ is measurable-$\Zsig/\borel(\mathcal{G})$, where $\borel(\mathcal{G})$
uses
the topology on $\mathcal{G}$ generated by
$\norm{\cdot}_{\Lp{r}(\P_{\{\Zt\}_{t=1}^n})}$.
\end{proposition}
The conditions on $U_n$ in Proposition \ref{prop:meas_V3_sieve}
are somewhat stronger than those used in Proposition \ref{prop:meas_V2_sieve}
since they imply that
$U_n$ is measurable-$(\Zsig\otimes\borel(\R^n))/\borel(\R)$
by using
\citet[Lemma 4.51]{aliprantis_infinite_2006}.
Notably, $\Crit$ will satisfy this stronger condition for many common choices of $\crit$, such as least squares or logistic loss.
}
Proposition \ref{prop:meas_V3_sieve} implies that
a mapping $\omega\mapsto\f$ from $\Omega\to\mathcal{G}$
exists and is measurable
whenever
$\crit(\Zt(\omega),\cdot):\R\to\R$
is continuous for all $\omega\in\Omega$,
and
$\measurableftns\subseteq \Lp{r}(\P_{\{\Zt\}_{t=1}^n})$
is a pointwise-separable class of functions such that
$\{f(\z):f\in\mathcal{G}\}\subset\R$
is compact for each $\z\in\R^{d_{\boldsymbol{W}}}$.
Proposition \ref{prop:meas_V3_sieve} is closely related to
\citet[Theorem 2.2]{wooldridge_white_1991}, which is commonly used to ensure the existence and measurability of sieve estimators (e.g., \citealp{chen_chapter_2007}).
Although \citet[Theorem 2.2]{wooldridge_white_1991} is applicable to metric spaces $(\mathcal{G},\rho)$ where $\rho$ may not be induced by an $\Lp{r}$-norm, they require that $(\mathcal{G},\rho)$ be a compact metric space.
Therefore,
Proposition \ref{prop:meas_V3_sieve}
adds generality by only requiring
$\mathcal{G}\subset
\Lp{r}(\P_{\{\Zt\}_{t=1}^n})$
such that
$\{f(\z):f\in\mathcal{G}\}\subset\R$
is compact for all $\z\in\R^{d_{\boldsymbol{W}}}$,
which does not imply that
$(\mathcal{G},\norm{\cdot}_{\Lp{r}(\P_{\{\Zt\}_{t=1}^n})})$
is a compact metric space.
This will be crucial when the sieve spaces are constructed using DNNs with unbounded weights as in Section \ref{sec:DNN}.
\subsection{Convergence rates without stationarity}\label{sec:ROC_NSD}
This subsection gives a convergence rate result for sieve estimators
applicable to very general estimation settings where $\{\Zt\}_{t\in\N}$ is possibly non-stationary.
The complexity of $\mathcal{G}$ will be controlled using a covering number as defined below.
$(\mathbb{M},\norm{\cdot})$
denotes
a semi-metric space,
induced by the norm $\norm{\cdot}$.
\begin{definition}\label{def:CoveringNumber}
\textbf{\textsc{(Covering Number)}}
Let
$\delta>0$,
and let
$(\delta_n,\norm{\cdot})$
be a semi-metric space.
\begin{enumerate}
\item
For $\mathcal{G}\subset \delta_n$,
the $\boldsymbol{\delta}$\textbf{-covering number} of $\mathcal{G}$,
denoted as $\cover(\delta,\mathcal{G},\norm{\cdot})$,
is the smallest $J\in\N$ such that there exists a collection
$
\{m_j\}_{j=1}^J
\subseteq
\delta_n,
$
where
$$
\mathcal{G}
\;\subseteq\;
\bigcup_{j=1}^J
\Big\{
g\in\mathcal{G}
:\,
\norm{g-m_j}<\delta
\Big\}
,
$$
and if no such $J\in\N$ exists let
$\cover(\delta,\mathcal{G},\norm{\cdot})=\infty$.
\item
When
$\delta_n$
is a space of
functions with elements $f:\R^{d_{\boldsymbol{W}}}\to \R$,
then, for any
$a\in\N$,
define
$
\delta_n|_{\{\Zt\}_{t=1}^a}
\coloneqq
\big\{
\big(f(\boldsymbol{W}_1), f(\boldsymbol{W}_2), \ldots, f(\boldsymbol{W}_a)\big)
:
f\in\delta_n
\big\}
.
$
For any
$r\geq 1$
let
$
\cover^{(\infty)}_{r}(\delta,\delta_n,a)
\coloneqq
\sup
\Big\{
\cover
\big(
\delta,\delta_n|_{\{\boldsymbol{W}_t(\omega)\}_{t=1}^a},\norm{\cdot}_{r,a}
\big)
:
\omega\in\Omega
\Big\}
.
$
\end{enumerate}
\end{definition}
For
$\mathcal{G}\subset\measurableftns$
and
a sample $\{\Zt\}_{t=1}^a$, note that
$
\cover
\big(
\delta,\mathcal{G}|_{\{\boldsymbol{W}_t\}_{t=1}^a},\norm{\cdot}_{r,a}
\big)
$
depends on $\omega\in\Omega$, but
$\cover^{(\infty)}_{r}(\delta,\mathcal{G},a)$
does not.
Following the usual convention, if $\mathcal{G}=\emptyset$ then
$\cover(\delta,\mathcal{G},\norm{\cdot})=1$, for any $\delta>0$.
Theorem \ref{thrm:ROC_V1} is the first main result of this section.
It provides a rate of convergence in probability using an approach similar to the consistency results in \cite{wooldridge_white_1991}.
For generality, the following will not impose the conditions of Proposition \ref{prop:meas_V3_sieve} to ensure the measurability of $\f$.
Instead, I will use outer probability, $\PrO$, as defined in \citet[\S1.2]{van_der_vaart_wellner_book_1996},
i.e., for an arbitrary set $B\subseteq\Omega$
$$
\PrO(B)= \inf\Big\{P(A):\; A\supseteq B,\; A\in\Zsig\Big\}.
$$
\begin{theorem}\label{thrm:ROC_V1}
For each $n\in\N$ let $(\measurableftns,\metric)$ be a (semi-) metric space.
Let $\mathcal{G}$
be a pointwise-separable class.
Suppose there exist
$\{\f\}_{n\in\N}$
satisfying \eqref{eq:fhat}, and $\{\err\}_{n\in\N}$ such that
$h_n = \Op(\err^2)$.
Then,
$\metric(\f,\fO)=\Opout(\err),$
if the following conditions hold:
\begin{enumerate}
[label=$\mathrm{(a.\arabic*)}$]
\item\label{RC:projection}
There exists a non-stochastic sequence
$\{f^{*}\}_{n\in\N}$
such that
$f^{*}\in\mathcal{G}$
and
$
\metric(f^{*},\fO)
\leq\err
$,
for all $n$.
\item\label{RC:pop_crit_quad}
There exist constants $\Cpcq,\CpcqB>0$ such that,
for any $n\in\N$ and
$f\in\mathcal{G}$,
$$
\Cpcq\,\metric(f,\fO)^2
\;\leq\;
\E\big[\Crit(f)\big] - \E\big[\Crit(\fO)\big]
\;\leq\;
\CpcqB\,\metric(f,\fO)^2
.
$$
\item\label{RC:crit_lip}
There exist
$
\mn: \R^{d_{\boldsymbol{W}}} \to [0,\infty),
$
measurable-$\borel(\R^{d_{\boldsymbol{W}}})/\borel([0,\infty))$,
and a positive, non-decreasing sequence
$\{\Bound\}_{n\in\N}$,
such that, for each $n\in\N$,
\begin{enumerate}
[label=$\mathrm{(\roman*)}$]
\item
for any
$f,f'\in\mathcal{G}$
and $\z\in\R^{d_{\boldsymbol{W}}}$
$$
\big| \crit\big(\z,f(\z)\big) - \crit\big(\z,f'(\z)\big) \big|
\leq
\mn(\z) \big|f(\z)-f'(\z)\big|
;
$$
\item
$
\lim_{n\to\infty}
\Pr\left(
\max_{t\in\{1,\ldots,n\}}
\mn(\Zt)
\geq
\Bound
\right)
=0
;
$
and
\item
for some $\Cui>0$ not depending on $n$,
$$
\sup\bigg\{
\E\Big[
\big|\crit\big(\Zt,f(\Zt)\big)\big|
\mathbbm{1}_{\{\mn(\Zt)\geq \Bound\}}
\Big]
:\,
f\in\mathcal{G},
\,
t\in\{1,...,n\}
\bigg\}
\leq
\Cui\,\err^2
.
$$
\end{enumerate}
\item\label{RC:prob_bound}
{
There exists
$
\Pboundl :
(0,\infty) \to (0,\infty)
$
such that,
\begin{enumerate}
[label=$\mathrm{(\roman*)}$]
\item
for any $\delta_n>0$, and
$f\in \mathcal{G}$,
\begin{equation*}
\resizebox{\linewidth}{!}{$
\begin{aligned}
\Pr\left(
\frac{1}{n}
\bigg|
\sum_{t=1}^{n} \Big(
\crit\big(\Zt,f(\Zt)\big)
\mathbbm{1}_{\{\mnZt < \Bound\}}
-
\E\big[
\crit\big(\Zt,f(\Zt)\big)
\mathbbm{1}_{\{\mnZt < \Bound\}}
\big]
\Big)
\bigg|
\geq
\delta_n
\right)
\lesssim
\Pboundl(\delta_n)
;
\end{aligned}
$}
\end{equation*}
\item
and, for any fixed $\delta_n>0$ sufficiently large,
\footnote{
$\mathcal{G}$ in the covering number may be replaced with
$
\left\{f\in\mathcal{G}:\, \err \sqrt{(\delta_n^2+\CpcqB+4\Cui)/\Cpcq} \leq \metric(f,\fO)\right\}
\subseteq
\mathcal{G}
.
$
}
\begin{equation*}
\begin{aligned}
\lim_{n\to\infty}
\left\{
\Pboundl
\hspace{-3pt}
\left(\delta_n\,\err^2\right)
\cdot
\cover^{(\infty)}_{1}
\hspace{-3pt}
\left(
\delta_n\,\err^2/\Bound
,\,
\mathcal{G}
,\,
n
\right)
\right\}
= 0
.
\end{aligned}
\end{equation*}
\end{enumerate}
}
\end{enumerate}
\end{theorem}
\vx
Condition \ref{RC:pop_crit_quad} is a curvature condition on $\crit$ near $\fO$ that is standard in nonparametric estimation (see e.g., \citealp{chen_sieve_1998}; \citealp{chen_chapter_2007}; \citealp{farrell_deep_2021}).
Condition \ref{RC:crit_lip}(i) is a Lipschitz condition on $\crit$.
In many estimation settings, these requirements are met by common choices of $\crit$, such as least squares or logistic regression (\citealp{farrell_deep_2021}).
Conditions
\ref{RC:crit_lip}(ii) and \ref{RC:crit_lip}(iii) are requirements on the tail behavior of $\mn$ and $\crit$, which are often satisfied when the extrema of $\{\Zt\}_{t=1}^n$, and $\mathcal{G}$, grow sufficiently slowly with $n$.
In many cases \ref{RC:crit_lip}(ii) will imply \ref{RC:crit_lip}(iii). This will be demonstrated in Subsection \ref{sec:DNN_prop}, with the use of Lemma \ref{lem:unif_int}.
The regularity conditions imposed by \ref{RC:crit_lip} are more general than those typically used in the nonparametric sieve estimation literature.
For instance, \cite{farrell_deep_2021}
requires that $\R^{d_{\boldsymbol{W}}}$ be compact,
and $\crit$ satisfy a Lipschitz condition like \ref{RC:crit_lip}(i), except $\mn$ must be a constant.
Condition A.4 of \cite{chen_sieve_1998}
requires that there exist
$s\in(0,2)$, $\gamma>4$, and $C>0$ such that,
for any $\delta>0$,
$
\sup_{\{f\in\mathcal{G}:\metric(f,g)\leq\delta\}}
| \crit\big(\z,f(\z)\big) - \crit\big(\z,h(\z)\big) |
\leq
\delta^s\,\mnZt,
$
and
$\sup_n\E[\mnZt^\gamma]<C$.
In either case, these conditions imply \ref{RC:crit_lip}.
Letting $\metric$ vary with $n$ is often necessary
in settings where $\{\Zt\}_{t\in\N}$ is non-stationary.
For instance, when $\metric$ is the metric induced by
$\norm{\cdot}_{\Lp{2}(\PrO_{\{\Zt\}_{t=1}^n})}$
condition
\ref{RC:pop_crit_quad} is easily verified in many estimation settings (see
Appendix \ref{sec:PROOF_DNN_ROC_V1}).
Note that condition \ref{RC:pop_crit_quad} will often not hold
for fixed
$\metric = \rho$,
such as
$
\metric=
\rho=
\norm{\cdot}_{\Lp{2}(\PrO_{\{\Zt\}_{t=1}^\infty})}
$
for all $n,$
when $\{\Zt\}_{t\in\N}$ is non-stationary.
Condition \ref{RC:prob_bound}(i) can often be
met by using an exponential inequality to obtain $\Pboundl$,
such as
\citet[Theorem 3.4]{wooldridge_white_1991},
or
\citet[Theorem 1]{merlevede_bernstein_2009}.
These inequalities can be applied without requiring $\crit$ to be bounded,
due to \ref{RC:crit_lip}(i) and the inclusion of $\mathbbm{1}_{\{\mnZt < \Bound\}}$ in \ref{RC:prob_bound}(i),
provided that $\mathcal{G}$ satisfies a boundedness condition, such as
$
\sup_{f\in\mathcal{G}}\norm{f}_{\infty}\lesssim \Bound.
$
\subsection{Nonasymptotic error bounds with stationarity and $\beta$-mixing}\label{sec:EB_SD}
This section gives finite sample error bounds for sieve estimators in a setting where $\{\Zt\}_{t\in\N}$ is strictly stationary and $\beta$-mixing.
When $\{\Zt\}_{t\in\N}$ is stationary let $\Pz=\P_{\Zt}$ for all $t\in\N$.
The
following definition for the $\beta$-mixing coefficient is from
\citet[Definition 3.1, p.19]{dehling_empirical_2002},
and is equivalent to many standard definitions.
\begin{definition}\label{def:beta_mixing}
\textbf{\textsc{($\boldsymbol{\beta}$-Mixing)}}
The $\beta$-mixing coefficient is defined as,
$$
\beta(j)
\coloneqq
\E\Bigg[
\sup\bigg\{
\Big|
\P\big(
B|\,
\sigma\big(\{\Zt\}_{1}^k\big)
\big)
-
\P\big(
B
\big)
\Big|
:\,
{B\in \sigma\left(\{\Zt\}_{k+j}^\infty\right)}
,\,
k\in\N
\bigg\}
\Bigg]
\quad
\text{ for }
j\in\N
.
$$
$\{\Zt\}_{t\in\mathbb{N}}$ is $\beta$-mixing
(or absolutely regular)
if
$
\lim_{j\to\infty}\beta(j)=0
$.
\end{definition}
Conditions that ensure $\beta$-mixing properties of stochastic sequences have been heavily studied
(e.g.,
\citealp{doukhan_mixing_1994};
\citealp{bradley_basic_2005}), and there are many examples of interesting
processes that are $\beta$-mixing with exponentially decreasing coefficients,
also referred to as geometric $\beta$-mixing.
For instance, conditions to ensure stationarity and geometric $\beta$-mixing for ARMA processes are given by
\citet{mokkadem_mixing_1988}
and for GARCH processes by
\citet{carrasco_mixing_2002}.
The following corollary provides an example for nonlinear ARCH processes, using
\citet[Theorem 1]{chen_geometric_2000}
with the discussion from
\citet[pp. 292,293]{dinh_tuan_mixing_1986}.
\begin{corollary}[\citealp{chen_geometric_2000}]\label{cor:beta_mix}
Consider model \eqref{eq:ex1}
with
$\Xt=(Y_{t-1},\ldots,Y_{t-d})^\tr$.
Given initial conditions $X_0=(Y_{0},\ldots,Y_{1-d})^\tr\in\R^d$, if
\begin{enumerate}
\item
$\{\upsilon_t\}$ is i.i.d. with a strictly positive and continuous density, such that $\E[\upsilon_t]=0$ and $\upsilon_t$ is independent of $X_{t-j}$ for all $j\in\N$;
\item
the function
$\fO$
is uniformly bounded;
\item
there exists a constant $c$ such that for all $\delta\geq0$,
$$
0<
c
\leq
\inf_{\x\in\R^d:\norm{\x}_{2,d}\leq\delta}
\eta(\x)
\leq
\sup_{\x\in\R^d:\norm{\x}_{2,d}\leq\delta}
\eta(\x)
<\infty;
$$
\item
there exist constants
$C>0$,
$c^{(f)}_{j}\geq0$,
and
$c^{(\eta)}_{j}\geq0$,
for $j=0,\ldots,d$
such that
$$
|\fO(\x)|
\leq
c^{(f)}_{0}
+
\sum_{j=1}^d
c^{(f)}_{j}
|x_j|
,
\quad
\eta(\x)
\leq
c^{(\eta)}_{0}
+
\sum_{j=1}^d
c^{(\eta)}_{j}
|x_j|
,
\quad
\forall
\x\in\R^d:\norm{\x}_{2,d}\leq C,
$$
and
$
\sum_{j=1}^d
\big\{
c^{(f)}_{j}
+
c^{(\eta)}_{j}
|\upsilon_j|
\big\}
<
1;
$
\end{enumerate}
then,
$\{\Zt\}_{t\in\N}$ is strictly stationary and $\beta$-mixing such that there exist constants $\Cbeta',\Cbeta>0$ where
$\beta(j)\leq \Cbeta'e^{-\Cbeta j}.$
\end{corollary}
This section's main result will use the pseudo-dimension to control the complexity of the sieve spaces $\mathcal{G}$.
The following definition is from \citet[Definition 2]{bartlett_nearly_tight_2019}.
\begin{definition}\label{def:pseudodimension}
\textbf{\textsc{(Pseudo-dimension)}}
Let $\mathcal{S}$ be a class of functions from $\mathbb{X}\rightarrow \R$.
The pseudo-dimension of $\mathcal{S}$, denoted as $\mathrm{Pdim}(\mathcal{S})$,
is the largest $p\in\N$ for which there exists
$(x_1,\ldots,x_p,y_1,\ldots,y_p)\in \mathbb{X}^p \times \R^p$
such that for any
$(b_1,\ldots,b_p)\in \{0,1\}^p$
there exists $s\in\mathcal{S}$ such that
$
\mathbbm{1}{\{s(x_i)- y_i >0\}}
= b_i,
$
for all $i=1,\ldots,p.$
\end{definition}
Pseudo-dimension, along with related complexity measures like Vapnik–Chervonenkis dimension, (see e.g. \citealp[Definition 1]{bartlett_nearly_tight_2019})
is often described as `scale insensitive' because, unlike the covering number, it does not depend on a specific threshold $\delta>0$.
Scale-insensitive complexity measures are particularly well-suited for function classes constructed with DNNs that have unbounded parameters,
for which
\citet{bartlett_nearly_tight_2019}
provides nearly tight pseudo-dimension bounds.
Pseudo-dimension can also be used to bound the covering number with \citet[Theorem 12.2]{anthony_bartlett_neural_1999} (also see Lemma \ref{lem:entropy_bound} herein).
Theorem \ref{thrm:ROC_V2}
extends
\citet[Theorem 2]{farrell_deep_2021} to general nonparametric sieve estimators in settings with dependent data that may take values in unbounded sets.
This extension relies on stationarity and $\beta$-mixing to apply a standard independent blocking technique,
and an exponential inequality from \cite{merlevede_bernstein_2009}.
Following \citet{farrell_deep_2021}, the proof of Theorem \ref{thrm:ROC_V2} differs from classic sieve analysis
(e.g., \citealp{wooldridge_white_1991};
\citealp[\S3.4]{van_der_vaart_wellner_book_1996};
\citealp{chen_sieve_1998})
by using a localization approach with scale-insensitive measures of complexity to offer straightforward applicability to DNN estimators that have unbounded parameters (see \citealp{farrell_deep_2021} for more discussion).
{
\begin{theorem}\label{thrm:ROC_V2}
Let $\{\Zt\}_{t\in\N}$ be strictly stationary,
$(\measurableftns,\norm{\cdot}_{\Lp{2}(\Pz)})$ be a (semi-) metric space,
$\crit(\Zt(\omega),\cdot):\R\to\R$
be continuous for all $\omega\in\Omega$,
and
$\mathcal{G}$
be a pointwise-separable class
such that
$\{f(\z):f\in\mathcal{G}\}\subset\R$
is compact for each $\z\in\R^{d_{\boldsymbol{W}}}$ and $n\in\N$.
Suppose
$\norm*{\fO}_{\infty}\leq 1$,
and
the following conditions hold:
\begin{enumerate}
[label=$\mathrm{(b.\arabic*)}$]
\item\label{CR2:mixing}
$\{\Zt\}_{t\in\N}$ is $\beta$-mixing with
$\beta(j)\leq \Cbeta'e^{-\Cbeta\,j } $
for some $\Cbeta,\Cbeta'>0$.
\item\label{CR2:projection}
For each $n\in\N$,
there exists a non-stochastic
$f^{*}\in\mathcal{G}$
where
$
\tilde{\epsilon}_n
\coloneqq
\norm*{f^{*}-\fO}_{\infty}
$
is such that
$
\lim_{n\to\infty}
\tilde{\epsilon}_n(\log n)(\log\log n)
=0
.
$
\item \label{CR2:pop_crit_quad}
There exist constants $\Cpcq,\CpcqB>0$ such that, $\Cpcq\leq 1$, and
for any $n$,
$f\in\mathcal{G}$,
$$
\Cpcq\norm*{f-\fO}^2_{\Lp{2}(\Pz)}
\;\leq\;
\E\big[\Crit(f)\big] - \E\big[\Crit(\fO)\big]
\;\leq\;
\CpcqB\norm*{f-\fO}^2_{\Lp{2}(\Pz)}
.
$$
\item\label{CR2:Pseudo}
$\mathcal{G}$ is
such that
$\Pdim(\mathcal{G})\geq 1$ for all $n$,
and
for some non-decreasing
sequence
$\{B\}_{n\in\N}$
such that
$B_1\geq 2$,
$
\sup_{f\in\mathcal{G}}
\norm{f}_{\infty}
\leq
B
<
\infty
$
for each $n$,
and
$$
\lim_{n\to\infty}
\frac{B}{\sqrt{n}}
\Big[
\sqrt{
{ \Pdim(\mathcal{G})}
\log(n)
}
+
\sqrt{
{\log\log(n)}
}
\Big]
=0
.
$$
\item \label{CR2:crit_lip}
There exists
$
\mn: \R^{d_{\boldsymbol{W}}} \to [0,\infty),
$
measurable-$\borel(\R^{d_{\boldsymbol{W}}})/\borel([0,\infty))$,
such that, for each $n$,
\begin{enumerate}
[label={(\roman*)}]
\item
for any
$f,f'\in\mathcal{G}$,
$\z\in\R^{d_{\boldsymbol{W}}}$
$$
\big| \crit\big(\z,f(\z)\big) - \crit\big(\z,f'(\z)\big) \big|
\leq
\mn(\z) \big|f(\z)-f'(\z)\big|
;
$$
and
\item
there exists a constant
$\Cuib\geq1$
and
a strictly positive sequence $\{\trunc\}_{n\in\N}$ such that
$$
\min\left\{
B
\E\big[
\mnZt\mathbbm{1}_{\{\mnZt>\CuibB\}}
\big]
\, ,\;
\sup_{f\in\left\{\mathcal{G}\cup\{\fO\}\right\}}
\E\Big[
\big|\crit\big(\Zt,f(\Zt)\big)\big|
\mathbbm{1}_{\{\mn(\Zt)\geq \CuibB\}}
\Big]
\right\}
\leq
\trunc
,
$$
and
$
\lim_{n\to\infty} \trunc=0
.
$
\end{enumerate}
\end{enumerate}
Then, there exists a constant $C>0$, depending only on
$\Cpcq$,
$\CpcqB$,
$\Cuib$,
$\Cbeta$,
and
$\Cbeta'$,
such that for any
$
n
\geq
\max\left\{
5
\,,\;
2\Pdim(\mathcal{G})
\,,\;
{16B^2}/{\log(n)}
\right\}
,
$
measurable mapping $\omega\mapsto\f$ satisfying \eqref{eq:fhat},
and constants
$n\in\N$,
$\delta>0$
where
\begin{gather*}
\max\left\{
2
\;,\;
\frac{\tilde{\epsilon}_n(\log n)(\log\log n)}{B}
\right\}
<
n
\leq
n/2
,
\quad
\text{and}
\quad
\sqrt{\delta}
\geq
\frac{
\tilde{\epsilon}_n\sqrt{n}
}{
\DNNBoundn
-
\tilde{\epsilon}_n{(\log n)(\log\log n)}
},
\end{gather*}
the following holds
\begin{equation*}
\resizebox{\hsize}{!}{$
\begin{aligned}
&
\P\Big(
\|\f-\fO\|_{\Lp{2}(\P_{\boldsymbol{W}})}
\leq
C\,
\err(\delta,n)
\Big)
\geq
1-
e^{-\delta}
-
2\log(n)
\left[
\frac{n\,\beta(n)}{n}
+
2\Pr\Big(
\max_{t\in\{1,\ldots,n\}}
\mn(\Zt)\geq\CuibB
\Big)
\right]
,
\quad \text{and}
\\
&
\Pr\left(
\norm*{\f-\fO}_{2,n}
\leq
C\,
\err(\delta,n)
\right)
\geq
1-
4e^{-\delta}
-
6\log(n)
\left[
\frac{n\,\beta(n)}{n}
+
2\Pr\Big(
\max_{t\in\{1,\ldots,n\}}\mnZt>\CuibB
\Big)
\right]
\end{aligned}
$}
\end{equation*}
for
$\,\err(\delta,n)
\coloneqq
B
\sqrt{\frac{n}{n}}
\Big[
\sqrt{
\Pdim(\mathcal{G})
\log(n)
}
+
\sqrt{
\log\log(n)+\delta
}
\Big]
+
\sqrt{
\tilde{\epsilon}_n^2
+
\trunc
+
h_n
}
.$
\end{theorem}
The existence of a measurable mapping
$\omega\to\f$
is not an extra assumption,
as it will follow from
Proposition \ref{prop:meas_V3_sieve}
and the assumptions in the statement of Theorem \ref{thrm:ROC_V2}.
Therefore it is not necessary to use outer probabilities/integrals when dealing with $\f$.
The requirement that
$
n
>
\frac{\tilde{\epsilon}_n(\log n)(\log\log n)}{B}
$
will not be binding for large $n$. This follows because \ref{CR2:projection} and \ref{CR2:Pseudo} imply
$
\lim_{n\to\infty}
\frac{\tilde{\epsilon}_n(\log n)(\log\log n)}{B}
=0
.
$
Note that $n\geq16B^2/\log(n)$ and $n\geq 2\Pdim(\mathcal{G})$ for all $n$ sufficiently large is implied by \ref{CR2:Pseudo}.
Requiring that $\Cuib\geq 1$ and $\Cpcq\leq 1$ is also without loss of generality. To see this, if \ref{CR2:crit_lip}(ii),
or \ref{CR2:pop_crit_quad} hold with $\Cuib< 1$ or $\Cpcq> 1$, respectively, then they will also hold with the constant replaced by one.
However, this requirement can be dropped if
$n$ is large enough
or
${(4\cdot288) \Cuib }
/{\Cpcq}
\geq
\sqrt{38}$
(see first paragraph of Appendix \ref{sec:step4} for more detials).
Conditions
\ref{CR2:pop_crit_quad} and
\ref{CR2:crit_lip}(i)
are analogous to
Conditions
\ref{RC:pop_crit_quad}
\ref{RC:crit_lip}(i),
and the discussion following Theorem \ref{thrm:ROC_V1} still applies.
Condition \ref{CR2:crit_lip}(ii) is related to \ref{RC:crit_lip}(iii), but Condition \ref{CR2:crit_lip}(ii) is somewhat more general since the uniform integrability type requirement on $\crit$ can be replaced with
$
\lim_{n\to\infty}
B
\mnZt\mathbbm{1}_{\{\mnZt>\CuibB\}} = 0
$.
Although Theorem \ref{thrm:ROC_V2} does not explicitly require
$
\lim_{n\to\infty}
\Pr\big(
\max_{t\in\{1,\ldots,n\}}\mnZt>\CuibB
\big)
=
0
$,
like Condition \ref{RC:crit_lip}(iii), this will be necessary for the error bounds to hold with probability approaching one.
In light of this, the requirement that $\crit(\Zt(\omega),\cdot):\R\to\R$
be continuous for all $\omega\in\Omega$,
imposes very little additional structure, since Condition \ref{CR2:crit_lip}(i) will imply this with probability approaching one whenever
$
\lim_{n\to\infty}
\Pr\big(
\max_{t\in\{1,\ldots,n\}}\mnZt>\CuibB
\big)
=
0
.
$
Theorem \ref{thrm:ROC_V2} may also imply a rate of convergence by choosing
$\delta=\delta_n$ and $n=a_n$ such that $\delta_n$ and $a_n$ go to infinity with $n$ at an appropriate rate.
For this, note that for any $\{a_n\}_{n\in\N}$ such that $\lim_{n\to\infty}a_n/\log(n)=\infty$ then
$\textstyle \log(n)\frac{n\,\beta(a_n)}{a_n}=0$.
To see this, choose $b_n=a_n\Cbeta \log^{-1}(n)$, so by Condition \ref{CR2:mixing}
$$
\frac{\log(n)n\,\beta(a_n)}{a_n}
\lesssim
\frac{\log(n)ne^{-\Cbeta a_n}}{a_n}
=
\frac{\log(n)n^{1-b_n}}{a_n}
=
\frac{\log(n)n^{1-\frac{a_n\Cbeta}{\log(n)} }}{a_n}
.
$$
}
\section{DNN estimators}\label{sec:DNN}
This section applies the results of Section \ref{sec:gen_res} to obtain theoretical properties for a class of DNN estimators commonly used in applications.
The proofs for this section are included in Appendix \ref{sec:PROOF_SEC_DNN}.
Throughout this section,
$\{\Zt=(\Yt,\Xt^{\tr})^{\tr}\}_{t\in\mathbb{N}}$
will be a stochastic sequence on the complete probability space $\probspace$,
such that,
$\Yt\in\R$
and
$\Xt\in\R^{d_{\boldsymbol{W}}}$
for each $t$.
The object to be estimated is $\fO:\R^{d_{\boldsymbol{W}}}\to[-1,1]$, which
satisfies
the following H\"older smoothness assumption.
\begin{assumption}\label{as:smoothness}
\textbf{\textsc{(Smoothness)}}
For a smoothness parameter
$\smooth \in \mathbb{N}$,
and each multi-index
$\kb \in
\big\{\N\cup\{0\}\big\}^d
$
with
$ \sum_{j=1}^d \k_j \leq \smooth-1$,
the regression function
$\fO:\R^{d_{\boldsymbol{W}}}\to\R$
is such that
$D^{\kb}\fO$
is continuous,
and
$
\norm*{D^{\kb}\fO}_\infty
\leq 1.
$
\end{assumption}
This type of smoothness assumption is standard in the nonparametric estimation literature (e.g. \citealp{stone_optimal_1982}; \citealp{chen_sieve_1998}; \citealp{chen_chapter_2007}; \citealp{chen_optimal_2015}; \citealp{farrell_deep_2021}),
and
\cite{yarotsky_error_2017}
shows that
DNNs approximate these functions well.
Note that this assumption is weaker than the structural assumptions imposed in much of the DNN literature (e.g.,
\citealp{kohler_nonparametric_2017};
\citealp{schmidt_hieber_nonparametric_2020};
\citealp{kohler_rate_2021}
).
For most of the results in this section,
$\beta$-mixing will be stronger than necessary.
In these cases, I use $\alpha$-mixing,
as defined by \citealp[Definition 3.1]{dehling_empirical_2002}.
\begin{definition}\label{def:alpha_mixing}
\textbf{\textsc{($\boldsymbol{\alpha}$-Mixing)}}
The $\alpha$-mixing coefficient is defined as,
$$
\alpha(j)
\coloneqq
\sup\bigg\{
\Big|
\P\big(A\cap B\big)
-
\P(A)\P(B)
\Big|
:\,
A\in \sigma\big(\{\Zt\}_{1}^k\big)
,\,
B\in \sigma\left(\{\Zt\}_{k+j}^\infty\right)
,\,
k\in\N
\bigg\}
\;\;
\text{ for }
j\in\N.
$$
$\{\Zt\}_{t\in\mathbb{N}}$ is $\alpha$-mixing
(or strong mixing)
if
$
\lim_{j\to\infty}\alpha(j)=0
$.
\end{definition}
Sufficient conditions for a process to be $\alpha$-mixing have been heavily studied
(e.g. \citealp[\S 2.4]{doukhan_mixing_1994}; or \citealp{bradley_basic_2005}),
and
an example is included in Corollary \ref{cor:log_alpha_con} of Subsection \ref{sec:DNN_log}.
It is well known that $\alpha(j)\leq\beta(j)$, so any $\beta$-mixing process is also $\alpha$-mixing with a rate at least as fast. Consequently, Corollary \ref{cor:beta_mix} can also be used to imply $\alpha$-mixing.
\subsection{DNN sieve spaces $(\mathcal{G})$}\label{sec:DNN_sieve_spaces}
\begin{figure}[t]
\centering
\includegraphics[width =0.5\textwidth, height = 4cm]{MLP_v1_2024_08_25.jpg}
\caption{Example of $\mathcal{G}$ architecture graph structure where
$\L=2$, $\Hi{1} =3$, $\Hi{2}=2$, $\W=20$, and $d=2$.}
\label{fig:DNN_MLP}
\end{figure}
The sieve spaces used here, $\mathcal{G}$, will be constructed using fully connected feedforward DNNs.
For concreteness, I focus on DNNs
equipped with the ReLU activation function
$\relu(x) = \max\{0,x\}$.
However, Corollary \ref{prop:nonReLU} shows that all results in the following subsections apply for any continuous piecewise-linear activation function.
Considerations for other feedforward architectures and activation functions are also discussed in Subsection \ref{sec:DNN_Alt_Arch}.
For $n\in\mathbb{N}$ the graph structure of the architecture $\mathcal{G}$ is characterized by its depth $\L\in\mathbb{N}$, and width vector $\Hb=[H_{n,0},H_{n,1},\ldots,H_{n,\L}] \in \N^{\L+1}$ where $H_{n,0}=d$ for all $n$.
Each hidden layer, $l\in\{1,\ldots,\L\}$, is comprised of $H_{n,l}$ ``hidden'' computational units, referred to as \textit{nodes}, and denoted as $\n_{l,h}$.
The $d$ inputs
$\x=[x_1,\ldots,x_d]'\in\R^{d_{\boldsymbol{W}}}$
are fed into each node in the first hidden layer $l=1$.
Then each node in layer $l=1$ feeds into each node of the next layer $l=2$.
This process repeats with each node in layer $l-1$ passing its output into each node in layer $l$ up to the last hidden layer $l=\L$.
Finally, each node in the last hidden layer, $l=\L$, feeds into the output layer of the network, $l=\L+1$, which consists of only one computation unit.
Note that nodes in layer $l$ receive inputs only from nodes in layer $l-1$, and none from layers $l-2$ or earlier.
See Figure \ref{fig:DNN_MLP} for an example of an $\mathcal{G}$ architecture.
Each node is a function taking values in $\R$ which depends on a real-valued
vector of parameters
$\gv_{l,h}$,
and takes as arguments the outputs of all nodes in the previous layer, which ultimately depend on the original input $\x$.
The parameters for each node consist of a
scalar intercept term
$\gamma_{l,h,0}$,
\footnote{The machine learning literature often refers to $\gamma_{l,h,0}$ as the bias, or threshold. To avoid confusion with similarly named objects in the econometrics literature I refer to this as the intercept term.}
and weights
$\gamma_{l,h,1},\ldots,\gamma_{l,h,H_{n,l-1}}$
such that
$$\gv_{l,h} = [\gamma_{l,h,0},\gamma_{l,h,1},\ldots,\gamma_{l,h,H_{n,l-1}}]^\tr \in \R^{H_{n,l-1}+1}.$$
The parameters for the entire network are collected together as
\begin{equation*}
\gV
\coloneqq
\{\g_{l,h,k}\}_{\forall\, l,h,k}
\in
\R^{\W},
\end{equation*}
where $\W$ is the total number of parameters in the network.
For a set of parameters, the output of node $h$, in layer $l$, is denoted as
$\n_{l,h}(\x\;|\; \gV)$
and the single node in the last layer ($l=\L+1$) is denoted as $\n_{\L+1,1}(\x\;|\; \gV)$
where the dependence on
$\gV$
will often be suppressed.
Then each node can be written as
\begin{equation}\label{eq:nodes}
\n_{l,h}\left(\x \;|\; \gV \right) \coloneqq \left\{
\begin{aligned}
&
\relu\bigg(
\sum_{i=1}^{d} \g_{1,h,i} \cdot x_i + \g_{1,h,0}
\bigg),
& l=1,
\\&
\relu\bigg(
\sum_{i=1}^{H_{n,l-1}} \g_{l,h,i} \cdot \n_{l-1,i}(\x) + \g_{l,h,0}
\bigg),
& 2\leq l \leq \L,
\\&
\sum_{i=1}^{H_{n,\L}} \g_{\L+1,1,i} \cdot \n_{\L,i}(\x) + \g_{\L+1,1,0},
& l = \L + 1.
\end{aligned}
\right.
\end{equation}
With this notation, $\n_{\L+1,1}: \R^{d_{\boldsymbol{W}}} \times \R^{\W} \rightarrow \R$. Then, define the architecture $\mathcal{G}$ as
\begin{equation}\label{eq:sievespace}
\resizebox{0.92\hsize}{!}{$
\mathcal{G}
=
\FMLPnon(\L,\Hb,B)
\coloneqq
\Big\{
f=
\n_{\L+1,1}\big(\cdot \,|\, \gV\big):
\;
\gV \in \R^{\W},
\;
\sup_{\x\in\R^{d_{\boldsymbol{W}}}}|\n_{\L+1,1}\big(\x \;|\; \gV\big)|
\leq B
\Big\}.
$}
\end{equation}
A particular network $f\in \mathcal{G}$ is determined by $\gV$ and can be written as
$f(\cdot) = \n_{\L+1,1}\left(\cdot \;|\; \gV\right):\R^{d_{\boldsymbol{W}}}\to[-B,B]$.
\begin{remark}\label{rem:FMLP}
$\mathcal{G}$ as in \eqref{eq:sievespace} has the following useful properties:
\begin{enumerate}
\item
The functions
$\n_{\L+1,1}: \R^{d_{\boldsymbol{W}}} \times \R^{\W} \rightarrow \R$
are continuous. This follows because for each $l\in \{1,\ldots,\L+1\}$, $h\in\{1,\ldots,H_{n,l}\}$ the function $\n_{l,h}$ is composed of compositions of the ReLU activation function ($\relu$) and linear combinations of $\gv_{l,h}$ and $\{\n_{l-1,h}\}_{h=1}^{H_{n,l-1}}$ (or $\x\in\R^{d_{\boldsymbol{W}}}$ when $l=1$).
\item
$\mathcal{G}$ is a pointwise separable class. To see this, let $\Rat$ denote the rationals and define
$\mathcal{G}^\Rat\coloneqq\big\{f\in\mathcal{G}: \gV \in \Rat^{\W} \big\}$.
Then,
$\mathcal{G}^{\Rat}$
is a countable dense subset of $\mathcal{G}$, since
$
\n_{\L+1,1}(\x|\;\cdot\;):\R^{\W}\to
\R
$
is continuous for each $\x\in\R^{d_{\boldsymbol{W}}}$, and $\Rat$ is a countable dense subset of $\R$ (for the standard topology on $\R$).
\item
Note that for each $n$ and $\x\in\R^{d_{\boldsymbol{W}}}$
$$
\{f(\x):f\in\mathcal{G}\}
=
[-B,B]
,
$$
which is compact.
To see this, consider the subset of $\mathcal{G}$ where all parameters are equal to zero except for the intercept term in the output node,
i.e.
\begin{equation*}
\mathcal{G}^*
\coloneqq
\Big\{
f\in\mathcal{G}:
\g_{\L+1,1,0}
\in[-B,B]
,
\;\;\text{ and }\;\;
\g_{l,h,k}=0
\;\; \forall
\{l,h,k\}\neq \{\L+1,1,0\}
\Big\}.
\end{equation*}
Clearly
$f^*(\x)=\g_{\L+1,1,0}$
for any
$f^*\in\mathcal{G}^*$, $\x\in\R^{d_{\boldsymbol{W}}}$.
Thus,
$
\{f^*(\x):f^*\in\mathcal{G}^*\}
=
[-B,B]
$
for each $\x\in\R^{d_{\boldsymbol{W}}}$.
Then the result follows since
$\mathcal{G}^*\subset\mathcal{G}$
and
$
\sup_{f\in\mathcal{G}}
\snorm{f}=B
$
by \eqref{eq:sievespace}.
\end{enumerate}
\end{remark}
In what follows, the mappings $\omega\mapsto\f$ from $\Omega$ to $\mathcal{G}$ will be considered measurable.
This will follow using Remark \ref{rem:FMLP} since Proposition \ref{prop:meas_V3_sieve} will be applicable in the estimation settings considered throughout this section.
\subsection{DNNs for nonparametric regression}\label{sec:DNN_prop}
I consider a general nonparametric regression estimation setting, that includes Example \ref{ex:reg} as a special case.
Theorem \ref{thrm:DNN_ROC_V1} will apply
Theorem \ref{thrm:ROC_V1} to show the convergence of DNN estimators in a very general setting without assuming stationarity.
Theorem \ref{thrm:DNN_ROC_V2} will apply
Theorem \ref{thrm:ROC_V2} to obtain
non-asymptotic probability bounds on the empirical and theoretical error of DNN estimators with stationary $\beta$-mixing data.
The goal is to estimate the function
$\fO=\E[\Yt|\Xt]$,
with a DNN sieve estimator $\f$ as in \eqref{eq:fhat}, using the least squares criterion
$$
\crit(\Zt,f)
\coloneqq
(\Yt-f(\Xt))^2
,
$$
and the DNN sieve spaces
$
\mathcal{G}
=
\big\{
f(\pi_{X}(\cdot))
:
f\in\mathcal{G}
\big\}
.
$
The regressands
$\{\Yt\}_{t\in\N}$
will be assumed to satisfy the following conditions.
\begin{assumption}\label{as:data}
For all $t\in\N$,
suppose
$\Yt\in\Lp{2}\probspace$
such that
there exists a non-decreasing sequence
$\{B\}_{n\in\N}$
with
$
B_1
\geq 2,
$
where
$$
\lim_{n\to\infty}
\Pr\left(\,
\max_{t\in\{1,\ldots,n\}}
|\Yt|\geq B
\right)
=0
.
$$
\end{assumption}
This assumption will be the key to ensuring Conditions \ref{RC:crit_lip}(ii)(iii) and \ref{CR2:crit_lip}(ii) hold
in this setting.
The two main results of this section will also specify some additional conditions to
control the dependence of $\{\Zt\}_{t\in\N}$ and
ensure $B$ grows sufficiently slowly.
In both cases, these requirements will be quite general for nonparametric estimation
as they will
hold without $\Yt$ taking values on a compact set
(\citealp[Assumption 1]{farrell_deep_2021}),
and without
$\E[\Yt^2|\{\Xt,\Y_{t-1}\}_{t=1}^n]$ being uniformly bounded
or
$\E[\Yt^{2+\delta}]<\infty$ for some $\delta>0$
(\citealp[Assumption 2]{chen_optimal_2015}).
Assumption \ref{as:data} can be verified
using results from extreme value theory for a wide variety of
$\{\Yt\}_{t\in\N}$
that are not uniformly bounded almost surely.
The following proposition gives two examples that use \citet[Theorem 3.4.1]{leadbetter_extremes_1983}, and \citet[Theorem 6.3.4]{leadbetter_extremes_1983}.
See Appendix \ref{sec:PROOF_SEC_DNN} for a proof showing these results can be applied.
\begin{proposition}
\label{cor:ext_val}
Suppose $\{\Yt\}_{t\in\N}$ is $\alpha$-mixing and one of the following holds:
\begin{enumerate}
\item
$\{\Yt\}_{t\in\N}$ is stationary,
and
$
\lim_{n\to\infty}
n
\P(|\Y_1|>B)
=
0
$;
\footnote{
Note that $\alpha$-mixing is stronger than necessary and could be replaced with
\citet[Condition $D(u_n)$, p.53]{leadbetter_extremes_1983} which is implied by $\alpha$-mixing
(see discussion on p.54 therein).
}
or
\item
$\{\Yt\}_{t\in\N}$ is
(possibly nonstationary)
such that for each $n\in\N$ the joint distribution of
$\{\Yt\}_{t=1}^n$
is an $n$-dimensional normal distribution;
$\;\alpha(j)<(36\sqrt{2})^{-1}$ for any $j\in\N;\;$
$\lim_{j\to\infty}\alpha(j)\log^2(j)=0;\;$
$\lim_{n\to\infty}
\textstyle
\sum_{t=1}^{n}
\P(|\Yt| \geq B)
=
0$;
and
$$
\lim_{n\to\infty}
\min_{t\in\{1,\ldots,n\}}
\left(
\min\left\{
\frac{B-\E(-\Yt)}{\sqrt{\mathrm{Var}(\Yt)}}
\,,\;\;
\frac{B-\E(\Yt)}{\sqrt{\mathrm{Var}(\Yt)}}
\right\}
\right)
=
\infty
.
$$
\end{enumerate}
\vspace{.1cm}
Then, in either case,
$
\lim_{n\to\infty}
\Pr\Big(\,
\max_{t\in\{1,\ldots,n\}}
|\Yt|
\geq
B
\Big)
=0
.
$
\end{proposition}
The following lemma shows that Assumption \ref{as:data}
implies a form of uniform integrability that will be used to verify \ref{CR2:crit_lip}(ii) and \ref{RC:crit_lip}(iii) for this subsection's main results.
See Appendix \ref{sec:PROOF_SEC_DNN} for the proof of Lemma \ref{lem:unif_int}.
\begin{lemma}\label{lem:unif_int}
If Assumption \ref{as:data} holds then
$\lim_{n\to\infty}
\Big\{
\max_{t\in\{1,\ldots,n\}}
\E\big[
\Yt^2\,\mathbbm{1}_{|\Yt|\geqB}
\big]
\Big\}
=
0
.
$
\end{lemma}
The first main result of this section is
Theorem \ref{thrm:DNN_ROC_V1}, which
applies Theorem \ref{thrm:ROC_V1}, to obtain a rate of convergence in probability for DNN estimators in general nonparametric regression settings with nonstationary $\alpha$-mixing data.
\begin{theorem}\label{thrm:DNN_ROC_V1}
Suppose Assumptions \ref{as:smoothness} and \ref{as:data} hold with $B\lesssim n^{\DNNBoundrate}$ for some $\DNNBoundrate\in[0,1/4)$.
Let $\{\Zt\}_{t\in\N}$ be an $\alpha$-mixing process with
$\alpha(j)\leq \Calpha'e^{-\Calpha\,j } $
for some $\Calpha,\Calpha'>0$.
Let
$\mathcal{G}= \FMLPnon(\L,\Hb,B)$
be defined as in \eqref{eq:sievespace}
where
the sequences
$\{\L\}_{n\in\N}$,
$\{\Hi{l}\}_{n\in\N}$
for each
$l\in\N$,
are non-decreasing,
$\Hi{l}\asymp \H$,
and
\begin{equation}\label{eq:architecture_growth1}
\L \asymp \log(n),
\qquad
\H
\asymp
n^{
\left(\frac{d}{\smooth+d/2}\right)
(1/4-\DNNBoundrate)
}
\log^2(n)
.
\end{equation}
Suppose
$\{\f\}_{n\in\N}$
satisfies \eqref{eq:fhat}
and there exists $\{\err\}_{n\in\N}$ such that
$h_n = \Op(\err^2)$,
and
for some $\upsilon>0$,
$$
\err
\gtrsim
\max
\left\{
n^{
-\left(\frac{\smooth}{\smooth+d/2}\right)
(1/4-\DNNBoundrate)}
\log^{2+\upsilon}(n)
\,,\;
\max_{t\in\{1,\ldots,n\}}
\sqrt{
\E\big[
\Yt^2\,\mathbbm{1}_{|\Yt|\geqB}
\big]
}
\right\}
.
$$
Then,
$\norm*{\f-\fO}_{\Lp{2}(\P_{\{\Xt\}_{t=1}^n})}=\Op(\err)$.
\end{theorem}
Theorem \ref{thrm:DNN_ROC_V1} shows that DNN estimators are consistent in very general settings, however
the convergence rate is strictly slower than $n^{-1/4}$.
The next result uses Theorem \ref{thrm:ROC_V2}
to obtain nonasymptotic error bounds which can imply faster rates of convergence in probability when
$\{\Zt\}_{t\in\N}$ is stationary and $\beta$-mixing.
\begin{theorem}\label{thrm:DNN_ROC_V2}
Suppose Assumptions \ref{as:smoothness} and \ref{as:data} hold with $B\asymp n^{\DNNBoundrate}$ for some $\DNNBoundrate\in[0,1/2)$.
Let $\{\Zt\}_{t\in\N}$ be a strictly stationary $\beta$-mixing process with
$\beta(j)\leq \Cbeta'e^{-\Cbeta\,j } $
for some $\Cbeta,\Cbeta'>0$.
Let
$\mathcal{G}= \FMLPnon(\L,\Hb,B)$
be defined as in \eqref{eq:sievespace}
where
the sequences
$\{\L\}_{n\in\N}$,
$\{\Hi{l}\}_{n\in\N}$
for each
$l\in\N$,
are non-decreasing,
$\Hi{l}\asymp \H$,
and
\begin{equation*}
\L \asymp \log(n),
\qquad
\H
\asymp
n^{
\left(\frac{d}{\smooth+d}\right)
(1/2-\DNNBoundrate)
}
\log^2(n)
.
\end{equation*}
Then,
for
$\{\f\}_{n\in\N}$ satisfying \eqref{eq:fhat}, and
$$
\err
=
n^{
-\left(\frac{\smooth}{\smooth+d}\right)
(1/2-\DNNBoundrate)
}
\log^6(n)
+
\sqrt{
\E\big[
\Yt^2\,\mathbbm{1}_{\{|\Yt|\geq B\}}
\big]
+
h_n
}
,
$$
there exists a constant $C>0$ independent of $n$, such that
for all $n$ sufficiently large
\begin{equation*}
\resizebox{\hsize}{!}{$
\begin{aligned}
&
\Pr\left(
\|\f-\fO\|_{\Lp{2}}
\leq
C\,\err
\right)
\geq
1-
e^{
-n^{
\left(\frac{\smooth}{\smooth+d}\right)
(1/2-\DNNBoundrate)
}
}
-
\frac{
2\Cbeta'n^{
1
-
\Cbeta
\log(n)
}
}{
\log(n)
}
-
4\log(n)
\Pr\Big(\,
\max_{t\in\{1,\ldots,n\}}
|\Yt|\geq B
\Big)
,
\\&
\Pr\left(
\|\f-\fO\|_{2,n}
\leq
C\,\err
\right)
\geq
1
-
4e^{
-n^{
\left(\frac{\smooth}{\smooth+d}\right)
(1/2-\DNNBoundrate)
}
}
-
\frac{
12\Cbeta'n^{
1
-
\Cbeta
\log(n)
}
}{
\log(n)
}
-
24\log(n)
\Pr\Big(\,
\max_{t\in\{1,\ldots,n\}}
|\Yt|\geq B
\Big)
.
\end{aligned}
$}
\end{equation*}
\end{theorem}
This result can imply convergence rates as fast as
$n^{
-\left(\frac{\smooth}{2(\smooth+d)}\right)
}
\log^6(n)
$
when $\Yt$ is almost surely bounded and $h_n$ converges sufficiently quickly.
In this case, the rate only differs by a factor of $\log^2(n)$ from the rate implied by \citet[Theorem 1]{farrell_deep_2021} for settings
where $\{\Zt\}_{t\in\N}$ forms an \iid sequence and $\Yt$ takes values in a compact set.
Notably, \citet{brown_inference_2024} demonstrates that the rates implied by Theorem \ref{thrm:DNN_ROC_V2} can be sufficiently fast to obtain $\sqrt{n}$-asymptotic normality of a finite-dimensional parameter following first stage nonparametric estimation of conditional expectations with DNN estimators in a partially linear regression under stationary $\beta$-mixing data with unbounded regressands.
The convergence rates from Theorem \ref{thrm:DNN_ROC_V1} and the rate implied by Theorem \ref{thrm:DNN_ROC_V2} will not depend on $\alpha(j)$ and $\beta(j)$ respectively, provided they are geometrically mixing.
Although, the probability bound from Theorem \ref{thrm:DNN_ROC_V1} will depend on $\Cbeta, \Cbeta'$.
\subsection{DNNs for binomial autoregressions with covariates}\label{sec:DNN_log}
{
This section considers logistic binomial autoregression models, building upon
Example \ref{ex:log}.
The main result of this section is Theorem \ref{thrm:DNN_ROC_LOGISTIC},
which
demonstrates the applicability of Theorem \ref{thrm:ROC_V1}
for DNN estimators in categorical autoregressive settings.
Such settings are of particular interest for DNNs since they are frequently employed for classification problems.
Although this binomial logistic setting is a simple case of the classification problem,
similar results for multinomial and non-logistic settings could be attained with the methods employed here.
Also, see \citet[Lemma 9]{farrell_deep_2021} for an example of how the criterion functions for Poisson, Gamma, and multinomial logistic models can be shown to satisfy the requirements of Theorem \ref{thrm:ROC_V1}.
The setting will be a special case of the models considered in \citet{truquet_strong_2021}.
For all $t\in\N$, let
$
\Zt\coloneqq(\Yt,\Xt)
$
such that
$\Yt\in\{0,1\}$
and
$
\Xt=(\boldsymbol{V}_{t-1},\Y_{t-1},\ldots\Y_{t-r})
\in
[0,1]^{d-r}\times\{0,1\}^r
\subset
\R^{d_{\boldsymbol{W}}}
$
for some random vector of covariates $\boldsymbol{V}_t\in[0,1]^{d-r}$, where $d>r$.
Suppose
\begin{equation}\label{eqlog:model}
\Yt
=
s\big(
\boldsymbol{V}_{t-1},
Y_{t-1},
\ldots,
Y_{t-r},
\upsilon_t
\big)
\coloneqq
s\big(
\Xt,
\upsilon_t
\big)
,
\quad
\forall
t\in\N,
\end{equation}
where
$\upsilon_t\in\R$
is some random noise,
and the function
$
s:
[0,1]^{d-r}
\times
\{0,1\}^{r}
\times
\R
\to
\{0,1\}
$
is measurable.
The estimation target will be
a scaled version of the function
$
\textstyle
\logp{
\frac
{\E[\Yt|\Xt]}
{1-\E[\Yt|\Xt]}
}
$
under the following logistic regression assumption.
\begin{assumption}\label{as:data_logistic}
For all $t\in\N$,
let
$
\E[\Yt|\Xt]
=
e^{B\fO}\big[1+e^{B\fO}\big]^{-1}
$
for some $B \geq 2$, and $\fO:\R^{d_{\boldsymbol{W}}}\to[-1,1]$.
\end{assumption}
It will be convenient to write the assumption this way since
$\snorm{\fO}\leq 1$
under Assumption \ref{as:smoothness}.
Indeed,
Assumption \ref{as:data_logistic} is equivalent to the usual assumption
$
\E[\Yt|\Xt]
=
e^{s_0}\big[1+e^{s_0}\big]^{-1}
$
where it is also assumed that
$\snorm{s_0}\leqB$,
by setting
$\fO=s_0/B$.
In what follows,
the particular value of $B$ will play no role in the convergence rate of the estimator.
Also note that Assumption \ref{as:smoothness} will require $\fO$ to be defined on $\R^{d_{\boldsymbol{W}}}$, which imposes some additional structure on $\fO$ since
$
\Xt
\in
[0,1]^{d-r}\times\{0,1\}^r
\subset
\R^{d_{\boldsymbol{W}}}
.
$
The goal is to estimate
$
\fO
=
B^{-1}
\logp{
\frac
{\E[\Yt|\Xt]}
{1-\E[\Yt|\Xt]}
}
,
$
using a DNN sieve estimator $\f$ as in \eqref{eq:fhat} where the criterion is
$$
\crit(\Zt,f)
\coloneqq
-\Yt B f(\Xt)
+
\logp{1+e^{B f(\Xt)}}
,
$$
and the DNN sieve spaces are
$
\mathcal{G}
=
\big\{
f(\pi_{X}(\cdot))
:
f\in\mathcal{G}
\big\}
.
$
\begin{theorem}\label{thrm:DNN_ROC_LOGISTIC}
Suppose Assumptions \ref{as:smoothness} and \ref{as:data_logistic} hold.
Let $\{\Zt\}_{t\in\N}$ be an $\alpha$-mixing process with
$\alpha(j)\leq \Calpha'e^{-\Calpha\,j } $
for some $\Calpha,\Calpha'>0$.
Let
$\mathcal{G}= \FMLPnon(\L,\Hb,2)$
be defined as in \eqref{eq:sievespace}
where
the sequences
$\{\L\}_{n\in\N}$,
$\{\Hi{l}\}_{n\in\N}$
for each
$l\in\N$,
are non-decreasing,
$\Hi{l}\asymp \H$,
and
\begin{equation}\label{eq:architecture_growth_logistic}
\L \asymp \log(n),
\qquad
\H
\asymp
n^{
\frac{1}{2}
\left(\frac{d}{\smooth+d}\right)
}
\log^2(n)
.
\end{equation}
For
$\{\f\}_{n\in\N}$
satisfying \eqref{eq:fhat}
if there exists $\{\err\}_{n\in\N}$ such that
$h_n = \Op(\err^2)$,
and
$$
\err
\gtrsim
n^{
-\frac{1}{2}\left(\frac{\smooth}{\smooth+d}\right)
}
\log^{5}(n)
,
$$
then $\norm*{\f-\fO}_{\Lp{2}(\P_{\{\Xt\}_{t=1}^n})}=\Op(\err)$.
\end{theorem}
To the best of my knowledge, this is the first result providing a convergence rate for DNN estimators in classification settings with dependent data.
Theorem \ref{thrm:DNN_ROC_LOGISTIC} provides a convergence rate in settings with
nonstationary $\alpha$-mixing data
that is identical, up to a logarithmic factor, to the rate implied by \citet[Theorem 1]{farrell_deep_2021} under \iid data.
In addition, this convergence rate is unaffected by the rate of decay of the $\alpha$-mixing coefficient, provided it is geometric mixing.
Theorem \ref{thrm:DNN_ROC_LOGISTIC} allows for very general forms of dependence, and includes many interesting examples.
The following corollary provides two examples of settings in which
Theorem \ref{thrm:DNN_ROC_LOGISTIC} can be applied without directly assuming the mixing condition for $\{\Zt\}_{t\in\N}$.
Corollary \ref{cor:log_alpha_con}(i)
follows from
\citet[Theorem 1]{truquet_strong_2021} and point 2 of the discussion following their result,
since
Assumption \ref{as:data_logistic} implies
$
\P(\Yt=1|\Xt)
\in
\left[
e^{-B}/
\big(1+e^{-B}\big)
\,,\;
e^{B}/
\big(1+e^{B}\big)
\right]
.
$
Corollary \ref{cor:log_alpha_con}(ii)
follows from
\citet[Theorem 3, Proposition 1]{truquet_strong_2021}.
\footnote{
To apply \citet[Proposition 1]{truquet_strong_2021} note that ergodicity of $\{\boldsymbol{V}_t\}$ is implied by stationarity and $\alpha$-mixing
(see e.g.
\citealp{bradley_basic_2005}
).
}
Using these results, one can also obtain similar sufficient conditions for mixing properties of $\{\Zt\}_{t\in\N}$ in the multinomial case, $\Yt\in\{0,1,\ldots,N\}$.
Let the mixing coefficient for $\{\boldsymbol{V}_t\}_{t=0}^\infty$ and $\{\Zt\}_{t\in\N}$ be denoted as $\alpha_{\boldsymbol{V}}$ and $\alpha_{\boldsymbol{W}}$ respectively.
\begin{corollary}[\citealp{truquet_strong_2021}]\label{cor:log_alpha_con}
Consider the model from \eqref{eqlog:model}.
Suppose $\{\boldsymbol{V}_t\}_{t=0}^{\infty}$ is strictly stationary and one of the following holds:
\begin{enumerate}
\item
$\{\boldsymbol{V}_t\}_{t=0}^\infty$ is
$\alpha$-mixing such that
$
\alpha_{\boldsymbol{V}}(j)
=
\bigO\big(e^{-C j}\big)
$
for some $C>0$, and
$\{\upsilon_t\}_{t\in\N}$ is an i.i.d. sequence, independent of
$\{\boldsymbol{V}_t\}_{t=0}^\infty$,
such that for any $y\in\{0,1\}$
\begin{equation*}
\begin{aligned}
\P\Big(
\Yt=y
\,
\big|
\,
\{\boldsymbol{V}_t\}_{t=0}^{\infty},
Y_{t-1}
,
Y_{t-2}
,\ldots
\Big)
=
\P\Big(
\Yt=y
\,
\big|
\,
\boldsymbol{V}_{t-1},
Y_{t-1}
,\ldots
, Y_{t-r}
\Big)
\equiv
\P\Big(
\Yt=y
\,
\big|
\,
\Xt
\Big)
;
\end{aligned}
\end{equation*}
or
\item
$\{\boldsymbol{V}_t\}_{t=0}^{\infty}$ is
$\alpha$-mixing such that
$
\alpha_{\boldsymbol{V}}(j)
=
\bigO\Big(e^{-C j^2}\Big)
$
for some $C>0$,
and
$\{\upsilon_t\}_{t=0}^{\infty}$ is uniformly distributed on $(0,1)$ such that,
for each $t\in \N$,
$\upsilon_t$ is independent of
$\sigma\big(
\{\boldsymbol{V}_{t-j},\upsilon_{t-j}\}_{j=1}^t
\big)
$
and for any $y\in\{0,1\}$
\begin{equation*}
\begin{aligned}
\P\Big(
\Yt=y
\,
\big|
\,
\boldsymbol{V}_{t-1},
Y_{t-1}
,
Y_{t-2}
,\ldots
\Big)
=
\P\Big(
\Yt=y
\,
\big|
\,
\boldsymbol{V}_{t-1},
Y_{t-1}
,\ldots
, Y_{t-r}
\Big)
\equiv
\P\Big(
\Yt=y
\,
\big|
\,
\Xt
\Big)
;
\end{aligned}
\end{equation*}
and the model \eqref{eqlog:model} satisfies
\begin{equation*}
\begin{aligned}
s\big(
\Xt,
\upsilon_t
\big)
=
0
\quad
\iff
\quad
0
<
\upsilon_t
\leq
\E[\Yt|\Xt]
.
\end{aligned}
\end{equation*}
\end{enumerate}
Then,
$\{\Zt\}_{t\in\N}$ is strictly stationary and $\alpha$-mixing with
$
\alpha_{\boldsymbol{W}}(j)
=
\bigO\big(e^{-\Calpha j}\big)
$,
for some $\Calpha>0$.
\end{corollary}
Corollary \ref{cor:log_alpha_con}(i) allows general distributions for $\upsilon_t$
but imposes a strict exogeneity assumption with respect to $\{\boldsymbol{V}_t\}_{t=0}^{\infty}$.
Corollary \ref{cor:log_alpha_con}(ii)
requires $\{\upsilon_t\}_{t=0}^{\infty}$ to be \iid with a uniform distribution, and $\{\boldsymbol{V}_t\}$ to be mixing at a faster rate,
but allows for some endogeneity, since $\boldsymbol{V}_t$ can depend on $\upsilon_t$,
provided $\boldsymbol{V}_t$ is
independent of $\upsilon_{t-j}$, for $j=1,\ldots,t$.
These results are
somewhat stronger than needed for Theorem \ref{thrm:DNN_ROC_LOGISTIC}, since the stationarity of $\{\Zt\}_{t\in\N}$, or $\{\boldsymbol{V}_t\}_{t=0}^{\infty}$ is not required.
\citet{truquet_strong_2021} suggests these results could generalize to the non-stationary case, but further work is needed to verify this.
}
\subsection{Extensions to alternative DNN architectures}\label{sec:DNN_Alt_Arch}
While the networks considered in the previous sections are standard, similar results can be obtained for other architectures.
The key to this will be obtaining results for the complexity--either covering number or pseudo-dimension--and approximation power of the DNN sieve spaces under consideration.
This section demonstrates that this is possible and provides extensions of this paper's results to alternative feedforward DNN architectures.
\begin{remark}\label{rem:Pdim}
In this section,
and in the proofs for Section \ref{sec:DNN}'s results,
I will use bounds on the
the Vapnik Chervonenkis dimension (see \citealp[Definition 1]{bartlett_nearly_tight_2019}) of DNNs that take values in $\{0,1\}$
from
\citet{anthony_bartlett_neural_1999}
and
\citet{bartlett_nearly_tight_2019}.
This is without loss of generality since these results can be directly applied to the pseudo-dimension of real valued DNNs using \citet[Theorem 14.1]{anthony_bartlett_neural_1999} (also see discussion following \citealp[Definition 2]{bartlett_nearly_tight_2019}).
\end{remark}
The first result uses \citet[Proposition 1]{yarotsky_error_2017} to show that
Theorems \ref{thrm:DNN_ROC_V1}, \ref{thrm:DNN_ROC_V2}, and \ref{thrm:DNN_ROC_LOGISTIC}
are directly applicable to fully connected feedforward networks with any continuous piecewise-linear activation function, $\varphi$, that has a fixed number of breakpoints, $b\in\N$. One important example of this is the `leaky' ReLU (LReLU) activation function
\begin{equation*}
\varphi^{\textrm{LReLU}}(x)
=
\left\{
\begin{aligned}
&
cx,
&\quad&
x<0,
\\&
x,
&\quad&
x\geq0,
\end{aligned}
\right.
\end{equation*}
for some small constant $c>0$ (often set to $c=0.01$) that is predetermined before optimizing over the parameters $\gV$.
The LReLU is often used in practice to address the vanishing gradient problem that arises with the ReLU where certain computation units may never have non-zero outputs.
\begin{corollary}\label{prop:nonReLU}
Let $b\in\N$ be a constant and $\varphi:\R\to\R$ be any continuous piece-wise linear function with $b$ breakpoints.
Let
$\mathcal{N}_{n,\varphi}$
be defined as in \eqref{eq:sievespace} except with the ReLU activation function
replaced by $\varphi$.
Then, Theorems \ref{thrm:DNN_ROC_V1}, \ref{thrm:DNN_ROC_V2}, and \ref{thrm:DNN_ROC_LOGISTIC}
also apply to $\mathcal{N}_{n,\varphi}$.
\end{corollary}
The proofs of Theorems \ref{thrm:DNN_ROC_V1}, \ref{thrm:DNN_ROC_V2}, and \ref{thrm:DNN_ROC_LOGISTIC} follow similarly when $\mathcal{G}$ is replaced with $\mathcal{N}_{n,\varphi}$.
To see this, note that with \citet[Proposition 1]{yarotsky_error_2017} the approximation result of Lemma \ref{lem:approximation} can be extended to DNNs with a continuous piecewise-linear activation function by only increasing $\H$ by a constant factor; and
the complexity results of Lemmas \ref{lem:Pdim_bound} and \ref{lem:entropy_order}
are applications of
\citet{bartlett_nearly_tight_2019}
and
\citet[Theorem 12.2]{anthony_bartlett_neural_1999}
which also apply to $\mathcal{N}_{n,\varphi}$.
\begin{figure}[t]
\centering
\includegraphics[width =0.5\textwidth, height = 4cm]{DNN_v1_2024_08_25.jpg}
\caption{Example of $\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}$ architecture graph structure where
$\L=2$,
$\W=17$,
and $d=2$.}
\label{fig:DNN}
\end{figure}
The next result provides a framework to obtain theoretical properties of
DNN estimators with a wide variety of activation functions and any feedforward graph structure, provided an approximation result like Lemma \ref{lem:approximation} exists for the DNN under consideration.
These DNNs will be denoted as $\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}$, which allow for any graph structure where
units in layer $l\in\{1,\ldots,\L+1\}$
takes inputs from any of the units in layers $l'\in\{0,\ldots,l-1\}$.
See Figure \ref{fig:DNN} for an example of $\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}$'s graph structure.
\begin{corollary}
\label{prop:FFN}
Let $\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}$ be any feedforward neural network with $\L$ layers, $\W$ parameters, $U_n$ computation units and some continuous activation function $\varphi:\R\to\R$, such that
$\sup_{f\in\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}}\snorm{f}\leq B$.
Define the complexity bound $\Xi_{n,\varphi}$ for the following three classes of activation functions:
\begin{enumerate}
\item
if $\varphi$ is piecewise-linear with $b\in\N$ breakpoints let
$
\Xi_{n,\varphi}
\coloneqq
\W\L \log(\W);
$
\item
if $\varphi$ is piecewise-polynomial with $b\in\N$ breakpoints where each piece is a polynomial with degree $\leq p \in\N$,
let
$
\Xi_{n,\varphi}
\coloneqq
\WU_n\log\big((p+1)b\big)
;
$
\item
if $\varphi$ is the sigmoid function,
$\textstyle\varphi(x)=\frac{1}{1+e^{x}}$,
let
$$
\Xi_{n,\varphi}
\coloneqq
\big((\W+2)U_n\big)^2
+
(\W+2)U_n\log_2\big(18(\W+2)U_n^2\big)
.
$$
\end{enumerate}
Suppose $\varphi$ is described by one of the three above classes,
then there exists $C\geq 1$ such that,
for any $n\in\N$,
$
\textstyle
\cover^{(\infty)}_{\infty}
\big(
\delta
,\,
\mathcal{G}
,\,
n
\big)
\leq
\left(
\frac
{2eB n}
{\delta \, \Xi_{n,\varphi}}
\right)^{C\,\Xi_{n,\varphi}}
$
\begin{equation*}
\Pdim(\mathcal{N}_{n,\varphi}^{\mathrm{FFN}})\;\leq\; C\,\Xi_{n,\varphi}
,
\quad
\text{and}
\quad
\cover^{(\infty)}_{\infty}
\big(
\delta
,\,
\mathcal{G}
,\,
n
\big)
\;\leq\;
\left(
\frac
{2eB n}
{\delta \, \Xi_{n,\varphi}}
\right)^{C\,\Xi_{n,\varphi}}
,
\end{equation*}
for all $\delta>0$, and $n\geq C\,\Xi_{n,\varphi}$.
Consequently,
Theorems \ref{thrm:ROC_V1} or \ref{thrm:ROC_V2}
can be applied
with
$
\mathcal{G}
=
\big\{
f(\pi_{X}(\cdot))
:
f\in\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}
\big\}
,
$
whenever
the conditions of these theorems hold
using the complexity bounds in the previous display.
\end{corollary}
The pseudo-dimension bounds follow from \citet[pp. 5,6]{bartlett_nearly_tight_2019} for the piecewise-linear case,
\citet[Theorem 10]{bartlett_nearly_tight_2019}
for the
piecewise-polynomial case, and \citet[Theorem 8.13]{anthony_bartlett_neural_1999} for the sigmoid case.
The covering number bound uses \citet[Theorem 12.2]{anthony_bartlett_neural_1999}.
To apply Theorems \ref{thrm:ROC_V1} or \ref{thrm:ROC_V2} with Corollary \ref{prop:FFN}, an approximation result, like Lemma \ref{lem:approximation}, will be needed for $\mathcal{N}_{n,\varphi}^{\mathrm{FFN}}$ to verify Conditions \ref{RC:projection} or \ref{CR2:projection} respectively.
\section{Summary and extensions}
\label{sec:Conc}
This paper addresses the lack of statistical foundation for empirical work using deep neural network (DNN) estimators under dependent data. By establishing general results for sieve estimators, I provide a flexible framework that applies to various DNN estimators in a wide range of dependent data settings. These results extend existing work to more complex and realistic scenarios, allowing
for non-\iid data with very general forms of dependence and taking values in unbounded sets.
I apply this framework to derive properties for DNN estimators in both nonparametric regression and classification contexts, focusing on architectures
that reflect modern applications--
featuring ReLU activation functions, unbounded parameters, fully connected feedforward structures, and depth and width that grow with sample size.
Notably, Corollary \ref{prop:nonReLU} shows that these results also apply when the ReLU activation function is replaced with any continuous piecewise-linear activation function, such as the leaky ReLU.
The practical relevance of my DNN results is demonstrated by \citet{brown_inference_2024} which considers a partially linear regression model with dependent data and unbounded regressands,
and shows that the estimator for the finite-dimensional parameter, constructed using DNN-estimated nuisance components, achieves $\sqrt{n}$-consistency and asymptotic normality.
While this work only considers standard DNN architectures, Subsection \ref{sec:DNN_Alt_Arch} demonstrates how the general sieve estimator results presented here offer a pathway for extending the analysis to more complex architectures.
Perhaps the most important avenue for future research on DNN estimators under dependent data is recurrent neural networks (RNNs).
RNNs are a class of DNN architectures that are specifically designed for time series settings, due to a recursive feedback loop that gives the network a form of `memory' for past events.
While the empirical results from \citet{lazcano_back_2024} indicate that the DNN architectures studied here outperform RNNs in simpler time-series models, RNNs have demonstrated superior empirical performance in more complex settings, such as in the work by \citet{bucci_realized_2020} on forecasting stock market volatility.
Thus far, very little work has been done on the theoretical properties of RNNs.
While \citet{kohler_recurrent_2020} provides an initial study of a specific recurrent architecture in time-series nonparametric regression, much remains to be understood, particularly for more general recurrent architectures, and dependence settings.
Following the ideas of Subsection \ref{sec:DNN_Alt_Arch}, the sieve estimator framework introduced in this paper could facilitate future research for RNNs once their approximation power and complexity are better understood.
Many other important aspects of DNNs are also not considered here,
such as computational efficiency or potential gains from alternative architectures and regularization techniques.
Another important class of DNNs not considered here are convolutional neural networks, which are standard in many important DNN applications, such as image recognition.
The results given here could also be adapted for classes of functions beyond the standard H\"older smoothness condition, using approximation results such as \citet{Imaizumi_DNN_nonsmooth_2019}.
These considerations are left for future research.
\pagebreak
\addappheadtotoc