Inference in Partially Linear Models under Dependent Data with Deep Neural Networks

Inference in Partially Linear Models under Dependent Data with Deep Neural Networks

{4pt} {4pt}

Introduction

Partially linear models were first considered by engle_semiparametric_1986 and robinson_1988. Since then, these models have been widely used for empirical work in time series settings (see e.g., gao_nonlinear_2007; and hardle_partially_2000; for a review). For instance, partially linear models have been employed by engle_semiparametric_1986 to study electricity sales, since the impact of temperature on electricity consumption is nonlinear, as both high and low temperatures lead to increased electricity demand; li_simultaneous_2024 to study the forward premium anomaly; and gao_adaptive_2000 to study the number of lynx trapped in the MacKenzie River district in the Canadian North-West Territories.

In particular, consider the partially linear model,

Let $\{\Zt\coloneqq(\Yt,\Tt,\Xt)\}_{t\in\N}$ be an stochastic sequence on the probability space $\probspace$. The data is $\{\Zt\}_{t=1}^n$, where $\Yt$ is the outcome, $\Tt$ is the policy or treatment variable, and $\Xt$ are additional covariates which affect $\Yt$ through an unknown nuisance function $\nus$ that is measurable. Note that this model can apply to nonlinear auto-regressive settings by letting $\Xt$ include past values $\Y_{t-j}$, $j=1,\ldots,r$ for $r\in\N$. For notational simplicity, we consider the case where $\Tt$ is a scalar treatment, although the results obtained here could easily generalize to the vector-valued case. We are interested in estimating and performing inference on the parameter $\paramO\in\R$ in settings where $\{\Zt\}_{t\in\N}$ is (strictly) stationary and $\beta$-mixing (see e.g. dehling_empirical_2002), in the sense that $\beta(j)\to0$ as $n\to\infty$ for $$ \beta(j) \coloneqq \E\Bigg[ \sup\bigg\{ \Big| \P\big( B|\, \sigma\big(\{\Zt\}_{1}^k\big) \big) - \P\big( B \big) \Big| :\, {B\in \sigma\left(\{\Zt\}_{k+j}^\infty\right)} ,\, k\in\N \bigg\} \Bigg] , \quad \text{ for } j\in\N , $$ where $\sigma\left(\{\Zt\}_{k}^n\right)$ denotes the $\sigma$-algebra generated by $\{\Zt\}_{t=k}^n$.

Using a procedure due to robinson_1988, the estimator for $\paramO$ will be constructed using DNN-estimated nuisance components. With the DNN results from brown_statistical_2024, I show that this estimator will obtain $\sqrt{n}$-consistency and asymptotic normality in settings with stationary $\beta$-mixing data under mild regularity conditions. By employing the ideas of chen_debiased_2022, I do this without sample splitting, which is particularly important in dependent data settings, where it can be difficult to construct independent validation sets.

This work primarily contributes to two growing bodies of literature. First, the literature on inference after machine learning (e.g., chernozhukov_doubledebiased_2018; farrell_deep_2021; chernozhukov_automatic_2022) largely focuses on \iid settings, whereas I provide results that deliver valid large sample inference under dependent data settings. Second, the work that studies partially linear models in \iid settings (see e.g., robinson_1988; fan_profile_2005; geng_estimation_2020; and citations therein) and time series settings (e.g., hardle_partially_2000; gao_nonlinear_2007; li_simultaneous_2024) has used more traditional nonparametric estimators, such as kernel estimators, whereas I use DNNs to estimate the infinite dimensional parameters. In addition, it also shows practical implications of the results from brown_statistical_2024, and introduces techniques that could be applied in settings beyond the simple partially linear model considered here.

{ Notation: For two sequences of non-negative real numbers $\{x_t\}_{t\in\mathbb{N}}$ and $\{y_t\}_{t\in\mathbb{N}}$, the notation $x_t\lesssim y_t$ means there exists a constant $0<C<\infty$ such that $x_t\leq Cy_t$ for all $t$ sufficiently large. We write $x_t \asymp y_t$ if $x_t \lesssim y_t$ and $x_t \gtrsim y_t$. For some set $\mathbb{X}$, and $A\subseteq\mathbb{X}$, the indicator function is denoted as $\mathbbm{1}_{A}:\mathbb{X}\to\{0,1\}$, where $\mathbbm{1}_{A}(x)=1$ if $x\in A$ and $\mathbbm{1}_{A}(x)=0$ if $x\in \mathbb{X}\setminus A$. For a measurable space $(\mathbb{X}, \Sigma)$, and a random variable $\X:\Omega \to \mathbb{X}$, define the measure $\Px(B) = \P(\X^{-1}(B))$ for any $B\in \Sigma$. Let $\Lp{r}\probspace$ denote the space of functions $ f: \Omega \to \R $ that are Borel measurable and $\norm{f}_{\Lp{r}\probspace}<\infty$, for the (pseudo-) norms

We write $\Lp{r}(\P)$ or $\norm{f}_{\Lp{r}(\P)}$ when no confusion may arise.

}

{

Inference in Partially Linear Models

We estimate $\paramO$ following a procedure due to robinson_1988. Note that $$ \E[\Yt|\Xt] = \E[\Tt|\Xt]\paramO + \nus(\Xt) + \E[u_t|\Xt] . $$ Let $\fOY\coloneqq\E[\Yt|\Xt]$ and $\fOT\coloneqq\E[\Tt|\Xt]$, then, taking the difference of (ref) and the previous display,

This provides the moment condition $ \E\big[\mom{\Zt}{\paramO}{\fOT}{\fOY}\big] = 0 , $ for the linear moment function

Now, we construct a two-stage method of moments estimator for $\paramO$. In the first stage, we estimate the conditional expectations $\fOY$ and $\fOT$ with DNN estimators $\fY$ and $\fT$ constructed using the framework of Section (ref). Then, we have the estimator

which exists whenver $\big[\Tt-\fT(\Xt)\big]^2\neq 0$ for atleast one $t\in\{1,\ldots,n\}$.

First Stage DNN Estimation

This section considers the preliminary DNN estimation of the conditional expectation functions $\fOT$ and $\fOY$.

DNN Architectures

First, we describe the DNN architectures used for the first stage DNN estimators. This will follow the framework of brown_statistical_2024. The architecture $\FMLP$, will be a fully connected feedforward DNN (also known as a multilayer perceptron) equipped with the ReLU activation function $\relu(x) = \max\{0,x\}$. This type of network is standard in practice, however the results given here also apply with $\relu$ replaced by any continuous piecewise-linear activation function, using brown_statistical_2024.

For each $n\in\mathbb{N}$, the graph structure of $\FMLP$ is determined by the number of hidden layers $\L$ and the number of units in each layer $\Hb=[H_{n,1},\ldots,H_{n,\L}] \in \N^{\L}$. See Figure (ref) for an example of $\FMLP$. Each layer $l\in\{1,\ldots,\L\}$ consists of computation units or nodes, which are denoted as $\n_{l,h}$, for $h\in \{1,\ldots,H_{n,l}\}$. Each node, $\n_{l,h}$, is a function taking values in $\R$ which depends on a vector of parameters $\gv_{l,h} = [\gamma_{l,h,0},\gamma_{l,h,1},\ldots,\gamma_{l,h,H_{n,l-1}}]' , $ and performs the computation

The network consists of $\W$ parameters collected together as $ \gV \coloneqq \{\g_{l,h,k}\}_{\forall\, l,h,k} \in \R^{\W} . $ The architecture $\FMLP$ is the space of functions defined by

where $\DNNBound\geq 2$ is the user-chosen upper bound. Given an architecture $\FMLP$, a particular network $f\in\FMLP$ is determined by a choice of parameters $\gV$, and can be written as $f(\cdot) = \n_{\L+1,1}\left(\cdot \;|\; \gV\right):\mathbb{X}\to[-\DNNBound,\DNNBound]$.

DNN Estimation

Following brown_statistical_2024, we estimate $\fOT$ and $\fOY$ using the method of sieves (grenander_abstract_1981; chen_chapter_2007), where the sieve spaces are constructed using DNNs. For the DNN architecture defined as in (ref), define the following DNN sieve spaces

for non-decreasing sequences

for all $l\in\N$. Let $\fT\in\FMLPT$, and $\fY\in\FMLPY$ be approximate sieve estimators satisfying

where the “plug-in” error $\theta_n:\Omega\to[0,\infty)$ is a random variable, such that $\theta_n = \op(1)$, and whenever feasible $\theta_n \coloneqq 0$. Note that by brown_statistical_2024 Remark 1, and Propositions 1 and 2, the infimums in (ref) will be measurable, and there will exist measurable mappings: $\omega\mapsto\fT$ from $\Omega$ to $\FMLPT$, and $\omega\mapsto\fY$ from $\Omega$ to $\FMLPY$. Henceforth, $\fT$, $\fY$, will always refer to such a measurable mapping. To estimate $\fOT$ and $\fOY$ with $\fT$ and $\fY$ we impose the following regularity assumptions.

Assumption (ref)(i) requires that $\{\Zt\}_{t\in\N}$ be geometrically $\beta$-mixing. Notably the particular value of $\Cbeta,\Cbeta'$ will play no role in the results of this paper. Stochastic-sequences that are $\beta$-mixing have been heavily studied (e.g., doukhan_mixing_1994; bradley_basic_2005). For example, conditions to ensure stationarity and geometric $\beta$-mixing for ARMA processes are given by mokkadem_mixing_1988, and for GARCH processes by chen_geometric_2000, and carrasco_mixing_2002. Also see brown_statistical_2024 for more discussion.

Assumptions (ref)(ii) and (iii) are standard regularity assumptions on the data. Assumption (ref)(ii) will hold without $\Yt$ taking values on a compact set (farrell_deep_2021), and without $\E[\Yt^2|\{\Xt,\Y_{t-1}\}_{t=1}^n]$ being uniformly bounded. See brown_statistical_2024 and leadbetter_extremes_1983 for examples of $\{\Yt\}_{t\in\N}$ that satisfy Assumption (ref)(ii). Notably, Assumption (ref)(ii) will also imply $\lim_{n\to\infty} \Big\{ \max_{t\in\{1,\ldots,n\}} \E\big[ \Yt^2\,\mathbbm{1}_{|\Yt|\geq\DNNBound} \big] \Big\} = 0 , $ by brown_statistical_2024.

Assumption (ref)(iv) is a H\"older smoothness condition that is widely used in the nonparametric estimation literature (e.g. stone_optimal_1982; chen_sieve_1998; chen_chapter_2007; chen_optimal_2015; farrell_deep_2021). Note that this is more general than the structural assumptions imposed in much of the DNN literature (e.g., kohler_nonparametric_2017; schmidt_hieber_nonparametric_2020; kohler_rate_2021 ).

Under Assumption (ref) the following corollary provides a rate of convergence in probability for $\fT$ and $\fY$. Corollary (ref) will follow immediately from brown_statistical_2024.

Asymptotic Properties of $\paramhat$

The following assumption will ensure that $\paramhat$ exists with probability approaching one and that Corollary (ref) can be used to obtain DNN estimators of $\fOT$, $\fOY$ that converge faster than $n^{-1/4}$.

First, note that Assumption (ref)(i) and Corollary (ref) imply $\paramhat$ exists with probability approaching one. Next, let

Then, under Assumptions (ref) and (ref), Corollary (ref) will imply $\err=\littleo(n^{-1/4}),$ and that $\norm*{\fT-\fOT}_{\Lp{2}(\Px)}$ and $\norm*{\fY-\fOY}_{\Lp{2}(\Px)}$ are $\Op(\err)$, whenever $\theta_n=\op(n^{-1/2})$. The upper bound on $\DNNBoundrate$ from Assumption (ref)(ii) is equivalent to requiring $\DNNBound\err=\littleo(n^{-1/4})$. The feasibility of the bounds on $\DNNBoundrate$ is not an additional assumption since the first part of Assumption (ref)(ii) implies $\smoothT>d$. With this, the following theorem will use the properties of $\fT$, and $\fY$ from Corollary (ref) to obtain $\sqrt{n}$-convergence and asymptotic normality of $\paramhat$.

The existence of $\sigma\geq0$ is not a new result and follows from bosq_1998. In most settings with $\E[\upsilon_t^2|\Xt,\Tt]>0$ the requirement that $\sigma>0$ is fairly mild, and will hold whenever the covariances of $ \big\{ {\upsilon_t}/{\big(\Tt-\ftnT(\Xt)\big)} \big\}_{t\in\N} $ don't sum to zero. To see this, note that by $\E[\upsilon_t|\Xt,\Tt]=0$, the definition of $\psi_t$, and Bienaym\'e's identity

\vx

The proof of Lemma (ref) is in Appendix (ref) below. Before the proof of this result, we first introduce the idea of independent blocking, commonly used when dealing with stationary $\beta$-mixing processes (e.g. chen_sieve_1998).

Independent blocks

The following is from brown_statistical_2024. Let $a\in\N$ be such that $1\leqa\leq n/2$, so $b \coloneqq \big\lfloor n/(2a)\big\rfloor$ is well defined. Then, we can divide $\{\Zt\}_{t=1}^n$ into $2b$ blocks of length $a$, and the remainder into a block of length $n-2\anBa$, using the index sets

Using Berbee's Lemma we can redefine $\{\Zt\}_{t=1}^n$ on a richer probability space (see brown_statistical_2024 for details) \footnote{We will continue to refer to the richer probability space as $\probspace$ since the extension preserves the distribution of random variables defined on the original space. } where there exists a random sequence $\{\overline{\boldsymbol{W}}_t\}_{t=1}^n$ with the following two properties: let $T,T' \in\{T_{1,1},T_{2,1},T_{1,2},\ldots,T_{2,b},T_{R}\} $, then (i) the block $\{\overline{\boldsymbol{W}}_t\}_{t\in T}$ is independent from the blocks $\{\overline{\boldsymbol{W}}_t\}_{t\in T'}$, $\{\Zt\}_{t\in T'}$ for any $T'\neq T$; and (ii) $\{\overline{\boldsymbol{W}}_t\}_{t\in T}$ has the same distribution as $\{\Zt\}_{t\in T}$, i.e. $\P_{\{\overline{\boldsymbol{W}}_t\}_{t\in T}}=\P_{\{\Zt\}_{t\in T}}$. By stationarity, all blocks of length $a$ are identically distributed so the sequence of blocks $ \{\overline{\boldsymbol{W}}_t\}_{t\inT_{1,1}},\{\overline{\boldsymbol{W}}_t\}_{t\inT_{2,1}},\{\overline{\boldsymbol{W}}_t\}_{t\inT_{1,2}},\ldots,\{\overline{\boldsymbol{W}}_t\}_{t\inT_{2,b}} $ is i.i.d., \footnote{ All blocks except $\{\overline{\boldsymbol{W}}_t\}_{t\inT_{R}}$ are of length $a$, and therefore i.i.d. However, $\{\overline{\boldsymbol{W}}_t\}_{t=1}^n$ is not an independent sequence since elements within a single block, $\{\overline{\boldsymbol{W}}_t\}_{t\in T}$, may be correlated. } and we have

Next, the usual $\beta$-mixing coefficient (e.g. dehling_empirical_2002) can be equivalently written as (see eberlein_weak_1984) $$ \beta(m) = \sup_{ A\times B \in \sigma\left( \{\Zt\}_{t=1}^k \right) \,\otimes\, \sigma\left( \{\Zt\}_{t=k+m+1}^\infty \right) } |\P(A \times B)- \P(A)\P(B)|. $$ Hence, for $j\in \{1,...,b-1\}$

By (ref) the conditions for eberlein_weak_1984 Lemma 2 are satisfied. So we apply this result, and use (ref), to obtain, for any measurable set $E$,

Then, by the triangle inequality and $ b \coloneqq \big\lfloor n/(2a)\big\rfloor < n/(2a) +1 , $

Proof of Lemma (ref)

The proof of Lemma (ref) will make use of the Rademacher complexity, defined as in brown_statistical_2024 (also see bartlett_local_2005). { [2] { \sup_{\{#2\}} \frac{1}{#1} \sum_{j=1}^{#1} \xi_j \normTj{f-\fO}{j}^2 }

}

{

} \vx

{

}

\vx \vx

}

\singlespacing