The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
92,528 characters
Automatic Debiased Machine Learning for Dynamic Treatment Effects and General Nested Functionals
\maketitle
\begin{abstract}
We extend the idea of automated debiased machine learning to the dynamic treatment regime and more generally to nested functionals. We show that the multiply robust formula for the dynamic treatment regime with discrete treatments can be re-stated in terms of a recursive Riesz representer characterization of nested mean regressions. We then apply a recursive Riesz representer estimation learning algorithm that estimates de-biasing corrections without the need to characterize how the correction terms look like, such as for instance, products of inverse probability weighting terms, as is done in prior work on doubly robust estimation in the dynamic regime. Our approach defines a sequence of loss minimization problems, whose minimizers are the mulitpliers of the de-biasing correction, hence circumventing the need for solving auxiliary propensity models and directly optimizing for the mean squared error of the target de-biasing correction. We provide further applications of our approach to estimation of dynamic discrete choice models and estimation of long-term effects with surrogates.
\end{abstract}
\section{Introduction}
Recent progress in the area of causal machine learning has shown how one can automatically de-bias causal estimands that take the form of a solution to a moment equation which involves nuisance regression functions \cite{chernozhukov2018automatic,smucler2019unifying,chernozhukov2020adversarial,Chernozhukov2021riesz}. Prominent examples include estimands of the form:
\begin{align}
\theta =~& \operatorname{\mathrm{E}}[m(Z; g)], \text{ for } g(X) := \operatorname{\mathrm{E}}[Y \mid X]
\end{align}
encompassing quantities such as the average treatment effect the average policy effect and the average marginal effect, under suitable conditional exogeneity conditions.
However, all prior work analyzes problems that fall into the static treatment regime setting, i.e. treatments are given at single period and not over time in a dynamic and adaptive manner. In this work we present the first automatic debiasing approach for the dynamic treatment regime.
The dynamic treatment regime has been well studied in the causal inference and biostatistics literature with many approaches for doubly robust \cite{robins1986new,robins2000marginal,gill2001causal,hernan2001marginal,vansteelandt2003causal,van2005history,bang2005doubly,yu2006double,orellana2010dynamic} and multiply robust \cite{scharfstein1999adjusting,murphy2001marginal,bang2005doubly,van2011targeted} estimation. Recent work has also extended this literature to the high-dimensional regime and to the incorporation of machine learning based regression and propensity estimators \cite{lewis2020double,bodoryevaluating,bradic2021high,singh2021finite}. However, all prior work use explicit de-biasing approaches that analytically characterize the form of the de-biasing term in order to achieve double robustness, such as for instance products of inverse propensity scores over time.
The key idea behind automatic de-biasing in the static regime, is that the de-biasing term can be equivalently phrased in terms of the Riesz representer of the linear functional implied by the estimand $\theta$. Hence, de-biasing boils down to estimation of the Riesz representer of a linear functional, given in an oracle manner and does not require analytic derivation.
We extend the idea of automated debiased machine learning to the dynamic treatment regime and we show that the multiply robust formula for the dynamic treatment regime with discrete treatments can be re-stated in terms of a recursive Riesz representer characterization of nested mean regressions. We then apply a recursive Riesz representer estimation learning algorithm that estimates de-biasing corrections without the need to characterize how the correction terms look like, such as for instance, products of inverse probability weighting terms, as is done in prior work on doubly robust estimation in the dynamic regime.
Our approach defines a sequence of loss minimization problems, whose minimizers are the mulitpliers of the de-biasing correction, hence circumventing the need for solving auxiliary propensity models and directly optimizing for the mean squared error of the target de-biasing correction. We also extend prior work on estimation rates of Riesz representers to account for the estimation error that stems from the prior steps in the recursive Riesz estimation process, which was not required in prior work in the static regime.
\section{Dynamic Treatment Regime}
We consider estimation of treatment effects in the dynamic treatment regime. We assume we have access to $n$ samples of trajectories
$$Z := (S_1, T_1, S_2, T_2, \ldots, S_M, T_M, Y),$$
with $S_t\in \ensuremath{{\cal S}}_t$ are time-varying confounders and $T_t\in {\mathcal T}_t$ are treatments over time and $Y$ a final outcome. For any time $t$, let $\bar{S}_t=\{S_1,\ldots, S_t\}$ and $\bar{T}_t=\{T_1,\ldots, T_t\}$ denote the sequence of the variables up until time $t$ and similarly, let $\underline{S}_t = \{S_t, \ldots, S_M\}$ and $\underline{T}_t=\{T_t,\ldots, T_M\}$. We will also denote with $\bar{s}_t, \bar{\tau}_t, \underline{s}_t, \underline{\tau}_t$, corresponding realizations of the latter random sequences. Moreover, we will be denoting with $(\bar{\tau}'_t, \underline{\tau}_{t+1})$, the sequences of potential treatment states that follows $\tau'$ up until time $t$ and then continues with $\tau$. We let $0\in {\mathcal T}_t$ denote a baseline policy value, which could be appropriately instantiated based on the context.
For any sequence of treatments $\tau=(\tau_1,\ldots, \tau_M)$, let $Y^{(\tau)}$ denote the counterfactual outcome under such a sequence of interventions (sequence of treatment states), equivalently in do-calculus notation $Y \mid do(\bar{T}_M=\bar{\tau}_M)$. Note that with this notation $Y\equiv Y^{(\bar{T}_M)}$.
Under this counterfactual notation, our target quantity of interest is:
\begin{align}\label{eqn:target-cntf}
\theta(\tau) := \operatorname{\mathrm{E}}\left[Y^{(\tau)}\right]
\end{align}
We assume that the data generating process satisfies the following sequential conditional randomization assumption:
\begin{assumption}[\emph{Sequential Conditional Exogeneity}]\label{ass:cond-ex} The data generating process satisfies the following conditional independence conditions:
\begin{equation}
\forall 1\leq t\leq M \text{ and } \forall \underline{\tau}_{t}\in \times_{k=t}^M {\mathcal T}_k: Y^{(\bar{T}_{t-1}, \underline{\tau}_{t})}\perp \!\!\! \perp T_{t} \mid S_{t} \tag{dynExog}
\end{equation}
\end{assumption}
\begin{figure}[htpb]
\centering
\hspace{-4em}
\includegraphics[scale=.45]{surrogates_causal_graph.png}
\caption{Causal diagram describing the causal relationships of the random variables in the time series.}
\label{fig:dag}
\end{figure}
This condition is for instance satisfied if the data generating process adheres to the causal graph presented in Figure~\ref{fig:dag}, as can be easily verified from the single-world-intervention graph (SWIG) in Figure~\ref{fig:swig}.
\vsedit{Note that even though we used a Markovian notation and the observational policy only depend on current state $S_t$ and the outcome $Y$ only depends on last state $S_M$, one should really interpret $S_t$ as the current sufficient statistic of the history up until time $t$. For instance, $S_t$} can contain all prior treatments and prior base states as part of it. For instance, suppose that we had an observed time series of $(X_1, T_1, \ldots, X_M, T_m, Y)$ and we wanted to allow all forward arrows in the causal graph. Then we could re-define $S_t = (\bar{X}_t, \bar{T}_{t-1})$ and apply our current formulation. This would lead to identical derivations, modulo this renaming. Thereby our setting is much more permissive than what one might believe at a first glance and encompasses the general dynamic treatment regime setting as a special case.
\vscomment{\textcolor{blue}{[VC: relation on Figure 2 to Assumption 1 is not crisp. This is mainly exposition issue.]}}
Moreover, we will assume a surrogacy assumption, that under an interventional future treatment policy, the effect of $T_{t-1}$ on future outcomes only goes through $S_t$. This is again satisfied if the data generating process adheres to the causal graph presented in Figure~\ref{fig:dag}, as can be easily verified from the single-world-intervention graph (SWIG) in Figure~\ref{fig:swig}. In fact, we will only require a conditional mean-independency assumption.
\begin{assumption}[\emph{Sequential Surrogacy}]\label{ass:dyn-surrogacy}
The data generating process satisfies the following conditional mean-independence conditions:
\begin{equation}
\forall 1\leq t \leq M \text{ and } \forall \underline{\tau}_{t}\in \times_{k=t}^M {\mathcal T}_k: Y^{(\bar{T}_{t-1}, \underline{\tau}_{t})} \perp \!\!\! \perp_{\ensuremath{\text{mean}}} (T_{t-1}, S_{t-1}) \mid S_{t} \tag{dynSurr}
\end{equation}
\end{assumption}
We note that if $S_t$ contains all past treatments and states as a subset, then this assumption is trivial since $T_{t-1}, S_{t-1}$ are deterministic random variables conditional on $S_t$ and hence are mean independent with any other random variable.
Finally, we also require a regularity condition of sequential positivity (aka overlap), which states that the conditional density of treatment is bounded away from zero a.s. To define sequential positivity, we will denote with $\pi(\tau_t, s_t)$ the marginal densities of the random variables $(T_t, S_t)$ and period $t\in [1,M]$. Then sequential positivity is defined as:
\begin{assumption}[Sequential Positivity]\label{ass:dynPos}
The density $\pi$ of the data generating processes satisfy that: $\ensuremath{\mathrm{Pr}}(T_t=\tau_t\mid S_t)>0$, whenever $\ensuremath{\mathrm{Pr}}(S_t)>0$, for all $1\leq t\leq M$.
\end{assumption}
\textsc{[VC: The notation $[\ensuremath{\mathrm{Pr}}(T_t=\tau_t\mid S_t)>0]$ and $\ensuremath{\mathrm{Pr}}(S_t)>0$ implies treatment and states are discrete; but indetification argument does not rely on that. Can change to $\pi(tau_t, s_t)>0$ when $pi(s_t)>0$?]}
\begin{figure}[htpb]
\centering
\hspace{-4em}
\includegraphics[scale=.45]{surrogates_swig.png}
\caption{Single world intervention diagram from intervening and setting the treatments to $\underline{\tau}_{t}$ from period $t$ and on-wards.}
\label{fig:swig}
\end{figure}
\section{Identification as Nested Regressions}\label{sec:dyn-reg}
We re-state the identification argument from classical work in the dynamic treatment regime \cite{robins1986new,robins2000marginal,gill2001causal,hernan2001marginal,vansteelandt2003causal,van2005history,bang2005doubly,yu2006double,orellana2010dynamic}, in a manner that will be convenient for our main theorem in the next section.
\begin{theorem}[Non-Parametric Identification]\label{thm:non-param-id}
If the data generating processes satisfy Assumption~\ref{ass:cond-ex} - Assumption~\ref{ass:dynPos}, then the target quantity $\theta(\tau)$ is non-parametrically identified via the following recursively defined estimands:
\begin{align}
\theta(\tau) =~& \operatorname{\mathrm{E}}[f_1(S_1, \tau_1)]\\
\forall 1\leq t < M: f_{t}(S_{t}, T_t) =~& \operatorname{\mathrm{E}}\left[f_{t+1}(S_{t+1}, \tau_{t+1})\mid S_{t}, T_t\right] \tag{recursive estimand}\\
f_{M}(S_{M}, T_M) =~& \operatorname{\mathrm{E}}\left[Y\mid S_{M}, T_M\right] \tag{base estimand}
\end{align}
\end{theorem}
\begin{proof}
For any $s_t\in \ensuremath{{\cal S}}_t$ and $\tau_t\in {\mathcal T}_t$, define: $$f_{t}(s_{t}, \tau_t) := \operatorname{\mathrm{E}}\left[Y^{(\bar{T}_{t-1}, \underline{\tau}_{t})} \mid S_{t}=s_{t}, T_t=\tau_t\right].$$
Then for any $t \geq 1$, we have the recursion:
\begin{align}
f_{t}(s_{t}, \tau_t) =~& \operatorname{\mathrm{E}}\left[Y^{(\bar{T}_{t-1}, \underline{\tau}_{t})} \mid S_{t}=s_{t}, T_t=\tau_t\right]\\
=~& \operatorname{\mathrm{E}}\left[Y^{(\bar{T}_{t}, \underline{\tau}_{t+1})} \mid S_{t}=s_{t}, T_t=\tau_t\right]\tag{consistency}\\
=~& \operatorname{\mathrm{E}}\left[\operatorname{\mathrm{E}}\left[Y^{(\bar{T}_{t}, \underline{\tau}_{t+1})}\mid S_{t+1}, S_{t}, T_t\right] \mid S_{t}=s_{t}, T_t=\tau_t\right] \tag{tower law}\\
=~& \operatorname{\mathrm{E}}\left[\operatorname{\mathrm{E}}\left[Y^{(\bar{T}_{t},\underline{\tau}_{t+1})}\mid S_{t+1}\right] \mid S_{t}=s_{t}, T_t=\tau_t\right] \tag{dynSurr}\\
=~& \operatorname{\mathrm{E}}\left[\operatorname{\mathrm{E}}\left[Y^{(\bar{T}_{t}, \underline{\tau}_{t+1})}\mid S_{t+1}, T_{t+1}=\tau_{t+1}\right] \mid S_{t}=s_{t}, T_t=\tau_t\right] \tag{dynExog + overlap}\\
=~& \operatorname{\mathrm{E}}\left[f_{t+1}(S_{t+1}, \tau_{t+1})\mid S_{t}=s_{t}, T_t=\tau_t\right]
\end{align}
Moreover, note that:
\begin{align}
f_{M}(s_{M}, \tau_M) =~& \operatorname{\mathrm{E}}\left[Y^{(\bar{T}_{M-1}, \tau_M)} \mid S_{M}=s_{M}, T_M=\tau_M\right]\\
=~& \operatorname{\mathrm{E}}\left[Y^{(\bar{T}_{M})} \mid S_{M}=s_{M}, T_M=\tau_M\right] \tag{consistency} \\
=~& \operatorname{\mathrm{E}} \left[Y \mid S_{M}=s_{M}, T_M=\tau_M \right] \tag{base case identification}
\end{align}
Thus we have that $f_{M}(s_{M}, \tau_M)$ is identified via the above equation and that by induction, if $f_{t+1}$ has been identified, then $f_{t}$ is identified in terms of $f_{t+1}$, via the recursive equation:
\begin{align}
f_{t}(s_{t}, \tau_t) =~& \operatorname{\mathrm{E}}\left[f_{t+1}(S_{t+1}, \tau_{t+1})\mid S_{t}=s_{t}, T_t=\tau_t\right] \tag{recursive identification}
\end{align}
Thus $f_{t}$ are identified for any $M\geq t \geq 1$.
Finally, note that:
\begin{align}
\theta(\tau) =~& \operatorname{\mathrm{E}}\left[Y^{(\tau)}\right]\\
=~& \operatorname{\mathrm{E}}\left[\operatorname{\mathrm{E}}\left[Y^{(\tau)}\mid S_1\right] \right] \tag{tower law}\\
=~& \operatorname{\mathrm{E}}\left[\operatorname{\mathrm{E}}\left[Y^{(\tau)}\mid S_1, T_1=\tau_1\right] \right] \tag{dynExog}\\
=~& \operatorname{\mathrm{E}}\left[f_1(S_1, \tau_1)\right]
\end{align}
which concludes the proof.
\end{proof}
\section{Automated Debiasing via Recursive Riesz Representers}
We recursively construct a Neyman orthogonal moment for our estimand $\theta(\tau)$. In particular, we recursively apply the Riesz representer theorem to introduce de-biasing terms.
First observe that our estimand is phrased as:
\begin{align}
\theta(\tau) = \operatorname{\mathrm{E}}\left[m_0(Z;f_1)\right] := \operatorname{\mathrm{E}}\left[f_1(S_1,\tau_1)\right]
\end{align}
which is a linear functional of the regression function $f_1$.
Thus we can de-bias the target estimand
with respect to errors in the estimation of $f_1$ by adding a de-biasing term, which will contain residuals
of $f_1$ with the target of the regression. In particular, if we let:
\begin{align}
L_1(g) = \operatorname{\mathrm{E}}\left[g(S_1,\tau_1)\right]
\end{align}
be a linear functional, and $a_1: \ensuremath{{\cal S}}_1\times {\mathcal T}_1\to \mathbb{R}$, \vsedit{be its Riesz representer function, i.e. the function that satisfies:
\begin{align}
L_1(g) = \operatorname{\mathrm{E}}\left[a_1(S_1, T_1) g(S_1, T_1)\right]
\end{align}
Such a Riesz representer is guaranteed to exist when the functional $L_1$ is Lipschitz continuous in the $L^2(P)$ space. This for instance, holds when treatments are discrete and sequential positivity holds. In any case, throughout this section we will make the abstract assumption that $L_1(g)$ has a Reisz representer.} Then the following is a de-biased moment with respect to $f_1$:
\begin{align}
\theta(\tau) = \operatorname{\mathrm{E}}\left[m_1(Z; f_1, a_1, f_2)\right] := \operatorname{\mathrm{E}}\left[f_1(S_1, \tau_1) + a_1(S_1, T_1) (f_2(S_2, \tau_2) - f_1(S_1,T_1))\right]
\end{align}
This moment now contains $f_2$ and is not orthogonal with respect to $f_2$. However, if we look at the linear functional:
\begin{align}
L_2(g) := \operatorname{\mathrm{E}}\left[m_1(Z;f_1, a_1, g)\right] = \operatorname{\mathrm{E}}\left[f_1(S_1, \tau_1) + a_1(S_1, T_1) (g(S_2, \tau_2) - f_1(S_1,T_1))\right]
\end{align}
Note that this functional is linear, and at $g=0$, it takes value \begin{align}
L_2(0)=\operatorname{\mathrm{E}}[f_1(S_1, \tau_1) - a_1(S_1, T_1) f_1(S_1, T_1)]=0, \tag{by definition of $a_1$}
\end{align}
\vsedit{Thus under similar conditions as for $L_1(g)$, it also has an inner product representation and hence a corresponding Riesz representer $a_2: \ensuremath{{\cal S}}_2\times {\mathcal T}_2\to \mathbb{R}$.} Moreover, note that the Riesz representer of $L_2$ is the same as the Riesz representer of the simpler functional $$L_2(g) = \operatorname{\mathrm{E}}[a_1(S_1, T_1) g(S_2, \tau_2)],$$ since this is the only part that depends on $g$.\footnote{Note here, that since this functional depends on $(S_1,S_2,T_1,T_2)$, the global RR $a_2^{global}$ has arguments $(S_1,S_2,T_1,T_2)$. Since $g$ only has arguments $(S_2,T_2)$, there always exists a minimal RR $a_2(S_2,T_2)=\operatorname{\mathrm{E}}[a_2^{global}(S_1,S_2,T_1,T_2)|S_2,T_2]$. In the remainder of the paper, when we talk about a Riesz representer, we will be referring to such a minimal one.}
Thus we can de-bias $m_1$ with respect to $f_2$ by adding a similar Riesz based correction term:
\begin{align}
\theta(\tau) = \operatorname{\mathrm{E}}\left[m_2(Z; f_1, a_1, f_2, a_2, f_3)\right] := \operatorname{\mathrm{E}}\left[m_1(Z;f_1, a_1, f_2) + a_2(S_2, T_2)(f_3(S_3, \tau_3) - f_2(S_2, T_2)\right]
\end{align}
Iteratively, we can then define the de-biased moment for every $t<M$:
\begin{align}
\theta(\tau) = \operatorname{\mathrm{E}}\left[m_t(Z; \bar{f}_{t+1}, \bar{a}_t)\right] = \operatorname{\mathrm{E}}\left[m_{t-1}(Z;\bar{f}_t, \bar{a}_{t-1}) + a_{t}(S_t, T_t) (f_{t+1}(S_{t+1}, \tau_{t+1}) - f_t(S_t, T_t))\right]
\end{align}
where $a_t: \ensuremath{{\cal S}}_t \times {\mathcal T}_t$ is the Riesz representer of the linear functional:
\begin{align}
L_t(g) := \operatorname{\mathrm{E}}\left[a_{t-1}(S_{t-1}, T_{t-1}) g(S_{t}, \tau_{t})\right].
\end{align}
Moreover, for $t=M$, we have that:
\begin{align}
\theta(\tau) = \operatorname{\mathrm{E}}\left[m_{M-1}(Z;\bar{f}_M, \bar{a}_{M-1}) + a_{M}(S_M, T_M) (Y - f_M(S_M, T_M))\right]
\end{align}
which concludes our iterative construction, since no further nuisance components are introduced in this final step.
Thus in the end, if we follow the notational convention of $f_{M+1}(S_{M+1}, \tau_{M+1}):= Y$, we have that an overall de-biased moment is of the form:
\begin{align}
\theta(\tau) = \operatorname{\mathrm{E}}\left[f_1(S_1, \tau_1) + \sum_{t=1}^M a_t(S_t, T_t) \left(f_{t+1}(S_{t+1}, \tau_{t+1}) - f_t(S_t, T_t)\right)\right]
\end{align}
where each $a_t: \ensuremath{{\cal S}}_t\times {\mathcal T}_t\to \mathbb{R}$ is recursively defined as the Riesz representer of the linear functional:
\begin{align}
L_t(g) := \operatorname{\mathrm{E}}\left[ a_{t-1}(S_{t-1}, T_{t-1}) g(S_t, \tau_{t})\right]
\end{align}
where we set $a_{0}(S_0, T_0):=1$.
This leads to the following theorem.
\begin{theorem}[Main Theorem]\label{thm:main}
Suppose Assumptions 1-3 hold. Let $f_{M+1}(S_{M+1}, \tau_{M+1}):= Y$ and $a_{0}(S_0, T_0):=1$. Then the estimand $\theta$ has the debiased representation:
\begin{align}
\theta(\tau) = \operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}_M, \bar{a}_M)\right] := \operatorname{\mathrm{E}}\left[f_1(S_1, \tau_1) + \sum_{t=1}^M a_t(S_t, T_t) \left(f_{t+1}(S_{t+1}, \tau_{t+1}) - f_t(S_t, T_t)\right)\right],
\end{align}
where $f_t$ are recursively defined in Theorem~\ref{thm:non-param-id} and $a_t$ are recursively defined as follows: for all $t\geq 1$, $a_t:\ensuremath{{\cal S}}_t\times {\mathcal T}_t\to \mathbb{R}$ is the Riesz representer of the linear functional:
\begin{align}
L_t(g) := \operatorname{\mathrm{E}}\left[a_{t-1}(S_{t-1}, T_{t-1}) g(S_t, \tau_t)\right].
\end{align}
Then (i) moment $m_M$ is Neyman orthogonal with respect to all nuisance functions $\bar{f}_M$ and $\bar{a}_M$; (ii) For any alternative values of the nuisance functions $\bar{f}^*_M, \bar{a}^*_M$, we have the following mixed bias property:
\begin{align}
\theta^*(\tau) - \theta(\tau) :=~& \operatorname{\mathrm{E}}\left[m_M(Z;\bar{{f}}^*_M, \bar{{a}}^*_M) - m_M(Z;\bar{f}_M, \bar{a}_M)\right]\\
=~& \sum_{t=1}^M \operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) \left(\tilde{f}_{t+1}(S_{t+1}, \tau_{t+1}) - \tilde{f}_{t}(S_t, T_t)\right)\right],
\end{align}
where $\tilde{a}_t := a^*_t - a_t$ and $\tilde{f}_t := f^*_t - f_t$. (iii) The latter property implies the double robustness: if for each $t$, either $\tilde{a}_t=0$ or $\tilde f_{t+1}=\tilde f_t =0$,
then ${\theta}^*(\tau)=\theta(\tau)$.
\end{theorem}
\begin{proof}
Consider the directional derivative with respect to $f_t$, in the direction $\tilde{f}_t$ evaluated at the true $\bar{f}_M, \bar{a}_M$:
\begin{align}
\partial_{f_t}\operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{f}_t] = \operatorname{\mathrm{E}}[a_{t-1}(S_{t-1}, T_{t-1}) \tilde{f}_t(S_t, \tau_t) - a_t(S_t, T_t) \tilde{f}_t(S_t, T_t)]
\end{align}
By the definition of the Riesz representer $a_t$ of functional $L_t(g)$, for $g=\tilde{f}_t$, we have:
\begin{align}
\operatorname{\mathrm{E}}\left[a_{t-1}(S_{t-1}, T_{t-1}) \tilde{f}_t(S_t, \tau_t)\right] = \operatorname{\mathrm{E}}\left[a_{t}(S_{t}, T_{t}) \tilde{f}_t(S_t, T_t)\right]
\end{align}
Thus we have that:
\begin{align}
\partial_{f_t}\operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{f}_t] = 0
\end{align}
Moreover, the directional derivative with respect to $a_t$, in the direction $\tilde{a}_t$ evaluated at the true $\bar{f}_M, \bar{a}_M$:
\begin{align}
\partial_{a_t}\operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{a}_t] =~& \operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) \left(f_{t+1}(S_{t+1}, \tau_{t+1}) - f_t(S_t, T_t)\right)\right] \\
=~& \operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) \left(\operatorname{\mathrm{E}}\left[f_{t+1}(S_{t+1}, \tau_{t+1})\mid S_t, T_t\right] - f_t(S_t, T_t)\right)\right] \tag{tower law}\\
=~& 0 \tag{recursive definition of $f_t$}
\end{align}
Hence, we conclude that the moment is Neyman orthogonal.
Moreover, note that the second order directional derivative is zero for any pair $(a_t, f_{t'})$ such that $t'\notin \{t, t+1\}$ and also it is zero for any pair $(a_t, a_t)$ and $(f_t, f_t)$. Moreover, for any pair $(a_t,f_t)$ the second order directional derivative is of the form:
\begin{align}
\partial_{a_t,f_t}\operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{a}_t, \tilde{f}_t] = -\operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) \tilde{f}_t(S_t, T_t)\right]
\end{align}
and for any pair $(a_t, f_{t+1})$ it is of the form:
\begin{align}
\partial_{a_t,f_{t+1}}\operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{a}_t, \tilde{f}_{t+1}] = \operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) \tilde{f}_{t+1}(S_t, \tau_{t+1})\right]
\end{align}
Thus by an exact second order functional Taylor expansion we can write for any alternative parameter values $\bar{{f}}^*_M$ and $\bar{\hat{a}}^*_M$, and $\tilde{a}_t := {a}^*_t - a_t$ and $\tilde{f}_t := {f}^*_t - f_t$:
\begin{align}
{\theta}^*(\tau) - \theta(\tau) :=~& \operatorname{\mathrm{E}}\left[m_M(Z;\bar{\hat{f}}_M, \bar{\hat{a}}_M) - m_M(Z;\bar{f}_M, \bar{a}_M)\right]\\
=~& \sum_{t=1}^M \operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) \left(\tilde{f}_{t+1}(S_{t+1}, \tau_{t+1}) - \tilde{f}_{t}(S_t, T_t)\right)\right]
\end{align}
\end{proof}
\begin{remark}[Clever co-variate adjustment] Note that instead of adding de-biasing corrections to the moment function, one could also implement the same de-biasing with clever covariate adjustment. In particular, if one has access to the Riesz representers $a_t$, then when running the regression of $f_{t+1}(S_{t+1}, \tau_t)$ on $S_t, T_t$ to estimate the function $f_t$, we add a partially linear regression component of the form: $g(S_t, T_t) + \epsilon_t\cdot a_t(S_t, T_t)$ for some non-linear function $g$, then we note that a square loss minimizer over this function space with an un-penalized $\epsilon_t$, will result in a function estimate $\hat{f}_t$, which satisfies the first order condition: $\operatorname{\mathrm{E}}\left[a_t(S_t,T_t)\, (f_{t+1}(S_{t+1}, \tau_t) - \hat{f}_t(S_t, T_t))\right]=0$, which is exactly the de-biasing correction term associated with $f_t$. Thus if we add such a clever co-variate in each of these regressions, then the resulting de-biasing terms will be identically zero and we can just perform plug-in estimation without further de-biasing. Thus an alternative approach to de-biasing is to first estimate the Riesz representer functions, then run a sequence of nested regressions where at each regression step we also add the Riesz representer as a co-variate in a partially linear manner. This is a Riesz representer based analogue of the clever co-variate adjustment introduced in \cite{bang2005doubly}.
\end{remark}
\section{Riesz Loss Based Estimation}
To estimate the Riesz representers $a_t$ of $L_t$, we will use the Riesz loss based approach introduced in \cite{Chernozhukov2021riesz}, which we provide here for concreteness. Consider the problem of estimating the Reisz representer function $a_0$ of a bounded linear operator:
\begin{equation}
L(g) = \operatorname{\mathrm{E}}[m(Z; g)] = \operatorname{\mathrm{E}}[a_0(Z)\cdot g(Z)]
\end{equation}
Let $\operatorname{\mathrm{E}}_n[\cdot]$ denote the empirical expectation over a sample of size $n$, i.e. $\operatorname{\mathrm{E}}_n[Z]=\frac{1}{n}\sum_{i=1}^n Z_i$. We consider a loss function based approach:
\begin{equation}
\hat{a} = \operatorname*{arg\,min}_{a\in A} \ \operatorname{\mathbb{E}_n}[a(Z)^2 - 2m(Z;a)]
\end{equation}
for some function space $A$ and $Z$ a random variable with support $\ensuremath{{\cal Z}}$. Let $\|\cdot\|_2$ denote the $\ell_2$ norm of a function of a random input, i.e. $\|a\|_2=\sqrt{\operatorname{\mathrm{E}}[a(Z)^2]}$. We also let $\|\cdot\|_{\infty}$ denote the $\ell_{\infty}$ norm, i.e. $\|a\|_{\infty} = \max_{z\in \ensuremath{{\cal Z}}} a(z)$.
\begin{theorem}[\cite{Chernozhukov2021riesz}]\label{thm:riesz-loss}
Let $\delta_{n}$ be an upper bound on the critical radius of the function spaces:
\begin{align}
\ensuremath{\text{star}}(A-a_0) =~& \{z \to \gamma\,(a(z) - a_0(z)): a\in A, \gamma \in [0,1]\}\\
\ensuremath{\text{star}}(m\circ A-m\circ a_0) =~& \{z \to \gamma\,(m(z; a)-m(z;a_0)): a\in A, \gamma \in [0,1]\}
\end{align}
Suppose that $m$ satisfies the mean-squared continuity property:
\begin{align}
\sqrt{\operatorname{\mathrm{E}}[(m(Z;a) - m(Z;a'))^2]} \leq \kappa\,\|a-a'\|_2
\end{align}
and that for all $f\in \ensuremath{\text{star}}(A-a_0)$ and $f\in \ensuremath{\text{star}}(m\circ A-m\circ a_0)$, $\|f\|_{\infty}\leq 1$. Then for some universal constant $C$, we have that w.p. $1-\zeta$:
\begin{align}
\|\hat{a}-a_0\|_2^2 \leq C\left(\delta_n^2\, (1 + \kappa^2) + \|a_*-a_0\|_2^2 + \frac{\kappa\log(1/\zeta)}{n}\right)
\end{align}
where $a_* = \operatorname*{arg\,min}_{a\in A} \|a-a_0\|_2$.
\end{theorem}
\section{Automated Riesz Estimation for Dynamic Effects}
We can thus apply Theorem~\ref{thm:riesz-loss} to the dynamic treatment effect setting for automated de-biasing: for each $t=1,\ldots,M$
\begin{enumerate}
\item Consider the loss function:
\begin{align}
{\mathcal L}_{t,n}(a_t) = \operatorname{\mathbb{E}_n}\left[a_t(S_t, T_t)^2 - 2 \hat{a}_{t-1}(S_{t-1}, T_{t-1}) a_t(S_t, \tau_t)\right]
\end{align}
\item Construct $\hat{a}_t$ by minimizing ${\mathcal L}_{t,n}$ over a class ${\mathcal A}_t$:
\begin{align}
\hat{a}_t = \operatorname*{arg\,min}_{a_t \in {\mathcal A}_t} {\mathcal L}_{t,n}(a_t)
\end{align}
\end{enumerate}
Note that this approach has the caveat that the loss function ${\mathcal L}_{t,n}$ is not simply the empirical analogue of the loss function ${\mathcal L}_t(a_t) := \operatorname{\mathrm{E}}[a_t(S_t, T_t)^2 - 2 a_{t-1}(S_{t-1}, T_{t-1}) a_t(S_t, \tau_t)]$, which is the Riesz loss associated with the linear functional ${\mathcal L}_t$, since we also replace $a_{t-1}$ with $\hat{a}_{t-1}$ in the above equation. Thus we need to be augment Theorem~\ref{thm:riesz-loss} to account for plug-in nuisance errors, of nuisance quantities that appear in our functional.
\subsection{Riesz Loss Based Estimation with Nuisances}
Consider the problem of estimating the Reisz representer function $a_0$ of a bounded linear operator:
\begin{equation}
L(g, h_0) = \operatorname{\mathrm{E}}[m(Z; g, h_0)] = \operatorname{\mathrm{E}}[a_0(Z)\cdot g(Z)]
\end{equation}
Let $\operatorname{\mathrm{E}}_n[\cdot]$ to denote the empirical expectation over a sample of size $n$, i.e. $\operatorname{\mathrm{E}}_n[Z]=\frac{1}{n}\sum_{i=1}^n Z_i$. We consider a loss function based approach:
\begin{equation}
\hat{a} = \operatorname*{arg\,min}_{a\in A} \operatorname{\mathbb{E}_n}[a(Z)^2 - 2m(Z;a, \hat{h})]
\end{equation}
where $\hat{h}$ is some estimate of $h_0$, $A$ is a function space, and $Z$ is a random variable with support $\ensuremath{{\cal Z}}$. As before, let $\|\cdot\|_2$ denote the $\ell_2$ norm of a function of a random input, i.e. $\|a\|_2=\sqrt{\operatorname{\mathrm{E}}[a(Z)^2]}$. We also let $\|\cdot\|_{\infty}$ denote the $\ell_{\infty}$ norm, i.e. $\|a\|_{\infty} = \max_{z\in \ensuremath{{\cal Z}}} a(z)$.
\begin{theorem}\label{thm:riesz-loss-nuisance}
Let $a_*=\operatorname*{arg\,min}_{a\in A} \|a-a_0\|_2$. Let $\delta_n\geq \sqrt{\frac{\log\log(n)}{n}}$ be an upper bound on the critical radius of the function spaces:
\begin{align}
\ensuremath{\text{star}}(A-a_*) =~& \{z \to \gamma\,(a(z) - a_*(z)): a\in A, \gamma\in [0, 1]\}\\
\ensuremath{\text{star}}(m\circ A \circ H - m\circ a_* \circ h_0) =~& \{z \to \gamma\,(m(z; a, h)-m(z;a_*, h_0)): a\in A, h\in H, \gamma \in [0,1]\}
\end{align}
Suppose that $m$ satisfies the following continuity properties:
\begin{align}
\forall h\in H, a,a'\in A: \sqrt{\operatorname{\mathrm{E}}[(m(Z;a, h) - m(Z;a', h))^2]} \leq~& \kappa\|a-a'\|_2\\
\forall a\in A, h\in H: \sqrt{\operatorname{\mathrm{E}}[(m(Z;a, h) - m(Z;a, h_0))^2]} \leq~& \kappa\|h-h_0\|_2\\
\forall a\in A, h\in H: |\operatorname{\mathrm{E}}\left[m(Z;a-a_*,h) - m(Z;a-a_*,h_0)\right]| \leq~& \kappa \|a-a_*\|_2 \|h-h_0\|_2
\end{align}
for some $\kappa$ and that for all $f\in \ensuremath{\text{star}}(A-a_*)$ and $f\in \ensuremath{\text{star}}(m\circ A \circ H-m\circ a_*\circ h_0)$, $\|f\|_{\infty}\leq 1$. Then for some universal constants $C, c_0, c_1$, if we let $\delta = \delta_n + c_0\sqrt{\frac{\log(c_1/\zeta)}{n}}$, we have that w.p. $1-\zeta$:
\begin{align}
\|\hat{a}-a_0\|_2^2 \leq~& O\left(\delta^2 (1 + \kappa^2) + \|a_* - a_0\|_2^2 + (1+\kappa^2)\|\hat{h}-h_0\|_2^2 \right)
\end{align}
\end{theorem}
\begin{proof}
Consider the following notation:
\begin{align}
{\mathcal L}(a, h) =~& \operatorname{\mathrm{E}}[a(Z)^2 -2m(Z;a, h)]\\
{\mathcal L}_n(a, h) =~& \operatorname{\mathrm{E}}_n[a(Z)^2 -2m(Z;a, h)]
\end{align}
Note that:
\begin{align}
{\mathcal L}(a, h_0) = \operatorname{\mathrm{E}}[a(Z)^2 - 2a_0(Z)a(Z)]
\end{align}
By the definition of the Reisz representer we have for any $a\in A$:
\begin{align}
{\mathcal L}(a, h_0) - {\mathcal L}(a_0, h_0) = \operatorname{\mathrm{E}}[a(Z)^2 - 2a_0(Z)a(Z)] + \operatorname{\mathrm{E}}[a_0(Z)^2] = \operatorname{\mathrm{E}}[(a(Z) - a_0(Z))^2] = \|a-a_0\|_2^2
\end{align}
Let $a_*=\operatorname*{arg\,min}_{a\in A} \|a-a_0\|_2$ and let:
\begin{align}
\ell(z; a, h) = a(z)^2 - 2m(z, a, h) - (a_*(z)^2 - 2m(z; a_*, h))
\end{align}
Note that $\ell(Z;a_*,h_0)=0$ and that $\ell$ is $6$-Lipschitz with respect to the vector $(m(z;a, h), m(z; a_*, h), a(z))$, since the gradient of the function $\ell$ with respect to these components is $(-2, 2, 2 a(z))$, which has an $\ell_2$ norm bounded by $6$ (since $|a(z)|\leq 1$).
By Lemma~11 of \cite{foster2019orthogonal}, and by our choice of $\delta:=\delta_n + c_0 \sqrt{\frac{\log(c_1/\zeta)}{n}}$, where $\delta_n$ is an upper bound on the critical radius of $\ensuremath{\text{star}}(A-a_0)$ and $\ensuremath{\text{star}}(m\circ A\circ H -m\circ a_*\circ h_0)$ and $\ensuremath{\text{star}}(m\circ a_*\circ H -m\circ a_*\circ h_0)$, w.p. $1-\zeta$: $\forall a\in A, h\in H$
\begin{align}
\left|{\mathcal L}_n(a, h) - {\mathcal L}_n(a_*, h) - ({\mathcal L}(a, h) - {\mathcal L}(a_*, h))\right| =~& \left| \operatorname{\mathbb{E}_n}[\ell(Z;a,h) - \ell(Z; a_*, h_0)] - \operatorname{\mathrm{E}}\left[\ell(Z; a,h) - \ell(Z;a_*, h_0)\right]\right|\\
\leq~& O\left(\delta \left(\|a-a_*\|_2 + \sqrt{\operatorname{\mathrm{E}}[(m(Z;a, h)-m(Z;a_*, h_0))^2]}\right)\right)\\
~& + O\left(\delta\left(\sqrt{\operatorname{\mathrm{E}}[(m(Z;a_*, h)-m(Z;a_*, h_0))^2]}\right) + \delta^2\right)\\
\leq~& O\left(\delta \left(\|a-a_*\|_2 + \sqrt{\operatorname{\mathrm{E}}[(m(Z;a, h)-m(Z;a_*, h))^2]}\right)\right)\\
~& + O\left(\delta\left(\sqrt{\operatorname{\mathrm{E}}[(m(Z;a_*, h)-m(Z;a_*, h_0))^2]}\right) + \delta^2\right)\\
=~& O\left(\delta\, \kappa\, \left(\|a-a_*\|_2 + \|h-h_0\|_2\right) + \delta^2\right) =: \epsilon_1(a, h)
\end{align}
Moreover, since $\hat{a}$ is the minimizer of ${\mathcal L}_n(a, \hat{h})$ over $A$ and since $a_*\in A$, we have that:
\begin{align}
{\mathcal L}_n(\hat{a}, \hat{h}) - {\mathcal L}_n(a_*, \hat{h}) \leq 0
\end{align}
Combining all the above we have:
\begin{align}
{\mathcal L}(\hat{a}, \hat{h}) - {\mathcal L}(a_*, \hat{h}) \leq~& {\mathcal L}_n(\hat{a}, \hat{h}) - {\mathcal L}_n(a_*, \hat{h}) + \epsilon_1
\leq \epsilon_1
\end{align}
By Lipschitzness of $m$ with respect to $h$, we also have that:
\begin{align}
\left|\left({\mathcal L}(\hat{a}, \hat{h}) - {\mathcal L}(a_*, \hat{h})\right) - \left({\mathcal L}(\hat{a}, h_0) - {\mathcal L}(a_*, h_0)\right)\right| =~&
2\left| \operatorname{\mathrm{E}}[m(Z;\hat{a}-a_*, \hat{h}) - m(Z;\hat{a}-a_*, h_0)]\right|\\
\leq~& 2\,\kappa \|\hat{a}-a_*\|_2\, \|\hat{h}-h_0\|_2
\end{align}
Finally, by the definition of the Riesz representer:
\begin{align}
{\mathcal L}(\hat{a}, h_0) - {\mathcal L}(a_*, h_0) =~& {\mathcal L}(\hat{a}, h_0) - {\mathcal L}(a_0, h_0) + {\mathcal L}(a_0, h_0) - {\mathcal L}(a_*, h_0)\\
=~& \|\hat{a}-a_0\|_2^2 - \|a_* - a_0\|_2^2
\end{align}
Hence, combining the above inequalities we have:
\begin{align}
\|\hat{a}-a_0\|_2^2 - \|a_* - a_0\|_2^2 =~& {\mathcal L}(\hat{a}, h_0) - {\mathcal L}(a_*, h_0)\\
\leq~& {\mathcal L}(\hat{a}, \hat{h}) - {\mathcal L}(a_*, \hat{h}) + O\left(\kappa\|\hat{a}-a_*\|_2\, \|h-h_0\|_2\right)\\
\leq~& \epsilon_1(\hat{a}, \hat{h}) + O\left(\kappa\|\hat{a}-a_*\|_2\, \|\hat{h}-h_0\|_2\right)\\
=~& O\left(\delta\, \kappa\, \left(\|\hat{a}-a_*\|_2 + \|\hat{h}-h_0\|_2\right) + \delta^2 + \kappa\|\hat{a}-a_*\|_2\, \|\hat{h}-h_0\|_2\right)
\end{align}
We can thus conclude that for some universal constant $C$:
\begin{align}
\|\hat{a}-a_0\|_2^2 \leq~& \|a_* - a_0\|_2^2 + C\left(\delta\, \kappa\, \left(\|\hat{a}-a_*\|_2 + \|\hat{h}-h_*\|_2\right) + \delta^2 + \kappa\|\hat{a}-a_*\|_2\, \|\hat{h}-h_0\|_2\right)\\
\leq~& \|a_* - a_0\|_2^2 + C\left(4C\delta^2\, \kappa^2 + \frac{1}{16C}\left(\|\hat{a}-a_*\|_2 + \|\hat{h}-h_*\|_2\right)^2 + \delta^2 + \kappa \|\hat{a}-a_*\|_2\, \|\hat{h}-h_0\|_2\right)\\
\leq~& \|a_* - a_0\|_2^2 + C\left(4C\delta^2\, \kappa^2 + \frac{1}{8C}\|\hat{a}-a_*\|_2^2 + \frac{1}{8C}\|\hat{h}-h_*\|_2^2 + \delta^2 + \kappa\|\hat{a}-a_*\|_2\, \|\hat{h}-h_0\|_2\right)\\
\leq~& \|a_* - a_0\|_2^2 + C\left(4C\delta^2\, \kappa^2 + \frac{1}{4C}\|\hat{a}-a_*\|_2^2 + \frac{1}{8C}\|\hat{h}-h_0\|_2^2 + \delta^2 + 4C\kappa^2 \|\hat{h}-h_0\|_2^2\right)\\
\leq~& \frac{1}{2} \|\hat{a}-a_0\|_2^2 + O\left(\delta^2 (1 + \kappa^2) + \|a_* - a_0\|_2^2 + (1+\kappa^2)\|\hat{h}-h_0\|_2^2 \right)
\end{align}
where we invoked repeatedly the AM-GM inequality ($a\cdot b = \sqrt{a^2/\sigma} \cdot \sqrt{\sigma b^2} \leq \frac{a^2}{2\sigma} + \frac{\sigma b^2}{2}$).
Re-arranging, yields the desired statement:
\begin{align}
\|\hat{a}-a_0\|_2^2 \leq~& O\left(\delta^2 (1 + \kappa^2) + \|a_* - a_0\|_2^2 + (1+\kappa^2)\|\hat{h}-h_0\|_2^2 \right)
\end{align}
\end{proof}
\subsection{Application to Recursive Riesz Estimation}
For every $t$ we can apply Theorem~\ref{thm:riesz-loss-nuisance} with $H=A_{t-1}$, $A=A_t$ and $L(g, h) = \operatorname{\mathrm{E}}\left[h(S_{t-1}, T_{t-1})\, g(S_t,\tau_t)\right]$. Note that the continuity properties are satisfied if (i) the function classes $A_t$ that we use for the Riesz representers are bounded in some finite range $[-H, H]$; and (ii) $\ensuremath{\mathrm{Pr}}(T_t=\tau_t\mid S_t)\geq \lambda > 0$:
\begin{align}
\operatorname{\mathrm{E}}\left[h(S_{t-1}, T_{t-1})^2 (a(S_t,\tau_t) - a'(S_t,\tau_t))^2\right] \leq~& H^2 \operatorname{\mathrm{E}}\left[(a(S_t,\tau_t) - a'(S_t,\tau_t))^2\right]\\
\leq~& \frac{H^2}{\lambda} \operatorname{\mathrm{E}}\left[ \ensuremath{\mathrm{Pr}}(T_t=\tau_t\mid S_t)\, (a(S_t,\tau_t) - a'(S_t,\tau_t))^2\right]\\
\leq~& \frac{H^2}{\lambda} \operatorname{\mathrm{E}}\left[ \ensuremath{\mathrm{Pr}}(T_t=\tau_t\mid S_t)\, \operatorname{\mathrm{E}}\left[(a(S_t,\tau_t) - a'(S_t,\tau_t))^2\mid S_t\right]\right]\\
\leq~& \frac{H^2}{\lambda}\operatorname{\mathrm{E}}\left[ (a(S_t,T_t) - a'(S_t,T_t))^2\right] = \frac{H^2}{\lambda} \|a-a'\|_2^2
\end{align}
Similarly:
\begin{align}
\operatorname{\mathrm{E}}\left[(h(S_{t-1}, T_{t-1}) -h_0(S_{t-1}, T_{t-1}))^2 a(S_t,\tau_t)^2\right] \leq H^2 \|h-h_0\|_2^2
\end{align}
and
\begin{align}
|\operatorname{\mathrm{E}}\left[m(Z;a-a_*,h) - m(Z;a-a_*,h_0)\right]| =~& |\operatorname{\mathrm{E}}\left[(h(S_{t-1}, T_{t-1}) -h_0(S_{t-1}, T_{t-1}))\, (a(S_t,\tau_t) - a_*(S_t,\tau_t))\right]|\\
\leq~& \|h-h'\|_2\, \sqrt{\operatorname{\mathrm{E}}\left[(a(S_t,\tau_t) - a_*(S_t,\tau_t))^2\right]}\\
\leq~& \|h-h'\|_2\, \sqrt{\kappa\operatorname{\mathrm{E}}\left[(a(S_t,T_t) - a_*(S_t,T_t))^2\right]}\\
=~& \frac{1}{\sqrt{\lambda}} \|h-h'\|_2\, \|a-a_*\|_2
\end{align}
where the third line uses the reasoning above.
Thus we have that as long as $\delta_n\geq \sqrt{\frac{\log\log(n)}{n}}$ upper bounds the critical radius of:
$\ensuremath{\text{star}}(A_t-a_{t,*})$ and $\ensuremath{\text{star}}(m\circ A_t \circ A_{t-1} - m\circ a_{t,*} \circ a_{t-1, 0})$ (with $a_{t,*}=\operatorname*{arg\,min}_{a\in A_t} \|a_t - a_{t,0}\|_2$ and $a_{t,0}$ is the true Riesz representer for functional $L_t$),
then we get the corresponding fast rate. The latter critical radius can also be upper bounded as a function of the entropy integral of the function class $A_t$ and $A_{t-1}$ separately.
Thus we can derive a bound on the rate of convergence of $\hat{a}$ outlined by the iterative Riesz estimation process. For simplicity of stating the corollary we will also assume that $a_{t, 0} \in A$, though a more general statement can be made, accounting also for the bias terms $\|a_{t,*}-a_{t,0}\|_2=\min_{a_t\in A} \|a_t-a_{t,0}\|_2$.
\begin{corollary}
Suppose: (i) correct specification $a_{t,0}\in A_t$; (ii) the function spaces $A_t$ contain uniformly bounded functions; (iii) the setting satisfies strict positivity, i.e. $\ensuremath{\mathrm{Pr}}(T_t=\tau_t\mid S_t)\geq \lambda >0$ a.s. for some constant $\lambda$; (iv) $\delta_{t,n}\geq \sqrt{\frac{\log\log(n)}{n}}$ upper bounds the critical radius of
$\ensuremath{\text{star}}(A_t-a_{t,0})$ and $\ensuremath{\text{star}}(m\circ A_t \circ A_{t-1} - m\circ a_{t,0} \circ a_{t-1, 0})$. Then, if we let $\delta_t = \delta_{t,n} + c_0\sqrt{\frac{\log(c_1/\zeta)}{n}}$ for some universal constants $(c_0,c_1)$, we have that w.p. $1-\zeta$:
\begin{align}
\|\hat{a}_t-a_{t,0}\|_2^2 \leq~& O\left(\delta^2 + \|\hat{a}_{t-1}-a_{t-1,0}\|_2^2 \right)
\end{align}
Hence, for any constant time horizon $M$, we have that w.p. $1-\zeta$:
\begin{align}
\forall t\in [1, M]: \|\hat{a}_t-a_{t,0}\|_2^2 \leq~& O\left(\max_{t'\leq t} \delta_{t'}^2\right)
\end{align}
\end{corollary}
\section{Extension: Nested Linear Moment Functionals}
We note that even though throughout we consider a counterfactual static treatment sequence $\tau$, all our results naturally extend to any deterministic or randomized policy counterfactual dynamic policy $\pi_t: \ensuremath{{\cal S}}_{t-1} \to {\mathcal T}_t$.
Moreover, note that even though we used a Markovian notation where observational policies and outcomes only depend on current state $\ensuremath{{\cal S}}_t$, one may interpret $\ensuremath{{\cal S}}_t$ as the current sufficient statistic of the history up until time $t$. For instance, $\ensuremath{{\cal S}}_t$ can contain all prior treatments and all prior states. Therefore our analysis applies to many settings.
Finally, our results also naturally extend to contrasts of counterfactual outcomes or counterfactual policies, or any other weighted linear combination of counterfactual outcomes. In general, if we can show that our estimand takes the form of a nested moment equation of the form:
\begin{align}
\theta =~& \operatorname{\mathrm{E}}[m_1(Z; f_1)]\\
\forall 1\leq t < M: f_{t}(S_{t}, T_t) =~& \operatorname{\mathrm{E}}\left[m_{t+1}(Z; f_{t+1})\mid S_{t}, T_t\right] \tag{recursive estimand}\\
f_{M}(S_{M}, T_M) =~& \operatorname{\mathrm{E}}\left[Y\mid S_{M}, T_M\right] \tag{base estimand}
\end{align}
where $m_t$ are linear moments in $f_t$, which is a generalization of the estimand presented in Theorem~\ref{thm:non-param-id}, then our approach easily extends. The linear functionals in this case take the form: $L_t(g) = \operatorname{\mathrm{E}}\left[a_{t-1}(S_{t-1},T_{t-1})\, m_t(Z;g)\right]$. Our main setting was a special case where $m_t(Z; g) = g(S_t, \tau_t)$. Linear combinations of counterfactual quantities fall into the latter more general category for more complex moment functions $m_t$ that involve evaluation of the function $g$ at multiple treatment points.
\section{Extension: Nested Non-Linear Functionals}
We now consider a more general type of estimand, defined as the solution to nested non-linear moment equations. In particular, we have that $\theta$ is defined via the set of equations:
\begin{align}
\operatorname{\mathrm{E}}[m_1(Z; \theta, f_1)] =~& 0 \label{eqn:non-lin-estimand}\\
\forall 1\leq t < M, \forall j\in [d_t]: f_{t,j}(X_{t,j}) =~& \operatorname{\mathrm{E}}\left[m_{t+1, j}(Z; f_{t+1})\mid X_{t,j}\right] \tag{recursive estimand}\\
f_{M, j}(X_{M,j}) =~& \operatorname{\mathrm{E}}\left[Y_j\mid X_{M,j}\right] \tag{base estimand}
\end{align}
where $m_{t+1}$ is a sequence of potentially non-linear vector-valued functionals, $f_t$ is a $d_t$-dimensional vector-valued function and $Y\in {\mathcal Y}$, $X_t \in {\mathcal X}_t$ are sub-vectors of the random vector $Z$.
\begin{theorem}[Main Theorem for Non-Linear Functionals]
For notational convenience define: $m_{M+1,j}(Z, f_{M+1}):= Y_j$ and $a_{0,j}(X_0):=1$ and let:
\begin{align}
m_M^*(Z; \theta, \bar{f}_M, \bar{a}_M) := m_1(Z; \theta, f_1) + \sum_{t=1}^M \sum_{j=1}^{d_t} a_{t,j}(X_{t,j})' \left(m_{t+1,j}(Z; f_{t+1}) - f_{t,j}(X_{t,j})\right)
\end{align}
where $\bar{f}_M = \{f_1, \ldots, f_M\}$, $\bar{a}_M=\{a_1, \ldots, a_M\}$, $f_t$ are recursively defined in Equation~\ref{eqn:non-lin-estimand} and $a_t$ are recursively defined as follows: for all $t\geq 1$ and $j\in [d_t]$, $a_{t,j}:{\mathcal X}_t\to \mathbb{R}$ is the Riesz representer of the linear functional, with respect to $g$:
\begin{align}
L_{t,j}(g; f_{t,j}) := \operatorname{\mathrm{E}}\left[\sum_{j=1}^{d_{t-1}} a_{t-1,j}(X_{t-1,j}) \frac{\partial}{\partial \tau} m_{t,j}(Z; f_{t,j} + \tau g, f_{t,-j})\bigr\vert_{\tau=0}\right]
\end{align}
Then the estimand $\theta_0$ can be identified by the moment equation:
\begin{align}
\operatorname{\mathrm{E}}\left[m_M^*(Z; \theta, \bar{f}_M, \bar{a}_M)\right] = 0
\end{align}
Moreover, the moment $m_M^*$ is Neyman orthogonal with respect to all nuisance functions $\bar{f}_M, \bar{a}_M$.
\end{theorem}
\begin{proof}
Consider the directional derivative with respect to $f_{t,j}$, in the direction $\tilde{f}_{t,j}$ evaluated at the true $\bar{f}_M, \bar{a}_M$:
\begin{align}
\partial_{f_{t,j}}\operatorname{\mathrm{E}}\left[m_M^*(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{f}_{t,j}] = L_{t,j}(\tilde{f}_{t,j}; f_{t,j}) - \operatorname{\mathrm{E}}[a_{t,j}(X_{t,j}) \tilde{f}_{t,j}(X_{t,j})]
\end{align}
By the definition of the Riesz representer $a_{t,j}$ of functional $L_{t,j}(g;f_{t,j})$, for $g=\tilde{f}_{t,j}$, we have:
\begin{align}
L_{t,j}(\tilde{f}_{t,j}; f_{t,j}) = \operatorname{\mathrm{E}}\left[a_{t}(X_{t,j})' \tilde{f}_t(X_{t,j})\right]
\end{align}
Thus we have that:
\begin{align}
\partial_{f_{t,j}}\operatorname{\mathrm{E}}\left[m_M^*(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{f}_{t,j}] = 0
\end{align}
Moreover, the directional derivative with respect to $a_{t,j}$, in the direction $\tilde{a}_{t,j}$ evaluated at the true $\bar{f}_M, \bar{a}_M$:
\begin{align}
\partial_{a_{t,j}}\operatorname{\mathrm{E}}\left[m_M^*(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{a}_{t,j}] =~& \operatorname{\mathrm{E}}\left[\tilde{a}_{t,j}(X_{t,j}) \left(m_{t+1,j}(Z; f_{t+1}) - f_{t,j}(X_{t,j})\right)\right] \\
=~& \operatorname{\mathrm{E}}\left[\tilde{a}_{t,j}(X_{t,j})' \left(\operatorname{\mathrm{E}}\left[m_{t+1,j}(Z; f_{t+1})\mid X_{t,j}\right] - f_{t,j}(X_{t,j})\right)\right] \\
=~& 0 \tag{recursive definition of $f_t$}
\end{align}
Hence, we conclude that the moment is Neyman orthogonal.
\end{proof}
Estimation of the recursive Riesz representers can be done in an identical manner as for linear nested moments. Albeit now we need to do it in a sequential manner where we first estimate the nested regression estimates $\hat{f}_t$, then calculate the functional derivatives at $\hat{f}_t$, i.e. if we let
$$\partial m_t(Z; f_t, g) = \frac{\partial}{\partial \tau} m_t(Z; f_t + \tau g)\bigr\vert_{\tau=0}$$
and we let:
\begin{align}
L_t(g; f_t) = \operatorname{\mathrm{E}}\left[a_{t-1}(X_{t-1})' \partial m_t(Z; f_t, g)\right]
\end{align}
then we will run the Riesz loss approach on the linear functional $L_t(g; \hat{f}_t)$. Assuming that the moments $m_t$ is sufficiently smooth with respect to $f$, then we can also approximate $\partial m_t$ by a finite difference:
\begin{align}
\hat{\partial} m_t(Z; f_t, g) \approx \frac{m_t(Z; f_t + \epsilon g) - m_t(Z; f_t)}{\epsilon}
\end{align}
which for $\epsilon = o(n^{-1})$ will add negligible extra error, i.e. $|\partial m_t(Z; f_t, g) - \hat{\partial} m_t(Z; f_t, g)| = o(\epsilon)$.
\subsection{Application: Dynamic Discrete Choice}
Consider a dynamic discrete choice problem where at each period a decision maker chooses either to renew $Y_{t}=1$ or not renew $Y_t=0$. Conditional on renewal, the next period state is independent of first period state. We have that:
\begin{align}
R_{jt} := D_j(X_t)'\theta + \epsilon_{jt}
\end{align}
with $D_{1}(X_t) = (1, 0, \ldots, 0)$ and $D_{00}(X_t) = 0$ and with $\epsilon_{jt}$ exogenous random shocks that are mean zero and identically distributed across time and actions and independently of $X_t$ and with a known distribution. Let $\sigma(x) = \ensuremath{\mathrm{Pr}}(Y_t=1\mid X_t=x)$ be the probability of renewal and let $V(x)$ denote the value function. Moreover, let:
\begin{align}
v_1(x) :=~& \theta_0 + \delta \operatorname{\mathrm{E}}[V(X_{t+1}) \mid Y_t=1] =: V_1\\
v_0(x) :=~& D_0(X_t)'\theta_0 + \delta \operatorname{\mathrm{E}}[V(X_{t+1}) \mid X_t=x, Y_t=0]
\end{align}
We assume that the decision maker chooses an action that maximizes the $\delta$-discounted reward. By the Bellman equation we can write:
\begin{align}
Y_t = \operatorname*{arg\,max}_{j\in \{0, 1\}} v_j(X_t) + \epsilon_{jt}
\end{align}
Note that:
\begin{align}
\sigma(x) := \ensuremath{\mathrm{Pr}}(Y_t=1\mid X_t=x) = \ensuremath{\mathrm{Pr}}(\epsilon_{0t} - \epsilon_{1t} \leq v_1(X_t) - v_0(X_t)\mid X_t=x) = \Lambda(v_1(x) - v_0(x))
\end{align}
where $\Lambda$ is the CDF of shock differences $\epsilon_{0t}-\epsilon_{1t}$. We further assume that $\Lambda$ is invertible and hence we can write $v_1(x) - v_0(x) = \Lambda^{-1}(\sigma(x))$.
Then note that we can write:
\begin{align}
V(x) =~& \int \max_{j\in \{0,1\}} \{v_j(x) + \epsilon_{j}\}f(\epsilon) d\epsilon
= V_1 + \int \max_{j\in \{0,1\}} \{v_j(x) - v_1(x) + \epsilon_{j}\}f(\epsilon) d\epsilon\\
=~& V_1 + \int \max\{\epsilon_{1}, - \Lambda^{-1}(\sigma(x)) + \epsilon_0\}f(\epsilon) d\epsilon =: V_1 + Q(\sigma(x))
\end{align}
Since $f$ is assumed to be known, the function $Q$ is also known.\footnote{For the case of type I extreme value distributed shocks we have that $\Lambda$ is the logistic function, i.e. $\Lambda(v) = \frac{1}{1 + \exp\{-v\}}$ and that $\Lambda^{-1}(\sigma) = \log(\sigma) - \log(1-\sigma)$ and $Q(\sigma)= \gamma_{E} - \log(1 - \sigma)$, where $\gamma_E$ is the Euler constant.} Thus we get that:
\begin{align}
\operatorname{\mathrm{E}}[V(X_{t+1})\mid Y_t=1] - \operatorname{\mathrm{E}}[V(X_{t+1})\mid X_t, Y_t=0]
= \operatorname{\mathrm{E}}\left[Q(\sigma(X_{t+1}))\mid Y_t=1\right] - \operatorname{\mathrm{E}}[Q(\sigma(X_{t+1}))\mid X_t, Y_t=0]
\end{align}
for some known function $Q$. If we let: $q := \ensuremath{\mathrm{Pr}}[Y_t=1]$
\begin{align}
\sigma_0(x) :=~& \operatorname{\mathrm{E}}[Q(\sigma(X_{t+1}))\mid X_t=x, Y_t=0] = \operatorname{\mathrm{E}}\left[\frac{Q(\sigma(X_{t+1})) (1 - Y_t)}{\sigma(X_t)}\mid X_t=x\right]\\
\sigma_1 :=~& \operatorname{\mathrm{E}}\left[Q(\sigma(X_{t+1}))\mid Y_t=1\right] = \operatorname{\mathrm{E}}\left[\frac{Q(\sigma(X_{t+1}))\, Y_t}{q}\right] \\
D(x) :=~& D_1(x) - D_0(x)
\end{align}
Then we can also write:
\begin{align}
\ensuremath{\mathrm{Pr}}(Y_t=1\mid X_t=x) =~& \Lambda(v_1(x) - v_0(x))\\
=~& \Lambda\left(D(X_t)'\theta + \delta\, (\operatorname{\mathrm{E}}[V(X_{t+1})\mid Y_t=1] - \operatorname{\mathrm{E}}[V(X_{t+1})\mid X_t=x, Y_t=0])\right)\\
=~& \Lambda\left(D(X_t)'\theta + \delta\, (\sigma_1 - \sigma_0(x)) \right)
\end{align}
which yields the nested non-linear identifying moment vector for $\theta$:
\begin{align}
m_1(Z;\theta, \sigma_0, \sigma_1) :=~& D(X_t)\, \Lambda'(u(X_t; \theta, \sigma_0, \sigma_1))\, (Y_t - \Lambda(u(X_t; \theta, \sigma_0, \sigma_1)))\\
u(X_t; \theta, \sigma_0, \sigma_1) :=~& D(X_t)'\theta + \delta\,(\sigma_1 - \sigma_0(X_t))
\end{align}
\paragraph{Casting as nested non-linear functionals.} Note that this is of the form in the previous section, where $m_1$ is as defined above, $Z := (X_t, Y_t, X_{t+1})$ and:
\begin{align}
f_1 :=~&(\sigma_0, \sigma_1) &
f_2 :=~& (\sigma, q)\\
(X_{1,1}, X_{1,2}) :=~& (X_t, \emptyset) &
(X_{2,1}, X_{2,2}) :=~& (X_{t+1}, \emptyset) &
(Y_1, Y_2) =~& (Y_t, Y_t)\\
m_{2, 1}(Z;f_2) =~& \frac{Q(f_{2,1}(X_{t+1}))\, (1 - Y_t)}{f_{2,1}(X_t)} &
m_{2, 2}(Z;f_2) =~& \frac{Q(f_{2,1}(X_{t+1}))Y_t}{f_{2,2}}\\
\end{align}
We note that to apply our general framework to this setting it suffices to have samples of ``transition triplets'' of the form $(X_t^{(i)}, Y_t^{(i)}, X_{t+1}^{(i)})$ from each unit $i$. However, having trajectories from each unit $i$ can always improve efficiency and we can perform averaging across the transition triplets within unit, not treating them as independent. So we can use the estimator:
\begin{align}
\frac{1}{n} \sum_{i=1}^n \frac{1}{T} \sum_{t=1}^T m^*(X_{t}^{(i)}, Y_{t}^{(i)}, X_{t+1}^{(i)}; \theta, \bar{f}_2, \bar{a}_2) = 0
\end{align}
But treat $\frac{1}{T} \sum_{t=1}^T m^*(X_{i,t}, Y_{i,t} X_{i, t+1})$ as the moment stemming from a single observation and perform asymptotics in $n$.
\section{Extension: Nested Non-Linear IV Functionals}
We now consider a more general type of estimand, defined as the solution to nested non-linear instrumental variable moment equations. In particular, we have that $\theta$ is defined via the set of equations:
\begin{align}
\operatorname{\mathrm{E}}[m_1(Z; \theta, f_1)] =~& 0 \label{eqn:non-lin-estimand-iv}\\
\forall 1\leq t < M, \forall j\in [d_t]: \operatorname{\mathrm{E}}\left[f_{t,j}(X_{t,j}) - m_{t+1, j}(Z; f_{t+1})\mid V_{t,j}\right] =~& 0 \tag{recursive estimand}\\
\operatorname{\mathrm{E}}\left[f_{M, j}(X_{M,j}) - Y_j\mid V_{M,j}\right] =~& 0 \tag{base estimand}
\end{align}
where $m_{t+1}$ is a sequence of potentially non-linear vector-valued functionals, $f_t$ is a $d_t$-dimensional vector-valued function and $Y_j, V_{t,j}, X_{t,j}$ are sub-vectors of the random vector $Z$.
\begin{theorem}[Main Theorem for Non-Linear Functionals]
For notational convenience define: $m_{M+1,j}(Z, f_{M+1}):= Y_j$ and $\mu_{0,j}(X_0):=1$ and let:
\begin{align}
m_M^*(Z; \theta, \bar{f}_M, \bar{a}_M) := m_1(Z; \theta, f_1) + \sum_{t=1}^M \sum_{j=1}^{d_t} \mu_{t,j}(V_{t,j})' \left(m_{t+1,j}(Z; f_{t+1}) - f_{t,j}(X_{t,j})\right)
\end{align}
where $\bar{f}_M = \{f_1, \ldots, f_M\}$, $\bar{\mu}_M=\{\mu_1, \ldots, \mu_M\}$, $f_t$ are recursively defined in Equation~\ref{eqn:non-lin-estimand-iv} and $a_t$ are recursively defined as follows: for all $t\geq 1$ and $j\in [d_t]$, $a_{t,j}:{\mathcal X}_t\to \mathbb{R}$ is the Riesz representer of the linear functional, with respect to $g$:
\begin{align}
L_{t,j}(g; f_{t,j}) := \operatorname{\mathrm{E}}\left[\sum_{j=1}^{d_{t-1}} \mu_{t-1,j}(V_{t-1,j}) \frac{\partial}{\partial \tau} m_{t,j}(Z; f_{t,j} + \tau g, f_{t,-j})\bigr\vert_{\tau=0}\right]
\end{align}
Then $\mu_{t,j}$ is defined based on $a_{t,j}$ as the solution to the following conditional moment equation:
\begin{align}
\operatorname{\mathrm{E}}\left[\mu_{t,j}(V_{t,j}) - a_{t,j}(X_{t,j}) \mid X_{t,j}\right] =0
\end{align}
Then the estimand $\theta_0$ can be identified by the moment equation:
\begin{align}
\operatorname{\mathrm{E}}\left[m_M^*(Z; \theta, \bar{f}_M, \bar{a}_M)\right] = 0
\end{align}
Moreover, the moment $m_M^*$ is Neyman orthogonal with respect to all nuisance functions $\bar{f}_M, \bar{a}_M$.
\end{theorem}
\begin{proof}
Consider the directional derivative with respect to $f_{t,j}$, in the direction $\tilde{f}_{t,j}$ evaluated at the true $\bar{f}_M, \bar{a}_M$:
\begin{align}
\partial_{f_{t,j}}\operatorname{\mathrm{E}}\left[m_M^*(Z;\bar{f}_M, \bar{\mu}_M)\right][\tilde{f}_{t,j}] =~& L_{t,j}(\tilde{f}_{t,j}; f_{t,j}) - \operatorname{\mathrm{E}}[\mu_{t,j}(V_{t,j}) \tilde{f}_{t,j}(X_{t,j})]\\
=~& \operatorname{\mathrm{E}}[a_{t,j}(X_{t,j}) \tilde{f}_{t,j}(X_{t,j})] - \operatorname{\mathrm{E}}[\mu_{t,j}(V_{t,j}) \tilde{f}_{t,j}(X_{t,j})]\\
=~& \operatorname{\mathrm{E}}[\operatorname{\mathrm{E}}\left[a_{t,j}(X_{t,j}) - \mu_{t,j}(V_{t,j}) \mid X_{t,j}\right]\, \tilde{f}_{t,j}(X_{t,j})] = 0
\end{align}
Moreover, the directional derivative with respect to $\mu_{t,j}$, in the direction $\tilde{\mu}_{t,j}$ evaluated at the true $\bar{f}_M, \bar{\mu}_M$:
\begin{align}
\partial_{a_{t,j}}\operatorname{\mathrm{E}}\left[m_M^*(Z;\bar{f}_M, \bar{\mu}_M)\right][\tilde{a}_{t,j}] =~& \operatorname{\mathrm{E}}\left[\tilde{\mu}_{t,j}(V_{t,j}) \left(m_{t+1,j}(Z; f_{t+1}) - f_{t,j}(X_{t,j})\right)\right] \\
=~& \operatorname{\mathrm{E}}\left[\tilde{\mu}_{t,j}(V_{t,j})' \operatorname{\mathrm{E}}\left[m_{t+1,j}(Z; f_{t+1}) - f_{t,j}(X_{t,j})\mid V_{t,j}\right]\right] \\
=~& 0 \tag{recursive definition of $f_t$}
\end{align}
Hence, we conclude that the moment is Neyman orthogonal.
\end{proof}
Note that we can relax the requirement on $\mu_{t,j}$, instead of being the solution to the non-parametric IV problem to simply satisfy the set of un-conditional moment restrictions:
\begin{align}
\sup_{f\in {\mathcal F}_{t,j}-{\mathcal F}_{t,j}} \operatorname{\mathrm{E}}\left[\left(\mu_{t,j}(V_{t,j}) - a_{t,j}(X_{t,j})\right) f(X_{t,j})\right] =0
\end{align}
where ${\mathcal F}_{t,j}$ is the function class that we assume $f_{t,j}$ lies in. Thus when the function class ${\mathcal F}_{t,j}$ is simple enough, then estimating the function $\mu_{t,j}(V_{t,j})$ is a much simpler problem than non-parametric IV.
\section{Extension: Change of Measure}
We now consider yet another generalization of the last section, where we can even allow the measure over which we take expectations in the moments the define the nested nuisances, to change for each function. This allows us to apply our methodology to cases where there is a co-variate shift between the distribution that is used to train a regression or IV regression function and the distribution over which we evaluate the moment that we plug it in, so as to get a target effect estimate. We will see that a prototypical application of this extension is the case of estimating long-term effects with surrogates from a combination of historical and short-term data.
In particular, we have that $\theta$ is defined via the set of equations:
\begin{align}
\operatorname{\mathrm{E}}_{Z\sim D}[m_1(Z; \theta, f_1)] =~& 0 \label{eqn:non-lin-estimand-iv-change}\\
\forall 1\leq t < M, \forall j\in [d_t]: \operatorname{\mathrm{E}}_{Z\sim D_{t,j}}\left[f_{t,j}(X_{t,j}) - m_{t+1, j}(Z; f_{t+1})\mid V_{t,j}\right] =~& 0 \tag{recursive estimand}\\
\operatorname{\mathrm{E}}_{Z\sim D_{M,j}}\left[f_{M, j}(X_{M,j}) - Y_j\mid V_{M,j}\right] =~& 0 \tag{base estimand}
\end{align}
where $m_{t+1}$ is a sequence of potentially non-linear vector-valued functionals, $f_t$ is a $d_t$-dimensional vector-valued function and $Y_j, V_{t,j}, X_{t,j}$ are sub-vectors of the random vector $Z$.
\begin{theorem}[Main Theorem for Non-Linear Functionals]
For notational convenience define: $m_{M+1,j}(Z, f_{M+1}):= Y_j$ and $\mu_{0,j}(X_0):=1$ and let $\bar{f}_M = \{f_1, \ldots, f_M\}$, $\bar{\mu}_M=\{\mu_1, \ldots, \mu_M\}$, $f_t$ are recursively defined in Equation~\ref{eqn:non-lin-estimand-iv} and $a_t$ are recursively defined as follows: for all $t\geq 1$ and $j\in [d_t]$, $a_{t,j}:{\mathcal X}_t\to \mathbb{R}$ is the Riesz representer of the linear functional, with respect to $g$:
\begin{align}
L_{t,j}(g; f_{t}) := \sum_{k=1}^{d_{t-1}} \operatorname{\mathrm{E}}_{Z\sim D_{t-1, k}}\left[ \mu_{t-1,k}(V_{t-1,k}) \frac{\partial}{\partial \tau} m_{t,k}(Z; f_{t,j} + \tau g, f_{t,-j})\bigr\vert_{\tau=0}\right]
\end{align}
and with respect to the $\cL^2(D_{t,j})$ inner-product space.
Then $\mu_{t,j}$ is defined based on $a_{t,j}$ as the solution to the following conditional moment equation:
\begin{align}
\operatorname{\mathrm{E}}_{Z\sim D_{t,j}}\left[\mu_{t,j}(V_{t,j}) - a_{t,j}(X_{t,j}) \mid X_{t,j}\right] =0
\end{align}
Define the moment:
\begin{align}
M^*(\theta, \bar{f}_M, \bar{\mu}_M) := \operatorname{\mathrm{E}}_{Z\sim D}\left[m_1(Z; \theta, f_1)\right] + \sum_{t=1}^M \sum_{j=1}^{d_t} \operatorname{\mathrm{E}}_{Z\sim D_{t,j}}\left[\mu_{t,j}(V_{t,j})' \left(m_{t+1,j}(Z; f_{t+1}) - f_{t,j}(X_{t,j})\right) \right]
\end{align}
Then the estimand $\theta_0$ can be identified by the moment equation:
\begin{align}
M^*(\theta, \bar{f}_M, \bar{a}_M) = 0
\end{align}
Moreover, the moment $M^*$ is Neyman orthogonal with respect to all nuisance functions $\bar{f}_M, \bar{a}_M$.
\end{theorem}
Finally, we note that the above theorem requires estimating the Riesz representer of a linear functional $L(g)=\operatorname{\mathrm{E}}_{Z\sim D}[m(Z;g)]$, in the ${\cal L}^2(D')$ space for some other distribution $D'$. Under the assumption that $L(g)$ is continuous in this inner product space, then such a Riesz representer is guaranteed to exist. Moreover, the statistical learning approach easily extends to this case.
Note that under by the Riesz definition:
\begin{align}
L(g) = \operatorname{\mathrm{E}}_{Z\sim D'}\left[ a(Z) g(Z) \right]
\end{align}
Thus we can consider the risk:
\begin{align}
R(a) = \operatorname{\mathrm{E}}_{Z\sim D'}\left[a(Z)^2\right] - 2\operatorname{\mathrm{E}}_{Z\sim D}\left[m(Z;g)\right]
\end{align}
Note that:
\begin{align}
R(a) - R(a_0) =~& \operatorname{\mathrm{E}}_{Z\sim D'}\left[a(Z)^2\right] - 2\operatorname{\mathrm{E}}_{Z\sim D}\left[m(Z;a)\right] - \operatorname{\mathrm{E}}_{Z\sim D'}\left[a_0(Z)^2\right] + 2\operatorname{\mathrm{E}}_{Z\sim D}\left[m(Z;a_0)\right]\\
=~&\operatorname{\mathrm{E}}_{Z\sim D'}\left[a(Z)^2\right] - 2\operatorname{\mathrm{E}}_{Z\sim D}\left[m(Z;a)\right] - \operatorname{\mathrm{E}}_{Z\sim D'}\left[a_0(Z)^2\right] + 2\operatorname{\mathrm{E}}_{Z\sim D'}\left[a_0(Z)^2\right]\\
=~&\operatorname{\mathrm{E}}_{Z\sim D'}\left[a(Z)^2\right] - 2\operatorname{\mathrm{E}}_{Z\sim D}\left[m(Z;a)\right] + \operatorname{\mathrm{E}}_{Z\sim D'}\left[a_0(Z)^2\right]\\
=~&\operatorname{\mathrm{E}}_{Z\sim D'}\left[a(Z)^2\right] - 2\operatorname{\mathrm{E}}_{Z\sim D'}\left[a_0(Z)\, a(Z)\right] + \operatorname{\mathrm{E}}_{Z\sim D'}\left[a_0(Z)^2\right]\\
=~&\operatorname{\mathrm{E}}_{Z\sim D'}\left[(a(Z) - a_0(Z))^2\right]
\end{align}
Thus it is equivalent to minimizing the mean-squared-error of $a$ with respect to $a_0$, over the distribution $D'$.
Moreover, consider the case where the moment is of the form $\theta - \operatorname{\mathrm{E}}[m_0(Z;f_1)]$ for some linear functional $m$ of $f_1$ and we are in the nested regression case (i.e. $V_{t,j}=X_{t,j}$); thus $a_{t,j}=\mu_{t,j}$. Let $\theta^*$ be the solution to $M^*(\theta; \bar{f}_M^*, \bar{a}_M^*)$ and $\hat{\theta}$ be the solution to $M^*(\theta; \hat{\bar{f}}_M, \hat{\bar{a}}_M)$, and we define as $\tilde{a}=a^* - \hat{a}$, $\tilde{f}=f^* - \hat{f}$, then we can derive an extension to the mixed bias property that:
\begin{align}
\theta^* - \tilde{\theta}
=~& \sum_{t=1}^M \sum_{j=1}^{d_t} \operatorname{\mathrm{E}}_{Z\sim D_{t,j}}\left[\tilde{a}_{t,j}(X_{t,j}) \left(m_{t+1, j}(Z;\tilde{f}_{t+1}) - \tilde{f}_{t}(X_{t,j})\right)\right],\\
\leq~& \sum_{t=1}^M \sum_{j=1}^{d_t} \sqrt{\operatorname{\mathrm{E}}_{Z\sim D_{t,j}}[\tilde{a}_{t,j}(X_{t,j})^2]}\left(\sqrt{\operatorname{\mathrm{E}}_{Z\sim D_{t,j}}[m(Z;\tilde{f}_{t+1})^2]} + \sqrt{\operatorname{\mathrm{E}}_{Z\sim D_{t,j}}[\tilde{f}_{t}(X_{t,j})^2]} \right)
\end{align}
If we further assume that the moment satisfies the following mean-squared-continuity property:
\begin{align}
\operatorname{\mathrm{E}}_{Z\sim D_{t,j}}[m(Z;\tilde{f}_{t+1})^2] \leq L\sum_{j=1}^{d_{t+1}} \operatorname{\mathrm{E}}_{Z\sim D_{t+1,j}}[\tilde{f}_{t+1,j}(X_{t+1,j})^2]
\end{align}
Then it suffices to control the mean-squared-errors of each regression function $f_{t,j}$, with respect to the distribution of data $D_{t,j}$ on which it is trained. Similarly, it suffices to control the mean-squared-errors of the Riesz representers $a_{t,j}$ with respect to the distribution $D_{t,j}$, which is exactly what the modified risk $R(a)$ is equivalent to.
\subsection{Example: Estimation of Long-Term Effects with Surrogates}
Consider the case when we have a short term data set that contains observations of $X, T, S$, where $X$ are controls, $T$ is a binary treatment and $S$ is a short-term surrogate of a long-term outcome. Moreover, we have a long-term data set that contains $X, S, Y$, where $Y$ is a long-term outcome.
We assume that even in the short-term data setting there is a latent long-term $Y$, coupled with each observation, which is not observed. Our goal is to estimate the effect of $T$ on $Y$.
Assume that $Y \perp T \mid S, X$ (surrogacy) and that for each $t\in \{0,1\}$ $Y^{(t)} \perp \!\!\! \perp T \mid X$ (conditional exogeneity). Let $D_s$ denote the distribution of the short-term dataset and $D_\ell$ the distribution of the long term data set. We will use the short-hand $\operatorname{\mathrm{E}}_{s},\operatorname{\mathrm{E}}_{\ell}$ for $\operatorname{\mathrm{E}}_{Z\sim D_s},\operatorname{\mathrm{E}}_{Z\sim D_{\ell}}$. Assume that $\operatorname{\mathrm{E}}_s[Y\mid S,X]=\operatorname{\mathrm{E}}_{\ell}[Y\mid S,X]$ (invariance). We can then write:
\begin{align}
\theta =~& \operatorname{\mathrm{E}}_s[Y^{(1)} - Y^{(0)}] = \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[Y^{(1)} - Y^{(0)}\mid X]]\\
=~& \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[Y^{(1)} \mid T=1, X] - \operatorname{\mathrm{E}}_s[Y^{(0)}\mid T=0, X]] \tag{conditional exogeneity}\\
=~& \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[Y \mid T=1, X] - \operatorname{\mathrm{E}}_s[Y\mid T=0, X]]\\
=~& \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[Y\mid S, T=1, X] \mid T=1, X] - \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[Y\mid S, T=1, X]\mid T=0, X]]\\
=~& \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[Y\mid S, X] \mid T=1, X] - \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[Y\mid S, X]\mid T=0, X]] \tag{surrogacy}\\
=~& \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_\ell[Y\mid S, X] \mid T=1, X] - \operatorname{\mathrm{E}}_s[\operatorname{\mathrm{E}}_\ell[Y\mid S, X]\mid T=0, X]] \tag{invariance}
\end{align}
Define as:
\begin{align}
h(S, X) =~& \operatorname{\mathrm{E}}_{\ell}[Y\mid S, X] &
g(T, X) =~& \operatorname{\mathrm{E}}_{s}[h(S, X) \mid T, X]
\end{align}
Then we can write:
\begin{align}
\theta_0 = \operatorname{\mathrm{E}}_{s}[g(1, X) - g(0,X)]
\end{align}
We see that this estimand falls into the extended framework presented in this section with $f_1=g$, $V_{1}=X_1=(T,X)$ and $f_2=h$, $V_2=X_2=(S,X)$ and linear moments $m_1(Z; \theta, f_1) = f_1(1, X) - f_1(0,X) - \theta$ and $m_2(Z;f_2)=f_2(S,X)$.
Thus we can apply our automated debiasing framework to arrive at a moment equation of the form:
\begin{align}
\operatorname{\mathrm{E}}_{s}\left[g(1,X) - g(0,X) + a_1(T,X)\,(h(S,X) - g(T,X))\right] + \operatorname{\mathrm{E}}_{\ell}\left[a_2(S,X) (Y - h(S,X))\right]
\end{align}
Note that this can be taken to data since the long-term outcome $Y$ appears only in the expectation in the historical data, where it is observed, and also the regression $h$ is viable in the historical data. Moreover, the treatment $T$ appears only in the expectation in the short-term data, where it is observed. In the work of \cite{athey2020estimating} the quantity $a_2(S,X)$, that comes out of our automatic de-biasing framework, is referred to as the surrogate score and the term $\operatorname{\mathrm{E}}_{\ell}\left[a_2(S,X)Y\right]$ is referred to as the surrogate score representation of the treatment effect.
Moreover, note that this setting falls under the linear moment functional setting that we expanded at the end of the last section, for which the estimand will also have the multiply robust mixed bias property. In this setting the formula simplifies to:
\begin{align}
\theta^* - \tilde{\theta} = \operatorname{\mathrm{E}}_s[\tilde{a}_1(T,X) (\tilde{h}(S,X) - \tilde{g}(S,X))] - \operatorname{\mathrm{E}}_\ell\left[\tilde{a}_2(S,X)\, \tilde{h}(S,X)\right]
\end{align}
Thus if $a_1$ or $a_2$ are correct, then the bias is $0$, or if $h,g$ are correct then the bias is $0$. Moreover, if we let $\|\cdot\|_s$ denote the RMSE with respect to $D_s$ and similarly $\|\cdot\|_\ell$, then it suffices to control the mean-squared-errors quantities:
\begin{align}
\|a_1-a_{1,0}\|_s (\|h-h_0\|_s + \|g-g_0\|_s) + \|a_2 - a_{2,0}\|_\ell \, \|h-h_0\|_\ell
\end{align}
The mean-squared-continuity property referred to in the last section, here boils down to assuming that $\|h-h_0\|_s \leq L \|h-h_0\|_\ell$, which is satisfied for instance if the density ratio of $S,X$ in the two settings is bounded. In that case, it suffices to bound the RMSE of $h$ over the data on which it is trained, which are drawn from $D_\ell$. In practice, it would be a beneficial to train $h$ in a manner that controls the RMSE under both $D_s$ and $D_\ell$ using co-variate shift techniques, since we have samples of co-variates $S,X$ from both domains.
\subsection{Application: Front-Door Criterion}
Consider identification under the front door criterion in settings where the data generating process is governed by the causal graph in Figure~\ref{fig:front}.
\begin{figure}
\centering
\begin{tikzpicture}[node distance=1.5cm]
\node[draw, circle] (D) {$D$};
\node[draw, circle, right of=D] (M) {$M$};
\node[draw, circle, right of=M] (Y) {$Y$};
\node[draw, circle, above of=M, yshift=0.5cm] (U) {$U$};
\draw[->] (U) -- (D);
\draw[->] (U) -- (Y);
\draw[->] (D) -- (M);
\draw[->] (M) -- (Y);
\end{tikzpicture}
\caption{Causal graph for front-door identification.}\label{fig:front}
\end{figure}
In this setting, we have by the front door identification approach that we can estimate the mean counterfactual response when we intervene on $D$, and set it to $d$, as:
\begin{align}
\operatorname{\mathrm{E}}[Y^{(d)}] =~& \operatorname{\mathrm{E}}[Y^{(M^{(d)})}] = \int \operatorname{\mathrm{E}}[Y^{(m)}\mid M^{(d)}=m]\, p(M^{(d)}=m) dm \\
=~& \int \operatorname{\mathrm{E}}[Y^{(m)}]\, p(M^{(d)}=m\mid D=d) dm\\
=~& \int \operatorname{\mathrm{E}}[\operatorname{\mathrm{E}}[Y^{(m)}\mid D]]\, p(M^{(d)}=m\mid D=d) dm\\
=~& \int \operatorname{\mathrm{E}}[\operatorname{\mathrm{E}}[Y^{(m)}\mid M=m, D]]\, p(M^{(d)}=m\mid D=d) dm\\
=~& \int \operatorname{\mathrm{E}}[\operatorname{\mathrm{E}}[Y\mid M=m, D]]\, p(M=m\mid D=d) dm\\
=~& \int \int \operatorname{\mathrm{E}}[Y\mid M=m, D=d'] p(D=d')\, p(M=m\mid D=d) dm
\end{align}
Let $D'$ denote a random variable that is drawn from the marginal distribution of $D$ but is independent of $M$ and let $M$ denote a random variable that is drawn from the conditional distribution of $M$ given $D$. Moreover, let $Y$ be the random variable drawn from the conditional distribution of $Y$ given $M, D', X$. Then we can write:
\begin{align}
\theta = \operatorname{\mathrm{E}}[Y^{(d)}] = \operatorname{\mathrm{E}}[\operatorname{\mathrm{E}}[\operatorname{\mathrm{E}}[Y\mid M, D']\mid D=d]]
\end{align}
which is a nested regression estimand. In particular, we can write:
\begin{align}
\theta =~& f(d)\\
f(D) =~& \operatorname{\mathrm{E}}[h(D', M)\mid D]\\
h(D, M) =~& \operatorname{\mathrm{E}}[Y\mid D, M]
\end{align}
Hence we can construct an automated debiased moment via nested Riesz estimation:
\begin{align}
\theta = f(d) + \operatorname{\mathrm{E}}_{(X, D', D)}[a_f(D) (h(D',M) - f(D)) ] + \operatorname{\mathrm{E}}_{(D, M, Y)}[a_h(D, M) (Y - h(D,M))]
\end{align}
Note that here we have a change of measure setting, since the regression $h$ is trained on data where $D,M$ are correlated conditional on $X$, but then it is applied on data were $D,M$ are independent conditional on $X$.
The dataset for training $f$, can be produced by using a correlated sample pair $(D,M)$ and then couple it with an independent sample of the treatment $D$ from the marginal distribution of treatments. For instance, we can achieve that by pairing two empirical samples $i, j$ and then using the vector $(D_i, D_j, M_j)$ as a sample of $(D', D, M)$ and running a regression of $h(D_i, M_j)$ on $D_j, X$. Similarly, the sample for training $h$ is produced by taking a sample from the data generating process $(D_i, M_i, Y_i)$ and regressing $Y_i$ on $D_i, M_i$.
In this setting, one could also mathematically characterize the two Riesz representers, which will be of the form:
\begin{align}
a_f(D) =~& \frac{1\{D=d\}}{\ensuremath{\mathrm{Pr}}(D=d)}\\
a_h(D, M) =~& \operatorname{\mathrm{E}}\left[\frac{p(M\mid D')}{p(M\mid D)}\mid D, M\right]
\end{align}
The implicit Riesz estimation avoids the explicit estimation of the density ratio.
\begin{figure}
\centering
\begin{tikzpicture}[node distance=1.5cm]
\node[draw, circle] (D) {$D$};
\node[draw, circle, right of=D] (M) {$M$};
\node[draw, circle, right of=M] (Y) {$Y$};
\node[draw, circle, above of=M] (U) {$U$};
\node[draw, circle, below of=M] (X) {$X$};
\draw[->] (U) -- (D);
\draw[->] (U) -- (Y);
\draw[->] (D) -- (M);
\draw[->] (M) -- (Y);
\draw[->] (X) -- (D);
\draw[->] (X) -- (M);
\draw[->] (X) -- (Y);
\end{tikzpicture}
\caption{Causal graph for front-door identification with observables.}\label{fig:front}
\end{figure}
If the causal graph contains observable characteristics $X$ that create correlations among the variables $D, M, Y$, then a generalization of the above identification strategy yields that:
\begin{align}
\theta =~& \operatorname{\mathrm{E}}[f(d, X)]\\
f(D, X) =~& \operatorname{\mathrm{E}}[h(D', M, X)\mid D, X]\\
h(D, M, X) =~& \operatorname{\mathrm{E}}[Y\mid D, M, X]
\end{align}
where $D'$ is a random variable the is drawn independently of $M$ conditional on $X$ from the conditional distribution of $D\mid X$. For binary treatment, we can write this as:
\begin{align}
\theta =~& \operatorname{\mathrm{E}}[f(d, X)]\\
f(D, X) =~& \operatorname{\mathrm{E}}[h(1, M, X) p(X) + h(0, M, X) (1 - p(X))\mid D, X]\\
p(X) =~& \operatorname{\mathrm{E}}[D\mid X]\\
h(D, M, X) =~& \operatorname{\mathrm{E}}[Y\mid D, M, X]
\end{align}
This falls again under the class of nested regression functionals. Albeit in this formulation the moment that defines $f$, is a non-linear moment with respect to the regressions $(h,p)$. Thus we would need to apply the non-linear automatic debiasing approach.
\section{Inference}
So far, we have characterized Neyman orthogonal moment functions for a broad class of recursive functionals. We have also defined sequential estimators for the recursive nuisance functions. In this section, we combine these results to prove consistency, asymptotic normality, and semiparametric efficiency for our estimator of the causal parameter.
\subsection{Estimator and confidence interval}
The final aspect of our inferential procedure is cross fitting, which is a classic idea in semiparametric statistics \cite{bickel1982adaptive,schick1986asymptotically,klaassen1987consistent}. Our overall procedure is a variant of debiased machine learning \cite{chernozhukov2018original}.
For the inference proof, it improves clarity to adorn "true" values with the superscript $*$, e.g. we write $\theta^*=\theta(\tau)$.
Consider the abstract notation
$$
\operatorname{\mathrm{E}}[\psi(Z;\theta^*,\eta^*)]=0
$$
where $Z$ concatenates random variables in the model, $\theta^*$ is the true causal parameter value, and $\eta^*$ is the true nuisance value.
\begin{algorithm}[Debiased machine learning]\label{alg:dml}
Given a sample $(Z_i)$ $(i=1,...,n)$, partition the sample into folds $(I_{(q)})$ $(q=1,...,Q)$. Denote by $I_{(q)}^c$ the complement of $I_{(q)}$.
\begin{enumerate}
\item For each fold $q$, estimate $\hat{\eta}_{(q)}$ from observations in $I_{(q)}^c$.
\item Estimate $ \theta^*$ as the solution to
$
n^{-1}\sum_{q=1}^Q\sum_{i\in I_{(q)}} \psi(W_i;\hat{\theta},\hat{\eta}_{(q)})=0
$.
\item Estimate its $95$\% confidence interval as
$
CI=\hat{\theta}\pm 1.96\hat{\sigma} n^{-1/2}$, where $1.96$ is the $1-0.95/2$ quantile of the standard Gaussian and $\hat{\sigma}^2
$ is defined below after introducing additional notation.
\end{enumerate}
\end{algorithm}
We will show
$$
\frac{\sqrt{n}}{\sigma}(\hat{\theta}-\theta^*)\overset{d}{\rightarrow}\mathcal{N}(0,1),\quad \hat{\sigma}^2\overset{p}{\rightarrow} \sigma^2 \implies \mathbb{P}(\theta^* \in CI)\rightarrow 0.95.
$$
\subsection{Generic asymptotic result}
Recall the recursive definition of $f^*_t$:
\begin{align}
\theta^*=~& \operatorname{\mathrm{E}}[m_1(Z; f^*_1)]\\
\forall 1\leq t < M: f^*_{t}(S_{t}, T_t) =~& \operatorname{\mathrm{E}}\left[m_{t+1}(Z; f^*_{t+1})\mid S_{t}, T_t\right] \tag{recursive estimand}\\
f^*_{M}(S_{M}, T_M) =~& \operatorname{\mathrm{E}}\left[Y\mid S_{M}, T_M\right] \tag{base estimand}
\end{align}
Recall the recursive definition of $a^*_t$: for all $t\geq 1$, $a^*_t:\ensuremath{{\cal S}}_t\times {\mathcal T}_t\to \mathbb{R}$ is the Riesz representer of the linear functional:
\begin{align}
L_t(g) := \operatorname{\mathrm{E}}\left[a^*_{t-1}(S_{t-1}, T_{t-1}) m_t(Z;g)\right]
\end{align}
\begin{theorem}[Inference for nested linear functionals]\label{thm:inference}
Suppose identification holds, i.e. $\theta^*=\operatorname{\mathrm{E}}[m_1(Z; f^*_1)]$. Consider the recursive definitions of $f_t^*$ and $a_t^*$ given above. Assume that
\begin{enumerate}
\item $\| m_{t}(Z;f_{t})\|_2 \leq \kappa \|f_{t}\|_2$
\item $|\theta^*|<C$ and $\operatorname{\mathrm{E}}[\left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)^2|S_t, T_t] \leq \bar{\sigma}^2$
\item $\|m_{t}(Z; \hat{f}_{t})\|_p\leq C$, $\|\hat{f}_t(S_t, T_t)\|_p \leq C$, and $\|\hat{a}_t(S_t, T_t)\|_{\infty}\leq C$ for some $p>2$
\item $\|\tilde{f}_t(S_t, T_t)\|_2=o_p(1)$ and $\|\tilde{a}_t(S_t, T_t)\|_2=o_p(1)$
\item $\sqrt{n}\|\tilde{a}_t(S_t, T_t)\|_2 \|\tilde{f}_t(S_t, T_t)\|_2 =o_p(1)$ and $\sqrt{n} \|\tilde{a}_t(S_t, T_t)\|_2 \|\tilde{f}_{t+1}(S_{t+1},T_{t+1})\|_2=o_p(1)$
\item $\sigma^2>c$ where
$$
\sigma^2=\operatorname{\mathrm{E}}\left[\{m_M(Z;\bar{f}^*_M, \bar{a}^*_M)-\theta^*\}^2\right].
$$
\end{enumerate}
Then for the class of nested linear functionals,
$$
\hat{\theta}\overset{p}{\rightarrow}\theta^*,\quad \frac{\sqrt{n}}{\sigma}(\hat{\theta}-\theta_0)\overset{d}{\rightarrow}\mathcal{N}(0,1),\quad \mathbb{P}(\theta^*\in CI)\rightarrow 0.95.
$$
\end{theorem}
\subsection{Proofs}
\subsubsection{Abstract conditions from previous work}
To begin, we quote an abstract theorem whose conditions we will verify. Suppose $p>2$ is some constant, as are $0<c_0\leq c_1$.
\begin{assumption}\label{assumption:DML1}
Assume that for all $n\geq 3$ and $\mathbb{P}$
\begin{enumerate}
\item The moment function is valid
\item The moment function is affine in $\theta$, i.e.
$$
\psi(z;\theta,\eta)=\psi^a(z;\eta)\theta+\psi^b(z;\eta)
$$
\item The moment function, viewed as a functional of the nuisances, is twice Gateaux differentiable with respect to the nuisances, i.e.
$$
\eta\mapsto \operatorname{\mathrm{E}} [\psi(Z;\theta,\eta)]
$$
\item The moment function is Neyman orthogonal
\item In the affine representation, the mean of the component that multiplies the causal parameter, i.e.
$$
J^*=\operatorname{\mathrm{E}}[\psi^a(Z;\eta^*)]
$$
has singular values that are between $c_0$ and $c_1$.
\end{enumerate}
\end{assumption}
\begin{assumption}\label{assumption:DML2}
Assume that for all $n\geq 3$ and $\mathbb{P}$
\begin{enumerate}
\item Within each fold, w.p. $1-\Delta_n$, $\hat{\eta}_{(q)} \in \mathcal{T}_n$ where $\eta^*\in \mathcal{T}_n$ and $\mathcal{T}_n$ satisfies the following conditions
\item Bounded moments:
\begin{align}
&\sup_{\eta\in \mathcal{T}_n}\|\psi(Z;\theta^*,\eta)\|_p \leq c_1 \\
&\sup_{\eta\in \mathcal{T}_n} \|\psi^a(Z;\eta)\|_p \leq c_1
\end{align}
\item The following rate conditions hold
\begin{align}
r_n&=\sup_{\eta\in \mathcal{T}_n}|\operatorname{\mathrm{E}}[\psi^a(Z;\eta)]-\operatorname{\mathrm{E}}[\psi^a(Z;\eta^*)]|\leq \delta_n\\
r_n'&=\sup_{\eta\in \mathcal{T}_n} \|\psi(Z;\theta^*,\eta)-\psi(Z;\theta^*,\eta^*)\|_2\leq \delta_n \\
\lambda_n'&=\sqrt{n}\sup_{r\in(0,1),\eta\in\mathcal{T}_n} |\partial_r^2 \operatorname{\mathrm{E}}[\psi(Z;\theta^*,\eta^*+r(\eta-\eta^*))]|\leq \delta_n
\end{align}
\item The variance of the score is non-degenerate: the eigenvalues of
$$
\operatorname{\mathrm{E}}[\psi(Z;\theta^*,\eta^*)\psi(Z;\theta^*,\eta^*)']
$$
are bounded below by $c_0$
\end{enumerate}
\end{assumption}
\begin{theorem}[Gaussian approximation \cite{chernozhukov2018original}]\label{thm:gaussian}
Suppose Assumptions~\ref{assumption:DML1} and~\ref{assumption:DML2} hold, and $n^{-1/2}\leq \delta_n$. Then
$$
\frac{\sqrt{n}}{\sigma}(\hat{\theta}-\theta^*)=\frac{1}{\sqrt{n}}\sum_{i=1}^n -\frac{(J^*)^{-1}}{\sigma} \psi(Z_i;\theta^*,\eta^*)+O_p(n^{-1/2}+r_n+r_n'+\lambda_n')
$$
where
$$
\sigma^2=(J^*)^{-1} \operatorname{\mathrm{E}}[\psi(Z;\theta^*,\eta^*)\psi(Z;\theta^*,\eta^*)']((J^*)^{-1} )'
$$
\end{theorem}
\begin{theorem}[Variance estimation \cite{chernozhukov2018original} ]\label{thm:variance}
Suppose Assumptions~\ref{assumption:DML1} and~\ref{assumption:DML2} hold, and $n^{-1/2 \wedge (1-2/p)}\leq \delta_n$. Consider the variance estimator
$$
\hat{J}^{-1} n^{-1}\sum_{q=1}^Q\sum_{i\in I_{(q)}} \psi(W_i;\hat{\theta},\hat{\eta}_{(q)})\psi(W_i;\hat{\theta},\hat{\eta}_{(q)})' (\hat{J}^{-1})',\quad \hat{J}=\frac{1}{n}\sum_{q=1}^Q\sum_{i\in I_{(q)}} \psi^a(Z_i;\hat{\eta}_{(q)})
$$
\end{theorem}
\begin{corollary}[Confidence intervals \cite{chernozhukov2018original}]\label{cor:ci}
Suppose the conditions of Theorem~\ref{thm:variance} hold. Then the confidence interval
$$
CI=\hat{\theta}\pm 1.96\frac{\hat{\sigma}}{\sqrt{n}}
$$
is uniformly valid, i.e.
$$
\sup_{\mathbb{P}\in \mathcal{P}_n}|\mathbb{P}(\theta^*\in CI)-1.96|\rightarrow 0.
$$
\end{corollary}
\subsubsection{Matching symbols}
So what remains is a verification of the conditions in Assumptions~\ref{assumption:DML1} and~\ref{assumption:DML2} for the various causal parameters of interest. We verify these conditions for
\begin{enumerate}
\item dynamic treatment effect
\item nested linear functionals
\end{enumerate}
In particular, the former are a special case of the latter, so we focus on the latter.
Recall the recursive definition of $f^*_t$:
\begin{align}
\theta^*=~& \operatorname{\mathrm{E}}[m_1(Z; f^*_1)]\\
\forall 1\leq t < M: f^*_{t}(S_{t}, T_t) =~& \operatorname{\mathrm{E}}\left[m_{t+1}(Z; f^*_{t+1})\mid S_{t}, T_t\right] \tag{recursive estimand}\\
f^*_{M}(S_{M}, T_M) =~& \operatorname{\mathrm{E}}\left[Y\mid S_{M}, T_M\right] \tag{base estimand}
\end{align}
Recall the recursive definition of $a^*_t$: for all $t\geq 1$, $a^*_t:\ensuremath{{\cal S}}_t\times {\mathcal T}_t\to \mathbb{R}$ is the Riesz representer of the linear functional:
\begin{align}
L_t(g) := \operatorname{\mathrm{E}}\left[a^*_{t-1}(S_{t-1}, T_{t-1}) m_t(Z;g)\right]
\end{align}
Recall the orthogonal moment function:
\begin{align}
\theta= \operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}^*_M, \bar{a}^*_M)\right] := \operatorname{\mathrm{E}}\left[m_1(Z; f^*_1) + \sum_{t=1}^M a^*_t(S_t, T_t) \left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)\right]
\end{align}
Clearly, the moment function is
\begin{equation}\label{eq:psi}
\psi(Z;\theta,\eta)=\theta-\left\{m_1(Z; f_1) + \sum_{t=1}^M a_t(S_t, T_t) \left(m_{t+1}(Z; f_{t+1}) - f_t(S_t, T_t)\right)\right\}
\end{equation}
where
\begin{equation}\label{eq:match}
\eta=(\bar{f}_M, \bar{a}_M),\quad \psi^a(Z;\eta)=1,\quad \psi^b(Z;\eta)=-\left\{m_1(Z; f_1) + \sum_{t=1}^M a_t(S_t, T_t) \left(m_{t+1}(Z; f_{t+1}) - f_t(S_t, T_t)\right)\right\}
\end{equation}
\subsubsection{New results}
\begin{lemma}[Verification of Assumption~\ref{assumption:DML1}]\label{lemma:DML1}
Suppose identification holds, i.e. $\theta^*=\operatorname{\mathrm{E}}[m_1(Z; f^*_1)]$. Consider the recursive definitions of $f_t^*$ and $a_t^*$ given above. Then the conditions of Assumption~\ref{assumption:DML1} hold.
\end{lemma}
\begin{proof}
We verify each condition.
\begin{enumerate}
\item The moment function is valid.
In this proposition, we take identification as given. Therefore the first and second term of~\ref{eq:psi} cancel in expectation when $\eta=\eta^*$ and $\theta=\theta^*$. The final terms are mean zero by the law of iterated expectations when $\eta=\eta^*$ due to the recursive definition of $f_t^*$.
\item The moment function is affine in $\theta$, i.e.
$$
\psi(z;\theta,\eta)=\psi^a(z;\eta)\theta+\psi^b(z;\eta)
$$
We verify this property in~\eqref{eq:match}
\item The moment function, viewed as a functional of the nuisances, is twice Gateaux differentiable with respect to the nuisances, i.e.
$$
\eta\mapsto \operatorname{\mathrm{E}}[\psi(Z;\theta,\eta)]
$$
The proof is identical to the proof of Theorem~\ref{thm:main}, replacing $f_{t+1}(S_{t+1},\tau_{t+1})$ with $m_{t+1}(Z; f_{t+1})$.
\item The moment function is Neyman orthogonal
The proof is identical to the proof of Theorem~\ref{thm:main}, replacing $f_{t+1}(S_{t+1},\tau_{t+1})$ with $m_{t+1}(Z; f_{t+1})$.
\item In the affine representation, the mean of the component that multiplies the causal parameter, i.e.
$$
J^*=\operatorname{\mathrm{E}}[\psi^a(Z;\eta^*)]
$$
has singular values that are between $c_0$ and $c_1$.
By~\eqref{eq:match}, $J^*=1$, which is bounded above and below.
\end{enumerate}
\end{proof}
\begin{lemma}[Verification of Assumption~\ref{assumption:DML2}]\label{lemma:DML2}
Assume that
\begin{enumerate}
\item $\hat{\eta},\eta^* \in \mathcal{T}_n$
\item $\| m_{t}(Z;f_{t})\|_2 \leq \kappa \|f_{t}\|_2$
\item $|\theta^*|<C$ and $\operatorname{\mathrm{E}}[\left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)^2|S_t, T_t] \leq \bar{\sigma}^2$
\item $\|m_{t}(Z; \hat{f}_{t})\|_p\leq C$, $\|\hat{f}_t(S_t, T_t)\|_p \leq C$, and $\|\hat{a}_t(S_t, T_t)\|_{\infty}\leq C$ for some $p>2$
\item $\|\tilde{f}_t(S_t, T_t)\|_2=o_p(1)$ and $\|\tilde{a}_t(S_t, T_t)\|_2=o_p(1)$
\item $\sqrt{n}\|\tilde{a}_t(S_t, T_t)\|_2 \|\tilde{f}_t(S_t, T_t)\|_2 =o_p(1)$ and $\sqrt{n} \|\tilde{a}_t(S_t, T_t)\|_2 \|\tilde{f}_{t+1}(S_{t+1},T_{t+1})\|_2=o_p(1)$
\item $
\sigma^2=\operatorname{\mathrm{E}}[\psi(Z;\theta^*,\eta^*)^2]>c_0
$
\end{enumerate}
Then the conditions of Assumption~\ref{assumption:DML2} hold.
\end{lemma}
\begin{proof}
We verify each condition.
\begin{enumerate}
\item Within each fold, w.p. $1-\Delta_n$, $\hat{\eta}_{(q)} \in \mathcal{T}_n$ where $\eta^*\in \mathcal{T}_n$ and $\mathcal{T}_n$ satisfies the following conditions.
Correct specification of $\eta^*$ is a sufficient condition.
\item Bounded moments:
\begin{align}
&\sup_{\eta\in \mathcal{T}_n}\|\psi(Z;\theta^*,\eta)\|_p \leq c_1 \\
&\sup_{\eta\in \mathcal{T}_n} \|\psi^a(Z;\eta)\|_p \leq c_1
\end{align}
\begin{enumerate}
\item For the former, write
\begin{align}
\|\psi(Z;\theta^*,\eta)\|_p
&=\left\|\theta^*-\left\{m_1(Z; f_1) + \sum_{t=1}^M a_t(S_t, T_t) \left(m_{t+1}(Z; f_{t+1}) - f_t(S_t, T_t)\right)\right\}\right\|_p \\
&\leq |\theta^*|+\|m_1(Z; f_1)\|_p+ \sum_{t=1}^M \|a_t(S_t, T_t)\|_{\infty} \|m_{t+1}(Z; f_{t+1}) - f_t(S_t, T_t)\|_p.
\end{align}
Finally, note that
\begin{align}
\|m_{t+1}(Z; f_{t+1}) - f_t(S_t, T_t)\|_p
&\leq \|m_{t+1}(Z; f_{t+1})\|_p+\|f_t(S_t, T_t)\|_p.
\end{align}
\item The latter is trivial since $\psi^a(Z;\eta)=1$.
\end{enumerate}
\item The following rate conditions hold
\begin{align}
r_n&=\sup_{\eta\in \mathcal{T}_n}|\operatorname{\mathrm{E}}[\psi^a(Z;\eta)]-\operatorname{\mathrm{E}}[\psi^a(Z;\eta^*)]|\leq \delta_n\\
r_n'&=\sup_{\eta\in \mathcal{T}_n} \|\psi(Z;\theta^*,\eta)-\psi(Z;\theta^*,\eta^*)\|_2\leq \delta_n \\
\lambda_n'&=\sqrt{n}\sup_{r\in(0,1),\eta\in\mathcal{T}_n} |\partial_r^2 \operatorname{\mathrm{E}}[\psi(Z;\theta^*,\eta^*+r(\eta-\eta^*))]|\leq \delta_n
\end{align}
\begin{enumerate}
\item The first inequality is trivial since $\psi^a(Z;\eta)=1$.
\item For the second inequality, write
\begin{align}
&\psi(Z;\theta^*,\eta)-\psi(Z;\theta^*,\eta^*)\\
&=\theta^*-\left\{m_1(Z; f_1) + \sum_{t=1}^M a_t(S_t, T_t) \left(m_{t+1}(Z; f_{t+1}) - f_t(S_t, T_t)\right)\right\} \\
&-\theta^*+\left\{m_1(Z; f^*_1) + \sum_{t=1}^M a^*_t(S_t, T_t) \left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)\right\} \\
&=-\left\{
m_M(Z;\bar{f}_M, \bar{a}_M)- m_M(Z;\bar{f}^*_M, \bar{a}_M)
+m_M(Z;\bar{f}^*_M, \bar{a}_M)-m_M(Z;\bar{f}^*_M, \bar{a}^*_M)\right\}
\end{align}
Grouping the initial two terms
\begin{align}
m_M(Z;\bar{f}_M, \bar{a}_M)- m_M(Z;\bar{f}^*_M, \bar{a}_M)
&=m_1(Z; \tilde{f}_1)+\sum_{t=1}^M a_t(S_t, T_t) \left(m_{t+1}(Z; \tilde{f}_{t+1}) - \tilde{f}_t(S_t, T_t)\right)
\end{align}
Grouping the final two terms
\begin{align}
m_M(Z;\bar{f}^*_M, \bar{a}_M)-m_M(Z;\bar{f}^*_M, \bar{a}^*_M)
&=\sum_{t=1}^M \tilde{a}_t(S_t, T_t) \left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)
\end{align}
Hence
\begin{align}
&\|\psi(Z;\theta^*,\eta)-\psi(Z;\theta^*,\eta^*)\|_2 \\
&\leq \|m_1(Z; \tilde{f}_1)\|_2 \\
&\quad + \sum_{t=1}^M \left\|a_t(S_t, T_t) \left(m_{t+1}(Z; \tilde{f}_{t+1}) - \tilde{f}_t(S_t, T_t)\right)\right\|_2 \\
&\quad +\sum_{t=1}^M \left\|\tilde{a}_t(S_t, T_t) \left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)\right\|_2
\end{align}
We focus on each term separately.
\begin{enumerate}
\item In the first term
\begin{align}
\|m_1(Z; \tilde{f}_1)\|_2 \leq \kappa \|\tilde{f}_1(S_1, T_1)\|_2
\end{align}
\item In the second term
\begin{align}
&\left\|a_t(S_t, T_t) \left(m_{t+1}(Z; \tilde{f}_{t+1}) - \tilde{f}_t(S_t, T_t)\right)\right\|_2 \\
&\leq \|a_t(S_t, T_t)\|_{\infty} \|m_{t+1}(Z; \tilde{f}_{t+1}) - \tilde{f}_t(S_t, T_t)\|_2 \\
&\leq \|a_t(S_t, T_t)\|_{\infty} \left(\|m_{t+1}(Z; \tilde{f}_{t+1})\|_2+\|\tilde{f}_t(S_t, T_t)\|_2\right) \\
&\leq \|a_t(S_t, T_t)\|_{\infty} \left(\kappa \|\tilde{f}_{t+1}(S_{t+1}, T_{t+1})\|_2+\|\tilde{f}_t(S_t, T_t)\|_2\right)
\end{align}
\item In the third term,
\begin{align}
&\left\|\tilde{a}_t(S_t, T_t) \left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)\right\|^2_2 \\
&=\operatorname{\mathrm{E}}[\tilde{a}_t(S_t, T_t)^2 \left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)^2] \\
&=\operatorname{\mathrm{E}}[\tilde{a}_t(S_t, T_t)^2 \operatorname{\mathrm{E}}[\left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)^2|S_t, T_t]] \\
&\leq \bar{\sigma}^2\operatorname{\mathrm{E}}[\tilde{a}_t(S_t, T_t)^2] \\
&=\bar{\sigma}^2 \|\tilde{a}_t(S_t, T_t)\|_2^2
\end{align}
Hence
$$
\left\|\tilde{a}_t(S_t, T_t) \left(m_{t+1}(Z; f^*_{t+1}) - f^*_t(S_t, T_t)\right)\right\|_2\leq \bar{\sigma} \|\tilde{a}_t(S_t, T_t)\|_2
$$
\end{enumerate}
\item For the third inequality, recall the second order derivatives from Theorem~\ref{thm:main}, replacing $f_{t+1}(S_{t+1},\tau_{t+1})$ with $m_{t+1}(Z; f_{t+1})$. The second order directional derivative is zero for any pair $(a_t, f_{t'})$ such that $t'\notin \{t, t+1\}$ and also it is zero for any pair $(a_t, a_t)$ and $(f_t, f_t)$. Moreover, for any pair $(a_t,f_t)$ the second order directional derivative is of the form:
\begin{align}
\partial_{a_t,f_t}\operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}_M, \bar{a}_M)\right][\tilde{a}_t, \tilde{f}_t] = -\operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) \tilde{f}_t(S_t, T_t)\right]
\end{align}
and for any pair $(a_t, f_{t+1})$ it is of the form:
\begin{align}
\partial_{a_t,f_{t+1}}\operatorname{\mathrm{E}}\left[m_M(Z;\bar{f}^*_M, \bar{a}^*_M)\right][\tilde{a}_t, \tilde{f}_{t+1}] = \operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) m_{t+1}(Z;\tilde{f}_{t+1})\right].
\end{align}
Therefore it is sufficient to bound, using Cauchy-Schwartz
\begin{align}
\sqrt{n}\left|\operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) \tilde{f}_t(S_t, T_t)\right]\right|&\leq \sqrt{n}\|\tilde{a}_t(S_t, T_t)\|_2 \|\tilde{f}_t(S_t, T_t)\|_2 \\
\sqrt{n}\left|\operatorname{\mathrm{E}}\left[\tilde{a}_t(S_t, T_t) m_{t+1}(Z;\tilde{f}_{t+1})\right]\right|
&\leq \sqrt{n}\|\tilde{a}_t(S_t, T_t)\|_2 \| m_{t+1}(Z;\tilde{f}_{t+1})\|_2 \\
&\leq \sqrt{n}\kappa \|\tilde{a}_t(S_t, T_t)\|_2 \|\tilde{f}_{t+1}(S_{t+1},T_{t+1})\|_2.
\end{align}
\end{enumerate}
\item The variance of the score is non-degenerate: the eigenvalues of
$$
\operatorname{\mathrm{E}}[\psi(Z;\theta^*,\eta^*)\psi(Z;\theta^*,\eta^*)'].
$$
are bounded below by $c_0$.
In our case, we simply assume
$$
\sigma^2=\operatorname{\mathrm{E}}[\psi(Z;\theta^*,\eta^*)^2]>c_0.
$$
\end{enumerate}
\end{proof}
\begin{proof}[Proof of Theorem~\ref{thm:inference}]
The result immediately follows from Lemmas~\ref{lemma:DML1} and~\ref{lemma:DML2}, which verify the conditions of Corollary~\ref{cor:ci}.
\end{proof}
\bibliographystyle{plain}
\bibliography{refs}