Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
125,168 characters · 28 sections · 98 citation commands
Policy Choice in Time Series by Empirical Welfare Maximization
Keywords: Causal inference, potential outcome time series, treatment choice, regret bounds, concentration inequalities. \thispagestyle{empty}
\setcounter{page}{1} \onehalfspacing
A central topic in economics is the nature of the causal relationships between economic outcomes and government policies, both within and across time periods. To investigate this, empirical research makes use of time-series data, with the aim of finding desirable policy rules. For instance, a monetary policy authority may wish to use past and current macroeconomic data to learn an interest rate policy that is optimal in terms of a social welfare criterion. Building on the recent development of potential outcome time series (white2010granger, angrist2018semiparametric, bojinov2019time, and rambachan2021when), this paper proposes a novel method to inform policy choice when the available data is a multivariate time series.
In contrast to the structural and semi-structural approaches that are common in macroeconomic policy analysis, such as dynamic stochastic general equilibrium (DSGE) models and structural vector autoregressions (SVAR), we set up the policy choice problem from the perspective of the statistical treatment choice proposed by manski2004statistical. The existing statistical treatment choice literature typically focuses on microeconomic applications in a static setting, and the applicability of these methods to a time-series setting has yet to be explored. In this paper, we propose a novel statistical treatment choice framework for time-series data and study how to learn an optimal policy rule. Specifically, we consider extending the conditional empirical success (CES) rule of manski2004statistical and the empirical welfare maximization rule of kitagawa2018should to time-series policy choice, and characterize the conditions under which these approaches can inform welfare-optimal policy. These conditions do not require functional form specifications for structural equations or the exact temporal dependence of the time-series observations, but can be connected to the structural approach under certain conditions.
In the standard microeconometric setting considered in the treatment choice literature, the planner has access to a random sample of cross-sectional units, and it is often assumed that the populations from which the sample was drawn and to which the policy will be applied are the same. These assumptions are not feasible or credible in the time-series context, which leads to several non-trivial challenges. First, the economic environment and the economy's causal response to it may be time-varying. Assumptions are required to make it possible to learn an optimal policy rule for future periods based on available past data. In addition, the outcomes and policies observed in the available data can be statistically and causally dependent in a complex manner, and accordingly, the identifiability of social welfare under counterfactual policies becomes non-trivial and requires some conditions on how past policies were assigned over time. Second, in defining an optimal policy in a time-series context, it is natural to consider both unconditional welfare and conditional welfare given the history of observables available at the time the policy is chosen. The conditional perspective differs from the unconditional one in that it evaluates welfare given the realized history, whereas the unconditional criterion averages these conditional measures across all possible histories. These welfare criteria are unique in the time-series setting, contrasting with the standard settings with cross-sectional microdata. Whether the unconditional or the conditional welfare criterion is more relevant depends on applications and the data environment. Third, when past data are used to inform policy, we have only a single realization of a time series in which the observations are dependent across the periods and possibly nonstationary. Such statistical dependence complicates the characterization of the statistical convergence of the welfare performance of an estimated policy. Fourth, if the planner wishes to learn a dynamic assignment policy, which prescribes a policy in each period over multiple periods on the basis of observable information available at the beginning of every period, the policy learning problem becomes substantially more involved. This is because a policy choice in the current period may affect subsequent policy choices through the current policy assignment and a realized outcome under the assigned treatment.
Taking into account these challenges, we propose time-series empirical welfare maximization (T-EWM) methods that construct an empirical welfare criterion based on a historical average of the outcomes and obtain a policy rule by maximizing the empirical welfare criterion over a class of policy rules. We then clarify the conditions on the causal structure and data-generating process under which T-EWM methods consistently estimate a policy rule that is optimal in terms of unconditional or conditional welfare. Extending the regret bound analysis of manski2004statistical and kitagawa2018should to time-series dependent observations, we obtain a finite-sample uniform bound for welfare regret. Moreover, we characterize the convergence of welfare regret and establish the minimax rate optimality of the T-EWM rule.
Our development of T-EWM builds on the recent potential outcome time-series literature including white2010granger, angrist2018semiparametric, bojinov2019time, and rambachan2021when. In particular, to identify the counterfactual welfare criterion, we employ the sequential exogeneity restriction considered in bojinov2019time. These studies focus on retrospective evaluation of the causal impact of policies observed in historical data, and do not analyze how to perform future policy choices based on the historical evidence.
Since the seminal works of manski2004statistical and Dehejia2005, statistical treatment choice and empirical welfare maximization have been active topics of research, e.g., stoye2009minimax, stoye2012minimax, QianMurphy2011, tetenov2012statistical, BhattacharyaDupas2012, Zhao2012JASA, kitagawa2018should, kitagawa2021equality, kallus2021more, athey2021policy, MT17, KST21, among others. These works focus on a setting where the available data is a cross-sectional random sample obtained from an experimental or observational study with randomized treatment, possibly conditional on observable characteristics. Viviano21 and Abhishek2020OptimalTreatment consider EWM approaches for treatment allocations where the training data features cross-sectional dependence due to network spillovers, while to our knowledge, this paper is the first to consider policy choice with time-series data. As a related but distinct problem, there is a large literature on the estimation of dynamic treatment regimes, murphy2003optimal, zhao2015new, han2021optimal, Sakaguchi21, and ida2024dynamic, among others. The problem of dynamic treatment regimes assumes that training data is a short panel in which treatments have been randomized both among cross-sectional units and across time periods. Recently, adusumilli2019dynamically consider an optimal policy in a dynamic treatment assignment problem with a budget constraint where the planner allocates treatments to subjects arriving sequentially. The T-EWM framework, in contrast, assumes observations are drawn from a single time series as is common in empirical macroeconomics.
A large literature on multi-arm bandit algorithms analyzes learning and dynamic allocations of treatments when there is a trade-off between exploration and exploitation. See lattimore2020bandit and references therein, and Dimakopoulou_et_al_2017, Kock2020functional, kasy2019adaptive, adusumilli2021risk, and Kitagawa_Rowley_2024 for recent works in econometrics. The setting in this paper differs from the standard multi-arm bandit setting in the following three respects. First, our framework treats the available past data as a training sample and focuses on optimizing short-run welfare. We are hence concerned with the performance of the method in terms of short-term regret rather than cumulative regret over a long horizon. Second, in the standard multi-arm bandit problem, subjects to be treated are assumed to differ across rounds, which implies that the outcome-generating process is independent over time. This is not the case in our setting, and we include the realization of outcomes and policies in the past periods as contextual information for the current decision. {Third, suppose that bandit algorithms can be adjusted to take into account the dependence of observations, our method is then analogous to the “pure exploration” class, involving a long exploration phase followed by a one-period exploitation at the very end. However, a major difference is that the bandit algorithm concerns data in a random experiment while our method is aimed at data in quasi-random experiments. }
The analysis of welfare regret bounds is similar to the derivation of risk bounds in empirical risk minimization, as reviewed by Vapnik1998Book and Lugosi2002. Risk bounds studied in the empirical risk minimization literature typically assume independent and identically distributed (i.i.d.) training data. A few exceptions, jiang2010risk, brownlees2021performance, and brownlees2021empirical, obtain risk bounds for empirical risk minimizing predictions with time-series data, but they do not consider welfare regret bounds for causal policy learning.
The rest of the paper is organized as follows. Section (ref) describes the setting using a simple illustrative model with a single discrete covariate. Section (ref) discusses the general model with continuous covariates and presents the main theorems. In Section (ref), we discuss extensions to our proposed framework, including a case of multi-period welfare functions, how T-EWM is related to the Lucas critique, as well as T-EWM's links with structural vector autoregressive models, Markov Decision Processes, and reinforcement learning. In Section (ref), we present an empirical application. Technical proofs, simulation studies, and other details are presented in Appendices.
In this section, we introduce the basic setting, notation, the welfare criterion we aim to maximize, and conditions on the data-generating process that are important for the learnability of an optimal policy. Then, we illustrate the main analytical tools used to bound welfare regret through a heuristic model with a simple dynamic structure.
We suppose that the social planner is at the beginning of time $T$. Let $W_t \in \{0, 1 \}$ denote a treatment or policy (e.g., nominal interest rate) implemented at time $t = 0,1,2,\dots$ To simplify the analysis, we assume that $W_t$ is binary (e.g., a high or low interest rate). The planner sets $W_T \in \{0, 1\}$, $T \geq 1$, making use of the history of observable information up to period $T-1$ to inform her decision. This observable information consists of an economic outcome (e.g., GDP, unemployment rate, etc.), $Y_{0:T-1}=(Y_0,Y_1,Y_2, \cdots, Y_{T-1})$, the history of implemented policies, $W_{0:T-1}=(W_0,W_1,W_2, \cdots, W_{T-1})$, contextual covariates (e.g., inflation), $Z_{0:T-1}=(Z_0, Z_1,Z_2, \cdots, Z_{T-1})$, and potential mediators between treatment and outcome, $M_{0:T-1}=(M_0, M_1,M_2, \cdots, M_{T-1})$. $Z_t$ and $M_t$ can be multidimensional vectors, but $Y_t$ and $W_t$ are assumed to be univariate.
Following bojinov2019time, we refer to a sequence of policies $w_{0:t} = (w_0, w_1,$ $ \dots, w_t) \in \{0,1\}^{t+1}$, $t \geq 0$, as a treatment path. A realized treatment path observed in the data $0 \leq t \leq T-1$ is a stochastic process $W_{0:T-1}$ drawn from the data-generating process. The timing of realizations is therefore
i.e., the transition between periods happens between the $Z_{t-1}$ is realised and before $W_t$ is realised. Furthermore, let
collect the observable variables for period $t$. For $t = 0,1,2,\dots$,. let $\mathcal{X}$ denote the sample space of $X_t$. Furthermore, we define the filtration $$\mathcal{F}_{t-1}=\sigma (X_{0:t-1}),$$ where $\sigma(\cdot)$ denotes the Borel $\sigma$-algebra generated by the variables specified in the argument. The filtration $\mathcal{F}_{T-1}$ corresponds to the planner's information set at the time of making her decision in period $T$.
Following the framework of bojinov2019time, we introduce potential outcome time series. At each $t = 0, 1, 2, \dots,$ and for every treatment path $w_{0:t} \in \{0 ,1 \}^{t+1}$, let $Y_t(w_{0:t}) \in \mathbb{R}$ be the realized period $t$ outcome if the treatment path from $0$ to period $t$ were $w_{0:t}$. Hence, we have a collection of potential outcome paths indexed by treatment path, $$ \{ Y_t(w_{0:t}) : w_{0:t} \in \{0, 1 \}^{t+1}, t=0,1,2,\dots \}, $$ which defines $2^{t+1}$ potential outcomes in each period $t$. This is an extension of the Neyman-Rubin causal model originally developed for cross-sectional causal inference. As maintained in bojinov2019time, the potential outcomes for each $t$ are indexed by the current and past treatments $w_{0:t}$ only. This imposes the restriction that any future treatment $w_{t+p}$, $p \geq 1$, does not causally affect the current outcome, i.e., an exclusion restriction for future treatments.
For a realized treatment path $W_{0:t}$, the observed outcome $Y_t$ and the potential outcomes satisfy $$ Y_t = \sum_{w_{0:t} \in \{ 0, 1\}^{t} } 1 \{W_{0:t} = w_{0:t} \} Y_t(w_{0:t}) $$ for all $t \geq 0$.
The baseline setting of the current paper considers the choice of policy $W_T$ for a single period $T$.\footnote{Section (ref) discusses how to extend the single-period policy choice problem to multi-period settings.} We denote the policy choice based on observations up to period $T-1$ by
where $\mathcal{X}^T$ denotes the sample space of $X_{0:T-1}$. The period-$T$ treatment is $W_T = g(X_{0:T-1})$, and we refer to $g(\cdot)$ as a decision rule.
We assume that the planner's preferences for policies in period-$T$ are embodied in a social welfare criterion. In particular, we define one-period welfare conditional on $\mathcal{F}_{T-1}$ (conditional welfare, for short) to be\footnote{Throughout the paper, we acknowledge that the expectation, $\mathop{\mbox{\sf E}}$, and the probability, $\Pr$, are corresponding to the outer measure whenever a measurability issue is encountered.}
As an alternative welfare criterion, we can define the unconditional welfare by averaging out the conditioning information in (ref),
Whether the planner's objective function should be conditional or unconditional welfare depends on applications and the goal of analysis. If the planner's goal is to perform policy choice in a potentially nonstationary environment given the realized path of scenarios, it is natural to set the conditional welfare as the criterion to maximize. This case contrasts with the setting of the cross-sectional treatment choice with micro data where the unconditional welfare that averages out the observable characteristics of a unit is common.\footnote{manski2004statistical also considers a conditional welfare criterion in the cross-sectional setting.}
On the other hand, there are cases in which studying optimal policy from the perspective of unconditional welfare is suitable. First, an optimal policy in terms of the unconditional welfare can inform an optimal policy action that the planner would choose under a counterfactual history. Such analysis could be of interest in the practice of ex post counterfactual policy analysis. Second, in settings where multiple units exist, learning an unconditionally optimal policy from the time-series data of a particular unit informs the planner of optimal policies for other similar units but with different histories. Third, an unconditionally optimal policy can also inform an optimal policy in the setting where the planner implements the same Markovian policy over multiple time periods, assuming the data-generating process is stationary.
Note that the T-EWM based on unconditional welfare yields the same empirical average as the EWM in cross-sectional data. The two, however, differ markedly in substance. In time-series settings, observed outcomes and policies are subject to statistical and causal dependence across periods, whereas cross-sectional data are typically assumed to be i.i.d. across the units. Due to this difference, external-validity type assumptions become more debatable in the time-series setting than in the cross-sectional setting.
Learning conditionally optimal policies is generally more challenging than learning unconditionally optimal ones since estimation of the unconditional welfare can use all the observations in the sample, whereas estimation of the conditional welfare may have to rely on a subset of observations. In Section (ref), however, we show that under certain assumptions, the unconditional welfare regret can bound the conditional welfare regret so that one can learn the conditionally optimal policy as fast as the unconditionally optimal one.
For the welfare targets (ref) and (ref), define the planner's optimal conditional and unconditional policies that maximize her one-period welfare as
respectively. The former, $g^{\ast}_{X_{0:T-1}}$, can be regarded as the optimal policy $g^{\ast}$ in the latter evaluated at the conditioning value of $X_{0:T-1}$. The planner does not know $g^{\ast}$, so she instead seeks a statistical treatment choice rule (manski2004statistical, manski2004statistical) $\hat{g}$, which is a decision rule selected on the basis of the available data $X_{0:T-1}$.
Our goal is to develop a way of obtaining $\hat{g}$ that performs well in terms of the conditional welfare criteria ((ref)) and (ref). Specifically, we assess the statistical performance of an estimated policy rule $\hat{g}$ in terms of the convergence of conditional and unconditional welfare regrets,
as well as their convergence rate with respect to the sample size $T$. When examining convergence, we accommodate statistical uncertainty over $\hat{g}$ by focusing on convergence with probability approaching one uniformly over a class of sampling distributions for $X_{0:T-1}$.
We begin our analysis with a simple illustrative model, which provides a heuristic exposition of the main idea of T-EWM and its statistical properties. We cover more general settings and extensions in Sections (ref) and (ref).
Suppose that the data consist of a bivariate time series $X_{0:T-1} = ((Y_t,W_t) \in \mathbb{R} \times \{0, 1 \} : t=0,1, \dots, T-1)$ with no other covariates. To simplify exposition for the illustrative model, we impose the following restrictions on the dynamic causal structure and dependence of the observations.
These assumptions significantly simplify the dynamic structure of the problem. Markovian exclusion, Assumption (ref)(i), says that only the current treatment $W_{t}$ and treatment in the previous period $W_{t-1}$ can have a causal impact on the current outcome. This allows the indices of the potential outcomes to be compressed to the latest two treatments $(w_{t-1},w_t)$, as in ((ref)). Markovian exogeneity, Assumption (ref)(ii), states that once one conditions on the policy implemented in the previous period $W_{t-1}$, the potential outcomes and treatment for the current period are statistically independent of any other past variables.
It is important to note that these assumptions do not impose stationarity: we allow the distribution of potential outcomes to vary across time periods. In addition, under Assumption (ref), we can reduce the class of policy rules to those that map from $W_{T-1} \in \{0,1 \} $ to $W_T \in \{0,1 \}$, i.e., $$g : \{0 , 1\} \to \{0, 1\}.$$ Consequently, welfare conditional on $\mathcal{F}_{T-1}$ can be simplified to
where the first equality follows from the definition of $\mathcal{F}_{T-1}$ and Assumption (ref)(i); the second equality follows from Assumption (ref)(ii).
To make sense of Assumption (ref) and illustrate the relationship between the potential outcome time series and the standard structural equation modeling, we provide a toy example.
Compatibility with Assumption (ref) can be seen as follows. Assumption (ref)(i) is implied by (ref), where the structural equation of $Y_t$ involves only $(W_t,W_{t-1})$ as the factors of direct cause. Assumption (ref)(ii) is implied by (ref), (ref), and the fact that the distribution of $V_{t}$ depends solely on $W_{t-1}$, i.e., under (ref),(ref), and (ref), we have $\Pr(W_t|\mathcal{F}_{t-1})=\Pr(W_t|W_{t-1}).$\\
To examine the learnability of the optimal policy rule, we further restrict the data-generating process. First, we impose a strict overlap condition on the propensity score.
The next assumption imposes an unconfoundedness condition on observed policy assignment.
This assumption states that the treatments observed in the data are sequentially randomized conditional on lagged observable variables. This is a key assumption to make unbiased estimation of the welfare feasible in each period in the sample, as employed in bojinov2019time and others. It is worth noting that the above assumption together with Assumption (ref)(ii) implies $Y_{t}(W_{t-1},w)\perp W_t|W_{t-1}.$
Under Assumption (ref), we have that, for any measurable function $f$ of the potential outcome $Y_{t}(W_{0:t-1})$ and treatment $W_t$, it holds
{\bf Example (ref) continued.} Assumption (ref) is satisfied if $0<p<1$ and $0<q<1$; Assumption (ref) is implied by (ref) and (ref).\\
Imposing Assumption (ref) and (ref) and assuming propensity scores are known, we consider constructing a sample analogue of ((ref)) conditional on $W_{T-1}=w$ based on the historical average of the inverse propensity score weighted outcomes,
where $T(w) = \#\{1\leq t \leq T-1: W_{t-1} = w\} $ is the number of observations where the policy in the previous period took value $w$, i.e., the subsample corresponding to $W_{t-1} = w$. Unlike the microeconometric setting considered in, e.g., kitagawa2018should, we do not necessarily have $\widehat{\mathcal{W}}(g|W_{T-1}=w)$ as a direct sample analogue for the planner's social welfare objective, since we allow a non-stationary environment in which the historical average of the conditional welfare criterion can diverge from the conditional welfare in the current period. Nevertheless, we refer to $\widehat{\mathcal{W}}(g|W_{T-1}=w)$ as the empirical welfare of the policy rule $g$.
Denoting $(\cdot|W_{T-1}=w)$ by $(\cdot|w)$, we define the true optimal conditional policy and its empirical analogue to be,
where $\hat{g}$ is constructed by maximizing empirical welfare over a class of policy rules (four policy rules in total). We call a policy rule constructed in this way the Time-series Empirical Welfare Maximization (T-EWM) rule. The construction of the T-EWM rule $\hat{g}$ is analogous to the conditional empirical success rule with known propensity scores considered by manski2004statistical in the i.i.d. cross-sectional setting. In the time-series setting, however, the assumptions imposed so far do not guarantee that $\widehat{\mathcal{W}}(g|w)$ is an unbiased estimator of the true conditional welfare $\mathcal{W}_{T}(g|w)$.
A major contribution of this paper is characterizing conditions that justify the T-EWM rule $\hat{g}$ in terms of the convergence of conditional welfare. This section clarifies these points in the context of our illustrative example.
To bound conditional welfare regret, our strategy is to decompose empirical welfare $\widehat{\mathcal{W}}(g|w)$ into a conditional mean component and a deviation from it. The deviation is the sum of a martingale difference sequence (MDS), and this allows us to apply concentration inequalities for the sum of MDS. Define an intermediate welfare function,
Under the strict overlap and unconfoundedness assumptions (i.e., Assumptions (ref) and (ref)), the difference between empirical welfare and (ref) is a sum of MDS. A concentration inequality for average MDS then implies that the empirical welfare concentrates around $\bar{\mathcal{W}}(g|w)$. Since $\bar{\mathcal{W}}(g|w)$ is not guaranteed to inform an optimal policy in terms of $\mathcal{W}_T(g|w)$, we impose the assumption:
Noting that the left-hand side of ((ref)) is nonnegative for any $g$ by construction and $c>0$, this assumption implies that $\bar{\mathcal{W}}(g^*|w)-\bar{\mathcal{W}}(g|w) > 0$ must hold whenever $\mathcal{W}_T(g^*|w)- \mathcal{W}_T(g|w) > 0$. That is, the optimality of $g^{\ast}$ in terms of the conditional value of welfare at $T$ is maintained in the historical average of the conditional values of welfare. Under this assumption, having an estimated policy $\hat{g}$ that attains a convergence of $\bar{\mathcal{W}}(g^*|w)-\bar{\mathcal{W}}(\hat{g}|w)$ to zero guarantees that $\hat{g}$ is also consistent for the optimal policy $g^{\ast}$ in terms of the conditional welfare at $T$.
{\bf Example (ref) continued.}\\ $\varepsilon_{t}$ remains an i.i.d. noise in the following settings.\\ (i) By Remark (ref), Assumption (ref) holds for Example 1 since $S_{t}(w)=\beta_{0}+\beta_{1}\cdot g+\beta_{2}\cdot w+\varepsilon_{t}$ is weakly stationary.
(ii) If we replace (ref) by \[ Y_{t}=\delta_{t}+\beta_{1}W_{t}+\beta_{2}W_{t-1}+\varepsilon_{t}, \] where $\delta_{t}$ is an arbitrary deterministic time trend. The process $Y_{t}$ is trend stationary (non-stationary), but Assumption (ref) still holds with $c=1$ since those deterministic trends are canceled out by differences, i.e., \[ \mathcal{W}_{T}(g^{*}|w)-\mathcal{W}_{T}(g|w)=\beta_{1}\left(g^{*}-g\right)=\bar{\mathcal{W}}(g^{*}|w)-\bar{\mathcal{W}}(g|w). \]
(iii) If we replace (ref) by \[ Y_{t}=\beta_0 + \beta_{1}W_{t}+\beta_{2}W_{t-1}+\sum_{i=0}^{t}\varepsilon_{i}, \] the process $Y_{t}$ is non-stationary with stochastic trends, but Assumption (ref) still holds with $c=1$ since the stochastic trends are canceled out by differences.
(iv) If we replace (ref) by \[ Y_{t}=\delta_t + \beta_{1,t}W_{t}+\beta_{2,t}W_{t-1}+\varepsilon_{t} \] to allow heterogeneous treatment effect, then
where $\bar{\beta}_{w}:=\frac{1}{T(w)}\sum_{1\leq t\leq T-1:W_{t-1}=w}\beta_{1,t}$. Since $\mathcal{W}_{T}(g^{*}|w)-\mathcal{W}_{T}(g|w)$ is non-negative by the definition of $g^{*}$ in (ref), Assumption (ref) holds if $\beta_{1,T}$ and $\bar{\beta}_{w}$ have the same sign and $\frac{|\beta_{1,T}|}{|\bar{\beta}_{w}|}\leq c.$.
Without loss of generality, we can assume that both $\beta_{1,T}$ and $\bar{\beta}_{w}$ are positive, then a sufficient condition for Assumption (ref) is: There are positive numbers $l$ and $u$, such that $0<l\leq\beta_{1,t}\leq u$ holds for all $t$. In this case, $c=\frac{u}{l}$.
Assumption (ref) implies that $g^*$ also maximizes $\bar{\mathcal{W}}$. Hence, if empirical welfare $\widehat{\mathcal{W}}(\cdot|w)$ can approximate $\bar{\mathcal{W}}(\cdot|w)$ well, intuitively the T-EWM rule $\hat{g}$ should converge to $g^{\ast}$. The motivation for Assumption (ref) is to create a bridge between $\mathcal{W}_{T}(g^*|w)-\mathcal{W}_{T}(\hat{g}|w)$, the population regret, and $\widehat{\mathcal{W}}(\cdot|w) - \bar{\mathcal{W}}(\cdot|w) $, which is a sum of MDS with respect to filtration $\{ \mathcal{F}_{t-1} : t = 1,2, \dots, T \}$.\footnote{Note that by (ref), $\mathop{\mbox{\sf E}}(\cdot|\mathcal{F}_{t-1})=\mathop{\mbox{\sf E}}(\cdot|W_{t-1} )$.} Specifically,
where the first inequality follows from Assumption (ref). The second inequality follows from the definition of T-EWM rule $\hat{g}$ in (ref).
To bound the right-hand side of (ref), define
Then, we can express (ref) and (ref) as
and
follows. Next, we impose
The proof of this proposition can be found in Appendix (ref). With Proposition (ref), we can apply a concentration inequality for sums of MDS to obtain a high-probability bound for $\widehat{\mathcal{W}}(g|w) - \bar{\mathcal{W}}(g|w)$ that is uniform in $g$.
The proof of Theorem (ref) can be found in Appendix (ref). Combining ((ref)) (an implication of Assumption (ref)) and ((ref)), we can conclude that the convergence rate of expected regret $\mathop{\mbox{\sf E}}[\mathcal{W}_{T}(g^*|w)-\mathcal{W}_{T}(\hat{g}|w)]$ is upper-bounded by $2c\cdot\frac{C}{\sqrt{T-1}}$ uniformly in $w\in \{0,1\}$.
Time-series models commonly used in macroeconomic policy analysis imply an infinite order Markovian structure. For example, we consider the following modification to Example (ref).
Without requiring stationarity or functional form restrictions, we can extend our framework to infinite Markovian order and obtain convergence of welfare regret conditional on a treatment path of infinite length, $\mathcal{W}_{T}(g^{*}|w_{-\infty:T-1})-\mathcal{W}_{T}(\hat{g}|w_{-\infty:T-1})$. Let $1< m < T$ and $\hat{g}$ be a policy that maximizes the empirical welfare $\widehat{\mathcal{W}}(g|w_{T-m:T-1})$ with conditioning policy path truncated to $w_{T-m:T-1}$. Appendix (ref) shows the following upper bound for the welfare regret of infinite order Markov models:
where $\bar{\mathcal{W}}(g|\mathcal{F}_{t-1})$ is the intermediate welfare defined by (ref) in Appendix (ref), around which the empirical welfare is expected to concentrate, and $\widetilde{\text{w-bias}}_{\infty}\left(m\right)$ is the welfare bias due to truncation of the empirical welfare defined by (ref) in Appendix (ref). The first term on the right-hand side represents the average of an MDS. Under the regularity conditions presented in Appendix (ref), we can show that this term converges at a rate of $\frac{1}{\sqrt{T-m}}$. For the second term on the right-hand side, we shall have $\text{plim}_{m\to\infty}\widetilde{\text{w-bias}}_{\infty}\left(m\right)=0$ under additional conditions that ensure the decay temporal dependence. See Appendix (ref) for more details. Furthermore, in Appendix (ref), we show that the infinite order T-EWM proposed in Appendix (ref) can be applied to the policy choice problem specified in Example (ref).
This section extends the illustrative example of Section (ref) by allowing $X_t$ to contain continuous variables. We first introduce the continuous setting in Section (ref). Then, Section (ref) presents a natural approach for handling the continuous policy variable, which employs a kernel function to construct an analogue of the conditional empirical welfare function (ref) presented in Section (ref). Finally, having introduced the unconditional welfare in the time-series setting in Section (ref), we formally present the unconditional T-EWM method in Section (ref), developed in parallel with the kernel-based approach.
In addition to $(Y_t,W_t)$, we incorporate general covariates $Z_t$ into $X_t \in \mathcal{X}$, which can be continuous. Now, $X_t=(Y_t,W_t,Z_t)$. For simplicity of exposition, we maintain the first-order Markovian structure similarly to the illustrative example, while modifying Assumptions (ref), (ref), and (ref) as follows. It is straightforward to incorporate a higher-order Markovian structure.
Similarly to ((ref)) in the illustrative example, Assumption (ref) implies that we can reduce the conditioning information of $\mathcal{F}_{t-1}$ to only $X_{t-1}$ and reduce the policy to a binary map of $X_{t-1}$ without any loss of conditional welfare. Following these reductions and considering the planner's focus on the policy choice in period $T$, we can formulate the planner's objective function as follows:
where the policy function $g$ is defined in (ref). We assume the strict overlap and unconfoundedness restrictions under the general covariates as follows.
{
Under Assumptions (ref) and (ref), we can generalize ((ref)) by including the set of covariates in the conditioning variables: for any measurable function $f$,
For continuous conditioning covariates $X_{T-1}$, a simple sample analogue of the objective function is not available due to the lack of multiple observations at any single conditioning value of $X_{T-1}$. One approach is to use nonparametric smoothing to construct an estimate for conditional welfare.
For simplicity, we let $X_{T-1}\in \mathbb{R}$. Then, with a kernel function $K(\cdot)$ and a bandwidth $h$, the empirical conditional welfare can be rewritten as\footnote{If the set of conditioning variables $X_{T-1}$ contains both continuous and discrete components, we can adapt a hyper method to construct a valid sample analogue combing kernel-smoothing (for continuous variables) and subsamples (for discrete variables). In this section, we focus on the case where the target welfare function is conditional on a univariate continuous variable} {
} where $(\cdot|x)$ denotes $(\cdot|X_{T-1}=x)$, $\widehat{\mathcal{W}}_t(g) := \frac{Y_{t}W_{t}}{e_t(X_{t-1})} g(X_{T-1})+ \frac{Y_{t}(1-W_{t})}{1- e_t(X_{t-1})}[1-g(X_{T-1})]$, and $K_h(a,b):=\frac{1}{h}K(\frac{a-b}{h})$.
Using the same notation, we can define $\mathcal{W}_{t}(g|x) =\mathop{\mbox{\sf E}}[ Y_{t}(W_{t-1},1)g(X_{t-1})+Y_{t}(W_{t-1},0)[1-g(X_{t-1})]|X_{t-1}=x]$. Then, similar to (ref) and (ref), we define
to be the maximizers of $\mathcal{W}_T(g|x)$ and $\widehat{\mathcal{W}}(g|x)$. Moreover, define an intermediate welfare function:
The invariance of welfare ordering assumption is modified to:
Similar to Assumption (ref), Assumption (ref) holds if the stochastic process $S_t(x):=Y_{T}(W_{T-1},1)$ $g(X_{T-1})+Y_{t}(W_{T-1},0)[1-g(X_{T-1})]|_{X_{T-1}=x}$ is weakly stationary.
The following theorem shows an upper bound for conditional regret in the one-dimensional covariate case (i.e., $X_t\in \mathbb{R}^1$). This result can be readily extended to the multiple-covariate case.
Setting $h = O(T^{-1/5})$, the right-hand side bound becomes $O(T^{-2/5})$. A proof is presented in Appendix (ref).
In this section, we shift the focus to maximizing unconditional welfare. For ease of exposition with the unconditional T-EWM, given a policy function $g:\mathcal{X}^{T} \to \{0,1\}$, we define the corresponding region in the space of the covariate vector for which the decision rule chooses $W_T =1$ to be
We refer to $G$ as a decision set. Under Assumption (ref), the unconditional welfare under policy $G$ is
The sample analogue of $\mathcal{W}_T(G)$ can be expressed as
For $\mathcal{G}$ being a class of decision sets, we define
In addition, define two intermediate welfare functions,
The need for an additional intermediate welfare function, $\widetilde{\mathcal{W}}(G)$, arises because we aim to bound the unconditional welfare. After centering the empirical welfare around its conditional mean, $\bar{\mathcal{W}}(G) $, we are left with another difference, $\bar{\mathcal{W}}(G)-\widetilde{\mathcal{W}}(G)$, which we will control below.
To obtain a regret bound for unconditional welfare, Assumption (ref) is modified to:
Note that by Assumption (ref), we have
where the first inequality follows from (ref), and the second inequality follows from an argument similar to the one for (ref).
Note that $\widehat{\mathcal{W}}(G)-\widetilde{\mathcal{W}}(G)$ is not a sum of MDS. Instead, it can be decomposed as
where $I:=\bar{\mathcal{W}}(G)-\widetilde{\mathcal{W}}(G)$ and $II:=\widehat{\mathcal{W}}(G)-\bar{\mathcal{W}}(G)$. Subject to assumptions specified later, Theorem (ref) below shows that $II$, which is a sum of MDS, converges at $\frac{1}{\sqrt{T-1}}$-rate, and Theorem (ref) below shows that $I$ converges at the same rate.
(ref) reveals that our proof strategy is considerably more complicated than the proof for the EWM model with i.i.d. observations of kitagawa2018should, although the rates are similar. Specifically, we need to derive a bound for the tail probability of the sum of martingale difference sequences. In addition, we need to handle complex functional classes induced by non-stationary processes. For the EWM model, the main task is to show the convergence rate of a sample analogue of $II$, which can be achieved with standard empirical process theory for i.i.d.\ samples. In comparison, we not only have to treat our $II$ more carefully due to time-series dependence, but we also have to deal with $I$.
Similar to our approach with the simple model described in Section (ref), we assume that $Y_t$, which is reintroduced in Section (ref), has a bounded support.
Define empirical welfare at time $t$ and its population conditional expectation as follows,
Then, we examine two summations:
For each $t=1, \dots, T-1$, define a function class indexed by $G \in \mathcal{G}$,
where the arguments of the function $h_t(\cdot;G)$ are $Y_{t}$, $W_t$, and $X_{t-1}$. In the following, we use $n$ to represent the number of summands since the endpoints of samples may vary across different settings in this section, subsequent sections, and appendices. For example, in the case of multi-period welfare functions, the endpoint of a sample is no longer fixed at $T-1$. Given the class of functions $\mathcal{H}_t$, we consider a martingale difference array $\{h_t(Y_{t},W_t,X_{t-1};G)\}_{t=1}^{n}$, and denote its average by $${\mathop{\mbox{\sf E}}}_{n} h\stackrel{\mathrm{def}}{=} \frac{1}{n}\sum_{t=1}^{n} h_t(Y_{t},W_t,X_{t-1}; G),$$ where $h \stackrel{\mathrm{def}}{=} \left\{ h_{1}(\cdot; G),h_{2}(\cdot;,G),\dots,h_{n}(\cdot;G)\right\} $, and we suppress $n$ and $G$ if there is no confusion in the context.
Since we do not restrict $X_t$ to be stationary, we shall handle a vector of function classes that possibly vary over $t$. To this end, we define the following set of notations. Let $H_t$ denote the envelope for the function class $\mathcal{H}_t$, and $\overline{H}_{n}=(H_{1},H_{2},\cdots,H_{n})^{\prime}$, and $\mathbf{H_{n}}=\mathcal{\mathcal{H}}_{1}\times \mathcal{\mathcal{H}}_{2}\times \dots \times \mathcal{H}_{n}.$ For a function $f$ supported on $\mathcal{X}$, define $\|f\|_{Q,r} \stackrel{\mathrm{def}}{=} (\int_{x\in\mathcal{X}} |f(x)|^r dQ(x))^{1/r} $, and for an $n$-dimensional vector $v=\left\{ v_{1},\dots,v_{n}\right\} $, its $l_{2}$ norm is denoted by $|v|_{2}\overset{\text{def}}{=}\left(\sum_{i=1}^{n}v_{i}^{2}\right)^{1/2}$. The covering number of a function class $\mathcal{H}$ w.r.t. a metric $\rho$ is denoted by $\mathcal{N}(\varepsilon, \mathcal{H}, \rho(.))$. For two series of functions $f=\{f_t \}_{i=1}^n$ and $g=\{g_t \}_{i=1}^n$, define the metrics $\rho_{2,n}(f,g) = (n^{-1}\sum_t|f_t-g_t|^2)^{1/2}$ and $\sigma_n(f,g) = \left( n^{-1} \sum_t \mathop{\mbox{\sf E}}[(f_t-g_t)^2| \mathcal{F}_{t-1}]\right)^{1/2}$. Let $\alpha_{n}$ denote an $n$-dimensional vector in $\mathbb{\mathbb{R}}^{n}$ and $\circ$ denote the element-wise product. In the next assumption, we want to bound the covering number of $\mathcal{N}(\delta|\overline{H}{}_{n}|_{2}, \mathbf{H_{n}}, \rho_{2,n})$ by the covering number of all its one-dimensional projection. Now, let $A\lesssim B$ denote that there exists some constant $c_0$ such that $A \leq c_0\cdot B$.
Assumption (ref) restricts the complexity of the function class to be of polynomial discrimination, and the complexity index $v$ appears in the derived regret bounds. See Appendix (ref) for a justification for this assumption.
$\rho_{2,n}(f,g)^2$ is the quadratic variation difference and $\sigma_n(f,g)^2$ is its conditional equivalent. It is evident that $\rho_{2,n}(f,g)^2- \sigma_n(f,g)^2$ involves martingale difference sequences. In the special case of i.i.d. observations, $\sigma_n(f,g)^2$ is equivalent to the sample average of unconditional expectations. Assumption (ref) can thus be viewed as specifying that $\rho_{2,n}(f,g)^2$ and $\sigma_n(f,g)^2$ are asymptotically equivalent in a probability sense. A similar condition can be seen, for example, in Theorem 2.23 of hall2014martingale.
Now, let $A\lesssim_p B$ denote $A=O_p(B)$. Then, we have for $II$:
The proof of Theorem (ref) is presented in Appendix (ref).
Here, we complete the process of bounding unconditional regret. Let us define
and
We can apply a similar technique in Theorem (ref) (See Lemma (ref) in Appendix (ref)) to bound the sum of $\overline{S}_{t}(G)$. The second term $\tilde{S}_{t}(G)$ is handled below. Recalling (ref) and (ref), we have $I=\frac{1}{T-1}\sum_{t=1}^{T-1}\tilde{S}_{t}(G)+ \frac{1}{T-1}\sum_{t=1}^{T-1}\overline{S}_{t}(G)$. Define the function classes,
Note that by Assumption (ref), we have $\mathop{\mbox{\sf E}}(S_{t}(G)| \mathcal{F}_{t-1}) = \mathop{\mbox{\sf E}}(S_{t}(G)| X_{t-1})$, and $\mathop{\mbox{\sf E}}(S_{t}(G)| \mathcal{F}_{t-2})=\mathop{\mbox{\sf E}}(S_{t}(G)| X_{t-2})$.
{It shall be noted that Assumption (ref) implies $\tilde{S}_{t}(G) = \tilde{g}_{t}({\varepsilon}_t,{\varepsilon}_{t-1}, \cdots )$, where $\tilde{g}$ is another measurable function of ${\varepsilon}$'s.}
The next assumption pertains to the tail of the underlying innovations of time series.
}
Assumption (ref) imposes that the time series $\tilde{S}_{t}(G)$ and $X_t$ can be expressed as measurable functions of i.i.d.\ innovations ${\varepsilon}_t$. Assumption (ref)(i) completes the Markov exogeneity (Assumption (ref)) in the presence of the covariate $Z_t$, which is introduced at the beginning of Section (ref). Assumption (ref)(ii) is a standard envelope assumption on the function class, and it states that the function of interest is enveloped by a function of $X_{t-2}$. Assumption (ref) implies that $\Phi_{\phi_{\tilde{v}}}(\tilde{S}_.(G))< \infty$ for $\tilde{v} = 1/2$ or 1. Assumption (ref) restricts the complexity of the function classes. Based on these assumptions, we have the following rate:
A proof is presented in Appendix (ref). The bound depends on the complexity of the function class, $V$ and $v$, and the time-series dependency, $\mbox{sup}_{G}\Phi_{\phi_{\tilde{v}}}(\tilde{S}_{.}(G))$. As we consider the sample analogue of unconditional welfare, all the observations are utilized, resulting in a $\frac{1}{\sqrt{T-1}}-$rate of convergence.
Now, we can obtain the overall bound for unconditional welfare, using (ref). Let $P_T$ be a joint probability distribution of a sample path of length $(T-1)$, $\mathcal{P}_T(M,\kappa)$ be the class of $P_T$, which satisfies Assumptions (ref) to (ref), and (ref) to (ref); let ${\mathop{\mbox{\sf E}}}_{P_{T}}$ be the expectation taken over different realizations of random samples. Recall that $x_{obs}$ is defined to be the observed value of $X_{T-1}$.
This theorem follows from (ref), Theorems (ref) and (ref), and analogous reasoning presented in Appendix (ref).
The kernel method for the conditional regret discussed in Section (ref) is a direct way to estimate an optimal policy with the conditional welfare criterion. However, the localization by bandwidth slows down the speed of learning; the regret of conditional welfare can only achieve a $\frac{1}{\sqrt{(T-1)h}}$-rate of convergence rather than a $\frac{1}{\sqrt{T-1}}$-rate. This slow convergence rate may limit its practicality when the covariates $X_{t-1}$ are multi-dimensional or when we extend the order of Markovian dependence to multiple periods. Therefore, in what follows, we instead pursue a novel approach that estimates a conditional optimal policy rule by maximizing an empirical analogue of unconditional welfare over a specified class of decision sets, $\mathcal{G}$. We show that under additional assumptions, this approach can lead to the convergence rate of the conditional welfare that is free from the curse of dimensionality. To this end, we rewrite the conditional welfare (ref) with the notation for the decision set $G$:
The relationship between $g$ and $G$ is shown in (ref). We first clarify how a maximizer of conditional welfare (ref) can be linked to a maximizer of unconditional welfare. With this result in hand, we can focus on estimating unconditional welfare and choosing a policy by maximizing it. In our setup, the complexity of the functional class needs to be specified by the user.
We will show that this approach can attain a $\frac{1}{\sqrt{T-1}}$ rate of convergence. Faster convergence relative to the kernel approach comes at the cost of imposing an additional restriction on the data-generating process, as we spell out in the next assumption.
This assumption ensures that maximizing unconditional welfare corresponds to maximizing conditional welfare over $\mathcal{G}$. A sufficient condition for the equivalence of maximizing conditional and unconditional welfare is that the specified class of policy rules, $\mathcal{G}$, includes the first best policy $G^*_{FB}$ for the unconditional problem, where
This sufficient condition states that the class of policy rules over which unconditional empirical welfare is maximized contains the set of points in $\mathcal{X}$ where the conditional average treatment effect $\mathsf{E}[Y_{T}(W_{T-1},1)-Y_{T}(W_{T-1},0)|X_{T-1}=x]$ is positive. This assumption thus restricts the distribution of potential outcomes at $T$ and its dependence on $X_{T-1}$. We refer to Assumption (ref) as `correct specification'.\footnote{kitagawa2018should, KST21, and Sakaguchi21 consider correct specification assumptions exclusively for unconditional welfare criteria. These assumptions correspond to $G_{FB}^{\ast} \in \mathcal{G}$.} The planner can be confident about Assumption (ref) if, for instance, $\mathsf{E}[Y_{T}(W_{T-1},1)-Y_{T}(W_{T-1},0)|X_{T-1}=x]$ is believed to be monotonic in $x$ (element-wise) and the class $\mathcal{G}$ consists of decision sets with monotonic boundaries MT17,KST21.
With Assumption (ref), we can shift the focus to maximizing unconditional welfare, even when the planner's ultimate objective function is conditional welfare. We also note several side-benefits of considering optimal policy in terms of unconditional welfare. The following proposition directly results from Assumptions (ref) (i), Assumption (ref), and the definition of $G^*_{FB}$.
Having assumed the relationship of optimal policies between the two welfare criteria, we now show how the unconditional welfare function can bound the conditional function. For $G \in \mathcal{G}$ and $X_{T-1}=x$, define conditional regret as \[ R_{T}(G|x)=\mathcal{W}_{T}(G_{*}|x)-\mathcal{W}_{T}(G|x). \] Note that the unconditional regret can be expressed as an integral of the conditional regret,
For $x' \in \mathcal{X}$, define
and let $x^{obs}$ denote the observed value of $X_{T-1}$. We assume the following:
The following lemma provides a bound for conditional regret $ R_{T}(G|x^{obs})$ using unconditional regret $ R_{T}(G)$.
The proof of this lemma can be found in Appendix (ref). Using Assumption (ref) and Lemma (ref), we proceed to bound the regret for conditional welfare $ \mathcal{W}_{T}(G_{*}|x^{obs})-\mathcal{W}_{T}(\hat{G}|x^{obs})$ by regret for unconditional welfare $\left[\mathcal{W}_{T}(G_{*})-\mathcal{W}_{T}(\hat{G})\right]$, and further by $\left[\mathcal{\widetilde{W}}(G)-\widehat{\mathcal{W}}{}(G)\right]$ (up to constant factors):
The first inequality follows from Lemma (ref) and Assumption (ref). The second inequality follows from Assumption (ref). The last inequality follows from an argument similar to the one below (ref). As a result, the bound of the conditional regret can be further established using Theorems (ref) and (ref).
In this section, we discuss a few possible extensions. Section (ref) introduces a multi-period policy-making framework. Section (ref) concerns the ability of the current methods to handle the Lucas critiques. Section (ref) discusses a few connections of T-EWM to the literature on optimal policy choice. More extensions are discussed in Appendix (ref).
So far, we have considered only the case of a one-period welfare function. This subsection discusses how to extend the current setting to a multiple-period policy framework. In the interest of space, we focus on cases with discrete covariates and extend the simple model in Section (ref) to a two-period welfare function. Extending this model beyond two periods is straightforward using similar reasoning. Recall Assumption (ref), which imposed Markov properties on the data-generating processes,
The planner chooses policy rules for two periods, $g_1(\cdot)$ and $g_2(\cdot): \{0,1\}\to \{0,1\}$, to maximise aggregate welfare over periods $T$ and $T+1$. The decision in the second period, $g_2(.)$, is not contingent on the functional form of $g_1(.)$. By Assumption (ref), the period $T+1$ welfare is determined only by the treatment choice in the period $T$, i.e., $W_T=g_1(W_{T-1})$. The two-period welfare function can be written as
where the last equality follows from Assumption (ref) and (ref). To be more specific with the analytical format, we suppress the $(\cdot)$ in $g_i(\cdot)$, when the meaning is clear from the context.
The second term of (ref) follows from
where the last equality follows from Assumption (ref). To estimate the above welfare function, we recall the definition of $T(w)=\#\{1\leq t\leq T-1:W_{t}=w\}$, and we define $T(g_{1}(w) )$ similarly. Then the empirical analogue of (ref) can be written as,
The maximizer of (ref), $\hat{g}_1$, $\hat{g}_2$, can be obtained by backward induction, a technique widely applied in the Markov decision process (MDP) and dynamic treatment regime literature. See Section (ref) and Appendix (ref) for more discussion on the relationship between T-EWM and MDP. To derive the theoretical property of the estimator, we also define
Similarly to the derivation in the previous sections, $\widehat{\mathcal{W}}_{T:T+1}(g_1,g_2|w)- \bar{\mathcal{W}}_{T:T+1}(g_1,g_2|w)$ is a (weighted) sum of MDS. Its upper bound can be shown by the method of Section (ref). We show in Appendix (ref) the extension to multi-period welfare with continuous conditioning covariates.
How can the framework of T-EWM handle the Lucas critique? In this section, we clarify a link between T-EWM and the SVAR approach based on a three-equation New Keynesian model where a choice of a policy regime can take into account the economy's policy response.
We start with a three-equation New Keynesian model. (See, e.g., Chapter 8 of walshmonetary.) At time $t$, let $\pi_{t}$ denote inflation, $x_{t}$ the output gap, and $i_{t}$ the interest rate.
Here $v_{t}$ is the monetary policy shock (baseline target rate), which can be viewed as a policy variable that the planner can manipulate. In the process of generating the sample, it is often assumed to follow an AR(1) process $ v_{t}=\rho v_{t-1}+e_{t}$. The AR coefficient of $v_t$, $\rho$, represents a monetary policy regime. We also assume that the shocks in the Phillips curve follow an AR(1), $\varepsilon_t=\gamma \varepsilon_{t-1}+\delta_t$. Define $ d_{t}=\left(
\right) , F=\left(
\right) $, and a vector of noises $\eta_t=\left(
\right) $. Then, the process for $d_t$ can be written as $d_t=Fd_{t-1}+\eta_t$.
Define the outcome variables of the system (ref) to be $\tilde{Y}_{t}=\left(
\right)$. At the end of time $T-1$, the goal of the planner is to minimize (or maximize) the expectation of some function of $\tilde{Y}_{T}$. For example, an objective function is the welfare cost that penalizes the time $T$ output gap and inflation, $Y_{T}= |x_{T}|^2+|\pi_{T}-\pi_0|^2$, where $\pi_0$ is the inflation target.
Appendix (ref) shows that the VAR-reduced form of the system (ref) can be expressed as:
where $M(\rho)$ is a non-random matrix defined in Appendix (ref). If the model (ref) is correctly specified, the solution to (ref) takes the Lucas critique into account since it solves for a deep parameter $\rho$. A change in $\rho$ incorporates both the direct effect of the policy regime (through $\rho$ in $d_t$ equation) and private agents' anticipation of the policy change ($\rho$ in $M(\rho)$).
Now we show how this is related to the T-EWM framework. The treatment $v_t$ in (ref) corresponds to $W_t$ in the previous sections. We can write $M(\rho)=\left(
\right) $. When $v_t$ is a binary variable (e.g., high target rate and low target rate), we can define the potential outcomes by setting $v_t = 1$ or $0$ in (\ref{eq:Yt_dt2}): $\tilde{Y}_t(1)=\left(
\right) $ and $\tilde{Y}_t(0)=\left(
\right) $. Both $\tilde{Y}_t(1)$ and $\tilde{Y}_t(0)$ depend on $\rho$, so we can write them as $\tilde{Y}_t(1;\rho)$ and $\tilde{Y}_t(0;\rho)$. Transforming $\tilde{Y}_t=(x_t,\pi_t)^{\prime}$ into $Y_t=|x_t|^2 + |\pi_t - \pi_0|^2$, we can define the potential outcomes for the welfare $(Y_t(1; \rho), Y_t(0; \rho))$, which also depend on $\rho$. The policy choice problem for $v_T \in \{0, 1\}$ can be set to minimize the expected welfare cost:
in $g(v_{T-1}) \in \{0, 1\}$, assuming that the one-time policy choice of $v_T$ does not change the value of $\rho$ governing the outcome-generating process ((ref)). Thus, one can view the framework of T-EWM in the previous sections concerns a choice of binary policy shock $v_T$, assuming the fixed deep parameter of $\rho$.
In contrast, consider the case where the policy choice of interest concerns regime $\rho$ instead of $v_T$. We can modify the T-EWM framework as follows in order to handle this case. Assume that the monetary policy shock $v_t$ is continuously distributed and let $f_{v_{t}|\mathcal{F}_{t-1}}( v;\rho)$ be the conditional density of $v_t$ given the filtration $\mathcal{F}_{t-1}$ evaluated at $v$ and $\rho$. In the outcome process of $t=T-1$ and earlier, the distribution $ f_{v_{t}|\mathcal{F}_{t-1}}( v;\rho)$ at the value of $\rho$ generates the outcomes up to $T-1$. In the counterfactual policy scenario of period $t=T$ and later, the planner manipulates the distribution of $v_T$ by changing the value of $\rho$ instead of directly setting a particular value of $v_T$. For such an intervention, we specify the planner's problem as a choice of $\rho$ to minimize
This objective function differs from the welfare objective function of (ref) in the following two aspects. First, in ((ref)), the policy rule for $v_T$ is deterministic, whereas in ((ref)), the planner chooses $\rho$ to change the conditional probability density function of the treatment $v_T$, i.e., a randomized policy for $v_T$. Second, we see that in ((ref)), the policy $v_T$ affects the outcome $Y_T$ only through $v_{T}$ with $\rho$ fixed, whereas in ((ref)), the policy $\rho$ affects the outcome through both the distribution of $v_T$ and the deep policy parameter $\rho$ underlying the potential outcomes.
Despite these differences, we can pursue a T-EWM approach for a statistical policy choice of $\rho$ as far as one can construct a sample analogue of the welfare criterion of ((ref)). For instance, let $\rho_t$ be the policy regime in the sampling period $t=1, \dots, T-1$, and assume $\rho_t$ is observable or estimable.\footnote{$\rho_t$ can be estimated by the method proposed in schorfheide2005learning, assuming that monetary policy follows a nominal interest rate rule that is subject to regime shifts.} For the sake of illustration, we will use a simplified model by assuming that $\rho_t$ takes values in a finite set $\Omega$ and $v_t\in\{0,1\}$. For a given $\rho\in\Omega$, we assume that $\rho_t$ is independent of $Y_{t}(v_t;\rho)$ conditional on $\mathcal{F}_{t-1}$, and we let Assumption (ref) hold with $W_{t} = v_{t}$, such that $\mathop{\mbox{\sf E}}(\cdot|\mathcal{F}_{t-1})=\mathop{\mbox{\sf E}}(\cdot|v_{t-1})$. Then, for a policy function $g_{\rho}:\{0,1\}\to \Omega$ and $v\in\{0,1\}$, we can define the empirical welfare estimating(ref) as
where $T(v):=\# \{ v_{t-1}=v\}$. We then maximize this empirical criterion with respect to $v=v_{T-1}$ to estimate an optimal policy regime.
It is worth noting that as long as $\rho_t$ is observable (or estimable), the empirical analogue of the welfare criterion is free from functional form restrictions of $Y_t(v; \rho)$ or a distributional restriction of $f_{v_t|\mathcal{F}_{t-1}}(v; \rho)$. On the other hand, whether the empirical analogue can estimate the welfare criterion ((ref)) well or not relies on whether the agents in the economy have correct knowledge of $\rho_t$ and behave in response to the shift of $\rho_t$. We leave for future research a rigorous characterization of the learnability of an optimal policy regime along this T-EWM approach and its implementability in practice.
In this section, we discuss T-EWM's relation to the literature on treatment and optimal policy analysis.
Markov Decision Process (MDP) is the standard framework for reinforcement learning algorithms commonly applied to decision-making problems in dynamic environments. See, e.g., kallenberg2016markov for a comprehensive introduction. We can relate the current T-EWM model to a special case of MDP with a finite horizon. The conditioning variables $X_{t-1}$ correspond to the Markov state at time $t$, and the welfare outcome $Y_{t}$ corresponds to the reward at $t$. Before the planner intervenes at time $T$, the Markov state transitions follow the data-generating process in which the transitions of the policies are prescribed by the propensity score. After the planner intervenes, the transition of policies is governed by a deterministic rule described by the (estimated) optimal policy function (ref). As we show in Appendix (ref), the population conditional welfare of T-EMW with a finite-horizon welfare target can be viewed as the value function of a finite-horizon MDP with a non-stationary solution. See Chapter 2 of kallenberg2016markov for more details about finite-horizon MDPs.
The reinforcement learning (RL) literature is vast and provides a rich toolbox to solve MDPs. There are, however, several main differences between T-EWM and RL approaches. First, T-EWM builds on the framework of potential outcome time series which allows for more flexibility in dynamic causal dependence than the reward-generating processes considered in the standard MDPs. Second, the framework of T-EWM can easily accommodate a nonstationary environment in both causal effects and data-generating processes without requiring modeling them explicitly. The third difference lies in the estimation algorithms. T-EWM obtains a policy by optimizing the empirical welfare criterion, whereas RL typically employs iteration algorithms to maximize the value function. Fourth, T-EWM performs an optimal policy choice with available data exogenously given to the planner. The RL literature in contrast studies the decision maker's joint strategies of sampling data and learning policies. That is, the T-EWM method can be regarded as an estimation approach for the optimal policy in an offline and off-policy RL problem with a finite horizon, building on the potential outcome time-series framework.
In empirical macroeconomics, a common practice is to measure the causal effects of policies using the impulse response functions (IRF) to the structural policy shocks; see sims1980macroeconomics, ramey2016macroeconomic, plagborg2016essays, stock2017twenty, among others. This approach is based on the representation that the endogenous outcome variables are expressed as a weighted sum of the structural shocks, in which the coefficients of the structural shocks correspond to IRFs. With our notation, it can be expressed as
where $W_t$ is the policy shock and $\epsilon_t$ is a non-policy structural shock that the planner cannot control. Setting the value of structural shocks exogenously to $w_{-\infty:t}$ without intervening in the joint distribution of the other shocks $\epsilon_{-\infty:t}$ defines the potential outcome $Y_t(w_{-\infty:t})$. The coefficients $(\theta_h, \phi_h)$, $h = 0, 1, 2, \dots,$ are IRFs of the $h$-period ahead outcome variable to the unit changes in policy and non-policy shocks, respectively.
Our approach of T-EWM based on the potential outcome time series differs from the approach of causal IRF analysis in several aspects. First, we assume that the policy shocks of interest $\{W_t: t=0, \dots, T-1 \}$ are directly observable in the available time-series data, while the common framework of IRF analysis such as SVAR considers settings where the policy shocks are not observable and need to be identified through identification of a linear simultaneous equation system. Second, our framework imposes little restrictions on the functional form of how the current and past shocks affect the outcome, while the IRF analysis based on ((ref)) assumes that the structural equation for $Y_t$ is linear in the structural shocks and additive over the horizons. Third, our framework allows unrestricted heterogeneity (i.e., nonstationarity) of the causal effects, whereas the shock-based causal models ((ref)) assume either stationary IRFs that depend only on horizon $h$ or nonstationary IRFs with explicit modeling of how they evolve over time.\footnote{See bojinov2019time and rambachan2021when for comparative discussions between the IRFs and the average treatment effects with potential outcome time series.} Fourth, the existing IRF analysis focuses mainly on estimating and inferring the IRFs, whereas the literature has not formulated how to perform statistical policy choice based on the estimated IRFs. Our T-EWM approach, in contrast, explicitly formulates the policy choice problem using potential outcome time series and analyzes the welfare performance of the statistical policy choice.
To this point, we have treated the propensity score function as known, but this is infeasible in many applications. Here we consider the case where the propensity score at each time $t$, $e_{t}(\cdot)$, is unknown. Estimation can be either parametric or non-parametric. Let $\hat{e}_{t}(\cdot)$ denote the estimator of the propensity score function, and $\hat{G}_{e}$ denote the optimal policy obtained using $\hat{e}_t$.
In this subsection, we adapt Theorem 2.5 of kitagawa2018should to our setting and obtain a new regret bound with estimated propensity scores. We show that, with estimated propensity scores, the convergence rate is determined by the slower one of the rate of convergence of $\hat{e}_{t}(\cdot)$ and the rate of convergence of the estimated welfare loss (given in Theorem (ref)).
To ensure that $\hat{e}_t(W_{t-1})$ is a valid estimator of ${e}_t(W_{t-1})$ in the sense of satisfying ((ref)), we need to restrict the dynamics of $W_t$. It is, however, important to note that the condition of ((ref)) does not require stationarity of the outcome variable $Y_t(.)$.
A proof is presented in Appendix (ref). This theorem shows that if the propensity score is estimated with sufficient accuracy (a rate of $\phi_{T}^{-1} \lesssim \sqrt{T}^{-1}$), we obtain a similar regret bound to the previous sections. It is not surprising to see that the rate is affected by the estimation accuracy of the propensity score, and it is the maximum of $\frac{1}{\sqrt{T-1}}$ and $\phi_{T}^{-1}$.
In this subsection, we briefly review various methods of estimating propensity score functions. The propensity score function $e_{t}(\cdot)$ can be estimated parametrically or nonparametrically. An example of a parametric estimator is the (ordered) probit model, which is employed by hamilton2002model, scotti2018bivariate, and angrist2018semiparametric. Under Assumption (ref), $e_{t}$ can be expressed as a function of $X_{t-1}$. Then, the structure of the propensity score can be given by a probit model,
A more complicated structure, such as the dynamic probit model (eichengreen1985bank; davutyan1995operations) can also be employed.
We can also use a nonparametric estimator to estimate $e_{t}(\cdot)$. For example, frolich2006non and park2017nonparametric extend the local polynomial regression of fan1995data to a dynamic setting. Their methods can be employed here. For simplicity, we assume that the propensity score function is invariant across times, i.e., $e_t(\cdot)=e(\cdot)$ for any $t$, and $e(\cdot)$ is continuous, and we set $\mathcal{X}\subset \mathbb{R}^1$. For a local polynomial of order $p=1$ and any $x\in\mathcal{X}$, a local likelihood logit model can be specified as $$\text{log}\left[\frac{e(x)}{1-e(x)}\right]=\alpha_x,$$ for a local parameter $\alpha_x$. By the continuity of the propensity score function $e(\cdot)$, for some $X_{t-1}$ close to $x$, we can find a local parameter $\beta_{x}$, such that $\text{log}\left[\frac{e(X_{t-1})}{1-e(X_{t-1})}\right]\approx\alpha_{x}+\beta_{x}(X_{t-1}-x)$. The estimated propensity score evaluated at $x$, $\hat{e}(x)$, can be obtained by solving
where $K(\cdot)$ is a kernel function, and $h$ is the bandwidth. Then, we have $\hat{e}(x)=\frac{\exp(\hat{\alpha}_{x})}{1+\exp(\hat{\alpha}_{x})}.$
In the previous pandemic of COVID-19, policymakers around the world faced the problem of making effective policies to respond to the Covid-19 pandemic. In this section, we illustrate the usage of T-EWM with an application to choosing the stringency of restrictions imposed in the United States during the pandemic. We let the treatment $W_{t}$ be a binary indicator for whether the government relaxes restrictions at week $t$. $W_{t}=1$ means the stringency of restrictions is maintained or increased from time $t-1$. $W_{t}=0$ means restrictions are relaxed. The stringency of restrictions is measured by the Oxford Stringency Index, which is described in Section (ref). We assume that a change in the stringency of restrictions at time $t$ will have a lagged effect on deaths, and set the outcome variable $Y_{t}$ to be $-1\times$two-week-ahead deaths. The time $t$ information set is,
The variables in $X_{t}$ are chosen to include the most important factors considered by policy makers when deciding the stringency of restriction. The inclusion of the economic conditions reflects policymakers' concerns over the economic effect of restrictions. Finally, the propensity score is estimated as a logit model with a linear index,
The dataset consists of weekly data for the United States. It runs from April 2020 to January 2022 and contains 92 observations. Data on cases, deaths, and vaccine coverage are downloaded from the website of the Centers for Disease Control and Prevention (CDC). These series are plotted in Figures (ref) and (ref) in Appendix (ref). We use the Oxford Stringency Index as a measure of the stringency of restrictions. This index is taken from the Oxford COVID-19 Government Response Tracker (OxCGRT), which tracks policy interventions and constructs a suite of composite indices that measure governments' responses.\footnote{The index is calculated based on a dataset that is updated by a professional team of over one hundred students, alumni, staff, and project partners. It is a composite index covering different types of restriction, such as school and workplace closures, restrictions on the size of gatherings, internal and international movements, facial covering policies etc. See hale2020variation for more details.} The left panel of Figure (ref) in Appendix (ref) plots the time series of the stringency index.
The economic impact of restrictions is a crucial factor that every policymaker has to consider. To measure economic conditions, we use the Lewis-Mertens-Stock weekly economic index (WEI). \footnote{The WEI is an index of ten indicators of real economic activity. It represents the common component of series covering consumer behavior, the labor market, and production. See lewis2021measuring for more details. }The right panel of Figure (ref) in Appendix (ref) plots the WEI. In general, none of these time series seems stationary over the sample period. They exhibit both seasonal fluctuations and changes with respect to different stages of the pandemic.
Before we proceed with the analysis, we discuss the validity of some of the T-EWM assumptions in this context. The Markov properties (Assumption (ref)) require that (i) the potential outcome at time $t$ depends only on the time $t$ and time $t-1$ treatments; (ii) conditional on $X_{t-1}$, the potential outcomes and treatment at time $t$ are not affected by the path $X_{0:t-2}$. (i) can be justified here as the treatment is the direction of change in the stringency of restrictions. While the level before $t-1$ may affect the outcome at $t$, its directional change may not have such a long-existing impact. (ii) requires that conditional on current cases and deaths, cases and deaths one week ago (as $X_{t}$ includes the first difference of cases and deaths, it includes the level of cases and deaths from one week ago), current economic conditions, the stringency of restrictions, vaccine coverage, and deaths in two weeks will be independent of lags of these variables if the lag is greater than one. For this assumption to hold, it must be that current infections are unrelated to deaths in more than three weeks. In general, it is unclear whether this is true, although there is some support in the literature. For example, based on data collected during the early stage of the pandemic in China, verity2020estimates calculate that the posterior mean time from infection to death is 17.8 days, with a 95%-confidence interval of {[}16.9, 19.2{]}.
Strict overlap (Assumption (ref)) requires that, for all $x\in\mathcal{X}$, the propensity score is strictly larger than 0 and smaller than 1. Figure (ref) in Appendix (ref) shows the histogram of estimated propensity scores. The strict overlap assumption can be verified by visual inspection: the smallest (estimated) propensity score is larger than 0.3, and the largest is smaller than 0.9. Sequential unconfoundedness (Assumption (ref)) requires that conditional on $X_{t-1}$, treatment assignment at time $t$ is quasi-random. This assumption cannot be tested in general. However, in the past two years, policymakers have had very limited knowledge of Covid-19 beyond the observable data. After controlling for the observables in ((ref)), it seems reasonable that the remaining random factors in two-week-ahead deaths and changes in the stringency of restrictions are independent.
In this subsection, we summarize the estimation results and discuss the policy recommendations. We show that T-EWM leads to sensible and robust policy decisions. After observing $X_{T-1}$, we aim to maximize expected welfare, $\mathop{\mbox{\sf E}}(-1\cdot\text{deaths}_{T+1})$, over a set of quadrant policies. A class of quadrant treatment rules with $k$ policy variables, $x=(x_{1}\dots x_k)^{\prime}$, is defined as
Let $X_{T-1}^{P}$ denote a vector of variables for policy choice. This can be any subvector of $X_{T-1}$. For our first set of results, we use $X_{T-1}^{P}=\left(\text{change in deaths}_{T-1},\text{restriction stringency}_{T-1}\right).$ Figure (ref) presents the estimated treatment region. The estimated optimal decision rule states that restrictions should not be relaxed (i.e., $W_{T}=1$), if the weekly fall in deaths is below $745$, and the current level of restrictions is lower than 62.6.
We can examine the robustness of this result by expanding the set of variables for policy choice. We first add vaccine coverage: $X_{T-1}^{P}=(\text{change in deaths}_{T-1},$ $\text{restriction stringency}_{T-1},$ $\text{vaccine coverage}_{T-1})$. The optimal estimated quadrant policy is then a 3d-quadrant. Figure (ref) in Appendix (ref) shows the projection of this 3d-quadrant onto the 2d-planes of $(\text{change in deaths}_{T-1},$ $\text{restriction stringency}_{T-1})$ and $(\text{restriction stringency}_{T-1}$,\\ $\text{vaccine coverage}_{T-1}$). We further expand the set of variables for policy choice by adding the change in cases, so that $X_{T-1}^{P}=(\text{change of deaths}_{T-1},$ $\text{restriction stringency}_{T-1},$ \\$vaccine coverage_{T-1}, change in cases_{T-1})$. The optimal estimated quadrant policy in this case is a 4d-quadrant. Figure \ref{fig:Policy_k=00003D4} in Appendix \ref{app_app} shows projections of this 4d-quadrant onto the 2d-planes of $(change in deaths_{T-1}$, $restriction stringency_{T-1})$ and $(vaccine coverage_{T-1},$ $change in cases_{T-1}$).
Table (ref) summarises the estimation results of Figures (ref), (ref), and (ref). It shows that the thresholds of the T-EWM policies are insensitive to the additions of the variables that the quadrant policies depend on. As the number of variables increases, the threshold of the change in weekly deaths decreases slightly from $-745$ to $-1007.5$, while the other thresholds remain stable. During the sample period, the mean of weekly cases is 594344.3. Therefore, the threshold in the last column and the last row corresponds to a $10\%$ fall relative to the average number of cases during the sample period. In sum, T-EWM suggests that the policymaker should not relax restrictions ($W_{T}=1$) if there are no significant drops in deaths and cases, current stringency is comparatively low, and vaccine coverage is comparatively low.
Including more than four policy variables presents a significant computational challenge for the grid-search method we have used to optimize quadrant treatment rules. \footnote{When five variables are included in $X_{T-1}^{P}$, the estimated running time is 12 hours; with six variables, the estimated running time skyrockets to 4459 hours. (These estimates were made on a computer with a 12th Gen Intel(R) Core(TM) i7-1265U 2.7 GHz processor and 32GB of RAM.)} To overcome this issue, we employ decision trees to search for an optimal policy based on time-series empirical welfare.
The decision tree approach (breiman1984classification) sequentially searches for policy variables and their threshold-based splits to maximize welfare. athey2021policy and zhou2023offline study the properties and implementation of this approach in maximizing the doubly robust empirical welfare criterion with brute force searches for tree partitions. ida2022choosing implement a decision tree with a heuristic two-step optimization in the context of estimating a rebate assignment policy in an electricity market. We implement a classification tree algorithm based on hastie2009elements. See Appendix (ref) for the details of our implementation.
Figure (ref) illustrates a policy rule obtained by a T-EWM decision tree, which considers all the seven policy variables within $X_{T-1}$, i.e., $ X_{T-1}^{P}=X_{T-1}=(\text{cases}_{T-1},\,\text{deaths}_{T-1},$ $\text{change in cases}_{T-1},\,\text{change in deaths}_{T-1},\,\text{restriction stringency}_{T-1},\,\text{vaccine coverage}_{T-1},$\\ $\text{economic conditions}_{T-1}).$ Due to the relatively small sample size of 92, we only generate trees with a depth of three.
The result from the decision tree indicates that if the economy performs relatively well, policymakers should adopt a more aggressive approach in maintaining or tightening Covid restrictions. (Note that the decision given by the left node of the second level aligns with the last one in the last row of Table (ref), which considers four policy variables.) Conversely, if the economy performs relatively poorly, policymakers should refrain from further tightening Covid restrictions unless the increase in cases is significant.
To wrap up, we illustrate the use and implementation of the proposed T-EWM approach in making policy decisions. Furthermore, utilizing decision trees with the T-EWM approach allows for the inclusion of more policy variables while keeping computation time manageable. More results from the T-EWM decision trees including a specification with a higher order of the Markovian structure of $q=2$ can be found in Appendix (ref).
This article proposes T-EWM, a framework and method for choosing optimal policies based on time-series data. We characterise assumptions under which this method can learn an optimal policy. We evaluate its statistical properties by deriving non-asymptotic upper and lower bounds of the unconditional and conditional welfare. We discuss its connections to the existing literature, including the Markov decision process and impulse response analysis. We present simulation results and empirical applications to illustrate the computational feasibility and applicability of T-EWM. As a benchmark formulation, this paper mainly focuses on a one-period social welfare function as a planner's objective. Extensions of the analysis to policy choices for the middle-run and long-run cumulative social welfare are left for future research.