Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
35,459 characters · 12 sections · 7 citation commands
Two Burning Questions on COVID-19: Did shutting down the economy help? Can we (partially) reopen the economy without risking the second wave?
\paragraph{Challenge of Choosing Between Interventions.} It is clear that the COVID-19 pandemic has led to an unprecedented disruption of modern society at a global scale. What is much less clear, however, is the effect that various interventions that have been put into place have had on health and economic outcomes. For example, perhaps a 30% and 60% clampdown in mobility have similar societal health outcomes, yet vastly different implications for the number of people who cannot go to work or file for unemployment. Having a clear understanding of the trade-offs between these interventions is crucial in charting a path forward on how to open up various sectors of society. A key challenge is that policy makers do not have the luxury of actually enacting a variety of interventions and seeing which has the optimal outcome. In fact, at a societal level, this is simply infeasible. Arguably, an even bigger challenge is that the COVID-19 pandemic, and the resulting policy choices ahead of us, are unprecedented in scale. Thus, it is difficult to reliably apply lessons from previous pandemics (e.g., SARS, H1N1). This is only further exacerbated when taking into the account the vastly different economic, cultural, and societal factors that make each town/city/state/country unique.
\paragraph{Synthetic Interventions.} In this memo, we introduce Synthetic Interventions SI, a method towards solving the problem laid out above. Specifically, Synthetic Interventions provides a data-driven and statistically principled way to perform {\em what-if} scenario planning, i.e., for policy makers to understand the trade-offs between different interventions before having to actually enact them. In essence, the Synthetic Interventions method leverages information from different interventions that have already been enacted across the world and fits it to a policy maker's setting of interest -- for example, to estimate the effect of mobility-restricting interventions on the U.S., we use daily death data from countries that enforced severe mobility restrictions to create a “synthetic low-mobility U.S.” and predict the {\em counterfactual} trajectory of the U.S. if it had indeed applied a similar intervention. We highlight a few desirable attributes of this methodology.
\paragraph{COVID-19 Case-Study using Synthetic Interventions.} In the next section, we detail the use of Synthetic Interventions to evaluate the effect of various mobility restriction interventions on health outcomes. We give attention to the data requirements of the method, some important algorithmic details, and how to evaluate whether the method is producing accurate forecasts. We hope this memo serves as a template for policy makers to make statistically informed decisions on the variety of weighty policy choices ahead of them.
Recall that the aim of Synthetic Interventions is to predict the counterfactual trajectories of a region under a variety of interventions of interest. Below, we explain how policy-makers can use the Synthetic Interventions method to help guide them in choosing between interventions. For concreteness, we use the case study of applying Synthetic Interventions to understand the effect of mobility restriction policies on the number of COVID-19 related deaths at a country level, as a running example.
We begin by listing some of the key modeling decisions and data requirements for the Synthetic Interventions method to produce reliable counterfactual predictions.
\paragraph{Choosing Metric of Interest.} First, a practitioner must decide upon his or her relevant metric of interest. Some examples include the number of (i) COVID-19 related deaths, (ii) COVID-19 related cases, (iii) ICU beds utilized, and (iv) unemployment claim filings. The counterfactual predictions produced by the Synthetic Interventions method will be with respect to the particular metric chosen.
\paragraph{Choosing Interventions of Interest.} Secondly, a practitioner must define a set of interventions that are of interest to the policy maker. For example, in the case of studying the effect of mobility, the interventions could be a (i) less than $10\%$ drop in mobility, (ii) $10-40\%$ drop in mobility, and (iii) greater than $40\%$ drop in mobility compared to the national baseline. The counterfactual predictions produced by the Synthetic Interventions method will be for the particular interventions chosen.
\paragraph{Choosing Pre- and Post-Intervention Time Periods.} The third decision that needs to be made is defining the pre- and post-intervention periods, i.e., the data that is fed into the algorithm must be divided into two segments:
\paragraph{Categorizing Geographical Locations by Intervention Received.} The fourth decision that needs to be made is categorizing the various geographic locations by the interventions they enact, which are observed in the post-intervention period. In our context, this would be categorizing the countries by the average drop in mobility compared to the national baseline in the pre-intervention period.
\paragraph{Algorithm Overview.} The method has two major steps:
\paragraph{Data Input.} The data input into the Synthetic Interventions algorithm is divided into two parts:
\paragraph{Data Output.} The result of applying the Synthetic Interventions method on the data input described above would be {\em a counterfactual trajectory} of the target unit in the post-intervention time period under {\em every intervention of interest}. In other words, for each donor subgroup (which receives a unique intervention), one counterfactual trajectory for the target unit will be produced.
\paragraph{Mathematical Formulation.} Please refer to Section (ref) for a formal description of the Synthetic Interventions method.
\paragraph{Choosing Metric of Interest - Daily Death Counts.} Although there are a myriad of ways to measure a region's health and economic outcomes, some metrics such as the number of cases are considered “noisier” due to the inconsistencies of testing across regions, and others such as the number of daily unemployment filings are sparse. As a result, given its greater reliability and availability, daily death count reports are used as our outcome variable of interest. Thus, we will apply the Synthetic Interventions method to predict how different interventions affect the number of COVID-19 related deaths within a region.
\paragraph{Choosing Interventions of Interest - Daily Mobility Rate.} Each country has implemented a variety of interventions simultaneously to combat the spread of COVID-19. This makes it difficult to analyze any particular intervention (e.g., stay-at-home orders vs. schools shutting down) in isolation. However, if we consider the causal chain of events, we observe that all of these interventions are directed towards restricting how individuals move and interact, which then affects the spread and, ultimately, deaths associated with COVID-19. In light of this, we adopt mobility as our notion of intervention, and investigate how a country's drop in mobility level translates to the number of potential COVID-19 related deaths\footnote{ To resolve any potential ambiguities, we will henceforth refer to change in the level of mobility within a country as an intervention, and refer to orders such as the closing of schools or the enforcing of individuals to stay at home as a policy (which eventually translates to an intervention). }. To that end, we use Google's excellent mobility reports google_mobility to define three distinct, mutually exclusive interventions as follows:
\paragraph{Choosing Pre- and Post-Intervention Time Periods - Number of Deaths in the Country.} As previously mentioned, it is crucial to have a well-defined pre- and post-intervention period, where all regions during the pre-intervention period operate under a no intervention setting, and the effects of their enacted interventions are only observed during the post-intervention period. Towards this, an assumption we rely on is that the enactment of mobility lockdown in a country is a function of the number of deaths in the country instead of a particular calendar date. This is indeed verified in a data-driven manner using the Google mobility reports -- we see that 20 days prior to 80 deaths in a country (and any time before that), essentially no country enacted a mobility lockdown. Thus, we align the daily death rate trajectories, and correspondingly the pre- and post-intervention periods, as before and after the country has 80 deaths. In other words, we refer to the day a country hits 80 deaths as “Day 0”, and the pre- and post-intervention periods then refer to the days before and after Day 0, respectively.
\paragraph{Categorizing Countries by Intervention Received -- Average (Lagged) Mobility Score.} Studies have shown that there is a median lag of 20 days from the onset of infection to the day of death (e.g., see lag_study). Hence, a country's daily death count is a function of the region's infection levels, which are in turn affected by mobility levels, from roughly 20 days prior. Thus, to compute the effect of a mobility intervention from Day 0 onwards, we compute a country's mobility score (i.e., intervention level) from Day -20 to Day -1. Given that the mobility score in google_mobility is changing every day, we take the average mobility score of a country from Day -20 to Day -1 and then bucket it into the three intervention groups defined above\footnote{Moving forward, we aim to apply more sophisticated clustering methods such as $K$-means clustering.}.
Here, we apply the Synthetic Interventions method to produce the counterfactual predictions of the daily death counts for all countries in the Google mobility data set under the three possible interventions detailed in Section (ref).
\paragraph{Validating Synthetic Interventions.} Our aim is to validate that the method is able to produce accurate counterfactual predictions across all three interventions. To that end, for every intervention (mobility restriction level), we display the counterfactual predictions associated with two representative countries that enacted that intervention\footnote{Similar results hold generally across all countries. Please contact authors for any specific country of interest.}. In the low mobility restricting regime, we show results for the United States and the United Kingdom in Figures (ref) and (ref), respectively, in Appendix (ref). Pleasingly, we observe that the output produced by Synthetic Interventions is able to accurately recreate the observed death rates in the post-intervention period. We note that the model is learned during the pre-intervention period (Day -20 to Day -1), and the dashed lines in Days 0 - 15 are the predicted values under all possible mobility restriction levels (while solid lines are the true values).
Similarly, in the moderate and severe mobility restricting regimes, we display the results for Turkey, Sweden, India, and Romania in Figures (ref), (ref), (ref), and (ref), respectively, in Appendix (ref). Again, we observe that the counterfactual predictions produced from the Synthetic Interventions method is able to closely match the observed death rates under all different interventions, i.e., mobility restrictions. Importantly, we note that in all of these models, the Synthetic interventions built for the target country is learned in the pre-intervention period, when no intervention has yet occurred. Still, we are able to transfer the learnt model into an interventional setting, i.e., when the interventions take effect within the donor countries.
For each of the countries listed above, we display their top four donor countries (under each intervention) that most closely resemble them. These are shown in Figures (ref), (ref), (ref), (ref), (ref), and (ref) in Appendix (ref).
\paragraph{Key Takeaway from Counterfactual Predictions.} A stark conclusion that is immediately apparent from the figures above is that uniformly across all countries, there is a significant drop in the number of deaths with a even a “small” drop in mobility (i.e, a 10-40% drop in mobility compared to the national baseline). After this point, gains by further restricting mobility seem to be diminishing. An ongoing line of research is to study how these various restrictions in mobility affect economic outcomes of a country. This will ideally allow policy-makers to precisely quantify the trade-off between various mobility restricting interventions.
\paragraph{What Can Happen if Mobility Restrictions are Lifted?} Using the output of Synthetic Interventions, a natural next question that arises is estimating the peak number of daily deaths if mobility restrictions are lifted as of today (April 23rd, 2020). To do so, we input the counterfactual estimates of the Synthetic Interventions method into an exponential curve-fitting algorithm. Specifically, for each country, we fit an exponential function to its current trajectory and each intervention level that is less restrictive than its current intervention level. We display the corresponding figures in Appendix (ref).
\paragraph{Key Takeaways.} In a similar vein to the previous section, we see from the figures in Appendix (ref), even small drops in mobility can significantly decrease the peak number of daily deaths in a country. Given that the peak number of daily deaths is the key metric in understanding how strained each nation's medical system will be, we see that moderate mobility restriction measures can lead to a significant “flatenning” of the curve.
\paragraph{Setup.} We consider the setting where there are $N$ units and $D$ interventions, and we observe the outcome variable of interest (e.g., death count) associated with each unit across $T$ time periods. Throughout, we let $n, t$, and $d$ be the indices associated with unit $n$, time $t$, and intervention $d$, respectively. Further, we denote $T_0 < T$ as the intervention point (e.g., Day 0), which partitions our time horizon into two distinct segments: the {\em pre-intervention} period ($t \le T_0$), where all units are assumed to be in a common setting (e.g., in the case study above, all countries operate as usual prior to any COVID-19 related outbreaks), and a {\em post-intervention} period ($t > T_0$), where each unit may receive one of $D$ interventions (e.g., a country goes into a low/moderate/severe mobility lockdown).
\paragraph{Potential Outcome Notation.} We let $y^{(d)}_{nt}$ denote the potential outcome for unit $n$ at time $t$ under intervention $d$. Without loss of generality, we will let $d=1$ represent the absence of any intervention or {\em null}-intervention (our assumed common pre-intervention setting). Finally, we let $\mathcal{I}^{(d)}$ represent the set of units that actually receive intervention $d$ (e.g., mobility level $d$) in the post-intervention period, and $N^{(d)} = \abs{\mathcal{I}^{(d)}}$ as the number of units within partition $\mathcal{I}^{(d)}$.
\paragraph{Input.} For each unit, we have access to a time series of $T$ observations\footnote{It is not necessary that the observations are indexed by time, though it is the most common way datasets are structured. Rather, we require $T$ (pre- and post-intervention) measurements for each of the $N$ units.} according to some outcome variable of interest. We can represent the available data (i.e., the dataset that is fed into the synthetic interventions method) as an $N \times T$ matrix $\boldsymbol{Y} = [y_{nt}] \in \mathbb{R}^{N \times T}$, with $(n,t)$-th entry expressed as follows:
where $\star$ indicates a missing value and $\mathbb{1}$ denotes the indicator function. That is, during the pre-intervention period, the observed outcomes associated with all units are associated with the null-intervention. During the post-intervention, however, we observe the outcome variable associated with each unit based on the intervention it received.
\paragraph{Output.} The output of the Synthetic Interventions algorithm is simply an augmentation of the observed data $\boldsymbol{Y}$ under all possible interventions. That is, the Synthetic Interventions algorithm predicts the counterfactual trajectory of the outcome variable under all $D$ interventions for every unit. Hence, for each unit $n$, synthetic interventions produces $\widehat{y}^{(d)}_{nt}$ for all $(t, d)$, where $t > T_0$ and $d \in [D]$. If the counterfactual predictions are accurate and reliable, then a practitioner now has insights as to how each intervention affects every unit across time, and can reason about the trade-offs between interventions. Importantly, we note that these trajectories are unit specific, i.e., the counterfactual trajectories produced will be unique to each unit.
For notational simplicity (and without loss of generality), we temporarily consider unit $n=1$ as our {\em target} unit of interest, which operates under intervention $d^*$ in the post-intervention period; we will refer to the remaining units as our {\em donor} units. In what follows, we describe how to predict the counterfactual outcome variables for our target unit if it was operating under a different intervention $d \neq d^*$ instead. The same procedure can then be applied towards all remaining units.
For convenience, we will denote $\boldsymbol{Y}^{(d, \text{pre})} = [y^{(1)}_{nt}]_{n \in \mathcal{I}^{(d)}, t \le T_0}$ and $\boldsymbol{Y}^{(d, \text{post})} = [y^{(d)}_{nt}]_{n \in \mathcal{I}^{(d)}, t > T_0}$ as the pre- and post-intervention matrices, respectively, associated with the donor units who received intervention $d$. Let their singular value decompositions (SVDs) be denoted as:
Below we give intuition for the two major steps of the Synthetic Interventions algorithm (described above).
\paragraph{Pre-Intervention - Synthetic Target Unit Construction.} In order to predict the counterfactual outcome variables for our target unit under intervention $d$, we first build a synthetic version of our target unit using the donor units within partition $\mathcal{I}^{(d)}$. That is, we find the set of weights, defined by $\widehat{\lambda}^{(d)}$ in (ref), that best approximates the outcome variables of our target unit in the absence of any intervention. Crucially, to ensure that we are not over-fitting the data beyond the inherent model complexity in the target unit and donor trajectories, the Synthetic Interventions method performs Singular Value Thresholding (SVT) (equivalently Principal Component Analysis) on $\boldsymbol{Y}^{(d, \text{pre})}$ and $\boldsymbol{Y}^{(d, \text{post})}$ before learning the linear weights $\widehat{\lambda}^{(d)}$, as seen in (ref). In other words, instead of simply performing Ordinary Least Squares (OLS) regression, we perform Principal Component Regression (PCR), which learns a model in the reduced subspace spanned by the columns of $\boldsymbol{\tilde{Y}}^{(d, \text{pre})}$, to ensure our model is appropriately de-noised and regularized. For more motivation on why we use PCR, please refer to robust_pcr.
\paragraph{Post-Intervention - Counterfactual Prediction.} Once the model for intervention $d$ is learned, we re-scale the observed outcome variables associated with our donor units in $\mathcal{I}^{(d)}$ during the post-intervention period by the set of weights $\widehat{\lambda}^{(d)}$ according to (ref). Effectively, (ref) performs a synthetic intervention $d$ for our target unit and provides the corresponding counterfactual outcome variables under such a setting. The key assumption that allows one to make valid counterfactual estimates using the Synthetic Interventions algorithm is that the relationship between units is invariant across interventions. Although this assumption may seem “unnatural”, it is pleasingly implied (and justified) by a tensor factor model imposed on the data, which can be thought of as a natural extension to the matrix factor model generally assumed in panel data in econometrics (and, in particular, synthetic control methods). Mathematical details on this are beyond the scope of this memo, and an interested reader should feel free to contact the authors. Lastly, just as in the pre-intervention phase, we perform SVT (i.e., PCA) on the donor post-intervention data in (ref) for de-noising and regularization purposes.
We acknowledge fruitful discussions with Rahul Panicker of Wadhwani AI in helping us focus on mobility as the lens to take with respect to interventions. We also thank Alberto Abadie of MIT Economics for helpful discussions.