Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
60,016 characters · 15 sections · 29 citation commands
Incentive-Aware Synthetic Control: Accurate Counterfactual Estimation via Incentivized Exploration
A ubiquitous task in the internet economy is to estimate counterfactual outcomes for a group of units (e.g. people, products, geographic regions) under different interventions (e.g. marketing campaigns, discount levels, legal regulations) over time. Such multi-dimensional data are often referred to as panel data, where the different units may be thought of as rows of a matrix and the time-steps as columns. A prominent framework for counterfactual inference using panel data is that of synthetic control methods (SCMs) abadie2003economic, abadie2010synthetic, which aim to estimate the counterfactual outcome under control, i.e. no treatment, for units that were treated.\looseness-1
In the SCM framework\footnote{See (ref) for a more in depth description of the SCM framework we consider.}, there is a pre-intervention time period, during which all units are under control (i.e. under no intervention). After the pre-intervention period there is a post-intervention time period, where each unit is given exactly one intervention from a set of possible interventions (which includes control).
The goal of SCMs is to estimate the counterfactual outcomes under control for a treated unit during the post-intervention period. To do this, SCMs broadly follow two steps:
While SCMs were originally applied to public policy domains (e.g. abadie2010synthetic, freire2018evaluating), they are now often used to estimate counterfactuals in various internet domains, ranging from surge pricing for ride-sharing applications uber.com_2019 to the effectiveness of advertisements across different geographic regions EBay_Partner_Network_Help_Center_2022.
From the description of SCMs laid out above, it is clear that for the counterfactual estimate to be valid, the treated unit's potential outcomes under control should be linearly expressible by the observed outcomes of the donor units.\footnote{See (ref) for a mathematical justification of this fact.} To provide statistical guarantees on the performance of SCMs, such intuition has traditionally been made formal by making an overlap assumption on the relationship between the outcomes of the donor units and the test unit; for instance of the following form:\looseness-1
More broadly, previous works that provide theoretical guarantees for SCMs assume there exists some underlying mapping $\omega^{(i,d)}$ (e.g. linear or convex) through which the outcomes of a treated unit (unit $i$) may be expressed by the outcomes of the $N_d$ donor units. Since such a condition appears to be necessary in order to make valid counterfactual inference, assumptions of this nature are ubiquitous when proving statistical guarantees about SCMs (e.g. abadie2010synthetic, amjad2018robust, agarwal2020synthetic, agarwal2023adaptive).\footnote{We discuss the role of the (ref) and related conditions in the literature in (ref).} \looseness-1
However, despite their ubiquity, such overlap assumptions may not hold in all domains in which one would like to apply SCMs. For example, consider a streaming service with two service plans (i.e. interventions): a yearly subscription (the treatment) and a pay-as-you-go model (the control). Suppose that all streamers (the units in this example) are initially signed up for the pay-as-you-go model, and may choose to switch to the subscription model after a trial period (the pre-intervention period). Furthermore, suppose that the streaming service wants to determine the effectiveness of its subscription program on user engagement (the unit outcome of interest) over the next year (the post-intervention period).
Under this setting, the subpopulation of streamers who self-select the subscription plan is most likely those who believe they will consume large amounts of content on the platform. In contrast, those who pay-as-they-go most likely believe they will consume less content. The two subpopulations may have very different experiences under the two business plans (due to their differing viewing habits), leading to two different sets of potential outcomes that may not overlap, i.e., are not linearly expressible in terms of each other. This could make it difficult to draw conclusions about the counterfactual user engagement levels of one subpopulation using the realized engagement levels of the other.\looseness-1
In this work, our goal is to leverage tools from information design to incentivize the exploration of different treatments by non-overlapping unit subpopulations in order to obtain valid counterfactual estimates using SCMs. Specifically, we adopt tools and techniques from the literature on incentivizing exploration in multi-armed bandits (e.g. mansour2019bayesian, slivkins2021exploration) to show how the principal---the platform running the SCM---can leverage knowledge gained from previous interactions with similar units to persuade the current unit to take an intervention they would not normally take. The principal achieves this by sending a signal or recommendation to the unit with information about which intervention is best for them. The principal's recommendation policy is designed to be incentive-compatible, i.e. it is in the unit's best interests to follow the intervention recommended to them by the policy. In our streaming service example, incentive-compatible signalling may correspond to recommending a service plan for each user based on their usage of the platform in a way that guarantees that units are better off in expectation when purchasing the recommended service plan. Our procedure ensures that the (ref) becomes satisfied over time, which enables the principal to do valid counterfactual inference using off-the-shelf SCMs after they have interacted with sufficiently many units.\looseness-1
{\bf Overview of paper.} We introduce an online learning model where the principal observes panel data from a population of units with differing preferences. We show that the (ref) is indeed a necessary condition to obtain valid counterfactual estimates in our setting. To achieve overlap, we introduce a recommender system for incentivizing exploration when there are two interventions: treatment and control. At a high level, we adapt the “hidden exploration” paradigm from incentivized exploration in bandits to the panel data setting: First, randomly divide units into “exploit” units and “explore” units. For all exploit units, recommend the intervention that maximizes their estimated expected utility, given the data seen so far. For every explore unit, recommend the intervention for which the principal would like the (ref) to be satisfied. Units are not aware of their exploit/explore designation by the algorithm, and the explore probability is chosen to be low enough such that the units have an incentive to follow the principal's recommendations. After recommendations have been made to sufficiently-many units, we show that with high probability, the (ref) is satisfied for all units and either intervention of our choosing. This enables the use of existing SCMs to obtain finite sample guarantees for counterfactual estimation for all units under control after running (ref).
We extend our methods to the setting of synthetic interventions agarwal2020synthetic, in which the principal may wish to estimate counterfactual outcomes under different treatments in addition to control (Appendix (ref)). We also provide two hypothesis tests for checking whether the (ref) holds for a given treated unit and set of donor units. Finally, we empirically evaluate the performance of our recommender system. We find that it enables consistent counterfactual outcome estimation when the (ref) does not hold a priori while existing SCMs alone do not.\looseness-1
{\bf Causal inference and synthetic control methods.} Popular methods for counterfactual inference using panel data include SCMs abadie2003economic, abadie2010synthetic, difference-in-differences harmless_econometrics, ashenfelter1984using, bertrand2004much, and clustering-based methods zhang2019quantile, dwivedi2022counterfactual. Within the literature on synthetic control, our work builds off of the line of work on robust synthetic control amjad2018robust, amjad2019mrsc, agarwal2020model, agarwal2020synthetic, agarwal2023adaptive, which assumes outcomes are generated via a latent factor model (e.g. chamberlain, liang_zeger, arellano) and leverages principal component regression (PCR) jolliffe1982note, massy1965principal, agarwal2019robustness, agarwal2020model to estimate unit counterfactual outcomes. Our work falls in the small-but-growing line of work at the intersection of SCMs and online learning chen2023synthetic, farias2022synthetically, agarwal2023adaptive, although we are the first to consider unit incentives in this setting. Particularly relevant to our work is the model used by agarwal2023adaptive, which extends the finite sample guarantees from PCR in the panel data setting to online settings. While harris2022strategyproof also consider incentives in SCMs (albeit in an offline setting), they only allow for a principal who can assign interventions to units (e.g. can force compliance). As a result, the strategizing they consider is that of units who modify their pre-intervention outcomes in order to be assigned a more desirable intervention. In contrast, we consider a principal who cannot assign interventions to units but instead must persuade units to take different interventions by providing them with incentive-compatible recommendations.\looseness-1
{\bf Unit overlap and similar conditions.} Our notion of unit overlap comes directly from the literature on robust synthetic control (see the previous paragraph for references). However, it is important to note that analogous assumptions appear more broadly in the literature on synthetic control. For example, abadie2010synthetic requires that the treated unit’s pre-treatment outcomes and covariates lie in the convex hull of the donor units pool. Beyond synthetic control, similar assumptions to the (ref) are also prevalent in the literature on other matching-based estimators such as difference-in-differences donald2007inference or clustering-based methods zhang2019quantile. Finally, it is worth noting that in causal inference, the term “overlap” is also commonly used to refer to propensity score overlap, which assumes that the probability a unit receives treatment is bounded in $(0, 1)$. This is in contrast to the unit outcome/latent factor overlap that we consider.
{\bf Incentivized exploration.} Our work draws on techniques from the growing literature on incentivizing exploration (IE) kremer2013wisdom, mansour2019bayesian, mansour2021bayesian, sellke2022price, immorlica2023incentivizing, sellke2023incentivizing, HuNSW22, NgoSS021. In IE, a principal interacts with a sequence of myopic agents over time. Each agent takes an action, and the principal can observe the outcome of each action. While each individual agent would prefer to take the action which appears to be utility-maximizing (i.e. to exploit), the principal would like to incentivize agents to explore different actions in order to benefit the population in aggregate. Motivation for IE includes online rating platforms and recommendation systems which rely on information collected by users to provide informed recommendations about various products and services.
{\bf Notation.} Subscripts are used to index the unit and time-step, while superscripts are reserved for interventions. We use $i$ to index units, $t$ to index time-steps, and $d$ to index interventions. For $x \in \mathbb{N}$, we use the shorthand $[\![x]\!] := \{1, 2, \ldots, x\}$ and $[\![x]\!]_0 := \{0, 1, \ldots, x-1\}$. $\mathbf{y}[i]$ denotes the $i$-th component of vector $\mathbf{y}$, where indexing starts at 1. We sometimes use the shorthand $T_1 := T - T_0$ for $T, T_0 \in \mathbb{N}_{>0}$ such that $T > T_0$. $\Delta(\mathcal{X})$ denotes the set of possible probability distributions over $\mathcal{X}$. $\mathbf{0}_d$ is shorthand for the vector $[0, 0, \ldots, 0] \in \mathbb{R}^d$. Finally, we use the notation $a \wedge b := \min\{a,b\}$ and $a \vee b := \max\{a,b\}$.\looseness-1
We consider an online setting in which the principal interacts with a sequence of $n$ units one-by-one for $T$ time steps each. As is standard in the literature on synthetic control, we posit that there is a pre-intervention period of $T_0$ time-steps, for which each unit $i \in [\![n]\!]$ is under the same intervention, i.e., under control. After the pre-intervention period, the principal either recommends the treatment to the unit or suggests that they remain under control (possibly using knowledge of the outcomes for units $j < i$). We denote the principal's recommended intervention to unit $i$ by $\widehat{d}_i \in \{0, 1\}$, where $1$ denotes the treatment and $0$ the control. After receiving the recommendation, unit $i$ chooses an intervention $d_i \in \{0,1\}$ and remains under intervention $d_i$ for the remaining $T-T_0$ time-steps.\footnote{We envision a scenario where both interventions are available to units and they can choose between them.} We use $\mathbf{y}_{i,pre} := [y_{i,1}^{(0)}, \ldots, y_{i,T_0}^{(0)}]^\top \in \mathbb{R}^{T_0}$ to denote unit $i$'s pre-treatment outcomes under control, and $\pv{\mathbf{y}}{d}_{i,post} := [y_{i,T_0+1}^{(d)}, \ldots, y_{i,T}^{(d)}]^\top \in \mathbb{R}^{T-T_0}$ to refer to unit $i$'s post-intervention potential outcomes under intervention $d$. We denote the set of possible pre-treatment outcomes by $\mathcal{Y}_{pre}$. See (ref) for a summary of our setting.
In order to impose structure on how unit outcomes are related for different units, time-steps, and interventions, we assume that outcomes are generated via the following latent factor model, a popular assumption in the literature on synthetic control (see references in (ref)).
While we assume the existence of such structure in the data, we do not assume that the principal knows or gets to observe $\mathbf{u}_t^{(d)}$, $\mathbf{v}_i$, or $\varepsilon_{i,t}^{(d)}$. In the literature on synthetic control, the goal of the principal is to estimate unit-specific counterfactual outcomes under different interventions. Specifically, our target causal parameter is the (counterfactual) average expected post-intervention outcome.\footnote{Our results hold in the more general setting where the target causal parameter is any linear function of the vector of expected post-intervention outcomes. We focus on the average as it is a common quantity of interest.}\looseness-1
The data generated by the interaction between the principal and a unit $i$ may be characterized by the tuple $(\mathbf{y}_{i, pre}, \widehat{d}_i, d_i, \pv{\mathbf{y}}{d_i}_{i, post})$, where $\mathbf{y}_{i,pre}$ are unit $i$'s pre-treatment outcomes, $\widehat{d}_i$ is the intervention recommended to unit $i$, $d_i$ is the intervention taken by unit $i$, and $\pv{\mathbf{y}}{d_i}_{i, post}$ are unit $i$'s post-intervention outcomes under intervention $d_i$.
We refer to the way the principal assigns interventions to units as a recommendation policy.
We assume that before the first unit arrives, the principal commits to a method for computing the sequence of recommendation policies $\{\pi_i\}_{i=1}^n$ which is fully known to all units. Whenever $\pi_i$ is clear from the context, we use the shorthand $\widehat{d}_i = \pi_i(H_i, \mathbf{y}_{i,pre})$ to denote the recommendation of policy $\pi_i$ to unit $i$.
In addition to having a corresponding latent factor, each unit has an associated belief over the effectiveness of each intervention. This is made formal through the following definitions.
In other words, a unit's type is the intervention they would take according to their prior without any additional information about the effectiveness of each intervention. We consider the setting in which the possible latent factors associated with each type lie in mutually orthogonal subspaces (i.e. the (ref) is not satisfied). See (ref) for a simple theoretical example of such a data-generating process and (ref) for an empirical example. If the principal could get the (ref) to be satisfied, they could use existing synthetic control methods off-the-shelf to obtain valid finite sample guarantees.
As is standard in the literature on IE, we assume that units are Bayesian-rational and each unit knows its place in the sequence of $n$ units (i.e., their index $i \in [\![n]\!]$). Thus given recommendation $\widehat d_i$, unit $i$ selects their intervention $d_i$ such that $d_i \in \arg \max_{d \in \{0,1\}} \; \mathbb E_{\mathcal{P}_{\mathbf{v}_i}}[\bar{y}_{i, post}^{(d)} | \widehat d_i]$, i.e. they select the intervention $d_i$ which maximizes their utility in expectation over their prior $\mathcal{P}_{\mathbf{v}_i}$, conditioned on receiving recommendation $\widehat d_i$ from the principal.
It is worth clarifying the behavioral interpretation of (ref). Formally, (ref) assumes that units are Bayesian-rational, understand the recommendation algorithm, and correctly condition on the information conveyed by receiving a particular recommendation. This assumption provides a clean and standard benchmark that ensures recommendations are followed and allows us to focus on the statistical question of inducing overlap. At the same time, we do not view this assumption as requiring literal Bayesian reasoning by agents in practice. In many real-world systems, users may instead follow recommendations with high but imperfect probability, rely on coarse beliefs about the system’s reliability, or exhibit heterogeneous levels of sophistication. To this end, we explore a setting in which units follow recommendations imperfectly in (ref), and we find that our recommendation schemes are empirically robust to unit non-compliance.
Suppose that the principal would like to estimate counterfactuals for some unit under intervention $d$. We begin by showing that the (ref) must be satisfied for this unit under intervention $d$ in order for any algorithm to obtain consistent counterfactual estimates under the setting described in (ref).
The proof of (ref) proceeds by constructing a family of problem instances under which the (ref) is not satisfied, before showing that no algorithm can consistently estimate $\mathbb E[\Bar{y}_{n^{(0)}+1,post}^{(0)}]$ on all instances in the family.
To infer the unit outcomes in the post-intervention period from outcomes in the pre-intervention period, we require the following linear span inclusion condition to hold on the latent factors associated with the pre- and post-intervention time-steps:
Much like the (ref), (ref) is ubiquitous in the literature on robust synthetic control (see, e.g. agarwal2020synthetic, agarwal2023adaptive, harris2022strategyproof). In essence, (ref) requires that the pre-intervention period be “sufficiently informative” about the post-intervention period. Indeed, (ref) may be thought of as the “transpose” of the (ref), where the “overlap” condition is required to hold on the time-steps instead of on the units.\footnote{Note that (ref) is not a substitute for the (ref) by (ref).} To gain intuition for why such an assumption is necessary, consider the limiting case in which $\mathbf{u}_t^{(0)} = \mathbf{0}_r$ for all $t \leq T_0$. Under such a setting, all expected unit outcomes in the pre-intervention time period will be $0$, regardless of the underlying unit latent factors, which makes it impossible for the principal to infer anything non-trivial about a unit's post-intervention outcomes from looking at their pre-intervention outcomes alone.
\paragraph{Vertical vs. horizontal regression.} The following proposition is an important structural result for our main algorithm, as it will allow us to transform the problem of estimating unit counterfactual outcomes from a regression over donor units (vertical regression) to a regression over pre-intervention outcomes (horizontal regression). While (ref) is known in the literature on learning from panel data ({e.g., \xspace} agarwal2023adaptive, harris2022strategyproof), we state it here for completeness.
Given historical data of the form $\{\mathbf{y}_{j,pre}, \mathbf{y}_{j,post}^{(d_j)}\}_{j=1}^{i-1}$, we can estimate $\boldsymbol{\theta}^{(d)}$ as $\widehat{\boldsymbol{\theta}}^{(d)}_{i}$ using some estimation procedure, which in turn allows us to estimate $\mathbb E[\bar{y}_{i, post}^{(d)}]$ as $\widehat \mathbb E[\bar{y}_{i,post}^{(d)}] := \frac{1}{T_1} \langle \widehat{\boldsymbol{\theta}}^{(d)}_i, y_{i,pre} \rangle$ for $d \in \{0, 1\}$. While this is not a synthetic control method in the traditional sense, as it does not construct a “synthetic control” trajectory from the trajectories of the donor units, it turns out that the point estimate it produces (and assumptions it requires) are equivalent to that of a vertical regression-based SCM in many settings (including ours). See shen2022same for more background on the similarities and differences between using horizontal regression-based approaches (like ours) and vertical regression-based approaches (like traditional SCMs) in panel data settings. We choose to use a horizontal regression-based approach because it is easier to work with in our setting, where units arrive sequentially (recall (ref)). Importantly, we would still need the (ref) to be satisfied for intervention $d$ if we used a vertical regression-based method.\looseness-1
For the reader familiar with bandit algorithms, it may be useful to draw the following connection between our setting and that of linear contextual bandits. In particular, it is sometimes helpful to think of $\mathbb E[y_{i, pre}]$ as the “context” of unit $i$, and $\mathbb E[\bar{y}_{i, post}^{(d)}]$ as the principal's expected reward of assigning intervention $d$ given $\mathbb E[y_{i, pre}]$. Since we observe $y_{i, pre}$ instead of $\mathbb E[y_{i, pre}]$ we cannot apply linear contextual bandit algorithms to our panel data setting out-of-the-box. However as we will show in (ref), one can combine ideas from incentivizing exploration in contextual bandits with principled ways of handling the noise in the pre-intervention outcomes to incentivize exploration in panel data settings.
In this section, we turn our focus to incentivizing type $1$ units (i.e. units who a priori prefer the treatment) to remain under control in the post-intervention period. The methods we present may also be applied to incentivize type $0$ units (i.e. units who a priori prefer the control) to take the treatment. We focus on incentivizing control amongst type $1$ units in order to be in line with the literature on synthetic control, which aims to estimate counterfactual unit outcomes under control.
The goal of the principal is to design a sequence of recommendation policies that convinces enough units of type $1$ to select the control in the post-treatment period, such that the (ref) is satisfied for both interventions for all units of type $1$. At a high level, the principal is able to incentivize units to explore because they have more information than any one particular unit (as they view the realized history $H_i$ before making a recommendation), and so they can selectively reveal information in order to persuade units to take interventions which they would not normally take. Our algorithm ((ref)) is inspired by the “detail free” algorithm for incentivizing exploration in multi-armed bandits in mansour2019bayesian. Furthermore, through our discussion in (ref), one may view (ref) as an algorithm for incentivizing exploration in linear contextual bandit settings with measurement error in the context.
The recommendation policy in (ref) is split into two stages. In the first stage, the principal provides no recommendations to the first $N_0$ units. Left to their own devices, these units take their preferred intervention according to their prior belief: type $0$ units take control, and type $1$ units take the treatment. We choose $N_0$ such that it is large enough for the (ref) to be satisfied for all units of type $1$ under the treatment with high probability.\footnote{The (ref) will always be satisfied w.h.p. for type $1$ units under the treatment after seeing sufficiently-many type $1$ units. The hard part is convincing enough type $1$ units to remain under control so that the (ref) is also satisfied for control.}\looseness-1
In the second stage, we use the initial data collected during the first stage to construct an estimator of the average expected post-intervention outcome for type $1$ units under treatment using PCR, as described in (ref). By dividing the total number of units in the second stage into phases of $L$ rounds each, the principal can randomly “hide” one explore recommendation amongst $L-1$ exploit recommendations. When the principal sends an exploit recommendation to unit $i$, they recommend the intervention which would result in the highest estimated average post-intervention outcome for unit $i$, where unit $i$'s post-intervention outcomes under treatment are estimated using the data collected during the first stage. On the other hand, the principal recommends that unit $i$ take the control whenever they send an explore recommendation.
Thus under (ref), if a type $1$ unit receives a recommendation to take the treatment, they will always follow the recommendation since they can infer they must have received an exploit recommendation. However if a type $1$ unit receives a recommendation to take the control, they will be unsure if they have received an explore or an exploit recommendation, and can therefore be incentivized to follow the recommendation as long as $L$ is set to be large enough (i.e. the probability of an explore recommendation is sufficiently low). This works because the sender's estimate of $\mathbb E[\Bar{y}_{i,post}^{(1)}]$ is more informed than unit $i$'s prior belief, due to the history of outcomes $H_i$ that the principal has observed.
Our algorithm requires that bounds on various aggregate statistics about the underlying unit population are common knowledge amongst the principal and all units. Such assumptions are standard in the literature on incentivized exploration.
We are now ready to present our main result: a way to initialize (ref) to ensure that the (ref) is satisfied with high probability.
Summary of (ref): (ref) posits that the principal and units know valid upper- and lower-bounds on various population-level statistics about the underlying unit population ({e.g., \xspace} the fraction of units of each type, the prior mean for each type and intervention). As these are population-level statistics, in practice, the principal may be able to learn these bounds from samples of the data and broadcast them to the unit population. See (ref) for more details.
(ref) says that after running (ref) with optimally-chosen parameters, the (ref) will be satisfied for all type $1$ units under both interventions with high probability. Therefore after running (ref) on sufficiently-many units, the principal can use off-the-shelf SCMs (e.g. agarwal2020synthetic, agarwal2023adaptive) to obtain counterfactual estimates with valid confidence intervals for all type $1$ units under control (w.h.p.). Note that all parameters required to run (ref) are computable by the principal in polynomial time. See (ref) for a concrete instantiation of (ref).\looseness-1
So far, our focus has been on designing algorithms to incentivize subpopulations of units to take interventions for which the (ref) does not initially hold. In this section we study the complementary problem of designing a procedure for determining whether the (ref) holds for a given test unit and set of donor units. In particular, we provide two simple hypothesis tests for deciding whether the (ref) holds, by leveraging existing finite sample guarantees for PCR. Our first hypothesis test is non-asymptotic, and thus is valid under any amount of data. While our second hypothesis test is asymptotic and therefore valid only in the limit, we anticipate that it may be of more practical use.
Our first hypothesis test proceeds as follows: Using the PCR techniques described in (ref), we learn a linear relationship between the first $T_0/2$ time-steps in the pre-intervention time period and the average outcome in the second half of the pre-intervention time period using data from the donor units. Using this learned relationship, we then compare our estimate for the test unit with their true average outcome in the second half of the pre-intervention time period. If the difference between the two is larger than the confidence interval given by PCR (plus an additional term to account for the noise), we can conclude that the (ref) is not satisfied with high probability.
In order for our hypothesis test to be valid we require the following assumption, which is analogous to (ref).
While (ref) requires that the post-intervention latent factors for any intervention fall within the linear span of the pre-intervention latent factors, (ref) requires that the second half of the pre-intervention latent factors fall within the linear span of the first half. This assumption allows us to obtain valid confidence bounds when regressing the second half of the pre-intervention outcomes on the first half. Let $y_{i,pre'} := [y_{i,1}^{(0)}, \ldots, y_{i,T_0/2}^{(0)}]$, $y_{i,pre''} := [y_{i,T_0/2 + 1}^{(0)}, \ldots, y_{i,T_0}^{(0)}]$, and $\Bar{y}_{i,pre''} := \frac{2}{T_0} \sum_{t=T_0/2 + 1}^{T_0} y_{i,t}^{(0)}$. Under (ref) we can estimate $\mathbb{E} [\Bar{y}_{i,pre''}]$ as $\widehat{\mathbb{E}} [\Bar{y}_{i,pre''}] := \langle \widehat{\boldsymbol{\theta}}_{i}, y_{i,pre'} \rangle$, where $\widehat{\boldsymbol{\theta}}_{i}$ is computed using the PCR techniques in (ref).
We are now ready to introduce our hypothesis test. In what follows, we will leverage the high-probability confidence interval of agarwal2023adaptive ((ref)). Under (ref), we can apply (ref) out-of-the-box to estimate $\mathbb{E} [\Bar{y}_{i,pre''}]$ using the first $T_0/2$ time-steps as the pre-intervention period and the next $T_0/2$ time-steps as the post-intervention period.
{\bf Hypothesis test.} Consider the hypothesis that (ref) is not satisfied for test unit $n$.\looseness-1
Therefore we can conclude that if $|\widehat{\mathbb E} [\Bar{y}_{i,pre''}] - \Bar{y}_{i,pre''}| > \alpha(\delta) + 2\sigma \sqrt{\frac{\log(1/\delta)}{T_0}}$, then the (ref) will not be satisfied for unit $i$ w.h.p.
Next we present a hypothesis test which, while only valid in the limit, may be of more practical use when compared to the non-asymptotic hypothesis test of (ref). At a high level, our asymptotic hypothesis test leverages (ref) and the guarantees for synthetic interventions (SI) in agarwal2020synthetic to determine whether the (ref) holds for a given test unit $n$. In particular, their results imply that under (ref) and the (ref), a rescaled version of $\widehat{\mathbb E} [\Bar{y}_{n,pre''}] - \mathbb E [\Bar{y}_{n,pre''}]$ converges in distribution to the standard normal distribution as $|\mathcal{I}|, T_0, T_1 \rightarrow \infty$. While the rescaling amount depends on quantities which are not computable, a relatively simple application of the continuous mapping theorem and Slutsky's theorem imply that we can compute a valid test statistic which converges in distribution to the standard normal.\looseness-1
{\bf Notation.} In what follows, let $\omega^{(n,0)} \in \mathbb{R}^{|\mathcal{I}|}$ be the linear relationship between the test unit $n$ and donor units $\mathcal{I}$ which is known to exist under the (ref), i.e. $\mathbb E[y_{n,t}^{(0)}] = \sum_{i \in \mathcal{I}} \omega^{(n,0)}[i] \cdot \mathbb E[y_{i,t}^{(0)}]$. Let $\widetilde{\omega}^{(n,0)}$ be the projection of $\omega^{(n,0)}$ onto the subspace spanned by the latent factors of the $\mathcal{I}$ test units. Let $\widehat{\omega}^{(n,0)}$ be the estimate of $\widetilde{\omega}^{(n,0)}$ and $\widehat{\sigma}$ be the estimate of the standard deviation of the noise terms $\{\epsilon_{i,t} \; : \; i \in \mathcal{I}, t \in [\![T]\!]\}$ given by SC. Finally, let $z_{\alpha}$ be the z score at significance level $\alpha \in (0,1)$, i.e. $\alpha = \mathbb{P}(x \leq z_{\alpha})$, where $x \sim \mathcal{N}(0,1)$.
{\bf Hypothesis test.} Consider the hypothesis that the (ref) is not satisfied for a test unit $n$.
In this section, we complement our theoretical results with a numerical comparison to standard SCMs which do not take incentives into consideration.
{\bf Experiment Setup.} We consider the setting of (ref): There are two interventions and two unit types. Type $1$ units prefer the treatment and type $0$ units prefer the control. Our goal is to incentivize type $1$ units to take the control so we can obtain accurate counterfactual estimates of all units under control. See (ref) for details about our data-generating process.
We apply the asymptotic hypothesis test of (ref) using all type $0$ units as the donor units and a single type $1$ unit as the test unit. We accept the null hypothesis, as the test statistic $\frac{\sqrt{T_1}}{\widehat{\sigma} \|\widehat{\omega}^{(n,0)}\|_2}|\widehat{\mathbb E} [\Bar{y}_{n,pre''}] - \Bar{y}_{n,pre''}| \approx 2.24$, which is much larger than the value of $z_{0.95} \approx 1.96$ needed for acceptance. That is, the asymptotic test (correctly) returns that the (ref) is not satisfied for a test unit of type $1$ under control.
Using (ref) (the formal version of (ref)), we can calculate a lower bound on the phase length $L$ of (ref) such that the BIC condition ((ref)) is satisfied for units of type $1$ who are recommended the control. We run three different sets of simulations and compare the estimation error $|\widehat{\mathbb{E}} \Bar{y}_{n,post}^{(0)} - \mathbb{E} \Bar{y}_{n,post}^{(0)}|$ for a new type $1$ unit $n$ under control when incentivizing exploration using (ref) (blue) to the estimation error without incentivizing exploration (orange). For both our method and the incentive-unaware ablation, we use the adaptive PCR method of agarwal2023adaptive to estimate counterfactuals. All experiments are repeated $50$ times and we report both the average prediction error and the standard deviation for estimating the post-treatment outcome of type $1$ units under control.\looseness-1
{\bf Varying Strength of Belief.} In (ref), we explore the effects of changing the units' “strength of beliefs”. Specifically, we vary the gap between the prior mean outcome under control and treatment for type $1$ units by setting $\mu_{i}^{(1)} - \mu_{i}^{(0)} \in \{ 0.2, 0.4, 0.6, 0.8 \}$. As the gap decreases it becomes easier to persuade units to explore, and therefore our counterfactual estimates improve.
{\bf Varying Latent Factor Size.} In (ref), we vary $r \in \{2, 4, 6, 8\}$ while keeping all other parameters of the problem instance constant. Increasing $r$ increases the batch size $L$, so we see a modest decrease in performance as $r$ increases.
In order to test the validity of the asymptotic hypothesis test in (ref), we use synthetic data for which we know the underlying data-generating process.
Specifically, we create two types of synthetic datasets: one where the (ref) holds (“UOA data”), and one where it does not (“non-UOA data”). For simplicity, we consider a setting where there are two interventions and two unit types.
For non-UOA data, we use the same data-generating process as described above, where if unit $i$ is of type $1$ (resp. type $0$), we generate a latent vector $\mathbf{v}_i = [0 \; \cdots \; 0 \; v_i[1] \; \cdots \; v_i[r/2]]$ (resp. $\mathbf{v}_i = [v_i[1] \; \cdots \; v_i[r/2] \; 0 \; \cdots \; 0]$), where $\forall j \in [r/2]: v_i[j] \sim \mathsf{Unif}(0, 1)$. This dataset then contains $500$ units of each type. On the other hand, UOA data contains $1000$ Type $0$ units only, where the latent vector is generated the same way as Type 0 units in non-UOA data. The pre-treatment and post-treatment latent factors are generated in the same manner as above.
Using this data-generating process, we generate 500 datasets of each type (for a total of 1000 datasets), and run the asymptotic hypothesis test in (ref) for each dataset. If we accept the null hypothesis, {i.e., \xspace} the (ref) is predicted not to be satisfied according to our test, then we say the 'predicted' label of the dataset is $1$; otherwise, the 'predicted' label is $0$. We say that a dataset has (true) label $1$ if the (ref) is not satisfied, and label $0$ if it is.
We report the true positive rate (TPR) and false positive rate (FPR) for this experiment. The resulting confusion matrix is:
with the $TPR = 0.956$ and $FPR = 0.05$. This experimental result supports (ref), where the FPR is at most 0.05.
We study the problem of non-compliance when estimating counterfactual outcomes using panel data. Our focus is on synthetic control methods, which canonically require an overlap assumption on the unit outcomes in order to provide valid finite sample guarantees. We shed light on this often overlooked assumption, and provide an algorithm for incentivizing units to explore different interventions using tools from information design and online learning. After running our algorithm, the (ref) will be satisfied with high probability, which allows for the principal to obtain valid finite sample guarantees for all units when using off-the-shelf SCMs to estimate counterfactual outcomes under control. We also extend our algorithm to satisfy the (ref) for all units and all interventions when there are more than two interventions, and we provide two hypothesis tests for determining if the (ref) hold for a given test unit and set of donor units. Finally, we complement our theoretical findings with numerical simulations, and observe that our procedure for estimating counterfactual outcomes produces significantly more accurate estimates when compared to existing methods which do not take unit incentives into consideration.
There are several exciting directions for future research. In some multi-armed bandit settings, lower bounds are known on the sample complexity of incentivizing exploration (sellke2022price, sellke2023incentivizing). It would be interesting to prove analogous bounds on the number of units required to incentivize sufficient exploration for the (ref) to be satisfied for all units and all interventions in our setting. Another avenue for future research is to design difference-in-differences or clustering-based algorithms for incentivizing exploration in other causal inference settings. Finally it would be interesting to relax other assumptions typically needed for synthetic control, such as the linear span inclusion assumption on the time-step latent factors.
We would like to thank the anonymous reviewers for their helpful feedback. For part of this work, DN was a Ph.D. student at the University of Minnesota and KH was a Ph.D. student at Carnegie Mellon University. VS was supported by NSF Award IIS-2337916 and ZSW was supported by an NSF CAREER Award.