Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
30,559 characters · 10 sections · 21 citation commands
SPORTSCausal: Spill-Over Time Series Causal Inference
Randomized controlled trial(RCT) has been playing a central role in causal inference. The easiest design of RCT is single-treatment complete randomized design (CRD), which is popular in business analysis and also clinical investigation. More modern design, such as randomized complete block design (RCBD), factorial design(mukerjee2007modern), latin-square design(bradley1958complete), suduko square design(sarkar2015sudoku) etc are also constructed for multi-factor complicated experiments.
Mixed effect model (as R package \verb+lme4+) was designed as a comprehensive tool for all of those experimental designs (bates2007lme4).It is simple as analysis of covariance (ANCOVA) when there is no confounding blocks or unit-level random covariates. Take the single-treatment CRD as an example, the averaged treatment effect and its corresponding statistical inference could be evaluated by regress the time $T$ outcome with time 0 outcome and the treatment indicator ($y_{it} \sim y_{i0} + d_i$ where $d_i$ is the treatment indicator). $y_{it}$ and $y_{0t}$ are always substituted by the average over time "after" and "before" the treatment.
However, similar as rubin1986comment described, most experimental designs require the following three assumptions for the traditional methodologies to be valid:
These assumptions simplify statistical inference but are often not valid in real-world settings. Statisticians continue to explore various modeling strategies to relax these assumptions and create more flexible models.
Grouping randomness is a key assumption in experimental design, yet it is not always achievable due to constraints in the experimental environment and sample size. For instance, as discussed in zheng2024bootstrap, in business analysis, a common approach is the tail-number experiment, where subjects with odd-numbered sample IDs are placed in the experimental group, while those with even-numbered IDs are placed in the control group. This design is random only if no other experiments are concurrently using similar ID tail-number information. Large sample theory ensures that when grouping randomness is present, all features are balanced between groups, reducing noise from unit-level variability and confounding covariates. Without grouping randomness, the experiment offers little advantage over observational studies, potentially leading to differences between the experimental and control groups even before treatment begins.
To address this issue, various methods have been developed, such as Difference-in-Differences (DID, luke1998interracial), Propensity Score Matching (PSM, rosenbaum1985constructing), and Synthetic Control (abadie2010synthetic), all of which aim to match experimental and control subjects on pre-treatment characteristics. Recent advancements, such as zheng2024bootstrap Bootstrap Matching, have been made to enhance the robustness of PSM in experimental settings.
Time independence is another critical assumption in experimental design, as it simplifies the inference process by avoiding the complexities of longitudinal time series data. Researchers often prefer longer pre- and post-treatment periods to strengthen their assumptions, although they typically average over time rather than fully utilizing the longitudinal information. However, approaches like those by robins2000marginal in epidemiology and the \verb+causalimpact+ R package developed by Google for Bayesian longitudinal causal inference brodersen2015inferring have started to integrate time series data into causal analysis.
The assumption of independence between treatment groups, encapsulated in the "Stable Unit Treatment Value Assumption" (SUTVA, rubin1986comment), posits that the treatment should only affect the experimental group and not the control group, with no reciprocal influence between the two. While this assumption generally holds in many study areas, there are situations where its violation leads to significant issues in causal inference. For instance, in business analysis, particularly in advertising strategy, if group A is subjected to a traditional strategy and group B to a new algorithm, the competition dynamics between the two groups can cause spillover effects. These effects occur when changes in group B indirectly influence group A, complicating the interpretation of the treatment effect. The observed difference between the groups is then a combination of both the direct (treatment) and indirect (spillover) effects. aronow2020spillover explored methods to estimate these spillover effects, and R packages like \verb+inferference+ and \verb+interferenceCI+ have been developed to address similar challenges. However, these methods often require sophisticated pseudo-experimental designs to isolate the behavior of the control group during treatment, without the interference of spillover effects.
In this paper, we introduce a novel statistical method, \verb+SPORTSCausal+, designed to perform causal inference in scenarios where group randomness is absent, but spillover effects are present. We demonstrate the application of \verb+SPORTSCausal+ using a real-world budget-control experiment, highlighting its effectiveness in handling complex causal relationships.
Let's formalize the question, taking a single treatment RCT as an example. More complicated design could be easily generated here to capture the spillover effect. Denote $i = 1,2,\cdots,m, m+1, \cdots, m+n$ be the number of subjects where the first $m$ subjects are randomly chosen to fit in the base group and the rest $n$ subjects in experiment group. Suppose $t = 1,2,\cdots, T_0$ be the time period before the treatment and $t = T_0 + 1, T_0+2, \cdots, T_0 + T$ be the time period after the treatment. Therefore, for the outcome, we have a matrix $Y = \left(y_{it}\right)_{(m+n)*(T_0 + T)}$ where each row is a subject and each column is a time point and $y_{it}$ is for example, the cost/revenue/ROI for subject $i$ at time point $t$. For simplicity, let's denote $D_i$ be the indicator for treatment effect. In other words, $D_i = 0, i\in \{1,2,\cdots,m\}$ and $D_i = 1, i \in \{m+1, \cdots, m+n\}$. We could also have a feature matrix $X = (x_{ij})_{(m+n)*k}$. Supposing each subject has $k$ features (such as age, gender, etc), each row is again a feature vector for subject $i = 1,2,\cdots, m+n$ and $j = 1,2,\cdots, k$. Finally the unknown treatment effect to be $d$.
Let's define previous several assumptions:
Let's here make a few remarks:
The most difficult assumption to be understood is assumption $C_3$. The following figure, Figure (ref) shows a graphical representation when assumption $C_3$ is satisfied or violated.
Several clarifications related to assumption $C_3$:
Those three assumptions would often be satisfied and people is always utilizing those three assumptions to make easier statistical inferences. We would review different algorithms when the previous assumptions is satisfied or violated. However, before that, we would like to relax assumption $C_2$ and $C_3$ a little bit, that would eventually benefit the algorithm development presented in this paper.
Statisticians are developing different algorithms under variety of assumptions. Here we only describe a few of them. Our target is to descirbe algorithm only requiring $C_4,C_5$.
This is the easiest algorithm for experimental design. R package \verb+lme4+ can be used to implement this algorithm. Without the subject specific covariates, we can also fit the model using \verb+lm+ or \verb+anova+.
When $C_1$ is violated, we pretty much have an observational study rather than experimental design. Many statisticians attempt to solve this problem. We describe a recent proposed algorithm zheng2024bootstrap name bootstrap matching, which is a robust version of PSM borrowing idea from Bootstrap and PSM.
The final step needs large-scale hypothesis testing tools (e.g., zheng2021mixtwice). Compared to traditional Propensity Score Matching (PSM), bootstrap matching offers greater robustness by employing random subsampling multiple times, which helps to reduce bias and variance in the estimates. Additionally, because the bootstrapping process can be performed in parallel, bootstrap matching can achieve better computational efficiency. This is particularly beneficial since the matching step, after calculating all propensity scores, is computationally intensive. A detailed discussion on this method can be found in zheng2024bootstrap.
brodersen2015inferring develops method \verb+causalimpact+ using time series data. The key idea here is to use structural bayesian time series model to predict the treatment-free behavior of experiment group, using the information of control. The gap between the predicted performance and the actual performance of treatment group after the experiment, is therefore the treatment effect. Compared to traditional analysis of covariance (ANCOVA), \verb+causalimpact+ relaxed the assumption of time independence but borrowed more information from longitudinal prospective.
In the \verb+causalimpact+, we always denote $y_t$ be the treatment series and $x_t$ be the control series. We use the following time series model to capture the relationship between control and treatment group:
where $\mu_t$ is the local linear trend that could be modeled using different types of time series model (for example, auto-regressive model, ARIMA model with certain degree of lag, etc), $\tau_t$ is a fixed effect to model seasonality, $\epsilon_t$ is the white noise with certain variance. It is remarkable that some preliminary analysis, such as matching step, is still beneficial but not necessary since we allow the coefficient $\beta \neq 0$ which evaluates the difference between treatment group and control, in the absence of experiment.
We now try to discuss why \verb+causalimpact+ might not work in the absence of assumption $C_3$.
Figure (ref) is only a toy example illustrating why the assumption $C_3$ is important for \verb+causalimpact+. The time series is generated using simple auto-regressive model. Left panel of Figure (ref) shows in the presence of assumption $C_3$. Under the statistical model of \verb+causalimpact+, people uses information from red line (before and after the treatment) to predict the performance of treatment group, without the treatment. Therefore the gap between the predicted mean and the actual mean in after the treatment is the unbiased and consistent estimator of treatment effect. However, in the absence of $C_3$ (right panel), we could suffer from using control group to predict the treatment since there is structural interference between groups when treatment affects. Moreover, the gap between red and blue is the summation of treatment effect and indirect interference effect.
However, when we relax assumption $C_3$ to $C_5$, with the help of $C_4$, we could still get benefits.
Claim:
Under the assumption of $C_4$, $C_5$, treatment effect is identifiable.
The prove of this claim indeed results in the algorithm:
In online advertising, budget optimization is one of the most critical tasks for maximizing the effectiveness of campaigns. While auction-based systems handle the majority of ad placements, subsidies can sometimes be strategically employed to enhance overall system performance. For instance, one common application of subsidies is the cold start problem, where new advertiser campaigns lack sufficient data for accurately predicting key metrics such as click-through rate (CTR) or conversion rate (CVR). To address this, a subsidy may be assigned to these new campaigns to support model learning, allowing the system to gather the necessary data to improve prediction accuracy over time. By temporarily boosting the visibility of these campaigns, the subsidy helps mitigate the initial disadvantage faced by new advertisers, facilitating a more efficient and effective advertising ecosystem. We call those experiments advertiser campaign budget optimization experiments.
Even when experiments are correctly randomized at the level of advertisers or ad campaigns, it is challenging to eliminate the spillover effect entirely. For example, in a scenario where the treatment group receives subsidies while the control group does not, the treated campaigns are more likely to win auctions and secure a greater number of ad impressions. This occurs because, with a relatively stable user base, the total number of impressions is unlikely to increase significantly in a short period. Consequently, campaigns with additional budget or subsidies are better positioned to outbid others and capture a larger share of impressions.
Moreover, the impact of the spillover effect can vary depending on the size of the treatment group. When only 5% of campaigns receive a subsidy, the remaining 95% may experience reduced opportunities as the subsidized campaigns outcompete them. In contrast, when 50% of the campaigns receive additional subsidies, the competitive landscape shifts, with half of the campaigns benefiting from the subsidy while the other half faces increased competition. This variation in the spillover effect based on treatment size highlights the complexities involved in designing and interpreting experiments in online advertising, particularly when financial interventions like subsidies are involved.
As shown in Figure (ref), there is a significant gap between the treatment group and the control group. However, the magnitude of this increase differs notably based on the size of the experiment: the treatment effect is approximately 37.35% when the experiment size is 5%, compared to 28.22% when the experiment size is 50%. This gap arises not only from the increase in the treatment group but also from a corresponding decrease in the control group. The observed inconsistency in treatment effects between low-traffic and high-traffic experiments, if left uncorrected, can lead to substantial misinterpretations and potentially misguided business decisions.
Addressing these variations is crucial to ensure that the insights derived from experiments accurately reflect the true impact of interventions across different scales. To achieve this, \verb+SPORTSCausal+ was applied to predict the control (shown in black), resulting in corrected treatment effects of 13.36% in the 5% experiment and 14.34% in the 50% experiment. By accounting for the spillover effect, \verb+SPORTSCausal+ provides a more consistent and accurate estimation of the treatment effect, ensuring that business decisions are based on reliable data across varying experiment sizes.
The methods applied to more than the settings of online advertising. In the field of economics, particularly in policy evaluation, the assumptions of group independence and randomness are often violated due to interconnected markets and spillover effects. For instance, a government subsidy in one region can influence prices and demand in neighboring regions, complicating the causal interpretation of policy impacts. Traditional econometric models might overestimate or underestimate the true effect of such interventions due to unaccounted spillovers. \verb+SPORTSCausal+ provides a robust framework to account for these complexities, enabling more accurate estimation of policy impacts by adjusting for time-series spillovers. This is particularly valuable in economic studies where interventions can have far-reaching and dynamic effects across different regions or markets. In bioinformatics, high-density peptide arrays, for example, are essential for studying peptide-protein interactions zheng2020disordered, mergaert2022rheumatoid, amjadi2024novel,parker2024novel, but their densely packed format can lead to spillover effects, where the binding of one peptide affects neighboring ones. These effects can distort estimates of true binding affinities, complicating the interpretation of results. \verb+SPORTSCausal+ helps address these challenges by accounting for spillover in time-series data, providing more accurate estimates of peptide-antibody interactions. This is crucial for applications like vaccine development, where precise identification of immunogenic peptides is vital. In social and behavioral sciences, particularly in psychology, interventions often influence not only the participants but also those around them, leading to spillover effects. For instance, studies like liu2022progressive highlight the importance of retrieval practice in enhancing memory. However, when applying such practices in group settings, spillover effects can occur, where the benefits of one participant's practice may influence others, potentially confounding the results. \verb+SPORTSCausal+ is well-suited to address these complexities by accounting for spillover effects in the analysis of memory interventions. By applying this method, researchers can obtain a clearer picture of the true impact of retrieval practices, ensuring that the observed improvements in memory are accurately attributed to the intervention itself, rather than external influences. This leads to more reliable conclusions that can better inform educational and therapeutic strategies.
In this paper, we have introduced \verb+SPORTSCausal+ and its R package sports, a novel approach to causal inference that addresses the challenges posed by the absence of group randomness and the presence of spillover effects. Traditional methods, such as Propensity Score Matching (PSM) and other matching techniques, often rely on strong assumptions that may not hold in real-world scenarios, leading to biased estimates and misleading conclusions. \verb+SPORTSCausal+ overcomes these limitations by accounting for both direct and indirect effects, providing a more accurate assessment of treatment impacts.
We demonstrated the effectiveness of \verb+SPORTSCausal+ through a real-world budget-control experiment, where traditional methods failed to capture the true treatment effect due to significant spillover. By correcting for these spillover effects, \verb+SPORTSCausal+ delivered robust and reliable estimates across different experiment scales. Our discussion extended beyond business applications, highlighting the broader relevance of \verb+SPORTSCausal+ in fields such as economics, bioinformatics, and social and behavioral sciences.
Overall, \verb+SPORTSCausal+ represents a significant advancement in causal inference, offering a flexible and robust tool for researchers across various domains. However, one limitation of the proposed approach is its lack of integration with experimental design principles. While \verb+SPORTSCausal+ provides substantial benefits in real-world settings where experiment setups are less than ideal, we believe that combining this method with efficient experimental design could further enhance its effectiveness.
In the context of online advertising, for example, one of the critical challenges is determining the minimum traffic needed to accurately estimate the true treatment effect within a reasonable timeframe. Many trials are exploratory, and diverting significant live traffic—especially when it's user-visible—is often undesirable. Simultaneously, business decisions must be made swiftly, meaning teams cannot afford to wait for data to fully converge. Given these constraints, along with the complexities introduced by spillover effects, we see significant opportunities for future research to refine and extend \verb+SPORTSCausal+, particularly in integrating it with optimized experimental design strategies.