Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs

Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs

\RUNAUTHOR \RUNTITLE{Data-Driven Switchback Experiments} \TITLE{Data-Driven Switchback Experiments: \\ Theoretical Tradeoffs and Empirical Bayes Designs\footnote{The preliminary version of this paper is circulated under the title “Bias-variance tradeoffs for designing simultaneous temporal experiments.” This version has more general theoretical results, a new design approach, and an empirical application. We would like to thank Susan Athey, Mohsen Bayati, Haoge Chang, Dean Eckles, Adam Glynn, and Ramesh Johari for their useful comments and suggestions.} }

\ARTICLEAUTHORS{ \AUTHOR{Ruoxuan Xiong} \AFF{Department of Quantitative Theory and Methods, Emory University, \EMAIL{[email removed]}} \AUTHOR{Alex Chin} \AFF{Motif Analytics, \EMAIL{[email removed]}} \AUTHOR{Sean J. Taylor} \AFF{Motif Analytics, \EMAIL{[email removed]}} }

\ABSTRACT{ We study the design and analysis of switchback experiments conducted on a single aggregate unit. The design problem is to partition the continuous time space into intervals and switch treatments between intervals, in order to minimize the estimation error of the treatment effect. We show that the estimation error depends on four factors: carryover effects, periodicity, serially correlated outcomes, and impacts from simultaneous experiments. We derive a rigorous bias-variance decomposition and show the tradeoffs of the estimation error from these factors. The decomposition provides three new insights in choosing a design: First, balancing the periodicity between treated and control intervals reduces the variance; second, switching less frequently reduces the bias from carryover effects while increasing the variance from correlated outcomes, and vice versa; third, randomizing interval start and end points reduces both bias and variance from simultaneous experiments. Combining these insights, we propose a new empirical Bayes design approach. This approach uses prior data and experiments for designing future experiments. We illustrate this approach using real data from a ride-sharing platform, yielding a design that reduces MSE by 33% compared to the status quo design used on the platform.

} \KEYWORDS{Time-Based Experiment, Carryover Effect, Simultaneous Intervention, Treatment Effect Estimation, Ride-Sharing Platform}

\setcounter{equation}{0} {\thesection.\arabic{equation}} \oldsection{Introduction}

Experimentation has become an increasingly popular and effective tool for testing and improving social and business policy in digitally-mediated economic and social settings. However, the scale and complexity of modern digital applications have given rise to scientific and statistical challenges for the design and analysis of experiments.

The leading example considered in this paper is a ride-hailing platform where a product team would like to measure the effects of their product (e.g., matching or pricing algorithm) changes through an experiment. The experiment is run on a geographically determined market for two weeks. The product team chooses an experimental design ex-ante that determines when the current and test product versions are used during the experiment.

Such a product change may affect users' behavior in ways that create interference and alter outcomes for other users on both the rider and driver side of the marketplace. To mitigate user-level interference, companies often aggregate all users in a market into one unit and employ switchback designs on this unit.\footnote{See examples in Amazon masoero2023efficient,Cooprider2023amazon, DoorDash Kastelman2018switchback, Lyft chamandy2016experiment, Tubi silbert2022switchback, LinkedIn and Netflix bojinov2020avoid. } Prior to the recent applications in marketplaces, switchback designs were originally used in agriculture cochran1941double and medicine mirza2017history. These designs randomly switch between treatment and control for the same unit over time chamandy2016experiment.Post experiment, a quantity we call the global average treatment effect (GATE) is commonly estimated. The GATE measures the difference in average outcomes over users and time between when the product change is deployed indefinitely (global treatment) versus when it is absent (global control).

Precise estimation of GATE is important in deciding whether to launch the product change indefinitely. The precision can be improved by using a better switchback design. Prior work (bojinov2020design,hu2022switchback among others) has studied the design problem concerning the carryover effects, which is a major source of the estimation error. Carryover effects measure the impact of past interventions on future outcomes. They are nonzero when interventions take time to change the marketplace to a new equilibrium state where the globally treated outcomes can be observed.\footnote{Carryover effects can be viewed as interference in the temporal dimension, which is distinct from interference across users.} To reduce the estimation error from carryover effects, switching less frequently can help.

In this paper, we show that the estimation error more generally depends on four factors: carryover effects considered in prior work, periodicity, correlated outcomes, and simultaneous interventions, as illustrated in Figure (ref). Specifically, periodicity arises from the significant variation in rider demand and driver availability over time in a day and days in a week. Outcomes close in time are correlated due to weather, traffic, or other external factors such as supply and demand shocks. Other product teams may test other product changes simultaneously in the same market, complicating the measurement of the marginal effect of each one.

Summary of Contributions

We study the design and analysis of switchback experiments, accounting for all four factors. While we use the ride-hailing platform as the leading example, our work also applies to other contexts where some or all of these factors exist.

For the design, we propose an empirical Bayes approach that uses knowledge from prior experiments to inform the design of new experiments. We illustrate this approach through a case study of a large corpus of prior experiments run in various geographically determined markets on a ride-sharing platform. This empirical Bayes approach first estimates an empirical distribution of cumulative effect curves (CECs), where each one is estimated from one experiment-market pair. The cumulative effect, as a measurement of carryover effects, is the total effect from current and past treatments,\footnote{Formally, the cumulative effect is the sum of instantaneous effect (effect of current treatment) and carryover effects (effects of past treatments).} which converges to GATE as the treatment duration grows to infinity. The shape of CEC shows the relationship between cumulative effect and treatment duration. In our case study, most CECs are non-monotonic, with 68% of CECs changing signs as the treatment duration changes. See two examples in Figure (ref). Non-monotonicity and changing signs can occur, for example, when testing a new surge pricing algorithm. The surge pricing may immediately reduce the demand and negatively affect the outcome (e.g., conversion rate). However, surge pricing attracts more drivers, gradually increases the supply, reduces the ETA, and then positively affects the outcome. With a non-monotonic curve similar to Curve \#2 in Figure (ref), the estimated GATE can have larger errors from carryover effects than that with a curve like Curve \#1.

This empirical Bayes approach then chooses a switchback design by comparing the performance of candidate designs in synthetic experiments. These synthetic experiments are run on historical data, and the CEC is randomly drawn from the empirical distribution of CECs, ensuring that all four factors are present. The best-performed design randomizes and, on average, doubles the switching lengths than the status quo design with fixed lengths, yielding a 33% reduction in MSE. Furthermore, the order of candidate designs encompasses a {\it hierarchical structure} in the effectiveness of three design principles: (a) {\it balancing} periodicity is the most effective; (b) carefully {\it selecting average switching periods} is moderately effective; (c) {\it tuning exact switching times} is mildly effective.

For the theoretical analysis, we provide a rigorous decomposition of bias and mean-squared error (MSE) of the estimated GATE from the standard Horvitz–Thompson estimator, showing the tradeoffs of four factors. The bias is decomposed into two sources of errors: (a) carryover effects from treatment at earlier times; (b) confounding effects from simultaneous interventions. The MSE is decomposed into squared bias and variance, where the variance is affected by three sources of randomness: (a) the measurement errors of outcomes and their covariance, determined by their distance in time; (b) the randomness of treatment assignments of primary and simultaneous interventions; (c) the randomness in event occurrence times.

The decomposition together with a careful simulation study explains how the design principles reduce the estimation error: (a) balancing periodicity reduces all sources of variance; (b) switching less frequently reduces bias from carryovers; (c) switching more frequently reduces variance from the randomness in measurement errors and treatment assignments; (d) randomizing interval start and end points reduces both bias and variance from simultaneous interventions. These insights manifest that the hierarchical structure identified in the ride-sharing setting is due to the large noise compared to the size of GATE; once the variance is reduced through balancing, bias dominates because of the non-monotonic CECs, and switching less frequently helps.

Related Work

Our work contributes to the literature on the design and analysis of experiments wu2011experiments, and more specifically to literature on the time-based experiments or crossover trials jones2003design. Our work is most closely related to the growing literature on the design and analysis of switchback experiments, starting from bojinov2020design. We complement this literature bojinov2020design,hu2022switchback,wu2022dynamic,ni2023design,li2023experimenting,chen2023switchback,masoero2023efficient and show that, in addition to carryover effects considered in this literature, factors including periodicity, correlation event outcomes, and simultaneous interventions can affect the performance of switchback design. Due to these factors, we show the value of using prior data and experiments, and propose new designs to improve efficiency.

Our design is also related to a number of other designs in time-based experiments. One design is the staggered rollout design for panel experiments, where the design selects an initial (and possibly different) treatment time for each unit, either under temporal interference xiong2019optimal,basse2019minimax or network interference cortez2022staggered,han2022detecting,boyarsky2023modeling. Another design is the synthetic control design for panel experiments, where the design selects units to be treated, allocates treatment to all of them in a single period, and forms a synthetic treated and control unit for treatment effect estimation doudchenko2019designing,doudchenko2021synthetic,abadie2021synthetic. Concerning periodicity and nonstationarity, we complement wu2022non,wu2023non and simchi2024non, and provide a solution to balance periodicity when experimenting on an aggregate unit.

In our setup, we aggregate the units such that interference between users can be abstracted away; however, a growing literature directly tackles interference using novel experimental ideas. For example, on network data, cluster-randomized designs are commonly used for mitigating interference ugander2013graph,eckles2017design,candogan2021near,holtzreducing, where the clusters are chosen to minimize edges that cut across clusters. Another popular method is the two-stage or multi-stage randomization, which has been used in public health hudgens2008toward,liu2014large, digital platforms ye2023cold, political science sinclair2012detecting, and social science crepon2013labor,baird2018optimal,basse2018analyzing. In the two-sided marketplace, recently proposed designs to mitigate interference bias include multiple randomization designs bajari2021multiple,johari2020experimental and designs that perturb treatments near equilibrium outcomes wager2021experimenting. li2021interference characterize the bias and variance of such experiments and describe how the design can be optimized in such settings. Besides using novel designs, there is a growing literature on developing new treatment effect estimators and inferential theory accounting for the interference (chin2018central,chin2019regression,forastiere2021identification,qu2021efficient,yuan2021causal,yuan2023two,leung2023network,farias2022markovian among others). Complementing this literature, our paper takes an agnostic approach to marketplace interference structure and leverages time-based experiments for treatment effect estimation.

Finally, this paper accounts for the impact of interventions tested simultaneously, which is common in practice. When multiple interventions are simultaneously applied to the same units, factorial design fisher1936design is commonly used wu2011experiments, which allows for estimating the effect of any treatment combination, especially useful in cross-sectional experiments with sufficiently low variance. In general, design and analysis are more complex when the number of interventions is large dasgupta2015causal.\footnote{This is because the number of treatment combinations increases exponentially in the number of interventions.} ye2023deep provide a novel solution to this problem using debiased deep learning. In this paper, we study the setting of time-based experiments, which is more challenging. We focus on designing and analyzing a single primary intervention while being agnostic to how the other experiments were designed.

\setcounter{equation}{0} {\thesection.\arabic{equation}} \oldsection{Problem Setup}

Suppose a decision maker runs an experiment on a geographically determined market between time $0$ and time $T$ to study the effect of a new intervention, such as a new pricing or matching algorithm. In this paper, we adopt a continuous time framework, which naturally captures the types of event stream data that are commonly observed on ride-hailing platforms and similar settings.

Before the experiment starts, the decision maker chooses the treatment design for the whole experiment horizon, i.e., $\bm{W} = \{W_{t} \in \{0, 1\},\, \forall t \in [0,T]\}$. Here $W_{t} = 1$ indicates that all users in this market are exposed to the intervention (treatment) at time $t$, and $W_{t}=0$ indicates otherwise (control). Note that here, the treatment decisions are made for a continuous time interval. Therefore, we propose to use a two-step design procedure: the decision maker first partitions the experimental horizon $[0,T]$ into $M$ disjoint intervals and then randomly chooses the treatment assignment of each interval. The design procedure is fully general and can essentially yield any design with an appropriate choice of $M$ and treatment assignment probability. Let $0 \leq \iota_{0} \leq \iota_{1} \leq \cdots \leq \iota_{M-1} \leq \iota_{M} = T$ be the endpoints that define the $M$ intervals, $\mathcal{I}_{m} = [\iota_{m-1}, \iota_{m}]$ be the $m$-th interval, and $|\mathcal{I}_{m}| = \iota_{m} - \iota_{m-1}$ be the length of the $m$-th interval.

As the treatment decisions are made at the interval level, the treatment assignments for all times within an interval are the same, i.e., \[W_{t} = W_{t^\prime}, \qquad \text{for all } t, t^\prime \in \mathcal{I}_{m}, ~~ \text{for all $m$}. \] Any design that allows for varying treatment assignments across intervals is a switchback design bojinov2020design and is the main focus of this paper. We will discuss switchback designs in more detail in Section (ref).

In this paper, we directly analyze the event stream data, which is usually the most granular data available in an experiment. The use of granular data can provide more information for the analysis and design of switchback experiments. For example, we might be interested in the event where a rider opens the app and checks the price.

We suppose there are $n$ events occurring in the marketplace between time $0$ and time $T$. For each event $i$, the occurrence time $t_i$ is random and, in the continuous time framework, can take any real value between $0$ and $T$. Let the outcome of event $i$ be $Y^{(i)}$. For example, $Y^{(i)}$ could be a binary variable indicating whether the rider requests a ride or not. Let $f(t): [0,T] \rightarrow \mathbb{R}^+$ be the density function from which events are sampled. We assume that $f(t)$ is bounded from below and from above for all $t$. For simplicity, it is possible to consider the uniform event density as in Example (ref). However, in many realistic settings, the density of events will exhibit periodic patterns due to the seasonality of human behavior. For instance, in ride-hailing, many ride requests occur during commute times, and relatively few occur during the late evening on weeknights. For these settings, a periodic event density, as in Example (ref), may be a more appropriate choice.

Besides the event outcome, we additionally define the marketplace outcome at time $t$ as $Y_t$. The marketplace outcome $Y_t$ can be viewed as the average outcome of all users in the marketplace, such as the average request rate at time $t$. Then the event outcome is a noisy measurement of the marketplace outcome, i.e., for all $i$, \[Y^{(i)} = Y_{t_i} + \varepsilon^{(i)} \, , \] where the measurement error $\varepsilon^{(i)}$ has mean zero. For example, when $Y^{(i)}$ is binary indicating whether rider $i$ requests a ride, we can model $Y^{(i)}$ as a random draw from the Bernoulli distribution with probability $\mathbf{P}(Y^{(i)} = 1) = Y_{t_i}$ of being one. Then $\varepsilon^{(i)}$ measures the difference between the binary outcome and the probability, i.e., $\varepsilon^{(i)}$ takes a value between $1-Y_{t_i}$ and $-Y_{t_i}$.

Importantly, measurement errors of events that are close in time can be correlated: \[\operatorname{Cov}(\varepsilon^{(i)}, \varepsilon^{(j)}) \neq 0 \qquad \text{for }~ t_i \neq t_j \, . \] The correlation can be caused by external factors like weather, supply conditions, and traffic. This correlation creates a nuisance dependence between event outcomes, which can affect the resulting variance of treatment effect estimates.

We further account for the possibility that other decision makers may run experiments simultaneously on the same market to test the effect of other interventions. We refer to the experiment that the primary decision maker runs as the primary experiment. For the simplicity of exposition, suppose only one experiment is run simultaneously in addition to the primary experiment in the main text. However, our theoretical results are shown for any number of simultaneous experiments. The insights for one and multiple simultaneous experiments are generally the same.

Let the treatment design of the simultaneous experiment be $\bm{W}^\mathrm{s}$, where $\bm{W}^\mathrm{s} = \{W_t^\mathrm{s} \in \{0, 1\},\, \forall t \in [0,T]\}$. We assume that the treatment designs of the simultaneous experiment are chosen independently of the primary experiment. We primarily focus on the case where the primary decision maker is {\it agnostic} to the treatment decisions of the simultaneous experiment. We further make the non-anticipating outcome assumption, i.e., the outcome at time $t$ is only affected by the treatment assignments up to time $t$ (basse2019minimax,bojinov2020design among others). We use $\bm{W}_t = \{W_u,~ \forall u \in [0, t]\}$ and $\bm{W}^\mathrm{s}_{t} = \{W_u^\mathrm{s}, ~\forall u \in [0, t]\}$ to denote the treatment assignments of the primary and simultaneous experiments up to time $t$.

Then, accounting for the simultaneous experiment and non-anticipating outcomes, we define the potential outcomes of the marketplace at time $t$ as \[ Y_t(\bm{w}_t, \bm{w}^\mathrm{s}_{t}) \, , \] where $\bm{w}_t$ is a realization of $\bm{W}_t$ and $\bm{w}^\mathrm{s}_{t}$ is a realization of $\bm{W}^\mathrm{s}_{t}$.\footnote{Suppose both the primary and simultaneous interventions are not applied to times outside of the experiment, i.e., $w_t = 0$ and $w^\mathrm{s}_{t} = 0$ for $t \not \in [0,T]$. Therefore, there are no carryover effects from treatments outside of the experiment, $ \mathbb{R} \backslash [0,T]$, to outcomes that occurred within the experiment, $[0,T]$. It is then reasonable to define potential outcomes only using treatment assignments within the experiment. } In this definition, potential outcomes are indexed by the treatment assignments of both primary and simultaneous experiments. The marketplace outcome satisfies $Y_t = Y_t(\bm{W}_t, \bm{W}^\mathrm{s}_{t})$. Given treatment designs $\bm{W}_t$ and $ \bm{W}^\mathrm{s}_{t}$ and event occurrence time $t_i$, there is no randomness in $Y_{t_i}$ anymore, and the randomness in event outcome $Y^{(i)}$ purely comes from the measurement error $\varepsilon^{(i)}$.

Note that the definition above generalizes the standard, binary definition of potential outcomes under the stable unit treatment value assumption (SUTVA) in two aspects. First, this definition allows potential outcomes to be jointly affected by the primary and simultaneous interventions. Second, this definition allows for temporal interference, i.e., the potential outcome of $t$ is not only affected by the treatment status at $t$ but also the treatment assignments at earlier times.

Estimands

Post-experiment, the primary decision maker uses the observed event outcomes $\{Y^{(i)}\}_{i\in \{1,\cdots, n\} }$ and treatment assignments $\bm{W}$ to estimate the effect of the primary intervention. The estimated effect will then be used to decide whether to deploy the intervention indefinitely. The estimand of primary interest for making this decision is the {\it global average treatment effect} (GATE), which measures the difference in average outcomes over time when an intervention is deployed indefinitely (global treatment) versus when it is absent (global control). The GATE is formally defined as \[\delta^\mathrm{gate} = \int \delta^\mathrm{gate}_{t} f(t) dt \, , \] which is the average of the total treatment effect $\delta^\mathrm{gate}_{t}$ at time $t$ weighted by the event density $f(t)$. The total treatment effect $\delta^\mathrm{gate}_{t}$ at time $t$ is defined as \[ \delta^\mathrm{gate}_{t} = Y_{t}(\bm{W}_t = \bm{1}_t, \bm{W}_t^\mathrm{s} = \bm{0}_t) - Y_{t}(\bm{W}_t = \bm{0}_t, \bm{W}_t^\mathrm{s} = \bm{0}_t) \, ,\] where $\bm{1}_t$ and $\bm{0}_t$ denote the marketplace being in the treatment and control state for a time duration of $t$, respectively. In the definition of GATE, simultaneous intervention is held in the global control state. This definition makes sense when the primary decision maker is interested in the effect of the primary intervention, while holding other conditions as the status quo. In cases where the primary decision maker is aware of the other intervention and wants to condition them in the treatment state, we can consider alternative definitions of $\delta^\mathrm{gate}$ and analyze them analogously.

Besides GATE, the decision maker may also want to estimate the cumulative effect, which is the total effect from current and past treatments. Formally, the cumulative effect is defined as the treatment effect at time $t$ when the treatment is employed from time $t-\Delta t$ to $t$ \[ \delta^\mathrm{cum}_{t}(\Delta t) = Y_{t}(\bm{W}_t = (\bm{0}_{t-\Delta t}, \bm{1}_{\Delta t}), \bm{W}_t^\mathrm{s} = \bm{0}_t) - Y_{t}(\bm{W}_t = \bm{0}_t, \bm{W}_t^\mathrm{s} = \bm{0}_t) \, ,\] where the notation $(\bm{0}_{t-\Delta t}, \bm{1}_{\Delta t})$ concatenates $\bm{0}_{t-\Delta t}$ and $\bm{1}_{\Delta t}$, meaning that the marketplace is in the control state for a duration of $t-\Delta t$ and then in the treatment state for a duration of $\Delta t$. The definition of $\delta^\mathrm{cum}_{t}(\Delta t)$ is visualized in Figure (ref). It is easy to see that the cumulative effect $\delta^\mathrm{cum}_{t}(\Delta t)$ converges to the total treatment effect $\delta^\mathrm{gate}_{t}$ as the treatment duration $\Delta t$ grows to infinity, i.e., \[\delta^\mathrm{gate}_{t} = \lim_{\Delta t \rightarrow \infty} \delta^\mathrm{cum}_{t}(\Delta t) \, . \]

However, an infinite treatment duration may not be necessary for the cumulative effect to stabilize and converge to the total treatment effect. Gaining insights into both the necessary duration for convergence and the dynamics of the cumulative effect (i.e., the CEC) is valuable for the decision maker when designing and analyzing the switchback experiments.

Post-Experiment Estimation

In this paper, we primarily focus on analyzing the widely used Horvitz-Thompson (HT) estimator for the treatment effect horvitz1952generalization. The HT estimator of $\delta^\mathrm{gate}$ using the $n$ observed event outcomes and treatment design takes the form of

where \[\pi = \int_{t \in [0, T]} \+E[W_{t}] f(t) d t \] is the fraction of times in the treatment state. If time $t$ has 50% probability of being treated, then $\mathbf{P}(W_{t} = 1) = 1/2$ and $\pi = 1/2$. If an event occurs in a treated interval, then its outcome is used to estimate the average outcome under global treatment; otherwise, its outcome is used to estimate the average outcome under global control.

We focus on the HT estimator instead of alternative treatment effect estimators (discussed in Section (ref)) for four reasons. First, the implementation of the HT estimator is simple. It relies neither on assumptions about carryover and correlation mechanisms of event outcomes, nor on the knowledge of treatment assignments of simultaneous interventions. Second, as shown in our case study, the HT estimator generally performs better than alternative estimators. Third, the HT estimator is simpler to analyze than alternative estimators while still conveying important design insights to improve estimation efficiency. Fourth, our case study shows that these design insights can not only reduce estimation errors for the HT estimator but also for alternative estimators.

Empirical Bayes Switchback Design

Before the experiment starts, the decision maker chooses the number of intervals $M$ and the interval switching points $0 \leq \iota_{1} \leq \cdots \leq \iota_{M-1} \leq T$, aiming to reduce the estimation error of GATE, post-experiment. An important metric for the decision maker is the MSE of $\hat{\delta}^\mathrm{gate}$, defined as

where the expectation is taken with respect to the treatment designs $\bm{W}$ and $\bm{W}^\mathrm{s}$, the measurement errors in event outcomes $\varepsilon^{(1)}, \cdots, \varepsilon^{(n)}$, and the event occurrence times $t_1, \cdots, t_n$. Here we focus on the randomized designs, where each time period is equally likely to be treated or untreated, i.e., $\mathbf{P}(W_{t} = 1) = 1/2$ and $ \mathbf{P}(W^\mathrm{s}_t = 1) = 1/2$ for all $t$.\footnote{We focus on the $1/2$ probability because this probability is shown to be efficient in many design problems.}

In Section (ref), we provide the expression of MSE as a function of the interval endpoints, which essentially applies to any switchback design. The expression shows that the MSE depends on assumptions on carryovers, outcome covariance, periodicity, and simultaneous interventions.

We propose a new empirical Bayes approach for switchback design that leverages historical data to account for these four factors in choosing a design for a new experiment. This approach has four phases, as illustrated in Figure (ref). In phase one, we analyze historical data and model the data-generating process. Specifically, the event density is estimated and then can be used as the input for the design. The periodic pattern is analyzed and then can be used in design to balance the periodicity. In phase two, we estimate an empirical distribution of CECs from prior experiments. Although the CEC of the new intervention is not known ex-ante, a reasonable prior is the empirical distribution of CECs. In phase three, we choose a design by comparing the performance of candidate designs through synthetic experiments. These experiments are run on historical data, and the CEC of the synthetic intervention is randomly drawn from the empirical distribution of CECs. Such experiments ensure that all factors affecting the MSE are present. In phase four, we select the best-performed design in synthetic experiments. The selected design is referred to as the empirical Bayes design.

In this paper, we focus on empirically evaluating the following three types of heuristic designs, in which the decision maker only needs to choose two parameters, and the design problem is more tractable. The comparison of these three designs fleshes out the important design insights to improve estimation efficiency. Figure (ref) illustrates the switching points of the three designs under the periodic event density. The first type is the fixed duration switchback (Example (ref)), which has constant interval lengths and is usually the design used in practice.

The second type is the Poisson duration switchback (Example (ref)), where the interval length is random and generated by the Poisson distribution. The Poisson duration switchback has similar, but slightly different, interval lengths compared to the fixed duration switchback. We study this design motivated by our finding in the MSE decomposition that randomizing interval lengths can reduce the error from the confounding effect of simultaneous interventions.

The third type is the change-of-measure switchback (Example (ref)), which has constant interval lengths after changing the measure of nonuniform event density to uniform density. Under the uniform event density, the change-of-measure switchback is the same as the fixed duration switchback. However, under the periodic density, the change-of-measure design has much shorter interval lengths in times of high density and much longer interval lengths in times of low density than in the other two designs. We can determine the interval lengths in change-of-measure switchback using empirical density estimated from historical data. We study this design because we find that such a pattern of interval lengths can reduce an important variance term in MSE.

In addition, we consider the following balanced design that imposes restrictions on the randomness of treatment assignments. This balanced design is relevant only if we explicitly account for the periodicity in the analysis and design, which is a key distinction from prior work on switchback designs. We study this design because, based on the MSE decomposition, we find that balancing is very effective for variance reduction, especially when the relative size of the treatment effect compared to noise is small.

In the example of a two-week experiment, when a balanced randomized switchback is used, the treatment assignments of the second week are opposite to those of the first week. As the heterogeneities in the outcomes tend to have a periodic pattern by week, the balanced design in a two-week experiment creates matched pairs for the same time in a week.\footnote{Balancing the periodicity is found effective in reducing variance in the switchback experiments via multiple randomization designs masoero2023efficient and nonstationary a/b tests wu2022non. } We note that the balancing approach in Example (ref) is used in practice and studied in this paper for illustration purposes. One could consider more complex balanced designs, such as balancing the treatment assignments of two consecutive days, hours, etc., while the insights would generally be the same.

\setcounter{equation}{0} {\thesection.\arabic{equation}} \oldsection{A Case Study on a Ride-Sharing Platform}

In this section, we analyze historical data from a ride-sharing platform and explore strategies for designing more efficient experiments. We have access to two sets of historical data. The first data set consists of the event-level data of the top 50 regions between June 2022 and March 2023, referred to as the historical control data hereafter. The second data set includes the event-level data of a large corpus of experiments conducted between June 2021 and March 2023, hereafter referred to as the historical experimental data. In both data sets, each event represents a rider session started from opening the app. The outcome is binary, denoting whether the rider requested the ride ($Y_{t_i} = 1$) or not ($Y_{t_i} = 0$).

We illustrate our proposed empirical Bayes design approach on the historical data to select a switchback design. By comparing the performance of designs introduced in Section (ref), we identify a hierarchical structure of the effectiveness of design principles in reducing the estimation error.

Analysis of Historical Data

In this subsection, we show the estimated event density, mean control outcomes, and variance of measurement errors from historical control data. These estimates demonstrate the behavioral changes based on the time of day and day of the week. We also show the empirical distribution of CECs estimated from prior experiments.

Estimates from Historical Control Data

\paragraph{Event density}

The estimated event density $f(t)$ for every minute of a week is shown in Figure (ref). There are two main observations. First, the event density has a periodic pattern, with high density during the peak hours, such as 6 PM, and low density during the off-peak times, such as 3 AM. Second, the event density is higher during peak hours on weekends (Fridays, Saturdays, and Sundays) than during peak hours on weekdays.

\paragraph{Global control outcome}

The estimated and standardized global control outcome (i.e., $Y_t(\bm{0}_t, \bm{0}_t)$ subtracted by its mean and divided by its standard deviation) for every minute in a week is shown in Figure (ref). The average global control outcome has a periodic pattern and is generally higher during the daytime.

\paragraph{Heteroskedastic measurement errors}

The estimated and standardized standard error of measurement errors for every minute in a week is shown in Figure (ref). We estimate the standard errors by assuming that the binary event outcome is randomly drawn from a Bernoulli distribution, with the probability of getting one being the average conversation rate (i.e., global control outcome). The standard error is heteroscedastic and has a periodic pattern, similar to that of the event density and global control outcome. Notably, the standard error of measurement errors tends to be lower in times of high event density, and higher in times of low event density. This is because, during periods of high event density, the mean control outcome is generally larger and lies in a regime where the variance is negatively correlated with the mean control outcome.

Empirical Distribution of CECs from Prior Experiments

We analyze a large corpus of the experiments run between June 2021 and March 2023 to construct a prior on CEC. We have $149$ two-week experiments run across $114$ markets. Different experiments were run in different subsets of the markets. There are $890$ distinct experiment-market pairs in total. These experiments employ a balanced, fixed-duration switchback design with a constant interval length of $56$ minutes (referred to as status quo design).

We first estimate a CEC for each experiment-market pair by taking the difference in outcomes between treated and control intervals independently for every minute since the switch. Then, we obtain a $56$-dimensional CEC vector whose $j$-th entry is the average cumulative effect given that the market has been treated for $j$ minutes.

Figure (ref) shows several representative estimated CECs. A notable observation is that the estimated cumulative effects vary substantially minute-to-minute with the treatment duration, indicating a high variance in the treatment effect estimation. Therefore, to reduce variance, we smooth out the estimated CECs by fitting a natural cubic spline to the vector of points. The natural cubic splines are particularly suitable for incorporating our priors on the shape of CECs, including a low tendency of cumulative effects to vary minute-to-minute with treatment duration, and the convergence of cumulative effects to GATE at the right boundary. In Appendix (ref), we document and interpret the constraints imposed in curve fitting. The smoothed curves by natural cubic splines are overlaid on the (raw) estimated CECs in Figure (ref). The $890$ smoothed CECs constitute the empirical distribution of CECs.

Two important observations arise from the smoothed CECs. First, for $68\%$ of the smoothed CECs, cumulative effects change signs as the treatment duration changes. This suggests that the treatment may first be effective in increasing the conversion rate and then become ineffective. The opposite case, where the treatment is first ineffective and then becomes effective, is also likely to happen. Second, in most cases, the cumulative effects do not vary monotonically with the treatment duration. For example, the cumulative effects can first increase and then decrease, or vice versa, as the treatment duration increases. Overall, it often takes quite a bit of time for the cumulative effects to converge to GATE. Because of these observations, we further consider more persistent carryovers and fit the $112$-minute CECs, as shown in Figure (ref) in Appendix (ref). The two observations for the $56$-minute curves continue to hold for the $112$-minute curves.

To validate our model fitting procedure, we compare the cross-validation error of smoothing by the natural cubic spline with smoothing by alternative methods, such as polynomial regression and local regression. We document our cross-validation procedure in Appendix (ref), with the results shown in Figure (ref) in Appendix (ref). The cross-validation error of the natural cubic spline is comparable to all the alternative methods. As the natural cubic spline is well suited to incorporate our priors on the curve shape and can capture complex curve shapes, we focus on it in the main text. See Appendix (ref) for more discussion. As a robustness check, we have run synthetic experiments using the smoothed effects by alternative methods, and the best-performed design is generally robust to the choice of a flexible smoothing method.

Empirical Bayes Design through Synthetic Experiments

We illustrate our proposed empirical Bayes approach to choosing a switchback design by running synthetic experiments on historical data. Decision makers can use a similar approach to select specific switchback designs tailored to their specific settings.

Setup of Synthetic Experiments

We consider the following six switchback designs to run two-week synthetic experiments: fixed duration (FD), balanced fixed duration (bal. FD), change-of-measure (CM), balanced change-of-measure (bal. CM), Poisson duration (Poisson), and balanced Poisson duration (bal. Poisson) switchback designs. For each design, we vary the average interval length across three specifications: $28$, $56$, and $112$ minutes. In the Poisson duration switchback, the average interval length corresponds to the mean parameter $\lambda$. In the change-of-measure design, the average interval length equals to the event occurrence probability in an interval multiplied by the experiment duration.

We conduct two types of synthetic experiments. In the first type, no other experiment is run simultaneously with the primary synthetic experiment. To implement this type, we randomly draw a market from the $50$ markets in the historical control data and then randomly draw two consecutive weeks of historical control data in this market. In the second type, one experiment starts and ends simultaneously with the primary synthetic experiment. To implement the second type, we randomly select one experiment-market pair and use its two-week historical experimental data. In both types, we assume that the primary synthetic intervention has not been applied to the historical (control or experimental) data.

Next, we randomly draw one CEC from the empirical distribution of CECs, and assume that the cumulative effects of the synthetic intervention follow this drawn curve. Then we use this curve to generate the two-week synthetic experimental data. Specifically, given a switchback design and every time point, we calculate the total effect from treatments received on and prior to that time point. Then we add the total effect to the two weeks of historical data drawn from the previous step, and obtain the synthetic experimental data.

Finally, we apply the HT estimator to the synthetic experimental data to estimate GATE and compute the estimation error of GATE using the drawn curve. We repeat this procedure and obtain the estimation error for each of the six switchback designs with each of the three average interval lengths, given a draw of two weeks of historical data and a CEC. We repeatedly draw the two-week historical data and CEC for $500$ times in total. Then for each switchback design with each average interval length, we obtain $500$ estimation errors in total.

Results

Figure (ref) visualizes the estimation error of GATE for each switchback design with each average interval length when one experiment runs simultaneously. If we use the balanced Poisson duration switchback with an average interval length of $112$ minutes, then the MSE is reduced by $33$% compared to the status quo fixed-duration switchback design with an interval length of $56$ minutes. Figure (ref) in Appendix (ref) shows the corresponding results with no experiment running simultaneously. Based on the results in Figures (ref) and (ref), we summarize the effectiveness of various design principles in reducing the estimation error into a hierarchical structure, as shown in Figure (ref).

As shown in Figures (ref) and (ref), balancing periodicity, achieved by using balanced switchback designs in Example (ref), is the most effective in reducing the estimation error. This observation indicates that variance from periodicity is a major source of error and can be reduced using a balanced design.

Carefully choosing the average interval lengths can also effectively reduce the estimation error. The effect is two-fold. On the one hand, when switchback designs are not balanced, fast switching reduces variance and then reduces the estimation error because variance is the major source of error for an unbalanced design. On the other hand, when switchback designs are balanced, long switching can reduce the chance of a large error and make the designs more robust to various shapes of CEC. We also note that when an experiment starts and ends simultaneously, the $56$-minute interval length can have a larger MSE than the $28$-minute one. This is because of the confounding effect of the simultaneous experiment. The reason is supported by Figure (ref) in Appendix (ref) that, without the simultaneous experiment, the $56$-minute interval length then has a lower error.

After the average interval lengths are selected, choosing the exact switching times can further reduce the MSE. A simple approach is to choose between fixed duration, change-of-measure, and Poisson duration designs depending on our prior for CEC. Figure (ref) shows that the balanced Poisson switchback with long interval lengths has smaller MSE than other balanced designs, when the prior follows the empirical distribution of CECs. The results indicate that randomizing switching times (e.g., through Poisson switchback) can mitigate the confounding effects of simultaneous experiments, which can effectively reduce the estimation error in practice.

We also directly compare results with and without one simultaneous experiment in Figure (ref) in Appendix (ref). The comparison shows that the presence of a simultaneous experiment generally increases the estimation error. The increase is more obvious if the primary and simultaneous experiments use the same $56$-minute interval length. Therefore, when impacts from simultaneous experiments are nonnegligible, it can be important to use designs that mitigate confounding effects. For example, effective approaches include choosing different interval lengths, staggering switching times across experiments, or launching experiments at different times.

Finally, we run synthetic experiments using the $112$-minute cumulative effect curves smoothed by natural cubic splines. As shown in Figure (ref) in Appendix (ref), the Poisson duration switchback with an average interval length of $112$ minutes continues to have the lowest MSE, and the hierarchical structure continues to hold.

\setcounter{equation}{0} {\thesection.\arabic{equation}} \oldsection{Analysis of Switchback Design}

In this section, we present the precise decomposition of the bias and MSE for any switchback design. The decomposition shows how the four factors affect the MSE and provides the theoretical foundation to explain how various switchback designs affect the MSE.

In Section (ref), we first lay out the assumptions required for the decomposition. Subsequently, we introduce several interval-level statistics in Section (ref). These interval-level statistics are then used in the decomposition of bias and MSE in Section (ref). The decomposition in this section assumes only one simultaneous intervention, but this is only for the purpose of exposition. The general results with multiple simultaneous interventions are stated in Appendix (ref), and the proof in Appendix (ref) allows for multiple simultaneous interventions.

Assumptions

We begin by assuming that the sampling of events is exogenous and independent of the treatment assignments of both the primary and simultaneous interventions. This assumption holds true for interventions that potential riders cannot discern a difference before opening the app and checking prices, and consequently do not affect the event density, such as changes to pricing or matching algorithms.

Moreover, for the purpose of exposition, we impose an assumption on the structure of the treatment effects. This assumption is based on the observation that the treatment effects can be decomposed as

where $e_t$ is a one-hot-encoded vector with the entry of time $t$ to be $1$ and all the remaining entries to be $0$, $\delta^\mathrm{inst}_{t}$ is the {\it instantaneous effect} of treatment at time $t$ on the outcome at time $t$, and $\delta^\mathrm{co}_{t}(\bm{w}_t)$ is the {\it carryover effect} of treatment assignments $\bm{w}_t$ prior to time $t$ on the outcome at time $t$. For notation simplicity, we let $\delta^\mathrm{co}_t \coloneqq \delta^\mathrm{co}_{t}(\bm{1}_t)$ be the carryover effect under global treatment. Then, for a sufficiently large $t$ and with a sufficiently long time in the treatment state, the total treatment effect has $\delta^\mathrm{gate}_t = \delta^\mathrm{inst}_t + \delta^\mathrm{co}_t$.

This assumption, as stated in Assumption (ref) below, imposes a structure on carryover effects; that is, they are additive and can be parameterized by a carryover kernel. This assumption is for exposition, which can be relaxed at the cost of more complex notations in the main result, while the insights are generally the same.

The carryover kernel $d^\mathrm{co}_{t}(t^\prime)$ can be quite general. Below, we provide two examples of commonly used carryover kernels in practice. In the case of the non-anticipating outcome, $d_{t}(t') = 0$ for all $t' > t$. Moreover, if the treatment assignments can only affect the outcomes for a duration of $h < \infty$ in the future (as assumed in bojinov2020design,basse2019minimax,xiong2019optimal among others)), then $d_{t}(t') = 0$ for all $t' < t - h$ and can take arbitrary values for $t - h \leq t^\prime \leq t$. Alternatively, if the duration is infinity, but carryover effects decay at a geometric rate (as assumed in hu2022switchback), then $d^\mathrm{co}_{t}(t^\prime)$ decays with $\exp[-(t - t^\prime)]$ for all $t^\prime < t$.

Below, we introduce a condition under which the effects of primary and simultaneous interventions are additive. This represents a special case of the confounding between primary and simultaneous interventions. We show how this condition simplifies the decomposition of MSE and makes the decomposition more interpretable.

This condition excludes intervention effects from being synergistic (combining two interventions leads to a larger effect than expected) or antagonistic (combining two interventions leads to a smaller effect than expected). Condition (ref) is reasonable for certain classes of distinct interventions; for example, we may often assume that a pricing change and a routing change act via different mechanisms and are thus additive.

Interval-Level Statistics

We introduce several interval-level statistics that quantify carryover effects, correlations in measurement errors, confounding effects from simultaneous interventions, and other components at the interval level. These interval-level statistics serve as building blocks of the bias and MSE decomposition in Section (ref).

\paragraph{Fraction of events.}

Let \[\mu^{(m)} = \int_{t \in \mathcal{I}_{m} } f(t) d t \] be the fraction of events occurring in the interval $\mathcal{I}_{m}$. $\mu^{(m)} $ ranges from 0 to 1 and the sum of $\mu^{(m)}$ over $m$ equals to $1$.

\paragraph{Mean control outcome.} Let

\[ \mu^{(m)}_{Y^\mathrm{ctrl}} = \int_{t \in \mathcal{I}_{m}} Y_{t}(\bm{0}_t, \bm{0}_t) f(t) dt\] be the integrated global control outcome $Y_{t}(\bm{0}_t, \bm{0}_t)$ over time $t$ in the interval $\mathcal{I}_{m}$.

\paragraph{Variance and covariance of measurement errors.} Let the integrated variance of the measurement error for events in the interval $\mathcal{I}_{m}$ be \[V^{(m)} = \int_{t_i \in \mathcal{I}_{m} } \+E_{\varepsilon}\left[(\varepsilon^{(i)})^2 \mid t_i \right] f(t_i) d t_i \, , \] where $\+E_{\varepsilon}\left[(\varepsilon^{(i)})^2 \mid t_i \right]$ represents the variance of measurement error for event $i$ occurring at time $t_i$ (as measurement error has mean zero).

Next, let the integrated covariance between measurement errors of events in the interval $\mathcal{I}_{\ell m}$ be \[C^{(m)} = \int_{t_i, t_j \in \mathcal{I}_{m} } \+E_{\varepsilon}\left[\varepsilon^{(i)} \varepsilon^{(j)} \mid t_i, t_j \right] f(t_i) f(t_j) d t_i d t_j \, ,\] where $\+E_{\varepsilon}\left[\varepsilon^{(i)} \varepsilon^{(j)} \mid t_i, t_j \right] $ is the covariance between the measurement errors of event $i$ occurred at time $t_i$ and event $j$ occurred at time $t_j$.

In practical settings, patterns often exist in how the covariance $\+E_{\varepsilon}\left[\varepsilon^{(i)} \varepsilon^{(j)} \mid t_i, t_j \right] $ varies with time $t_i$ and time $t_j$, e.g., decays monotonically or periodically with the distance between time $t_i$ and time $t_j$. Therefore, a kernel function can be used to parameterize and capture the patterns in $\+E_{\varepsilon}\left[\varepsilon^{(i)} \varepsilon^{(j)} \mid t_i, t_j \right] $. See examples in Figure (ref).

\paragraph{Integrated total treatment effects.}

Let the integrated total treatment effect over time $t$ in the interval $\mathcal{I}_m$ be

Following the definition of $\delta^\mathrm{gate}$, the sum of $\Xi^{(m)}$ over $m$ equals $\delta^\mathrm{gate}$. Moreover, if $\delta^\mathrm{gate}_{t}$ is constant in time $t$, then $\Xi^{(m)} = \delta^\mathrm{gate} \mu^{(m)}$ for any $m$.

\paragraph{Integrated carryover effects.} Let the integrated carryover effect of primary treatments in $\mathcal{I}_k$ to outcomes in $\mathcal{I}_m$ be \[I^{(m,k)} = \int_{t \in \mathcal{I}_{m}} \left[\delta^\mathrm{co}_{t} \int_{t^\prime \in \mathcal{I}_{k}} d^\mathrm{co}_{t}(t^\prime) f(t^\prime) d t^\prime \right] f(t) dt \, . \] For simplicity in notation, we let $I^{(m)} = I^{(m,m)}$ be the integrated carryover effect of treatments on outcomes in the same interval. The integrated carryover effect $I^{(m,k)}$ increases with the length of both $\mathcal{I}_{m}$ and $\mathcal{I}_{k}$, and increases with the size of carryover effect $\delta^\mathrm{co}_{t}$ for time $t \in \mathcal{I}_{m}$. The sum of $I^{(m,k)}$ over both $m$ and $k$, which is the integrated carryover effect of primary treatments of all intervals on the outcomes of all intervals, is equal to the average of carryover effect $\delta^\mathrm{co}_t$ over time $t$ weighted by event density $f(t)$ (which is denoted by $\delta^\mathrm{co}$ hereafter). Moreover, if the carryover effect $\delta^\mathrm{co}_t$ is constant in $t$, then the sum of $I^{(m,k)}$ over $k$, which is the integrated carryover effect of the treatment of all intervals on the outcomes in the interval $\mathcal{I}_{m}$, is equal to $\delta^\mathrm{co} \mu^{(m)}$. Therefore, $I^{(m,k)}$ can be viewed as the building block of $\delta^\mathrm{co}$.

\paragraph{Confounding effects from the simultaneous intervention.} For any time $t$, let \[\delta^\mathrm{simul}_t(\bm{W}_t) = \+E_{\bm{W}^\mathrm{s}_t}\left[Y_t(\bm{W}_t, \bm{W}^\mathrm{s}_t) - Y_t(\bm{W}_t, \bm{0}_t) \mid \bm{W}_t, t \right] \] be the expected treatment effects from the simultaneous intervention at time $t$, conditional on $\bm{W}_t$. Here, the expectation is taken with respect to the distribution from which the treatment design of the simultaneous intervention is drawn. If the simultaneous intervention has nonzero treatment effects, then $\delta^\mathrm{simul}_t(\bm{W}_t)$ is generally nonzero, which can then bias the HT estimator of $\delta^\mathrm{gate}$.

We introduce a quantity below that measures the integrated bias from the treatment effects of simultaneous intervention \[S^{(m)} = \int_{t \in \mathcal{I}_{m}} \Phi^\mathrm{simul}_t f(t) dt \, , \] where $\Phi^\mathrm{simul}_t$ is defined as \[\Phi^\mathrm{simul}_t = \+E_{\bm{W}^{(-m)} }\big[ \delta^\mathrm{simul}_t(\bm{W}^{(-m)}, W^{(m)} = 1 ) \big] - \+E_{\bm{W}^{(-m)} }\big[ \delta^\mathrm{simul}_t(\bm{W}^{(-m)}, W^{(m)} = 0 ) \big] \, .\] Here $W^{(m)}$ and $\bm{W}^{(-m)}$ are the treatment assignments of interval $\mathcal{I}_m$ and of all the intervals excluding $\mathcal{I}_m$, respectively. As treatments are assigned at the interval level, $\bm{W}_t$ is uniquely determined given $(W^{(m)},\bm{W}^{(-m)})$, so we can also write $\delta^\mathrm{simul}_t(\bm{W}_t)$ as $\delta^\mathrm{simul}_t(\bm{W}^{(-m)}, W^{(m)})$.

The quantity $\Phi^\mathrm{simul}_t$ measures the imbalance of expected treatment effects from the simultaneous intervention at time $t$ between when the $m$-th interval is treated ($W^{(m)} = 1$) versus when this interval is not treated ($W^{(m)} = 0$). Here, the expectation is taken with respect to the distribution from which $\bm{W}^{(-m)}$ is drawn. If the imbalance $\Phi^\mathrm{simul}_t$ is larger, then the integrated bias from simultaneous interventions $S^{(m)}$ is generally larger.

Note that $\Phi^\mathrm{simul}_t$ can be zero in some special cases, such as when the effects of primary and simultaneous interventions are additive (Condition (ref) holds), as illustrated in the following example.

However, when the effects of primary and simultaneous interventions are not additive (e.g., two interventions are useful only when both are present), $\Phi^\mathrm{simul}_t$ is generally nonzero and is a source of bias of the HT estimator.

Main Results

We provide the decomposition of the bias and MSE of $\hat{\delta}^\mathrm{gate}$ from the HT estimator in terms of the interval-level statistics in Theorems (ref) and (ref) below. The decomposition shows how different components in the outcome affect the estimation error of $\hat{\delta}^\mathrm{gate}$. In the theorem statements, we use $W^{\mathrm{s}(m)}$ to denote the $m$-th interval's treatment assignment of the simultaneous intervention.

Theorem (ref) shows there are two sources of bias in the HT estimator. The first source of bias comes from the carryover effects and is measured by $\operatorname{Bias}(\mathcal{E}_\mathrm{carryover})$. If carryover effects are zero, (i.e., $\delta^\mathrm{co}_{t} = 0$ for all time $t$), then $\operatorname{Bias}(\mathcal{E}_\mathrm{carryover}) = 0$. If carryover effects are nonzero, then $\operatorname{Bias}(\mathcal{E}_\mathrm{carryover})$ quantifies the bias in the HT estimator arising from using direct treated and control outcomes to approximate globally treated and control outcomes, respectively.

The second source of bias comes from the confounding effects of simultaneous interventions and is measured by $\operatorname{Bias}(\mathcal{E}_\mathrm{simul})$. When the effects of main and simultaneous interventions are additive, as illustrated in Example (ref), $S^{(m)}$ is zero and then $\operatorname{Bias}(\mathcal{E}_\mathrm{simul})$ is also zero. However, when effects are not additive, $\operatorname{Bias}(\mathcal{E}_\mathrm{simul})$ is generally nonzero and tends to increase with the effect size of the simultaneous intervention.

Both sources of bias can be reduced by properly choosing a switchback design. To mitigate the bias from carryover effects, switching less frequently helps. For example, in the setting of uniform event density and fixed-duration switchback, $I^{(m)}=\delta^\mathrm{co}(1/M - h/(2T))$ as shown in Example (ref), and then the carryover bias is $|\operatorname{Bias}(\mathcal{E}_\mathrm{carryover})| = \delta^\mathrm{co} Mh/(2T)$, which increases with the number of intervals $M$. To reduce the bias from simultaneous experiments, randomizing and staggering the switching times helps, such as by using Poisson duration switchback. This can be clearly seen in the toy Example (ref) in Appendix (ref) and simulation results in Figure (ref) below.

Next, we show the decomposition of the MSE of $\hat{\delta}^\mathrm{gate}$.

Theorem (ref) demonstrates that, in addition to the bias terms, the MSE is affected by three sources of variance. The first source of variance arises from the randomness in event measurement errors and is quantified by $\operatorname{Var}(\mathcal{E}_\mathrm{meas})$. Note that $\operatorname{Var}(\mathcal{E}_\mathrm{meas})$ consists of two parts: the first part $V^{(m)}$, which measures the variance of event measurement error, and the second part $C^{(m)}$, which measures the covariance between measurement errors of two events. As the number of events grows, the first part diminishes and the second part dominates. If the correlation in measurement errors is persistent, then $C^{(m)}$ is larger, leading to a larger MSE.

The second source of variance stems from the randomness in treatment assignments of the primary intervention and is measured by $\operatorname{Var}(\mathcal{E}_\mathrm{inst}+\mathcal{E}_\mathrm{carryover})$. The expression of $\operatorname{Var}(\mathcal{E}_\mathrm{inst}+\mathcal{E}_\mathrm{carryover})$ shows that: (1) it increases with the size of the instantaneous effect, following that the term $\Xi^{(m)}$ increases with the instantaneous effect; (2) it increases with the size of the carryover effect, following that both $\Xi^{(m)}$ and $I^{(m,m^\prime)}$ increase with the carryover effect; and (3) it increases with the scale of the mean outcome, following that the term $\mu^{(m)}_{Y^\mathrm{ctrl}} $ increases with the mean outcome.

The third source of variance arises from the randomness in treatment assignments of the simultaneous intervention and affects $\+E[\mathcal{E}_\mathrm{simul}^2]$. In Proposition (ref) below, we present the expression for $\+E[\mathcal{E}_\mathrm{simul}^2]$ under the condition of additive primary and simultaneous effects. This expression shows that this source of variance increases with the magnitude of instantaneous and carryover effects of simultaneous intervention. Furthermore, this term increases with the overlap between intervals of the primary and simultaneous interventions.

In addition to the bias and variance terms, the MSE includes a cross term $\+E[(\mathcal{E}_\mathrm{inst}+\mathcal{E}_\mathrm{carryover}) \cdot \mathcal{E}_\mathrm{simul}] $. Proposition (ref) also provides the expression for this term under the condition of additive primary and simultaneous effects.\footnote{A preliminary version of this proposition appears in xiong2023bias. Theorems (ref) and (ref) are more general as they both allow for non-additive effects of the primary and simultaneous interventions. } Generally, if the variance terms from primary and simultaneous interventions are larger, then this cross term is larger.

The MSE decomposition makes it clear how the design affects the MSE. First and foremost, the switching interval length has a mixed effect on MSE. Switching frequently reduces most variance and cross terms because the number of interval-level observations increases, thereby increasing the effective sample size.

This can be seen clearly by analyzing the order of the terms in the setting of uniform event density and fixed-duration switchback. Specifically, in this setting, $\operatorname{Var}(\mathcal{E}_\mathrm{meas}) = O(1/M)$ decreases with $M$, following that $C^{(m)} = O(1/M^2)$ as shown in Example (ref). Moreover, the term $\sum_{m = 1}^M \big(\Xi^{(m)} + 2 \mu^{(m)}_{Y^\mathrm{ctrl}}\big)^2 = O(1/M)$ also decreases with $M$, given that $\Xi^{(m)} = O(1/M)$ and $\mu^{(m)}_{Y^\mathrm{ctrl}} = O(1/M)$. Analogously, for the variance from the simultaneous intervention, both $\+E[\mathcal{E}_\mathrm{simul}^2] $ and $\+E[(\mathcal{E}_\mathrm{inst}+\mathcal{E}_\mathrm{carryover}) \cdot \mathcal{E}_\mathrm{simul}]$ are $O(1/M)$ under the additivity condition.

On the other hand, switching frequently increases the carryover bias. This is because the carryover effects from treatments in other intervals increase with switching frequency. Therefore, with the tradeoff involved, the optimal value of $M$ depends on the relative size of instantaneous and carryover effects, the scale of the global control outcomes, and the duration and mechanism of carryover effects.

Second, the choice of interval endpoints also affects the MSE. To see this more clearly, consider the setting with constant $\delta^\mathrm{gate}_{t}$ in $t$ and constant $Y_t(\bm{0}, \cdots, \bm{0})$ in $t$ (equals to $\bar{Y}^{\mathrm{ctrl}}$). Then $\sum_{m = 1}^M \big(\Xi^{(m)} \big)^2 = (\delta^\mathrm{gate} + 2 \bar{Y}^{\mathrm{ctrl}})^2 \sum_{m = 1}^M \big( \mu^{(m)} \big)^2$, which is minimized at $\mu^{(m)} = 1/M$ for all $m$ with the same fraction of events in each interval. This implies that when event density $f(t)$ varies with $t$, this term can be reduced by switching more frequently in times of high event density and less frequently in times of low event density. The change-of-measure switchback has such switching patterns and is therefore considered in our case study.

In addition, the second term of $\+E[\mathcal{E}_\mathrm{simul}^2] $ is affected by how much the intervals of primary and simultaneous interventions overlap, and is the largest when the interval endpoints of primary and simultaneous interventions are the same. Therefore, randomizing and staggering the switching times of different interventions can reduce $\+E[\mathcal{E}_\mathrm{simul}^2] $. This then motivates us to consider the Poisson switchback in our case study.

Last but not least, balancing periodicity can reduce the MSE. Recall the setting of a two-week experiment with the balanced design in Example (ref). Under mild assumptions, the mean control outcome $\mu^{(m)}_{Y^\mathrm{ctrl}}$ can be canceled out in both the variance term $\operatorname{Var}(\mathcal{E}_\mathrm{inst}+\mathcal{E}_\mathrm{carryover}) $ and the cross term $\+E[(\mathcal{E}_\mathrm{inst}+\mathcal{E}_\mathrm{carryover}) \cdot \mathcal{E}_\mathrm{simul}] $. The variance term is then equal to \[\operatorname{Var}(\mathcal{E}_\mathrm{inst}+\mathcal{E}_\mathrm{carryover}) = \sum_{m = 1}^M \left(\Xi^{(m)} \right)^2 + \sum_{m=1}^M \sum_{m^\prime \neq m} \left(\left[I^{(m,m^\prime)}\right]^2 + I^{(m,m^\prime)} I^{(m^\prime, m)}\right) \, .\] Furthermore, under Condition (ref), the cross term is equal to

Canceling out $\mu^{(m)}_{Y^\mathrm{ctrl}}$ can substantially reduce the variance when the relative treatment effects are small (not more than a few percent of the outcome mean). This is exactly the case in our case study on a ride-sharing platform. In fact, the HT estimator coincides with the Hajek estimator when the balanced designs are used, but this is not the case for unbalanced designs. The Hajek estimator is generally more efficient than the HT estimator; this gives another perspective as to why balanced designs are more efficient than unbalanced designs.

Inference of Treatment Effects

To draw inferences on the treatment effect, one approach is to run a randomization-based test for the sharp null of no effect from the primary intervention, holding the treatment assignments of the simultaneous experiment as the realized ones: \[\mathcal{H}_0 : Y_t(\bm{w}_t, \bm{W}^\mathrm{s}_t) = Y_t(\bm{0}_t, \bm{W}^\mathrm{s}_t ) \, , \qquad \forall \bm{w}_t ~\text{and}~ t \, . \] Under this sharp null hypothesis, the outcomes would have been identical to the observed ones regardless of what treatment assignments $\bm{w}_t$ are used in the primary experiment, and $\delta^\mathrm{gate} = 0$. Then we can calculate the $p$-value of this hypothesis using the same procedure as bojinov2020design. Specifically, we construct an empirical distribution of the estimated GATE under the sharp null and use it to compute the $p$-value. To construct the empirical distribution, suppose we draw the treatment designs and re-calculate the GATE for $J$ times. In the $j$-th iteration, let $\bm{W}^{[j]}$ be the drawn treatment design, which is used with the observed outcomes $Y^{(1)}, \cdots, Y^{(n)}$ to estimate $\delta^\mathrm{gate}$

We then use $\hat{\delta}^{\mathrm{gate},[1]}, \cdots, \hat{\delta}^{\mathrm{gate},[J]}$ from the $J$ iterations to estimate the $p$-value \[\hat{p} = \frac{1}{J} \sum_{j = 1}^J \bm{1}\left(|\hat{\delta}^\mathrm{gate}| \geq |\hat{\delta}^{\mathrm{gate},[j]}| \right) \, , \] where $\bm{1}(|\hat{\delta}^\mathrm{gate}| \geq |\hat{\delta}^{\mathrm{gate},[j]}| )$ is an indicator function that takes value of one if $|\hat{\delta}^\mathrm{gate}| \geq |\hat{\delta}^{\mathrm{gate},[j]}|$, and zero otherwise. Besides calculating the $p$-value, we can invert the randomization-based test similar to tritchler1984inverting to construct the confidence interval of $\hat{\delta}^\mathrm{gate}$ under the sharp null.

Alternative Estimators

It is possible to reduce the estimation error of GATE by using alternative estimators that leverage prior knowledge of carryover and correlation mechanisms and information on other interventions. Specifically, to reduce the carryover bias, we can specify and leverage the structure of the carryover mechanisms when estimating GATE. For example, a commonly made assumption is that the fixed-duration carryover effect, i.e., the effect of treatment carries over to no more than $h$ periods in the future, for some $h$, such as in Examples (ref) and (ref). Under this assumption, we can burn the first $h$ periods since switch (either from control to treatment, or from treatment to control) hu2022switchback, and use the following modified HT estimator

This estimator only uses the observations at times that have been in the treatment or control states for at least time $h$ and, therefore, encompasses a bias-variance tradeoff: It is unbiased under the assumption of a fixed-duration carryover effect, but it can have a larger variance due to a reduced sample size. In Figure (ref) in Appendix (ref), we compare the estimation error with various $h$ under the same synthetic experiment setup as that in Section (ref). The hierarchical structure shown in Figure (ref) continues to hold with various $h$ and the balanced Poisson design with long switches has the lowest MSE for all $h$. However, it turns out that the estimator with zero burn-ins, i.e., the HT estimator analyzed in this paper, has the lowest MSE regardless of which design is used. This empirical evidence supports our choice of estimator, and occurs because our data is noisy.

To reduce the variance from correlated outcomes, we can specify and use the structure of the correlation mechanisms to reweigh the event outcomes. Moreover, if we know the design of the simultaneous experiment, we can use the generalized least squares (GLS) estimators to jointly estimate the instantaneous and carryover effects for all interventions, while taking advantage of the inverse error covariance weighting. GLS can be more efficient but requires the correct model specification. Nevertheless, the design insights from the HT estimator can carry over to the GLS estimator, with a few numerical examples shown in xiong2019optimal.

Another possible future direction to improve estimation efficiency is to model the demand and supply, and use machine learning to estimate the demand and supply adam2023machine. The estimated model of demand and supply could be used as the input in the estimation of GATE.

\setcounter{equation}{0} {\thesection.\arabic{equation}} \oldsection{Simulation}

In this section, we compute values for error components in the MSE in the settings where error components are on a similar scale, fleshing out the interesting tradeoffs in designing a switchback experiment. The computation is based on Theorems (ref) and (ref). The results confirm the findings in our case study. We use fixed-duration designs in the base case to characterize the tradeoffs involved, and the general insights carry over to other heuristic designs. In the base case, a linear decay carryover kernel is used with a bandwidth of $h_{\mathrm{carryover}} = 60$ (i.e., the duration of carryover effects is 60 minutes). Moreover, a linear decay covariance kernel is used with a bandwidth of $h_{\mathrm{covariance}} = 60$ (i.e., two event outcomes are correlated only if they are within 60 minutes apart). Both the instantaneous and carryover effects are constant in time and equal to $\delta^\mathrm{inst}_{t} = 1$ and $\delta^\mathrm{co}_{t} = 1$ for all $t$. In the base case, there is no simultaneous experiment, and the primary experiment lasts one day, i.e., $T = 1,440$ minutes. We vary the parameters in the base case and illustrate how error components vary and affect the performance of various switchback designs.

Instantaneous and Carryover Effects Only

Figure (ref) illustrates the tradeoff between the bias from carryover effects, $\operatorname{Bias}(\mathcal{E}_\mathrm{carryover}) $, and variance from instantaneous and carryover effects, $\operatorname{Var}(\mathcal{E}_\mathrm{inst}+\mathcal{E}_\mathrm{carryover})$. If the carryover effects last longer, i.e., $h_{\mathrm{carryover}}$ is larger, then the bias $\operatorname{Bias}(\mathcal{E}_\mathrm{carryover}) $ tends to be larger, and switching less frequently reduces the MSE from instantaneous and carryover effects, as shown in the comparison between Figures (ref) and (ref). The relative size between instantaneous and carryover effects also matters for the tradeoff. If the instantaneous effect is relatively larger than the carryover effect, then switching more frequently reduces the MSE, as shown in the comparison between Figures (ref) and (ref).

Total Treatment Effects with Measurement Errors

Figure (ref) summarizes the tradeoffs between error components from randomness in treatment assignments and measurement errors. Switching frequently generates more comparisons, reducing variance from measurement errors and increasing carryover bias from previous intervals. If correlation in measurement errors are persistent (i.e., large $h_{\mathrm{covariance}}$), then $\operatorname{Var}(\mathcal{E}_\mathrm{meas}) $ has a larger impact on the MSE, and switching more frequently helps to reduce the MSE, as shown in the comparison between Figures (ref) and (ref). Moreover, if event outcomes are noisier with a larger $\operatorname{Var}_{\sigma,t}$, then switching more frequently helps, as shown in the comparison between Figures (ref) and (ref).

Simultaneous Interventions

Figure (ref) shows the tradeoffs involved in the presence of simultaneous interventions. The MSE of the primary intervention is affected by three factors: the number of simultaneous interventions $K$, switching frequency, and the offset in switching times between simultaneous experiments. When more experiments are run simultaneously, the optimal switching frequency increases. This is because switching more frequently helps reduce the confounding effects of simultaneous interventions. Moreover, properly staggering the switching times of the primary and simultaneous interventions also decreases the MSE of the primary intervention, with the effect being more pronounced with a low switching frequency due to the increased finite-sample correlation between the designs. As shown in Figure (ref), Poisson switchbacks, which stagger through randomizing switching times, can be more effective unless the fixed duration designs are explicitly staggered well.

Periodic Event Density

Figure (ref) shows results from a periodic density using a fixed duration switchback. When the design has a period that aligns with density, the offset parameter $q$ determines how the alignment alters the bias and variance. Switching at times of high density yields a design with low variance from measurement errors $\operatorname{Var}(\mathcal{E}_\mathrm{meas}) $. On the other hand, switching at times of low density reduces carryover bias from the preceding interval $\operatorname{Bias}(\mathcal{E}_\mathrm{carryover}) $ and reduces MSE from treatment effects. Combining measurement errors and treatment effects, the optimal switching times are somewhere in between high and low density times. Therefore, knowledge of the density of events can improve the efficiency of the design by leveraging the best absolute times for bias- or variance-minimizing switching points.

\setcounter{equation}{0} {\thesection.\arabic{equation}} \oldsection{Discussion and Conclusion}

This paper studies the design and analysis of simultaneous switchback experiments from both a theoretical and practical point of view. We provide a theoretical analysis of how the bias and variance of the HT estimator of the GATE are affected by four factors: carryover effects, periodicity, correlations in event outcomes, and effects of simultaneous interventions. Empirical and simulation studies show how these factors trade off each other and provide insights into how one can design efficient switchback experiments. Based on these insights, we propose an empirical Bayes design that utilizes historical data to model and select the parameters of the new experiment's design.

Perhaps the most general conclusion we can draw is that designing experiments in this context involves considering a complex set of tradeoffs and critically depends on the assumptions experimenters would make using prior knowledge. Our case study on a ride-sharing platform illustrates this point, showcasing the value of leveraging prior experiments in the empirical Bayes approach when designing new experiments.

While we motivate this study by applications in the ride-sharing setting, the theory and practical guidelines presented can find broader applications in other contexts. Indeed, various settings exist where cross-sectional interventions are not possible or outcomes cannot be easily attributed to treatment decisions. Estimating the effectiveness of traditional media advertising aligns well with our problem setup, and a privacy-friendly approach to online advertising might employ temporal variation in campaign spending linked to sales through timestamps only. Prior work has explored time-varying interventions in financial or cryptocurrency markets krafft2018experimental or in self-experimentation for personalized medicine karkar2016framework. An important goal of this work is to expand the use of time-based experiments to settings where they are not currently used.

\bookmarksetup{startatroot}