Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
47,525 characters · 19 sections · 15 citation commands
TEA-Time: Transporting Effects Across Time
Causal effects identified by randomized controlled trials (RCTs) are inherently local—not only to the population studied but also to the time at which the trial was conducted. A marketing intervention tested during summer may perform differently in holiday season; a job training program evaluated during economic expansion may have different effects during recession; a drug's efficacy may vary with seasonal disease prevalence. Despite the ubiquity of such temporal variation, the causal inference literature has devoted considerably more attention to transporting effects across populations than across time.
This paper develops a framework for temporal transportation: using data from trials conducted at one time to estimate treatment effects at different times. We consider settings where an organization has access to multiple historical RCTs—potentially comparing different interventions—and seeks to predict what the effect of a specific treatment would be if administered at a new target time. This problem arises naturally in e-commerce (where promotional effects exhibit strong seasonality), digital advertising (where effectiveness varies with attention cycles), labor market policy (where program effects depend on market tightness), and clinical trials (where drug efficacy may vary with seasonal disease prevalence).
The challenge of temporal transportation differs fundamentally from cross-population generalization. When transporting across populations, we typically observe covariates in both source and target and can reweight to adjust for distributional shifts dahabreh2019generalizing, stuart2011use. When transporting across time, we cannot observe outcomes under the target timing—by definition, we are extrapolating to a period where the relevant trial was not conducted. Identification therefore requires structural assumptions about how treatment effects vary with time.
Our key insight is that other trials --- comparing potentially different treatments --- can serve as temporal anchors. Figure (ref) illustrates the core idea. Suppose we observe a primary trial comparing treatments $a$ versus $b$ at a source time, and wish to estimate what this effect would be at a target time. If anchor trials testing other treatment comparisons were conducted at both source and target times, we can use them to learn how outcomes or effects evolve temporally.
\paragraph{Contributions.} Our paper makes three contributions:
First, we formalize the temporal transportation problem and introduce the transported average treatment effect (TATE) as the target estimand. Under a separable temporal effects assumption—a multiplicative decomposition of potential outcomes into unit-specific and time-specific components—we show that the TATE equals the product of an observed average treatment effect and an identifiable temporal ratio.
Second, we provide two identification strategies with distinct data requirements. The replicated trials strategy uses pairs of trials comparing identical treatments at different times, permitting flexible temporal structures but requiring exact replication. The common arm strategy identifies the temporal ratio from any treatment arm observed at multiple times—such as a control arm appearing across many trials—but imposes a stronger restriction that temporal effects operate only through measurement time.
Third, we develop doubly robust estimators for the TATE under each strategy. Our estimators are consistent if either the outcome model or the propensity scores are correctly specified, and achieve the semiparametric efficiency bound when both are correct. For the common arm strategy with multiple anchors, we derive the optimally weighted combination.
We evaluate our methods through simulations and an application to over 22,000 A/B tests from the Upworthy Research Archive matias2021upworthy. The simulations confirm near-nominal coverage, with the common arm strategy achieving approximately 50% lower RMSE when its assumptions hold. The empirical application reveals a variance-bias tradeoff: the common arm approach offers greater precision but incurs bias when the temporal factor depends on both intervention and measurement times.
\paragraph{Notation.} Let $\mathcal{T}$ denote discrete time periods, $\mathcal{P}$ a population of units, and $\mathcal{A} \cup \{0\}$ the set of treatments with $0$ as control. We observe a collection of RCTs $\mathcal{K}$. Each trial $k \in \mathcal{K}$ draws a sample $\mathcal{S}_k \subset \mathcal{P}$ of $n_k$ units, compares treatment $a_k$ against baseline $b_k$, administers treatment at time $t_{0k}$, and measures outcomes at time $t_{1k} \geq t_{0k}$. Units in trial $k$ are randomly assigned to $A \in \{a_k, b_k\}$. Let $X$ denote pre-treatment covariates and $S \in \mathcal{K}$ denote trial membership.
Define $Y_{t_1}(a, t_0)$ as the potential outcome under treatment $a$ administered at $t_0$ and measured at $t_1$. The observed outcome for a unit in trial $k$ is $Y_{t_{1k}} = Y_{t_{1k}}(A, t_{0k})$.
\paragraph{Standard assumptions.} We maintain standard experimental assumptions: (i) random assignment: $A \perp \{Y_{t_1}(a, t_0)\}_{a, t_0, t_1} \mid S = k$ for each trial $k$; and (ii) SUTVA: no interference between units and no hidden treatment versions.
\paragraph{Separable temporal effects.} To enable temporal transportation, we impose multiplicative structure on how outcomes vary with timing.
This structure is analogous to a rank-one factor model: $\theta_a(X)$ serves as unit-specific loadings and $\Lambda(t_0, t_1)$ as the time factor. Under Assumption (ref), the individual treatment effect $Y_{t_1}(a, t_0) - Y_{t_1}(b, t_0) = \tilde{\theta}_{a,b}(X) \cdot \Lambda(t_0, t_1)$, where $\tilde{\theta}_{a,b}(X) := \theta_a(X) - \theta_b(X)$.
\paragraph{Target estimand.} The transported average treatment effect (TATE) for trial $k^\star$ is $\tau(a_{k^\star},b_{k^\star},t_{0k^\star},t_{1k^\star},\delta_0, \delta_1) = \tau_{k^\star}(\delta_0, \delta_1) = $
representing the ATE for the trial $k^\star$ population had treatment occurred at time $t_{0k^\star} + \delta_0$ with outcomes measured at $t_{1k^\star} + \delta_1$.
We establish identification of the TATE in two steps: first decomposing it into an observed ATE and a temporal ratio, then providing two strategies for identifying the ratio from auxiliary trials.
Our first result shows that under separable temporal effects, the TATE factors into two components: the treatment effect observed in the original trial and a ratio capturing how temporal conditions scale outcomes.
The intuition is straightforward. Under separability, both the observed and transported treatment effects share the same unit-specific component $\bar{\tilde{\theta}}_{a_{k^\star}, b_{k^\star}, k^\star} := \mathbb{E}[\tilde{\theta}_{a_{k^\star}, b_{k^\star}}(X) \mid S = k^\star]$, differing only in their temporal multipliers. Taking ratios cancels this common factor, leaving only the ratio of temporal modifiers.
Theorem (ref) reduces TATE identification to identifying the temporal ratio $\Lambda(t_{0k^\star} + \delta_0, t_{1k^\star} + \delta_1)/\Lambda(t_{0k^\star}, t_{1k^\star})$. We now present two strategies for achieving this, each with distinct data requirements and structural assumptions.
The first strategy identifies temporal ratios using trials that compare the same treatment pair at different times. The key insight is that if two trials test identical treatments but at different times, the ratio of their ATEs reveals how temporal conditions scale effects.
For treatments $a, b \in \mathcal{A} \cup \{0\}$ with $a > b$, define $\mathcal{K}(a, b) := \{\ell \in \mathcal{K} : (a_\ell, b_\ell) = (a, b)\}$ as the set of trials comparing $a$ against $b$, and $\mathcal{T}(a, b) := \{(t_{0\ell}, t_{1\ell}) : \ell \in \mathcal{K}(a, b)\}$ as the timing pairs at which this comparison was conducted.
The random sampling condition ensures that trial-specific effects reflect population quantities. The anchor pair conditions guarantee that ATE ratios are well-defined and informative. The temporal coverage condition requires that the anchor comparison was conducted at both the original and target times.
Under random sampling, both trials draw from the same population, so their average unit-specific effects are identical. Taking ratios of observed ATEs cancels this common factor, yielding the temporal ratio. Notably, the anchor pair $(a^*, b^*)$ need not equal the target comparison $(a_{k^\star}, b_{k^\star})$—any replicated comparison spanning the relevant times can serve as an anchor.
The second strategy identifies temporal ratios from a single treatment arm observed at multiple times. This approach is often more practical, as it does not require exact replication of treatment comparisons, but it imposes a stronger structural restriction.
This assumption is plausible when temporal variation reflects conditions at outcome measurement—such as seasonal demand patterns, economic conditions, or user engagement cycles—rather than conditions at treatment administration.
For treatment $c \in \mathcal{A} \cup \{0\}$, define $\mathcal{K}[c] := \{\ell \in \mathcal{K} : c \in \{a_\ell, b_\ell\}\}$ as trials where $c$ appears as either arm, and $\mathcal{T}^{(c)} := \{t_{1\ell} : \ell \in \mathcal{K}[c]\}$ as the measurement times at which $c$ was observed.
Note that Assumption (ref)(ii) requires non-zero mean outcome under $c^*$, rather than a non-zero treatment effect as in Strategy 1. This condition is typically weaker when outcomes are naturally positive.
Under Assumption (ref), the expected outcome for units receiving $c^*$ is $\mathbb{E}[Y_{t_1} \mid A = c^*] = \bar{\theta}_{c^*} \cdot \Lambda(t_1)$. Since $\bar{\theta}_{c^*}$ is constant across trials by random sampling, the ratio of conditional means at different times equals the temporal ratio. The anchor arm $c^*$ can be any treatment—not necessarily control—providing flexibility when certain treatments appear more frequently across trials.
The two strategies impose different structural requirements. Strategy 1 permits $\Lambda(t_0, t_1)$ to depend on both treatment administration and measurement times, accommodating settings where intervention timing matters—for instance, if treatment effects decay with elapsed time since administration. However, Strategy 1 requires trials comparing the exact same treatment pair at different times, which may be demanding in practice.
Strategy 2 requires only that some treatment arm appears at multiple measurement times, a condition often satisfied by control arms or standard treatments appearing across many trials. The cost is the restriction $\Lambda(t_0, t_1) = \Lambda(t_1)$, which rules out dependence on intervention timing. When both strategies are feasible, comparing their estimates provides a robustness check; discrepancies suggest potential violations of Assumption (ref).
We develop estimators for the TATE based on the identification results of Section (ref). Our approach proceeds in three steps: we derive efficient influence functions from the identified functionals, construct doubly robust estimators, and characterize their asymptotic properties.
We observe i.i.d.\ data $\{(Y_i, A_i, S_i, X_i)\}_{i=1}^{n}$. Define the nuisance functions $\pi_k(X) := P(S = k \mid X), \quad e_k(a, X) := P(A = a \mid S = k, X), \quad \mu_{a,k}(X) := \mathbb{E}[Y \mid A = a, S = k, X].$
Our identification results express the TATE as products and ratios of conditional means. We derive influence functions for these building blocks, then apply the delta method to obtain influence functions for the TATE.
\paragraph{Building blocks.} The marginal mean $\bar{\mu}_{a,k} := \mathbb{E}[Y \mid A = a, S = k]$ has efficient influence function \(\phi_{\bar{\mu}_{a,k}}(O) =\)
The ATE for trial $k$, $\tau_k := \tau_k(0,0) = \bar{\mu}_{a_k,k} - \bar{\mu}_{b_k,k}$, has influence function $\phi_{\tau_k}(O) = \phi_{\bar{\mu}_{a_k,k}}(O) - \phi_{\bar{\mu}_{b_k,k}}(O)$.
\paragraph{TATE influence functions.} Applying the delta method to the identified functionals yields influence functions for each strategy. For Strategy 1, the TATE is $\psi_1 = g(\tau_{k^\star}, \tau_j, \tau_{j'})$ where $g(x,y,z) = xy/z$. The gradient $\nabla g(x,y,z) = (y/z, x/z, -xy/z^2)^\top$ evaluated at the true parameters gives:
Both influence functions decompose into three terms reflecting distinct sources of uncertainty: the target trial ATE (first term) and both components of the temporal ratio (second and third terms). The coefficients have natural interpretations: $R_1$ (or $R_2$) scales the target trial's contribution since $\psi = \tau_{k^\star} \cdot R$, while the terms $\psi_1/\tau_j$ and $\psi_1/\tau_{j'}$ reflect how ratio estimation error propagates through the product.
These influence functions satisfy Neyman orthogonality with respect to the nuisance functions. Let $\eta = (\pi_k, e_k, \mu_{a,k})$ denote the nuisance vector, then the pathwise derivative of $\mathbb{E}[\phi_{\bar{\mu}_{a,k}}(O; \eta)]$ with respect to $\eta$ vanishes at the true parameter values \( \frac{\partial}{\partial r} \mathbb{E}[\phi_{\bar{\mu}_{a,k}}(O; \eta + r(\tilde{\eta} - \eta))]\Big|_{r=0} = 0 \) for all directions $\tilde{\eta} - \eta$ in the tangent space. This orthogonality implies that first-order errors in nuisance estimation do not affect the estimator's first-order bias, enabling valid inference even when nuisances are estimated at slower-than-parametric rates.
The TATE is a smooth function of identified quantities --- $\psi_1 = g(\tau_{k^\star}, \tau_j, \tau_{j'})$ where $g(x,y,z) = xy/z$ --- so we estimate it by plugging in efficient estimates of the building blocks. Define the doubly robust score $\varphi_{a,k}(O; \eta) = $
where $\eta = \{\pi_k, e_k, \mu_{a,k}\}$. This score satisfies $\mathbb{E}[\varphi_{a,k}(O;\eta)] = \bar{\mu}_{a,k}$, with centered version $\phi_{\bar{\mu}_{a,k}}(O) = \varphi_{a,k}(O;\eta) - \bar{\mu}_{a,k}$ being the EIF. Let $\varphi_{\tau_k}(O; \eta) = \varphi_{a_k,k}(O; \eta) - \varphi_{b_k,k}(O; \eta)$.
Given estimated nuisances $\hat{\eta}$, the building blocks $\hat{\bar{\mu}}_{a,k} = \mathbb{P}_n[\varphi_{a,k}(O; \hat{\eta})]$ and $\hat{\tau}_k = \mathbb{P}_n[\varphi_{\tau_k}(O; \hat{\eta})]$ are asymptotically linear with the EIFs under Assumption (ref). The TATE estimators follow by plug-in:
A Taylor expansion combined with asymptotic linearity of the building blocks yields $\hat{\psi}_1 - \psi_1 = \mathbb{P}_n[\phi_{\psi_1}(O)] + o_p(n^{-1/2})$, establishing that the plug-in estimator achieves the semiparametric efficiency bound (Appendix (ref)). Alternative constructions such as TMLE are asymptotically equivalent (Appendix (ref)).
The influence functions characterize asymptotic variances via $V = \mathbb{E}[\phi_\psi(O)^2]$. When the target trial is distinct from anchor trials, the influence function components have disjoint support, yielding:
The first term in each expression represents uncertainty from estimating the target trial's ATE, scaled by the squared temporal ratio. The second term captures uncertainty from estimating the temporal ratio itself, scaled by the squared TATE. When $|R_1|$ or $|R_2|$ is large (substantial temporal scaling), target trial uncertainty dominates; when the TATE is large, ratio uncertainty becomes more consequential.
\paragraph{Efficiency comparison.} These expressions reveal why Strategy 2 typically achieves greater precision. The key difference lies in the second term: Strategy 1 involves $V_{\tau_j}/\tau_j^2$ and $V_{\tau_{j'}}/\tau_{j'}^2$, while Strategy 2 involves $V_{\bar{\mu}_{c^*,\ell}}/\bar{\mu}_{c^*,\ell}^2$ and $V_{\bar{\mu}_{c^*,\ell'}}/\bar{\mu}_{c^*,\ell'}^2$.
For Strategy 1, the variance of a treatment effect satisfies:
Under randomization, units receive either treatment or control but not both, so the covariance term is typically small. Thus $V_{\tau_k} \approx V_{\bar{\mu}_{a_k,k}} + V_{\bar{\mu}_{b_k,k}}$, meaning Strategy 1's ratio involves roughly twice the variance of Strategy 2's.
Moreover, treatment effects are typically smaller in magnitude than outcome means: $|\tau_k| \ll |\bar{\mu}_{a,k}|$ in many applications. This implies $V_{\tau_k}/\tau_k^2 \gg V_{\bar{\mu}_{a,k}}/\bar{\mu}_{a,k}^2$—the coefficient of variation for treatment effects exceeds that for means. Both factors compound to give Strategy 2 substantially lower variance for the temporal ratio, which propagates to the final TATE estimator. This efficiency gain comes at the cost of Assumption (ref); when this assumption fails, Strategy 2 incurs bias while Strategy 1 remains valid.
When $m \geq 2$ anchor arms satisfy Assumption (ref), each yields a ratio $R_{2,j} = \bar{\mu}_{c_j,\ell_j}/\bar{\mu}_{c_j,\ell'_j}$ identifying the same temporal quantity. Let $\hat{\mathbf{R}} = (\hat{R}_{2,1}, \ldots, \hat{R}_{2,m})^\top$ with influence functions:
The covariance matrix $\mathbf{V}_{\mathbf{R}} = \mathbb{E}[\boldsymbol{\phi}_{\mathbf{R}}(O)\boldsymbol{\phi}_{\mathbf{R}}(O)^\top]$ captures both the variance of individual ratios and their correlations (which arise when anchors share trials at common time points).
\paragraph{Optimal combination.} Among estimators of the form $\hat{R} = \mathbf{w}^\top\hat{\mathbf{R}}$ with $\mathbf{w}^\top\mathbf{1} = 1$, the minimum-variance choice is:
This yields $\hat{R}_2^* = \mathbf{w}^{*\top}\hat{\mathbf{R}}$ and $\hat{\psi}_2^* = \hat{\tau}_{k^\star} \cdot \hat{R}_2^*$. The optimal variance $V_{R^*}$ is no larger than any individual $V_{R_{2,j}}$, with strict improvement when anchors provide non-redundant information. In practice, $\mathbf{V}_{\mathbf{R}}$ is replaced by its sample analog $\hat{\mathbf{V}}_{\mathbf{R}} = \mathbb{P}_n[\hat{\boldsymbol{\phi}}_{\mathbf{R}}(O)\hat{\boldsymbol{\phi}}_{\mathbf{R}}(O)^\top]$.
We establish formal guarantees under regularity conditions.
The overlap condition ensures inverse probability weights remain bounded. Bounded fourth moments guarantee finite variance of the influence function. Non-degeneracy ensures the temporal ratio is well-defined; division by zero would invalidate the identification strategy.
\paragraph{Variance estimation and inference.} The asymptotic variance is consistently estimated by:
where $\hat{\phi}_\psi(O)$ substitutes estimated nuisances and parameters into the influence function. This yields asymptotically valid confidence intervals $\hat{\psi} \pm z_{1-\alpha/2}\sqrt{\hat{V}/n}$ and Wald tests of hypotheses about the TATE.
When using flexible machine learning methods for nuisance estimation, $K$-fold cross-fitting ensures Assumption (ref) holds under weak conditions: partition data into $K$ folds, estimate nuisances on $K-1$ folds, and evaluate scores on the held-out fold. This sample-splitting avoids overfitting bias without sacrificing efficiency asymptotically.
Having established our framework, we now position it relative to existing literature. Our contribution sits at the intersection of several research streams, but differs from each in important ways.
\paragraph{Transportability and external validity.} The literature on generalizing experimental findings to new populations huang2024towards, degtiar2023review, bareinboim2016causal, cole2010generalizing, westreich2017transportability typically assumes access to covariate information in the target population and leverages selection-on-observables and positivity assumptions. Our setting differs in that the “target” is defined by temporal shift rather than covariate shift.
\paragraph{Meta-analysis.} Meta-analytic methods borenstein2021introduction, higgins2009re, riley2011interpretation, parikh2025double synthesize evidence across studies and can incorporate time as a study-level moderator. However, the goal is typically pooling—estimating an average effect or characterizing heterogeneity across observed studies—rather than extrapolation to counterfactual timing.
\paragraph{Factor models, synthetic control, and difference-in-differences.} Our separable temporal effects assumption (Assumption (ref)) shares structural similarities with several literatures that leverage factor structure for causal inference. Interactive fixed effects models in panel econometrics bai2009panel decompose outcomes into unit-specific loadings and time-specific factors; matrix completion approaches athey2021matrix view the potential outcome matrix as approximately low-rank; and synthetic control abadie2010synthetic, abadie2015comparative and difference-in-differences methods callaway2021difference, sun2021estimating, goodman2021difference, dechaisemartin2020two, borusyak2024revisiting implicitly rely on factor structure to construct counterfactuals, a connection made explicit by xu2017generalized and arkhangelsky2021synthetic.
Our separability assumption (A.(ref)) is similar to this broader literature. However, our setting and goals differ in three key respects. First, these methods estimate effects or impute missing outcomes within the observed temporal support of the study; we extrapolate to timing configurations where the relevant trial was not conducted. Second, our setup involves integrating information from multiple trials comparing different intervention or treatment pairs. Third, existing methods use outcomes from control units to impute missing potential outcome for the treated unit; we instead use outcomes from trials with entirely different treatments to identify how effects scale across time.
\paragraph{Semiparametric estimation.} Our estimation approach builds on the semiparametric efficiency literature robins1994estimation, bang2005doubly, kennedy2022semiparametric, parikh2025double, particularly doubly robust methods and debiased machine learning chernozhukov2018double.
We conduct Monte Carlo simulations to evaluate finite-sample performance, assessing bias, variance, and coverage while comparing the two identification strategies.
\paragraph{Data generating process.} We generate data according to Assumption (ref). Units have covariates $X = (X_1, X_2)^\top$ with $X_1 \sim \mathcal{N}(0, 1)$ and $X_2 \sim \text{Bernoulli}(0.5)$. The temporal modifier captures seasonal variation through $\Lambda(t) = 1 + \gamma \sin(2\pi t / 12)$ with $\gamma = 0.3$, producing a 12-period cycle. For treatment $a \in \{0, 1, 2\}$, potential outcomes follow
where $\theta_0(X) = 2 + 0.5 X_1 + 0.3 X_2$, $\theta_1(X) = \theta_0(X) + 1 + 0.4 X_1 + 0.2 X_2$, and $\theta_2(X) = \theta_0(X) + 0.5 + 0.2 X_1 + 0.1 X_2$.
\paragraph{Trial structure.} We simulate $K = 6$ trials as detailed in Table (ref). The target trial $k^\star = 1$ compares treatment 1 against control at $(t_{0k^\star}, t_{1k^\star}) = (1, 3)$, and we estimate the TATE at $(t_0, t_1) = (7, 9)$---a half-cycle displacement with true value $\psi \approx 0.77$. Trials 2--3 provide anchors for Strategy 1; trials 4--5 provide anchor arms for Strategy 2.
\paragraph{Estimators.} We compare five estimators: S1 implements Strategy 1 using trials 2 and 3; S2-C and S2-T implement Strategy 2 with control ($c^* = 0$) and treatment 2 ($c^* = 2$) as anchors, respectively; S2-M combines both anchors via inverse-variance weighting; and Oracle uses the true temporal ratio, providing a lower bound on achievable variance. Nuisance functions are estimated via gradient boosting with 5-fold cross-fitting. We vary $n \in \{600, 1200, 2400\}$ with $B = 500$ replications.
Table (ref) presents bias, RMSE, the ratio of estimated to empirical standard errors (SE Ratio), and coverage of 95% confidence intervals.
\paragraph{Bias and consistency.} All estimators exhibit negligible bias across sample sizes (absolute bias $< 0.03$ even at $n = 600$), confirming that identification assumptions hold and doubly robust estimators achieve the consistency guaranteed by Theorem (ref).
\paragraph{Efficiency comparison.} Strategy 2 substantially outperforms Strategy 1 in RMSE. At $n = 2400$, S2-M achieves RMSE of 0.071 versus 0.143 for S1---a 50% reduction. This gain arises because Strategy 2 estimates temporal ratios from conditional means rather than treatment contrasts, which have higher variance. The multi-anchor estimator S2-M performs comparably to single-anchor variants.
\paragraph{Variance estimation and coverage.} SE Ratios near 1.0 confirm accurate variance estimation. S2-T shows mildly elevated ratios (1.28--1.38), reflecting finite-sample variability with a single anchor. Coverage rates are at or above nominal 95% for all estimators, with slight overcoverage for S2-T consistent with conservative variance estimation.
\paragraph{Comparison to oracle.} The Oracle provides a lower bound on achievable RMSE by eliminating uncertainty in the temporal ratio. At $n = 2400$, S2-M achieves RMSE of 0.071 versus 0.061 for Oracle, indicating that estimation error stems primarily from the target trial ATE rather than temporal ratio estimation.
\paragraph{Summary.} The simulations validate our theoretical results: both strategies yield consistent estimators with accurate variance estimation and near-nominal coverage. When Assumption (ref) holds, Strategy 2 reduces RMSE by approximately 50% compared to Strategy 1, and combining multiple anchors provides modest additional gains.
We apply our framework to headline A/B tests from the Upworthy Research Archive matias2021upworthy, one of the largest publicly available collections of randomized digital experiments. Full details appear in Appendix (ref).
\paragraph{Data and setup.} Upworthy conducted over 22,000 headline A/B tests between 2013--2015, randomly assigning visitors to different headline-image combinations and measuring click-through rates (CTR). Following matias2021upworthy, we exclude tests from June 2013--January 2014 where randomization problems were detected.
To apply our framework, we must identify treatment arms appearing across multiple tests at different times. Since each test uses unique headline text, we cluster semantically similar headlines using Sentence-BERT embeddings reimers2019sentence with a constrained clustering procedure ensuring no two headlines from the same test share a cluster (see Appendix (ref)). This yields 50 headline clusters, each appearing across multiple months.
\paragraph{Research question.} We estimate TATEs for two A/B tests conducted in late 2013 and early 2014, transporting observed effects to subsequent months of 2014. We compare both identification strategies against a “ground truth” TATE constructed by pooling all available comparisons of the same cluster pair at each target time.
\paragraph{Results.} Figure (ref) and Table (ref) reveal a variance-bias tradeoff. Strategy 2 yields substantially smaller standard errors than Strategy 1 (mean SE of 0.0019 vs.\ 0.0054 for Trial A), reflecting its use of conditional means rather than treatment contrasts. However, Strategy 2 exhibits systematic bias: estimates remain nearly constant across months while the true TATE varies considerably---even changing sign in some months. Strategy 1, despite wider confidence bands, tracks these dynamics (correlation with true TATE: 0.71 vs.\ 0.35 for Trial A). This pattern suggests violations of Assumption (ref): the temporal factor depends not only on measurement time but also on intervention time. This occurs when the gap between intervention and measurement matters---for instance, if treatment effects decay over time. Strategy 1 accommodates such dependence by allowing $\Lambda(t_0, t_1)$ to vary with both arguments.
We have developed a framework for transporting treatment effects across time, introducing the transported average treatment effect (TATE) and providing identification and estimation strategies under a separable temporal effects assumption. We conclude by discussing key considerations and limitations.
\paragraph{The separability assumption.} Our identification relies on potential outcomes decomposing multiplicatively into unit-specific and temporal components (Assumption (ref)). This structure is plausible when temporal variation operates as a scaling factor affecting all units proportionally---for instance, seasonal demand shocks in retail, attention cycles in digital advertising, or labor market tightness for employment interventions. The assumption may fail when ceiling/floor effects attenuate temporal scaling for some units, or when treatments exhibit qualitative interactions with time (helping some subgroups in certain periods while hurting them in others). When multiple anchor arms are available, there are testable implications providing a diagnostic.
\paragraph{Strategy selection.} The replicated trials strategy (Strategy 1) permits flexible temporal structures where $\Lambda(t_0, t_1)$ depends on both treatment and measurement times, but requires trials comparing identical treatment pairs across time---a demanding data requirement. The common arm strategy (Strategy 2) requires only that some treatment arm appears at multiple measurement times, offering greater practical flexibility since control arms often span many trials. The cost is the stronger restriction $\Lambda(t_0, t_1) = \Lambda(t_1)$, appropriate when temporal effects reflect conditions at outcome measurement rather than treatment administration. When both strategies are feasible, comparing estimates provides a robustness check.
\paragraph{Limitations.} The separability assumption is fundamentally untestable from data at observed times alone. Furthermore, our approach relies on population stability across time; if population compositions drift over time, temporal effects are conflated with compositional changes. We also assume treatment stability---that the “same” treatment does not differ in implementation across time.
\paragraph{Future Work.} Two promising extensions address key limitations of our framework. Sensitivity Analysis for Separability Violations: Our identification relies critically on Assumption (ref), which is fundamentally untestable. A natural extension develops sensitivity analysis tools quantifying how departures from multiplicative separability propagate to bias in the TATE. This would transform our point identification results into partial identification intervals, providing more robust uncertainty quantification when separability is violated. Covariate-Dependent Temporal Modifiers: Our temporal modifier $\Lambda(t_0, t_1)$ is assumed common across units, ruling out heterogeneous temporal effects across subpopulations. Future work would accommodate covariate-dependent modifiers $\Lambda(t_0, t_1; X)$.
\paragraph{Conclusion.} This paper makes three contributions. First, we formalized temporal transportation through the TATE and showed it decomposes into an observed ATE and a temporal ratio under separability. Second, we provided two identification strategies with different data requirements and structural restrictions, offering practitioners flexibility based on available trials. Third, we developed doubly robust estimators achieving semiparametric efficiency, with valid inference through influence function-based variance estimation. Temporal variation in treatment effects is ubiquitous yet understudied relative to population heterogeneity. As organizations increasingly rely on continuous experimentation, transporting findings across time becomes essential: a promotion tested in January must inform March decisions; a program evaluated during expansion must guide policy during contraction. Our framework provides principled tools for such extrapolation, grounded in explicit assumptions that can be deliberated in context.