Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
39,191 characters · 18 sections · 31 citation commands
Validating Causal Message Passing Against Network-Aware Methods on Real Experiments
\begingroup \footnotetext{\textsuperscript{*}\;Amazon \quad \textsuperscript{$\dagger$}\;Stanford University} \addtocounter{footnote}{-1} \endgroup
Randomized experiments form the foundation of evidence-based decision-making across domains ranging from technology services to public health programs. The classical analysis of such experiments relies on the Stable Unit Treatment Value Assumption (SUTVA), which posits that each unit's outcome depends only on its own treatment assignment cox1958planning. However, in networked settings, where units interact through social ties, shared resources, or competitive dynamics, this assumption fails. A unit's outcome may depend not only on its own treatment but also on the treatments received by connected units, a phenomenon known as network interference or spillover halloran1995causal, sobel2006what, hudgens2008toward.
The methodological response to interference has followed two broad paths. The first path leverages explicit knowledge of the network structure, developing estimators that condition on or adjust for network exposure aronow2017general, leung2022causal. Recent advances in bipartite experimental designs exemplify this approach: when treatment units (e.g., service providers) and outcome units (e.g., customers) form a two-sided structure, methods can exploit the observed interaction graph to define exposure mappings and construct interference-aware estimators pouget2019variance,doudchenko2020causal,zigler2021bipartite,harshaw2023design,tan2025estimating. These network-aware methods achieve strong performance when the interaction structure is observed, but they require data that is often unavailable, proprietary, or measured with substantial error.
The second path addresses settings where the network is unknown or only partially observed. Recent work has developed methods that substitute temporal or design-based variation for network knowledge johari2022experimental, cortez2022staggered,munro2021treatment. Among these, causal message passing (CMP) offers a principled framework that leverages a key insight: the mechanisms governing how treatment effects propagate through a network exhibit structural invariance over time shirani2024causal, shirani2025can, shirani2025evolution. By observing how outcomes evolve across time periods, CMP learns the aggregate dynamics of interference without requiring observation of individual network connections.
This paper bridges these two methodological streams by empirically validating CMP against a network-aware benchmark. We apply CMP to two large-scale field experiments previously analyzed using the bipartite graph methodology of tan2025estimating, which explicitly incorporates the interaction structure between treatment and outcome units. Our main finding is that CMP, despite operating without any network information, produces total treatment effect estimates that closely match those from the network-aware approach and correctly recover the direction of interference bias predicted by domain knowledge.
Our contributions are:
Understanding and estimating the effects of treatments and interventions is a foundational problem across many fields, and has attracted increasing attention in recent years halloran1995causal,sobel2006what,hudgens2008toward,tchetgen2012interference,manski2013identification,aronow2017general,saveski2017detecting,athey2018exact,viviano2020experimental,hu2022average,han2022detecting,leung2022causal,agarwal2022network,belloni2022neighborhood,li2022network,li2022random,farias2022markovian,candogan2023correlated,ni2023design,viviano2023causal,harshaw2023design,imbens2024causal,eichhorn2024low,johari2024does,Larsen02042024,peng2025differencesinneighborsnetworkinterferenceexperiments.
The current work connects several strands of the interference literature, ranging from network-aware designs to settings with unknown network structure and to marketplace experiments. Specifically, network-aware experimental designs have been developed by ugander2013graph and eckles2017design, while bipartite interference settings have received substantial attention in pouget2019variance,doudchenko2020causal,zigler2021bipartite,NEURIPS2022_f7f043c3,harshaw2023design,tan2025estimating.
In the context of service experiments, a growing body of research has examined interference effects across a range of economic environments and experimental designs blake2014marketplace,holtz2020reducing,wager2021experimenting,munro2021treatment,johari2022experimental,10.1145/3485447.3512063,bajari2023experimental,farias2023correcting,zhu2024seller,wu2024switchbackpriceexperimentsforwardlooking,bright2022reducing,zhang2025debiasing,johari2025estimationtreatmenteffectsnonstationarity. Leveraging temporal observations and structure has also proven promising for addressing interference farias2022markovian,hu2022switchback,bojinov2023design,li2023experimenting,ni2023design,han2024population,jia2025clustered.
Methods for settings with unknown network structure include mean-field-type approaches wager2021experimenting,johari2022experimental,munro2021treatment, design-based experimental strategies cortez2022staggered,hu2022switchback, and total-effect estimators that do not require explicit network information yu2022estimating. This paper utilizes a recent methodology in this stream, particularly the causal message passing (CMP) framework introduced in shirani2024causal and extended to settings with higher-order dynamics in bayati2024higher. While these initial versions of CMP perform well when sufficient temporal or cross-sectional variation is available, their effectiveness can be limited in short experiments or in environments with complex outcome-treatment dynamics. To address this challenge, shirani2025can proposes a distribution-preserving network bootstrap that constructs multiple representative subpopulations from a single experiment, thereby increasing effective sample size for estimation; we adopt this approach in the present work. More recently, shirani2025evolution provides an axiomatic foundation for evolution-based methods through an exposure-mapping perspective. Our contribution is to provide a direct empirical comparison between bipartite, network-aware designs and network-blind causal message passing using real experimental data.
This section provides summaries of the two methodologies we compare: the network-aware bipartite approach and causal message passing. But we defer for a detailed description to the original papers. Specifically, to tan2025estimating and shirani2024causal,shirani2025can. Both target the same estimand, the total treatment effect under full deployment, but differ fundamentally in their information requirements.
Consider an experiment with $N$ units indexed by $i = 1, \ldots, N$, observed over time periods $t = 0, 1, \ldots, T$. Let $W_t^i \in \{0,1\}$ denote the treatment assignment for unit $i$ at time $t$, and let $Y_t^i(\mathbf{W})$ denote the potential outcome for unit $i$ at time $t$ under the full treatment allocation matrix $\mathbf{W}$, where $\mathbf{W}$ is an $N \times T$ matrix with entry $(i,t)$ being $W_t^i$.
When running an experiment, practitioners face choices about how treatment assignments evolve over time. One approach fixes each unit's treatment status throughout the experiment: units are assigned to treatment or control at the outset, and experimenters observe outcomes as they stabilize to an equilibrium reflecting the treatment's full effect. An alternative approach allows treatment assignments to vary over time, with units potentially switching between treatment and control across periods. A particularly common design in this latter category is the staggered rollout, where all units begin in the control state, and at some point during the experiment, units transition to treatment and remain treated thereafter. In this paper, we primarily focus on settings consistent with staggered rollout designs. Consequently, both outcomes and treatments are naturally represented as $N \times T$ matrices, capturing the temporal evolution of each unit's status and response.
The total treatment effect (TTE) compares the average outcome under full treatment against the average outcome under no treatment:
where $\mathbf{1}$ denotes the all-treated assignment and $\mathbf{0}$ denotes the all-control assignment across all time periods. This estimand captures both the direct effect of treatment on treated units and the indirect spillover effects that propagate through the network and over time.
In bipartite settings, treatment is applied to units on one side of the graph (e.g., service providers), while these treatment units interact with units on the other side, which we refer to as connected units throughout this paper. We note that tan2025estimating, and broadlly the bipartite-experiments literature, refer to these connected units as “outcome units”; we deliberately avoid this terminology to prevent confusion, for reasons we now explain. In bipartite experimental designs, outcomes are often observed at the edge level, that is, at each interaction between a treatment unit and a connected unit. For example, when a service provider serves a user, an outcome may be recorded for that specific interaction, such as the quantity of service delivered. These edge-level outcomes sometimes can then be aggregated to either side of the bipartite graph, referred by the linear additive edges assumption in tan2025estimating. One can aggregate to the treatment side by summing all edge outcomes across connected units served by each treatment unit, or aggregate to the connected-unit side by summing all edge outcomes across treatment units serving each connected unit. In this paper, we focus on outcomes aggregated to the treatment side: whenever we refer to outcomes $Y_t^i$, we mean the aggregate outcome for treatment unit $i$ at time $t$, computed by summing the edge-level outcomes across all connected units that treatment unit $i$ interacts with. This aggregation convention is why we avoid calling the right-hand-side units “outcome units,” as doing so would conflate the units with the outcomes themselves and obscure the aggregation structure.
A further distinction comes up regarding which treatment units are eligible for the intervention. tan2025estimating emphasize that treatment units often are of two type: those eligible for treatment and those ineligible due to operational or design constraints. One of their main contributions is formalizing this structure and introducing two corresponding estimands. The Primary Total Treatment Effect (PTTE) measures the impact on eligible treatment units when they receive the intervention, while the Secondary Total Treatment Effect (STTE) captures how outcomes of ineligible treatment units may be affected through spillovers, even though these units cannot themselves be treated. In this paper, we focus exclusively on eligible units: our indexing $i = 1, \ldots, N$ refers only to treatment units that are eligible for the intervention (but we allow existence of additional ineligible units). Consequently, the total treatment effect we study corresponds to what tan2025estimating call the PTTE, and this is precisely the estimand they report for their real experimental analyses. We adopt the same focus. Studying how one might estimate the STTE using temporal methods such as causal message passing is an interesting direction, but it lies beyond the scope of this paper.
The network-aware approach of tan2025estimating does not operate directly on temporal data. Instead, it works with outcomes aggregated across time at the unit level---specifically, the difference between post-treatment and pre-treatment average outcomes for each unit. This aggregated outcome structure is precisely what we refer to as the input to our SUTVA-based “Basic Method.” Treatment status is represented as a single indicator $W^{(j)} \in \{0,1\}$ for each treatment unit $j$, denoting whether that unit was assigned to treatment or control.
The methodology leverages the observed bipartite graph connecting treatment units and connected units. For each treatment unit $j$, the method defines two exposure variables. The direct exposure captures whether unit $j$ is itself treated and has connections in the graph:
where $\mathcal{C}_j$ denotes the set of connected units linked to treatment unit $j$. The indirect exposure captures spillover from other treated units through shared connected units:
where $\mathcal{T}_c$ denotes the set of treatment units connected to connected unit $c$, and $\mathbb{I}(\cdot)$ is the indicator function.
The outcome for treatment unit $j$ is modeled as a function of these exposures:
where $X_j$ are covariates and $\Psi$ is estimated using machine learning methods such as kernel ridge regression. The PTTE is then estimated by comparing predicted outcomes under full treatment versus no treatment:
where $\mathcal{T}$ denotes the set of eligible treatment units. A key requirement of this approach is observation of the bipartite graph structure linking treatment units and connected units.
Causal message passing (CMP) takes a fundamentally different approach. Rather than conditioning on network structure, CMP exploits the temporal invariance of interference mechanisms. The key insight is that treatment effects propagate through the network according to rules that remain structurally stable over time, analogous to invariant physical laws in interacting particle systems shirani2024causal, shirani2025can.
CMP models the evolution of population-level outcome distributions through state evolution equations:
where $Y_t$ and $W_{t+1}$ are single-dimensional random variables, representing the limiting distribution of outcomes and treatments at time $t$, respectively, while $f_t$ captures the aggregate dynamics of how outcomes evolve. The mapping $f_t$ incorporates both direct treatment effects and indirect spillover effects without requiring knowledge of individual network connections.
The estimation procedure proceeds in three steps:
The total treatment effect is estimated by comparing the predicted counterfactual evolutions:
where $\widehat{\text{CFE}}_T(\mathbf{w})$ denotes the estimated counterfactual evolution at the final time period under treatment allocation $\mathbf{w}$.
The key advantage of CMP is that it only requires observed outcomes over time, without any knowledge of the network data (i.e., $\omega_{ij}$'s used by the Network-Aware method).
In the following section, we compare the Network-Aware and CMP methodologies against a simpler baseline that we call the Basic Method. This Basic Method assumes SUTVA holds and ignores interference entirely. As discussed above, the Network-Aware approach operates on aggregated outcomes at the unit level, specifically the first differences between post-treatment and pre-treatment periods. The Basic Method uses the same aggregated outcome structure, collapsing the time series into a single pre-post difference for each unit, but applies a ML-enhanced version of standard difference-in-differences estimator that assumes SUTVA. By aggregating time before estimation, the Basic Method effectively discards the intermediate temporal dynamics that CMP leverages to disentangle interference. The Network-Aware methodology can be viewed as a more sophisticated version of the Basic Method, one that incorporates the bipartite graph structure through exposure mappings. In contrast, CMP takes a fundamentally different approach by working directly with temporal outcome data and learning the dynamics of how treatment effects propagate over time.
Table (ref) summarizes the key differences among these three methodologies.
We apply all methodologies to two large-scale field experiments conducted in a service environment where treatment units interact with connected units through a bipartite structure. Due to confidentiality constraints by the data provider, we can only describe the experiments in general terms.
Both experiments involve interventions applied to a subset of treatment-side units (e.g., drivers in a ride-sharing service). The bipartite structure arises naturally from the interaction patterns: connected units may connect to multiple treatment units, and treatment units serve multiple connected units.
Importantly, only a subset of treatment units are eligible for the intervention due to operational constraints. This creates an eligibility-constrained setting where randomization occurs within the eligible set, but ineligible units continue to interact with outcome units and may generate spillover effects.
The bipartite interaction graphs in both experimental settings are sparse, with the average degree of treatment units being small relative to the population size. This sparsity presents a non-trivial challenge for network-aware estimation, as the signal for exposure-response functions must be learned from relatively few interactions per unit.
Experiment A involved approximately 7,000 eligible treatment units and ran for multiple ($T<100$) time periods with varying treatment probabilities across experimental stages. The intervention modified a feature affecting how treatment units interact with connected units.
Experiment B involved approximately 4,000 eligible treatment units with a similar multi-stage design. The intervention targeted a related but different aspect of the interaction between the treatment and connected units.
Both experiments collected outcome data at the treatment unit level across all time periods, enabling application of both network-aware and CMP methodologies.
We evaluate three pre-specified metrics, denoted $M_1$, $M_2$, and $M_3$:
We implement the bipartite methodology of tan2025estimating, using the observed interaction graph to construct exposure variables and generalized propensity scores. LightGBM with parameter auto-tuning estimates the outcome function, with bootstrap inference for confidence intervals. We also implement the CMP framework of shirani2025can, computing outcome and treatment summary statistics at each time period. We use ridge regression with cross-validation to estimate state evolution parameters, applying the distribution-preserving network bootstrap to generate training samples. We also implemented the Basic method described above that ignores interference.
Table (ref) presents our main results, comparing the Basic (naive), Network-Aware, and CMP approaches across both experiments and all three metrics.
The most striking finding is the close alignment between CMP and Network-Aware estimates across both experiments and all metrics. In every case, CMP produces effect estimates with the same directional sign as the Network-Aware approach. This alignment is notable given that CMP operates without any knowledge of the bipartite interaction structure that the Network-Aware method explicitly leverages.
For Experiment A, both Network-Aware and CMP find:
For Experiment B, both methods find:
CMP reaches statistical significance on metric $M_1$ where Network-Aware does not. One possible explanation is the difference in data utilization: while the Network-Aware method collapses the temporal dimension into a single pre-post difference per unit, CMP utilizes the full time series of outcomes. However, the two methods differ in many other respects, modeling assumptions, estimation procedures, and sources of identifying variation, making it difficult to attribute the difference in significance to any single factor. A rigorous comparison of statistical efficiency between these fundamentally different approaches is beyond the scope of this paper.
While CMP matches Network-Aware perfectly on the primary decision metric $M_3$, agreeing in both sign and statistical significance across both experiments, the two methods differ in statistical significance on metric $M_1$: CMP finds significant effects in both experiments where Network-Aware does not. Whether this reflects a genuine difference in statistical efficiency, differences in the estimands being targeted, or other methodological factors cannot be determined from our comparison alone. What we can conclude is that temporal dynamics provide sufficient signal to match the directional conclusions of the network-aware approach on all metrics studied.
For metric $M_1$, economic theory predicts that ignoring spillovers should bias estimates in a specific direction: the Basic method should yield estimates that are more positive than the true total treatment effect. Both experiments confirm this prediction, and importantly, CMP's significant findings on this metric align with theoretical expectations.
In Experiment A, the Basic method shows a positive and significant effect, while both Network-Aware and CMP show negative effects. The difference (Basic $-$ Network-Aware) and (Basic $-$ CMP) are both positive, matching the expected bias direction. CMP's finding of a significant negative effect, where Network-Aware finds a non-significant negative effect, is consistent with domain knowledge about this metric. If the true effect is indeed negative as theory predicts, this pattern would suggest CMP may have greater power to detect it, though confirming this hypothesis would require further investigation.
In Experiment B, while all methods show positive effects, the observed bias (Basic more positive than the interference-aware estimates) still matches the expected direction. Again, CMP finds significance where Network-Aware does not. If the true effect is positive, this would be consistent with CMP having greater sensitivity, though other explanations cannot be ruled out.
This pattern demonstrates that CMP, like the Network-Aware approach, correctly identifies and adjusts for the bias that arises when spillovers are ignored. For metric $M_1$, where domain knowledge provides strong directional predictions, CMP's significant findings align with theoretical expectations. The temporal information that CMP extracts appears to be a reliable substitute for network information in these experimental settings, though understanding the precise trade-offs between temporal and network-based approaches remains an important direction for future research.
The primary decision metric $M_3$ reveals the practical importance of accounting for interference.
In Experiment A:, the Basic method suggests a negative treatment effect that does not reach statistical significance. In contrast, both Network-Aware and CMP find positive and statistically significant effects. This reversal is consequential: a statistically significant increase in $M_3$ implies a deterioration on the primary decision metric, which the Basic method would fail to detect. The interference-aware methods provide a clear signal that the naive analysis obscures. Importantly, CMP and Network-Aware agree exactly on this decision-critical metric, same sign and significance conclusion.
In Experiment B, both Network-Aware and CMP find negative but non-significant effects on $M_3$, while the Basic method finds positive but non-significant effects. Although the directional disagreement exists, all methods lead to the same practical conclusion: no statistically significant effect detected. The interference-aware methods again agree with each other.
The agreement between CMP and Network-Aware on $M_3$ across both experiments is the key validation result of this paper. For the metric that determines whether an intervention should be launched at scale, the two methodologies, one requiring network data, one requiring only temporal data, reach identical conclusions.
The strong alignment between CMP and Network-Aware estimates, particularly the perfect agreement on the primary decision metric $M_3$, supports the theoretical premise of causal message passing: that temporal variation in outcomes encodes information about interference structure. When treatment effects propagate through a network, they induce predictable patterns in how outcome distributions evolve over time. By learning these patterns, CMP effectively reconstructs the aggregate effect of network interference without observing the network itself.
On metric $M_1$, CMP reaches statistical significance where Network-Aware does not. While one plausible explanation is that CMP's use of outcome data across all time periods effectively increases the sample available for estimation, the two methods differ in too many dimensions to draw definitive conclusions about relative efficiency. Understanding the trade-off between network information and temporal information, and the conditions under which one dominates the other, is an important question that our empirical comparison raises but cannot resolve. What our results do suggest is that practitioners need not view the absence of network data as a fundamental obstacle to interference correction.
We hypothesize that the observed agreement arises when interference mechanisms are relatively stable over time, sufficient temporal variation exists to identify state evolution mappings, and the network structure generates interference patterns that manifest clearly in outcome dynamics. Future work should characterize conditions under which the methods may diverge and develop diagnostics for practitioners.
Several limitations warrant acknowledgment. Our validation is based on two experiments, and generalization to other domains requires additional study. Neither method provides the “true” treatment effect; we compare methods against each other and against directional predictions from theory. CMP requires multi-period outcome data and assumes relatively stable interference mechanisms, rapidly changing network structures may violate these assumptions. shirani2025evolution provides an axiomatic treatment that can help understand such limitations better.
For practitioners facing settings where network data is unavailable, costly, or unreliable, CMP offers a principled alternative that correctly recovers bias direction and, in Experiment A, reverses a misleading conclusion that would arise from standard SUTVA-based analysis. More broadly, this work demonstrates empirical convergence between network-aware and network-blind methods on real data, suggesting that the information content of temporal dynamics and network structure may be more closely related than previously recognized.