Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
55,612 characters · 23 sections · 63 citation commands
Algorithm or Creative? A Three-Arm Experimental Design for Decomposing Algorithmic Bias in Platform A/B Tests
Online advertising platforms host hundreds of thousands of advertiser-run A/B tests at any moment; on Meta alone, burtch2025characterizing document roughly $182{,}000$ active comparisons. These tests are the dominant decision tool for allocating a global digital-advertising budget that now exceeds \$700 billion per year. Their interpretation is, however, complicated by a feature of modern delivery infrastructure that the experimenter cannot turn off: the platform's algorithm observes each creative and routes impressions to the users it predicts are most likely to engage. Two creatives in the same A/B test therefore reach two systematically different audiences, a phenomenon termed divergent delivery lambrecht2019algorithmic, ali2019discrimination, burtch2025characterizing.
Divergent delivery turns every two-arm A/B test into a sum of two effects: what the creative does to a fixed audience, and what the algorithm does to the composition of that audience. A manager who sees creative $B$ outperform creative $A$ cannot tell which channel produced the gap, and the two answers prescribe opposite next moves --- scale the creative or scale the audience signal. The misattribution is not a small error: in protected categories such as housing, credit, and employment, the algorithmic channel carries regulatory exposure that the creative channel does not. The measurement problem and the fairness problem are, in this setting, the same problem.
Two natural remedies do not solve it. Holding the audience fixed by tight targeting is undone by the algorithm itself ali2019discrimination, burtch2025characterizing. Adjusting for the realized audience post hoc by adding it as a regression control is biased because the audience is a post-treatment mediator: it responds to the treatment and inherits its endogeneity gerber2012field, acharya2016explaining. The existing literature documents the imbalance but does not give the advertiser a tool to recover the creative-only effect from a single experiment.
We propose a three-arm experimental design that does. A new arm exposes the platform's algorithm to the treatment metadata signal while holding the user-facing creative identical to the control. Under three transparent assumptions --- random arm assignment, exclusion, and a verifiable match between the realized mediator distributions in the new arm and the treatment arm --- the design point-identifies the natural indirect effect (the algorithm's targeting response) and the natural direct effect (the user-visible creative effect), without invoking the cross-world sequential-ignorability assumption that observational mediation analysis requires pearl2001direct, robins1992identifiability, imai2013experimental.
We apply the design to a live Meta traffic-optimization campaign in which the treatment is the presence of a women-targeted text fragment. At the high-bid level the algorithmic channel raises the female impression share by $+2.07$ percentage points; the visible-creative channel moves it by $-0.68$ percentage points; the total effect is $+1.39$ percentage points. Roughly three-quarters of the absolute reallocation is algorithmic, and the two channels work in opposite directions: the visible creative partially undoes the algorithm's targeting move. A disaggregated decomposition shows the algorithm concentrating new impressions on $35$--$44$ female users while reallocating away from the $65$+ band. A conventional two-arm A/B test, which observes only the total effect, attributes the entire shift to the creative and understates the algorithmic channel by roughly a factor of two.
Three contributions follow. First, methodologically, the paper introduces a portable experimental design that identifies direct and indirect channels in the presence of post-treatment mediation by a delivery algorithm, with assumptions that can be checked against production data rather than asserted. Second, empirically, we provide what is to our knowledge the first causal --- not descriptive --- decomposition of divergent delivery on a major advertising platform, and we show that the algorithmic channel dominates the conventional A/B contrast in both magnitude and policy interpretation. Third, for IS research and policy, the framework connects the measurement problem to the algorithmic-fairness debate: the indirect effect we identify is, by construction, the platform's contribution to differential exposure, separable from any choice the advertiser makes about creative content. The design generalizes to any delivery outcome (demographic share, geographic share, conversion mix) on any platform whose targeting metadata is separable from rendered content.
The remainder of the paper is organized as follows. Section (ref) reviews the relevant literatures and locates the contribution. Section (ref) develops the three-arm design and the identification result. Section (ref) validates the decomposition in a calibrated auction simulation. Section (ref) reports the Meta field experiment and the per-cell decomposition.
Three streams of research converge on the problem we study, and a fourth provides the design tools.
\paragraph{Field experiments on digital advertising platforms.} A substantial literature uses platform A/B tests to measure the causal effect of digital advertising aral2011platform, gordon2019comparison, tucker2014social, matz2017psychological. gordon2019comparison show, on Meta data, that observational and experimental estimates of ad effectiveness can disagree by an order of magnitude, motivating the field's move to randomized comparisons as the default measurement primitive. None of this work, however, separates the algorithmic delivery channel from the creative channel within a single randomized comparison; the conventional two-arm design folds them together by construction.
\paragraph{Divergent delivery and algorithmic bias in ad delivery.} A more recent strand documents that the delivery algorithm itself produces large audience differences across cells of an A/B test, even when targeting, budget, and placement are held constant. lambrecht2019algorithmic show that an ostensibly gender-neutral STEM career ad is delivered disproportionately to men; ali2019discrimination reproduce the pattern on Meta across protected demographic categories; burtch2025characterizing characterize the phenomenon at scale and propose campaign-configuration adjustments that compress, but do not eliminate, the imbalance. Audit studies in adjacent settings speicher2018potential, imana2024apparent confirm the same logic in search and display delivery. This literature establishes the empirical fact of divergent delivery and proposes configuration-side remedies, but it treats the imbalance as a descriptive quantity rather than isolating the algorithmic contribution as a causal estimand.
\paragraph{Causal mediation and direct/indirect effect identification.} The identification machinery we use is the causal-mediation tradition of pearl2001direct, robins1992identifiability, imai2010general, imai2013experimental, and vanderweele2015explanation. The baron1986moderator-style regression of the outcome on the treatment and a post-treatment mediator is now understood to be biased gerber2012field, acharya2016explaining, heckman2015econometric; the credibility of any decomposition rests on the source of variation in the mediator. imai2013experimental formalize designs --- “parallel” and “encouragement” --- in which the mediator is randomized independently of the treatment, sidestepping the cross-world sequential-ignorability assumption that observational mediation analysis requires robins2003semantics, imai2010identification.
\paragraph{Where this paper sits.} These streams have not been integrated. The divergent-delivery literature has the right object of measurement but treats it descriptively. The mediation literature has the right identification tools but has not been applied to a platform delivery algorithm acting as the mediator. The digital-advertising experimental literature has the right setting but the wrong design for separating the two channels. This paper builds the bridge. We treat the delivery algorithm as a Pearl--Robins mediator on the causal path from creative to outcome, and we adapt the parallel-design logic of imai2013experimental to the operational reality that the experimenter cannot directly set the algorithm's state but can manipulate the metadata signal that the algorithm conditions on. The result is a design that point-identifies the indirect (algorithmic) and direct (creative) channels under assumptions verifiable from the experiment's own data, placing divergent delivery on a causal footing suited to both managerial decision-making and IS-policy analysis.
We compare two treatments, $A$ and $B$, with respect to an outcome $Y$. Although treatment is assigned at random, each subject's exposure to treatment is shaped by a delivery mechanism $S$ --- in our application, an ad delivery algorithm that determines which advertisements are served to which users. Because $S$ is itself a post-treatment variable that responds to treatment and in turn influences the outcome, it satisfies the formal definition of a mediator pearl2001direct, and the decomposition of the total effect into direct and indirect channels applies.
Because $S$ is itself a response to treatment, $S$ takes value $S(A)$ under $A$ and value $S(B)$ under $B$, and the two distributions generally differ. We adopt the potential-outcomes framework of neyman1923,rubin1974, and write $Y_i(a, s)$ for the outcome of unit $i$ under treatment $a \in \{A, B\}$ and mediator value $s$, and $S_i(a)$ for the mediator value unit $i$ would realize under $a$. The observed outcome under treatment $a$ is $Y_i(a, S_i(a))$, sometimes called the composite potential outcome vanderweele2015explanation.
A naive comparison of treatment arms estimates
which is the total effect of $A$ versus $B$. Equation (ref) is a well-defined causal contrast under randomization of treatment alone, but it is not the contrast a researcher typically wants when $S$ is itself altered by the treatment: it confounds the direct effect of receiving $A$ rather than $B$ with the indirect effect operating through changes in the mediator. As pearl2001direct and robins1992identifiability make explicit, when treatment shifts both the direct channel and the mediator, the total effect mixes the two channels in an unidentifiable proportion under a single-stage design.
A common but biased remedy is to include $S$ as a control variable in a regression of $Y$ on the treatment indicator:
This is the structure underlying baron1986moderator. gerber2012field formalize the bias of this approach in their Chapter 10 appendix; the argument is also developed by green2010enough, imai2010general, and acharya2016explaining. Conditioning on a post-treatment variable that is itself caused by the treatment opens a non-causal path between $D$ and $\varepsilon$, so $\hat{\beta}$ in (ref) does not estimate the direct effect even under random assignment of $D$. The unbiasedness conferred by randomization is broken the moment a post-treatment regressor is introduced. The only assumption-light remedy is to make $S$ itself respond to a randomization that is independent of treatment assignment --- in other words, to redesign the experiment.
We add a third arm in which subjects receive treatment $A$, but the mediator is intervened upon and set to a value drawn from the distribution that $S$ takes under $B$. Concretely, subjects are randomly assigned to one of three arms:
Arm 2 is the new ingredient: subjects assigned to it receive the substantive treatment $A$, but the mediator is fixed by the experimenter to a value drawn from the distribution of $S(B)$. This decoupling of treatment from mediator is the central design feature; it mirrors the “parallel design” of imai2013experimental and the mechanism-experiment logic discussed by ludwig2011mechanism.
Given the three observed means above, define
The natural indirect effect (NIE) isolates the mediator: treatment is held fixed at $A$ while $S$ shifts from its $A$-distribution to its $B$-distribution, capturing the part of the total effect that operates through algorithmic targeting alone. The natural direct effect (NDE) isolates the substantive treatment channel: the mediator is held fixed at its $B$-distribution while treatment shifts from $A$ to $B$, capturing the part operating through the user-visible content. With this convention, a positive NIE on a delivery outcome to a target demographic indicates that the algorithm allocates more impressions to that demographic in response to the treatment-metadata signal. The decomposition
follows algebraically from (ref)--(ref) and is exact pearl2001direct, vanderweele2015explanation.
The crucial point is that, in the three-arm design, each of the three means in (ref)--(ref) is identified by a simple sample mean within an arm. No cross-equation regression and no sequential ignorability assumption is required. The only assumptions invoked are:
Assumption (A1) is delivered by the randomization. Assumption (A2) is the standard SUTVA-style excludability assumption angrist1996identification. Assumption (A3) is the only design-specific assumption and is verifiable: one can compare the empirical distribution of $S$ in arm 2 to that of $S$ in arm 3, and one can test the equality of those distributions by a Kolmogorov--Smirnov or randomization-based test.
Under (A1)--(A3), the sample analogues
are unbiased for the corresponding population quantities, where $\bar{Y}_z$ is the mean outcome in arm $z$. Standard errors follow from Neyman's variance for differences of arm means neyman1923, gerber2012field, and a heteroskedasticity-consistent variance estimator (HC2) gives valid inference in finite samples.
A subtlety merits emphasis. The decomposition in (ref) as defined here uses the marginal distribution of $S(B)$ when imposing $S$ in arm 2, not each subject's own counterfactual $S_i(B)$. In the language of pearl2001direct and vanderweele2015explanation, this identifies a population-average natural decomposition, or equivalently a controlled direct effect (CDE) marginalized over the distribution of $S(B)$. The unit-level natural direct effect, $\mathbb{E}[Y_i(A, S_i(B)) - Y_i(B, S_i(B))]$, would additionally require the cross-world independence assumption $Y(a, s) \perp\!\!\!\perp S(a') \mid X$ for $a \neq a'$ robins2003semantics. This is empirically irrefutable and is precisely the assumption the three-arm design is intended to avoid.
For most applied purposes the population-average decomposition is the policy-relevant object: it answers the counterfactual “what would the contrast $A - B$ look like if delivery were equalized at $B$-levels?” Researchers wishing the unit-level decomposition must either invoke sequential ignorability imai2010general or use a sensitivity analysis along the lines of imai2010identification to bound the resulting bias.
Three issues warrant attention at the design stage.
\paragraph{Spillovers and SUTVA.} If subjects communicate or if $S$ has equilibrium feedback, the three-arm design must be combined with cluster-level randomization or exposure-mapping analysis aronow2017estimating, gerber2012field. The decomposition above continues to hold within exposure conditions but the conditioning set must be made explicit.
\paragraph{Power.} Splitting the sample three ways reduces the precision of any single pairwise contrast by roughly a factor of $\sqrt{2/3}$ compared with a two-arm design at the same total sample size. Because arm 2 enters both $\widehat{\mathrm{NDE}}$ and $\widehat{\mathrm{NIE}}$, an unequal allocation favoring arm 2 (e.g., $1:2:1$) often minimizes the total variance of the decomposition; see imai2013experimental for the relevant optimization.
Let $Z_i \in \{1, 2, 3\}$ index the arm of unit $i$ and let $n_z = \sum_i \mathbf{1}\{Z_i = z\}$. The point estimators are simple group differences; valid standard errors come from the heteroskedasticity-consistent estimator
where $s_z^2$ is the within-arm sample variance neyman1923.\footnote{Equation (ref) is the population-level Neyman variance, valid when impressions within an arm are independent and homoskedastic. In production ad-delivery experiments these assumptions are typically violated: the algorithm's daily state induces within-arm structure, and outcome variance is heteroskedastic across cells. We therefore use the heteroskedasticity-robust HC3 estimator mackinnon1985some as our primary SE, computed on the weighted regression at the (day $\times$ time-slot $\times$ arm) level, with weights equal to total impressions. A dispersion diagnostic at the (day $\times$ time-slot) level (Appendix (ref)) confirms that the Bernoulli SE in (ref) understates the true variance by a factor of approximately $5\times$ for the delivery outcome but coincides with HC3 for engagement (CTR). Multiple-testing across the three pairwise contrasts (and, in the disaggregated analyses of Section (ref), across $36$ cell-level contrasts) is controlled by the Romano-Wolf step-down procedure of romano2005stepwise, list2019multiple, using the wild bootstrap of liu1988bootstrap, wu1986jackknife with Rademacher weights and centered t-statistics davidson2008wild. The wild bootstrap is the standard pairing with HC3 because it preserves the design matrix exactly and resamples only residual signs. Implementation details are in the replication code; cluster-robust alternatives bertrand2004much, cameron2008bootstrap, mackinnon2023cluster produce qualitatively similar conclusions but are not used as the headline specification because the 7-day panel has only $G = 7$ day clusters.} In a regression representation,
and contrasts of the $\hat{\mu}_z$ recover $\widehat{\mathrm{NIE}}$, $\widehat{\mathrm{NDE}}$ and $\widehat{\mathrm{TE}}$. With HC2 standard errors mackinnon1985some the inference in (ref) coincides asymptotically with that of (ref). Pre-treatment covariates may be added via the interacted specification of lin2013agnostic to improve precision without compromising unbiasedness.
The three-arm design achieves unbiased identification of the direct and indirect channels for two reasons. First, each estimand in the decomposition is the difference of two arm-means whose populations are made comparable by physical randomization, not by statistical adjustment. Under (A1)--(A3) the expectations $\mathbb{E}[Y(A, S(A))]$, $\mathbb{E}[Y(A, S(B))]$, and $\mathbb{E}[Y(B, S(B))]$ are each identified as the population mean of $Y$ within the corresponding arm. Second, because arm 2 is constructed to satisfy a single experimentally enforced property --- that $S$ follows the marginal distribution of $S(B)$ --- the design replaces the unverifiable cross-world independence assumption used in observational mediation analysis with an assumption that can be checked by comparing the realized distributions of $S$ in arms 2 and 3. The decomposition is therefore robust to unmeasured confounding between $S$ and $Y$, which is the principal source of bias in regression-based mediation gerber2012field, acharya2016explaining. As heckman2015econometric emphasize, the credibility of a mediation decomposition rests entirely on the source of variation in the mediator; in the design proposed here, that source is randomization.
Before applying the three-arm design to live ad-delivery data, we validate the decomposition in a controlled simulation where the population values of $\mathrm{NIE}$, $\mathrm{NDE}$, and $\mathrm{TE}$ are known by construction. The full simulation specification is documented in the companion paper fb_ad_auction_model.tex; we summarise the model and the key findings here.
The simulation instantiates a Meta-style ad auction. $N_u = 10{,}000$ users with gender share $\pi_M = 0.462$ are matched to $N = 500$ advertisers in two stages. Each user is assigned the top $\lceil q N \rceil = 50$ advertisers by a matching signal $s_{u,i}$, which equals the true click rate $r_{u,i}$ plus noise: $s_{u,i} = r_{u,i} + \varepsilon_{u,i}$. Among the matched advertisers, the highest bidder wins a single second-price auction and serves the impression. Click rates are i.i.d.\ across $(u, i)$ pairs from a $\mathrm{Beta}(\alpha, \beta)$ distribution with mean $\mu_r = 0.02$, and bids are i.i.d.\ $\mathrm{LogNormal}$ across advertisers and held fixed within batches.
The mediator $S$ in the methodology paper instantiates as the noise variance $\sigma^2_{i, g}$ of the matching signal. The two potential mediator distributions are: $S(A)$ --- every advertiser has a perfect predicted click rate ($\sigma^2_{i, g} = 0$ for all $i, g$); $S(B)$ --- advertisers running creative $B$ have a noisy predicted click rate on female users only ($\sigma^2_{B, F} = \sigma^2 > 0$, all other cells zero). The three arms then map onto the methodology paper's $(D, S)$ conditions:
Under the maintained no-creative-effect assumption (click rate primitive $r_{u,i}$ does not depend on creative), the population $\mathrm{NDE}$ is exactly zero. This makes arm-3 minus arm-2 a built-in placebo: any non-zero estimate must come from finite-sample noise. The population $\mathrm{NIE}$ and $\mathrm{TE}$ coincide and equal the change in delivery induced by the noisy signal regime.
We run $R = 1{,}000$ Monte Carlo rounds and report the female share of focal-advertiser impressions as the primary delivery outcome. The arm-level summary at the high focal bid ($b_0 = \$3.63$, the 99.5th percentile of the bid distribution) is:
The decomposition is reported in Table (ref).
The pattern matches the methodology paper's logic. The $\mathrm{NIE}$ recovers the algorithm-induced reallocation under the noisy signal regime ($+18.45$ percentage points). The $\mathrm{NDE}$ is statistically indistinguishable from zero ($-0.09$ ppt, 95% CI $[-0.46, +0.28]$), as the no-creative-effect specification dictates. The $\mathrm{TE}$ equals the $\mathrm{NIE}$ within sampling noise, confirming the algebraic decomposition $\mathrm{TE} = \mathrm{NIE} + \mathrm{NDE}$ at the sample level.
Sweeping the focal bid across the 95th, 98th, and 99.5th percentiles of the valuation distribution produces:
{ $^{\dagger}$ 95% CI includes zero (placebo passes).}
The $\mathrm{NIE}$ rises monotonically with bid: a higher focal bid converts more of the matches that arm 2's noisy signal generates into impressions, so the indirect channel becomes more visible at the impression level. The $\mathrm{NDE}$ remains a placebo at every bid, as it should. The match-level decomposition (reported in the companion paper, Appendix A of fb_ad_auction_model.tex) is bid-independent because the matching stage does not see bids; this is the cleanest version of the indirect channel.
The empirical $\mathrm{NIE}$ on female share at high bid (Table (ref) below) is $+2.07$ ppt. The simulation produces $+18.45$ ppt under the calibrated $\sigma = 0.030$. The gap is a deliberate calibration choice: the simulation $\sigma$ is set well above plausible production-system noise so that the indirect channel is unmistakable, and the displayed magnitude is intended as a stress test of the methodology rather than a calibration to the empirical scale. Setting $\sigma \approx 0.004$ reproduces the empirical $+2.07$ at the focal advertiser's actual bid percentile. Inverting the simulation to recover production $\sigma$ from the empirical $\mathrm{NIE}$ is a natural follow-up.
The companion document fb_ad_auction_model.tex contains the full specification, including the match-level decomposition (which is robust to bid choice) and the engagement-side decomposition on $\mu_r$ and $\mu_s$ (which separates platform-predicted CTR from realised CTR and recovers the noisy-signal selection cost). The simulation code is in fb_ad_auction_experiment.R.
We apply the three-arm design to an ad-delivery experiment run on Meta. The experiment manipulates whether a women-targeted text fragment (denoted “wd F”) is present in the ad metadata (visible to the delivery algorithm) and whether that fragment is also rendered to the user. The full design crosses the three arms with two bid levels, giving a parallel three-arm experiment at high and low bids.
The data are recorded at the level of (ad creative $\times$ age band $\times$ gender $\times$ day $\times$ time-of-day) cells, with impressions, link clicks, reach, and spend aggregated within each cell. The campaign objective is traffic optimization; six demographic age bands (18--24, 25--34, 35--44, 45--54, 55--64, 65+) and three reported gender categories (female, male, unknown) span the observed audience.
The three arms within each bid level correspond to the methodology's $(D, S)$ conditions of Section (ref) as follows:
Figure (ref) shows the user-facing creative for each arm as it appeared in the news feed. The control creative (Arm 1) and the metadata-only creative (Arm 2) are pixel-identical to the user; the two arms differ only in the campaign metadata that the delivery algorithm conditions on. The visibly female-coded creative (Arm 3) replaces the generic body text with “Women applicants encouraged.”
The design is replicated at high and low bid levels, providing a second source of variation that we exploit in Section (ref).
\paragraph{Campaign features held constant.} The following configurations are identical across the three arms within each bid level:
\paragraph{Features intentionally varied.} The ad creative differs across the three arms (the manipulation of interest), and the bid amount differs across the two parallel three-arm experiments (a secondary source of heterogeneity).
The structural symmetry of these configurations ensures that total realized impressions across arms differ by less than $1\%$ at high bid (33{,}621, 33{,}554, 33{,}860 in arms 1, 2, 3 respectively), consistent with the budget cap binding symmetrically across arms.
We first audit pairwise audience imbalance across arms using the diagnostic framework of burtch2025characterizing. For each of the three pairwise comparisons (1 vs. 2, 2 vs. 3, 1 vs. 3) within each bid level, we compute two-sided t-tests of the difference in impression share for every (age $\times$ gender) cell, and the corresponding standardized mean differences (SMDs). The arm pairings carry distinct substantive interpretations under our design: 1-vs-2 reveals the indirect channel (algorithmic targeting in response to metadata, with visible creative held fixed at $A$); 2-vs-3 audits assumption (A3) (whether the realized $S$ in arm 2 matches that in arm 3); and 1-vs-3 is the conventional divergent-delivery imbalance reported in the existing literature.
Plots of the empirical CDF of $p$-values and the distribution of SMDs are presented in Appendix Figures (not reproduced here for brevity); the underlying tabulation by (age $\times$ gender) is the primary descriptive object. Aggregating across cells, the female share of impressions at high bid is $34.49\%$, $36.56\%$, and $35.86\%$ in arms 1, 2, and 3 respectively.
We use the same inference layer for every contrast reported in this section. For each outcome $y$ (impression share to demographic $g$, or click-through rate), we fit a weighted ordinary least squares regression on the (day $\times$ time-slot $\times$ arm) panel with treatment dummies, weighted by total impressions: \[ y_{d,t,z} = \alpha + \beta_1 \mathbf{1}\{z = \mathrm{T}_1\} + \beta_2 \mathbf{1}\{z = \mathrm{T}_2\} + \varepsilon_{d,t,z}. \] Standard errors on $\hat{\beta}_1$, $\hat{\beta}_2$, and their difference $\hat{\beta}_2 - \hat{\beta}_1$ are HC3 robust mackinnon1985some, computed from the full covariance matrix of the WLS estimator. The contrast $\hat{\beta}_2 - \hat{\beta}_1$ uses the covariance-corrected variance $\operatorname*{Var}(\hat{\beta}_1) + \operatorname*{Var}(\hat{\beta}_2) - 2\operatorname*{Cov}(\hat{\beta}_1, \hat{\beta}_2)$. The three pairwise contrasts $\widehat{\mathrm{NIE}} = \hat{\beta}_1$, $\widehat{\mathrm{TE}} = \hat{\beta}_2$, and $\widehat{\mathrm{NDE}} = \hat{\beta}_2 - \hat{\beta}_1$ are recovered from this single regression.
Multiple-testing within each family (the three pairwise contrasts for the aggregate analysis; the $36$ contrasts for the per-cell analysis of Section (ref)) is controlled by the Romano-Wolf step-down procedure romano2005stepwise, list2019multiple, implemented via the wild bootstrap liu1988bootstrap, wu1986jackknife, davidson2008wild. For each bootstrap iteration $b = 1, \ldots, B = 5{,}000$, we draw $\omega_i \in \{-1, +1\}$ i.i.d. for each observation $i$, construct $y^*_i = \hat{y}_i + \omega_i \hat{\varepsilon}_i$, re-fit the regression, recompute HC3 standard errors, and form centered $t$-statistics $t^*_{b,k} = (\hat{\beta}^*_{b,k} - \hat{\beta}_k)/\widehat{\mathrm{SE}}_{\mathrm{HC3}}(\hat{\beta}^*_{b,k})$. The step-down algorithm of romano2005stepwise delivers adjusted $p$-values $p_{\mathrm{adj}}$ that control family-wise error rate while accounting for the joint dependence structure across hypotheses.
The decision rule between Bernoulli SE and HC3 SE is data-driven, following the dispersion-diagnostic recommendation of Appendix (ref). For delivery outcomes the diagnostic shows substantial over-dispersion (ratio of observed to Bernoulli variance approximately $25\text{--}80$ depending on cell), so HC3 is the appropriate primary SE. For engagement (CTR) outcomes the dispersion ratio is close to one at the day level, indicating Bernoulli SE is approximately correct; HC3 coincides numerically. We therefore report HC3 throughout for consistency, with the understanding that Bernoulli SE would yield identical conclusions for CTR and substantially anti-conservative conclusions for delivery.
For delivery outcomes specifically, the indirect effect captures the differential allocation of impressions across arms produced by the algorithmic mediator alone. We estimate this on the female share of impressions, which corresponds to summing the delivery to all female cells across age bands. Inference uses the heteroskedasticity-robust HC3 estimator with wild-bootstrap-based Romano-Wolf step-down adjusted p-values (see Section (ref)).
The two channels work in opposite directions. The indirect channel is positive in point estimate: the algorithm's response to the metadata fragment shifts the female share of impressions upward by $2.07$ percentage points ($t = +1.12$ on HC3 SE). The direct channel is negative: making “wd F” visible to users reduces the female share by $0.68$ percentage points relative to the metadata-only condition ($t = -0.32$). The total effect is positive in point estimate ($+1.39$ pp, $t = +0.63$) but smaller than the indirect channel alone, because the direct channel partially offsets the algorithm's targeting. The indirect channel's absolute magnitude accounts for approximately $\tfrac{2.07}{2.07 + 0.68} \approx 75\%$ of the absolute decomposition.
None of the three aggregate contrasts crosses conventional significance under wild-bootstrap-adjusted inference (all $p_{\mathrm{adj}} \geq 0.48$). The point estimates are stable across specifications --- the algorithmic-targeting channel sign and magnitude survive every variation of inference we examine --- but the 7-day pilot panel yields HC3 standard errors approximately $5\times$ the naïve Bernoulli baseline (Appendix (ref)), so this sample lacks the power to identify a $2$-percentage-point algorithmic shift at the $5\%$ level. The non-zero point estimate of $\widehat{\mathrm{NDE}}$ on a pure delivery outcome is interpretable as evidence that engagement feedback has partially updated $S^{(3)}$ away from $S^{(2)}$ within the analysis window: were the two algorithmic states identical throughout, $\widehat{\mathrm{NDE}}$ on delivery would be exactly zero.
Decomposition by individual (age $\times$ gender) cell, presented in the next subsection, sharpens this picture: the aggregate $+2.07$ pp on female share is the sum of several substantial cell-level reallocations that the aggregate measurement compresses.
The aggregate decomposition in Section (ref) averages across age bands. Because the algorithm targets on multiple demographic dimensions simultaneously, the cell-level decomposition reveals a richer reallocation pattern than the gender-only summary suggests. Table (ref) reports $\widehat{\mathrm{NIE}}$, $\widehat{\mathrm{NDE}}$, and $\widehat{\mathrm{TE}}$ on impression share for each of the twelve (age $\times$ gender) cells at high bid (excluding the “unknown” gender category, which constitutes less than $1\%$ of impressions). Cells are scaled to impression-share percentage points; $t$-statistics use HC3 robust standard errors. The $p_{\mathrm{adj}}$ column reports wild-bootstrap Romano-Wolf step-down adjusted p-values, jointly controlling family-wise error rate across the 36 cell-level contrasts.
Three substantive patterns emerge in the point estimates. Although no individual cell-level contrast survives joint family-wise multiple-testing correction at the 5% level under wild-bootstrap Romano-Wolf (the smallest adjusted $p$-value is $0.33$ for the 65+ male $\widehat{\mathrm{NIE}}$), the directional pattern is consistent and internally coherent across cells.
\paragraph{The algorithm targets middle-aged women.} The largest positive cell-level $\widehat{\mathrm{NIE}}$ point estimates appear in the 35--44 female cell ($+1.50$ pp), 45--54 female ($+0.70$ pp), and 25--34 female ($+0.49$ pp) cells. The algorithm's response to the metadata fragment is concentrated in working-age women rather than spread uniformly across all female sub-populations. The 18--24 and 65+ female cells receive much smaller positive (or, in the case of 65+, negative) $\widehat{\mathrm{NIE}}$. This pattern is consistent with the algorithm operationalizing “targeting women” as “targeting working-age women,” the demographic most likely to engage with traffic-optimized campaigns at the prevailing bid.
\paragraph{Compensating reallocation away from older users.} To deliver the impression shift toward middle-aged women under a fixed budget, the algorithm reallocates impressions away from other cells. The 65+ band shows the most pronounced negative $\widehat{\mathrm{NIE}}$ for both genders ($-1.09$ pp for females, $-1.58$ pp for males). The 25--34 male and 55--64 male cells also lose impressions. These cells function as the “budget source” for the algorithm's targeting move: the metadata signal causes the delivery algorithm to depreciate predicted relevance for these audiences and reallocate the freed budget to working-age women. The aggregate $+2.07$ pp on female share is therefore the net result of two-sided reallocation, not a uniform shift.
\paragraph{The visible content broadens delivery back toward older users.} The direct-effect column shows a striking sign-inversion of the indirect-effect pattern. Cells with the largest negative $\widehat{\mathrm{NIE}}$ (65+ male, 65+ female, 55--64 male) display the largest positive $\widehat{\mathrm{NDE}}$ point estimates ($+1.92$, $+1.42$, $+1.12$ pp respectively). Cells with the largest positive $\widehat{\mathrm{NIE}}$ (35--44 female, 25--34 female, 45--54 female) display negative $\widehat{\mathrm{NDE}}$. The visible “wd F” creative therefore systematically appears to undo part of the algorithm's metadata-driven targeting move, restoring impressions to the older demographic that the algorithm initially de-prioritized and pulling them away from the working-age women the algorithm initially targeted.
\paragraph{Two interpretations.} The cell-level sign-inversion of $\widehat{\mathrm{NDE}}$ relative to $\widehat{\mathrm{NIE}}$ is consistent with two distinct mechanisms, which the present design cannot fully separate. First, the visible female-coded creative may broaden the ad's appeal to older audiences (intrinsic-content channel). Second, engagement feedback from arm 3 may have updated $S_t^{(3)}$ away from $S(B)$ within the analysis window in a direction that partially counteracts the metadata signal --- a dynamic-contamination channel operating through arm-3 user response. The systematic cell-level mirroring between $\widehat{\mathrm{NIE}}$ and $\widehat{\mathrm{NDE}}$ point estimates is more readily reconcilable with the second mechanism; we defer a fuller treatment of dynamic considerations to future work (see also further_methodology).
\paragraph{The total effect understates the algorithmic channel.} The $\widehat{\mathrm{TE}}$ column shows that the two channels approximately cancel for several cells (18--24, 65+, 55--64). A standard A/B-test analysis would report only the $\widehat{\mathrm{TE}}$ and conclude that the treatment moves delivery little in these bands. The three-arm design reveals that the algorithm does substantially reallocate impressions across these cells: it pulls them out via metadata and partially restores them via visible content. Conventional A/B-test reporting therefore understates the magnitude of algorithmic differential treatment by approximately a factor of two for the cells where the two channels offset.
Replicating the decomposition at the low-bid experiment provides a second source of heterogeneity. Table (ref) reports the decomposition of female impression share at each bid level under HC3 inference with wild-bootstrap Romano-Wolf adjusted p-values.
The composition of the decomposition reverses across bid levels in point-estimate terms, though the low-bid contrasts are statistically null on their own. At high bid the indirect channel dominates in magnitude ($+2.07$ pp) and the direct channel is small and opposite-signed ($-0.68$ pp); at low bid the indirect channel's point estimate goes near zero ($-0.83$ pp) and the direct channel rises in magnitude ($+4.08$ pp). The cross-bid differences in NIE and NDE are large in absolute terms but imprecisely estimated --- the low-bid arms accumulated approximately $1{,}600$ impressions total, less than $5\%$ of the high-bid sample, so HC3 standard errors at low bid exceed $6$ percentage points and no contrast crosses 5% significance. The directional pattern is consistent with a coherent substantive interpretation: at higher bids the algorithm wins more competitive auctions and has greater latitude to differentiate audiences by metadata, magnifying the indirect channel; at lower bids the algorithm has less room to optimize, and the user-visible content carries proportionally more of the cross-arm difference. This is consistent with the configuration-mitigation argument of burtch2025characterizing, who show that uniform campaign configurations reduce divergent delivery.