Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
66,079 characters · 17 sections · 30 citation commands
Two-Sided Prioritized Ranking: A Coherency-Preserving Design for Marketplace Experiments
\thispagestyle{empty}
Online platforms such as e-commerce sites and online marketplaces rely heavily on randomized controlled experiments to guide product decisions. These experiments help platforms evaluate changes safely, improve user experience, and increase engagement and sales, while providing timely and credible feedback on new features kohavi2020trustworthy, bojinov2022online, xia2019safe, xu2018sqr, kohavi2009online.
Standard experimental designs rely on the Stable Unit Treatment Value Assumption (SUTVA), which rules out spillovers across units rubin1974estimating, imbens2015causal. In online marketplaces this assumption is frequently violated. Items compete for users' limited attention and demand within ranked lists, so modifying treated items (for example, via discounts or price increases) can change outcomes for untreated items through substitution or complementarity. Such interference (spillovers, network dependence) has been documented in ridesharing platforms chamandy2016experimentation and online pricing experiments choi2019monetizing. When interference is ignored, estimates from randomized experiments can be substantially biased blake2014marketplace, fradkin2019simulation, hudgens2008toward. For example, if we discount Hotel A in a search result, users who would have booked Hotel B now book A instead, making B appear to perform worse, not because of its true quality, but because of demand spillovers from the treated item.
Under interference, treatment effects depend on how treatment is distributed across units. A natural policy-relevant contrast is the total effect of moving from a fully untreated world to a fully treated world. This contrast is formalized as the global treatment effect in the interference literature hudgens2008toward, manski2013identification, munro2024treatmenteffectsmarketequilibrium.
Interference alone does not preclude credible estimation of global treatment effect, but the experimental designs that would address it are often infeasible in marketplace settings due to operational constraints. User-level randomization, where all items shown to treated users receive treatment, would avoid mixing treated and untreated items within the same query. However, this approach violates what we call coherency: the requirement that all users observe the same realized price (or other treatment attribute) for any given item, and that all users retain access to the full catalog of items throughout the experiment.
Price parity is sometimes a legal requirement, and more often a reputational one. Overt price variation is tightly regulated under European competition law eu_tfeu_art102, ec_application_art102, which makes user-level price experiments, that show different users different prices, difficult to deploy without raising compliance and reputational concerns. Even when not explicitly illegal, platforms face acute trust and brand risks. Consumer reactions to visible price dispersion are typically strong, and perceived unfairness can dominate any short-run learning benefits ccakir2025price.
Recent reporting describes tests on Instacart in which shoppers were charged different prices for identical items at the same store, a difference that could impact annual spending by as much as $\$1,200$ kravitz2025instacart_ai_pricing_bibtex. A September 2025 Consumer Reports survey of 2,240 U.S. adults found that 72 percent of Instacart users opposed such price discrimination for any reason consumerreports2025_aes_september_omnibus. Following public scrutiny, Instacart announced in December 2025 that it would end “item price tests,” noting that showing different prices for the same item at the same store fell short of customer expectations instacart2025ending_item_price_tests. These concerns are longstanding; for instance, Amazon's 2000 DVD pricing experiment triggered immediate backlash and led to necessitating public apologies and refunds wired2000priceamends. Disclosing that a price difference is part of an experiment not only risks reputational costs but also undermines internal validity by altering user behavior. Workarounds such as coupon codes or targeted promotions introduce their own confounding incentives and can complicate interpretation of price effects.
A second coherency requirement is full catalog access. Many platforms cannot remove items from search results or show different item sets across users without degrading the user experience, distorting substitution patterns, and harming revenue. Designs that vary availability conflate the effect of the intervention with the effect of restricting choice sets, and they may induce strategic seller or user responses that do not reflect business-as-usual behavior.
These two constraints, along with practical considerations, rule out many standard experimental designs. User-level randomization violates coherency by showing different prices to different users. Item-level randomization preserves coherency but suffers from substantial bias under interference blake2014marketplace. While cluster randomization can reduce interference by grouping related units ugander2013graphclusterrandomizationnetwork, ugander2023randomized, holtz2024reducing, saveski2017detecting, it suffers from low statistical power and requires well-defined clusters that align with spillover patterns, which is often difficult in marketplaces where interactions evolve dynamically, and can be computationally expensive to implement candogan2023correlated.
Recent advances in network interference, such as exposure-based designs, typically require researchers to specify an exposure model or mapping that defines how treatment spillovers propagate across units (e.g., aronow2017estimating, harshaw2023design), while analysis-based corrections can debias naive estimates when the platform intermediates spillovers through a known mechanism such as matching bright2025reducing, chin2019regression. However, this is difficult to specify and validate in marketplace settings where substitution patterns are context-dependent and high-dimensional. Switchback testing alternates treatment assignments over time for the same units brown1980crossover, robins1986new, sneider2019experiment, bojinov2023design. While switchbacks can support causal identification in time-varying environments, frequent treatment fluctuations can confuse users and distort engagement patterns. For salient interventions such as prices, these fluctuations may also create carryover effects that undermine internal validity.
Two-sided randomization (TSR) methods apply randomization on both the user side and the item side johari2022experimental, bajari2023experimental, li2022interference. Standard TSR implementations apply treatment only when a treated user interacts with a treated item, which can lead different users to see different versions of the same item (including different prices). This violates the coherency requirement that motivates our setting. This creates a methodological gap: how can platforms credibly estimate treatment effects while maintaining both price parity and full catalog access under interference?
We introduce Two-Sided Prioritized Ranking (TSPR), an experimental design for item-side interventions in ranked-list marketplaces that maintains price parity and full catalog access while addressing interference. Under plausible conditions, we show that TSPR identifies the proportional effect of global treatment.
TSPR exploits a feature of modern marketplaces: centralized recommender systems rank items for each user query, and users exhibit strong position bias, allocating disproportionate attention to top-ranked items craswell2008experimental, friedberg2022causal, joachims2017accurately, richardson2007predicting. Empirical evidence shows a steep decline in click probability as an item moves down the ranking. These models imply that observed clicks combine position and relevance effects. The key insight is that while we cannot vary treatment status across users or remove items (coherency), we can vary users' exposure to treatment by systematically reordering the ranked list.
Specifically, TSPR randomizes users into two groups and reorders each user's ranked list so that treated items are prioritized at the top for one group and untreated items are prioritized for the other. This induces systematic variation in treatment exposure through position bias while preserving coherency: all users retain access to the same underlying item set, and each item's treatment status remains consistent across users.
Prior work on interference in ranking experiments often focuses on evaluating or improving ranking algorithms goli2024bias, zhan2024estimating, nandy2021b, ursu2018power. Our objective is different: we do not treat the recommender system as the object of experimentation, but instead use it as the mechanism through which item-side treatment exposure is shifted while preserving coherency. This differs from approaches that rely on naturally occurring ranking noise as exogenous variation, and from interleaving-style methods that primarily optimize ranking quality rather than deliver coherent item-side interventions.
Using an open-source Expedia hotel search dataset, we estimate behavioral models of click and booking decisions and conduct Monte Carlo simulations to evaluate performance of our method in estimating global treatment effect. TSPR substantially reduces both bias and variance relative to item-level A/B tests, and strongly outperforms cluster-randomized designs on the variance of estimates. These results demonstrate that ranking-based designs can credibly estimate treatment effects in settings where standard methods fail due to interference or operational constraints.
Section (ref) formally defines the TSPR design and derives conditions under which it identifies the global treatment effect. Section (ref) describes the semi-synthetic simulation framework calibrated to Expedia hotel search data. Section (ref) presents Monte Carlo evidence that TSPR substantially reduces bias and improves efficiency relative to Bernoulli-randomized and cluster-randomized baselines. Section (ref) interprets these findings, characterizing when TSPR offers the greatest advantages and when simpler designs suffice. Section (ref) concludes.
We model a two-sided platform as a matching mechanism between a set of queries $q \in Q$, which represent user inputs, and a set of items $i \in I$, which represent the available options. The platform uses a recommender system to compute relevance scores $r_{q,i} \in \mathbb{R}$ for each query–item pair based on attributes of the query and the item, such as user preferences and item features. When a user submits query $q$, the platform ranks all available items in descending order of $r_{q,i}$ and displays the ordered list to the user. After viewing the list, the user may interact with some of the displayed items, and these interactions generate outcomes $y_{q,i}$. For simplicity, we assume that all items begin with outcome value zero and that $y_{q,i}$ takes non-negative real values after user interaction, representing clicks, bookings, or revenue. Because each user submits exactly one query in our setting, we use the terms “user’’ and “query’’ interchangeably.
In this environment, standard item-level A/B testing fails to produce unbiased estimates of treatment effects because items shown together in the same query can affect each other’s outcomes. This violates the Stable Unit Treatment Value Assumption (SUTVA) due to interference between items. Any experimental design for this setting must also satisfy two operational constraints. First, users must retain access to the full catalog of items during the experiment. Second, all users must observe a coherent realization of item treatment status, meaning that every user sees the same version of each item throughout the experiment. These constraints rule out many existing designs and motivate the structure of our Two-Sided Prioritized Ranking approach.
Due to item-side interference, the effect of a binary treatment $T_i \in \{0,1\}$ on item–query outcomes $y_{q,i}$ depends on how treatments are distributed across items. This motivates our focus on the global lift ($\Phi$), which captures both direct effects and spillovers by comparing expected outcomes under full treatment and full control. Because our estimand is defined at the query level, we aggregate item outcomes within each query and work with $Y_q = \sum_i y_{q,i}$. For notational simplicity we omit the query index and write $Y$.
We define global lift as
where $I$ is the set of all items. The numerator corresponds to the expected query-level outcome when all items are treated, and the denominator corresponds to the expected outcome when all items are untreated. Since in practice each item can only be in one treatment state at a time, only one of these two quantities is observed, which makes $\Phi$ fundamentally a counterfactual estimand. This estimand is in one-to-one correspondence with the total average treatment effect emphasized in the interference literature hudgens2008toward, manski2013identification, munro2024treatmenteffectsmarketequilibrium.
The proposed method rests on several assumptions. First, we assume that items at the top of the listing exert a disproportionate influence on user behavior craswell2008experimental, and that this influence declines rapidly with rank. Effective exposure to the treatment therefore depends on the extent to which treated items appear near the top of the ranked list, since these positions receive most of the user’s attention. By strategically altering the ordering of items, we manipulate users’ effective exposure to treated versus untreated items.
Second, the method requires that each query contains a sufficiently large set of relevant items. This ensures that the repositioning scheme can meaningfully increase the exposure of one group of queries to treated items while decreasing it for the other group.
Third, we assume that user-side interference is negligible. This corresponds to a slack-supply environment in which inventory or availability constraints are not binding over the experiment horizon. Under slack supply, one user’s actions do not affect item availability for others, and interference arises entirely within queries, across items displayed in the same ranked list. In our model, within-query interference operates through two mechanisms: (i) limited attention to early ranks and (ii) unit-demand substitution, since booking one item reduces the probability that other items in the same query are chosen.\footnote{Extending the design to settings with binding capacity constraints or other forms of user-side interference across queries or users is left to future work.}
Our proposed experimental design for estimating total lift is summarized in Table (ref), with Figure (ref) illustrating the two-sided randomization scheme and group-specific listing priorities for query results.
{2pt} {2pt}
As outlined in Table (ref), after specifying the experiment intensity parameter $p$, we begin by partitioning items into three subsets: Treated, Untreated, and Placebo, with probabilities $p$, $p$, and $1 - 2p$, respectively. Only items in the Treated subset receive the intervention. The inclusion of a Placebo subset is essential for maintaining balance in the experiment.
In marketplace experiments, the probability of assignment to either treatment or control is typically well below 0.5, often on the order of a few percent, in order to limit opportunity costs and to mitigate potential negative effects on user experience if the new feature performs worse than the existing one ha2020counterfactual. Without a Placebo subset, the Untreated subset would be substantially larger than the Treated subset. This would create an asymmetric effect in step 3 of our design. In particular, for queries in $Q^A$, where non-treated items are prioritized, the larger Untreated pool would produce top-ranked items of higher average quality than the top-ranked items drawn from the smaller Treated pool shown to $Q^B$. Such an imbalance would cause the recommender system modification to affect the two query groups differently, confounding the estimation of the intervention’s effect.
The Placebo subset prevents this imbalance by ensuring that the Treated and Untreated subsets are of comparable size. As a result, the expected match quality of top-ranked items is similar across the two user groups, which allows the variation induced by the prioritization scheme to isolate the treatment effect rather than reflect differences in pool size or quality.
Placebo items also create a buffer between treated and untreated items in the ranked list. This separation sharpens the interpretation of rank depth as treatment exposure in our partial-outcome contrasts: at small depths, outcomes are driven primarily by exposure to the prioritized block rather than by immediate mixing of treated and untreated items. As a result, placebo reduces contamination of the control arm from treated items and improves the signal-to-noise ratio of the within-depth contrasts that identify lift.
In the next step, incoming queries are randomized into $Q^A$ or $Q^B$ with equal probability. Item priorities are then assigned so that queries in $Q^A$ receive items in the order Untreated, Placebo, Treated, while queries in $Q^B$ receive items in the reverse order: Treated, Placebo, Untreated. This prioritization induces systematic differences in exposure to treated items across the two query groups.
This section introduces the analytical framework used throughout the paper. We begin by outlining the setup and notation, then define the estimand of interest that captures treatment effects under varying ranking and attention conditions. We next formalize the identifying assumptions required for consistent estimation and describe the estimator that operationalizes these ideas in practice.
The parameter $\Phi$ captures the relative (multiplicative) effect of treatment. It is defined as the proportional lift in the total outcome under universal treatment (all items treated) relative to universal control (all items untreated) (Equation (ref)). Thus, $\Phi$ represents the percentage change in expected total outcomes when all items are treated.
A TSPR experiment is characterized by the set of treatment-prioritized queries $Q^B$, the set of control-prioritized queries $Q^A$, the set of items $\mathcal{I}$, the treatment intensity $p$, the treatment type $T$, and the randomization and re-ranking scheme implemented according to Algorithm (ref). Let $y_{q,i}$ denote the user/query $q$'s outcome of the item displayed at rank $i$.
For notational simplicity, we drop the query index $q$ and refer to query-level outcomes as $Y$. All expectations are taken over queries within a given experimental arm.
Assumption (ref) implies that if treatment were rolled out to all items but still under original recommender system, the expected partial outcome would satisfy
We now characterize how moving from the platform’s original ranking to the TSPR ranking experiment (Table (ref)) changes expected partial outcomes. There are two channels. First, re-ranking can change user attention, meaning how attention is allocated across positions (for example, time spent evaluating items and clicks). Second, it can change outcomes through the treatment itself. Assumptions (ref) and (ref) formalize the distortion induced by re-ranking and how it affects partial outcomes under TSPR. Assumptions (ref) and (ref) then describe how the partial treatment exposure queries receive affects partial outcomes, both when treated items are prioritized at the top for group $B$ queries and when treated items are down-ranked for group $A$ queries.
In a TSPR experiment, the platform perturbs its baseline relevance ordering to induce exogenous variation in exposure. Such changes alter how user attention is distributed across the list. Because the baseline recommender maximizes outcomes by favoring highly relevant items near the top, these perturbations generally lower total and partial outcomes. When items that would appear lower under the platform’s baseline ranking are moved into early positions, users may click less, search less deeply, or abandon sooner. The magnitude of this perturbation is governed by the treatment assignment probability $p$ from the experimental design: larger $p$ implies a larger expected deviation from the platform’s baseline ordering within early ranks. We capture this channel by allowing the baseline attention function $F(l)$ to be attenuated under TSPR, and denote the distorted attention function by $D(l;p)$.
Assumption (ref) is motivated by the symmetry of the TSPR design. Items are randomly assigned to Treated, Untreated, and Placebo labels, independently of their baseline relevance. As a result, the distributions of baseline relevance among Treated and Untreated items are identical in expectation. The two query arms then apply mirror-image block prioritization rules: arm $B$ promotes the Treated block while arm $A$ promotes the Untreated block, and in both arms items are otherwise the same and only re-ordered within a fixed candidate set. When within-block ordering follows the platform’s baseline ranking, the primary source of perturbation is the block swap itself, whose magnitude is governed by the treatment assignment probability $p$. Under these conditions, the expected quality of the top-$l$ positions, and therefore the induced attenuation in attention, is the same across arms, implying $d_A(l;p)=d_B(l;p)$ for all $l$.
Equation (ref) characterizes partial outcomes under full treatment. Under TSPR, however, treatment is applied to only a small subset of items, but the re-ranking scheme uses position bias to maximize exposure to treated items for queries in $Q^B$ and minimize exposure for queries in $Q^A$. Partial treatment and ranking distortion therefore require a more general formulation.
Invoking Assumptions (ref), (ref), and (ref), and writing $d(l)$ for the distortion function at a fixed treatment probability $p$, the expected partial outcome at rank $l$ for a query assigned to group $B$ under TSPR satisfies
Similarly, for a query in group $A$ in TSPR,
We now introduce two functions, $\tau(\cdot)$ and $\nu(\cdot,\cdot)$, that characterize how treatment exposure interacts with ranking in treatment-dominated ($Q^B$) and control-dominated ($Q^A$) listings. The function $\tau(\cdot)$ captures the scaling of treatment effects when treated items fill the top positions in group $B$, reflecting substitution or complementarity across these items. The function $\nu(\cdot,\cdot)$ captures the contamination effect for group $A$, where a small number of treated items may appear in lower ranks and influence expected outcomes.
Building on equation (ref), for a query in group $B$ with $l\leq n_b$ treated items at the top:
Assumption (ref) ensures that partial lift has the same sign as the full-treatment effect, and that $\phi(l)=\Phi\,\tau(l)$ converges to $\Phi$ as the treated block grows. The sign of $\tau(1)-1$ is a reduced-form summary of net interference at shallow depths. When items are substitutes, treating a small block at the top amplifies the per-item effect: with $\Phi<0$, for instance, the single treated item at rank 1 loses demand to the many untreated items below it, making $\phi(1)$ more negative than $\Phi$ and hence $\tau(1)>1$. Conversely, if items are complements, where the treatment effect grows with the number of co-treated items, treating a single item in isolation produces a smaller effect than treating the full catalog, yielding $\tau(1)<1$. As the treated block expands and fewer untreated items remain to absorb substitution (or contribute complementarities), $\tau(l)$ declines toward 1.
In our main application (an Expedia-like marketplace), items are substitutes in expectation: booking one hotel forgoes others in the same query. This substitution channel implies $\tau(1)>1$, with $\tau(l)$ decreasing toward $1$ as $l$ grows.
For a query in group $A$, with $n_u$ untreated items, $n_p$ placebo items, and $n_a$ treated items appearing at the tail of the list, we model partial outcomes for $l\leq n_u+n_p$ as:
Here $\nu(\cdot) \in [0,1)$ represents the fractional effective exposure to treatment reaching the top of the control-prioritized arm, scaling outcomes in the direction of $\Phi$.
We now define the partial lift of treatment at block size $l$ by conditioning on queries whose prioritized block contains exactly $l$ items, that is, $n_b=l$ for queries in $Q^B$ and $n_u=l$ for queries in $Q^A$:
In practice, the contamination term $\nu(\cdot)$ is small whenever treated items in the tail of group $A$ listings receive little effective attention, either because attention decays sharply with rank, because the list is long, or because $p$ is small so that the placebo buffer separating untreated and treated blocks is wide.
We therefore adopt the approximation $\nu \approx 0$ for the remainder of the analysis, reducing the partial lift to the single-function form $\phi(l) = \Phi\,\tau(l)$. This simplification is what allows us to recover $\Phi$ from the depth profile of $\widehat{\phi}(l)$ using either the parametric or nonparametric estimators described below.
Given data from a TSPR experiment, we first construct the empirical partial lift at each depth $l$. For each block size $l$, we compare queries from $Q^B$ that have exactly $l$ Treated items in the top positions to queries from $Q^A$ that have exactly $l$ Untreated items in the top positions, computing the partial outcome up to position $l$ in both cases. The empirical partial lift is
where $\widehat{\mathbb{E}}[Y^l_B]$ and $\widehat{\mathbb{E}}[Y^l_A]$ denote the sample means of the partial outcome in the treatment-prioritized and control-prioritized arms, respectively, conditional on block size $l$.
With empirical values $\widehat{\phi}(l)$ for a range of depths $l = 1,\dots,L$, and under standard regularity conditions (e.g., sufficient support across $l$), we recover the global lift $\Phi$ using either a parametric or a nonparametric approach. Both methods exploit the relationship $\phi(l) = \Phi\,\tau(l)$, where $\tau(l) \to 1$ as $l \to \infty$, so that $\phi(l)$ converges to the full-treatment effect as depth increases.
We impose a parsimonious parametric form on the depth function that satisfies the qualitative restrictions derived above; namely, that $\tau(l)$ is smooth, positive, and converges to unity as $l$ grows:
This specification\footnote{While we also explored a generalized specification $\tau(l) = 1 + \alpha/l$ where $\alpha$ is estimated as a free parameter, we found that this added flexibility yielded no significant improvement in bias reduction; consequently, we maintain the simpler, parameter-free form for all subsequent results. In general, more flexible functional forms can be used depending on the empirical setting.} has no free parameters beyond the global lift $\Phi$ itself. At $l=1$, $\tau(1) = 2$: when only a single treated item occupies the top position, the partial lift is twice the full-treatment effect, reflecting the concentrated exposure of the treatment block. As $l$ increases, $\tau(l)$ declines monotonically toward $1$, so that $\phi(l) \to \Phi$.
Since the model $\phi(l) = \Phi\,\tau(l)$ is linear in $\Phi$ for known $\tau$, the estimator admits a closed-form weighted least squares solution. We minimize
where $w(l)$ is a nonnegative precision weight proportional to the number of queries contributing to the $l$-th partial outcome. Setting the first-order condition to zero yields
This is equivalent to dividing each $\widehat{\phi}(l)$ by its model-implied scaling factor $\tau(l)$ and taking a precision-weighted average of the resulting depth-specific estimates of $\Phi$. Standard errors are computed via bootstrap resampling at the query level.
As a robustness check that avoids imposing a specific functional form on $\tau(l)$, we estimate $\Phi$ using isotonic regression barlow1972statistical. This approach follows recent work in marketplace analytics that utilizes shape restrictions to correct for rank-based biases goli2024bias. While goli2024bias employ isotonic regression to improve recommender system accuracy, we leverage it here to nonparametrically recover the global lift $\Phi$ from item-level interventions. The key shape restriction is that $|\phi(l)|$ is monotonically decreasing in $l$: as more items in the top block receive treatment (or control), the partial lift converges toward the full-treatment effect. When $\Phi < 0$, this implies that $\phi(l)$ is increasing; when $\Phi > 0$, $\phi(l)$ is decreasing.
We fit a weighted isotonic regression of $\widehat{\phi}(l)$ on $l$, using the same precision weights $w(l)$ as above, subject to the appropriate monotonicity constraint. Block sizes with fewer than a minimum number of queries in either arm are excluded prior to fitting to reduce noise from imprecisely estimated cells.
Let $\widetilde{\phi}(l)$ denote the isotonic-regression fitted values. Because $\phi(l) \to \Phi$ as $l$ grows, we estimate the global lift as a weighted average of the fitted values at large depths:
where $\mathcal{L}_{\mathrm{top}}$ contains the largest observed block sizes, selected so that the fitted values have converged close to $\Phi$. The threshold defining $\mathcal{L}_{\text{top}}$ balances two competing forces: restricting to the largest block sizes ensures that the fitted values $\widetilde{\phi}(l)$ have converged close to $\Phi$, while including more block sizes improves precision.
To illustrate our methodology, we use an open-source dataset of hotel search impressions from Expedia expedia-personalized-sort. The data capture consumer queries and their subsequent search behavior over an eight-month period. Our training and calibration sample consists of a 20% subset of the cleaned data, comprising nearly 2 million observation-level records across approximately 80,000 unique search impressions.
Consumers interact with the platform in three stages. First, consumers initiate queries by specifying trip details. Second, they receive a ranked list of hotel results. A key feature of this dataset is the experimental variation in ranking: approximately 30% of search impressions were randomly sorted, while the remainder followed the platform's original relevance-based recommender system. This variation allows us to disentangle the causal effect of display position from item relevance. Finally, users engage by clicking on hotels to view details and may subsequently complete a booking.
To evaluate our experimental design, we implement Monte Carlo simulations that replicate this two-sided marketplace. We model user interactions as a function of a latent utility $v_{ij}$. The platform's relevance score $r$ is modeled as $r = v + \epsilon$, where $\epsilon \sim N(0, \sigma^2)$. While the original ranking is sorted decreasing in $r$, the random ranking allows for unbiased estimation of position effects.
Table (ref) presents summary statistics for the search impressions used in our analysis, highlighting the baseline differences in performance between the ranking mechanisms.
Click behavior is modeled using a logistic function incorporating rank-based attention and sequential behavior. For each item $j$ at position $p$ for user $i$, the probability of a click is:
We estimate these parameters in two stages to resolve the endogeneity of position in relevance-sorted results. In the first stage, we use only the randomly sorted subset to estimate the position coefficients ($\beta_1, \beta_2$) and click-history effects ($\beta_3, \beta_4$). Because positions are assigned randomly, these estimates represent pure attention effects. In the second stage, we fix these coefficients as an offset and estimate the utility coefficient $\beta_5$ and intercept $\beta_0$ on the full mixed sample. The resulting parameters, shown in Table (ref), show a clear initial decline in attention (negative $\beta_1$) and a strong negative pressure on subsequent clicks once an initial click has occurred ($\beta_4$).
The resulting estimates (Table (ref)) reveal a nuanced search process. The large negative coefficient for the click indicator ($\beta_4 = -3.60$) captures the expected “satisficing” effect, where any initial click significantly lowers the marginal probability of further search. However, the positive coefficient for the cumulative click count ($\beta_3 = 0.65$) identifies latent searcher heterogeneity: conditional on not stopping, users with higher click counts exhibit a higher baseline propensity for exhaustive search. This specification ensures the simulation reflects a marketplace populated by both “quick-search” and “high-intensity” consumers. Figure (ref) (left) shows that our click model closely fits the click-through-rate based on item position to the real Expedia click impressions data.
Conditional on the set of clicked items $C_i$, user $i$ either books one item or takes the outside option (no booking). We model this choice with a multinomial logit that includes a click-order effect to capture cascade behavior in purchasing decisions. The probability of booking item $k \in C_i$ is
where $\text{click\_order}_{ik}$ records the sequential position in which item $k$ was clicked (1\,=\,first clicked, 2\,=\,second, etc.) and the “$1$” in the denominator represents the outside option. The coefficient $\gamma_2$ captures the empirical regularity that earlier-clicked items are more likely to be booked, consistent with directed search in which users click the most promising options first. All parameters are estimated by maximum likelihood on the subsample of clicked items.
The negative coefficient for click order ($\gamma_2$) in Table (ref) confirms a “first-mover” advantage in search, where items discovered earlier in the process are more likely to be converted. A likelihood ratio test strongly rejects the standard multinomial logit (without click order) in favor of the cascade specification ($p < 0.001$; see Appendix (ref)), so we adopt the cascade model for all main results. Figure (ref) (right) confirms that the cascade model closely matches observed booking rates by click order.
Treatment is introduced as a constant shift $\delta$ in the latent utility: $v_{ij}^* = v_{ij} + \delta T_{ij}$. This shift propagates through both the click and booking stages. Crucially, as shown in Equation (ref), an increase in the utility of a treated item increases its own booking probability while simultaneously decreasing the probability for all other items in $C_i$. This within-query substitution is the primary source of interference we aim to address with the TSPR design.
We conduct counterfactual simulations for 20{,}000 queries using the estimated models of click and booking behavior. To establish a simulated ground truth for lift, we simulate the marketplace under two extreme scenarios: one in which no items receive treatment and one in which all items are treated. The treatment enters as a constant reduction in the latent utility of an item, which represents the effect of a platform-wide price or markup increase and implies a 6.9 percent decline in bookings under full treatment. The recommender system is held fixed in both simulations. The resulting proportional change in total bookings serves as the benchmark against which we evaluate the lift estimates produced by each experimental design.
We then implement our Two-Sided Prioritized Ranking (TSPR) experimental design, described in Table (ref) and estimate the total lift using both the parametric (Section (ref)) and non-parametric (Section (ref)) approaches. To compare performance of our experimental design and two estimators, we use bernoulli-randomized and cluster-randomized experiments, both common benchmarks for marketplace experiments. All baselines target the same estimand $\Phi$ defined in Equation (ref); differences in performance reflect interference-induced bias and variance, not differences in the target parameter. The main results are under treatment probability $p=0.25$ but we show sensitivity to choice of $p$ in Appendix (ref).
As a baseline, we consider an item-side randomized experiment in which items are Bernoulli randomized at the listing level. Figure (ref) contrasts this design with the Two-Sided Prioritized Ranking (TSPR) setup. In the item-side A/B test (panel a), treated and untreated items are randomly interleaved within the same ranked list, so treated items compete directly with untreated items for user attention, generating within-list interference. TSPR instead induces structured variation in treatment exposure by simply re-ordering items within the list: In the treatment arm ($Q^B$, panel b), treated items are promoted to the top of the ranking, whereas in the control arm ($Q^A$, panel c), untreated items are prioritized, with placebo items buffering the two blocks. We simulate both designs and compare the resulting lift estimates.
To estimate the lift in total outcome, we extend the horvitz1952generalization logic to the two-sided marketplace setting using item-level randomization. In this baseline configuration, randomization occurs only at the level of items. Each item $i$ is independently assigned to treatment ($Z_i = 1$) with probability $p$ or control ($Z_i = 0$) with probability $1-p$, forming the sets $T = \{i : Z_i = 1\}$ and $C = \{i : Z_i = 0\}$. Letting $y_{q,i}$ denote the outcome for item $i$ in query $q \in Q$, we estimate the mean total outcome per query under global treatment ($\mu_B$) and global control ($\mu_A$) using the estimators $\hat{\mu}_B^{IS} = \frac{1}{|Q|} \sum_{q \in Q} \sum_{i \in T} \frac{y_{q,i}}{p}$ and $\hat{\mu}_A^{IS} = \frac{1}{|Q|} \sum_{q \in Q} \sum_{i \in C} \frac{y_{q,i}}{1-p}$. Our estimand of interest is the lift $\Phi_{IS} = \frac{\mu_B}{\mu_A} - 1$. By taking the ratio of our HT estimators, the $1/|Q|$ terms cancel, yielding the item-side lift estimator:
This baseline utilizes only item-level Bernoulli randomization and provides a simple comparison point that ignores query-level randomization.
As a second baseline, we compare our estimates to lift ratio estimates obtained from cluster-randomized experiments. Cluster randomization reduces interference bias because units within a cluster share the same treatment assignment, which limits spillover across treatment arms. However, clustering methods often exhibit substantially larger variance, as the clusters are frequently large, which effectively reduces the number of independent units of randomization. Furthermore, implementing cluster randomization requires detailed knowledge of the underlying network structure and is often costly. When it can be applied correctly, it preserves user experience coherency under our definition. For this reason, cluster-randomized experiments provide a relevant benchmark for evaluating the performance of TSPR.
To construct this baseline in our setting, we use the real search impression data from Expedia described in Section (ref). Each observation consists of a property $j$ appearing in a search query $i$. Similar to the approach in holtz2024reducing, we begin with the co-occurrence of properties across different search queries to construct our clusters. We employ Truncated SVD to find a lower-dimensional dense representation of the co-occurrence matrix and subsequently use $k$-means to cluster the resulting embeddings.
The dimension of the SVD and the number of clusters $k$ are treated as hyperparameters. We perform a grid search to identify the optimal combination based on the Modularity score. Networks with high modularity have dense connections between the nodes within modules but sparse connections between nodes in different modules. The grid search resulted in selecting 100 dimensions for SVD and 200 clusters for K-means. The resulting modularity score exceeds $0.93$, indicating that this method was able to identify highly segregated clusters of properties.
After clustering, we randomly assign clusters to treatment with a probability $p_{\text{treat}}$. Let $T_c = \{i : \text{cluster}(i) \in \text{treated clusters}\}$ and $C_c = \{i : \text{cluster}(i) \in \text{control clusters}\}$. We compare the mean total outcome per query between treated and untreated items, and estimate lift using the same Horvitz–Thompson-style ratio structure discussed earlier for the item-side baseline:
Before comparing estimators, we illustrate how TSPR recovers the global lift from partial-list contrasts. For the isotonic estimator, we set $\mathcal{L}_{\text{top}}$ to the largest 30% of observed block sizes (with a minimum of two queries); results are similar for thresholds between 20% and 50%.
Figure (ref) illustrates the estimation using a representative simulation. The empirical partial lifts $\hat{\phi}(l)$ are initially higher in magnitude than the global effect $\Phi$, reflecting concentrated treatment exposure at low block depths. As the block size $l$ increases, the observed lifts converge toward the ground truth. Both the parametric WLS fit and the isotonic regression capture this transition, averaging the noisy empirical points according to their query-level precision weights. The right panel highlights the structural decay of $\tau(l)$, the scaling function that maps partial lifts back to the full-treatment effect.
The central results of our simulation study are summarized in Figure (ref) and Table (ref).
Figure (ref) compares the performance of the parametric (WLS) and non-parametric (isotonic) estimators under a TSPR experiment to Bernoulli-randomized and cluster-randomized baselines. Across 500 simulation replications, TSPR delivers a substantial improvement over standard marketplace experimental designs, regardless of the estimation method. Relative to the Bernoulli-randomized (naive) baseline, TSPR effectively eliminates the severe interference bias that arises when treated and untreated items compete for attention within the same ranked list. While cluster randomization produces a mean estimate close to the ground truth (low bias), it does so at the cost of much higher variance. Specifically, the cluster-randomized design exhibits standard deviations that are 8.3$\times$ and 4.9$\times$ those of the TSPR parametric and non-parametric estimates, respectively.
Table (ref) quantifies these patterns in terms of bias, dispersion, and overall accuracy. The Bernoulli-randomized design exhibits substantial downward bias (mean $\hat{\Phi}=-0.1028$ vs.\ $\Phi_{true}=-0.0691$), yielding the largest bias in magnitude ($-0.0337$) and a correspondingly high RMSE (0.0438). Cluster randomization largely removes bias (bias $=0.0086$), but its performance is dominated by extreme variability (empirical SD $=0.1345$), producing by far the worst RMSE (0.1348). In contrast, both TSPR estimators achieve a markedly better bias–variance tradeoff. The parametric TSPR estimator attains the lowest RMSE (0.0210) and the smallest empirical SD (0.0162), despite a modest bias ($-0.0133$). The non-parametric TSPR estimator exhibits a very low bias (bias $=0.0069$) while maintaining low variance (SD $=0.0275$) and a low RMSE (0.0283). Overall, the table shows that TSPR improves accuracy relative to Bernoulli randomization by sharply reducing interference-driven bias, while avoiding the prohibitive variance costs of clustering.
We assess sensitivity to the treatment probability $p$. While our main results use $p=0.25$, we stress-test TSPR over $p \in [0.10, 0.45]$, which corresponds to placebo buffer sizes ranging from 80% to 10%. As shown in Appendix (ref), TSPR remains robust across this range, consistently delivering substantially lower bias and variance than the Bernoulli-randomized baseline.
Bootstrap confidence intervals for the TSPR non-parametric estimator achieve 93% empirical coverage at the nominal 95% level, indicating well-calibrated uncertainty quantification (Appendix (ref)).
The Bernoulli-randomized item-side estimator is unbiased only when each item's outcome is independent of all other items in the list. In marketplace search, this no-interference condition fails: users substitute across items, so that booking one option crowds out others within the same query. Position bias amplifies the resulting distortion, because the items most affected by substitution are those competing in high-attention ranks where the bulk of user engagement occurs. When interference and position bias act together, the naive estimator systematically overstates the magnitude of the treatment effect, conflating the causal impact of treatment with the reallocation of demand across treated and untreated items within the same list.
TSPR addresses this problem by leveraging position bias as the source of identifying variation. By reordering ranked lists so that one query group receives concentrated exposure to treated items and the other to untreated items, TSPR creates meaningful differences in aggregate outcomes across arms without requiring interference to be absent. The design reduces bias from within-list substitution, and remains effective precisely in the environments where naive item-level experiments fail most severely. If user behavior were order-invariant, with clicking and booking depending only on the set of utilities $\{v_{qi}\}$ and not their positions, then permuting the ranking would not change outcomes, and TSPR would have no identifying variation. TSPR is therefore most valuable in environments where position bias is strong, a condition that is well documented in search and recommendation settings.
Cluster randomization provides an alternative by assigning groups of related items to the same treatment status, eliminating within-cluster spillovers. When cluster boundaries align with the true interference structure, this approach can achieve low bias, as in our simulation where SVD-based embedding followed by k-means yielded modularity above 0.93. The efficiency cost, however, is substantial: with roughly 200 clusters instead of 20{,}000 query-level randomization units in TSPR, the effective sample size falls by about two orders of magnitude, producing roughly 8 times the standard deviation of TSPR’s parametric estimator. Moreover, clean clusters are difficult to construct in practice. Substitution patterns are often diffuse and context-dependent, and misspecified boundaries reintroduce the very spillovers clustering is meant to remove. TSPR avoids this fragility and does not depend on knowing the interference network.
This paper introduces Two-Sided Prioritized Ranking (TSPR), an experimental design for item-side interventions in online marketplaces that maintains price parity and full catalog access while addressing interference through position-based exposure variation. In simulations calibrated to hotel search data, TSPR substantially reduces bias relative to Bernoulli-randomized A/B tests and achieves an order-of-magnitude reduction in standard deviation compared to cluster randomization, even when clusters are cleanly defined.
TSPR can be implemented by adjusting ranking priorities within an existing recommender system, making it straightforward to deploy on platforms that already support re-ranking logic. The design avoids the operational and data burdens that often accompany marketplace experimentation, such as user-level pricing changes, catalog partitioning, or explicitly modeling the interference network. Although TSPR does not target the global treatment effect without bias, it offers a strongly favorable bias–variance tradeoff in practice: relative to Bernoulli randomization it substantially reduces both bias and variance, and relative to cluster randomization it delivers far higher precision.
TSPR is best suited to settings with strong position bias, slack supply, and short experiment horizons that limit dynamic feedback. The identifying assumptions of treatment-attention separability and symmetric re-ranking distortion are plausible in many marketplace settings, but they can fail if the treatment meaningfully changes user engagement or if high treatment probabilities induce ranking perturbations that users do not tolerate. Practitioners should assess these risks using pre-experiment diagnostics and the sensitivity analyses reported in the appendix.
Natural extensions include adapting TSPR to supply-constrained settings with binding capacity constraints, extending it to continuous or multi-arm treatments, and allowing for user-side interference across queries. More broadly, the results underscore that platform structure, here ranked attention, can provide identifying variation even under interference. We view the coherency constraints that motivate TSPR as realistic operational requirements, and a useful guide for methodological development in platform experimentation.