Hannah Li, Geng Zhao, Ramesh Johari, Gabriel Y. Weintraub
arXiv 25 Apr 2021 · Statistics — Methodology · publishedProceedings of the ACM Web Conference 2022 (2022) · 23 citations (OpenAlex)
arXiv:2104.12222 · PDF · DOI · OpenAlex · Extracted main text
Two-sided marketplace platforms often run experiments to test the effect of an intervention before launching it platform-wide. A typical approach is to randomize individuals into the treatment group, which receives the intervention, and the control group, which does not. The platform then compares the performance in the two groups to estimate the effect if the intervention were launched to everyone. We focus on two common experiment types, where the platform randomizes individuals either on the supply side or on the demand side. The resulting estimates of the treatment effect in these experiments are typically biased: because individuals in the market compete with each other, individuals in the treatment group affect those in the control group and vice versa, creating interference. We develop a simple tractable market model to study bias and variance in these experiments with interference. We focus on two choices available to the platform: (1) Which side of the platform should it randomize on (supply or demand)? (2) What proportion of individuals should be allocated to treatment? We find that both choices affect the bias and variance of the resulting estimators but in different ways. The bias-optimal choice of experiment type depends on the relative amounts of supply and demand in the market, and we discuss how a platform can use market data to select the experiment type. Importantly, we find in many circumstances, choosing the bias-optimal experiment type has little effect on variance. On the other hand, the choice of treatment proportion can induce a bias-variance tradeoff, where the bias-minimizing proportion increases variance. We discuss how a platform can navigate this tradeoff and best choose the treatment proportion, using a combination of modeling as well as contextual knowledge about the market, the risk of the intervention, and reasonable effect sizes of the intervention.
appendix boundary found by appendix_command · 64% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | D. Holtz, R. Lobel, I. Liskovich, and S. Aral (2020) Reducing interference bias in online marketplace pricing experiments, 2020 | 1.000 | 6 | 5 | 100% |
| 2 | R. Johari, H. Li, I. Liskovich, and G. Weintraub (2021) Experimental design in two-sided platforms: An analysis of bias, 2021 | 1.000 | 6 | 5 | 100% |
| 3 | P. Bajari, B. Burdick, G. Imbens, J. McQueen, T. Richardson, and I.… (2019) Multiple randomization designs for interference | 0.843 | 3 | 3 | 100% |
| 4 | T. Blake and D. Coey (2014) Why marketplace experimentation is harder than it seems: The role of test-control interference | 0.737 | 3 | 2 | 100% |
| 5 | A. Fradkin (2015) Search frictions and the design of online marketplaces | 0.737 | 3 | 2 | 100% |
| 6 | G. W. Imbens and D. B. Rubin (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction | 0.737 | 3 | 2 | 100% |
| 7 | N. Chamandy (2016) Experimentation in a ridesharing marketplace, Dec 2016 | 0.737 | 3 | 2 | 100% |
| 8 | M. Saveski, J. Pouget-Abadie, G. Saint-Jacques, W. Duan, S. Ghosh, Y… (2017) Detecting network effects: Randomizing over randomized experiments | 0.644 | 2 | 2 | 100% |
| 9 | S. Wager and K. Xu (2019) Experimenting in equilibrium | 0.644 | 2 | 2 | 100% |
| 10 | Y. Xu, W. Duan, and S. Huang (2018) Sqr: Balancing speed, quality and risk in online experiments, 2018 | 0.644 | 2 | 2 | 100% |
Showing the top 10 of 25 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.