Ruohan Zhan, Shichao Han, Yuchen Hu, Zhenling Jiang
arXiv 20 Jun 2024 · Econometrics · 3 citations (OpenAlex)
arXiv:2406.14380 · PDF · DOI · OpenAlex · Extracted main text
Recommender systems are essential for content-sharing platforms by curating personalized content. To evaluate updates to recommender systems targeting content creators, platforms frequently rely on creator-side randomized experiments. The treatment effect measures the change in outcomes when a new algorithm is implemented compared to the status quo. We show that the standard difference-in-means estimator can lead to biased estimates due to recommender interference that arises when treated and control creators compete for exposure. We propose a "recommender choice model" that describes which item gets exposed from a pool containing both treated and control items. By combining a structural choice model with neural networks, this framework directly models the interference pathway while accounting for rich viewer-content heterogeneity. We construct a debiased estimator of the treatment effect and prove it is $\sqrt n$-consistent and asymptotically normal with potentially correlated samples. We validate our estimator's empirical performance with a field experiment on Weixin short-video platform. In addition to the standard creator-side experiment, we conduct a costly double-sided randomization design to obtain a benchmark estimate free from interference bias. We show that the proposed estimator yields results comparable to the benchmark, whereas the standard difference-in-means estimator can exhibit significant bias and even produce reversed signs.
appendix boundary found by none_found · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Farrell MH, Liang T, Misra S (2021) a) Deep learning for individual heterogeneity: An automatic inference framework | 1.000 | 14 | 3 | 100% |
| 2 | Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W… (2018) Double/debiased machine learning for treatment and structural parameters | 1.000 | 5 | 3 | 100% |
| 3 | Goli A, Lambrecht A, Yoganarasimhan H (2023) A bias correction approach for interference in ranking experiments | 0.928 | 4 | 3 | 100% |
| 4 | Johari R, Li H, Liskovich I, Weintraub GY (2022) Experimental design in two-sided platforms: An analysis of bias | 0.928 | 4 | 3 | 100% |
| 5 | Chernozhukov V, Demirer M, Lewis G, Syrgkanis V (2019) Semi-parametric efficient policy learning with continuous actions | 0.843 | 3 | 3 | 100% |
| 6 | Farias V, Li H, Peng T, Ren X, Zhang H, Zheng A (2023) Correcting for interference in experiments: A case study at douyin | 0.644 | 2 | 2 | 100% |
| 7 | Foster DJ, Syrgkanis V (2023) Orthogonal statistical learning | 0.644 | 2 | 2 | 100% |
| 8 | Sävje F, Aronow P, Hudgens M (2021) Average treatment effects in the presence of unknown interference | 0.644 | 2 | 2 | 100% |
| 9 | Ye Z, Zhang Z, Zhang D, Zhang H, Zhang RP (2023) b) Deep learning based causal inference for large-scale combinatorial experiments: Theory and empirical evidence | 0.644 | 2 | 2 | 100% |
| 10 | Bright I, Delarue A, Lobel I (2022) Reducing marketplace interference bias via shadow prices | 0.511 | 2 | 1 | 100% |
Showing the top 10 of 44 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | 2501.11996 | 0.511 | 2 | 1 |
| 2 | ARMA-Design: Optimal Treatment Allocation Strategies for A/B Testing in Partially Observable Experiments | 0.405 | 1 | 1 |
| 3 | Experimental Design for Matching | 0.405 | 1 | 1 |
| 4 | Personalized Policy Learning through Discrete Experimentation: Theory and Empirical Evidence | 0.405 | 1 | 1 |