Rina Friedberg, Karthik Rajkumar, Jialiang Mao, Qian Yao, YinYin Yu, Min Liu
arXiv 12 May 2022 · Econometrics
arXiv:2205.06363 · PDF · DOI · OpenAlex · Extracted main text
Information retrieval systems, such as online marketplaces, news feeds, and search engines, are ubiquitous in today's digital society. They facilitate information discovery by ranking retrieved items on predicted relevance, i.e. likelihood of interaction (click, share) between users and items. Typically modeled using past interactions, such rankings have a major drawback: interaction depends on the attention items receive. A highly-relevant item placed outside a user's attention could receive little interaction. This discrepancy between observed interaction and true relevance is termed the position bias. Position bias degrades relevance estimation and when it compounds over time, it can silo users into false relevant items, causing marketplace inefficiencies. Position bias may be identified with randomized experiments, but such an approach can be prohibitive in cost and feasibility. Past research has also suggested propensity score methods, which do not adequately address unobserved confounding; and regression discontinuity designs, which have poor external validity. In this work, we address these concerns by leveraging the abundance of A/B tests in ranking evaluations as instrumental variables. Historical A/B tests allow us to access exogenous variation in rankings without manually introducing them, harming user experience and platform revenue. We demonstrate our methodology in two distinct applications at LinkedIn - feed ads and the People-You-May-Know (PYMK) recommender. The marketplaces comprise users and campaigns on the ads side, and invite senders and recipients on PYMK. By leveraging prior experimentation, we obtain quasi-experimental variation in item rankings that is orthogonal to user relevance. Our method provides robust position effect estimates that handle unobserved confounding well, greater generalizability, and easily extends to other information retrieval systems.
appendix boundary found by appendix_command · 88% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Craswell, N., Zoeter, O., Taylor, M., and Ramsey, B (2008) An experimental comparison of click position-bias models | 0.585 | 3 | 1 | 100% |
| 2 | Joachims, T., Swaminathan, A., and Schnabel, T (2017) Unbiased learning-to-rank with biased feedback | 0.511 | 2 | 1 | 100% |
| 3 | Agarwal, A., Zaitsev, I., Wang, X., Li, C., Najork, M., and Joachims… (2019) Estimating position bias without intrusive interventions | 0.405 | 1 | 1 | 100% |
| 4 | Angrist, J. D., Imbens, G. W., and Rubin, D. B (1996) Identification of causal effects using instrumental variables | 0.405 | 1 | 1 | 100% |
| 5 | Swaminathan, A. and Joachims, T (2015) Counterfactual risk minimization: Learning from logged bandit feedback | 0.405 | 1 | 1 | 100% |
| 6 | Aggarwal, G., Muthukrishnan, S., Pal, D., and Pál, M (2009) General auction mechanism for search advertising | 0.405 | 1 | 1 | 100% |
| 7 | Wang, X., Golbandi, N., Bendersky, M., Metzler, D., and Najork, M (2018) Position bias estimation for unbiased learning to rank in personal search | 0.405 | 1 | 1 | 100% |
| 8 | Li, Y (2020) Handling position bias for unbiased learning to rank in hotels search | 0.405 | 1 | 1 | 100% |
| 9 | Rosenman, E. T. R. and Owen, A. B (2021) Designing experiments informed by observational studies | 0.405 | 1 | 1 | 100% |
| 10 | Pandey, S., Roy, S., Olston, C., Cho, J., and Chakrabarti, S (2005) Shuffling a stacked deck: The case for partially randomized ranking of search engine results | 0.405 | 1 | 1 | 100% |
Showing the top 10 of 12 scored citations.