Brett R. Gordon, Robert Moakler, Florian Zettelmeyer
arXiv 18 Jan 2022 · Econometrics · publishedMarketing Science (2022) · 81 citations (OpenAlex)
arXiv:2201.07055 · PDF · DOI · OpenAlex · Extracted main text
Despite their popularity, randomized controlled trials (RCTs) are not always available for the purposes of advertising measurement. Non-experimental data is thus required. However, Facebook and other ad platforms use complex and evolving processes to select ads for users. Therefore, successful non-experimental approaches need to "undo" this selection. We analyze 663 large-scale experiments at Facebook to investigate whether this is possible with the data typically logged at large ad platforms. With access to over 5,000 user-level features, these data are richer than what most advertisers or their measurement partners can access. We investigate how accurately two non-experimental methods -- double/debiased machine learning (DML) and stratified propensity score matching (SPSM) -- can recover the experimental effects. Although DML performs better than SPSM, neither method performs well, even using flexible deep learning models to implement the propensity and outcome models. The median RCT lifts are 29%, 18%, and 5% for the upper, middle, and lower funnel outcomes, respectively. Using DML (SPSM), the median lift by funnel is 83% (173%), 58% (176%), and 24% (64%), respectively, indicating significant relative measurement errors. We further characterize the circumstances under which each method performs comparatively better. Overall, despite having access to large-scale experiments and rich user-level data, we are unable to reliably estimate an ad campaign's causal effect.
appendix boundary found by appendix_titled_section at “Appendix” · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Gordon, B. R., F. Zettelmeyer, N. Bhargava, and D. Chapsky (2019) A comparison of approaches to advertising measurement: Evidence from big field experiments at facebook self | 1.000 | 9 | 4 | 100% |
| 2 | Imbens, G. and D. B. Rubin (2015, April) (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction\/ (1st ed.) | 0.928 | 4 | 3 | 100% |
| 3 | Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W… (2018) Double/debiased machine learning for treatment and structural parameters | 0.874 | 5 | 2 | 100% |
| 4 | Shapiro, B., G. Hitsch, and A. Tuchman (2021) Tv advertising effectiveness and profitability: Generalizable results from 288 brands | 0.843 | 3 | 3 | 100% |
| 5 | Tunuguntla, S (2021) Display ad measurement using observational data: A reinforcement learning approach | 0.811 | 4 | 2 | 100% |
| 6 | Hoban, P. and N. Arora (2018) Measuring display advertising response using observational data: The impact of selection biases | 0.737 | 3 | 2 | 100% |
| 7 | Dehejia, R. H. and S. Wahba (2002, February) (2002) Propensity score matching methods for non-experimental causal studies | 0.644 | 2 | 2 | 100% |
| 8 | Gui, G., H. Nair, and F. Niu (2022) Auction throttling and causal inference of online advertising effects | 0.644 | 2 | 2 | 100% |
| 9 | Johnson, G. A (2022) Inferno: A guide to field experiments in online display advertising | 0.644 | 2 | 2 | 100% |
| 10 | LaLonde, R. J (1986) Evaluating the econometric evaluations of training programs with experimental data | 0.511 | 2 | 1 | 100% |
Showing the top 10 of 43 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.