arXiv 10 May 2024 · Econometrics
arXiv:2405.06779 · PDF · DOI · OpenAlex · Extracted main text
Can the causal effects estimated in an experiment be generalized to real-world scenarios? This question lies at the heart of social science studies. External validity primarily assesses whether experimental effects persist across different settings, implicitly presuming the consistency of experimental effects with their real-life counterparts. However, we argue that this presumed consistency may not always hold, especially in experiments involving multi-dimensional decision processes, such as conjoint experiments. We introduce a formal model to elucidate how attention and salience effects lead to three types of inconsistencies between experimental findings and real-world phenomena: amplified effect magnitude, effect sign reversal, and effect importance reversal. We derive testable hypotheses from each theoretical outcome and test these hypotheses using data from various existing conjoint experiments and our own experiments. Drawing on our theoretical framework, we propose several recommendations for experimental design aimed at enhancing the generalizability of survey experiment findings.
appendix boundary found by appendix_command · 71% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Bordalo, Pedro and Gennaioli, Nicola and Shleifer, Andrei (2012) Salience theory of choice under risk | 1.000 | 6 | 3 | 100% |
| 2 | Jenke, Libby and Bansak, Kirk and Hainmueller, Jens and Hangartner,… (2021) Using eye-tracking to understand decision-making in conjoint experiments | 1.000 | 5 | 3 | 100% |
| 3 | Bansak, Kirk and Hainmueller, Jens and Hopkins, Daniel J and Yamamot… (2021) Beyond the breaking point? Survey satisficing in conjoint experiments | 0.928 | 5 | 3 | 80% |
| 4 | Bordalo, Pedro and Gennaioli, Nicola and Shleifer, Andrei (2013) Salience and consumer choice | 0.737 | 3 | 2 | 100% |
| 5 | Egami, Naoki and Hartman, Erin (2023) Elements of external validity: Framework, design, and analysis | 0.737 | 3 | 2 | 100% |
| 6 | Chetty, Raj and Looney, Adam and Kroft, Kory (2009) Salience and taxation: Theory and evidence | 0.644 | 2 | 2 | 100% |
| 7 | Dafoe, Allan and Zhang, Baobao and Caughey, Devin (2018) Information equivalence in survey experiments | 0.644 | 2 | 2 | 100% |
| 8 | Hainmueller, Jens and Hangartner, Dominik and Yamamoto, Teppei (2015) Validating vignette and conjoint survey experiments against real-world behavior | 0.644 | 2 | 2 | 100% |
| 9 | Slough, Tara and Tyson, Scott A (2024) Sign-Congruence, External Validity, and Replication | 0.644 | 2 | 2 | 100% |
| 10 | Barabas, Jason and Jerit, Jennifer (2010) Are survey experiments externally valid? | 0.511 | 2 | 1 | 100% |
Showing the top 10 of 55 scored citations.