Tamara Broderick, Ryan Giordano, Rachael Meager
arXiv 30 Nov 2020 · Statistics — Methodology · 15 citations (OpenAlex)
arXiv:2011.14999 · PDF · DOI · OpenAlex · Extracted main text
Study samples often differ from the target populations of inference and policy decisions in non-random ways. Researchers typically believe that such departures from random sampling -- due to changes in the population over time and space, or difficulties in sampling truly randomly -- are small, and their corresponding impact on the inference should be small as well. We might therefore be concerned if the conclusions of our studies are excessively sensitive to a very small proportion of our sample data. We propose a method to assess the sensitivity of applied econometric conclusions to the removal of a small fraction of the sample. Manually checking the influence of all possible small subsets is computationally infeasible, so we use an approximation to find the most influential subset. Our metric, the "Approximate Maximum Influence Perturbation," is based on the classical influence function, and is automatically computable for common methods including (but not limited to) OLS, IV, MLE, GMM, and variational Bayes. We provide finite-sample error bounds on approximation performance. At minimal extra cost, we provide an exact finite-sample lower bound on sensitivity. We find that sensitivity is driven by a signal-to-noise ratio in the inference problem, is not reflected in standard errors, does not disappear asymptotically, and is not due to misspecification. While some empirical applications are robust, results of several influential economics papers can be overturned by removing less than 1% of the sample.
appendix boundary found by appendix_command · 87% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Hampel, F., Ronchetti, E., Rousseeuw, P., and Stahel, W (1986) Robust statistics: the approach based on influence functions | 1.000 | 9 | 3 | 100% |
| 2 | Angelucci, M., Karlan, D., and Zinman, J (2015) Microcredit impacts: Evidence from a randomized microcredit program placement experiment by Compartamos Banco | 0.928 | 4 | 3 | 100% |
| 3 | Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J.,… (2012) The Oregon health insurance experiment: Evidence from the first year | 0.874 | 8 | 2 | 100% |
| 4 | Angelucci, M. and De Giorgi, G (2009) Indirect effects of an aid program: How do cash transfers affect ineligibles' consumption? | 0.874 | 6 | 2 | 100% |
| 5 | Meager, R (2019) Understanding the average impact of microcredit expansions: A Bayesian hierarchical analysis of seven randomized experiments self | 0.874 | 6 | 2 | 100% |
| 6 | Meager, R (2020) Aggregating distributional treatment effects: A Bayesian hierarchical analysis of the microcredit literature self | 0.874 | 5 | 2 | 100% |
| 7 | Belsley, D., Kuh, E., and Welsch, R (1980) Regression diagnostics: Identifying influential data and sources of collinearity | 0.811 | 4 | 2 | 100% |
| 8 | Hampel, F (1974) The influence curve and its role in robust estimation | 0.737 | 3 | 2 | 100% |
| 9 | Krantz, S. and Parks, H (2012) The implicit function theorem: History, theory, and applications | 0.737 | 3 | 2 | 100% |
| 10 | Young, A (2019) Consistency without inference: Instrumental variables in practical application | 0.737 | 3 | 2 | 100% |
Showing the top 10 of 73 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | Sample Fit Reliability | 1.000 | 5 | 4 |
| 2 | Testing Most Influential Sets | 1.000 | 5 | 3 |
| 3 | Finding Most Influential Sets | 0.874 | 6 | 4 |
| 4 | Generalizability with ignorance in mind: learning what we do (not) know for archetypes discovery | 0.644 | 2 | 2 |
| 5 | How Robust are Robustness Checks? | 0.405 | 1 | 1 |