Tomas Havranek, Zuzana Irsova, Martina Luskova, T. D. Stanley
arXiv 25 Jul 2026 · Econometrics
arXiv:2607.23174 · PDF · Extracted main text
Meta-analysts routinely face estimates that look too large or extreme. Yet, how to handle them is left to the reviewer's judgment. The methods for detecting such estimates are well known. What is missing is an informed assessment of how much alternative handling choices might change a meta-analysis' conclusions. We fill this gap by analyzing the effects of four pre-registered handling treatments across 358 behavioral science meta-analyses with at least ten estimates. Each outlier handling treatment is estimated by two estimators (random effects and unrestricted weighted least squares), and compared to the 'do-nothing' baseline on three outcomes: the pooled effect, statistical significance, and whether the effect reaches the smallest effect size of interest (|d| >= 0.20). Our entire analysis and comparison pipelines were pre-registered. Alternative outlier handling treatments have little effect on the meta-analysis mean as the median absolute change in Cohen's d is at most 0.047 and often much less. Yet, at least one of these four treatments in combination with one of these estimators reverses the statistical significance of 11.5% of meta-analyses and the smallest-effect-of-interest assessment in 15.9%. Winsorizing has the least effect and DFBETAS the most. Categorical changes are found almost entirely among results already close to the decision boundary; strongly significant results essentially never change. These findings give applied meta-analysts, methods specialists, and reviewers a reference point for how much this under-reported choice matters and provide yet another reason for meta-analysts to publicly pre-specify their methods and handling treatments.
appendix boundary found by none_found · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Viechtbauer W, Cheung M W-L (2010) Outlier and Influence Diagnostics for Meta-Analysis | 1.000 | 11 | 3 | 100% |
| 2 | Langan D, Higgins JPT, Simmonds M (2015) An Empirical Comparison of Heterogeneity Variance Estimators in 12\,894 Meta-Analyses | 1.000 | 5 | 3 | 100% |
| 3 | Gelman A, Loken E The Statistical Crisis in Science | 0.843 | 3 | 3 | 100% |
| 4 | Meng Z, Wang J, Lin L, Wu C (2024) Sensitivity Analysis with Iterative Outlier Detection for Systematic Reviews and Meta-Analyses | 0.843 | 3 | 3 | 100% |
| 5 | Havranek T, Irsova Z, Luskova M, Stanley TD (2026) Outlier and Influence Handling in Meta-Analysis: Pre-Analysis Plan; 2026 | 0.843 | 3 | 3 | 100% |
| 6 | Aung NM, Jurak I, Mehmood S, Axon E (2026) Sensitivity Analysis in Meta-Analysis: A Tutorial | 0.644 | 2 | 2 | 100% |
| 7 | Bartos F, Lusková M, Bortnikova K, Hozová K, Kantova K, Irsova Z, Ha… (2025) Effect of Exercise on Cognition, Memory, and Executive Function: A Study-Level Meta-Meta-Analysis Across Populations and Exercis… | 0.644 | 2 | 2 | 100% |
| 8 | Simmons JP, Nelson LD, Simonsohn U (2011) False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant | 0.644 | 2 | 2 | 100% |
| 9 | van Zwet E, Gelman A, Wiecek W (2026) A Statistical Case for Qualified Scientific Optimism; 2026 | 0.644 | 2 | 2 | 100% |
| 10 | Stanley TD, Ioannidis JPA, Maier M, Doucouliagos H, Otte WM, Bartos F (2026) Why the Unrestricted Weighted Least Squares Should Be Routinely Reported in Medical Meta-Analyses; 2026 | 0.511 | 2 | 1 | 100% |
Showing the top 10 of 32 scored citations.