Graham Elliott, Nikolay Kudrin, Kaspar Wuthrich
arXiv 16 Jun 2019 · Econometrics · publishedEconometrica (2022) · 58 citations (OpenAlex)
arXiv:1906.06711 · PDF · DOI · OpenAlex · Extracted main text
We theoretically analyze the problem of testing for $p$-hacking based on distributions of $p$-values across multiple studies. We provide general results for when such distributions have testable restrictions (are non-increasing) under the null of no $p$-hacking. We find novel additional testable restrictions for $p$-values based on $t$-tests. Specifically, the shape of the power functions results in both complete monotonicity as well as bounds on the distribution of $p$-values. These testable restrictions result in more powerful tests for the null hypothesis of no $p$-hacking. When there is also publication bias, our tests are joint tests for $p$-hacking and publication bias. A reanalysis of two prominent datasets shows the usefulness of our new tests.
appendix boundary found by none_found · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Simonsohn, U., Nelson, L. D., and Simmons, J. P (2014) P-curve: a key to the file-drawer | 1.000 | 7 | 4 | 100% |
| 2 | Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., and Jennions, M. D (2015) The extent and consequences of p-hacking in science | 0.928 | 4 | 4 | 100% |
| 3 | Brodeur, A., Lé, M., Sangnier, M., and Zylberberg, Y (2016) Star wars: The empirics strike back | 0.874 | 5 | 2 | 100% |
| 4 | Cox, G. and Shi, X (2020) Simple adaptive size-exact testing for full-vector and subvector inference in moment inequality models | 0.737 | 3 | 2 | 100% |
| 5 | Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., and Jennions, M. D (2016) Data from: The extent and consequences of p-hacking in science | 0.644 | 4 | 1 | 100% |
| 6 | Simonsohn, U., Simmons, J. P., and Nelson, L. D (2015) Better p-curves: Making p-curve analysis more robust to errors, fraud, and ambitious p-hacking, a reply to Ulrich and Miller (20… | 0.644 | 2 | 2 | 100% |
| 7 | Beare, B. K. and Moon, J.-M (2015) Nonparametric tests of density ratio ordering | 0.511 | 2 | 1 | 100% |
| 8 | Brodeur, A., Lé, M., Sangnier, M., and Zylberberg, Y (2016) Replication data for: Star wars: The empirics strike back | 0.511 | 2 | 1 | 100% |
| 9 | R Core Team (2020) R: A Language and Environment for Statistical Computing | 0.405 | 1 | 1 | 100% |
| 10 | Andrews, I. and Kasy, M (2019) Identification of and correction for publication bias | 0.405 | 1 | 1 | 100% |
Showing the top 10 of 32 scored citations.