arXiv 24 Jun 2025 · Econometrics
arXiv:2506.20035 · PDF · DOI · OpenAlex · Extracted main text
This paper proposes a test that is consistent against every detectable form of selective reporting and remains interpretable even when the t-scores are not exactly normal. The test statistic is the distance between the smoothed empirical t-curve and the set of all t-curves that would be possible in the absence of any selective reporting. This novel projection test can only be evaded in large meta-samples by selective reporting that also evades all other valid tests of restrictions on the t-curve. A second benefit of the projection test is that under the null we can interpret the projection residual as noise plus bias incurred from approximating the t-score's exact distribution with the normal. Applying the test to the Brodeur et al. (2020) meta-data, we find that the t-curves for RCTs, IVs, and DIDs are more distorted than could arise by chance. But an Edgeworth Expansion reveals that these distortions are small enough to be plausibly explained by the only approximate normality of the individual t-scores. The detection of selective reporting in this meta-sample is therefore more fragile than previously known.
appendix boundary found by appendix_command · 79% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Brodeur, A., N. Cook, and A. Heyes (2020, November) (2020) Methods matter: p-hacking and publication bias in causal analysis in economics | 1.000 | 9 | 3 | 100% |
| 2 | Kudrin, N (2024) Testing for and evaluating the extent of selective reporting | 1.000 | 5 | 3 | 100% |
| 3 | Elliott, G., N. Kudrin, and K. Wüthrich (2024) The power of tests for detecting $p$-hacking | 0.928 | 4 | 3 | 100% |
| 4 | Elliott, G., N. Kudrin, and K. Wüthrich (2022) Detecting p-hacking | 0.874 | 14 | 2 | 100% |
| Andrews | unmatched citation key Andrews | 0.644 | 2 | 2 | 100% |
| 6 | Stellato, B., G. Banjac, P. Goulart, S. Boyd, E. Anderson, V. Bansal… (2024) osqp: Quadratic Programming Solver using the 'OSQP' Library | 0.644 | 2 | 2 | 100% |
| 7 | Brodeur, A., M. Lé, M. Sangnier, and Y. Zylberberg (2016, January) (2016) Star wars: The empirics strike back | 0.405 | 1 | 1 | 100% |
| 8 | Carrasco, M. and J.-P. Florens (2011) A spectral method for deconvolving a density | 0.405 | 1 | 1 | 100% |
| 9 | Havranek, T., Z. Irsova, L. Laslopova, and O. Zeynalova (2024, 09) (2024) Publication and attenuation biases in measuring skill substitution | 0.405 | 1 | 1 | 100% |
| 10 | Head, M., L. Holman, L. Lanfear, A. Kahn, and M. Jennions (2015) The extent and consequences of p-hacking in science | 0.405 | 1 | 1 | 100% |
Showing the top 10 of 15 scored citations. 1 of these could not be matched to a bibliography entry, so only the citation key is shown.