Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
64,037 characters · 13 sections · 48 citation commands
Scaling Reproducibility: An AI-Assisted Workflow for Large-Scale Replication and Reanalysis
\thispagestyle{empty} \doublespace
\setcounter{page}{1} \abovedisplayskip=5pt \belowdisplayskip=5pt \doublespacing
Reproducibility is fundamental to research credibility and cumulative scientific progress. In empirical social science, reproducible analyses allow researchers to verify published claims, scrutinize identifying assumptions, and assess the practical relevance of new methodological developments. Access to real-world data and code has therefore become important both for assessing research credibility and for advancing methodology through systematic reanalysis.
Institutional norms have expanded the availability of replication packages---bundles of data, code, and documentation deposited by authors to enable reproduction of published results. Leading journals in political science and economics now require authors to post replication packages, and some conduct replication checks before publication. Yet availability alone does not ensure reproducibility at scale. Replication packages vary widely in environments, structure, and execution logic, so reproducing results across many papers remains costly and fragile. The bottleneck is operational: executing heterogeneous packages in a standardized and auditable manner requires substantial effort.
This paper develops an AI-assisted workflow for automated full-paper replication to address this execution bottleneck. The workflow reconstructs computational environments, executes code, and verifies published results. Its purpose is to determine whether existing empirical analyses can be reproduced reliably and at scale, not to introduce new estimators or diagnostics. Once data and code are standardized through this process, they can be readily reused, lowering the cost for methodological researchers to develop and apply new methods on a common empirical basis.
A central design principle is the separation of scientific reasoning from computational execution. Researchers specify the empirical targets to be reproduced—model specifications and reported quantities of interest as defined by the paper's narrative and replication package. Conditional on these inputs, replication reduces to execution-oriented tasks: acquiring replication packages, reconstructing environments, locating and running specifications, extracting analysis datasets, and harmonizing outputs. Throughout this paper, we use reproducibility to mean re-execution of existing code and data nasem2019reproducibility, distinguishing it from replicability—obtaining similar findings in new studies. Earlier literature sometimes uses “replication” for the former king1995replication.\footnote{The terminology varies across fields. In political science, king1995replication used “replication” to encompass both re-execution of existing analyses and new studies testing the same hypothesis. Economics has a similar tradition vilhuber2020reproducibility. We follow the distinction codified by the National Academies nasem2019reproducibility and adopted by recent work in both economics vilhuber2020reproducibility and political science alvarez2022reproduce: reproducibility denotes re-execution of the original code and data, while replicability denotes obtaining consistent findings with new data. The compound “replication package”---the standard term for the bundle of code and data deposited by authors---retains the older usage; the property we measure when re-executing those packages is reproducibility.}
Using this workflow, we study a large set of top political science publications. We define the universe as all empirical and quantitative papers in the American Political Science Review (APSR), American Journal of Political Science (AJPS), and Journal of Politics (JOP) from 2010 to 2025, totaling \textcolor{black}{3,464} papers. For this universe, we measure stated data availability using automated extraction of availability statements. Combined with replication outcomes, this allows us to compare trends in stated availability and actual reproducibility over time and relate them to changes in journal policies, particularly the introduction of verification requirements.
We then apply the workflow to two tasks. First, we conduct full-paper replication for a stratified sample of 384 studies (8 papers per journal-year). For each paper, we execute the complete codebase and attempt to reproduce point estimates reported in regression tables, excluding figures, descriptive tables, and simulation results. This defines a consistent unit of evaluation across heterogeneous designs and software environments. The estimated full-paper reproducibility rate rises from \textcolor{black}{29.6%} before DA-RT adoption to \textcolor{black}{79.8%} after, using journal-specific implementation dates (APSR: 2016; AJPS and JOP: 2015). This increase is driven by journal policies requiring public data archiving and, especially, in-house or third-party verification. Conditional on accessible materials, \textcolor{black}{237} of \textcolor{black}{251} papers (\textcolor{black}{94.4%}) are fully reproducible and \textcolor{black}{239} of \textcolor{black}{251} (\textcolor{black}{95.2%}) are fully or largely reproducible regardless of publication year---failures trace almost entirely to missing or restricted data, not to code quality.
Second, we apply the workflow to instrumental variable (IV) designs. Building on lal2024much, we analyze 92 studies and extract 215 specifications for standardized diagnostics, illustrating how automated execution enables systematic analysis across heterogeneous empirical settings. Among the 215 specifications, the workflow achieves an 82% end-to-end autonomous success rate (55 of 67 original papers); all failures reflect unavailable replication archives rather than execution errors. Conditional on accessible materials, the workflow reproduces benchmark estimates exactly and completes all diagnostic tests.
This level of reliability is achieved through an adaptive, version-controlled process. Replication packages span multiple programming languages, estimation commands, directory structures, and coding conventions, and many failure modes arise only when new materials are encountered. Recurring issues are encoded as generalized rules in the execution layer and versioned across runs. Coverage expands across versions, while numerical behavior remains fixed within each version. The Supplementary Materials document these failure patterns and the corresponding adjustments.
In political science, king1995replication argued that scholarly norms should require sufficient data and code to reproduce published results. This vision was formalized through the Data Access and Research Transparency (DA-RT) initiative and the 2014 Journal Editors’ Transparency Statement (JETS), building on public data repositories---principally the Harvard Dataverse king2007introduction, ICPSR, and the Open Science Framework---that provided the infrastructure for deposit and retrieval. The three journals in our corpus adopted verification requirements at different times JETS2014, key2016data, providing institutional variation that allows us to assess whether these policies are associated with improved reproducibility in practice.
These transparency norms made possible a series of large-scale reanalysis projects hainmueller2019much, lal2024much, chiu2023causal, in which much of the effort was devoted to reconstructing environments and harmonizing replication packages rather than developing new methods. Those experiences suggest that, once empirical targets are specified, most remaining work is procedural and therefore automatable. The present workflow operationalizes this insight.
More broadly, this project connects to long-standing concerns about research credibility, from Leamer’s call to “take the con out of econometrics’’ leamer1983let to the credibility revolution angrist2010credibility, torreblanca2026credibility and the replication crisis in psychology open2015estimating. These discussions emphasize verifiable analysis and expose the consequences of fragile research pipelines.
The main contribution of this paper is twofold. First, we make computational reproducibility measurable at scale and document that journal transparency policies---particularly verification requirements---are associated with substantially higher reproducibility rates in political science. Second, we show that automated execution enables standardized diagnostics across published studies, illustrated with IV designs, and can facilitate methodological research by providing reusable data and outputs.
The scope of this study is empirical social science, with data drawn from top political science journals where norms for sharing replication packages are relatively strong. The workflow requires usable code and data and does not recover results when packages are missing or fundamentally flawed. The findings therefore reflect what is feasible under current best practices. Although the analysis focuses on political science, the framework is general and can be extended to other fields with established transparency norms.
This section describes the AI-assisted workflow used to conduct full-paper replication and design-specific diagnostics. The workflow targets the execution bottleneck identified in the introduction: executing heterogeneous replication packages in a standardized and auditable manner. It does not automate methodological reasoning or introduce new statistical procedures. Instead, it standardizes and accelerates execution when usable data and code are available.
\paragraph*{Design principles.}
Large-scale reproducibility involves a basic tension between heterogeneity and determinacy. Replication materials vary widely across studies in programming language (Stata, R, and Python), directory structure, naming conventions, and documentation quality. At the same time, reproducibility requires determinacy: for a fixed pipeline version and fixed inputs, numerical outputs must not depend on ad hoc decisions, platform-specific defaults, or stochastic behavior.
The workflow resolves this tension through three separation principles. First, adaptive coordination is separated from fixed computation. An LLM orchestrator routes tasks, interprets failures, and selects among predefined recovery steps. All numerical work---data preparation, model estimation, and diagnostic computation---is executed by version-controlled program code. For a fixed pipeline version and fixed inputs, the workflow produces identical numerical outputs and retains a complete audit trail. Second, scientific reasoning is separated from execution. Researchers define the empirical targets and, where applicable, the diagnostic templates. The workflow handles the operational tasks: acquiring replication packages, reconstructing environments, executing code, and harmonizing outputs. Third, execution is separated from verification. Running an author’s code successfully does not by itself establish that the paper’s reported results have been reproduced. The workflow therefore treats code execution and result verification as distinct phases, with an explicit gate between them.
\paragraph*{Architecture.} The system is organized as a three-layer architecture. At the top layer, a large language model (LLM), Claude, serves as an orchestrator that dispatches tasks, interprets errors, and determines how the pipeline proceeds. The middle layer consists of structured skill descriptions that define each stage’s input--output contract and record previously resolved failure patterns. At the bottom layer, deterministic agent code and statistical scripts perform all file operations and computation. The orchestrator controls task routing but is excluded from numerical estimation and inference. Figure (ref) illustrates the architecture.
Execution proceeds through three phases. In Phase A (acquisition and execution), the system extracts metadata and replication links from the paper’s PDF, downloads the replication package from public repositories, prepares the code for automated execution by resolving path dependencies and environment assumptions, and executes the complete codebase. All estimation commands encountered during execution are instrumented to capture structured output, including coefficients, standard errors, sample sizes, and clustering. The output is a comprehensive log of every regression the code produces.
\afterpage{
}
In Phase B (reproducibility verification), the system extracts all coefficients reported in the paper’s regression tables and compares them against the code-generated outputs. Matching uses precision-aware rounding: each code coefficient is rounded to the same number of decimal places as the paper-reported value, then checked for exact equality. Optimal assignment across paper--code coefficient pairs ensures that each reported value is matched to the most appropriate code output. Phase B concludes with a reproducibility verdict: papers classified as fully or largely reproducible proceed to Phase C; others enter a bounded fix loop (up to three iterations) before the verdict is finalized.
In Phase C (diagnostic evaluation), the system identifies the research design used in each reproduced specification---currently supporting IV and difference-in-differences designs---and applies the corresponding diagnostic template. For IV designs, this includes first-stage $F$-statistics, Anderson--Rubin tests, bootstrap confidence intervals, jackknife sensitivity, and OLS comparisons, implemented through the ivDiag package ivDiag2024. Phase C concludes with a verification check confirming that all diagnostics completed without error. The system then compiles a standardized report combining the reproducibility assessment from Phase B and the diagnostic results from Phase C.
Each phase produces explicit intermediate artifacts---structured logs, extracted datasets, matching reports, diagnostic outputs---written to disk. Agents share no hidden state. This modularity ensures that every step is inspectable, and execution can resume from any phase without restarting the pipeline.
\paragraph*{Adaptation.} The workflow was developed iteratively through repeated encounters with diverse replication packages. Replication materials span multiple programming languages, estimation commands, directory structures, and coding conventions, and many failure modes arise only when new materials are encountered. When a recurring failure pattern is identified, it is encoded as a generalized rule in the execution layer and version-controlled between runs. Coverage expands across pipeline versions, while numerical behavior remains fixed within each version. The executor is kept design-agnostic: extending to new research designs requires specifying a new diagnostic template rather than modifying the underlying architecture.
We apply the workflow described in Section (ref) to publications from three leading political science journals. The analysis proceeds at three levels. First, we define a universe of empirical and quantitative papers and measure stated data availability at the population level. Second, we draw a stratified sample and attempt full-paper replication. Third, we apply a fixed diagnostic template to a corpus of IV designs. Each level narrows the scope and increases the cost per paper. The first requires only text classification; the second requires executing replication packages; the third requires additional design-specific diagnostics. Table (ref) summarizes the two evaluation tasks.
The three journals adopted transparency requirements at different times and through different mechanisms. The key distinction is between availability policies, which require authors to deposit data and code in a public repository, and verification policies, which require that deposited materials be executed and checked before publication.
The discipline-wide catalyst was the 2014 JETS, organized through the DA-RT initiative. Twenty-seven political science journals---including all three in our corpus---committed to requiring that “cited data are available at the time of publication through a trusted digital repository’’ by January 15, 2016. JETS formalized a norm; implementation varied by journal.
AJPS moved earliest. It introduced a mandatory data archiving policy in 2012 and announced a formal replication policy in March 2015 AJPSReplicationGuide2015. That same year, AJPS contracted with the Odum Institute for third-party verification of accepted manuscripts---one of the earliest systematic external verification programs in the social sciences. Under this policy, corresponding authors of accepted manuscripts must upload replication files to the AJPS Dataverse, and publication is contingent on successful verification. AJPS thus combined availability and verification before the JETS deadline.
JOP transitioned during 2015, coinciding with a publisher change to the University of Chicago Press and an editorial transition. By mid-2015, published JOP articles carried a standardized statement that replication packages were available in the JOP Dataverse---a shift from earlier practice, when authors were asked only to note the location of packages without mandatory deposit key2016data. JOP adopted in-house verification around early 2021, when published articles began stating that the analysis had been “successfully replicated by the JOP replication analyst.’’
APSR signed JETS in 2014 and revised its submission guidelines effective January 2016, but enforcement accelerated under a new editorial team beginning in August 2016. Authors of conditionally accepted manuscripts must now submit a reproducibility package to the APSR Dataverse, and the journal reviews packages to reproduce tables and figures prior to publication.
These policy shifts define three regimes: (i) pre-policy, when data sharing was voluntary or weakly enforced; (ii) post-availability, when journals required data deposit but did not verify reproducibility; and (iii) post-verification, when journals required that submitted code and data reproduce the published results as a condition for acceptance. The timing of each transition differs by journal.
We accessed articles published in the APSR, AJPS, and JOP between 2010 and 2025. Each article was classified by an LLM as empirical and quantitative or not, based on full text, following the criteria in torreblanca2026credibility.\footnote{For this task, we used Claude Sonnet 4 Batch API; model identifier claude-sonnet-4-20250514.} This classification identifies papers whose primary analysis is based on quantitative data from real-world sources, excluding purely theoretical, qualitative, or simulation-based studies. After applying the current screened political-science inclusion filter, the resulting universe contains \textcolor{black}{3,464} papers: \textcolor{black}{902} from APSR, \textcolor{black}{955} from AJPS, and \textcolor{black}{1,607} from JOP.
For each paper in the universe, we extract the data availability statement from the full text and record two indicators: (i) whether the paper includes a stated commitment to make replication packages available, and (ii) whether the statement provides a link to a public repository where the packages can be located. These indicators measure stated availability at the population level. Whether the packages can actually be downloaded and executed is assessed only for the replication sample described below.
From the universe of \textcolor{black}{3,464} empirical papers, we draw a stratified sample of \textcolor{black}{384} studies: \textcolor{black}{8} papers per journal-year across \textcolor{black}{48} cells. Because the number of empirical papers varies across journal-years, we apply inverse probability weights to ensure that sample-level estimates reflect the composition of the universe.
For each sampled paper, we execute the authors' complete codebase and compare every regression coefficient it produces against every coefficient reported in the paper's regression tables. Point estimates are the most precisely defined and consistently reported quantities across heterogeneous studies; exact coefficient agreement implies that the full specification---controls, fixed effects, sample restrictions, and data transformations---was correctly reproduced. We exclude descriptive tables, simulation results, and figures: descriptive tables lack the standardized structure of regression output, and figures would require computer vision to extract numerical targets. Standard errors and other uncertainty measures are captured for downstream diagnostics but are not used as matching criteria, as they are reported inconsistently across papers and can depend on bootstrap seeds. Each code-generated coefficient is rounded to the same number of decimal places as the paper-reported value, then checked for exact equality. Code-generated and paper-reported coefficients are then matched via optimal one-to-one assignment.
We classify a paper as fully reproducible if 100% of point estimates in main-text regression tables are matched, largely reproducible if more than 80% but fewer than 100% are matched, partially reproducible if 50--80% are matched, and not reproducible otherwise.
As a secondary application, we apply the same workflow to a corpus of IV designs. The starting point is the 67 IV studies analyzed in lal2024much, drawn from the same three journals (2010--2022) and satisfying a common set of design restrictions: linear IV models with a single endogenous regressor. The original project manually verified benchmark 2SLS point estimates for each study, which serve as fixed ground truth.
We extend this corpus by incorporating 25 additional IV studies published between 2023 and 2025 under identical inclusion criteria. For each study, the workflow replicates up to three IV specifications, including the baseline and key robustness variants. The total number of evaluated specifications increases from 70 to 215.
Unlike the full-paper task, the IV evaluation targets only the selected specifications rather than all main-text tables (Table (ref)). Each specification is verified through a three-step chain: (i) the authors' code reproduces the paper's coefficients, (ii) an independent R re-estimation matches the code output, and (iii) only fully verified specifications proceed to diagnostic analysis. For the 25 newly incorporated studies, we verify that the automated pipeline reproduces the reported 2SLS estimate.
We trace a single study through the full workflow to illustrate both tasks described in Section (ref). We select Rueda2017, which was previously analyzed in lal2024much, because it exemplifies a rigorous empirical design and provides a clear illustration of how the workflow operates: a clean instrumental variable strategy, a well-organized Stata replication package with clear documentation, and results spanning both IV and non-IV specifications across multiple tables, even though our workflow is capable of handling more complex cases. The paper studies the relationship between polling station size and vote buying in Colombia, instrumenting average polling station size with the maximum size set by election authorities. The replication code is written in Stata---the dominant environment in our evaluation corpus. The workflow processes this paper in two steps: full-paper replication and, for the IV specifications, design-specific diagnostics.
\paragraph*{Full-paper replication.} Before applying design-specific diagnostics, the workflow attempts to reproduce all coefficients reported in the paper’s regression tables. It executes the complete replication package---25 Stata scripts in this case---and compares every code-generated coefficient against the corresponding paper-reported value using precision-aware rounding and optimal assignment.
Table (ref) summarizes the results. Of the 35 coefficients reported across five main-text tables, 31 are exactly reproduced, yielding an overall match rate of 88.6%. The paper is classified as largely reproducible. This verdict reflects the restricted LAPOP microdata rather than any issue with the authors' code; all coefficients that can be evaluated are reproduced exactly.
All four unmatched coefficients trace to a single cause: the LAPOP survey microdata file (final_LAPOP.dta) is not included in the public replication package. Table 4 relies entirely on this file and cannot be reproduced; one specification in Table 6 also depends on it. The remaining 31 coefficients, spanning OLS, IV, and reduced-form estimates across four tables, are reproduced exactly. The LAPOP microdata are subject to third-party access restrictions, and their exclusion from the public replication package is consistent with standard practice for restricted-use survey data. Automating the retrieval and integration of such externally hosted datasets is a natural extension of the workflow.
\paragraph*{IV Diagnostic analysis.} The workflow identifies all eight IV specifications reported in the main-text tables of Rueda2017: four with outcome e_vote_buying and four with outcome sum_vb, each using a progressively richer set of controls. All eight share the same instrument (lz_pob_mesa_f), treatment (lm_pob_mesa), and clustering variable (muni_code).
For the IV specifications identified in Table 5, the workflow applies a three-step verification chain before proceeding to diagnostics: (i) the authors’ code reproduces the paper’s coefficients, (ii) an independent R re-estimation matches the code output, and (iii) only fully verified specifications receive diagnostic analysis. All eight specifications in Rueda2017 pass this chain.
The diagnostic template, described in Section (ref) in the Supplementary Materials, is then applied to each verified specification. Table (ref) reports the results. All eight specifications display strong first stages. The effective $F$-statistics range from 60 to 8{,}598, well above the conventional cutoff of 10, and no model is flagged for weak instruments. Diagnostic details---bootstrap confidence intervals, jackknife sensitivity, and OLS comparisons---are reported in Table (ref). The workflow applies the same criteria uniformly and reports the results without substantive interpretation; the full diagnostic output is included in the standardized report. The reanalysis does not assess the credibility of the core identification assumptions. When there is a single instrument for a single endogenous regressor, these assumptions are not directly testable, and the diagnostics cannot adjudicate their validity.
\paragraph*{Standardized report.} The workflow compiles all outputs---replication results, verification chain, and diagnostic analyses---into a standardized report. The report includes an executive summary listing reproducibility verdicts and diagnostic summaries, followed by specification-level diagnostics with coefficient comparison plots, first-stage $F$-statistics, bootstrap confidence intervals, and jackknife sensitivity analyses. The format is identical across papers, facilitating cross-study comparison.
The full pipeline---from PDF ingestion and package retrieval to report generation---completed in less than 4 minutes of wall-clock time. Diagnostic computation accounted for most of the runtime, and no human intervention was involved. A sample diagnostic report for Rueda2017 is included in the Supplementary Materials (Section (ref)).
\FloatBarrier
This section reports three sets of findings from deploying the AI-assisted workflow at scale. First, we document trends in stated data and code availability across the three journals over the study period. Second, we report full-paper replication results for the stratified sample of \textcolor{black}{384} papers, including inverse-probability-weighted reproducibility rates and a diagnostic classification of non-reproducible cases. Third, we present results from the IV diagnostic corpus, showing that the automated pipeline reproduces the core patterns documented in lal2024much. Across the political-science replication tasks, the workflow processed \textcolor{black}{3,382} targeted models and executed \textcolor{black}{2,012} code scripts with no human intervention.
Before examining replication outcomes, we summarize the transparency environment in which the workflow operates. For each paper in the universe of \textcolor{black}{3,464} empirical articles, the Profiler agent extracts whether the paper includes a data availability statement and whether a repository URL is provided. These indicators measure stated availability at the population level. For the stratified replication sample ($n = \textcolor{black}{384}$), we go further and verify whether replication packages are actually downloadable and whether they contain usable data.
\afterpage{
}
Figure (ref) presents both perspectives. The upper panel plots population-level stated availability by journal. Availability is near zero for all three journals before 2012. For APSR, the sharp increase coincides with the adoption of DA-RT around 2016, with availability climbing steeply after 2019--2020 as the journal adopted in-house verification. For AJPS and JOP, stated availability has been high since the early 2010s. By the end of the study period, stated availability exceeds 90% in all three journals.
The lower panel provides a more intensive validation using the replication sample. Three lines track progressively stricter definitions of availability: stated availability, actually downloadable replication packages, and packages containing usable (non-restricted) data. The gap between these lines is revealing. Even for AJPS and JOP, where stated availability appears high early on, consistent data availability---where downloadable packages reliably contain usable data---only emerges after the adoption of DA-RT around 2015--2016. The shaded area between the downloadable and usable-data lines captures papers with available replication packages but entirely restricted data. In the post-DA-RT period, restricted data emerges as the primary remaining barrier to reproducibility: authors comply with archiving requirements by posting code and non-restricted data, but key analyses depend on proprietary, confidential, or third-party datasets that cannot be included. Addressing this gap---through secure remote execution, synthetic data, or coordinated access agreements---is an important frontier for the field.
These trajectories reflect a broader institutional transformation. Journal mandates were effective in part because authors could deposit packages in established public repositories---principally the Harvard Dataverse, ICPSR, and the Open Science Framework---that provide persistent access, DOI-based citation, and programmatic retrieval. The infrastructure preceded the mandates and made compliance feasible at scale. The implication for reproducibility is direct: without retrievable replication packages, reproduction cannot begin. Papers published before mandatory archiving are far less likely to have accessible packages, and this shapes the results reported in the next subsection.
We now turn to the main empirical task: full-paper replication of \textcolor{black}{384} studies sampled from the three journals across 2010--2025. For each paper, the workflow attempts to download the replication package, execute all code, extract every coefficient from the paper’s regression tables, and match code outputs to paper-reported values.
Of the \textcolor{black}{384} sampled papers, \textcolor{black}{275} (71.6%) have replication files available in a public repository; the remaining \textcolor{black}{109} have no replication package posted. Among papers with available files, \textcolor{black}{251} (65.4% of the full sample) have data that can be accessed and code that can be executed; \textcolor{black}{24} papers are blocked by restricted or proprietary data. Of the \textcolor{black}{251} papers with accessible data and executable code, \textcolor{black}{237} (94.4%) are classified as fully reproducible (Figure (ref)).
The dominant source of non-replication is the absence of replication packages, not code failure: missing packages (\textcolor{black}{109}) and restricted data (\textcolor{black}{24}) together account for 133 of the 147 papers that are not fully reproducible.\footnote{That said, our workflow was able to fix repairable coding errors in 10 studies; these are classified as fully reproducible. See Table (ref) in the Supplementary Materials.} Only \textcolor{black}{14} papers with accessible materials receive a verdict below fully reproducible: \textcolor{black}{2} are largely reproducible, \textcolor{black}{9} are partially reproducible, and \textcolor{black}{3} are not reproducible---see Table (ref) in the Supplementary Materials for a detailed breakdown. The adaptive execution layer contributes materially to coverage by repairing common author-side packaging errors that would otherwise block execution, but it cannot overcome packages that were never posted.
\afterpage{
}
\paragraph*{Reproducibility by journal and year.} Figure (ref) disaggregates these outcomes by journal and publication year. The upper panel shows paper-level verdicts: papers published before DA-RT adoption are dominated by missing replication packages (tan bars), while papers published after are predominantly fully reproducible (green bars). The timing of this shift closely tracks the adoption of mandatory data archiving and verification policies documented in Figure (ref), strongly suggesting that institutional policy change---not secular trends in author behavior---drove the improvement. The lower panel shifts from paper-level verdicts to model-level outcomes, restricting attention to papers with downloadable, non-restricted replication packages. For each journal-year cell, bars show the number of models the workflow attempted to replicate, with solid shading for replicated models and lighter shading for non-replicated ones. The contrast between panels is informative: while the upper panel shows substantial non-replication driven by missing data, the lower panel reveals that conditional on data availability, the share of non-reproducible models is very small across all journals and time periods.
\paragraph*{Population-level estimates.} Table (ref) quantifies these patterns using inverse-probability-weighted rates, with journal-specific DA-RT implementation dates as cutoffs (APSR: 2016; AJPS and JOP: 2015). Because the number of empirical papers varies across journal-years---from \textcolor{black}{27} to \textcolor{black}{177}---the inverse-probability-weighting (IPW) estimator reweights each cell to reflect its share of the publication universe. Across all years, the estimated pooled reproducibility rate is \textcolor{black}{67.2%} with a 95% CI of [\textcolor{black}{62.9%}, \textcolor{black}{71.4%}]. The rate is substantially higher in the post-DA-RT period: \textcolor{black}{79.8%} ([\textcolor{black}{74.6%}, \textcolor{black}{84.9%}]) compared to \textcolor{black}{29.6%} ([\textcolor{black}{22.1%}, \textcolor{black}{37.0%}]) before DA-RT adoption.
In the post-DA-RT period, the three journals show broadly similar rates: AJPS at \textcolor{black}{84.4%}, JOP at \textcolor{black}{81.2%}, and APSR at \textcolor{black}{73.0%}. AJPS, which introduced third-party verification earliest (2015), has the highest rate; APSR (\textcolor{black}{73.0%}), which adopted verification only around 2021, lags despite requiring data deposit since 2016. Pre-DA-RT rates vary more widely---AJPS at \textcolor{black}{47.0%} versus APSR at \textcolor{black}{16.9%} and JOP at \textcolor{black}{20.6%}---reflecting AJPS’s earlier culture of data sharing even before formal mandates. The timing of the pre-to-post shift closely tracks verification adoption at each journal, consistent with the conclusion that active verification---not merely archiving mandates---drove the improvement in reproducibility.
\paragraph*{Runtime.} The workflow processes the full sample of \textcolor{black}{384} papers, and processing is fully parallelizable. Runtime varies substantially by outcome category. For papers where no public replication package is found, the pipeline terminates after profiling and retrieval attempts, with a median wall-clock time under 2 minutes. For papers that enter full execution---downloading materials, running code, extracting and matching coefficients---the median end-to-end time is under 10 minutes, with the most complex codebases requiring up to roughly one hour. Most wall-clock time is spent downloading replication packages and executing authors’ code; LLM calls account for a small share of total computation. Once the execution layer is stabilized, the marginal cost per additional paper is low. By comparison, manual full-paper replication at this scale would require person-years of effort.
As a secondary application, we evaluate the workflow on the IV diagnostic corpus described in Section (ref). This corpus provides a validation anchor: the 67 original studies in lal2024much have manually verified benchmark estimates, allowing direct assessment of the workflow’s execution reliability.
For the original 67 papers, the workflow autonomously retrieves, executes, and analyzes 55 (82%). All 12 failures occur at the material retrieval stage---replication archives for these pre-2020 papers are no longer publicly available. Using archived materials retained from the earlier project, the workflow reproduces all 67 papers: conditional on accessible data and code, specification extraction, execution, and diagnostic analysis succeed in every case. Performance generalizes to the 25 newly incorporated papers, all of which succeed end to end, yielding 215 successfully processed specifications across the combined corpus (Table (ref) in the Supplementary Materials). The high conditional success rate reflects the adaptive nature of the execution layer, the strong compliance with verification requirements among post-DA-RT publications, and the deterministic nature of the diagnostic template.
\paragraph*{Empirical patterns.} We apply the diagnostic template from lal2024much to the extended corpus of 215 specifications (92 papers). Figure (ref) mirrors Figure 5 in lal2024much. The core empirical finding is preserved: 2SLS estimates systematically exceed OLS estimates in magnitude, and the 2SLS-to-OLS ratio is negatively correlated with first-stage strength among observational designs ($p < 0.001$, standard errors clustered at the study level) but not among experimental designs ($p = 0.391$).
The extended sample confirms the original finding: 2SLS magnitudes systematically exceed OLS (median ratio 3.0), and the discrepancy grows as first-stage strength weakens among observational designs, consistent with bias amplification under violations of instrument validity. The pattern is absent among experimental designs. These results reinforce the argument in lal2024much that many observational IV estimates rest on fragile identification assumptions. The key difference is scale and speed: whereas the original study required approximately four years of manual data collection, processing, and reanalysis, the automated workflow processed the expanded corpus within days.
\FloatBarrier
This paper shows that computational reproducibility in political science can be verified at scale using an AI-assisted workflow, and that journal verification requirements are associated with substantially higher reproducibility rates. Among \textcolor{black}{384} sampled papers from three top journals over 2010--2025, the estimated population-level reproducibility rate rises from \textcolor{black}{29.6%} before DA-RT adoption to \textcolor{black}{79.8%} after. The workflow also enables standardized diagnostics at scale, reproducing and extending the IV analysis of lal2024much across 215 specifications in days rather than years. Together, these results demonstrate that automated verification is feasible, that existing transparency policies have worked, and that the remaining frontier is expanding data access.
These results rest on institutional infrastructure that took decades to build. king1995replication argued that scholarly norms should require sufficient data and code to reproduce published results. The DA-RT initiative and the 2014 JETS formalized that argument into journal policy. Public repositories---the Harvard Dataverse king2007introduction, ICPSR, the Open Science Framework---provided the deposit and retrieval layer that made compliance feasible. AJPS's contract with the Odum Institute for third-party verification (2015), and the AEA Data Editor system in economics (2019) vilhuber2020reproducibility, demonstrated that active verification, not merely archiving, is what drives compliance. Our results show, at scale, that these investments paid off: the pre-to-post DA-RT shift accounts for the bulk of the difference in reproducibility rates. Making data and code publicly available is an important first step; verification infrastructure ensures that posted materials actually work. The present workflow can substantially reduce the cost of providing that verification.
The workflow scales evaluation rather than defining evaluative standards: it relies on diagnostic templates specified by human experts and applies them uniformly across studies. Human-led replication initiatives---journal data editor programs, research transparency organizations such as OSF and BITSS, the Institute for Replication (I4R), and independent replication communities---remain indispensable for evaluating identification strategies, measurement choices, and research design.
All results are conditional on the existence and accessibility of replication packages. The workflow does not take the extra step of fetching data that may be publicly available but not included in the replication package, such as public-use surveys or government datasets hosted on external platforms. When such data are absent from the package, the paper is classified as having inaccessible packages, even though a researcher willing to locate and download the data separately could potentially reproduce the results. The high conditional reproducibility rate (\textcolor{black}{94.4%} among papers with accessible packages) therefore characterizes the quality of self-contained packages, not the reproducibility of the literature as a whole. Similarly, the matching criterion targets point estimates only; extending verification to figures, uncertainty estimates, and other reported quantities is a natural next step. The workflow also does not currently handle studies that are highly computationally intensive, such as large-scale simulation studies and Bayesian analyses requiring extended MCMC sampling. These studies can in principle be incorporated with additional computational resources and are a natural target for future extension.
The workflow can be extended along three dimensions. First, we plan to apply it to additional research designs for which structured corpora already exist, including two-way fixed effects studies chiu2023causal and heterogeneous treatment effect analyses using linear interaction models hainmueller2019much, liu2025practical. Second, the scope of replication can be deepened beyond benchmark specifications to reconstruct complete tables and figures, verify internal coherence across reported results, and automate diagnostics such as cluster jackknife, leave-one-out influence, and alternative inference procedures. Third, the workflow can integrate into the research and publication pipeline---assisting authors in preparing replication packages at submission, and enabling journals or third-party verification services to conduct reproducibility checks more routinely. A companion study extending this framework across six fields is in preparation.
Beyond these extensions, the broader implications concern how verification, reporting, and methodological development are organized.
\paragraph*{Lowering the cost of verification dramatically.}
The most immediate implication is a substantial reduction in the marginal cost of verification. When replication packages are available, recomputing estimates and applying standardized diagnostics becomes considerably less expensive than under current practice. This does not resolve disputes about identification or theory, but it changes incentives. At present, systematic verification is rare because its cost often exceeds its expected benefit. As that cost falls, more journals may find it feasible to require in-house or third-party reproducibility checks as a condition for acceptance. Authors, anticipating a higher likelihood of auditing, may adopt more disciplined coding practices and address influence, clustering, and resampling concerns ex ante. Verification thus becomes more closely integrated into the publication process rather than applied only after controversies arise.
\paragraph*{Standardizing diagnostic reporting.}
Uniform diagnostic protocols can also reshape reporting norms. If weak-instrument tests, robust inference procedures, and sensitivity analyses are implemented automatically and summarized in standardized formats, discretion in how robustness is selected and presented declines. Journals may require structured diagnostic summaries alongside main results, much as data-availability statements have become routine. Referees may increasingly expect influence diagnostics and alternative clustering checks as part of the baseline empirical presentation. Over time, graduate training may adapt to treat such diagnostics as integral components of empirical analysis rather than supplementary exercises.
\paragraph*{Enabling large-scale reanalysis and accelerating methodological research.}
Harmonized analysis datasets and structured metadata enable large-scale reanalysis under consistent criteria. Each of our previous large-scale reanalysis projects---hainmueller2019much, lal2024much, and chiu2023causal---took three to four years of sustained effort. Much of that time was spent harmonizing replication packages, clarifying benchmark estimands, and standardizing robustness checks across heterogeneous applications. With the present workflow, many of these steps can be automated, substantially reducing the time required to conduct comparable large-scale reanalyses.
In the near term, this infrastructure may support more frequent and systematic reassessment of empirical literatures. Research groups, professional associations, or journals could periodically revisit published findings using updated diagnostic standards without incurring multi-year coordination costs. As harmonized corpora accumulate, empirical claims may compete not only on substantive grounds but also on demonstrated stability under shared diagnostics.
A large quantity of harmonized data will have profound implications for methodological research. In computer science, benchmark datasets such as ImageNet Deng2009, MS COCO Lin2014, SQuAD Rajpurkar2016, and GLUE Wang2018GLUE structured progress by providing common evaluation environments. Researchers could compare algorithms under identical tasks and metrics, which facilitated cumulative improvement. Analogously, a large collection of harmonized empirical datasets with standardized diagnostic outputs can serve as a benchmark platform for causal and statistical methods. Methodologists could evaluate new estimators and inference procedures across diverse real-world applications rather than relying primarily on stylized simulations. By lowering the cost of empirical validation, the workflow may help shift methodological research toward cumulative comparison under shared empirical settings.
Taken together, these implications suggest that agentic AI workflows for reproducibility, with humans in the loop, can function as research infrastructure. They do not replace researchers' substantive judgment, but make systematic evaluation easier to conduct and harder to avoid. By lowering the cost of verification, standardizing diagnostics, and accelerating methodological development, they may help make transparency and cumulative scrutiny part of routine empirical practice.
\onehalfspacing