The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
43,179 characters
CausalVerify: An Execution-Grounded Benchmark for LLM Causal Inference Workflows
\maketitle
\begin{abstract}
Existing causal-inference benchmarks for LLMs often evaluate text
descriptions of methods or whether generated code can run. They rarely
test whether the executed workflow recovers the target causal estimate.
CausalVerify studies this verification problem for structured
econometric causal-estimation workflows by separating realistic
interpretation from verifiable computation. The benchmark draws on 259
published economics papers, each represented by a reconstructed research
question, data description, and institutional context, together with 100
fixed-seed synthetic scenarios that realise CSV datasets for
difference-in-differences (DID), event study (ES), instrumental variables
(IV), and regression discontinuity design (RDD). Experiment~A (real-paper text agreement) scores
method-family and direction agreement against four-LLM consensus labels.
Experiment~B (synthetic execution) runs model-written R code and checks
whether the extracted treatment-effect estimate matches a canonical
estimator on the same realised dataset; we call this execution-grounded
correctness layer L2b+, as distinct from L2b, which records only whether
the code executes. A calibration arm asks whether self-reported confidence
separates correct from incorrect workflows. On Experiment~B, seven evaluated LLMs reach final L2b+ pass rates
between 10\% and 88\% at the default 50\% tolerance (9\% to 84\% at
a stricter 25\%; Appendix~\ref{app:tolerance_sweep}), and 66 of the 426
workflows that execute (15.5\%) return a wrong estimate.
Execution ranking (L2b) agrees with reference-agreement correctness
(L2b+) far better than text-direction scoring (L4, direction agreement
with consensus labels):
Kendall $\tau=0.81$ and Spearman
$\rho=0.93$, versus Kendall $\tau$ between $-0.20$ and $0.10$ for L4. A
robustness run with Llama-3.3-70B-Instruct shows the same qualitative
gap, and reported confidence does not reliably separate correct from
incorrect workflows. The claims are confined to standardized single-shot
workflows in these four design families under the evaluated R backend and
model panel; the benchmark does not measure general causal-inference
ability. Code, data, cached outputs, and a datasheet are released at
\url{https://github.com/causalverify/causalverify}.
\end{abstract}
\section{Introduction}
Evaluating LLMs for empirical causal inference requires more than text
scoring of methods and effect signs: it requires verifying that the
executed analysis recovers a target estimate. A valid empirical claim
requires an estimand, an identification argument, an estimator,
executable analysis code, and a numerical result whose interpretation
matches the target causal quantity~\citep{rubin1974estimating,
rosenbaum1983central,holland1986statistics,pearl2009causality,
imbens2015causal,angrist2009mostly}. This structure exposes a failure mode that standard
LLM evaluations can miss: a model may name the right empirical design,
produce code that runs without error, and still estimate the wrong
coefficient. In applied research, such failures are more dangerous than
obvious failures because the response appears fluent, complete, and
confident while the causal claim remains unverified. This concern sits
within broader verification pressure in scientific publishing, where
more than 10{,}000 papers were retracted in 2023, a record high
reported by \citet{vannoorden2023retractions}. As LLMs move from text
assistance toward software engineering, scientific-analysis agents, and
laboratory-instrumentation control
~\citep{gao2023pal,yao2023react,schick2023toolformer,
liu2023agentbench,wu2023autogen,huang2024mlagentbench,
jimenez2024swe,majumder2024discoverybench,chen2024scienceagentbench,
xie2025autonomouslab},
causal-inference evaluation must ask not
only whether a model sounds right, but whether its executed workflow
recovers the right estimate.
Existing benchmarks capture only parts of this problem. Prior work
and benchmarks on LLM causal reasoning study causal direction, graph
reasoning, counterfactuals, correlation-to-causation judgments,
intervention design, surveys of benchmark validity, and real-world
identification tasks~\citep{kiciman2023causal,zecevic2023causalparrots,
jin2023cladder,jin2024corr2cause,chen2024clear,chi2024unveiling,
yang2024criticalreview,ma2025causalsurvey,zhou2024causalbench,
acharya2025causcibench,lee2025econcausal,shi2026intervenebench,
sawarni2026causalreasoningbenchmark}. Code benchmarks evaluate whether
programs can be synthesized, executed, or made to pass
tests~\citep{chen2021codex,austin2021program,hendrycks2021apps,
li2022alphacode,lu2021codexglue,cassano2023multiple,lai2023ds1000,
gu2024cruxeval,jimenez2024swe,jain2024livecodebench,
zhuo2024bigcodebench}. These
settings are valuable, but causal workflows require a stricter check:
after a model interprets a study, chooses a design, writes analysis
code, and reports a treatment effect, does the executed workflow
recover the intended estimate? This gap matters because code can
pass surface checks while remaining semantically
wrong~\citep{liu2023chatgptcode}.
We introduce \textsc{CausalVerify}, an execution-grounded benchmark for
structured econometric causal-estimation workflows: tasks that begin after
the research question, data description, and institutional context have
been specified and end with an executed treatment-effect estimate in one of
four quasi-experimental design families. The benchmark deliberately separates
ecological realism from executable verifiability. Real published papers
provide realistic institutional context, but rarely provide the cleaned
data, final runnable specification, and counterfactual treatment effect
needed for direct verification. We therefore use 259 published
economics papers to measure text-level agreement on method family and
effect direction, not as verified causal correctness. Conversely,
fixed-seed synthetic DGPs sacrifice some realism but provide executable
targets: 100 realised CSV datasets across difference-in-differences,
event study, instrumental variables, and regression discontinuity
designs~\citep{imbens1994identification,angrist1996identification,
thistlethwaite1960regression,imbens2008rdd,lee2010rdd,
calonico2014robust,mackinlay1997event,bertrand2004trust,
abadie2005semiparametric,callaway2021did,goodmanbacon2021did,
sun2021eventstudy,santanna2020drdid,roth2023trending}.
Models write R code, the
benchmark executes it, and L2b+ checks whether the extracted
treatment-effect estimate matches a canonical estimator on the same
realised dataset. The benchmark does not measure general causal-inference
ability: problem formulation, data collection, design selection from
ambiguous field constraints, matching, propensity-score, doubly-robust,
mediation, and heterogeneous-effect workflows, other software ecosystems,
and interactive human--model collaboration lie outside its scope
(Sections~\ref{sec:benchmark} and~\ref{sec:limitations}).
\textsc{CausalVerify} separates four signals that are often conflated:
textual interpretation, code executability, numerical coefficient
recovery, and self-assessed confidence. Across seven evaluated LLMs,
final L2b+ pass rates range from 10\% to 88\%. Execution success is
strongly aligned with reference-agreement correctness (Kendall
$\tau = 0.81$, Spearman $\rho = 0.93$), whereas text-direction
agreement is weakly or negatively aligned with Exp~B L2b+ ranking
(Kendall $\tau \in [-0.20, 0.10]$). Execution is not
sufficient---some code runs and estimates the wrong coefficient---but
it is a stronger proxy for execution-grounded agreement with the
reference implementation under our reference-label construction than
L4 direction agreement. A calibration arm further shows that naive
retrospective confidence reports do not cleanly separate correct from
incorrect workflows under our prompt,
connecting the benchmark to work on calibration, verbalized
uncertainty, selective prediction, and
self-checking~\citep{guo2017calibration,geifman2017selective,
jiang2021know,lin2022uncertaintywords,mielke2022linguistic,
kadavath2022know,kuhn2023semantic,manakul2023selfcheckgpt,
tian2023calibration,xiong2024uncertainty}. Our contributions are a
benchmark design separating real-paper agreement from execution-grounded
reference-agreement correctness, L2b+ as a metric for agreement with a
benchmark-fixed reference implementation, and empirical evidence that, within
structured econometric causal-estimation workflows, plausible text,
runnable code, agreement with the reference implementation, and
confidence can diverge sharply.
\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{fig1_causalverify_overview.pdf}
\caption{\textsc{CausalVerify}{} benchmark overview. Exp~A uses published economics
papers to form text-agreement tasks for method family and direction.
Exp~B uses fixed-seed synthetic DGPs and realised datasets to evaluate
executable model-written R workflows against canonical estimators.
The calibration arm returns each model its own Exp~B answer and compares
self-assessed confidence with L2b+ correctness.}
\label{fig:benchmark_overview}
\end{figure}
\section{Related Work and Positioning}
Prior work evaluates important pieces of the LLM causal-analysis
workflow, but usually not the executed causal estimate. Causal
benchmarks test textual reasoning about direction, counterfactuals,
graphs, interventions, or identification. Code benchmarks test whether
generated programs execute or pass functional tests. Science-agent
benchmarks evaluate broader discovery or tool-use workflows.
\textsc{CausalVerify} differs in its verification target: the primary
correctness signal is not a text label or a generic program test, but
whether executed model-written analysis code recovers a
benchmark-defined target estimate on the realised dataset.
Among existing benchmarks, CauSciBench~\citep{acharya2025causcibench}
is nearest in scope: it evaluates end-to-end causal-analysis performance
over curated real, synthetic, and textbook tasks. \textsc{CausalVerify}
asks a complementary measurement question: when a model appears to
understand a causal task, which layer of the workflow is externally
verified? It separates real-paper text agreement, controlled-DGP
execution verification, and retrospective calibration rather than
collapsing them into a single causal-ability score.
Concurrent work by Sawarni et
al.~\citep{sawarni2026causalreasoningbenchmark} disentangles causal
identification from estimation on real-world datasets, and its finding
that strategy identification is easier than full specification is
consistent with our name-versus-compute gap. The methodological
difference is the unit of verification: \textsc{CausalVerify} verifies
generated and executed R code on controlled DGP data, extracts the
reported coefficient, and checks it against a realised-data canonical
estimate. This isolates the failure mode where code runs but computes
the wrong causal quantity. Recent workflow-oriented evaluations such as
BioMysteryBench
place models in bioinformatics environments with real data, tool
access, and objective final
answers~\citep{anthropic2026biomysterybench}; \textsc{CausalVerify}{} is complementary:
it fixes a realised-data causal reference estimate so that
model-written causal code can be audited layer by layer rather than
graded only on a final answer. Table~\ref{tab:benchmark_comparison}
summarizes the distinction from adjacent benchmarks.
\begin{table}[H]
\centering
\caption{Comparison with adjacent benchmarks. The table reports
evaluation affordances rather than overall benchmark quality. A check
mark (\textcolor{passgreen}{\ding{51}}) indicates that a benchmark contains an explicit mechanism
for that dimension, not that it is superior in all respects.
\textcolor{orange}{$\sim$}~= partial; \textcolor{failred}{\ding{55}}~= absent. $^{\dagger}$EconCausal counts
10{,}490 triplets across 2{,}595 studies (Lee et al.\ 2025).}
\label{tab:benchmark_comparison}
\footnotesize
\setlength{\tabcolsep}{2.5pt}
\begin{tabularx}{\textwidth}{lclcccc}
\toprule
\textbf{Benchmark} & \textbf{N} & \textbf{Target} &
\textbf{Real ctx.} & \textbf{Exec. coeff.} &
\textbf{Layered Dx} & \textbf{Self-assess.} \\
\midrule
CLadder / CORR2CAUSE~\citep{jin2023cladder,jin2024corr2cause} & 11k / 400 & Causal QA / direction
& \textcolor{orange}{$\sim$} & \textcolor{failred}{\ding{55}} & \textcolor{failred}{\ding{55}} & \textcolor{failred}{\ding{55}} \\
CausalBench~\citep{zhou2024causalbench} & 3 & Graph + effect
& \textcolor{failred}{\ding{55}} & \textcolor{failred}{\ding{55}} & \textcolor{orange}{$\sim$} & \textcolor{failred}{\ding{55}} \\
EconCausal~\citep{lee2025econcausal} & 10{,}490$^{\dagger}$ & Economic causal sign
& \textcolor{passgreen}{\ding{51}} & \textcolor{failred}{\ding{55}} & \textcolor{failred}{\ding{55}} & \textcolor{failred}{\ding{55}} \\
CauSciBench~\citep{acharya2025causcibench} & 367 & End-to-end CI pipeline
& \textcolor{passgreen}{\ding{51}} & \textcolor{orange}{$\sim$} & \textcolor{orange}{$\sim$} & \textcolor{failred}{\ding{55}} \\
InterveneBench~\citep{shi2026intervenebench} & 744 & Intervention design
& \textcolor{passgreen}{\ding{51}} & \textcolor{orange}{$\sim$} & \textcolor{orange}{$\sim$} & \textcolor{failred}{\ding{55}} \\
CausalReasoningBench~\citep{sawarni2026causalreasoningbenchmark} & 173 & ID + estimate
& \textcolor{passgreen}{\ding{51}} & \textcolor{failred}{\ding{55}} & \textcolor{orange}{$\sim$} & \textcolor{failred}{\ding{55}} \\
Code benchmarks~\citep{chen2021codex,jimenez2024swe} & -- & Program tests
& \textcolor{failred}{\ding{55}} & \textcolor{orange}{$\sim$} & \textcolor{orange}{$\sim$} & \textcolor{failred}{\ding{55}} \\
Discovery / science agents~\citep{majumder2024discoverybench,chen2024scienceagentbench} & 264 / 102 & Discovery workflow
& \textcolor{passgreen}{\ding{51}} & \textcolor{orange}{$\sim$} & \textcolor{orange}{$\sim$} & \textcolor{failred}{\ding{55}} \\
\rowcolor{lightblue}
\textsc{CausalVerify}{} & 259 + 100 & Causal workflow
& \textcolor{passgreen}{\ding{51}} & \textcolor{passgreen}{\ding{51}} & \textcolor{passgreen}{\ding{51}} & \textcolor{passgreen}{\ding{51}} \\
\bottomrule
\end{tabularx}
\end{table}
\section{Benchmark Construction and Controls}
\label{sec:benchmark}
\textsc{CausalVerify} separates three evaluation regimes because no single
setting provides both ecological realism and executable reference estimates.
Real papers test whether models parse applied research contexts, but their
reference labels are interpretive: the original cleaned data, exact runnable
specification, and counterfactual effect are usually unavailable. This
reflects the structure of empirical causal work, where claims depend on
the alignment of estimand, identification, estimator, and
data~\citep{angrist2009mostly,imbens2015causal}.
Synthetic DGPs give up some realism but make coefficient correctness
executable. Calibration then asks the deployment-facing question: when no
external verifier is available, can the model's own confidence serve as a
useful warning signal?
\paragraph{Scope.}
Every task begins after the research question, data description, and
institutional context have been specified. The benchmark therefore
evaluates the downstream portion of empirical practice --- design
recognition through estimation and reporting --- for four
quasi-experimental families: DID, ES, IV, and RDD. It does not evaluate
problem formulation, data acquisition, design selection from ambiguous
field constraints, or iterative robustness analysis, and it does not cover
matching, propensity-score, doubly-robust, mediation, or
heterogeneous-effect workflows. These are scope statements, not
afterthoughts: competition-style evaluations that overweight modeling at
the expense of subject-matter reasoning are ``somewhat detached from the
practice of causal inference''~\citep{hernan2019comment}, and we claim
nothing outside the stated boundary.
\paragraph{Exp A: real-paper context.}
Exp~A contains 259 retained published economics papers after excluding
two source failures. Each paper is represented by three reconstructed
fields: research question, data description, and institutional context.
Seven models are run on every paper, for 1813 outputs. Exp~A is
intentionally a text-level benchmark, comparable to prior LLM
causal-reasoning evaluations that score textual causal direction, method
recognition, sign judgments, or identification/estimation
judgments~\citep{jin2023cladder,jin2024corr2cause,lee2025econcausal,
sawarni2026causalreasoningbenchmark}. It measures method-family agreement (L3) and direction agreement (L4)
against four-LLM consensus reference labels from a pool of Claude
Opus~4.7, GPT-4o, Kimi, and Gemini 2.5 Flash; this pool overlaps with
the evaluated panel and we discuss the implied structural circularity in
Section~\ref{sec:limitations}. The benchmark does not treat these labels as
human-adjudicated causal claims; it uses real papers to test whether
models read applied study contexts in the same way as a structured
consensus procedure. The execution-grounded reference-agreement
endpoint is reserved for Exp~B.
\paragraph{Exp B: synthetic DGP execution.}
Exp~B contains 100 fixed-seed synthetic DGP scenarios across standard
applied causal-inference designs: difference-in-differences and event
studies, instrumental variables, and regression discontinuity
designs~\citep{callaway2021did,sun2021eventstudy,mackinlay1997event,
imbens1994identification,angrist1996identification,
thistlethwaite1960regression,imbens2008rdd,lee2010rdd}. The 100 scenarios
comprise 30 DID, 24 Event Study, 24 IV, and 22 RDD tasks; the realised
datasets and canonical estimates are fixed
before model evaluation. Models receive the research question, data
description, and a preview of the realised CSV, then produce runnable R
code. We execute the code and compare the extracted treatment-effect
estimate to a canonical estimator computed on the same realised dataset.
This realised-data comparison avoids penalizing models for finite-sample
deviations between the realised sample and the ideal DGP parameter.
Recent large-scale multi-analyst evidence shows that empirical claims can
vary under alternative defensible analyses of the same research question
and data~\citep{silberzahn2018many,botviniknezer2020variability,aczel2026analyticalrobustness}.
We therefore treat the
Exp~B canonical estimator as a fixed reference path for executable
evaluation, not as the only scientifically defensible analysis.
\paragraph{Calibration: self-assessment.}
We attempt a calibration query for each frozen Exp~B output; response
availability yields 646 calibration records. The model is shown its own
previous analysis without the canonical estimate or pass/fail label and
asked to report confidence in the method choice, specification, and
numerical estimate. This is a
retrospective verbalized-confidence protocol rather than token-probability
calibration, connecting the benchmark to prior work on neural-network
calibration, LLM self-knowledge, and confidence
elicitation~\citep{guo2017calibration,kadavath2022know,
tian2023calibration,xiong2024uncertainty}. The resulting confidence is
compared to L2b+ correctness. This protocol deliberately mimics deployment:
a user may see a model's confidence even when no canonical estimator is
available.
\paragraph{Controls and claim boundaries.}
The benchmark uses a multi-stage evaluation pipeline rather than an
autonomous causal scientist. Separate stages construct contexts, audit
leakage, collect model responses, extract coefficients, aggregate labels,
and elicit calibration, reducing ``task writer = respondent = grader''
circularity. The synthetic scenarios are not free-form LLM inventions:
mathematical templates, treatment-effect signs, sample sizes, seeds,
realised CSV files, and canonical estimators are fixed by the benchmark.
Natural-language descriptions can vary, but a model cannot pass L2b+ by
guessing the sign or recognizing a phrase; it must produce code whose
estimated coefficient agrees with the realised-data reference. These
components support different claims: Exp~A is realistic but text-derived,
Exp~B is executable but synthetic, and calibration measures
confidence--correctness alignment. Exp~A and Exp~B are not two components
of a single combined leaderboard: Exp~A supplies realistic contexts in
which objective numerical verification is generally unavailable, Exp~B
supplies controlled tasks in which correctness is directly observable, and
the disagreement between their evaluation signals is part of the finding
rather than a reason to aggregate them.
\begin{table}[t]
\centering
\caption{Benchmark controls. Each component supports a different claim
and protects against a different failure mode.}
\label{tab:controls}
\small
\setlength{\tabcolsep}{3pt}
\begin{tabularx}{\textwidth}{p{0.18\textwidth}p{0.40\textwidth}X}
\toprule
\textbf{Component} & \textbf{Implementation} & \textbf{Claim supported} \\
\midrule
Real context & 259 papers; reconstructed RQ/DD/IC; consensus reference
labels & Text-level method and direction agreement \\
Executable truth & 100 fixed-seed DGPs; realised CSVs; canonical
estimators & Numerical coefficient correctness \\
Role separation & Separate stages for construction, response, labeling,
extraction, calibration & Reduces circular grading \\
Leakage control & Method-name blacklist and prompt leakage audit &
Prevents trivial label recovery \\
Scorer audit & Coefficient extraction judge plus human validation audit
& Reduces brittle post-processing errors \\
Claim boundaries & Exp~A = agreement; Exp~B = execution; calibration =
confidence alignment & Prevents overclaiming \\
\bottomrule
\end{tabularx}
\end{table}
\section{Evaluation Layers}
The benchmark uses six scoring labels, but they apply to different
regimes. L1--L2b+ are Exp~B workflow-implementation layers, while L3/L4
are Exp~A text-agreement diagnostics. L1 checks that an output exists.
L2a checks for a parseable R code block. L2b executes the code with
\texttt{Rscript} in the fixed evaluation environment. L2b+ is the
execution-grounded correctness layer: it checks whether the extracted
treatment-effect estimate matches the realised-data canonical estimate
within the default relative-error tolerance. L3 measures method-family
agreement and L4 measures direction agreement against consensus reference
labels. Only L2b+ is a numerical correctness layer; L1/L2a are
bookkeeping diagnostics, L2b is execution, and L3/L4 are text agreement.
\paragraph{Why L2b+ uses a canonical estimator.}
The benchmark does not ask models to recover an ideal population
parameter directly. In finite samples, even a textbook estimator may
differ from the DGP parameter. We therefore compare model output to a
canonical estimator applied to the same realised dataset. For scalar
extracted estimate $\hat{\beta}_m$ and realised-data canonical estimate
$\hat{\beta}_c$, L2b+ passes when
\[
\frac{|\hat{\beta}_m-\hat{\beta}_c|}{|\hat{\beta}_c|} \le \tau ,
\]
with default tolerance $\tau=0.50$. Scenarios with
$|\hat{\beta}_c|<10^{-9}$ are marked unscoreable rather than divided. For
Event Study scenarios, accepted post-event windows and scale conversions
are frozen before scoring; valid cumulative post-event estimates are
accepted only when they can be deterministically converted to the
canonical per-day abnormal-return scale. This mapping is fixed before
model comparison.
Appendix~\ref{app:canonical_estimators} lists the canonical reference
estimate and accepted reported coefficient for each Exp~B method family.
L2b+ is therefore best read as agreement with a benchmark-fixed reference
implementation, not as a verdict on causal-econometric correctness in
general; a model that selects a methodologically more appropriate
estimator (e.g.\ Callaway--Sant'Anna~\citep{callaway2021did} or
Sun--Abraham~\citep{sun2021eventstudy}) may diverge from the canonical
reference, and we treat such divergence as benchmark coverage limitation
rather than model error.
All 30 Exp~B DID scenarios use common adoption timing, in which
Callaway--Sant'Anna and Sun--Abraham reduce to TWFE; Sun--Abraham
implementations on three sampled scenarios pass L2b+ at 50\% tolerance
with relative errors below 32\%
(\texttt{audit/rebuttal\_kit/cs\_sa\_estimator\_check.csv}), confirming
that the TWFE canonical reference is appropriate for this design space.
\paragraph{Why L3/L4 are supporting metrics.}
Exp~A institutional context necessarily describes identifying variation,
so method family can often be inferred by a trained reader. We therefore
treat L3/L4 as agreement diagnostics, not verified causal correctness.
They are useful diagnostics of how models parse real research contexts, but the
load-bearing correctness evidence comes from Exp~B.
\paragraph{Scorer validation.}
A preliminary 30-case audit of L2b-pass/L2b+-fail examples found that
regex extraction produced many scorer-side failures. The final scorer
therefore uses a separate coefficient-extraction judge to identify the
reported treatment-effect estimate from executed stdout, followed by
deterministic numerical comparison against the canonical estimate. The
extraction judge sees the executed code, stdout, scenario context, and
method family, but not the evaluated model identity, canonical estimate,
or L2b+ label. A completed 50-cell blinded human audit of this extraction
instrument finds 90.9\% numeric agreement among human-scoreable
effect-present cells and 88.6\% agreement on the induced L2b+ pass/fail
decision; Appendix~\ref{app:l2b-judge-human} gives denominators and
disagreement cases. The judge can still err, so we treat L2b+ as an
audited execution-grounded metric rather than an infallible oracle.
\paragraph{Threshold choice.}
The default L2b+ tolerance is relative error at most 50\%. This
threshold is intentionally broad: it allows finite-sample and
implementation variation while still rejecting estimates that are off by
a factor large enough to change practical interpretation. The binary
threshold is a coarse workflow-success endpoint; the relative-error
profiles and tolerance sweep test whether conclusions depend on a single
cutoff.
\section{Results}
The results follow the same separation as the benchmark design. Exp~A first
shows what real-paper text agreement can and cannot tell us. Exp~B then
tests the executable claim: whether the model-written workflow recovers the
canonical estimate. Finally, calibration asks whether the models themselves
can identify when that workflow is wrong under a retrospective self-report
prompt.
\begin{figure}[t]
\centering
\includegraphics[width=\textwidth]{fig2_l2b_plus_cascade.pdf}
\caption{Headline result. \textbf{(a)} L2b execution and L2b+ coefficient
correctness are distinct. \textbf{(b)} L2b ranking tracks L2b+ correctness
(Kendall $\tau=+0.81$), while L4 agreement against consensus direction
labels has weak association with Exp~B L2b+ ranking
($\tau\in[-0.20,+0.10]$). \textbf{(c)} Text-agreement diagnostics: Exp~A L3
(method family) and L4 (direction) agreement with four-LLM consensus
labels across the seven primary models. These measure agreement with
reference labels that carry label noise
(Section~\ref{sec:limitations}), not executable correctness. Throughout the paper, model markers follow a fixed
(color, shape) mapping: Opus $\bullet$, Sonnet $\blacktriangle$, GPT-4o
$\blacksquare$, GPT-5 $\bigstar$, o3 $\blacklozenge$, Kimi
$\boldsymbol{+}$, Gemini $\boldsymbol{\times}$.}
\label{fig:headline}
\end{figure}
\paragraph{Finding 1: real-paper text agreement is useful but not decisive.}
On Exp~A, L3 is scoreable for 187/259 papers and L4 for 92/259 papers
(unscoreable cases are predominantly papers where the 4-LLM consensus
did not converge on a single canonical method-family label or
positive/negative direction label).
GPT-4o leads L3 (88.8\%) and Sonnet leads L4 (85.9\%); the rankings
do not collapse to one textual ability score.
The 30-paper blinded human ambiguity audit reinforces this caution:
agreement with the 4-LLM consensus is only fair for direction labels
(Cohen's $\kappa = 0.294$), so L4 scores partly reflect reference-label
noise rather than purely model behavior.
Figure~\ref{fig:headline}(c) summarizes the L3/L4 separation;
Figure~\ref{fig:expa_text} breaks L3 down by method family, and
Event Study has the lowest observed cross-model L3 mean in this sample
with smaller corpus size limiting precision. Exp~A is therefore best
read as a real-context text diagnostic, not as the paper's primary
correctness endpoint.
\begin{figure}[t]
\centering
\includegraphics[width=0.90\textwidth]{fig3_method_dotplot.pdf}
\caption{Text-agreement diagnostic: Exp~A L3 method-family agreement by
method family. Colored markers
show the seven model-level agreement rates; black diamonds and whiskers show
the cross-model mean with paper-cluster bootstrap 95\% intervals.
Event Study has the lowest observed mean (64\%, $n=13$); with $n=13$
papers the cross-model 95\% bootstrap CI is wide and Event Study
agreement should be treated as exploratory rather than as a comparative
finding. L3 is a text-agreement diagnostic, not an executable
correctness endpoint.}
\label{fig:expa_text}
\end{figure}
\paragraph{Finding 2: runnable code is not necessarily correct.}
Figure~\ref{fig:headline} shows that L2b execution and L2b+ correctness
separate the model panel. Final ES-aware canonical L2b+
rates span 10--88\% across the seven primary commercial models: Opus
88\%, GPT-5 72\%, GPT-4o 62\%, Sonnet 50\%, o3 46\%, Gemini 32\%, and
Kimi 10\%. This spread is invisible if evaluation stops at whether the
model wrote code. Of the 426 workflows that execute, 66 (15.5\%) return a
wrong estimate, and Appendix~\ref{app:tolerance_sweep} shows that this
share stays between 14.1\% and 27.2\% across the 10\%--100\% tolerance
sweep.
Appendix~\ref{app:failure_taxonomy} decomposes Exp~B failures into execution,
reporting, target-coefficient, event-window, IV-stage, and RDD cutoff/sign
categories.
\begin{figure}[t]
\centering
\includegraphics[width=0.90\textwidth]{fig4_cascade.pdf}
\caption{Execution cascade on Exp~B. Each bar partitions 100 scenarios for one
model
into L2b+ correct estimates, executed-but-wrong coefficients, execution
failures, and missing or unparseable code. The decomposition shows why
runnable code alone is not sufficient.}
\label{fig:cascade}
\end{figure}
\paragraph{Finding 3: execution ranking tracks reference-agreement correctness.}
Across the seven models, the L2b ranking strongly matches the final L2b+
ranking (Kendall $\tau=+0.81$, scenario-clustered 95\% bootstrap CI
$[0.62, 0.90]$; Spearman $\rho=+0.93$, $[0.75, 0.96]$; 1{,}000 resamples,
seed 20260507; every replicate exceeds the L4-vs-L2b+ upper bound $+0.10$).
Appendix~\ref{app:rank_stability} reports leave-one-model-out stability:
Kendall $\tau$ remains in $[0.733,0.867]$, with $\tau=0.714$ when adding
Llama as robustness-only. Appendix~\ref{app:lodo} reports
leave-one-design-out stability: removing any one of the four design
families leaves the L2b+ ranking within Kendall $\tau\in[0.81,1.00]$ of
the full panel, with the top-two and bottom-two model sets unchanged. By contrast, L4 agreement against consensus
direction labels has weak or negative association with Exp~B L2b+
ranking ($\tau\in[-0.20,+0.10]$ across two frozen scorer variants).
Execution is not sufficient, but it is a much stronger proxy for
reference-agreement correctness than text-level direction.
Appendix~\ref{app:expb-robustness} reports
$P(\mathrm{L2b+}\mid\mathrm{L2b})$ to separate execution failures from
executed-but-wrong estimates.
\paragraph{Finding 4: L2b+ rankings are not threshold artifacts.}
The L2b+ relative-error distributions show that stronger models
concentrate executed estimates near zero, while weaker models accumulate
large-error tails. Thus the ranking is better understood as a
distributional difference in numerical accuracy rather than an artifact
of the default 50\% cutoff. Appendix~\ref{app:additional-diagnostics}
reports the full relative-error profiles and tolerance sweep.
Appendix~\ref{app:tolerance_sweep} reports 10\%--100\% tolerance sweeps;
stricter cutoffs lower absolute pass rates but preserve the main
text-versus-execution separation.
\paragraph{Finding 5: naive retrospective confidence is a weak triage signal.}
Calibration asks each model to rate its own previous Exp~B output
without seeing the L2b+ pass/fail label. Actual L2b+ rates vary sharply, but
confidence gaps remain small for the main models. GPT-5 illustrates
the asymmetry: it is second on L2b+ but has a slightly negative
confidence gap. Under our retrospective confidence prompt, the
self-assessment signal therefore does not provide a reliable deployment
triage signal; this does not rule out stronger calibration interfaces.
Appendix~\ref{app:additional-diagnostics} reports reliability diagrams
for the retrospective confidence protocol.
\begin{table}[t]
\centering
\caption{Calibration summary against final L2b+ v2 correctness. Gap is
mean numerical confidence on correct outputs minus mean confidence on
incorrect outputs. Small gaps indicate weak triage value.}
\label{tab:calibration}
\small
\begin{tabular}{lccc}
\toprule
\textbf{Model} & \textbf{n} & \textbf{L2b+ v2} & \textbf{Conf. gap} \\
\midrule
Opus & 100 & 88.0 & +0.075 \\
GPT-5 & 99 & 71.7 & -0.045 \\
GPT-4o & 100 & 62.0 & +0.016 \\
Sonnet & 99 & 49.5 & -0.011 \\
o3 & 100 & 46.0 & -0.010 \\
Gemini & 48 & 31.2 & +0.234 \\
Kimi & 100 & 10.0 & +0.027 \\
\bottomrule
\end{tabular}
\end{table}
\begin{figure}[t]
\centering
\includegraphics[width=0.90\textwidth]{calibration_gap_scatter.pdf}
\caption{Correctness versus self-assessment. Each point is one model.
The horizontal axis is execution-grounded L2b+ correctness; the vertical
axis is the mean numerical-confidence gap between correct and incorrect
workflows. High L2b+ accuracy does not imply a reliable confidence signal.}
\label{fig:calibration-gap}
\end{figure}
\paragraph{Robustness check: the execution--correctness gap reproduces for an open-weights model.}
As a single cross-vendor check, we evaluate Llama-3.3-70B-Instruct on the
same 100 Exp~B scenarios using the same prompt and scoring pipeline. It
reaches L2b $=41\%$ and final ES-aware L2b+ $=20\%$, placing it between
Kimi and Gemini and reproducing the same qualitative gap between runnable
code and numerically correct code. Adding Llama as an eighth model
lowers Kendall $\tau$ to $0.714$ (Appendix~\ref{app:rank_stability})
without changing the qualitative L2b--L2b+ ordering or L2b--L4
separation. We treat this as a robustness check, not a complete
benchmark of the open-source ecosystem.
\section{Reproducibility and Human Audits}
This build is tied to a frozen audit state. The released
artifacts include reconstructed Exp~A paper fields, four-LLM consensus
reference labels, 1813 Exp~A outputs, 700 primary Exp~B model-scenario
execution records, 100 realised CSV datasets with canonical-estimator
outputs, 646 calibration records, prompt hashes, scorer decision records,
and cached model responses. Deterministic scoring can be rerun without
new LLM calls; because commercial APIs are not bit-reproducible, the
release treats cached JSON outputs as the reproducible model-output layer.
Appendix~\ref{app:reproducibility} lists artifact paths and checksums.
Human annotation is intentionally an ambiguity audit rather than a
training or tuning input. A blinded 30-paper Exp~A slice was labeled
from the original PDFs after the automated benchmark freeze without
access to Phase-2 LLM votes, consensus labels, or model outputs, and
these human labels are not used to change frozen consensus labels,
prompts, outputs, or scoring thresholds. Agreement with the four-LLM
consensus is method 60.0\% / direction 47.6\% (Cohen's
$\kappa = 0.606$ and $0.294$), supporting our interpretation of Exp~A
as a real-context agreement diagnostic with substantial ambiguity;
Exp~B supplies executable correctness.
A separate blinded human audit validates the coefficient-extraction
instrument used for L2b+. The completed 50-cell audit finds 90.9\%
numeric agreement among human-scoreable effect-present cells and 88.6\%
agreement on the induced L2b+ pass/fail decision; Appendix~\ref{app:l2b-judge-human}
gives denominators and disagreement cases. This audit validates the
measurement instrument but does not change the frozen headline L2b+
rates.
\section{Limitations}
\label{sec:limitations}
First, Exp~A labels are reference labels rather than L2b+ correctness
labels: the final labels come from a four-LLM consensus, and the
completed blinded 30-paper ambiguity audit agrees with consensus on
60.0\% of method labels and 47.6\% of direction labels, with Cohen's
$\kappa = 0.294$ for direction. The audit is an ambiguity check, not a
full human re-adjudication of the 259-paper corpus. Stratified by consensus
strength, human agreement with the consensus method label is 100\% where
all four models agree (4/4) and where three of four agree (5/5), 56\%
under bare-plurality consensus (9/16), and 0\% on ties and splits (0/5);
among the 25 audited papers with both method labels available, agreement
is 72\% ($\kappa = 0.61$), against 47.6\% ($\kappa = 0.29$) for direction
on the 21 comparable papers. Disagreement concentrates in the
low-consensus strata, so consensus strength flags cases that warrant human
review; it does not make the labels uniformly reliable, and the audit
cannot support a re-ranking of models. The consensus pool
also overlaps with the evaluated panel (two exact matches, two
same-provider siblings; see Section~\ref{sec:benchmark}), so L3/L4 are agreement diagnostics
under structural overlap rather than independent text-correctness
measurements; L2b+ in Exp~B does not use this pool. Because Exp~A uses
published papers, we also cannot fully rule out model familiarity with
some studies from pretraining or public metadata.
Second, Exp~B is synthetic and structured. This is necessary for
executable reference estimates, but it measures whether models can
implement textbook workflows on known CSV data, not whether they can conduct an
entire empirical research project. The DGPs cover four major method
families but cannot represent all design choices in applied economics
and finance. In particular, Exp~B does not test literature review, data
cleaning, design selection from ambiguous field constraints, or
iterative robustness analysis. Matching, propensity-score, doubly-robust,
mediation, and heterogeneous-effect workflows are outside the benchmark
and are future work; adding a further design family would not make the
benchmark a measure of general causal inference, and we do not claim one.
Third, Exp~B fixes R as the execution backend, held constant so that the
package ecosystem and canonical implementations are identical across
models. L2b+ therefore measures executable workflow reliability under this
released backend rather than language-invariant causal ability, and the
reported magnitudes are backend-specific. A 30-scenario Python-backend
pilot run before submission (Opus, GPT-5, o3; not part of the released v1
artifact) also produced executed-but-wrong estimates, so the failure mode
is not specific to R, but that pilot supports only the existence of the
failure mode under a second backend, not its magnitude; backend sensitivity
across Python, Stata, Julia, or other environments remains an important
extension.
Fourth, calibration is retrospective and prompt-specific. The protocol
tests whether models can assess their own prior outputs under our
confidence-elicitation prompt, not whether all possible uncertainty
interfaces would fail. Calibration coverage is also incomplete for some
models. The negative result should be read as evidence that naive
retrospective self-reported confidence is not enough, not as proof that no
calibrated interface can be designed.
Fifth, the evaluated model panel is limited. The primary panel consists
of closed or hosted frontier systems, which is appropriate for the
deployment question that motivates the paper but does not characterize
the full open-source ecosystem. We add one open-weights model as a
cross-vendor robustness check; a complete benchmark of multiple Llama,
Qwen, Mistral, DeepSeek, and domain-specific systems is left for future
work. Model-level rank correlations should also be read as descriptive
diagnostics over this evaluated panel, not population-level estimates
over all possible LLMs, and conclusions about open-weight systems rest on
one robustness check rather than on coverage of that ecosystem.
Sixth, L2b+ is an audited measurement instrument rather than an oracle.
It depends on the canonical specification, the relative-error tolerance,
event-study scale conversions, and coefficient extraction from executed
stdout. We mitigate these risks through frozen scorer rules, decision
records, and a completed coefficient-extraction human audit, but
individual-cell mistakes remain possible and should be treated as part
of the benchmark's measurement uncertainty.
Finally, the benchmark is intentionally single-shot and non-interactive.
Models receive fixed prompts and data previews, and the benchmark scores
the returned workflow rather than allowing extended human-in-the-loop
debugging or iterative robustness analysis. This design makes evaluation
reproducible and comparable across models, but it does not measure how a
supervised analyst might use an LLM over multiple rounds to improve a
causal study. The scores are therefore estimates of standardized one-shot
workflow reliability, not of upper-bound interactive performance, and the
benchmark's pass/fail signal is an oracle that is not available in real
use.
\section{Conclusion}
\textsc{CausalVerify} shows that, for structured econometric
causal-estimation workflows, the right unit of evaluation is the executed
workflow: a model may produce
plausible causal text and runnable code while still estimating the
wrong causal quantity. Across seven evaluated LLMs, final L2b+ pass
rates range from 10\% to 88\%; execution ranking tracks
reference-agreement much better than L4 direction agreement, and
naive retrospective confidence does not reliably separate correct
from incorrect workflows. The question for causal inference is
therefore not whether the model sounds right, but whether its
workflow executes, agrees with a specified target, and carries
audited uncertainty. These conclusions are confined to the four design
families, the R backend, and the single-shot protocol evaluated here.
Within that scope the broader evaluative lesson stands: successful
execution is not itself evidence of scientific correctness, and evaluation
of quantitative scientific agents should validate the target quantity
rather than rely on a proxy for producing it.
\phantomsection
\label{sec:references-start}
\bibliographystyle{plainnat}
\bibliography{references}
\clearpage