Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
18,148 characters · 7 sections · 20 citation commands
An Auditable AI Agent Loop for Empirical Economics: A Case Study in Forecast Combination $~$
\doublespacing
\noindentKeywords: Agent loops, Autoresearch, Specification search, Researcher degrees of freedom, Responsible AI, Forecast combination\\ \noindentJEL Codes: C53, C52, C18
Empirical researchers have always explored alternative specifications. AI makes that process cheap and fast. An AI coding agent can rewrite an estimation script, run a fixed evaluator, observe the score, and try again hundreds of times in one session. Once that loop is in place, the practical boundary between coding assistance and undisclosed specification search becomes thin. For a field long concerned with researcher degrees of freedom Leamer1983,White2000,GelmanLoken2013,Miguel2021, the central question is therefore not whether AI agents can improve an empirical score, but whether their search can be made transparent and auditable.
This paper studies one response. Building on Karpathy2026's (Karpathy2026) open-source autoresearch, I adapt a simple agentic coding loop to a research protocol for empirical economics. The protocol has four ingredients: a written instruction contract, an immutable evaluator, a single editable script, and a complete experiment log. I then add an explicit post-search holdout evaluation. Together, they do not eliminate adaptive search, but they make it inspectable, bounded, and easier to disclose, while also making validation beyond the search sample explicit. The point is therefore not a new agent architecture, but a way to use an existing one in empirical work with clearer external validation.
The empirical illustration uses the forecast-combination problem of DieboldShin2019. An AI coding agent is asked to find a look-ahead-free real-time tuning rule for peLASSO while leaving the evaluator fixed. On the original rolling evaluation, three independent runs improve on the simple average and on the paper's ex post peLASSO benchmark. A post-search holdout for 2017Q1--2025Q4 then provides the key validation: two discovered methods continue to outperform standard benchmarks, while one does not.
That split result is the main message. The experiment logs show that nominal hyperparameter tuning readily drifts into broader method discovery. Some departures survive external evaluation; others do not. The value of the framework is therefore twofold: the underlying loop records how the agent searched, and the added holdout evaluation allows the resulting departures to be judged rather than hidden. For empirical economics, that is the relevant promise of agentic search: not automation without discretion, but automation with an audit trail and explicit post-search holdout evaluation.
Recent work on AI for science spans a wide range of autonomy. On one end, systems such as AlphaEvolve NovikovEtAl2025 and DS-STAR ChenEtAl2025 use large language models to iteratively rewrite code and refine analyses against automated evaluators. On the other end, systems such as The AI Scientist LuEtAl2024 and AI co-scientist GottweisEtAl2025 pursue broader research automation across idea generation, experiment design, analysis, and communication. In economics, Korinek2025 surveys AI agents for research, while DawidEtAl2025 develop broader agentic workflows. Our contribution is narrower: we study evaluator-locked local search within a fixed empirical workflow, where the researcher sets the question, data, evaluator, editable surface, and search budget. The goal is not full automation, but an auditable protocol for standard empirical practice.
The remainder of the paper describes the protocol (Section (ref)), presents the empirical illustration (Section (ref)), and concludes (Section (ref)).
Consider a researcher who starts with a baseline empirical model and wants to explore nearby alternatives using an AI coding agent.\footnote{By AI coding agent we mean a general-purpose coding assistant such as Claude Code or OpenAI Codex that plans, writes code, and executes tool calls within a development environment.} We formalize the search as follows. Let $C$ denote a written instruction contract that specifies the research objective, admissible modifications, and search budget. Let $\mathcal{T}(C)$ denote the class of admissible editable scripts under $C$. Let the available data be partitioned as $D = (D^{S}, D^{H})$, where $D^{S}$ is the search sample and $D^{H}$ is a post-search holdout reserved for external validation. Let $S(\tau; D^{S})$ denote the scalar score returned by an immutable evaluator when script $\tau \in \mathcal{T}(C)$ is run on search sample $D^{S}$. Both $\mathcal{T}(C)$ and $S$ are fixed ex ante by the researcher as part of the search design. In practice, the contract fixes the evaluator, editable surface, and no-look-ahead constraint, but it need not impose a machine-checkable restriction that candidate scripts belong to a particular model family; the effective admissible set $\mathcal{T}(C)$ may therefore be broader than the researcher's semantic intent. A complete experiment log $L$ records every attempted modification and its outcome. These four elements---contract, evaluator, editable script, and log---constitute the protocol.
When lower scores are preferred, the search problem is \[ \tau^{\star} \in \arg\min_{\tau \in \mathcal{T}(C)} S(\tau; D^{S}). \] In practice, the agent sequentially generates and evaluates a finite sequence of candidate scripts $\{\tau_{0}, \tau_{1}, \ldots, \tau_K\}$, each informed by the outcomes of previous attempts, and returns \[ \hat{\tau}_{K} \in \arg \min_{\tau \in \{\tau_{0}, \tau_{1}, \ldots, \tau_{K} \}} S(\tau; D^{S}). \] This distinction matters. $\tau^{\star}$ is the best admissible script in the full class $\mathcal{T}(C)$. $\hat{\tau}_K$ is the best script found by a particular bounded search. The audit log records the outcome of every experiment as an ordered sequence $L_K = \bigl((k, \tau_k, s_k, d_k)\bigr)_{k=0}^{K}$, where $s_k = S(\tau_k; D^{S})$ when the run succeeds, $s_k = +\infty$ when it crashes, and $d_k \in \{\textit{keep},\, \textit{discard},\, \textit{crash}\}$ records the outcome status. Algorithm (ref) summarizes the protocol.
The protocol does not impose a model of how the sequence of candidates is generated; that depends on the underlying large language model and the coding environment. Because the agent observes what it has already tried, the search is path dependent and stochastic: independent runs from the same starting point will generally follow different paths and may reach different solutions. The search is adaptive but bounded. The agent searches freely within the editable script, but the evaluator, search sample, and scoring rule are fixed ex ante. This separation between immutable search design and editable implementation is analogous to a pre-analysis plan, except that the audit trail is generated automatically and covers every specification the agent considered. Implementation details, including the instruction contract, are in the Appendix.
Once the search is complete, the researcher may evaluate the selected script $\hat{\tau}_K$ on the holdout $D^{H}$. Let $S^{H}(\tau; D^{H})$ denote a scoring function computed on the holdout. The realized holdout score $S^{H}(\hat{\tau}_K; D^{H})$ provides an external check on whether improvements discovered during adaptive search on $D^{S}$ generalize beyond the search sample. Because $D^{H}$ need not even exist during the search, the holdout evaluation is a researcher action, not part of the agentic loop.
We illustrate the protocol using the real-time tuning problem in DieboldShin2019's (DieboldShin2019) peLASSO forecast combination. peLASSO first uses LASSO to select a subset of forecasters from the Survey of Professional Forecasters and then shrinks the surviving weights toward equality using an egalitarian penalty. In the two-step implementation, two regularization parameters must be chosen at each forecast origin using only information available at that time. DieboldShin2019 evaluate the procedure with ex post optimal parameters and do not provide a data-based rule for real-time selection. Instead, they propose other methods such as subset averaging, inspired by the ex post analysis of peLASSO, that perform well on their evaluation sample. The present application takes the tuning problem of peLASSO, which was left unresolved, as the object of agentic search.
The instruction contract asks the agent to find a look-ahead-free lambda selection rule for peLASSO, editing only a single forecasting script (train.R). The immutable evaluator (prepare.R) implements rolling-origin RMSE evaluation under the original DieboldShin2019 design: at each date $t$, only data through $t-1$ is available, and the score is the RMSE over the full evaluation period. We conduct three independent runs, each starting from the simple average as the initial script and an approximate budget of $K = 200$ experiments.
We also evaluate the discovered methods on a post-search holdout (2017Q1--2025Q4, 36 quarters) that was withheld from the evaluator during the agentic search. As additional benchmarks, we include two methods from DieboldShin2019: best $\leq N_{\max}$-average ($N_{\max} = 6$, their Table 4) and best $(\leq N_{\max}, \leq W_{\max})$-average (their Table 5), both of which exhaustively search over forecaster subsets at each forecast origin.
Table (ref) reports RMSE for the search sample (1999Q3--2016Q4) and the holdout (2017Q1--2025Q4). On the search sample, all three agent runs beat both the simple average and the peLASSO ex post benchmark: Run 1 achieves a relative RMSE of 0.858, Run 3 reaches 0.808, and Run 2 achieves 0.510. The two benchmark methods from the original paper also outperform the simple average, with relative RMSEs of 0.954 (best $\leq N_{\max}$-average) and 0.916 (best $(\leq N_{\max}, \leq W_{\max})$-average).
On the holdout, Run 2 remains the strongest method, with a relative RMSE of 0.811 (0.739 excluding the COVID quarters of 2020). Run 1 also holds up, at 0.945 (0.861 excluding COVID). The two benchmark methods from the original paper generalize modestly, each at 0.974. In contrast, Run 3 (1.089) performs worse than the simple average on the holdout, and peLASSO ex post shows essentially no improvement (0.995). The holdout therefore shows that agentic search can uncover methods that generalize, but it also reveals that some discovered improvements are sample specific.
The instruction contract asks the agent to tune regularization parameters for peLASSO. The experiment logs\footnote{Appendix (ref) reproduces an excerpt from the experiment log of Run 3, and Appendix (ref) gives a detailed description of the method discovered by each run.}, however, reveal that two of the three runs drifted beyond that task: Run 1 toward stability selection and performance weighting, and Run 2 toward a ranking-based method with bias correction. Run 3 stayed closest to the original task, using adaptive LASSO with forward cross validation for lambda selection, though it added an egalitarian elastic net blend. The holdout makes these departures evaluable. Run 2's bias correction, which adjusts for recent forecast bias, turns out to generalize and accounts for most of its holdout gain. Run 3's egalitarian elastic net blend, which drove much of its search-sample improvement, degrades holdout performance. The protocol does not prevent the agent from leaving the intended design space; it ensures that every departure is recorded, so that a holdout or other external evidence can later distinguish structural insight from sample adaptation.
Agent loops can widen researcher degrees of freedom, but they need not do so invisibly. This paper adapts a minimal agent loop architecture to an auditable protocol for agentic specification search and adds a post-search holdout evaluation. In the forecast combination illustration based on DieboldShin2019, three independent runs improve on the search sample, but a genuine holdout shows that only two of the three discovered methods generalize. The contribution of the framework is not to eliminate adaptive search, but to make it visible, bounded, and evaluable. Because the protocol disciplines the search process rather than the subject matter, it applies wherever automated specification search is used. Appendix (ref) sketches how the same architecture maps to regression specification search, structural VAR identification, and other standard empirical tasks.
Several limitations deserve emphasis. First, a language model may encode background knowledge that overlaps with the holdout period, so residual look-ahead concerns remain. Second, the model's internal trade-offs between score improvement and contract boundaries are opaque; it is often unclear how those objectives are balanced. Third, the effective search class may recombine existing ideas more readily than it generates novel ones. Fourth, the current search over $\mathcal{T}(C)$ is greedy: propose one candidate, evaluate it, then keep or discard it. More structured strategies, such as the evolutionary population-based search in AlphaEvolve NovikovEtAl2025 or tree search in AygunEtAl2025, could explore the space more efficiently. Fifth, the protocol is a practical guardrail against data-adaptive search, but it is not a substitute for formal post-selection inference or other corrections for p-hacking.