Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
65,637 characters · 17 sections · 52 citation commands
Can AI Master Econometrics? Evidence from Econometrics AI Agent on Expert-Level Tasks
\setlist{noitemsep} \onehalfspacing
\affil[a]{School of Economics, Shandong University} \affil[b]{HKU Business School, The University of Hong Kong} \affil[c]{Tsinghua University} \affil[d]{School of Engineering, Hong Kong University of Science and Technology}
\thispagestyle{empty}
\centerline{\bf ABSTRACT}
Can AI effectively perform complex econometric analysis traditionally requiring human expertise? This paper evaluates AI agents' capability to master econometrics, focusing on empirical analysis performance. We develop “MetricsAI”, an Econometrics AI Agent built on the open-source MetaGPT framework. This agent exhibits outstanding performance in: (1) planning econometric tasks strategically, (2) generating and executing code, (3) employing error-based reflection for improved robustness, and (4) allowing iterative refinement through multi-round conversations. We construct two datasets from academic coursework materials and published research papers to evaluate performance against real-world challenges. Comparative testing shows our domain-specialized AI agent significantly outperforms both benchmark large language models (LLMs) and general-purpose AI agents. This work establishes a testbed for exploring AI's impact on social science research and enables cost-effective integration of domain expertise, making advanced econometric methods accessible to users with minimal coding skills. Furthermore, our AI agent enhances research reproducibility and offers promising pedagogical applications for econometrics teaching.
{\bf Keywords}: Econometrics, Empirical Replication, AI Adoption, Agentic AI, Expert System
{\bf JEL classification:} A20, B41, C49, C87
The rapid advancement of artificial intelligence (AI), particularly through large language models (LLMs) like ChatGPT (OpenAI:2024), has sparked widespread discussion regarding its potential to enhance productivity, decision-making, and efficiency across diverse sectors. In recent years, these developments have gained notable traction in academia, especially in fields such as economics, business, and social sciences. Several studies (Li:2024; Chakraborty:2025) have already demonstrated that LLM-based tools can effectively perform tasks traditionally reserved for human participants, such as perceptual analysis and personnel selection, with remarkable accuracy and minimal professional intervention.
However, despite these promising advances, the adoption of AI for domain-specific analytical tasks remains limited, particularly in areas requiring complex econometric reasoning. Current LLM applications tend to focus on relatively simple, language-based tasks or the generation of basic numerical summaries from unstructured text. Scholars such as Korinek:2023 and Xu:2024 have highlighted persistent limitations in LLMs' ability to execute multi-step, logically constrained analytical workflows, especially when domain-specific methodological knowledge is required. Ling:2024 further pointed out that fine-tuning LLMs for specialized fields like econometrics incurs substantial costs and struggles with maintaining pace with rapidly evolving academic frontiers.
From a broader economic perspective, the barriers to acquiring econometric and applied statistical skills present significant productivity losses and restrict the democratization of evidence-based research capabilities. In the absence of automation and AI support, performing rigorous empirical research typically demands years of specialized education, fluency in statistical programming, and deep familiarity with econometric theory and diagnostic conventions. This creates high entry barriers, disproportionately excluding students from under-resourced regions, non-English-speaking countries, and disciplines where quantitative literacy is not traditionally emphasized.
Economic theory predicts that lowering the cost of learning applied statistics and econometrics generates substantial productivity and welfare gains. In a simple production function model of knowledge accumulation, reducing the skill acquisition cost raises the equilibrium number of agents engaging in empirical analysis and policy-relevant research, thereby generating additional societal value. In particular, democratizing access to rigorous empirical tools is crucial for bridging global educational inequalities, especially in developing economies lacking experienced academic instructors in econometrics and applied quantitative research.
Building upon these motivations, this paper introduces MetricsAI (Econometrics AI Agent) — a specialized AI agent designed to perform applied econometric analysis via interactive, no-code workflows. Developed on the open-source MetaGPT framework, MetricsAI overcomes traditional LLM limitations by integrating a bespoke econometric tool library, a zero-shot learning framework, and a multi-round conversational workflow. This agent strategically plans econometric tasks, automatically selects appropriate econometric models based on data types and research objectives, generates and executes Python code, and iteratively refines results through error-based reflection and user feedback loops.
This innovation arrives at a time when AI agent frameworks are rapidly transforming industrial applications but remain under-explored in supporting academic research. Major technology leaders have emphasized the imminent expansion of agentic workflows. Andrew Ng (Stanford University), speaking at the AI Ascent 2024 Conference, predicted that "the set of tasks AI could do will expand dramatically this year because of agentic workflows." Similarly, Jensen Huang, CEO of Nvidia, projected at the 2024 Gartner IT Symposium/Xpo that his company would one day employ 50,000 people alongside over 100 million AI assistants. These endorsements reflect a growing consensus in both academia and industry that AI agents will redefine complex task automation in the coming years.
While AI agents such as Amazon’s Alexa+ already manage household devices through conversational interfaces, their academic counterparts remain scarce. Existing AI research tools either focus on unstructured data summarization or rely on static, fine-tuned models ill-equipped for dynamic, iterative research processes. Our paper directly addresses this gap by designing an AI agent tailored for the intricate demands of applied econometrics, social science, and business research.
Technically, MetricsAI represents a substantial advance over conventional LLM workflows. Most contemporary LLMs struggle to manage multi-step empirical workflows involving data preprocessing, model selection, estimation, inference, robustness checks, and diagnostic interpretation. MetricsAI resolves these issues through three core innovations:
1. A custom econometric tool library implementing essential methods such as OLS, PanelOLS, IV-2SLS, DID, RDD, and Propensity Score techniques; 2. A zero-shot learning framework allowing the agent to integrate new econometric procedures without costly LLM retraining; 3. A multi-round interactive workflow enabling users to iteratively refine tasks through natural language queries, supported by contextual memory management and an error-based feedback mechanism.
To empirically validate MetricsAI’s performance, we conducted extensive tests using two real-world datasets: (i) doctoral-level econometrics coursework assignments and (ii) published empirical economics papers. The experimental results reveal substantial performance gains. While baseline LLMs such as GPT-4o achieved under 50% success rates in complex tasks, and general-purpose AI agents achieved partial replication rates around 30%, MetricsAI consistently attained directional replication rates exceeding 90%, with perfect replication exceeding 50% for coursework assignments and 27% for published paper replications.
Beyond technical performance, this paper examines the broader economic, educational, and labor market implications of AI-driven econometric analysis tools. First, MetricsAI lowers the skill threshold for empirical social science research. By providing a no-code, interactive analysis environment, it allows undergraduate students and non-specialist professionals to replicate published econometric studies, a task traditionally reserved for doctoral students or seasoned researchers. This democratization effect is expected to expand the population of competent empirical researchers, improving the reach, readership, and citation potential of academic work.
Second, MetricsAI addresses barriers posed by language and technical fluency. The agent processes instructions in natural language, including non-English languages when necessary, and generates multilingual explanations of statistical outputs. This feature is particularly valuable for researchers and students in non-English-speaking regions, where access to quantitative economics training remains limited. By bridging these divides, MetricsAI contributes to educational equality and fosters the dissemination of evidence-based policy knowledge in under-resourced countries.
Third, the system promotes rapid adoption of state-of-the-art econometric methodologies. Empirical researchers frequently struggle to implement new techniques due to steep learning curves and sparse training materials. MetricsAI’s modular, zero-shot architecture allows newly published econometric methods to be incorporated into the tool library without requiring LLM retraining. Researchers can immediately access these methods via natural language queries, enhancing research productivity and reducing the prevalence of outdated or misapplied models in empirical studies.
Fourth, the agent contributes to improved research reproducibility. Replicability remains a significant concern in applied economics and business research (Mueller-Langer et al., 2019). By standardizing analytical workflows, automating diagnostic checks, and generating machine-readable results, MetricsAI facilitates rigorous, transparent, and easily audited empirical work. Its iterative, conversational interface also allows users to revise model specifications and robustness checks interactively, a capability largely absent in existing AI research tools.
Fifth, the deployment of MetricsAI has labor market implications for economics and business schools. As econometric literacy becomes increasingly accessible, universities are likely to expand their course offerings in applied statistics, business analytics, and causal inference. This expansion is expected to raise demand for economics instructors specializing in empirical research methods, potentially increasing employment opportunities in academic and professional training institutions. Moreover, enhanced research capacity in developing countries may stimulate local labor markets for empirical researchers and policy analysts, contributing to regional human capital development.
Finally, by dramatically reducing the technical and linguistic barriers to applied empirical analysis, MetricsAI paves the way for quantitative reasoning to become a basic educational competency akin to high-school mathematics. In the long run, it may help position data literacy and causal reasoning as foundational components of undergraduate education across disciplines, from economics and business to sociology and political science.
MetricsAI’s extensibility further enhances its long-term value. The system’s zero-shot learning framework allows new econometric techniques and domain-specific modules to be seamlessly integrated through Python functions and internal prompts, without altering the core LLM. This modularity ensures the agent’s continued relevance in a rapidly evolving research environment. Its architecture could also be transferred to other specialized fields, such as macroeconomics, finance, and quantitative public health, enabling the efficient development of AI agents equipped with field-specific analytic tools.
In sum, this paper makes three major contributions. First, we introduce a highly capable, cost-effective AI agent for applied econometric analysis. Second, we empirically demonstrate its superior performance relative to LLM baselines and general-purpose AI agents. Third, we offer an economic and educational analysis of MetricsAI’s potential impact on labor markets, higher education, research productivity, and global educational equality. The agent is open-source and available for public use at \url{https://github.com/HKU-Business-AI-Center/Econometrics-Agent}.
The remainder of the paper is organized as follows. Section (ref) reviews the emerging literature on LLM and AI agent applications in business, economics, and social sciences, positioning our contributions within this context. Section (ref) details the MetricsAI agent’s structural design, focusing on its domain-specific tool integration and task decomposition capabilities. Section (ref) outlines our empirical evaluation methodology and presents comparative performance results against other LLM-based solutions. Section (ref) provides detailed case studies illustrating MetricsAI’s operation and advantages. Section (ref) discusses the broader economic implications and future applications, concluding with remarks on its potential to reshape applied social science research.
Our paper relates to the growing literature on LLM applications in business, economics, and social sciences, where LLMs primarily serve as tools for analyzing unstructured textual data, replacing traditional natural language processing techniques. Research has demonstrated LLMs' fundamental content creation capabilities in various settings. Stroube:2024 used GPT-4 to generate synthetic gender-neutral movie pitches to control for gender influence in quality evaluation. Yoganarasimhan:2024 applied GPT-3.5 Turbo to calculate polarization scores for articles and topics in social media analysis. Abraham:2024 utilized ChatGPT to generate ESG scores from PE firms' websites, improving upon dictionary-based measurements. Armstrong:2024 applied the GPT-3.5 Turbo to quantify the firm exposure in Securities Exchange Commission agencies but received insignificant enhancement. Niu:2024 verified the consistency of “bag-of-words" technique by GPT-4 when measuring the effect of manufacturers’ service offerings on demand variability. Noailly:2024 utilized EnvP-BERT fine-tuned with environmental policies to capture climate-related indexes, which obtained resembling accuracy. Apart from generating numeric data by analyzing the implication of the context, LLMs are additionally exerted in detecting the similarity by tokenizing context as vectors and afterward calculating the distances in different norms (Bursztyn:2022; Graeber:2024; Ahrens:2024; Gorodnichenko:2024; Curti:2023; Balsmeier:2024).
Most existing research focuses on direct applications of LLMs' language capabilities, primarily using simple prompts to extract information from textual data. Due to LLMs' limited capabilities in specific domains like econometrics, their application in these areas remains restricted. Our paper advances this literature by endowing LLMs with domain knowledge through the MetricsAI's framework. This enables social science researchers to efficiently implement advanced econometric methodologies, expanding LLMs' utility beyond basic content generation.
Specifically, there have been recent studies discussing potential revolutionary roles of LLMs in econometrics. From a theoretical perspective, Manning:2024 investigated LLMs' capability in social science research through automated structural causal model development and in silico simulations. Ludwig:2025 derived conditions under which LLMs can effectively conduct economic measurement prediction and estimation tasks. From an empirical perspective, Han:2025 developed a multi-step prompt engineering method to validate LLMs' capability in identifying valid IVs for causal discovery. Our paper contributes directly to LLMs' empirical application in econometrics, focusing on automating researchers' complete hands-on workflow. Moreover, thanks to its extensible workflow and tool package design, our MetricsAI's framework supports smooth integration of fast-evolving AI-driven econometrics methodologies from academia, while serving as a platform for deploying these research outcomes in real-world applications.
Our second contribution relates to the literature exploring AI's economic impact and human-AI interaction. Previous research has extensively studied productivity effects. Hui:2024 found that ChatGPT usage among freelancers reduces skill-based gaps in highly affected occupations, impacting both earnings and employment. Brynjolfsson:2024 demonstrated heterogeneous productivity gains among customer service workers using LLM-based conversational guidance systems. Similar productivity effects were found by Noy:2023 in professional writing and Dell:2023 in consulting, where AI augmentation improved performance within AI-capable tasks but showed limitations beyond the AI's capability frontier.
These productivity shifts have broader implications, as Meltzer:2024 discussed in analyzing potential job market changes across manufacturing and international trade, emphasizing the need for regulatory frameworks. Beyond productivity, Generative AI influences other economic behaviors. Leib:2023 found that AI-generated dishonesty-promoting suggestions affect human honesty similarly to human advice, while Doshi:2024 revealed that LLMs enhance individual creative writing but may reduce global novelty by channeling creativity in similar directions.
Research has also examined LLMs' decision-making patterns compared to humans. Goli:2024 found LLMs prefer lower discount rates and earlier rewards in intertemporal choices, with effects moderated by language features. Chen:2025 identified varying risk and certainty preferences across different LLM models in operations management decisions. Regarding occupational assessment, Gmyrek:2024 found GPT-4 generally aligns with human judgments, except for overestimating digital economy roles and underestimating non-conventional occupations.
While these studies explore LLMs' economic impact across various dimensions, their research designs focus on LLMs' current capabilities and limitations. None examines the impact of domain-specific AI agents, which represent a more advanced application of LLMs with enhanced robustness and task-solving capabilities. Our paper addresses this gap. Through our agent design, we demonstrate how LLMs can transcend basic content generation to become effective task solvers and productivity enhancers. By studying how professionally-equipped AI agents impact practical users, we expand the scope of social science research into advanced AI applications.
Technically, our paper directly links to the concept of expert system, the computer program that emulates the behavior of a human expert within a well-defined domain of knowledge (Liebowitz:1995), which is widely discussed since the last century (e.g., Tzafestas:1993; Tan:2017). The most traditional expert system is based on pre-defined rules, high-specified knowledge and various possible keywords, which strictly limit users' flexibility when interacting with the system to solve complex tasks. Another emerging branch applies neural network models and focuses more towards the learning capability of the system. Difficulty appears when facing insufficient training samples or requiring reasoning processes. Our MetricsAI enlightens a new direction for the future development of expert systems, combining the rapidly developing LLM techniques with the knowledge library and hence allowing the flexible functionality towards domain knowledge tasks.
Our paper makes another technical contribution addressing the challenge of enhancing LLMs' domain knowledge performance through a cost-effective solution. Current approaches primarily rely on expensive and time-consuming methods. These include fine-tuning models with curated domain-specific data, such as BloombergGPT (Wu:2023) for finance and Med-PaLM 2 (Singhal:2023) for medicine, or using lightweight fine-tuned LLMs to guide other LLMs (Yao:2023). Alternative approaches focus on complex prompt engineering to generate domain knowledge content (Liu:2025). However, without proper workflow and self-reflection mechanisms, these direct LLM applications remain vulnerable to hallucination issues that can significantly impact output quality.
MetricsAI's structural design implements a zero-shot learning framework that ensures high task-solving performance at minimal cost. Our approach minimizes hallucinations through two key mechanisms. First, domain knowledge toolkit integration prevents hallucinations in code generation and execution, limiting potential errors to tool selection and calling procedures. Second, self-reflection processes further reduce hallucinations by validating execution outputs and implementing error corrections. This design framework can be extended to other domains, particularly those with limited textual training samples or frequent knowledge updates, enabling efficient development of customized AI agents with domain-specific tools.
Current LLM tools face two primary limitations: they struggle to deliver complete workflows and perform poorly in knowledge-dense domains. MetricsAI addresses both challenges through a customized AI Agent system and a zero-shot learning framework, enabling accurate and executable workflows without the substantial costs of LLM retraining.
LLM-based AI agents are autonomous systems that perceive instructions, reason about complex tasks, and execute actions to achieve specified goals through code execution and tool usage. While generic LLM agents can handle open-ended queries, they struggle in expert domains such as econometrics, where problem-solving requires adherence to established analytical steps and specialized methods. Recent AI agent frameworks demonstrate that incorporating structured domain knowledge significantly enhances performance. MetaGPT (Hong et al., MetaGPT:2024a) encodes procedural knowledge as Standardized Operating Procedures within prompt sequences, using specialized sub-agents to verify intermediate results and decompose complex tasks. The Data Interpreter agent (Hong et al., datainterpreter:2024b) uses a hierarchical task graph and iterative refinement modules for end-to-end data-science workflows.
Building on these advances, MetricsAI features a customized architecture specifically designed to embed econometric domain expertise into its core operations. This specialization moves beyond the capabilities of general-purpose agents or direct LLM interactions by incorporating specific structural enhancements tailored for econometric analysis.
A cornerstone of this specialized structure lies in enhanced task decomposition. When presented with an econometric problem (User Input in Figure (ref)), the agent's Plan Generation module follows predefined templates reflecting standard econometric research paths---such as causal inference strategies and time-series analysis workflows---rather than relying solely on generic planning. This approach decomposes complex requests into logically sequenced, econometrically meaningful sub-tasks. These sub-tasks are categorized using specific econometric actions like “instrumental variable selection” and “difference-in-differences pre-trend check,” enabling more precise execution than generic labels.
Crucially, executing these specialized sub-tasks requires more than the LLM's reasoning capabilities. Thus, another key structural enhancement is the integration of a unique Econometrics Tool Library. During the Tool Selection and Program Execution phases (Figure (ref)), the agent accesses a bespoke set of Python functions implementing various econometric methods (e.g., OLS, PanelOLS, IV-2SLS, DID, RDD, propensity score methods), including options for robust standard errors and fixed effects. As detailed in Section (ref), these tools are described with internal prompts that enable the LLM core to understand their functionality and application context. This structured approach, where the agent selects and invokes validated tools, minimizes LLM hallucination risk for complex algorithms and ensures adherence to econometric practices---a distinct advantage over general agents lacking such domain-specific implements.
Furthermore, recognizing empirical analysis as an iterative process, MetricsAI incorporates sophisticated multi-round conversational capabilities with intent recognition and memory. When users provide subsequent instructions (multi dialogue loop in Figure (ref)), an Intent Recognition mechanism assesses whether the new input relates to the ongoing analysis or initiates a new task. For continuing analyses, the agent retrieves context from its Memory---including current data state, generated code, existing plan, and environment variables within its sandboxed execution environment. It then updates the Plan Generation phase, modifying or adding sub-tasks as needed, and resumes execution from the appropriate point. For new tasks, it initializes a fresh environment. This mechanism enables users to iteratively refine analyses, correct misunderstandings, or explore alternative specifications conversationally, ensuring outputs align with evolving research requirements.
These core structural enhancements---particularly the domain-tailored task decomposition and the unique econometrics toolkit---work in concert to ensure MetricsAI's effectiveness. The agent’s step-by-step reasoning is grounded in econometric best practices, while code execution relies on robust, pre-defined functions, leading to more reliable and accurate outcomes. This built-in domain expertise substantially reduces logical errors and methodological misapplications compared to general agents.
A key advantage of this design is its extensibility through a zero-shot prompting framework with tool use, allowing the agent to incorporate new econometric methods without LLM retraining. Unlike the costly and often infeasible process of fine-tuning an LLM to keep pace with rapid academic advances in developing new techniques, our agent can be updated simply by adding new tool functions and descriptions to the prompt library. This modularity allows the agent’s knowledge base to expand alongside the field’s developments, making the integration of recently published procedures as straightforward as adding new modules. Such capability ensures the agent's state-of-the-art performance while potentially enabling the transfer of this structured approach to other specialized domains.
In short, by combining LLM-driven reasoning with domain-specific planning structures and a dedicated toolkit, MetricsAI achieves performance, robustness, accuracy, and adaptability beyond the capabilities of generic AI agents. The architecture executes complete econometric analyses end-to-end, delivering rigorous results while dynamically incorporating user feedback through its interactive workflow.
LLMs are generative pre-trained transformers that learn from massive amounts of textual data across various knowledge domains, enabling strong performance in general topics (Wang:2020). However, achieving professional-level capability in specialized domains remains challenging, particularly in econometrics where two major limitations affect LLM fine-tuning.
First, econometric methods and algorithms rapidly evolve through academic journals, creating an expanding frontier of new knowledge. LLMs struggle to keep pace with these developments: frequent fine-tuning or retraining incurs prohibitive time and financial costs (Xia:2024), while limited availability of training data---both textual explanations and code---for new methods constrains learning capabilities. Second, novel econometric methodologies often face slow adoption and knowledge diffusion among empirical researchers and practitioners, resulting in insufficient training samples for LLM learning and performance. Even for the emerging LLM-based tools with searching engines or retrieval-augmented generation (RAG) techniques, their capability in solving users' novel tasks will still be constrained when there are merely several academic papers available on the Internet.
The framework of MetricsAI achieves proper understanding and accurate application of econometric knowledge through a zero-shot learning approach. This is accomplished via an econometrics algorithm tool package designed specifically for LLM use rather than human users. Unlike traditional packages familiar to econometricians in Stata and R, our Python-scripted package incorporates system prompts engineered for LLM comprehension. The current tool package supports popular empirical econometric applications including general linear models, propensity score methods, IV-2SLS regression, static/staggered difference-in-differences, and sharp/fuzzy RDD. All econometric models and methodologies use function calling format (\url{https://platform.openai.com/docs/guides/function-calling}), supporting flexible natural language inputs for econometric tasks.
More importantly, each function in the tool library includes a carefully designed internal prompt that introduces the background knowledge of the relevant econometric method and provides implementation guidance. These prompts define and explain hyper-parameters that enable flexible tool usage based on user requirements---for example, choosing between regular and robust standard errors in OLS. These system prompts enhance the LLM's understanding of different econometric tools' functionality, allowing accurate tool selection through reasoning when addressing user tasks.
An example of tool functions and internal prompts is provided in Figure (ref) in Appendix (ref). These tool packages are written in Python format to match MetricsAI's code execution environment. As previously introduced, an LLM serves as the agent's core, understanding available tools, selecting appropriate ones for given tasks, and carefully extracting inputs from user commands, including datasets, econometric methodologies, and related hyper-parameters. Each tool features carefully designed functional arguments that meet researchers' needs while maintaining sufficient flexibility to handle diverse, customized econometric tasks.
The tool packages' functionality builds upon established Python packages for simpler econometric algorithms (like OLS and logistic regression), while complex algorithms lacking available packages (such as staggered DID event study and fuzzy RDD) are developed on our own following standard Python package formats. Each tool package includes a detailed description prompt---effectively a “manual for LLMs”---that instructs the core LLM about the package's use. This design implements zero-shot learning Larochelle:2008,Lampert:2014, allowing the LLM to learn directly from its environment and access relevant knowledge without training or fine-tuning costs.
An example of the complete tool-selection-and-implementation workflow is provided in Figure (ref) in Appendix (ref). During initialization, the LLM processes each registered tool's internal prompt, summarizing four key aspects: target scenario, input requirements, output structure, and special requirements (if any). MetricsAI ranks available tools based on their target scenarios, selecting the highest-scoring tool for the current task. The LLM then generates code to ensure inputs meet tool package requirements while outputs fulfill user needs.
One might question whether LLM code generation capabilities could replace tools specifically designed for LLM use, suggesting that direct code generation might suffice given that Python and Stata package documentation is publicly available and likely included in LLM training data. However, limited training samples for specific econometric methods prevent LLMs from mastering domain knowledge and ensuring reliable performance. The test results in Sections (ref) and (ref) demonstrate the critical importance of our tool package zero-shot learning framework.
Our design has several key advantages. First, it requires no econometrics-specific LLM tuning. General-domain LLMs (such as GPT-4o and LLaMA 3) perform effectively within MetricsAI, as demonstrated in our empirical tests using unmodified GPT-4o. Second, since all tool methods are fixed functions---although accepting flexible inputs---hallucination within econometric algorithms is eliminated. The LLM's role is limited to understanding tool application scenarios and selecting/calling appropriate tools, with any remaining errors readily caught by the code compiler. Last but not least, the zero-shot learning framework enables simple integration of new algorithms and methodologies in Python function format, allowing continuous expansion of the econometric “ecosystem” and enhancement of the MetricsAI's capabilities.
We evaluate MetricsAI through two sets of inquiries. The first comprises 18 exercises from the coursework assignments of a doctoral-level course titled “Applied Econometrics” at the University of Hong Kong, with Python-generated standard solutions. These exercises cover OLS & PanelOLS regression, propensity score matching, IV-2SLS regression, Difference-in-Differences (DID) analysis, and Regression Discontinuity Design. The second set consists of test datasets from randomly selected seminal articles in reputable journals, primarily accompanied by Stata-based replication packages.
Table (ref) and Figure (ref) show the distribution of econometric algorithms across both test datasets. OLS and PanelOLS emerge as the predominant methodologies in empirical research in business, economics, and social sciences, while advanced causal analysis techniques like IV and DID also make significant contributions. Our MetricsAI supports these algorithms, enabling automated research workflows with minimal development time and cost. All test datasets are available in the study's online appendix.
Our empirical test adopts a structural prompt template to construct detailed questions. To demonstrate the MetricsAI's capabilities, we present results against control groups using standardized evaluation metrics.
While LLMs can process prompts of varying lengths, tones, attitudes, languages, and formats, their performance may be affected by potential latent factors in these prompts He:2024. To ensure consistent performance evaluation, we standardized the prompt format across all test questions. Each prompt must specify the data source, proposed econometric methodology, treatment variable, dependent variable, control variables, and customized requirements (such as data processing, fixed effects, robust standard errors, or methodology-specific settings). These components are integrated into a fixed prompt structure. For illustration, consider this example from Anderson:2008 using staggered DID event study to estimate safety belt legislation effects on traffic fatalities:
In this prompt template, boldface elements vary across questions to accommodate different econometric tasks, while the remaining structure stays fixed. For researchers familiar with econometric analysis, the workflow typically involves data exploration and cleaning, programming language selection, code implementation, and debugging. However, those without sufficient econometric experience face a more challenging process, often requiring extensive trial-and-error iterations.
The generated prompts are fed directly to MetricsAI, which automatically executes Python code in a Jupyter kernel and iteratively updates based on error messages. As detailed in Section (ref), the program terminates upon completion, storing results in JSON format. All scripts and outcomes are preserved for review, allowing users to interact with the AI agent through multi-round conversations for result revision and additional tasks.
Performance evaluation compares the generated coefficient, standard error, and p-value against standard solutions from coursework materials or original papers. We assess coefficient direction and $L_1$-norm distance (coefficients and standard errors in percentage, p-values in absolute distance). Two replication standards are defined: perfect replication requires all three errors to be below $1\%$, while partial replication requires coefficient and standard error gaps below $5\%$.
To demonstrate MetricsAI's capabilities, we establish three control groups:\footnote{To ensure clear performance differentiation across all the four groups, we consistently use GPT-4o as the base model for both direct response generation and agent empowerment. For the Stata-based tests, we manually collect metrics from summary tables, bypassing JSON file generation due to Stata's language constraints.}
Table (ref) summarizes the performance of the four groups on the coursework assignment problems. MetricsAI demonstrates superior performance with a 96.11% directional replication rate and very low level of coefficient error (median value of errors only around 0.01%). In contrast, both GPT-generated Python and Stata control groups show incorrect directions in over half of test cases. While the general AI Agent achieves a 77.78% directional replication rate, its coefficient values frequently deviate significantly from true values, with the median value of coefficient errors reaching 11.06%.
In addition, MetricsAI perfectly replicates around 58% of cases, compared to approximately 10% $\sim$ 27% for control groups. Its robust econometric toolkit and systematic internal prompts yield higher replication rates for advanced algorithms and complex procedures. Furthermore, both AI agents - MetricsAI and general-purpose AI agent (Data Interpreter) - achieve almost perfect compilation rates through error-feedback mechanisms, while direct LLM generation groups compile successfully less than 36% of the time.
Table (ref) presents results from the paper replication dataset. Given Stata's dominance in empirical research across business, economics, and social sciences, the Stata code generation group holds a natural advantage over Python-based approaches. Indeed, GPT-generated Stata code outperforms GPT-generated Python code across most evaluation metrics. However, MetricsAI achieves superior performance even in this Stata-favorable context, with a 88% directional replication rate - 52% higher than the Stata control group. Academic paper tasks present greater complexity than coursework, requiring detailed specifications, customized model structures, and various covariate adjustment methods. Nevertheless, MetricsAI consistently outperforms all control groups across key metrics: coefficient estimation error, standard error estimation error, p-value estimation error, and partial replication rates for different econometric algorithms and procedures. These results demonstrate the Agent's capability to handle complex econometric tasks autonomously while delivering reliable results.
Despite substantially outperforming control groups, MetricsAI does show room for improvement. For example, its performance declines for complex econometric methods like DID and RDD compared to simpler approaches such as OLS and IV-2SLS. Similarly, results slightly deteriorate when moving from straightforward coursework problems to more sophisticated paper replication tasks. However, these limitations can be addressed through the AI agent's domain knowledge architecture---specifically by developing customized tools and enhancing prompt instructions to better support complex algorithms and detailed requirements.
Following the empirical results presented in Section (ref), we demonstrate the MetricsAI's problem-solving approach through a comprehensive case study, comparing its procedures with the three control groups. The case study is drawn from Almond:2005 with the theme of estimating the effect of maternal smoking during pregnancy on infant weights applying propensity score-related approaches.
The first component is derived from the doctoral-level econometrics coursework materials and employs a propensity score-based regression adjustment method to conduct the analysis. This method was first introduced in ROSENBAUM:1983 and discussed by economists since Card:1988. With its advantages in maintaining sample completeness and supporting flexibility in non-linear & interaction effects, as well as its model simplicity, this method is commonly used as the first-step analysis among studies with propensity score-based methods. The question prompt is as follows:
The empirical research process begins with data pre-processing and matching, including data file merging, variable selection (dependent, independent, and control variables), and handling missing values. The analysis requires:
Using the provided dataset, the standard solution yields an ATE of $-207.7272$ with standard error $5.508$. One may also add the covariates into the second-stage OLS regression, which would produce an alternative ATE of $-212.9892$ with standard error $5.071$.
In the following, we compare MetricsAI with the three control groups in terms of problem-solving procedures, highlighting common LLM tool limitations and demonstrating how MetricsAI addresses them. All code generation and execution records are available online and can be accessed via \url{https://huggingface.co/datasets/troyhan/econometric_ai_agent_testset/tree/main}.
Direct code generation by LLMs appears to be a straightforward solution given the clear requirements and simple knowledge base. However, LLM hallucinations frequently disrupt code generation, where even minor errors can cause program termination without any meaningful results. The GPT-generated Python script demonstrates this vulnerability. As propensity score regression adjustment lacks built-in Python implementation, LLMs must generate complete data processing and regression procedures. Without self-correction capabilities, hallucinations occur in two areas, causing coding errors and execution failure:
Unlike direct LLM code generation, the general-purpose AI agent avoids coding errors through its reflection capabilities. However, Figure (ref) in Appendix (ref) reveals a behavioral bias of the agent. Since logistic regression, the first step of this analysis, is common in machine learning, the agent follows machine learning conventions by splitting data into training and test sets. This approach conflicts with standard econometric practice, which emphasizes empirical explanation rather than prediction, making such splitting unnecessary. The resulting estimates deviate significantly from the correct answer, yielding an ATE of $-103.7577$ with standard error $15.8834$.
Far from coincidental, this behavioral bias reflects a systemic issue stemming from the substantially greater accessibility of machine learning materials compared to econometric code resources. MetricsAI addresses this bias through two mechanisms: detailed internal prompts that enforce econometric standards, and a specialized tool library that ensures adherence to econometric conventions.
Since Stata provides numerous built-in data pre-processing methods and econometric algorithms, LLMs generate Stata code more effectively than Python code for econometric analysis. Moreover, Stata's specialized focus on econometrics, rather than machine learning, means its scripts in training samples naturally guide LLMs toward standard econometric approaches. However, Stata's function library lacks a built-in function specifically for propensity score regression adjustment. After obtaining propensity scores, the optimal approach is to directly apply OLS for final estimation. Instead, as Figure (ref) in Appendix (ref) shows, the hallucination issue arises, leading the LLM to mistakenly select propensity score matching for the second-step estimation. Although the results closely match the standard answer (ATE: $-218.9029$, standard error: $7.5627$), this methodological deviation render these estimates unreliable.
The knowledge mismatch error is another aspect of the hallucination issue that requires careful attention. For tasks related to propensity score, although OLS regression adjustment is simpler, LLMs tend to select the more popular and well-documented propensity score matching method, following their tendency to generate the most probable tokens.
In contrast, MetricsAI has two advantages that prevent this error: First, its tool library supports user-customized functions tailored to specific needs, including less common methods. For this task specifically, the Agent is equipped with both propensity score regression adjustment functionality and various other propensity score-based econometric methods. Second, the selection of econometric algorithms (and their corresponding tool functions) does not rely on LLM generation capabilities. Instead, the tool recommendation procedure introduced in Section (ref) first identifies the tool that best matches the task requirements. Only after that does LLM generate corresponding codes to correctly apply the tool and finish the task.
MetricsAI decomposes this task into three steps. The first two steps---“Load and preprocess the dataset from a specified file path” and “Perform exploratory data analysis on the dataset”---follow standard empirical econometric paradigms under internal prompt instructions. These steps complete essential data preparation, including categorical variable one-hot encoding and data cleaning. Furthermore, as Figure (ref) in Appendix (ref) shows, MetricsAI's precise workflow requirements prevent the coding errors observed in control groups.
For the final step, “Apply propensity score regression, controlling for specified variables and trimming samples,” MetricsAI utilizes built-in tool functions to construct propensity scores and conduct regression adjustment analysis, as shown in Figure (ref) in Appendix (ref). After tool selection, the LLM focuses solely on preparing correct inputs to meet task requirements. This approach significantly reduces both the knowledge mismatch error from LLM hallucination and the behavioral bias through standardized tool functions. The final estimates (ATE: $-207.8559$, standard error: $5.4845$) closely match the standard answer, validating the results.
Beyond the discussions in Almond:2005, subsequent studies have further explored propensity score methods for treatment effect identification. Take Cattaneo:2010 as an example. Based on their analytical frameworks, we formulate another propensity score matching task, which embeds the more commonly-used propensity score-based methodology, with the following instruction prompt:
The manual solution procedure follows similar initial steps as the propensity score regression adjustment method. For ATE estimation, a nearest-neighbor matching method is applied to both treatment and control groups using estimated propensity scores. The final estimate is derived from the mean difference between groups across all included entities. Under the specified settings, this procedure yields an ATE estimate of $-210.9683$.
Since propensity score matching is a built-in Stata function, GPT-generated Stata code provides succinct solutions. However, for Python implementations, where no popular packages exist for this method, the generated code must implement the method step-by-step. Here, the knowledge hallucination issue emerges consistently in our tests: as Figure (ref) in Appendix (ref) shows, GPT models claim to calculate ATE while actually producing ATET (Average Treatment Effect on the Treated) estimates. This error stems from LLMs' domain knowledge limitations. While general-purpose AI agents can use internal prompts to guide LLM behavior, such guidance is not sufficient and cannot ensure comprehensive knowledge of every domain-specific detail.
This example therefore provides direct evidence for the advantage of MetricsAI's tool library design. Figure (ref) in Appendix (ref) demonstrates how the core LLM, supported by a well-established knowledge and tool library, avoids knowledge hallucination under MetricsAI's framework. Rather than solely relying on LLMs' content generation ability to perform domain knowledge-driven tasks---which are often complicated, in-depth, and sometimes uncommon and ambiguous---the tool library greatly simplifies LLMs' generation and decision processes while guaranteeing domain knowledge capability. Beyond this straightforward case example and simple methodology, the tool library's robustness, flexibility, and extensibility ensure the elimination of knowledge hallucination across more complex tasks and algorithms.
The AI era has brought significant productivity gains across various domains, yet challenges persist in fields requiring deep domain expertise. We introduce MetricsAI, a LLM-driven specialized system that automates econometric analysis. Through a carefully customized agent structure, it executes complete econometric analyses while dynamically adapting to user feedback. The agent incorporates a simple yet robust zero-shot learning framework that enables continuous functionality expansion, both within econometrics and across other knowledge domains. We demonstrate the agent's superior performance through empirical testing on coursework assignments and economics papers, complemented by in-depth case study comparisons against standard LLMs and general-purpose AI agents.
MetricsAI's excellent performance demonstrates its significant potential for future applications. Beyond reducing learning barriers for econometrics students, it provides academic researchers and industry practitioners with efficient tools for research tasks. As leading academic journals increasingly require original data and replicable procedures, manual paper proofreading has become time-consuming and challenging. MetricsAI can serve as an AI-driven digital referee for empirical research papers, significantly boosting proofreading efficiency (Mueller:2019) while ensuring content quality and validity.
Our work's extensibility manifests in two key dimensions. First, MetricsAI's capabilities can be expanded by incorporating additional tool packages. For instance, addressing growing concerns about p-hacking in empirical research in business, economics, and social sciences, we developed a tool package using inverse optimization to analyze potential decision procedures regarding empirical test settings within given optimization spaces and datasets. This cost-effective addition helps detect potential p-hacking risks in empirical research, offering an efficient alternative to manual review.
Second, our zero-shot learning framework can extend to other knowledge-intensive domains. Through targeted instruction prompt refinement and domain-specific tool package development, this one-off setup enables new AI agents to deliver consistent, high-quality performance in their respective domains. This extensibility opens possibilities for future applications in areas such as quantitative investing and macroeconomics.
\linespread{1.3}\selectfont \printbibliography