Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
138,100 characters · 22 sections · 168 citation commands
A Unifying Framework for Robust and Efficient Inference with Unstructured Data
\abstract{To analyze unstructured data (text, images, audio, video), economists typically first extract low-dimensional structured features with a neural network. Neural networks do not make generically unbiased predictions, and biases will propagate to estimators that use their predictions. While structured variables extracted from unstructured data have traditionally been treated as proxies – implicitly accepting arbitrary measurement error - this poses various challenges in an era where constantly evolving AI can cheaply extract data. Researcher degrees of freedom (e.g., the choice of neural network architecture, training data or prompts, and numerous implementation details) raise concerns about p-hacking and how to best show robustness, the frequent deprecation of proprietary neural networks complicates reproducibility, and researchers need a principled way to determine how accurate predictions need to be before making costly investments to improve them. To address these challenges, this study develops MAR-S (Missing At Random Structured Data), a semiparametric missing data framework that enables unbiased, efficient, and robust inference with unstructured data, by correcting for neural network prediction error with a validation sample. MAR-S synthesizes and extends existing methods for debiased inference using machine learning predictions and connects them to familiar problems such as causal inference, highlighting valuable parallels. We develop robust and efficient estimators for both descriptive and causal estimands and address inference with aggregated and transformed neural network predictions, a common scenario outside the existing literature.}
Economists frequently use unstructured data in empirical research. These include images (pixel data), text (token sequences from a high-dimensional vocabulary), audio (waveforms or spectrograms), and videos (image sequences). However, unstructured data are rarely used directly in econometric analyses because they are high-dimensional, computationally unwieldy, and uninterpretable. Instead, researchers extract meaningful low-dimensional features - referred to as structured data - and use these in statistical analyses.
For instance, commonly used datasets extract structured information on governance, institutions, political stability, policy uncertainty, conflict, and violence from news articles and other text data.\footnote{e.g., the Economic Policy Uncertainty Index, the Armed Conflict Dataset, the Conflict Recurrence Dataset, the Correlates of War, the Global Terrorism Database, the Polity5 Dataset, Internal Wars & Failures of Governance, the Mass Mobilization Data, the Armed Conflict Location & Event Data, and the Geopolitical Risk Index.} Researchers impute sentiment, topics, and a variety of other structured data from government transcripts, firm filings, earnings calls, patent data, and web texts.\footnote{ e.g., FOMC transcripts, various sources for earnings calls, USPTO and Google patent data, and web texts} Satellite images of nighttime lights are used to measure economic activity, development, and urbanization.\footnote{ See gibson2020night for a review.} Imputations from remote sensing data frequently supplement ground measurements of temperature, precipitation, pollution, agricultural output, land use, illicit activities, and deforestation.\footnote{These include datasets on temperature and precipitation (e.g., MODIS, ERA5, CHIRPS, TRMM), agriculture and land use (Harvest, Landsat, Sentinel), air quality and atmospheric composition (e.g., OMI/Aura and TROPOMI/Sentinel-5P, MERRA-2, CAMS), and deforestation and illicit activity (e.g., PlanetScope,UNODC-ICMP)}
Traditionally, extracting structured information required costly manual processing or complex, human-engineered rules, and hence structured data were often created through large-scale initiatives. Recent years have witnessed a dramatic decline in the costs of processing unstructured data, with the machine learning literature showing that deep neural networks excel at cheap, large-scale feature extraction from unstructured data goodfellow_deep_2016. Today, individual researchers are increasingly using neural networks to process unstructured data, and a growing economics literature reviews their promise (e.g., gentzkow_text_2019,dell_deep_2025).
However, neural networks will not generically make unbiased predictions in finite samples. Choices related to network architecture, the distribution of training data, and various implementation details can introduce systematic biases. Moreover, the use of nonlinear transformations at each layer of the neural network and the frequent application of neural networks to binary or multiclass classification problems also violate classical measurement error assumptions. When neural networks serve as first-step estimators to predict structured features from unstructured data, their biases will propagate to estimators that rely on their predictions, affecting point estimates and uncertainty quantification.
Structured variables extracted from unstructured data are often treated as proxies, implicitly accepting the possibility of arbitrary measurement error. However, in a world of inexpensive AI models, such as commercial large language models (LLMs), treating first-step predictions as proxies poses a variety of serious challenges to the integrity of empirical work. First, different first-step imputations - resulting from the choice of different prompts, models, training data distributions, or implementation details - will plausibly have different biases, raising concerns about researcher degrees of freedom and p-hacking. Researchers often attempt to address these concerns by examining many different first-step predictors, which can lead to unwieldy robustness appendices without fully addressing bias concerns. Relatedly, off-the-shelf models create the possibility of “AI slop”, by making it very cheap to generate a large number of proxies that may not be well-defined or rigorously related to the underlying hypothesis of interest. Moreover, economists frequently use proprietary neural networks, raising replication concerns when these models are deprecated. Yet commercial models cannot always be replaced by an open-source alternative. Finally, when working with open-source neural networks, economists can improve them through costly investments - by training larger models, collecting more or higher-quality training data, training for longer, and refining implementation details. However, it is unclear how to assess how many costly investments to make without a principled way to account for how first-step prediction errors affect the estimation of target parameters.
Addressing these challenges requires a shift from thinking of structured data as proxies to being precise about what the measurement aims are and how well they are being achieved. We would like a broadly-applicable framework that corrects the bias introduced by imputing structured variables from unstructured data, yielding target parameters that are robust to the choice of first-step predictors and hence reducing concerns about bias and researcher degrees of freedom. This would also allow the replication of similarly unbiased - albeit not numerically identical - estimates when neural networks are deprecated. Investments in better first-step predictors (e.g., training a highly performing customized neural network over using a model off-the-shelf) would enable more precise estimation of the target parameters, making the cost-benefit calculations of such investments clearer.
Fortunately, foundational work on semiparametric inference with missing data - in particular the rubin_inference_1976 missing at random (MAR) mechanism - provides just such a framework\footnote{See also little_statistical_2019,robins_estimation_1994,robins_analysis_1995,robins_semiparametric_1995,bang_doubly_2005}. This study introduces MAR-S (Missing At Random Structured Data), a framework for conducting valid, efficient, and robust inference on estimands that incorporate unstructured data through first predicting their low-dimensional features. MAR-S frames inference with unstructured data as a missing data problem - because raw unstructured datasets typically lack the low-dimensional summaries that are relevant to economic analyses - tailoring Rubin's foundational MAR mechanism to structured summaries of unstructured data. The estimators arising from this framework use a validation sample with ground truth values to estimate the bias in the imputed structured data and adjust the estimates accordingly. The validation sample must meet the “missing at random” (MAR) assumption rubin_inference_1976: after adjusting for observable factors, annotated and unannotated structured data should be comparable in their ground truth values.
Ground truth - as used in machine learning - does not mean fundamental truth. Rather, ground truth derives from an implementable definition of the structured data specified by the researcher. It can be accessed through some non-scaleable technology, such as the judgment of the researcher or a costly measurement process. While economists sometimes use unstructured data to measure concepts that lack an undisputed definition, specifying and evaluating the intended measurement allows unbiased inference conditional on the stated definition. It also requires researchers to be precise in developing their measurement goals and facilitates important discussions about what we should be measuring in the first place.
The missing at random (MAR) requirement for the validation sample is a powerful complement to deep neural networks, because it allows us to make minimal assumptions about the neural network, an extremely complex data generating process. A fundamental principle in machine learning is that predictive accuracy deteriorates with domain shift away from the training data distribution, which often includes a split of the annotated data ben2010theory. Biases shift in complex ways with the unstructured inputs, and humans are not very good at predicting how domain shift affects performance vafa2024large. MAR-S addresses this fundamental feature of neural networks by requiring a representative validation sample, in order to allow robust, unbiased inference with a black box imputation function. If MAR is not feasible, it is necessary to open this black box and place more restrictions on the imputation function, such as assuming a stable relationship between the imputed structured data and ground truth across annotated and unannotated samples\footnote{See rambachan2024program for a framework using remote sensing data where ground truth is unavailable for the sample of interest.}. While sometimes necessary and plausible, the complexity of domain shift can call such assumptions into question. Fortunately, when researchers create their own annotations, it is often straightforward to ensure they satisfy MAR. Researchers also need to ensure that annotations are of high quality, as we cannot correct arbitrary imputation error using an annotated sample measured with arbitrary error.\footnote{As one would expect, progress can be made only if one is willing to make assumptions about the form of the measurement error in the annotated data.}
The value of validation data has long been recognized in the econometrics literature on measurement error schennach_recent_2016, and debiasing with a ground truth sample is also central to recent frameworks for valid statistical inference using predictions from black box AI models (e.g., angelopoulos_prediction-powered_2023,egami_using_2023,ludwig_large_2024). In the context of these literatures, MAR-S makes three sets of related contributions. First, it provides a theoretical synthesis, showing that recent work on inference with black-box AI models—developed independently across disciplines with limited interactions—arise from and can be extended by a semiparametric missing data framework. MAR-S clarifies inference with unstructured data - and the value of validation data - for an economist audience by connecting this problem to familiar literatures on measurement error (e.g., schennach_recent_2016, chen_measurement_2005,chen_semiparametric_2008), debiased machine learning (e.g., chernozhukov_doubledebiased_2018, chernozhukov_riesznet_2022, chernozhukov_automatic_2022), and causal inference (e.g., rubin1978bayesian, imbens_causal_2015, robins_estimation_1994). For example, the MAR assumption - which could alternatively be termed “annotation on observables” - parallels the “selection on observables” assumption in causal inference, a closely related missing data problem where counterfactual outcomes are missing. A variety of insights from these closely connected literatures can be transferred to inference with unstructured data, reducing the risk of “reinventing the wheel” in what may seem at first glance a novel problem (inference with AI predictions).
We depart from most recent work on statistical inference with black box AI by emphasizing a semiparametric approach, which not only clarifies necessary and sufficient statistical independence and function approximation assumptions, but also provides new insights about estimator efficiency. For an estimator to achieve asymptotic efficiency, the imputation of missing structured data should depend not only on unstructured data but also on context-specific structured variables that help estimate the target parameter (e.g., other covariates in a regression model).
Centrally, MAR-S makes it feasible to apply debiasing to a wide variety of empirical scenarios. While debiasing is intuitive and built upon long-established methods, it has not been widely used in inference with unstructured data in economics, and common empirical scenarios fall beyond the existing literature. This study aims to bridge that gap. We derive robust, efficient estimators for descriptive moments, linear regression model coefficients, treatment effects identified through linear IV models, modern difference-in-differences estimands, and regression discontinuity estimands under local randomization. Because MAR-S employs a semiparametric approach, it can be integrated with recent advances in automatic debiasing chernozhukov_automatic_2022, van_der_laan_automatic_2025 and automatic (functional) differentiation luedtke_simplifying_2024 to further extend its applicability. To facilitate its application, MAR-S is accompanied by an implementation package.\footnote{\url{https://github.com/jscarlson/mar-s}}
Importantly, existing debiasing approaches assume that validation data are available for the imputed variable(s) used in the relevant estimating equation. However, in economic applications, validation data are typically available at a granular level (e.g., for individual texts or images), while the parameter of interest often reflects a (possibly nonlinear) aggregation of individual predictions (e.g., log economic policy uncertainty in the U.S. in 1999, constructed from many newspapers articles). Collecting ground truth at the aggregate level is often infeasible, because it would require many labels or because the measure of interest is a population-level quantity whose value is not observed in any finite sample. Hence, we develop a framework that leverages MAR-S for debiased inference when ground truth data are available only at a disaggregated level, while the parameter of interest is derived from structured data that are aggregated and potentially transformed. We also consider issues that arise when the structured data of interest represent a “rare event.” For example, a text that pertains to a specific topic can be relatively rare compared to a corpus's size.
This paper is organized as follows. Section (ref) situates MAR-S within the relevant literature. Section (ref) introduces the MAR-S framework, Section (ref) develops estimators for functionals widely used in empirical economics, and Section (ref) extends MAR-S to common empirical settings. Section (ref) demonstrates empirical applications of MAR-S, and Section (ref) concludes.
This study builds on extensive literatures in econometrics, statistics, biostatistics, and machine learning (ML) to offer a theoretical synthesis and empirical guidance on inference with unstructured data.
Specifically, MAR-S addresses common applied economics scenarios that fall outside a recent literature on inference with black-box flexible function approximators (e.g.,, LLMs). Notable contributions to this literature include a series of papers on “prediction-powered inference” (PPI) angelopoulos_prediction-powered_2023, angelopoulos_ppi_2024, zrnic_cross-prediction-powered_2024, zrnic_active_2024, ji_predictions_2025, kluger_prediction-powered_2025, as well as “design-based supervised learning” egami_using_2023, egami_using_2024 and the applied econometric framework of ludwig_large_2024. This literature assumes that ground truth is available at the level of the parameter of interest. In contrast, in empirical economics, ground truth is usually available at a granular level—e.g., individual texts or images—while the parameter of interest often reflects a (possibly nonlinear) aggregation of record-level predictions or a population-level quantity. MAR-S addresses this, as well as more generally considering efficient and robust inference in common empirical settings and offering new insights into data requirements for efficiency in these settings.
MAR-S moreover provides a general theoretical framework that synthesizes contributions from the emerging literature on black-box AI—much of which originates in disciplines outside economics and does not aim to provide a general framework—with longstanding results in semiparametric statistics and econometrics. angelopoulos_prediction-powered_2023, angelopoulos_ppi_2024 - in seminal work on inference with black-box AI predictions - introduce a baseline framework for prediction-powered inference (PPI), which for the sake of simplicity assumes away several common features in empirical applications. For example, it assumes predictions are fixed outputs from a pre-trained black-box model (excluding model training and cross-fitting), annotated data are a simple random sample (excluding non-uniform or unknown labeling procedures), and it does not pursue a semiparametric approach or speak to semiparametric efficiency, with angelopoulos_ppi_2024 only briefly commenting on connections to semiparametrics. These limitations are addressed individually by follow-up work: zrnic_cross-prediction-powered_2024 incorporates training and cross-fitting; zrnic_active_2024 allows for non-uniform annotation while maintaining fixed predictions; and ji_predictions_2025 adopts a semiparametric approach that includes training and cross-fitting but assumes known, uniform annotation. They treat the model predictions as fixed, incorporating training through a tripartite sample-splitting scheme to separately estimate nuisances, learn an imputed-loss function to recalibrate the predictions, and perform inference. MAR-S instead follows the simpler, ubiquitous pre-train, fine-tune paradigm - which overwhelmingly predominates in machine learning applications - by directly calibrating (non-fixed) model predictions through finetuning with a bipartitie sample splitting scheme. Save for kluger_prediction-powered_2025, these methods only apply to estimands which are the minimizers of “nice” (e.g., convex, smooth) population losses; a motivation of kluger_prediction-powered_2025 is to improve on this paradigm. kluger_prediction-powered_2025 do not consider semiparametric efficiency, but do allow for non-uniform (though known) annotation probability. Appendix Section (ref) provides further comparison between MAR-S and the existing theoretical literature.
By tailoring general semiparametric methods to unstructured data, MAR-S accommodates all the aforementioned complexities within a single framework. MAR-S also unifies other recent contributions with a more applied orientation, such as list_using_2024 and ludwig_large_2024. Such synthesis can facilitate cross-fertilization across distinct literatures and reduce the risk of “reinventing the wheel”, which in our reading has sometimes been an issue given limited interactions between AI/ML and econometrics. There are legibility advantages to providing the simplest possible framework and then making individual modifications, especially given that much of the black box AI literature targets an interdisciplinary audience (e.g., angelopoulos_prediction-powered_2023 is published in Science). MAR-S remains legible by drawing on foundational contributions in econometrics that are likely to be familiar to an economist audience.
MAR-S adapts longstanding semiparametric methods, utilizing Rubin’s missing data framework (rubin_inference_1976; see also little_statistical_2019, robins_estimation_1994, robins_analysis_1995, robins_semiparametric_1995, bang_doubly_2005). This underscores that even with transformative new technologies—such as LLMs—classical statistical frameworks remain foundational. Semiparametric/nonparametric frameworks make relatively weak assumptions about the data generating process (DGP), allowing the data to inform estimation as much as possible (e.g., pfanzagl_contributions_1982, bickel_efficient_1998, newey_asymptotic_1994, van_der_vaart_asymptotic_1998, ackerberg_asymptotic_2014, kennedy_semiparametric_2016, kennedy_semiparametric_2018, chernozhukov_locally_2022). Deep neural networks - whose complexity mitigates against making simplifying assumptions about their DGP - are well-suited to such assumption-lean methods. Semiparametric inference is also grounded in theories of minimax-style efficiency, providing principled benchmarks for comparing the performance of different estimators newey_asymptotic_1994, van_der_vaart_asymptotic_1998.
This study emphasizes insights from connecting MAR-S to semiparametric literatures on measurement error, causal inference, and debiased machine learning. MAR-S builds directly on an econometric literature on measurement error correction with auxiliary validation data, extending the semiparametric framework of chen_semiparametric_2008 to high-dimensional unstructured data. In doing so, it bridges classical econometric theory on measurement error (see chen_nonlinear_2011,schennach_recent_2016,schennach_measurement_2022 for reviews) with contemporary machine learning applications. A key insight from this literature is that validation samples with ground truth values provide a general, model-agnostic method for correcting non-classical measurement error in nonlinear models chen_measurement_2005,chen_semiparametric_2008. ludwig_large_2024 likewise center measurement error in their applied econometric framework for LLM predictions.
The imputations used by MAR-S estimators can also be viewed as “imperfect surrogates” in the sense of kallus_role_2024 and ji_predictions_2025: proxies for the ground truth of interest that, when combined with some ground truth labels, can be used to improve estimation efficiency even when typical (stringent) surrogacy conditions are not met athey_surrogate_2019. This contrasts to settings without a ground truth for the sample of interest, where establishing the relevance of the surrogate (imputed) values to the target estimand requires more careful theoretical justification rambachan2024program, nakamura_surrogate_2025.
MAR-S moreover relates closely to causal inference, underscoring the close connections between Rubin's MAR framework and the Rubin Causal Model neyman1923application, rubin1974estimating, rubin1978bayesian, imbens_causal_2015. Both causal inference and inference with unstructured data can be conceptualized as special cases of a more general missing data problem little_statistical_2019, ding_causal_2018, hirano_efficient_2003. Counterfactual outcomes are missing for causal inference, whereas structured data summaries are missing in inference with unstructured data. The MAR-S framework builds on this connection, with notation and structure that mirror key aspects of causal inference. Insights from causal inference can elucidate core challenges in inference with unstructured data.
A prominent example of a semiparametrically efficient estimator is the augmented inverse propensity weighted (AIPW) estimator, widely used for estimating treatment effects in causal inference robins_estimation_1994, robins_semiparametric_1995, scharfstein_adjusting_1999. The AIPW estimator belongs to a broader class of “doubly robust” estimators, which offer protection against misspecification by relaxing the rate requirements for estimating nuisance parameters. The nuisance parameter estimators are “first-steps” that do not directly estimate the primary parameter of interest (e.g., a causal effect or other functional) but are essential for constructing the final estimator (e.g., by imputing missing data). Many of this study's estimators follow the structure of the AIPW estimator, and their double robustness is an important feature, because it allows us to avoid placing strong regularity or rate conditions on the first-step imputation of missing structured data.
The literature on double/debiased machine learning (DML) likewise yields doubly robust estimators, developing tools for valid inference on low-dimensional target parameters while relying on flexible, nonparametric first-step estimators—particularly in causal inference and in the estimation of structural parameters chernozhukov_doubledebiased_2018, chernozhukov_riesznet_2022, chernozhukov_automatic_2022, ichimura_influence_2022. Building on this tradition, we construct efficient and robust estimators for missing data functionals, under assumptions tailored to unstructured data and deep neural networks.
This section introduces the MAR-S framework, which extends the foundational missing data mechanism of rubin_inference_1976 to inference with unstructured data. We first establish a terminology and model for data missingness in the context of unstructured data (Section (ref)) and then outline the key assumptions of MAR-S (Section (ref)). Both the notation and underlying assumptions closely parallel those in semiparametric causal inference neyman1923application, rubin1974estimating, rubin1978bayesian, imbens_causal_2015 and semiparametric analysis of missing data more broadly tsiatis_semiparametric_2006. Section (ref) defines robustness and efficiency, important properties of MAR-S estimators. We then consider issues posed by deep neural networks and big data and why the classic missing at random framework remains well-suited for these novel challenges (Section (ref)). Finally, we outline the MAR-S algorithm (Section (ref)).
To enable robust and efficient inference with unstructured data, we recast the problem as inference on missing structured data. Structured data, denoted \(M \in \mathcal{M}\), are low-dimensional data that can be used directly in estimating equations. They contrast with unstructured data \(U \in \mathcal{U}\) (e.g., raw RGB values, audio waveforms, or sparse term vectors), which are high-dimensional and unsuitable for direct use in estimation.
Under MAR-S, structured data are observed through a process termed annotation. A non-scaleable measurement technology, such as a human expert or costly measurement device, uses a well-specified definition of the missing structured data to record \(M\), which corresponds to a given \(U\). The annotation process is described by the annotation score function $\pi(x):=P(A = 1 \mid X = x)$, where $X \in \mathcal{X}$ are covariates observed by the researcher.
Since annotation is too expensive to scale, researchers learn an imputation function \(\hat{\mu}\) to impute missing structured data. This allows the researcher to leverage the full unstructured dataset for more precise estimation. Deep neural networks typically serve as the imputation function, because they are the state-of-the-art method for mapping unstructured data to low-dimensional outputs. However, their predictive accuracy can vary significantly depending on task complexity, model architecture, and the distributions of input and training data.
MAR-S---and the rubin_inference_1976 framework that it builds upon---are closely linked to the Rubin Causal Model neyman1923application, rubin1974estimating, rubin1978bayesian, imbens_causal_2015, as causal inference is fundamentally a missing data problem. Thus, we incorporate potential outcomes notation. This provides a unified notation for discussing missing structured data, alongside other forms of data missingness, such as those arising from causal inference.
Suppose that structured data \(M \in \mathcal{M}\) are subject to some data missingness given by an annotation indicator variable \(A \in \{0,1\}\). A consistency of potential outcomes assumption then allows us to write:
where \(M^{a=1}\) is the “ground truth” potential outcome, observed when the unstructured data instance is labeled, and \(M^{a=0}\) is the “missing” potential outcome, obtained in the state where the instance has not been annotated. We keep the argument \(a\) explicit for now, to distinguish it from potential outcomes that depend on other random variables, such as treatment status.
The probability distribution of the random variable \(M^a\) is the distribution of \(M\) in a counterfactual world where \(A = a\) with probability 1, e.g., the distribution of structured data when all data have been annotated. The researcher is interested in functionals of this distribution.
Without loss of generality, we can set \(M^{a=0} = 0\) with probability 1, since, given an assumption on the consistency of potential outcomes, we only need to enforce that \(M^{a=0}\) carries no information about \(M^{a=1}\). For brevity of notation, relabel \(M^* := M^{a=1}\). Hence:
This notation follows Rubin's canonical missing data models, which assume that a researcher observes the (iid) dataset \(\{(A_i, A_iM_i^*, \ldots)\}_{i=1}^n\) and is interested in inference on \(M^*\).
We make no general assumptions about the relationship between $M^*$ and $U$. In some settings, such as when $U$ is solely used to create $M^*$ without consultation of any other sources of information, it may be reasonable to assume that $M^*$ is a deterministic function of $U$; in other words, $M^*$ is $U$-measurable.\footnote{C.f., nakamura_surrogate_2025, which explicitly makes this assumption.} In other settings - such as common remote sensing settings where ground truth comes from ground measurements and cannot be inferred soley from satellite images themselves, or any annotation that consults external information - $M^*$ is unlikely to be a deterministic function of $U$. Nonetheless, $U$ may explain much of the variation in $M^*$.
Just as we cannot perform valid causal inference without assigning some units to treatment, we cannot perform valid inference on \(M^{a=1}\) without annotating some data, ensuring that \(P(A = 1) > 0\). Hence, MAR-S is not suited for unsupervised settings, where the aim is to uncover patterns in entirely unlabeled data. For econometric methods involving unstructured data without a defined ground truth, see, for example, battaglia_inference_2024. MAR-S is also a superpopulation-based framework that assumes data are independently and identically distributed (iid), although we discuss extensions that allow for clustering and other error structures.
This section outlines the assumptions underlying the MAR-S framework. Centrally, the first-step imputation function---used to predict missing structured data---operates under weak assumptions, because of our knowledge of annotation.
In causal inference, consistency of potential outcomes requires that the treatment is well-defined and that each observation’s outcome depends solely on its own treatment status (non-interference). In settings like the present with one-sided data missingness, the concept relies on analogous principles. Annotation status must be well-defined—each observation is either labeled or not—and the ground truth label for any given observation should depend only on its own annotation status, not on the annotation status of other observations. This is ensured by using a rubric for annotation that remains unchanged throughout the annotation process.
Assumption (ref) requires that annotation reveals $M^*$. Researchers that anticipate labeling to reveal ground truth plus well-understood noise could model this as an alternative to Assumption (ref). If the researcher is not willing to take a stand on the absence or form of measurement error in annotations, debiasing as proposed by MAR-S is out of reach: one cannot correct arbitrary error in the imputation of structured data using annotations measured with arbitrary error.\footnote{See nakamura_surrogate_2025 for an example of a framework that permits labeling noise under additional assumptions.}
The second assumption is the cornerstone of the MAR-S framework. It states that Rubin's “missing at random” (MAR) assumption holds for the ground truth potential outcomes, given observables $X$ the researcher can access rubin_inference_1976,little_statistical_2019.
This is analogous to the selection on observables assumption in causal inference hirano_efficient_2003, and following that nomenclature, Assumption (ref) could also be labeled “annotation on observables.” After adjusting for observables $X$, annotated and unannotated structured data (and the associated unstructured data) are comparable in their ground truth values. There are no unaccounted confounders determining whether an instance of unstructured data is annotated. In the era of deep learning, it is increasingly common for researchers to create their own annotated datasets, and Assumption (ref) provides guidance on how to do so in order to allow unbiased, robust, efficient inference.
The third assumption is that the annotation mechanism is known and can be bounded such that no instances of unstructured data are guaranteed to be either annotated or unannotated.
The naming convention “annotation score function” mimics the terminology of the propensity score function, which serves an analogous role in causal inference.
Assuming a bounded annotation score function is equivalent to the assumption of “strict overlap” commonly used in observational causal inference. In such settings, the plausibility of strict overlap diminishes as the dimensionality of the variable ensuring unconfoundedness increases damour_overlap_2021. In most economic applications with unstructured data, $X$ is low-dimensional, in contrast to high-dimensional $U$. Notably, the machine learning literature sometimes specifies $X$ as a low-dimensional representation of $U$, a perspective we will revisit when discussing annotation in practice (Section (ref)).
In social science, existing annotation practices often violate this assumption. Researchers frequently use keyword-based filtering when working with text data, assigning a nonzero annotation probability only to texts containing specific keywords. This is a particularly common approach when the data are class imbalanced - e.g., texts of the class of interest are rare compared to the size of a corpus. We would plausibly expect a LLM's prediction error to depend on the terms in the text, and hence the bias observed in the annotated sample may not reflect that in the full unannotated data. This is problematic when all text is used to estimate the parameter of interest. A similar issue arises when annotation data are collected for one time period but the quantities of interest are estimated using imputed structured data for a longer or different period. For example, the remote sensing literature shows that a model trained to distinguish crop type for a specific time period may perform worse on years outside this period due to distributional shifts in farming practices or weather wang2020mapping.
One way to address this issue is to redefine the population of interest to include only the unstructured data instances from which the annotation sample is drawn. While this approach will be appropriate in some cases, it changes the interpretation of the resulting estimands and estimates. Section (ref) discusses alternative strategies for selecting the most informative annotation sample in class-imbalanced data, while putting positive probability on sampling all unstructured data instances.
When the researcher does not annotate their own data, the annotation function may need to be estimated. Assumption (ref) can be relaxed, provided the researcher is willing to impose asymptotic estimation rate requirements on the imputation function and the (estimated) annotation function. We elaborate on these requirements when discussing the final MAR-S assumption, to which we turn now.
This assumption concerns the ideal function for imputing missing structured data, which takes the general form: \[ \mu(\tilde x) := E[M\mid A=1, \tilde X = \tilde x], \] where $\tilde X := (X, U, V)$. Here, $V$ is a set of context specific random variables that influence the estimation of our quantity of interest, such as controls in an ordinary least squares regression. As we will see in the following section, these may be necessary for efficient estimation.
In practice, $\mu$ is unknown and must be approximated. We refer to the estimator $\hat\mu$ as the imputation function. Accurate imputation captures meaningful information about the ground truth distribution of structured data and improves precision when the predictions are used to estimate a functional of interest. Here, as in other semiparametric causal inference settings, we make an assumption about the quality of an estimator $\hat\mu$ for $\mu$ in the large sample limit.
Intuitively, this condition says that we need the expected $L^2(P)$ error of our estimator to go to zero as the amount of data we train the estimator with goes to infinity. Assumption (ref), sometimes referred to as “universal consistency” wager_causal_2024, is mild in the context of deep neural networks: recent theoretical work has shown that certain classes of deep neural networks learned with gradient descent - as will be deployed with MAR-S - are universally consistent drews_universal_2024.
If we relax Assumption (ref), we lose some robustness in our estimators. We must now estimate the annotation score function, and Assumption (ref) (universal consistency of the imputation function) must be replaced with a stronger set of assumptions on (MSE or $L^2(P)$) convergence rates for both the imputation function and the estimated annotation score function in order to guarantee efficient, $\sqrt{n}$-consistent, and asymptotically normal inference. Intuitively, the researcher now needs to assume that both the convergence of the imputation and estimated annotation score functions occurs at a sufficiently fast rate. For discussion of these rates, see e.g., chernozhukov_doubledebiased_2018,kennedy_semiparametric_2023,wager_causal_2024. For instance, it is sufficient that both the imputation and estimated annotation score functions converge at order $n^{-1/4}$ rates in many settings of interest. Such rates are shown to be achievable for various classes of neural networks chen_improved_1999,farrell_deep_2021.
Alternatively, Assumption (ref) can also be relaxed while maintaining Assumption (ref), and - as long as the other assumptions still hold - estimates will still be unbiased, albeit not efficient. To understand why this is the case, we now turn to a discussion of robustness.
Robust estimation, in the current context, focuses on constructing estimators that relax the asymptotic estimation rate requirements for first-step (nuisance) parameter estimates in multi-step procedures. First-step parameter estimators are not directly used to estimate the primary parameter of interest (e.g., a causal effect or functional) but are essential for constructing the final estimator, e.g., by imputing structured data. Intuitively, robust estimators are designed to tolerate trade-offs in estimation error among their first-step components while maintaining asymptotic normality and $\sqrt{n}$-consistency. Worse performance in one component can be offset by better performance in another. An estimator is strongly doubly robust wager_causal_2024 when there are two nuisance functions, and the estimator can balance errors between them without losing $\sqrt{n}$-consistency.
In MAR-S, Assumptions (ref) and (ref) characterize robustness. The first-step estimator $\hat{\mu}$ (e.g., a deep neural network that imputes structured data) is subject to weak conditions because we have access to the most accurate possible first-step estimator for the annotation function $\pi$, which is $\pi$ itself. Intuitively, knowing the annotation score function significantly enhances the robustness of semiparametric estimation in MAR-S, analogous to how knowing the propensity score function would strengthen the robustness of semiparametric causal inference.
In the literature on inference with black box AI, the imputation function is typically treated as an arbitrary, fixed function $f$ (e.g., a pre-trained LLM). The baseline MAR-S framework assumes tuning the neural network on a split of the annotated data but can likewise accommodate a fixed, arbitrary $f$ by dropping the universal consistency assumption. Using a fixed $f$ (e.g., an off-the-shelf commercial LLM) will not yield efficiency - intuitively, fine-tuning on empirical examples drawn from the distribution of interest will generally reduce prediction error, increasing precision - but double robustness ensures the validity of inference even when the imputation function is fixed and highly misspecified (provided that the annotation score function is known).
We now turn to a discussion of efficiency in MAR-S. A semiparametrically efficient estimator of a functional achieves the lowest possible asymptotic variance among all regular, asymptotically linear (RAL) estimators of that functional (see, e.g., Theorem 25.20 of van_der_vaart_asymptotic_1998). In other words, an efficient estimator attains the semiparametric variance lower bound, which serves as a benchmark for optimality within this framework. This lower bound corresponds to the variance of the efficient influence function (EIF) associated with the functional. We denote the EIF by $\varphi_\theta(W_i; P)$ for a data instance $W_i \in \mathcal{W}$, a joint distribution $P \in \mathcal{P}$ where $W_i \overset{\text{iid}}{\sim} P$, and a functional $\theta \in \Theta \subseteq \mathbb{R}$.\footnote{Efficiency is also defined for vector-valued parameters $\theta \in \Theta \subseteq \mathbb{R}^p$, though, for simplicity, we only consider scalar-valued functionals in this paper.} Simplifying the notation to $\varphi(W_i)$, an “optimal” RAL estimator $\hat{\theta}$ satisfies \[ \sqrt{n}(\hat{\theta} - \theta) \xrightarrow{d} N(0, \text{Var}(\varphi(W_i))). \]
Intuitively, an influence function captures how a small change in the data impacts the value of a functional (e.g., the mean), or the probability limit of an estimator. Functionals are associated with influence functions, sometimes referred to as “influence curves” kennedy_semiparametric_2023,hines_demystifying_2022; estimators themselves can be linked to influence functions through their probability limits newey_asymptotic_1994,ichimura_influence_2022.
In fully nonparametric statistical models-which is to say there are no restrictions placed on $\mathcal{P}$-any RAL estimator is necessarily efficient, as there is only one valid influence function for all RAL estimators, which is the efficient influence function chen_overidentification_2018, kennedy_semiparametric_2023, schuler_introduction_2024. However, in semiparametric models, multiple influence functions may exist for a RAL estimator, but only one achieves the semiparametric efficiency bound and corresponds to the EIF. Because the annotation score function $\pi$ is known in the MAR-S framework, the statistical model is semiparametric, which we label $\mathcal{P}_\pi$. As a result, any RAL estimator is not automatically efficient. Multiple influence functions may exist, and the efficient influence function is the one with the lowest variance.
When we relax Assumption (ref) and the annotation function is estimated, the statistical model under which inference is performed becomes fully nonparametric: $P \in \mathcal{P}$, as opposed to $P \in \mathcal{P}_\pi$. However, for all functionals of a certain class, we will show that the EIF remains the same under $\mathcal{P}_\pi$ or $\mathcal{P}$; as such, all EIFs we derive in the application section of this paper remain the same when relaxing Assumption (ref) to unknown but estimable annotation score functions (see Lemma (ref)). Hence, the construction of efficient estimators also remains the same.
The intuition behind constructing robust and efficient estimators within the MAR-S framework is broadly similar to that in the semiparametric inference literature for missing data: impute the missing data using a sufficiently accurate first-step imputation function and debias using ground truth values. However, several factors differentiate unstructured data from other contexts.
First, imputation functions $\hat{\mu}$ are typically deep neural networks, in particular transformers vaswani_attention_2017, which are often needed to ensure that $\mu \approx \hat{\mu}$ either asymptotically or in finite samples. Although the convergence rates for these models are unknown, Assumptions (ref) and (ref) imply that we do not need to focus on this but rather on universal consistency, a weak requirement for deep neural networks drews_universal_2024. Moreover, a massive empirical literature demonstrates the success of deep neural networks in prediction tasks involving unstructured data, suggesting that convergence occurs quickly in practice klaassen_doublemldeep_2024, following pre-training on massive-scale data.
Relatedly, features $\tilde X$ are high-dimensional because high-dimensional unstructured data $U$ is a component of $\tilde X$. Classic results from nonparametric statistics indicate that minimax rates for learning functions conditional on high-dimensional covariates can be poor when the conditional expectation function lacks sufficient smoothness stone_optimal_1982. This has motivated the development of higher-order influence function (HOIF) estimators robins_higher_2008,robins_minimax_2017. However, Assumption (ref) obviates the need to implement HOIF-based estimators: when the annotation score function is known, a first-order influence function-corrected estimator has no (asymptotic) bias, which is what HOIF-based estimators seek to correct liu_semiparametric_2023. Moreover, we only need universal consistency of the high-dimensional imputation function (e.g., the neural network). In practice, nonparametric regression using deep neural networks on high-dimensional unstructured data appears to exhibit fast rates of convergence under common data-generating processes klaassen_doublemldeep_2024, likely due to the massive-scale datasets used for pretraining. Relatedly, recent theory reveals that neural networks can be especially well suited for estimating nonparametric first-steps/nuisances for treatment effects under selection on a diverging number of confounders chen_causal_2024.
In order to be highly predictive, deep neural networks need to be pre-trained on massive-scale datasets. In various settings, this raises concerns about data contamination. For example, if GPT saw predictions from a benchmark dataset in pre-training, it may perform well on the benchmark not because it can generalize, but simply because it has memorized the answers sainz2024data. In economics, contamination concerns have arisen when a neural network estimates a conditional expectation of \( M^* \) in a sample that is contaminated given some information set, and then is applied to some non-contaminated setting, leading to concerns such as lookahead bias sarkar_lookahead_2024. With MAR-S, in contrast, the neural network is just a tool for regression adjustment within a doubly robust estimator.\footnote{See Section (ref) for more on connections to regression adjustment.} Assumptions 1-3 (about annotation) deliver an unbiased estimator, regardless of whether the neural network has seen $U$ during pre-training. The only source of sampling variability in the imputation function \( \hat{\mu} \) in MAR-S arises from the annotated fine-tuning data, and the pre-trained model is fixed (with respect to sampling).
Finally, unstructured datasets can be massive, and the structured data of interest to the econometrician may represent a “rare event.” For example, the structured data may indicate whether a piece of text in a large database (e.g., all social media posts or historical news articles) pertains to a specific topic, which is rare relative to the dataset's enormity. In rare event estimation, a common concern is the large “coefficient of variation,” defined as the ratio of the variance of the event indicator to the probability of the event. This can make uncertainty quantification less informative. To address this, one might optimize the annotation score function $\pi$ to reduce the variance of estimates by incorporating elements of importance sampling mcbook, zrnic_active_2024, as elaborated in Section (ref).
Massive unstructured datasets also connect to the notion of “decaying overlap” zhang_double_2023: the marginal probability of annotation may be small given the size of the full corpus, prompting interest in alternative asymptotic thought experiments. We assume a conventional asymptotic regime where the marginal annotation probability $P(A=1)$ is bounded away from zero. However, as kallus_role_2024 make clear in their analysis of treatment effects (an analogous problem), there is no need to change the MAR-S estimator itself under an asymptotic analysis where the number of annotations is diverging but the ratio of labeled data to unlabeled data is converging to zero. Appropriate asymptotic efficiency analysis shows the variance of the estimator is primarily driven by the size of the labeled dataset, as opposed to the full dataset size.\footnote{See, e.g., Theorem 4.2 of kallus_role_2024.} A key difficulty explored in kallus_role_2024 and zhang_double_2023 is estimating a propensity score (equivalent to the MAR-S annotation score) in the decaying overlap setting. Such difficulties are obviated in MAR-S by the assumption that the annotation score is known.\footnote{In applications where this does not hold, researchers could apply the propensity score estimators of zhang_double_2023 or kallus_role_2024 as annotation score estimators.}
We now provide an overview of how to implement the MAR-S framework in a general setting. MAR-S follows a standard procedure for semiparametric efficient inference pfanzagl_contributions_1982 that has gained significant traction in biostatistics and econometrics, especially for causal inference (e.g., bang_doubly_2005,van_der_laan_targeted_2006,chernozhukov_doubledebiased_2018). We specifically build on frameworks for deriving efficient and robust semiparametric estimators in hines_demystifying_2022 and kennedy_semiparametric_2023. MAR-S adopts this canonical procedure to missing structured data using the following steps:
More intuition is provided through the various examples developed in the following section.
We illustrate the MAR-S framework in five empirical settings of particular interest to economists: descriptive moments, linear regression, linear instrumental variables (IV) models, difference-in-differences (DiD) designs, and regression discontinuity designs (RDD) under local randomization.\footnote{MAR-S is limited to pathwise differentiable functionals, in the sense that $\sqrt{n}$-consistent estimators are not guaranteed to exist for nonpathwise differentiable functionals, and as such efficiency would be ill-defined.} We also show how MAR-S can unify recent work on inference with black-box AI models—developed independently across disciplines with limited interactions—and connect this work to classic results from semiparametric inference and widely used inference methods that incorporate machine learning-based first steps chernozhukov_doubledebiased_2018,chernozhukov_locally_2022,chernozhukov_riesznet_2022,chernozhukov_automatic_2022.
We develop each example by assigning a single variable to be constructed from missing structured data $M$ (e.g., an outcome or treatment), although MAR-S can be equally applied to settings where alternative - or multiple - variables are imputed from unstructured data.
Before deriving specific estimators, it is useful to define a core class of functionals of interest.
Many functionals of missing structured data---including all the functionals of missing structured data considered in this paper---can be written as MAR-S mean functionals.\footnote{Note that the definition of a MAR-S mean functional implies that $[(U,M^*,V) \perp\!\!\!\perp A] \mid X$ by the contraction property of conditional independence.} For such functionals, we have the following identification result.
Intuitively, if we observe some ground truth values of the structured data $\tilde M^*$ (i.e., $P(A=1)>0$), then we can identify $\theta$.\footnote{Similar identification results are possible by simply letting $\tilde X = X$ or $\tilde X = (X,V)$. The importance of incorporating unstructured data $U$ in $\tilde X$ is, naturally, to produce the most accurate possible imputations of the missing structured data, for which $X$ or $V$ may contain very little signal. The intuition mimics that of (nonparametric) regression adjustment. For formal results on the importance of imputation based on unstructured data in the settings we consider, see Appendix Section (ref).}
We now state the following lemma, which greatly simplifies the derivations of efficient influence functions for the applications considered.
Under the MAR-S framework, the statistical model is semiparametric when the annotation score function $\pi$ is known. Deriving efficient influence functions under semiparametric models is typically more challenging than under fully nonparametric models, for which there is only one influence function, which is the efficient influence function. Lemma (ref) shows that the EIF for a MAR-S mean functional $\theta$ under a nonparametric statistical model is also the correct EIF for $\theta$ under the semiparametric statistical model, simplifying derivation. Intuitively, perturbing the distribution given by the annotation score does not change the value of the functional: if we had labeled the data in a different (but valid) way, the (in-population) value of the parameter being estimated would remain unchanged.\footnote{This agrees with several related results in chen_semiparametric_2008, e.g., the asymptotic variance lower bound of parameters identified using “verify-in-sample” auxiliary datasets is unaffected by knowledge of the “propensity score” (which is $1-\pi(x)$ in the MAR-S framework).}
With Lemma (ref), we can compute the efficient influence function for a generic MAR-S mean functional, which we introduce in the next proposition.
We begin by applying MAR-S to descriptive moments - means of appropriately defined random variables - which are relevant to many analyses and lay the foundation for analyzing more complicated estimands.
Consider the dataset $\{W_i\}_{i=1}^n$ where $W_i := (M_i, A_i, X_i, U_i)$, with variable of interest $M_i \in \mathcal{M} \times \{0\} \subseteq \mathbb{R}$, annotation indicator $A_i \in \{0,1\}$, observed covariates $X_i \in \mathcal{X} \subseteq \mathbb{R}^k$, and unstructured data $U_i \in \mathcal{U} \subseteq \mathbb{R}^\ell$. We further assume that $W_i \overset{\text{iid}}{\sim} P$, for some arbitrary distribution $P$. We define $\tilde X_i := (X_i, U_i)$.
We wish to compute a descriptive moment of the data, and without loss of generality we will focus on the mean.\footnote{This is without loss of generality insofar as $M$ may simply be defined as some structured random variable raised to a particular power.} Assuming that $E[|M_i|] < \infty$, we define the estimand of interest, denoted by $\theta$, as the expected value of $M_i^*$: \[ \theta := E[M_i^*]. \] We use the terms estimand, functional, and parameter interchangeably to refer to $\theta$.
By Proposition (ref), letting $\tilde M_i^* = g(M_i^*,1) = M_i^*$, we can identify $\theta$ as
where $\mu(\tilde X_i) := E[M_i \mid A_i = 1, \tilde X_i] = E[M_i \mid A_i = 1, X_i, U_i]$. From Proposition (ref), it is then immediate that the efficient influence function for MAR-S mean functional $E_P\left[\mu(\tilde X_i)\right]$ is \[ \varphi(W_i) = \mu(\tilde X_i) + \frac{A_i}{\pi(X_i)}(M_i - \mu(\tilde X_i)) - \theta, \] where $\pi$ is the annotation score function.
As such, we can form the (doubly) robust and efficient one-step estimator as \[ \hat \theta = \frac{1}{|\mathcal{I}|}\sum_{i \in \mathcal{I}} \left[ \hat\mu(\tilde X_i) + \frac{A_i}{\pi(X_i)}(M_i - \hat\mu(\tilde X_i)) \right],\] where $\mathcal{I}$ is the set of indices of the data allocated to the “estimation” partition of a random data split, and where $\hat\mu$ (the deep neural network) is estimated on the other “tuning” partition, represented by the set of indices $\mathcal{I}'$. That is, $\hat\mu$ is a random function of $\{W_{i'}\}_{i'\in \mathcal{I}'}$. This takes the form of the augmented inverse probability weighted (AIPW) estimator, which arises as the efficient estimator of a potential mean in causal inference robins_estimation_1994,rotnitzky_semiparametric_1998,scharfstein_adjusting_1999, a closely related missing data problem.
The following proposition confirms that this estimator is indeed efficient (attains the semiparametric variance lower bound) and robust (only requires the weak Assumption (ref)).
Denote $\pi(X_i)=|\mathcal{I}\cap \mathcal{J}|/|\mathcal{I}|$, where $\mathcal{J}$ is the set of indices corresponding to annotated data points. Then:
The second expression (PPI) is equivalent to the expression for mean estimation in angelopoulos_prediction-powered_2023, whereas the last expression (FRA) is reported in recent work on “flexible regression adjustment” list_using_2024. While these are two distinct literatures, both can be viewed as special cases unified under the banner of efficient and robust semiparametric inference for missing data. Term A in the second expression is the best imputation-based guess of $E[M_i^*]$ in the estimation sample (the naive “plug-in” estimator), and term B is a bias correction term, that can be thought of as an estimate of the measurement error of the imputation function in the annotated sample. In big data settings, the variance of term A will be small, leading to seemingly precise estimates when imputation error is ignored. The variance of term B will depend on the predictive performance of $\hat \mu$, as well as the size of the annotated set $\mathcal{J}$. If the imputation function is not very predictive, it will produce limited information about the underlying true distribution of data, which term B accounts for. The third expression illustrates an alternative perspective: term C is the best estimate of the quantity of interest using only ground truth data. We then leverage the imputation function $\hat\mu$ as a form of nonparametric regression adjustment in term D, with the same intuition as a linear regression adjustment.
MAR-S also relates closely to the double/debiased machine learning (DML) framework chernozhukov_doubledebiased_2018. For example, in the context of estimating a potential mean under the “selection on observables” assumption in a causal inference setting, the DML framework proposes an estimator derived via a Neyman orthogonal score, which is likewise identical to the AIPW estimator robins_estimation_1994. Deriving a Neyman orthogonal score can be viewed as an “estimating equations” approach to constructing semiparametrically efficient estimators kennedy_semiparametric_2023,schuler_introduction_2024, as opposed to alternative methods such as the one-step influence function-based correction used in MAR-S.\footnote{chernozhukov_locally_2022, which generalizes the original DML results, further makes clear that Neyman orthogonal moments can be viewed as influence function-based corrections to moment conditions.} Because MAR-S is based on the same fundamental semiparametric analysis as DML, there are likely many ways to apply insights from the DML framework to MAR-S. Consider a recent strand of the DML literature on “automatic” or data-driven ways of implementing debiasing corrections chernozhukov_locally_2022,chernozhukov_riesznet_2022,chernozhukov_automatic_2022. Although many functionals considered under MAR-S lead to Riesz representers with simple analytic expressions, there are other missing structured data settings that induce more complicated functionals where automatic debiasing could be quite useful.
We now apply MAR-S to linear regression. For the sake of illustration, we consider the case where an outcome variable is imputed from unstructured data, although the MAR-S framework can readily impute one or more regressors (as well as the outcome).
Consider the dataset $\{W_i\}_{i=1}^n$, where $W_i=( M_i, C_i, A_i, X_i, U_i)$, with outcome $ M_i \in \mathcal{M} \times \{0\} \subseteq \mathbb{R}$, regressors $C_i \in \mathcal{C} \subseteq \mathbb{R}^d$, annotation indicator $A_i \in \{0,1\}$, observed covariates $X_i \in \mathcal{X} \subseteq \mathbb{R}^k$, and unstructured data $U_i \in \mathcal{U} \subseteq \mathbb{R}^\ell$. We further assume that $W_i \overset{\textrm{iid}}{\sim} P$ for some distribution $P$, and that $ [C_i \perp\!\!\!\perp A_i] \mid (X_i, U_i, M_i^*)$.
We assume that \[ M_i^* = C_i^\mathtt{T} \theta + \varepsilon_i, \quad E[\varepsilon_i \mid C_i ] = 0,\] where $\theta \in \Theta \subseteq \mathbb{R}^p$. We are interested in identifying and estimating the $j$-th regression coefficient $\theta_{j}$. By the Frisch-Waugh-Lovell theorem, we have that \[ \theta_{j} = E\left[{C_{i,j}^\perp}^2\right]^{-1} E\left[ C_{i,j}^\perp M_i^*\right], \] where $ C_{i,j}^\perp:=C_{i,j}-E^*[C_{i,j} \mid 1, C_{i,1}, \ldots, C_{i,j-1}, C_{i,j+1}, \ldots, C_{i,d}]$, and where $E^*$ is the linear projection operator. We assume that the full sample available to the researcher is large enough that $C_{i,j}^\perp$ may be treated as known, a plausible assumption in many big data settings that motivate the application of deep neural networks.\footnote{If this assumption does not hold, then the MAR-S framework can still be applied, but the efficient influence function will change. Intuitively, when $C_{i,j}^\perp$ is not known, our functional hides other component functionals stemming from the population regression coefficients in the residualized variable.} We will further assume that $C_{i,j}^\perp$ is bounded with probability 1 (e.g., because all covariates are bounded almost surely), though this assumption may be relaxed if Assumption (ref) is strengthened to hold conditionally.
Let $\tilde X_i := (X_i, U_i, C_i)$. The numerator of the above expression is a MAR-S mean functional with $\tilde M_i^* = g(M_i^*,C_{i,j}^\perp)=C_{i,j}^\perp M_i^*$, so by Proposition (ref): \[ \theta_{j} := \theta_{j,\text{den}}^{-1} \theta_{j,\text{num}}= E\left[{C_{i,j}^\perp}^2\right]^{-1} E\left[ C_{i,j}^\perp \mu(\tilde X_i)\right] \] where $ \mu(\tilde X_i) = E\left[ M_i \mid A_i = 1, \tilde X_i\right]$.
Because the numerator functional is a MAR-S mean functional, it is likewise immediate from Proposition (ref) that the EIF for the numerator functional is
where $\pi$ is the annotation score function. The EIF for the denominator functional $E_P[{C_{i,j}^\perp}^2]$ is the well-known form of the EIF for any unconditional mean schuler_introduction_2024\footnote{Because the denominator functional is just concerned with the marginal distribution of the random variable ${C_{i,j}^\perp}^2$, we can effectively treat the relevant statistical model as nonparametric, as, borrowing notation used in the appendix proofs, $\nabla_h \theta_{j,\text{den}} = \nabla_{h_{{C_{j}^\perp}^2}} \theta_{j,\text{den}} = \nabla_{h_\pi} \theta_{j,\text{den}}$.} \[ \varphi_\text{den}(W_i) = {C_{i,j}^\perp}^2 - \theta_{j,\text{den}}. \]
The robust, efficient one-step estimators for the numerator and denominator functionals are: \[ \hat \theta_{j,\text{num}} = \frac{1}{|\mathcal{I}|}\sum_{i\in \mathcal{I}} C_{i,j}^\perp \left[\hat\mu(\tilde X_i) + \frac{A_i}{\pi( X_i)}( M_i -\hat\mu(\tilde X_i)) \right], \quad \hat \theta_{j,\text{den}} = \frac{1}{|\mathcal{I}|}\sum_{i\in \mathcal{I}} {C_{i,j}^\perp}^2. \] Combining these estimators, we can form the efficient estimator for $\theta_{j}$ as \[ \hat \theta_j := \hat \theta_{j,\text{num}} \hat \theta_{j,\text{den}}^{-1} = \frac{\sum_{i\in \mathcal{I}} C_{i,j}^\perp\left[\hat\mu(\tilde X_i) + \frac{A_i}{\pi( X_i)}( M_i - \hat\mu(\tilde X_i)) \right]}{\sum_{i\in \mathcal{I}} { C_{i,j}^\perp}^2}. \] This is just a residualized regression with “pseudo-outcome” $\hat \varphi_i := \hat\mu(\tilde X_i) + \frac{A_i}{\pi( X_i)}( M_i - \hat\mu(\tilde X_i))$, underscoring the connection to mean estimation:\footnote{The term “pseudo-outcome” mirrors similar language used by egami_using_2023 and in causal inference.} \[ \hat \theta_j := \hat \theta_{j,\text{num}} \hat \theta_{j,\text{den}}^{-1} = \frac{\sum_{i\in \mathcal{I}} C_{i,j}^\perp\hat \varphi_i}{\sum_{i\in \mathcal{I}} {C_{i,j}^\perp}^2}. \]
One interesting implication is that the imputation function $\hat \mu$ is a function of context-specific variables (the controls in the least squares regression), as $\tilde X_i := (X_i, U_i, C_i)$. Under Assumption (ref), the imputation function should asymptotically approximate the function $ E[M \mid A= 1, X=x, U=u, C=c ]$ for estimation to be efficient. This has not been emphasized in the existing literature, which often treats the imputation function as an arbitrary black box. Learning a function of unstructured and structured (tabular) data is termed “wide and deep learning” cheng_wide_2016 and has recently been studied in econometrics klaassen_doublemldeep_2024.
If $[C_i \perp\!\!\!\perp M_i^* ] \mid (X_i,U_i)$, then the optimal imputation function would no longer need to be a function of $C$. Alternatively, if $M^*$ is $U$-measurable - it can be accessed solely from unstructured data $U$ without consulting any external information - then the optimal imputation function would no longer need to account for $X$ or $C$, as $ E[M \mid A= 1, X, U, C ]=E[M^* \mid U] = M^*$ almost surely, and the only relevant function to learn is an approximation of $E[M \mid A=1, U=u]$. While this describes simple information extraction tasks, it fails in most remote sensing contexts - where integration of information from ground sources (e.g., weather stations, household surveys) is necessary to interpret images, as well as in many tasks where knowledge of some outside information helps extract $M^*$. For instance, the date when a document was written may be needed to most accurately assess which individuals the document refers to arora2024contrastive. Imputation functions that ignore relevant information would still yield valid inference under Assumption (ref), but would not yield (asymptotically) efficient inference.
We now extend MAR-S to linear IV, illustrating the case where treatment is imputed from unstructured data and following the terminology and setup of blandhol_when_2022. This is for illustrative purposes, and it is straightforward to apply MAR-S when any (or multiple) variables are the missing structured data.
Consider dataset $\{W_i\}_{i=1}^n$ with \[ W_i = (Y_i, M_i, C_i, Z_i, A_i, X_i, U_i), \] where $Z_i \in \mathcal{Z} \subseteq \mathbb{R}$ is a candidate instrumental variable (IV), $C_i \in \mathcal{C} \subseteq \mathbb{R}^d$ are covariates (which contain a constant), $ M_i \in \mathcal{M} = \{0,1\}$ is a treatment of interest, $Y_i \in \mathcal{Y} \subseteq \mathbb{R}$ is an outcome of interest, $A_i \in \{0,1\}$ is the annotation indicator, $X_i \in \mathcal{X} \subseteq \mathbb{R}^k$ are observed covariates relevant to annotation, and $U_i \in \mathcal{U} \subseteq \mathbb{R}^\ell$ are unstructured data. We further assume that $W_i \overset{\text{iid}}{\sim} P$ for some joint distribution $P$, and that $[(C_i, Z_i) \perp\!\!\!\perp A_i] \mid (X_i, U_i, M_i^*)$.
Potential outcomes are denoted $\{Y_i^m\}_{m\in\mathcal{M}}$ and potential treatments are denoted $\{ M_i^z\}_{z \in \mathcal{Z}}$. Assuming consistency of potential outcomes and treatment, we have \[ Y_i =\sum_{m \in \mathcal{M}} \mathbb{I}( M_i=m) Y_i^m,\quad M_i =\sum_{z \in \mathcal{Z}} \mathbb{I}(Z_i=z) M_i^z. \]
We are interested in identifying and estimating the average treatment effect: \[ \theta := E[Y_i^{m=1} - Y_i^{m=0}]. \] Per blandhol_when_2022, we assume that the candidate instrumental variable is relevant and exogenous, that treatment effects are homogeneous, and that the conditional expectation of the potential outcome is a linear function of the covariates. We can then express $\theta$ as \[ \theta =E[Z_i^\perp M_i^*]^{-1}E[Z_i^\perp Y_i],\] where $Z_i^\perp$ is $Z_i$ linearly residualized with $C_i$, using the same notation and assumptions (known, almost surely bounded) as in the previous application. Under these assumptions, $\theta$ is the two stage least squares (TSLS) estimand.\footnote{If we were to drop Assumptions CLE and LIN, per blandhol_when_2022, and instead impose their “ordered strong monotonicity” (OSM) assumption, and if $C_i$ were “rich,” e.g., $C_i$ were saturating indicators of the relevant discrete covariate space for (conditional) exogeneity to hold, then $\theta$ would still have a “weakly causal” interpretation.}
Let $\tilde X_i := (X_i, U_i, Z_i, C_i)$. Then, by Proposition (ref), noting that the denominator of this expression is a MAR-S mean functional with $\tilde M_i^* = g(M_i^*, Z_i^\perp) = Z_i^\perp M_i^*$, we can identify $\theta$ as \[ \theta = \theta_{\text{den}}^{-1} \theta_{\text{num}} =E[Z_i^\perp \mu(\tilde X_i)]^{-1}E[Z_i^\perp Y_i] \] where $\mu(\tilde X_i) := E[ M_i \mid A_i=1, \tilde X_i]$.
The efficient influence function for the numerator functional (under the semiparametric statistical model $\mathcal{P}_\pi \ni P$ induced by Assumption (ref)) is \[ \varphi_\text{num}(W_i) = Z_i^\perp Y_i - \theta_{\text{num}},\] which is again the EIF for an unconditional mean. The efficient influence function for the denominator functional (using Proposition (ref) directly) is \[ \varphi_\text{den}(W_i) = Z_i^\perp \left[ \mu(\tilde X_i) + \frac{A_i}{\pi(X_i)}( M_i - \mu(\tilde X_i))\right] - \theta_{\text{den}}, \]
As with linear regression, we can form the efficient estimator for $\theta$ by first forming efficient one-step estimators for $\theta_{\text{num}}$ and $\theta_{\text{den}}$, \[ \hat\theta_\text{num}=\frac{1}{|\mathcal{I}|}\sum_{i \in \mathcal{I}} Z_i^\perp Y_i, \quad \hat\theta_\text{den} = \frac{1}{|\mathcal{I}|}\sum_{i \in \mathcal{I}} Z_i^\perp \left[ \hat\mu(\tilde X_i) + \frac{A_i}{\pi( X_i)}( M_i - \hat\mu(\tilde X_i)) \right], \] and then combining them \[ \hat \theta = \hat\theta_\text{num}\hat\theta_\text{den}^{-1} = \frac{\sum_{i \in \mathcal{I}} Z_i^\perp Y_i}{\sum_{i \in \mathcal{I}} Z_i^\perp\left[ \hat\mu(\tilde X_i) + \frac{A_i}{\pi(X_i)}( M_i - \hat\mu(\tilde X_i)) \right]}. \] Note that the efficient estimator is just a TSLS regression with pseudo-treatment $\hat\varphi_i := \hat\mu(\tilde X_i) + \frac{A_i}{\pi(X_i)}( M_i - \hat\mu(\tilde X_i))$, or \[ \hat \theta = \hat\theta_\text{num}\hat\theta_\text{den}^{-1} = \frac{\sum_{i \in \mathcal{I}} Z_i^\perp Y_i}{\sum_{i \in \mathcal{I}} Z_i^\perp \hat\varphi_i}. \]
The optimal imputation function is a function of $Z$ and $C$, absent further assumptions, e.g., $M^*$ is $U$-measurable.
In this application, we focus on the nonparametric difference-in-differences (DiD) estimator introduced in callaway_difference--differences_2021.
Consider the dataset $\{W_i\}_{i=1}^n$: \[ W_i = (M_{i1}, \ldots M_{iT}, D_{i1}, \ldots, D_{iT}, A_{i1}, \dots, A_{iT}, X_{i1}, \dots, X_{iT}, U_{i1}, \dots, U_{iT}), \] where $M_{it} \in \mathcal{M} \times \{0\} \subseteq \mathbb{R}$ is an outcome of interest, $D_{it} \in \mathcal{D}=\{0,1\}$ are treatment indicators, $A_{it} \in \{0,1\}$ are annotation indicators, $X_{it} \in \mathcal{X} \subseteq \mathbb{R}^k$ are observed covariates, and $U_{it} \in \mathcal{U} \subseteq \mathbb{R}^\ell$ are unstructured data. Suppose that $W_i \overset{\text{iid}}{\sim} P$ for some joint distribution $P$.
Let $G_{ig}$ be a binary indicator for unit $i$ first being treated at time $g$ and let $C_i$ be an indicator for units that are never treated. Furthermore, we define $M_{it}^{g=0}$ to be the untreated potential outcome of unit $i$ at time $t$ and $M_{it}^{g}$ to be the potential outcome of unit $i$ at time $t$ if they first became treated in period $g$. Assuming consistency of potential outcomes, we have \[ M_{it}=M_{it}^{g=0}+\sum_{g=2}^{T}\left(M_{it}^g-M_{it}^{g=0}\right) G_{ig}. \]
We are interested in the estimand \[ \theta = E\left[{M_{it}^*}^g -{M_{it}^{*}}^{g=0} \mid G_{ig}=1\right],\] the average treatment effect on those treated in cohort $g$ at time $t$. Under the framework of callaway_difference--differences_2021, this functional is the crucial input to any other substantive DiD estimand of interest, which will just be a known or identified weighted average of these estimands. Without loss of generality, we set $g = 2$ and $t = 2$, the canonical two-period DiD setup. Under Assumptions 1–4 of callaway_difference--differences_2021, assuming that their Assumptions 3 and 4 hold unconditionally, as well as under their non-anticipation assumption, we can express $\theta$ as
for $M_i^* := M_{i2}^*-M_{i1}^*$.
Let $M_i := M_{i2}-M_{i1}$, $\tilde X_i := (X_{i1}, X_{i2}, U_{i1}, U_{i2}) $, and $A_i := A_{i1}\cdot A_{i2}$. We will assume that $[(C_i, G_{i2}) \perp\!\!\!\perp A_{i}]\mid (X_i, U_i, M_i^*)$. Then, observing that both terms are MAR-S mean functionals under Proposition (ref) with $\tilde M_i^* = g(M_i^*, G_{i2}P(G_{i2}=1)^{-1}) = P(G_{i2}=1)^{-1} G_{i2} M_i^*$ (respectively, with second argument $P(C_{i}=1)^{-1} C_i$) we can identify $\theta$ as
where $\mu_G(\tilde X_i) = E[ M_i \mid G_{i2} = 1, A_i = 1, \tilde X_{i}]$ and $\mu_C(\tilde X_i) = E[ M_i \mid C_i = 1, A_i = 1, \tilde X_{i}]$.
The efficient influence functions for functionals $E_P[\mu_G(X_i) \mid G_{i2}=1]$ and $E_P[\mu_C(X_i) \mid C_i=1]$ are, respectively,
from Proposition (ref).
We can form efficient one-step estimators for each $\theta_G$ and $\theta_C$ as
where it is assumed that the marginal probabilities $P(G_{i2}=1),P(C_{i}=1)$ are effectively known, given the size of the full sample.\footnote{As with the residualization discussion in the previous sections, if this is not a viable assumption for a given empirical application then the MAR-S framework still applies, but the EIF and corresponding estimator will change.} We can combine them to form the efficient estimator for $\theta$ as
Lastly, we consider a regression discontinuity design (RDD) under MAR-S. We focus on sharp RDD under the local randomization framework, as opposed to the continuity framework cattaneo_regression_2022.\footnote{Under the continuity framework, the analogous sharp RDD estimand involves functionals of the form \[ \theta(r):= E[M_i^* \mid R_i = r], \] where $R_i \in \mathcal{R}$ is continuous, and $\theta(r)$ is to be evaluated at a particular (limiting) point in $\mathcal{R}$. In the literature on semiparametric inference, it is well known that functionals of this type are not pathwise differentiable, and hence there is no regular, $\sqrt{n}$-consistent estimator of the quantity. That said, frameworks for valid asymptotic inference on such functionals (that incorporate flexible first-step estimators) do exist bibaut_data-adaptive_2017,noack_flexible_2024, and resemble strategies arising from inference on conditional average treatment effects with continuous conditioning covariates in the causal inference literature kennedy_nonparametric_2017,semenova_debiased_2021.}
Consider the dataset $\{W_i\}_{i=1}^n$ with $W_i = (M_i, R_i, D_i, A_i, X_i, U_i)$, where $M_i \in \mathcal{M} \times \{0\} \subseteq \mathbb{R}$ is an outcome of interest, $R_i \in \mathcal{R} \subseteq \mathbb{R}$ is the running variable, $D_i \in \mathcal{D} = \{0,1\}$ is a treatment indicator, $A_i \in \{0,1\}$ is the annotation indicator, $X_i \in \mathcal{X} \subseteq \mathbb{R}^k$ are observed covariates, and $U_i \in \mathcal{U} \subseteq \mathbb{R}^\ell$ are unstructured data. We assume $W_i \overset{\text{iid}}{\sim} P$ for some joint distribution $P$, and that $[(R_i, D_i) \perp\!\!\!\perp A_i] \mid (X_i, U_i, M_i^*)$.
Potential outcomes $\{M_i^d\}_{d\in\mathcal{D}}$ are related to observed outcomes via the assumption of consistency: \[ M_i = D_i M_i^{d=1} + (1-D_i) M_i^{d=0}. \]
We are interested in the estimand \[ \theta = E\left[{M_i^*}^{d=1}-{M_i^*}^{d=0} \mid R_i \in \mathcal{B}\right] \] for a set (or “window”) $\mathcal{B} \subset \mathcal{R}$. Under Assumptions LR1 and LR2 of cattaneo_practical_2024, which enforce that local randomization holds in $\mathcal{B}$, we can write $\theta$ as \[ \theta = E[M_i^* \mid R_i \in \mathcal{B}, D_i=1] - E[M_i^* \mid R_i \in \mathcal{B}, D_i=0]. \]
Let $\tilde X_i := (X_i, U_i)$. Then, by Proposition (ref), noting that both terms are MAR-S mean functionals with $\tilde M_i^* = g(M_i^*, \mathbb{I}\{R_i \in \mathcal{B}, D_i=d\}P(R_i \in \mathcal{B}, D_i=d)^{-1})=\mathbb{I}\{R_i \in \mathcal{B}, D_i=d\}P(R_i \in \mathcal{B}, D_i=d)^{-1} M_i^*$, we can identify $\theta$ as $\theta = \theta_{1} - \theta_{0}$ where \[ \theta_{d} = E[\mu_d(\tilde X_i) \mid R_i \in \mathcal{B}, D_i=d], \] with $\mu_d(\tilde X) = E[M_i\mid R_i \in \mathcal{B}, D_i=d,A_i=1, \tilde X_i]$.
From Proposition (ref), the efficient influence function for $E_P[\mu_d(\tilde X_i) \mid R_i \in \mathcal{B}, D_i=d]$ is \[ \varphi_d(W_i) = \frac{\mathbb{I}\{R_i \in \mathcal{B}, D_i=d\}}{P(R_i \in \mathcal{B}, D_i=d)} \left[ \mu_{d}(\tilde X_i) + \frac{A_i}{\pi(X_i)}(M_i-\mu_{d}(\tilde X_i)) \right]- \theta_{d}. \] Similarly to the DiD application, we form efficient one-step estimators for $\theta_1$ and $\theta_0$ as
where again it is assumed the marginal probabilities $ P(R_i \in \mathcal{B}, D_i=1), P(R_i \in \mathcal{B}, D_i=0)$ are known or can be appropriately treated as known (though this can be relaxed). We can combine them to form an efficient estimator of $\theta$ as \[ \hat\theta = \hat\theta_1 - \hat\theta_0, \] just as in the DiD application.
The MAR-S framework can be extended to address aggregated and transformed imputations and class-imbalanced data, which arise frequently in empirical applications.
The baseline MAR-S framework—and other methods for debiasing black-box AI estimates—are conceptually straightforward. However, they leave common empirical applications unaddressed.
In particular, a bedrock assumption of this literature is that ground truth data are available for the imputed variable(s) used in the relevant estimating equation. This often fails in practice because ground truth are available at the granular level of individual texts or images, whereas the variable of interest is a (potentially non-linear) function of granular missing data. For example, missing structured data might be imputed for millions of individual social media posts or newspaper articles, and then (non-linearly) aggregated over time or across space. This scenario is pervasive in empirical economics, where unstructured data are typically observed for individual texts or images, while other variables in the analysis are often aggregated across geography, time, firms, or other dimensions. In some cases, the missing variable of interest may itself be best represented as a functional of granular missing structured data, e.g., a population-level mean, for which ground truth is not observed in any finite sample.
Consider the dataset $\{W_{ij}\}_{i\in [N_j], j\in [n]}$, $W_{ij}\overset{\text{iid}}{\sim} P_1$, $N_j \overset{\text{iid}}{\sim}P_2$, where $i$ indexes the granular, “record” level at which the unstructured data live, and $j$ indexes the level of aggregation. Let $W_{ij} = (M_{ij}, A_{ij}, X_{ij}, U_{ij})$, with variable of interest $M_{ij} \in \mathcal{M} \times \{0\} \subseteq \mathbb{R}$, annotation indicator $A_{ij} \in \{0,1\}$, observed covariates $X_{ij} \in \mathcal{X} \subseteq \mathbb{R}^k$, and unstructured data $U_{ij} \in \mathcal{U} \subseteq \mathbb{R}^\ell$. Suppose that Assumptions (ref) and (ref) hold at the “record” level $i$ and that the functional of interest is a descriptive mean for the distribution of structured data aggregated at the level $j$: \[ \theta = E[M_j^*], \quad M_j^* = \alpha\left(\{M_{ij}^*\}_{i=1}^{N_j}\right),\] where $\alpha$ is an operator that returns a scalar. This missing data functional is irregularly identified khan_irregular_2010 if we do not observe ground truth for $M_j^*$, since the probability of annotating all $N_j$ records for any $j$ by chance is exceedingly small when $N_j$ is large. However, if the aggregation is some known linear combination of the record-level structured data, we can recover regular point identification. This setting is described by the following assumption.
If Assumption (ref) holds, by the law of total expectation we have that
allowing us to write \[ \theta = \theta_{1} \theta_{2},\quad \theta_{1} := E\left[\omega_j N_j \right], \quad \theta_{2} := E\left[M_{ij}^* \right]. \] This intuitive decomposition of \(\theta\) can be interpreted as the product of the expected aggregate level cardinality \(\theta_{1}\) and the expected record-level mean \(\theta_{2}\). Importantly, $\theta_2$ is regularly point identified, because we observe ground truth at the record level. If $\omega_j = 1$ (we are interested in sums) then $\theta = E[N_j]E[M_{ij}^* ]$. If $\omega_j = 1/N_j$ (we are interested in averages), then $\theta = E[M_{ij}^* ]$.
As long as Assumptions (ref) and (ref) also hold at granular record level $i$, the results from Theorem (ref) show that an efficient estimator for $\theta_{2}$ is \[ \hat \theta_{2} = \frac{1}{|\mathcal{I}|}\sum_{ij \in \mathcal{I}} \left[ \hat\mu(\tilde X_{ij}) + \frac{A_{ij}}{\pi(X_{ij})}(M_{ij} - \hat\mu(\tilde X_{ij})) \right]\] where $\hat\mu(\tilde X_{ij}) := \hat E[ M_{ij} \mid A_{ij}=1, \tilde X_{ij}]$. We can efficiently estimate $\theta_{1}$ with a plug-in method, and these two estimates can be combined to produce an efficient estimator $\hat\theta$ for $\theta$, using identical logic to that of the estimators in the linear regression and linear IV examples.
Unfortunately, this leaves many scenarios unaddressed, as imputed structured data are often aggregated and then non-linearly transformed (e.g., via a logarithm). While it is sometimes reasonable to approximate a transformation linearly—e.g., with a Taylor expansion (Appendix Section (ref))—this approach quickly becomes cumbersome. Moreover, it does not address scenarios where the missing variable of interest is conceptualized as a population level mean.
In the common setting where a researcher is interested in running a regression analysis with a regressor that is a functional (or aggregation) of granular missing structured data, we develop a more broadly applicable method for using MAR-S, which readily handles non-linear transformations by a simple application of the delta method. Conceptually, this approach closely parallels estimation of coefficients in linear models with regressors estimated using randomly sampled survey data deaton_panel_1985, fuller_measurement_1987. Unlike the random surveys considered by Deaton, neural networks cannot be assumed to generate classical measurement error. Importantly, however, valid debiasing ensures that the noise remaining after debiasing is classical.
We can use MAR-S to create debiased first-step estimates of the regressors of interest at the relevant aggregation (e.g., mean annual economic policy uncertainty, estimated from newspaper article level imputations). Our estimates of functionals of $M^{\ast}$ may be quite noisy if the imputation function performs poorly, but valid debiasing ensures that only classical measurement error remains in the debaised aggregates. These measures can be used directly in regression analyses, correcting for attenuation bias from the remaining classical measurement error using well-trodden methods that accommodate common empirical scenarios.
To start, consider the dataset $\{W_i\}_{i=1}^n$, where $W_i \overset{\text{iid}}{\sim} P$, and $W_i = (X_i^*, Y_i)$. $Y_i \in \mathcal{Y} \subseteq \mathbb{R}$ is an outcome of interest and $X_i^* \in \mathcal{X} \subseteq \mathbb{R}^K$ are regressors. Without loss of generality, let $E[W_i]=0$ (both variables are centered). We consider the linear model \[ Y_i = X_i^{*\mathtt{T}} \beta + \varepsilon_i, \quad E[\varepsilon_i\mid X_i^*] = 0. \] This model assumes that the conditional expectation of $Y_i$ given $X_i^*$ is linear. However, even if this assumption does not hold, $\beta$ still represents the coefficients of the best linear approximation to the true conditional expectation function, an interpretation that many researchers find suitable for their regression applications. Dropping subscripts, we can write $ Y = X^* \beta + \varepsilon, $ where $X^*$ is a $n \times K$ matrix, $\beta$ is a $K \times 1$ vector, and $Y$ and $\varepsilon$ are each $n \times 1$ vectors.
In particular, we consider the case where the $k$-th regressor for all units $i$, $X_{i,k}^*$, is unobserved and has been estimated with a MAR-S first-step, $X_{i,k}$. Accordingly, the MAR-S framework ensures that: \[ X_{i,k} = X_{i,k}^* + \eta_{i,k}, \quad (\varepsilon_i, X_{i,k}^*) \perp\!\!\!\perp \eta_{i,k}, \quad \eta_{i,k} \overset{d}{\approx} N(0, \sigma^2_{\eta,i,k} ), \quad\sigma^2_{\eta,i,k} := |\mathcal{I}_{i,k}|^{-1}\text{Var}(\varphi)_{i,k}. \] This setup nests the case where only some regressors are generated by a MAR-S first-step, in which case $\sigma^2_{\eta,i,k}=0$ for non-MAR-S regressors.
Consider the case of homogeneous error variances: $ \sigma^2_{\eta,i,k} = \sigma^2_{\eta,k}$ for all $i$. Homogeneity is straightforward to relax (Appendix Section (ref)). This setup corresponds to a typical setting where the same functional is being estimated across $i$ using data from highly similar populations with similar estimation sample sizes.
We will therefore also assume \[ \eta_i \overset{\text{iid}}{\sim} N(0, \Sigma), \quad \Sigma := \text{diag}(\sigma^2_{\eta,1}, \dots, \sigma^2_{\eta,K}), \] given two further considerations: (a) each MAR-S first-step estimate of \( X_{i,k} \) is typically derived from a separate sample, and (b) for large \( \mathcal{I}_{i,k} \), a normal approximation is expected to be accurate. We treat \( \Sigma \) as known, which is reasonable when the sample sizes \( \mathcal{I}_{i,k} \) are sufficiently large.
Under this model, note that $\beta$ satisfies the GMM moment condition \[ E\left[g(X_i, Y_i, \beta)\right] = 0, \quad g(X_i, Y_i, \beta) = X_i(Y_i - X_i^\mathtt{T} \beta) + \Sigma \beta. \] Correspondingly, the classic, plug-in GMM estimator is given by \[ \hat \beta := \left(\frac{1}{n}\sum_{i=1}^nX_i X_i^\mathtt{T} - \Sigma\right)^{-1} \left( \frac{1}{n}\sum_{i=1}^nX_i Y_i\right) = (X^\mathtt{T}X - n \Sigma)^{-1} X^\mathtt{T}Y, \] and possesses the usual just identified GMM asymptotic distribution, under additional mild regularity conditions: \[ \sqrt{n} (\hat \beta - \beta) \xrightarrow[d]{} N(0, G^{-1} \Xi G^{-1}), \] where $G = E[\nabla_\beta g(X_i, Y_i, \beta)] = -(E[X_i X_i^\mathtt{T}] - \Sigma)$, and $\Xi = \text{Var}(g(X_i, Y_i, \beta))$.
Using Isserlis' theorem, we can also re-write the asymptotic variance as \[ \text{AVar} (\hat\beta) = \underbrace{\Omega^{-1}E\left[\varepsilon_i^2 X_i^* X_i^{*\mathtt{T}}\right] \Omega^{-1}}_{\text{A}} + \underbrace{\Omega^{-1}\Pi\Omega^{-1}}_{\text{B}} \] where \[ \Omega = E[X_i^* X_i^{*\mathtt{T}}],\quad \Pi := E[\varepsilon_i^2] \Sigma +\beta^{\mathtt{T}} \Sigma \beta\left(\Omega+\Sigma\right)+ \Sigma \beta \beta^\mathtt{T} \Sigma, \] which is the usual sandwich asymptotic variance of the OLS estimator without measurement error (term A), plus an inflation factor (term B) which becomes smaller the more precise our MAR-S estimates are. (In particular, note that $\Pi = 0$ if $\Sigma = 0$, as we would expect.)
We can form consistent plug-in estimators for all of these quantities, such that \[ \hat V^{-1/2} \sqrt{n} (\hat \beta - \beta) \xrightarrow[d]{} N(0, I) \] where \[ \hat V = \hat G^{-1} \hat \Xi \hat G^{-1} = (\hat E[X_i X_i^\mathtt{T}] - \Sigma)^{-1}\widehat{\text{Var}}(X_i(Y_i - X_i^\mathtt{T} \hat\beta)+\Sigma \hat\beta)(\hat E[X_i X_i^\mathtt{T}] - \Sigma)^{-1}. \] Therefore, we are able to produce asymptotically valid confidence intervals with desired coverage.
This approach is conceptually straightforward and enables the application of MAR-S to common empirical settings that lie outside existing frameworks. It extends naturally to clustered data by invoking the appropriate clustered central limit theorem and readily adapts to panel data, the original context in deaton_panel_1985. Several additional extensions are also straightforward, such as accommodating settings where the outcome variable is also estimated via a MAR-S first-step; relaxing the assumption of normally distributed measurement error; or addressing cases where $\Sigma$ must be estimated rather than assumed to be known fuller_measurement_1987.\footnote{For an alternative framing of this regression-centric method entirely in terms of nonparametrically identified functionals, see Appendix Section (ref).}
Unlike in observational causal inference, where the propensity score function is typically unknown, researchers are able to design the annotation score function when they are the ones labeling data. While choosing the annotation score function is sometimes simple - e.g., in cases where annotating a random sample is suitable - it is less obvious when data are class-imbalanced.
In particular, the missing structured data sometimes constitute a “rare event”, particularly with massive-scale data. For example, if the dataset is the universe of all social media posts on a major platform, the share of content about almost any given topic will be small, given the vast diversity of content. With rare event estimation, the “coefficient of variation” (the ratio of the variance of the indicator of the event over the probability of the event) can be large, making confidence intervals uninformative. The literature on rare event estimation suggests annotation that incorporates some element of importance sampling to reduce the variance of estimates mcbook. Such an approach is developed in machine learning work on “batch active inference” zrnic_active_2024, which we now relate to the MAR-S framework.
In the setting of descriptive moment estimation, recall that the asymptotic variance of the estimator $\hat\theta$ is given by
where $c$ is a constant that collects all terms that, under our assumptions, do not vary with $\pi(X)$. To find the importance sampling-based annotation score function, we then solve
where $(\lambda, \kappa)$ are Karush-Kuhn-Tucker multipliers that enforce $0 \leq E[\pi(X)] \leq 1$. In light of Assumption (ref), the optimal $\pi_{IS}$ is given by \[ \pi_{IS}(X) \propto E\left[(M^* - \mu(\tilde X))^2 \mid X\right]^{1/2}. \] This annotation function is infeasible, as it depends on $M^*$, which we do not observe prior to annotation, but it provides a useful intuition: the annotation score function should place more weight on sampling data points that are harder to impute, in the sense of MSE. As advocated by zrnic_active_2024, we can consider implementing a feasible annotation score function that captures this intuition, such as \[ \pi_{FIS}(X) \propto \text{err}(X) \approx E\left[(M^* - \mu(\tilde X))^2 \mid X\right]^{1/2} \] where $\text{err}(x)$ is some proxy function that captures the uncertainty of a fixed imputation function. For example, distributional characteristics of the softmax scores of deep neural encoder outputs or verbalized confidence scores from LLMs may serve as such proxies yang_verbalized_2024. While the cited work provides valuable guidance, selecting the annotation score function in the context of highly class imbalanced datasets remains an important area for ongoing research.
Related questions about annotation include: how many labels should be collected? Is there an optimal way to do so with a limited labeling budget? Is there a power analysis that can be deployed to inform labeling practices? For consideration of optimized labeling practices with a budget see zrnic_active_2024,fisch_strat_2024,angelopoulos_cost-optimal_2025, and for power analysis see broska_mixed_2025. broska_mixed_2025 define the “effective sample size” $n_0$ in what we call a descriptive moments estimation problem as \[ n_0 := |\mathcal{I} \cap \mathcal{J}| \times \frac{|\mathcal{I}|}{|\mathcal{I} \cap \mathcal{J}|+|\mathcal{I} \cap \mathcal{J}^c|\left(1-\max\{\tilde{\rho}^2, 0\}\right)}, \] where $\mathcal{J}$ is the set of annotated indices, $\mathcal{J}^c$ is the set of unannotated indices and $\tilde\rho$ is the asymptotic correlation between the ground truth only estimator and the imputation only estimator---in this setting, $\tilde\rho:=\text{Cor}(M^*, \mu(\tilde X))$. The effective sample size $n_0$ indicates how many ground truth labels the combination of validation data and imputations effectively corresponds to. The more predictive the imputation function, the greater the effective sample size, which would equal the full size of the dataset if the imputation function were perfect. $\tilde\rho$ - which governs the power analysis - is unknown before labeling, but researchers may have priors about it or may estimate it in a “pilot” analysis.
This section illustrates MAR-S with three empirical examples. The first two examples apply MAR-S to prominent papers in the economics literature that make use of unstructured data---baker_measuring_2016 and caldara_measuring_2022---while the final example probes MAR-S design choices using a mean estimation example that builds upon annotated data collected by the authors dell_american_2023.
To be included, a study must provide a publicly available validation set for the imputed missing data, and we must be able to access all data required for replication, through replication packages and/or accessible proprietary databases (e.g., Proquest). These criteria narrow the pool substantially, as most studies where text or image data are central either lack a validation sample or the data needed for replication are not fully available. Typically, only the imputed structured data, and not the raw texts or images, are included in replication packages. As a result, we include studies that used keyword classifiers—rather than neural network classifiers—for imputation, since keyword methods dominated until recent years. We supplement the original keyword classifiers by training transformer LLM-based classifiers using the labeled data provided by the original authors.
baker_measuring_2016 introduce a quantitative index of economic policy uncertainty (EPU). The EPU index calculates the proportion of articles published in leading newspapers that discuss economic policy uncertainty at each point in time, measuring this with a keyword-based binary classifier.\footnote{For details on the keyword classifier training algorithm, see Section III.A of baker_measuring_2016.} These proportions are scaled and normalized to produce the index values.
Each entry in the EPU index can be viewed as an estimate of the probability that a newspaper article reflects economic policy uncertainty, as defined by baker_measuring_2016, at a specific point in time for a given set of newspapers. This is an inference problem where unstructured data (newspaper articles) are used to estimate a functional of missing structured data (the expectation of a binary indicator for economic policy uncertainty at a given point in time).
Consider the dataset $\{W_{it}\}_{i=1}^n$, where $W_{it} = (M_{it}, A_{it}, U_{it})$. Here, $M_{it} \in \{0,1\}$ is a binary random variable indicating the expression of economic policy uncertainty by newspaper article $i$ at time $t$, $A_{it} \in \{0,1\}$ is an annotation indicator for article $i$ at time $t$, and $U_{it} \in \mathcal{U} \subseteq \mathbb{R}^\ell$ is newspaper article text for article $i$ at time $t$. We assume $W_{it} \overset{\textrm{iid}}{\sim} P_t$, that is, the newspaper articles are an iid sample from some superpopulation of articles of interest, period by period. Moreover, in this empirical setting $(U_{it}, M_{it}^*) \perp\!\!\!\perp A_{it}$ and the annotation score $\pi$ is constant.
We may represent the EPU index's value at time $t$ as \[ \text{EPU}_t := c_t\theta_t, \quad \theta_t := E_{P_t}[M_{it}^*], \] where $c_t$ is a constant specific to time $t$ that incorporates known information on overall economic uncertainty at time $t$, as well as any other normalizing constants of interest. Written this way, it is clear that $\theta_t$ is a descriptive moment functional of the form discussed in Section (ref), and as such we know that, under Assumptions (ref), (ref), (ref), and (ref), the robust and efficient estimator of $\theta_t$ is given by \[ \hat \theta_t = \frac{1}{|\mathcal{I}|}\sum_{i \in \mathcal{I}} \left[ \hat\mu(U_{it}) + \frac{A_{it}}{\pi}(M_{it} - \hat\mu(U_{it})) \right], \] where $\hat\mu(U_{it}) := \hat E[M_{it} \mid A_{it} = 1, U_{it}]$.
We can therefore compute the “MAR-S EPU index” as $\{c_t\hat\theta_t\}_{t\in\mathcal{T}}$, for any given set of time indices $\mathcal{T} \subset \mathbb{N}$. In particular, we compute a yearly MAR-S EPU index where $\mathcal{T} = \{1985, 1986,\dots,2010\}$ (which baker_measuring_2016 term the “modern” EPU index) using two different imputation functions: the keyword classifier from baker_measuring_2016 and a deep neural classifier based on Answer.AI's ModernBERT model warner_smarter_2024, a lightweight model well suited to texts of this length. To remain faithful to the definition of economic policy uncertainty implemented by baker_measuring_2016, we leverage only the ground truth annotations that these authors created for their human audit sample and additionally randomly sample unannotated article abstracts for each time period from ProQuest at a ratio of 9 unannotated articles for every annotated article. Given that all data are iid randomly sampled, the annotation score function for our final sample is $\pi=0.1$.\footnote{As in the original paper, all data, both annotated and unannotated, are randomly sampled conditional on the text including specific keywords pertaining to the economy and to uncertainty. As such, the proper interpretation of the estimand $\theta_t$ is one that construes the superpopulation of interest as leading newspaper articles that contain specific economics and uncertainty keywords. The constants $c_t$ then modify the interpretation of the EPU index, effectively changing the denominator of the index values to be reflective of the full population of newspaper articles of interest, under the key assumption that no articles discussing EPU are devoid of the aforementioned economics and uncertainty keywords used for filtering.} We employ a simple data-splitting technique, in which half the labeled data (across all periods) is used to train and validate the imputation function and the other half is used to estimate the EPU index. Cross-fitting could be used to further improve precision, although care must be taken in how training and validation data for the classifier are partitioned from estimation data bach_hyperparameter_2024.
The ModernBERT large model is small relative to modern decoder LLMs (396 million parameters) and obtained an accuracy of 81.5% in the estimation sample. The baker_measuring_2016 keyword classifier, by contrast, achieves an accuracy of 65.3% in the estimation sample.
In Figure (ref), we plot both MAR-S EPU indices along with counterpart EPU indices that have not been debiased, which we write as $\{c_t\tilde\theta_t\}_{t\in\mathcal{T}}$, where $\tilde \theta_t = \frac{1}{|\mathcal{I}|}\sum_{i \in \mathcal{I}} \hat\mu(U_{it})$. We compute 95% confidence intervals using consistent, plug-in estimators of the asymptotic variances based on the known form of the efficient influence function.
Figure (ref) illustrates that the confidence intervals for the EPU index are substantially wider if one properly accounts for the significant uncertainty due to correction for systematic measurement error. The unadjusted EPU index confidence intervals are only generically valid if one is interested in inference on $E[\hat \mu(U_{it})]$ for arbitrary $\hat\mu$, as opposed to $E[M_{it}^*]$, or if one assumes that $\hat\mu$ is an unbiased estimator of $\mu$, which is seemingly rejected by apparent, though often mild, downward bias in many years of the index.
The more accurate the imputation function is, the narrower the confidence intervals will be for a given $n$. Hence, the MAR-S EPU index based on a deep neural classifier (81.5% accuracy) has shorter confidence intervals than the MAR-S EPU index based on the baker_measuring_2016 keyword classifier (65.3% accuracy). While we do not show estimates that use only the ground truth data - to avoid cluttering the graph - neural MAR-S EPU estimates have shorter confidence intervals on average across years (average width of 19.11 versus 21.58), whereas keyword MAR-S EPU estimates have longer confidence intervals on average across years (average width of 25.26 versus 21.58). When a predictor is poor enough (as with a weakly performing keyword classifier, where we cannot assume universal consistency), one may be better off using ground truth labels alone. This said, a massive literature on supervised learning documents many settings where neural networks perform extremely well - with sufficient training data and a straightforward, well-defined task - thereby offering substantial gains in precision. The number of labels required when using ground truth only can also become very large in some applications - e.g, suppose we wanted an EPU index measured at the year x county level using thousands of local newspapers - making highly predictive neural networks a practical requirement to tackle various questions.
baker_measuring_2016 use the EPU index in regression analyses.\footnote{Another application of the EPU index is to provide real-time tracking of economic policy uncertainty. In such applications, since it is not possible to place positive probability on annotating texts that will be created in the future, valid debiasing requires additional assumptions, such as assuming future texts are drawn from the same distribution (conditional on observables) as the annotated data.} We now briefly revisit a representative regression, reanalyzing the baseline specification in Table IV, column (5): \[ \Delta \text{Emp}_{it} = \beta\Delta \log \mathrm{EPU}_t \times \text{intensity}_{it} + \gamma\Delta \frac{\text {Federal purchases}_t}{\text {GDP}_t} \times \text{intensity}_{it} + \alpha_i + \psi_t + u_{it}, \] where $i$ indexes firms and $t$ indexes years; $\alpha_i$ and $\psi_t$ are firm and year fixed effects, respectively; $\text{intensity}_{it}$ is a firm-year policy exposure intensity variable, $\Delta \text{Emp}_{it}$ is a firm-year employment growth variable, $\Delta \frac{\text {Federal purchases}_t}{\text {GDP}_t}$ is a relevant control variable; and $u_{it}$ is an error term. The estimand of interest is $\beta$.
We apply the framework for aggregated and transformed structured data (Section (ref)) - specifically, the extended framework for independent but not identically distributed measurement error outlined in Appendix Section (ref) - to recover valid inference on $\beta$. To elaborate, per MAR-S, we have asymptotic normality of the first-step estimator: $\sqrt{|\mathcal{I}_t|} \left(\hat \theta_t - \theta_t \right) \xrightarrow[d]{} N(0, \text{Var}(\varphi_t))$, where $\varphi_t$ is the asymptotic variance of the EIF. Since $\text{EPU}_t := c_t \theta_t$ (the MAR-S debiased mean times a known constant), $\sqrt{|\mathcal{I}_t|} \left(\widehat{\text{EPU}}_t - \text{EPU}_t \right) \xrightarrow[d]{} N(0, c_t^2\text{Var}(\varphi_t))$. Using the delta method $\sqrt{|\mathcal{I}_t|} \left(\log\widehat{\text{EPU}}_t - \log \text{EPU}_t \right) \xrightarrow[d]{} N(0, \theta_t^{-2}\text{Var}(\varphi_t))$.
Motivated by this central limit theorem result, we assume that \[ \log\widehat{\text{EPU}}_t = \log\text{EPU}_t + \tilde \eta_t, \quad \tilde \eta_t \overset{\text{ind}}{\sim} N(0, \sigma_t^2), \] where $\sigma_t^2 := |\mathcal{I}_t|^{-1}\theta_t^{-2}{\text{Var}}(\varphi_t)$.
Then, by the independence of MAR-S measurement errors, we also have that \[ \Delta \log\widehat{\text{EPU}}_t = \Delta \log\text{EPU}_t + \eta_t, \quad \eta_t \overset{\text{ind}}{\sim} N(0, \sigma_t^2 + \sigma_{t-1}^2). \] And \[ \Delta \log\widehat{\text{EPU}}_t \times \text{intensity}_{it} = \Delta \log\text{EPU}_t \times \text{intensity}_{it} + \nu_{it}, \] where $\nu_{it} := \eta_t \times \text{intensity}_{it}$.
Measurement error in the interacted MAR-S-first step regressor, $\nu_{it}$, is not strongly classical, since $\nu_{it}$ and $\Delta \log\text{EPU}_t \times \text{intensity}_{it}$ share $\text{intensity}_{it}$. Fortunately, the measurement error is still weakly classical, since $\eta_t $ is mean zero and independent of $\Delta \log\text{EPU}_t$ schennach_measurement_2022. Accordingly, identification of $\beta$ still holds, though
We assume that $E[\text{intensity}_{it}^2]$ varies over time, and that $E[\text{intensity}_{it}^2]$ for any given $t$ is effectively known, given the size of the panel.
As such, our final estimator is given by \[ \hat \beta = ( X^\mathtt{T} X - n\bar\Sigma)^{-1} X^\mathtt{T}Y, \] where $X$ is a (residualized) design matrix and $\bar \Sigma := n^{-1}\sum_{it}\Sigma_{it}$.
The left side of Figure (ref) plots $\hat \beta$ using the MAR-S EPU index and measurement error-corrected least squares (ME-LS). Confidence intervals use firm level clustering, per baker_measuring_2016. The right plots OLS estimates, when the MAR-S EPU index is used (ignoring the remaining classical measurement error) and when the unadjusted EPU index is used (ignoring all measurement error).\footnote{Note that the coefficient estimate based on the unadjusted, keyword-imputation-based EPU index and the OLS estimator produced in Figure (ref) differs somewhat from the conceptually comparable estimate reported in baker_measuring_2016. This is because the regression results reported in this table are: (1) based on a sample of data from 1985-2010, as opposed to 1985-2012, as in the original paper, given what can currently be accessed through our university's Proquest subscription; and (2) the EPU index deployed in this paper is based on deploying the keyword classification algorithm of baker_measuring_2016 in the validation sample collected for this paper, and as such the final keywords in the classifier differ slightly from those of the original EPU index.}
The estimates that use OLS with the debiased EPU index show the expected attenuation bias relative to the ME-LS estimates. The unadjusted EPU index - which ignores measurement error altogether - contains both this random component (that attenuates estimates) and a systematic component (see Figure (ref)). These happen to roughly offset each other - and hence the ME-LS estimates have a similar mangitude to the coefficients from an OLS regression using the unadjusted index - but this will not be true generically.
The more precisely estimated MAR-S EPU index, which was constructed via deep neural network imputation, yields a narrower confidence interval for the coefficient of interest than the noisier keyword-based MAR-S EPU index, and does not include zero, showing the returns to more accurate imputation. Standard errors for the uncorrected estimates are understated, compared to those that account for measurement error in imputation.
An intellectual successor to baker_measuring_2016, caldara_measuring_2022 construct a quantitative index measuring geopolitical risk (GPR). Similar to the EPU index, the GPR index at each point in time is calculated as the (normalized) share of articles from prominent newspapers that discuss rising geopolitical risks, based on a detailed keyword-based text query.\footnote{The query is Boolean and can be interpreted as a binary keyword-based classifier. However, unlike the EPU index, there is no explicit training algorithm: caldara_measuring_2022 constructed the query using data external to the analytic sample and did not update it based on the analytic sample. For the full query and details of its construction, see Section A.3 of the online appendix in caldara_measuring_2022.} The GPR index estimates the probability that a newspaper article discusses rising geopolitical risks, at a specific point in time for a given set of newspapers. This involves using newspaper text to estimate a functional of missing structured data (the expected value of a binary indicator for articles discussing geopolitical risk).
We follow an identical formal setup to the one described in the previous section and produce GPR indices (for $\mathcal{T} = \{1900, 1901,\dots,2015\}$), with and without application of the MAR-S framework. We again form multiple indices based on two distinct imputation functions: the original keyword query for GPR, and a deep neural classifier based on Microsoft's DeBERTaV3 model he_debertav3_2021.\footnote{Specifically, we use the large variant.} As with the EPU index, we remain faithful to the definition of geopolitical risk crafted by caldara_measuring_2022 by leveraging only the ground truth annotations that were created for their human audit sample. The labeled data from this human audit sample is iid randomly sampled from a large universe of ProQuest articles specified by caldara_measuring_2022, and we use the entirety of this universe of articles as unlabeled data for imputing the index.\footnote{The universe of articles in question conditions on first meeting certain keyword inclusion requirements; this influences the interpretation of downstream estimands and estimates, and motivates correction by a known period-by-period constant, under certain assumptions. See appendix B.6 of caldara_measuring_2022 for more details. The annotation score function using this universe is $\pi=0.00067$.} We employ an identical data splitting procedure as for the MAR-S EPU indices. In the estimation sample, the lightweight deep neural classifier achieves an accuracy of 81.0%, and the keyword query achieves an accuracy of 66.2%, using the human audit annotations as ground truth.
Figure (ref) plots the MAR-S GPR and unadjusted GPR indices, constructed using neural and keyword classifiers. The uncorrected GPR keyword index systematically and substantially underestimates geopolitical risk relative to the ground truth sample, as many articles in the audit sample discuss rising geopolitical risk but do not contain the requisite keywords. While level differences are large, variation across all indices is driven primarily by the World Wars (and September 11th, to a lesser extent). Confidence intervals are (erroneously) much narrower for the non-MAR-S indices than for the MAR-S indices. Moreover, the more accurate neural classifier generates narrower MAR-S confidence intervals than the keyword classifier.\footnote{Compared to estimates that only use the ground truth labels, the neural MAR-S GPR estimates are on average shorter (average width of 0.063 versus 0.069) and the keyword MAR-S GPR estimates are on average longer (average width of 0.082 versus 0.069).} Regression analyses in caldara_measuring_2022 could be approached similar to the example above from baker_measuring_2016, by using ME-LS with the debaised GPR index.
In supplementary materials, caldara_measuring_2022 show robustness of their target estimates to different keyword queries that make different tradeoffs between recall and precision, a common approach to address researcher degrees of freedom. Using neural networks significantly expands researcher degrees of freedom, relative to a world where data are created by some external source. Correcting for imputation bias addresses concerns that different choices (e.g., using a different prompt, model, training data set, or choice of hyperparameters) will lead to different estimates, by producing valid point estimates and confidence intervals as long as the relatively weak MAR-S assumptions are met. We view this as a significant advantage, as it is much easier to interpret - and plausibly easier to produce - a single debiased estimate than a lengthy appendix exploring different first-step estimators.
To explore design choices within the MAR-S framework, we use the familiar setting of mean estimation, developing a third example focused on estimating the share of articles about politics in local U.S. newspapers. Most annotated datasets in economics studies are small, so we instead use a dataset created to study machine learning methods dell_american_2023. Articles were randomly selected for annotation from a large-scale dataset of historical U.S. newspapers. Politics is examined because it is one of the few topics common enough for simple random sampling to yield a relatively balanced annotation sample. Here, $M_i \in \{0,1\}$ is a binary indicator for whether a newspaper article discusses politics. Under Assumptions (ref)-(ref), the efficient estimator for the probability that an article is about politics is given by $\hat\theta$, defined in Section (ref).
Figure (ref) plots estimates of $\hat\theta$ based on models trained and debiased with an increasing number of annotations. The total annotated dataset is size $n=4157$, and estimates use 5%, 10%, 20%, 50%, and 100% of this annotated data, respectively, evenly split between model training and debiasing. All estimates are produced using the ModernBERT base model (further pretrained with the Nomic Embed framework) warner_smarter_2024,nussbaum_nomic_2024.
Regardless of the size of the annotated data, point estimates appear unbiased. Confidence intervals are very large for the smallest sample of annotated data. Larger annotation sets lead to narrower confidence intervals, reflecting accuracy improvements in the classifier when more labeled training data are available, as well as a larger sample size for debiasing. However, beyond a modest amount of labeled data, the improvements in precision (and classifier accuracy) are fairly marginal. As discussed in Section (ref), the effective number of observations will depend on the asymptotic correlation between the ground truth only and imputation only estimates.
Figure (ref) plots imputation function accuracy alongside point estimates and confidence intervals for the share of articles discussing politics, using various text classification models and the 20% annotation dataset from Figure (ref) for training and debiasing. The rest of the annotated dataset serves as the “unlabeled data”, allowing comparison of the MAR-S estimates to a ground truth oracle. Specifically, we examine performance and estimates from three zero-shot classification approaches—the keyword classification strategy from dell_american_2023, a zero-shot encoder transformer following yin_benchmarking_2019, and a popular open-source generative AI model (the smallest Llama 3.1 instruction-tuned decoder transformer grattafiori_llama_2024)—as well as two neural encoder models fine-tuned on a split of the annotated data: the original BERT encoder transformer devlin_bert_2019 and a successor, ModernBERT warner_smarter_2024. The fine-tuned methods perform better, translating into narrower confidence intervals but falling short of the oracle's precision. How much researchers should invest in improving predictor accuracy will depend on their precision requirements.
Unstructured data holds remarkable potential for enriching social science research. However, neural networks, while powerful, do not generically produce unbiased predictions. Their accuracy can often be improved by gathering more or higher-quality training data and increasing model size, but such efforts are costly. To assess when predictions are sufficiently accurate—and to draw credible conclusions using them—researchers must take prediction error seriously. This also plays a crucial role in addressing serious concerns about researcher degrees of freedom and reproducibility that arise when researchers use neural networks to impute structured data.
The MAR-S framework addresses these challenges by framing the analysis of unstructured data as a missing structured data problem. This approach enables the recovery of $\sqrt{n}$-consistent estimators with valid uncertainty quantification and offers a foundation for building estimators that are both efficient and robust. We further extend this framework to settings where the parameter of interest is a (potentially nonlinear) function or functional of missing structured data, and where ground truth is observed only at the granular level. This scenario---overlooked in the existing literature---is common in empirical economics because imputed structured data (and corresponding ground truth annotations) are typically at the level of individual texts or images, while other variables of interest are available only at a coarser level of aggregation.
Crucially, viewing inference with unstructured data through the lens of missing data requires researchers to clearly define the low-dimensional summaries they wish to predict. While there are many interesting applications of unsupervised machine learning - where there is by definition no ground truth - MAR-S makes clear that being explicit when possible about measurement goals and rigorous in their implementation conveys significant benefits. It allows researchers to make only very weak assumptions about the DGP and facilitates the statistically principled use of deep neural networks for estimation. Moreover, specifying and evaluating measurement goals guards against the risk of generating massive volumes of data without thinking carefully about how they will inform the underlying social science questions, an increasingly tempting proposition as the costs of data generation plummet. As unstructured data become more central to empirical work, MAR-S offers a practical path toward credible and interpretable inference that applies to a wide range of common empirical scenarios.
\begingroup \singlespacing \endgroup
\setcounter{table}{0} \setcounter{figure}{0} \setcounter{section}{0}