Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
131,261 characters · 29 sections · 115 citation commands
Inference for Regression with Variables Generated by AI or Machine Learning
}
JEL Codes: C11, C51, C55
Keywords: Measurement Error, Artificial Intelligence, Large Language Models, Topic Models, Inference
\thispagestyle{empty} \pagenumbering{arabic}
\onehalfspacing
Economists now routinely use artificial intelligence (AI) or machine learning (ML) algorithms to generate new variables. These technologies are used to quantify unstructured data such as text and images, to measure subtle concepts like uncertainty and sentiment, and to create new data sets of variables that were previously too costly, labor-intensive, or otherwise infeasible to collect.
The variables generated by AI and ML algorithms are rarely of interest in themselves, but rather are used in econometric models to address questions of cause and effect, produce forecasts, or estimate counterfactuals. In pioneering work, bakerMeasuringEconomicPolicy2016 quantifies economic policy uncertainty from news text and uses it as a covariate in regressions and vector autoregressions (VARs). In more recent examples, magnolfiTripletEmbeddingsDemand2025 measures product differentiation from survey data and compianiDemandEstimationText2025 measures product substitutability with text and images from online platforms. Both papers then use the derived measures in demand models. gorodnichenkoVoiceMonetaryPolicy2023 measures tone-of-voice from audio recordings of central bank press conferences, then runs predictive regressions of financial variables on tone. gabaixAssetEmbeddings2023 imputes firm characteristics from investor holdings data and uses them to explain asset returns. einavProducingHealthMeasuring2024 measures patient health status from surveys then uses it in an econometric model of nursing home value added. vafaDecomposingChangesGender2023 measures labor market experience from CVs and uses it to study the gender wage gap.
In standard practice, AI- or ML-generated variables are treated as regular numerical data when estimating and performing inference in downstream econometric models. We refer to this as the two-step strategy: variables are generated in the first step, then used as covariates in the second step. While a pragmatic initial approach, the two-step strategy has largely unknown statistical properties. One natural concern is that estimators are biased due to measurement error in the AI/ML-generated variables. Another is that inference suffers from a generated regressor problem paganEconometricIssuesAnalysis1984. Conversely, results in the time-series literature suggest that plugging in estimated variables need not lead to inference problems stockForecastingUsingPrincipal2002,bernankeMeasuringEffectsMonetary2005,baiConfidenceIntervalsDiffusion2006. Without a coherent framework for analyzing the problem, it is difficult to assess which of these perspectives is correct. More generally, characterizing the statistical guarantees\textemdash or lack thereof\textemdash of the two-step strategy is an important step in developing reliable inference methods for working with variables generated by AI or ML, an area that is still very much in its infancy.
This paper makes two main contributions. First, we show formally that the two-step strategy can lead to invalid inference on downstream regression parameters, even in modern settings where high-performance algorithms are deployed on large data sets. Second, more constructively, we propose two methods for valid inference: (i) bias-corrected confidence intervals, and (ii) joint estimation of the regression coefficients and latent variables. We document the performance of the methods in several empirical settings.
We consider a downstream regression where the outcome variable $Y_i$ depends on vectors of latent variables $\boldsymbol\theta_i$ and observed variables $\mathbf{q}_i$. For each observation, the researcher also has an unstructured or high-dimensional data set $\mathbf{x}_i$ for estimating $\boldsymbol\theta_i$. To accommodate many scenarios, we stay agnostic on the form that $\mathbf{x}_i$ takes. For instance, it may be a sequence of words with textual data, an array of RGB values with image data, or a sequence of amplitudes with audio data. One scenario is label imputation, where $\boldsymbol\theta_i$ is a vector of binary labels (e.g., race or gender indicators). In this case, $\mathbf{x}_i$ (e.g., images) is used as an input to a classifier which produces predicted values $\hat{\bs \theta}_i$. Another scenario is dimensionality reduction, in which a low-dimensional representation (or embedding) $\hat{\bs \theta}_i$ of $\mathbf{x}_i$ is generated with an unsupervised learning model, as in hansenTransparencyDeliberationFOMC2018, bybeeBusinessNewsBusiness2024, and ashMoreLawsMore2025 to name a few. A third scenario is index construction. For example, $\mathbf{x}_i$ could be a set of texts (e.g., articles, paragraphs, or sentences) which are individually classified as containing positive or negative sentiment, then aggregated and normalized to produce a sentiment score $\hat{\bs \theta}_i$, as in bakerMeasuringEconomicPolicy2016, caldaraMeasuringGeopoliticalRisk2022, and gorodnichenkoVoiceMonetaryPolicy2023.
In the two-step strategy, the researcher first computes an estimate $\hat{\boldsymbol\theta}_i$ of $\boldsymbol{\theta}_i$ from $\mf x_i$ for each observation, then regresses $Y_i$ on $\hat{\boldsymbol\theta}_i$ and $\mathbf{q}_i$, and reports point estimates and confidence intervals using standard OLS methods (i.e., treating $\hat{\boldsymbol\theta}_i$ as regular numeric data). Depending on the context, one may wish to do inference on the coefficients of the latent or observed variables. In either case, the key question is whether this approach leads to valid inference.
To this end, we introduce an asymptotic framework in which the magnitude of the measurement error and sampling uncertainty remain comparable as the sample size increases. This framework delivers tractable approximations to the finite-sample distribution faced in practice, in which both sources of error play a role.\footnote{Our use of sequences of DGPs to better approximate the finite-sample behavior of estimators has a precedent in a number of contexts in economics. See, e.g., phillipsUnifiedAsymptoticTheory1987, chesherEffectMeasurementError1991, staigerInstrumentalVariablesRegression1997, and hahnAsymptoticallyUnbiasedInference2002.} It also captures the prevailing trend of analyzing increasingly large data sets with increasingly accurate algorithms.
In this framework, we derive two new results about the two-step strategy. First, the asymptotic distribution of OLS estimators has a first-order bias due to measurement error. The bias is increasing in the scale of measurement error relative to sampling uncertainty in the downstream model. Second, the asymptotic variance of the OLS estimator is the same as if $Y_i$ were regressed on the true $\boldsymbol{\theta}_i$ and $\mathbf{q}_i$. Moreover, OLS standard errors are consistent. As a result, two-step confidence intervals have the correct width but incorrect centering, making them invalid for inference. This differs from a generated regressor problem, where the variance is inflated but there is no location shift. To the extent that the empirical economics literature acknowledges the two-step strategy might be a problem, concerns typically focus on standard errors. Our analysis shows these concerns are misplaced: the primary issue is bias, not incorrect standard errors.
For the case of imputed labels, the potential for AI/ML-generated variables to bias downstream estimators has been flagged in recent work, mainly in data science and political science. See fongMachineLearningPredictions2021, allonMachineLearningPrediction2023, angelopoulosPredictionpoweredInference2023,angelopoulosPPIEfficientPredictionPowered2023, zhangDebiasingMachineLearningAIGenerated2023, zrnicCrosspredictionpoweredInference2024 miaoTaskAgnosticMachineLearningAssistedInference2024, klugerPredictionPoweredInferenceImputed2025 and sanfordAdversarialDebiasingUnbiased2025 for general ML-generated variables, and egamiUsingImperfectSurrogates2023,egamiUsingLargeLanguage2024 and ludwigLargeLanguageModels2025 for variables generated by large language models.\footnote{There is also recent work in economics that considers the complementary problem of imputed dependent variables as opposed to imputed covariates; see rambachanProgramEvaluationRemotely2025 and modarressiCausalInferenceOutcomes2025.} These works demonstrate the inconsistency of OLS estimators in settings where the magnitude of measurement error remains fixed as the sample size increases. However, this asymptotic framework isn't necessarily appropriate in modern use cases, where high-quality algorithms are deployed on large data sets. Our analysis provides a new set of results for such cases.
Furthermore, these works propose bias corrections that require a validation sample in which both the true $\bs \theta_i$ and its AI/ML-generated estimate $\hat{\bs \theta}_i$ are observed alongside $(Y_i,\mathbf q_i)$.\footnote{This approach is related to an older literature on estimation with auxiliary data chenSemiparametricEfficiencyGMM2008.} The idea is to use the validation data to estimate bias, then bias-correct estimates from the main sample in which only $\hat{\bs \theta}_i$ is available. Such an approach is possible when the researcher can, albeit at some cost, scrutinize $\mf x_i$ and assign a ground-truth $\bs \theta_i$. But in that case one could simply estimate the model on the validation data alone: the AI/ML-generated data is only useful insofar as it may help improve efficiency. More problematically, $\bs \theta_i$ is latent in most economic use cases---for instance, one never observes true policy uncertainty, risk, or sentiment---so validation data is unavailable and these existing methods are inapplicable.
Our first inference approach is based on bias correction, but unlike existing approaches it does not require validation data. Instead, we rely on our theoretical results, which characterize the first-order asymptotic bias of OLS estimators and establish consistency of two-step standard errors. This allows us to perform an analytical bias correction, then re-center the usual confidence intervals at the bias-corrected estimator to perform valid inference. Our bias corrections are general and widely applicable. We specialize them to AI/ML-generated binary labels and dimension reduction. For the former, bias-correction can be performed without validation data provided one has a measure of the classifier's expected false-positive rate.\footnote{The expected false-positive rate may be estimated from a validation sample, but it may also be available externally. To give a recent example, bursztynImmigrantNextDoor2024 uses a ML algorithm to classify charitable donors' names by ethnicity. They estimate the accuracy of the classifier using an external sample of North Carolina voter registration data which contains self-reported ethnicity (but not data on donations or other controls).} For the latter, bias correction can be performed using the estimated low-dimensional representation.
It is important to note that the measurement error in AI/ML generated variables may be “nonclassical” (i.e., correlated with the true latent $\bs \theta_i$). This makes it difficult for researchers to know even the sign of the bias ex ante: there may be attenuation or amplification. Indeed, Section (ref) shows that bias can be positive or negative in the case of index construction. Nonclassical measurement error is also much more difficult to correct for than classical measurement error, requiring specialized methods\textemdash see, e.g., schennachMeasurementSystems2022 for a discussion.
Our bias corrections are convenient to apply, but they may not be available for all types of AI or ML algorithms. They also rely on the magnitudes of measurement error and sampling uncertainty being comparable. To perform inference without validation data in settings where this is not the case, we introduce a second approach based on joint maximum likelihood estimation of the models for latent variable estimation and regression. In this approach, the model linking $\mf x_i$ with the latent $\bs \theta_i$ is analogous to an “observation equation” in state-space models. This requires some more careful modeling of how $\bs \theta_i$ and $\mf x_i$ are related, but we demonstrate its feasibility with three distinct and non-exhaustive applications: AI/ML generated binary labels, dimension reduction, and AI/ML generated indices.
While joint estimation is straightforward in theory, it presents a computational challenge due to the large number of latent $\bs \theta_i$ that must be integrated out of the likelihood. To address this, we use Hamiltonian Monte Carlo, a Markov Chain Monte Carlo algorithm that uses information on the gradient of a distribution to sample from it. Implementation is greatly simplified with the use of modern probabilistic programming languages: one simply specifies the likelihood in code, which is then “automatically” compiled to perform sampling.\footnote{Previous papers that have performed inference using the joint likelihood approach with unstructured data include gentzkowMeasuringGroupDifferences2019, ruizSHOPPERProbabilisticModel2020, and munroLatentDirichletAnalysis2022. These typically require custom code to estimate, which makes adapting the model difficult for non-specialists.}
We introduce three applications to illustrate the theoretical results. The first illustrates label imputation. hansenRemoteWorkJobs2023 uses a Large Language Model to classify each job posting in the Lightcast dataset as offering remote work or not. These imputed labels can be merged with other posting-level metadata to study the causes and consequence of remote work adoption. We focus on the relationship between wage inequality and remote work by regressing the posted wage on the remote indicator and controls. The classifier achieves a high test-set accuracy of 99%, so one might expect that measurement error is inconsequential. But our theory shows that the important quantity is measurement error relative to sampling uncertainty, and the Lightcast dataset has hundreds of millions of individual observations. We show via a case study that bias correction and joint estimation both estimate notably stronger effects of remote work on posted wages than the two-step strategy.
The second application illustrates regressors derived from dimension-reduction algorithms. bandieraCEOBehaviorFirm2020 conducts a time-use survey to document behavioral differences among CEOs and their impact on firm performance. The authors use latent Dirichlet allocation bleiLatentDirichletAllocation2003\textemdash a factor model for discrete data\textemdash to represent CEO time-use behavior in a low-dimensional space. This representation is then included as a covariate along with other firm controls in a sales regression. We replicate this two-step strategy and find the estimated impact of behavior on performance aligns with estimates obtained via bias correction and the joint estimation strategy. Our theory predicts this will hold when measurement error is low compared to sampling uncertainty. We then re-estimate the model using a 10% subsample of time units for each CEO to scale-up measurement error. Here we find the two-step strategy produces insignificant behavioral effects, while both corrections produce significant effects.
The third application illustrates index construction via classification and aggregation. Central bank communication has become a major research and policy topic over the past decade, and linking market reactions to communication often involves quantifying the latter from unstructured data. We replicate the hawkish sentiment measure from gorodnichenkoVoiceMonetaryPolicy2023 which classifies individual paragraphs of FOMC statements as hawkish or dovish. Paragraphs are then aggregated to form a meeting-level share which proxies continuous, latent sentiment. Following the two-step strategy, we regress the path factor gurkaynakActionsSpeakLouder2005\textemdash a measure of movement in the long end of the yield curve\textemdash on sentiment and find weakly positive effects. With joint estimation, however, the estimated effect size and $R^2$ are both nearly three times larger, which shows the value of our correction for prediction as well as for inference.
Finally, in simulation exercises calibrated to the empirical applications, we find that the two-step strategy performs poorly\textemdash in terms of bias in estimated coefficients and coverage of confidence intervals\textemdash relative to both bias correction and joint estimation. This provides further evidence that measurement error distorts inference in the two-step strategy, while our corrections reduce bias and restore valid inference even in challenging empirical settings.
Our overall message is that the increasingly common practice of using regressors generated by AI or ML can lead to invalid inference, but practical solutions exist. We view our proposed bias correction and joint estimation approaches as robust, widely applicable starting points for empirical analysis. For instance, an emerging line of research uses text-derived sentiment indices as inputs into forecasting models. Our analysis can be extended to show how errors in these indices lead to biased forecasts, while our solutions can improve the performance of these forecasting methods. Likewise, the industrial organization literature increasingly uses embedded representations of firms and products to model market behavior and demand. Our solutions can be adapted to these settings as well. Going forward, it is important to establish which algorithms and econometric models are most susceptible to measurement error and associated inference problems. More generally, inference problems arising from the use of AI/ML-generated variables should more widely recognized in order to fully harness the potential of AI/ML methods in empirical economics.
The rest of the paper proceeds as follows. Section (ref) provides a simple setting illustrating why the two-step strategy leads to biased inference and how our proposed solutions can help. Section (ref) introduces the more general framework and presents three empirical applications. Sections (ref) and (ref) present, respectively, the main theoretical analysis of the two-step strategy and the proposed solutions. Section (ref) presents simulation results and Section (ref) concludes.
This section presents a simple model to illustrate how the two-step strategy leads to biased inference, and how our proposed methods can restore valid inference.
The model is loosely based on bakerMeasuringEconomicPolicy2016. Suppose we are interested in the effect $\gamma$ of $\theta_i$ (policy uncertainty in month $i$) on $Y_i$ (employment or investment, say, in month $i+1$) in the regression model
Policy uncertainty is a nebulous concept that is difficult to precisely define let alone observe. bakerMeasuringEconomicPolicy2016 forms EPU indices from monthly counts of articles in ten newspapers containing certain terms, which they convert to an index. Evidently there is measurement error due to the sampling of articles: one could change the set of newspapers surveyed and obtain a quantitatively different (but related) measure.\footnote{Misclassification of articles is a second source of measurement error. We sidestep this for now for sake of exposition, but account for it in later sections.} To capture this, consider
where $C_i$ is the number of articles sampled in month $i$, $X_i$ is the number of these that relate to uncertainty, and $\theta_i$ is true policy uncertainty. We observe $X_i$, $Y_i$, and $C_i$ but not $\theta_i$. One can estimate $\theta_i$ using $\hat\theta_i = X_i/C_i$, as done by bakerMeasuringEconomicPolicy2016 (bakerMeasuringEconomicPolicy2016).
In this example, the two-step strategy computes the OLS estimate $\hat \gamma$ from regressing $Y_i$ on $\hat{\theta}_i$, then performs standard OLS inference for $\gamma$. This approach ignores the fact that $\hat\theta_i$ is a noisy estimate of $\theta_i$, potentially leading to biased estimates and invalid inference.
We use asymptotics to tractably approximate the finite-sample problem faced by the researcher, where $\hat \gamma$ is computed from $(Y_i,X_i,C_i)_{i=1}^n$. Both measurement error and sampling error affect the properties of $\hat \gamma$ in finite samples. We therefore consider a sequence of populations indexed by the sample size $n$, where the distribution of $(Y_i,X_i,\theta_i)$ conditional on $C_i$ is fixed but the distribution of $C_i$ is changing with $n$ so that
In this sequence of DGPs, the variance of $\hat \theta_i$, which is proportional to $C_i^{-1}$, is of the same order of magnitude as sampling uncertainty. The parameter $\kappa$ controls the relative importance of measurement error, with larger values of $\kappa$ giving relatively greater importance to measurement error. Working with this sequence of DGPs therefore allows us to gain insights about how $\hat{\gamma}$ behaves when both measurement and sampling error are present.
Under suitable regularity conditions (see Theorem (ref)), one can show that
where $\bar \theta_n$ is the sample mean of $\hat \theta_i$ and $\hat \epsilon_i$ is the OLS residual. The first result shows $\hat \gamma$ is asymptotically normally distributed with the same variance as if $Y_i$ was regressed on the true latent $\theta_i$, but with a centering that differs from zero when $\kappa > 0$. The second result shows OLS standard errors are consistent, irrespective of $\kappa$. Taken together, these results imply that two-step confidence intervals (given by $\hat \gamma$ $\pm$ $1.96$ times the OLS standard error) have the correct width, but incorrect centering whenever $\kappa > 0$. Moreover, the asymptotic bias of $\hat \gamma$, and therefore the degree of under-coverage of two-step CIs, is increasing in $\kappa$.
Our first proposed solution is a straightforward bias correction. This approach simply constructs an estimate of the bias and adds it back to the two-step estimator $\hat \gamma$. The bias correction follows easily from ((ref)), so the bias corrected estimator $\hat \gamma^{bc}$ is \[ \hat \gamma^{bc} = \left( 1 + \frac{\hat \kappa}{\sqrt n} \frac{\sum_{i=1}^n \hat \theta_i (1-\hat \theta_i)}{\sum_{i=1}^n (\hat \theta_i - \bar \theta_n)^2} \right) \hat \gamma , \] where $\hat \kappa = \frac{1}{\sqrt n} \sum_{i=1}^n C_i^{-1}$. Bias corrected confidence intervals are then simply $\hat \gamma^{bc}$ $\pm$ $1.96$ times the OLS standard error. See Theorem (ref) for a formal justification for this approach.
A second approach is joint maximum likelihood estimation of ((ref)) and ((ref)). This approach treats ((ref)) analogously to an observation equation in a state-space model, with $\theta_i$ as a latent variable.
We start by assuming the error terms in ((ref)) have probability density function $\sigma^{-1} f(\varepsilon/\sigma)$. Combining with ((ref)), this yields the likelihood \[ f(Y_i,X_i|C_i,\theta_i;(\gamma,\alpha,\sigma)) \propto \frac 1 \sigma f \left( \frac{Y_i - \alpha - \gamma \theta_i}{\sigma} \right) (\theta_i)^{X_i}(1-\theta_i)^{C_i - X_i}. \] Since $\theta_i$ is latent we proceed in the spirit of random effects and assume $\theta_i$ is drawn from a distribution with probability density function $g$ on $[0,1]$.\footnote{One can easily allow the distribution of $\theta_i$ to depend on covariates, as in correlated random effects. We suppress this for now, but adopt such an approach in the empirical applications.} Note that the effect of this prior will be dominated by the data as $C_i$ becomes large. We then integrate out $\theta_i$ to produce a likelihood in terms of the observed data: \[ f(Y_i,X_i|C_i;(\gamma,\alpha,\sigma)) = \int_0^1 f(Y_i,X_i|C_i,\theta_i;(\gamma,\alpha,\sigma)) g(\theta_i) \, d \theta_i. \] We estimate $\gamma$ by maximizing the log-likelihood \[ L_n((\gamma,\alpha,\sigma)) = \sum_{i=1}^n \log f(Y_i,X_i|C_i;(\gamma,\alpha,\sigma)). \] Inference is performed using standard asymptotics for maximum likelihood estimators.
In the general model, we wish to estimate and perform inference on the parameters $\bs \gamma$ and $\bs \alpha$ of the linear regression model
where $\boldsymbol{\theta}_i$ is now extended to be a vector of latent variables of interest, $\mathbf{q}_i$ is a vector of observed quantitative variables, and $\mb E \left[ \varepsilon_i (\boldsymbol{\theta}_i , \mathbf{q}_i) \right] = 0 $. For each observation $i$ we also have unstructured or high-dimensional data $\mf x_i$, from which an estimate $\hat{\boldsymbol{\theta}}_i$ of $\boldsymbol{\theta}_i$ can be derived. Thus, the researcher's dataset is a random sample $(Y_i, \mf q_i, \mf x_i)_{i=1}^n$. The parameter $\boldsymbol \gamma$ is typically the key object of interest, but in some cases (e.g., aviviArePatentExaminers2024) $\boldsymbol \alpha$ may be the focus, with $\boldsymbol{\theta}_i$ serving as a control variable derived from unstructured data.
The dominant two-step strategy can be summarized as follows:
Evidently there is a measurement error problem: the estimates $\hat{\boldsymbol{\theta}}_i$ are proxies for the true latent covariates $\boldsymbol{\theta}_i$ in ((ref)). Step (ii) overlooks this issue and treats the estimates $\hat{\boldsymbol{\theta}}_i$ as regular numeric data. This raises the possibility that two-step estimators of $\bs \gamma$ and $\bs \alpha$ are biased. Moreover, conventional standard errors and confidence intervals do not account for the additional variation arising from using $\hat{\boldsymbol{\theta}}_i$ instead of $\boldsymbol{\theta}_i$, raising the possibility of a generated regressors problem. To understand the forces at play, in Section (ref) we shall analyze the two-step strategy and formally demonstrate why it can lead to biased estimates and inference. Many specific cases can be captured by this setup. We consider three here.\footnote{The previous version of the paper considers further extensions, for example regression onto similarity measures between vector representations of unstructured data.}
\paragraph{Application 1: AI/ML-Generated Labels.} Economists now routinely use AI or ML methods to impute missing covariates. A leading use case involves regressions of an outcome $Y_i$ on a latent binary variable $\theta_i$ (e.g., indicating positive/negative sentiment of a news article or racial group membership) and observed controls $\mf q_i$.\footnote{We present the case of scalar $\bs\theta_i$ in the main text and defer the case of multiple categories to Appendix (ref).} Examples include goldsmith-pinkhamGenderGapHousing2023, adams-prasslFirmConcentrationJob2023, argyleRacialDisparitiesBias2025, and wuBehavioralResponsesEstate2024. Unstructured data $\mf x_i$ (e.g., article text or voter registration data) is often used to predict $\theta_i$ using a classification algorithm. The two-step strategy entails first generating a prediction $\hat{\theta}_i$ of $\theta_i$ then regressing $Y_i$ on $\hat{\theta}_i$ and $\mf q_i$. Here the source of measurement error is misclassification: $\hat \theta_i$ may differ from $\theta_i$ for some observations. Although sophisticated modern classifiers have low error rates, they are often used to impute missing observations for large data sets. As a result, measurement error from misclassification may be non-negligible relative to sampling error in the downstream regression, invalidating two-step inference.
\paragraph{Application 2: Topic Models.}
A large empirical literature uses topic models to reduce the dimension of unstructured data. Examples include hansenTransparencyDeliberationFOMC2018, muellerReadingLinesPrediction2018, larsenValueNewsEconomic2019, thorsrudWordsAreNew2020, adamsDeathCommitteeAnalysis2021, bybeeBusinessNewsBusiness2024, and ashMoreLawsMore2025 with text, dracaHowPolarizedAre2021, and munroLatentDirichletAnalysis2022 with survey data, and nimczikJobMobilityNetworks2017 and olivellaDynamicStochasticBlockmodel2021 with network data.
Here $\mathbf{x}_i = (x_{i,j})_{j=1}^V$ is a $V$-dimensional vector of feature counts, with $x_{i,j}$ counting the number of times feature $j$ appears in observation $i$. For instance, in text applications, $x_{i,j}$ counts the number of times word $j$ appears in document $i$. The vector $\mathbf{x}_i$ follows a Multinomial distribution with a factor structure. There are $K < V$ distributions $\boldsymbol{\beta}_1,\ldots,\boldsymbol{\beta}_K \in \Delta^{V-1}$, the $(V-1)$-dimensional simplex. Each $\boldsymbol{\beta}_k$ represents a common factor (or “topic”). Each observation $i$ is characterized by a latent vector $\boldsymbol{w}_i \in \Delta^{K-1}$. The elements $w_{i,k}$ of $\bs w_i$ represent the weight of $\boldsymbol{\beta}_k$ in generating $\mathbf{x}_i$. The count probabilities for observation $i$ are $\mathbf{p}_i = \sum_{k=1}^K \boldsymbol{\beta}_{k} w_{i,k} = \mathbf{B}^T \boldsymbol{w}_i$, where $\mathbf{B}^T = [\boldsymbol{\beta}_1,\ldots,\boldsymbol{\beta}_K]$. Combining these elements yields
where $C_i = \sum_{v=1}^V x_{i,v}$ is the total feature count for observation $i$. Finally, the sub-vector $\bs \theta_i$ of $\bs w_i$ collects the topic weights for inclusion in the regression.
In the two-step approach, $\mf B$ and $(\bs \theta_i)_{i=1}^n$ are estimated using Latent Dirichlet Allocation bleiLatentDirichletAllocation2003 or more recent methods (e.g., bingOptimalEstimationSparse2020, wuSparseTopicModeling2023, keUsingSVDTopic2022), then $Y_i$ is regressed on $\hat{\bs \theta}_i$ and $\mf q_i$. The source of measurement error is sampling error in the estimated topic weights $(\hat{\boldsymbol{\theta}_i})_{i=1}^n$, which is proportional to $C_i^{-1}$. The quantity $\mb E[C_i^{-1}]$ controls the overall rate of measurement error.
\paragraph{Application 3: AI/ML-Generated Indices.}
A third use case involves constructing indices by classification and aggregation. For instance, bakerMeasuringEconomicPolicy2016 constructs policy uncertainty indices by first classifying news articles based on whether they relate to policy uncertainty, and then aggregating the results over time into monthly or quarterly indices. See also caldaraMeasuringGeopoliticalRisk2022 and gorodnichenkoVoiceMonetaryPolicy2023, among others.
Extending the simple example of Section (ref), let $C_i$ denote the number of articles to be classified in month $i$, and let $N_i$ represent the number of articles classified as pertaining to policy uncertainty. The quantity $\hat \theta_i = N_i/C_i$ is a natural measure of the true latent uncertainty $\theta_i \in [0,1]$, where $\theta_i = 0$ indicates no uncertainty and $\theta_i = 1$ indicates maximal uncertainty. As each individual article classification may be subject to some error, there are now two sources of measurement error: misclassification error and sampling uncertainty (the set of articles are a sample from the broader corpus of news). Both can be accounted for using a topic model. Suppose the misclassification rates are constant across observations. Then $\mf n_i = (N_i, C_i - N_i)^T$ follows the distribution in ((ref)), with \[ \mathbf B^T = \left[
\right], \quad \quad \bs w_i = \left[
\right], \] where $\beta_1$ is the probability that an article relating to uncertainty is correctly classified, and $\beta_0$ is the probability that an article not relating to uncertainty is misclassified.
As in the simple example in Section (ref), we consider two strategies to correct bias and restore valid inference. The first strategy involves bias-corrected estimators and confidence sets, which we formally develop and provide theoretical justification for in Section (ref). The second is joint estimation, the implementation details of which are discussed in Section (ref). In the remainder of the section, we illustrate the problems with two-step estimation\textemdash and how the proposed solutions fix them\textemdash across three applications drawn from diverse empirical literatures. Each application corresponds to one of the examples above.
Since the COVID-19 pandemic, the incidence of remote work has risen remarkably barreroWhyWorkingHome2021,aksoyWorkingHomeWorld2022. But much of the evidence on remote work comes from surveys which are limited in sample size. This makes tracking the evolution of remote work across narrow geographies and firms infeasible. hansenRemoteWorkJobs2023 instead develops a dataset (available at \url{https://wfhmap.com/}) that measures remote work from a vast corpus of online job postings provided by Lightcast. Each job posting contains metadata on occupation, firm, location, job title, and the posted wage, and a textual description of the job. hansenRemoteWorkJobs2023 uses the posting text to impute a binary label indicating whether or not the posting offers remote work. The authors collect human labels from Amazon Mechanical Turk for a sample of postings and use them to fine-tune DistilBERT sanhDistilBERTDistilledVersion2020, a Large Language Model. The classifier achieves high overall accuracy in a held-out test set (98%-99%). The authors then impute a remote work label for every posting in the corpus.
The data can be used to answer numerous questions about the causes and consequences of remote work. One important question is the degree of wage inequality in remote work arrangements, which the data has previously been used to study lambertResearchGrowingInequality2023. To explore this issue, we use regression (ref) taking $Y_i$ as the log of the advertised wage for posting $i$.\footnote{We form the advertised wage by averaging the salary_from and salary_to fields in the Lightcast data.} Here $\theta_i \in \{0,1\}$ is a latent indicator of whether posting $i$ offers remote work. The two-step approach replaces the latent $\theta_i$ with the predicted label $\hat{\theta}_i \in \{0,1\}$ from the classifier. Given the remote work classifier achieves high test-set accuracy, one may believe that such classification error is unlikely to affect inference meaningfully in this application. However, what matters is not measurement error per se but its magnitude relative to sampling error. In regressions with a large number of observations, sampling error is potentially small enough that even small classification errors can distort inference.
To illustrate the impact of classification error, we begin by organizing the data into NAICS2 industry $\times$ city cells for all job postings in 2022 and 2023 in the United States. These years coincide with peak post-pandemic incidence of remote work. Our theory in Section (ref) shows in which situations classification error is likely to produce the largest distortions, and we use it to select for case study NAICS 72 industry (Accommodation and Food Services) postings in San Diego. After removing internships and observations with missing salary data, there are 16,315 observations. The first column of Table (ref) contains two-step estimates of $\gamma$. In the initial regression model, we include no controls and find an estimated 65 log point effect on wages of remote work. Since this may in part reflect occupation composition, we next include SOC2 fixed effects in $\mathbf{q}_i$, which reduces the estimated $\gamma$ to 36 log points.
The positive association between remote work and posted wages is consistent with the descriptive evidence in lambertResearchGrowingInequality2023. Nevertheless, its strength may be distorted in the two-step approach. A key quantity governing the bias is the expected false positive rate (FPR), which can be estimated by reading a random sample of postings and counting the number that were mis-classified as positive. In this example, we took a random sample of 1000 postings, read the 26 that were classified as positive, and found nine were classified incorrectly, so the expected FPR is 0.009. Despite the small FPR, bias correction produces much larger effects: with no controls, the effect increases by 62% (to 105 log points) and, with controls, by 76% (to 64 log points).\footnote{Bias-corrected confidence intervals are wider to account for the uncertainty in the estimated FPR.} In many economic applications of AI and ML methods, classification accuracy is well below its level here. Finding large effects even in this setting shows that classification error can be of first-order importance.
In addition, we formulate a joint model over $\theta_i$, $\hat{\theta_i}$, and $Y_i$. While this produces a smaller estimated $\gamma$ without controls, in the preferred specification with occupation effects we continue to find a larger effect that falls outside the two-step confidence intervals. One reason why joint estimation and bias correction may not coincide is that human labels themselves may not represent the ground truth even in the best-designed audit. For example, the 2025 Economic Report of the President councilofeconomicadvisersEconomicReportPresident2025 discusses strengths and weaknesses of the hansenRemoteWorkJobs2023 database and points out that not every posting will explicitly mention remote work even when the firm in practice offers it.\footnote{The relevant passage is
}
The role of CEOs in shaping firm performance is important for many academic and policy debates, but until recently little data existed on what CEOs do with their time. To fill this evidence gap, bandieraCEOBehaviorFirm2020 collects and analyzes survey data on CEO time use in a sample of manufacturing firms. The paper describes salient differences in executive time use and relates those differences to firm and CEO characteristics and firm outcomes.
The survey consists of five questions with categorical responses: (Q1) the type of activity (meeting, public event, etc.); (Q2) duration of activity (15m, 30m, etc.); (Q3) whether the activity is planned or unplanned; (Q4) the number of participants in the activity; (Q5) the functions of the participants in the activity (HR, finance, suppliers, etc.). Survey responses are recorded for each 15-minute interval of a given week, e.g. Monday 8am-8:15am, Monday 8:15am-8:30am, and so forth. The sample consists of 916 CEOs.
The data are modeled as a topic model, with $V = 654$ answer combinations observed across the five questions, and $x_{i,j}$ denoting the number of times combination $j$ appears in the diary of CEO $i$. The authors reduce the dimensionality of the feature space using LDA with $K=2$ topics. The estimated $\hat{\boldsymbol{\beta}}_1$ places relatively higher mass on features associated with “management” like visiting production sites and one-on-one meetings with employees or suppliers, while $\hat{\boldsymbol{\beta}}_2$ places relatively higher mass on features associated with “leadership” like communicating with other C-suite executives and holding large, multi-function meetings. The leadership weight $\hat{\theta}_i$\textemdash which the authors call a “behavior index”\textemdash is thus a measure of the tendency of CEO $i$ to engage in leadership activities. bandieraCEOBehaviorFirm2020 regress log sales on $\hat{\theta}_i$ and firm controls, finding a positive association between leadership and firm performance. The paper's use of the two-step strategy may, however, lead to invalid inference.
To explore this possibility, we first replicate the authors' two-step strategy, estimating $\theta_i$ by LDA, then regressing the log sales of each CEO's firm on $\hat \theta_i$ and controls $\mf q_i$ including log employment, country fixed effects, and survey-wave fixed effects. Results reported in Table (ref) show that, according to the two-step approach, moving from a CEO who only spends time in management to one who only spends time in leadership is associated with a 40 log point increase in sales. What's more, neither bias correction nor joint estimation shifts the estimated effect size by a large amount. This shows that bias is not an inevitable part of using AI/ML-generated variables, but rather depends on the empirical setting.
One reason why measurement error is less important here lies in the $\kappa$ expression (ref) for the simple model. While not directly applicable here as the topic model structure is more complex, it still allows one to qualitatively compare sampling error (reflected by $\sqrt{n}$) and measurement error (reflected by $\mb E \left[ C_i^{-1} \right]$). The empirical analogue of this expression is $0.44$. In other words, there are a relatively large number of survey responses per CEO compared to the number of surveyed CEOs, meaning the measurement error in $\hat \theta_i$ is small relative to sampling error. To increase measurement error, we take a random 10% subsample of survey responses for each CEO, which can be thought of as observing half a day of behavior rather than a five-day workweek. The two-step approach now estimates a smaller insignificant effect, while both bias correction and joint estimation continue to estimate a significant effect.
Public communication is an increasingly important tool for central banks blinderCentralBankCommunication2008,blinderCrystalBallDarkly2018 and disentangling its effects is an ongoing research challenge. Communication events are often accompanied by rich, unstructured data like the content of speeches and press conferences, which require some quantification prior to econometric analysis. To illustrate this, we consider market reactions to FOMC policy announcements. Using high-frequency yield curve movements around FOMC announcements, gurkaynakActionsSpeakLouder2005 extracts separate “target” and “path” factors which account for large shares of observed variation in short- and long-run yields, respectively. One view is that policy rate news drives short rates (i.e., the target factor) while communication in the form of written statements drives long rates (i.e., the path factor) by shifting expectations of future actions.
To test this, we estimate (ref) with $Y_i$ as the path factor for FOMC meeting $i$ (as updated in acostaConstructingHighfrequencyMonetary2024), $\theta_i$ as the hawkish sentiment of the FOMC written statement, and $\mathbf{q}_i$ as a constant and the shadow short rate wuMeasuringMacroeconomicImpact2016, which accounts for policy variation during the zero-lower-bound period. The sample period is Feb 1995 through June 2023, during which 200 FOMC meetings take place.\footnote{The sample is determined by the availability of the path factor and shadow short rate data.}
Of course, hawkish sentiment $\theta_i$ is latent. To estimate it, we replicate the approach of gorodnichenkoVoiceMonetaryPolicy2023 which uses classification and aggregation to build a sentiment index.\footnote{gorodnichenkoVoiceMonetaryPolicy2023 studies the market impact of the tone of voice of FOMC Chairs during post-meeting press conferences, and uses the hawkish sentiment of the written statement as a control.} gorodnichenkoVoiceMonetaryPolicy2023 provides 1,243 sentences with human labels from FOMC statements from 1997 through 2010. Each sentence assigned to one of three mutually exclusive categories: hawkish (243 sentences), dovish (511), or neutral (489). We split the sentences into 1,118 training observations and 125 test observations and fine-tune BERT devlinBERTPretrainingDeep2019\textemdash a well-known Large Language Model\textemdash on the training data for label prediction.\footnote{More precisely, we fine tune bert-base-uncased via the Hugging Face library in Python.} We assign each sentence in the test data to its most likely class based on the trained model and obtain test-set accuracy of 0.85, slightly higher than in the original paper (0.81). We then predict a label for all paragraphs from FOMC statements in our sample.\footnote{Here we follow gorodnichenkoVoiceMonetaryPolicy2023 and classify individual paragraphs in the full sample although BERT is trained on human-labeled sentences. The optimal level of document granularity for classification prior to aggregation is an open question. gorodnichenkoVoiceMonetaryPolicy2023 classifies individual paragraphs of FOMC press conferences in addition to FOMC statements, but press conferences only began in 2011 where statements have been released since the mid-1990s.} The overall sentiment measure for FOMC meeting $i$ is $\hat\theta_i = N_i/C_i$ where $C_i$ is the number of sentences classified as hawkish or dovish, and $N_i$ is the number classified as hawkish. Figure (ref) shows the evolution of this measure over the sample period.
The first column of Table (ref) contains the results of two-step estimation. There is some evidence consistent with communication rather than policy driving the path factor. The coefficient on $\hat\theta_i$ is positive and significant while that on the policy rate is essentially zero. At the same, the effect is somewhat weak. The standard deviation of the path factor in the sample is 0.103, so the estimated $\hat\gamma$ implies a one-third standard deviation effect of $\theta_i$ moving from $0$ to $1$. The regression also has a low $R^2$.
We do not develop a bias correction for index construction.\footnote{Our bias corrections are valid provided measurement error and sampling error are comparable. The relatively short length of FOMC announcements and large misclassification errors we report below suggests measurement error may be sizable in this case. Joint estimation remains valid in such settings.} We do however implement joint estimation. The second column of Table (ref) shows the results. The effect size on sentiment nearly triples, as does the $R^2$ value. Confidence intervals are a little wider using this method, as it accounts for the fact that some FOMC announcements are relatively short, making their corresponding $\theta_i$ values difficult to infer. Figure (ref) shows that estimated sentiment from the two approaches co-move strongly, but estimated sentiment from the joint model is smoother.\footnote{While the joint approach is based on maximizing the integrated likelihood, our inference algorithm is based on sampling from the posterior distribution over $\theta_i$ and regression parameters. Sentiment from the joint model is the average value of $\theta_i$ draws.} In short, we find results consistent with measurement error in the two-step sentiment estimate which weakens the estimated market reaction to FOMC statements.
These results also have implications for the large literature that regresses outcomes on sentiment. shapiroMeasuringNewsSentiment2022 compares a variety of classifiers for predicting human labels for news article sentiment, including BERT, and reports substantial error even for the highest-performing methods (see Tables 1-3 in the paper). Approaches like ours can restore valid inference in such settings.
Table (ref) also shows estimates of $\beta_0$ and $\beta_1$, and shows the latter is well below that implied by the classifier. Economic theory predicts that the difference between realized and expected hawkishness is the relevant object for generating market reactions, whereas the human labels only classify realized hawkishness. While a full analysis is outside the scope of the paper, joint estimation in principle allows $\beta_0$ and $\beta_1$ to adjust to account for this broader notion of misclassification.
The empirical applications show that measurement error in AI/ML-generated variables can bias inference in a variety of settings. Each application has a particular algorithm for estimating $\hat{\bs\theta}_i$ that differs in important respects: supervised vs unsupervised learning, whether $\hat{\bs\theta}_i$ is the aggregation of multiple or single classified documents, and so forth. Further varieties are also common in the literature. To better understand the biases arising from measurement error and how to fix them, we begin by abstracting away from algorithmic-specific details of how $\hat{\bs\theta}_i$ is estimated. This allows us to develop a general, broadly applicable framework, which we then tailor to some specific use cases.
We first introduce some notation. Let \[ \bs \psi = \left[
\right] , \quad \quad \bs \xi_i = \left[
\right] , \quad \quad \hat{\bs \xi}_i = \left[
\right] . \] The OLS estimator of $\boldsymbol{\psi}$ in the two-step strategy is given by
The OLS estimators $\hat{\bs\gamma}$ and $\hat{\bs\alpha}$ of $\bs\gamma$ and $\bs \alpha$ are the upper and lower blocks of $\hat{\bs \psi}$.
We use asymptotic theory to derive tractable approximations to the finite-sample distribution of $\hat{\bs \psi}$. Both measurement error in $\hat{\bs \theta}_i$ and downstream sampling error determine the finite-sample distribution. To ensure that asymptotics deliver a useful approximation, we adopt a framework in which both sources of error remain present as the sample size $n$ becomes large. We do so be allowing the precision of $\hat{\bs\theta}_i$ to increase with $n$ at an appropriate rate. Formally, we consider a sequence of populations in which the distribution of $(Y_i,\boldsymbol{\theta}_i,\mathbf{q}_i)$ is held fixed and the conditional distribution of $\hat{\boldsymbol{\theta}}_i$ given $(Y_i,\boldsymbol{\theta}_i,\mathbf{q}_i)$ varies with $n$, so that
as $n \to \infty$, where $\bs \Omega$ is a finite non-random matrix and $\kappa$ is a non-negative constant. The constant $\kappa$ represents the relative magnitudes of measurement error and sampling error. The matrix $\bs \Omega$ is related to the variance of measurement error in settings where the error in $\hat{\bs \theta}_i$ is “classical”. However, condition ((ref)) also allows for “non-classical” measurement error, which is needed to accommodate a number of important use cases\textemdash including imputed categorical labels aignerRegressionBinaryIndependent1973. We show below that ((ref)) holds for AI/ML-generated labels and topic models and derive expressions for $\kappa$ and $\bs \Omega$, illustrating how $\kappa$ links measurement error in $\hat{\bs \theta}_i$ to the sample size.
We view this asymptotic framework as appropriate for approximating modern use cases where high-performance algorithms are deployed on large data sets.\footnote{In such scenarios, conventional measurement error frameworks in which the size of the measurement error remains fixed seem inappropriate, since measurement error bias eventually dominates and estimators are inconsistent. For completeness, we provide a set of results for this case in Appendix (ref).} While it is not meant to be taken literally, a heuristic interpretation is that it captures the prevailing trend whereby increasingly large datasets are analyzed by increasingly accurate algorithms.
Having introduced the asymptotic framework, we now introduce the assumptions. We first present a general “high-level” set of assumptions, then verify them below within the context of the running examples. In what follows, notions of convergence in probability and distribution should be understood as holding along this sequence of populations satisfying condition ((ref)). Let $\hat \varepsilon_i = Y_i - \hat{\boldsymbol{\psi}}^T \hat{\boldsymbol{\xi}}_i$. Throughout, we let $\mf 0$ denote a matrix or vector of zeros whose dimension is determined by the context.
Assumption (ref)(ref) is standard. Assumption (ref)(ref) can be verified using appropriate laws of large numbers. The first part of Assumption (ref)(ref) imposes a standard CLT condition while the second part is only used to establish consistency of standard errors. We focus on the case where the data are independent and identically distributed to simplify exposition, though this can be relaxed and the results can easily be extended to general types of dependence.
We now present our main result for this section, which shows that $\hat{\boldsymbol{\psi}}$ is consistent, derives its asymptotic distribution, and establishes consistency of standard errors. Let \[ \mathbf V = \mb E\Big[\boldsymbol{\xi}_i^{\phantom{T}}\!\!\!\;\boldsymbol{\xi}_i^T\Big]^{-1} \mb E\Big[\varepsilon_i^2 \boldsymbol{\xi}_i^{\phantom{T}}\!\!\!\;\boldsymbol{\xi}_i^T\Big] \mb E\Big[\boldsymbol{\xi}_i^{\phantom{T}}\!\!\!\;\boldsymbol{\xi}_i^T\Big]^{-1} \] denote the asymptotic variance of the OLS estimator in the infeasible regression of $Y_i$ on the true latent $\bs \theta_i$ and $\mathbf q_i$. Also let
denote the covariance matrix estimator for the regression of $Y_i$ on $\hat{\bs \theta}_i$ and $\mathbf q_i$.
Theorem (ref) shows that two-step inference is invalid when $\kappa > 0$. In this case, standard errors are consistent but the asymptotic distribution is centered away from the origin due to measurement error bias. As a result, confidence intervals based on the usual two-step strategy have the correct width but incorrect centering, leading to coverage rates below nominal coverage.\footnote{We omit discussion of the case $\kappa = +\infty$ where measurement error dominates sampling error. In that case, the coverage rates of standard OLS confidence intervals approach zero as $n$ becomes large.} The bias---and thus the degree of under-coverage---increases in $\kappa$. Simulations reported in Section (ref) show that coverage distortions can be severe even for small values of $\kappa$. Moreover, because measurement error can be nonclassical, it can be difficult to know even the sign of the bias: there may be attenuation or amplification. These critiques apply to inference on $\bs \alpha$ as well as $\bs \gamma$, and are therefore relevant for researchers using unstructured data to create control variables.
On the other hand, two-step inference is valid when $\kappa = 0$. In this case, measurement error is of smaller order than sampling error and can effectively be ignored. Here $\hat{\bs \psi}$ has the same $N(\mathbf 0, \mathbf V)$ asymptotic distribution as the (infeasible) OLS estimator obtained by regressing $Y_i$ on the true latent $\boldsymbol{\theta}_i$ and standard errors computed using $\hat{\boldsymbol{\theta}}_i$ are consistent.
We now derive expressions for $\kappa$ and $\Omega$ in the running examples. We shall use these in the next section to perform bias corrections.
To mimic scenarios where high-performance classifiers are deployed at scale, we consider a sequence of DGPs where the distribution of $(Y_i, \bs \theta_i, \mf q_i)$ is fixed but the distribution of $\mf x_i|(Y_i, \bs \theta_i, \mf q_i)$ varies with $n$ so that $\mf x_i$ becomes increasingly more informative about $\bs \theta_i$. For brevity we present conditions for the case of a deterministic classifier: $\hat \theta_i = \pi(\mf x_i)$ for some function $\pi$ taking values in $\{0,1\}$. We treat $\pi$ as deterministic, conditioning on the external training data set where necessary. Appendix (ref) presents a more general treatment allowing stochastic classifiers and multiple categories.
Assumption (ref)(ref) is the key drifting-sequence condition. It says that the false-positive rate $\mb E[\hat \theta_i(1-\theta_i)]$ goes to zero at rate $n^{-1/2}$ (or faster). This allows for the classifier to produce misclassifications which individually occur with low probability, but whose cumulative effect will be non-negligible relative to sampling error when $\kappa > 0$. Assumption (ref)(ref) is standard. Assumption (ref)(ref) says $\mathbf q_i$ and the prediction errors $\hat \theta_i - \theta_i$ are orthogonal. It is straightforward to relax this condition; doing so will simply add nonzero off-diagonal terms in the bias expression in ((ref)) without altering our main point: the two-step strategy can lead to biased inference. Finally, Assumption (ref)(ref) says the true regression errors $\varepsilon_i$ are uncorrelated with the AI/ML-generated prediction $\hat \theta_i$. As $\varepsilon_i$ and $\theta_i$ are assumed uncorrelated, in effect this condition simply requires that the prediction error $\hat \theta_i - \theta_i$ and $\varepsilon_i$ are uncorrelated.
In many modern empirical settings, there may be a large number of observations (large $n$) and a large amount of unstructured data per observation (large $C_i$). To mimic such scenarios, we consider a sequence of populations in which the distribution of $(Y_i,\mf q_i,\bs w_i)$ is fixed and the conditional distribution of $(\mf x_i, C_i)$ given $(Y_i,\mf q_i,\bs w_i)$ varies with $n$ so that ((ref)) holds and
Smaller values of $\kappa$ correspond to settings where there is a relatively more unstructured data per observation, and hence relatively less measurement error. Also note that since $C_i$ enters via its inverse, if most documents are large but a few are small, then $\kappa$ may still be large.
To simplify some expressions, in what follows we implicitly assume that the document size $C_i$ is independent of $(\boldsymbol{w}_i, \mathbf{q}_i,Y_i)$. We also assume that $\mf x_i$ and $\mf q_i$ are independent conditional on $(C_i,\boldsymbol{w}_i)$, and that $\varepsilon_i$ and $(\mf x_i,C_i)$ are independent conditional on $(\boldsymbol{w}_i,\mathbf q_i)$. In effect, the latter two assumptions ensure the multinomial sampling error and regression errors are uncorrelated. These assumptions seem very reasonable and can be relaxed: doing so simply complicates the expressions below. We also slightly strengthen ((ref)) to require that $\mb E[\varepsilon_i(\bs w_i, \mf q_i)] = \mf 0$. That is, no relevant topic weights have been omitted from the regression.
Assumption (ref)(ref) is the key drifting sequence condition. Assumption (ref)(ref) says that none of the topics are redundant. bingOptimalEstimationSparse2020, wuSparseTopicModeling2023, and keUsingSVDTopic2022 show various estimators $\hat{\mathbf{B}}$ converge at the optimal rate $(nC)^{-1/2}$ (up to log terms) where, for simplicity, all $C_i$ are of the same order $C$ (i.e., $C_i \asymp C$). Hence, their estimators all satisfy Assumption (ref)(ref) when $C$ grows with $n$, as we have here by ((ref)). Assumption (ref)(ref) imposes some structure on the $\hat{\bs \theta}_i$ to facilitate derivations. This condition is not vacuous: we have $\boldsymbol{\theta}_i = \boldsymbol S(\mathbf{B} \mathbf{B}^T)^{-1}\mathbf{B} \mb E[\mathbf x_i/C_i|C_i, \boldsymbol w_i]$ by display ((ref)) and Assumption (ref)(ref). Hence, one could take $\hat{\boldsymbol{\theta}}_i = \boldsymbol{S}(\hat{\mathbf{B}} \hat{\mathbf{B}}^T)^{-1} \hat{\mathbf{B}} (\mathbf x_i/C_i)$, in which case Assumption (ref)(ref) trivially holds. Assumption (ref)(ref) is standard. Assumption (ref)(ref) is made to simplify arguments establishing consistency of standard errors and can be relaxed. This condition trivially holds in view of ((ref)) when all $C_i$ are of the same order, since in that case $C_i \gtrsim n^{1/2}$. We note that these conditions implicitly assume $\mf B$ is identified. We defer discussion of identification to Appendix (ref).
We now propose two methods for performing valid inference on $\bs \gamma$ and $\bs \alpha$: (1) bias corrected estimators and confidence intervals, and (2) joint estimation of the upstream and downstream regression models. These are the approaches illustrated in the applications in Section (ref).
Each method has its strengths and weaknesses. The bias correction is simple to implement and scales well, but the formulas for the bias we develop are for particular settings. While these encompass leading applications, they are not exhaustive, and other formulas will need to be derived for other settings by specializing our general characterization of bias. Joint estimation is more flexible and can handle cases where the general bias formula may be difficult to specialize, but it is also more computationally demanding.
Theorem (ref) shows that the asymptotic bias of the two-step estimator $\hat{\bs \psi}$ takes the form \[ \boldsymbol{b} = -\kappa \, \mb E\Big[\boldsymbol{\xi}_i^{\phantom{T}}\!\!\!\;\boldsymbol{\xi}_i^T\Big]^{-1} \left[
\right] \boldsymbol \psi. \] We use this formula to construct bias-corrected estimators and confidence intervals (CIs) for $\bs \gamma$ and $\bs \alpha$. Given consistent estimators $\hat \kappa$ and $\hat{\bs \Omega}$ of $\kappa$ and $\bs \Omega$, one can construct the following bias-corrected estimators
The first estimator $\hat{\boldsymbol{\psi}}^{bca} $ performs an additive bias correction to the OLS estimator $\hat{\boldsymbol \psi}$. Simulations below show that this estimator performs well when the bias in $\hat{\boldsymbol \psi}$ is relatively small. The additive correction may not be sufficient when the bias in $\hat{\boldsymbol \psi}$ is large, as it relies upon $\hat{\boldsymbol \psi}$ being a reasonable estimator of $\boldsymbol \psi$. The bias corrected estimator $\hat{\boldsymbol{\psi}}^{bcm}$ performs a more aggressive (multiplicative) bias correction. We recommend this second estimator when the bias in $\hat{\boldsymbol \psi}$ is expected to be large, provided the maximum eigenvalue of \[ \frac{\hat \kappa}{\sqrt n}\left( \frac 1n \sum_{i=1}^n \hat{\boldsymbol{\xi}}_i^{\phantom{T}}\!\!\!\;\hat{\boldsymbol{\xi}}_i^T \right)^{-1} \left[
\right] \] is less than one in absolute value.\footnote{This condition ensures invertibility of $ \left( \mathbf I - \frac{\hat \kappa}{\sqrt n}\left( \frac 1n \sum_{i=1}^n \hat{\boldsymbol{\xi}}_i^{\phantom{T}}\!\!\!\;\hat{\boldsymbol{\xi}}_i^T \right)^{-1} \left[
\right] \right). $ } We discuss how to consistently estimate $\kappa$ and $\bs\Omega$ within the context of our running examples below.
Bias corrected CIs for the regression coefficients are constructed by centering at the bias-corrected estimators and using OLS standard errors. A valid $100(1-a)$% CI for the $j$th component $\psi_j$ of $\bs \psi$ is given by \[ \mathrm{CI}_n(\psi_j) = \left[ \hat \psi_j^{bc} - z_{1-a/2} \frac{\hat \sigma_j}{\sqrt n} , \hat \psi_j^{bc} + z_{1-a/2} \frac{\hat \sigma_j}{\sqrt n} \right], \] where $\hat \psi_j^{bc}$ denotes the $j$th entry of $\hat{\bs \psi}^{bca}$ or $\hat{\bs \psi}^{bcm}$, $z_{1-a/2}$ is the $1-a/2$ quantile of the normal distribution (e.g., 1.96 for a 95% CI), and $\hat \sigma_j$ denotes the square root of the $j$th diagonal entry of $\hat{\mathbf V}$ from ((ref)).
The following result shows that the bias-corrected estimators are asymptotically normal with the correct centering, and the bias-corrected confidence intervals are valid:
We now show how to apply bias correction in the context of our two running examples. In each case, we propose consistent estimators $\hat \kappa$ and $\hat{\bs\Omega}$ of $\kappa$ and $\bs\Omega$. Theorem (ref) then implies that the bias-corrected CIs have correct coverage. We also recommend reporting $\hat \kappa$ as a diagnostic for assessing the relative importance of measurement error and sampling error.
Here $\kappa = \lim_{n \to \infty} \sqrt n \, \mb E[ \hat \theta_i(1-\theta_i)]$ and $\bs \Omega = 1$. One may estimate $\kappa$ by taking a sample of observations of size $m \ll n$. Then, each observation $i = 1,\ldots,m$ for which $\hat \theta_i = 1$ is inspected, and assigned a true label $\theta_i$. The estimator of $\kappa$ is \[ \hat \kappa = \sqrt n \widehat{FPR}, \quad \quad \widehat{FPR} = \frac 1m \sum_{i=1}^m \hat \theta_i (1-\theta_i). \] We establish validity of this approach allowing $m/n \to 0$ asymptotically. This is important for accommodating modern use cases where ML/AI methods are deployed to impute labels on massive data sets (large $n$), but where correctly labeling data can be costly (small $m$). For instance, boxellJournalistIdeologyProduction2022 impute binary labels representing political slant for a corpus of millions of newspaper articles using a validation sample size in the tens of thousands. Other approaches recently advocated in the literature for correcting measurement error (e.g., fongMachineLearningPredictions2021,allonMachineLearningPrediction2023,egamiUsingImperfectSurrogates2023) are shown to be valid when $m/n \to c > 0$, so that the validation and original sample sizes are comparable. As far as we are aware, the theoretical properties of these proposed methods are unknown in modern scenarios where $m/n \to 0$.
We emphasize that our approach does not require constructing a full validation sample of size $m$ (and thus performing costly inspection of all $m$ observations), but only those for which $\hat \theta_i = 1$. This can greatly reduce the burden on the researcher. For instance, in our empirical application to remote work, we take a subsample of size $m = 1000$. Of these, only 26 observations have $\hat \theta_i = 1$, so only 26 job postings need to be inspected. By contrast, constructing a validation sample would require inspecting all $m = 1000$ observations.
Finally, our bias correction can be implemented using external data in which $Y_i$ and/or $\mf q_i$ are missing, since it only requires a subsample of $\theta_i$ and $\hat \theta_i$. This makes our approach more broadly applicable than other methods that require a full validation data set. For instance, bursztynImmigrantNextDoor2024 use a ML algorithm to classify a data set of charitable donors' names by ethnicity. As true ethnicity is latent, they estimate the accuracy of the classifier using an external sample of North Carolina voter registration data which contains self-reported ethnicity (but is missing data on charitable donations).
The following result shows that $\hat \kappa$ is consistent, allowing $m/n \to 0$.
Consistency of $\hat\kappa$ suffices for asymptotic validity of the bias-corrected CIs. However, the estimation of $\hat \kappa$ from a small subsample can introduce additional variability that, while asymptotically negligible, may be important to account for in finite samples. To this end, we introduce the following finite-sample correction to standard errors. Recall $\hat{\mf V}$ from ((ref)). Also let \[ \hat{\bs \Gamma} = \left( \frac 1n \sum_{i=1}^n \hat{\boldsymbol{\xi}}_i^{\phantom{T}}\!\!\!\;\hat{\boldsymbol{\xi}}_i^T \right)^{-1} \left(
\right). \] The adjusted covariance matrix estimators $\hat{\mf V}^{bca}$ and $\hat{\mf V}^{bcm}$ for $\hat{\bs \psi}^{bca}$ and $\hat{\bs \psi}^{bcm}$ are given by \[
\] respectively. The form of these adjustments follows from the law of total variance. Under the conditions of Theorem (ref) and Lemma (ref), we have $\hat{\mf V}^{bca} \to_p \mf V$ and $\hat{\mf V}^{bcm} \to_p \mf V$, with $\mf V$ the asymptotic variance derived in Theorem (ref). In practice, we recommend reporting standard errors computed from $\hat{\mf V}^{bca}$ if using $\hat{\bs \psi}^{bca}$ or $\hat{\mf V}^{bcm}$ if using $\hat{\bs \psi}^{bcm}$.
Codes to implement these bias corrections and standard error formulas are available in the Python package ValidMLInference.
In view of the expressions for $\kappa$ and $\bs \Omega$ derived in Theorem (ref), consider \[
\] where $\bar{\bs w}_n = \frac 1n \sum_{i=1}^n \hat{\bs w}_i$. The following result shows that these estimators are consistent.
Our second approach for correcting the bias from the two-step strategy begins by formulating a likelihood $l\left(Y_i, h(\mathbf{x}_i), \bs \theta_i \mid \mathbf{q}_i, \mathbf{v}_i, \bs\gamma, \bs\alpha, \bs\zeta\right)$. Here $h$ is a function of the high-dimensional or unstructured observables, $\mathbf{v}_i$ are covariates that potentially enter the model beyond the downstream regression, and $\bs\zeta$ are nuisance parameters. We discuss how to form a likelihood below. Integrating the latent $\bs \theta_i$ out yields a likelihood $l\left(Y_i, h(\mathbf{x}_i) \mid \mathbf{q}_i, \mathbf{v}_i, \bs\gamma, \bs\alpha, \bs\zeta\right)$ depending only on observables, which can then be used for maximum likelihood estimation of model parameters. This idea can be applied generically, and here we adapt it to each of the three applications in Section (ref).
In the remote work application, $Y_i$ is log posted wages, $\theta_i \in \{0,1\}$ is an indicator for whether job posting $i$ offers remote work ($\theta_i = 1$) or not ($\theta_i = 0$), and $\mathbf{x}_i$ is job posting text. We use $h(\mathbf{x}_i) = \hat\theta_i \in \{0,1\}$ and thereby formulate a likelihood over the predicted class label associated with $\mathbf{x}_i$ rather than over $\mathbf{x}_i$ directly. No additional covariates enter the model so $\mathbf{v}_i$ is empty. The integrated likelihood is
for $d \in \{0,1\}$, $\bs \omega = (\omega_{00}, \omega_{10}, \omega_{01}, \omega_{11}) \in \Delta^3$ with $\omega_{ab} = \Pr(\hat \theta_i = a, \theta_i = b)$, and $MN(\,\cdot\,;\bs \lambda) = \sum_{l=1}^L \lambda_{l} \phi(\,\cdot\,;\lambda_{\mu l},\lambda_{\sigma^2 l})$ is a mixture of normal distributions with $L$ components and mixing weights $\lambda_{1},\ldots,\lambda_{L} \in \Delta^{L-1}$, where $\phi(\,\cdot\,;\mu,\sigma^2)$ denotes the $N(\mu,\sigma^2)$ density, and the component means are normalized so that $\sum_{l=1}^L \lambda_{l} \lambda_{\mu l} = 0$. Thus, here $\bs \zeta = (\bs \omega, \bs \lambda_0,\bs \lambda_1)$. We use $L = 3$ for the results in Section (ref).
In the CEO time use application, $Y_i$ is log sales of firm $i$, $\theta_i \in [0,1]$ is a behavior index, and $\mathbf{x}_i$ is the vector of counts of time-use feature combinations across all surveyed 15-minute time intervals. Here the likelihood is directly over $\mathbf{x}_i$, i.e. $h(\mathbf{x}_i) = \mathbf{x}_i$. In the spirit of correlated random effects in panel data models, we specify a distribution for the behavior index $\theta_i$ conditional on $J$ covariates $\mathbf g_i$, which may include $\mathbf q_i$ or other variables.\footnote{robertsStructuralTopicModels2014 presents a model in which a logistic normal distribution over topic shares is parameterized by covariates but without a downstream regression. bleiSupervisedTopicModels2010 and ahrensBayesianTopicRegression2021 present models in which linear combinations of topic shares explain a normally distributed response variable, but do not allow covariates to enter the distribution over topic shares.} The likelihood is implied by
where $\mathbf{w}_i = (1-\theta_i, \theta_i)^T$. bandieraCEOBehaviorFirm2020 shows log employment and an indicator for whether the CEO has an MBA are correlated with behavior, so we include these in $\mathbf g_i$ together with a constant. As the likelihood also depends on the number of observed time units $C_i$, we have $\mathbf{v}_i = (C_i, \mathbf g_i)$.
In this application, $Y_i$ is the path factor in FOMC meeting $i$, $\theta_i \in [0,1]$ is the hawkish sentiment of the FOMC written statement, and $\mathbf{x}_i$ is the text of the statement. We take $h(\mathbf{x}_i) = (N_i, C_i)$ and form a likelihood over the number of paragraphs in FOMC statements classified as hawkish ($N_i$) conditional on the total number classified as hawkish or dovish ($C_i$). We treat $\mathbf{v}_i$ as empty, but our analysis could be extended to allow the prior for $\theta_i$ to depend on covariates. The likelihood is implied by
In addition, we include additional terms in the likelihood for the testing data:
where $N_{ab}$ is the number of test-set observations classified an $a$ with true label $b$.\footnote{In the application, $N_{11} = 14$, $N_{00} = 54$, $N_{10} = 1$, $N_{01} = 1$.}
In certain cases, one can directly integrate-out $\bs \theta_i$ from the likelihood, as in (ref). However, in more complex problems, such as (ref), there are two challenges. First, the integration for computing the likelihood has no closed-form solution and must be performed numerically. Moreover, this numerical integration must be done observation-by-observation. As such, standard likelihood-based estimation may not be computationally feasible.
In such cases, one may use Bayesian computation to perform valid frequentist inference. In this approach, we introduce a prior for the model parameters $\boldsymbol \delta = (\bs\gamma, \bs\alpha, \bs\zeta)$ and treat the latent $\boldsymbol{\theta}_i$ as “parameters” drawn from a prior distribution as illustrated above. We sample from the posterior distribution of $\left(\bm\delta,(\boldsymbol \theta_i)_{i=1}^n\right)$ conditional on the observed data $(Y_i, h(\mathbf x_i), \mathbf q_i, \mathbf v_i)_{i=1}^n$. The marginal draws for $\boldsymbol \delta$ represent draws from the posterior distribution for $\boldsymbol \delta$ based on the integrated likelihood $l\left(Y_i, h(\mathbf{x}_i) \mid \mathbf{q}_i, \mathbf{v}_i, \bs\gamma, \bs\alpha, \bs\zeta\right)$. Thus, $\boldsymbol{\theta}_i$ is implicitly integrated of the likelihood as part of the sampling procedure.
It is important to emphasize that while our suggested approach uses Bayesian computation, inference is frequentist. The maximum likelihood estimator $\hat{\boldsymbol{\delta}}$ of $\boldsymbol{\delta}$ is asymptotically normal under standard regularity conditions (e.g., Theorem 5.41 of vaartAsymptoticStatistics1998). By the Bernstein--von Mises Theorem (see Theorem 10.1 of vaartAsymptoticStatistics1998 and discussion), the posterior mean $\bar{\boldsymbol{\delta}}$ of $\boldsymbol{\delta}$ is first-order asymptotically equivalent to the MLE $\hat{\boldsymbol{\delta}}$. Moreover, the posterior distribution of $\boldsymbol\delta$ is asymptotically normal with mean $\bar{\boldsymbol \delta}$ and variance (when appropriately scaled with $n$) equal to the asymptotic variance of the MLE. As such, Bayesian credible sets for elements of $\boldsymbol{\delta}$\textemdash or any of its components such as $\boldsymbol{\gamma}$ or $\boldsymbol{\alpha}$\textemdash are valid frequentist confidence sets. This approach is also efficient for inference on $\boldsymbol{\delta}$ and its components, as it is asymptotically equivalent to likelihood-based inference.
We implement this approach for the CEO time use and central bank communication applications. For the former, we follow standard practice in the topic modeling literature and use $\text{Dirichlet}(0.2)$ priors for $\bm\beta_1$ and $\bm\beta_2$. For the logistic normal prior for $\boldsymbol{\theta}_i$, we set $\sigma_\theta^2 = 1$ and use a $\text{Normal}(0,4)$ prior for each element of $\bs\phi$. We also use $\text{Normal}(0, 100)$ priors for all downstream regression coefficients, and a $\text{Gamma}(1,10)$ prior for $\sigma_Y$.
For the central bank communication application, we use the same priors in the downstream regression. We use a $\text{Beta}(2, 5)$ prior for $\beta_0$ and a $\text{Beta}(5, 2)$ prior for $\beta_1$. We adopt asymmetric priors on the classification probabilities to resolve the label switching problem inherent in topic models.
We perform sampling with Hamiltonian Monte Carlo implemented in NumPyro phanComposableEffectsFlexible2019. See sacherHamiltonianMonteCarlo2024 for more details on HMC and probabilistic programming.
In this section, we provide simulation evidence illustrating the finite-sample performance of our methods for correcting bias and performing valid inference.
This simulation is calibrated to the remote work empirical application in Section (ref). The same contains 16,315 job postings. Bias-corrected estimates of the intercept and slope are approximately 10 and 1. The residual standard deviation is approximately 0.3 (respectively, 0.5) for observations with $\hat \theta_i = 0$ ($\hat \theta_i = 1$). We therefore generate data according to \[ Y_i = 10 + \theta_i + (0.3 + 0.2 \theta_i) \varepsilon_i, \] with $\varepsilon_i \sim N(0,1)$. The sample mean of $\hat \theta_i$ is 0.025 and $\widehat{FPR} = 0.009$, which corresponds to $\hat \kappa \approx 1.1$. We draw samples of size $n =$ 8,000, 16,000, and 32,000, with $\theta_i \sim \text{Bernoulli}(p)$ for $p =$ 0.025, 0.05, and 0.5, and $\kappa =$ 0.5, 1, and 2, with 1000 simulations for each configuration.
We implement the two-step strategy, additive and multiplicative bias corrections, and joint estimation. For the latter, we use the likelihood from ((ref)) with a single Gaussian component ($L = 1$). To implement both bias corrections, we generate a sample of $(\theta_i,\hat \theta_i)$ of size $m = 1000$. We use two estimators of the false-positive rate. The first is the empirical frequency $\widehat{FPR} = \frac 1m \sum_{i=1}^m \hat \theta_i(1-\theta_i)$. We may interpret this as the posterior mean of $r := \mb E[\hat \theta_i(1-\theta_i)]$ under an improper $r^{-1}(1-r)^{-1}$ prior. This prior puts most of its mass at the endpoints of the interval $[0,1]$. However, our approach is based on the premise that $r$ should be small. We therefore consider a Bayes estimator $\widehat{FPR}_B = \frac{\sum_{i=1}^m \hat \theta_i(1-\theta_i) + \frac 12}{m + \frac{5}{2}}$, which is the posterior mean of $r$ under a proper $r^{-1/2}(1-r)$ prior. Results are presented in Tables (ref)-(ref).
Our baseline set of simulations with $p = 0.025$ (Table (ref)) is very challenging: in a sample of size 8,000, we expect only 200 observations with $\theta_i = 1$. When $\kappa = 1$ about half of these will be incorrectly imputed with $\hat \theta_i = 0$, and when $\kappa = 2$ almost all will be incorrectly imputed. In either case, we expect a very large bias of two-step estimators, and the results in Table (ref) confirm this, with large bias and zero coverage. Our bias corrections rely on the two-step estimator having a bias that is of the same order as sampling uncertainty. One might therefore expect they will perform poorly in small samples when two-step estimators are severely biased. Table (ref) shows this is not necessarily the case. Consider the configuration with $n =$ 16,000 and $\kappa = 1$, which is closest to the empirical application. The two-step estimator under-estimates the coefficient by about 32%, the additive bias corrections by about 10%, while the multiplicative bias corrected estimators are approximately unbiased, even in this challenging design. Indeed, the multiplicative bias correction appears to perform well across all configurations except when $n$ is smallest and $\kappa$ is largest.
Bias corrected confidence sets appear to under-cover by about 10% with $n =$ 16,000 and $\kappa = 1$, but the results for $n =$ 32,000 show their coverage moves toward nominal coverage as $n$ increases. Coverage is much closer to nominal coverage in the less challenging designs with $p = 0.05$ (Table (ref)) and $p = 0.5$ (Table (ref)), even with small $n$. Coverage also seems to be improved using the Bayes estimator $\widehat{FPR}_B$ instead of the empirical frequency.
Finally, joint estimation produces approximately unbiased estimators across all designs, including the most challenging cases. It also produces confidence sets with coverage closest to nominal coverage and, as expected, the lowest risk of the competing approaches. Overall, these results demonstrate the sound performance of both our proposed solutions in a challenging, empirically calibrated setting.
These simulations are calibrated to the empirical application to AI/ML-generated indices in Section (ref). We draw a latent share $\theta_i \sim U[0,1]$ then generate $N_i \sim \text{Binomial}(C_i,\beta_1 \theta_i + \beta_0 (1-\theta_i))$ with $\beta_1 = 0.9 $ and $\beta_0 = 0.1$ to introduce misclassification error. We set \[Y_i = -0.05 + 0.11 \theta_i + 0.1 \varepsilon_i,\] where $\varepsilon_i \sim N(0,1)$ and parameters are similar to the estimates in the empirical application. We generate samples of $(Y_i,N_i,C_i)$ of size $n = $ 200 (as in the application), 800, and 3,200. For $n = 200$ we use $C_i$ from the empirical application so that $\kappa \approx 4.57$. We then increase $C_i$ by a factors of two and four, to generate samples with $\kappa \approx 2.28$ and $\kappa \approx 1.14$, respectively. For the larger sample sizes we replicate the empirical $C_i$ and multiply by factors of two and four so $\kappa$ remains constant. We generate 1000 samples for each configuration.
We implement the two-step strategy, both bias corrections, and joint estimation. For the latter, we use the likelihood described in (ref). For the two-step strategy, we require a suitable estimate of $\theta_i$. We produce estimates based on the topic model representation from Section (ref) in order to account for potential misclassification error. To provide a clear comparison with joint estimation, we use the estimates of $\beta_0$ and $\beta_1$ from joint estimation, then estimate $\theta_i$ in accordance with the discussion in Section (ref), using \[ \hat \theta_i = \max(0, \min( 1, \bs S (\hat{\mf B}^T)^{-1} (\mf n_i/C_i))), \] where \[ \bs S = \left[
\right], \quad \hat{\mf B} = \left[
\right] , \quad \mf n_i = \left[
\right]. \] We also implement additive and multiplicative bias corrections using the formulas derived for topic models with the $\bs S$ and $\hat{\mf B}$ as above. Results are presented in Table (ref).
First consider the bias results. In our baseline setting with $n = 200$ and $\kappa = 4.57$ as in the empirical application, the median relative bias of the two-step estimate of $\gamma$ is $-0.446$, showing that the two-step strategy under-estimates $\gamma$ by nearly half. By contrast, joint estimates are nearly unbiased. The additive bias correction improves upon the two-step strategy, producing an estimate about 85% that of the true effect size when $\kappa = 4.57$, and approximately unbiased estimates for smaller $\kappa$. Conversely, the multiplicative correction tends to over-estimate the true effect size, suggesting it performs too aggressive a bias correction in small samples. In larger samples, the additive correction produces approximately unbiased estimates, and the performance of the multiplicative correction improves, especially for smaller values of $\kappa$. The second panel of Table (ref) also suggests the additive bias correction is preferred from a root median square error perspective, producing values that are on par with joint estimation.
Turning to coverage, we see that the two-step CIs have coverage well below nominal coverage, especially for larger values of $n$ and $\kappa$. Conversely, joint estimation produces CIs with coverage close to nominal coverage even for the smallest sample size. Coverage of the additively bias-corrected CIs is significantly better than that of the multiplicatively bias-corrected CIs which, in turn, is better than two-step coverage. For large $n$ and small $\kappa$, CIs based on the additive bias correction have coverage close to nominal coverage.
We conclude this section by revisiting some of the empirical applications in light of the simulation evidence presented above. First consider the application to remote work (Section (ref)). The bias-corrected estimates reported in Section (ref) are based on the multiplicative bias correction, which the above results showed performed best in simulations mimicking this design, and the empirical estimate $\widehat{FPR}$. Results for the different bias corrections with different estimates of the false-positive rate are presented in Table (ref). As before, estimated effect sizes are larger for the multiplicative correction, and the multiplicatively corrected CIs are slightly to the right of those using the additive correction. Additively corrected CIs overlap slightly with two-step CIs reported in Section (ref), while multiplicatively corrected CIs do not.
In view of the simulation evidence above for the topic model, the bias corrections reported in Section (ref) use the additive correction, but the multiplicative correction produced values that agreed to within $\pm0.01$.
For central bank communication application in Section (ref), we implemented the two-step strategy using the empirical share $\hat \theta_i = N_i/C_i$ to estimate $\theta_i$. Table (ref) presents a set of simulation results for this case as well. This share estimate is prone to two sources of measurement error: misclassification error and upstream sampling error. The results in Table (ref) hold the former fixed and let the latter go to zero appropriately with the sample size. Evidently, these measurement errors work in different directions. With small $n$ and large $\kappa$,\footnote{Note that the $\kappa$ here is defined within the context of the topic model for the simulations in Section (ref), where upstream sampling error is the only source of measurement error. An appropriate $\kappa$ as in ((ref)) for the share variable $\hat \theta_i = N_i/C_i$ must account for both misclassification error and upstream sampling error.} upstream sampling error is more important. This is a “classical” measurement error that causes attenuation bias. With $n = 200$ and $\kappa = 4.57$, as in the empirical application, the bias is comparable to that obtained by implementing the two-step strategy using the topic model, underestimating the true effect size by around 43%. However, bias goes in a different direction when $n$ is large and $\kappa$ is small, so that misclassification error is more important than upstream sampling error. For instance, with $n =$ 3,200 and $\kappa = 1.14$, two-step regression onto shares now over-estimates the true effect size by around 16%. Correspondingly, the two-step confidence interval has coverage well below nominal coverage.
An important take-away from these results is that the bias of two-step estimators using AI/ML-generated variables can behave differently than in classical measurement error settings, making it difficult for researchers to determine even the sign of the bias.
The leading approach for analyzing unstructured or high-dimensional data follows a two-step strategy. First, latent variables of economic interest are estimated using an AI-powered information retrieval algorithm or other ML method. Second, the AI- or ML-generated variables are plugged-in to downstream econometric models, and are treated as regular numeric “data” for the purposes of estimation and inference.
This paper highlights, both theoretically and empirically, how measurement error introduced in the first step leads to biased estimates and invalid inference for the downstream regression coefficients. The degree of bias, and therefore the degree to which it distorts inference, depends on the relative importance of measurement error and sampling error, but it can be substantial in practice.
To address this problem, we propose two robust alternative inference methods: (1) an explicit bias correction with bias-corrected confidence intervals; and (2) joint maximum likelihood estimation. In a series of simulations and applications involving label imputation, dimensionality reduction, and index construction via classification and aggregation, we show that the two-step strategy produces material biases whereas both proposed methods perform well.
\FloatBarrier