Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
95,447 characters · 17 sections · 62 citation commands
EnsembleIV: Creating Instrumental Variables from Ensemble Learners for Robust Statistical Inference
\doublespacing
Empirical researchers in the social sciences are increasingly leveraging a combination of supervised machine learning and statistical inference in a hybrid, two-phase process. In the first phase, a supervised machine learning model is trained to predict a target outcome based on a set of features. In the second phase, predicted values of the target are used as an independent variable, usually within a regression model, for statistical inference purposes. However, because predictions from the first-phase machine learning model are typically imperfect, prediction errors will manifest as measurement error in the second-phase regression model, leading to estimation biases and threatening the validity of inferences.
This two-phase approach is an example of statistical inference with generated regressors pagan1984econometric,oxley1993econometric. It has seen growing adoption in a number of social science disciplines, partly due to impressive advances in machine learning techniques that enable the efficient extraction of useful information from large amounts of both structured and unstructured data. To name just a few examples, cengiz2022seeing built a boosted tree model to predict which workers were affected by changes to the minimum wage based on workers' demographics and then examined the effect of those changes on workers' labor market outcomes. moreno2014doing mined text sentiment from user-generated comments on an online service platform, and subsequently estimated the impact of (predicted) sentiment on buyers' purchasing decisions. zhang2021makes used deep learning techniques to measure the quality of property images on Airbnb, and studied how image quality, in turn, affects property demand. wu2020air estimated the effect of a geography's (predicted) long-term air pollution exposure on the rate of COVID-19 mortality.
While early work that adopted this two-phase approach has largely ignored the measurement error issue that this approach implies, researchers focused on statistical methods have increasingly attended to it over the last few years yang2018mind,wang2020methods,qiao2021correcting,fong2021machine,yang2022achieving. Compared to the settings in which measurement error has traditionally been addressed, wherein the errors are unobserved, and estimation biases are therefore hard to address, measurement error deriving from a machine learning model's prediction errors is quite unique, in that additional information is typically available about the error that can be leveraged in bias correction. Because the training and evaluation of a machine learning model typically require a manually collected labeled dataset, where the true values of the target outcome (e.g., text sentiment or image quality) are known, one can directly observe the measurement error in the generated variable on the labeled data and quantify its properties (e.g., distributional characteristics or correlations with other variables of interest). This has enabled the use of some existing error-correction techniques to mitigate estimation biases Stefanski1995,Kuchenhoff2006,yang2018mind, as well as the development of new approaches qiao2021correcting,fong2021machine,yang2022achieving,wei2022unstructured,allon2023machine.
In this paper, we develop a novel method that leverages instrumental variables to address the measurement error problem. The method consists of three key ingredients, respectively focusing on the generation, transformation, and selection of instruments. First, we propose to use ensemble learning techniques (e.g., random forest) to build the first-phase machine learning model. Doing so generates a set of individual learners (e.g., individual trees in a random forest) whose predictions can serve as candidate instruments for each other. However, these instruments are “imperfect" in that they are not guaranteed to satisfy the exclusion condition. Accordingly, second, we propose a technique, based on earlier work by nevo2012identification, that transforms candidate instruments to ensure they comply with the exclusion condition. Third, to deal with the potential challenge of weak instruments (i.e., instruments that only barely satisfy the relevance condition), we propose and evaluate several approaches to selecting strong instruments, which are subsequently used in instrumental variable regressions to obtain unbiased estimates. We henceforth refer to this method as EnsembleIV.
Under standard and mild regularity conditions, we prove the consistency and asymptotic normality of EnsembleIV estimator, which enable asymptotically valid inference. We also carry out empirical evaluations of EnsembleIV to understand its properties and performance on both synthetic and real-world data. Simulation studies with synthetic data demonstrate EnsembleIV's ability to substantially mitigate estimation biases on machine-learning-generated variables in several common regression models. Further, applying EnsembleIV to a real-world dataset of user-generated content on social media, we illustrate how it can be used together with modern deep-learning techniques.
EnsembleIV represents a novel methodological contribution to the nascent literature on robust statistical inference with machine learning-generated variables. While the measurement error problem and its solutions have been extensively studied in the literature, EnsembleIV stands out as an approach that leverages ensemble machine learning techniques to create valid instrumental variables. It also improves upon existing approaches in several important aspects. First, EnsembleIV has a more general theoretical foundation, as the method can accommodate (i) both continuous and binary generated variables and (ii) both independent and correlated measurement errors carroll2006measurement. In contrast, many existing approaches are more limited, e.g., SIMEX Stefanski1995,yang2018mind only addresses classical measurement error and MC-SIMEX Kuchenhoff2006 works only for binary misclassification. Second, prior approaches that use instrumental variables for bias correction, such as ForestIV yang2022achieving, typically rely on the “diversity" of individual learners within an ensemble Breiman2001 to discover valid instruments. EnsembleIV is much less dependent on this property. Even if the ensemble learning technique does not automatically produce diverse individual learners, and thus naturally valid instruments, EnsembleIV can still create valid instruments, leveraging the aforementioned IV transformation technique. This allows EnsembleIV to be applied with common ensemble learning algorithm (e.g., random forest, boosting, stacking), greatly enhancing its practical applicability. Third, as will be shown later, EnsembleIV outperforms benchmarks such as ForestIV and regression calibration in terms of estimation efficiency, producing estimates with smaller standard errors in the same sample.
In this section, we outline the theoretical components needed to create valid instrumental variables from ensemble learners, based on which we design the EnsembleIV algorithm. We use the notations in Table (ref) throughout the paper.
We first formulate the measurement error problem that arises when a machine-learning-generated variable is incorporated into a regression as an independent covariate, then describe the IV approach to address it. Without loss of generality, consider the following regression equation:\footnote{For generalized linear models (e.g., logistic regressions), the same measurement error problem can be formulated based on the latent variable model wooldridge2002, where the latent outcome $Y^*$ is linearly related to independent covariates in the same way as specified in (ref).}
where the idiosyncratic error term $\varepsilon$ is assumed to satisfy the following two assumptions:
Assumption I ensures that, in the absence of measurement error, we are working with a correctly-specified regression equation free from other sources of endogeneity. This allows us to focus on measurement error as the core identification challenge. Assumption II means that prediction errors associated with each individual learner do not have a direct impact on outcome $Y$ beyond the covariates already included in the regression. This setup follows that employed in previous work that deals with correcting estimation bias from machine-learning-generated covariates yang2018mind,qiao2021correcting,fong2021machine,yang2022achieving,allon2023machine,zhang2023debiasing.
Under the setting that we consider, $X$ can only be observed in a relatively small set of labeled data ($D_{label}$) but is unobserved in the much larger unlabeled data ($D_{unlabel}$, and $|D_{unlabel}| \gg |D_{label}|$). A typical reason for the size disparity between labeled and unlabeled data is the cost of labeling $X$. As an example, in studies that investigate the relationships between textual sentiment and certain dependent variables of interest tirunillai2012does,goh2013social,moreno2014doing, researchers often need to manually label the sentiment (e.g., by hiring crowd workers, such as Amazon Mechanical Turkers, who read the text and assign sentiment labels).\footnote{We acknowledge that such manual labeling processes can potentially introduce inter-labeler disagreement or error into the labeled data. This can be especially challenging for subjective or open-ended labeling tasks, where different labelers may have different subjective beliefs about “ground truth" (e.g., labeling “beauty" based on the image of a person). There are strategies that researchers can adopt to mitigate this type of error, e.g., adding more labelers for each data instance, refining labeling instructions, measuring inter-rater reliability among labelers, and resolving discrepancies via discussion hopkins2007extracting. Here, we focus strictly on the prediction error from machine learning models, as labeling error is fundamentally different -- labeling disagreement tends to reflect uncertainty in the labeling process whereas prediction error represents imperfect machine learning predictions.} Doing so for a large volume of text can be very cost-prohibitive. Of course, one can directly estimate regression equation (ref) using $D_{label}$; however, the limited size of $D_{label}$ may result in imprecise estimates, potentially impairing subsequent statistical inference and the ultimate managerial or policy decision. To make use of the larger $D_{unlabel}$, one increasingly common approach has been to build machine learning models on $D_{label}$ to predict the values of $X$ on $D_{unlabel}$.
However, the predicted values $\widehat{X}$ generally contain some degree of prediction error, defined as $e := \widehat{X} - X$. When $\widehat{X}$ is added into regression equation (ref) as a surrogate for $X$, the prediction errors become measurement errors, and the regression that is actually estimated can be written as:
We denote $u := \varepsilon - e\beta$ as the error term in the estimated regression, and $Cov(\widehat{X}, u) = Cov(X + e, \varepsilon - e\beta) = -\beta (Cov(X,e) + Var(e))$. Aside from a very unlikely scenario wherein $Cov(X,e)$ precisely cancels out $Var(e)$, the regression will suffer from endogeneity and produce biased and inconsistent estimates.
It is important to note that we do not make assumptions regarding the distributions of $X$ or $\widehat{X}$ -- they can be continuous or binary -- nor do we restrict the relationship between $e$ and $X$ (i.e., $e$ could be independent of, or correlated with, $X$). In other words, our proposed approach is applicable for both continuous and binary mismeasured covariates, and under both independent and correlated measurement error.
A standard solution to the measurement error problem is the instrumental variable approach buzas1996instrumental,greene2003econometric,hu2008instrumental. A valid instrumental variable (IV), $Z$, needs to satisfy two conditions, namely (i) relevance, $Cov(Z, \widehat{X}) \neq 0$, i.e., the IV should be correlated with the mismeasured covariate; and (ii) exclusion, $Cov(Z, u) = -\beta Cov(Z, e) = 0$, i.e., the IV should be uncorrelated with the measurement error. If such an IV can be identified, one can obtain consistent estimates via IV regression, e.g., two-stage least-squares greene2003econometric,wooldridge2002. IV regression is known to be consistent in the limit and significantly mitigates estimation bias in (large) finite samples.
Despite the attractive theoretical properties of the IV approach, in practice, it is often challenging to identify instruments that satisfy the exclusion condition because the measurement error (or, more generally, the source of endogeneity) is not directly observable. As such, the exclusion condition is typically argued conceptually conley2012plausibly. Again, in our context, this is not the case; measurement error can be observed directly in the labeled sample of data that is used to train and test the machine learning model.
We now describe the key theoretical components of EnsembleIV. Suppose the first-phase machine learning model is an ensemble model with $M$ individual learners (e.g., a random forest with $M$ trees). Instead of the common practice of using the aggregated predictions, $\widehat{X} = \frac{1}{M} \sum_i \widehat{X}^{(i)}$, in the second-phase regression, we consider using the predictions from one individual learner, $\widehat{X}^{(i)}$, employing those predictions as a mismeasured variable, and relying on the predictions from other individual learners, $\widehat{X}^{(j)} (j \neq i)$, as candidate instruments.\footnote{In numeric prediction tasks, $\widehat{X}^{(i)}$ is comprised of the raw numerical predictions of the $i$-th learner. In binary classification tasks, it is the probability prediction of the $i$-th learner.}
Because the different individual learners are all predictive of the target ground truth to some extent, their predictions will generally be correlated with one another, i.e., $Cov(\widehat{X}^{(j)}, \widehat{X}^{(i)}) \neq 0$. This satisfies the relevance condition for valid IVs. However, there is no theoretical guarantee that these candidate IVs will satisfy the exclusion condition, which requires that $Cov(\widehat{X}^{(j)}, \widehat{X}^{(i)} - X) = 0$. To address the exclusion violation, we apply an IV transformation technique developed based on prior work by nevo2012identification.
Specifically, suppose a candidate IV, $Z$, is “imperfect" in the sense that it violates the exclusion condition, i.e., $Cov(Z,u) = -\beta Cov(Z,e) \neq 0$. nevo2012identification, relaxing the assumption of strict exclusion, allow an instrumental variable to be correlated with the error term, and apply the weaker assumptions that (i) the correlation between the instrumental variable and the error term has the same sign as the correlation between the endogenous regressor and the error term and (ii) that the instrumental variable is less correlated with the error term than is the endogenous regressor. They derive analytic bounds for the parameters and demonstrate the application of their method to produce set estimates.
Leveraging portions of their theoretical observations and the fact that we observe measurement error directly in our setting, we develop a procedure to transform a candidate IV so that it satisfies the exclusion restriction. Denote correlation coefficient $\rho_{Zu} = \frac{Cov(Z,u)}{\sigma_Z \sigma_u}$ and $\rho_{\widehat{X}u} = \frac{Cov(\widehat{X},u)}{\sigma_{\widehat{X}} \sigma_u}$ and consider the following quantity:
Conceptually, $\lambda$ measures how $Z$ compares to $\widehat{X}$ in terms of the degree to which each variable violates the exclusion condition, i.e., how much “better" (or “worse") $Z$ is than $\widehat{X}$ in this regard. Consequently, $\lambda$ can be used to construct a valid IV, as described in the following Lemma.
Proof of this lemma is included in Appendix (ref). In a typical measurement error setting, $\lambda$ is unknown because the measurement error component, $e$, is unobservable. However, when the mismeasured variable is generated by a machine learning model, we can empirically estimate $\lambda$ using the labeled data. Specifically, for a pair of individual learners $i \neq j$, where $\widehat{X}^{(i)}$ serves as the endogenous covariate and $\widehat{X}^{(j)}$ serves as the (imperfect) IV, the following Algorithm (ref) first estimates $\lambda$ based on $D_{test}$ and then employs it to transform the IV into a valid instrument.
Ideally, to transform the IV on $D_{unlabel}$ (lines 5-6 in Algorithm (ref)), one would need $\widehat{\lambda}_{unlabel}$ defined based on predicted and ground truth values on $D_{unlabel}$. This quantity cannot be estimated in practice (because the ground truth values of $X$ on $D_{unlabel}$ are not observable), which is why we rely on $\widehat{\lambda}$ estimated on $D_{test}$ for IV transformation. Importantly, we establish in Appendix (ref) (specifically, in Lemma (ref)) that $\widehat{\lambda}$ estimated on $D_{test}$ tends to (the non-computable) $\widehat{\lambda}_{unlabel}$ (under mild regularity conditions). Therefore, $\widehat{\lambda}$ estimated on a finite (sufficiently large) $D_{test}$ should still provide a reliable approximation of $\widehat{\lambda}_{unlabel}$ to enable IV transformation.
Taking $\widehat{X}^{(i)}$ as the mismeasured covariate, we apply Algorithm (ref) to each $\widehat{X}^{(j)}$ where $j \in \{1, \ldots, M\} \setminus i$, which produces $M-1$ transformed IVs denoted as $\widetilde{Z}^{(j)}$. Following this procedure will often yield numerous asymptotically valid instruments for the IV estimation, but it may be sub-optimal to employ all of them in this manner for several reasons. First, some IVs may only be weakly correlated with the mismeasured variable, and having weak instruments is known to produce inconsistent estimates bound1995problems,stock2002testing,andrews2019weak. Second, with a finite dataset, it is possible that the value of $\widehat{\lambda}$ estimated from $D_{test}$ may not be sufficiently close to (the uncomputable) $\widehat{\lambda}_{unlabel}$ for some pairs of individual learners $i \neq j$. As a result, although the transformed IVs likely improve upon the original (non-transformed) ones in terms of exclusion, they still may not be perfectly excluded. With these “almost valid" instruments, it becomes even more important to select strong IVs to use in estimation murray2006avoiding. Finally, having too many instruments can overfit the endogenous covariate and fail to isolate the exogenous variations roodman2009note. Because of these reasons, we next carry out an “IV selection" step where a subset of strong IVs is selected from the pool of transformed IVs. We specifically consider three different approaches for IV selection, all of which are carried out on $D_{unlabel}$ because this step relies on the transformed IVs $\widetilde{Z}^{(j)}$:
Combining the IV transformation and selection procedures, we now describe the EnsembleIV estimation procedure in Algorithm (ref). Note that we do not need to specify which technique is used to build the “ensemble model" because EnsembleIV is generally applicable to ensembles of machine learning models -- this includes common ensemble learning techniques such as random forest, gradient boosting, and stacking, as well as ensembles of different individual learners (e.g., an ensemble of neural networks with varying architectures or hyperparameters).\footnote{In boosting-based ensemble techniques (such as XGBoost or LightGBM), $\widehat{X}^{(i)}$ in Algorithm (ref) should be the $i$-th cumulative learner, as will be discussed in Section (ref).}
The “IV Estimation" step amounts to any standard IV regression approach established in the econometrics literature, e.g., two-stage least-squares greene2003econometric,wooldridge2002 for linear models and two-stage residual-inclusion terza2008two,angrist2008mostly,wan2018general for generalized linear models (GLMs). Notably, instead of relying on the IV estimates associated with a single individual learner (and its instruments), we average over the IV estimates obtained from each individual learner. Doing so can produce more precise estimations.\footnote{For example, in the case where different $\boldsymbol{\widehat{\beta}}_{IV}^{(i)}$ are independent of each other, averaging can reduce the standard errors by a factor of $\sqrt{M}$. Even if different $\boldsymbol{\widehat{\beta}}_{IV}^{(i)}$ are correlated, variance reduction will still occur, though it will be impaired to the degree correlation exists.}
Under a few mild regularity assumptions, we prove that the EnsembleIV estimator is consistent and asymptotically normal (see Theorem 1 in Appendix (ref)). That said, we also bootstrap our entire procedure to estimate standard errors, given the bootstrap often yields an approximation sampling distribution of an estimator that is more accurate in finite samples than first-order asymptotic approximations hall1992bootstrap, horowitz2019bootstrap,davidson2006boostrap, including in the context of IV regression coefficients davidson2008bootstrap. In particular, we draw from each data partition $\{D_{train}, D_{test}, D_{unlabel}\}$ separately, with replacement, and carry out Algorithm (ref) on each bootstrapped sample. This represents an end-to-end approach to approximate the uncertainty in EnsembleIV estimates and to support inference.
When applying EnsembleIV in practice, the testing partition ($D_{test}$) plays a uniquely important role. It first serves as a dataset to evaluate the out-of-sample performance of the machine learning model trained on $D_{train}$ (enabling model fine-tuning and selection). In addition, it is used for the estimation of $\lambda$ and IV transformation. Given a fixed $D_{label}$, the common practice in building machine learning models is to have a relatively small $D_{test}$ as compared to $D_{train}$. This, however, may lead to noisy $\lambda$ estimates and jeopardize the validity of transformed IVs. At the same time, while having a large $D_{test}$ generally improves $\lambda$ estimates, this results in a small $D_{train}$ partition, which generally produces a poorer machine learning model and greater measurement error.
To address this conundrum, we implement a “cross-fitting" strategy to enhance EnsembleIV's performance given a relatively small $D_{test}$. The idea of cross-fitting is to use part of the available data to estimate certain nuisance variables (i.e., variables that are not of interest themselves but are needed to estimate the target parameters, such as $\lambda$ in our case), use the rest of the data for target parameter estimation, and then swap the role of each partition and repeat. Finally, we average the resulting estimates of the target parameter to obtain our final estimates. This process allows one to leverage all available data for estimation (which improves precision) while continuing to employ different samples of data for nuisance variable and target parameter estimation (thereby avoiding over-fitting). This strategy has proven successful in other contexts, most notably the Double Machine Learning estimator of chernozhukov2018double. In Algorithm (ref), we integrate cross-fitting with EnsembleIV as part of a sample-splitting procedure.
Finally, we provide a remark on EnsembleIV's computational complexity. When using EnsembleIV, the most time-consuming aspect will typically relate to the training of the first-phase machine learning model and its application to both $D_{test}$ and $D_{unlabel}$, to obtain predictions from the individual learners. The complexity of this step is increasing in $M$ (although the complexity may vary depending on the specific ensemble learning technique and its software implementation). However, importantly, this step only needs to be done once (or $K$ times with cross-fitting). Next, during the IV transformation and selection step, the theoretical complexity is $O(M^2)$, because the algorithm iteratively designates each individual learner as the endogenous covariate and performs transformation and selection over the other $M-1$ individual learners. That being said, this step is much less time-consuming because IV transformation/selection mainly involves computing correlations across previously calculated variables. Lastly, the IV estimation step involves running $M$ separate IV regressions (or $KM$ regressions with cross-fitting). This step is, again, quite efficient. Overall, the majority of computational overhead relates to training the first-phase machine learning model, while the part of EnsembleIV that is quadratic in $M$ can be implemented very efficiently.
EnsembleIV offers a systematic approach to generate, transform, select, and use individual learners within an ensemble machine learning model as instrumental variables to address the measurement error problem. In this section, we carry out comprehensive empirical evaluations to understand the effectiveness of the approach. In the first set of evaluations, we rely on synthetic data and simulations to assess EnsembleIV's correction performance for several widely used regression specifications. In the second set of evaluations, we apply EnsembleIV on a real-world dataset consisting of unstructured textual data to demonstrate how the estimator can be used in combination with deep learning techniques.
Recall the two-phase approach of combining machine learning with statistical inference. In the first, prediction phase, a supervised learning model is trained using the labeled data that can predict the variable of interest in the unlabeled data. In the second, statistical inference phase, the predicted values are incorporated into a regression model as an independent covariate. In our simulation studies, we leverage public datasets to implement the machine learning phase, and we synthesize data atop those real-world samples to inform our assessments of the statistical inference phase.
We employ two publicly available datasets as the basis for the first-phase prediction, namely the “Bike Sharing" dataset fanaee2014event and the “Bank Marketing" dataset moro2014data. Both datasets are available from the UCI Machine Learning Repository\footnote{Bike Sharing data: \url{https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset}; Bank Marketing data: \url{https://archive.ics.uci.edu/ml/datasets/bank+marketing}.} and are commonly used as benchmarking datasets in the machine learning literature. The Bike Sharing dataset contains 17,379 bike rental records. From this sample, we use date and timestamps in tandem with weather-related features to predict the logarithm of hourly rental volumes (denoted as $lnCnt$). The Bank Marketing dataset contains 45,211 phone calls conducted as part of a Portuguese bank's telemarketing campaign. From this sample, we use client features (e.g., demographics and financial status) to predict conversion, i.e., subscription to a term deposit account with the bank (denoted as $Deposit$). Notably, the target variable to be predicted is continuous in the Bike Sharing dataset and binary in the Bank Marketing dataset, allowing us to evaluate EnsembleIV's performance under both variable types.
We partition each dataset into $D_{label}$ and $D_{unlabel}$. Later in the cross-fitting process, we further partition $D_{label}$ into 4 folds (i.e., $K=4$), equivalent to using 75% of labeled data for training the machine learning model and 25% of labeled data for testing. The following Table (ref) summarizes the key aspects of each dataset used in the machine learning phase. In Appendix (ref), we provide descriptive evidence for the effectiveness of the IV transformation and selection steps, i.e., that they can indeed create (approximately) valid instruments and select strong IVs.
Next, we simulate data atop these samples for use in the statistical inference phase. We consider two widely-used regression specifications: a linear regression and a logistic regression. The data generation process of each specification is included as follows:
where $MLV$ refers to the machine-learning-generated variable (i.e., the target variable in the machine learning phase, $MLV \in \{lnCnt, Deposit\}$); $W_1 \sim Uniform(-10,10)$ and $W_2 \sim N(0, 10^2)$ are independent covariates (i.e., control variables); $\varepsilon \sim N(0, 2^2)$ represents the independent error term in linear specifications; and $Y$ is the dependent variable.
For each specification, we run three sets of regressions: (a) a “Biased" regression, using the aggregated prediction values produced by the random forest for $MLV$ on $D_{unlabel}$; (b) an “Unbiased" regression, using the true values for $MLV$ on $D_{label}$, and (c) the “EnsembleIV" regression with cross-fitting. While each IV selection method can improve EnsembleIV's estimation, it is hard to determine a priori which method would work the best on a specific dataset; we therefore report EnsembleIV estimates under all three methods (i.e., top-3, PCA with top 3 components, and LASSO). The Biased regression produces estimates that one obtains when combining machine learning with statistical inference absent any bias correction effort. The Unbiased regression, in contrast, produces the most precise estimates one can get using the ground truth values available only in the labeled data. We expect the EnsembleIV estimates to be less biased than the Biased estimates (due to the use of IV for correction), and more precise than the Unbiased estimates (due to its ability to leverage the much larger unlabeled data for estimation).
To obtain the empirical distributions of coefficient estimates, we repeat each simulation experiment, end-to-end, 100 times (including data partitioning, ensemble model training, regression data synthesis, and statistical estimation). We report the mean and standard deviation of each coefficient estimate across 100 simulation runs as the point estimate and standard error, respectively. Further, to quantify the degree of estimation bias, as well as the effectiveness of correction, we report the Estimation Mean-Squared Error (MSE) of each regression, defined as
where $\boldsymbol{\beta} = (\beta_0, \beta_{MLV}, \beta_{W_1}, \beta_{W_2}) = (1, 0.5, 2, 1)$ denote the true coefficient values. The estimation MSE is computed as the sum of bias and variance over all coefficients estimated in the regression, with smaller estimation MSE intuitively corresponding to `better' estimates, i.e., estimates that are statistically closer to the true coefficients. We report the simulation results on the Bike Sharing data in Table (ref) and those on the Bank Marketing data in Table (ref).
We observe that EnsembleIV is consistently able to reduce estimation bias (compared to the Biased estimates) and improve estimation precision (compared to the Unbiased estimates). Across all specifications on both datasets, EnsembleIV produces (i) point estimates that are closer to the true coefficient values than the Biased estimates, and (ii) standard errors that are smaller than those of the Unbiased estimates (these precision gains are driven by the much larger $D_{unlabel}$). EnsembleIV's advantages are also reflected in the estimation MSE metric -- EnsembleIV estimates have lower estimation MSE than both the Biased and Unbiased estimates (with a single exception -- the LASSO-selected IVs and linear second phase regression on the Bank Marketing dataset -- though we still observe bias reduction). Further, IV selection based on Top-3 and PCA largely results in similar EnsembleIV estimates, whereas the LASSO selection approach tends to produce somewhat worse point estimates, yet lower standard errors. The above simulation studies demonstrate EnsembleIV's utility to mitigate estimation biases caused by the measurement error problem, while also leveraging the larger $D_{unlabel}$ to improve estimation precision.
An important advantage of EnsembleIV is that it can be applied with different common ensemble learning techniques. Owing to the IV transformation step, valid IVs can be created even if the resulting individual learners do not have inherently low error correlations. In this section, we demonstrate how EnsembleIV can be used with gradient boosting and evaluate its performance using the two synthetic datasets.
We choose gradient boosting here because it represents a starkly different type of ensemble learning technique than random forest. Under gradient boosting, a collection of $M$ boosting trees are trained in a sequential manner, which we denote with a tuple $(1, \ldots, M)$. Each boosting tree $i > 1$ aims to predict the errors (i.e., residuals) of the earlier boosting trees $1 \leq j < i$. To apply EnsembleIV with gradient boosting, we first use the original $M$ boosting trees to construct $M$ “cumulative" learners, where the $i$-th cumulative learner is the aggregation of all boosting trees $j \in (1, \ldots, i)$. We use the predictions from these $M$ cumulative learners as the endogenous variables and their candidate instruments. Intuitively, different cumulative learners are all somewhat predictive of the target ground truth, thereby supporting the relevance condition of IVs. However, unlike in a random forest, there is little reason to believe that the prediction errors of cumulative learner $i$ are only weakly correlated with the predictions of cumulative learner $j$, because the two cumulative learners have a non-trivial overlap in their constituent boosting trees. The likely violation of the exclusion condition makes this a challenging and meaningful test of EnsembleIV's ability to create valid IVs from data.
We apply EnsembleIV with XGBoost chen2016xgboost, a popular gradient boosting algorithm, on both the Bike Sharing dataset and the Bank Marketing dataset with a linear and logit second-phase regression, respectively. The simulation setups are the same as before, and the results are reported in Tables (ref) and (ref) for the two datasets, respectively.
On the Bike Sharing dataset, the XGBoost model achieves a prediction RMSE of 0.55 (evaluated on $D_{test}$). This is quite low considering that the target ground truth ($lnCnt$) ranges from 0 to 6.88 with a standard deviation of 1.48. Accordingly, the relatively small measurement error in the Biased regression does not result in substantial estimation biases (especially under the linear regression specification). In this case, we observe that EnsembleIV estimates are still very close to the true coefficient values though with smaller standard errors and estimation MSEs than the Unbiased estimates. In other words, EnsembleIV does not introduce any additional biases in cases where measurement error is small to begin with. On the Bank Marketing dataset, directly using the XGBoost model's predictions in the Biased regression has resulted in over 50% underestimation of $\beta_{MLV}$. Importantly, EnsembleIV is again able to mitigate the estimation biases and maintain smaller standard errors and estimation MSEs than the Unbiased estimates. All three IV selection approaches produce similar EnsembleIV estimates in terms of the estimation MSE scores. On the Bank Marketing dataset, we again observe that the LASSO selection approach leads to slightly worse point estimates yet smaller standard errors than the other two selection approaches.\footnote{Based on the simulation results in Tables (ref)-(ref), we note that the performance of the different IV selection methods in finite samples is likely affected by idiosyncrasies in data, machine learning techniques, and second phase regression specifications. In practice, instead of choosing one selection method up front, researchers could repeat estimation with different methods to establish robustness.}
While the previous section demonstrates EnsembleIV's bias correction effectiveness on both synthetic and real-world datasets, in this section we focus on benchmarking EnsembleIV against alternative bias correction approaches and evaluating EnsembleIV's performance robustness. For simplicity, we present EnsembleIV results with PCA-based IV selection in this section, although the other selection methods yield qualitatively similar findings.
Using the same simulation setup as before, we compare the correction performance between EnsembleIV (with 4-fold cross-fitting) and its “closest kin", namely ForestIV yang2022achieving. We note that the original design of ForestIV does not have a cross-fitting component. For a more comprehensive comparison, we therefore report ForestIV estimates both with and without 4-fold cross-fitting. The results are presented in Tables (ref) and (ref), for the two datasets, respectively.
We find that EnsembleIV estimates have lower standard errors than ForestIV estimates whether or not we implement ForestIV with cross-fitting. This offers empirical support to EnsembleIV's advantage in estimation efficiency. Point estimates from EnsembleIV are also closer to the true coefficient value than those from ForestIV, except in the case of a linear second-phase regression on Bank Marketing data. Overall, this set of benchmarking analyses suggests that EnsembleIV represents a more efficient estimator than ForestIV (even after the enhancement of cross-fitting). EnsembleIV's efficiency benefits can be attributed to its design. ForestIV identifies the single “best" combination of individual learners that would yield estimates exhibiting the smallest deviation from the unbiased estimates obtained on $D_{label}$, whereas EnsembleIV averages over all combinations. Accordingly, in a given dataset, EnsembleIV will generally be more efficient than ForestIV.
Importantly, we note that EnsembleIV's advantage over ForestIV also goes beyond estimation efficiency. First, the validity of ForestIV depends on the specific theoretical properties of a single machine learning technique, the random forest algorithm. Further, that validity has only been formally established for scenarios involving continuous machine-learning-generated variables yang2022achieving. In contrast, EnsembleIV can be applied with ensembles of machine learning models -- this includes common ensemble learning techniques such as random forest, gradient boosting, and stacking, as well as ensembles of different individual learners (e.g., an ensemble of neural networks with varying architectures or hyperparameters). EnsembleIV is also provably valid for both binary and continuous machine-learning-generated variables (see EnsembleIV's asymptotic properties, shown in Appendix (ref), which do not depend on the type of machine-learning-generated variables). Second, whereas ForestIV is passive, relying on the chance discovery of individual learners from a trained random forest that happens to yield predictions meeting the exclusion condition, EnsembleIV is active, purposefully transforming all candidate instruments to achieve compliance with the exclusion condition. For this reason, EnsembleIV has a more solid theoretical grounding and it provides the ability to average over multiple IV estimates to improve estimation efficiency.\footnote{We note that the manuscript introducing ForestIV yang2022achieving also experimented with the idea of using subsets of trees as endogenous covariates and IVs yang2022achieving, but the authors found it to have limited effectiveness. We repeat the same exploration and again find it to result in generally worse correction performance. See Appendix (ref) for more details.}
Next, we benchmark EnsembleIV with two other bias correction approaches, respectively Regression Calibration gleser1992importance,carroll1995measurement and a GMM estimator proposed by fong2021machine, which we refer to as “FT-GMM" for short. While Regression Calibration is a classic method from the measurement error literature, FT-GMM represents a recently developed method to address the same estimation bias problem that we consider in this paper.\footnote{Another representative bias correction method is Simulation Extrapolation Stefanski1995,Kuchenhoff2006. We note that ForestIV has been shown to outperform SIMEX in yang2022achieving, and we therefore omit SIMEX as a benchmark here.}
The Regression Calibration approach relies on a subsample of data where both the ground truth values and the predicted values of the machine-learning-generated variable can be observed, i.e., $D_{test}$ in our context. Using this set of data, it learns a calibration model by regressing the ground truth values onto the predicted values (controlling for other exogenous covariates). Then, the calibration model is applied to $D_{unlabel}$ to obtain calibrated predictions. Finally, the second-phase regression model is estimated using those calibrated predictions (rather than the original predictions from the first-phase machine learning model).
The FT-GMM approach starts by following the same steps as Regression Calibration --- using $D_{test}$ (“validation data” in the authors' notation) to learn a projection of the ground truth values onto the predicted values (although the authors interpreted this step as the first phase of a 2SLS estimation), then apply this projection to $D_{unlabel}$ (“primary data” in the authors' notation). Then, unlike Regression Calibration, the FT-GMM approach seeks to combine the 2SLS estimates on $D_{unlabel}$ with the OLS estimates on $D_{train} \cup D_{test}$ via GMM, concatenating the moment conditions of the two estimators. Doing so potentially allows FT-GMM to achieve higher estimation efficiency. Importantly, however, the proposed FT-GMM approach does not support GLMs, hence we cannot apply it to a scenario involving Logistic regression in the second phase.
Therefore, we apply both benchmark approaches to the Bike Sharing and Bank Marketing datasets in the case of a linear second-phase regression. However, we limit our consideration to the Regression Calibration approach when it comes to a Logistic second-phase regression. In each case, we compare the benchmark methods' correction performance with that of EnsembleIV (with 4-fold cross-fitting). Neither benchmark was initially designed with the cross-fitting component; however, we report their performance both with and without cross-fitting, for the sake of completeness. The results are presented in Tables (ref) and (ref).
With a linear second-phase regression, all three methods arrive at fairly similar estimation results on the Bike Sharing data. EnsembleIV produces better point estimates whereas the two benchmarks have smaller standard errors. After applying the cross-fitting enhancement, the two benchmarks can achieve slightly smaller empirical estimation MSE values. In comparison, on the Bank Marketing data, EnsembleIV has smaller standard errors than the benchmarks (even after applying cross-fitting), though the point estimate is not as close to the true value. Under a logistic second-phase regression, EnsembleIV achieves a smaller estimation MSE than Regression Calibration in most cases, except on Bank Marketing data where EnsembleIV has a similar estimation MSE as the cross-fitting-enhanced Regression Calibration. Overall, EnsembleIV performs competitively against the benchmarks and, unlike FT-GMM, naturally accommodates GLM second-phase regressions without the need for regression-specific modifications to the correction procedure.
In addition to benchmarking, we also derive a better understanding of EnsembleIV's performance sensitivity with changes to several parameters that researchers have to decide upon when applying the approach:
For simplicity, we focus on the Bike Sharing data and the linear second-phase regression. We start with the same parameter configuration as our main simulation studies (Table (ref)) and vary one parameter at a time in a series of experiments. Specifically, we vary $|D_{label}| \in \{1000, 2000, 3000, 4000, 5000\}$, $M \in \{50, 100, 150, 200, 250\}$, $\sigma_{\varepsilon} \in \{0.5, 1.0, 2.0, 3.0, 4.0\}$, and $K \in \{2, 3, 4, 5, 6\}$. In the following Figure (ref), we plot the point estimates of the machine-learning-generated covariate and its empirical 95% confidence intervals of the Biased, Unbiased, and EnsembleIV estimators as we vary each parameter.
We discuss several patterns that can potentially assist researchers in setting these parameters. In the top-left plot, as more labeled data becomes available, the EnsembleIV point estimate converges nicely to the true value of the coefficient. Such a convergence is likely the result of two factors. First, having a larger $|D_{label}|$ means that more training data (i.e., $\frac{K-1}{K}$ of $|D_{label}|$) are available, which can improve the predictive performance of the first-phase ensemble model. Indeed, across the five sizes of $|D_{label}|$, we observe a consistent decline of prediction RMSE values (averaged over 4-fold cross-validation) of 0.68, 0.60, 0.58, 0.55, and 0.52, aligned with better EnsembleIV estimates. Second, increasing $|D_{label}|$ also results in more testing data (i.e., $\frac{1}{K}$ of $|D_{label}|$), which benefits the precision of $\lambda$ estimation and IV transformation. Meanwhile, we note that the confidence interval of the Unbiased estimate expectedly shrinks. The top-right plot shows that EnsembleIV estimates are somewhat insensitive with respect to the number of individual learners. This is likely because, on the Bike Sharing data, varying $M$ from 50 to 250 does not significantly affect the predictive performance.\footnote{The prediction RMSE values (averaged over 4-fold cross-validation) are 0.572, 0.576, 0.577, 0.568, 0.569, respectively.}
The bottom-left plot shows that, as the standard deviation of the regression error term becomes larger, all three estimators have wider confidence intervals, while the point estimates of EnsembleIV and Unbiased regression remain very close to the true coefficient value. This demonstrates the robustness of EnsembleIV to increases in exogenous noise in the second-phase regression model. Hence, efforts to include relevant control variables in the regression, which can reduce the standard deviation of the error term, would benefit EnsembleIV by producing more precise estimates. Finally, the bottom-right plot indicates that EnsembleIV is fairly insensitive with respect to the number of folds used in cross-fitting on the Bike Sharing dataset, suggesting that it is not necessary to use a large number of folds when applying EnsembleIV, given cross-fitting with more folds is computationally more costly.
We repeat these sensitivity analyses on the Bank Marketing dataset. Because this dataset is larger, we also impose a larger increment in the size of labeled data, i.e., $D_{label} \in \{1500, 3000, 4500, 6000, 7500\}$. Variations of the other parameters are kept the same as above. The results are presented in Figure (ref).
We observe patterns consistent with those noted above. EnsembleIV again shows a trend of convergence as the size of labeled data increases, given the increased accuracy of the first-phase predictive model and more testing data for IV transformation. As we increase the number of individual learners from 50 to 100, there is a bit more bias in EnsembleIV's point estimates -- this also aligns with the fact that the first-phase random forest model with 50 trees achieved a slightly smaller validation loss than that with 100 trees. Overall, our results when varying $D_{label}$ and $M$ indicate that having a better-performing first-phase model generally improves EnsembleIV's correction performance. Therefore, researchers' efforts to fine-tune the first-phase machine learning models (e.g., via parameter tuning or model selection) are instrumental to EnsembleIV's performance. Notably, in the bottom-right plot, we see that a larger number of folds in cross-fitting hurts EnsembleIV's performance, likely because the size of testing data becomes too small to enable sufficiently good IV transformation. Considering these observations in tandem with our observations on Bike Sharing data, we recommend using a small number of folds in cross-fitting. Doing so simultaneously benefits correction performance and reduces computational overhead.
In addition to the parameters examined above, researchers also need to decide $n$, the number of strong IVs, if Top-$n$ or PCA-based methods are used for IV selection. In the case of Top-$n$, selected IVs are those having the strongest correlation with the endogenous covariate; and in the case of PCA, the top $n$ components (each being a linear combination of original IVs) are also those that account for the most variation in the endogenous covariate. Theoretically, using a small number of strong IVs is preferred over using a large number of potentially weaker IVs because the latter can lead to misleading inferential results stock2002survey. Under the same empirical setup as before, we vary the number of top IVs used from 1 to 5. The results, summarized in Table (ref), show that changes in the number of IVs yield very little differences, across the board. Therefore, a small number of strong IVs is generally sufficient in practice. In terms of exactly how many strong IVs one should use, we note that donald2001choosing propose an approach based on minimizing an approximated MSE, which can serve as a guideline.
Finally, we note that EnsembleIV's correction performance under different IV selection methods (and with different numbers of selected IVs) in finite samples is likely affected by idiosyncrasies in data, machine learning techniques, and second-phase regression specifications. Accordingly, in practice, researchers may be best served by repeating estimation with different selection methods to establish robustness.
We now conduct an in-vivo evaluation of EnsembleIV using a real-world dataset collected from Facebook business pages. Business pages on Facebook are public spaces managed by firms to share marketing content (e.g., product promotions), who seek to interact with Facebook users (e.g., answering their questions or complaints). Several prior studies have examined firm- and user-generated content in this context, and their impact on user engagement or firm performance goh2013social,lee2018advertising,yang2019understanding. For this evaluation, we estimate how the sentiment (positive or negative) of a user-generated post affects content engagement, measured as the number of comments that the post receives from other users. This evaluation serves a few purposes. First, it represents a realistic empirical context, where neither the data-generation process nor the true coefficient values are known. Second, it demonstrates how EnsembleIV can be combined with modern deep-learning techniques for language tasks. Finally, because the underlying data-generation process is unknown, there are no guarantees that the assumptions behind EnsembleIV (as discussed in Section (ref)) will hold. We use this real-world dataset as a testbed to develop a data-driven diagnostic procedure that can assess the satisfaction of Assumption II.\footnote{Note that we still maintain Assumption I, i.e., the mean independence of the regression error term with respect to regression covariates, in the absence of measurement error. If this assumption fails, it implies that other types of endogeneity exist in the regression that go beyond our focus.}
We use the dataset collected by yang2019understanding, which includes a total of 429,015 user-generated posts from 2012, appearing on the business pages of 41 Fortune 500 companies. To build a sentiment classifier, a random subset of 10,157 posts had been manually labeled by 5 independent Amazon Mechanical Turk workers, and their ground truth sentiment labels (positive or negative) were determined via majority voting (i.e., $|D_{label}| = 10157$). The remaining 418,858 unlabeled posts make up $D_{unlabel}$.
For the first-phase machine learning task, we train a binary classifier to predict the sentiment of each post. We consider two different machine learning techniques when building the sentiment classifier. In the first approach, which we refer to as “BoW + RF", we adopt a traditional bag-of-words representation for posts, namely a term-frequency inverse document-frequency (TF-IDF) matrix.\footnote{We follow standard text pre-processing steps to construct the TF-IDF matrix, including tokenization, punctuation removal, lower-casing, and stop-word removal. We further remove words that are too frequent or too infrequent (showing up in more than 10,000 documents or fewer than 50 documents). The final vocabulary contains 785 unique words.} We train a random forest classifier comprised of 100 trees, taking the TF-IDF matrix as input. In the second approach, which we refer to as “BERT + RF", we encode each post as a 768-dimension embedding vector using the BERT pre-trained model devlin2018bert. As we have learned from the sensitivity analyses in Section (ref), it is beneficial to fine-tune the first-phase model and also restrict the number of folds in cross-fitting. Therefore, we consider three different ensemble techniques (random forest, XGBoost, and LightGBM) and fine-tune the number of individual learners in each model via 3-fold cross-validation. In this application, the LightGBM technique ke2017lightgbm ultimately outperforms random forest and XGBoost. We summarize the accuracy and F-scores of the best-performing models in Table (ref).
For the second-phase inference task, we estimate a linear regression on $D_{unlabel}$ with the number of comments received by each post serving as the dependent variable, log-transformed to reduce skewness. The sentiment of each post, denoted as $sentiment$, takes a value of 1 if the predicted sentiment is positive and 0 otherwise. This variable serves as the mismeasured explanatory variable. In addition, we control for several other covariates, including (i) the (log-transformed) number of words in a post ($wordcount$) and (ii) the content type of the post ($type$, one of photo, status, video, or link). In Table (ref), we report estimation results from the Biased regression, Unbiased regression, and EnsembleIV, all with bootstrapped standard errors. For EnsembleIV, we apply 3-fold cross-fitting and the PCA-based IV selection (the other IV selection approaches produce similar results). Note that the unbiased estimates do not rely on machine learning predictions and thus remain fixed across results associated with each of the two sentiment classifiers.
Although the true coefficient values are unknown, we can nevertheless use the unbiased estimates as a reasonable baseline. Several findings are worth noting. First, directly using the aggregated predictions from LightGBM in the second-phase regressions results in nontrivial bias. The magnitude of the coefficient on $sentiment$ is significantly underestimated. Second, EnsembleIV can mitigate the estimation bias on the sentiment variable in both cases, producing point estimates on $sentiment$ that are close to the unbiased value and with smaller standard errors. This again demonstrates EnsembleIV's utility to reduce bias compared to the biased estimate and improve precision compared to the unbiased estimate.
Recall from Section (ref) that the validity of EnsembleIV relies on two assumptions regarding the error term of the second-phase regression, $\varepsilon$, namely that (i) in the absence of measurement error, $\varepsilon$ is mean independent of regression covariates, and (ii) conditional on the regression covariates, $\varepsilon$ is uncorrelated with the prediction errors of all individual learners. While Assumption I simply means that there are no sources of endogeneity other than measurement error, Assumption II is needed to establish the validity of using (transformed) individual learners as IVs. Recent work has begun to consider scenarios in which Assumption II is violated wei2022unstructured,allon2023machine. This can happen when the machine learning models exploit “peripheral features" of inputs that affect $Y$ but are not included in $X$ or $\boldsymbol{W}$.
We develop a data-driven diagnostic procedure that a researcher can employ to evaluate whether this peripheral feature challenge is likely to be present in the variables generated by an ensemble learning technique on a particular dataset. The procedure uses a subset of the labeled data to evaluate the correlations between individual learners' prediction errors and the residuals of the measurement-error-free regression (as an unbiased and consistent estimate of $\varepsilon$). Specifically, the diagnostic is designed as follows:
Because $D_{diagnostic}$ is a random sample that shares the same distribution as the actual $D_{unlabel}$, the estimated $Corr(\widetilde{Z}^{(j)} - X, r_{diagnostic})$ contains testable evidence whether generated and transformed IVs in this dataset tend to exhibit correlation with the regression error term. In practice, researchers can select their preferred test of correlation and significance level to assess whether the empirical correlation is statistically significant. Here we provide one exemplary illustration of how this could be done. Let $\widetilde{Z}^{(j)}_{i}$ denote a transformed IV for endogenous covariate $\widehat{X}^{(i)}$ ($i \neq j$). Given an ensemble model of $M$ individual learners, the EnsembleIV procedure uses $M(M-1)$ such transformed IVs (i.e., $M-1$ transformed IVs for each of the $M$ individual learners designated as the endogenous covariate). One could construct a joint test of all $M(M-1)$ correlations, which offers an empirical assessment of how severe the challenge of “peripheral features" may be. This joint test is illustrated in the following hypothesis:
One example test statistic that can be used to test the above hypothesis is:
i.e., the average absolute correlation between the error of a transformed IV and the residual term. To perform a hypothesis test, we would need the distribution of this test statistic under the null of zero correlation. Deriving the theoretical sampling distribution of this test statistic is challenging. However, we can rely on a standard permutation test welch1990construction to approximate the distribution. Specifically, we permute the ordering of records in $r_{diagnostic}$ and re-compute the test statistic. Over a large number of permutations, this procedure generates an empirical distribution of the test statistic under the null of zero correlation. Then, we can compute the empirical $p$-value by comparing the observed test statistic from the original (“un-permuted") sample to this empirical distribution.\footnote{Note that the procedure we propose considers forming a single sample of $D_{diagnostic}$, providing a single $TS$ to test. However, Type II errors can be decreased with a $K$-fold sample-splitting procedure, similar to what we implemented in the “cross-fitting” enhanced version of Ensemble IV. Specifically, $D_{label}$ can be randomly partitioned into $K$ folds and, in each iteration, a unique fold is utilized as $D_{diagnostic}$, producing $K$ independent test statistics and associated $p$-values. The $p$-values from these tests can be combined using standard methods fishers-method to leverage the full $D_{label}$ for testing.}
Should the null hypothesis of zero correlation be rejected (i.e., the empirical $p$-value falls below the significance threshold chosen by the researcher), indicating a nontrivial peripheral feature challenge, the researcher will need to take steps to address the issue. For this purpose, we develop an extension to EnsembleIV that explicitly accounts for the potential correlation between prediction errors and peripheral features (see Appendix (ref) for details). That said, if the null hypothesis cannot be rejected, then the researcher may have some confidence that the peripheral features challenge is absent or is too weak to be detectable in the given empirical context (and may, therefore, not pose a significant threat to estimation and inference). In this case, the researcher can proceed by applying the standard form of EnsembleIV, as presented earlier.
We carry out the diagnostic procedure on the Facebook dataset. Among 10,157 labeled data, we reserve 1,000 as $D_{diagnostic}$. We train a LightGBM model with the BERT representation because model evaluation shows that it produces worse performance than the bag-of-words representation (perhaps the model has missed some important features, making the “peripheral features" challenge more likely to be present in this setting). We provide a descriptive summary of the empirical correlations in Table (ref) below.
We find that the correlations between individual learners' prediction errors and the regression residual are small even before IV transformation, potentially indicating that the BERT representation already captures a sufficient amount of textual information. After IV transformation, the correlations become even weaker -- a paired $t$-test shows significantly weaker correlations after IV transformation ($p < 0.001$). Applying the permutation test discussed above, we fail to reject the null hypothesis of zero correlation ($TS = 0.016, p = 0.5982$, computed based on 10,000 permutations). While this is not confirmation that the null is true, it does indicate that the empirical correlations we observed are consistent with those we might expect under zero correlation. Therefore, the problem of “peripheral features" appears sufficiently negligible in this specific dataset and using the original EnsembleIV method is reasonable.
In addition to the above experiment on the Facebook data, we have constructed a set of simulations that help demonstrate that the proposed diagnostic procedure is useful for detecting invalid IVs when there are known strong correlations with peripheral features. Details of these simulations are provided in Appendix (ref).
Finally, we acknowledge that more research is needed to fully understand (1) the prevalence and challenge of “peripheral features" in practice and (2) how robust different correction methods (EnsembleIV included) are to this challenge. Recent work including wei2022unstructured and allon2023machine represent laudable steps towards a better understanding of these issues, and we expect the literature will continue proposing approaches that expand the ability to perform inference in the presence of different degrees of the “peripheral features" challenge.
Despite increasing popularity in empirical studies, the integration of machine-learning-generated variables into regression models for statistical inference suffers from a measurement error problem, which can bias estimation and threaten the validity of inference. In this paper, we develop a novel approach to alleviate associated estimation biases. Our proposed approach, EnsembleIV, creates valid and strong instrumental variables from individual learners in an ensemble model and uses them to obtain corrected estimates that are robust to measurement error.
The core idea behind EnsembleIV is that individual learners in an ensemble (e.g., individual trees from a random forest) typically produce correlated predictions. As a result, for a specific individual learner, predictions from other individual learners can serve as relevant but not perfectly excluded instrumental variables. Furthermore, the technique based on the work of Nevo and Rosen nevo2012identification enables the transformation of these candidate IVs into (asymptotically) valid ones. Our proposed EnsembleIV procedure follows this idea to generate, transform, and select IVs in a systematic and data-driven manner and use them to correct estimation bias arising from the measurement error.
Our empirical evaluations, using both synthetic and real-world datasets, show that EnsembleIV can effectively reduce estimation biases for both linear and generalized linear regression specifications, and its effectiveness is not restricted by the type of machine-learning-generated variable (binary or continuous) or the specific ensemble learning technique (random forest or boosting). For unstructured data (e.g., text), EnsembleIV can be easily used in conjunction with representations learned by deep learning networks. Compared with several alternative bias correction methods, including ForestIV yang2022achieving, regression calibration gleser1992importance, and GMM fong2021machine, EnsembleIV can improve the estimation efficiency over regression calibration and ForestIV, while offering greater flexibility than GMM (i.e., EnsembleIV does not need regression-specific adjustments). Moreover, decreased predictive performance of the first-phase ensemble learning model (e.g., due to having a harder predictive task or fewer predictive features) translates to wider confidence intervals, but not increased biases, of EnsembleIV estimates. Overall, EnsembleIV represents a versatile and highly practical tool for drawing robust statistical inference with machine-learning-generated variables.
Our work also opens up several directions for future research. As shown in our sensitivity analyses with ForestIV, it appears that EnsembleIV's correction effectiveness is best when the size of the labeled data partition is at least moderately large. Future work may seek to improve the small sample performance of EnsembleIV -- this can enhance EnsembleIV's usefulness in settings where it is simply infeasible to obtain more than a small amount of labeled data. Further, recognizing the emerging popularity of Double/Debiased Machine Learning chernozhukov2018double, it would also be interesting to combine EnsembleIV with DML from both theoretical and empirical perspectives.