Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
55,290 characters · 16 sections · 41 citation commands
Achieving Reliable Causal Inference with Data-Mined Variables: A Random Forest Approach to the Measurement Error Problem
\doublespacing
Advances in predictive machine learning have enabled researchers to extract useful information from various types of data, such as text and images, which would otherwise be difficult or costly to codify at scale. For example, recent academic work has highlighted that cutting-edge prediction techniques are now capable of inferring the socioeconomic properties of a localized population (e.g., income/racial distribution) from the models and makes of cars appearing in Google Street View images gebru2017using and detecting adverse drug events based on drug attributes ryu2018deep. These measurements, now available at scale and with little cost, can enable empirical investigations of important questions in economics, health care, and many other domains.
Indeed, many researchers have begun doing exactly that, first using predictive machine learning to construct or populate a variable of interest, e.g., using text mining tools to predict text sentiment, and then including that variable into econometric models as an independent covariate. This practice has become prevalent in multiple social science domains, including economics jelveh2015political, political science Fong2017, and management yang2018mind.
However, recent studies have also noted that attempts to draw inferences based on this recipe are likely to suffer from endogeneity due to measurement error yang2018mind. This is because predictions from machine learning models are inevitably imperfect, and prediction errors carry over to subsequent econometric models as measurement error, leading to biased and inconsistent parameter estimates. Measurement error may lead to overestimation or underestimation of coefficients loken2017measurement, and the degree of bias can be substantial, even when the machine learning model achieves reasonable predictive performance yang2018mind. The estimation biases stemming from measurement error in machine-learning-generated covariates can thus undermine the validity of subsequent causal inferences and decision making.
In this paper, we propose a novel approach to address this problem. Our approach is based on the notion of instrumental variable regression, a well-established method of resolving endogeneity in the econometrics literature, including endogeneity deriving from measurement error greene2003econometric. We make use of a notably unique feature of this problem setting of applying machine learning followed by regression. Specifically, we leverage the fact that predictive machine learning models are typically trained and evaluated using data for which true labels (assumed to be perfectly measured) are available and which are used to quantify prediction error and model performance. This perfectly measured set of data offers a unique opportunity to overcome difficulties commonly associated with evaluating the validity of instruments.
To find candidate instruments, we rely on random forest Breiman2001, an ensemble learning approach that aggregates a collection of individual decision trees (weak learners) to arrive at accurate predictions. Prior work has demonstrated that the performance of random forests depends jointly on (i) the degree to which trees comprising the forest yield correlated predictions and (ii) the degree to which the the trees yield weakly correlated prediction errors Breiman2001,bernard2010study. We demonstrate that these notions are closely related to the relevance and exclusion criteria that underpin valid instrumental variables. Based on this, we look to explore the random forest ensemble to identify sets of individual trees such that one tree's predictions might serve as the endogenous covariate in the econometric model of interest, while other trees' predictions serve as its instruments, thereby alleviating estimation biases due to measurement error. Drawing on theories from the instrumental variable and random forest literature, we develop algorithms to implement this idea, empirically selecting the best set of individual trees to mitigate biases in coefficient estimates. We term our procedure ForestIV.
We conduct two sets of comprehensive simulation experiments, considering the cases where a data-mined covariate is continuous vs. binary, and thus respectively suffers from either continuous measurement error or misclassification. In both cases, we show that ForestIV can effectively mitigate estimation biases. We also report sensitivity analyses of ForestIV and benchmark its performance against three alternative bias correction methods.
It should be noted that ForestIV provides a general-purpose method for correcting bias with machine-learning-generated covariates, whether derived from structured or unstructured data (e.g., text or images). For scenarios involving structured data, random forest is widely applicable to various supervised machine learning problems, and, for a large number of real world prediction problems, random forest is among the most accurate techniques fernandez2014we. However, even in scenarios involving unstructured data, where other techniques (e.g., deep neural networks) may be the state-of-the-art, random forest can be usefully combined with them. For example, random forest can be stacked with the intermediate representations learned by a neural network; that is, the output of an intermediate layer of the network, which encodes informative, high-level features learned from the unstructured data, can serve as the input features to the random forest algorithm. Notably, this practice is quite common in transfer learning, where a supervised machine learning model is built based on features produced by another technique goodfellow2016deep.
Our paper makes several notable contributions. First, we theoretically and empirically show that the proposed ForestIV approach effectively addresses the estimation biases in econometric models due to measurement error in machine-learning-generated covariates. Therefore, ForestIV improves the robustness of causal inferences and decisions derived from procedures combining machine learning with econometric analyses. Second, we design data-driven procedures that leverage the labeled data (used to build and evaluate the machine learning model) to empirically select instruments that are most suitable for bias correction purposes. Third, ForestIV represents a novel method to automatically obtain candidate instruments from the output of the random forest technique. This provides a viable solution to the often-challenging problem of identifying valid instruments.
Our work is informed by the econometrics literature on measurement error, instrumental variables, and generated regressors, as well as the machine learning literature on random forests. We provide a review of relevant literature in Appendix (ref). In this section, we provide a description of the measurement error problem resulting from machine learning predictions of a continuous covariate. We then introduce the theoretical justifications and implementation details of our proposed ForestIV solution.
We begin by setting up the measurement error problem for a continuous covariate. Note that, for expositional simplicity, we use a simple linear regression as a representation of the econometric model to be estimated, but the underlying theoretical arguments can be generalized to other econometric specifications, e.g., generalized linear models.\footnote{For example, a binary response model -- Probit or Logit -- can be formulated as a latent linear model with a binary transformation of the dependent variable, with the measurement error specified for the latent linear model.}
Consider a linear regression model,
where $Y$ represents the dependent variable, $\{X,\boldsymbol{Z}\}$ represent the independent covariates ($\boldsymbol{Z}$ includes control variables and a constant term), $\varepsilon$ is the exogenous random error term, and $\boldsymbol{\beta}=\{\beta_X,\boldsymbol{\beta_Z}\}$ denotes the model coefficients to be estimated. Importantly, $X$ is not directly observed in the data, and we instead rely on its surrogate, $\widehat{X}$, which is based on the predictions of a machine learning model, e.g., random forest. For instance, if $X$ represents poverty level in a neighborhood, then $\widehat{X}$ could be the predicted poverty level mined from Google Street View images. In contrast, covariates $\boldsymbol{Z}$ are directly observed in the data and are measured precisely (with no measurement error). Therefore, the actual estimation being conducted is a regression of $Y$ on $\{\widehat{X},\boldsymbol{Z}\}$. In this section, we assume $X$ and $\widehat{X}$ to be continuous variables, whereas $\boldsymbol{Z}$ can contain both continuous and categorical variables. We discuss the case when $X$ and $\widehat{X}$ are binary variables in later sections.
Because predictions of a machine learning model inevitably have some degree of error, $\widehat{X}$ is in general an imperfect surrogate for $X$, and contains continuous measurement error. The existence of measurement error in an independent covariate is known to result in biased regression estimates. As an illustrative example, consider the case of additive independent (otherwise known as classical) measurement error, where $\widehat{X}=X+e$, and $e$ is independent of $X$. The estimated regression equation is therefore
Because $Cov(\widehat{X},(\varepsilon-e\beta_X))=-\beta_X\sigma_e^2\neq0$, the regression suffers from endogeneity due to measurement error in $\widehat{X}$, resulting in biased coefficient estimates greene2003econometric. (See Appendix (ref) for a formal setup of this problem.)
Consider the task of building a random forest (or any predictive machine learning model) to predict $X$. A typical approach is to collect some amount of labeled data, where the outcome to be predicted is actually observed. More concretely, denote the entire dataset that a researcher has access to as $D$. Suppose the researcher takes a random subsample from $D$, denoted as $D_{label}$, and obtains the (precisely-measured) ground truth for that subsample, e.g., via manual labeling. Then, $D_{label}$ would be randomly partitioned into $D_{train}$ and $D_{test}$, where $D_{train}$ would be used to build the random forest model and $D_{test}$ to evaluate the resulting model's performance. For data in the remaining unlabeled dataset, $D_{unlabel}=D \setminus D_{label}$, the random forest model would then be deployed to generate predictions $\widehat{X}$.
As is common in a growing number of policy-relevant contexts, the dataset of interest often includes a large amount of unlabeled data. Due to the cost of obtaining ground truth labels, the size of $D_{label}$ is typically much smaller than the size of $D_{unlabel}$. As a result, researchers may not be able to estimate their econometric models of interest on $D_{label}$ with a satisfactory degree of statistical power given the likely large variance. That is, estimating the econometric models using information in $D_{unlabel}$ presumably has the potential to deliver substantial improvements in the precision of estimates (due to its larger sample size).
In this section, we consider the generation of instruments from random forest, which is at the core of the ForestIV approach. We first establish the setting and list assumptions, before providing formal theoretical results. We consider a random forest model with $M$ individual trees, indexed by $\{1,\dots ,M\}$, which is built based on $n$ training samples and $p$ features. On a new data point represented by feature vector $\mathbf{f} = \{f_1,\ldots,f_p\}$, denote the prediction of individual tree $i\in\{1,\dots ,M\}$ as $\widehat{X}^{(i)}$, and the prediction of the forest as $\widehat{X}$, where $\widehat{X} = \frac{1}{M} \sum_{i=1}^M \widehat{X}^{(i)}$. Given the ground truth, $X$, the prediction error is correspondingly defined as $e^{(i)}=\widehat{X}^{(i)} - X$. We make three assumptions underlying our theoretical results. The first two assumptions are adopted from scornet2015consistency, which establishes the consistency of random forest predictions, and the third assumption is standard in the measurement error literature.
Assumption 1 scornet2015consistency. The ground truth can be expressed as $X = \sum_{j=1}^p m_j(f_j) + \zeta$, where features $\{f_1,\ldots,f_p\}$ are uniformly distributed over $[0,1]^p$, $\zeta$ represents independent, centered Gaussian noise with finite variance, and each component $m_j(.)$ is continuous. This assumption states that the ground truth is the sum of univariate functions of input features. Although random forest is a nonparametric model, analysis of its properties is often facilitated within the framework of additive models scornet2015consistency.
Assumption 2 scornet2015consistency. Denote $t_n$ as the number of leaves in each tree, and $a_n$ as the number of training data points used to build each tree. Let $a_n \rightarrow \infty$, $t_n \rightarrow \infty$, we assume $t_n (\log a_n)^9/a_n \rightarrow 0$. This assumption, as a regularity condition, controls the rate at which the trees in the random forest grow.
Assumption 3 (Classical measurement error). As $n \rightarrow \infty$, the prediction error of an individual tree takes the classical form, i.e., $\lim_{n \rightarrow \infty} \mathbb{E}_i\mathbb{E}_{\mathbf{f}} Cov(X, e^{(i)}) = 0$. This assumption, notably common in the theoretical measurement error literature, implies that (asymptotically) the prediction error of an individual tree is uncorrelated with the ground truth hausman1991measurement,newey2001flexible,li2002robust,schennach2004estimation,schennach2013nonparametric.
This result, the proof of which can be found in Appendix (ref), implies that the expected covariance between one tree's prediction and another tree's prediction error goes to 0, which establishes an asymptotic guarantee for instrument validity and, therefore, the theoretical foundation for considering trees within a random forest as instruments for one another. To provide intuition for this foundation, we revisit the estimated regression $Y=\widehat{X}\beta_X+\boldsymbol{Z\beta_Z}+(\varepsilon-e\beta_X)$, where $\widehat{X}$ is endogenous. A valid instrument, $W$, should satisfy two conditions: (a) $Cov(W,\widehat{X}) \neq 0$, i.e., the instrument is correlated with the endogenous (mis-measured) regressor (i.e., the relevance condition); and (b) $Cov(W,\varepsilon-e\beta_X)=0$, i.e., the instrument is not correlated with the regression error term (i.e., the exclusion condition). Recall that $\varepsilon$ is exogenous in (ref), the true underlying regression equation of interest. Consequently, condition (b) is equivalent to $Cov(W,e)=0$, i.e., a valid instrument is not correlated with the measurement error in $\widehat{X}$. Now suppose we replace $\widehat{X}$ with $\widehat{X}^{(i)}$ (i.e., predictions of individual tree $i$ in the forest), and $W \equiv \widehat{X}^{(j)}$ (i.e., predictions of individual tree $j$ in the forest). We then have that $Cov(\widehat{X}^{(i)}, \widehat{X}^{(j)}) = Var(X) \neq 0$, meeting condition (a), and $Cov(e^{(i)}, \widehat{X}^{(j)}) \rightarrow 0$ as $n\rightarrow \infty$ (Theorem (ref)), meeting condition (b). As a consequence, $\widehat{X}^{(j)}$ is an asymptotically valid instrument for $\widehat{X}^{(i)}$.
More generally, Theorem (ref) tells us that, asymptotically, and under mild assumptions, in a random forest of $M$ trees, we can use predictions from any individual tree as the endogenous covariate in the regression model, and use predictions from the other $M-1$ individual trees in the random forest as valid instruments. Though this result provides a valuable theoretical foundation for the random forest algorithm as a generator of valid instruments, we do not yet know at what sample size the asymptotic properties will manifest. Therefore, at a finite sample size we can only consider the individual trees as candidate instruments, and we must carry out additional steps to identify which of these potential instruments provides evidence of having reached their asymptotic state. Moreover, the asymptotic result implies that with infinite data all individual trees are valid instruments for all other trees. However, in practice, only one pair of (endogenous and valid instrumenting) trees is required, which (given the asymptotic result) would appear plausible even in finite data.
We note that, in finite samples, the above asymptotic results with respect to random forest are in fact also reflected in empirical evidence. For example, bernard2010study show that the performance of random forest statistically improves with increases in the accuracy of individual trees and decreases in the correlation between their prediction errors. In other words, a well-performing random forest should consist of individual trees that are relatively accurate (high strength) and have only weakly correlated errors (low correlation) Breiman2001. We make the key observation that high strength and low correlation are closely related to the requirements of a valid instrumental variable.
This highlights an interesting and perhaps counter-intuitive characteristic of ForestIV. Because the predictive performance of an individual tree is typically worse than that of the entire random forest, we likely induce larger estimation biases by drawing on predictions from an individual tree as the endogenous covariate (rather than the aggregate prediction from the overall forest). However, this initial sacrifice is accompanied by the opportunity to leverage predictions from other trees as instruments, to address the estimation bias. In other words, for the purposes of causal inference, a more nuanced use of the entire random forest ensemble (rather than a traditional use of its aggregate predictions) allows the mitigation of the measurement error that is inevitably present in the model's predictions.
In practice, when using the predictions from a focal individual tree in a random forest as the endogenous covariate, it is necessary to select only a subset of the other trees' predictions for use as instruments, for several reasons. First, when there is only a finite set of training data, it is unlikely that all other trees have reached their (asymptotic) state of valid instrumentation for the focal tree. Second, due to randomness in constructing the random forest with finite data, even valid instruments can be invalid by chance, i.e., empirically $Cov(e^{(i)}, \widehat{X}^{(j)})$ is large. Third, using an excessive number of instruments can lead to over-fitting of the endogenous variables, including the endogenous components a researcher seeks to eliminate, and create estimation challenges roodman2009note. Finally, again due to randomness in the construction of a random forest, some instruments may be only weakly correlated with the endogenous covariate, despite meeting the exclusion requirement. Including these weak instruments in an instrumental variable regression can be counterproductive, yielding biased and inconsistent estimations hausman2001mismeasured.
In our setting, because we have access to a set of labeled data, we can assess instrument validity empirically, to a degree. Thus, after obtaining the predictions from each individual tree in the random forest, we focus on identifying a “desirable" subset of trees: one to be used as the endogenous covariate, and the remaining to serve as valid and strong instruments that can mitigate estimation biases. We decompose this task into three distinct steps, as follows:
The additional “validity checks" in Step 3 are necessary because, even though Steps 1-2 are designed to remove invalid instruments and select strong ones, there is limited theoretical guarantee that all selected instruments are always valid and strong in finite samples. The validity checks in the third step thus attempt to identify for which subset of trees the asymptotic properties appears to be present, reducing the likelihood that our approach produces erroneous results when predictions from all individual trees in the random forest are not suitable instruments. Moreover, with a finite sample, it is unlikely that instrument validity, particularly the exclusion restriction, will be satisfied exactly. Therefore, our procedure is consistent with the practice of using “plausibly exogenous" instruments for estimation in finite samples conley2012plausibly.
To rule out invalid instruments for a given endogenous covariate, we rely on information from the labeled data. Recall that the random forest model is built on $D_{train}$, and its performance is then evaluated on $D_{test}$. As a result, for $D_{test}$, we observe the ground truth, the model-predicted values, and thus the prediction errors (the difference between ground truth and prediction). Using this information on $D_{test}$, we can gauge the validity of using individual trees' predictions as instruments, and empirically rule out the invalid instruments that are strongly correlated with the measurement errors.
Grounded in prior work on instrument selection Belloni2012, we adapt a lasso-based heuristic procedure to identify and discard instruments violating the exclusion requirement. Without loss of generality, suppose predictions from the first individual tree on $D_{test}$, $\widehat{X}^{(1)}$, serve as the endogenous covariate, and the corresponding prediction error is denoted as $e^{(1)}$. We estimate a lasso regression of $e^{(1)}$ on the predictions of other individual trees on $D_{test}$, i.e., $e^{(1)} \sim \{\widehat{X}^{(2)}, \dots, \widehat{X}^{(M)}\}$. The set of regressors for which the lasso yields non-zero coefficients are then dropped, because their linear combination is determined to be a strong predictor of $e^{(1)}$, which implies that those regressors violate the exclusion restriction with respect to $\widehat{X}^{(1)}$. Conversely, the set of regressors with zero coefficients are retained, denoted by $\boldsymbol{V_1}$, because the lasso fails to provide evidence to suggest that they violate the exclusion restriction.
To select a set of sufficiently strong instruments from the remaining set of candidate instruments, for a given endogenous covariate, we adopt another heuristic approach motivated by Belloni2012. Consider $\widehat{X}^{(1)}$ as the endogenous covariate, and $\boldsymbol{V_1}$ as the set of instruments obtained in Step 1. We estimate a lasso regression in which the endogenous covariate is regressed on all available instruments (similar to the first stage of a 2SLS estimation), i.e., $\widehat{X}^{(1)} \sim \boldsymbol{V_1}$. The lasso attempts to shrink the coefficients of instruments that are conditionally uncorrelated with the endogenous covariate (i.e., weak in the presence of the other instruments) to zero, with the non-zero coefficients indicating a set of regressors predictive of $\widehat{X}^{(1)}$, which intuitively correspond to strong instruments. Because this step requires only the predictions from individual trees, it is carried out on $D_{test} \cup D_{unlabel}$.
Importantly, if the set of instruments changes during the two-step selection process (i.e., certain instruments are determined to violate exclusion or relevance requirements, and are thus dropped), we will repeat both of the lasso selection steps (1 and 2) with the remaining instruments, until the selection ceases to change. This iterative approach increases the likelihood that our selected instruments are simultaneously valid and strong. Moreover, we expect this procedure should work well with sufficient data, because when $\boldsymbol{V_i}$ contains only excluded instruments (satisfied asymptotically based on Theorem (ref)), Belloni2012 shows that asymptotically $\boldsymbol{S_i}$ will be the linearly optimal set of instruments. We offer greater detail about the instrumental variable selection procedure outlined in Steps 1 and 2 in Appendix (ref). In general, for each $\widehat{X}^{(i)}, i \in \{1, \dots, M\}$, we can use this procedure to select a set of strong, excluded instruments, $\boldsymbol{S_i}$.
Consider the econometric model of interest, $Y = X\beta_X + \boldsymbol{Z\beta_Z} + \varepsilon$. If all variables in the econometric model, i.e., $\{Y,X,\boldsymbol{Z}\}$, are directly observable in $D_{label}$, then one can obtain unbiased estimates of $\{\beta_X,\boldsymbol{\beta_Z}\}$, denoted as $\widehat{\boldsymbol{\beta}}_{label}$, by estimating the econometric model on the error-free $D_{label}$. In practice, the size of $D_{label}$ is often limited, due to the cost of acquiring labels. As a result, $\widehat{\boldsymbol{\beta}}_{label}$ may exhibit a large standard error and would not be particularly suitable for drawing causal inferences. Nonetheless, the estimates are unbiased and can be employed as a useful baseline for determining the quality of instrument estimations.\footnote{We note that if $D_{label}$ is sufficiently large, implying that researchers can afford to acquire a large set of labeled data with sufficient statistical power to estimate the econometric model of interest, then it is unnecessary to use machine learning models to construct / mine variables in the first place; the researcher should simply estimate the econometric model on $D_{label}$.} In this sub-section, we discuss the final selection procedure, based on a comparison of 2SLS estimates and the unbiased estimation using precisely measured covariates in $D_{label}$, i.e., $\widehat{\boldsymbol{\beta}}_{label}$.
Specifically, following Steps 1-2, we obtain a set of 2SLS estimates for each pair of $(\widehat{X}^{(i)}, \boldsymbol{S_i})$, using $\widehat{X}^{(i)}$ as the endogenous covariate, and $\boldsymbol{S_i}$ as its instruments, denoted as $\widehat{\boldsymbol{\beta}}_{IV}^i$. Denote the variance-covariance matrices of $\widehat{\boldsymbol{\beta}}_{IV}^i$ and $\widehat{\boldsymbol{\beta}}_{label}$ as $\widehat{\boldsymbol{\Sigma}}_{IV}^i$ and $\widehat{\boldsymbol{\Sigma}}_{label}$, respectively. To compare $\widehat{\boldsymbol{\beta}}_{IV}^i$ with $\widehat{\boldsymbol{\beta}}_{label}$, we use the Hotelling's $T^2$ test with unequal variance seber2009multivariate. This test is a multivariate generalization of the $T$ test, designed to evaluate a null hypothesis of mean equality between two vectors of random variables, whose joint distributions have unequal variances. The test statistic is: $H_i = (\widehat{\boldsymbol{\beta}}_{IV}^i - \widehat{\boldsymbol{\beta}}_{label})^T \left( \widehat{\boldsymbol{\Sigma}}_{IV}^i + \widehat{\boldsymbol{\Sigma}}_{label} \right)^{-1} (\widehat{\boldsymbol{\beta}}_{IV}^i - \widehat{\boldsymbol{\beta}}_{label})$. $H_i$ is asymptotically distributed as $\chi^2(K)$, where $K$ represents the total number of covariates seber2009multivariate. Therefore, if $H_i$ is larger than the critical value of $\chi^2(K)$ at a user-chosen significance level (we pick $\alpha=0.05$ for illustration purposes), then $\widehat{\boldsymbol{\beta}}_{IV}^i$ is significantly different from the unbiased $\widehat{\boldsymbol{\beta}}_{label}$, indicating that one or more instruments likely violate the relevance and/or exclusion requirement. Such $\widehat{\boldsymbol{\beta}}_{IV}^i$ is then discarded.
Meanwhile, suppose more than one $(\widehat{X}^{(i)}, \boldsymbol{S_i})$ pair has an associated Hotelling statistic below the critical value, such that each pair empirically yields estimates that are not significantly different from $\widehat{\boldsymbol{\beta}}_{label}$. We then measure the empirical bias and variance of each set of estimates, using empirical MSE, as follows: $$MSE_i = tr\left((\widehat{\boldsymbol{\beta}}_{IV}^i - \widehat{\boldsymbol{\beta}}_{label}) (\widehat{\boldsymbol{\beta}}_{IV}^i - \widehat{\boldsymbol{\beta}}_{label})^T + \widehat{\boldsymbol{\Sigma}}_{IV}^i\right)$$. Finally, we select the estimates that have the smallest empirical MSE, i.e., the estimates that have the smallest sum of empirical bias and variance. We note that testing these multiple hypotheses, without control, will potentially inflate our Type I error: the probability of incorrectly rejecting a truly valid tuple. Given that we likely have multiple tuples to choose from, our preference is to be conservative (over-reject tuples) and we prefer to make no inference rather than make an erroneous inference.
Note that the 2SLS variances (i.e., diagonal elements of $\widehat{\boldsymbol{\Sigma}}_{IV}^i$) associated with ForestIV estimates do not reflect the estimator's true variability, because we intentionally select estimates with low variances. To properly quantify the variability of ForestIV estimator, we propose to bootstrap $\{D_{label},D_{unlabel}\}$, apply the ForestIV procedure to each bootstrap sample, and obtain the empirical sampling distributions of ForestIV estimates. The variances of the sampling distributions serve as straightforward estimation of ForestIV variances. The pseudocode for ForestIV is summarized in the Appendix (ref).
Taking the proposed ForestIV procedure as a baseline, we also consider three possible alternative design choices, related to instrument discovery, selection, and estimation. First, a key feature of ForestIV is that we “break up" a random forest and exploit individual trees to discover instruments for bias correction. As an alternative, one might instead consider a sample splitting approach, where $D_{train}$ is split into two independent subsets and a random forest is built on each subset. Then, predictions from one forest might serve as an instrument for predictions from the other. This approach is intuitively appealing because the predictive performance of a forest is arguably better than that of a single tree, thereby reducing the measurement error problem to start with. In the same vein, a second alternative design of the instrument selection procedure in ForestIV is to use the aggregate predictions from a subset of trees (rather than predictions from a single tree) as the endogenous covariate and instrumental variables. Third, in the estimation step of ForestIV, while we propose to select the estimates that minimize empirical MSE, an alternative design is to average across all estimates that are not rejected by the Hotelling's $T^2$ test. In general, the relative utility of these alternative design choices is not clear a priori, thus we explore all of them empirically as part of the simulation experiments discussed in the following section.
Our first demonstration leverages the Bike Sharing Data fanaee2014event, which contains 17,379 instances of hourly bike rental activities. It is a commonly used dataset for benchmarking and evaluating machine learning models giot2014predicting. Moreover, this dataset assist in emulating an example of how machine learning could enable empirical studies of questions on the impact of mobility on important economic or social outcomes.
We randomly partition the dataset, such that 1,000 observations serve as $D_{train}$, 200 observations serve as $D_{test}$, and the remaining 16,179 observations serve as $D_{unlabel}$. This represents a realistic scenario where $D_{unlabel}$ is much larger than $D_{label}$. Using $D_{train}$, we build a random forest model of 100 trees to predict the log-transformed count of total hourly bike rentals (denoted as $lnCnt$, log-transformed to reduce skewness), based on 12 features, including the time of the rental as well as weather and seasonal information. Importantly, the random forest generates aggregate (ensemble) predictions, as well as predictions from each of its individual trees. We denote $\widehat{lnCnt}$ as the aggregate predictions, and $\widehat{lnCnt}_i$ as the predictions from individual tree $i \in \{1,\ldots,100\}$.
Next, we simulate an econometric model with $lnCnt$ as an independent covariate. The model specification is $Y=1+0.5lnCnt+2Z_1+Z_2+\varepsilon$, where $Z_1 \sim Uniform[-10,10]$, $Z_2 \sim N(0,100)$, and $\varepsilon \sim N(0,4)$. In the Bike Sharing dataset, $\sigma_{lnCnt}=1.5$, which is smaller than the standard deviation of the regression error term. Therefore, this simulation setup represents a realistic scenario where the variation in “noise" is comparable to that of the “signal". We keep the true regression coefficients fixed, i.e., $\beta_0=1, \beta_{lnCnt}=0.5, \beta_{Z_1}=2, \beta_{Z_2}=1$, in order to quantify the degree of estimation bias and the effectiveness of any corrections.
The above simulation procedure is repeated for 100 rounds. Within each simulation round, a random forest model is built based on a random split of the data (as described earlier), and an artificial dataset is generated for the econometric estimations. Specifically, we first estimate the biased regression, $Y \sim \beta_0+\beta_1 \widehat{lnCnt}+\beta_2Z_1+\beta_3Z_2$, where $\widehat{lnCnt}$ is the aggregate predictions from that round's random forest. This is precisely what would be done if one were to use the machine-learning-predicted covariate directly in the econometric model, without considering measurement error, as is commonly the case. We then apply ForestIV to obtain the corrected coefficients. Because an independent set of $\{D_{train}, D_{test}, D_{unlabel}\}$ is drawn in each simulation round, the ForestIV estimates across 100 simulation rounds naturally form the empirical sampling distributions. We report the standard deviations of the sampling distributions as the standard errors of ForestIV estimates.\footnote{We also obtained standard errors from our bootstrap approach, and note that the estimates are nearly identical.} Finally, besides the biased and ForestIV estimates, we also report the unbiased estimates from directly running the regression on $D_{label}$.
We provide some descriptive evidence regarding the effectiveness of our two-step lasso regression procedure to select valid and strong instruments from candidates in Appendix (ref). Next, we plot the sampling distributions of biased, unbiased, and ForestIV estimation on $lnCnt$ (i.e., the covariate generated by machine learning) across all 100 simulation runs in Figure (ref). We report complete results in Appendix (ref).
We make several observations. First, directly using the $\widehat{lnCnt}$ predicted by random forest in the regression model clearly results in biases. The coefficient on $lnCnt$ is overestimated on average. Second, compared to the biased regression, ForestIV effectively mitigates estimation bias on $lnCnt$. Third, compared to the unbiased estimation, ForestIV estimation has a “narrower" distribution (i.e., has smaller standard errors), indicating sizable increase in estimation precision.
We conduct several sensitivity analyses to understand performance of ForestIV with respect to several parameters. We summarize the key insights from the sensitivity analyses as follows, while referring to Appendix (ref). for detailed results. First, we examine how ForestIV estimations change as the size of the unlabeled dataset increases (Appendix (ref)). This helps illustrate the asymptotic properties of ForestIV estimations. We found that ForestIV estimates converge as more unlabeled data is added, with shrinking confidence intervals. The converged estimates substantially reduce bias and achieve smaller standard errors than the unbiased estimates. Second, we further increase the amount of noise in the data used for econometric estimation, and check the corresponding performance of ForestIV (Appendix (ref)). While standard errors of ForestIV estimates become larger, we still observe effective bias correction. Third, we vary the total number of trees in the random forest model, and found that ForestIV requires a reasonably large random forest to perform well (Appendix (ref)). Moreover, the Hotelling $T^2$ statistics turn out to be a useful metric to help determine the size of random forest.
Finally, we implement and empirically evaluate the relative benefit of the three alternative design choices noted earlier: (i) sample splitting (Appendix (ref)), (ii) using predictions from subsets of trees as the endogenous covariate and instruments (Appendix (ref)), and (iii) averaging across all ForestIV estimates not rejected by the Hotelling $T^2$ test (Appendix (ref)). We refer readers to the appendices for a detailed discussion of each simulation-based evaluation. On the whole, we find that none of these alternative designs outperforms our proposed ForestIV method (at least in the context of our Bike Sharing data and associated simulation setup). That said, the possibility remains that one or more of these alternatives could yield improvements under certain conditions, and each holds promise as an avenue of further exploration in future work.
In this section, we turn to the case of a binary misclassified (and thus endogenous) covariate, as would be generated by a machine-learning classifier model. It turns out that ForestIV still exhibits an ability to generate (and select) instrumental variables that produce improved estimates, although the underlying mechanisms are somewhat different from the case of a continuous endogenous covariate.
Suppose the binary outcome label can have values of 0 (negative class) or 1 (positive class). Consider a random forest classifier with $M$ decision trees. We again use notations $X$, $\widehat{X}$, and $\widehat{X}^{(i)}$ to represent the ground truth, prediction of the forest, and prediction of individual tree $i$. The prediction error of individual tree $i$ is defined as $e^{(i)} = \widehat{X}^{(i)} - X$. For any given data point, $e^{(i)}$ can take three possible values: 0 (correct prediction), 1 (false positive), and $-1$ (false negative).
Meanwhile, based on the econometric literature, e.g., angrist2008mostly, instrumental variables with binary endogenous covariates can be applied in the same manner as in the continuous case; that is, treat the variable as continuous, and employ a 2SLS estimation. The first stage of this 2SLS estimation amounts to a linear probability model. The predicted values for the endogenous covariate recovered from the first stage regression thus reflect exogenous variation in continuous class probabilities. These values would then be used in the second stage regression. Intuitively, to evaluate whether predictions from one tree, $j$, can serve as a valid instrument for predictions from another tree, $i$, we need to assess (1) $Cov(\widehat{X}^{(i)}, \widehat{X}^{(j)})$ (the relevance condition) and (2) $Cov(e^{(i)}, \widehat{X}^{(j)})$ (the exclusion restriction).
The first condition is typically satisfied, because two individual trees from a reasonably well-performing random forest are both somewhat predictive of the outcome, i.e., $Cov(\widehat{X}^{(i)}, \widehat{X}^{(j)})\neq 0$. The second condition can be written as $Cov(e^{(i)}, e^{(j)} + X)$. Accordingly, we next provide several theoretical results to characterize $Cov(e^{(i)}, e^{(j)})$ and $Cov(e^{(i)}, X)$, respectively.
All proofs are included in Appendix (ref). This theorem suggests that a well-performing random forest would have relatively small $Corr(|e^{(i)}|, |e^{(j)}|)$. Typically, because $Corr(|e^{(i)}|, |e^{(j)}|) \neq Corr(e^{(i)}, e^{(j)})$, this result seems to indicate that the exclusion restriction of instrument validity, $Cov(e^{(i)}, e^{(j)} + X) = 0$, is not satisfied. However, the following two theorems show that $Cov(e^{(i)}, X)$ is always non-zero (i.e., the classical measurement error assumption is never true for binary misclassification), and can be offset, or cancelled out, $Cov(e^{(i)}, e^{(j)})$, thereby making the exclusion condition plausible.
For notation simplicity, denote the probability that $X = \alpha, \widehat{X}^{(i)} = \beta, \widehat{X}^{(j)} = \gamma$ as $p_{\alpha \beta \gamma}$ ($\alpha, \beta, \gamma \in \{0,1\}$), and denote the probability that $X = \alpha$ as $p_{\alpha \bullet \bullet}$.
Theorem (ref) shows that the prediction error of an individual tree is always negatively correlated with the ground truth. Theorem (ref) suggests that, a few corner cases aside, the correlation between prediction errors of two individual trees is positive.\footnote{One such corner case that $Cov(e^{(i)}, e^{(j)}) < 0$ is when $p_{011}=p_{100}=0$ and $(p_{010}-p_{101})(p_{110}-p_{001}) < 0$, which means that the predictions from the two trees are never wrong at the same time, and their misclassification patterns satisfy a stringent condition. Such corner case is not easily realized.} As a result, $Cov(e^{(i)}, X)$ is likely to offset $Cov(e^{(i)}, e^{(j)})$, leading to a relatively small value of $Cov(e^{(i)}, e^{(j)} + X)$.\footnote{This is empirically supported in our simulation experiments using the Bank Marketing data, discussed in the next section. In particular, the average covariance between prediction errors of any two trees, $\mathbb{E}_{i\neq j}Cov(e^{(i)}, e^{(j)})$, is 0.071 (correlation is 0.465). Meanwhile, the average covariance between a tree's prediction errors and the ground truth, $\mathbb{E}_i Cov(e^{(i)}, X)$, is -0.079 (correlation is -0.601). As a result, the average covariance between one tree's predictions and another tree's prediction errors, $\mathbb{E}_{i\neq j}Cov(e^{(i)}, \widehat{X}^{(j)})$, is only -0.008 (correlation is -0.061).} In other words, in the case of a binary endogenous covariate generated by a random forest classifier, other trees' predictions can still plausibly serve as instrumental variables.
We demonstrate the performance of ForestIV in the case of binary classification using the Bank Marketing data moro2014data as an example dataset, which contains 45,211 records related to a bank's telemarketing efforts. We randomly partitioned the data into 1,500 observations to serve as $D_{train}$, 500 observations to serve as $D_{test}$, and the remaining 43,211 observations to serve as $D_{unlabel}$. Using the training data, we build a random forest classifier comprised of 100 trees to predict a binary outcome, $Deposit$, representing whether a client subscribed to a term deposit as a result of the phone call, based on 16 attributes describing the client and the marketing campaign. This dataset is an example of how machine learning can enable empirical studies of questions in direct marketing or customer relationship management.
We next simulate an econometric model: $Y=1+0.5Deposit+2Z_1+Z_2+\varepsilon$, where $Z_1 \sim Uniform[-1,1]$, $Z_2 \sim N(0,1)$, and $\varepsilon \sim N(0,4)$. As before, we repeat the simulation for 100 rounds. Within each round, we estimate the biased regression (directly using the random forest predictions, $\widehat{Deposit}$, in the regression), the unbiased regression obtained on $D_{label}$, and the corrected coefficient obtained from the ForestIV procedure on $D_{unlabel}$. We plot the distributions of biased, unbiased, and ForestIV estimation on $Deposit$ in Figure (ref), and report the full results in Appendix (ref).
Consistent with previous simulations, we observe that directly using the random forest's predictions in our regression leads to severe biases. The coefficient on $Deposit$ is underestimated. ForestIV is again effective in mitigating the biases. Finally, compared to unbiased estimation, ForestIV achieves gains in estimation precision, as indicated by its “narrower" distribution. In general, this set of simulations confirms the effectiveness of ForestIV for a binary endogenous covariate. We repeat all the sensitivity analyses that were conducted previously in the continuous covariate scenario and observe consistent insights. All results are described in Appendix (ref).
In addition to the above simulation experiments, we conduct benchmarking analyses to compare the relative correction performance of ForestIV with those of three other approaches from the existing literature: Simulation-Extrapolation Cook1994, Latent Instrumental Variables ebbes2005solving, and regression adjustment for nonparametrically generated regressors meng2016linear. First, SIMEX is an alternative correction approach that has previously been demonstrated to perform well in the same problem context. We run two sets of simulations to show that ForestIV achieves better correction performance than SIMEX. We also identify a systematic “blindspot" in the SIMEX correction procedure that occurs when measurement error is correlated with other precisely-measured covariates in an econometric model. While SIMEX will produce provably erroneous correction results in the presence of such correlation, empirical results suggest that our ForestIV approach does not suffer from this limitation. Second, with the LatentIV approach, a latent instrumental variable is modeled and jointly estimated with the main regression to address the endogeneity issue. Our simulation again shows that ForestIV outperforms LatentIV in bias correction. Third, in the generated regressors literature, meng2016linear have proposed a method to explicitly adjust coefficient estimates to account for biases due to measurement error in linear regressions, deriving from mis-measured, nonparametrically generated regressors. Once more, our simulation shows that ForestIV outperforms this method in bias correction. Thus, in sum, ForestIV is found to outperform all three benchmarks in our simulation experiments (see Appendix (ref) for details).
To summarize, we introduce a new approach, ForestIV, which addresses bias in regression estimates that is attributable to (predictive) measurement error in data-mined covariates. With a continuous endogenous covariate, the intuition behind ForestIV is that a high-performing random forest will be comprised of (i) trees that are individually accurate in their predictions and, thus, “overlap", offering “repeated measures" of true, exogenous variation in the data-mined variable, and (ii) trees that exhibit low correlations in their prediction errors, which in tandem with the former point implies that trees make “different" mistakes, and thus embed orthogonal measurement errors. Our approach is closely connected to the idea of using multiple error-prone measures as instrumental variables blackburn1992unobserved,hausman1995nonlinear,lewbel2019using. With a binary endogenous covariate, while trees no longer necessarily have low error correlations, we show that instrument validity is nonetheless plausible.
The application of our approach in empirical contexts has the potential to improve the precision and robustness of estimations, and thus subsequent decision-making. Meanwhile, our approach shows the possibility of automatically generating candidate instruments based on an ensemble learning technique, which complements the emerging literature on the use of machine learning methods for causal inference athey2016recursive,mcfowland2018efficient. At the core of ForestIV is the fundamental trade-off between an estimator's bias and variance, which together describe its statistical risk. ForestIV attempts to provide an estimate with reduced overall risk, one with substantially lower bias than the biased regression and lower variance than the unbiased regression. We provide several additional guidelines for using ForestIV in practice in Appendix (ref).
Several future research directions are worth pursuing. For example, while this paper focuses on selecting valid and strong instruments from a given random forest, future work might look to leverage a specialized random forest algorithm that explicitly aims to minimize individual trees' prediction error correlations. The Rotation Forest algorithm blaser2016random and the Dynamic Random Forest algorithm bernard2012dynamic represent two such attempts. Another direction to pursue is to generalize ForestIV to bagging-based machine learning models, more broadly. Intuitively, because individual learners from a bagging model are trained on different bootstrap samples of the training data, they are likely to generate correlated predictions and weakly correlated prediction errors. Future work can investigate whether general bagging models can produce useful instruments, whether the types of individual learners (e.g., decision tree or other techniques) affect the validity of instruments, and the performance of those alternative ensembles relative to ForestIV.