Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
80,312 characters · 19 sections · 60 citation commands
Estimation and Inference of Treatment Effects with L2-Boosting in High-Dimensional Settings
\doublespacing
Boosting algorithms are popular in machine learning and have proven to be useful for prediction and variable selection (nr:buhlmann.hothorn:2007). Nevertheless, in many applications, researchers are interested in inference on pre-specified variables, and this requires a different approach. Often these variables are so-called treatment or policy variables that researchers would like to learn and make inferences about, particularly in high-dimensional settings.
It is well-known that inference after model selection may lead to invalid results when the selection step is not taken into account or safeguarded against. For an overview of the pitfalls and challenges of post-selection inference in general, we refer to nr:leeb.potscher:2005. In this paper, we describe our approach to address this issue. First, we substantially extend the techniques in SL_2016 to provide a set of novel convergence results on iterated post-$L_2$-Boosting and orthogonal $L_2$-Boosting. These results are derived under approximate sparsity and without assuming the beta-min condition. Second, we provide results for valid inference in high-dimensional settings in which post- or orthogonal $L_2$-Boosting is applied for variable selection and estimation. For this, we combine the convergence results with the techniques of orthogonalized moment conditions as described in CCDDHN:2016.
Boosting has proven very valuable for prediction. We show in this paper that it can also be applied to (causal) inference. In particular, we consider the case of the estimation of a treatment effect with many control variables, and the case of instrumental variable (IV) estimation such as two stage least squares procedure with many potential instruments. The first case can also be interpreted as estimation and inference on a prespecified variable in a high-dimensional linear regression model estimated with $L_2$-Boosting, where all other variables are considered as nuisance variables. As the first ingredient of our estimation method, it relies on the so-called orthogonalized or Neyman moment conditions. This theory was developed by Belloni, Chernozhukov, Hansen, and coauthors in a series of papers. The case of instrumental variables is analyzed in BCCH:2012, and the treatment effect case in BelloniChernozhukovHansen2011. Papers with extensions of the general idea are CCDDHN:2016 and CHS:2016. The second ingredient is rates of convergence that are fast enough to estimate the nuisance part of the orthogonal moment condition. Our results only rely on the standard assumptions in the high-dimensional literature, and nearly-Lasso convergence rates are derived for the iterated post-$L_2$-Boosting ($\mathtt{I}$-$\mathtt{pBA}$) and orthogonal $L_2$-Boosting ($\mathtt{oBA}$) algorithms. In addition, a data-driven early stopping criterion is proposed and we show that such stopping criterion achieves nearly-Lasso convergence rate under our assumptions. These results might be of independent interest since Boosting is widely used and the results for Boosting in the high-dimensional case under such weak assumptions have been missing in the literature to date.
Additionally, we conduct extensive simulation studies and compare the Boosting algorithm with Lasso in various high-dimensional settings. These results might give guidance to empirical researchers to decide when to choose which algorithm. We conclude that $L_2$-Boosting with a theoretical-grounded stopping criterion is the more efficient method with respect to computation time and also provides the best estimation accuracy in our simulation study. We also show the computational advantage of $L_2$-Boosting over Lasso by comparing the CPU time in a high-dimensional (causal) inference framework. Finally, we conduct an empirical analysis of how the pre-merger overlap of bank branch networks in the United States can affect the post-merger stock returns of acquirer banks. In this setting, many potential instrumental variables are available. We show how $L_2$-Boosting can be used to select these in a data-driven, principled way and to estimate the treatment effect of interest.
Boosting algorithms are one of the major recent advances in machine learning and statistics. One of the earliest examples is Freund and Schapire's AdaBoost algorithm for classification (freund:1997), which has led to numerous variants that have proven to be competitive in terms of their prediction accuracy in a variety of applications and their strong resistance to overfitting as shown in nr:buhlmann.hothorn:2007. Boosting methods were originally proposed as ensemble methods, which rely on the principle of generating multiple predictions and majority voting (averaging) of the individual classifiers. An important development in the analysis of Boosting algorithms was Breiman’s interpretation of Boosting as a gradient descent algorithm in a function space (nr:breiman:1996, nr:breiman:1998), inspired by numerical optimization and statistical estimation. Building on this insight, nr:friedman.hastie.ea:2000 and nr:friedman:2001 embedded Boosting algorithms into the framework of statistical estimation and additive basis expansion. This also enabled Boosting to be applied to regression analysis. Boosting for regression was proposed by nr:friedman:2001, and then nr:buhlmann.yu:2003 defined and introduced $L_2$-Boosting. nr:zhang.yu:2005 show the consistency of $L_2$-Boosting and derive the rate of convergence in low-dimensional settings, and buhlmann2006sparse show the consistency of $L_2$-Boosting in high-dimensional settings without stating the rate. SL_2016 derive rates for boosting and variants in high dimensions under a beta-min condition and exact sparsity assumption. An extensive overview of the development of Boosting and its manifold applications is given in nr:buhlmann.hothorn:2007.
The paper is structured as follows: In Section (ref), we discuss the problem of estimating treatment effects in high-dimensional settings. In Section (ref), we introduce $L_2$-Boosting and two variants. In Section (ref), we present the formal results for the predictive performance of Boosting and for valid inference on (low-dimensional) treatment effects in a possibly high-dimensional setting. In Sections (ref) and (ref), we provide a simulation study and an empirical application. Finally, we conclude in Section (ref). Proofs and additional results of the simulation study are available in Appendices (ref) and (ref).
The goal is to estimate the treatment effect $\alpha$ of a treatment variable $D$ on an outcome variable $Y$ in a high-dimensional regression model, namely
where $\gamma$ denotes the intercept and $\varepsilon$ a statistical error term. The coefficient $\alpha$ of treatment variable $D$ is the treatment effect of interest. There are two reasons for including covariates $X=(X_1,\ldots,X_p)$ in model ((ref)) for the estimation of the treatment effect. First, in randomized control trials the treatment variable is assigned randomly, and additional covariates improve the precision of the estimate of the average treatment effect. This argument has already been made in cox1958 and, more recently, lin2013agnostic. Second, in observational studies, covariates might be need to be included to establish unconfoundedness, in order to ensure that $\mathbb{E}(\varepsilon|D,X)=0$. This means that given the variables $X$, the treatment can be regarded as if it had been randomly assigned and there are no unobserved confounders, as described in ImbensRubin:2015.
A related question is that of which variables to include in model ((ref)) from a set of potential covariates. In high-dimensional settings, when the number of covariates $p$ is larger than the sample size $n$, variable selection is inevitable because the least squares estimate is not well defined. Even when $p$ is smaller than $n$ but the ratio $p/n\rightarrow c$ for some positive constant $c>0$, ordinary least squares estimates are unreliable and, again, variable selection is needed because including too many (noisy) covariates might overfit and disguise the true treatment effect. In the empirical study in Section (ref), we analyze how the pre-merger overlap of bank branch networks in the US affects the post-merger stock returns of acquirer banks. The data set contains 442 observations and 153 potential instrumental variables, comprising a high-dimensional setting in which variable selection is needed in the first-stage regression.
In a naive approach to estimating the treatment effect in model ((ref)), one might first select the relevant covariates by using classical $t$-tests or modern machine learning methods, such as Lasso or $L_2$-Boosting, and then estimate the treatment effect by including only the selected variables and continue with standard inference methods. However, this procedure, while often used in applied work, might fail to provide valid post-selection inference. This is because many modern methods like Lasso or Boosting yield consistent model selection only under very strong assumptions that are often unrealistic, particularly in applications in economics.
In general, machine learning methods may miss key variables that are moderately correlated with the outcome but highly correlated with the treatment variable. Although this does not hurt predictive performance, it introduces an omitted variable bias that leads to invalid post-selection inference.
We demonstrate this by a simple simulation study with a binary treatment variable and $100$ covariates. The data generating process is given by $$y_i=d_i\alpha_0+ x_i^\intercal \beta+\varepsilon_i\quad\text{and}\quad d_i=1_{\{0.2\cdot x_i^\intercal \gamma+\nu_i>0\}}$$ with $\alpha_0=0$ for $i=1,\dots,100$. The high-dimensional coefficient vectors $\beta\in\mathbb{R}^{100}$ and $\gamma\in\mathbb{R}^{100}$ are sparse with $\|\beta\|_0=3$ and $\|\gamma\|_0=3$, respectively. The noise $\varepsilon_i$ is normally distributed with $\varepsilon_i \sim N(0,3)$ and $\nu_i$ following a standard logistic distribution. For the high-dimensional vector $x_i=(x_{i,1},\ldots,x_{i,100})^\intercal$, we set \[x_i\sim \tiny{N\left (0, \left[
\right] \right)}.\] The results for $10,000$ repetitions of the estimate $\hat \alpha_0$ are displayed in Figure (ref). The resulting distribution using the naive approach is highly biased, shows heavy tails and is not in line with a standard normal distribution (see Figure (ref)).
To deal with such a problem and provide valid post-selection inference, we apply the double selection approach, which is described in detail in Section (ref) and implicitly creates an orthogonal moment condition. Figure (ref) shows the empirical distribution of the estimates when employing the double selection methods introduced in BelloniChernozhukovHansen2011 and reviewed and extended in CCDDHN:2016. The estimates are nearly unbiased and can be approximated by a normal distribution. The intuition of the double selection method is that it cures the omitted variable bias seen with machine learning methods by running an auxiliary regression/step. This is equivalent to using an orthogonal moment function. This property means that estimating the nuisance parameters with machine learning methods and plugging it into the moment function has no first order effect on the estimation of the target parameter, as long as the machine learning estimator does not converge too slowly.
In this section, we describe the $L_2$-Boosting algorithm, namely the original Boosting algorithm for regression defined in nr:buhlmann.yu:2003 and two variants thereof (iterated post-$L_2$-Boosting and orthogonal $L_2$-Boosting).
To define the Boosting algorithm for linear models, we consider the following regression setting:
where $x_i=(x_{i,1},\ldots,x_{i,p_n})^\intercal$ is a vector that consists of $p_n$ predictor variables. $\beta$ is a $p_n$-dimensional coefficient vector and $\varepsilon_i$ is a random, zero-mean error term with $\mathbb{E}[\varepsilon_i|x_i]=0$. We allow the dimension of the predictors $p_n$ to grow with the sample size $n$. Also, the case $dim(\beta)=p_n\gg n$ is allowed. In this setting, a so-called sparsity condition is unavoidable. This means that there is a large set of potential variables, but the number of variables that have non-zero coefficients, denoted by $s$, is small compared to the sample size, i.e., $s<n$. In our theoretical analysis, we weaken this assumption to approximate sparsity. In the rest of this paper, we drop the dependence of $p_n$ on the sample size and denote it by $p$ so no confusion will arise. Let $X$ denote the $n \times p$ design matrix where the single observations $x_i$ form the rows of $X$. Let $X_j$ denote the $j$th column of the design matrix, and $x_{i,j}$ is the $j$th component of the vector $x_i$. For a set $\widetilde{T}\subset \{1,2,...,p\}$, we define $X_{\widetilde{T}}$ as the submatrix of $X$ with respect to column indices in $\widetilde{T}$. We assume a fixed design with $\max_{1\le j\le p}x_{i,j}\le C$ for all $i=1,\dots,n$ and $c\le\min_{1\le j\le p}\mathbb{E}_n[x_{i,j}^2]$ for absolute constants $0<c<C<\infty$. Without loss of generality, we consider standardized regressors, i.e., $\mathbb{E}_n[x_{i,j}]=0$ and $\mathbb{E}_n[x_{i,j}^2]=1$ for $j=1,\ldots,p$. Further assumptions will be imposed in the next sections.
The basic principle of $L_2$-Boosting is as follows: The criterion function that we want to minimize is the sum of squared residuals, as is the case with ordinary least squares (OLS). We initialize the estimator $\hat \beta$ to zero (strictly speaking, a $p$-dimensional vector consisting of zeros) and subsequently calculate the residuals, which in this case are equivalent to the observations in this first step. Then, we conduct $p$ univariate regressions by regressing the residuals (in the first round, the observations) on each of the $p$ regressors separately, resulting in $p$ univariate regressions. After this, we select the variable that explains most of the residuals and update the coordinate of our estimated coefficient vector in this direction. The above procedure is repeated until some stopping criterion is reached. The algorithm for $L_2$-Boosting with componentwise least squares is given below.
The version above and the orthogonal version, introduced later, are also known in deterministic settings as the pure greedy algorithm (PGA) and the orthogonal greedy algorithm (OGA), respectively. Boosting is a gradient descent method. In the $L_2$-case, the gradient of squared loss equals the residuals, and the residuals are iteratively fitted by a so-called base learner -- here, componentwise univariate regressions. Hence the gradient is approximated in each step by a simple parametric model, namely a univariate linear model. In the low-dimensional case, the estimator converges to the OLS solution. In the high-dimensional case, overfitting can occur in the absence of the stopping criteria because the algorithms will ultimately build a model that explains all the variances in $y$.
The act of stopping is crucial for Boosting algorithms because stopping too late or never stopping leads to overfitting, and therefore some kind of penalization is required. A suitable solution is to stop early, i.e., before overfitting takes place. Early stopping can be considered an unusual penalization/regularization scheme. Similar to Lasso, early stopping might induce a bias through shrinkage. A potential way to decrease the bias is to perform the post-$L_2$-Boosting approach defined in the next section.
Post-$L_2$-Boosting ($\mathtt{pBA}$) is a post-model selection estimator that applies ordinary least squares (OLS) to the model selected by $L_2$-Boosting in the first step. To formally define this estimator, we make the following definitions: $T_0:=supp(\beta)$ and $\hat{T}:=supp(\beta^{m})$, which are the support of the true model and the support of the model estimated by $L_2$-Boosting as described in Algorithm (ref) stopping at $m$. Further, $T^c$ denotes the complement of a set $T$ with regard to $\{1,\ldots,p\}$.
In the context of Lasso, OLS after model selection was analyzed in belloni:2013. Given the above definitions, the post-model selection estimator or post-$L_2$-Boosting estimator takes the form
where $Q_n(\beta)$ denotes the squared sum of residuals defined as $\sum_{i=1}^n (y_i - x_i^\intercal \beta)^2$.
In our theoretical analysis, we consider a modified version of post-$L_2$-Boosting called iterated post-$L_2$-Boosting ($\mathtt{I}$-$\mathtt{pBA}$), which is given in Algorithm (ref) below. Instead of one final projection, a projection is conducted regularly, described by a sequence $l_m$, $m=0,1,...$
Another variant of the Boosting algorithm is orthogonal Boosting ($\mathtt{oBA}$), described in Algorithm (ref), or the OGA in its deterministic version. Only the updating step is changed in comparison to Algorithm (ref). In every step, an orthogonal projection of the response variable is carried out on all the variables that have been selected up to that point. The advantage of this method is that any variable is selected at the most once in this procedure, whereas in the previous version the same variable might be selected at different steps making the analysis much more complicated. More formally, the method is described in Algorithm (ref) by modifying step (4) in Algorithm (ref).
As mentioned, early stopping is crucial for Boosting to avoid overfitting. Therefore, in the last step of Algorithms (ref), (ref) or (ref), an early stopping criteria is needed. The standard approaches for determining the “optimal” stopping time are cross-validation and a corrected Akaike information criterion (nr:buhlmann:2006). Although both lack a theoretical foundation in a high-dimensional setting, they are applied by practitioners and often give competitive results. We provide a data-driven early stopping criterion later in Section (ref), first developed in SL_2016, with beta-min condition and exact sparsity needed for guaranteeing theoretical performance. In this paper, we prove the nearly-Lasso convergence rate of such early stopping criterion without the strong beta-min and exact sparsity conditions. We demonstrate the good performance of such early stopping criterion in our simulation study.
friedman2001elements were the first to point out a strong relationship between $L_2$-Boosting with componentwise linear least squares and Lasso. Although these methods are generally not equivalent, efron2004least proved an approximate equivalence between $L_2$-Boosting and Lasso and confirmed that $L_2$-Boosting and Lasso are closely related. Compared to Lasso, Boosting uses an unusual “penalization scheme" by early stopping. $L_2$-Boosting can therefore be interpreted as an approximate and implicit regularized optimization, whereas Lasso directly solves a complex penalized optimization problem.
In this paper, we show that iterated post-$L_2$-Boosting and orthogonal $L_2$-Boosting achieve nearly the same rate of convergence as Lasso in a high-dimensional linear regression model. This aligns with the work of nr:buhlmann.hothorn:2007 who did not find Lasso to be superior overall to $L_2$-Boosting (or vice versa) in terms of empirical performance for prediction. More recently, hepp2016approaches also performed an extensive simulation study comparing $L_2$-Boosting and Lasso. Their results suggest that in high-dimensional settings when $p>n$, Boosting has a slight advantage in terms of prediction accuracy, whereas Lasso leads, on average, to sparser models. They also point out that the amount of regularization imposed on the models clearly determines the quality of the estimation methods.
Next, we extend these results and perform simulation studies that compare $L_2$-Boosting and Lasso within advanced inferential procedures in high-dimensional linear regression models. A simulation of this nature is lacking in the literature and in general little is known about the empirical performance and computation time of machine learning methods in causal/inferential settings (see, for example, knaus2020double). As discussed, $L_2$-Boosting solves univariate regressions and is therefore computationally efficient and feasible in high dimensions, having a linear complexity in the number of predictor variables $p$ (cf. buhlmann2006sparse, mayr2014evolution). The computational complexity of $L_2$-Boosting, including $\mathtt{I}$-$\mathtt{pBA}$ and $\mathtt{oBA}$, is $O(npm_1)$, where $m_1=O(s\log(n))$ denotes the number of steps. Lasso has a computational complexity of $O(npm_2)$, where $m_2=O(\min\{n,p\})$ (cf. efron2004least). This leads to a computational superiority of $L_2$-Boosting over Lasso in high-dimensional settings if $s\log(n)<< n$. This phenomenon has been observed in nr:buhlmann.hothorn:2007 who compare the computing time of $L_2$-Boosting to that of Lasso in high-dimensional regressions. Boosting is employed in practice in cases where explicitly solving regularized optimization problems is not practical. This usually happens in very high-dimensional settings when $p>>n$, see, e.g., efron2004least. In our simulation studies in Section (ref), we confirm the computational advantage of Boosting over Lasso by comparing the CPU time in high-dimensional (causal) inference settings. Such advantage can be substantial when $n$ and $p$ become large.
Moreover, in settings where the number of observations $n$ is very large (\textquotedblleft big data\textquotedblright), an advantage of $L_2$-Boosting is that the estimation can be easily parallelized and distributed to different cores (e.g. CPUs) or machines. It is a gradient based method. In each step, $p$ univariate regressions are calculated and are independent of each other and hence can be computed in a distributed fashion. This allows for efficient implementation and the application to big data.
In this section, we provide the theoretical results of our paper, including novel results on the convergence rate of iterated post-$L_2$-Boosting ($\mathtt{I}$-$\mathtt{pBA}$) and orthogonal $L_2$-Boosting ($\mathtt{oBA}$) in subsection (ref). Note that, compared to existing literature, the results rely only on approximate sparsity, and no beta-min condition is required. Based on these results, we present, in subsection (ref), inference results for treatment effects in a setting with high-dimensional controls and in an instrumental variable model.
Consider the following high-dimensional approximate sparse linear regression model $$y_i=x_i^\intercal \beta+r_{i}+\varepsilon_i, \quad i=1,\ldots,n,$$ where $r_i$ denotes the approximation error of the sparse model and $\varepsilon_i$ is i.i.d. across $i$ with variance $\sigma^2$.\footnote{In this subsection we consider homoscedastic error terms. Our proofs can be easily generalized to heteroscedastic errors.} We assume the following conditions to establish new rates of convergence for $L_2$-Boosting:
For the next assumption, we first define the smallest and largest restricted eigenvalues of a matrix:
Next, we require the following assumption on the tails of $X$ and $\varepsilon$:
For the theoretical analysis, first, we consider the (iterated) Post-$L_2$-Boosting algorithm that is defined in Algorithm (ref).
The next theorem provides the same rates of convergence for the orthogonal $L_2$-Boosting ($\mathtt{oBA}$) algorithm that has also been defined in Section (ref).
The proofs of Theorem (ref) and Theorem (ref) and the corresponding corollaries are given in Appendix (ref).
Similar to Corollary (ref), the following corollary is a natural extension of Theorem (ref).
In this section, we present inference results for treatment effects using $L_2$-Boosting. We first consider the case in which a researcher is interested in estimating the treatment effect $\alpha_0$ of a treatment variable $d$ after selecting among many control variables as discussed in Section (ref). Then, we provide new results for inference in an instrumental variable model with potentially many instruments when post- or orthogonal $L_2$-Boosting is used for the variable selection. Essential ingredients for valid inference in these settings are the rates of convergence of $L_2$-Boosting that have been provided in the previous subsection.
We consider the model
where $r_{1,i}$ and $r_{2,i}$ are approximation errors, $i=1,\dots,n$. The estimation method consists of the following three steps, the first two of which involve model selection with $L_2$-Boosting:
The estimated regression coefficient of the treatment variable in step (3) above is the double selection estimator $\check{\alpha}$. To analyze this estimator based on $L_2$-Boosting, we impose the following assumptions:
Assumptions A.(ref)--A.(ref) allow us to apply Theorem (ref) and Theorem (ref) to ensure sufficiently fast estimation rates for $L_2$-Boosting in models ((ref)) and ((ref)). Assumption A.(ref) (i) imposes weak technical conditions on the moments of the error terms to allow for valid inference. Assumption A.(ref) (ii) restricts the growth of the number of parameters. Moreover, we assume that, in the $L_2$-Boosting algorithms, early stopping takes place. This ensures that the conditions for the stopping time $m^*$ in Theorem (ref) and Theorem (ref) are satisfied. With these assumptions, we can now formulate our first main theorem regarding inference.
The proof of Theorem (ref) is given in Appendix (ref). This result can be used to conduct valid inference on the regression coefficient $\alpha_0$. The construction of uniformly valid confidence intervals is given in the following corollary.
In this section, we consider the following instrumental variable model with potentially very many instruments
with instrument function $D_i=D(z_i)=\mathbb{E}[d_i|z_i]= z_i^\intercal \gamma+r_i$. Again, we allow for approximate sparsity in the second equation. For simplicity, in our technical analysis, we consider the model above without controls $x_i$ in the first equation and a regular fix design $Z$ with observations $z_i$:
To estimate the coefficient $\alpha_0$ of the endogenous treatment variable, we employ the following two-stage least squares (2SLS) procedure: In the first step, we estimate and predict the instrument $\hat{D}_i=z_i^\intercal \hat{\gamma}$ by iterated post- or orthogonal $L_2$-Boosting. Finally, we estimate $\hat{\alpha}_0$ by regressing the outcome variable $y$ on the predicted instrument $\hat{D}_i$. To analyze this estimator based on $L_2$-Boosting, we impose the following assumptions:
Assumptions B.(ref)-B.(ref) are essentially the same as Assumptions A.(ref)-A.(ref) except for some small modifications in B.(ref) related to the different underlying setting in subsection (ref). It is worth noting that the growth condition B.(ref)(ii) is slightly weaker than the growth condition A.(ref)(ii) because Assumption A.(ref)(ii) implies Assumption B.(ref)(ii). Assumption A.(ref)(ii)(b) is needed to ensure consistency of the respective variance estimators. With these assumptions, we can show that the instrumental variable estimator $\hat{\alpha}_0$ following the two-stage least squares (2SLS) procedure is asymptotically normally distributed. This result is provided by the following theorem.
This also enables us to construct uniformly valid confidence intervals for the treatment effect as in Corollary (ref). The proof of Theorem (ref) is given in Appendix (ref).
In this section, we present simulation results for both settings.
First, we consider the following i.i.d. data generating process:
where $x_i \sim N(0, \Sigma)$. The second auxiliary equation captures the confounders. We consider two different covariance structures $\Sigma$: First, an i.i.d. structure with $\Sigma=I_p$ being the $p \times p$ identity matrix and secondly a structure with correlated covariates, for which we generate a Toeplitz design $\Sigma_{kj}=0.5^{|j-k|}$ but then perform a random covariate rotation. The parameter of interest, $\alpha_0$, is equal to $0.5$. We consider both a sparse setting (Control-1) and an approximately sparse setting (Control-2). In the sparse setting, the first $s=20$ coefficients are set to one and all other parameters $p-s$ are equal to zero. In the approximately sparse setting, the first $s=10$ coefficients are set to one and the remaining coefficients are declining:
The residuals $\xi_i\sim N(0,\sigma^2)$ and $\nu_i\sim N(0,\sigma^2)$ are generated independently with $$\sigma = \sqrt{\theta_g^\intercal\Sigma\theta_g/\mathrm{SNR}}$$ to control the empirical signal-to-noise ratio ($\mathrm{SNR}$), $$\mathrm{SNR}=\mathbb{E}_n[(x_i^\intercal\theta_g)^2]/\mathbb{E}_n[\xi_i^2]=\mathbb{E}_n[(x_i^\intercal\theta_m)^2]/\mathbb{E}_n[\nu_i^2].$$ We vary the sample size $n$, the number of covariates $p$ and the $\mathrm{SNR}$. The number of repetitions is $R=500$. For estimation of the treatment effect $\alpha_0$ we use the double selection procedure presented in Section (ref). Because we use different approaches for variable selection, we are able to compare $L_2$-Boosting and Lasso in terms of how capable they are in treatment effect estimation. The results are provided in subsection (ref).
In the setting with many instrumental variables, we consider the following i.i.d. data generating process, which is similar to that used in the simulation experiment in BCCH:2012:
with
The parameter of interest is set to $\alpha_0=1$. The regressors $z_i=(z_{i1}, \ldots, z_{ip})^\intercal$ are drawn from a normal distribution $N(0, \Sigma_Z)$ for the same covariance structures as in subsection (ref). Further, we set $corr(\varepsilon,\nu)=0.1$ and $\sigma^2_\varepsilon=1$. Let $\sigma^2_{\nu}=1-\gamma^\intercal\Sigma_z \gamma$ such that the unconditional variance of the endogenous variable equals one. The first-stage coefficients are set according to $\gamma= C \widetilde{\gamma}$. For the high-dimensional regression coefficient $\widetilde{\gamma}$, we use a sparse design, i.e., $\widetilde{\gamma}=(1, \ldots,1,0,\ldots,0)$ with $s$ coordinates equal to one and all other $p-s$ equal to zero. The constant $C$ is set in such a way that we generate data with an SNR of $1$ and $3$. This leads to the concentration parameter $\mu^2 = \frac{n \gamma^\intercal\Sigma_z \gamma}{\sigma_{\nu}^2}$ of size $n$ or $3n$, which determines the behavior of the instrumental variable estimators as described in hansen2008estimation. As in subsection (ref), we also vary the sample size $n$ and the number of covariates $p$. The number of repetitions in the simulations study is again $R=500$. We subsequently employ two-stage least squares with $L_2$-Boosting and Lasso to compare the performance in estimating the treatment effect $\alpha_0$ in a high-dimensional setting with many instrumental variables.
The results of both simulation settings described in subsection (ref) and (ref) are summarized in Tables (ref) -- (ref) in Appendix (ref). We present the mean absolute error (MAE) of $\hat{\alpha}$, $$\frac{1}{R}\sum\limits_{r=1}^R|\hat{\alpha}_r-\alpha_0|,$$ the standard deviation of the estimators and the empirical rejection rates at a significance level $0.05$. In our analysis, pBA refers to post-$L_2$-Boosting, I-pBA to the iterated version, and oBA to the orthogonal variant as in Section (ref). Each of these uses the theoretically justified stopping criterion from Corollary (ref) and Corollary (ref). The Lasso method refers to the Post-Lasso estimator with theoretically justified selection of the shrinkage parameter $\lambda$. For comparison, we also use an implementation of Post-$L_2$-Boosting with cross-validated stopping (CV-pBA) as well as a Lasso implementation with cross-validated $\lambda$. The estimators based on $L_2$-Boosting yield a low MAE throughout all these settings. On average, the methods based on $L_2$-Boosting also provide empirical rejection rates that are close to the nominal level of $0.05$. Compared to Lasso, the MAE provided by $L_2$-Boosting is slightly lower and, on average, the empirical rejection rates are slightly closer to the nominal level of $0.05$. Nevertheless, the differences in MAE between $L_2$-Boosting and Lasso are all within one standard error of each other. In summary, both $L_2$-Boosting and Lasso provide empirical rejection rates that are close to the nominal level of $0.05$. This is in line with the theory that both $L_2$-Boosting and Lasso can be used for valid estimation of treatment effects in high-dimensional settings. Further, the cross-validated methods yield a higher MAE and higher standard deviations than the methods with a theoretically justified tuning parameter. Table (ref) below summarizes the results for the setting with $n=600$ and $SNR=1$.
In addition, we provide the average CPU time of the proposed methods for a single iteration in Tables (ref) -- (ref) in the Appendix (ref). These results refer to an implementation of all methods in Python to make the different computation times of the applied methods comparable. \footnote{We are using the Python version 3.6.5. The operating system is Linux/Ubuntu. The hardware is Intel(R) Xeon (R) CPU E5-2650 v3 @2.30GHz.} We provide the results for the setting with $n=600$ and $SNR=1$ again in Figure (ref) below. It is clearly visible that $L_2$-Boosting has a computational advantage over Lasso. For example, in the sparse setting (Control-1) for i.i.d. covariates with $p=1800$, post-Lasso requires a CPU time of $0.213$ seconds per iteration, whereas post-$L_2$-Boosting (pBA) takes only $0.096$ seconds. On average, the CPU time for Lasso is more than twice that for post-L2-Boosting. For an overview of how the computation time increases with sample size $n$, we refer to Figure (ref) in the appendix. Obviously, the methods with cross-validated penalty are significantly slower than those with a theoretically justified tuning parameter, cf. Appendix (ref). In summary, we conclude that $L_2$-Boosting with a theoretically justified stopping criterion is the most efficient method with respect to computation time, and it also provides the best estimation accuracy in our simulation study.
In this section, we apply $L_2$-Boosting for instrumental variable estimation in a setting with many instruments. We build on the empirical study in levine2020bank and show how our methodology can improve IV estimation in high-dimensional settings.
We use a data set from levine2020bank, who study how the pre-merger overlap of bank branch networks in the US can affect the post-merger stock returns of acquirer banks over the period 1984 to 1995. The target variable of interest is the cumulative abnormal returns (CAR) of the acquirer bank within a window of five days around the announcement of the acquisition, denoted CAR(-2,2).\footnote{Acquirer CAR(-2, 2) is the 5-day CAR (cumulative abnormal returns) during the window (-2, +2), where day 0 is the announcement date of the acquisition. Abnormal returns are defined by using the difference between actual and projected returns. For a definition of the projected returns we refer to levine2020bank.}
We are interested in estimating the causal effect of the overlap of bank branch networks on the outcome of the acquisition, i.e., CAR(-2,2). This overlap is measured by three variables, namely the Overlap, Correlation, and Cosine Distance. The Overlap variable is measured as the quotient of the number of states in which both banks are present and the total number of states in which at least one bank is present. Correlation and Cosine Distance are derived from the vector for each bank, which denotes if they are present in each state. For details on the construction of the three measurements of bank branch network overlap, we refer to levine2020bank. Additionally, 13 variables are available as controls.\footnote{The covariates are: Acquirer run-up (-200, -11), Acquirer Net Income, Acquirer Tobin’s Q, Deal size adjusted by Acquirer Assets, Cash deal dummy, Stock Deal dummy, Attitude dummy, Target public dummy, Percentage of shares acquired, Acquirer Total Assets, Acquirer/Target Assets Ratio, a dummy for whether the acquiring banks acquired another target during the last three years, and number of offers that the target received. The regressions also contain Year fixed effects and Acquirer Bank State*Target Bank State fixed effects.}
The endogeneity problem arises because measures of the overlap of bank branch networks and bank merger decisions and outcomes may be affected by unobservable hidden factors. In many states in the US, banks were not allowed to set up branches in other states. In the 1980s and 1990s, states gradually relaxed this restriction, which was largely exogenous to the individual banks (Jayaratne). Therefore, a connection forms when a state allows another state’s banks to open branches, and these connections generate a directed spatial state network between the 50 states in the United States and Washington D.C.. As a result, interstate deregulation events generate exogenous changes of the state network, which allows information from such state network to serve as instruments to the endogenous bank network, e.g., the three overlap measures mentioned above.
It is common to utilize neighborhood structure of nodes in a network to serve as instruments to deal with endogeneity in the literature, e.g., Chandrasekhar. Specifically, for any pairs of nodes $S_1,S_2$, we can define $d(S_1,S_2)$ as the minimum number of edges that link $S_1$ to $S_2$. For any $k\in \mathbb{Z}^+$, we can define the $k^{th}$ order neighbor of a node $S_1$, denoted as a set $N_k(S_1)$, as the collection of those nodes $S_2$ such that $d(S_1,S_2) = k$.
Note that the state network has a spatial structure. Consider the case that a bank (the acquirer) in $S_1$ acquires a bank in $S_2$. Consider measuring “overlap" in the state network for two states $S_1,S_0$ subjective to the target bank's state, $S_2$. We can construct $k^{th}$ order (geographical) distance-weighted overlap between two states $S_1$ and $S_0$, subject to $S_2$, denoted as $k\textrm{-}NN(S_0,S_1,S_2)$, as follows:
where $GDR(S_0,S_1,S_2)$ is a function that measures “geographic distance ratio", and $\mathcal{S}$ is the collection of all states and Washington D.C., which has a cardinality of $51$. In levine2020bank, the authors define three different ways to construct geographic distance ratio (GDR) and $k$-$NN$. For details, see Appendix (ref).
Specifically, in (ref), we call $1\textrm{-}NN$ as FNN, and $2\textrm{-}NN$ as SNN, referring to “first" and “second order neighbor's distance weighted network overlap", respectively. Note that for each acquisition, given $S_1,S_2$, there exists $51$ possible $S_0\in \mathcal{S}$ and three ways to measure FNN, leading to $51\times 3=153$ FNN instruments.
Using the state network, levine2020bank construct three IVs based on SNNs, namely weighted Overlap, Correlation Coefficient, and Cosine Distance of states that allowed market entry.\footnote{Details on the construction can be found in the online appendix of levine2020bank.} We instead use the $153$ FNN variables as potential IVs. Because we have $442$ observations, this leads to a high $p/n$ ratio, only a small set of which, however, might be relevant.\footnote{Note that the number of observations differ slightly to the original study because we corrected a minor data mistake in the sample. This mistake is confirmed after communication with one of the authors.} In fact, our later analysis shows that there are only four to six FNNs with non-zero coefficients after applying iterated post- or orthogonal $L_2$-Boosting, meaning that most FNNs are filtered out during the selection step.\footnote{Note that in our estimation with $L_2$-Boosting, we do not include the original, constructed IVs that levine2020bank use. However, if we include them in the selection process, the results do not change – indeed, these are still better than those obtained using the combination of simple instrumental variables.}
The FNNs appear to be much weaker than SNNs because none of the FNNs are found to be significantly correlated to the explanatory variable in the first stage. For this reason, levine2020bank use SNNs instead of FNNs. However, discarding the FNNs leads to a loss of efficiency. The advantage of $L_2$-Boosting algorithm is that it utilizes the useful information of FNNs and selects a combination of FNNs to form IVs that may be even better than SNNs.
We apply iterated post-$L_2$-Boosting and orthogonal $L_2$-Boosting on the bank merger data, conduct inference as described in Section (ref), and present the estimates in Table (ref). We control for all firm-year characteristics, year fixed effects, and acquirer-state times target-state fixed effects as was done in the original study. The numbers in parentheses are standard errors.
Columns 1-3 in Table (ref) give the estimated coefficients and standard errors using original IVs (SNNs) of levine2020bank.\footnote{Note that the estimates differ slightly to the original study since we corrected a minor data mistake in the sample.} Columns 4-6 give the estimates of the coefficients and standard errors using iterated post-$L_2$-Boosting (I-pBA). Rows 4-6 in Table (ref) give the FNNs that are ultimately selected for use. For example, as shown in Row 4, six FNNs ($FNN1_{DC}$, $FNN1_{IA}$, $FNN1_{VA}$, $FNN2_{MD}$, $FNN2_{NJ}$, $FNN2_{KS}$) are selected to jointly instrument for Overlap in our estimation procedure. These six instruments provide a smaller standard error (3.53) than the original IVs of levine2020bank (5.27) -- indicating the selection of more efficient IVs. The standard errors using the IVs selected from I-pBA for Correlation Coefficient and Cosine Distance are also smaller than those obtained using the original IVs. This shows that the estimation is more efficient when using our selection procedure. Moreover, all estimated coefficients using I-pBA are also statistically significant.
Orthogonal $L_2$-Boosting also works well on this sample. Columns 7-9 present the estimates of the coefficients and standard errors using orthogonal $L_2$-Boosting. Rows 7-9 in Table (ref) present the FNNs that are selected. For example, as shown in Row 7, six IVs ($FNN1_{AR}$, $FNN1_{DC}$, $FNN1_{IA}$, $FNN1_{VA}$, $FNN2_{MD}$, $FNN2_{NJ}$) are selected to jointly instrument for Overlap and produce a smaller standard error (3.52) than that produced by the SNNs (5.27). The standard errors using the combined FNNs selected from orthogonal $L_2$-Boosting for Correlation Coefficient and Cosine Distance are also smaller than the SNNs in levine2020bank.
In the last three Columns 10-12 in Table (ref), we present the estimates using post-Lasso for a comparison. As shown in the bottom row of the table, the computing time for post-Lasso is higher than that for post-$L_2$-Boosting.
In this paper, we apply $L_2$-Boosting, namely the iterated post- and orthogonal versions, for estimation and inference of treatment effects in the setting of many controls and many instruments. We derive uniformly valid results for the asymptotic distribution of estimated treatment effects. We use the framework of orthogonalized moment conditions introduced by Belloni, Chernozhukov, Hansen, and coauthors in a series of papers to derive these results. We also provide new results on the rate of convergence of iterated post- and orthogonal $L_2$-Boosting, which are needed as an ingredient of our approach, but might also be of independent interest.
To derive these rates, we do not require the beta-min condition, and we rely only on approximate sparsity, which is a substantial improvement on earlier work. In an extensive simulation study, we show that our proposed method works well and has a computational advantage over Lasso. Finally, we analyze how the pre-merger overlap of bank branch networks in the US can affect the post-merger stock returns of acquirer banks. Our results show that IV selection with our procedure gives more efficient estimates than standard approaches and is capable of selecting the relevant instruments from among a large set of potential instruments.