Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
13,936 characters · 6 sections · 13 citation commands
$L_2$Boosting for Economic Applications
\pubMonth{Month} \pubYear{Year} \pubVolume{Vol} \pubIssue{Issue}
In the recent years more and more high-dimensional data sets, where the number of parameters $p$ is high compared to the number of observations $n$ or even larger, are available for applied researchers. Boosting algorithms represent one of the major advances in machine learning and statistics in recent years and are suitable for the analysis of such data sets. While Lasso has been applied very successfully for high-dimensional data sets in Economics, boosting has been underutilized in this field, although it has been proven very powerful in fields like Biostatistics and Pattern Recognition. We attribute this to missing theoretical results for boosting. The goal of this paper is to fill this gap and show that boosting is a competitive method for inference of a treatment effect or instrumental variable (IV) estimation in a high-dimensional setting. First, we present the $L_2$Boosting with componentwise least squares algorithm and variants which are tailored for regression problems which are the workhorse for most Econometric problems. Then we show how $L_2$Boosting can be used for estimation of treatment effects and IV estimation. We highlight the methods and illustrate them with simulations and empirical examples. For further results and technical details we refer to SL_2016 and SL_2017 and to the online supplement of the paper.
To define the boosting algorithm for linear models, we consider the following regression setting:
with vector $x_i=(x_{i,1},\ldots,x_{i,p_n})$ consisting of $p_n$ predictor variables, $\beta$ a $p_n$-dimensional coefficient vector, and a random, mean-zero error term $\varepsilon_i$, $\textrm{E}[\varepsilon_i|x_i]=0$.
We allow the dimension of the predictors $p_n$ to grow with the sample size $n$, and even be larger than the sample size, i.e., $dim(\beta)=p_n\gg n$. But we will impose a sparsity condition. This means that there is a large set of potential variables, but the number of variables which have non-zero coefficients, denoted by $s$, is small compared to the sample size, i.e. $s \ll n$.
$X$ denotes the $n \times p$ design matrix where the single observations $x_i$ form the rows. $X_j$ denotes the $j$th column of the design matrix, and $x_{i,j}$ the $j$th component of the vector $x_i$. We consider a fixed design for the regressors. We assume that the regressors are standardized with mean zero and variance one, i.e., $\textrm{E}_n[x_{i,j}]=0$ and $\textrm{E}_n[x_{i,j}^2]=1$ for $j=1,\ldots,p$,
The basic principle of Boosting can be described as follows. We follow the interpretation of nr:breiman:1998 and nr:friedman:2001 of Boosting as a functional gradient descent optimization (minimization) method. The goal is to minimize a loss function, e.g., an $L_2$-loss or the negative log-likelihood function of a model, by an iterative optimization scheme. In each step the (negative) gradient which is used to update the current solution is modeled and estimated by a parametric or nonparametric statistical model, the so-called base learner. The fitted gradient is used for updating the solution of the optimization problem. A strength of boosting, besides the fact that it can be used for different loss functions, is its flexibility with regard to the base learners. We then repeat this procedure until some stopping criterion is met. The act of stopping is crucial for boosting algorithms, as stopping too late or never stopping leads to overfitting and therefore some kind of penalization is required. A suitable solution is to stop early, i.e., before overfitting takes place. \textquotedblleft Early stopping\textquotedblright\ can be interpreted as a form of penalization. Similar to LASSO, early stopping might induce a bias through shrinkage.
The literature has developed many different forms of boosting algorithms. In this paper we consider $L_2$Boosting with componentwise linear least squares, as well as two variants. All three are designed for regression analysis. \textquotedblleft $L_2$\textquotedblright refers to the loss function, which is the typical sum-of-squares of the residuals $Q_n(\beta)=\sum_{i=1}^n (y_i - x_i' \beta)^2$ typical in regression analysis. In this case, the gradient equals the residuals. \textquotedblleft Componentwise linear least squares\textquotedblright refers to the base learners. We fit the gradient (i.e. residuals) against each regressor ($p$ univariate regressions) and select the predictor/variable which correlates most highly with the gradient/residual, i.e., decreases the loss function most, and then update the estimator in this direction. We next update the residuals and repeat the procedure until some stopping criterion is met. We consider $L_2$Boosting and two modifications: the \textquotedblleft classical\textquotedblright one which was introduced in nr:friedman:2001 and refined in nr:buhlmann.yu:2003 for regression analysis, an orthogonal variant and post-$L_2$Boosting. In signal processing and approximation theory, the first two methods are known as the pure greedy algorithm (PGA) and the orthogonal greedy algorithm (OGA) in the deterministic setting, i.e. in a setting without stochastic error terms.
Moreover, we consider two variants, namely post-$L_2$Boosting (post-BA) and orthogonal $L_2$Boosting (oBA).
Post-$L_2$Boosting is a post-model selection estimator that applies ordinary least squares (OLS) to the model selected by the first-step, namely $L_2$Boosting. To define this estimator formally, we make the following definitions: $T:=supp(\beta)$ and $\hat{T}:=supp(\beta^{m^*})$, the support of the true model and the support of the model estimated by $L_2$Boosting as described above with stopping at $m^*$. A superscript $C$ denotes the complement of the set with regard to $\{1,\ldots,p\}$. In the context of LASSO, OLS after model selection was analyzed in belloni:2013. Given the above definitions, the post-model selection estimator or OLS post-$L_2$Boosting estimator will take the form
where $Q_n(\beta)=\sum_{i=1}^n (y_i - x_i' \beta)^2$. For oBA, only the updating step is changed: an orthogonal projection of the response variable is conducted on all the variables which have been selected up to this point. The advantage of this method is that any variable is selected at most once in this procedure, while in the previous version the same variable might be selected at different steps which makes the analysis far more complicated. More formally, the method can be described as follows by modifying Step (4):
Define $X_o^m$ as the matrix which consists only of the columns which correspond to the variables selected in the first $m$ steps, i.e. all $X_{j_k}$, $k=0,1,\ldots,m$. Then we have:
In many cases the researcher is interested in valid inference of a low-dimensional parameter $\alpha$ in the presence of a high-dimensional nuisance parameter $\eta$ where modern methods of machine learning are used to estimate the nuisance parameter $\eta$. To achieve this, two conditions are sufficient. First, it is important that estimating equations used to draw inferences about $\alpha$ satisfy a key orthogonality or immunization condition. When estimation and inference for $\alpha$ are based on the system of equations $M(\alpha, \eta)=0$, this condition is fulfilled if $$ \partial_{\eta} M(\alpha, \eta)=0.$$ This is an important element in providing an inferential procedure for $\alpha$ that remains valid when $\eta$ is estimated using regularized machine learning methods, like Lasso or boosting. This orthogonality condition can generally be established. Second, it is important to use high-quality, structured estimators of $\eta$. Additional structure is usually required for $\eta$ and the imposed estimator shall replicate this structure. It has been shown that under a sparsity condition Lasso is such a high-quality estimator. For a detailed description of this approach to valid post selection inference we refer to another article in this session, CCDDHN:2016, and CHS:2016.
SL_2016 show that both post and orthogonal $L_2$Boosting fulfill the conditions for a high quality estimator and have the same rate of convergence as Lasso.\footnote{They also derive an upper bound for the rate of convergence of $L_2$Boosting which is slower than the Lasso rate. Under additional assumptions an improvement might be possible.} Hence, those two variants of boosting can be used in a high-dimensional setting to estimate the nuisance parameter $\eta$ and finally provide valid inference for the target parameter of interest, $\alpha$. Formal statements of those results are provided in SL_2017. In the following sections we show how $L_2$Boosting can be used in practical economic problems, namely IV estimation with many potential IVs, and inference on treatment effects after selection among high-dimensional controls with $L_2$Boosting. BCCH12 and BelloniChernozhukovHansen2011 provide the underlying theory for high-dimension, including the immunized moment conditions.
In this section we report briefly the results from the simulation studies and the applications. A detailed description can be found in the supplement to the paper.
Here we use boosting for estimation of the first stage in an IV setting with very many potential instrumental variables. The simulations show that boosting performs well in common settings, giving a lower bias than post-Lasso and rejection rates close to the nominal $5\%$ level. In the application the influence of property rights protection, measured by federal appellate court decisions regarding in eminent domain, on GDP is analyzed. The boosting estimates replicate the Lasso estimates but with smaller standard errors. The economic conclusions remain unchanged.
For estimation and inference of a treatment effect in a setting with very many control variables we apply the so-called double selection method. It consists of two separate selection steps to determine the final controls for the regression of the outcome variable on the treatment variable and the selected controls. In the simulations Lasso shows a lower bias, but the rejection rates are too small compared to the nominal $5\%$ level. The boosting estimates are close to the nominal level. The application, analyzing the convergence hypothesis in Macroeconomics, replicates the Lasso estimates.
In this paper we define and explain briefly $L_2$Boosting and two variants, namely post-$L_2$Boosting and orthogonal boosting. We show, how these methods can be used within the orthongal moment conditions framework for valid post-selection inference on treatment effects, either in an IV estimation setting with many instruments or in a setting with very many controls. Although only post- and orthogonal boosting have been shown to have rate of convergence in prediction norm, allowing valid post-selection inference, all three versions show very good properties in the simulations. In the applications we present boosting replicates the results from Lasso estimates. In sum, boosting seems to be an useful tool for the (micro-)econometrician's toolbox. The strength of boosting is particular in more complex models and offers many interesting questions for future research.