Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
60,208 characters · 16 sections · 43 citation commands
Lassoed Boosting and Linear Prediction in the Equities Market
\doublespace
JEL Classification: C18, C21
Keywords: Lassoed boosting, linear regression, variable selection, return prediction, parameter attribution
\doublespace
To analyze consumer behavior, predict sales, and track price movement, business and economic researchers must routinely sift through massive amount of data to select relevant variables. In an influential paper, tibshirani1996lasso proposed a shrinkage method called lasso to simplify estimation, and it has become a critical tool in high-dimensional analysis. Extensions include the elastic net in zou2005elasticnet and the group lasso in yuan2006grouplasso. fanandli2001SCAD,zhang2010MCplus,mazumderetal2011 also discuss nonconvex penalty function approaches. buhlmann2011highdimenstats, hastie2015slsparcity provide thorough expositions of the lasso and related methods.
With so many variable-selection methods available, data analysts will profit from some general advice on their use. hastie2017extended recently conducted a large-scale simulation to compare the performance of (a) the lasso; (b) forward stepwise selection, which generates models by sequentially adding the regressor that most improves the fit; (c) best subset selection of regressors for each model size; (d) the relaxed lasso in meinshausen2007relaxed, which emerged as the overall winner with good accuracy and sparsity recovery. The paper also commented on many other competitors of the lasso, such as the Dantzig selector and square-root lasso, and concluded that their performance is either close to the lasso or somewhere between the lasso and the best subset.
This paper investigates whether we can design a new estimator that is as simple as the relaxed lasso but even more effective under certain scenarios. We propose one example: lassoed boosting. The relaxed lasso uses linear interpolation between every lasso solution and the corresponding least-squares (LS) solution to create additional coefficient paths; the linear interpolation forces coefficients to grow proportionally toward an LS solution. In lassoed boosting, we use the lasso in the first stage to screen variables and, for each subset of variables, we use LS-boost (friedman2001gbm) to grow coefficients in the second stage. We hypothesize that, for some data, a good solution may appear outside the grid of (proportional) solutions generated by the relaxed lasso; using boosting to rebuild coefficients allows us to explore possibly better solutions. Our method complements the use of the relaxed lasso in practice.
Both lassoed boosting and the relaxed lasso can be connected to a strand of literature on refitting strategies (see, e.g., chzhenetal2019lassorefit and references therein). The simple idea of combining the lasso with LS-boost comes with some obvious benefits. Using a large iteration number in the second stage, LS-boost should (a) mitigate the overshrinkage problem of the lasso; (b) remove the proportional constraints when spawning solutions so that coefficient paths can grow freely; and (c) give the estimation procedure a second chance, increasing the likelihood of finding a sparser model. Moreover, tuning both the lasso and LS-boost procedures should lead to better solutions.
This paper discusses the method of lassoed boosting and its good performance in the simulation experiment in hastie2017extended and an application. Based on the results in freundetal2017boosting (hereafter FGM), we also discuss the convergence rate of lassoed boosting estimator. We apply lassoed boosting to predict equity returns and compare the results to several other methods. An R package lboost that implements our method can be found at \url{https://github.com/xhuang20/lboost}.
The rest of the paper is organized as follows. Section 2 discusses the convergence property of LS-boost and lassoed boosting. Section 3 introduces several other two-stage methods. Section 4 discusses the simulation experiment, and Section 5, the application to equity-returns prediction. Section 6 concludes. The online supplement contains all proofs, additional discussions, and figures.
We begin by defining the lasso and LS-boost procedures. Consider $n$ observations $\{(x_i,y_i)\}_1^n$, where $x_i = (x_{i1},\ldots, x_{ip})$ is the $1 \times p$ row vector of variables and $y_i$ is the $i$th response variable. In matrix notation, define the $n \times 1$ vector $\boldsymbol{\mathrm{y} }$ and the $n \times p$ matrix $\boldsymbol{\mathrm{X}}$ with the $j$th column $\boldsymbol{\mathrm{x}}_j$ and the $i$th row $x_i$. Let $\beta^*$ be the true coefficient vector and $u_i \sim (0, \sigma^2)$. Let $\lVert \cdot \rVert_1$ and $\lVert \cdot \rVert_2$ be the $\ell_1$ and $\ell_2$ norms, respectively. Consider the linear regression model
The LS solution $\hat{\beta}_{\text{LS}}$ is obtained by minimizing the following loss function:
The lasso estimate, $\hat{\beta}^{\lambda}$, results from minimizing
for some tuning parameter $\lambda > 0$. A sequence of $\lambda$s, $\left\{\lambda_q\right\}_0^Q$, is used to tune the coefficient solutions with $\lambda_0 = \max_j |\frac{1}{n}\langle\boldsymbol{\mathrm{x}}_j,\boldsymbol{\mathrm{y} }\rangle| > \lambda_1 > \cdots > \lambda_Q$. Let $\mathcal{A}_q$ be the active set of selected variables at step $q$ and $\hat{\beta}^{\lambda_q}$ be the coefficient estimate. When $\lambda = \lambda_0$, no variable is selected and $\mathcal{A}_0$ is empty; the size of the active set increases as more variables are included in $\mathcal{A}_q$ when $\lambda$ decreases. When $\lambda=0$, $\mathcal{A}_q$ includes all variables and (ref) reduces to (ref).
The LS-boost algorithm works by choosing the variable $x_{j_k}$ that best fits the current residual $\hat{\boldsymbol{\mathrm{u}} }^{k-1}$ at each iteration step $k$ and then updating the $j_k$th regression coefficient. Choose a learning rate $0<\varepsilon<1$. Initialize $\hat{\beta}^0 = 0$ and $\hat{\boldsymbol{\mathrm{u}}}^0 = \boldsymbol{\mathrm{y}}$. For each iteration $k \geq 1$,
Step 1. Select the variable $\boldsymbol{\mathrm{x}}_{j_k}$ with
Step 2. Update $\hat{\beta}^k$ and $\hat{\boldsymbol{\mathrm{u}}}^k$ by
Iterating between Step 1 and Step 2 until we reach a prespecified stopping criterion gives the solution paths. LS-boost can sometimes generate coefficient paths similar to those of the lasso, but they are two different methods in general.
Lassoed boosting works by rebuilding coefficient paths for variables in each $\mathcal{A}_q$ using LS-boost.
A few remarks are in order.
In this section, we discuss the asymptotic convergence result for lassoed boosting. Let $\hat{\beta}^{\lambda_q, k}$ be the boosting solution at step $k$ for the variables in the active set $\mathcal{A}_q$ associated with $\lambda_q$. Let $K_{\mathcal{A}_q} = |\mathcal{A}_q|$ be the number of elements in $\mathcal{A}_q$. The active set for the true model is $\mathcal{A}=\{1,\ldots,s\}$ and $K_{\mathcal{A}} = s$. We make the following assumptions for (ref).
We also implicitly assume $\log(p)/n \rightarrow 0$ in the proof of (ref) in the supplement (Section S.1), but this assumption is not needed for (ref).
The following proposition shows that lassoed boosting shares the same loss-function convergence rate with the relaxed lasso in Theorem 6 in meinshausen2007relaxed.
The proof is given in Section S.1. The convergence rate for lassoed boosting is faster than the lasso's in Theorem 5 in meinshausen2007relaxed.
The relaxed lasso and lassoed boosting share the same fast convergence rate because both of their solution paths include the LS solution when the lasso correctly identifies the variables. This observation suggests that any lasso-based two-stage method that includes the LS solution in the second step will also enjoy the rate in (ref). (ref) summarizes this result. Let $\hat{\beta}_{\text{two-stage}}^{\lambda_q, \mathcal{K}}$ be a two-stage estimator that uses either the lasso solution or the active set $\mathcal{A}_q$ to generate solution paths that include the full LS solution for each $\mathcal{A}_q$, and $\mathcal{K}$ is the vector of all tuning parameters in the second stage.
See Section S.1 for the proof. (ref) indicates that the relaxed lasso and lassoed boosting are two examples of a large class of two-stage estimators.
We begin our discussion of some linear convergence results for LS-boost and extend it to lassoed boosting on an active set $\mathcal{A}_q$. Let $\hat{\beta}^{k}$ be the LS-boost solution at step $k$; $\hat{\beta}^{k}_{\text{LS}}$ a possibly non-unique least squares solution at step $k$; and $\hat{\beta}_{\text{LS}}$ the full LS solution. Note that $\hat{\beta}_{\text{LS}}$ is non-unique when $p > n$. Theorem 2.1 in FGM gives the linear convergence result for $\lVert \boldsymbol{\mathrm{X}} \hat{\beta}^k - \boldsymbol{\mathrm{X}} \hat{\beta}_{\text{LS}} \rVert_2$. First, we investigate the convergence result for $\lVert \boldsymbol{\mathrm{X}} \hat{\beta}^k - \boldsymbol{\mathrm{X}}\beta^* \rVert_2$.
Without any identification assumption, both $\hat{\beta}_{\text{LS}}$ and $\beta^*$ are underidentified, as shown in (ref). The boosting solution converges to $\hat{\beta}_{\text{LS}}$, which is one of many solutions in the flat sample solution region, or the irregular shape at the center of the contour plot. The linear model in (ref) is also underidentified at the population level, leading to a flat population solution region in (ref). In general, the two regions do not completely overlap. (ref) show where they might intersect, and in (ref), $L_n(\hat{\beta}_{LS}) = L_n(\beta^*)$.
Let $\lambda_{\text{pmin}}(\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{X}})$ be the smallest nonzero eigenvalue of $\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{X}}$ and define
FGM show that $0.75 \leq \gamma < 1$. Let $\nabla L_n(\beta)$ be the gradient vector at $\beta$.
A proof is given in the online supplement. Compared to Theorem 2.1 in FGM, (ref) has an extra term that relates to the gradient vector and the $\ell_2$ error of $\hat{\beta}_\text{LS}$. Without additional assumptions, this extra term will not disappear as $k \rightarrow \infty$.
Clearly, (ref) indicates that, in a finite sample case when $n \not\to \infty$, LS-boost prediction will not recover the true sparse regression function, $\boldsymbol{\mathrm{X}} \beta^*$. Theorem 12.2 in buhlmann2011highdimenstats and Theorem 1 in buhlmann2006boostinghigh show that, as both $n \to \infty$ and $k \to \infty$, $\lVert \boldsymbol{\mathrm{X}} \hat{\beta}^k - \boldsymbol{\mathrm{X}}^T \beta^* \rVert_2^2/n = o_p(1)$. This result does not contradict the nonasymptotic result in (ref). A heuristic argument follows.
Next, we discuss the faster convergence rate of lassoed boosting. Let $|\mathcal{A}_q| = p_q$. Consider two active sets $\mathcal{A}_{q_1}$ and $\mathcal{A}_{q_2}$ that are generated on the lasso solution path. We focus on the specific case when $p_{q_1} < p_{q_2} < n$ with either $n < p$ or $n \ge p$ and discuss why it is worth considering.
The linear convergence rate $\gamma$ plays a critical role in determining the speed of convergence in (ref) and Theorem 2.1 in FGM. Figure 4 in FGM shows a general pattern: $\gamma$ decreases, and $\lambda_{\text{pmin}}(\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{X}})$ increases as $p$ increases. However, the two relationships are not strictly monotonic, suggesting that the convergence will be faster as $p$ increases, in agreement with result (ii) in Theorem 2.1 of FGM
A similar conclusion can be drawn for the convergence result in (ref). We would expect the opposite: the convergence for the estimator and the prediction will slow down when $p$ increases and more variables add “noise" and competition to the variable selection process.
We provide an alternative explanation to complement the results in Figure 4 in FGM. Notice it represents cases when $p > n$ with $n=50$ and $p\geq 73$. The same figures will give a different pattern when $p < n$. In lassoed boosting, we apply LS-boost sequentially to variables in $\left\{\mathcal{A}_q\right\}_{q=1}^{Q}$. If the lasso does a good job, and the model really is sparse, we expect that in the early stage of variable selection, some of the $\mathcal{A}_q$ will include the correct variables and $|\mathcal{A}_q|$ will be much smaller than $n$ ($\ll n$). For a sparse model, those active sets with $|\mathcal{A}_q| \ll n$ are arguably the most interesting since the boosting solutions spawned on these sets will be more likely to mimic the true and sparse elements in $\beta^*$. LS-boost on each $\mathcal{A}_q$ can be viewed as separate exercises. We can add the subscript $\mathcal{A}_q$ to results in (ref) and (ref). Consider (ref) for the active set $\mathcal{A}_q$ with $|\mathcal{A}_q| \ll n $.
where all quantities are restricted to the active set $\mathcal{A}_q$ and
Since $|\mathcal{A}_q| < n$, all parameters can be identified.
Next, we show a simulation result that describes $\gamma_{\mathcal{A}_q}$ and $\lambda_{\text{pmin}}(\boldsymbol{\mathrm{X}}_{\mathcal{A}_q}^T\boldsymbol{\mathrm{X}}_{\mathcal{A}_q})$ as a function of $|\mathcal{A}_q|$. (ref) describes the relationship of the linear convergence rate and minimum eigenvalue with $p_q(=|\mathcal{A}_q|)$ when $p_q < n$. It reverses the patterns in Figure 4 in FGM. We stress that both figures are correct, but (ref) helps to explain the convergence rate when $p_q < n$. As the lasso penalty parameter $\lambda_q$ decreases, and $p_q$ increases, the minimum eigenvalue of $\boldsymbol{\mathrm{X}}_{\mathcal{A}_q}^T\boldsymbol{\mathrm{X}}_{\mathcal{A}_q}$ will decrease monotonically and remain positive. Such decrease in minimum eigenvalue as matrix dimension increases is a standard result in matrix theory (see, e.g., Theorem 4.3.8 in hornjohnson1985matrixanalysis). (ref) and (ref) imply (ref). Hence, for two active sets $\mathcal{A}_{q_1}$ and $\mathcal{A}_{q_2}$ with $p_{q_1} < p_{q_2} < n$, we have $\gamma_{\mathcal{A}_{q_1}} < \gamma_{\mathcal{A}_{q_2}}$ and the convergence rate for $\hat{\beta}^k_{\mathcal{A}_q}$ in (ref) is faster when boosting on $\mathcal{A}_{q_1}$ than on $\mathcal{A}_{q_2}$. This result applies to both (ref) and (ref) after we replace $\boldsymbol{\mathrm{X}}^T\boldsymbol{\mathrm{X}}$ with $\boldsymbol{\mathrm{X}}_{\mathcal{A}_q}^T\boldsymbol{\mathrm{X}}_{\mathcal{A}_q}$ in lassoed boosting. It is beneficial, particularly in the early stages of the lasso; when $\mathcal{A}_q$ includes correct variables and $p_q < n$, the parameters are identified; convergence is faster; and the convergence rate for prediction in (ref) is also faster. When $|\mathcal{A}_q| > n$, parameters are underidentified and $\hat{\beta}_{\text{LS},\mathcal{A}_q}^k $ in (ref) may differ for each $k$ and $\mathcal{A}_q$.
In this section, we give several additional examples of two-stage estimators.
The forward stagewise regression algorithm in hastie2009elements is a popular method for building coefficients and the regression function in small steps. In the $k$th step, it identifies the predictor $\boldsymbol{\mathrm{x}}_{j_k}$ that best correlates with the current residual $\boldsymbol{\mathrm{r}}_k$ and makes the following update:
This algorithm can also be implemented on lasso-generated active sets. We call it lassoed forward stagewise regression.
Like (ref), (ref) in the online supplement gives a convergence result for the prediction function of the estimator in (ref). In the case of $p_q < n$, applying (ref) to the active set $\mathcal{A}_q$ and using the same argument as for lassoed boosting, we conclude that the convergence rate for lassoed forward stagewise regression is faster.
hastie2009elements write, “one can use the lasso to select the set of non-zero predictors, and then apply the lasso again, but using only the selected predictors from the first step. This is known as the relaxed lasso (Meinshausen, 2007).” The idea of using the lasso twice refers to the “Simple Algorithm" in meinshausen2007relaxed, where after obtaining the active set $\mathcal{A}_q$, the lasso is applied again on $\mathcal{A}_q$ with the penalty parameter on $[0,\lambda_q]$. (ref) describes the procedure of lassoing twice, and we call it twiced lasso.
The key difference between twiced lasso and the relaxed lasso is, in step 4 of (ref), the former always starts at $\lambda_1$ while the latter starts at $\lambda_{q+1}$. Obviously, starting from $\lambda_1$ for every $\mathcal{A}_q$ creates computational redundancy. However, the formulation in (ref) allows direct comparison with lassoed boosting in (ref) and helps to explain the change in the convergence rate.
Adapt the restricted eigenvalues condition in equation (11.10) in hastie2015slsparcity to the active set $\mathcal{A}_q$ and we have, as a constant $\tilde{\lambda} > 0$,
The constrained set $\mathcal{C}$ defines directions along which parameters can be identified. The discussion on lassoed boosting also applies here. When the lasso does a good job selecting variables in step 3 of (ref), $\mathcal{A}_q$ contains the correct variables, and $p_q < n$. In this case, the least-squares loss is strictly convex ($\nabla^2 L_n(\beta) = \boldsymbol{\mathrm{X}}_{\mathcal{A}_q}^T\boldsymbol{\mathrm{X}}_{\mathcal{A}_q}/n $ is positive definite) and (ref) holds for all $\nu \in \mathbb{R}^{p_q}$. Hence, $\tilde{\lambda} = \lambda_{\text{pmin}}(\boldsymbol{\mathrm{X}}_{\mathcal{A}_q}^T\boldsymbol{\mathrm{X}}_{\mathcal{A}_q}/n)$, and as (ref) suggests, $\tilde{\lambda}$ decreases as $p_q$ increases. Theorem 11.1 in hastie2015slsparcity implies
where $\lambda_n$ is the tuning parameter such that $\lambda_n \geq 2 \lVert \boldsymbol{\mathrm{X}}_{\mathcal{A}_q}^T \boldsymbol{\mathrm{u}} \rVert_{\infty}/n > 0$. A smaller $p_q$ will tighten the bound in (ref), implying a faster convergence rate.
When $s \leq p_q < n$, and $\mathcal{A}_q$ correctly includes the nonzero variables, the non-zero elements of $\beta^*_{\mathcal{A}_q}$ are equal to those in $\beta^*$.
Another extension uses boosting twice. Step 1 in (ref) uses LS-boost to screen variables and organizes them into sequentially increasing sets. Step 2 uses LS-boost again to grow the coefficient paths on each active set. The iteration stopping criterion in step 1 is when the active set includes all variables, and we can use the standard AIC-type criterion to stop boosting in the second stage.
(ref) differs from the twin boosting algorithm in buhlmann2010twinboosting. Twin boosting uses the estimate in the first stage to guide variable selection in the second stage, similar to the idea of the adaptive lasso in zou2006adaptivelasso.
Our Monte Carlo simulations study the performance of five estimators: forward stepwise, lasso, lassoed boosting, relaxed lasso, and twiced lasso (see online supplement). We do not consider the best subset selection method in bertsimas2016bestsubset due to its computation cost. Our simulation design is the same as that in hastie2017extended except that we use 50, rather than 10, equally spaced values between $0$ and $1$ as weights for the relaxed lasso.
The data-generating process (DGP) is determined by a combination of $4$ beta types, $4$ sample-size and sparsity patterns, $3$ error correlation values, and $10$ signal-to-noise (SNR) levels, for a total of $480(=4\times4\times3\times10)$ DGPs.
Beta types. We consider four coefficient settings for the sparse coefficient $\beta^*$.
The first three beta-types are considered in bertsimas2016bestsubset; beta-type 5 is added in hastie2017extended. We do not consider beta-type 4 in bertsimas2016bestsubset due to its similar result to beta-type 3.
Size and sparsity. We consider the following data size and sparsity combinations: \[
\]
Error correlation. We try $\rho = 0, 0.3$ and $0.7$ in the DGP.
SNR levels. The SNR level takes $10$ values: $v=(0.05,0.09,0.14,0.25,0.42,0.71,1.22,2.07,3.52,6.00)$.
Let $\hat{\beta}$ be the estimated coefficient from one of the five methods.
Evaluation metrics. We apply the following four evaluation metrics on the test data:
The simulation study has the following steps:
Parameter tuning again follows that in hastie2017extended. We use the R package glmnet to generate the lasso solutions and to select variables for lassoed boosting. We use $50$ values of $\lambda$ for the lasso at the low setting and $100$ values of $\lambda$ in the other three settings. We also use $50$ values of equally spaced weights between $0$ and $1$ for the relaxed lasso. The forward stepwise procedure is tuned up to $50$ steps. We use the R package $mboost$ to apply LS-boost to each active set of variables. Early stopping in boosting typically prevents an LS solution, while the relaxed lasso can always reach a full LS solution. To reduce the difference between lassoed-boosting and relaxed-lasso solutions, we use the corrected AIC in hurvich1998smoothing to obtain the iteration number and then double it as the final stopping criterion in LS-boost. In the interval between $1$ and the final stopping number, we use $50$ equally spaced steps and extract the corresponding LS-boost coefficients on each $\mathcal{A}_q$. The learning rate is set to $0.01$.
Like hastie2017extended, for all methods, we use a training set to generate solution paths, a validation set to select the best solution, and a test set to compute the final prediction. Hence, our results require no direct tuning of the lasso penalty parameter $\lambda$.
Lassoed boosting is more computationally expensive than the relaxed lasso. By the time the lasso and the relaxed lasso finishes, lassoed boosting just gets started at generating solutions for each of the active set. We can employ parallelization and warm starts to improve the speed and reduce redundancy in computation.
This discussion focuses on cases for beta-type 2 and $\rho = 0.35$ based on validation tuning. The supplement includes the complete set of results for both validation and oracle tuning, where parameters are chosen to minimize the average risk over $10$ replications. We also skip the results for the twiced lasso because they resemble those for the relaxed lasso, but they are reported in all figures in the supplement.
(ref) plot the average of relative risk (RR), relative test error (RTE), proportion of variance explained (PVE), and number of nonzero (NNZ) coefficients for each of the data size-and-sparsity combinations over $10$ replications. The dotted line in the RTE plots is the RTE of the null model; the dotted line in the PVE plots is the perfect score SNR/(1 + SNR); the horizontal dotted line in the number of nonzeros plots is the value of $s$. On top of each NNZ plot, we print the average number of correctly identified variables over 10 replications for each method at each of the 10 SNR levels, allowing us to infer the true positive rate.
In the RR plots across (ref), the performance of lassoed boosting and the relaxed lasso are much the same, except for a small difference in (ref). In (ref) with small and medium settings, lassoed boosting and the relaxed lasso outperform forward stepwise and the lasso when SNR is low; the performance of all four methods starts to converge when SNR increases. In (ref) with large $p$, forward stepwise and the lasso seem to outperform the other two methods when SNR is small. However, some low SNR results must be interpreted with caution. For example, in the NNZ plot, the lasso, lassoed boosting, and the relaxed lasso can barely identify any correct variable, and the RR score is larger than $1$. Forward stepwise sticks to the null model at very low SNR values and obtains a null score of $RR=1$.
In all RTE plots in (ref), lassoed boosting and the relaxed lasso perform almost identically, and significantly outperform the other two methods when SNR is relatively high. The only exception appears in (ref), where the RTE of forward stepwise is the smallest of the four when SNR is relatively large, though the numerical difference is small compared to that of lassoed boosting and the relaxed lasso.
In the PVE plots in (ref), all four methods give good results that seems to improve as SNR increases. This result is no surprise since all methods better identify the correct variables at higher SNR. In (ref), lassoed boosting and relaxed lasso perform similarly and better than the other options at most of the SNR levels.
In the NNZ plots across all four figures, lassoed boosting and the relaxed lasso show similar capability in sparsity recovery. In (ref), when SNR is relatively high, all methods recover the true variables; the relaxed lasso achieves slightly sparser models than lassoed boosting. However, when SNR decreases, lassoed boosting sometimes achieves sparser models while identifying, on average, the same number of nonzero parameters. The NNZ plot in (ref) seems to suggest that the relaxed lasso performs better when SNR is low, which is not generally the case (see the third row in (ref) for instances when lassoed boosting yields sparser models at low SNR levels). Again, in (ref), almost no nonzero parameter is recovered when SNR is low, making any comparison dubious.
The first three metrics seems to suggest that lassoed boosting and the relaxed lasso share similar coefficient paths, but the NNZ plots in (ref) demonstrate that they differ. Several examples of beta-types 1 and 3 in (ref) from the supplement show that lassoed boosting produces sparser models in quite a few cases, and while the lasso can recover the true parameters, its false inclusion rate is high. Furthermore, from the NNZ plots, we conclude that all methods may fail to recover variables correctly when SNR is very low.
Overall, we can roughly conclude that the performance of lassoed boosting is comparable to that of the relaxed lasso in the first three metrics, and it can produce a sparser model under certain scenarios, which happens more likely when the SNR is moderately large in validation tuning (see (ref) for several examples). We can also easily spot multiple occasions when the relaxed lasso gives sparser models.
In this section, we compare the prediction accuracy of various methods to lassoed boosting's. Sections S.6 and S.7 also discuss the use of path integrated gradient to compare the path differences between the LS-boost and the lasso.
Greenetal2017stockreturns use $94$ variables from the databases CRSP, Compustat and I/B/E/S to study the determinants of average monthly US stock returns in a series of Fama-MacBeth regressions. We update their data to 2018 and use data from 2010 to 2018 for a total of $376544$ firm-month observations in $108$ months. The number of stocks (firms) in each month varies from $3269$ to $3883$. We add $8$ variables that were removed in Greenetal2017stockreturns due to collinearity concerns. The dummy variable ipo is removed since it varies little during certain time periods. In total, we have $101$ ($94+8-1$) variables (see (ref) in (ref)). For example, we have $3883$ observations for January 2010. $\boldsymbol{\mathrm{y} }$ becomes a $3883 \times 1$ vector of one-month-ahead, cross-sectional stock returns, and $\boldsymbol{\mathrm{X} }$ includes all variables plus a column of ones for the intercept. Missing values are replaced by the mean of the variable. We use $50\%$, $25\%$, and $25\%$ of the $3883$ observations for training, validation, and testing, respectively, and record the mean-squared prediction error (MSPE) for the test set of that month. After repeating this exercise for the remaining $107$ months, we report the mean and median of the $108$ MSPEs.
The goal of our exercise is to find a good linear model to predict stock returns, rather than to identify what determines the cross-section of expected stock returns, which is addressed in the large finance literature on anomalies. Active arbitrage in the market also implies that many analysts may have exploited all the variables under consideration, so we do not expect any linear model to exhibit broad predictive power. The data ignore many macro variables and such popular variables as the Fama-French factors and the q-factors.
(ref) reports the prediction results from different methods. The tuning procedure in simulation is also used for the lasso, forward stepwise (FS), twiced lasso (TLasso), relaxed lasso (RLasso), and lassoed bossing ($\text{LB}_1$) in (ref). $\text{LB}_2$ reports lassoed boosting with a learning rate equal to $0.001$ and a stopping criterion equal to the corrected AIC. GHZ represents a linear regression with $12$ variables identified in Greenetal2017stockreturns as significant in explaining the cross-section of stock returns but only for the non-microcap return data before $2003$, and only two are significant for data after $2003$. We use the $12$-variable regression model for simple benchmarking purposes.
The differences in prediction among the methods are small. The average MSPE of $\text{LB}_1$ is smaller than that of the lasso and the relaxed lasso but its median MSPE is larger. The relaxed lasso outperforms the lasso with a smaller mean MSPE and a smaller median MSPE. The results of the twiced lasso closely follow those of the lasso and the relaxed lasso. When tuned with a learning rate of $0.001$, lassoed boosting starts to perform even better. $\text{LB}_2$ has the smallest mean MSPE and median MSPE among all the methods. Note that the lasso and lasso-based methods all outperform the FS method and the $12$-variable GHZ model.
The numerical difference between the relaxed lasso and lassoed boosting ($\text{LB}_2$) in (ref) is also very small. The difference in mean MSPE is $0.000031$. Does such a small difference matter? We note that MSPE is defined as
where $n_T$ is the size of the test set. MSPE measures the average squared distance between prediction and the true value. By taking the square root of the MSPE, we obtain the average absolute distance, which is $0.0056$ ($=0.56\%$), which translates to $56$ cents of difference when trading $100$ dollars. A practitioner can better assess its economic significance. Based on the $108$ MSPEs, a paired one-sided t-test for the null hypothesis $\text{MSPE}_{\text{RLasso}} < \text{MSPE}_{\text{LB}_2} $ gives a p-value of $0.007$. Hence, we reject the null and conclude that the MSPE of $\text{LB}_2$ is smaller than that of the relaxed lasso.
The last row of (ref) reports the average number of selected variables across the $108$ test sets. On average, the number for $\text{LB}_2$ is less than half that for the lasso. Compared to the relaxed lasso, lassoed boosting also offers a sparser solution on average. (ref) plots the number of nonzeros identified in lassoed boosting ($\text{LB}_2$) vs. the lasso (left) and the relaxed lasso (right). The dotted line is $45\text{\textdegree}$ line. Out of $108$ test sets, lassoed boosting yields a sparser model in $91$ cases; the lasso in only $6$. Compared to the relaxed lasso, the number of nonzeros identified by lassoed boosting is smaller in $25$ cases, larger in $14$ cases, and the same in $69$ cases. Overall, our empirical results are consistent with simulation: lassoed boosting can yield sparser model in certain cases. The large variation in model size also indicates model instability, though a stable model like GHZ underperforms the unstable models.
Combining the information in (ref) and (ref), we conclude that, based on this application, lassoed boosting can deliver better prediction with a more compact model.
This paper finds that lassoed boosting, a refitting strategy based on the lasso, has good finite-sample properties in both our simulation experiment and an application. We also introduce the idea of using path integrated gradient to study the difference between the lasso and LS-boost in parameter attribution.
Our application uses one data set and the simulation compares only five variable-selection methods. It takes one line of R code to connect our method to the bestsubset package in hastie2017extended and to reproduce the simulation results (see the github page for instructions.) We invite readers to try lassoed boosting in more simulations and applications to further study its properties.
Our work can be extended in several directions. First, lassoed boosting should be compared to other methods in a classification exercise such as credit rating and default analysis. Second, a valid p-value should be attached to the estimates. Third, (ref) suggests that both the relaxed lasso and lassoed boosting may be examples of many other two-stage approaches. An analyst should explore other possibilities to find a better method.
I am grateful to two referees for their insightful comments and suggestions. The computation and research support from the Office of Research and Coles College of Business at Kennesaw State University are greatly acknowledged.
\spacing{1.45}
\setcounter{page}{1} \spacing{1.42}