Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
64,675 characters · 13 sections · 48 citation commands
Estimation in high-dimensional linear regression: Post-Double-Autometrics as an alternative to Post-Double-Lasso
In high-dimensional linear regression, covariate selection becomes a key issue when the purpose is to obtain a good estimate of a parameter of interest, such as an average treatment effect. It is important not to miss relevant variables correlated to the outcome and the treatment variable (to avoid bias), and not to include irrelevant variables (to avoid loss of accuracy). For this purpose, Post-Double-Lasso (belloni2014) is becoming the most popular method. This method allows the use of many covariates, to avoid omitted variable bias, while controlling overfitting issue and loss of accuracy with a Lasso variable selection method.
However, wuthrich2023 demonstrate that this method can experience significant omitted variable bias in finite samples, even in straightforward settings favorable to Lasso. They also show that the performance of this method is very sensitive to the choice of the regularization parameter of the Lasso, and no specific choice of this parameter systematically outperforms the others.
In this paper, we consider an alternative variable selection method, Autometrics (doornik2009), which is based on statistical inference. This method performs automatic selection of variables based on “Hendry's theory of reduction” (see Chapter 6 of Hendry2014) and an automatic general-to-specific model selection (Hendry2000). Its main hyperparameter, the target size $\alpha$, is the significance level used for inference. Simulation results show that this parameter allows to determine approximatively the frequency of irrelevant explanatory variables included in the terminal model (Hendry2014, epprecht2021, flachaire2024).
Firstly, we propose a new method called Post-Double-Autometrics, that extends Post-Double-Lasso by using Autometrics instead of Lasso to do variable selection. We show through extensive Monte Carlo experiments that the proposed method provides the best variable selection method and outperforms Post-Double-Lasso in many cases. To illustrate, Figure (ref) provides a numerical illustration of this phenomenon in a simple simulation experiment, with $n=400$ observations and $p=210$ covariates of which only $10$ are relevant. The error term is {\em i.i.d.} and the covariates are moderately correlated. The Lasso penalization parameter $\lambda^{min}$ is obtained by cross-validation, and the Autometrics target size is 5%. The left panel shows the behavior of the Post-Double-Lasso estimator of a treatment effect whose coefficient is equal to $0$ while the right panel shows the empirical distribution of our proposed Post-Double-Autometrics estimator. In this setting, the Post-Double-Lasso estimator is upward biased while the Post-Double-Autometrics estimator is approximately unbiased and behaves almost like the oracle estimator. More details on this simulation will be given in Section (ref).
Secondly, we revisit a standard economic growth application, on the convergence hypothesis that the poor catch up the rich countries, with the Barro1994 data consisting of a panel of countries for the period of 1960 to 1985. This hypothesis is often tested from a linear regression of growth rates on the initial level of GDP and additional characteristics of countries. Given that the number of covariates that describe such characteristics can be comparable to the sample size, covariate selection is an issue. Empirical results based on Lasso selection method support the convergence hypothesis (Belloni2011b, Belloni2013c). We show that this conclusion is sensitive to the choice of the penalization term, and that the convergence hypothesis is rejected with Post-Double-Autometrics whatever the target size. Further analysis shows that including or missing one specific covariate is most likely the source of support or lack of support for the convergence hypothesis.
The reminder of the paper is structured as follows. Section 2 presents the pitfalls of Lasso-based methods and introduce our new approach. Section 3 is devoted to Monte Carlo experiments and Section 4 to an empirical application. The paper concludes with Section 5.
In this section, we first present the framework, Post-Double-Lasso and finally Post-Double-Autometrics.
In accordance with the literature, we consider the following standard linear regression model
for $ i=1,\dots,n$, where $y_i$ is the outcome, $d_i$ is the treatment variable, $\boldsymbol{X}_i = \left(x_{1,i}, \dots, x_{p,i}\right) \in \mathbb{R}^p$ is a vector of $p$ covariates, and $\epsilon_i$ and $\eta_i$ are error terms. The parameter of interest ${\delta}$ is the average treatment effect. The covariates might include transformations of raw variables in order to capture non-linear effects (e.g., powers of raw variables, interaction terms).
In observational studies, the unconfoundedness condition is crucial to obtain unbiased estimation of the average treatment effect ${\delta}$. This condition requires $\boldsymbol{X}$ to contain all the confounder variables which explain both the outcome $y$ and the treatment $d$. Otherwise, the estimation may suffer from substantial omitted variable bias (OVB) or counfounding bias. In practice, it is difficult to account for all counfounder variables. Here, we consider the strategy of including many covariates and using a variable selection method to select the relevant covariates (to make the estimate as efficient as possible).
An early approach developed in the literature is Lasso, which performs estimation and variable selection simultaneously (tibshirani1996). Considering the linear regression in ((ref)), Lasso solves the following penalized regression problem
where $\boldsymbol{Z}_i=(\boldsymbol{X}_i,d_i)$, $\boldsymbol{\theta}=(\boldsymbol{\beta}^\top,\delta)^\top$ and $\omega_j = 1 \ \forall j$. Estimation is feasible in a high dimensional setting, that is, when the number of covariates $p$ is large and can exceed the number of observations $n$. The penalization restricts the magnitude of the coefficients, making some of them exactly zero with a sufficiently large $\lambda$. Several approaches have been proposed in the literature to set $\lambda$. For example, bickel2009 proposed to set $\lambda^{bya} = 2\sigma \sqrt{2n^{-1}\left(1+\tau\right)\log p}$, where $\sigma$ is the standard deviation of the error term, and $\tau>0$. belloni2012 proposed a penalization choice, denoted as $\lambda^{bcch}$, to take into account heteroscedastic and clustered errors. Another popular approach is to set $\lambda$ by cross-validation, with the value minimizing the prediction error, ${\lambda}^{min}$, or the value associated to the most parsimonious model within a 1-standard-error interval, ${\lambda}^{1se}$. See Homrighausen2013,Homrighausen2014 and Chetverikov2021 for theoretical results on the choice of $\lambda$ by cross-validation.
The penalization shrinks all coefficients towards zero, introducing some bias into the coefficient estimates associated with relevant covariates. Since the interest is on the estimation of the parameter $\delta$, it is recommended to use the Post-Lasso (belloni2013) which relies on two steps. First, estimate (ref) by Lasso to select a subset of variables, denoted $\boldsymbol{X}^*$. Second, estimate $\delta$ from the OLS regression of $y$ on $\boldsymbol{X}^*$ and $d$. To mitigate the bias introduced in the coefficients of relevant variables, the Adaptive-Lasso can also be used (Zou2006). It solves the penalized regression problem in ((ref)), with weights $\omega_j = |\hat{\mu}_j|^{-\eta}$, where $\eta > 0$ and $\hat{\mu}_j$ is a consistent estimator of $\theta_j$. Here, smaller weights are assigned to larger coefficients in the constraint and, thus, large non-zero coefficients shrink more slowly to zero as $\lambda$ increases.
belloni2014 show that Lasso usually fails to select some relevant variables in the first step, which may lead to a biased estimate of the parameter of interest $\delta$. This typically occurs when some covariates in $\boldsymbol{X}$ are correlated with $d$, but only have a small effect on $y$. To solve this issue, they proposed the Post-Double-Lasso. The standard linear regression model defined in (ref)-(ref) implies the following reduced form model:
where $\boldsymbol{\alpha} = \boldsymbol{\beta}+\boldsymbol{\gamma} \delta$ and $\nu_i = \epsilon_i + \eta_i\delta$. In order to select relevant control variables, Post-Double-Lasso relies on the following two steps of Lasso
where ${w}_{1,j} = {w}_{2,j} = 1 \ \forall j$. Finally, $\delta$ is estimated from the OLS regression of $y$ on $d$ and the union of selected control variables, such as
where $\hat{\boldsymbol{I}}_{\boldsymbol{X}^{*}} = supp\left(\hat{\boldsymbol{\alpha}}\right) = \{j:\hat{\alpha}_j \ne 0\}$ and $\hat{\boldsymbol{I}}_{\boldsymbol{X}^{**}} = supp\left(\hat{\boldsymbol{\gamma}}\right) = \{j:\hat{\gamma}_j \ne 0\}$. This method thus gives covariates omitted in the first step a chance to be selected in the second step, especially those correlated to $d$ and likely to lead to an omitted variable bias. Similarly to the Post-Lasso, several choices can be considered for the penalization parameters $\lambda_1$ and $\lambda_2$, and the Lasso can be replaced by the Adaptive-Lasso.
Although the Post-Double-Lasso is a substantial improvement over the Post-Lasso, wuthrich2023 show that it can still lead to an OVB. Specifically, under certain assumptions, the authors theoretically demonstrate that the probability of not selecting any relevant control variables, given that at least one exists, becomes high when $p$ is relatively large. This occurs when the magnitude of the control variable coefficients is not sufficiently large compared to the penalty parameter. In such cases, the Lasso method may fail to capture relevant variables in both steps of the Post-Double-Lasso procedure and induce an OVB on $\delta$. Additionally, the authors show through simulations that the magnitude of the bias depends on the choice of the regularization parameters $\lambda_1$ and $\lambda_2$, and that no single popular choice consistently leads to better results.
To address this OVB issue, we propose to use another variable selection method: Autometrics (doornik2009). Autometrics is a popular variable selection model among econometricians because it relies on statistical inference to select the relevant variables. There exists at least six different software implementations of automatic “general-to-specific” (GETS) model selection (Matlab, OxMetrics, Scilab, STATA, EViews and R). Autometrics is one of them and is implemented in OxMetrics. See Sucarrat2020 for more details on general-to-specific modelling and its implementations.
More specifically, Autometrics is an algorithm performing automatic variables selection with the GETS model selection approach (hendry1999, Hendry2000). As Lasso, this method starts from a generalized unrestricted model (GUM) which includes all potential relationships between the outcome $y$ and the control variables $\boldsymbol{X}$, such as lag values, trends, step indicators or any other transformations of the raw variables aiming at capturing non-linearities. The algorithm then performs a battery of tests to remove insignificant variables from $\boldsymbol{X}$ and to find a congruent parsimonious model. The final model is then estimated by OLS.\footnote{For more details on Autometrics, see doornik2009 and Castle_etal2023.} Using a multiple block-search algorithm, Autometrics can also be used with more observations than covariates.
Autometrics has three main advantages. First, unlike the Lasso performing variable selection based on the magnitude of coefficients (wuthrich2023), Autometrics relies on statistical inference. Therefore, Autometrics-based methods will select variables whenever their coefficients are significant, independently of the magnitude of their coefficients. Second, the tuning parameter of Autometrics is the targeted nominal size $\alpha$. It allows the user to control the expected proportion of irrelevant variables included in the final model. This approach contrasts sharply with Lasso, where $\lambda$ directly controls the number (or proportion) of variables selected in the final model. Third, Autometrics may invalidate a reduction when the reduced model does not pass some pre-defined misspecification tests.
The target size $\alpha$ refers to the pre-specified significance level or threshold that guides the inclusion or exclusion of variables in the model during the selection process of Autometrics. Specifically, it determines the acceptable probability of incorrectly rejecting a true null hypothesis (i.e., including a variable that is not truly relevant). By setting a target size, Autometrics controls the false-positive rate of variable selection, balancing between overfitting (selecting too many irrelevant variables) and underfitting (failing to select relevant variables). In practical terms:
The choice of target size directly impacts the number of control variables selected, which can influence both the results and the robustness of the model. However, unlike for Lasso, the tuning parameter of Autometrics is more straightforward and intuitive to select.
Lasso methods and Autometrics both start with a general model containing potentially many covariates, from which the goal is to select the most relevant ones. However, unlike Lasso, which estimates parameters through a single optimization process, Autometrics employs an iterative procedure, reducing the model using statistical tests and a tree search algorithm. As a result, deriving theoretical results for Autometrics is more challenging, and its properties are primarily assessed through Monte Carlo simulations, as shown in Hendry2011. Nonetheless, theoretical insights on similar general-to-specific model selection procedures and algorithms can be found in Johansen2009, Hendry2011, Hendry2015, and Johansen2016.
Autometrics allows to force the presence of variable $d$ in Equation (ref) in the terminal model. However, Autometrics is expected to miss relevant control variables with a low degree of non-centrality (i.e., true $t$-statistic) and, if these variables are strongly correlated with $d$, this can lead to an OVB for $\delta$, similarly to Post-Lasso.
Therefore, following the approach of belloni2014, we propose an extension of Autometrics, which we call Post-Double-Autometrics. Our method is similar to the Post-Double-Lasso described above, with Lasso being replaced by Autometrics in the first two steps. By giving covariates omitted in the first step a chance to be recovered in the second step, this method is expected to improve over the usual implementation of Autometrics. We expect Post-Double-Autometrics to address the OVB issue of Post-Double-Lasso, as Autometrics selects variables based on statistical inference rather than coefficient magnitude. This means Autometrics will not overlook relevant variables, even if their coefficient magnitudes are small, as long as they are statistically significant.
Post-Double-Autometrics shares similarities with the indicator saturation method proposed by hendry2006. Their approach consists in splitting the initial sample in two blocks, and applying a general-to-specific variable selection method on each block. A final variable selection is then applied on the union of variables selected in each block. However, the dependent variable considered in both blocks of the indicator saturation method is the same ($y$). Moreover, the main objective of the indicator saturation method is to detect outliers or breaks and not estimating an average treatment effect as in the Post-Double-Autometrics.
In this section, we use Monte Carlo simulations to study the finite sample properties of different methods for estimating a parameter of interest in high-dimensional linear regression. We consider two single-selection methods, the naive Post-Lasso and Autometrics, and three double-selection methods, the benchmark Post-Double-Lasso, the extension Post-Double-Adaptive-Lasso, and our proposed Post-Double-Autometrics.\footnote{For the adaptive-Lasso, we set $\eta = 1$ and $\hat{\mu}_j$ to the estimate of an OLS regression when $p+1 < N/2$ and a ridge regression otherwise.} Monte Carlo simulations are performed on $1{,}000$ replications.
The data generating process (DGP) corresponds to Equations (ref)-(ref), where $\boldsymbol{X}_i \overset{i.i.d}{\sim} \mathcal{N}\left(\mathbf{0_p}, \boldsymbol{\Sigma}_{\boldsymbol{X}}\right)$, $\boldsymbol{\Sigma}_{\boldsymbol{X}}$ is a Toeplitz matrix with element $(k,l)$ set to $\rho^{|k-l|}$ for $k,l=1,\ldots,p$, $\epsilon_i \overset{i.i.d}{\sim} \mathcal{N}\left(0, \sigma^2_\epsilon\right)$ and $\eta_i \overset{i.i.d}{\sim} \mathcal{N}\left(0, \sigma^2_\eta\right)$. The parameter $\rho$ controls the degree of correlation between the covariates. The DGP is general enough to consider cases where variables are positively or negatively correlated (when $\rho$ is respectively positive or negative) but also weakly or strongly correlated (i.e., when $|\rho|$ is respectively close to 0 or to 1).
To assess the variable selection properties of the different methods, we consider a few relevant variables ($\beta_j\neq0,\gamma_j\neq0$, for $j=1,\dots,10$) and many irrelevant variables ($\beta_j=\gamma_j=0$, for $j=11,\dots,p$). Instead of choosing arbitrary values for the non-zero coefficients, we calibrate their individual statistical significance as follows:
where $\psi_j^y$ and $\psi_j^d$ are the non-centrality parameters of the relevant covariates $\boldsymbol{X}_j$, $j=1,\dots,10$, in (ref)-(ref), $\boldsymbol{Z} = \left(\boldsymbol{X}, d\right)$, and
The values of $\psi_j^y$ and $\psi_j^d$ correspond to the expected $t$-statistics for, respectively, the null hypotheses $H_0: \beta_j=0$ in Equation (ref) and $H_0: \gamma_j=0$ in Equation (ref). They allow to control the relevance of covariates, since the probability to reject a null hypothesis is high (low) for large (small) value of the non-centrality parameter. To simplify the notation, we will denote by $\psi^y$ (resp. $\psi^d$) the non-centrality parameters of all non-zero coefficients $\beta_j$ (resp. $\gamma_j$) of the 10 relevant covariates $\boldsymbol{X}_j$, $j=1,\dots,10$.
To study the finite sample performance of the different methods mentioned above, we report the average bias and root mean squared error (RMSE) of $\hat\delta$ as well as the proportion of selected relevant (potency) and irrelevant (gauge) variables included in the terminal model. High potency is important to avoid biased estimation, while for a given level of potency, a smaller gauge is expected to lead to more efficiency (i.e., lower variance of $\hat\delta$).
In the first simulation, we set $n=400$, $p=210$, $\rho = 0$, i.e., $\boldsymbol{\Sigma}_X=I_p$, $\sigma^2_\epsilon = \sigma^2_\eta = 1$, and our parameter of interest $\delta =0$. In this case, the covariates are orthogonal which is the most favorable case for all methods. Moreover, we use $\psi_j^y = 2.5$ and $\psi_j^d = 4$ for $j = 1, \dots, 10$, so that coefficients of the relevant variables have expected $t$-statistics equal to $2.5$ for $H_0: \beta_j=0$ in Equation (ref), and equal to 4 for $H_0: \gamma_j=0$ in Equation (ref). The probability for these variables to be kept in the final model is therefore quite low in the first equation, and higher in the second equation. In this framework, double-selection methods are expected to outperform single-selection methods.
Table (ref) shows bias and RMSE of the treatment effect $\hat\delta$, as well as the average proportion of relevant (potency) and irrelevant (gauge) variables. Note that for the Post-Double-Lasso and Post-Double-Autometrics the gauge and potency are computed on the union of $\boldsymbol{X}^*$ and $\boldsymbol{X}^{**}$, and we consider several choices of tuning parameters $\lambda$ and $\alpha$.
First, double-selection methods provide substantial improvements over single selection methods, regardless of the choice of the variable selection method considered. The potency is always increased in the double-selection approaches, leading to lower bias and RMSE. Overall, the results suggest that single-selection methods are always biased in this setting, unlike double-selection methods. Therefore, we focus on double-selection methods in the following and leave single-selection methods out of the discussion.
Second, the performance of Post-Double-Lasso methods depends on the choice of the $\lambda$ penalty term. Most of the relevant variables are missed (potency close to 0) with $\lambda^{bya}$ and $\lambda^{bcch}$, leading to a treatment effect poorly estimated (large bias and RMSE). The bias is close to 0 for Post-Double-Lasso with $\lambda^{min}$ as almost all relevant variables are selected (potency close to 1). However, $\lambda^{min}$ selects too many (more than 20%, i.e. more than 40) irrelevant variables which translates into a small loss of efficiency. Interestingly, Post-Double-Lasso with $\lambda^{1se}$ selects very few irrelevant variables (gauge close to 2%) but also misses more than 20% of relevant variables which translated into a much larger bias. Results suggest that $\lambda^{min}$ is the best choice for the penalty term of Post-Double-Lasso methods as it leads to an unbiased estimate of $\delta$ and a small RMSE.
Third, both the Post-Double-Adaptive-Lasso with $\lambda^{min}$ and $\lambda^{1se}$ are unbiased and have a RMSE slightly lower than Post-Double-Lasso with $\lambda^{min}$. Post-Double-Adaptive-Lasso with $\lambda^{1se}$ performs particularly well because it has a high potency (close to 95%) and a gauge lower than 10%.
Fourth, Post-Double-Autometrics leads to large potency, and thus to small bias, regardless of the choice of $\alpha$. Moreover, as Autometrics controls the proportion of irrelevant selected variables, while selecting most relevant variables, the gauge of Post-Double-Autometrics remains relatively small unlike Post-Double-(Adaptive-)Lasso with $\lambda^{min}$. The Post-Double-Autometrics method applies Autometrics successively to two different models and the expected gauge of each model is $\alpha$. Therefore, if the number of redundant variables is very large (like in our simulation), approximatively $2 \alpha$% of redundant variables are expected to be retained when considering the union between $\boldsymbol{X}^*$ and $\boldsymbol{X}^{**}$, which is exactly what we observe in Table (ref).\footnote{Note that the gauge for the standard Autometrics method is approximately equal to $\alpha$.}
Finally, although OLS using all variables is unbiased, its RMSE is relatively large compared to those of unbiased double-selection methods. For example, the RMSE of OLS is about $35\%$ larger than Post-Double-Autometrics with $\alpha = 0.05$.
To conclude, Post-Double-Autometrics with $\alpha=0.05$ competes very well with Post-Double-Lasso with $\lambda^{min}$ and Post-Double-Adaptive-Lasso with $\lambda^{min}$ and $\lambda^{1se}$ in this very simple and unrealistic framework of independent covariates.
This section complements the previous simulation results by first allowing the covariates to be correlated and then by varying the non-centrality parameter $\psi^d$ of the 10 relevant covariates in Equation (ref).
Figure (ref) displays bias and RMSE of $\hat\delta$ as well as the potency and gauge for $\psi^y = 2.5$, $\psi^d = 4$, $n=400$ and $p=210$ dependent control variables, with $\rho \in \left[-0.9; 0.9\right]$, for OLS and double-selection methods, across $1{,}000$ replications. The case $\rho=0$ corresponds to the results in Table (ref).\footnote{Results for single-selection methods are available upon request.}
The conclusions from Table (ref) are confirmed, but some interesting new results are emerging. Indeed, Post-Double-Lasso methods are again found to be sensitive to the selection method of $\lambda$, Post-Double-Lasso with $\lambda^{min}$ can lead to more accurate estimates than OLS when $\rho>-0.4$, and the Post-Double-Adaptive-Lasso improves the results compared to standard Lasso methods. Importantly, the performance of all Lasso-based methods deteriorates when $\rho$ becomes strongly negative, i.e., the bias and RMSE increase while the potency largely decreases.\footnote{Hendry2014 found similar results, demonstrating that Lasso's effectiveness decreases significantly when variables are negatively correlated.}
In contrast, Post-Double-Autometrics provides reliable results regardless of $\alpha$ and $\rho$. The bias and RMSE of Post-Double-Autometrics remain close to 0 for all values of $\rho$, whereas the gauge is still close to $2\alpha$. The results thus suggest that Post-Double-Autometrics outperforms other estimation methods as it provides the best treatment effect estimation (lowest bias and RMSE) and the best variable selection (high potency and controlled gauge), while being insensitive to $\alpha$ and to the magnitude of the correlation between the covariates.
Finally, Figures (ref)-(ref) complement previous results by displaying respectively the bias, RMSE, potency and gauge as a function of the correlation coefficient, with $\rho\in [-0.9,0.9]$, and of the non-centrality parameter $\psi^d$, with $\psi^d \in [1,8]$, for $\psi^y = 2.5$, $n=400$ and $p=210$. The higher $\psi^d$ is, the greater the probability that these variables will be selected, i.e., the more relevant they are. Therefore, the potency is expected to increase with $\psi^d$, resulting in lower values of the bias and RMSE. Note that the case $\rho = -0.6$ and $\psi^d = 4$ corresponds to the results displayed in Figure (ref), while those of Figure (ref) have been obtained for $\psi^d = 4$ and $\rho\in [-0.9,0.9]$. For the sake of clarity, we focus on the best performing methods, i.e., Post-Double-Lasso for $\lambda = \{\lambda^{min}, \lambda^{1se}\}$, Post-Double-Adaptive-Lasso for $\lambda = \{\lambda^{min}, \lambda^{1se}\}$ and Post-Double-Autometrics for $\alpha = \{5\%, 1\%\}$.\footnote{Results for other methods are available upon request.} In the figures, these methods are denoted, respectively, PDL-min, PDL-1se, PDAL-min, PDAL-1se, PDA-5% and PDA-1%.
The bias in Lasso-based methods is influenced not only by the parameters $\lambda$ and $\rho$, but also by the value of $\psi^d$. Even for large $\psi^d$, these methods remain biased when $\rho$ is strongly negative, and the bias only disappears when $\psi^d \ge 7.5$. Similar patterns are observed for the RMSE. Furthermore, the results indicate that the gauge increases with $\psi^d$, reaching levels above 30% with $\lambda^{min}$. Comparable findings hold for the adaptive-Lasso methods, though with slightly better performance in terms of bias and RMSE. In summary, while Lasso-based double-selection methods can yield accurate estimates of treatment effects, their performance depends on the choice of $\lambda$, the correlation among covariates ($\rho$), and the individual relevance of these covariates ($\psi^d$).
In contrast, the performance of Post-Double-Autometrics is remarkable in all cases. While the bias remains close to 0 for all values of $\psi^d$ due to a potency close to 1, the gauge does not increase and is close to $2\alpha$ regardless of the value of $\psi^d$. Moreover, the RMSE of Post-Double-Autometrics is the lowest among all methods, and especially for $\alpha = 0.05$. Results thus suggest that Post-Double-Autometrics outperforms other methods as it provides the most accurate treatment effect, regardless of the choice of $\alpha$, the correlation between the covariates and the non-centrality parameter $\psi^d$.
In Appendix 2, additional simulation results are presented with Elastic Net, which combines Lasso and Ridge penalizations. This method can overcome some limitations of the Lasso, especially with group of highly correlated covariates (zouhastie2005). Used in a post-double selection procedure, Figure (ref) shows better results than Post-Double-Lasso (lower bias) but worse than Post-Double-Autometrics (higher gauge and RMSE) in Figures (ref)-(ref). Note that the Elastic Net method requires the calibration of two hyperparameters, which greatly increases the calculation time, reason why we only carried out 100 replications for this method.
Finally, we study the robustness of double-selection methods in a high-dimensional linear regression model with more variables than observations. More precisely, we set $n = 200$ and keep all other parameters unchanged, so that $p=210>n$. Figures (ref)-(ref) display respectively the bias, RMSE, potency and gauge as a function of the correlation coefficient, with $\rho\in [-0.9,0.9]$, and $\psi^d \in [1,8]$, for $\psi^y = 2.5$. Results are similar to those of Figures (ref)-(ref), but some differences can be observed.
First, the bias of Lasso-based methods is substantially larger than 0 for $p>n$ despite a larger potency. Indeed, as the sample size $n$ decreased compared to the previous simulation but the non-centrality parameters $\psi^y$ and $\psi^d$ remained constant, the non-zeros coefficients in $\boldsymbol{\beta}$ and $\boldsymbol{\gamma}$ increased. Therefore, as Lasso selects variables based on the magnitude of the penalized coefficients (in absolute value), more relevant control variables are included in the terminal model, resulting in a larger potency. However, the effect of missed relevant control variables also increases (as $\beta$ is larger), leading to a larger bias.
Second, the effect of the adaptive-Lasso over the Lasso is unclear for $p>n$. While the bias and RMSE of Post-Double-Adaptive-Lasso and Post Double-Lasso are similar for $\psi^d \le 4$, they differ for larger values of $\psi^d$. Indeed, the bias and RMSE of Post-Double-Adaptive-Lasso increases with $\psi^d$, whereas they decrease towards 0 for Post-Double-Lasso when $\psi^d>0$ as for the case $p<n$.
Third, Post-Double-Autometrics is still unbiased whatever the values of $\rho$ and $\psi^d$. As Autometrics selects variables based on their significance, potency and bias remain similar. However, the gauge of Post-Double-Autometrics is slightly larger than $2\alpha$. Such result comes from the block-search performed by Autometrics when $p> n$, which leads to a small increase of the gauge.
To conclude, Post-Double-Autometrics also outperforms Post-Double(-Adaptive)-Lasso when $p>n$ and represents a reliable alternative when OLS is unfeasible.
In this section, we revisit an international economic growth example. In the empirical growth literature, estimating the effect of an initial level of GDP per capita on the growth rates of GDP per capita is a key issue. Indeed, an important finding of the Solow-Swan-Ramsey growth model is the hypothesis of convergence, stating that poorer countries should catch up with richer ones over time as they grow faster. Following this hypothesis, the effect of the initial level of GDP on its growth rate, for a given country, should be negative. To assess whether or not this assumption holds empirically, we use $n = 90$ complete observations extracted from the database of Barro1994.\footnote{The data is available in the R library hdm of hdm using the command data("GrowthData").} The original dataset consists of a panel of 138 countries for the period of 1960 to 1985, and includes $p = 60$ control variables ($\boldsymbol{X}$), such as variables measuring the education, trade openness or mortality rate. The dependent variable ($y$) corresponds to the national growth rates in GDP per capita for the periods 1965-1975 and 1975-1985, and the variable of interest ($d$) is the real GDP per capita in 1965. See Table (ref) in Appendix 1 for a complete description of the variables included in the dataset.
Many contributions in the literature investigated the convergence hypothesis. Using a simple bivariate regression model of growth rates on the initial level of GDP, Barro1995 showed that the convergence hypothesis is rejected. However, this estimate may be biased as the model does not take into account other countries' characteristics potentially affecting the growth rate, the initial level of GDP, or both. The literature thus focused on estimating this effect conditional on some characteristics (Levine1992, Sala1997, Sala2004). Given that the number of control variables is close to the number of observations, covariate selection is critical and the findings of these articles were severely criticized as they relied on ad-hoc procedures. More recently, using the same sample of $90$ countries than in our application, Belloni2011b and Belloni2013c shed light on the convergence hypothesis using Lasso-based approaches. The authors find that a negative and statistically significant effect of the initial level of GDP on its growth rate, supporting the convergence hypothesis of the Solow-Swan-Ramsey growth model. We revisit this application using the different methods presented in Section (ref) to show whether or not our results are supporting those of the literature, and ultimately if the convergence hypothesis is verified.
Table (ref) displays the results obtained for Post-Lasso and Post-Double-Lasso with $\lambda = \{\lambda^{bya}, \lambda^{min}, \lambda^{1se}, \lambda^{bcch}\}$, Post-Adaptive-Lasso and Post-Double-Adaptive-Lasso with $\lambda = \{\lambda^{min}, \lambda^{1se}\}$, Autometrics and Post-Double-Autometrics with $\alpha = \{0.05, 0.01\}$, as well as OLS regressions with and without control variables. We also report the results of an OLS regression involving all control variables with the correction of Cattaneo2018 to control for high-dimension and heteroscedasticity. The results of the OLS methods without control variables suggest that the convergence hypothesis is rejected as the coefficient is not significant at the 10% nominal level, which is consistent with the finding of Barro1995. This result should be taken with caution as this model is subject to omitted variable bias due to the non-inclusion of any control variables. The inclusion of 60 control variables in the model (with or without correction of the standard error of $\hat\delta$) does not change the conclusion. However, this result should be interpreted with caution due to the high number of variables in the model, leading to a relatively large standard error.
To obtain more accurate estimates of $\delta$, we rely on the methods presented in Section (ref) to select the relevant control variables. The results show that the coefficient $\hat \delta$ is always negative, except for the two methods that do not select any covariates, i.e., OLS without control variables and Post-Lasso with $\lambda^{bya}$. Consistent with the findings of our Monte Carlo simulations, results of the Post-Lasso and Post-Double-Lasso are very sensitive to the choice of $\lambda$. While $\delta$ is significantly different from 0 at the 10% nominal level for both Post-Adaptive-Lasso and Post-Double-Adaptive-Lasso, the results for Post-Lasso and Post-Double-Lasso methods depend on the choice of $\lambda$. Specifically, $\delta$ is significantly different from 0 for all Post-Lasso and Post-Double-Lasso methods except for Post-Lasso with $\lambda^{bya}$ and Post-Double-Lasso with $\lambda^{min}$.
Among the Post-Double-Lasso methods, $\lambda^{min}$ selects more than 10 additional control variables compared with $\lambda^{bya}$, $\lambda^{1se}$ and $\lambda^{bcch}$ and out of these four methods is the only one allowing to reject the hypothesis of convergence (or equivalently to fail to reject the null hypothesis $H_0: \delta=0$). Naturally, decreasing the penalization parameter increases the number of selected control variables, which affects the statistical inference on $\delta$.\footnote{Similarly, Belloni2011b show in their application that despite the fact that the number of selected control variables increases as $\lambda$ decreases, their main conclusion remained unaffected by the choice of $\lambda$.} It echoes with the recommendation of wuthrich2023 to check the sensibility of the results to the choice of $\lambda$ by varying $\lambda$ in a grid around the chosen `optimal' value. Similar results are observed for the Post-Double-Adaptive-Lasso, even though $\delta$ remains significant for both values of $\lambda$. \footnote{Additional results with Elastic Net are presented in Appendix 3. $\delta$ is significantly different from 0 for Post-ElasticNet, not for Post-Double-ElasticNet which gives the same results as OLS.}
Autometrics results differ between the two target sizes considered in the application. Specifically, while $\delta$ is significantly different from 0 for $\alpha=0.01$ (with only 3 control variables selected), $\delta$ is not significantly different from 0 for $\alpha=0.05$ (with 12 control variables selected).
In contrast, Post-Double-Autometrics provides consistent results for both values of $\alpha$ and suggests that the convergence hypothesis is not verified (i.e., $H_0:\delta=0$ is not rejected). This conclusion does not support the previous findings of the literature obtained with Lasso-based methods.\footnote{Note that our results are also consistent with those of the Post-Double-Lasso with $\lambda^{min}$, which has been shown to be the overall best Lasso-based method in our previous section across several Monte Carlo simulation experiments.} Moreover, the results obtained are more accurate than those of OLS methods with all control variables as the standard errors of $\hat\delta$ are much smaller.
To illustrate the (lack of) robustness of the Post-Double-Lasso, Figure (ref) (left panel) displays the estimates and $90\%$ confidence intervals of $\hat\delta$ with $\lambda$ in the first two steps of the Post-Double-Lasso set to $\tau\lambda^{min}$ with $\tau \in \left(0,5\right)$.\footnote{Note that when $\tau = 0$, the results are identical to those of the OLS method with all control variables, and when $\tau = 1$ the results are those displayed in Table (ref).} The results show the sensitivity of the statistical inference on $\delta$ for Post-Double-Lasso to the value of $\lambda$. Indeed, while the coefficient is not significant for $\tau < 1.75$, it becomes negative and significant for $\tau \ge 1.75$. Results obtained for $\lambda^{bya}$, $\lambda^{1se}$, $\lambda^{bcch}$ are qualitatively similar (not reported here) Therefore, the conclusion of Post-Double-Lasso and Post-Double-Adaptive-Lasso regarding the convergence hypothesis strongly depends on the choice of $\lambda$.
On the other hand, the results of Post-Double-Autometrics are robust to the choice of $\alpha$. Figure (ref) (right panel) displays the estimate and $90\%$ confidence interval of $\hat\delta$ for $\alpha \in [1\%,10\%]$. The results show that the estimates and confidence intervals are stable across all considered values of $\alpha$, and that the convergence hypothesis is never satisfied. In summary, unlike Belloni2011b and Belloni2013c, our results do not support the convergence hypothesis of the Solow-Swan-Ramsey growth model.
Finally, Table (ref) displays the control variables selected by Post-Double-Autometrics, Post-Double-Lasso and Post-Double-Adaptive-Lasso. Among all control variables, the most important ones are the black market premium (bmp1l), which is a proxy of trade openness, the life expectancy at birth (lifee065), the percentage of workers within the population (worker65), and the ratio of export to GDP (ex1) as they are selected by several methods. Interestingly, the variable measuring the ratio of import to GDP (im1) is only selected by the three double-methods for which the convergence hypothesis is not verified, i.e., Post-Double-Autometrics with $\alpha = \{0.05, 0.01\}$ and Post-Double-Lasso with $\lambda^{min}$. This variable is likely the primary factor influencing the significance or non-significance of $\delta$ in the competing post-double methods, thereby contributing to the OVB. To validate this finding, we analyze the role of variable im1 in the model.
First, Table (ref) reports the $t$-statistic associated with the coefficient of im1 in the two steps of Post-Double-Autometrics with $\alpha = \{0.05, 0.01\}$. The results confirm that im1 is relevant, as it is selected in both steps. While the $t$-statistics are positive in the first step, they are negative in the second step. This suggests that, conditional on the other selected control variables, the correlation between $d$ and im1 is negative. Furthermore, the $t$-statistics are relatively small (ranging from 2 to 4 in absolute value). This mirrors previous Monte Carlo simulation findings, where we found that when the non-centrality parameters of relevant variables were small both in Equations (ref) and (ref), Post-Double-Autometrics selects relevant variables more frequently than Post-Double-Lasso.
Second, Table (ref) shows the confidence intervals for the three methods, with and without variable im1 in the final OLS regression. Interestingly, while the estimated effect of initial GDP, i.e., $\delta$, is not significant when im1 is included in the final OLS regression, it becomes significant when it is omitted. $\hat \delta$ remains negative, regardless of its significance, but the coefficient of im1 is positive when included in the final model (0.274, 0.355, and 0.444 for Post-Double-Lasso with $\lambda^{min}$ and Post-Double-Autometrics at $\alpha = \{0.05, 0.01\}$, respectively), leading to an insignificant $\delta$. This suggests that omitting im1 from the model introduces a negative bias. This finding highlights the importance of including im1 and explains why models excluding this variable result in a significant coefficient for initial GDP. Contrary to \Citet{Levine1992}, we thus find that the share of imports is not substitutable to export share, but that each of these two variables should be included in the terminal model.
Finally, these three methods also select other variables in common, such as variables related to the population not working (pop1565 and pop6565) or to education (nof65, seccm65 and sf65). The results thus suggest that the most important covariates allowing to reject the convergence hypothesis are mainly related to trade openness, infant life expectancy, proportion of worker in the population, export and import (relative to the GDP), and education.\footnote{Note that one variable related to education (pf65) is only selected by both Post-Double-Autometrics methods.} Furthermore, Post-Double-Lasso with $\lambda^{min}$ selects several additional variables compared to Post-Double-Autometrics. These ones are mainly related to education, but also to tariff restriction, political instablity, exchange rate and government expenditures. However, given that Post-Double-Lasso with $\lambda^{min}$ selects on average many irrelevant variables (see previous section), most of these variables are probably irrelevant and could be dropped from the analysis.
In this paper, we have investigated the shortcomings of the Post-Double-Lasso method for estimating a parameter of interest, as an average treatment effect, in linear regression models with numerous covariates. While Post-Double-Lasso has gained popularity for its ability to mitigate omitted variable bias and manage overfitting through Lasso variable selection, we highlighted its vulnerabilities, particularly its sensitivity to the choice of the regularization parameter and the possibility of substantial omitted variable bias when some of the variables are negatively correlated.
As an alternative to Post-Double-Lasso, we introduced Post-Double-Autometrics, which leverages the Autometrics variable selection method, a method based on statistical inference to select relevant variables. Our findings demonstrate that Post-Double-Autometrics can effectively mitigate bias in estimating treatment effects by retaining relevant variables that might be overlooked by Lasso-based methods. Through extensive Monte Carlo simulations and an empirical illustration, we showed that Post-Double-Autometrics outperforms Post-Double-Lasso, yielding more reliable estimates of the treatment effect.
We would like to thank the participants of the São Paulo School of Advanced Science on High Dimensional Models (Brasil), the 2024 French Stata conference in applied econometrics (Marseille) and the 2024 QFFE conference (Marseille) for their valuable comments.
The project leading to this publication has received funding from the French government under the “France 2030” investment plan managed by the French National Research Agency (reference :ANR-17-EURE-0020 and ANR-21-CE26-0007-01) and from Excellence Initiative of Aix-Marseille University - A*MIDEX.