Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
56,039 characters · 20 sections · 27 citation commands
Valid Simultaneous Inference in High-Dimensional Settings (with the HDM Package for R)
\pagestyle{myheadings} \markboth{\sc Simultaneous Inference with HDM}{\sc }
Valid simultaneous inference becomes very important when multiple hypotheses are tested at the same time. For instance, suppose a researcher wants to estimate a potentially large number of regression coefficients and to assess which of these coefficients are significantly different from zero at a given significance level $\alpha$. It is well known that in such a situation, the naive approach, i.e. simply ignoring multiple testing issues, will generally lead to flawed conclusions due to a large number of mistakenly rejected hypotheses. Indeed, the actual probability of incorrectly rejecting one or more hypotheses will in general exceed the claimed/desired level $\alpha$ by large.
The statistical literature has proposed various approaches to mitigate the consequences of testing multiple hypotheses at the same time. These methods can be grouped into two approaches according to the underlying criterion. The first approach, initiated by the famous Bonferroni correction, seeks to control the probability of at least one false rejection which is called the family-wise error rate (FWER). Since the definition of the FWER refers to the probability of making at least one type I error, the FWER-criterion is appealing from an intuitive point of view. However, FWER control is often criticized to be conservative and, instead, the false discovery rate (FDR) control is used as a criterion in many test procedures leading to the second major class of multiple testing correction methods, e.g. bh1995. The FDR refers to the expected share of falsely rejected null hypotheses and, hence, results from FDR-procedures differ from classical tests results in terms of interpretation. Various approaches aim at maintaining control of the FWER and reducing conservativeness at the same time by incorporating a stepwise procedure, for instance, the stepdown method of holm1979. Moreover, taking the dependence structure of test statistics into consideration allows to reduce conservativeness of FWER-procedures, as in the stepdown procedure of romanowolf2005, romanowolf2 which is based on resampling methods.
In the following, we review classical methods and will describe how valid simultaneous inference can be conducted in a high-dimensional regression setting, i.e. if the number of covariates exceeds the number of observations, and give examples how the presented methods can be applied with the statistical software {\normalfontR }{.} It is well-known that classical regression methods, such as ordinary least squares, break down in high-dimensional settings. Instead, regularization methods, e.g. the lasso, can be used for estimation. However, post-selection inference is non-trivial and requires modification of the estimators.
The {\normalfontR } package {hdm} implements the double-selection approach of BCH2011restud that allows to perform valid inference based on model-selection with the lasso. Moreover, {hdm} provides powerful tests for a large number of hypotheses using the multiplier bootstrap, potentially in combination with the Romano and Wolf stepdown procedure (starting with Version 0.3.0) . The implemented methods are less conservative / more powerful than traditional approaches as they allow to take the dependence structure among test statistics into account and proceed in a stepwise manner. At the same time, the implemented procedures are attractive from an intuitive point of view as they guarantee control of the family-wise error rate (FWER).
The remainder of the paper is organized as follows. First, the setting is introduced and an overview on valid post-selection inference in high dimensions is provided. Second, a short and selective review on traditional and recent methods to adjust for multiple testing is presented. Third, we give an overview on the tools for valid post-selection inference in high-dimensions available in the {\normalfontR } package {hdm}. Fourth, the use of the software is illustrated in a simulated and a replicable real-data example. A conclusion is provided in the last section.
We are interested in testing a set of $K$ hypotheses $H_1, ..., H_K$ in a high-dimensional regression model, i.e. a regression where the number of covariates $p$ is large, potentially much larger than the number of observations $n$, i.e. we have $p\gg n$. The ultimate objective in this setting is to perform inference on a subset of the regression coefficients, i.e. a vector of so-called “target” coefficients $\theta_k$ with $k = 1,..., K$ and $K<n$.\footnote{The setting with a potentially infinite-dimensional vector of target coefficients is theoretically considered in belloni2017 and BCK2015.}
where $\beta_0$ is an intercept and $\beta$ denote the regression coefficients of the control variables $x_i$. Moreover, it is assumed that $E_n[\epsilon_i x_i] = 0$. In this setting, $K$ hypotheses are tested for the coefficients that correspond to the effect of the “target” variables $d_i$ on the outcome $y_i$
For instance, such a high-dimensional regression setting arises in causal program evaluation studies, where a large number of regressors is included to approximate a potentially complicated, non-linear population regression function using transformations with dictionaries, e.g. interactions, splines or polynomials. Alternatively, an analysis of heterogeneous treatment effects across possibly many subgroups as in the example in Section (ref) might require a large number of interactions of the regressors.
Suppose, there is a test procedure for each of the hypotheses leading to test statistics $t_1, ..., t_K$ and unadjusted p-values $p_1, ..., p_K$. In the context of multiple testing, it is often helpful to sort the p-values in an increasing order (i.e. the “most significant” test result as the first in the row) and the hypotheses likewise, i.e. $p_{(1)}, ..., p_{(K)}$ and $H_{(1)}, ..., H_{(K)}$ with $p_{(1)} \le p_{(2)} \le ... \le p_{(K)}$. Also the test statistics are ordered by the same logic $\vert t_{(1)} \vert \ge \vert t_{(2)} \vert \ge ... \ge \vert t_{(K)} \vert$. A researcher decides whether to accept or to reject a null hypothesis if the corresponding p-value $p_k$ is above or below a prespecified significance level $\alpha$. Generally, the significance level corresponds to the probability of erroneously rejecting a true null hypothesis. However, if the conclusions are based on a comparison of unadjusted p-values and the significance level, the probability of incorrectly rejecting at least one of the hypotheses will, potentially by large, exceed the claimed level $\alpha$. Hence, adjustment for multiple testing becomes necessary to draw appropriate inferential conclusions.
In high-dimensional settings traditional regression methods such as ordinary least squares break down and testing the $K$ hypotheses will severely suffer from the shortcomings of the underlying estimation method. Penalization methods, for instance the lasso or other machine learning techniques, provide an opportunity to overcome the failure of traditional least squares estimation as they regularize the regression problem in Equation (ref) by introducing a penalization term. In the example of lasso, the ordinary least squares minimization problem is extended by a penalization of the regression coefficients using the $l_1$-norm. The lasso estimator is the solution to the maximization problem
with $\| \bullet \|_1$ being the $l_1$-norm, $\lambda$ is a penalization parameter and $\hat{\psi}$ denotes a diagonal matrix of penalty loadings. More details on the choice of $\lambda$ and $\hat{\psi}$ as implemented in the {hdm} package can be found in the package vignette available at \href{https://cran.r-project.org/web/packages/hdm/index.html}{CRAN} and rjournalhdm. As a consequence of the $l_1$-penalization, some of the coefficients are shrunk towards zero and some of them are set exactly equal to zero. In general, inference after such a selection step is only valid under the strong assumption of perfect model selection, i.e. the lasso does only set those coefficients to zero that truly have no effect on $y_i$. However perfect model selection and the underlying assumptions are often considered unrealistic in real-world applications leading to a breakdown of the naive inferential framework and, thus, flawed inferential conclusions.
In contrast to the naive procedure, the so-called double-selection approach of BCH2011restud tolerates small model selection mistakes so that valid confidence intervals and test procedures can be based on the lasso. The double-selection method is based on orthogonalized moment equations and introduces an auxiliary (lasso) regression step for each of the target coefficients in order to avoid that variable selection erraneously excludes variables that are related to both the outcome and the target regressors by setting their coefficients equal to zero. Double selection proceeds as follows: (1) For each of the target variables in $d_{j,i}$, $j = 1,\ldots,K$, a lasso regression is estimated to identify the most important predictors among the covariates $x_i$ and the remaining target variables $d_{-j,i}$. (2) A lasso regression of the outcome variable $y_i$ on all explanatory variables, except for $d_{j,i}$, is estimated to identify predictors of $y_i$. This step is executed for each of the target variables $d_j$ with $j = 1,\ldots, K$. (3) The target coefficients $\theta$ are estimated from a linear regression of the outcome on all target variables as well as all covariates that have been selected in either step (1) or (2). As a consequence of the double-selection procedure, the risk of an omitted variable bias that might arise due to imperfect model selection is reduced. It can be shown that the double-selection estimator $\hat{\theta}^{DS}_k$ is asymptotically normally distributed under a set of regularity assumptions. Probably, the most important of these assumptions is (approximate) sparsity. This assumption states that only a subset of the regressors suffice to describe the relationship of the outcome variable and the covariates and that all other regressors have no or only a negligible effect on the outcome. In general, valid post-selection inference is compatible with other tools from the machine learning literature, for instance elastic nets or regression trees, as long as these methods satisfy some regularity conditions BCK2015, BCH2011restud.
In the multiple testing scenario described above, the double-selection approach can be used to test the $K$ null hypotheses $H_{0,k}$, $k=1,...,K$. BCK2015 show that a valid ($1-\alpha$)-confidence interval can be constructed by using the multiplier boostrap as established in CCK2013. Moreover, CCK2013 and BCK2015 show that a multiplier bootstrap version of the Romano-Wolf method can be used to construct a joint significance test in a high-dimensional setting such that asymptotic control of the FWER is obtained.
An advantage of the stepdown method of romanowolf2005, romanowolf2 is that it is based on resampling methods. Hence, it is able to account for the dependence structure underlying the test statistics and to give less conservative results as compared to methods such as the Bonferroni and Holm correction. The idea of the Romano-Wolf procedure is to construct rectangular simultaneous confidence intervals in subsequent steps whereas in each step, the coverage probability is kept above a level of $(1-\alpha)$. If in step $j$, the confidence set does not contain zero in dimension $k$, the corresponding $H_k$ is rejected. In step $j+1$, the algorithm proceeds analogously by constructing a rectangular joint confidence region for those coefficients for which the null hypotheses has not been rejected in step $j$ or before. The algorithm stops if no hypothesis is rejected anymore. To take the dependence structure of the test hypotheses into account, the classical Romano-Wolf stepdown procedure uses the bootstrap to compute the constant $c_{(1-\alpha)}$ which is needed to construct a rectangular confidence interval. This constant is estimated by the ($1-\alpha$)-quantile of the maxima of the bootstrapped test statistics in each step to guarantee the coverage probability of ($1-\alpha$). The computational burden of the Romano-Wolf stepdown procedure can be reduced by using the multipler bootstrap. This bootstrap method requires the calculation of the solution of an orthogonal moment equation only once and then operates on pertubations by realizations of an independently and standard normally distributed random variable CCK2013.
A basic question frequently arising in empirical work is whether the Lasso regression has explanatory power, comparable to a F-test for the classical linear regression model. The construction of a joint significance test follows CCK2013 (Appendix M). Based on the model $ y_i = \beta_0 + d_i'\theta + x_i'\beta + \epsilon_i $ with intercept $\beta_0$, the null hypothesis of joint statistical in-significance is $H_0: (\theta', \beta')'=\mathbf{0}$. The null hypothesis implies that $$ \mathbb{E} \left[ (y_i - \beta_0) x_i \right] = 0,$$ and the restriction can be tested using the sup-score statistic: $$S = \| \sqrt{n} \mathbb{E}_n \left[ (y_i - \hat \beta_0) x_i \right] \|_\infty,$$ where $\hat \beta_0 = \mathbb{E}_n [y_i]$. The critical value for this statistic can be approximated by the multiplier bootstrap procedure, which simulates the statistic: $$ S^* = \| \sqrt{n} \mathbb{E}_n \left[ (y_i - \hat \beta_0) x_i g_i \right] \|_\infty,$$
where $g_i$'s are i.i.d. $N(0,1)$, conditional on the data. The $(1-\alpha)$-quantile of $S^*$ serves as the critical value, $c(1-\alpha)$. We reject the null if $S > c(1-\alpha)$ in favor of statistical significance, and we keep the null of non-significance otherwise.
The FWER is defined as the probability of falsely rejecting at least one hypothesis. The goal is to control the FWER and to secure that it does not exceed a prespecified level $\alpha$. We assume that for the individual tests the significance level is set uniformly to $\alpha$.
According to the Bonferroni correction the cutoff of the p-values is set to $\alpha^{*}=\alpha/K$ and all hypotheses with p-values below the adjusted level $\alpha^{*}$ are rejected. Bool's inequality then gives directly that the FWER is smaller or equal to $\alpha$. Instead of adjusting the level of $\alpha$ to $\alpha^{*}$, it is possible to adjust the p-values so that we reject a hypothesis $H_k$ if $p^{*}_k = K \cdot p_{k} < \alpha$. A drawback of the procedure is that it is quite conservative, meaning that in many applications, in particular in high-dimensional settings when many hypotheses are tested simultaneously, often no or very few hypotheses are rejected, increasing the risk of accepting false null hypotheses (i.e. of a type II error).
We again assume that the p-values are ordered (from lowest to highest) $p_{(1)} \le \ldots \le p_{(K)}$ with corresponding hypotheses $H_{(1)}, \ldots, H_{(K)}$. The Bonferroni-Holm procedure controls the FWER by the following procedure: Let $k$ be the smallest index such that the corresponding p-value exceeds the adjusted cutoff $\alpha^{*}$.
Reject the null hypothesis $H_{(1)}, \ldots, H_{(k-1)}$ and accept $H_{(k)}, \ldots, H_{(K)}$. The Bonferroni-Holm procedure can be considered as general improvement over the Bonferroni correction that maintains control of the FWER and reduces the risk of a type II error at the same time. The adjusted p-value according to the Bonferroni-Holm correction are computed as $p_{(j)}^{*} = \max_{l \le j} \min \{(K-j+1)p_{(j)}, 1 \}$ with $l = 1, \ldots, j$.
BCK2015 derive valid ($1-\alpha$)-confidence regions for the vector of target coefficients, $\theta$, in the high-dimensional regression setting in Equation (ref) estimated with lasso. The confidence regions which are constructed with the multiplier bootstrap can be used equivalently to a joint signficance test of the $K$ hypotheses. Accordingly, the null hypotheses $H_{0,k}: \theta_{k} = 0$, $k = 1, \ldots, K$, would be rejected at the level $\alpha$ if the simultaneous ($1-\alpha$)-confidence region does not cover zero in dimension $k$.
While the Bonferroni and Bonferroni-Holm correction do not take into account the dependence structure of the test statistics, further improvements in the control of the type II error can be achieved by modeling the dependence structure using resampling. A very popular and powerful method in this regard is the Romano-Wolf stepdown procedure. Stepdown methods proceed in several rounds where in each round a decision is taken on the set of hypotheses being rejected. The algorithm continues until no further hypotheses are rejected. The Romano-Wolf stepdown procedure guarantees asymptotic control of the FWER at level $\alpha$ by constructing a sequence of simultaneous tests. We present the recent version of the Romano-Wolf method from CCK2013 and BCK2015 who prove the validity of the procedure in combination with the multiplier bootstrap.
The p-adjustment algorithm parallels that in romanowolf2016 with the only difference that the bootstrap test statistics are computed efficiently with the multiplier bootstrap procedure instead of the classical bootstrap. In contrast to traditional bootstrap methods, the multiplier bootstrap does not require re-estimation of the lasso regression for each bootstrap sample. romanowolf2, romanowolf2016 recommend a high number of bootstrap repetitions $B\ge 1000$.
If the data stem from a randomized experiment, the method introduced in list2016 can be used. It is a variant of the Romano-Wolf procedure under uncounfoundedness. Moreover, it allows to compare the effect of different treatments and several outcome variables simultaneously.
The FWER is a very strict criterion which is often very conservative. This means that in settings when thousands or hundred thousands of hypotheses are tested simultaneously, the FWER does often not detect useful signals. Hence, in large-scale settings frequently a less strict criterion, the so-called false discovery rate (FDR) is employed. The false discovery proportion (FDP) is defined as the ratio of the number of hypotheses which are wrongly classified as significant (false positives) and the total number of positives. If the latter is zero, it is defined as zero. The FDR is defined as the expected value of the FDP : $FDR=\mathbb{E}(FDP)$. The FDR concept reflects the tradeoff between false discoveries and true discoveries.
To control the FDR, the Benjamini-Hochberg (BH) procedure ranks the hypotheses according to the corresponding p-values and then chooses a cutoff along the ranking to control the FDR at a prespecified level of $\gamma \in (0,1)$. The BH procedure first uses a stepup comparision to find a cutoff p-value:
and then rejects all hypotheses $H_{j}, j=1,\ldots,k$. In most applications, $\gamma = 0.1$ is chosen.
Estimation of the high-dimensional regression model in Equation (ref) and simultaneous inference on the target coefficients is implemented in the R package {hdm} available at \href{https://cran.r-project.org/web/packages/hdm/index.html}{CRAN}. {hdm} provides an implementation of the double-selection approach of BCK2015 using the lasso as the variable selection device. The function rlassoEffects() does valid inference on a specified set of target parameters and returns an object of S3 class rlassoEffects. This output object is used to perform simultaneous inference subsequently as described in the following. More details on the {hdm} package and introductory examples are provided in the {\texttt{hdm}} vignette available at \href{https://cran.r-project.org/web/packages/hdm/index.html}{CRAN} and rjournalhdm. The package {\texttt{hdm}} offers three ways to perform valid simultaneous inference in high-dimensional settings:
The simulation study provides a finite-sample comparison of different multiple testing corrections in a high-dimensional setting, i.e. the Bonferroni method, the Bonferroni-Holm procedure, Benjamini-Hochberg adjustment and the Romano-Wolf stepdown method. In addition, the study illustrates the failure of the naive approach that ignores the problem of simultaneous hypotheses testing, i.e. without any correction of the significance level or p-values.
We consider a regression of a continuous outcome variable $y_i$ on a set of $K = 60$ regressors, $d_i$,
with $\varepsilon_i \sim N(0,\sigma^2)$ and variance $\sigma^2 = 3$. In our setting, the realizations of $d_i$ are generated by a joint normal distribution $d_i \sim N(\mu, \Sigma)$ with $\mu = \mathbf{0}$ and $\Sigma_{j,k} = \rho^{|j-k|}$ with $\rho = 0.9$. We consider the case of an i.i.d. sample with $n = 200$ and $n = 500$ observations. The setting is sparse in that only few, $s = 12$, regressors are truly non-zero whereas the location of the non-zero coefficient is only known by an inaccessible oracle. Thus, the lasso is used to select the set of explanatory variables with a non-zero coefficient. Figure (ref) presents the regression coefficients in the simulation study.
The regression in Equation (ref) is estimated wih post-lasso and inference for all regression coefficients is based on double selection.\footnote{More details on implementation of the simulation study are provided in the appendix and the supplemental material available at \url{https://www.bwl.uni-hamburg.de/en/statistik/forschung/software-und-daten.html}.} The $K=60$ hypotheses are tested simultaneously
The simulation results are summarized in Table (ref). The reported results refer to averages over $R=5000$ repetitions in terms of correct and incorrect rejections of null hypotheses at a prespecified level of $\alpha = 0.1$ as well as the empirical FWER and FDR.
The results show that multiple testing adjustment is of great importance since naive inferential statements might be invalid. If each of the hypotheses is tested at a significance level of $\alpha = 0.1$ without adjustment of the p-values, the naive procedure leads to at least one incorrect rejection with a probability of almost 1. Also the Benjamini-Hochberg procedure with $\gamma = 0.1$ incorrectly rejects a true null in more than 3000 out of the 5000 repetitions. On average, more than 5 ($n=200$) and more than 4 ($n=500$) true hypotheses are rejected without adjustment of p-values. The simulation study illustrates that control of the FDR is achieved by the Benjamini-Hochberg correction. Accordingly, one would incorrectly reject more than 1 hypothesis on average. Over all 5000 repetitions 9.5% ($n=200$) and 8.7% ($n=500$) of all rejections are incorrect (false positives) which is below the specified level of $\gamma = 0.1$.
Methods with asymptotic control of the FWER are much less likely to erraneously reject true null hypotheses. In the setting with $n=200$ observations, the number of incorrect rejections is on average around 0.1 to 0.17 with the Bonferroni correction being most conservative. The empirical familywise error rates are very close to the desired level $0.1$, despite the small number of observations and the relatively large number of tested hypotheses. With larger sample size ($n=500$) the empirical FWERs approach the level $0.1$. The price for control of the probability of at least one type I error is paid in terms of power with less correct rejections for the FWER-methods as compared to the Benjamini-Hochberg correction. The largest number of correct rejections while still maintaining control of the FWER is achieved by the Romano-Wolf stepdown procedure. Hence, taking the dependence structure of the test statistics into account is favorable in case of dependencies among test statistics.
The following section demonstrates the methods for valid simultaneous inference implemented in the package {hdm} and provides a comparison of the classical correction methods in a replicable real-data example. The gender wage gap, i.e. the relative difference in wages emerging between male and female employees, is a central topic in labor economics. Frequently, studies report an average gender wage gap estimate, i.e. how much less women earn as compared to men in terms of average (i.e. mean or median) wages. However, it might be helpful for policy makers to gain a more detailed impression on the gender wage gap and to assess whether and to what extent the gender wage gap differs across individuals. A simplistic although frequently encountered approach to assess the wage gap heterogeneity is to compare the relative wage gap across female and male employees in subgroups that are defined in terms of a particular characteristic, e.g. industry. It is obvious that this approach neglects the role of other characteristics relevant for the wage income and, hence, the wage gap, e.g. educational background, experience etc. As an examplary illustration, the gender gap in average (mean) earnings in 12 different industrial categories is presented in Figure (ref), suggesting that the wage gap differs by large across the subgroups. In contrast to the approach that is simply based on descriptive statistics, an extended regression equation including interaction terms of the gender indicator with observable characteristics is able to take the role of other labor market characteristics into account and, hence, allows to give insights on the determinants of the gender wage gap. As the regression approach leads to a large number of coefficients that are tested simultaneously, an appropriate multiple testing adjustment is required. Thus, the heterogeneous gender wage gap example is used to demonstrate the adjustment methods provided in the {\normalfontR } package {hdm}. The presented example is an illustration of the more extensive analysis of a heterogeneous gender wage gap in gendergap.
The examplary data is a subsample of the 2016 American Community Survey.\footnote{It can be replicated with the documentation “Appendix: Replicable Data Example” that is available online. Further information and the code is provided at \url{https://www.bwl.uni-hamburg.de/en/statistik/forschung/software-und-daten.html}.} The data provide information on civilian full-time working (35+ hours a week, 50+ weeks a year) White, non-Hispanic employees aged older than 25 and younger than 40 with earnings exceeding a the federal minimum level of earnings (\$12,687.5 of yearly wage income). First, the data set is loaded from the {hdm} package and prepared for the analysis.
In order to answer the question whether the gender wage gap differs according to the observable characteristics of female employees in a valid way, it is necessary to account for regressors that affect women's job market environment. In the example, variables on marriage, the presence of own children, geographic variation, job characteristics (industry, occupation, hours worked), human capital variables (years of education, experience (squared)), and field of degree are considered. A wage regression is set up that includes all two-way interactions of female with the available characteristics in addition to the baseline regressors, $x_i$.
The analysis begins with constructing a model matrix that implements the regression relationship of interest.
Accordingly, the regression model considered has $p=123$ regressors in total and is estimated on the basis of $n = 70473$ observations. The wage Equation (ref) is estimated with the lasso with the theory-based choice of the penalty term (“rlasso”). To answer the question whether the included regressors have any explanatory power for the outcome variable, the global test of overall significance is run by calling summary() on the output object of the rlasso() function.
The hypotheses that all coefficients in the model are zero can be rejected at all common significance levels. The main objective of the analysis is to estimate the magnitude of the effects associated with the gender interactions and to assess whether these effects are jointly significantly different from zero. The so-called “target” variables, in total $62$ regressors, are specified in the index option of the rlassoEffects() function. Hence, it is necessary to indicate the columns of the created model matrix that correspond to interactions with the female dummy.
The output presented in the appendix shows the $K =62$ estimated coefficients together with t-statistics and unadjusted p-values. The next step is to adjust the p-values for multiple testing. Starting with Version 0.3.0, the {hdm} offers the S3 method p_adjust() for objects inheriting from classes rlassoEffects and lm. It hosts the correction methods from the function p.adjust() of the {\texttt{stats}} Package, e.g. Bonferroni, Bonferroni-Holm, Benjamini-Hochberg as well as no correction at all. First, the naive approach without a multiple testing correction given a significance level $\alpha$ is presented. Table (ref) shows the number of rejections at significance levels $\alpha \in \{0.01, 0.05, 0.1 \}$.
Thus, without a correction for multiple testing, $18$, $21$, and $25$ hypotheses could be rejected given the significance levels of 1%, 5% and 10%. If one returns to the initial example on variation by industry, one would find significant variation of the wage gap by industry (as compared to the baseline category “Wholesale”) in $3$ categories, namely “Agriculture”, “Finance, Insurance, and Real Estate” and “Professional and Related Services” at a significance level of 0.1.
Second, classical correction methods like the Bonferroni, Bonferroni-Holm, and the Benjamini-Hochberg adjustments are used to account for testing the $62$ hypotheses at the same time.
As a general improvement, the Holm-corrected p-values are smaller or equal to those obtained from a Bonferroni adjustment. At significance levels 1%, 5% and 10%, it is possible to reject fewer hypotheses if p-values are corrected for multiple testing.
According to the Benjamini-Hochberg (BH) correction bh1995 of p-values that achieves control of the FDR it is possible to reject $10, 19,$ and $21$ null hypotheses at specified values of the FDR, $\gamma$, at 0.01, 0.05 and 0.1.
Regarding variation by industry, the Bonferroni and Holm procedure find a significantly different wage gap (at the 10% significance level) only for industry “Finance, Insurance, and Real Estate” whereas the Benjamini-Hochberg correction with $\gamma = 0.1$ leads to the same conclusion as obtained without any adjustment. These results can now be compared to results obtained from joint significance test with and without the Romano-Wolf stepdown procedure. We can start with construction of a joint $0.9$-confidence region for the $62$ coefficients using the option joint=T in the confint() function for objects of the class rlassoEffects.
The results from the confidence intervals are equivalent to a test at significance level $\alpha=0.1$ so that $11$ hypotheses can be rejected. However, the Romano-Wolf stepdown procedure allows to increase power. For instance with a significance level of 5%, the stepdown correction allows to reject one hypothesis more than with the joint confidence interval. The p-values can be adjusted according to the Romano-Wolf-stepdown algorithm by setting the option method = “RW” (default) of the p_adjust() call. The number of repetitions can be varied by specifying the option B, $B=1000$ by default.
Using the joint confidence interval and the Romano-Wolf stepdown adjustment allows to reject more hypotheses than with traditional methods at significance levels 5% and 10%. Hence, taking into account the dependence of test statistics is beneficial in terms of power in the real-data example.
The previous sections provide a short overview on important methods for multiple testing adjustment in a high-dimensional regression setting. Throughout the paper, our intention was to present the concepts and the necessity of a multiple adjustment in a comprehensive way. Similarly, the tools for valid simultaneous inference in high-dimensional settings that are available in the {\normalfontR } package {hdm} are intended to be easy to use in empirical applications. The demonstration of the methods in the real-data example are intended to motivate applied statisticians to (i) use modern statistical methods for high-dimensional regression, i.e. the lasso, and (ii) to appropriately adjust if multiple hypotheses are tested simultaneously. Since the {hdm} provides user-friendly adjustment methods for objects of the S3 class lm, we hope that uses will use the correction methods more frequently even in classical least squares regression.