On the feasibility of parsimonious variable selection for Hotelling's $T^2$-test

On the Feasibility of Parsimonious Variable Selection for Hotelling's $T^2$-test

\vskip4pt

{\it This work is dedicated to the memory of Somesh Das Gupta, my colleague, teacher, and friend.} \vskip4pt

{\bf 1. Introduction.} This study is motivated by a re-examination of the variable-selection problem for Hotelling's $T^2$-test (closely related to variable selection for linear discriminant analysis). After some notational preliminaries in \S1.1, Hotelling's $T^2$ is reviewed in \S1.2. The variable-selection problem is described in \S1.3, where the substance of this investigation is described.

{\it 1.1. The noncentral $f$-distribution.} Let $\chi_m^2(\lambda)$ denote a noncentral chi-square random variable with $m$ degrees of freedom and noncentrality parameter $\lambda>0$. The noncentral $f_{m,n}(\lambda)$ distribution (nonnormalized) with $m$ and $n$ degrees of freedom and noncentrality parameter $\lambda>0$ is the distribution of the ratio $\chi_m^2(\lambda)/\chi_n^2$ (also denoted by $f_{m,n}(\lambda)$), where the numerator and denominator are independent chi-square random variables and $\chi_n^2\equiv\chi_n^2(0)$. The upper $\alpha$-quantile of $f_{m,n}\equiv f_{m,n}(0)$ is denoted by $f_{m,n}^\alpha$, so that

The {\it noncentral $f_{m,n}$-test of size} $\alpha\ge0$ for the problem of testing $\lambda=0$ vs. $\lambda>0$ has power function given by

see Das Gupta and Perlman [DGP] (1974), eqn.(2.1). Clearly $\pi_\alpha(\lambda;m,n)$ is decreasing in $\alpha$, with $\pi_0(\lambda;m,n)=0$. Because $f_{m,n}(\lambda)$ has strictly monotone likelihood ratio in $\lambda$, $\pi_\alpha(\lambda;m,n)$ is strictly increasing in $\lambda$.

It will be convenient to work with the (central) {\it beta distribution} $b_{m,n}$:

clearly, $b_{m,n}= 1-b_{n,m}$. The upper and lower $\alpha$-quantiles of $b_{m,n}$ are denoted by $b_{m,n}^\alpha$ and $b_{m,n;\alpha}$, respectively, so that

Thus by (ref),

The probability density function of $b_{m,n}$ is given by

{\it 1.2. Hotelling's $T^2${-}test.} Let $X_i:p\times 1$, $i=1,\dots,N$ ($N\ge p+1$) be a random sample from the $p$-dimensional multivariate normal distribution $N_p(\mu,\Sigma)$, where $\mu\ (p\times1)\equiv(\mu_1,\dots,\mu_p)' \in\mathbb{R}^p$ and $\Sigma\ (p\times p)\equiv(\sigma_{ij})$ is positive definite. The problem of testing

with $\Sigma$ unknown is invariant under the group action $X_i\to A X_i$, $i=1,\dots,N$, where $A\in GL(p)$, the group of all nonsingular $p\times p$ matrices. The maximal invariant statistic under $GL(p)$ is given by Hotelling's $T^2$ statistic:

where $\bar X=\sum_{i=1}^NX_i$ and $S=\sum_{i=1}^N(X_i-\bar X)(X_i-\bar X)'$. Its distribution is

where

the maximal invariant parameter. Therefore the uniformly most powerful invariant size-$\alpha$ test rejects $H_0$ if $T^2>f_{p,N-p;\alpha}$, with power function $\pi_\alpha(\Lambda;p,N-p)$; cf. [A] Theorem 5.6.1).\footnote{ Among the larger class of all tests, invariant or non-invariant, the $T^2$ test is admissible [S], proper Bayes [KS], and locally and asymptotically minimax for small and large values of $\Lambda$, respectively [GK].}

It is useful to express $\Lambda$ in terms of scale-free parameters, that is,

where $R\equiv(\rho_{ij})$ is the $p\times p$ correlation matrix determined by $\Sigma$ and

The testing problem (ref) can be stated equivalently as that of testing

with $R$ unknown. \vskip4pt

{\it 1.3. The $T^2$ variable-selection problem.} Denote the components of $\bar X$ by $\bar X_j$, $j=1,\dots,p$, and those of $S$ by $s_{jk}$, $j,k=1,\dots,p$. Let $\Omega_p$ be the collection of all nonempty subsets of $\{1,\dots,p\}$. For $\omega\in\Omega_p$ denote the $\omega$-subvector of $\bar X$ by $\bar X_\omega$, the $\omega$-submatrix of $S$ by $S_\omega$, and similarly define $\gamma_\omega$ and $R_\omega$. The $T^2$-statistic based on $(\bar X_\omega,S_\omega)$ is given by

($T_{\Omega_p}^2=T^2$, $\Lambda_{\Omega_p}=\Lambda\equiv\Lambda(\gamma,R)$.) The test that rejects $H_0$ if $T_\omega^2>f_{|\omega|,N-|\omega|;\alpha}$ has size $\alpha$ for $H_0$, and its power function is given by

This $T_\omega^2$-test is not invariant under $GL(p)$ but it is admissible for testing $H_0$ vs. $K$, being a unique proper Bayes test for a prior distribution that, under $K$, assigns mass 1 to $\{\mu\mid \mu_{\Omega_p\setminus\omega}=0\}$; cf. [KS], [MP].

The $T^2$ variable-selection problem is that of choosing a parsimonious subset $\omega$ such that the $T_\omega^2$-test maintains high power against alternatives deemed likely in some sense. Because $(\gamma,R)$ is unknown, variable selection in practice is traditionally approached by forward and/or backward selection procedures based on a preliminary sample that yields estimates of $(\gamma,R)$; see the Appendix. In general, all $2^p-1$ nonempty subsets $\omega$ must be considered.

Recently I consulted on such a variable-selection problem. The investigator, a web-page designer, had observed 20 physiological variables (blood pressure, temperature, heart rate, etc.) on each of 100 subjects (the numbers are approximate). He wished to compare their responses to a new web-page design with their responses to the current design. The overall $T^2$-statistic, based on a linear combination of all 20 variables, indicated a significant difference between the two sets of responses. However, the client desired to find a more readily interpretable measure of difference, namely a $T_\omega^2$-statistic based on a very small subset $\omega$ of the 20 variables, hopefully with $|\omega|=1$ or 2. Such a desire is not atypical of investigators presented with a multivariate data analysis. Therefore it occurred to me to wonder just how much power would be lost by restricting variable selection to small subsets $\omega$, for example to single variables or pairs of variables.

To state this more precisely, define

Thus $\^\omega_\alpha(\gamma,R)$ is the (not necessarily unique) subset $\omega$ of variables that maximizes the power of the size-$\alpha$ $T_\omega^2$-test to detect the alternative $(\gamma,R)$ if the actual value of $(\gamma,R)$ were revealed by an oracle. Whereas the admissibility of the overall size-$\alpha$ $T^2$-test dictates that its power cannot be dominated by that of the size-$\alpha$ $T_\omega^2$-test when $\omega\ne\Omega_p$, might it happen that $|\^\omega_\alpha(\gamma,R)|$ is small, perhaps 1 or 2, over a fairly wide range of parameter values $(\gamma,R)$? If so, then one might with some confidence limit variable selection to consideration of single variables (univariate $t^2$-tests) or pairs of variables (bivariate $T^2$-tests) -- thus the overall $p$-variate $T^2$-test would be ruled out {\it a priori}.

Of course, such a radical suggestion flies in the face of 100 years of multivariate statistical theory, of which I have been but one of many proponents. Nonetheless, this report presents some evidence, admittedly fragentary and incomplete, indicating that this suggestion might not be entirely inappropriate in applications. In Sections 2, 3, and 4, a few very special cases are considered where tractable algebraic expressions for the asymptotic ($\Lambda_\omega\to\infty$), local ($\Lambda_\omega\to0$), and/or exact values of $\pi_\alpha(\Lambda_\omega;|\omega|, N-|\omega|)$ are available. These in turn can be utilized to compare the powers of $T_\omega^2$ and $T^2$.

Examples 2.1 and 3.1 treat only the simplest possible case: the bivariate case ($p=2$) with $N=3$.\footnote{ However, Giri, Kiefer, and Stein [GKS] also began their study of Hotelling's $T^2$ test by considering only this simplest case $p=2$, $N=3$.} Here it is shown that $|\^\omega_\alpha(\gamma,R)|=1$ over large portions of the asymptotic and local regions of the alternative hypothesis $K$. This implies that the power of at least one of the two univariate Student $t^2$-tests ($|\omega|=1$) exceeds that of the overall (bivariate) $T^2$-test for most alternatives $(\gamma,R)$ in these regions.

In Example 4.4 this result is extended to the entire alternative hypothesis $K$, both for $N=3$ and $N=5$, but only under the highly restrictive and impracticably vague condition that $\alpha$ be sufficiently small, with “sufficiently small" determined by the value of the unknown noncentrality parameter -- see (ref) and (ref).

Examples 2.2 and 3.2 go beyond the bivariate case. Here $p\ge3$, $N=p+2$, and the powers of all possible bivariate $T_\omega^2$-tests ($|\omega|=2$) are compared to the power of the overall $p$-variate $T^2$ test, again only for asymptotic and local alternatives and, furthermore, only for very special configurations of $\gamma$ and $R$. In these cases, admittedly highly restrictive, the bivariate $T_\omega^2$-tests dominate the overall $T^2$ over a substantial portion of the alternative hypothesis $K$. This does not establish that $|\^\omega_\alpha(\gamma,R)|=2$ but again suggests that variable search might be limited to small variable subsets $\omega$.

Of course, a much more comprehensive study is needed to confirm the efficacy of such an approach to variable selection. It is hoped that this report might encourage such an investigation. \vskip6pt

{\bf 2. Some asymptotic power comparisons.} The power function of the $T_\omega^2$-test is

(recall (ref)-(ref)). It follows from eqn. (3.4) in [MP] that as $\Lambda_\omega\to\infty$,

Thus for two subsets $\omega,\,\omega'$ s.t. $\omega\subset\omega'$, $\exists\;\Lambda_{|\omega|,|\omega'|,N;\alpha}^*>0$ s.t.

Therefore power comparisons of $T_\omega^2$ and $T_{\omega'}^2$ for distant alternatives\footnote{ We do not claim to know the values of $\Lambda_{|\omega|,|\omega'|,N;\alpha}^*$, even approximately.} require determination of the lower quantiles $b_{n,m;\alpha}$. This can be done explicitly in Examples 2.1 and 2.2 below. Although these examples are of very limited scope\footnote{ But see Footnote (ref).} they begin to suggest that variable subset selection sometimes can be limited to very small subsets $\omega\in\Omega_p$, e.g., singletons in the bivariate Example 2.1, or pairs (including singletons) in Example 2.2.

To simplify the notation, set

The quantile $b_{n,m;\alpha}$ satisfies

For the simple cases $n=2$ or $m=2$,

{\bf Example 2.1.} In the bivariate case $p=2$, abbreviate the singleton subsets $\{1\}$ and $\{2\}$ of $\Omega_2$ by $1$ and $2$ respectively. We shall compare the powers $\pi_\alpha(\Lambda_1;\,1,\,2)$ and $\pi_\alpha(\Lambda_2;\,1,\,2)$ of the two univariate size-$\alpha$ $t^2$-tests to the power $\pi_\alpha(\Lambda;\,2,\,1)$ of the overall (bivariate) size-$\alpha$ $T^2$-test for distant alternatives.

Assume that $\gamma_1\ne0$ (recall (ref)) and set

where $-1<\rho<1$ , so by (ref) and (ref),

Without loss of generality we can assume that $|\gamma_1|\ge|\gamma_2|$, so $0\le\eta^2\le1$ and $\max(\Lambda_1,\Lambda_2)=\gamma_1^2$. Thus the alternative hypotheses $K$ can be represented as

while $\^\omega_\alpha(\gamma,R)$ can be re-expressed as $\^\omega_\alpha(\gamma_1,\eta,\rho)$.

Because $\max(\Lambda_1,\Lambda_2)\le\Lambda$, it follows from (ref) and (ref) that

In the simplest case $N=3$, (ref) yields the explicit expression

the inequality in (ref) is equivalent to

Note that

The quadratic function $h_{\alpha,\eta}(\rho)$ ($-1\le\rho\le1$) satisfies

It is easily seen that if $\alpha\le\frac{2}{3}$ then $Q_{1,2,3;\alpha}\le\f12$, so $h_{\alpha,\eta}(0)\le0$ for all $\eta\in[-1,1]$. Thus if $\alpha\le\frac{2}{3}$ then $h_{\alpha,\eta}(\rho)$ must have one root in $[-1,0]$ and one root in $[0,1]$. The two roots are given by

note that $\^\rho_{\alpha,-\eta}^{\pm}=-\^\rho_{\alpha,\eta}^{\mp}$.

It follows that if $\alpha\le\frac{2}{3}$ then for sufficiently large $\gamma_1^2$, i.e., $\gamma_1^2\ge\frac{1}{3}\Lambda_{1,2,3;\alpha}^*$,

that is, at least one of the two univariate $t^2$-tests is more powerful than the overall (bivariate) $T^2$-test. Specifically, when $\gamma_1^2\ge\frac{1}{3}\Lambda_{1,2,3;\alpha}^*$, $|\^\omega_\alpha(\gamma_1,\eta,\rho)|=1$ in the $(\eta,\rho)$-regions of the parameter space indicated in Table 1. From this it is seen that for $p=2$, $N=3$, large $\gamma_1^2$, and the common (small) values of $\alpha$, the overall size-$\alpha$ $T^2$-test is dominated by at least one of the two univariate size-$\alpha$ $t^2$-tests over a large portion of the alternative hypothesis $K$.$\square$ \vskip4pt

{\bf Example 2.2.} Suppose that $p\ge3$ and $N=p+2$. The powers of the ${p\choose2}$ bivariate size-$\alpha$ $T^2$-tests and the overall $p$-variate size-$\alpha$ $T^2$-test will be compared for distant alternatives, which requires comparison of the powers

From (ref),

Therefore for sufficiently large values of $\Lambda^{(2)}$, namely $\Lambda^{(2)}\ge\Lambda_{2,p,p+2;\alpha}^*$, at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall (p-variate) size-$\alpha$ $T^2$-test\footnote{ Note that by itself this does not establish that $|\^\omega_\alpha(\gamma,R)|=2$.} provided that

From (ref) we obtain the explicit expression

If we set $\nu_p=\frac{2}{p}$ and $U_{p;\alpha}=\frac{\nu_p}{Q_{2,p,p+2;\alpha}}$ then

Table 2 shows that $Q_{2,p,p+2;\alpha}$ decreases rapidly to 0 as $p\to\infty$, which suggests that (ref) might hold over substantial regions of the alternative hypothesis $K$. We proceed to exhibit several such regions. \vskip4pt

{\it Case 1: $\gamma_1=\cdots=\gamma_p=:\delta$ and $R$ has the intraclass form}

Here $-\frac{1}{p-1}<\rho<1$ and ${\bf e}_p=(1,\dots,1)':p\times1$. Then

so by (ref) and (ref),

Thus $\Lambda^{(2)}\ge\Lambda_{2,p,p+2;\alpha}^*$ holds for all feasible $\rho$ if $\delta^2\ge (p+2)^{-1}\Lambda_{2,p,p+2;\alpha}^*$. Also, if we set $\nu_p=\frac{2}{p}\ (\le\frac{2}{3})$ then (ref) is equivalent to each of the inequalities

Because $(p-1)U_{p;\alpha}>1$ for common (small) values of $\alpha$ (see (ref) and Table 2), in such cases (ref) in turn is equivalent to

Table 2 shows that in Case 1, $\~\psi_{p;\alpha}^-$ is close to the lower limit of the feasible range $(-\frac{1}{p-1},1)$ for $\rho$. Thus by (ref), if $\delta^2\ge (p+2)^{-1}\Lambda_{2,p,p+2;\alpha}^*$ then at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall (p-variate) size-$\alpha$ $T^2$-test for most of the region in the alternative hypothesis specified in Case 1. \vskip4pt

{\it Case 2: $\gamma_i=\gamma_j=:\delta$ for some $\{i,j\}\subset\{1,\dots,p\}$, $\gamma_k=0$ for $k\ne i,j$, and $R$ has the intraclass form $R_\rho$ in (ref).} By (ref) and (ref),

so again $\Lambda^{(2)}\ge\Lambda_{2,p,p+2;\alpha}^*$ holds for all feasible $\rho$ if $\delta^2\ge (p+2)^{-1}\Lambda_{2,p,p+2;\alpha}^*$. Also, abbreviating $Q_{2,p,p+2;\alpha}$ by $Q$, (ref) is equivalent to each of the inequalities

Since $h_{p;\alpha}(0)=Q-1<0$ for common (small) values of $\alpha$ (see (ref)-(ref) and Table 2), $h_{p;\alpha}(\rho)$ has two real roots $\~\rho_{p;\alpha}^-<0<\~\rho_{p;\alpha}^+$ (found numerically). Therefore $0>h_{p;\alpha}(\rho)$ for $\~\rho_{p;\alpha}^-<\rho<\~\rho_{p;\alpha}^+$.

Table 2 shows that in Case 2, the interval $(\~\rho_{p;\alpha}^-,\~\rho_{p;\alpha}^+)$ covers almost all of the feasible range $(-\frac{1}{p-1},1)$ for $\rho$. Thus if $\delta^2\ge (p+2)^{-1}\Lambda_{2,p,p+2;\alpha}^*$ then at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall p-variate size-$\alpha$ $T^2$-test for most of the region in the alternative hypothesis specified by Case 2.

\vskip4pt

{\it Case 3: $\gamma_i=\delta$ and $\gamma_j=-\delta$ for some $\{i,j\}\subset\{1,\dots,p\}$, $\gamma_k=0$ for $k\ne i,j$, and $R$ has the intraclass form $R_\rho$.} By (ref) and (ref),

Thus $\Lambda^{(2)}\ge\Lambda_{2,p,p+2;\alpha}^*$ again holds for all feasible $\rho$ if $\delta^2\ge (p+2)^{-1}\Lambda_{2,p,p+2;\alpha}^*$, while (ref) is equivalent to $1>Q_{2,p,p+2;\alpha}$, which holds for most $p,\alpha$ (see (ref)-(ref) and Table 2). Therefore if $\delta^2\ge (p+2)^{-1}\Lambda_{2,p,p+2;\alpha}^*$ then at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall p-variate size-$\alpha$ $T^2$-test for the {\it entire} region in the alternative hypothesis covered by Case 3. \vskip4pt

{\it Case 4: $p=:2l$ is even, $\gamma_i=\delta$ for $l$ indices in $\{1,\dots,p\}$, $\gamma_i=-\delta$ for the remaining $l$ indices, and $R$ has the intraclass form $R_\rho$.} By (ref) and (ref),

Thus $\Lambda^{(2)}\ge\Lambda_{2,p,p+2;\alpha}^*$ again holds for all feasible $\rho$ if $\delta^2\ge (p+2)^{-1}\Lambda_{2-2,p+2;\alpha}^*$, while (ref) is equivalent to the inequality

Because $\frac{1-|\rho|}{1-\rho}\le1$, while $U_{p;\alpha}>1$ for holds for most $p,\alpha$ (see (ref) and Table 2), we see that if $\delta^2\ge (p+2)^{-1}\Lambda_{2,p,p+2;\alpha}^*$ then at least one of the bivariate size-$\alpha$ $T^2$-tests is more powerful than the overall p-variate size-$\alpha$ $T^2$-test over the {\it entire} region in the alternative hypothesis in Case 4.$\square$ \vskip4pt

{\bf 3. Some local power comparisons.} From (ref)-(ref) and (ref), as $\Lambda_\omega\downarrow0$ the power function $\pi_\alpha(\Lambda_\omega)\equiv\pi_\alpha(\Lambda_\omega;|\omega|,N-|\omega|)$ of the $T_\omega^2$-test satisfies

Thus for two subsets $\omega,\,\omega'$ s.t. $\omega\subset\omega'$, $\exists\;\Lambda_{|\omega|,|\omega'|,N;\alpha}^{**}>0$ s.t.

where, from (2.2) and (2.3) in [DGP],

Therefore power comparisons of $T_\omega^2$ and $T_{\omega'}^2$ for local alternatives\footnote{ We do not claim to know the values of $\Lambda_{|\omega|,|\omega'|,N;\alpha}^{**}$, even approximately.} require determination of the lower tail probabilities $c_{m,n;k;\alpha}$ (see (ref)), which in turn require the lower quantiles $b_{n,m;\alpha}$.

In parallel with Section 2, this is done explicitly in Examples 3.1 and 3.2. As in Examples 2.1 and 2.2 these examples begin to suggest that variable selection might be limited to very small subsets $\omega\in\Omega_p$, e.g., singletons in the bivariate Example 3.1, or pairs (plus singletons) in Example 3.2. \vskip4pt

{\bf Example 3.1.} As in Example 2.1 consider the bivariate case $p=2$. Repeat the first two paragraphs from Example 2.1 verbatim, except replace “distant alternatives" by “local alternatives". Because $\max(\Lambda_1,\Lambda_2)\le\Lambda$, it follows from (ref) and (ref) that

In the simplest case $N=3$, it follows from (ref), (ref), and (ref) that

First note that in (ref),

The quadratic function $h_{\alpha,\eta}(\rho)$ ($-1\le\rho\le1$) satisfies

It is easily seen that if $\alpha\le\frac{1}{2}$ then $Z_{1,2,3;\alpha}\ge2$, so $h_{\alpha,\eta}(0)\le0$ for all $\eta\in[-1,1]$. Therefore, if $\alpha\le\frac{1}{2}$ then $h_{\alpha,\eta}(\rho)$ must have one root in $[-1,0]$ and one root in $[0,1]$. The two roots are given by

again $\check\rho_{\alpha,-\eta}^{\pm}=-\check\rho_{\alpha,\eta}^{\mp}$. Thus, if $\alpha\le\frac{1}{2}$ and $\rho\in(\check\rho_{\alpha,\eta}^-,\,\check\rho_{\alpha,\eta}^+)$ then (ref) must hold.

To conclude that $|\^\omega_\alpha(\gamma_1,\eta,\rho)|=1$, $\gamma_1^2$ must be sufficiently small, i.e.,

Because

for fixed $\eta$, (ref) will be satisfied provided that

It is straightforward to show that (ref) holds for $|\eta|<1$ but not for $|\eta|=1$.

Thus, if $\alpha\le\frac{1}{2}$, $|\eta|<1$, and (ref)-(ref) are satisfied then $|\^\omega_\alpha(\gamma_1,\eta,\rho)|=1$, in which case at least one of the two univariate $t^2$-tests are more powerful than the overall (bivariate) $T^2$-test. This occurs in the $(\eta,\rho)$-regions of the parameter space indicated in Table 3, provided that $\gamma_1^2\textstyle<\frac{1}{6}(1-\check m_{\alpha,\eta})\Lambda_{1,2,3;\alpha}^{**}$. Thus, for $p=2$, $N=3$, sufficiently small $\gamma_1^2$, and the common (small) values of $\alpha$, the overall (bivariate) size-$\alpha$ $T^2$-test is dominated by at least one of the two univariate size-$\alpha$ $t^2$-tests over a large portion of the alternative hypothesis.$\square$ \vskip4pt

{\bf Example 3.2.} Suppose that $p\ge3$ and $N=p+2\ge3$. We shall compare the powers of the ${p\choose2}$ bivariate size-$\alpha$ $T^2$-tests and the overall $p$-variate size-$\alpha$ $T^2$-test for local alternatives, which requires comparison of the powers

From (ref),

Therefore for sufficiently small values of $\Lambda$, namely $\Lambda\le\Lambda_{2,p,p+2;\alpha}^{**}$, at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall $p$-variate size-$\alpha$ $T^2$-test\footnote{ As in Example 2.2, this does not establish that $|\^\omega_\alpha(\gamma,R)|=2$.} whenever

From (ref), (ref)-(ref), (ref), and some algebra, the explicit expression

is obtained. Setting $\nu_p=\frac{2}{p}\ (\le\frac{2}{3})$ and $V_{p;\alpha}:=\nu_p Z_{2,p,p+2;\alpha}$, we have

Table 4 shows that $Z_{2,p,p+2;\alpha}$ increases rapidly to $\infty$ as $p\to\infty$, which suggests that (ref) might hold over substantial regions of the alternative hypothesis. Several such regions are now exhibited. \vskip4pt

{\it Case 1: $\gamma_1=\cdots=\gamma_p=:\delta$ and $R$ has the intraclass form (ref).} Here $-\frac{1}{p-1}<\rho<1$ and as in (ref),

Here (ref) is equivalent to each of the inequalities

Because $(p-1)V_{p;\alpha}>1$ for common (small) values of $\alpha$ (see (ref) and Table 4), in such cases (ref) in turn is equivalent to

To conclude that (ref)-(ref) holds, $\delta^2$ must be sufficiently small, i.e.,

However, $\rho>\breve\psi_{p;\alpha}^-$ implies that

Therefore (ref) will be satisfied provided that

If $p$ is large and $\alpha$ is small, Table 4 shows that in Case 1, $\breve\psi_{p;\alpha}^-$ is close to the lower limit of the feasible range $(-\frac{1}{p-1},1)$ for $\rho$. Here, by (ref), if $\delta^2\textstyle<\breve m_{p,\alpha}\Lambda_{2,p,p+2;\alpha}^{**}$ then at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall ($p$-variate) size-$\alpha$ $T^2$-test for most of the region in the alternative hypothesis covered by Case 1.

\vskip4pt

{\it Case 2: $\gamma_i=\gamma_j=:\delta$ for some $\{i,j\}\subset\{1,\dots,p\}$, $\gamma_k=0$ for $k\ne i,j$, and $R$ has the intraclass form $R_\rho$ in (ref).} As in (ref),

Abbreviating $Z_{2,p,p+2;\alpha}$ by $Z$, (ref) is equivalent to each of the inequalities

Since $h_{p;\alpha}(0)=1-Z<0$ (cf. (ref)), $h_{p;\alpha}(\rho)$ has real roots $\breve\rho_{p;\alpha}^-<0<\breve\rho_{p;\alpha}^+$ (found numerically). Therefore $0>h_{p;\alpha}(\rho)$ for $\breve\rho_{p;\alpha}^-<\rho<\breve\rho_{p;\alpha}^+$.

To conclude that (ref)-(ref) holds, $\delta^2$ must be sufficiently small, i.e.,

Because $\frac{(1-\rho)[1+(p-1)\rho]}{1+(p-3)\rho}$ is decreasing in $\rho$, $\rho<\breve\rho_{p;\alpha}^+$ implies that

Therefore (ref) will be satisfied provided that

Table 4 shows that in Case 2, the interval $(\breve\rho_{p;\alpha}^-,\breve\rho_{p;\alpha}^+)$ covers almost all of the feasible range $(-\frac{1}{p-1},1)$ for $\rho$. Thus if $\delta^2<\breve m_{p,\alpha}'\Lambda_{2,p,p+2;\alpha}^{**}$ then at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall p-variate size-$\alpha$ $T^2$-test for most of the region in the alternative hypothesis determined by Case 2. \vskip4pt

{\it Case 3: $\gamma_i=\delta$ and $\gamma_j=-\delta$ for some $\{i,j\}\subset\{1,\dots,p\}$, $\gamma_k=0$ for $k\ne i,j$, and $R$ has the intraclass form $R_\rho$.} By (ref) and (ref),

Here (ref) is equivalent to $Z_{2,p,p+2;\alpha}>1$, which holds for all $p,\alpha$ (see (ref)).

To conclude that (ref)-(ref) holds, $\delta^2$ must be sufficiently small, i.e.,

for all $\rho\in(-\frac{1}{p-1},1)$. This requires that $\rho$ be bounded below 1, that is, $\rho<1-\epsilon$ for some $\epsilon>0$, whence (ref) will be satisfied if

Therefore if (ref) holds then at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall $p$-variate size-$\alpha$ $T^2$-test if $\rho<1-\epsilon$, which covers almost all of the region in the alternative hypothesis determined by Case 3. \vskip4pt

{\it Case 4: $p=:2l$ is even, $\gamma_i=\delta$ for $l$ indices in $\{1,\dots,p\}$, $\gamma_i=-\delta$ for the remaining $l$ indices, and $R$ has the intraclass form $R_\rho$.} By (ref) and (ref),

Here (ref) is equivalent to the inequality

Because $\frac{1-|\rho|}{1-\rho}\le1$, while $V_{p;\alpha}>1$ for holds for most $p,\alpha$ (see (ref) and Table 4), (ref) is satisfied for most $p,\alpha$.

To conclude that (ref)-(ref) holds, $\delta^2$ must be sufficiently small, i.e.,

for all $\rho\in(-\frac{1}{p-1},1)$. This again requires that $\rho$ be bounded below 1, that is, $\rho<1-\epsilon$ for some $\epsilon>0$, whence (ref) will be satisfied if

Therefore if (ref) holds then at least one of the bivariate size-$\alpha$ $T^2$-tests will be more powerful than the overall $p$-variate size-$\alpha$ $T^2$-test if $\rho<1-\epsilon$, which covers almost all of the region in the alternative hypothesis determined by Case 4.$\square$ \vskip4pt

\vskip6pt

{\bf 4. Some exact power comparisons for the bivariate case.} The results in Sections 2 and 3 compare the power of the overall $T^2$-test with those of univariate or bivariate $T^2$-tests based on the original variates. However these power comparisons are asymptotic or local, relevant only for noncentrality parameters $\Lambda$ that approach $\infty$ or 0. In this section we consider the bivariate case $p=2$ and attempt to compare the exact power functions of the $T^2$-test and the two univariate $t^2$-tests for all values of $\Lambda$. Two conjectures are presented; the first of these is confirmed in Proposition 4.3 and applied in Example 4.4 for only two simple cases. \vskip4pt

{\bf Conjecture 4.1 (weak).} Suppose that $p=2$ and $N$ is odd: $N=2l+1$. Then for each $\lambda>0$, $\exists\, 0<\alpha_l^*(\lambda)<1$ such that

with equality when $\alpha=\alpha_l^*(\lambda)$.$\square$ \vskip4pt

Conjecture 4.1 is established below for $l=1,2$ and we expect it to hold for all $l\ge3$ as well. However, it is unsatisfactory in that if $\alpha_l^*(\lambda)$ depends nontrivially on $\lambda$ then we cannot conclude that, at least for small $\alpha$, one or both of the two univariate size-$\alpha$ $t^2$-tests dominate the bivariate size-$\alpha$ $T^2$ test in a large region of the alternative hypothesis. For this the following stronger result would be needed. \vskip4pt

{\bf Conjecture 4.2 (strong).} Conjecture 4.1 holds with $\alpha_l^*(\lambda)$ not depending on $\lambda$, i.e., $\alpha_l^*(\lambda)=\alpha_l^*$.$\square$ \vskip4pt

At this time we do not have evidence either for or against Conjecture 4.2. If valid, it would be essential to determine or approximate the values of $\alpha_l^*$. \vskip4pt

{\bf Proposition 4.3.} Conjecture 4.1 is valid for $l=1$ and 2. \vskip4pt

{\bf Proof.} By (ref),

if and only if

From (ref) and (ref) we find that

Set $u=1-b$ in (ref)-(ref), then differentiate w.r.to $\alpha$ to obtain

Next,

Therefore a sufficient condition that $\frac{d}{d\alpha}s_{\delta,\lambda}^{(l)}(\alpha)> \frac{d}{d\alpha}t_{\delta,\lambda}^{(l)}(\alpha)$ is that for all $k\ge0$,

with strict inequality for at least one $k$.

Thus for $\alpha=0$, a sufficient condition that $\frac{d}{d\alpha}s_{\delta,\lambda}^{(l)}(\alpha=0)> \frac{d}{d\alpha}t_{\delta,\lambda}^{(l)}(\alpha=0)$ is that for all $k\ge0$,

with strict inequality for at least one $k$. After some algebra, (ref) can be written equivalently as

where $R_{k,\delta}\sim{\rm Binomial}(k,\frac{1}{1+\delta})$. Because $R_{k,\delta}$ is (strictly) stochastically decreasing in $\delta$ (for $k\ge1$) while $\frac{\Gamma(l+\f12+R_{k,\delta})} {\Gamma(\f12+R_{k,\delta})}$ is (strictly) increasing in $R_{k,\delta}$, the left side of (ref) $\equiv$ (ref) is (strictly) decreasing in $\delta$ (for $k\ge1$).

For $k=0$ both sides of (ref) $=\frac{\Gamma(l+\frac{1}{2})}{\Gamma(\frac{1}{2})}$. For $k=1$, (ref) is equivalent to the inequality

which is equivalent to $\delta\le\frac{1}{2l-1}$. Therefore the sufficient condition (ref) for

will be satisfied for all $\delta\le\frac{1}{2l-1}$ if (ref) holds for $\delta=\frac{1}{2l-1}$ for all $k\ge2$, with strict inequality for at least one $k\ge2$.

Because $s_{\delta,\lambda}^{(l)}(0)=t_{\delta,\lambda}^{(l)}(0)=0$, it follows from (ref)-(ref) that (ref), with strict inequality for some $k\ge2$, is a sufficient condition that for each $\lambda>0$, $\exists\, 0<\alpha_l^*(\lambda)<1$ such that

with equality when $\alpha=\alpha_l^*(\lambda)$.

For the simplest case $l=1$ ($N=3$), (ref) with $\delta=\frac{1}{2l-1}=1$ becomes

which by (ref) can be reduced to the equivalent form

It is straightforward to verify (ref) by induction on $k$, with strict inequality holding for large $k$ because $\frac{\Gamma(\frac{3}{2}+k)}{\Gamma(1+k)}=O(k^{\f12})$. Therefore we conclude that (ref) holds for $l=1$; that is, for each $\lambda>0$, $\exists\, 0<\alpha_1^*(\lambda)<1$ such that

with equality when $\alpha=\alpha_1^*(\lambda)$. \vskip4pt

Next consider the case $l=2$ ($N=5$). With $\delta=\frac{1}{2l-1}=\frac{1}{3}$, (ref) becomes

which can be reduced to the equivalent form

Interestingly, (ref) holds with equality for $k=2$ as well as for 0 and 1. Rewrite (ref) in the equivalent form

To verify (ref) by induction on $k$, it suffices to show that for $k\ge2$,

After simplification, this is equivalent to the inequality

which holds for all $k\ge1$, with strict inequality for $k+1\ge3$. Therefore we conclude that (ref) holds for $l=2$: for each $\lambda>0$, $\exists\, 0<\alpha_2^*(\lambda)<1$ such that

with equality when $\alpha=\alpha_2^*(\lambda)$. $\square$ \vskip6pt

{\bf Example 4.4.} Return to the bivariate Example 2.1, where $p=2$ and

(recall (ref)). For $N=3$ it follows from (ref) and (ref) that for each $\gamma_1^2>0$,

Furthermore,

The two roots of $h_\eta(\rho)$ are $\textstyle\^\rho_\eta^\pm=\frac{\eta\pm\sqrt{12-3\eta^2}}{4}$; note that $\^\rho_{-\eta}^{\pm}=-\^\rho_{\eta}^{\mp}$. Some values appear in Table 5. Thus, if $\alpha<\alpha_1^*(3\gamma_1^2)$ then

that is, at least one of the two univariate $t^2$-tests is more powerful than the overall bivariate $T^2$-test. This occurs in the $(\eta,\rho)$-regions of the parameter space indicated in Table 5, which constitute a substantial part of the alternative hypothesis.

Similarly, for $N=5$ it follows from (ref) and (ref) that for each $\gamma_1^2>0$,

Furthermore,

The two roots of $\~h_\eta(\rho)$ are $\textstyle\~\rho_\eta^\pm=\frac{3\eta\pm\sqrt{40-15\eta^2}}{8}$; note that $\~\rho_{-\eta}^{\pm}=-\~\rho_{\eta}^{\mp}$. Some values appear in Table 5. Thus, if $\alpha<\alpha_2^*(5\gamma_1^2)$ then

that is, at least one of the two univariate $t^2$-tests is more powerful than the overall bivariate $T^2$-test. Again this occurs in the $(\eta,\rho)$-regions of the parameter space indicated in Table 5.

Thus for $p=2$, $N=3$ or 5, and sufficiently small $\alpha$ (but depending on $\gamma_1^2$), the overall size-$\alpha$ $T^2$-test is dominated by at least one of the two univariate size-$\alpha$ $t^2$-tests over a fairly large portion of the alternative hypothesis.$\square$

\vskip6pt

{\bf 5. Concluding remarks.} For the purpose of stimulating future research, the questions raised in this report are stated formally as follows: \vskip4pt

{\it The Oracular Variable-Selection Problem (OVSP)} is that of determining the function $\^\omega_\alpha(\gamma,R)$, as defined in (ref), and using this to determine the regions

\vskip4pt {\it The Parsimonious Variable-Selection Problem (PVSP)} asks if $A_\alpha(i)$ comprises a substantial portion of the alternative hypothesis $K$ for small values of $i$, e.g., $i=1,2$. \vskip4pt

If the answer to the PVSP is positive, then variable selection in some applied investigations can be limited to small, easily interpretable subsets of variables. \vskip6pt

{\bf Acknowledgment.} Special thanks are due to David Perlman for raising the questions addressed here and providing some supporting data. \vskip24pt

\centerline{\bf References} \vskip6pt

[A] Anderson, T. W. (2003). {\it An Introduction to Multivariate Statistical Analysis, 3rd edition,} Wiley & Sons, New York.

[DGP] Das Gupta, S. and Perlman, M. D. (1974). On the power of the noncentral $F$-test: effect of additional variates on Hotelling's $T^2$-test. {\it J.Amer. Statist. Assoc.} {\bf 69} 174-180.

[GK] Giri, N. and Kiefer, J. (1964). Local and asymptotic minimax properties of multivariate tests. {\it Ann. Math. Statist.} {\bf 35} 21-35.

[GKS] Giri, N., Kiefer, J., and Stein, C. (1963). Minimax character of Hotelling's $T^2$-test in the simplest case. {\it Ann. Math. Statist.} {\bf 34} 1524-1535.

[KS] Kiefer, J. and Schwartz, R. (1965). Admissible Bayes character of $T^2$-, $R^2$, and other fully invariant tests for classical multivariate normal testing problems. {\it Ann. Math. Statist.} {\bf 36} 747-770.

[MP] Marden, J. and Perlman, M. D. (1980). Invariant tests for means with covariates. {\it Ann. Statist.} {\bf 8} 25-63.

[S] Stein, C. (1956). The admissibility of Hotelling's $T^2$-test. {\it Ann. Math. Statist.} {\bf 27} 616-623. \vskip24pt

\centerline{\bf Appendix} \vskip4pt

{\bf A. Testing for additional information.} Variable selection for the $T^2$-test and related linear discriminant analysis was thoroughly studied in the 1970s and 1980s, an era of limited computer power, and subsequently by several authors with greater ability to consider all-subsets methods; a list of references appears below. Almost all of these studies were based on testing for additional information (= increased Mahalanobis distance), as now described.

For any two nested subsets $\omega\subset\omega'$ in $\Omega_p$, in general $\Lambda_\omega\le\Lambda_{\omega'}$. The question of whether the power of the $T_{\omega'}$-test exceeds that of the $T_{\omega}$-test for the testing problem (ref) usually was formulated as the problem of {\it testing for additional information (TAI),} namely, testing

based on a preliminary sample -- see [R] \S8c.4. This formulation was adopted by many researchers, even while citing the following result of [DGP] which implies that this standard formulation of TAI is inappropriate.

It was shown in [DGP, Theorem 2.1] that for fixed $\lambda>0$, the power function $\pi_\alpha(\lambda;m,n)$ (recall (ref)) of the non-central $f$-test is strictly decreasing in $m$ and strictly increasing in $n$.\footnote{ In line 2 of the second column on p.179 of [DGP], “conclude" should be“include". In the line following the third display in the second column on p.179, “$j$" should be“$f$". In Remark 4.1 on p.180, “increasing in $m$" should be “decreasing in $m$". In the next line, “$m\to\infty$" should be “$n\to\infty$".} Therefore for any integer $1\le q\le n-1$ there exists a unique real number

such that

Here $g_\alpha(0)=0$ and $g_\alpha(\lambda)$ is strictly increasing in $\lambda$; cf. [DGP, Theorem 3.1]. Thus the power is increased only if

Therefore [DGP, Section 4] introduced the problem of {\it testing for increased power (TIP),} namely, testing

and proposed several (approximate) tests.

This proposal was noted by subsequent authors but never implemented for variable selection, possibly because of difficulties in computing the functions $g_\alpha(\cdot)$, especially if many pairs $(\omega,\omega')$ must be considered. However, if as suggested above, variable selection might be limited to very small subsets of variables in practical applications, then replacing the TAI by the TIP might be feasible. \vskip4pt

{\bf Remark A.1.} The relation (ref) in Conjecture 4.1 can be stated equivalently in terms of $g_\alpha$:

with equality when $\alpha=\alpha_l^*(\lambda)$. Thus the relations (ref) and (ref) in Proposition 4.3 also can be stated equivalently in terms of $g_\alpha$:

with equality when $\alpha=\alpha_1^*(\lambda)$;

with equality when $\alpha=\alpha_2^*(\lambda)$. \vskip24pt

\centerline{\bf Additional References for the Appendix} \vskip6pt

[H1] Hand, D. J. (1981). {\it Discrimination and Classification,} Wiley & Sons, New York. [Chapter 6]

[H2] Hawkins, D. M. (1976). The subset problem in multivariate analysis of variance. {\it J. Royal Statist. Soc, Series B} {\bf 38} 132-139.

[JW] Jain, A. K. and Waller, W. G. (1978). On the optimal number of features in the classification of multivariate Gaussian data. {\it Pattern Recognition} {\bf 10} 103-109.

[JWT] Jiang, W., Wang, K., and Tsung, F. (2012). A variable-selection-based multivariate EWMA chart for process monitoring and diagnosis. {\it J. Quality Technology} {\bf 44} 209-230.

[McC] McCabe, G. P. Jr. (1975). Computations for variable selection in discriminant analysis. {\it Technometrics} {\bf 17} 259-263.

[McK] McKay, R. J. (1976). A graphical aid to selection of variables in two-group discriminant analysis. {\it Appl. Statist.} {\bf 27} 259-263.

[McL1] McLachlan, G. J. (1976). On the relationship between the $F$ test and the overall error rate for variable selection in two-group discriminant function. {\it Biometrics} {\bf 36} 501-510.

[McL2] McLachlan, G. J. (1980). A criterion for selecting variables for the linear discriminant function. {\it Biometrics} {\bf 32} 529-534.

[McL3] McLachlan, G. J. (1992). {\it Discriminant Analysis and Statistical Pattern Recognition,} Wiley & Sons, New York. [Chapter 12]

[Mu] Murray, G. D. (1977). A cautionary note on selection of variables in discriminant analysis. {\it Appl. Statist.} {\bf 26} 246-250.

[NT] Nobuo, S. and Takahisa, I. (2016). A variable selection method for detecting abnormality base on the $T^2$ test. {\it Comm. Statist. - Theory and Methods} {\bf 46} 501-510.

[R] Rao, C. R. (1973). {\it Linear Statistical Inference and its Applications, 2nd edition,} Wiley & Sons, New York.

[S] Schaafsma, W. (1982). In {\it Handbook of Statistics, Vol. 2,} P. R. Krishnaiah and L. N. Kanal, eds., 857-881. North Holland, Amsterdam.