Comparing distributions by multiple testing across quantiles or CDF values

Comparing distributions by multiple testing across quantiles or CDF values

\doublespacing \singlespacing

\iftoggle{BLINDED}

\doublespacing \onehalfspacing

\iftoggle{SUPPLEMENTAL}

Introduction

Increasingly, economists compare not only means, but entire distributions. This includes comparing income distributions (over two time periods, geographic areas, or demographic subpopulations) and a variety of economic outcomes in experiments and program evaluation, either comparing unknown “treated” and “untreated” distributions (i.e., two-sample inference), or comparing an unknown “treated” distribution to a known population distribution (i.e., one-sample inference). For example, in a paper garnering 308 citations to date,\footnote{Source: Google Scholar, July 23, 2017.} BitlerEtAl2006 study a Connecticut welfare reform program using quantile treatment effects on earnings and other outcomes (Section V), arguing that mean effects miss the substantial amount of economically important heterogeneity. In other recent empirical work using quantile treatment effects, DjebbariSmith2008 study the PROGRESA conditional cash transfer program in Mexico, JacksonPage2013 study the Tennessee STAR class size reduction data, and BitlerEtAl2008 study a Canadian welfare reform experiment. Our new methodology for comparing distributions complements the quantile treatment effect methods used in these papers.

To compare distributions, the most common statistical tests answer one of two questions: 1) Are the distributions identical or different? 2) Do the distributions differ at the median (or another pre-specified quantile)? Often, the question with the most economic and policy relevance is instead: 3) Across the entire distribution, at which quantiles or values do the distributions differ? For example, one may care not just whether two (sub)populations have different income distributions, but where (at which quantiles or values) they differ. In an experimental setting, the question is at which values the treatment effect is statistically significant; see (ref) for an empirical example.

We contribute to answering question (3). First, we formalize the question as multiple testing of a continuum either of CDF hypotheses at different values ($r\in\mathbb{R}$) or quantile hypotheses at different quantile indices ($\tau\in(0,1)$), which we call “quantile multiple testing.” This framework appears to be new to the literature on distributional inference. Second, we show that the Kolmogorov--Smirnov (KS) test can be interpreted as answering question (3) and that it appropriately controls the probability of having at least one false rejection, i.e., controls the familywise error rate (FWER). \footnote{There are other reasonable ways to quantify control of type I errors for multiple testing, such as the false discovery rate (FDR) of BenjaminiHochberg1995 and the $k$-FWER and false discovery proportion (FDP) of LehmannRomano2005fwer.} Third, we propose a new approach to answer question (3). Like the KS, our approach is nonparametric and appropriately controls FWER, without being conservative (like the Bonferroni method). Unlike the KS, our approach maintains “even sensitivity” (as quantified by pointwise size) across the continuum of quantile hypotheses. This addresses the long noted problem of the KS test's poor power against deviations in the tails Eicker1979. Fourth, we provide a new formula to instantly compute our method as well as related goodness-of-fit $p$-values and uniform confidence bands for an unknown CDF. Fifth, we refine our basic method with stepdown and pre-test procedures to improve power without sacrificing strong control of FWER.

Question (3) cannot necessarily be answered by methods addressing questions (1) or (2). The answer to question (1) is only “yes” or “no.” Using a method that answers question (2), if separate hypothesis tests are run at many different quantiles, each with size $\alpha$, then the well-known multiple testing problem is that the overall probability of making any false rejection (a.k.a.\ “false discovery”) is above $\alpha$; e.g., see RomanoEtAl2010. Even with two identical distributions (so all quantiles are identical), such a naive procedure may falsely reject equality for at least one quantile $30\%$ of the time even if $\alpha=5\%$. This $30\%$ overall false rejection probability (formalized later) is the FWER. Even a more sophisticated “multiple quantile treatment effect” approach like in the aforementioned BitlerEtAl2006 has drawbacks: a) the choice of quantiles is arbitrary, b) standard asymptotic approximations break down in the tails (where extremal quantile methods are required), and c) no finite-sample results are available. Our approach addresses all three of these problems.

The KS test Kolmogorov1933,Smirnov1939,Smirnov1948 is a goodness-of-fit (GOF) test that is only intended to answer question (1) above, but we show that it readily identifies a set of values at which the population and null distribution functions differ in a way that controls FWER. We call this the “KS-based” multiple testing procedure to distinguish it from the “KS test” for GOF. Using other GOF approaches like Cram\'{e}r--von Mises, Anderson--Darling, and permutation tests, identifying where two distributions differ is not possible.

The KS test, however, is known to suffer from low sensitivity (i.e., low power) in the tails. More specifically, the allocation of sensitivity is uneven: it is concentrated in the middle of the distribution, as has been discussed formally in the literature since (at least) Jaeschke1979,Eicker1979. The KS test also has an uneven distribution of sensitivity to deviations above versus below the null distribution at any point $x$ away from the median.

The KS test's uneven sensitivity can lead to obviously incorrect inferences. For example, let sample size $n=20$, with null hypothesis distribution $\mathrm{Unif}(0,1)$. Even if five of the $20$ observations exceed one million, any of which alone clearly contradicts the null hypothesis, the KS test still fails to reject at a $10\%$ level. \footnote{R code: ks.test(c(1:15/21,10\string^6+1:5),punif) results in D = 0.25, p-value = 0.1376.} This is not a small-sample issue: with $500$ of $n=\num{200000}$ observations exceeding one million, the KS still fails to reject at a $10\%$ level.\footnote{R code: n=200000;k=500;ks.test(c(1:(n-k)/(n+1),10\string^6+1:k),punif) results in D = 0.0025, p-value = 0.1641.}

For GOF testing, i.e., testing the global null $H_0 \colon F(\cdot)=F_0(\cdot)$, as well as uniform confidence bands, BujaRolke2006 appear to be the first to achieve “even sensitivity” by using the probability integral transform. The probability integral transform reduces the problem to order statistics from a standard uniform distribution, whose finite-sample distribution is known. Their one-sample uniform confidence band (from Section 5.1)\ was eventually detailed and published by AldorNoimanEtAl2013. BujaRolke2006 also construct a two-sample permutation test for equality. Although computation of our one-sample procedure is equivalent to computation of their uniform band, they do not propose any multiple testing procedure or discuss FWER, and our two-sample procedure is entirely new. Other papers have explored implications of the same probability integral transform, such as MoscovichEtAl2016 and MoscovichNadler2017 in the computer science literature, but only for GOF testing or uniform confidence bands, never quantile multiple testing.

The same probability integral transform underlies our methods. It provides finite-sample distributions while being distribution-free, and it facilitates finite-sample control of both overall FWER and pointwise type I error rates. The tradeoff is that iid sampling is required. However, the finite-sample sampling distribution (of the true CDF evaluated jointly at all order statistics) turns out to be equivalent to the finite-sample posterior distribution from the continuity-corrected Bayesian bootstrap in Banks1988: in the iid case, the uniform confidence band of AldorNoimanEtAl2013 is also a Bayesian uniform credible band. \footnote{This is shown formally in a prior version of this paper; see Proposition 10 in \url{https://faculty.missouri.edu/ kaplandm/pdfs/GK2016_dist_inf_KStype_longer.pdf}.} We are hopeful that further work will show that our iid assumption may be “relaxed” by using the Bayesian bootstrap to allow sampling weights Lo1993 and clustering CameronEtAl2008 while maintaining exact finite-sample properties in the iid case.

We provide a closed-form calibration formula that allows not only our multiple testing procedure but also the uniform confidence band and GOF $p$-values of BujaRolke2006 to be computed instantly. This formula replaces just-in-time simulations that can last (depending on sample size) minutes or even hours.

Many of our new multiple testing results rely on viewing the problem from the quantile perspective rather than the probability perspective of BujaRolke2006 and related papers. Seeing the problem as testing multiple quantiles helps us establish FWER properties and is critical for our procedures to improve power. Our strategy is to test $n$ different quantile hypotheses using the $n$ order statistics (i.e., ordered sample values). In contrast, papers like BujaRolke2006 apply the null CDF $F_0(\cdot)$ to the order statistics $X_{n:1}<\cdots<X_{n:k}<\cdots<X_{n:n}$, comparing $F_0(X_{n:k})$ to certain critical values for $k=1,\ldots,n$. When the true $F(\cdot)$ equals $F_0(\cdot)$, it is easy to analyze such a test's properties, and such is sufficient for GOF testing. However, if $F(x)=F_0(x)$ only over a proper subset of $\mathbb{R}$, then it is difficult to compute the false rejection probability: $X_{n:k}$ is random, so $F_0(X_{n:k})=F(X_{n:k})$ is true in some samples but not others. The quantile perspective avoids this difficulty: each pointwise null hypothesis concerns only one fixed population quantile value, $F^{-1}(\tau)$, which is tested with one order statistic.

This quantile multiple testing perspective facilitates procedures to improve power. It leads naturally to a stepdown procedure in the spirit of Holm1979, where if any quantile hypotheses are rejected by the initial test, then the remaining ones may be tested with a smaller critical value. Further, for one-sided testing, we propose a pre-test to determine at which quantiles the null hypothesis inequality constraint may be binding, and the pointwise test levels are recalibrated with attention restricted to this subset, similar to LintonEtAl2010, among others.

In the literature, using the probability integral transform for GOF testing dates back to Fisher1932, Pearson1933, and Neyman1937. An extension especially relevant to us is that the joint distribution of $F(X_{n:1}),\ldots,F(X_{n:n})$ for order statistics $X_{n:1}<\cdots<X_{n:n}$ is the same as that of the order statistics from a $\mathrm{Unif}(0,1)$ distribution; ScheffeTukey1945 seem to be the first to note this DavidNagaraja2003. Using a closely related sampling distribution, nonparametric (empirical) likelihood-based GOF testing and uniform confidence bands are respectively proposed by BerkJones1979 and Owen1995. However, they do not discuss multiple testing or two-sample inference, and our methods compute faster and spread sensitivity more evenly.

For multiple testing concepts like FWER and stepdown procedures, see Chapter 9 in LehmannRomano2005text, RomanoEtAl2010, and references therein.

(ref) contains results for the KS-based multiple testing procedure. (ref) describe our new methods and their properties. (ref) contain empirical examples and simulation results, respectively. (ref) contains additional methods, (ref) contains proofs, (ref) contains computational details, and (ref) contains additional simulations.

Notationally, we use $\alpha$ for FWER and $\tilde\alpha$ for pointwise type I error rate. Acronyms and abbreviations used include those for confidence interval (CI), data generating process (DGP), empirical distribution function (EDF), familywise error rate (FWER), goodness-of-fit (GOF), Kolmogorov--Smirnov (KS), multiple testing procedure (MTP), and rejection probability (RP). Random and non-random vectors are respectively typeset as, e.g., $\boldsymbol{\mathbf{X}}$ and $\boldsymbol{\mathbf{x}}$, while random and non-random scalars are typeset as $X$ and $x$, and random and non-random matrices as $\underline{\boldsymbol{\mathbf{X}}}$ and $\underline{\boldsymbol{\mathbf{x}}}$; $\operatorname{\mathds{1}}\mathopen{}\mathclose\bgroup\originalleft\{\cdot\aftergroup\egroup\originalright\}$ is the indicator function. The Dirichlet distribution with parameters $a_1,\ldots,a_K$ is written $\textrm{Dir}(a_1,\ldots,a_K)$, the beta distribution $\mathrm{Beta}(a,b)$, and the uniform distribution $\mathrm{Unif}(a,b)$; in some cases these stand for random variables following such distributions. The $\alpha$-quantile of the $\mathrm{Beta}(k,n+1-k)$ distribution is denoted by $B_{k,n}^\alpha$.

Setup

First, we define multiple testing terms following LehmannRomano2005text.

Given (ref), when we establish strong control of FWER, then weak control is directly implied. In our results, we will establish strong control of FWER over a set of distributions, similar to establishing “size control” by showing that type I error rates are controlled over a set of distributions.\footnote{We do not allow the population distribution to drift asymptotically to consider “uniformity,” but we conjecture that at least our basic methods do not suffer such issues.}

Second, we maintain the following assumptions throughout.

(ref) is applicable in many cases (such as our empirical example), but excludes settings with sampling weights or dependence. As noted in (ref), explorations of weakening this assumption through the connection with Banks1988 are in progress.

(ref) excludes discrete distributions; in such cases, our methods are conservative (like the KS test). The support is taken to be $\mathbb{R}$ for simplicity; any subset of $\mathbb{R}$ is fine. The continuity in (ref) allows the probability integral transform to be used, $F(X_i)\stackrel{iid}{\sim}\mathrm{Unif}(0,1)$. The strict monotonicity implies the CDF is invertible (without having to define the generalized inverse), so $F^{-1}(\cdot)$ is the quantile function, and $F^{-1}\bigl(F(r)\bigr)=r$ as well as $F\bigl(F^{-1}(\tau)\bigr)=\tau$.

Third, we address the following tasks. Unlike with a GOF test, which has a single global hypothesis and corresponding single decision (reject or not), (ref) each involve a continuum of pointwise hypotheses that each require a decision.

Quantile and distribution (CDF) tests are equivalent in the one-sample setting, but not two-sample. (ref) are equivalent to CDF tests: given (ref), if $F(r)>F_0(r)=\tau$, then $F^{-1}(\tau)<F_0^{-1}(\tau)$. In contrast, for (ref), the null hypothesis does not determine the value of $F_X(r)$ or $F_Y(r)$. Consequently, a method with strong control of FWER for (ref) does not necessarily have strong control of FWER for the corresponding quantile hypotheses, although it does have weak control of FWER; (ref) contains details.

Although different than the economic interpretation of quantile hypotheses, the interpretation of CDF hypotheses is still simple and meaningful. For example, imagine $F_Y(\cdot)$ is the distribution of hourly wage for individuals who did a job training program, and $F_X(\cdot)$ is the hourly wage distribution without the program, both in dollars per hour. If $H_{0r} \colon F_Y(r) \ge F_X(r)$ is rejected in favor of $F_Y(r)<F_X(r)$ for $r=15$, then the data suggest that the program increases the probability of an individual making at least \$15/hr.

KS-based multiple testing procedures

The one-sample and two-sample KS GOF tests are well known, including the simulation of finite-sample exact $p$-values. We present the corresponding “KS-based” multiple testing procedures (MTPs) and establish their strong control of FWER. Although seemingly intuitive, we are unaware of such a presentation in the literature. Last in this section, we discuss the problem of uneven sensitivity.

For the one-sample, two-sided KS-based MTP, given notation in (ref), let

for all $x\in\mathbb{R}$. Let $c_n(\alpha)$ denote the exact critical value with sample size $n$, so

It is well known that $c_n(\alpha)$ is distribution-free, depending only on $n$ and $\alpha$. Alternatively, the asymptotic $c_\infty(\alpha)$ can be used, such that \[ \operatorname{P}\biggl(\sup_{t\in[0,1]}\lvert B(t)\rvert > c_\infty(\alpha) \biggr) = \alpha \] for standard Brownian bridge $B(\cdot)$.

The KS test proper is a GOF test that rejects $H_0 \colon F(\cdot)=F_0(\cdot)$ when $\sup_{x\in\mathbb{R}}D_n^{x,0} > c_n(\alpha)$. Under $H_0$, this occurs with probability $\alpha$.

The corresponding MTP addressing (ref) is intuitive: reject $H_{0x} \colon F(x) = F_0(x)$ for any $x\in\mathbb{R}$ such that $D_n^{x,0}>c_n(\alpha)$. (To directly address (ref): if $H_{0x}$ is rejected, then $H_{0\tau}$ is rejected for $\tau=F_0(x)$.) Weak control of FWER is immediate from the GOF test's size control: when $F(\cdot)=F_0(\cdot)$, the probability of at least one pointwise rejection is equivalent to the probability of $\sup_{x\in\mathbb{R}}D_n^{x,0}>c_n(\alpha)$, which is exactly $\alpha$. Strong control of FWER is also straightforward to establish.

In the two-sample case, let

for all $r\in\mathbb{R}$. Under $H_0 \colon F_X(\cdot) = F_Y(\cdot)$, the critical value $c_{n_X,n_Y}(\alpha)$ is again distribution-free, and it converges to the same $c_\infty(\alpha)$ as for the one-sample test as $n_X,n_Y\to\infty$ at the same rate. However, in finite samples, we only have an inequality: under $H_0$,

Equality is impossible for most $\alpha$ because the distribution of $D_{n_X,n_Y}$ is discrete: it depends only on the ordering (i.e., permutation) of the $X_i$ and $Y_i$, and with finite $n_X$ and $n_Y$ the number of such orderings is finite.

The corresponding two-sample MTP addressing (ref) is intuitive: reject $H_{0r} \colon F_X(r) = F_Y(r)$ for any $r\in\mathbb{R}$ such that $D_{n_X,n_Y}^r>c_{n_X,n_Y}(\alpha)$. Weak control of FWER is again immediate from the GOF test's size control. Strong control of FWER can also be established, as in (ref). \footnote{Asymptotically, and usually not framed in terms of FWER, stronger results in more complex models exist, such as the nonparametric, uniform (over $\tau$) confidence band for the difference of two conditional quantile processes in QuYoon2015, or the “uniform inference” on the quantile treatment effect process in FirpoGalvao2015.} The key is that, given $\alpha$, $n_X$, and $n_Y$, rejection of $H_{0r}$ depends only on $\hat{F}_X(r)$ and $\hat{F}_Y(r)$, whose distributions are independent (by (ref)) and depend only on $F_X(r)$ and $F_Y(r)$. Such a property extends over multiple $r$ values jointly, too. This allows us to link the FWER with a probability under $F_X(\cdot)=F_Y(\cdot)$, which is bounded by the size of the global GOF test. (Implicitly, this was actually the one-sample argument for (ref), too.) Since this general proof structure is used later, part of the argument is given in the following lemma.

Although strong control of FWER is helpful, the KS-based MTPs suffer from uneven sensitivity to deviations from the null. One symptom of this was seen in the example in (ref), where the one-sample KS test could not reject that the population was $\mathrm{Unif}(0,1)$ even with five out of $n=20$ observations exceeding one million. More generally, the one-sample, two-sided KS test does not reject at a $10\%$ level even if $F_0(X_{20:16})=1$ or if $F_0(X_{20:5})=0$, which any reasonable test should and which our test does. This uneven sensitivity results in “low power in the tails” and is well documented in the literature. For example, Eicker1979 says that the KS is “sensitive asymptotically only in the central range given by $\{\tau:(\log\log n)^{-1} < F(\tau) < 1-(\log\log n)^{-1}\}$” (p.\ 117).

(ref) visualizes the intuition for the KS test's low sensitivity in the tails. The figure shows sample paths (realizations) of a standard Brownian bridge along with the two-sided asymptotic $10\%$ KS critical value, $1.225$. That is, $\operatorname{P}\bigl(\sup_{t\in[0,1]}\lvert B(t)\rvert > 1.225 \bigr)=0.1$, so only $10\%$ of sample paths wander above $1.225$ or below $-1.225$, leading to (asymptotic) weak control of FWER. (The figure oversamples paths exceeding the critical value to avoid a visual mess.) However, as can be seen, such excesses are much more likely to occur near the median ($t=0.5$) than in the tails. In the figure, the six paths deviating beyond the critical value do so only in $t\in[0.25,0.75]$, and the clumping of sample paths near $B(t)=0$ for $t$ near zero or one shows the difficulty of having a large deviation in the tails. This is due to the pointwise variance being highest at $t=0.5$, with $\operatorname{Var}\bigl(B(0.5)\bigr)=t(1-t)=0.25$, and lowest as $t\to0$ or $t\to1$, where the variance approaches zero.

(ref) visualizes (un)even sensitivity from a different perspective. The KS-based MTP is labeled “KS” in the legend; the “weighted KS” is a weighted version described below. “Dirichlet” is our new MTP, detailed in (ref). FWER is exactly $10\%$ for all methods shown, but sensitivity is allocated differently across the distribution. The probability of $X_{n:k}$ causing some $H_{0r} \colon F(r) = F_0(r)$ to be rejected is simulated\footnote{The simulation uses a standard uniform distribution for $F(\cdot)$, but the results are distribution-free: one could simply transform the data by $F_0(\cdot)$ and test against a standard uniform.} for $k=1,\ldots,n$. The resulting pattern is a more systematic account of the intuition in (ref): the pointwise rejection probability (RP) due to central order statistics is much higher than the RP due to extreme order statistics, and RP goes to almost zero at the sample minimum and maximum. This pattern is already clear with $n=20$ (left) and becomes more exaggerated with $n=100$ (right). Practically, this shows that although one is technically testing across the entire distribution, the KS-based MTP (implicitly) weights the middle of the distribution much more than the tails, which may not be desired. The corresponding uneven allocation of KS pointwise power is illustrated in (ref).

(ref) shows the same qualitative pattern for the two-sample KS-based MTP. The horizontal axis now just shows the value $r\in[0,1]$ for which $H_{0r} \colon F_X(r) = F_Y(r)$ is rejected. As in (ref), the pointwise RP peaks near the median and goes to zero in the tails.

If this uneven sensitivity stems from (ref) having uneven pointwise variance $t(1-t)$, then the simple solution is to divide by the standard deviation $\sqrt{t(1-t)}$ to achieve equal, unit variance at each $t$. In the one-sample KS context, this means dividing by $\sqrt{F_0(t)[1-F_0(t)]}$. AndersonDarling1952 gave exactly this solution in their Example 2 (p.\ 202), where they note it was first suggested by L.\ J.\ Savage (Footnote 2); they say, “In a certain sense, this function assigns to each point of the distribution $F(x)$ equal weights” (pp.\ 202--203).\footnote{The same paper includes a weighted Cram\'{e}r--von Mises test that is most commonly called the Anderson--Darling test.} However, they note that their results require the tails to have zero weight (pp.\ 210--211), which undermines the goal of even sensitivity. If nonetheless the weight is applied even in the tails, then the tails become overly sensitive. This is characterized by Eicker1979 as the weighted KS being “sensitive only in the moderate tails given by, e.g., $\bigl\{\tau : n^{-1}\log n < F(\tau) < \bigl((\log\log n)\log\log\log n\bigr)^{-1}\bigr\}$.” Other discussions of the unintended (bad) consequences of this weighting scheme are found in Jaeschke1979 and Lockhart1991, among others. For the corresponding MTP, the “weighted KS” line in (ref) shows that, indeed, the pointwise RP is much higher in the tails than near the median.

(ref) each show a line labeled “Dirichlet” that achieves a great degree of even sensitivity. These are the basic MTPs we propose in (ref).

One-sample Dirichlet approach

We propose methods for multiple testing of quantiles based on the probability integral transform and Dirichlet distribution, including stepdown and pre-test procedures to improve power. The Dirichlet distribution is used for GOF testing and uniform confidence bands in BujaRolke2006, but our methods, their properties, and the quantile multiple testing framework itself are novel.

The Dirichlet approach uses the same pointwise type I error rate, $\tilde\alpha$, for multiple quantile tests across the distribution, while choosing the value of $\tilde\alpha$ to ensure strong control of finite-sample FWER at level $\alpha$.

Basic method, FWER, and computation

All of our methods use the probability integral transform. The following results are from Wilks1962, with some notational changes.

(ref) determines the finite-sample size of a single quantile test based on an order statistic. Specifically, consider the test of $H_{0\tau} \colon F^{-1}(\tau) \ge F_0^{-1}(\tau)$ that rejects when $X_{n:k}<F_0^{-1}(\tau)$ for some $k$. Under $H_0$, the type I error rate is bounded (tightly) by

i.e., the $\mathrm{Beta}(k,n+1-k)$ CDF evaluated at $\tau$. This CDF can be computed immediately by any modern statistical software. The only difficulty is if a specific $\alpha$ is desired for a specific $\tau$, in which case one cannot find an exact, non-randomized test BeranHall1993,GoldmanKaplan2017a.

Here, instead of testing a specific $\tau$, we presume that the econometrician desires to test a wide range of quantiles. By choosing $\tau$ values that allow exact testing using order statistics $X_{n:k}$, we can get finite-sample results for a growing number ($n$) of quantiles. For comparison, GoldmanKaplan2017b provide a confidence set for a fixed number of exactly pre-specified $\tau$ (and $\alpha$), with $O(n^{-1})$ coverage probability error.

Our strategy is to use each of the $n$ order statistics to test a different $\tau$-quantile null hypothesis. Pointwise, let $B_{k,n}^{\tilde\alpha}$ denote the $\tilde\alpha$-quantile of the $\mathrm{Beta}(k,n+1-k)$ distribution. For any $\tilde\alpha\in(0,0.5)$, let the tested quantiles be

so $\operatorname{P}\bigl(X_{n:k}<F^{-1}(\ell_k)\bigr)=\tilde\alpha$ and $\operatorname{P}\bigl(X_{n:k}>F^{-1}(u_k)\bigr)=\tilde\alpha$, using (ref). A one-sided test of $H_{0\ell_k} \colon F^{-1}(\ell_k) \ge F_0^{-1}(\ell_k)$ that rejects when $X_{n:k}<F_0^{-1}(\ell_k)$ thus has exact size $\tilde\alpha$, and similarly for the test of $H_{0u_k} \colon F^{-1}(u_k) \le F_0^{-1}(u_k)$ that rejects when $X_{n:k}>F_0^{-1}(u_k)$. By using the same $\tilde\alpha$ across the entire distribution, we achieve even sensitivity. More precisely, we achieve the same finite-sample size $\tilde\alpha$ for the pointwise tests at the $n$ quantile indices $\ell_k$ or $u_k$, $k=1,\ldots,n$.

Moreover, using the Dirichlet distribution in (ref), one can solve for the $\tilde\alpha$ value such that

This choice of $\tilde\alpha$ achieves strong control of finite-sample FWER at level $\alpha$. To see this, let $K\equiv\{k:F^{-1}(\ell_k)\ge F_0^{-1}(\ell_k)\}$, the set of true hypotheses. Then,

A parallel argument applies to the other one-sided case with $u_k$.

To extend testing of the $n$ quantiles to the continuum of $H_{0\tau}$ for all $\tau\in(0,1)$, without affecting FWER, monotonicity of the quantile function is sufficient. If $X_{n:k}<F_0^{-1}(\ell_k)$, then $H_{0\ell_k} \colon F^{-1}(\ell_k) \ge F_0^{-1}(\ell_k)$ is rejected. If $X_{n:k}<F_0^{-1}(\tau)$ for another $\tau<\ell_k$, then $H_{0\tau} \colon F^{-1}(\tau)\ge F_0^{-1}(\tau)$ is also rejected. If we add this event, or rather its complement, into (ref), however, it disappears because \[ \{F(X_{n:k})\ge \ell_k\} \cap \{F(X_{n:k})\ge \tau\} = \{F(X_{n:k})\ge \ell_k\} \] since the event $\{F(X_{n:k})\ge \tau\}\supset \{F(X_{n:k})\ge \ell_k\}$ for any $\tau<\ell_k$.

The full one-sided and two-sided MTPs are now described, followed by their strong control of exact FWER, and finally a computational improvement.

Additionally, we contribute a new, fast approximation that can be used not only for (ref) but also to compute GOF $p$-values as well as uniform confidence bands for $F(\cdot)$. Solving (ref) requires approximating the $n$-variate Dirichlet distribution by either numerical integration or simulation (i.e., drawing standard uniform order statistics), which can be slow for large $n$. Extensive simulations have revealed a closed-form formula to approximate the necessary $\tilde\alpha$ as a function of $\alpha$ and $n$ with a high degree of accuracy. Computation now takes only a couple seconds even for $n=\num{100000}$.

For one-sided multiple testing, to apply (ref), the initial $\alpha$ can be adjusted using Theorem 5.1 of MoscovichEtAl2016, which is asymptotically exact and slightly conservative in finite samples. Specifically, the two-sided FWER $\alpha_2$ and one-sided FWER $\alpha_1$ (either lower or upper) are related by $\alpha_2 = 2\alpha_1 - \alpha_1^2$ asymptotically; in finite samples, $2\alpha_1 \ge \alpha_2 \ge 2\alpha_1 - \alpha_1^2$.

The accuracy of the (ref) formulas is shown in (ref). As stated in (ref), across all $\alpha$ and $n$, the relative FWER error never exceeds $20\%$ (and is usually close to zero), or $11\%$ when excluding $\alpha=0.001$. For example, for $\alpha=0.1$, true FWER is always between $0.089$ and $0.111$. To see more specific values, the right panel of (ref) shows a histogram of true minus nominal FWER differences. For the most commonly used $\alpha\in\{0.01,0.05,0.1\}$, these are seen to err slightly on the conservative side (i.e., true FWER is below nominal) and are often close to zero. In our code, we use a version of (ref) with coefficients specific to $\alpha$, which further increases the accuracy.

By monotonicity of the mapping $\tilde\alpha(\alpha,n)$ in $\alpha$ and $n$, additional approximation error from interpolation between the given values of $n$ and $\alpha$ is small. We conjecture that the formulas are accurate even outside the ranges given, especially in $n$; moreover, few economic applications require $n>10^6$, $\alpha<0.001$, or $\alpha>0.9$.

(ref) may also be used to quickly compute GOF $p$-values and uniform confidence bands; see BujaRolke2006, AldorNoimanEtAl2013, our code, and an earlier version of this paper for details.

(ref) also illustrates the cost of multiple testing across the entire distribution instead of focusing power on a single quantile. Instead of a single quantile test with size $\alpha$, the MTP has pointwise size $\tilde\alpha$, which may be much smaller and goes to zero as $n\to\infty$. (ref) already visualized two examples with FWER $\alpha=0.1$, showing $\tilde\alpha\approx0.01$ when $n=20$ and $\tilde\alpha\approx0.005$ when $n=100$. (ref) visualized two-sample examples. The one-sample lookup table included in the replication materials on the latter author's website shows $\tilde\alpha$ values for many more $n$ and $\alpha$.

Procedures to improve power

Stepdown procedure

Within our quantile multiple testing framework, a stepdown procedure to improve power is possible. The general stepdown strategy dates back to Holm1979; see also LehmannRomano2005text. Although the stepdown procedure strictly improves power (against false hypotheses), it does not “dominate” in the general decision-theoretic sense since FWER also increases in some cases, though never exceeding $\alpha$.

In our context, consider one-sided multiple testing with $\ell_k$. If at least one $H_{0\ell_k}$ is rejected by the basic MTP, then we proceed to test the remaining hypotheses as if the rejected ones are indeed false. In that case, we can remove such $\ell_k$ from the calibration equation (ref). Intuitively, with fewer true quantile hypotheses, we can test the remainder with greater pointwise size while maintaining the same overall FWER control. Mechanically, this is accomplished by changing which order statistic is used to test a hypothesis, to make it more likely to reject.\footnote{If instead $\tilde\alpha$ were increased, then the $\ell_k$ would change. This may be reasonable practically, but it would complicate matters and require a different strategy to prove strong control of FWER.} As in other stepdown procedures, the “trick” is that if one of the initially rejected hypotheses was in fact true, then a “familywise error” has already been made, so rejecting additional hypotheses has no effect on the FWER.

(ref) in (ref) describes a two-sided stepdown procedure.

Pre-test procedure

For one-sided multiple testing, a pre-test (actually “pre-MTP”) can improve pointwise power. It can also improve power of the corresponding global test, which in the one-sided case has first-order stochastic dominance as the null hypothesis. \footnote{Alternatively, one may test a null of non-dominance as suggested by DavidsonDuclos2013; there are also big differences between frequentist and Bayesian inference for first-order stochastic dominance, even asymptotically and with nonparametric methods, as pointed out by KaplanZhuo2017a and Zhuo2017.} Intuitively, the basic MTP for $H_{0\tau} \colon F^{-1}(\tau) \ge F_0^{-1}(\tau)$ over $\tau\in(0,1)$ must control FWER even for the most difficult $F(\cdot)$, where $F^{-1}(\tau)=F_0^{-1}(\tau)$ for all $\tau\in(0,1)$. In the GOF context, this $F^{-1}(\cdot)=F_0^{-1}(\cdot)$ is commonly called the “least favorable configuration”: it is the $F(\cdot)$ that maximizes the type I error rate. If (by pre-testing) $F(\cdot)$ can be restricted to a subset that excludes the least favorable configuration, then we can increase RPs while still controlling FWER.

Specifically, the pre-test determines at which $\tau$ the constraint $H_{0\tau} \colon F^{-1}(\tau) \ge F_0^{-1}(\tau)$ appears slack, i.e., where we can reject $F^{-1}(\tau)\le F_0^{-1}(\tau)$ in favor of $F^{-1}(\tau)>F_0^{-1}(\tau)$. Then, we recalibrate $\tilde\alpha$ using only the unrejected $H_{0\tau}$ to improve power.

Falsely inferring that the constraint is slack leads to over-rejection of the resulting MTP, so the probability of doing so should be small. This probability is the FWER of the pre-test. If $\alpha_p$ is the FWER level of the pre-test, then $\alpha_p\to0$ ensures zero asymptotic size distortion.\footnote{This idea is found in LintonEtAl2010, whose (13) has $c_N\to0$, and in DonaldHsu2016, whose (3.4) has $a_N\to-\infty$, among others.} Of course, in any finite sample, $\alpha_p>0$, so $\alpha_p$ should be tolerably small. We suggest $\alpha_p=\alpha/\ln\mathopen{}\mathclose\bgroup\originalleft[\ln\mathopen{}\mathclose\bgroup\originalleft(\max\{n,15\}\aftergroup\egroup\originalright)\aftergroup\egroup\originalright]$.

The pre-test implemented in our code is described in (ref) (and its strong control of FWER in (ref)) in (ref). The overall method (of which the pre-test is the first step) is described in (ref).

The FWER upper bound $\alpha+\alpha_p$ in (ref) is usually far from binding. It assumes that a false pre-test rejection always leads to a false rejection, whereas in reality the probability is only somewhat increased. Simulations show the FWER level to be much closer to $\alpha$ than $\alpha+\alpha_p$.

Additional modifications to improve power

\paragraph{Shape restrictions}

Shape restrictions may be imposed by making additional rejections (implied by the restrictions) after our initial MTP is run, without affecting the FWER (assuming the shape restrictions are correct). \footnote{We thank Matt Webb for this suggestion.} For example, consider a prior belief that the hypotheses $H_{0\tau}$ are false in a single, contiguous range of $\tau$ values. \footnote{This is implied by $F^{-1}(\cdot)-F_0^{-1}(\cdot)$ being quasiconcave if $H_{0\tau} \colon F^{-1}(\tau) \le F_0^{-1}(\tau)$, or quasiconvex if $\ge$.} If the initial MTP rejects the $H_{0\tau}$ in two disjoint ranges of $\tau$, then additionally rejecting all the values in between does not affect FWER if the shape restriction is correct, as seen by considering the following two possibilities. The first possibility is that the initial rejections are all correct: then, the additional rejections must also be correct. The second possibility is that at least one of the initial rejections is incorrect: then, a familywise error has already been made, so falsely rejecting additional $H_{0\tau}$ does not affect FWER. \footnote{This may raise the question whether FWER is the best measure to use, but that is left to other papers.}

The same arguments apply to imposing shape restrictions to the two-sample MTP in (ref).

\paragraph{Uneven sensitivity}

Although we have focused on even sensitivity, the Dirichlet framework can be used to direct power in other patterns, too. For example, perhaps only the $H_{0\tau}$ for $\tau\in(0,0.5)$ are of interest. \footnote{We thank a referee for this point.} Power over those $H_{0\tau}$ can be increased by ignoring $\tau\in(0.5,1)$ and determining $\tilde\alpha$ accordingly. That is, the equation (like (ref)) that determines $\tilde\alpha$ can be modified to use only the first $n/2$ order statistics. More generally, (ref) can be changed to have $\tilde{\alpha}(k)$ depend on the order statistic, $k$, instead of $\tilde{\alpha}(k)=\tilde{\alpha}$ constant for all $k$. For example, to shift power toward the lower half of the distribution without ignoring the upper half, one could set $\tilde\alpha(k)=3a$ if $k<n/2$ and $\tilde\alpha(k)=a$ otherwise. Then, a simulation can determine the value of $a$ (and thus $\tilde\alpha(k)$) that delivers correct FWER. As long as $\tilde\alpha(k)=g(k,a)$ for some chosen function $g(\cdot)$ and scalar $a$, it is easy to solve for the $a$ that yields exact FWER $\alpha$.

The downsides of such flexibility are that 1) it takes time to think about the best $\tilde\alpha(k)$, 2) it takes time to simulate the proper $\tilde\alpha(k)$ since our formula cannot be used, and 3) it is tempting to snoop around for the $\tilde\alpha(k)$ that provides the most rejections in a particular dataset. Consequently, even if even sensitivity is not exactly desired, it may be more convenient and defensible to use our methods as-is anyway.

Two-sample Dirichlet approach

Main method and results

Similar to the two-sample KS-based MTP and the two-sample GOF test in BujaRolke2006, our two-sample MTP depends only on the ordering of $X$ and $Y$ observations (see below). The difference is which orderings trigger rejections of which hypotheses. Compared to the KS-based MTP, as seen in (ref), our MTP allocates pointwise size (and thus power) more evenly across the distribution.

Our two-sample MTP differs from the two-sample GOF test in BujaRolke2006. Like our one-sample MTP, our two-sample MTP determines a particular $\tilde\alpha$ that depends only on the sample sizes and $\alpha$, after which order statistics are compared to different beta distribution quantiles to determine rejection. The BujaRolke2006 GOF test uses permutations of the observed data. Our approach has two advantages. First, computationally, our MTP's pointwise $\tilde\alpha$ may be pre-computed (given $\alpha$, $n_X$, and $n_Y$, as we have done in a large reference table), whereas permutations of observed data require just-in-time computation for each new dataset. Second, regarding FWER control, it is not clear that the MTP based on the BujaRolke2006 GOF test satisfies the assumption of (ref); it may still have strong control of FWER, but it would be more difficult to prove.

The bands $[\hat{\ell}_X(\cdot), \hat{u}_X(\cdot)]$ and $[\hat{\ell}_Y(\cdot), \hat{u}_Y(\cdot)]$ are uniform confidence bands, but they do not have $1-\alpha$ coverage probability: if they did, then the FWER level would be below $\alpha$. Instead, $\tilde\alpha$ is chosen to make the FWER level as close to $\alpha$ as possible. Plotting the bands shows the results of the MTP: for any $r$ where they do not overlap, $H_{0r}$ is rejected. (ref) in (ref) shows an empirical example of this.

As seen in (ref), whether or not there is at least one $H_{0r}$ rejected (vs.\ zero rejected) depends only on the ordering of $X$ and $Y$ values in the sample. For example, if $n_X=n_Y=2$, any sample with $X_{2:1}<X_{2:2}<Y_{2:1}<Y_{2:2}$ has the same ordering, XXYY; either all samples with that ordering reject at least one $H_{0r}$ (possibly with different $r$), or all samples accept all $H_{0r}$. Consequently, there is only a finite number of $\alpha$ (FWER level) that may be achieved exactly. Equivalently, the GOF $p$-value distribution is discrete. The same issue of a finite number of attainable $\alpha$ applies to the two-sample KS approach since it also depends on (only) the ordering. \footnote{Of course, we could propose a randomized MTP that rejects randomly for certain orderings to achieve exact $\alpha$ FWER level, but randomized tests are (appropriately) not popular in practice.}

Similar to (ref), the key to FWER control for (ref) is the choice of $\tilde\alpha$. (ref) chooses $\tilde\alpha$ such that the FWER is no greater than $\alpha$ under the global null $H_0 \colon F_X(\cdot) = F_Y(\cdot)$; i.e., weak control of FWER is ensured by construction. Computationally, we now describe two alternative ways to simulate the mapping from $\tilde\alpha$ to $\alpha$ (given $n_X$ and $n_Y$) when $F_X(\cdot)=F_Y(\cdot)$.

The first strategy for simulating $\alpha$ given $\tilde\alpha$ (and $n_X$ and $n_Y$) employs a convenient, order-preserving transformation. From the probability integral transform (and (ref)), $F_Y(Y_i)\stackrel{iid}{\sim}\mathrm{Unif}(0,1)$, and under $H_0 \colon F_X(\cdot) = F_Y(\cdot)$, then $F_Y(X_i)=F_X(X_i)\stackrel{iid}{\sim}\mathrm{Unif}(0,1)$, too. From (ref), $F_Y(\cdot)$ is order-preserving. Thus, we may simulate independent standard uniform samples of sizes $n_X$ and $n_Y$ to compute the FWER of (ref) given any $\tilde\alpha$, as suggested in (ref). Since FWER is monotonic in $\tilde\alpha$, which is a scalar, a simple numerical search finds the $\tilde\alpha$ that leads to a desired $\alpha$. These simulations may be done ahead of time to generate a reference table of $\tilde\alpha$ values, as we provide along with our code.

The second strategy for simulating $\alpha$ given $\tilde\alpha$ uses permutations. The distribution of $(X_1,\ldots,X_{n_X},Y_1,\ldots,Y_{n_Y})$ under $H_0 \colon F_X(\cdot) = F_Y(\cdot)$ is the same as that of any permutation of that vector, satisfying the “randomization hypothesis” in Definition 15.2.1\ of LehmannRomano2005text, for example. Given this, BujaRolke2006 propose a GOF test based on permutations of the observed data, implicitly following Theorem 15.2.1\ of LehmannRomano2005text. Alternatively, we follow Theorem 15.2.2\ of LehmannRomano2005text and use the fact that each of the $\binom{n_X+n_Y}{n_X}$ orderings is equally likely under $H_0$. This argument is again distribution-free, so our $\tilde\alpha$ is only a function of $\alpha$, $n_X$, and $n_Y$ and can be computed ahead of time.

The strong control of FWER in (ref) comes from the weak control of FWER (by construction) combined with (ref). For the one-sided case, as shown formally in the proofs, $F_X(\cdot)=F_Y(\cdot)$ is the least favorable configuration, so FWER is only lower for other distributions satisfying $H_0 \colon F_X(\cdot) \le F_Y(\cdot)$.

Two-sample quantile MTP: difficulties

(ref) does not have a corollary for interpreting (ref) as a quantile MTP for the hypotheses $H_{0\tau} \colon F_X^{-1}(\tau) = F_Y^{-1}(\tau)$. \footnote{We thank the referees for pushing us to determine this definitively.} In terms of the proof of (ref), the use of (ref) would not be valid: rejection of $H_{0\tau}$ depends on order statistics $X_{n_X:k}$ and $Y_{n_Y:m}$ for some $k$ and $m$, but their finite-sample distributions depend on more than just $F_X^{-1}(\tau)$ and $F_Y^{-1}(\tau)$. Specifically, from (ref), $F_X(X_{n_X:k})\sim\mathrm{Beta}(k,n_X+1-k)$, so the distribution of $X_{n_X:k}$ depends on all of $F_X^{-1}(\cdot)$ in finite samples.

More simply, a quantile version of (ref) for (ref) cannot be proved because it is false. A counterexample shows this. Let $X=0.5$ be a degenerate random variable. \footnote{Technically, this violates (ref), but $X \sim \mathrm{Unif}(0.5-e, 0.5+e)$ for arbitrarily small $e>0$ results in FWER that is arbitrarily close to the $e \to 0$ limit. The same comment applies to the mass points in the distribution of $Y$.} Let $\operatorname{P}(Y=0)=\operatorname{P}(Y=1)=0.5-e$ and $\operatorname{P}(Y=U)=2e$, for $U \sim \mathrm{Unif}(0,1)$ and small $e>0$. Thus, among the $H_{0\tau} \colon F_X^{-1}(\tau)=F_Y^{-1}(\tau)$, only the $\tau=0.5$ hypothesis is true. Let $n_X=6$ and $n_Y=12$. For $\alpha=0.05$, (ref) has $\tilde{\alpha} = \num[round-mode=figures,round-precision=3]{0.15351700}$. Thus, $B^{\tilde{\alpha}}_{n_X,n_X}>0.5$, $B^{1-\tilde{\alpha}}_{1,n_X}<0.5$, $B^{\tilde{\alpha}}_{9,n_Y}>0.5$, and $B^{1-\tilde{\alpha}}_{4,n_Y}<0.5$, so $H_{0,\tau=0.5}$ is rejected if $Y_{n_Y:9}=0$ or $Y_{n_Y:4}=1$. As $e\to0$, $\operatorname{P}(Y_{n_Y:9}=0)$ is the probability that no more than $n_Y-9=3$ observations have $Y_i=1$, which is the $\mathrm{Binomial}(n_Y,0.5)$ CDF evaluated at $n_Y-9$, which is $\num[round-mode=figures,round-precision=3]{0.07299805}$. By symmetry, $\operatorname{P}(Y_{n_Y:9}=0) = \num[round-mode=figures,round-precision=3]{0.07299805}$, too. Thus, the quantile FWER is $\num[round-mode=figures,round-precision=3]{0.1459961}$, slightly below $\tilde{\alpha}$ but well above $\alpha=0.05$. If $\alpha=0.01$, $n_X=6$, and $n_Y=11$, then $\tilde{\alpha}=\num[round-mode=figures,round-precision=3]{0.07164299}$, and similar calculations show $\mathrm{FWER}=\num[round-mode=figures,round-precision=3]{0.06542969}$, again slightly below $\tilde{\alpha}$ but well above $\alpha$.

The preceding counterexample's intuition follows. For testing $H_{0\tau} \colon F_X^{-1}(\tau)=F_Y^{-1}(\tau)$ for a single $\tau$, using a Dirichlet-based method similar to ours, GoldmanKaplan2017b show the importance of setting $\tilde{\alpha}$ based not only on $\alpha$, $n_X$, and $n_Y$, but also the ratio of the quantile function derivatives at $\tau$, $Q_X'(\tau) / Q_Y'(\tau)$. (This relates to the ratio of asymptotic variances of the corresponding sample quantiles.) They show that when this ratio equals one, $\tilde{\alpha}$ should be much larger than $\alpha$ to have near-exact size, but when the ratio approaches zero or infinity, $\tilde{\alpha}=\alpha$ is required. In (ref), $\tilde{\alpha}$ is calibrated to the case when $F_X(\cdot)=F_Y(\cdot)$ and thus implicitly $Q_X'(\tau)/Q_Y'(\tau)=1$ for all $\tau\in(0,1)$. In the counterexample, this implicit assumption is violated: $Q_X'(0.5)=0$ and $Q_Y'(0.5) \to \infty$ as $e \to 0$, so $Q_X'(0.5) / Q_Y'(0.5) = 0$. Consequently, the rejection probability is closer to $\tilde{\alpha}$ than $\alpha$. However, $\tilde{\alpha} \to 0$ as $n_X,n_Y \to \infty$, so $\tilde{\alpha}>\alpha$ only in very small samples. This provides a helpful bound on FWER with this particular DGP (and one could thus control FWER by ensuring $\tilde{\alpha} \le \alpha$), but it is unclear whether such a bound applies to more complex DGPs with multiple true $H_{0\tau}$.

Unlike strong control of FWER, weak control of FWER can be established for the quantile hypotheses. For two-sided hypotheses, as in (ref), weak control of FWER for the quantile hypotheses $H_{0\tau}$ means $\mathrm{FWER} \le \alpha$ if all $H_{0\tau}$ are true, i.e., if $F_X^{-1}(\tau)=F_Y^{-1}(\tau)$ for all $\tau\in[0,1]$, or more simply if $F_X^{-1}(\cdot)=F_Y^{-1}(\cdot)$. Since $F_X^{-1}(\cdot)=F_Y^{-1}(\cdot)$ is equivalent to $F_X(\cdot)=F_Y(\cdot)$, and since (ref) has $\mathrm{FWER} \le \alpha$ in that case (i.e., has any rejection with less than $\alpha$ probability), then the quantile interpretation would also have $\mathrm{FWER} \le \alpha$ in that case.

However, a test with only weak control of FWER is in principle no more informative than a GOF test. Consider the MTP that rejects $H_{0\tau}$ for all $\tau\in(0,1)$ whenever a level-$\alpha$ GOF test rejects, and otherwise the MTP rejects none of the $H_{0\tau}$. Under $F_X^{-1}(\cdot)=F_Y^{-1}(\cdot)$, the GOF test's rejection probability is below $\alpha$, so the MTP's familywise rejection probability (and thus FWER) is also below $\alpha$, satisfying weak control of FWER. However, if enough of the $H_{0\tau}$ are false that the GOF test rejects $80\%$ of the time (as it should), then the MTP has $80\%$ FWER since it falsely rejects all the true $H_{0\tau}$ along with the false $H_{0\tau}$. Not only does this technically violate strong control of FWER, it makes the MTP's rejection of a particular $H_{0\tau}$ uninformative in practice: we do not know if that $H_{0\tau}$ is actually false, or if it is rejected only because some other $H_{0\tau}$ is false. Our own MTP is not nearly so egregious, with only small FWER distortion even in our highly contrived example (and with proper FWER control in many other examples we tried), but we are reluctant to endorse a method that we know lacks strong control of FWER for the foregoing reason.

If quantiles are truly desired and distributional hypotheses do not suffice, then one-sample uniform confidence bands for the two quantile functions could be combined, but the result will be conservative. The two true quantile functions have probability $1-\alpha$ of both lying in their respective $\sqrt{1-\alpha}$ uniform confidence bands, so the quantile difference function $F_Y^{-1}(\cdot)-F_X^{-1}(\cdot)$ has at least $1-\alpha$ probability of lying in the “difference” of the two bands. Note that our (ref) MTP is also constructed (implicitly) using uniform confidence bands, but with lower than $1-\alpha$ coverage, whereas $\sqrt{1-\alpha}>1-\alpha$, so this approach is conservative.

Two-sample quantile MTP: methods

As detailed in (ref), as an alternative to (ref), we propose an MTP for a smaller number of quantile hypotheses. Stepdown and pre-test procedures to improve power are also described. Although performance in simulations is reasonable, these methods are not as elegant as the one-sample methods. They test fewer than $n$ quantile hypotheses, and we provide only heuristic asymptotic justification.

Extensions

In this section, we discuss how to apply our MTPs to regression discontinuity designs and to conditional distributions. It may also be possible to extend our approach to consider families of distributions (e.g., Gaussian), regression residuals, and the changes-in-changes model in AtheyImbens2006, among other possibilities.

Regression discontinuity

Our two-sample MTP may be applied to regression discontinuity (RD) designs. We show how recent results from CattaneoEtAl2015, ShenZhang2016, and CanayKamat2017 justify the use of our MTP in both sharp and fuzzy RD designs. (ref) contains an empirical example.

Notationally, let $Y_0$ and $Y_1$ denote the untreated and treated potential outcomes, respectively. Let $X$ be the running variable, with threshold $x_0$. Let $T$ be the treatment dummy, so the observed outcome is $Y = T Y_1 + (1-T) Y_0$.

Statistically, our MTP considers the family of hypotheses

or the one-sided hypotheses replacing $=$ with $\ge$ or $\le$. This $\Delta(\cdot)$ is the “reduced-form distributional effect” of ShenZhang2016.

Economically, under certain assumptions, testing (ref) corresponds to testing (local) distributional treatment effects for either sharp or fuzzy RD. For sharp RD, ShenZhang2016 provide conditions (Assumptions 1.4 and 2) under which $\Delta(\cdot)$ equals the distributional treatment effect at $X=x_0$, $F_{Y_1|X}(\cdot\mid x_0)-F_{Y_0|X}(\cdot\mid x_0)$. For fuzzy RD, they provide conditions (Assumptions 1.1--1.3) under which $\Delta(\cdot)$ has the same sign as the “local” (i.e., for compliers) distributional treatment effect at $X=x_0$.

\Citet{CattaneoEtAl2015} consider when the “local experiment” idea of RD can be taken literally in finite samples. Given our (ref), instead of using their Assumption 1 (p.\ 4), the most direct way to justify finite-sample inference is to assume there exists $\underline{x}$ such that $F_{Y|X}(\cdot \mid x)=F^{-}_{Y}(\cdot)$ is independent of $x$ over $x\in[-\underline{x},x_0)$, and similarly $F_{Y|X}(\cdot \mid x)=F^{+}_{Y}(\cdot)$ over some $x\in[x_0,\bar{x}]$. If we draw an iid sample including some observations with $X_i\in[-\underline{x},x_0)$ and some with $X_i\in[x_0,\bar{x}]$, then our (ref) is satisfied. If additionally $F^{-}_{Y}(\cdot)$ and $F^{+}_Y(\cdot)$ satisfy our (ref), then (ref) establishes the strong control of finite-sample FWER of our MTP for (ref).

\Citet{CanayKamat2017} provide weaker conditions that asymptotically justify a permutation test. Their Theorem 4.1 also justifies our MTP. Imagine ordering the sample observations by their $X_i$ values and looking at the values closest to $x_0$. For some $q$, let

denote the observed $X_i$ just to the left and right of $x_0$. Let the corresponding outcomes be $Y^{-}_{[j]}=Y_k$ if $X^{-}_{(j)}=X_k$, and similarly $Y^{+}_{[j]}=Y_k$ if $X^{+}_{(j)}=X_k$. Under assumptions on continuity and positive probability of $X$ near $x_0$, Theorem 4.1 in CanayKamat2017 states that for a fixed $q$ as $n\to\infty$, the joint distribution of the $Y^{-}_{[j]}$ and $Y^{+}_{[j]}$ is asymptotically equivalent to independent, iid samples from $\lim_{x\uparrow x_0} F_{Y|X}(\cdot \mid x)$ and $\lim_{x\downarrow x_0} F_{Y|X}(\cdot \mid x)$, respectively. These are the two distributions defining $\Delta(\cdot)$ in (ref).

We now formally state the method and asymptotic strong control of FWER.

We leave optimal selection of $q$ to future work. Smaller $q$ is better for FWER control but worse for power. For now, other proposals could be used. For example, CattaneoEtAl2015 provide a procedure to determine the largest neighborhood around $x_0$ where the local experiment assumption seems plausible, based on covariate balance.

Conditional distributions

Both our one-sample and two-sample MTPs may be applied to conditional distributions. \footnote{We thank a referee for this suggestion.} We focus on the two-sample, two-sided case here. It is often of interest to determine where two conditional distributions differ, such as income distributions conditional on different demographic and other characteristics. Under conditional independence MaCurdyEtAl2011, such a comparison identifies conditional distributional treatment effects.

Let $Y$ denote the outcome (like income), $\boldsymbol{\mathbf{X}}$ a conditioning vector (like education, age, etc.), and $T$ a binary variable (like a dummy for being male). The distributional hypotheses are

comparing the distribution of $Y$ given $(\boldsymbol{\mathbf{X}}=\boldsymbol{\mathbf{x}}_0, T=0)$ to the distribution of $Y$ given $(\boldsymbol{\mathbf{X}}=\boldsymbol{\mathbf{x}}_0, T=1)$.

If $\boldsymbol{\mathbf{X}}$ contains only discrete variables, then our MTPs can apply immediately. Assuming the sample contains some observations with $(\boldsymbol{\mathbf{X}}_i,T_i)=(\boldsymbol{\mathbf{x}}_0,0)$ and some with $(\boldsymbol{\mathbf{X}}_i,T_i)=(\boldsymbol{\mathbf{x}}_0,1)$, the MTP can use the corresponding $Y_i$ as the two respective (local) samples.

If $\boldsymbol{\mathbf{X}}$ contains any continuous variables, then $\operatorname{P}(\boldsymbol{\mathbf{X}}=\boldsymbol{\mathbf{x}})=0$ for any $\boldsymbol{\mathbf{x}}$, so we must smooth over observations with $\boldsymbol{\mathbf{X}}_i$ near (but not equal to) $\boldsymbol{\mathbf{x}}_0$. First, among observations with $T_i=0$, order the $\boldsymbol{\mathbf{X}}_i$ by their distance from $\boldsymbol{\mathbf{x}}_0$ according to some norm, $\lVert\cdot\rVert$. Specifically, let $\boldsymbol{\mathbf{X}}_{(k),T=0}$ satisfy $\lVert\boldsymbol{\mathbf{X}}_{(1),T=0}-\boldsymbol{\mathbf{x}}_0\rVert \le \lVert\boldsymbol{\mathbf{X}}_{(2),T=0}-\boldsymbol{\mathbf{x}}_0\rVert \le \cdots$, with each $\boldsymbol{\mathbf{X}}_{(k),T=0}=\boldsymbol{\mathbf{X}}_i$ for some $i$ where $T_i=0$ (with each $i$ used exactly once). Define $\boldsymbol{\mathbf{X}}_{(k),T=1}$ similarly for the observations with $T_i=1$. If the norm is chosen to make $\lVert\boldsymbol{\mathbf{x}}\rVert$ extremely large when the discrete components of $\boldsymbol{\mathbf{x}}$ are non-zero, then the smallest values of $\lVert\boldsymbol{\mathbf{X}}_i-\boldsymbol{\mathbf{x}}_0\rVert$ will have the discrete components of $\boldsymbol{\mathbf{X}}_i$ exactly equal those of $\boldsymbol{\mathbf{x}}_0$. Second, let $Y_{[k],T=0}$ and $Y_{[k],T=1}$ denote the corresponding outcomes: $Y_{[k],T=0}=Y_i$ iff $\boldsymbol{\mathbf{X}}_{(k),T=0}=\boldsymbol{\mathbf{X}}_i$, and similarly for $Y_{[k],T=1}$. Third, apply our two-sample MTP to the local samples

We leave optimal selection of $q_1$ and $q_0$ to future work; as in (ref), smaller $q$ is better for FWER but worse for power.

Theoretically, as in (ref), asymptotic justification of (ref) comes from Theorem 4.1 of CanayKamat2017. Their result implies that as $n\to\infty$, the local samples in (ref) converge in distribution to independent, iid samples from the conditional distributions of interest, $F_{Y|\boldsymbol{\mathbf{X}},T}(\cdot \mid \boldsymbol{\mathbf{x}}_0 , 0)$ and $F_{Y|\boldsymbol{\mathbf{X}},T}(\cdot \mid \boldsymbol{\mathbf{x}}_0 , 1)$.

Empirical examples

Gift wage experiment

We revisit data from GneezyList2006. Results may be replicated using code from the latter author's website. The global $p$-values are from the method proposed by BujaRolke2006, implemented in our code with our faster computation.

The experiment of GneezyList2006 pays control group individuals an advertised hourly wage and treatment group individuals an unexpectedly larger “gift” wage upon arrival. The “gift exchange” question is whether the higher wage induces higher effort in return. The experiment is run separately for library data entry and door-to-door fundraising tasks. The sample sizes are small: $10$ and $9$ for control and treatment (respectively) for the library task, and $10$ and $13$ for fundraising. With small samples, our methods' finite-sample FWER control is especially desirable.

The main finding of GneezyList2006 is that the “gift wage” treatment raises productivity significantly in the first time period but not thereafter. Complementing the original results, we examine heterogeneity in the period 1 treatment effect, testing across the productivity distribution.

(ref) shows the two-sided bands used by (ref), $[\hat{\ell}_X(\cdot),\hat{u}_X(\cdot)]$ and $[\hat{\ell}_Y(\cdot),\hat{u}_Y(\cdot)]$. Wherever the bands do not overlap, the pointwise null hypothesis $H_{0r} \colon F_X(r) = F_Y(r)$ is rejected. With such small sample sizes, discreteness precludes an exact $10\%$ FWER level, so exact $8.5\%$ (library) and $9.3\%$ (fundraising) FWER levels are used instead.

For the library task, with $8.5\%$ FWER level, our MTP does not reject equality of the treatment and control productivity CDFs at any point. However, there is almost a rejection near 56--58 books, around the upper quartile; increasing the FWER level to $14\%$ triggers rejection here. With one-sided global testing, i.e., testing first-order stochastic dominance, the Dirichlet test cannot reject that the treatment distribution dominates the control distribution ($p=0.996$), whereas it does reject at a $10\%$ level that the control distribution dominates the treatment distribution ($p=0.076$) because of the pointwise rejection near the $0.8$-quantile. In contrast, the KS test fails to reject at a $10\%$ level: its one-sided $p$-value is $0.13$.

For the fundraising task, with $9.3\%$ FWER level, (ref) shows two ranges near the lower quartile of the distributions where a zero treatment effect is rejected: 8--14 and 21--26 dollars raised. Opposite the library task, where the upper part of the distribution showed the most significant (statistically and economically) treatment effect, the gift wage treatment most affects the lower part of the fundraising distribution. Testing first-order stochastic dominance, the Dirichlet test cannot reject that the treatment dominates the control distribution ($p=1$), whereas it can reject at a $5\%$ level that the control distribution dominates the treatment distribution ($p=0.021$). The one-sided KS $p=0.034$: higher, but still below $0.05$. For two-sided testing of zero treatment effect, the Dirichlet test rejects at a $5\%$ level while the KS cannot: the Dirichlet $p=0.041$, while the KS $p=0.069$.

(ref) summarizes the results from running the Dirichlet and KS-based MTPs. In addition to the Dirichlet rejecting some $H_{0r}$ in many cases where the KS-based MTP cannot, the Dirichlet MTP rejects more $H_{0r}$ in cases where both MTPs reject at some $r$. The economic interpretation of the rejection of $H_{0r}$ for $r\in(56,58)$ for the library task (one-sided, $\alpha=0.1$) can be expressed as: the data suggest that the gift wage treatment increases the probability of an individual entering at least $57$ books.

As discussed in (ref), a prior belief that the gift wage treatment should only affect a single, contiguous range of $r$ values would lead to additional rejections in the fundraising data. The two-sided $H_{0r}$ for all $r\in(8,26)$ would be rejected, and similarly $r\in(8,26)$ or $r\in(6,27)$ for one-sided $H_{0r}$ with $\alpha=0.05$ or $\alpha=0.1$, respectively.

Regression discontinuity

Applying our MTP to a regression discontinuity (RD) design, we study the incumbency advantage in the U.S.\ Senate using the same data from CattaneoEtAl2015. \footnote{\url{https://sites.google.com/site/rdpackages/rdlocrand/r/rdlocrand_senate.csv}} The running variable $Z$ measures the Democratic candidate's margin of victory in an election, in percentage points; it is negative if the Democrat loses to the Republican. The outcome $Y$ is the vote share of the Democratic candidate in the next election for the same Senate seat, six years later. This is a sharp RD where the incumbency “treatment” dummy is simply $D=\operatorname{\mathds{1}}\mathopen{}\mathclose\bgroup\originalleft\{ Z>0 \aftergroup\egroup\originalright\}$. For other data details, see CattaneoEtAl2015.

The economic question is how incumbency (i.e., currently holding the seat) affects voting. \Citet{CattaneoEtAl2015} argue that for close elections, the results (and thus the incumbent in the next election) are as good as random. To quantify “close,” CattaneoEtAl2015 apply their window selection procedure (based on tests of covariate balance) to get a percentage Democratic margin of victory of $-0.75 \le Z \le 0.75$. We take their suggestion and apply our two-sample Dirichlet MTP ((ref)) to the “control” group outcomes $\{ Y_i : -0.75 \le Z_i < 0\}$ and the “treatment” group outcomes $\{ Y_i : 0 < Z_i < 0.75 \}$.

(ref) show the results. Like CattaneoEtAl2015, we find a very low $p$-value for the global null hypothesis that the control and treatment distributions are identical. Unlike the permutation test in CattaneoEtAl2015, our MTP assesses each pointwise $H_{0r}$. \footnote{This can be done with a permutation test only by invoking additional strong assumptions like a constant treatment effect, as in Assumption 3 of CattaneoEtAl2015. Section 2.3 of CattaneoEtAl2015 also discusses quantile treatment effect confidence intervals, but they are based on intervals for the quantiles of the marginal distributions, which makes them conservative, and they are only pointwise for individual $\tau$, not the full distribution.} With one-sided FWER level $\alpha=0.05$, our MTP rejects $H_{0r}$ for almost all $r\in[43.2, 56.6]$ (percent). Beyond just saying that incumbency affects some (unspecified) part of the vote share distribution, the MTP results show statistical significance for the beneficial incumbency effect across most of the distribution, implying at least a restricted first-order stochastic dominance relationship.

Simulations

All simulations may be replicated with code from the latter author's website. For comparison with our Dirichlet methods, we use KS-based MTPs as described in (ref). The unweighted KS-based MTP uses the KS implementation ks.test in the stats package in R R.core. For the weighted KS, asymptotic critical values from Jaeschke1979 and ChicheporticheBouchaud2012 were inaccurate,\footnote{Jaeschke1979 appropriately warns, “Since\ldots the rate of convergence\ldots is very slow, we would not encourage anyone to use the confidence intervals based on the asymptotic analysis.”} so we simulate exact critical values. However, this simulation is time-consuming, which is a practical disadvantage.

Earlier, (ref) showed simulation results on the uneven sensitivity of weighted and unweighted KS-based MTPs and the (relatively) even sensitivity of the Dirichlet MTPs, in terms of pointwise type I error rates. Intuitively, those differences translate into differences in pointwise power, as shown in (ref). The Dirichlet's more even sensitivity also achieves generally better global power than the (GOF) KS test. This is illustrated in (ref), as well as in Table 1 and Figure 8 of AldorNoimanEtAl2013, who also show a power advantage over the Anderson--Darling (i.e., weighted Cram\'er--von Mises) test for a variety of distributions. That is, there is not a tradeoff between even sensitivity and global power; the Dirichlet approach has both more even sensitivity and better global power.

In this section, we focus on our methods' strong control of FWER, the power improvements of stepdown and pre-test procedures, and the computational benefit of (ref).

FWER

(ref) shows weak control of FWER for one-sample, two-sided MTPs, i.e., it shows FWER simulated under $F(\cdot)=F_0(\cdot)$. Since all MTPs considered are distribution-free under (ref), the DGP is $X_i\stackrel{iid}{\sim}\mathrm{Unif}(0,1)$. For our (ref) (“Dirichlet”), this is exact by construction, up to the approximation error in (ref). This error is negligible in (ref). Earlier, (ref) showed simulated FWER for additional $n$ and nominal $\alpha$ when using (ref).

(ref) shows strong control of FWER for one-sample, one-sided MTPs: the basic Dirichlet test in (ref), as well as the stepdown and pre-test procedures in (ref), respectively. The null distribution $F_0$ is $\mathrm{Unif}(-1,1)$ and $H_{0\tau} \colon F^{-1}(\tau) \le F_0^{-1}(\tau)$. (ref) shows $F_0^{-1}(\cdot)$ and $F^{-1}(\cdot)$ for each row in (ref). The Dirichlet MTP in (ref) always controls FWER, but well below the required level $\alpha=0.1$ when $F(\cdot)\ne F_0(\cdot)$. The FWERs for the stepdown method and combined pre-test/stepdown method are higher but still below $\alpha=0.1$, as desired. Of course, all else equal, a higher error rate is never desired, but (as seen later) there is a corresponding gain in power, so the tradeoff may be beneficial from a minimax risk sort of perspective: a slight increase in FWER when FWER is near zero is not very costly, while improving worst-case power near zero is very beneficial.

(ref) shows weak control of FWER for two-sample, two-sided MTPs, i.e., FWER under $H_0 \colon F_X(\cdot) = F_Y(\cdot)$. Since all MTPs shown are distribution-free in this case, both samples are iid $\mathrm{Unif}(0,1)$.

(ref) shows our MTP's nearly exact FWER. The asymptotic KS-based MTP is somewhat conservative in these cases, as is the “exact” KS-based MTP. The exact and asymptotic KS can be identical due to the discreteness of the GOF $p$-value distributions (as discussed in (ref)), if the exact and asymptotic $p$-values lie on the same side of $\alpha=0.1$ for every possible data ordering (permutation). This discreteness makes the exact KS notably conservative when $n_X=n_Y=30$ and even $n_X=n_Y=100$, but the effect vanishes when reducing $n_X$ by one so that $n_X\ne n_Y$. The effect of discreteness on the Dirichlet MTP is negligible in all cases.

(ref) is the two-sample analog of (ref), showing strong control of FWER for one-sided MTPs. (ref) again visualizes the quantile functions for each row of the table, but now $F_Y=F_0$ and $F_X=F$. We compare MTPs for (ref) with $\alpha=0.05$: the basic Dirichlet MTP in (ref), the joint quantile difference MTP in (ref), the stepdown procedure in (ref), and the combined pre-test/stepdown procedure in (ref). For (ref), we forgo the adjustment of $\alpha$ for one-sided testing in favor of using our $\tilde\alpha$ reference table for faster computation.

(ref) shows strong control of FWER for all four methods. The stepdown and pre-test procedures' strong control of FWER in (ref) supports our heuristic arguments. Overall, the patterns are similar to those in (ref).

Power comparison

We now illustrate the power improvement from the stepdown and pre-test procedures. For pointwise and global power comparisons with the KS-based MTP, see (ref).

For one-sample multiple testing, (ref) compares the pointwise (by $\tau$) RPs of the same methods shown in (ref). The DGPs are the same as the rows in (ref) where $\{\tau : H_{0\tau}\textrm{ is true}\}=[0,0.5]$, visualized in the second and fourth panels in (ref). Since all methods (correctly) have RP near zero for $\tau<0.5$, only larger $\tau$ are shown. Compared with the basic Dirichlet MTP, the stepdown procedure weakly increases pointwise power, and adding the pre-test does, too. The pre-test is only helpful in the right panel where the null hypothesis constraint is slack for $\tau<0.5$.

For two-sample multiple testing, (ref) compares the pointwise (by $\tau$) RPs of the methods shown in (ref). For the basic MTP that tests $H_{0r}$ (with $r\in\mathbb{R}$) instead of $H_{0\tau}$, we plot the RP of $H_{0r}$ at $\tau=F_Y(r)=(r+1)/2$. The DGPs are the same as the rows in (ref) where $\{\tau : H_{0\tau}\textrm{ is true}\}=[0,0.5]$. The power improvements due to the stepdown and pre-test procedures are similar to (ref): modest but noticeable over a range of $\tau>0.5$. A bigger power difference is between the basic MTP and the joint quantile difference MTP (iteration $i=0$ of (ref)). These MTPs' powers differ because the latter MTP explicitly focuses on fewer quantiles, in this case only $10$, so more power can be focused on each quantile. This may be a reasonable way to improve power, especially in small samples, or if one assumes the quantile differences do not vary too quickly with $\tau$. One could further increase pointwise power by examining yet fewer quantiles, but the choice of quantiles becomes arbitrary and subject to manipulation.

Computation time

(ref) shows computation times for one-sample, two-sided methods: the Dirichlet MTP, the asymptotic KS test, and the exact KS test. Each value in the table has been averaged over at least four repetitions, using a standard desktop computer (8GB RAM, 3.2GHz processor). The time to simulate $\tilde\alpha$ (to the same degree of precision as (ref)) is also shown; this is the time saved by (ref) compared with just-in-time simulation as in BujaRolke2006. The simulation time depends on the starting value of $\tilde\alpha$ in the numerical search; we use five search iterations to be comparable to AldorNoimanEtAl2013, who report a runtime of $10$ seconds for $n=100$ (compared to $9.47$ seconds in our table).

In (ref), the asymptotic KS test runs instantly even for $n=\num{100000}$. The exact KS slows significantly around $n=\num{100000}$, requiring over $20$ seconds per test. With (ref), the Dirichlet MTP only takes a few seconds even with $n=\num{100000}$, faster than the exact KS and orders of magnitude faster than just-in-time simulation.

Empirical-based DGP

A DGP based on the “gift wage” empirical example in (ref) was constructed as follows. For both the library and fundraising tasks, using the Period 1 data, for both treatment and control groups, first a piecewise linear quantile function was interpolated between points $(\tau,F^{-1}(\tau))$ consisting of $(k/(n+1),Y_{n:k})$, $(0,0)$, and $(1,Y_{n:n}+10)$. Second, this was modified to only include integer values (as in the data) by applying the floor function $\lfloor\cdot\rfloor$ to generate a step function $F^{-1}(\cdot)$. These are the true population quantile functions for our simulations. The sample sizes are the same as in our empirical example (library: $10$ control, $9$ treatment; fundraising: $10$ control, $13$ treatment), as is the nominal one-sided FWER level $\alpha=0.1$. There were $\num{10000}$ simulation replications.

We compare five methods: “Basic” is our Dirichlet-based MTP in (ref), “KS” is the KS-based MTP as in (ref), “Joint” is a joint quantile difference test (iteration $i=0$ from (ref)), “Stepdown” is (ref), and “Pre+Step” is (ref). Basic and KS test $H_{0r} \colon F_T(r) \ge F_C(r)$ over each integer $r$ between $0$ and the maximum possible value, where subscript $T$ stands for “treatment” and $C$ for “control.” In the library task, $H_{0r}$ is true for $0 \le r \le 27$; in the fundraising task, $H_{0r}$ is true for $42 \le r \le 44$. The quantile tests evaluate $H_{0\tau}$ for $\tau\in\{0.30, 0.50, 0.70\}$ for the library task and $\tau\in\{ 0.22, 0.41, 0.59, 0.78 \}$ for fundraising; all $H_{0\tau}$ are false.

The FWER is nearly zero for both the Basic and KS MTPs: $0.002$ for the library DGP and $0.001$ for fundraising (for both methods). The FWER is close to zero because $H_{0r}$ is true for relatively small ranges of $r$. This is similar to the FWER in the last row of (ref), where it is shown how FWER only gets close to the nominal level when nearly all $H_{0r}$ are true.

The Basic MTP has the best global power against $H_0 \colon F_T(\cdot) \ge F_C(\cdot)$. That is, it has the highest probability of rejecting at least one $H_{0r}$. Next best are the joint quantile tests (which are all the same because the pre-test does not help for these DGPs and the stepdown cannot increase global power). The KS has the worst global power. Although not surprising that the Basic MTP has better global power than the KS, it is surprising that it fares better than the joint quantile tests that focus power on a smaller number of points where all $H_{0\tau}$ are false. The simple explanation may be that these few $\tau$ do not match up with the most statistically obviously false $H_{0\tau}$. So at least here, the “evenly sensitive” approach of the Basic MTP actually leads to the best global power, too.

(ref) shows pointwise rejection probability (RP), similar to (ref). The joint quantile tests generally have higher pointwise RP, although the magnitude partly depends on whether $r$ is compared with $F_T^{-1}(\tau)$ (as in (ref)) or $F_C^{-1}(\tau)$. The pre-test is useless because all $H_{0\tau}$ are false. The stepdown procedure improves pointwise RP by a few percentage points, sometimes more, sometimes less. Most notably, even though we used a nominal FWER level slightly above $10\%$ for the KS MTP and slightly below $10\%$ for the Dirichlet MTP, the Dirichlet MTP has significantly higher pointwise RP (i.e., power) than the KS MTP across a range of $r$.

Conclusion

We have considered the question, “At which quantiles or CDF values do two distributions differ?” Framed as multiple testing across the continuum of quantiles $\tau\in(0,1)$ or values $r\in\mathbb{R}$, we have shown KS-based multiple testing procedures to have strong control of FWER, for both one-sided and two-sided, one-sample and two-sample inference. Our newly proposed Dirichlet-based procedures also have strong control of finite-sample FWER, along with other advantages: more even sensitivity than KS, improved global power, stepdown and pre-test power improvements, and fast computation.

Extensions to conditional distributions and regression discontinuity have also been provided here. Future work may include extensions to other models, stepdown and pre-test procedures to enhance (ref), derivation of FWER bounds for the two-sample quantile MTP, and exploration of the connection with the continuity-corrected Bayesian bootstrap of Banks1988.

\iftoggle{SUPPLEMENTAL}{ \singlespacing

\doublespacing}

\iftoggle{SUPPLEMENTAL}{\setcounter{page}{2}}