Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
84,147 characters · 14 sections · 43 citation commands
Finite- and Large-Sample Inference for Ranks using Multinomial Data with an Application to Ranking Political Parties
KEYWORDS: Confidence sets, Multinomial Data, Multiple Testing, Polls, Ranks, Surveys
JEL classification codes: C12, C14, D31, I20, J62
\thispagestyle{empty}
Preferences over different categories are often assessed by means of data on choices. It is natural to summarize this type of data by ranking the different categories according to the share of support each one receives in the data. A prominent example is provided by surveys or polls of support for political candidates. In this case, individuals choose one political candidate or party from among those available. The resulting rankings may shape public discussion, inform campaigns, and be used as inputs into consequential decisions before the actual election. For example, in U.S. Presidential Elections, the decision about which candidates to feature in nationally televised political debates may hinge on their performance in different polls leading up to the election. Another prominent example is choice-based conjoint analysis, in which respondents select which of several options they would purchase or otherwise choose if given the option. Such analyses are regularly used in marketing analysis both to assess which product features are most valued and thereby inform decisions about which products to introduce. In both these examples, however, it is important to acknowledge the uncertainty surrounding these rankings.
Such data on choices, including polls of political attitudes, commonly feature limited sample sizes and/or categories whose true share of support is small. As explained further below, these features pose challenges to inference methods justified using large-sample arguments. In contrast, this paper considers the problem of constructing confidence sets for the rank of each category that are valid in finite samples, even when some categories are chosen with probability close to zero. We consider two types of confidence sets: marginal confidence sets for the rank of a particular category, by which we mean random sets that contain the rank of a particular category with probability no less than some pre-specified level, as well as simultaneous confidence sets for the ranks of all categories, by which we mean random sets that contain the ranks of all categories with probability no less than some pre-specified level. The former confidence sets provide a way of accounting for uncertainty when answering questions pertaining to the rank of a particular category, whereas the latter confidence sets provide a way of accounting for uncertainty when answering questions pertaining to the ranks of all categories. Our constructions are based off of testing a family of one-sided null hypotheses concerning differences in pairs of success probabilities in a way that controls the familywise error rate in finite samples. In order to do so, we exploit the multinomial structure of the data, which enables the use of a simple conditioning argument.
As a second contribution, we develop bootstrap methods for the construction of these confidence sets. Their validity is justified using large-sample arguments as in Mogstad:2024aa. However, unlike in Mogstad:2024aa's applications, the estimators of the success probabilities of different categories are necessarily dependent and the bootstrap procedure proposed in this paper explicitly accounts for this dependence. As described in more detail below, the results in Brown:2001p5787 suggest that such bootstrap methods may perform poorly when sample sizes are small and/or some categories are chosen with small probabilities. In particular, approaches that explicitly or implicitly (such as the bootstrap) rely on asymptotic normality of the estimators of the success probabilities perform poorly when the true success probability is small. In such a case, it is well known that the Poisson distribution is in fact a better approximation than the normal. In our simulations, we find not only that the bootstrap-based confidence sets (with or without studentization) may have coverage probability considerably below the desired level, but also that they may be excessively wide. In contrast, the finite-sample confidence sets have coverage probability no less than the desired nominal level and may even be considerably shorter.
We apply our inference procedures to re-examine the ranking of political parties in Australia using data from the 2019 Australian Election Survey. We find that the finite-sample (marginal and simultaneous) confidence sets are remarkably informative across the entire ranking of political parties, even in Australian territories with few survey respondents and/or with parties that are chosen by only a small share of the survey respondents. To illustrate this point further, consider one particular Australian territory, Greater Melbourne. We find that the finite-sample confidence sets are either of similar length to or substantially shorter than their bootstrap-based counterparts (with or without studentization). For instance, at conventional significance levels, the finite-sample marginal confidence set for the rank of the Green Party contains only rank 4. In contrast, the bootstrap-based marginal confidence sets (with or without studentization) contain the ranks 3 to 7, thus exhibiting substantially more uncertainty about the true rank of the Green Party. The studentized procedure leads to especially wide confidence sets for the ranks of parties that are chosen only by a small share of respondents. We find similar patterns in the eight most populous territories, while confidence sets in the remaining seven least populous territories are uninformative due to very small sample sizes. Unlike in Greater Melbourne, however, in some other territories bootstrap-based confidence sets (with or without studentization) may be slightly smaller than their finite-sample counterparts.
These findings motivate us to compare the different confidence sets in a simulation study modeled after our empirical application. The findings of this exercise can be summarized as follows. First, finite-sample marginal confidence sets have coverage probabilities no less than the desired nominal level in all simulation designs, including those with very small sample sizes and/or with parties that are chosen by only a small share of the respondents. Second, bootstrap-based confidence sets without studentiziation also have coverage probabilities no less than the nominal level, except when sample sizes are very small. In contrast, bootstrap-based confidence sets with studentization may have coverage probabilities less than the nominal level when sample sizes are small and/or the number of parties to be ranked is not too small. Third, the finite-sample confidence sets may produce considerably shorter confidence sets for the ranks compared to the bootstrap-based ones, especially when there are parties that are chosen by only a small share of respondents. However, there are also situations in which the latter methods produce shorter confidence sets than the former, so neither approach always dominates the other in terms of length of their confidence sets.
Our paper is most closely related to the aforementioned paper by Mogstad:2024aa. We emphasize that the primary contribution of our analysis relative to theirs is to show how one may exploit additional structure given by the multinomial data to construct confidence sets that enjoy finite-sample validity. Importantly, the finite sample guarantee allows for data-generating processes with success probabilities that are arbitrarily close to zero, whereas the asymptotic arguments justifying the bootstrap require these probabilities to be bounded away from zero. Second, we propose a bootstrap method that accounts for the dependence in the estimators of the multinomial success probabilities. Our paper is also related to a recent paper by Klein:2020oi, who consider the problem of constructing confidence sets analogous to those in Mogstad:2024aa. We show how a modification of their procedure can also be used to construct confidence sets that are valid in finite samples in the presence of multinomial data. In our simulations, we find that the resulting confidence sets are often of comparable length to our finite-sample confidence sets, but sometimes meaningfully larger. We refer the reader to Mogstad:2024aa for additional comparisons. Other related work includes Goldstein:1996re, who propose a different bootstrap-based confidence set to account for uncertainty in reported ranks. As explained by Hall:2009oi, Xie:2009oi and most recently by Mogstad:2024aa, however, this method performs poorly in the presence of categories that are chosen with similar frequencies (i.e., in the context of our simulations, when some parties are nearly tied). We confirm this finding in our simulations.
Our paper also draws motivation from earlier work by Brown:2001p5787, who demonstrate that conventional confidence intervals for the probability of success using binomial data may behave poorly in the sense of exhibiting undercoverage, especially when the success probability is close to zero or one, and may also behave erratically in the sense that coverage probabilities may be volatile and non-monotonic in the sample size. In our simulations, we find similar patterns concerning the coverage probabilities for the differences in pairs of success probabilities using multinomial data. For this reason, we view insistence upon finite-sample validity for our confidence sets to be especially compelling in this setting. On the other hand, we find that this poor behavior of confidence sets for the differences in the success probabilities need not translate into similar behavior for the implied confidence sets for the ranks.
We note some key differences between the problems considered in this paper and those of two recent papers in econometrics, Andrews:2018qf and gu2020invidious. In the context of the multinomial setting studied here, Andrews:2018qf develop methods for inference on the true success probability for the randomly selected category whose estimated rank is highest. In contrast, as the discussion above makes clear, we develop methods for inference on the true ranks themselves. gu2020invidious develop decision rules for selecting the most popular categories (i.e., those with the highest success probabilities), which is more closely related to a literature on subset selection (see Gupta:1979hi for a review). We show, however, how our simultaneous confidence sets may be used to create a complimentary object that we refer to as the confidence set for the $\tau$-best. For given value of $\tau$, such a confidence set is a random set that contains the identities of (all of) the categories whose rank is less than or equal to $\tau$ with probability approximately no less than some pre-specified level.
Finally, there is also a large literature about the analysis of datasets that contain observations of elicited rankings (e.g., Marden:1995aa), but this differs from our setting in which we assume only an individual's top choice is observed rather than their ranking of all available options.
The remainder of the paper is organized as follows. In Section (ref), we introduce our general setup, including a formal description of the different types of confidence sets we consider and the general testing problem involved in their constructions. Suitable tests that lead to confidence sets that are valid in finite samples are then described in (ref). The construction of confidence sets for the $\tau$-best that are valid in finite samples is briefly summarized in (ref). Section (ref) described bootstrap-based versions of these same confidence sets. In Section (ref), we apply our inference procedures to re-examine the ranking of political parties in Australia. Finally, in Section (ref), we examine the finite-sample behavior of our inference procedures via a simulation study modeled after our empirical application.
Let $j \in J \equiv \{1, \ldots, p\}$ index categories of interest, e.g., parties in an election. There are $n$ independent observations, and each observation falls in category $j$ with probability $\theta_j$. Let $X_j$ denote the observed count for category $j$ from the $n$ observations, e.g. the number of votes party $j$ receives from $n$ voters. Hence, $X\equiv (X_1,\ldots,X_p)'$ is distributed according to the multinomial distribution with parameters $n$ and $\theta\equiv (\theta_1,\ldots,\theta_p)'$.
The rank of category $j$ is defined as $$r_j \equiv 1 + \sum_{k \in J} \mathds{1}\{\theta_k > \theta_j \},$$ where $\mathds{1}\{A\}$ is equal to one if the event $A$ holds and equal to zero otherwise. Let $r\equiv (r_1,\ldots,r_p)'$. Before proceeding, it is useful to provide a simple example to illustrate the way in which ties are handled with this definition of ranks: if $\theta = (0.4,0.1,0.1,0.2,0.2)'$, then $r = (1,4,4,2,2)'$.
The primary goal is to construct confidence sets for the rank of a particular category or for the ranks of multiple categories simultaneously. Let $J_0\subseteq J$ denote the categories of interest. For a given value of $\alpha \in (0,1)$, we use data $X$ to construct (random) sets $$R_{n} \equiv \prod_{j\in J_0} R_{n,j},$$ where the (random) sets $R_{n,j}$, $j\in J_0$, are such that
If $J_0$ is a singleton, then sets $R_{n}$ satisfying (ref) are referred to as {\it marginal confidence sets for the rank of a single category}. If $J_0=J$, then sets $R_{n}$ satisfying (ref) are referred to as {\it simultaneous confidence sets for the ranks of all categories}. The remainder of the paper, however, allows $J_0$ to be any subset of $J$. In our constructions, $R_{n,j}$ are subsets of $J$ for each $j\in J_0$, allowing for the possibility that the lower endpoint is 1 or the upper endpoint is $p$ to permit both one-sided and two-sided inference.
In addition, we consider the goal of constructing confidence sets for the identities of all categories whose rank is less than or equal to a pre-specified value $\tau \in J$, i.e., for a given value of $\alpha \in (0,1)$, we construct (random) sets $R^{\tau-\rm{best}}_n$ that are subsets of $J$ and satisfy
where $$R_0^{\tau-\rm{best}} \equiv \{j \in J : r_j \leq \tau \}~.$$ Sets satisfying (ref) are referred to as {\it confidence sets for the $\tau$-best categories}.
As in Mogstad:2024aa, the construction of confidence sets for ranks can be based on tests of the hypotheses $$H_{j,k}\colon \theta_j\leq \theta_k $$ for pairs of indices $(j,k)\in J^2$. Which pairs are relevant depends on whether the desired confidence sets for the ranks indicated by $J_0$ are lower, upper or two-sided confidence bounds:
Suppose a family of tests of the hypotheses $H_{j,k}$ is given. Then, for each $j\in J_0$, let
indicate the set of hypotheses that are rejected in favor of $\theta_{j}<\theta_k$ and
the set of hypotheses that are rejected in favor of $\theta_{j}>\theta_k$. Consider the goal of constructing a two-sided marginal confidence set for the rank of a category $j_0$, i.e., $J_0=\{j_0\}$. Then, $\text{Rej}_{j_0}^{-}$ contains all categories $k\neq j_0$ whose parameter $\theta_k$ is claimed to be strictly larger than $\theta_{j_0}$. If these claims were correct, then the lower bound on the rank of category $j_0$ must be equal to the number of such categories $k$, denoted by $|\text{Rej}_{j_0}^{-}|$, plus one. Similarly, $\text{Rej}_{j_0}^{+}$ contains all categories $k\neq j_0$ whose parameter $\theta_k$ is claimed to be strictly smaller than $\theta_{j_0}$. Again, if these claims were correct, then the upper bound on the rank of category $j_0$ must be the total number of categories, $p$, minus the number of categories with smaller probability, denoted by $|\text{Rej}_{j_0}^{+}|$. Therefore, if all claims made in $\text{Rej}_{j_0}^{-}$ and $\text{Rej}_{j_0}^{+}$ are correct, the set
contains the rank of category $j_0$, $r_{j_0}$. More generally, for an arbitrary set of indices $J_0\subseteq J$, if all claims made in $\text{Rej}_{j}^{-}$ and $\text{Rej}_{j}^{+}$, $j\in J_0$, are correct, then the set
contains the ranks of all categories in $J_0$, $(r_j\colon j\in J_0)$. Of course, it cannot be guaranteed that tests of $H_{j,k}$ never falsely reject, but the probability of such mistakes can be controlled. More specifically, for the set $R_{n}$ to satisfy the coverage statement in (ref), the number of false claims must be controlled in the sense that the familywise error rate for testing $H_{j,k}$ for the relevant pairs of indices $I\subset J^2$ is no larger than $\alpha$, i.e.,
For two-sided confidence sets, the relevant set of indices $I$ is $J^{\rm two-sided}$ and, for one-sided confidence sets, it is either $J^{\rm lower}$ or $J^{\rm upper}$. The following theorem is a slight generalization (allowing for a general set $J_0$ of indices) of Theorem 3.4 in Mogstad:2024aa and summarizes the above discussion.
Instead of controlling the coverage probability in finite samples as in (ref), the confidence sets proposed in Mogstad:2024aa only asymptotically control the coverage probability. Their constructions assume the availability of bootstrap confidence sets that simultaneously cover the differences $\theta_j-\theta_k$ for all relevant pairs of indices $(j,k)$. While their paper does not explicitly show how these can be constructed when $X_1,\ldots,X_p$ are not independent (which by construction is the case with multinomial data), it is not difficult to propose an appropriate bootstrap procedure (see Appendix (ref)).
The general approach in Mogstad:2024aa does not require $X$ to follow a multinomial distribution. The purpose of the remainder of this paper is to show that with this additional distributional assumption it is possible to construct confidence sets for ranks that control the coverage probability not only asymptotically, but in finite samples.
In light of the previous discussion, for the construction of a confidence set satisfying (ref), it remains to propose a procedure for testing the family of hypotheses $H_{j,k}$, $(j,k)\in I$, that controls $FWER_I$. In this section, we propose a test of the individual hypothesis $H_{j,k}$ with nominal level $\beta_{j,k}$ and then choose the constants $(\beta_{j,k}\colon (j,k)\in I)$ in a way that controls $FWER_I$ in the sense of (ref).
Let $S_{j,k}\equiv X_j + X_k$. One can show that the conditional distribution of $X_j$ given $S_{j,k}=s$ is binomial based on $s$ trials and success probability $\theta_j/(\theta_j+\theta_k)$; a simple proof appears in the proof of Theorem (ref). Notice that $H_{j,k}$ is equivalent to $\theta_{j,k} \le 1/2$, where $\theta_{j,k} = \theta_j / ( \theta_j + \theta_k )$. Conditioning on $S_{j,k}$ eliminates nuisance parameters and reduces the testing problem to a one-parameter problem of testing a binomial probability. An exact level $\beta_{j,k}$ test may be therefore be easily constructed. In particular, the (possibly randomized) test of $H_{j,k}$ defined by the critical function
with constants $\gamma(s)$, $C(s)$ determined by
is an exact level $\beta_{j,k}$ test of $H_{j,k}$. The test has rejection probability equal to $\beta_{j,k}$ when $\theta_{j,k} = 1/2$. Moreover, since the binomial family of distributions has monotone likelihood ratio, the test has rejection probability strictly less than $\beta_{j,k}$ whenever $\theta_{j,k} < 1/2$. In fact, Theorem (ref) below shows that, for testing $H_{j,k}$, the test $\phi (X_j, S_{j,k} )$ is uniformly most powerful level $\beta_{j,k}$ among all level $\beta_{j,k}$ unbiased tests based on $(X_1 , \ldots , X_p )$; i.e., it is UMPU. If $\gamma(s)>0$ and one wishes to avoid randomization of the test, then one may simply reject $H_{j,k}$ iff $X_j > C( S_{j,k} )+1$. The $p$-value for this slightly conservative approach of testing $H_{j,k}$ when $S_{j,k}=s$ can be written as
The following theorem summarizes the above discussion.
This theorem shows that $\phi$ defines a level $\beta_{j,k}$ test of $H_{j,k}$. To satisfy (ref) one could combine the individual tests, i.e., choose the $(\beta_{j,k}\colon (j,k)\in I)$, by a Bonferroni correction or by the Holm:1979xy procedure, for example. Theorem (ref) then implies that the confidence set $R_n$, based on such a procedure, has coverage probability no less than $1-\alpha$.
The steps necessary for construction of the proposed confidence sets for the ranks, using the non-randomized test with $p$-value in (ref), are summarized as follows:
Since the Holm procedure rejects at least as many hypotheses as Bonferroni (with probability one) and thus leads to confidence sets that are at least as short as those based on Bonferroni, it is to be preferred. However, in simulations (Section (ref)), we find that the two methods lead to almost identical confidence sets, and both methods are optimal in many high-dimensional settings (Lehmann:2022aa).
Theorems (ref) and (ref) imply that the resulting confidence set for the ranks is valid in finite samples:
One important aspect to note is that we have not imposed any assumptions on the vector of probabilities $\theta$, besides it being the vector of probabilities associated with a multinomial distribution for $p$ categories. In particular, the confidence set $R_n$ satisfies the coverage result (ref) regardless of whether any of the elements of $\theta$ are equal to each other (“ties”) or close to each other (“near-ties”). This is an important feature of our confidence sets for the ranks because it ensures that the coverage property does not break down when some categories are observed with equal or similar counts. In contrast, a “naive” bootstrap confidence set is valid only in the absence of ties and may substantially under-cover when there are near-ties (see Remark (ref) and the simulations in Section (ref)).
Let $R_n \equiv \prod_{j\in J_0} R_{n,j}$ be a simultaneous lower confidence bound on the ranks of all categories, i.e., each $R_{n,j}$ has upper bound equal to $p$ and $R_n$ satisfies (ref) for $J_0=J$. Then, the projection
is a confidence set for the $\tau$-best categories:
The construction of such a confidence set using the non-randomized test with $p$-values in (ref) can thus be summarized as follows:
As was previously seen, confidence sets for ranks can be based on simultaneous tests of the hypotheses $H_{j,k}$ with $(j,k) \in I$ for an appropriate set of indices $I \subset J^2$. In a similar way, inference for ranks can also be based on simultaneous bootstrap confidence sets $C_n ( 1- \alpha , I)$ for the differences $(\theta_j - \theta_k\colon (j,k)\in I)$, which we now describe.
Let $X^*\equiv(X_1^*,\ldots,X_p^*)$ denote a bootstrap draw from the the multinomial distribution with parameters $n$ and $\hat{\theta}\equiv X/n$. Define the bootstrap estimator $\hat{\theta}^*\equiv X^*/n$ and
Consider the bootstrap statistic $$T_{{\rm lower}, n}^*(I) \equiv \max_{(j,k) \in I} \frac{\hat{\theta}^*_j - \hat{\theta}^*_k - (\hat{\theta}_j-\hat{\theta}_k)}{\hat{\sigma}^*_{j,k} / \sqrt{n} }, $$ where we adopt the convention that $0/0 = 0$ and $c/0 = \text{sign}(c)\infty$ for $c\neq 0$, and denote by $c_{{\rm lower},n}(1-\alpha,I)$ the $(1-\alpha)$-quantile of $T_{{\rm lower}, n}^*(I)$ conditional on the data.\footnote{In a given bootstrap sample, the ratio inside the max of $T_{{\rm lower},n}^*$ can have a zero denominator and/or zero numerator. For instance, when two categories $j$ and $k$ both have small success frequencies in the data ($\hat{\theta}_j$ and $\hat{\theta}_k$ are small), then it is possible that a given bootstrap sample does not contain any success for either of the two categories, i.e., $\hat{\theta}_j^*=\hat{\theta}_j^*=\hat{\sigma}_{j,k}^* = 0$, and the denominator is zero. When $\hat{\theta}_j<\hat{\theta}_k$, then the resulting critical value $c_{{\rm lower},n}(1-\alpha,I)$ equals $\infty$. On the other hand, when the frequencies in the data are equal ($\hat{\theta}_j=\hat{\theta}_k$), then both the numerator and denominator of the ratio are zero and the maximum is determined by other bootstrap samples that produce a positive ratio.} We can then construct lower confidence bounds for the vector of differences $\Delta_I\equiv (\theta_j-\theta_k\colon (j,k)\in I)$ by
with $$C_{{\rm lower}, n,j,k}(1 - \alpha,I) \equiv \Biggr [ \hat{\theta}_j - \hat{\theta}_k - c_{{\rm lower}, n}(1- \alpha,I) \frac{ \hat{\sigma}_{j,k}}{\sqrt{n}}, \infty \Biggr ) $$ and $\hat \sigma_{j,k} \equiv \sqrt{\hat{\theta}_j ( 1- \hat{\theta}_j ) + \hat{\theta}_k (1- \hat{\theta}_k ) + 2 \hat{\theta}_j \hat{\theta}_k}$. As long as all $\theta_j$, $j\in J$, are nonzero, this confidence set covers the vector of true differences with probability $1-\alpha$, asymptotically as the sample size $n$ tends to infinity:
Appendix (ref) provides a formal justification of this claim and further shows that the coverage probability is no less than $1-\alpha$ when some $\theta_j=0$. Let $I$ be equal to one of the sets $J^{\rm lower}$, $J^{\rm upper}$, or $J^{\rm two-sided}$ depending on which type of confidence set for the ranks is desired. Consider the test that rejects $H_{j,k}$ iff $C_{{\rm lower},n,j,k}(1-\alpha,I)$ lies entirely above zero. Then, based on this test, form the sets $\text{Rej}_{j}^{-}$ and $\text{Rej}_{j}^{+}$ as in (ref) and (ref). The bootstrap confidence set for the ranks of categories in $J_0$ can then be constructed as in (ref); denote the resulting confidence set by $R_n^{\rm boot} \equiv \prod_{j\in J_0} R_{n,j}^{\rm boot}$. By an argument similar to that in Theorem 3.3 in Mogstad:2024aa, the probability that this confidence set covers the true ranks is bounded from below by the probability that the vector of differences, $\Delta_I$, is covered by $C_{\rm lower, n}(1 - \alpha,I)$. Therefore, the validity of the bootstrap in the sense of (ref) implies that the bootstrap confidence set for the ranks also covers the true ranks with probability at least $1-\alpha$ in the limit as $n\to \infty$. The following result formalizes this discussion and shows that the validity of the bootstrap does not require any further assumptions:
In this section, we apply the inference procedures from Sections (ref) and (ref) to examine the ranking of political parties by their share of voters' support in the Australian Election Study (AES). The AES has fielded representative surveys after every federal election since 1987 and provides the most comprehensive source of evidence on political attitudes in Australia cameronAES. We use AES data from 2019 with address-based stratified random sampling from the Geocoded National Address File (G-NAF)beanAES.\footnote{The original sampling methodology description reads: “Within the parameters outlined above, the new AES sample was selected from the G-NAF database using a stratified sample design in accordance with the geographical distribution of the Australian residential population aged 18 years and over. GNAF sample selections were supplied by the MasterSoft Group. A total of 3,944 sample records were randomly generated within 15 geographic strata (see Table 2) to ensure sufficient sample was utilised to achieve the desired number of responses for the AES"beanAES. We interpret this sampling as being i.i.d. within territories.} Table (ref) shows a total of 3,944 sampled eligible voters from all fifteen Australian territories resulting in 1,211 respondents. In the subsequent analysis we work with respondents and refer to them as “sample".
To examine which political parties are on the top and the bottom of the ranking in each Australian territory, we use respondents' answers to the survey question “Generally speaking, do you usually think of yourself as Liberal, Labor, National, Greens or other (specify)?". The answer categories include political parties; “Skipped" and “No answer" categories that we group in a single “No answer" category; “Independent", “Swing Voter" and “No party" categories that we group in a single “No party" category. For more populous territories, we observe between seven and ten categories with positive support shares.
By applying the inference procedures from Sections (ref) and (ref) we compute (i) the marginal confidence set for the rank of a particular political party and (ii) the simultaneous confidence set for the ranks of all parties. Thus, (i) is relevant if one is interested whether a particular party is on the top (bottom) of ranking by the voters' support, and (ii) is relevant if one is interested in the entire ranking of parties.
Consider first a particular territory, Greater Melbourne. Figure (ref) shows the point estimates and standard errors for voters' support share for each category with positive number of supporters. The leftmost panel in row A shows considerable variation in point estimates across categories from 0.371 for the most supported (Labor Party) to 0.004 for the least supported (National Party). Support shares on the top and the bottom of the ranking are close to each other, while the shares in the middle are better separated.
The middle panel in row A of Figure (ref) presents the 95% marginal confidence sets for the rank of each category implemented using five procedures: the exact Holm (“exactHolm") described in the Algorithm (ref), Clopper-Pearson (“CP") as in Remark (ref), non-studentized (“boot") and studentized (“bootStud") versions of the bootstrap as in Section (ref) and the “naive" bootstrap (“naive") as in Remark (ref). The first two methods have been shown to be valid in finite samples, and the studentized and non-studentized bootstrap are motivated by asymptotic validity. The “naive" bootstrap is asymptotically valid in the absence of (near-)ties (see Remark (ref)), but invalid otherwise.
The resulting confidence sets exhibit four pronounced patterns. First, the “naive" bootstrap confidence sets are the tightest. As indicated in Remark (ref), the “naive" bootstrap produces confidence sets that fail to cover the true ranks with the desired probability when there are (near-) ties. Our simulations in Section (ref) confirm that, in datasets like the one from Greater Melbourne, the “naive" bootstrap does indeed produce short confidence sets at the expense of its coverage frequency lying substantially below the desired nominal level. Second, the exact Holm procedure produces weakly shorter confidence sets than Clopper-Pearson and the studentized and non-studentized bootstraps. For example, the exact Holm confidence sets for the ranks of the categories “No party" and “Greens” contain only ranks three and four, respectively, while Clopper-Pearson and the studentized and non-studentized bootstraps produce confidence sets containing at least two ranks. Third, both the studentized and non-studentized bootstrap confidence sets are wide in the middle of the ranking. Their length for the 4th category, “Greens", is equal to four compared to the maximum possible length of six. Fourth, the studentized bootstrap produces extremely wide confidence sets at the bottom of the ranking for the last two categories. These confidence sets cover the entire ranking and correspond to infinite critical values. In contrast, the length of the finite-sample valid confidence sets for the last two categories is two. These patterns suggest that the finite-sample valid confidence sets for parties in Greater Melbourne are informative, and the exact Holm procedure is the most informative among the valid procedures (i.e., excluding the “naive" bootstrap).
As discussed in Footnote (ref), the studentized bootstrap confidence sets are wide when the ratio in the bootstrap test statistic has a denominator that is equal to zero and a positive numerator in a substantial fraction of the bootstrap samples. This circumstance arises when at least two categories have small but positive shares in the data so that, in the bootstrap samples, $\hat{\theta}_j^* = \hat{\theta}_k^* = \hat{\sigma}_{j,k}^* = 0$ while, in the data, $|\hat{\theta}_j - \hat{\theta}_k| > 0$. One solution is to group the categories with small shares together. The leftmost panel in row B of Figure (ref) shows support shares when we group “National Party", “One Nation" and “No answer" into a single category “Other". The middle panel in row B shows that in the middle or at the bottom of the ranking, both the studentized and non-studentized bootstraps no longer produce confidence sets as wide as in row A. Notably, the exact Holm confidence sets are still tighter than Clopper-Pearson confidence sets.
An alternative solution to the division by zero in bootstrap samples is to reduce the confidence level. The rightmost column of Figure (ref) shows the 90% marginal confidence sets for the ranks using both the original set of categories in row A and with the three smallest categories grouped in row B. Indeed, with lower confidence level we no longer observe studentized bootstrap confidence sets covering the entire ranking. However, both finite-sample methods still produce weakly smaller confidence sets than both types of bootstrap. Furthermore, the panel with grouped small categories in row B shows that the exact Holm confidence sets for “Greens" are less informative than for the original categories, regardless of whether the confidence level is 90% or 95%.
Next, we move beyond the example of Greater Melbourne and consider all fifteen Australian territories. Appendix Figures (ref) and (ref) show the point estimates for support shares in all territories, and Appendix Figures (ref) and (ref) show the 95% marginal confidence sets for the rank of each category in each territory computed using the same five procedures as for Greater Melbourne. Similar to our illustration for Greater Melbourne, both bootstrap procedures produce wide confidence sets in the middle and at the bottom of the ranking in the majority of populous territories. Due to small sample sizes the seven least populous territories have mostly uninformative confidence sets for all categories. Therefore, we focus our analysis on the eight most populous territories.
Figure (ref) shows that none of the valid methods produce tighter confidence sets than all other valid methods uniformly across all categories in all territories. For example, the finite-sample valid confidence sets are weakly tighter than both types of bootstrap confidence sets for each category in Greater Melbourne. In comparison, the bootstrap confidence sets are weakly tighter in the top part of the ranking in Greater Sydney. We summarize this finding in the top panel of Table (ref) that shows the percentage of category$\times$territory cases across the eight most populous territories where each method produces strictly wider 95% marginal confidence sets for the rank than other methods. For example, the first row shows that the exact Holm confidence sets are strictly wider than Clopper-Pearson confidence sets in 6.2% of category$\times$territory cases, strictly wider than the non-studentized bootstrap confidence sets in 12.3% of cases, and strictly wider than the studentized bootstrap confidence sets in 1.5% of cases. The first column shows that Clopper-Pearson confidence sets are strictly wider than the exact Holm confidence sets in 3.1% of category$\times$territory cases, the non-studentized bootstrap confidence sets are strictly wider than the exact Holm in 29.2% of cases, and the studentized bootstrap confidence sets are strictly wider than the exact Holm confidence sets in 47.7% of cases. Notably, the share of cases where the studentized and non-studentized bootstrap confidence sets are strictly wider than Clopper-Pearson or the exact Holm confidence sets is substantially larger than the share of cases where Clopper-Pearson and the exact Holm confidence sets are strictly wider than both types of bootstrap confidence sets.
The bottom panel of Table (ref) shows the same comparisons when we group all categories except “Liberal", “Labor", “Greens" and “No party" into a single category “Other". As discussed above, this grouping prevents zeroes in the bootstrap test statistic denominator and excessively wide bootstrap confidence sets. As a result, the percentage of cases in rows 3 and 4 where both types of bootstrap confidence sets are strictly wider than the finite-sample valid confidence sets decreases. Interestingly, the percentage of cases where the exact Holm confidence sets are strictly wider than Clopper-Pearson confidence sets is lower with grouping, and the percentage of cases where Clopper-Pearson confidence sets are wider than the exact Holm confidence sets is larger.
The analysis of marginal confidence sets answers questions about the rank of a particular party, but one may be interested in the ranking of all parties described by simultaneous confidence sets. Figure (ref) compares 95% marginal confidence sets for the ranks to 95% simultaneous confidence sets for the ranks produced by the exact Holm, Clopper-Pearson, the studentized and non-studentized bootstrap procedures for categories in Greater Melbourne. Naturally, simultaneous confidence sets are weakly wider than marginal confidence sets for each procedure. This feature is more pronounced for the studentized bootstrap confidence sets due to infinite critical values in the bootstrap test statistic for categories at the bottom of the ranking. In contrast, the finite-sample valid simultaneous confidence sets are still informative, and the exact Holm produces weakly tighter confidence sets than all other procedures. Appendix Figure (ref) shows that with confidence level reduced to 90% we no longer observe studentized bootstrap confidence sets as wide as on Figure (ref) since the critical value becomes finite.
The top panel of Table (ref) shows that our findings hold in the eight most populous Australian territories with the original set of choice categories. The studentized bootstrap almost always produces strictly wider 95% simultaneous confidences sets than other methods and never produces tighter confidence sets. The exact Holm simultaneous confidence sets are never strictly wider than Clopper-Pearson or the studentized bootstrap confidence sets, and are strictly wider than the non-studentized bootstrap confidence sets in only 9.2% of category$\times$territory cases. In contrast, both Clopper-Pearson and the non-studentized bootstrap simultaneous confidence sets are strictly wider than the exact Holm confidence sets in 26.2% of category$\times$territory cases.
In the bottom panel of Table (ref) we group all categories except “Liberal", “Labor", “Greens" and “No party" into a single category “Other". As a result, both types of bootstrap confidence sets become tighter. In particular, the non-studentized bootstrap simultaneous confidence sets are never strictly wider than the confidence sets produced by any other inference procedure. The exact Holm confidence sets, however, are still never wider than Clopper-Pearson or the studentized bootstrap confidence sets and are strictly wider than the non-studentized bootstrap confidence sets in only 5% of category$\times$territory cases.
In this section, we examine the finite-sample performance of the following approaches to constructing confidence sets for ranks:
All simulations are based on $1,000$ Monte Carlo samples and nominal coverage of $95\%$. Bootstrap confidence sets are based on $10,000$ bootstrap samples except in Section (ref), where for computational reasons we use $1,000$ bootstrap samples. Coverage is defined as in Remark (ref), i.e., coverage of the set of possible ranks $R_j$.
We consider three different designs, starting with one that is calibrated to the dataset on which the empirical application is based. Second, we investigate whether the confidence sets exhibit erratic behavior in coverage frequencies similar to that reported for confidence sets for binomial proportions (Brown:2001p5787). The first two designs consider only data generating processes with three or seven categories to be ranked. In the final simulation design, we analyze the behavior of the confidence sets as the number of categories increases.
The simulation design in this subsection is calibrated to the AES data for Greater Melbourne as in Section (ref). The estimated vector of success probabilities is $$\hat{\theta}_{AES}=(0.372, 0.321, 0.179, 0.090, 0.026, 0.009, 0.004)$$ and the number of respondents is $n_{AES}=234$. The vector of success probabilities employed in the simulations, $\theta$, is parametrized as $$\theta = (1-\kappa)\frac{1}{p}\iota + \kappa \hat{\theta}_{AES}, $$ where $\iota$ denotes a vector of ones and $\kappa \in [0,1]$. So, when $\kappa=1$, then the vector of probabilities is the same as in the data set. When $\kappa=0$, then all probabilities are equal, and values of $\kappa$ between $0$ and $1$ generate probabilities between the two extremes. A parameter $\tau\in \{0.5,1,2\}$ is introduced to vary the sample size as $n = \tau\, n_{AES}. $ So, when $\tau=1$, then the sample size in the simulation is equal to the one in the data set, but we also consider half and double that sample size.
We begin by recalling the four main findings about 95% marginal confidence sets for the ranks of categories in Greater Melbourne from Section (ref):
Below we explore each of these findings in depth by focusing on lengths and empirical coverage frequencies of confidence sets for the rank of the 1st (top of the ranking), 4th (middle of the ranking) and 7th (bottom of the ranking) categories in Table (ref).
First, naive bootstrap confidence sets are the tightest for all three categories in all simulations. Table (ref) shows that this tightness comes at the cost of severe under-coverage for the 1st and the 7th categories. For the 7th category naive bootstrap confidence sets under-cover in all but two simulations, and empirical coverage frequency may be as low as 59.5%. Note that even for $\kappa = 1$ the success probabilities for the bottom categories are not well separated, which explains more pronounced under-coverage for the 7th category. In contrast, finite-sample methods (exactHolm and CP) cover the rank with the frequency no smaller than the desired level for all parametrizations and sample sizes. Both boot and bootStud confidence sets cover the 1st category with the frequency close to the nominal level, especially when success probabilities are equal ($\kappa=0$) and the sample size is small ($\tau = 0.5$). When the categories are better separated ($\kappa=1$) as in the dataset, then both finite-sample and bootstrap procedures cover the true rank with probability (close to) one.
Second, in contrast to our finding for Greater Melbourne, the finite-sample valid confidence sets are not uniformly tighter than the bootstrap confidences sets in simulations. In most parametrizations where boot and bootStud confidence sets are tighter, however, the difference in the size of the average confidence sets is below 0.1 and never exceeds 0.3, where the reference size of the confidence set covering the entire ranking is 6.0.
Third, both CP and exactHolm methods produce much tighter confidence sets than both types of bootstrap for the 4th category when $\kappa = 1$, i.e., when success probabilities are equal to point estimates from Greater Melbourne respondents' sample. Specifically, the average length of exactHolm confidence sets is less than one-third of the average length of boot and bootStud confidence sets when $\tau = 1$ and less than one-quarter of their average size when $\tau = 2$. Furthermore, the average length of exactHolm confidence sets is more than 20% lower than the average length of CP confidence sets in both instances. Despite the shorter average length, exactHolm confidence sets' empirical coverage frequency is above the desired level. Notice that both finite-sample valid confidence sets are of length zero in the majority of simulations with $\kappa = 1$ and $\tau = 2$ and contain only a single value 4. In contrast, most boot and bootStud confidence sets are of length one or above, meaning that they contain at least two values and are not as informative as finite-sample valid confidence sets.
Fourth, when the success probabilities are equal to point estimates from the respondents' sample of Greater Melbourne ($\kappa =1$), CP and exactHolm produce much tighter confidence sets for the 7th category than bootStud. For $\tau = 0.5$ and $\tau = 1$ the average length of CP and exactHolm confidence sets is more than two times smaller than the average length of bootStud confidence set. The difference becomes less pronounced with the increase in sample size, but the average length of the exactHolm confidence set is still more than 50% smaller than the average length of bootStud confidence set for $\tau = 2$. Furthermore, the exactHolm confidence set is substantially shorter than the boot and CP confidence sets for all values of $\tau$. Tighter exactHolm confidence sets for the 7th category still provide empirical coverage frequency above the desired level.
Table (ref) also highlights common features of the inference procedures. For equal success probabilities ($\kappa=0$) the set of ranks $R_j$ is $[1,7]$ for all categories $j$, and the average length of all confidence sets barely decreases for larger sample sizes $\tau$. When success probabilities differ ($\kappa>0$) larger sample size $\tau$ means differences in success probabilities are easier to detect, and all confidence sets for the rank decrease in length. Similarly, the average length of all confidence sets except for bootStud for 7th category decreases as we increase $\kappa$, which means differences in success probabilities are larger and thus, again, easier to detect. For bootStud at the bottom of the ranking, the effect of better separation for $\kappa>0$ is mitigated by decreasing success probabilities leading to more frequent division by zero events in bootstrap test statistics and thus larger critical values.
Finally, in addition to the five confidence sets we used in Section (ref) we include exactBonf in all simulations. Table (ref) shows that, as expected, exactBonf confidence sets are uniformly wider than exactHolm confidence sets, but the difference in the average length is not substantial.
Brown:2001p5787 found that coverage frequencies of some confidence intervals for binomial proportions may vary in highly non-monotonic ways with the sample size and the success probability. Furthermore, they found that coverage frequencies may be far below the desired level, especially for small sample sizes and/or small success probabilities. Motivated by this “erratic” behavior of coverage in the binomial case, in this subsection, we compare our confidence set (exactBonf), which is valid in finite samples, with the bootstrap confidence sets (boot and bootStud), which are justified by asymptotic validity. In particular, we are interested in their coverage properties in small samples and/or scenarios with small success probabilities. To this end we consider three categories and set the vector of success probabilities as $\theta = (\pi, \pi, 1-2\pi)$, where $\pi$ is varied between $1/100$ and $1/3$. The sample size is varied between $10$ and $100$.
For the different values of $\pi$, Figure (ref) shows the frequencies of the confidence set $C_{{\rm symm},n}(1-\alpha,I)$ simultaneously covering all differences involving the first category, $\Delta_I$, where $I=J^{\rm two-sided}$ with $J_0=\{1\}$. The coverage frequencies are plotted as functions of the sample size $n$ for the bootstrap with (panel (a)) and without (panel (b)) studentization. Both bootstrap approaches lead to simultaneous confidence sets for the differences that considerably under-cover for small sample sizes and or small probabilities $\pi$, analogously to the findings in Brown:2001p5787. The coverage probability for $\pi=1/100$ can be even lower than $0.4$ when sample sizes are small ($n\leq 20$).
Figure (ref) shows the coverage frequencies of the resulting confidence sets for the rank based on the two bootstrap approaches (panels (a) and (b)) and, for comparison, also the coverage frequencies of the finite sample method (panel (c)). Perhaps somewhat surprisingly, the coverage frequencies for the bootstrap methods are above $1-\alpha$ in all scenarios, even when success probabilities and/or sample sizes are small. Hence, under-coverage of the differences observed in Figure (ref) does not lead to under-coverage of the rank. The reason for this phenomenon is that correct coverage of the rank only requires that the confidence sets for the differences do not lead to incorrect claims about the signs of the differences. While not necessarily covering the true values of the differences, the bootstrap confidence sets for the differences lead to the correct determination of their signs and thus to coverage of the rank. As expected, the finite-sample method covers the rank with the desired probability even for small samples sizes and/or small success probabilities.
In this subsection, we consider a simulation design in which the success probabilities are all equal, i.e. $\theta_j=1/p$ for all $j=1,\ldots,p$, and we increase $p$ from $5$ to $50$. For the different values of $p$, Figures (ref) and (ref) show the coverage frequencies and lengths of the different confidence sets as functions of the sample size $n$.
First of all, the finite-sample methods exactBonf, exactHolm, and CP cover the rank with coverage frequencies close to one in all scenarios. As expected the naive bootstrap fails to cover the true rank with the desired probability. Its coverage may be considerably below $0.4$ and even approaches zero when there are many categories. Interestingly, however, the bootstrap method based on the studentized statistic also considerably under-covers when there are (moderately) many categories. For instance, with $p=20$ categories, its coverage frequency can be well below $0.8$ for small sample sizes. For more categories ($p=50$), the under-coverage occurs up to larger sample sizes.
In terms of length, the exactBonf, exactHolm, CP, and boot perform similarly. As expected bootStud and naive lead to shorter confidence sets in those scearios in which they under-cover.