Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
47,114 characters · 10 sections · 22 citation commands
Estimating the effect of treatments allocated by randomized waiting lists.
Keywords: Waiting lists, non-takers, non compliance, instrumental variable, local average treatment effect, randomized controlled trials
JEL Codes: C21, C23
Often times, some individuals who apply for a treatment are non-takers. They decline to get treated when they receive an offer, for instance because they then realize that their benefit from treatment is lower than they thought. When a treatment is oversubscribed but some applicants are non-takers, an appealing way of allocating the available seats is to use randomized waitlists. First, applicants are ranked randomly. Then, if $S$ seats are available, an initial round of offers takes place, whereby the first $S$ applicants get an offer. If $r$ of them decline it, a subsequent round of offers takes place whereby the next $r$ applicants get an offer. Offers stop when all the seats have been filled. This allocation method is fair: each taker has the same probability of being treated; it is also efficient: no seat for treatment remains unused, despite the presence of non-takers. Therefore, oversubscribed treatments with non-takers are often allocated by randomized waitlists. We conducted a survey, and found 43 articles studying treatments allocated by randomized waitlists, ranging from charter schools in the USA to agricultural trainings in Liberia. These treatments often have capacity constraints for various groups of applicants. For instance, a charter school may have 20 seats available in 7th grade and 25 seats in 8th grade. Then, a lottery takes place in each group.
As applicants are ranked randomly, it may be possible to form two comparable groups with different likelihoods of getting an offer. One could then compare those two groups to estimate the effect of the treatment. In practice, researchers have used two types of comparisons. Some researchers have compared applicants getting and not getting an initial offer, thus giving rise to the so-called initial-offer (IO) estimators. Other researchers have compared applicants ever and never getting an offer, thus giving rise to the so-called ever-offer (EO) estimators. When several lotteries were conducted, as in the charter school example above, researchers have often included waitlist fixed effects in their specifications, to ensure they compare applicants within and not across waitlists. In our survey, 22 articles used the EO estimator, 20 used the IO estimator, and a handful used other estimators. Overall, practices are not standardized.
We start by showing that the expected proportion of takers is strictly greater among applicants ever getting an offer than among applicants never getting one. Intuitively, this is because offers continue until sufficiently many takers have gotten an offer. Moreover, when waitlist fixed effects are included in the estimation, they induce an endogenous reweighting of waitlists that usually further increases this imbalance between the two groups, as we explain in more detail in Section (ref). Then, we show that due to this imbalance, the EO estimator is inconsistent when the number of waitlists goes to infinity. In our survey, we find that articles using randomized waitlists often pool data from a large number of small waitlists, thus motivating the asymptotic sequence we consider. By contrast, if the number of applicants and takers per waitlist goes to infinity, the asymptotic bias of the EO estimator goes to 0. Accordingly, in simulations we find that the EO estimator is more biased when waitlists have fewer applicants and takers.
It turns out that what creates the imbalance between applicants getting and not getting an offer is the fact that in each waitlist, the last applicant getting an offer must by construction be a taker. Indeed, we show that dropping that applicant in each waitlist is sufficient to restore the comparability between those two groups. Based on this result, we propose a new estimator of the treatment effect. It is built out of comparisons of applicants that get and do not get an offer in each waitlist, downweighting applicants that accept their offer by an amount equivalent to dropping one of them. Then, our estimator takes a weighted average of those within-waitlist comparisons, with a weighting scheme that avoids the endogeneous reweighting induced by the waitlist fixed effects. We refer to those estimators as the doubly-reweighted ever-offer estimators (DREO). We show that our estimator is consistent and asymptotically normal when the number of waitlists goes to infinity.
Contrary to subsequent-round offers, initial offers are only a function of applicants' random ranks in the waitlist. Therefore, applicants getting and not getting an initial offer are statistically comparable, and the IO estimator is also consistent. However, we find in simulations that the variance of that estimator is much larger than that of the DREO estimator, so using it will often result in large efficiency losses.
We use our results to revisit blattman2016, who studied the effects of an agricultural training. The DREO estimator is significantly and economically different from the EO estimator computed by the authors for some of the outcomes they considered.\footnote{A Stata adofile computing the DREO estimator is available from the authors' website.}
The remainder of the paper is organized as follows. Section (ref) uses a simple example to give the intuition of our results. Section (ref) presents our main results. Section (ref) presents our empirical application. Appendix (ref) presents the proofs. In our web appendix, we present our survey of articles that have used randomized waitlists, we show that some of the assumptions adopted in the paper can be relaxed, we present some simulations, and we revisit another application.
We start with a simple example. We consider a waitlist where five applicants compete for three seats. Four applicants are takers ($T$) and one is a non-taker ($NT$), meaning that she will refuse to get treated if she gets an offer. Applicants are randomly ranked, and treatment offers are made following that ranking until all seats are filled. Table (ref) displays the five possible orderings of the takers and the non-taker. For each ordering, applicants getting an offer are depicted in italics, while those not getting an offer are depicted in bold. In orderings 1 and 2, the first three applicants are takers, so offers stop after the third offer. In orderings 3, 4, and 5, one of the first three applicants is a non-taker, so a fourth offer is made; then the next applicant is a taker so offers stop as the available seats have been filled.
The first issue with the EO estimator is that, on average, applicants getting an offer bear a higher proportion of takers than applicants not getting an offer. Each ordering has a 0.20 probability of being selected. Across the five orderings, the expected share of takers among applicants getting an offer is $0.2\times \left(1+1+3/4+3/4+3/4\right)=17/20.$ On the other hand, the expected share of takers among applicants not getting an offer is $0.2\times \left(1/2+1/2+1+1+1\right)=4/5.$ Intuitively, this imbalance arises because offers stop when sufficiently many takers have accepted an offer. This endogenous stopping rule creates a positive correlation between getting an offer and being a taker. When the average potential outcomes of takers and non takers differ,\footnote{This is often the case. abadie2002 and crepon2015 are just a few examples of the many papers that have found large differences between the average potential outcomes of takers and non-takers.} this imbalance implies that applicants getting and not getting an offer are not statistically comparable: those two groups have different average potential outcomes.
The second issue with the EO estimator arises from the inclusion of fixed effects when pooling waitlists. Assume that one pools waitlists that all have four takers, one non-taker, and three seats. In some waitlists, the realized ordering of takers and non-takers is ordering 1 in Table (ref), in other waitlists the realized ordering is ordering 2, etc. With several waitlists, it follows from, e.g., Equation (3.3.7) in angrist2008, that the EO estimator with waitlist fixed effects is a weighted average of the EO estimators in each waitlist, that gives more weight to waitlists where the share of applicants getting an offer is closer to 1/2. In our example, 2/3 of applicants get an offer in waitlists with ordering 1 or 2, while 4/5 of applicants get an offer in waitlists with ordering 3, 4, or 5. Accordingly, waitlists with ordering 1 or 2 receive more weight. But those are precisely the waitlists where the proportion of takers among applicants getting an offer is the highest. Therefore, the reweighting of waitlists induced by the fixed effects aggravates the over-representation of takers among applicants getting an offer.
The DREO estimator we propose addresses those two issues. Firstly, in our example dropping the last taker getting an offer is sufficient to solve the endogenous stopping rule issue. Table (ref) shows that then, the expected share of takers among applicants getting an offer is equal to $0.2\times \left(1+1+2/3+2/3+2/3\right)=4/5,$ the same as among applicants not getting an offer. Still, dropping the last taker getting an offer is arbitrary: dropping the first or the second would have the same effect. Besides, doing so reduces the sample size and statistical precision. Instead, one can give to the three of them a weight equal to 2/3: this reduces the expected share of takers among applicants getting an offer by the same amount as dropping one. Secondly, instead of using fixed effects to pool waitlists, we simply take an average of the estimators in each waitlist, weighting waitlists proportionally to their number of applicants. These weights are independent of how many offers one has to make to fill the available seats, which solves the second issue of the EO estimator. Table (ref) shows that this second reweighting is necessary. Even after downweighting takers getting an offer, including waitlist fixed effects would still lead to over-represent takers among applicants getting an offer. Indeed, doing so gives more weight to waitlists with ordering 1 or 2, where 1/2 of applicants get an offer, while those are the waitlists where the proportion of takers among applicants getting an offer is the highest.
Throughout the paper, we consider the following set-up.
In Section (ref) in the Web appendix, we consider various extensions of this set-up. For instance, we show that our results remain unchanged if we allow for the possibility that some applicants manage to get treated even if they do not receive an offer. Similarly, we allow for the possibility that some applicants may participate in several waiting-lists, or that the treatment may not be binary. But for now we focus on the basic set-up outlined in Assumption (ref).
Then, we assume that ranks are randomly assigned to applicants. Let $R_{ik}$ denote the rank assigned to applicant $i$ in waitlist $k$, let $L_k$ denote the number of applicants getting an offer in waitlist $k$, and let $Z_{ik}=1\{R_{ik}\leq L_k\}$ denote whether applicant $i$ gets an offer, the so-called ever-offer instrument. Let $D_{ik}(1)$ denote her potential treatment if she gets an offer, and let $D_{ik}$ denote her observed treatment. Under point c) of Assumption (ref), $D_{ik}=Z_{ik}D_{ik}(1)$. For every $d\in \{0,1\}$, let $Y_{ik}(d)$ denote her potential outcome if $D_{ik}=d$,\footnote{We implicitly assume that getting an offer does not have a direct effect on the outcome, the so-called exclusion restriction, see Angrist96.} and let $Y_{ik}=Y_{ik}(D_{ik})$ denote her observed outcome. Let $$\mathcal{P}_k=\left(\left(D_{1k}(1),Y_{1k}(0),Y_{1k}(1)\right),...,\left(D_{N_kk}(1),Y_{N_kk}(0),Y_{N_kk}(1)\right)\right)$$ be a vector stacking the potential treatments and outcomes of the applicants in waitlist $k$. For any integer $j$, let $\Pi_j$ denote the set of permutations of $\{1..j\}$. Let $\mathcal{R}_k=(R_{1k},...,R_{N_kk})$ denote the ranks assigned to applicants $1$ to $N_k$ in waitlist $k$.
Assumption (ref) requires that the ranks assigned to applicants be independent of their potential treatments and outcomes, and uniformly distributed on $\Pi_{N_k}$. It implies that each applicant has the same probability of being in the first, second, ..., or last rank.
Finally, we consider a last assumption. Let applicants with $D_{ik}(1)=1$ (resp. $D_{ik}(1)=0$) be referred to as takers (resp. non-takers). For every $k \in \{1..K\}$, let $T_k=\sum_{i=1}^{N_k}D_{ik}(1)$ denote the number of takers in waitlist $k$.
Assumption (ref) requires that each waitlist have at least two seats. This can be assessed from the data, so waitlists with less than two seats can just be dropped. Assumption (ref) also requires that each waitlist have strictly more takers than seats. This cannot be assessed from the data. When all the seats available in a waitlist get filled, it must be that $S_k\leq T_k$, but it is still possible that $S_k=T_k$: all applicants not getting an offer might be non-takers. Still, we show in Section (ref) in the Web appendix that Assumption (ref) is testable.
Let $T=\sum_{k=1}^K T_k$ denote the total number of takers. Our parameter of interest is $$\Delta_K=E\left(\frac{1}{T}\sum_{(i,k):D_{ik}(1)=1}\left[Y_{ik}(1)-Y_{ik}(0)\right]\right),$$ the local average treatment effect of the takers.
Let $N=\sum_{k=1}^K N_k$ and $\overline{N}=\frac{N}{K}$ respectively denote the total number of applicants and the average number of applicants per waitlist. Let $\mathcal{I}=\{(i,k)\in \mathbb{N}^2:i\in \{1..N_k\},k\in \{1..K\}\}$, and for every $(i,k)\in \mathcal{I}$, let $w_{ik}=1-\frac{Z_{ik}D_{ik}}{S_k}.$ $w_{ik}$ is equal to $1-\frac{1}{S_k}$ for applicants that get and accept an offer, and to $1$ for everyone else. As $S_k$ takers receive an offer in each waitlist, weighting applicants getting an offer by $w_{ik}$ decreases the share of takers among them by the same amount as dropping one taker, as illustrated in the numerical example in Section (ref).
The DREO estimator of $\Delta_K$ is defined as
$\widehat{\Delta}$ can be computed through a 2SLS regression. Let $L=\sum_{k=1}^KL_k$, and let
be a weighting scheme combining $w_{ik}$ with propensity score reweighting. One can show that $\widehat{\Delta}$ is equal to the coefficient of $D_{ik}$ in a 2SLS regression of $Y_{ik}$ on $D_{ik}$ using $Z_{ik}$ as the instrument, and weighted by $w_{ik}^{DR}$. Importantly, note that under Assumption (ref), $S_k=\sum_{i=1}^{N_k}Z_{ik}D_{ik}$, so observing $(Z_{ik},D_{ik},Y_{ik})_{(i,k)\in \{1..N_k\}\times\{1..K\}}$ is sufficient to compute $\widehat{\Delta}$.
Our main result relies on the following lemma:
The intuition of point a) of the theorem goes as follows. As the numerical example in Section (ref) illustrates, one can show that in each waitlist, $w_{ik}$-reweighted applicants getting an offer are statistically comparable to applicants not getting an offer. Therefore, the only difference between these two groups is that one receives an offer and not the other one. Accordingly, $\frac{1}{L_k-1}\sum_{i:Z_{ik}=1}w_{ik}Y_{ik}-\frac{1}{N_k-L_k}\sum_{i:Z_{ik}=0}Y_{ik}$, the difference between the average outcome of the two groups, is an unbiased estimator of $E\left(\frac{1}{N_k}\sum_{i=1}^{N_k}\left[Y_{ik}(D_{ik}(1))-Y_{ik}(0)\right]\right),$ the intention to treat effect of getting an offer on applicants' outcome in waitlist $k$. The numerator of $\widehat{\Delta}$ is an average of those unbiased within-waitlist comparisons, that gives to each waitlist a weight proportional to its number of applicants. Therefore, this numerator is an unbiased estimator of $E\left(\frac{1}{N}\sum_{(i,k)\in \mathcal{I}}\left[Y_{ik}(D_{ik}(1))-Y_{ik}(0)\right]\right)$, the intention to treat effect among all applicants. The intuition of point b) is similar.
We now derive the asymptotic distribution of $\widehat{\Delta}$. In our survey of articles that have used randomized waitlists, the median number of waitlists used in the analysis is equal to 64. Therefore, we consider a sequence where $K$, the number of waitlists, goes to infinity. An alternative would be to consider a sequence where the number of applicants per waitlist goes to infinity, but in our survey the median of waitlists divided by applicants per waitlist is equal to 1.9, so the former asymptotic may be more appropriate in a majority of applications. For all $k\in\{1..K\}$, let $RF_k=\frac{N_k}{\overline{N}}\left[\frac{1}{L_k-1}\sum_{i:Z_{ik}=1}w_{ik}Y_{ik}-\frac{1}{N_k-L_k}\sum_{i:Z_{ik}=0}Y_{ik}\right]$ and $FS_k=\frac{N_k}{\overline{N}}\frac{1}{L_k-1}\sum_{i:Z_{ik}=1}w_{ik}D_{ik}.$ Let also $FS=\underset{K\rightarrow +\infty}{\lim}\frac{1}{K}\sum_{k=1}^KE\left(FS_k\right)$ and $\Delta=\underset{K\rightarrow +\infty}{\lim}\Delta_K$, where Assumption (ref) below ensures that those limits exist. Finally, for all $k$ let $\Lambda_k=\frac{RF_k-\Delta FS_k}{FS}$.
Typically, the lotteries determining applicants' ranks are independent across waitlists, so by design the vectors $(\mathcal{R}_k)_{1\leq k\leq K}$ are mutually independent, and $(\mathcal{R}_k)_{1\leq k\leq K}$ is independent of $(\mathcal{P}_k)_{1\leq k\leq K}$. Then, point a) of Assumption (ref) only requires that the vectors $(\mathcal{P}_k)_{1\leq k\leq K}$ be mutually independent. This is often plausible, for instance when the waitlists correspond to different schools. If point a) is not plausible, then Theorem (ref) below still holds conditional on applicants' potential treatments and outcomes, as in abadie2014 or li2017. Point b) requires that the number of takers be independent of their average treatment effect. If point b) does not hold, Theorem (ref) below still holds, except that $\Delta_K$ has to be replaced by $\Delta$, and more technical assumptions have to be made. Point c) requires that the number of applicants per waitlist be uniformly bounded by some constant $N^+$. Points d), e), and f) are technical conditions ensuring we can apply Liapunov's central limit theorem and Kolmogorov's strong law to $(RF_k)_{k\in \mathbb{N}}$, $(FS_k)_{k\in \mathbb{N}}$, and $(\Lambda_k)_{k\in \mathbb{N}}$. One can show that d) and f) hold if the potential outcomes $Y_{ik}(0)$ and $Y_{ik}(1)$ have a bounded support.
Let $\sigma^2=\underset{K\rightarrow +\infty}{\lim}\frac{1}{K}\sum_{k=1}^K V(\Lambda_k),$ $\sigma^2_+=\underset{K\rightarrow +\infty}{\lim}\left[\frac{1}{K}\sum_{k=1}^K E(\Lambda_k^2)-\left(\frac{1}{K}\sum_{k=1}^KE\left(\Lambda_k\right)\right)^2\right],$ \\ $\widehat{\Lambda}_k=\frac{RF_k-\widehat{\Delta}FS_k}{\frac{1}{K}\sum_{k=1}^KFS_k}$, and $\widehat{\sigma}^2_+=\frac{1}{K}\sum_{k=1}^K \left(\widehat{\Lambda}_k-\frac{1}{K}\sum_{j=1}^K\widehat{\Lambda}_j\right)^2.$
Theorem (ref) implies that $\widehat{\Delta}$ is an asymptotically normal estimator of $\Delta_K$ when the number of waitlists goes to infinity. As is usually the case for estimators constructed using independent but not identically distributed random variables (see e.g. liu1995, liu1995), the asymptotic variance $\sigma^2$ of $\widehat{\Delta}$ can only be conservatively estimated: we provide a consistent estimator of $\sigma^2_+$, an upper bound of $\sigma^2$. That estimator can then be used to build conservative confidence intervals for $\Delta_K$.\footnote{Conservative variance estimators also arise in other articles studying treatment effect estimation in randomized experiments (see e.g. Neyman1923, Neyman1923).} When all the $\Lambda_k$ have the same expectation, something that for instance happens when all waitlists have the same number of applicants, the same expectation of the proportion of takers, and the same expectations of takers' and non takers' potential outcomes, $\sigma^2_+=\sigma^2$ so those confidence intervals are exact. Finally, in simulations shown in Section (ref) of the Web appendix, we find that the asymptotic distribution in Theorem (ref) approximates the distribution of $\widehat{\Delta}$ well if 20 waitlists or more are used in the analysis. This suggests that articles using more than 20 waitlists may rely on Theorem (ref) for inference, while articles using less than 20 waitlists may not.
Let $\widehat{\beta}^{E}_{FE}$ be the coefficient of $D_{ik}$ in a 2SLS regression of $Y_{ik}$ on $D_{ik}$ and waitlist fixed effects, using $Z_{ik}$ as the instrument for $D_{ik}$. We refer to $\widehat{\beta}^{E}_{FE}$ as the EO estimator. The derivation of its limit relies on Assumption (ref), another technical assumption, that is stated in the proofs. Assumption (ref) is similar to points d) to f) in Assumption (ref), and it ensures that the limits in the definition of $w_k$ and $B$ below exist. Let
Under Assumptions (ref)-(ref), $\widehat{\beta}^{E}_{FE}$ converges towards the sum of two terms. The first is a weighted average of the LATEs of takers in each waitlist. If those LATEs vary across waitlists, this weighted average is not equal to the LATE of all takers, because it overrepresents waitlists with a ratio of seats to takers closer to 1/2.\footnote{This can be seen from the fact that $\frac{S_k\left(N_k-S_k\frac{N_k+1}{T_k+1}\right)}{N_k}=T_k\frac{S_k}{T_k}\left(1-\frac{S_k}{(T_k+1)N_k/(N_k+1)}\right)\approx T_k\frac{S_k}{T_k}\left(1-\frac{S_k}{T_k}\right)$.} The second term, $B$, is a bias term. As explained in Section (ref), this bias arises from the endogenous stopping of offers in each waitlist, and from the waitlist fixed effects.
We start by performing comparative statics on $\left|B\right|$, assuming that waitlists are homogeneous: there exist real numbers $N_0$, $T_0$, $S_0$, and $\Delta_{Y(0)}$ such that for all $k$, $N_k=N_0$, $T_k=T_0$, $S_k=S_0$, and $E\left(\left[\frac{1}{T_k}\sum_{i:D_{ik}(1)=1}Y_{ik}(0)-\frac{1}{N_k-T_k}\sum_{i:D_{ik}(1)=0}Y_{ik}(0)\right]\right)=\Delta_{Y(0)}$. Then,
where $t_0=T_0/N_0$ and $s_0=S_0/N_0$ respectively denote the proportion of takers and the ratio of seats to applicants in the waitlist. One can show that the right hand side of (ref) is decreasing in $N_0$, decreasing in $t_0$, increasing in $s_0$, and increasing in $\left|\Delta_{Y(0)}\right|$.
Then, we study how waitlists' heterogeneity affects $\left|B\right|$. Let $(S^a_0,S^b_0)\in\{2..T_0-1\}^2$, let $(T^a_0,T^b_0)\in\{3..N_0\}^2$, and let $\Delta_{Y(0),k}=E\left[\frac{1}{T_k}\sum_{i:D_{ik}(1)=1}Y_{ik}(0)-\frac{1}{N_k-T_k}\sum_{i:D_{ik}(1)=0}Y_{ik}(0)\right]$. The three following results hold:
Overall, $\left|B\right|$ seems to be higher when waitlists have heterogeneous numbers of applicants, takers, and seats. The impact of waitlists' heterogeneity on $\left|B\right|$ can be large. For instance, if $\left(N_k,S_k,\Delta_{Y(0),k}\right)=\left(40,20,\Delta_{Y(0)}\right)$, $\left|B\right|$ is 17.1% larger if $50$% of waitlists have $25$ takers and $50$% have $35$ takers than if all have $30$ takers.
Let $Z'_{ik}=1\{R_{ik}\leq S_k\}$ be an indicator for applicants in the initial round of offers, the so-called initial-offer instrument. Let $S=\sum_{k=1}^{K}S_k$. Let $w^{I}_{ik}=Z'_{ik}\times\frac{S}{N}\times\frac{N_k}{S_k}+\left(1-Z'_{ik}\right)\times\frac{N-S}{N}\times\frac{N_k}{N_k-S_k}$ be the propensity score weights attached to initial offers. Let $\widehat{\beta}^{I}_{PS}$ be the coefficient of $D_{ik}$ in a 2SLS regression of $Y_{ik}$ on $D_{ik}$, using $Z'_{ik}$ as the instrument, and weighted by $w^{I}_{ik}$. We call $\widehat{\beta}^{I}_{PS}$ the IO estimator.
Under Assumptions (ref)-(ref) and a technical condition similar to Assumption (ref), $\sqrt{K}\left(\widehat{\beta}^{I}_{PS}-\Delta_K\right)$ converges towards a normal distribution. Contrary to $Z_{ik}$, $Z'_{ik}$ is only a function of applicants' random numbers and of the number of seats in their waitlist. Thus, it satisfies the random instrument assumption in Imbens1994. Under Assumption (ref), it also satisfies the monotonicity condition therein. Then, one can show that $\widehat{\beta}^{I}_{PS}$ is an asymptotically normal estimator of the LATE of applicants complying with an initial offer. As those are a random subset of the takers, this LATE is equal to $\Delta_K$.
However, using $\widehat{\beta}^{I}_{PS}$ instead of $\widehat{\Delta}$ may result in a large loss of precision. In simulations shown in Section (ref) in our Web appendix, we find that the variance of $\widehat{\Delta}$ is between 27.6 and 57.3% smaller than that of $\widehat{\beta}^{I}_{PS}$, depending on the design we consider. This may reflect the fact that $\widehat{\beta}^{I}_{PS}$'s first stage is lower than that of $\widehat{\Delta}$, as some takers that do not get an initial offer get one in a subsequent round and get treated.
After the second Liberian civil war, some ex-fighters started engaging in illegal activities, and working abroad as mercenaries. blattman2016\footnote{blattman2016 is one of the few articles in our survey in Section (ref) whose data is not proprietary.} study the effect of an agricultural training on their employment and on their social networks. By improving their labor market opportunities, the program hoped to reduce their interest in illegal and mercenary activities, and to sever their relationships with other ex-combatants. To allocate the treatment, the authors divided applicants into 70 waitlists, according to the training site they applied for, their former military rank, and their community. In each waitlist, they randomly ranked applicants, and offers were made following that ranking until the seats available were filled.
blattman2016 estimate the training's effect on 62 outcomes, that are either applicants' answers to survey questions, or indexes averaging their answers to several related questions. To preserve space, we only consider some outcomes. Here are the rules we used to make our selection: we chose indexes rather than questions averaged into an index; among questions not averaged into an index, we discarded those asking applicants to give a subjective opinion; finally, we discarded a few measures the authors did not comment on in the paper. We end up with four measures of employment, one measure of applicants' interest in working as mercenaries, and five measures of their social network.
For each outcome, Table (ref) below shows the EO estimator computed by the authors, and the DREO estimator computed with the same controls as those used by the authors.\footnote{The DREO estimator with controls is defined in Section (ref) of the Web Appendix.} An estimate of $\widehat{\sigma}_+/\sqrt{K-1}$ is shown next to each DREO estimator.\footnote{To account for the controls included in the estimation, $Y_{ik}$ and $D_{ik}$ are regressed on the controls, and then the residuals from those two regressions are used instead of $Y_{ik}$ and $D_{ik}$ in the computation of $\widehat{\sigma}_+$.} The standard errors next to each EO estimator are computed using a bootstrap clustered at the waitlist level.\footnote{It follows from Theorem 2 in liu1995 that under point a) of Assumption (ref) and the technical conditions therein, this bootstrap yields a conservative estimate of the variance of the EO estimator.} The table also shows the p-value of a t-test that the EO and DREO estimators are equal, also computed using the bootstrap. Finally, the table shows the estimated difference between the mean of $Y_{ik}(0)$ among non-takers and takers. The EO and DREO estimators are close for all employment outcomes, but they significantly differ for three of the other outcomes. For those outcomes, the differences between the estimators are large: for applicants' interest in mercenary work, the DREO estimator is 51.0% larger in absolute value than the EO one; for applicants' relations with their ex-commanders, the DREO estimator is 47.4% larger, and it is statistically significant while the EO estimator is not; for applicants' social network quality, the DREO estimator is three times larger, but none of the two estimators is significant. For the first two outcomes, the estimated difference between the mean of $Y_{ik}(0)$ of takers and non-takers is large, which may explain why the EO and DREO estimators differ.
When the seats available for a treatment are allocated using randomized waitlists, we show that applicants getting and not getting an offer are not statistically comparable. Accordingly, a commonly used estimator of the treatment effect, the ever-offer estimator, is inconsistent when the number of waitlists goes to infinity. We propose a new estimator, the doubly-reweighted ever-offer (DREO) estimator, and we show that it is consistent and asymptotically normal. Simulations show that the DREO estimator is more efficient than another consistent estimator, the initial-offer estimator. Overall, we recommend that practitioners use the DREO estimator when they analyze randomized waitlists.