Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
83,580 characters · 14 sections · 74 citation commands
Kernel regression analysis of tie-breaker designs
In this paper we study a nonparametric regression approach to tie-breaker studies. In the settings of tie-breaker studies, there is a costly treatment while the control is inexpensive or even free. In addition, an investigator can decide how to allocate the costly treatment using a priority ordering on the subjects. The priority ordering could be based on how deserving of the treatment each subject is, or based on how strongly each subject is expected to respond to the treatment. Examples include offering scholarships or school placement to students Angrist2020Nebraska, offering a drug rehabilitation program to people of varying needs CappelleriXX, or assigning interventions to reduce risk factors for child abuse and neglect kran:2022. In these settings a randomized controlled trial (RCT) is inappropriate because it is extremely inefficient economically, or even ethically questionable. The natural, even automatic, approach to settings like this is to rank the subjects $i=1,\dots,N$ according to their value of a running variable $x_i$ and assign the treatment to only those subjects with the highest values of $x_i$. For simplicity one can assume that the number of subjects to treat is fixed and that the treatment is offered to subject $i$ if and only if $x_i> t$ for some threshold $t$.
The problem with a deterministic treatment based on $x_i$ is that it complicates causal inference of the effect of the treatment. One can use regression discontinuity analysis this:camp:1960,cattaneo2022regression but the regression discontinuity design (RDD) is known to give treatment effect estimates a very high variance. See, for example, Gelman_dont_do_high_order_polynomial. In a parametric regression model the treatment and running variable are highly correlated, making for an inefficient design gold:1972,jacob2012practical. In a tie-breaker design (TBD), the subjects are given the treatment with a probability that increases with $x_i$. The top ranked subjects get the treatment with probability one, the bottom ranked subjects do not get the treatment and subjects in between are randomly assigned to either treatment or control. The tie-breaker design interpolates between two extremes: the RCT and the RDD, trading off statistical efficiency with the short term economic value of aligning treatment to the running variable.
We believe that there are many more good uses for TBDs. Many companies interact electronically with their customers and partners. Perks, such as service upgrades, can easily be assigned with some randomization. Because the treatment is costly, it is important to evaluate the treatment efficacy later. This provides a strong motivation to introduce randomization. When the perk is simply a gift to some subjects there is less ethical concern over whether it goes to the most loyal customers or introduces some randomization. We also expect that tie-breakers will be useful in evaluating governmental programs such as the one in kran:2022, as well as educational programs such as those in Angrist2020Nebraska.
TBDs have been primarily studied as experimental designs using parametric regression modeling assumptions. While the design literature focuses on parametric models, the RDD literature primarily uses nonparametric regression methods. In this paper, we quantify the statistical gains to be obtained by conducting a TBD instead of an RDD using nonparametric regression.
Our main theoretical contributions are as follows. We study a kernel weighted local linear regression with a slope and intercept for both treated and control subjects and bandwidth $h$. The RDD can consistently estimate the treatment effect only at $x=t$, and so we focus our comparison at that point. We find an expression for the optimal bandwidth for estimating the treatment effect at $x=t$ under the TBD. We then compare the optimal mean squared error at $x=t$ for the two designs. For the popular triangular kernel, a TBD reduces the asymptotic mean squared error (AMSE) by a factor of about $2.27$ compared to an RDD of the same sample size, $N$. For other popular kernels, the AMSE is reduced by a slightly greater factor. In this setting, the AMSE decreases proportionally to $N^{-4/5}$, and using $N$ points in an RDD is comparable to using only $0.36N$ points in a TBD. The asymptotic analysis has a bandwidth that converges to zero. Since this convergence is at the very slow $N^{-1/5}$ rate, we cannot assume that in practice $h$ will be small enough such that subjects without randomized treatment are discarded. Therefore, we also compare the designs when $h$ is fixed and large enough to include nonnrandomized subjects in the regression. In this setting, the efficiency ratio, which we define as the relative variance of treatment effect estimators under the two designs, can be as large as four. For a fixed bandwidth and for the triangular and boxcar kernels, we also find that the efficiency ratio is monotone non-decreasing in the proportion of subjects who are given a randomized treatment assignment.
A further advantage of the TBD is that it can give consistent nonparametric estimates of the treatment effect for any value of $x$ in the randomization window. It can also be used to estimate the average causal effect over that window. These additional advantages are described more explicitly in Section (ref).
An outline of this paper is as follows. Section (ref) reviews the literature on tie-breakers as well as the much larger literature on RDDs. In Section (ref), we define a causal parameter of interest that can be used to compare the TBD to the RDD. We also introduce the causal identification assumptions needed and the local linear regressions used for estimating that parameter. In Section (ref), we compare the mean squared error (MSE) in asymptotically optimal estimation of our causal parameter of interest under an RDD to that under a TBD. In that asymptotic setting, the optimal bandwidth decreases at the slow $O(N^{-1/5})$ rate and then the local linear regression is eventually supported entirely in the experimental region of the TBD. In Section (ref), we investigate another regime where the bandwidth $h$ is fixed and is assumed to be larger than the radius $\Delta$ of the experimental region. For this setting, deferring an investigation of the bias to Appendix (ref), we study the variance of our estimator as a function of $\Delta$ and find the efficiency ratio to be monotone in $\Delta$ for the triangular and boxcar kernels. Section (ref) shows how one can compute efficiency ratios empirically using one's actual assignment variable levels, focusing on the Israeli classroom size data from AngristLavy as an example. The curves of the empirical efficiency ratios are quite similar to the ones obtained theoretically. Section (ref) presents a discussion. Appendix (ref) extends our results to TBDs in which each subject in the experimental group is given the treatment with probability $p \neq 1/2$. Appendices (ref), (ref), (ref), and (ref) contain some of our proofs.
Here we survey the small TBD literature and some recent developments in the much larger RDD literature. We also note connections to the experimental design literature. Most of the TBD literature has focused on global parametric models. Those are also the dominant model for experimental design. The TBD is usually compared to the RDD, for which nonparametric models are the norm.
Here we present some concepts from the regression discontinuity literature drawing heavily on cattaneo2022regression. We begin with a setting where there is a running variable $x_i$ (also called a score or index) for subject $i=1,\dots,N$. Subjects with $x_i>t$ are given the treatment and others get the control. The treatment levels are typically $T_i\in\{0,1\}$ with $T_i=0$ being the control. For the TBD setting it is more convenient to use $Z_i\in\{-1,1\}$ with $Z_i=-1$ indicating the control. The potential outcomes for subject $i$ are $Y_{i+}$ if treated and $Y_{i-}$ for control.
There are two main approaches to RDD in causal inference, continuity-based and local randomization-based. The continuity-based approach assumes that the mean response for treated subjects is continuous in $x$ as is that for control subjects. If the mean response for all subjects shows a discontinuity at $x=t$, then the magnitude of this discontinuity is defined to be the causal effect of treatment on subjects at $x=t$, and one then considers how to estimate that effect. The version from hahn2001identification has IID tuples $(x_i,Y_{i+},Y_{i-})$ where $Y_i$ equals $Y_{i+}$ for $x_i>t$ and $Y_i$ equals $Y_{i-}$ otherwise. The treatment effect is then $$ \tau = \lim_{x\downarrow t}\mu_+(x) -\lim_{x\uparrow t}\mu_-(x) $$ where $\mu_{\pm} = \mathbb{E}(Y_{\pm}\!\mid\! X=x)$. We will work primarily with a superpopulation setting where the subjects in the study are sampled from a joint distribution. cattaneo2022regression discuss this setting along with some other settings that focus on causal inference for the given subjects.
The local randomization approach from cattaneo2015randomization assumes the existence of a window $\mathcal{W}=[t-h,t+h]$ such that for $x\in \mathcal{W}$ the treatment variable is `as good as randomized'. In the local randomization approach we assume {\bf a)} that the joint distribution of the $Z_i$ for $x_i\in\mathcal{W}$ is known, and, {\bf b)} that the potential outcomes $(Y_{i+},Y_{i-})$ are independent of $x_i$. In particular, both mean responses must be constant functions of $x\in\mathcal{W}$. A variant of local randomization has the treatment based on a threshold of $x$ where $x$ is a noisy version of a latent variable $u$ eckl:igna:wage:2020 and where we have outside information under which the probability of treatment given $u$ is known.
Both frameworks have challenges. The obvious difficulty with local randomization is choosing the window $\mathcal{W}$ (or knowing the treatment probability given $u$). A smaller window provides a smaller dataset to use while making the window larger will normally increase the discrepancy between the model and the ground truth. The TBD can be viewed as a strategy to impose by design the first assumption in the local randomization approach while not making the second assumption.
The challenge in the continuity framework is in estimating the necessary limits. In a parametric model, those limits are estimated from all the data but have a bias due to lack of fit of the parametric model. As a result, nonparametric regression methods based on local polynomial models are favored. The challenge there is that one must choose a bandwidth $h$, analogously to the window size from the local randomization framework. Because the mean responses are only locally polynomial we must contend with a bias-variance tradeoff in estimating the limits.
Our theoretical and numerical results compare the TBD to the RDD in the continuity framework. We think that this is the more likely alternative analysis for our motivating applications if randomization had not been used, because the local randomization assumptions do not seem natural in those applications. We focus on the accuracy of point estimation. There is also a large literature on constructing confidence intervals around the RDD estimate (see cattaneo2022regression). We describe some of those concepts in this paper, but we do not develop confidence intervals for the TBD due to space constraints.
There are many different settings where treatments depend in a discontinuous way on $x$. In a sharp design, the treatment is $Z_i=1$ if and only if $x_i>t$. In a fuzzy design, the assignment to treatment or control might not perfectly match $x_i>t$ versus $x_i\leqslant t$, for reasons beyond the control of the investigator. For instance there may be subjects that do not comply with their assigned treatment. A related issue is that some subjects might be able to manipulate their value of the running variable in order to get (or avoid) the treatment. rosenman2019optimized discuss some ways to counter that problem. In the settings we consider, the investigator has control of the treatment and so we study the sharp design. We also do not address issues of subject compliance, as we suppose that the effect of the treatment assignment or the intent to treat is of sufficient interest to the investigator.
The RDD setting has been extended well beyond the simple framework described above. There are versions with treatments at more than two levels as well as versions with continuous treatments. The cutoff can be defined in terms of a vector of covariates yielding a discontinuity set of dimension one less than the vector has. The treatment discontinuity could be defined by geographical boundaries. There are multi-cutoff settings where subject $i$ gets the treatment when $x_i>t_i$. There are models where it is the derivative of $\mathbb{E}(Y\!\mid\! x,Z=1)-\mathbb{E}(Y\!\mid\! x,Z=-1)$ that has a step discontinuity at $t$. For discussion and references to the variants above, see cattaneo2022regression.
While we study TBDs in comparison to regression discontinuity, they can also be considered within an experimental design framework such as the covariate-dependent designs considered by metelkina2017information. That paper emphasizes sequential problems, and like most of the design literature it works primarily with parametric models. They find conditions where the treatment policies converge to optimal deterministic functions of a covariate vector. The TBD does not use deterministic allocations which is an advantage if the response distribution is subject to change between experiments.
Experimental design, especially in a sequential setting is closely related to bandit methods. We are motivated by problems where the responses $Y_i$ arrive too slowly for bandit methods to be suitable. In a business setting, the responses may arrive after a year or calendar quarter while the effect of a scholarship on graduation rates can only be seen years later.
The simplest tie-breaker design replaces the threshold $t$ by two thresholds $t\pm\Delta$. Subjects with $x_i>t+\Delta$ get the treatment, subjects with $x_i<t-\Delta$ get the control and other subjects are randomized to either treatment or control. The simplest choice has
camp:1969 describes the $\Delta=0$ version of this design. Some subjects are exactly at the threshold $t$ and then randomization breaks the ties among them. boru:1975 considers positive values of $\Delta$ such that differences in the running variable among subjects with $|x-t|\leqslant \Delta$ are essentially arbitrary because $x$ is an imperfect measure. abdu:etal:2022 study the New York school system that breaks ties among applicants by lottery, or standardized test, or audition, depending on the program. We only consider randomized tie-breaking.
gold:1972 considers a simple two line regression model that in our notation is
for IID errors $\varepsilon_i$ with mean zero and variance $\sigma^2$. He finds that an RDD estimates these coefficients with a variance that is asymptotically $\pi/(\pi-2)\approx2.75$ times as large as it would be under an RCT. His setting has Gaussian $x_i$ and $\beta_4=0$. jacob2012practical generalizes the above model to polynomials of degree two or three in $x$ with or without interactions between $x$ and $Z$. Their Table 6 shows that an RCT is 4 times as efficient as the RDD for a uniformly distributed running variable with $t$ at the midpoint of its range. They also provide similar efficiency estimates for other polynomial models for both uniform and Gaussian $x$ and include settings where $t$ is not at the median of the distribution of $x$.
The above comparisons of RCTs to RDDs do not include tie-breakers. CDP94_TBD_power3Delta compare small, medium and large randomization windows in which 20%, 35% and 50%, respectively, of the subjects get a randomized treatment. They tabulate the sample sizes needed to attain a certain level of statistical power for three treatment effect sizes in these TBDs as well as in an RDD and in an RCT. All designs had half of the subjects getting the treatment. The running variable $x$ was normally distributed. The model was (ref) with $\beta_4=0$, making the treatment effect constant. The power calculations were done by Monte Carlo sampling. The required sample sizes became smaller with increased randomization at any level of power and effect size.
owen:vari:2020 work out the asymptotic variance of $\hat\beta$ in the model (ref) as a function of $\Delta$. They consider both $\mathbb{U}[-1,1]$ and $\mathcal{N}(0,1)$ distributions for $x$ and a threshold $t$ at the median of $x$'s distribution. The estimated treatment effect is $2(\hat\beta_3+\hat\beta_4x)$ and they find for uniform $x$ that this estimate has asymptotic variance proportional to $16(1+3x^2)/[1+3\Delta^2(2-\Delta^2)]$ where $\Delta=0$ describes the RDD and $\Delta=1$ is the RCT. This decreases monotonically in $\Delta$ while increasing monotonically in $|x|$.
They also consider the opportunity cost of experimentation compared to the RDD. For $x_i\sim\mathbb{U}[-1,1]$ the expected value of $\sum_{i=1}^NY_i$ is approximately $(\beta_1 + \beta_4(1-\Delta^2)/2) N$. If larger $Y_i$ are better and $\beta_4>0$ then the opportunity cost grows proportionally to $\beta_4\Delta^2$. They discuss how one might trade off this opportunity cost against statistical efficiency.
The TBD has so far been analyzed for simpler methods than the RDD has. This can be understood by comparing their workflows. In a TBD we measure $x_i$, then sample $Z_i$ and then some time later observe $Y_i$. For an RDD we usually get $(x_i,Z_i,Y_i)$ all at once. The investigator planning a TBD only has $x_i$, must decide how to assign the $Z_i$, and may not know what model will be fit later, and then chooses some specific model to design for. When one studies the TBD theoretically, one does not even have the $x_i$ and then it is natural to assume a distribution for them. The TBD is prospective while the RDD is retrospective.
When vectors $\boldsymbol{x}_i$ of covariates are available, owen:vari:2020 describe how to investigate numerically the efficiency of a TBD that fits a regression model on some collection of features of $\boldsymbol{x}_i$ that interact with $Z_i$ where the treatment window is based on a linear combination of $\boldsymbol{x}_i$.
TimArt22 study multiple regression for a tie-breaker in a regression model $Y_i = \boldsymbol{x}_i^\mathsf{T}\beta + Z_i\boldsymbol{x}_i^\mathsf{T}\gamma+\varepsilon_i$ with $\Pr(Z_i=1)=p_i\in[0,1]$. They study a prospective $D$-optimality criterion that maximizes the determinant of $\mathbb{E}(\mathcal{X}^\mathsf{T}\mathcal{X})$, where $\mathcal{X}$ is the (random) design matrix built from $\boldsymbol{x}_i$ and $Z_i$. For any known $\boldsymbol{x}_i$, the finite sample optimal $p_i$ can be computed by convex optimization. For as yet unobserved $\boldsymbol{x}_i$ the prospective $D$-optimality criterion averages over both random $Z_i$ and random $\boldsymbol{x}_i$ from an assumed distribution for $\boldsymbol{x}_i$. For random $\boldsymbol{x}_i$, they study a three level tie-breaker with running variable $\boldsymbol{x}_i^\mathsf{T}\eta$ and treatment probabilities $0$, $0.5$ and $1$.
owen:vari:2020 consider replacing the simple trichotomy (ref) by various sliding scales where $\Pr(Z_i=1 \!\mid\! x_i )$ is a monotone function of $x_i$. They find no advantage to such alternatives when $x_i$ has a symmetric distribution about $t$ and half the subjects are treated. HarrisonArt22 revisit that problem for the two line model and find optimal designs for general $x_i$ distributions and general fractions of treated subjects without assuming that half of the subjects will be treated. These optimal designs can greatly improve upon the design defined in (ref). They still have $\Pr(Z_i=1 \!\mid\! x_i)$ as piecewise constant functions of $x_i$. If we impose monotonicity $\Pr(Z_i=1 \!\mid\! x_i )\geqslant\Pr(Z_{i'}=1 \!\mid\! x_{i'})$ whenever $x_i\geqslant x_{i'}$, then only two treatment probability levels are needed.
A limitation of previous comparisons between RDDs and TBDs is that they all assume parametric regression models for the response $Y_i$. We compare them using local linear regression. For simplicity, we restrict our attention to the three level version of the TBD in (ref).
Our model assumes an additive error on top of smooth functions of $x$ for the data points where $|x-t|>\Delta$. When $h>\Delta$, the causal estimate we consider merges deterministic and randomized treatment allocations and then cannot be analyzed in a potential outcomes framework. It is common in causal inference to ignore such data. For instance, a rule of thumb in crump2009dealing is to omit data where the treatment probability is outside $[0.1,0.9]$. Asymptotically, $h<\Delta$ and then a potential outcomes analysis is available. Otherwise, to stay within the potential outcomes framework, one must choose between ignoring some data and using the additive error model like we do.
Throughout the text we will compare the TBD to the RDD. In our comparison, we will define $t$ to be the putative RDD threshold and $\Delta$ to be the experimental radius, and we consider allocation of the treatments to the $N$ subjects according to the 3-level tie-breaker design (ref).
Next we discuss the estimands of interest. For each subject we consider the assignment variable $X\in\mathbb{R}$, the treatment $Z\in\{-1,1\}$ and two potential outcomes: $Y_+=Y(Z=1)$ and $Y_-=Y(Z=-1)$. Defining
the treatment effect at $X=x$ is $$\tau(x) =\mathbb{E}( Y_+ -Y_-\!\mid\! X=x) = \mu_+(x)-\mu_-(x).$$ If the investigator chooses an RDD with a threshold at $t$, then under certain regularity conditions, the causal estimand $$\tau_{\mathrm{thresh}} \equiv \tau(t)$$ can be consistently estimated. In particular, we assume IID samples (or sufficiently weak dependence between the samples) and that:
Under these conditions, $\tau_{\mathrm{thresh}}$ can be consistently estimated by local linear regression with $O_p(N^{-2/5})$ errors ImbensKalyanaraman_optimalBW. On the other hand, the conditions above do not suffice to let an RDD consistently estimate $\tau(x)$ for any $x\ne t$.
If an investigator runs a TBD with $\Delta>0$, then assumptions like those above replacing $t$ by $x$ allow consistent estimation of $\tau(x)$ for any $x\in(t-\Delta,t+\Delta)$. Furthermore, as long as $\mathrm{var}(Y_{\pm})<\infty$, $$\tau_{\text{ATE}}(\Delta) \equiv \mathbb{E}( \tau(X) \!\mid\! t-\Delta <X<t+\Delta) $$ can be consistently estimated with error $O_p(N^{-1/2})$ in a TBD without requiring assumptions (i), (ii), and (iii).
The discussion above leaves open the possibility that an RDD could be better than a TBD when estimating $\tau_{\mathrm{thresh}}$. Therefore, for the remainder of the paper our primary focus will be on showing that even if the only goal is estimating $\tau_{\mathrm{thresh}}$, it is still beneficial to run a TBD rather than an RDD. Our other focus will be to show that when the only goal is to estimate $\tau_{\mathrm{thresh}}$, it is beneficial to pick a larger $\Delta$ in the experimental design stage when the option is available. Picking a larger $\Delta$ has other benefits as well such as making $\tau(x)$ identifiable for more values of the assignment variable and making $\tau_{\text{ATE}}(\Delta)$ more representative of the overall population and easier to estimate. Naturally, there are non-statistical reasons to keep $\Delta$ smaller.
In keeping with current RDD practice, we suppose that {under an RDD} $\tau_{\mathrm{thresh}}$ will be estimated with local linear regression. In particular, we assume that a parameter vector $\beta$ defined by
will be fit for some symmetric kernel function $K(\cdot)\geqslant0$ and bandwidth parameter $h>0$, and that $\tau_{\mathrm{thresh}}$ will be estimated with
While this formulation of estimating $\hat{\tau}_{\mathrm{thresh}}$ may be less familiar than the approach of fitting separate local linear regressions for the treatment and control groups, it is easy to check that the two formulations yield the same estimator.
Throughout the paper, we suppose that under a TBD, $\hat{\tau}_{\mathrm{thresh}}$ will also be estimated using local linear regression according to (ref) and (ref). We do not use the same bandwidth $h$ for the TBD and RDD. Indeed, in the next section we see that the optimal bandwidth choice (in terms of AMSE) is different for the two designs.
Because kernels with unbounded support are not typically used in RDD analysis (cattaneo2022regression), we only consider kernels with bounded support. We assume without loss of generality, that the kernel is supported on $[-1,1]$. We have a special interest in a uniform (boxcar) kernel $K_{\mathrm{BC}}(x)=1_{|x|\leqslant 1}$ because it is a popular kernel choice and is a local version of the regression model (ref). We are also interested in a triangular spike kernel $K_{\mathrm{TS}}(x)=(1-|x|)_+$ where $z_+=\max(0,z)$. This kernel was shown by cheng1997automatic to optimize a bias-variance tradeoff for extrapolation from $x_i>t$ to $\mathbb{E}(Y\!\mid\! x=t)$ and has been advocated for RDD analysis by ImbensKalyanaraman_optimalBW and calonico2014robust among others.
The local linear regression estimator from (ref) has a bias and variance that both depend on the bandwidth $h$. Larger $h$ typically bring greater bias because the true regression is not precisely linear over a region centered on $t$. Smaller $h$ bring greater variance because then fewer data points are in the regression. ImbensKalyanaraman_optimalBW develop a method for choosing the bandwidth $h$ that is asymptotically mean squared optimal for the RDD. In the next section, we compare the AMSE of the TBD with that of the RDD, when each of them has their asymptotically optimal bandwidth choice.
In this paper, we focus on the accuracy of the estimated treatment effect. The RDD literature includes several papers devoted to the construction of confidence intervals. There it is necessary to account for the bias in a local polynomial regression. A simple approach is to choose $h$ to undersmooth the regression function, resulting in a bias of lower order than the standard error, and this simplifies confidence interval construction. Undersmoothing, however, brings less accuracy calonico2014robust. See calo:catt:farr:2019 for a discussion of bandwidth choices to optimize estimation, or optimize confidence interval construction, or to get robust (asymptotically valid) confidence intervals using the bandwidth that is optimal for estimation.
In this section, we demonstrate the advantage of the TBD over the RDD when each design's bandwidth is chosen to minimize the AMSE in the estimation of $\tau_{\mathrm{thresh}}=\mu_+(t)-\mu_-(t)$. Following ImbensKalyanaraman_optimalBW, we assume the following more general regularity conditions for estimating the causal effect at $X=t$:
Under an RDD, $Z_i=1$ if $X_i> t$ and is $-1$ otherwise, so these assumptions imply Assumptions 3.1--3.6 that ImbensKalyanaraman_optimalBW make for an RDD. To allow for analysis in the TBD setting, our assumptions (i)--(vi) are slightly stronger than those in ImbensKalyanaraman_optimalBW. For example, unlike in our Assumption (iii), ImbensKalyanaraman_optimalBW make no assumptions on $\mu_+(\cdot)$ in the interval $(-\infty,t)$ or on $\mu_-(\cdot)$ in the interval $(t, \infty)$. Regarding assumption (vi), ImbensKalyanaraman_optimalBW also consider the case where $\mu_+^{(2)}(t) = \mu_-^{(2)}(t)$ and show that in this case, their proposed method of estimating $\tau_{\mathrm{thresh}}$ has error $O_p(N^{-3/7})$ rather than $O_p(N^{-2/5})$. We do not consider the case where $\mu_+^{(2)}(t) = \mu_-^{(2)}(t)$ in detail for the TBD as the result should be similar to that for the RDD and is of less interest for our head-to-head comparison of TBD with RDD.
Because our assumptions (i)--(vi) imply Assumptions 3.1--3.6 in ImbensKalyanaraman_optimalBW for an RDD, if we let $$\tilde{\nu}_j \equiv \int_0^{\infty} u^j K(u) \,\mathrm{d} u \quad \text{ and } \quad \tilde{\pi}_j \equiv \int_0^{\infty} u^j K^2(u) \,\mathrm{d} u$$ for $j \in \mathbb{N}$, and let
and define
then Lemma 3.1 of ImbensKalyanaraman_optimalBW holds. We reproduce the statement of this lemma below.
Because we wish to compare the RDD to the tie-breaker design, we derive a similar result for the asymptotic MSE for the tie-breaker design. The TBD counterparts to the RDD quantities above are
for $j \in \mathbb{N}$,
and
The proof of this lemma is very similar to the proof of Lemma 3.1 in ImbensKalyanaraman_optimalBW, from their appendix. Instead of pointing to their proof and noting the parts of their proof that differ in the tie-breaker design setting, we write out the proof of Lemma (ref) in Appendix (ref) to ensure there are no subtle issues with using their proof in the tie-breaker design setting.
{The leading order MSE formulas are derived by evaluating and summing the leading order terms for both the squared-bias and the variance.} In formulas (ref) and (ref) for the leading order MSE, the first term gives the leading order squared-bias while the second term gives the leading order variance. See formulas (ref) and (ref) for explicit calculations of the leading order bias and variance in the TBD case, and see the formulas for `B' and `V' in the appendix of ImbensKalyanaraman_optimalBW for explicit calculations of these quantities in the RDD case. It is not surprising that the formulas for the leading order squared-bias, variance and MSE are different for the two design types because for an RDD, estimation of $\tau_{\mathrm{thresh}}$ involves estimation of the mean functions at a boundary point, whereas for a TBD, estimation of $\tau_{\mathrm{thresh}}$ involves estimation of mean functions at an interior point.
In Figure (ref), we plug in scalar multiples of the optimal bandwidth for the RDD given in Lemma (ref) to the first and second terms of formulas (ref) and (ref) to visualize the trade-off for the leading order squared-bias and variance in a tie-breaker design compared to a regression discontinuity design. The formulas simplify when defining the quantity
which does not depend on $h$, $N$ or the kernel choice.
In practice, the optimal bandwidth is not known and must be estimated. For both the RDD and the TBD, the optimal bandwidth depends on the quantity
which must be estimated from the observed data. We consider the regularized estimator for $\gamma$ of ImbensKalyanaraman_optimalBW. We take the estimated optimal bandwidth $\hat{h}_{\text{opt}}$ proposed in their Section 4.2 and set $\hat{\gamma}_{\text{RDD}}= (4 \tilde{C}_1/\tilde{C_2} )^{1/5}\hat{h}_{\text{opt}} N^{1/5}$. It can be seen from the proof of Theorem 4.1 in ImbensKalyanaraman_optimalBW that under assumptions (i)--(vi), $\hat{\gamma}_{\text{RDD}} \xrightarrow{p} \gamma$. In the TBD case, we know a consistent estimator of $\gamma$ exists. For example, if we let $\hat{\gamma}_{\text{TBD,naive}}$ be an estimator of $\gamma$ that is constructed similarly to $\hat{\gamma}_{\text{RDD}}$ using only the subset of the data which looks like an RDD, $\hat{\gamma}_{\text{TBD,naive}} \xrightarrow{p} \gamma$. Of course such an estimator of $\gamma$ is inefficient; in practice one should instead use an estimator of $\gamma$ that does not throw out all of the control samples for which $x>t$ and all of the treated samples for which $x<t$. For our theoretical comparison of TBDs with RDDs, we are not concerned with the actual form of $\hat{\gamma}_{\text{TBD}}$ as long as it is consistent. Therefore, in the TBD case we will let $\hat{\gamma}_{\text{TBD}}$ be any estimator that satisfies $\hat{\gamma}_{\text{TBD}} \xrightarrow{p} \gamma$. We make a few remarks about estimation of $\gamma$ in the TBD setting in the discussion section.
To compare the AMSE for the RDD versus the TBD, we will assume that if the investigator were to run an RDD and were seeking mean squared optimal estimation of $\tau_{\mathrm{thresh}}$, they would ultimately use the bandwidth
where $\hat{\gamma}_{\text{RDD}}$ is the consistent estimator for $\gamma$ described above and $\tilde{C}_j$ are defined at (ref). We will also assume that if the investigator were to run a TBD seeking mean squared optimal estimation of $\tau_{\mathrm{thresh}}$, they would ultimately use the bandwidth
where $\hat{\gamma}_{\text{TBD}}$ is any consistent estimator of $\gamma$ and $C_j$ are defined at (ref).
The following theorem compares the RDD with $N$ points to a TBD with $\theta N$ points for some $\theta>0$. We will use the value of $\theta$ that provides equal MSEs for estimation of $\tau_{\mathrm{thresh}}$ as a metric to compare the two designs.
Theorem (ref) uses the assumption that $\Pr(Z_i=1 \!\mid\! x_i )=1/2$ for $x_i$ in the randomization window. If $\sigma^2_+(t)\ne\sigma^2_-(t)$, then we might prefer to offer the treatment with probability $p\ne1/2$. In Appendix (ref), we study a treatment probability $p\in(0,1)$. When $p = \sigma_+(t)/(\sigma_+(t) +\sigma_-(t))$, the asymptotic MSE is minimized, though an investigator would also want to account for the cost of the treatment. If one chooses $p$ using poor prior estimates of $\sigma_\pm$ it is possible that the resulting TBD will have a higher asymptotic MSE than the RDD. However, for any of the kernels in Table (ref), one can protect against that by choosing $p\in[0.18,0.82]$.
We now use Theorem (ref) to compare the MSE in estimating $\tau_{\mathrm{thresh}}$ for the RDD versus the TBD, under optimal bandwidth choices for various kernels of interest. See Table (ref). If an investigator is deliberating between an RDD with $N$ samples versus conducting a TBD (for a fixed $\Delta>0$) with $N$ samples, and either experimental design is to be analyzed with the asymptotically optimal bandwidth choice for the prespecified kernel, then the ratio of the MSEs will converge in probability to $\big((\tilde{C}_1 \tilde{C}_2^4)/(C_1 C_2^4)\big)^{1/5}$ as $N \to \infty$. Using formulas (ref) and (ref), the fourth column of Table (ref) gives the value of the quantity $\big((\tilde{C}_1 \tilde{C}_2^4)/(C_1 C_2^4)\big)^{1/5}$ rounded to 2 decimal places. For the boxcar and triangular kernels respectively, this quantity is precisely $64^{1/5}$ and $60.46618^{1/5}$ without rounding.
It is also interesting to consider the quantity given by
As a result of Theorem (ref), an experimental designer deciding to use a TBD rather than an RDD would only need to collect $\theta_*$ times as many samples in order to achieve the same asymptotic MSE in estimating $\tau_{\mathrm{thresh}}$.
Table (ref) shows that the kernel choice has a remarkably small impact on the relative benefit of using a TBD rather than an RDD to estimate $\tau_{\mathrm{thresh}}$. It is well known in the usual kernel smoothing setting that there is little difference in performance among the widely used kernels. See wand:jone:1994. If $\tau_{\mathrm{thresh}}$ is to be estimated with local linear regression using one of the seven popular kernel choices exhibited in Table (ref), then the RDD has an asymptotic MSE that is about 2.3 times as large as that of the TBD, and the TBD will require 64 to 65 percent fewer samples than the RDD in order to achieve the same asymptotic MSE.
We think that a version of Theorem (ref) will hold also for unbounded kernels such as the $\mathcal{N}(0,1)$ density under reasonable but stronger regularity conditions on $f(\cdot)$, $\mu_\pm(\cdot)$ and $\sigma_\pm(\cdot)$. We do not develop such a result as cattaneo2022regression state that kernels with unbounded support are not used in RDD analysis.
The AMSE comparison in Section (ref) depends upon the optimal TBD bandwidth, $\hat{h}_{\text{opt,TBD}}$, eventually becoming smaller than the positive experimental radius $\Delta$. However, the optimal $h$ converges to zero only at the very slow rate $N^{-1/5}$. Furthermore, the constant in that rate includes the factor $|\mu_+^{(2)}(t)-\mu_-^{(2)}(t)|^{-2/5}$ which could be very large. We believe that in many applied settings the optimal value of $h$ will not be smaller than $\Delta$. Then $\Delta/h$ is not necessarily within the support of the kernel and $Y$ values data from outside the experimental region are included in the local linear regression.
In this section, we complement the prior analysis with one where $h$ is fixed and larger than $\Delta$. We assume a symmetric kernel function that is Lipschitz continuous on its support.
The kernel regression estimate of $\hat\tau_{\mathrm{thresh}}$ has a leading bias of $O(h^2)$. In the regime where the bandwidth is bigger than $\Delta$, mean squared optimality analysis for estimating $\tau_{\mathrm{thresh}}$ is complicated by the fact that for the TBD there will often exist an $h> \Delta$ such that the constant in this $O(h^2)$ term vanishes. Remarkably, such a bandwidth depends only on the experimental radius $\Delta$ and the kernel $K$. It does not depend on $\mu_+$, $\mu_-$, $f$, or $N$. In Appendix (ref), we prove that under certain regularity conditions on $f$, $\mu_{\pm}$, and $K$, a bandwidth $h$ that solves $\nu_2^2 = 4 \int_{\Delta/h}^{\infty} uK(u) \,\mathrm{d} u \int_{\Delta/h}^{\infty} u^3K(u) \,\mathrm{d} u$ removes the leading order bias, and moreover, such a solution exists. See Table (ref) for numerical solutions of this equation for the kernel choices considered previously. We find that for these kernel choices, the bandwidth removing the leading order bias ranges from approximately $3.13 \Delta$ for the Boxcar kernel to approximately $4.84 \Delta$ for the Triweight kernel. We caution investigators against picking this bandwidth because it does not shrink with $N$. It could place too little weight on reducing variance for small $N$ and the third order bias term will be $O(1)$.
Due to the existence of a fixed bandwidth bigger than $\Delta$ that removes the leading order bias of $\hat{\tau}_{\mathrm{thresh}}$, analysis of bias and mean squared optimality using second order Taylor expansions of $\mu_{\pm}(\cdot)$ would be misleading. Hence, we do not conduct an analysis similar to that seen in Section (ref) for the regime where $h>\Delta$. For that regime, we instead restrict our attention to the variance in estimating $\tau_{\mathrm{thresh}}$ at a fixed bandwidth $h$.
The variance of the local linear estimator $\hat{\tau}_{\mathrm{thresh}}$ given in (ref) and (ref) can be computed as follows. The design matrix for the regression is $\mathcal{X}\in\mathbb{R}^{N\times 4}$ with $i$'th row $(1,x_i,Z_i,x_iZ_i)$. The response is $\mathcal{Y} = (Y_1,\dots,Y_N)^\mathsf{T}$. For simplicity we assume $\mathrm{var}(\mathcal{Y} \!\mid\! \mathcal{X})=\sigma^2 I_{N}$ and without loss of generality we assume $t=0$. The kernel weights are $K(x_i/h)$, and we let $\mathcal{W}=\mathcal{W}(h)\in\mathbb{R}^{N\times N}=\mathrm{diag}(K(x_i/h))$. Then
and under the assumption that $\mathrm{var}(\mathcal{Y} \!\mid\! \mathcal{X})=\sigma^2 I_{N}$ we have
Formula (ref) for $\hat\beta$ matches the familiar generalized least squares formula for the case where $\mathrm{var}(\mathcal{Y}\!\mid\! \mathcal{X})=\mathcal{W}\sigma^2$. Here $\mathcal{W}$ arises from weights that are not of inverse variance type and hence the formula for $\mathrm{var}(\hat\beta\!\mid\!\mathcal{X};\Delta)$ involves a $\mathcal{W}^2$ factor and less cancellation than we might have expected. The boxcar kernel is special because then $K(x_i/h)\in\{0,1\}$ equals its own square. In that case $\mathrm{var}(\hat\beta\!\mid\!\mathcal{X};\Delta)=(\mathcal{X}^\mathsf{T}\mathcal{W}\mathcal{X})^{-1}\sigma^2$. The estimator is $\hat{\tau}_{\mathrm{thresh}} = 2\hat{\beta}_3$. Therefore, we study $\mathrm{var}(\hat\beta_3\!\mid\!\mathcal{X};\Delta)$ under a tie-breaker design as $(\mathrm{var}(\hat\beta\!\mid\!\mathcal{X};\Delta))_{3,3}$ using the expression in (ref).
At the stage where the experiment is being designed and $\Delta$ is being chosen, the investigator does not have much information about $\mathcal{X}\in\mathbb{R}^{N\times 4}$ but we will later see, quite a bit is known about the quantity $N \times \mathrm{var}(\hat\beta_3\!\mid\!\mathcal{X};\Delta)/\sigma^2$. For $x_i$ from a real dataset, we see in Section (ref) (e.g. Figure (ref)) that this quantity does not vary much for different simulations of the random treatment assignments $(Z_i)_{i=1}^N$. To get theoretical insight, we turn our attention to the uniformly spaced setting with $x_i=(2i-N-1)/N$ to develop tractable theoretical results. We give an asymptotic justification for this assumption using results from fan1996local in Section (ref). This rank transformation is also used in owen:vari:2020.
For $x_i = (2i-N-1)/N$, the matrices $\mathcal{X}^\mathsf{T}\mathcal{W}\mathcal{X}/N$ and $\mathcal{X}^\mathsf{T}\mathcal{W}^2\mathcal{X}/N$ contain elements that can be approximated by integrals of the form
for integer exponents $r$, $s$ and $t$. Our expressions will simplify somewhat because $Z^2=1$ making every $\mathcal{I}^{r,2,t}=\mathcal{I}^{r,0,t}$ and also because both $x$ and $\mathbb{E}(Z\!\mid\! x;\Delta)$ are antisymmetric functions of $x$ making them orthogonal to $K(x/h)$ which we have assumed to be symmetric. The error in those moment approximations is $O_p(N^{-1/2})$ if the $Z_i$ are independent random variables. The error can be much less with other sampling schemes. For instance, we could use stratified sampling, forming pairs of subjects $(i,i+1)$ in the experimental region and randomly setting $Z_i=\pm1$ and $Z_{i+1}=-Z_i$. We will use $\approx$ to describe approximations that are $O_p(N^{-1/2})$ or better.
Applying first $Z^2=1$ and then using symmetry and anti-symmetry
Because $K^2(\cdot)$ is also a symmetric function we also get $$ \frac1N\mathcal{X}^\mathsf{T}\mathcal{W}^2\mathcal{X} \approx
. $$
From all of the symmetries involved in the 32 components of these two matrices, we need to consider at most six distinct integrals. We rewrite those matrices, beginning with
where
Note that $\kappa_0$ and $\kappa_2$ may depend on $h$ but they do not depend on $\Delta$. A similar argument shows that
for
Now we are ready to describe the asymptotic variance of $\hat\beta_3$.
Our Lipschitz condition on the kernel $K$ is present for technical reasons. Using a formulation with fixed and discrete $x_i=(2i-N-1)/N$, this condition allows us to obtain the same error rate of $O_p(N^{-1/2})$ as would be obtained using the formulation with random $x_i \stackrel{\text{IID}}{\sim} \mathbb{U}[-1,1]$. Without the Lipschitz condition, an adversarially chosen kernel $K(\cdot)$ might have point discontinuities at every rational multiple of the bandwidth $h$. We remark that our Lipschitz condition can be loosened to a $1/2$-Hölder continuity condition, with details available from the first author upon request.
The variance formula in Theorem (ref) does not require the linear model (ref) to hold. When it does not hold there will generally be some bias where $\mathbb{E}(2\hat\beta_3\!\mid\!\mathcal{X};\Delta)\ne \mu_+(0)-\mu_-(0)$. We suppose that the user will choose an $h$ to appropriately navigate the bias variance tradeoff, but that step takes place after the outcomes $Y_i$ are observed, which are not available when $\Delta$ is chosen, so we resort to comparing the variance for any choice of $h$.
We are primarily interested in comparing the asymptotic variance of $\hat{\tau}_0=2\hat\beta_3$ for various choices of $\Delta$. We especially want to compare the efficiency of tie-breaker designs with $\Delta>0$ to the RDD with $\Delta=0$. To do this we consider the efficiency ratio
Using Theorem (ref), $\mathrm{Eff}^{(N)}(\Delta)$ converges in probability to the asymptotic efficiency ratio
using quantities that we defined at (ref) and (ref).
In this subsection we present the efficiency ratios under the conditions of Theorem (ref) for the two kernels of greatest interest: the boxcar kernel and the triangular kernel. We work with $x_i = (2i-N-1)/N$ throughout this subsection.
For the boxcar kernel $K_{\mathrm{BC}}(u) = 1_{|u|\leqslant 1}$, we can assume without loss of generality that $h\leqslant 1$ because there are no data with $|x_i-t|=|x_i|>1$, and then any $h>1$ will give the same estimate as $h=1$. We find for this kernel that
Using some foresight, we define the local tie-breaker constant $\delta=\Delta/h$. This is the fraction of the local regression region in which the treatment was assigned at random.
Choosing $h=1$ makes the local regression a global one. We then get the same efficiency ratio as in equation (6) from owen:vari:2020. By taking derivatives it is easy to show that the efficiency ratio in (ref) is strictly increasing as the local amount of experimentation $\delta$ varies over the interval $0<\delta< 1$. Figure (ref) plots $\mathrm{Eff}_{\mathrm{BC}}$ versus $\delta$.
The triangular spike kernel $K_{\mathrm{TS}}(x) = (1 -\vert x \vert )_+$ (triangular kernel for short) is more complicated than the boxcar kernel because for it, $K^2$ is not proportional to $K$. Once again, we assume that $h \in [0,1]$. For this kernel we compute $$\kappa_0= \frac{h}{2},\quad \kappa_2 = \frac{h^3}{12}, \quad \lambda_0 = \frac{h}{3},\quad\text{and}\quad \lambda_2=\frac{h^3}{30}$$ and then using $\delta = \Delta/h$, we get $$\phi(\Delta)= \frac{h^2}{6} (1- 3 \delta^2+ 2 \delta^3)\quad\text{and}\quad\psi(\Delta)= \frac{h^2}{12} (1- 6 \delta^2+ 8 \delta^3 - 3 \delta^4).$$
The second panel in Figure (ref) shows $\mathrm{Eff}_{\mathrm{TS}}$ versus the local experiment size $\delta$. The efficiency curve has a similar monotone increasing shape as we saw for the boxcar kernel. The maximum efficiency ratio, at $\delta=1$, is $18/5=3.6$ instead of $4$. The efficiency ratio is a rational function of $\delta$ with a numerator of degree $12$ and a denominator of degree $7$. It is strictly increasing on the interval $0<\delta<1$, though the proof is lengthy enough to move to the Appendix.
We explored the efficiency ratio for the tie-breaker design for $x_i$ with a uniform distribution. While that can be arranged by using ranks, in other situations we might prefer to use the original value of a running variable and those might not be uniformly distributed. We show how to do this using a dataset from AngristLavy on classroom sizes.
AngristLavy studied the causal effect of classroom size on test performance of elementary school students in Israel. In Israel, the Maimonides rule mandates that elementary school classes cannot exceed 40 students. If a school has 41 students enrolled in a particular grade that grade must be split into two classes. Note that grades that have 40 or fewer enrolled students are allowed to split into multiple classes and that grades with slightly more than 40 students occasionally violate the Maimonides rule and do not split into multiple classes. Despite this, we can consider this a setting for RDD where the treatment variable is whether or not the school is legally mandated to split a particular grade into smaller classes.
The dataset, published on the Harvard Dataverse ALHarvard2, has verbal and math scores for 3rd, 4th and 5th graders across Israel. We chose to focus exclusively on 4th grade verbal scores as our response variable and 4th grade enrollments as our assignment variable because AngristLavy suggest that a slightly significant effect of the treatment on 4th grade verbal scores exists. Even though the data were not generated by a tie-breaker we can still compute the relative efficiency that a tie-breaker design would have had.
To simplify the analysis, we removed all schools that either had more than 80 students or more than two 4th-grade classes from the dataset. We further removed all schools that had NA entries for either class size or verbal scores, leaving $N=711$ schools in our filtered dataset. See Figure (ref) for a visualization of the distribution of the 4th grade enrollments and Figure (ref) for visualizations of the local linear regression based-RDD on this dataset using boxcar and triangular kernels. We use the bandwidths $h_{\mathrm{IK}}$ given by the ImbensKalyanaraman_optimalBW procedure, which were computed using that paper's MATLAB code. The apparent benefit from smaller classrooms is positive but small and it turns out, not statistically significant in this analysis. The 95% confidence interval (assuming homoscedastic errors) for the effect size at the boundary of the local linear regression-based RDD was $(-1.5,9.2)$ when a boxcar kernel with bandwidth $h_{\mathrm{IK},\mathrm{BC}} = 7.09$ was used. The 95% confidence interval for the effect size at boundary of this RDD was $(-2.4, 9.4)$ when a triangular kernel with bandwidth $h_{\mathrm{IK},\mathrm{TS}} = 9.02$ was used.
Next we illustrate how an investigator can estimate the efficiency ratio of tie-breaker designs as a function of $\Delta$ on sample values of the assignment variable. First we translate the data, replacing $x_i$ by $x_i-40.5$ to move the threshold from $t=40.5$ to $t=0$. Next, for each $\Delta$ of interest we use $1000$ Monte Carlo samples to estimate $\mathrm{var}(\hat\beta_3\!\mid\!\mathcal{X};\Delta)$ and also $\mathrm{var}(\hat\beta_3\!\mid\!\mathcal{X} ;0)$, both up to a constant $\sigma^2$. That gives us $1000$ efficiency ratios $\mathrm{Eff}^{(N)}(\Delta)=\mathrm{var}(\hat\beta_3\!\mid\!\mathcal{X};0)/\mathrm{var}(\hat\beta_3\!\mid\!\mathcal{X};\Delta)$ for each $\Delta$. In each of our $1000$ samples, we simulate random assignments $Z_i$ for a tie-breaker design at the given experimental radius $\Delta$. The random assignments are stratified: in each consecutive pair of classroom sizes in the experimental region, one was randomly chosen to have $Z=1$ and the other got $Z=-1$. The $x_i$ and the random $Z_i$ let us compute the matrices $\mathcal{X}$ and $\mathcal{W}$ defined in the beginning of Section (ref), from which we compute a non-asymptotic $\mathrm{var}(\hat{\beta}_3\!\mid\!\mathcal{X};\Delta)$ using (ref). We do not simulate any $Y_i$ values because efficiency only depends on $\mathcal{X}$, and we are retaining the bandwidths from the ImbensKalyanaraman_optimalBW procedure on the original data. A more detailed simulation randomizing the bandwidth choice is out of scope. Our simulations demonstrate that the TBD is more efficient at each fixed $h$, so we expect that it will also be more efficient at a randomly chosen $h$. There could be exceptions if the bandwidth is adversarially correlated with the estimation errors but we do not think that is likely.
Figure (ref) shows boxplots of $1000$ simulated $\mathrm{Eff}^{(N)}(\Delta)$ values for various choices of $\Delta \in \mathbb{N}$ to plot the full efficiency curve. It is clear from Figure (ref) that with stratified allocations the efficiency is very reproducible. Figure (ref) shows results for different bandwidths, ranging from $h_{\mathrm{IK}}/2$ to $3h_{\mathrm{IK}}/2$. Because the efficiencies are so reproducible given the bandwidth, we just plot curves of the mean and standard deviations of estimated $\mathrm{Eff}$ values. For both the boxcar and triangular kernels, we see that the tie-breaker design is reproducibly more efficient than the RDD and the effect increases as $\delta =\Delta/h$ increases for all $h$ we studied. The efficiency curves for this dataset under various bandwidth choices look similar to the theoretical efficiency curves derived in Section (ref) for the case of a uniform assignment variable.
For a further discussion of the Maimonides rule, see angr:etal:2019. They consider different data sets and also investigate the possibility that the class sizes are sometimes manipulated to be above the threshold triggering a classroom split.
Our theoretical analysis in Section (ref) is for a uniformly spaced assignment variable. We can offer one explanation for why the empirical efficiencies on non-uniformly distributed data look so similar to the theoretical ones for uniformly distributed data (see the left panels in Figure (ref)). The explanation uses some results about non-parametric regression from fan1996local. Nonparametric regression estimates $\hat\mu(t)$ typically have an asymptotic variance where the leading term is proportional to $1/f(t)$ where $f$ is the probability density of the $x_i$. This arises because the local sample size is asymptotically proportional to $f(t)$. Hence, when considering nonuniform distributions, the $1/f(t)$ factors in the leading order variance terms will cancel out when computing the efficiency ratios. Some of the nonparametric regression estimators, such as the Nadaraya-Watson estimator, have a lead term in their bias that depends on the derivative $f'(t)$, and while $f'(t)=0$ for uniformly distributed data, it is not zero in general. Kernel weighted least squares methods (with symmetric $K(\cdot)$) do not have a dependency on $f'(t)$ in their bias. There is a curvature bias from $\mu''(t)$ but that is not related to the sampling distribution of the $x_i$. The lead terms in bias and variance for local linear regressions do not distinguish between distributions with the same value of $f(t)$ but different $f'(t)$. Thus the effects of non-uniformity of $X$ are asymptotically negligible.
If an investigator is able to implement a 3-level tie-breaker design with any experimental radius $\Delta \in (0,\Delta_{\text{max}})$, our results show that the TBD has considerable statistical advantages over the RDD.
The most obvious advantage is that the TBD allows estimation of multiple causal parameters of interest including the average treatment effect over subjects with $x \in (t- \Delta,t+\Delta)$ as well as the expected treatment effect at any particular $x \in (t- \Delta,t+\Delta)$. The former is estimable at a faster rate and with fewer assumptions, whereas the latter may still be of interest for choosing a future policy threshold. Meanwhile, the RDD only allows estimation of $\tau_{\mathrm{thresh}}$, the expected treatment effect at $x=t$.
Even if the only goal is estimation of $\tau_{\mathrm{thresh}}$, our results indicate a statistical advantage to running a TBD rather than an RDD and an advantage to picking a larger experimental radius $\Delta \in (0,\Delta_{\text{max}})$. As seen in Section (ref), to achieve the same asymptotic MSE in mean squared optimal estimation of $\tau_{\mathrm{thresh}}$, a TBD would require roughly 64 percent fewer samples than would be needed for an RDD. Moreover, the asymptotic advantage for a TBD is largely driven by its lower variance (Figure (ref)). Hence, if the convenient, but controversial, method of undersmoothing to construct asymptotically valid confidence intervals for $\tau_{\mathrm{thresh}}$ is used instead of more nearly optimal approaches, the TBD would exhibit even greater advantages over the RDD. We point readers to the introduction of Calonico_dont_undersmooth for an overview of the history of undersmoothing, and calo:catt:farr:2019 for a modern approach to constructing confidence intervals that has better coverage properties than undersmoothing has.
In terms of the statistical advantages of picking a larger $\Delta$, owen:vari:2020 found an efficiency advantage for the tie-breaker in a global regression, wherein the estimation variance decreased monotonically in $\Delta$. We provide a comparable finding for the now more standard local linear regression approach: for any fixed bandwidth $h$, we see a theoretical efficiency that increases with the amount $\Delta$ of experimentation. We have not investigated the effect of $\Delta$ on the subsequent choice of $h$ when $\hat{h}_{\text{opt,TBD}} > \Delta$, although one candidate choice is an $h> \Delta$ that removes the leading order bias term, which we derived in Appendix (ref).
There is room for an improved estimator of $\gamma$ in the TBD context which uses data from both treatments on both sides of the threshold $t$. We leave this for further work. A critical ingredient is the estimation of $\mu_{\pm}^{(2)}(t)$. Compared to the method in ImbensKalyanaraman_optimalBW, one could use a bandwidth tuned for an internal point $t$ instead of one tuned for an endpoint. Also the curvature estimates in ImbensKalyanaraman_optimalBW use local quadratic regressions while fan1996local suggest using local cubic regressions for curvature estimation at an interior point.