Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
118,214 characters · 15 sections · 59 citation commands
A general characterization of optimal tie-breaker designs
Companies, charitable institutions, and clinicians often have ethical or economic reasons to prefer assigning a binary treatment to certain individuals. If this preference is expressed by the values of a scalar running variable $x$, a natural decision is to assign the treatment to a subject if and only if their $x$ is at least some threshold $t$. This is a regression discontinuity design, or RDD this:camp:1960. Unfortunately, treatment effect estimates from an RDD analysis typically have very high variance jacob2012practical,gold:1972,gelman2017high, relative to those from a randomized control trial (RCT) that does not preferentially treat any individuals. To trade off between these competing statistical and ethical objectives, investigators can use a tie-breaker design (TBD). In a typical tie-breaker design, the top ranked subjects get the treatment, the lowest ranked subjects are in the control group and a cohort in the middle are randomized to treatment or control. The earliest tie-breaker reference that we are aware of is camp:1969 where $x$ was discrete and the randomization broke ties among subjects with identical values of $x$.
Past settings for tie-breaker designs include the offer of remedial English to incoming university students based on their high school English proficiency aike:west:schw:carr:hsiu:1998, a diversion program designed to reduce juvenile delinquency lips:cord:berg:1981, scholarship offerings for two and four year colleges based on a judgment of the applicants' needs and academic strengths abdulkadiroglu2017impact, angrist2020, and clinical trials Trochim92, where they are known as cutoff designs.
The tie-breaker design problem is to choose treatment probabilities $p_i$ for subjects $i=1,\dots,n$ based on their running variables $x_i$. These probabilities are chosen before observing the response values $y_1,\dots,y_n$ but with the running variables $x_1,\dots,x_n$ known. We assume throughout that $p_i=p_{i'}$ whenever $x_i=x_{i'}$.
As is common in the optimal experimental design literature, the statistical objective is an “efficiency" criterion that measures estimation precision. Specifically, our criterion will be a function $\Psi(\cdot)$ of the information (scaled inverse variance) matrix $\mathcal{I}_n(p_1,\dots,p_n)$ for the model parameters $\beta = (\beta_0,\beta_1,\beta_2,\beta_3)^{\top}$ in a two line model relating the response $y_i$ to the running variable $x_i$ and a treatment indicator $z_i\in\{-1,1\}$:
This simple working model nonetheless poses some challenging design problems. In Section (ref) we describe some more general modeling settings for tie-breaker models.
Throughout we assume that the running variable is centered, i.e.\ $(1/n)\sum_ix_i=0$, and that the $\varepsilon_i$ have common variance $\sigma^2$. Here $z_i=1$ indicates treatment and so $p_i = \Pr(z_i=1) = (1+\mathbb{E}(z_i))/2$). For model (ref) the information matrix is $\mathcal{I}_n = \mathbb{E}(\mathcal{X}_i\mathcal{X}_i^{\top})$ where $\mathcal{X}_i = (1,x_i,z_i,x_iz_i)^{\top} \in \mathbb{R}^4$ and the expectation is taken over the treatment assignments $z_i$, conditional on the running variables $x_i$ (whose values are known). The ordinary least squares estimate $\hat{\beta}$ of $\beta$ satisfies $\mathbb{E}(\mathrm{Var}(\hat\beta)^{-1})=n\mathcal{I}_n/\sigma^2$. Common examples of efficiency criteria $\Psi(\cdot)$ in the literature, such as the D-optimality criterion $\Psi_D(\cdot) = \log(\det(\cdot))$, are concave in both $\mathcal{I}_n$ and $p = (p_1,\ldots,p_n)^{\top}$ boyd:vand:2004. However, our theoretical results only require continuity of $\Psi(\cdot)$.
Our preference for treating individuals with higher running variables $x$ is expressed as an equality constraint on the scaled covariance $\overline{xp}\equiv(1/n)\sum_{i=1}^n x_ip_i$ between treatment and the running variable (recall the latter is known, hence viewed as non-random). Under the two-line model (ref), this constraint has the following economic interpretation. We take $y$ to be something like economic value or student success, where larger $y$ is better. We expect that $\beta_3>0$ holds in most of our motivating problems. The expected value of $y$ per customer under (ref) is then
where $\bar{p} \equiv (1/n)\sum_{i=1}^n p_i$. Equation (ref) shows that the expected gain is unaffected by $\beta_0$ or $\beta_1$. Furthermore, we assume the proportion of treated subjects is fixed by an external budget, i.e., an equality constraint $\bar{p} = \widetilde{p}$ for some $\widetilde{p} \in (0,1)$. For instance, there might be only a set number of scholarships or perks to be given out. The only term affected by the design in (ref) is then $\beta_3 \cdot \overline{xp}$, as pointed out by owen:vari:2020. For $\beta_3>0$, the short term average value per customer grows with $\overline{xp}$ and we would want that value to be large. Similar functionals are also commonly studied as regret functions in bandit problems goldenshlugerzeevi2013, metelkinapronzato2017.
We are now ready to formulate the tie-breaker design problem as the following constrained optimization problem. Given real values $x_1\leqslant x_2\leqslant \cdots\leqslant x_n$:
for some constants $\widetilde{p}$ and $\widetilde{xp}$. The first equality constraint in (ref) is a budget constraint due to the cost of treatment and the second constraint is on the short term gain mentioned above. We consider two different sets $\mathcal{A}$ in detail. The first is $[0,1]^n$. The second is $\{ \boldsymbol{p}\in[0,1]^n\mid 0\leqslant p_1\leqslant p_2\leqslant \cdots \leqslant p_n\leqslant 1\}$ which requires treatment probabilities to be non-decreasing in the running variable $x$. Such a monotonicity constraint prevents more qualified students from having a lower chance of getting a scholarship than less qualified ones or more loyal customers having a lower chance for a perk than others. It also eliminates perverse incentives for subjects to lower their $x_i$. To our knowledge, such a monotonicity constraint has not been received much attention in the optimal design literature, though it is enormously appealing in our motivating applications.
When the efficiency criterion $\psi(\cdot)$ is concave in $\boldsymbol{p}$, then a solution to (ref) can be found numerically via convex optimization, as mvtiebreaker do for vector valued $x_i$, and as metelkinapronzato2017 mention for a similar problem. However, our particular setting with univariate $x_i$ is tractable enough to provide a simple yet complete analytical characterization of the optimal $p_i$, even if the efficiency criterion is not concave. We show, under general conditions, that we can always find optimal treatment probabilities that are piecewise constant in $x$, with the number of pieces small and independent of $n$.
There is a well-developed literature for optimal experiment design in the presence of multiple objectives. Early examples of a constrained optimization problem of the form (ref) were designed to account for several of the standard efficiency objectives simultaneously stigler1971,lee1987,lee1988. lauter1974,lauter1976 proposed maximizing a convex combination of efficiency objectives, a practice now typically referred to as a “compound” design approach. It is now well known cookwong1994,clydechaloner1996 that in many problems with concave objectives, optimal constrained and compound designs are equivalent. In this paper, we provide another approach to reduce the constrained problem (ref) to a compound problem that can handle the monotonicity constraint. At the same time, we provide simple ways to compute our optimal designs that are based directly on the parameters $\widetilde{p}$ and $\widetilde{xp}$ in our constrained formulation (ref), and do not require specifying the Lagrange multipliers appearing in the corresponding compound problem. Those Lagrange multipliers involve ratios of information gain to economic gain where each of those quantities is only known up to a multiplicative constant.
Problems similar to (ref) have received significant attention in the sequential design of clinical trials. Biased-coin designs, beginning with the simple procedure of efron1971, have been developed as a compromise between treatment balance and randomization; see atkinson2014 for a review. Covariate-adaptive biased-coin designs often replace the balance objective with an efficiency criterion such as D-optimality atkinson1982, rosenbergersverdlov2008. Response-adaptive designs also optimize for some efficiency objective but simultaneously seek to minimize the number of patients receiving the inferior treatment for ethical reasons hurosenberger2006. Various authors such as bandyopadhyaybiswas2001 and huetal2015 propose sequential designs to effectively navigate this trade-off. When they also account for covariate information, they are called covariate-adjusted response-adaptive (CARA) designs zhangetal2007, zhanghu2009.
In the CARA literature especially, there has been significant recent interest in optimal design for nonlinear models sverdlovetal2013,metelkinapronzato2017, biswasbhattacharya2018. Unlike optimal designs in linear models such as (ref), designs in nonlinear models can typically only be locally optimal, meaning that their optimality depends on the values of the unknown parameters chernoff1953. While we may be able to obtain increasingly reliable estimates of these parameters over time in sequential settings, in non-clinical settings subjects typically enter a tie-breaker study non-sequentially, i.e., we know the running variables for all subjects before designing the experiment. In these applications --- such as measuring the impact of a scholarship on future educational attainment --- it can take several years to collect a single set of responses on which to compute a parameter estimate. Locally optimal designs are therefore of limited utility in this setting, and so we focus on optimal design under the linear model (ref) in a non-sequential setting, which already presents a sufficient challenge.
The existing literature on problems like (ref) typically considers the running variable $x$ to be random. For example, Section 7 of owen:vari:2020 study tie-breaker designs under the assumption that the running variable is either uniform or Gaussian, and exactly half the subjects are to be treated. They consider the typical three level tie-breaker design where subjects with running variable $x$ above some threshold $\Delta$ always get the treatment, subjects with running variable below $-\Delta$ never get the treatment, and the remaining subjects are randomized into treatment with probability $1/2$. They find that a $c$-optimality criterion of statistical efficiency is monotonically increasing in the width $\Delta$ of the randomization window, with the RCT ($\Delta\to\infty$) being most efficient and the RDD ($\Delta=0$) least efficient. Conversely, the short-term gain is decreasing in $\Delta$. They also show the three level design is optimal for any given level of short-term gain. In this article we show strong advantages to moving away from that three level design when the running variable is not symmetric, or we cannot treat half of the subjects.
metelkinapronzato2017 studied a further generalization of the optimal tie-breaker design problem, motivated by CARA designs. In particular, their Example 1, an illustration of their Corollary 2.1, is similar\footnote{There are minor differences such as the lack of a treatment fraction constraint, an inequality constraint on short-term gain as opposed to an equality constraint, and the use of a common intercept for the treated and untreated individuals, i.e., assuming that $\beta_2=0$ in (ref).} to a random-$x$ generalization of (ref). Crucially, however, the proof for their Corollary 2.1 does not generalize to the case where we require the treatment probabilities be monotone. Even without the monotonicity constraint, we provide a sharper characterization of the solutions to our more specific problem (Section (ref)).
We introduce a random-$x$ generalization of the problem (ref) in Section (ref). We show it encompasses both (ref) and the problem studied by owen:vari:2020 as special cases. Then, Section (ref) presents the main technical results characterizing the solutions to this more general problem. In particular, Theorem (ref) shows that, under the monotonicity constraint, there always exists a solution to (ref) corresponding to a simple, two-level stratified design where all subjects with running variable below some threshold $t'$ have the same probability of treatment, all subjects with running variable above $t'$ have an identical, higher probability of treatment, and those subjects with $x=t'$ have a treatment probability between these two values. Section (ref) then presents some results on the trade-off between a $D$-optimality efficiency criterion and short-term gain for these optimal designs; examples of this trade-off for some specific running variable distributions are given in Section (ref). That section also includes a fixed-$x$ application based on Head Start, a government assistance program for low-income children. It also shows how to compute our optimal designs when $x$ is either fixed or random. Finally, Section (ref) provides summarizes the main results.
Our random-$x$ generalization of (ref) assumes the running variables $x_i$ are samples from a common distribution $F$, which we hereafter identify with the corresponding cumulative distribution function. It considers an information matrix that averages over both the random treatment assignments and randomness in the running variables. This allows us to characterize optimal designs for an as yet unobserved set of running variable values, when their distribution is known.
Before presenting the random-$x$ tie-breaker design problem, we briefly review the standard setting of optimal design in multiple linear regression models; see e.g. atkinsonetal2007 for further background. In the simplest case, the user assumes the standard linear model $y_i = \bm{x}_i^{\mathsf{T}} \bm{\beta} + \varepsilon_i$ with the goal of selecting covariate values $\bm{x}_1,\dots,\bm{x}_n \in \mathbb{R}^p$ to optimize an efficiency criterion that is a function of the information matrix $\mathcal{X}^\mathsf{T} \mathcal{X}$, where $\mathcal{X} \in \mathbb{R}^{n \times p}$ is the design matrix with $i$-th row $\bm{x}_i^{\mathsf{T}}$. Perhaps the most common such criterion is D-optimality, which corresponds to maximizing $\log(\det(\mathcal{X}^\mathsf{T} \mathcal{X}))$. Another popular choice is $c$-optimality, which minimizes $c^\mathsf{T}(\mathcal{X}^\mathsf{T}\mathcal{X})^{-1}c$ for some choice of $c \in \mathbb{R}^p$. This can be interpreted as minimizing $\mathrm{Var}(c^{\top}\hat{\beta} \mid \mathcal{X})$, where $\hat{\beta}$ is the ordinary least squares estimator. The study of optimal design is often simplified by the use of design measures. A design measure $\xi$ is a probability distribution from which to generate the covariates $\bm{x}_i$. The relaxed optimal design problem involves selecting a design measure $\xi$ instead of a finite number of covariate values $\bm{x}_1,\ldots,\bm{x}_n$. The objective is to optimize for the desired functional of the expected information matrix $\mathcal{I}(\xi) \equiv \mathbb{E}_{\xi}[\mathcal{X}^{\mathsf{T}}\mathcal{X}]$ over some space $\Xi$ of design measures $\xi$. For instance, a design measure $\xi^*$ is D-optimal (for the relaxed problem) if $\xi^* \in \operatorname*{arg\,max}_{\xi \in \Xi} \det(\mathcal{I}(\xi))$, and $c$-optimal if $\xi^* \in \operatorname*{arg\,min}_{\xi \in \Xi} c^{\top}\mathcal{I}(\xi)^{-1}c$. The original optimal design problem restricts $\Xi$ to only consist of discrete probability distributions supported on at most $n$ distinct points with probabilities that are multiples of $1/n$.
For the tie-breaker design problem, our regression model (ref) includes both the running variables $x_i$ and the treatment indicators $z_i$ as covariates. But the experimenter does not have control over the entire joint distribution of $(x_i,z_i)$. The running variable is externally determined, so they can only specify the conditional distribution of the treatment indicator $z_i$ given the running variable $x_i$. This conditional distribution is specified by a design function $p:\mathbb{R} \to [0,1]$ such that $p(x) \equiv \Pr(z_i = 1 \mid x_i=x)$. As mentioned above, we assume $x_i \sim F$ for a known, fixed distribution $F$. This allows us to drop subscripts $i$ when convenient. For any two design functions $p$ and $p'$ we say $p=p'$ whenever $\Pr_F(\{x:p(x)=p'(x)\})=1$. We only need a minimal assumption on $F$, which can be continuous, discrete, or neither:
The mean-centeredness part of Assumption (ref) loses no generality, due to the translation invariance of estimation under the two-line model (ref). All expectations involving $x$ hereafter omit the subscript $F$ from all such expectations with the implicit understanding that $x \sim F$.
The random-$x$ tie-breaker design problem is as follows:
Here $\widetilde{z}$ and $\widetilde{xz}$ are constants analogous to $\bar{p}$ and $\overline{xp}$, respectively, in (ref), $\mathcal{F}$ is a collection of design functions, and $\mathcal{I}(p)$ is the expected information matrix under the model (ref), averaging over both $x \sim F$ and $z \mid x \sim p$.
This problem can be viewed as a constrained relaxed optimal design problem under the regression model (ref) where the set $\Xi$ of allowable design measures is indexed by the design functions $p \in \mathcal{F}$.
To interpret the equality constraints in (ref) it is helpful to note that
for any $a \geqslant 0$ with $\mathbb{E}(|x|^a)<\infty$. In particular, for each positive integer $a$, there exists an invertible linear mapping $\varphi_a:\mathbb{R}^{a+1} \to \mathbb{R}^{a+1}$ that does not depend on the design $p$ and maps $(\mathbb{E}_p(z),\ldots,\mathbb{E}_p(x^a z))$ to $(\mathbb{E}(p(x)),\ldots,\mathbb{E}(x^ap(x)))$. For example, $\varphi_1(x,y)=(1/2)(1+x,y)$. Taking $a=0$ in (ref), we see the constraint $\mathbb{E}_p(z)=\widetilde{z}$ in (ref) is equivalent to requiring the expected proportion of subjects to be treated to be $(1+\widetilde{z})/2$. This proportion is typically determined by the aforementioned budget constraints. Taking $a=1$ in (ref) shows that the second constraint $\mathbb{E}_p(xz)=\widetilde{xz}$ in (ref) sets the expected level of short-term gain. In Section (ref), we provide some guidance on how to choose $\widetilde{xz}$ in practice.
From computing the expected information matrix $\mathcal{I}(p)$ in Section (ref), we will see the problem (ref) reduces to the finite-dimensional problem (ref) when $F$ is discrete, placing probability mass $n^{-1}$ on each of the known running variable values $x_1,\ldots,x_n$. Thus, to solve (ref) it suffices to solve the problem (ref) for any $F$ satisfying Assumption (ref), which must hold for any discrete distribution with finite support.
For convenience, we introduce some notation for certain forms of the design function $p$. We will commonly encounter designs of the form
for a set $A \subseteq \mathbb{R}$. Another important special case consists of two level designs
for treatment probabilities $0 \leqslant \ell \leqslant u \leqslant 1$ and a threshold $t\in \overline{\mathbb{R}}$. For example, $p_{0,1,t}$ is a sharp RDD with threshold $t$, while for any $t$, $p_{\theta,\theta,t}$ is an RCT with treatment probability $\theta$.
The condition $\ell \leqslant u$ ensures that $p(x)$ is nondecreasing in $x$; we refer to such designs as monotone. Under a monotone design, a subject cannot have a lower treatment probability than another subject with lower $x$. We also define a symmetric design to be one for which $p(-x)=1-p(x)$; for instance, $p$ might be the cumulative distribution function (CDF) of a symmetric random variable. Finally, the three level tie-breaker design from owen:vari:2020 is both monotone and symmetric and defined for $\Delta \in [0,1]$ by
when $F$ is the $\mathbb{U}(-1,1)$ distribution. Note that for all $\Delta$, $p_{3,\Delta}$ always treats half the subjects, i.e., $\widetilde{z}=0$. The generalization to other $\widetilde{z}$ and running variable distribution functions $F$ is
where $a(\widetilde{z},\Delta) = F^{-1}((1-\widetilde{z})/2-\Delta)$ and $b(\widetilde{z},\Delta)=F^{-1}((1-\widetilde{z})/2+\Delta)$.
Before studying optimal designs, we impose lower and upper bounds on the possible short-term gain constraints $\widetilde{xz}$ to consider, for each possible $\widetilde{z} \in (-1, 1)$. For an upper bound we use $\widetilde{xz}_{\max}(\widetilde{z})$, the maximum $\widetilde{xz}$ that can be attained by any design function $p$ satisfying the treatment fraction constraint $\mathbb{E}_p(z)=\widetilde{z}$. It turns out that this upper bound is always uniquely attained. If the running variable distribution $F$ is continuous, it is uniquely attained by a sharp RDD. We remind the reader that uniqueness of a design function satisfying some property means that for any two design functions $p$ and $p'$ with that property, we must have $\Pr(p(x)=p'(x))=1$ under $x \sim F$.
By symmetry, the design that minimizes $\mathbb{E}_p(xz)$ over all designs $p$ with $\mathbb{E}_p(z)=\widetilde{z}$ is $p_{1,0,s}$ where $s = F^{-1}((1+\widetilde{z})/2)$. Notice that $\mathbb{E}_{p_{1,0,s}}(xz) = \widetilde{xz}_{\min}(\widetilde{z}) \equiv 2\mathbb{E}(x\bm{1}(x < s)) < 0$. We impose a stricter lower bound of $\widetilde{xz} \geqslant 0$ in the context of problem (ref). This is motivated by the fact that the running variable $x$ has mean 0 (Assumption (ref)), meaning that $\mathbb{E}_p(xz)=0$ whenever the design function $p$ is constant, corresponding to an RCT. Designs with $\widetilde{xz} < 0$ exist for all $\widetilde{z} \in (-1,1)$ but would not be relevant in our motivating applications, as they represent scenarios where subjects with smaller $x$ are more preferentially treated than in an RCT. We hence define the feasible input space $\mathcal{J}$ by
Any design function $p$ for which the moments $(\mathbb{E}_p(z),\mathbb{E}_p(xz))$ lie within the feasible input space $\mathcal{J}$ is referred to as an input-feasible design function.
If the design $p$ is input-feasible, we can write $\mathbb{E}_p(xz) = \delta \cdot \widetilde{xz}_{\max}(\mathbb{E}_p(z))$ for some $\delta \in [0,1]$. The parameter $\delta$ corresponds to the amount of additional short-term gain attained by the design $p$ over an RCT, relative to the amount of additional short-term gain attained by the generalized RDD $p_{\mathbb{E}_p(z)}$ that treats the same proportion of subjects as $p$. For instance, $\delta=0.4$ means that the design $p$ has a short-term gain that is 40% of the way from that of an RCT to the maximum attainable short-term gain under the treatment fraction constraint.
We now explicitly compute the expected information matrix
where \[ C =
\quadand\quad D =
\] and we have omitted the dependence of the expectations on the design $p$ for brevity. We emphasize that $\mathcal{I}$ depends on $F$ as well, though the experimenter can only control $p$. Furthermore, when $F = (1/n)\sum_{i=1}^n \delta_{x_i}$ and the running variable values $x_1,\dots,x_n$ are mean-centered, the expected information matrix $\mathcal{I}(p)$ is precisely the fixed-$x$ information matrix $\mathcal{I}_n(p_1,\ldots,p_n)$, identifying $p_i \equiv p(x_i)$. This shows that indeed, the random-$x$ problem (ref) is strictly more general than the fixed-$x$ problem (ref). Equation (ref) also shows that any efficiency objective $\Psi(\mathcal{I}(p))$ only depends on the treatment indicators $z$ through their marginal distributions conditional on $x$, and not their joint distribution. In the fixed-$x$ setting, this makes it easier to obey an exact budget constraint $n^{-1}\sum_{i=1}^n z_i = \widetilde{p}$ by stratification. For instance, given five subjects with $p_i=0.4$ we could randomly treat exactly two of them, instead of randomizing each subject independently and possibly going over budget.
While we will characterize solutions to the optimal design problem (ref) for any continuous efficiency criterion $\Psi(\cdot)$, in Section (ref) we will prove some additional results for the $D$-optimality criterion $\Psi_D(\cdot) = \log(\det(\cdot))$. We show that $D$-optimality is of particular interest in this setting, as it happens to correspond exactly with the $c$-optimality efficiency criterion of owen:vari:2020. They aim to minimize the asymptotic variance of $\hat{\beta}_3$ and do so by observing that if $\mathcal{I}$ is invertible, then when $(x_i,z_i)$ are independent, by the law of large numbers \[ n\textnormal{Var}(\hat{\beta}\!\mid\! \mathcal{X}) = n (\mathcal{X}^\mathsf{T}\mathcal{X})^{-1}\, \stackrel{\mathrm{a.s.}}{\to}\, \mathcal{I}^{-1}. \] We have assumed $\sigma^2=1$ WLOG as the $D$-optimal design does not depend on $\sigma^2$. Then by standard block inversion formulas
where
Equation (ref) shows that minimizing the asymptotic conditional variance of $\hat{\beta}_3$ is equivalent to maximizing $\textnormal{Eff}(p) := \det(M(p))/M_{11}(p)$, which under the present formalization of the problem is further equivalent to $c$-optimality for the expected information matrix (ref) with $c=(0,0,0,1)^{\top}$. The following result shows that $M_{11}(p)>0$ for any input-feasible design $p$. It follows that $\textnormal{Eff}(p)$ is always well-defined and nonnegative for any input-feasible design.
On the other hand, because $\mathcal{I}$ is the expected value of a positive semi-definite rank one matrix, it is also positive semi-definite. Thus \[ 0 \leqslant \det(\mathcal{I}) = \det(D)\det(M) = \mathbb{E}(x^2)\det(M). \] This shows that $\det(M) \geqslant 0$ with inequality iff $\mathcal{I}$ is invertible. Additionally, since $M_{11}$ only depends on $p$ through $\mathbb{E}_p(z)$ and $\mathbb{E}_p(xz)$, any two input-feasible designs $p$ and $p'$ satisfying the equality constraints in (ref) must have $M_{11}(p)=M_{11}(p')$. It follows that the solutions to (ref) under the $D$-optimality criterion $\Psi(\mathcal{I}(p)) = \Psi_D(\mathcal{I}(p))$ and the $c$-optimality criterion $\Psi(\mathcal{I}(p))=\textnormal{Eff}(p)$ must be identical whenever $(\widetilde{z},\widetilde{xz}) \in \mathcal{J}$.
To solve the constrained optimization problem (ref), we begin by observing that the expected information matrix $\mathcal{I}(p)$, computed in (ref), only depends on the design function $p$ through the quantities $\mathbb{E}_p(z)$, $\mathbb{E}_p(xz)$, and $\mathbb{E}_p(x^2z)$. Then the same is true for any efficiency objective $\Psi(\mathcal{I}(p))$. Consequently for any continuous $\Psi$ we can write $\Psi(\mathcal{I}(p))) = g_{\Psi}(\mathbb{E}_p(z),\mathbb{E}_p(xz),\mathbb{E}_p(x^2z))$ for some continuous $g_{\Psi}:\mathbb{R}^3 \to \mathbb{R}$ that may depend on the running variable distribution $F$.
Fixing $(\widetilde{z},\widetilde{xz}) \in \mathcal{J}$ and the set $\mathcal{F}$ of permissible design functions, we say a feasible design $p \in \mathcal{F}$ is one that satisfies the equality constraints in (ref), i.e. $\mathbb{E}_p(z)=\widetilde{z}$ and $\mathbb{E}_p(xz)=\widetilde{xz}$. Thus, the efficiency criterion $\Psi(\mathcal{I}(p))$ can only vary among feasible designs $p$ through the single quantity $\mathbb{E}_p(x^2z)$. Furthermore, any two feasible designs $p$ and $q$ with $\mathbb{E}_p(x^2z)=\mathbb{E}_q(x^2z)$ must have the same efficiency. Thus, we can break down the problem (ref) into two steps. First, we find a solution
where
is the set of values of $\mathbb{E}_p(x^2z)$ attainable by some feasible design $p \in \mathcal{F}$. Then we must find a feasible design $p \in \mathcal{F}$ that satisfies $\mathbb{E}_p(x^2z)=\widetilde{x^2z}^*(\widetilde{z},\widetilde{xz};\Psi)$.
The next result shows that when $\mathcal{F}$ is convex, $I_{\mathcal{F}}(\widetilde{z},\widetilde{xz})$ is an interval. For our two choices of $\mathcal{F}$ of interest, Propositions (ref) and (ref) will show it is a closed interval, so (ref) will always have a solution when $\Psi(\cdot)$ is continuous.
The endpoints of $I_{\mathcal{F}}(\widetilde{z},\widetilde{xz})$ can be computed as the optimal values of the following constrained optimization problems:
Given solutions $p_{\max}$ and $p_{\min}$ to the problems (ref), Lemma (ref) shows that the design
solves the problem (ref) for $\lambda = (\mathbb{E}_{p_{\max}}(x^2z)- \widetilde{x^2z}^{*}(\widetilde{z},\widetilde{xz};\Psi))/(\mathbb{E}_{p_{\max}}(x^2z)-\mathbb{E}_{p_{\min}}(x^2z))$. If $\mathbb{E}_{p_{\max}}(x^2z)=\mathbb{E}_{p_{\min}}(x^2z)$ then all feasible designs $p$ have the same efficiency, so any one of them is optimal.
The remainder of this section is concerned with characterizing the solutions to the problems (ref) for two specific choices of design function classes $\mathcal{F}$: the set of all measurable functions into $[0,1]$, and the set of all such monotone functions. For these two choices of $\mathcal{F}$, solutions $p_{\max}$ and $p_{\min}$ exist for any $(\widetilde{z},\widetilde{xz}) \in \mathcal{J}$ and are unique. Our argument uses extensions of the Neyman-Pearson lemma neymanpearson1933 in hypothesis testing. These extensions are in dantzigwald1951, whose two authors discovered the relevant results independently of each other. We use a modern formulation of their work, adapting the presentation by lehmannromano2005:
Lemma (ref) equates a constrained optimization problem (item (ref)) and a compound optimization problem (item (ref)). Unlike typical equivalence theorems, it does not require $\mathcal{F}$ to be the set of all measurable design functions, and uses an entirely different proof technique. Following whittle1973, equivalence theorems in optimal design are now popularly proven using the concept of Fr\'{e}chet derivatives on the space of design functions (measures). However, such approaches often do not apply when $\mathcal{F}$ is restricted to be the set of all monotone design functions. Most relevant to our problem, the proof of Corollary 2.1 in the supplement of metelkinapronzato2017 involves Fr\'{e}chet derivatives in the direction of design functions supported at a single value of $x$, which are not monotone. However, the use of Lemma (ref) requires an objective linear in $p$, where typical equivalence theorems only require concavity.
We now solve the design problem (ref) in the case that $\mathcal{F}$ is the set of all measurable functions $p:\mathbb{R}\to[0,1]$. We first explain how the results of metelkinapronzato2017 do not adequately do so already. Identifying our design functions $p$ with their design measures $\xi$, Corollary 2.1 of metelkinapronzato2017 does not provide any information about what an optimal solution $p_{\textnormal{opt}}(x)$ to (ref) would be for values of $x$ where $G_1(p_{\textnormal{opt}}(\cdot);x)=G_2(p_{\textnormal{opt}}(\cdot);x)$. Here $G_1$ and $G_2$ are quantities derived from the aforementioned Fr\'{e}chet derivatives, depending on the constraints $\widetilde{z}$ and $\widetilde{xz}$. Unfortunately, this lack of information about $p_{\textnormal{opt}}$ holds for all $x$ both in their Example 1 and in our setting. Their example skirts this limitation by noting some moment conditions on $p_{\textnormal{opt}}$ implied by the equality $G_1=G_2$ when the running variable is uniform and the efficiency criterion is $D$-optimality, and then manually searching for some parametric forms of $p_{\textnormal{opt}}$ for which it is possible to satisfy these conditions. By contrast, the results in this section apply Lemma (ref) with $\mathcal{F}$ the set of all design functions, and show a simple stratified design function is always optimal for any running variable distribution $F$ and continuous efficiency criterion. This enables optimal designs to be systematically and efficiently constructed (Section (ref)).
We will apply Lemma (ref) with $m=2$ constraints pertaining to $h_1(x)=1$ and $h_2(x)=x$. Our objective function is based on $h_3(x)=x^2$. When the running variable distribution $F$ is continuous, recalling the notation (ref) the solutions to (ref) take the forms $p_{\max} = p_{[a_1,a_2]^c}$ and $p_{\min}=p_{[b_1,b_2]}$ for some intervals $[a_1,a_2]$ and $[b_1,b_2]$.
As we will see in Section (ref), when $\widetilde{z} < 0$ we frequently encounter $p_{\textnormal{opt}}=p_{\max}$ under $D$-optimality. In this case it is intuitive that there is an efficiency advantage to strategically allocate the rare level $z=1$ at both high and low $x$, compared to a three level tie-breaker. But such a design is usually unacceptable in our motivating problems. We will thus constrain $\mathcal{F}$ to the set of monotone design functions in Section (ref).
Before doing that, we present an alternative solution to (ref) assuming the running variable distribution has an additional moment. This result shows that when the running variable is continuous, an optimal design with no randomization always exists. However, randomized assignment becomes essential once we restrict our attention to monotone designs in Section (ref), as the only non-randomized monotone designs are generalized RDDs (Remark (ref)).
We now apply Lemma (ref) to solve (ref) in the case of principal interest, where $\mathcal{F}$ is the set of all monotone design functions. Note that the lower bound $\widetilde{xz} \geqslant 0$ that we imposed in Section (ref) does not exclude any monotone designs. If $p(x)$ is monotone then $x$ and $z$ necessarily have a nonnegative covariance $\mathbb{E}_p(xz)$.
Our argument follows the outline of Section (ref). Suppose that $p_{\max}^{\dag}$ and $p_{\min}^{\dag}$ are solutions to (ref) with $\mathcal{F}$ the set of monotone design functions, which we distinguish from the optimal designs $p_{\max}$ and $p_{\min}$ of Section (ref). As $\mathcal{F}$ is convex, Lemma (ref) applies, and thus a solution $p_{\textnormal{opt}}^{\dag}$ to (ref) is given by (ref), replacing $p_{\max}$ and $p_{\min}$ by $p_{\max}^{\dag}$ and $p_{\min}^{\dag}$, respectively. Note that $\widetilde{x^2z}^*(\widetilde{z},\widetilde{xz})$ may differ from its value in Section (ref) since $\mathcal{F}$ has changed.
We now characterize the designs $p_{\max}^{\dag}$ and $p_{\min}^{\dag}$. As in Proposition (ref), these designs always exist and are unique for any $(\widetilde{z},\widetilde{xz}) \in \mathcal{J}$. When $F$ is continuous, these are monotone two level designs $p_{\max}^{\dag}=p_{\ell,1,t}$ and $p_{\min}^{\dag}=p_{0,u,s}$ as defined in (ref). For general $F$ the designs $p_{\max}^{\dag}$ and $p_{\min}^{\dag}$ may differ from these designs at at the single discontinuity.
Analogous to Theorem (ref), if we assume the running variable has a third moment then we have a solution to (ref) of a simpler form than (ref). When the running variable $x$ is continuous, this solution will be a two level design $p_{\ell',u',t'}$ for some $0 \leqslant \ell' \leqslant u' \leqslant 1$. In general, when $x$ is not continuous, we may need a different treatment probability at the discontinuity $t'$.
In general, the optimal designs derived in Theorems (ref) and (ref) are not unique when $\widetilde{x^2z}^*(\widetilde{z},\widetilde{xz},\Psi)$ is not on the boundary of $I_{\mathcal{F}}(\widetilde{z},\widetilde{xz})$. For example, in nondegenerate cases the solution $p_{\textnormal{opt}}^{\dag}$ in (ref) typically has two levels, while the solution in (ref) (with the monotonicity constraint) will have three levels. As another example, the three-level tiebreaker found by owen:vari:2020 to be optimal when $F$ is uniform and $\widetilde{z}=0$ does not take the form (ref) whenever $\widetilde{xz} < \widetilde{xz}_{\max}(0)$. Conversely, Propositions (ref) and (ref) guarantee a unique optimal design when $\widetilde{x^2z}^*(\widetilde{z},\widetilde{xz},\Psi)$ is one of the endpoints of $I_{\mathcal{F}}(\widetilde{z},\widetilde{xz})$.
As discussed in Section (ref), owen:vari:2020 showed that when $\widetilde{z}=0$ and $F \sim \mathbb{U}(-1,1)$, the efficiency (under their criterion $\textnormal{Eff}(\cdot)$) of the three level tie-breaker (ref) is monotonically increasing in the width $\Delta$ of the randomization window. As $\Delta$ is a strictly decreasing function of $\widetilde{xz}$, and the three level tie-breaker solves (ref) for all $\widetilde{xz}$, they conclude that there is a monotone trade-off between short-term gain and statistical efficiency. In other words, greater statistical efficiency from an optimal design requires giving up short-term gain.
We now extend these results to general $\widetilde{z}$ and other running variable distributions. Hereafter $p_{\textnormal{opt}}=p_{\textnormal{opt};\widetilde{z},\widetilde{xz}}$ denotes an optimal design without the monotonicity constraint, to be contrasted with $p_{\textnormal{opt}}^{\dag}$ of Section (ref). Note we have made the dependence of these designs on $(\widetilde{z},\widetilde{xz}) \in \mathcal{J}$ explicit. We use the same efficiency criterion $\textnormal{Eff}(\cdot)$ as owen:vari:2020. Recall this is a $c$-optimality criterion corresponding to the scaled asymptotic variance of the OLS estimate for $\beta_3$ in (ref), and equivalent to $D$-optimality for our problem (ref) by Section (ref).
It turns out, however, that the gain versus efficiency trade-off is no longer monotone under the monotonicity constraint. Indeed, our next theorem shows that whenever $\widetilde{z} \neq 0$, if $F$ is symmetric (or indeed, not extremely skewed), the fully randomized design $p_{\theta,\theta,0}$ is inadmissible for any $\theta \neq 1/2$, in the sense that there exists a different monotone design $p$ with $\mathbb{E}_p(z)=\widetilde{z}$ but both $\textnormal{Eff}(p) > \textnormal{Eff}(p_{\theta,\theta,0})$ and $\mathbb{E}_p(xz) > \mathbb{E}_{p_{\theta,\theta,0}}(xz)$. In other words, the RCT is no longer admissible under $\textnormal{Eff}(\cdot)$ when $\widetilde{z} \neq 0$.
In this section, we compute the optimal exploration-exploitation trade-off curves investigated in Section (ref) for several specific running variable distributions $F$. We can obtain large gains in efficiency under the criterion $\textnormal{Eff}(\cdot)$ by moving from the three level tie-breaker design to $p_{\textnormal{opt}}^{\dag}$, without sacrificing short-term gain. We see further (generally smaller) improvements when we remove the monotonicity constraint and move from $p_{\textnormal{opt}}^\dag$ to $p_{\textnormal{opt}}$.
To generate these curves we compute optimal designs $p_{\textnormal{opt};\widetilde{z},\widetilde{xz}}$ and $p_{\textnormal{opt};\widetilde{z},\widetilde{xz}}^{\dag}$ and evaluate their efficiency for various fixed $\widetilde{z} \in (-1,1)$ as we vary the short-term gain constraint $\widetilde{xz}$ over a fine grid covering $[0, \widetilde{xz}_{\max}(\widetilde{z})]$. For interpretability we write $\widetilde{xz}=\delta \cdot \widetilde{xz}_{\max}(\widetilde{z})$ and specify short-term gain with the normalized parameter $\delta \in [0,1]$, as discussed in Section (ref). When $F$ is continuous, solutions $p_{\textnormal{opt}}$ and $p_{\textnormal{opt}}^{\dag}$ to (ref) are computed by noting that we can write $p_{\max}=p_{[a_1,a_2]^c}$, $p_{\min}=p_{[b_1,b_2]}$, $p_{\max}^{\dag}=p_{\ell,1,t}$, and $p_{\min}^{\dag}=p_{0,u,s}$ by Propositions (ref) and (ref). Each of these designs has two unknown parameters that must be the unique solutions to the two feasibility constraints $\mathbb{E}_p(z)=\widetilde{z}$ and $\mathbb{E}_p(xz)=\widetilde{xz}$. Given these parameters, we can apply (ref) to compute $p_{\textnormal{opt}}$ and $p_{\textnormal{opt}}^{\dag}$. If $\lambda \notin \{0,1\}$ we could also get an optimal design of the form in Theorem (ref). First we compute $\widetilde{x^2z}^*(\widetilde{z},\widetilde{xz};\textnormal{Eff})$ via (ref), noting that the endpoints of $I_{\mathcal{F}}(\widetilde{z},\widetilde{xz})$ are $\mathbb{E}_{p_{\min}(\widetilde{z},\widetilde{xz}}(x^2z)$ and $\mathbb{E}_{p_{\max}(\widetilde{z},\widetilde{xz})}(x^2z)$. Then (ref) is simply maximizing a continuous function over a closed interval, so it can be handled by standard methods such as Brent's algorithm brent1973. Given $\widetilde{x^2z}^*(\widetilde{z},\widetilde{xz};\textnormal{Eff})$ we can then numerically search for $a_1$, $a_2$, $a_3$ such that $p_{\textnormal{opt}}=\bm{1} (x \leqslant a_1) + \bm{1}(a_2 \leqslant x \leqslant a_3)$ is feasible with $\mathbb{E}_{p_{\textnormal{opt}}}(x^2z)=\widetilde{x^2z}^*(\widetilde{z},\widetilde{xz};\textnormal{Eff})$. By Theorem (ref) such a solution will exist and be optimal. We can do a similar search for an optimal two level design $p_{\textnormal{opt}}^{\dag}$ under the monotonicity constraint, by Theorem (ref).
We begin with the case $F \sim \mathbb{U}(-1, 1)$. This is the distribution most extensively studied by owen:vari:2020, and allows closed form expressions for the parameters in $p_{\max}$, $p_{\min}$, $p_{\max}^{\dag}$, and $p_{\min}^{\dag}$, given in Table (ref). Figure (ref) shows plots of $\textnormal{Eff}(p)^{-1}$ versus $\widetilde{xz}$ for $\widetilde{z} \in \{0,-0.2,-0.5,-0.7\}$ under different designs: the three level tie-breaker $p_{3;\widetilde{z},\Delta}$, a globally optimal design $p_{\textnormal{opt};\widetilde{z},\widetilde{xz}}$, and an optimal monotone design $p_{\textnormal{opt};\widetilde{z},\widetilde{xz}}^{\dag}$. Since $F$ is symmetric, the curves would be identical if $\widetilde{z}$ were replaced with $-\widetilde{z}$.
As shown in owen:vari:2020, under the constraint $\widetilde{z}=0$ the three level tie-breaker is optimal for all $\delta$, and thus the three level tie-breaker, $p_{\textnormal{opt}}$, and $p_{\textnormal{opt}}^{\dag}$ all attain the optimal efficiency, as can be seen in the top left panel of Figure (ref). The proof of Theorem (ref) shows this would hold for any continuous, symmetric running variable distribution $F$. As $\widetilde{z}$ moves away from 0, however, we see that the three level tie-breaker becomes increasingly less efficient relative to both the optimal monotone design and the optimal design. At the same time, the range of short-term gain values $\widetilde{xz}$ attainable by three level tie-breaker designs becomes smaller relative to the full range achievable by arbitrary designs. Note that Figure (ref) plots the reciprocal of the efficiency criterion $\textnormal{Eff}(\cdot)$, so that it can be interpreted as an asymptotic variance for $\hat{\beta}_3$ via (ref), and compared with the plots in owen:vari:2020.
Table (ref) extends Table 2 of owen:vari:2020, referring to a setting in which only 15% of subjects are to be treated ($\widetilde{z} = -0.7$). That table shows the inverse efficiency of the sharp RDD $p_{0,1,0.7}$ is 223.44, while the three level tie-breaker $p_{3;-0.7,0.05}$ reduces this by about 40% to $137.56$, at the cost of around 2% of the short-term gain of the sharp RDD over the RCT. Then $\textnormal{Eff}(p_{\textnormal{opt}}^{\dag})^{-1}=54.90$ and $\textnormal{Eff}(p_{\textnormal{opt}})^{-1}=42.37$, further improving efficiency for designs $p_{\textnormal{opt}}$ and $p_{\textnormal{opt}}^{\dag}$ achieving the same short term gain as the three level tie-breaker. For this example, we can directly compute with (ref) that $p_{\textnormal{opt};-0.7,0.25}^{\dag} = p_{\max;-0.7,-0.25}^{\dag} = p_{\ell,1,t}$ is the unique optimal monotone design, where by Table (ref), $\ell=0.0034$ and $t = 0.7059$. In other words, the unique optimal monotone tie-breaker design deterministically assigns treatment to the top 14.7%, and gives the other subjects an equal, small (0.34%) chance of treatment.
A limitation of this analysis is that in many practical settings, the two-line regression model will not fit very well over the entire range of $x$ values. In that case the investigator might use a narrower data range, essentially fitting a less asymmetric two-line model, as illustrated in owen:vari:2020. This is equivalent to using a local linear regression with a rectangular “boxcar” kernel. In this setting, we know from Figure (ref) that when the treatment proportion is not exactly 50%, we can always do better than the three level tie-breaker using monotone two level design. Even with a small asymmetry, e.g. 40% treatment ($\widetilde{z} = -0.2$), we see a noticeable efficiency increase between the three level tie-breaker and an optimal monotone design across all values of $\delta$.
Finally, consistent with the results of Section (ref), we observe in Figure (ref) that $\textnormal{Eff}(p_{\textnormal{opt}})$ decreases with the gain parameter $\delta$ for each $\widetilde{z}$, while near $\delta =0$, $\textnormal{Eff}(p_{\textnormal{opt}}^{\dag})$ increases with $\delta$ for all $\widetilde{z} \neq 0$. This clearly demonstrates the inadmissibility of the fully randomized design from Theorem (ref). For example, if we fix $\widetilde{z}=-0.5$ (so 25% of the subjects are to be treated), the fully randomized design $p_{0.25,0.25,0}=p_{\textnormal{opt};-0.5,0}^{\dag}$ has efficiency 0.25 and no short-term gain ($\delta=0$), while $p_{\textnormal{opt};-0.5,0.1}^{\dag}$ has higher efficiency (0.28) with short-term gain $\delta = 0.27 > 0$. However, if we remove the monotonicity constraint, by Theorem (ref) $p_{\textnormal{opt};-0.5,0}$ is the most efficient design over all attainable gain values, attaining efficiency 0.33 with $\delta=0$.
We now repeat the analysis of Section (ref) for a skewed running variable distribution $F$:
for $x \in (-2,\infty) = I$. This corresponds to a mean-centered Weibull distribution with shape parameter $0.5$ and scale parameter $1$. Figure (ref) shows the trade-off curves under this distribution $F$. We see, as expected by Theorem (ref), that once again the fully randomized design is inadmissible, even within the class of monotone designs, when $\widetilde{z} \neq 0$.
Another notable feature when $F$ is not symmetric is that the three level tie-breaker is no longer optimal, even in the balanced case $\widetilde{z}=0$. While the unconstrained optimal design attains the lower bound $\mathbb{E}(x^2z) = \widetilde{x^2z}^*(0,\widetilde{xz};\textnormal{Eff}) = 0$ for a wide range of short-term gains, Figure (ref) shows the three level tie-breaker does not, except in the case $\widetilde{z}=\widetilde{xz}=0$ corresponding to the RCT. In Figure (ref), we see the optimal design is over 100 times as efficient as the three level tiebreaker for sufficiently large $\delta$, even in the balanced setting $\widetilde{z}=0$. In the unbalanced treatment cases we also see a range of values for which optimal designs with and without the monotonicity constraint attain the same value of $\mathbb{E}(x^2z)$. In those situations there exists a globally optimal design that is also monotone.
We now illustrate how to compute optimal designs for the original fixed-$x$ problem (ref) using a real data example. ludwigmiller2007 used an RDD to analyze the impact of Head Start, a U.S.\ government program launched in 1965 that provides benefits such as preschool and health services to children in low-income families. When the program was launched, extra grant-writing assistance was provided to the 300 counties with the highest poverty rates in the country. This created a natural discontinuity in the amount of funding to counties as a function of $x$, a county poverty index based on the 1960 U.S. Census. The distribution of $x$ over $n=2{,}804$ counties is shown in Figure (ref). The data is made freely available by cattaneo2017.
If the government had deemed it ethical to somewhat randomize the 300 counties receiving the grant-writing assistance, it could have more efficiently estimated the causal impact of this assistance using our $p_{\textnormal{opt}}^{\dag}$, while still ensuring poorer counties are preferentially helped, and no county has a lower chance of getting the assistance than a more well-off county. As in the data example of klug:owen:2022:tr, we do not observe the potential outcomes, so we cannot actually implement such a design and compute any estimators. However, we can still study statistical efficiencies, which depend only on the expected information matrix $\mathcal{I}$.
We fix the treatment fraction at $300/2804$, corresponding to $\widetilde{z} \approx -0.79$. Varying the short-term gain constraint $\widetilde{xz}$ we seek to compute $p_{\max;\widetilde{z},\widetilde{xz}}^{\dag}$ and $p_{\min;\widetilde{z},\widetilde{xz}}^{\dag}$. We describe how to compute the former. Because $F$ is discrete it suffices to only consider discontinuity points $t \in \{x_1,\dots,x_n\}$ where $F$ places positive probability mass, as every design of the form of $p_{\max}^{\dag}$ in (ref) has a representation in that form with such $t$. Also, given the values of the discontinuity $t$ and $p(t)=\epsilon$, there is at most one value $\ell=\ell(t,\epsilon) \in [0,1]$ such that the resulting design $p$ in the form of $p_{\max}^{\dag}$ in (ref) satisfies the treatment fraction constraint $\mathbb{E}_p(z)=\widetilde{z}$. When such an $\ell$ exists for some $(t,\epsilon)$, call the corresponding design $p^{(t,\epsilon)}$ (note we suppress the dependence on $\widetilde{z}$). From Appendix (ref) we deduce that $\mathbb{E}_{p^{(t,\epsilon)}}(xz) < \mathbb{E}_{p^{(t',\epsilon')}}(xz)$ and $\ell(t,\epsilon) > \ell(t',\epsilon')$ if $t > t'$, or if $t=t'$ and $p(t)<p'(t)$. This shows we can efficiently find the unique $(t,\epsilon)$ so that $p^{(t,\epsilon)}$ satisfies the desired short-term gain constraint $\mathbb{E}_{p^{(t,\epsilon)}}(xz)=\widetilde{xz}$. In particular we compute $t = \max\{s \in \{x_1,\ldots,x_n\} \mid \mathbb{E}_{p^{(s,1)}}(xz) \geqslant \widetilde{xz} \}$ via a binary search on $\{x_1,\ldots,x_n\}$, then solve for $\epsilon$ to satisfy $\mathbb{E}_{p^{(t,\epsilon)}}(xz)=\widetilde{xz}$. Given sorted $x$, this entire procedure computes $p_{\max}^{\dag}$ in $O(n)$ operations, as for each $(t,\epsilon)$, $\ell(t,\epsilon)$ and $\mathbb{E}_{p^{(t,\epsilon)}}(xz)$ can be computed in constant time using (ref) given the partial sums $\{\sum_{i=1}^m x_i\}_{m=1}^n$.
After computing $p_{\min}^{\dag}$ with a similar approach, we can apply (ref) to compute an optimal design $p_{\textnormal{opt}}^{\dag}$. As in the continuous case, we can alternately obtain a solution $p_{\textnormal{opt}}^{\dag}$ of the form in Theorem (ref) by finding $\ell'$, $u'$, $t'$, and $\epsilon'$ such that $\mathbb{E}_p(z)=\widetilde{z}$, $\mathbb{E}_p(xz)=\widetilde{xz}$, and $\mathbb{E}_p(x^2z)=\widetilde{x^2z}^*(\widetilde{z},\widetilde{xz};\textnormal{Eff})$ for $p(x) \equiv \ell' \bm{1}(x < t') + \epsilon' \bm{1}(x=t') + \bm{1}(x >t')$. Unlike the continuous case, we now have 4 unknown parameters instead of 3. We can search for an optimal set of these parameters by looping through the finite possible values of $t'$ and then doing a univariate search for $\epsilon'$, noting that knowledge of $t'$ and $\epsilon'$ determines $\ell'$ and $u'$ by the equality constraint parameters $\widetilde{z},\widetilde{xz})$. We implemented this search, along with the procedure to compute $p_{\max}^{\dag}$ and $p_{\min}^{\dag}$ described above, in the R language R. The code is freely available online\footnote{https://github.com/hli90722/optimal_tiebreaker_designs}.
The right panel of Figure (ref) shows the inverse efficiency for the three level tie-breaker (ref) versus the best two level monotone design obtained by applying the above procedure to the $x_i$ in the Head Start data. It turns out that for these $x_i$ and our choice of $\widetilde{z}$, $p_{\max}^{\dag}$ is optimal for all $\widetilde{xz}$ (and hence the unique optimal design, by Proposition (ref)). We note that with a normalized short term gain $\delta \approx 0.958$, which corresponds to random assignment for about 150 counties in the 3-level tie-breaker, the optimal monotone two level design has inverse efficiency 0.030, compared to 0.050 for the three level tie-breaker. That is, confidence intervals for $\beta_3$ using the three level tie-breaker would be about 29% wider than for the optimal monotone two-level design, without additional short-term gain. The sharp RDD would give 62% wider intervals than the optimal monotone two-level design with only about 4.2% additional short-term gain.
Our results provide a thorough characterization of the solutions to a constrained optimal experiment design problem. Considering a linear regression model for a scalar outcome involving a binary treatment assignment indicator $z$, a scalar running variable $x$, and their interaction, we seek to specify a randomized treatment assignment scheme based on $x$ --- a tie-breaker design --- that optimizes a statistical efficiency criterion that is an arbitrary continuous function of the expected information matrix under this regression model. We have equality constraints on the proportion of subjects receiving treatment due to an external budget, and on the covariance between $x$ and $z$ due to a preference for treating subjects with higher values of $x$. Critically, our proof techniques, which deviate from those typically used to show equivalence theorems, enable an additional monotonicity constraint. This allows our results to handle the ethical or economic requirement that a subject cannot have a lower chance of receiving the treatment than another subject with a lower value of $x$.
In a setting where the running variable $x$ is viewed as random from some distribution $F$ --- and thus part of the randomness in the expected information matrix defining the efficiency criterion --- we prove the existence of constrained optimal designs that stratify $x$ into a small number of intervals and assign treatment with the same probability to all individuals within each stratum. In particular, with the monotonicity constraint that is essential in our motivating applications, we only need three strata, one of which only contains a single running variable value. We also provide strong conditions on which the optimal tie-breaker design is unique. We emphasize the generality of our results, which apply for any continuous efficiency criterion, any running variable distribution $F$ (subject only to weak moment existence conditions), and the full range of feasible equality constraints. The problem an investigator faces in practice, where there are a finite number of running variable values $x_1,\ldots,x_n$ known (hence non-random) at the time of treatment assignment, is a special case of our more general problem where $F$ is discrete and takes on values $x_1,\ldots,x_n$ with equal probability. This enables optimal designs to be easily computed in practice, as described in Section (ref).
We believe that this work provides a useful starting point to study optimal tie-breaker designs. For results on tie-breaker designs beyond the two line parametric regression, see mvtiebreaker for a multivariate regression context and klug:owen:2022:tr for local linear regression models with a scalar running variable.
This work was supported by the US National Science Foundation under grants IIS-1837931 and DMS-2152780. The authors would like to thank Kevin Guo, Dan Kluger, Tim Morrison, and several anonymous reviewers for helpful comments.