Inference under Covariate-Adaptive Randomization with Imperfect Compliance

Inference under Covariate-Adaptive Randomization with Imperfect Compliance

\relax \hypersetup{pageanchor=false}

\hypersetup{pageanchor=true}

\thispagestyle{empty}

KEYWORDS: Covariate-adaptive randomization, stratified block randomization, treatment assignment, randomized controlled trial, strata fixed effects, saturated regression, imperfect compliance.

JEL classification codes: C12, C14

\thispagestyle{empty}

\hypersetup{pageanchor=true} \setcounter{page}{1}

Introduction

This paper studies inference in a randomized controlled trial (RCT) with covariate-adaptive randomization (CAR) with a binary treatment. As in bugni/canay/shaikh:2018,bugni/canay/shaikh:2019, CAR refers to randomization schemes that first stratify according to baseline covariates and then assign treatment status so as to achieve “balance” within each stratum. As these references explain, CAR is commonly used to assign treatment status in RCTs in all parts of the sciences.\footnote{See rosenberger/lachin:2016 for a textbook treatment focused on clinical trials and duflo/glennerster/kremer:2007 and bruhn/mckenzie:2008 for reviews focused on development economics.}

In contrast to bugni/canay/shaikh:2018,bugni/canay/shaikh:2019, this paper allows for imperfect compliance. That is, the participants of the RCT can endogenously decide their treatment, denoted by $D$, which may or may not coincide with the assigned treatment status, denoted by $A$. This constitutes an empirically relevant contribution to this growing literature, as imperfect compliance is a common occurrence in many RCTs. For recent examples of RCTs that use CAR and have imperfect compliance, see angrist/lavy:2009, attanasio/kugler/meghir:2011, dupas/karlan/robinson/ubfal:2018, mcintosh/alegria/ordonez/zenteno:2018, somville/vandewalle:2018, among many others.

Our goal is to study the effect of the treatment on an outcome of interest, denoted by $Y$. We consider the potential outcome framework with $Y=Y(1)D + Y(0)(1-D)$, where $Y(1)$ denotes the outcome with treatment and $Y(0)$ denotes the outcome without treatment. We also consider a potential decision framework with $D = D(1)A + D(0)(1-A)$, where $D(1)$ denotes the decision when assigned to treatment and $D(0)$ denotes the decision when not assigned to treatment. In the context of imperfect treatment compliance, a key causal parameter of interest is the so-called local average treatment effect (LATE) introduced in angrist/imbens:1994, and given by

In words, the LATE is the average treatment effect for those individuals who decide to comply with their assigned treatment status, i.e., the “compliers”.\footnote{The definition of LATE presumes the presence of compliers in the population of interest. This will be formalized in Assumption (ref). Under imperfect compliance, the literature also considers other parameters, such as the intention to treat, defined as $\text{ITT}=E[Y|A=1]-E[Y|A=0]$, or the average treatment effect on the treated, defined as $\text{TOT}=E[ Y( 1) -Y( 0) |D(1)=D(0)=1]$. In this paper, we prefer the LATE over these alternative parameters. First, it can be shown that $\text{ITT} = \text{LATE} \times P(D(0)=0,D(1)=1)$. Unlike the ITT, the LATE represents the average treatment effect for a subset of the population (i.e., the compliers), which makes it preferable. Second, it can be shown that $\text{TOT} = \text{LATE}$ if there are no “always takers”, i.e., individuals who adopt the treatment regardless of the assignment. However, if always takers are present, the TOT is not identified under our assumptions.} As in bugni/canay/shaikh:2019, we consider inference on the LATE based on simple linear regressions. In Section (ref), we study the properties of an estimator of the LATE derived from a “fully saturated” instrumental variable (IV) linear regression, i.e., a linear regression of the outcome on all indicators for all strata and their interaction with the treatment decision, where the latter is instrumented with the treatment assignment. We show that its coefficients can be used to consistently estimate the LATE under very general conditions. We show that the proposed LATE estimator is asymptotically normal, and we characterize its asymptotic variance in terms of primitives of the problem. As expected, we show that the asymptotic variance is different from the one under perfect compliance derived in bugni/canay/shaikh:2018,bugni/canay/shaikh:2019. We provide consistent estimators of these new standard errors and asymptotically exact hypothesis tests. In addition, we show that the results of the fully saturated regression can be used to estimate all of the primitive parameters of the problem.

In the special case when the target proportion of units being assigned to each of the treatments does not vary across strata, we also consider two other regression-based estimators of the LATE. Section (ref) proposes an estimator of the LATE based on the “strata fixed effects” IV linear regression, i.e., a linear regression of the outcome on indicators for all strata and the treatment decision, where the latter is instrumented with the treatment assignment. In turn, Section (ref) proposes an estimator of the LATE based on a “two sample regression” IV linear regression, i.e., a linear regression of the outcome on a constant and the treatment decision, where the latter is instrumented by the treatment assignment. We show that the LATE estimators produced by both of these regressions are asymptotically normal, and we characterize their asymptotic variances in terms of primitives of the problem. We show how to estimate the corresponding standard errors and generate asymptotically exact hypothesis tests by relying on results from the fully saturated regression.

Sections (ref), (ref), and (ref) characterize the asymptotic variance of the three regression-based IV estimators of the LATE in terms of the primitive parameters of the problem. This allows us to understand how the parameters of the RCT with CAR affect the standard errors. In principle, this information can be used to propose strategies to minimize the asymptotic variance of the LATE estimator of a hypothetical RCT, possibly with the aid of data from a pilot RCT. We consider this RCT design problem in Section (ref), and we establish several interesting results. First, we show that it is optimal to use a CAR method that imposes the highest possible level of “balance” on the treatment assignment within each stratum, such as stratified block randomization.\footnote{This finding extends the results obtained by bugni/canay/shaikh:2018,bugni/canay/shaikh:2019 to the case of imperfect compliance.} Within this class of CAR methods, our second result in Section (ref) establishes that the asymptotic variance of the estimators of the LATE cannot increase when the collection of strata becomes finer.\footnote{See bai:2022 for analogous results in the context of perfect compliance.} In addition, we show how to use the data from a pilot RCT to estimate the asymptotic variance that would result from using a finer set of strata in a hypothetical RCT. Our third and final result in Section (ref) provides an expression for the optimal treatment propensity in a hypothetical RCT in terms of its primitive parameters. To exploit this result in practice, we provide a consistent estimator of the optimal treatment propensity based on data from a pilot version of the RCT.

In recent work, ansel/hong/li:2018 also consider inference for the LATE in RCTs with a binary treatment and imperfect compliance. While most of their paper focuses on the case in which treatment assignment is done via simple random sampling, they consider inference on RCTs with CAR in Section 4. In contrast, our paper is entirely focused on RCTs with CAR. This allows us to tailor our assumptions to the problem under consideration, and it enables us to give more detailed formal arguments.\footnote{An example of this is the proof of Lemma (ref), where we modify the arguments in bugni/canay/shaikh:2018 to allow for the presence of imperfect compliance.} Second, ansel/hong/li:2018 consider IV regressions without fully specifying the set of covariates in these regressions. Consequently, they derive the asymptotic variance of their LATE estimators in terms of high-level expressions. In contrast, we fully specify the covariates in our IV regressions according to the specification typically used by practitioners. This allows us to obtain explicit expressions for the asymptotic variance of our LATE estimators, detailing which of these depend on the underlying population and which are chosen by the researcher implementing the RCT. In this sense, our expressions reveal the underlying forces determining the asymptotic variance and enable researchers to choose the RCT parameters to improve the efficiency of their estimators. We consider the topic of RCT design in Section (ref).

The remainder of the paper is organized as follows. In Section (ref), we describe the setup of the inference problem and we specify our assumptions. The next three sections consider the problem of inference on the LATE based on different IV regression models. Section (ref) considers the “fully saturated” IV linear regression, Section (ref) considers the “strata fixed effects” IV linear regression, and Section (ref) considers the “two-sample regression” IV linear regression. In each one of these sections, we propose a consistent estimator of the LATE, we characterize its asymptotic distribution, and we propose a consistent estimator of their standard errors and asymptotically valid hypothesis tests. In Section (ref), we consider the problem of designing a hypothetical RCT with CAR based on data from a pilot RCT with CAR. In Section (ref), we study the finite sample behavior of our hypothesis tests based via Monte Carlo simulations. Section (ref) illustrates the practical relevance of our results by an empirical application based on the RCT in dupas/karlan/robinson/ubfal:2018. Section (ref) provides concluding remarks. All proofs and several intermediate results are collected in the appendix.

Setup and notation

We consider an RCT with $n$ participants. For each participant $i=1,\dots,n$, $Y_i \in \mathbb{R}$ denotes the observed outcome of interest, $Z_i \in \mathcal{Z}$ denotes a vector of observed baseline covariates, $A_i \in \{0,1\}$ indicates the treatment assignment, and $D_i \in \{0,1\}$ indicates the treatment decision. Relative to the setup in bugni/canay/shaikh:2018,bugni/canay/shaikh:2019, we allow for imperfect compliance, i.e., for $D_i \neq A_i$.

We consider potential outcome models for both outcomes and treatment decisions. For each participant $i=1,\dots,n$, we use $Y_i(D)$ to denote the potential outcome of participant $i$ if he/she makes treatment decision $D$, and we use $D_i(A)$ to denote the potential treatment decision of participant $i$ if he/she has assigned treatment $A$. These are related to their observed counterparts in the usual manner:

Following the usual classification in the LATE framework in angrist/imbens:1994, each participant in the RCT can only be one of four types: complier, always taker, never taker, or a defier. An individual $i$ is said to be a complier if $\{D_i(0)=0,D_i(1)=1\}$, an always taker if $\{D_i(0)=D_i(1)=1\}$, a never taker if $\{D_i(0)=D_i(1)=0\}$, and a defier if $\{D_i(0)=1,D_i(1)=0\}$. As usual in the literature, we later impose that there are no defiers in our population of participants in order to identify the LATE. It is convenient to use $C$ to denote a complier, $AT$ to denote an always taker, $NT$ to denote a never taker, and $DEF$ to denote a defier. Our goal in this paper is to consistently estimate the LATE $ \beta \equiv E[Y(1)-Y(0)|C]$ and to test hypotheses about it. In particular, for a prespecified choice of $\beta_0 \in \mathbb{R}$, we are interested in the following hypothesis testing problem

at a significance level $\alpha \in (0,1)$.

Following bugni/canay/shaikh:2018,bugni/canay/shaikh:2019, we use $P_n$ to denote the distribution of the observed data

and denote by $Q_n$ the distribution of the underlying random variables, given by

Note that $P_n$ is jointly determined by (ref), $Q_n$, and the treatment assignment mechanism. We therefore state our assumptions below in terms of the restrictions on $Q_n$ and the treatment assignment mechanism. In fact, we will not make reference to $P_n$ for the remainder of the paper, and all the operations are understood to be under $Q_n$ and the treatment assignment mechanism.

Strata are constructed from the observed, baseline covariates $Z_i$ using a prespecified function $S:\mathcal{Z} \to \mathcal{S}$, where $\mathcal{S}$ is a finite set. For each participant $i=1,\dots,n$, let $S_i \equiv S(Z_i)$ and let $S^{(n)} = (S_i:i=1,\dots,n)$. By definition, we note that $S^{(n)}$ is completely determined by the covariates in $W^{(n)}$.

We begin by describing our assumptions on the underlying data generating process (DGP) of $W^{(n)}$.

Assumption (ref) requires the underlying data distribution to be i.i.d., i.e., $Q_{n} = Q^n$, where $Q$ denotes the common marginal distribution of $(Y_i(1),Y_i(0),D_i(0),D_i(1),Z_i)$. In addition, the assumption imposes several requirements on $Q$. Assumption (ref)(a) demands that the potential outcomes have finite second moments, which is important to develop our asymptotic analysis. Assumption (ref)(b) requires all strata to be relevant. Assumption (ref)(c) corresponds to angrist/imbens:1994, and imposes the standard “no defiers” or “monotonicity” condition that is essential to identify the LATE. In other words, this condition implies that there are no participants who will decide to defy the treatment assignment, i.e., decide to both adopt the treatment when assigned to the control and decide to adopt the control when assigned to treatment. To interpret Assumption (ref)(d), we note that Lemma (ref) provides the following expression of the probability of each type of participant conditional on the stratum $s \in \mathcal{S}$:

By (ref), Assumption (ref)(d) imposes that every stratum has a non-trivial amount of participants who will decide to comply with the assigned treatment status. This corresponds to a strata-specific version of the so-called “relevance condition” imposed in angrist/imbens:1994.

Next, we describe our assumptions on the treatment assignment mechanism. As explained earlier, we focus our analysis on CAR, i.e., on randomization schemes that first stratify according to baseline covariates and then assign treatment status to as to achieve “balance” within each stratum. To describe our assumption more formally, we require some further notation. Let $A^{(n)} = (A_i:i=1,\dots,n)$ denote the vector of treatment assignments. For any $s \in \mathcal{S}$, let $\pi_A(s) \in (0,1)$ denote the “target” proportion of participants to assign to treatment in stratum $s$, determined by the researcher implementing the RCT. Also, let

denote the number of participants assigned to treatment in stratum $s$, and let

denote the number of participants in stratum $s$. With this notation in place, we now specify our assumption regarding the treatment assignment mechanism.

In principle, it is possible that $n(s)=0$ for some $s\in \mathcal{S}$, and so the term $n_{A}(s)/n(s)$ may not be properly defined. In any case, we note that Assumption (ref) implies that $n(s)/n \overset{p}{\to} p(s)>0$ for all $s\in \mathcal{S}$, and so the zero denominator issue is only a small sample problem and does not affect our asymptotic analysis.\footnote{To avoid defining objects that have a zero denominator, we abuse the notation and redefine the ratio $a/b$ as an arbitrary number (say, zero) whenever $b=0$. We adopt this rule in Assumption (ref) and throughout the rest of this paper. For the reasons explained earlier, this alternative ratio definition only serves the purpose of avoiding a zero denominator in small samples and does not affect our asymptotic analysis.}

Assumption (ref)(a) essentially requires that the treatment assignment vector $A^{(n)}$ is a function of the strata vector $S^{(n)}$ and an exogenous randomization device. Assumption (ref)(b) imposes that the fraction of units assigned to treatment in the stratum $s$ converges in probability to a target proportion $\pi_A(s)$ as the sample size diverges. As we show in Section (ref), Assumption (ref) imposes sufficient structure of the CAR mechanism to analyze the asymptotic distribution of the LATE estimator in the “fully saturated” (SAT) IV regression. However, as we show in Sections (ref) and (ref), Assumption (ref) will not be enough to guarantee the consistency of the LATE estimator in the “strata fixed effects” (SFE) and “two sample” (2S) IV regressions. To analyse the asymptotic properties of these estimators, we replace Assumption (ref) with the following condition, which mildly strengthens it.

Note that Assumption (ref)(a) and Assumption (ref)(a) coincide. Assumption (ref)(b) strengthens the convergence Assumption (ref)(b), and it requires the fraction of units assigned to the treatment in the stratum $s$ to be asymptotically normal, conditional on the vector of strata $S^{(n)}$. For each stratum $s \in \mathcal{S}$, the parameter $\tau(s) \in [0,1]$ determines the amount of dispersion that the CAR mechanism allows on the fraction of units assigned to the treatment in that stratum. A lower value of $\tau(s)$ implies that the CAR mechanism imposes a higher degree of “balance” or “control” of the treatment assignment proportion relative to its desired target value.\footnote{Assumption (ref)(b) is slightly weaker than bugni/canay/shaikh:2019 in that we require the condition holding w.p.a.1 instead of a.s. We establish that the w.p.a.1 version of the assumption is sufficient to establish all of our formal results.} Finally, Assumption (ref)(c) imposes that the target value for the treatment assignment does not vary by strata. As we show in Sections (ref) and (ref), this condition is key to the consistency of the LATE estimators produced by the SFE and 2SR regressions.

As explained by bugni/canay/shaikh:2018,bugni/canay/shaikh:2019 and rosenberger/lachin:2016, Assumptions (ref) and (ref) are satisfied by a wide array of CAR schemes. We briefly consider three popular schemes that can easily be seen to satisfy this assumption.

“Fully saturated” (SAT) IV regression

In this section, we study the asymptotic properties of an IV estimator of the LATE based on a linear regression model of the outcome of interest on the full set of indicators for all strata and their interaction with the treatment decision, where the latter is instrumented with the treatment assignment. Following the nomenclature in bugni/canay/shaikh:2019, we refer to this as the SAT IV regression. We show in this section that this SAT IV regression can consistently estimate the LATE for each stratum, and we derive their joint asymptotic distribution. These estimators can then be combined to produce a consistent estimator of the LATE. We show that this estimator is asymptotically normal and we characterize its asymptotic variance in terms of the primitives parameters of the RCT. We also show that the coefficients and residuals of the SAT IV regression can be used to consistently estimate these primitive parameters, which allows us to propose a consistent estimator of the standard errors of the LATE estimator. All of this allows us to propose hypothesis tests for the LATE that are asymptotically exact, i.e., their limiting rejection probability under the null hypothesis is equal to the nominal level.

In terms of our notation, the SAT IV regression is the result of regressing $Y_{i}$ on $(1[S_{i}=s]:s\in \mathcal{S})$ and $(D_i 1[S_{i}=s]:s\in \mathcal{S})$. Since the treatment decision $D_{i}$ is endogenously decided by the RCT participant, we instrument it with the exogenous treatment assignment $A_i$. To define these IV estimators precisely, set

The IV estimators of the coefficients in SAT regression are

where $\hat{\gamma}_{\text{sat}}(s)$ corresponds to the IV estimator of the coefficient on $1[S_{i}=s]$ and $\hat{\beta}_{\text{sat}}(s)$ corresponds to the IV estimator of the coefficient on $D_i 1[S_{i}=s]$.\footnote{In principle, it is possible that ${\mathbf{Z}_{n}^{\text{sat}}}'\mathbf{X}_{n}^{\text{sat}}$ is singular. In any case, we show in Lemma (ref) that ${\mathbf{Z}_{n}^{\text{sat}}}'\mathbf{X}_{n}^{\text{sat}}/n$ converges in probability to a non-singular matrix. Thus, the singularity is only a small sample problem and does not affect our asymptotic analysis. We can avoid this problem by using $M^{-1}$ to denote any generalized inverse of $M$.}

Under Assumptions (ref) and (ref), Theorem (ref) in the appendix shows that, for each stratum $s \in \mathcal{S}$,

This last equation in (ref) reveals that $\hat{\beta}_{\mathrm{sat}}$ is a consistent estimator of the vector of strata-specific LATE. To define a consistent estimator of the LATE based on these, all we then need is a consistent estimator of the probability a participant belongs to each strata conditional on being a complier, i.e., $P( S=s|C)$ for $s \in \mathcal{S}$. To this end, for every $s \in \mathcal{S}$, let

where

Under Assumptions (ref) and (ref), Theorem (ref) also shows that $\hat{P}( S=s|C)$ is a consistent estimator of $P( S=s|C)$ for every $s \in \mathcal{S}$. It is then natural to propose the following estimator of the LATE:

It follows from our previous discussion, the continuous mapping theorem, and the law of iterated expectations that $\hat{\beta}_{\text{sat}}$ is a consistent estimator of the LATE. The following theorem confirms this result, and also characterizes its asymptotic distribution in terms of primitive parameters of the RCT.

Several remarks about Theorem (ref) are in order. First, we note that $\hat{\beta}_{\mathrm{sat}}$ is related to the IV estimator $\hat{\beta}_3$ considered by ansel/hong/li:2018. In fact, these two estimators coincide if one specifies their covariates as a full vector of strata dummies. By specifying the set of covariates in our regression, we are able to obtain a closed-form expression of the asymptotic variance of $\hat{\beta}_{\mathrm{sat}}$ in terms of the primitive parameters of the RCT. This will become useful in Section (ref), where we consider the problem of choosing the parameters of the RCT to improve efficiency.

Second, we note that bugni/canay/shaikh:2019 derive the asymptotic distribution of $\hat{\beta}_{\mathrm{sat}}$ under perfect compliance (i.e., $\pi _{D(1)}(s)=1$, $\pi _{D(0)}(s)=0$). This means that we can understand the consequences of imperfect compliance by comparing Theorem (ref) and bugni/canay/shaikh:2019. First and foremost, we note that imperfect compliance means that the probability limit of $\hat{\beta}_{\mathrm{sat}}$ is no longer the ATE, but rather the LATE. Second, we note that imperfect compliance introduces significant changes to the asymptotic variance of $\hat{\beta}_{\mathrm{sat}}$. Imperfect compliance not only changes the expressions of $V_{{Y}}^{\mathrm{sat}} = V_{{Y} ,1}^{\mathrm{sat}}+V_{{Y},0}^{\mathrm{sat}}$ and $V_{H}^{\mathrm{sat}}$, but it also adds two new terms, $V_{D,1}^{\mathrm{sat}}$ and $V_{D,0}^{\mathrm{sat}}$. All of this implies that the consistent estimator of $V_{\mathrm{sat}}$ proposed in bugni/canay/shaikh:2019 no longer applies, and a new one is required. We do this in Theorem (ref).

Third, it is notable that Assumption (ref) is not required to derive Theorem (ref). In other words, the details of the CAR mechanism are not relevant to the asymptotic distribution of $\hat{\beta}_{\mathrm{sat}}$. This was pointed out in the case of perfect compliance by bugni/canay/shaikh:2019, and Theorem (ref) reveals that it also extends to the present setup.

Fourth, we note that Theorem (ref) allows for $\pi _{D( 0) }( s)=P(AT|S=s)=0$ or $1-\pi _{D( 1) }( s)=P(NT|S=s)=0$, but this requires a mild abuse of notation. If $\pi _{D( 0) }( s)=P(AT|S=s)=0$, the mean and the variance of $\{Y(1)|AT,S=s\}$ are not properly defined, but we can set

Similarly, if $1-\pi _{D( 1) }( s)=P(NT|S=s)=0$, the mean and the variance of $\{Y(0)|NT,S=s\}$ are not properly defined, but we can set

In particular, in the special case of perfect compliance (i.e., $\pi _{D(1)}(s)=1$, $\pi _{D(0)}(s)=0$), Theorem (ref) then holds with

which can be shown to coincide with the corresponding result in bugni/canay/shaikh:2019.

As promised earlier, the next result provides a consistent estimator of $V_{\mathrm{sat}}$.

We can propose hypothesis tests for the LATE by combining Theorems (ref) and (ref). For completeness, this is recorded in the next result.

“Strata fixed effects” (SFE) IV regression

In this section, we consider the asymptotic properties of an IV estimator of the LATE based on a linear regression model of the outcome of interest with a full set of indicators for all strata and the treatment decision, where the latter is instrumented with the treatment assignment. Following the nomenclature in bugni/canay/shaikh:2018,bugni/canay/shaikh:2019, we refer to this as the SFE IV regression. Under certain conditions, we show that this SFE IV regression consistently estimates the LATE. We show that this estimator is asymptotically normal and we characterize its asymptotic variance in terms of the primitives parameters of the RCT. We also propose a consistent estimator of this asymptotic variance by using the results of the SAT IV regression in Section (ref). This allows us to propose hypothesis tests for the LATE that are asymptotically exact, i.e., their limiting rejection probability under the null hypothesis is equal to the nominal level.

In terms of our notation, the SFE IV regression is the result of regressing $Y_{i}$ on $(1[S_{i}=s]:s\in \mathcal{S})$ and $D_i$. Since the treatment decision $D_{i}$ is endogenously decided by the RCT participant, we instrument it with the exogenous treatment assignment $A_i$. To define this IV estimator precisely, set

The estimators of the coefficients in IV SFE regression are

where $\hat{\gamma}_{\text{sfe}}(s)$ corresponds to the IV estimator of the coefficient on $1[S_{i}=s]$ and $\hat{\beta}_{ \text{sfe}}$ corresponds to the IV estimator of the coefficient on $D_i$.

Under Assumptions (ref) and (ref), Theorem (ref) in the appendix shows that

where $(\omega (s):s\in \mathcal{S})$ are non-negative weights defined by

These equations show that $\hat{\beta}_{ \text{sfe}}$ is not necessarily a consistent estimator of the LATE under Assumptions (ref) and (ref). By inspecting (ref), it follows that the consistency of the estimator can be restored provided that the treatment propensity does not vary by strata, i.e., Assumption (ref)(c). For this reason, we maintain this condition for the remainder of this section.

The following result reveals that $\hat{ \beta}_{\text{sfe}}$ is a consistent and asymptotically normal estimator of the LATE. The result characterizes the asymptotic distribution of this estimator in terms of primitive parameters of the RCT.

We now give several remarks about Theorem (ref). First, we note that $\hat{\beta}_{\mathrm{sfe}}$ is related to the IV estimator $\hat{\beta}_2$ considered by ansel/hong/li:2018. In fact, these two estimators coincide if one specifies their covariates as a full vector of strata dummies. As pointed out in Section (ref), specifying the covariates in the regression allows us to obtain a closed-form expression of the asymptotic variance $\hat{\beta}_{\mathrm{sfe}}$ in terms of the primitive parameters of the RCT.

As we have done in Section (ref) with the SAT IV regression, we can analyze the consequences of imperfect compliance for the SFE IV regression by comparing Theorem (ref) and bugni/canay/shaikh:2018 or bugni/canay/shaikh:2019. First, note that imperfect compliance means that the probability limit of $\hat{\beta}_{\mathrm{sfe}}$ is not the ATE, but rather the LATE. Second, we note that imperfect compliance introduces significant changes to the asymptotic variance of $\hat{\beta}_{\mathrm{sfe}}$. This implies that the consistent estimators of $V_{\mathrm{sfe}}$ proposed in bugni/canay/shaikh:2018 or bugni/canay/shaikh:2019 do not apply, and a new one is required. We provide this in Theorem (ref).

Third, we note that Theorem (ref) relies on Assumption (ref), which is stronger than Assumption (ref) used to derive Theorem (ref). First, and as discussed earlier, Assumption (ref)(c) is important to guarantee that $\hat{\beta}_{\mathrm{sfe}}$ is a consistent estimator of the LATE. Second, we note that the derivation of the asymptotic distribution of $\hat{\beta}_{\mathrm{sfe}}$ relies on the details about the CAR mechanism provided in Assumption (ref)(b). These types of details were not required to derive the asymptotic distribution of $\hat{\beta}_{\mathrm{sat}}$ in Theorem (ref).

Fourth, it is relevant to note that $V_{\mathrm{sfe}}-V_{\mathrm{sat}}=V_{A}^{\mathrm{sfe}} \geq 0$, which reveals that $\hat{\beta}_{\mathrm{sat}}$ is equally or more efficient than $\hat{\beta}_{\mathrm{sfe}}$. In particular, both estimators have the same asymptotic distribution if and only if $V_{A}^{\mathrm{sfe}} =0$. By inspecting (ref), this occurs if the RCT is implemented with either $\pi=1/2$ or $\tau(s)=0$ (e.g., by using SBR as described in Example (ref)).\footnote{These results resemble those obtained by lin:2013 in the context of finite population inference, perfect compliance, and assignment using SRS.}

Fifth, we note that Theorem (ref) allows for $\pi _{D( 0) }( s)=P(AT|S=s)=0$ or $1-\pi _{D( 1) }( s)=P(NT|S=s)=0$, by using the same abuse of notation as in Section (ref). In particular, in the special case of perfect compliance (i.e., $\pi _{D(1)}(s)=1$, $\pi _{D(0)}(s)=0$), Theorem (ref) then holds with

which coincides with the corresponding results in bugni/canay/shaikh:2018 and bugni/canay/shaikh:2019.

As promised earlier, the next result provides a consistent estimator of $V_{\mathrm{sfe}}$.

To conclude the section, we can propose hypothesis tests for the LATE by combining Theorems (ref) and (ref). For completeness, this is recorded in the next result.

“Two sample” (2S) IV regression

We now consider the asymptotic properties of an IV estimator of the LATE based on a linear regression model of the outcome of interest on a constant and the treatment decision, where the latter is instrumented with the treatment assignment. Following the nomenclature in bugni/canay/shaikh:2018, we refer to this as the 2S IV regression. Under certain conditions, the 2S IV regression can consistently estimate the LATE. We show that this estimator is asymptotically normal and we characterize its asymptotic variance in terms of the primitives parameters of the RCT. We also propose a consistent estimator of this asymptotic variance by using the results of the SAT IV regression in Section (ref). This allows us to propose hypothesis tests for the LATE that are asymptotically exact, i.e., their limiting rejection probability under the null hypothesis is equal to the nominal level.

In terms of our notation, the 2S IV regression is the result of regressing $Y_{i}$ on $1$ and $D_i$. Since the treatment decision $D_{i}$ is endogenously decided by the RCT participant, we instrument it with the exogenous treatment assignment $A_i$. To define this IV estimator precisely, set

The estimators of the coefficients in IV 2S regression are

where $\hat{\gamma}_{\mathrm{2s}}$ corresponds to the IV estimator of the coefficient on $1$ and $\hat{\beta}_{ \mathrm{2s}}$ corresponds to the IV estimator of the coefficient on $D_i$.

Under Assumptions (ref) and (ref), Theorem (ref) in the appendix shows that

with $\bar{\pi} _{A} \equiv \sum_{s\in \mathcal{S}}p(s)\pi _{A}(s)$. This equation reveals that $\hat{\beta}_{\mathrm{2s}}$ is not necessarily a consistent estimator of the LATE under Assumptions (ref) and (ref). However, if we additionally impose Assumption (ref)(c), it follows that $\hat{\beta}_{\mathrm{2s}}$ becomes a consistent estimator of the LATE. For this reason, we maintain this condition for the remainder of this section.

The following result reveals that $\hat{ \beta}_{\mathrm{2s}}$ is a consistent and asymptotically normal estimator of the LATE. The result characterizes the asymptotic distribution of this estimator in terms of primitive parameters of the RCT.

Several remarks about Theorem (ref) are in order. First, we note that $\hat{\beta}_{\mathrm{2s}}$ coincides with IV estimator $\hat{\beta}_1$ considered by ansel/hong/li:2018. Relative to their results, we provide a closed-form expression of the asymptotic variance $\hat{\beta}_{\mathrm{2s}}$ in terms of the primitive parameters of the RCT.

As we have done in the previous sections, we can analyze the consequences of imperfect compliance for the 2S IV regression by comparing Theorem (ref) and bugni/canay/shaikh:2018. First, note that imperfect compliance means that the probability limit of $\hat{\beta}_{\mathrm{2s}}$ is not the ATE, but rather the LATE. Second, we note that imperfect compliance introduces significant changes to the asymptotic variance of $\hat{\beta}_{\mathrm{2s}}$. This implies that the consistent estimator of $V_{\mathrm{2s}}$ proposed in bugni/canay/shaikh:2018 does not apply, and a new one is required. We provide this in Theorem (ref).

Third, we note that Theorem (ref) relies on Assumption (ref), which is stronger than Assumption (ref) used to derive Theorem (ref). The argument here is the same as in Section (ref). First, Assumption (ref)(c) is important to guarantee that $\hat{\beta}_{\mathrm{2s}}$ is a consistent estimator of the LATE. Second, the derivation of the asymptotic distribution of $\hat{\beta}_{\mathrm{2s}}$ relies on the details about the CAR mechanism provided in Assumption (ref)(b).

Fourth, it is relevant to note that $V_{\mathrm{2s}}-V_{\mathrm{sat}}=V_{A}^{\mathrm{2s}} \geq 0$, which reveals that $\hat{\beta}_{\mathrm{sat}}$ is equally or more efficient than $\hat{\beta}_{\mathrm{2s}}$. In particular, both estimators have the same asymptotic distribution if and only if $V_{A}^{\mathrm{2s}} =0$. By inspecting (ref), this occurs if the RCT is implemented with $\tau(s)=0$ (e.g., by using SBR as described in Example (ref)). We also note that $\tau(s)=0$ implies that $V_{\mathrm{2s}} = V_{\mathrm{sfe}} = V_{\mathrm{sat}}$ and $\pi_A=1/2$ implies that $V_{\mathrm{2s}} \geq V_{\mathrm{sfe}} = V_{\mathrm{sat}}$. Other than these special cases, $V_{\mathrm{2s}}$ and $V_{\mathrm{sfe}}$ cannot be ordered unambiguously (See bugni/canay/shaikh:2018 for a similar point in the context of perfect compliance).\footnote{As in the previous section, these results resemble those obtained by lin:2013 in the context of finite population inference, perfect compliance, and assignment using SRS.}

Fifth, we note that Theorem (ref) allows for $\pi _{D( 0) }( s)=P(AT|S=s)=0$ or $1-\pi _{D( 1) }( s)=P(NT|S=s)=0$, by using the same abuse of notation as in Section (ref). In the special case of perfect compliance (i.e., $\pi _{D(1)}(s)=1$, $\pi _{D(0)}(s)=0$), Theorem (ref) then holds with

which coincides with the result in bugni/canay/shaikh:2018.

As promised earlier, the next result provides a consistent estimator of $V_{\mathrm{2s}}$.

To conclude the section, we can propose hypothesis tests for the LATE by combining Theorems (ref) and (ref). For completeness, this is recorded in the next result.

Designing an RCT based on pilot RCT data

This section considers the situation of a researcher interested in designing a hypothetical RCT with CAR to estimate the LATE. The researcher is in charge of choosing the parameters of the RCT, which entail the randomization scheme, the stratification function, and the treatment probability vector. To make these decisions, the researcher observes the results of a previous pilot RCT (also with CAR) conducted on the same population of interest. Therefore, the researcher can consistently estimate the main features of the population based on the pilot RCT data.

Throughout this section, we assume that both RCTs satisfy the sampling framework in Section (ref). We use $n_P$ to denote the sample size of the pilot RCT and $n$ to denote the sample size of the hypothetical RCT. We presume that $n_P$ and $n$ are sufficiently large, so our asymptotic analysis in previous sections accurately represents both RCTs. This framework allows us to exploit the pilot RCT information to improve the asymptotic efficiency of the IV LATE estimators in the hypothetical RCT.\footnote{We recognize that, in practice, there may be a wide array considerations in designing the hypothetical RCT. The fact that we can focus on the asymptotic efficiency of the IV LATE estimator is a by-product of our asymptotic framework.}

Choosing the randomization scheme

In this section, we consider a hypothetical RCT with CAR with a given stratification function $S:\mathcal{Z} \to \mathcal{S}$. Our objective is to consider the effect that the randomization scheme has on the asymptotic distribution of the LATE estimators.

Under Assumptions (ref) and (ref), Theorem (ref) reveals the randomization scheme has no influence on the asymptotic distribution of the LATE estimator based on the SAT IV regression. If we additionally impose Assumption (ref), then Theorems (ref) and (ref) show that the randomization scheme affects the asymptotic variance of the LATE estimators based on the SFE IV and 2S IV regressions via the parameter $(\tau(s):s \in \mathcal{S})$, which appears on $V_A^{\mathrm{sfe}}$ and $V_A^{\mathrm{2s}}$, respectively. For both of these estimators, the optimal choice of the randomization scheme is to set $\tau(s)=0$ for all $s \in \mathcal{S}$. This decision is optimal regardless of the information in the pilot RCT. Under this optimal choice, the asymptotic distribution of the LATE estimators in the SFE IV and 2S IV regressions coincide with the LATE estimator in the SAT IV regression, given in Theorem (ref). In practice, $\tau(s)=0$ for all $s \in \mathcal{S}$ can be achieved by implementing the CAR using SBR.

Based on these arguments, it is natural to focus the remainder of this section on the case in which the hypothetical RCT has $\tau(s)=0$ for all $s \in \mathcal{S}$.

Choosing the stratification function

This section considers the effect of the choice of the stratification function of the hypothetical RCT on the asymptotic variance of the LATE estimators. For the sake of simplicity, we impose some restrictions on the hypothetical RCT under consideration. First, we assume that the hypothetical RCT satisfies Assumptions (ref) and (ref) with $\tau(s) =0$ for all $s\in \mathcal{S}$. This choice is motivated by the discussion in Section (ref), and has the additional benefit that we can describe the asymptotic distribution of all three LATE estimators in a single statement.\footnote{Note that Assumption (ref)(c) imposes a constant treatment assignment probability. In principle, we could generalize our analysis at the expense of substantially complicating the notation.}

The next result establishes that if the strata of the hypothetical RCT becomes coarser and all else remains equal, then the asymptotic variance of the IV estimators will either remain constant or increase.

One implication of Theorem (ref) is that, all else equal, a finer strata structure is always preferable from the point of view of the asymptotic efficiency of the LATE estimator. Since the strata are defined based on the baseline covariate $Z$, the finest possible strata structure is one in which each point in the support of $Z$, denoted by $\mathcal{Z}$, is assigned to its own stratum. This idea is feasible if $Z$ is discretely distributed and $\mathcal{Z}$ is finite. On the other hand, this conclusion would fail if $\mathcal{Z}$ takes infinitely many values, as our formal analysis is based on the presumption that $\mathcal{S}$ is a finite set and $p(s)>0$ for all $s \in \mathcal{S}$. In this sense, our asymptotic framework limits our ability to make extreme recommendations based on Theorem (ref).

It is relevant to connect Theorem (ref) with the recent contributions by tabordmeehan:2020 and bai:2022. Unlike our work, both references consider inference for the ATE under perfect compliance. bai:2022 considers the problem of treatment assignment in an RCT and shows that the optimal stratified randomization scheme (in the sense of minimizing the asymptotic variance of the ATE estimator) is achieved by a certain matched-pair design, i.e., strata formed by pairs of individuals. While our asymptotic framework does not allow matched-pair designs, one could interpret them as the limiting result of repeatedly splitting up our strata. In this sense, Theorem (ref) is compatible with bai:2022's optimality result. In turn, tabordmeehan:2020 considers treatment assignment in an RCT using a randomization procedure referred to as {\it stratification trees} in the context of perfect compliance. The main result in tabordmeehan:2020 shows that these stratification trees can be used to find an optimal stratification function (again, in the sense of minimizing the asymptotic variance of the ATE estimator). To derive this result, he restricts attention to strata functions with a fixed level of complexity or {\it tree depth}. Theorem (ref) implies that the asymptotic variance of the ATE estimator cannot increase if we consider the optimal stratification tree in tabordmeehan:2020 and we further divide any of its branches. Note that this is compatible with the main results in tabordmeehan:2020, as the further divided tree would have a level of complexity not allowed in the optimization problem in his paper.

Consider the situation of a researcher who has completed a pilot RCT with CAR. Motivated by Theorem (ref), the researcher interested in gaining efficiency in the estimation of the LATE would want to run a hypothetical RCT with a finer strata partition than that of the pilot RCT. In this case, the researcher would naturally be interested in predicting the asymptotic variance of the LATE estimator in this hypothetical RCT based on the pilot RCT data (that is, before implementing the hypothetical RCT). This is precisely the problem addressed in Theorem (ref). This result provides a consistent estimator of the asymptotic variance of the LATE estimator in the hypothetical RCT with finer strata than the pilot RCT, based exclusively on the data from the pilot RCT.

Theorem (ref) shows how to use the pilot RCT data to estimate the asymptotic variance of the LATE estimators in a hypothetical RCT with finer strata than that of the pilot RCT. In principle, one could also consider a similar exercise in the case when the hypothetical RCT has coarser strata than that of the pilot RCT. We omit this result for the sake of brevity, as Theorem (ref) shows that such a hypothetical RCT would produce less efficient estimators than those from the pilot RCT.

Choosing the treatment propensity

This section considers the effect of the choice of the treatment propensity vector $(\pi_A(s):s\in \mathcal{S})$ on the asymptotic variance of the LATE estimators, for a given the stratification function $S:\mathcal{Z} \to \mathcal{S}$. Under appropriate assumptions, Theorems (ref), (ref), and (ref) provide an asymptotic distribution of our LATE IV estimators for any treatment propensity vector $(\pi_A(s):s\in \mathcal{S})$. The following result calculates the optimal treatment propensity in the sense of minimizing their asymptotic variance, and provides a strategy to estimate it based on data from a pilot RCT.

Monte Carlo simulations

In this section, we explore the various inference methods proposed in the paper via Monte Carlo simulations. This exercise has multiple goals. First, we hope to show that we can accurately estimate the LATE and the asymptotic variance of the various LATE estimators. Second, we seek to confirm the accuracy of our asymptotic normal approximation by showing that the empirical coverage rate of our proposed confidence intervals is close to our desired coverage level. Third, we will use these simulations to explore how the asymptotic variance of the various estimators change as we vary the parameters of the DGP. In particular, we will numerically explore the theoretical predictions in Section (ref) regarding the optimal RCT design.

We consider four simulation designs, which we describe in Table (ref). We consider CAR based on either four or eight strata, i.e., $\mathcal{S} \in \{4,8\}$. These were generated by intersecting the support of $L$ binary covariates with $L \in \{2,3\}$. Design 1 is our baseline design, where we consider an RCT with four strata, and with treatment assignment probabilities and strata-specific LATE constant across strata. Design 2 is similar to our baseline design, but we divide each stratum into two, effectively increasing the number of strata from four to eight. Design 3 is similar to our baseline design, but we allow the strata-specific LATE to vary across strata. Finally, Design 4 is similar to Design 3, but we also enable treatment assignment probabilities to vary by strata.

For each simulation design, we show the average results of computing the estimators in $5,000$ independent replications of the RCT with sample size $n=200$. For each RCT, we consider that the researcher assigns the treatment according to the following CAR mechanisms: SRS (as described in Example (ref)), SBR (as described in Example (ref)), and the minimization methods in pocock/simon:1975 (PSM) and hu/hu:2012 (HHM) (as described in Example (ref)). The implementation of the minimization methods requires several tuning parameters. For Designs 1, 3, and 4, (i.e., designs with two binary covariates), we follow the guidance in hu/hu:2012, and we set the biasing probability to $\lambda=0.85$, uniform PSM weights (i.e., $w_{m,1}=w_{m,2} = 0.5$, and $w_{o}=w_{s}=0$ by definition), and HHM weights to $(w_{o}, w_{m, 1}, w_{m, 2}, w_{s}) = (0.3, 0.1, 0.1, 0.5)$. For Design 2, we keep the biasing probability at $\lambda=0.85$ and uniform PSM weights (i.e., $w_{m,1}=w_{m,2}=w_{m,3} = 1/3$, and $w_{o}=w_{s}=0$ by definition), and we change the HHM weights to $(w_{o}, w_{m, 1}, w_{m, 2},w_{m,3}, w_{s}) = (0.04, {1}/{60}, {1}/{60}, {1}/{60}, 0.91)$. We note that our HHM weights for Designs 1, 3, and 4 satisfy Conditions (A)-(B) in hu/hu:2012 and those for Design 2 satisfy Condition (C) in hu/hu:2012. Thus, our implementation of HHM satisfies Assumption (ref) with $\tau(s) = 0$ for all $s \in \mathcal{S}$. In turn, our implementation of PSM satisfies Assumption (ref), but it is not currently known whether it satisfies Assumption (ref)(b). For this reason, we restrict attention to the SAT IV regression when presenting results for PSM.

Design 1

We begin our description of Design 1 by specifying the features known to the researcher implementing the RCT. This researcher knows that there are two binary covariates (i.e., $L=2$), resulting in four strata (i.e., $|\mathcal{S}|=4$), and that RCT participants are assigned into treatment or control by using SRS, SBR, PSM, or HHM with constant treatment assignment probabilities equal to $\pi_A(s)=1/2$ for all $s \in \mathcal{S}$.

We next describe those aspects of Design 1 unknown to the researcher implementing the RCT. First, we make all strata to be equally likely, i.e., $p(s)=1/4$ for all $s \in \mathcal{S}$. Second, RCT participants belong to each type according to i.i.d.\ draws of a multinomial distribution with

Third, conditional on their type and their strata, the RCT participants have potential outcomes that are i.i.d.\ drawn according to a normal distribution with

and

Note that (ref) implies that the strata-specific LATE is $\beta(s) =E(Y(1)-Y(0)|C,S=s) = 1$ and, thus, the LATE is $\beta = 1$.\footnote{As discussed in earlier sections, the ITT is equal to the LATE multiplied by $P(C)=0.7$.} Also, while the distribution of treatment effects for compliers are homogeneous across strata, the corresponding distribution for always takers and never takers is not.\footnote{We have also conducted simulations in a case in which there is homogeneity across strata for all types. These results are omitted for brevity and available upon request.}

The simulation results for Design 1 are provided in Table (ref). The results reveal that all the proposed estimators are very close to the true LATE (equal to one). Across our simulations, the bias is almost zero and the squared error of estimation is close to the asymptotic variance. We now describe the behavior of the asymptotic variance of these estimators. As shown by our formal results, the asymptotic variance is constant under SBR and HHM (i.e., $\tau(s)=0$), and the asymptotic variance of the SAT IV estimator is the same across CAR mechanisms. The homogeneity of the distribution of treatment effects for the compliers across strata implies that $V_A^{\mathrm{sfe}}=0$ and, thus, the asymptotic variance of the SFE IV estimator is the same across CAR mechanisms. Finally, the heterogeneity of the distribution of potential outcomes for never takers and always takers causes that $V_A^{\mathrm{2s}}>0$.\footnote{This is the main difference with the simulations in which there is homogeneity across strata for all types. In that case, $V_A^{\mathrm{2s}}=0$ and thus all estimators under consideration have the same asymptotic variance under SRS and SBR.} This explains why the asymptotic variance of the 2S IV estimator is higher under SRS than under SBR and HHM. It is relevant to note that, for all estimators, the asymptotic variance under SBR and HHM is smaller or equal than that under SRS, as demonstrated in Section (ref). In all cases, the average value of our proposed asymptotic variance estimate is very close to the true asymptotic variance. From these results and the fact that the normal asymptotic approximation is accurate, it follows that the empirical coverage rate of the true LATE is very close to the desired coverage rate of 95%.

The simulation results in Table (ref) were obtained using a treatment assignment probability of $\pi_A(s)=1/2$ for all $s \in \mathcal{S}$. The most efficient LATE estimator in Table (ref) is the SAT IV estimator, which has an asymptotic variance equal to 14.5306. Section (ref) shows us to improve on the efficiency of this estimator by optimizing the treatment assignment probabilities. According to Theorem (ref), the optimal treatment assignment probability vector is $(\pi_A^*(s):s \in\mathcal{S}) = (0.6362, 0.6339, 0.6303, 0.6256)$ and the optimal constant treatment assignment probability is $\pi_A^* = 0.6314$. For the SAT IV estimator, the former yields an asymptotic variance of 13.5913 and the later yields an asymptotic variance of 13.5922. To put this into perspective, we can compute the efficiency gain in terms of “effective sample size”. This calculation reveals that using $\pi_A(s)=1/2$ for all $s \in \mathcal{S}$ is approximately equivalent to discarding 6.458% of the sample relative to the optimal constant treatment assignment probability and 6.465% of the sample relative to the optimal treatment assignment probability vector.

Design 2

In Section (ref), we showed that the asymptotic variance of the proposed LATE estimators does not increase if the strata of the RCT becomes finer and all else remains equal. We explore this prediction in Design 2, where we are splitting each stratum in Design 1 into two equally-sized strata. This produces a DGP with eight strata (i.e., $|\mathcal{S}|=8$). As in Design 1, RCT participants are still assigned into treatment or control by using SRS, SBR, PSM or HHM with constant treatment assignment probabilities equal to $\pi_A(s)=1/2$.

We now describe the features of Design 2 that are unknown to the researcher. All strata remain equally likely, i.e., $p(s)=1/8$ for all $s \in \mathcal{S}$, and RCT participants are assigned into types according to i.i.d.\ draws of a multinomial distribution with probabilities as in (ref). Conditional on their type and their strata, the RCT participants have potential outcomes that are i.i.d.\ drawn according to a normal distribution with

and

The parameters of the DGP in Design 2 are the result from splitting each stratum in Design 1 into two equally-sized strata. In particular, the law of iterated expectations and the law of total variance imply that (ref) and (ref) are compatible with (ref) and (ref). Note also that (ref) implies that the strata-specific LATE is $\beta(s) =E(Y(1)-Y(0)|C,S=s) = 1$ and, thus, the LATE is $\beta = 1$.

The simulation results for Design 2 are provided in Table (ref). These results are qualitatively similar to those in Table (ref). The estimators of the LATE and the various asymptotic variances are very accurate, and the empirical coverage rate is very close to the desired coverage level. The main noticeable difference between the tables is that the asymptotic variance of each estimator in Design 2 is smaller than the corresponding one in Design 1, as expected from Theorem (ref). For example, the asymptotic variance of the SAT IV estimator is 14.5306 in the RCT with four strata and becomes 12.4898 in the RCT with eight strata. In terms of “effective sample size”, this means that using four strata instead of eight is approximately equivalent to discarding 14.04% of the sample relative to using eight strata.

We can also consider optimizing the treatment assignment probability in these simulations. Focusing on the SAT IV estimator, the optimal treatment assignment probability vector yields an asymptotic variance of 11.366 and the optimal constant treatment assignment probability yields an asymptotic variance of 11.3678. In terms of “effective sample size”, this means that using $\pi_A(s)=1/2$ for all $s \in \mathcal{S}$ is approximately equivalent to discarding 8.984% of the sample relative to the optimal constant treatment assignment probability and 8.998% of the sample relative to the optimal treatment assignment probability vector.

Design 3

Relative to Design 1, Design 3 introduces heterogeneity of the strata-specific treatment effects. In terms of the observable features to the researcher, however, Design 3 is identical to Design 1. There are four strata (i.e., $|\mathcal{S}|=4$) and RCT participants are assigned into treatment or control by using SRS, SBR, PSM or HHM with constant treatment assignment probabilities equal to $\pi_A(s)=0.7$ for all $s \in \mathcal{S}$.

We now describe the features of Design 3 that are unobserved to the researcher. As in Design 1, all strata are equally likely, i.e., $p(s)=1/4$ for all $s \in \mathcal{S}$, and RCT participants are assigned into types according to i.i.d.\ draws of a multinomial distribution with probabilities as in (ref). Conditional on their type and their strata, RCT participants have potential outcomes that are i.i.d.\ drawn according to a normal distribution. The conditional variance of this distribution is as in (ref). The conditional mean of this distribution for never takers and always takes is as in (ref), while that for compliers is as follows

Note also that (ref) implies that the strata-specific LATE is $(\beta(s):s\in \mathcal{S}) =(E(Y(1)-Y(0)|C,S=s) :s\in \mathcal{S}) = [-1,1,1,3]$. When this information is combined with $P(C|S=s)=0.7$ and $p(s)=1/4$ for all $s\in \mathcal{S}$, it follows that the LATE is (still) $\beta = 1$.

The simulation results for Design 3 are provided in Table (ref). Most of the results are qualitatively similar to those in previous simulations. The estimators of the LATE and the various asymptotic variances are very accurate, and the empirical coverage rate is very close to the desired coverage level. As predicted by our results, all LATE IV estimators have the same asymptotic variance when SBR or HHM is used. The main difference with respect to the results in Design 1 appears when SRS is used. In Design 1, the SAT and SFE IV estimators have the same asymptotic variance under SRS. In Design 3, the SFE IV estimator has larger asymptotic variance than the SAT IV estimator when SRS is used. This is compatible with Theorem (ref), as the heterogeneity of the strata-specific treatment effect in Design 3 implies that $V_{A}^{\mathrm{sfe}}>0$.

Finally, we could also explore the effects of optimizing the treatment assignment probability. We decided to avoid this here and in the next design for the sake of brevity.

Design 4

Design 4 considers a researcher who implements CAR with an heterogeneous treatment assignment probability. The RCT in Design 4 has four strata, just like in Design 1 and 3. Unlike all previous designs, RCT participants are assigned into treatment or control by using SRS, SBR, PSM or HHM with treatment assignment probability vector equal to

We now describe the features of Design 4 that are unobserved to the researcher. First, all strata are equally likely, i.e., $p(s)=1/4$ for all $s \in \mathcal{S}$. Second, RCT participants are assigned into types according to i.i.d.\ draws of a multinomial distribution with probabilities

Conditional on their type and strata, RCT participants have potential outcomes that are i.i.d.\ drawn according to a normal distribution with conditional variance as in (ref), and conditional mean given by

These parameters in (ref) imply that the strata-specific LATE is $(\beta(s):s\in \mathcal{S}) =(E(Y(1)-Y(0)|C,S=s) :s\in \mathcal{S}) = [-5.6,2.8,4.4,1.4]$. When this information is combined with $(P(C|S=s):s\in \mathcal{S})$ in (ref) and $p(s)=1/4$ for all $s\in \mathcal{S}$, we can verify that the LATE is (still) $\beta = 1$.

The simulation results for Design 4 are provided in Table (ref). Given the heterogeneity of the treatment assignment probability in (ref), neither the SFE IV estimator nor the 2S IV estimator are guaranteed to be consistent for the LATE. In fact, under our current conditions, Theorem (ref) in the appendix implies that $\hat{\beta}_\mathrm{sfe} \overset{p}{\to} 1.0974 \neq 1=\beta $ and Theorem (ref) in the appendix implies that $\hat{\beta}_\mathrm{2s} \overset{p}{\to} 2.0422 \neq 1=\beta$. Note that this occurs for all CAR mechanisms. Since these estimators are not consistent for the parameter of interest, there is no point in discussing their asymptotic variance or their coverage. In contrast, the SAT IV estimator remains consistent in this scenario. In addition, the estimator of the asymptotic variance of the SAT IV estimator is accurate and the empirical coverage level is close to the desired coverage level. These simulations confirm that the inference based on the SAT IV estimator is valid under more general conditions than that based on the other two IV estimators.

Empirical Illustration

In this section, we consider an empirical illustration based on dupas/karlan/robinson/ubfal:2018. The authors use an RCT to investigate the economic impact of expanding access to basic bank accounts in several countries: Malawi, Uganda, and Chile.\footnote{The data are publicly available at \url{https://www.aeaweb.org/articles?id=10.1257/app.20160597}.} These countries differ significantly in their level of development and banking access, with Uganda being the intermediate country in both of these respects. For our illustration, we focus on their RCT conducted in Uganda.

We now briefly summarize the empirical setting; see dupas/karlan/robinson/ubfal:2018 for a more detailed description. Bank accounts are important to daily economic life, but the rate of opening a bank account in developing countries is relatively low compared to developed countries. During the time of the RCT, the authors report that 74% of households in Uganda were unbanked. Among the many benefits of having access to bank accounts, the authors focus on its arguably most primary function: safekeeping.

dupas/karlan/robinson/ubfal:2018 selected a random sample of 2,159 Ugandan households who did not have a bank account in 2011. These households were assigned into a treatment or a control group. Treated households were given a voucher for a free savings account at their nearest bank branch, without any fees for the RCT duration, along with assistance to complete the necessary paperwork required to open this account. Households in the control group were not provided with these vouchers. We use the binary variable $A_i \in \{0,1\}$ to indicate if household $i$ was given the voucher for the free savings account. The treatment assignment was stratified by gender, occupation, and bank branch, which generated 41 strata, i.e., $s\in \mathcal{S}=\{1,\cdots,41\}$. Within each stratum, households were randomly assigned to treatment or control using SBR with $\pi_A(s)=1/2$ for all $s\in\mathcal{S}$. As a result, 1,080 households were selected to receive the treatment, while the remaining 1,079 were placed in the control group. The households in the sample were re-interviewed thrice during 2012-2013. In these follow-up surveys, the authors collected information about their saving behavior and several other related outcomes.

This RCT featured imperfect compliance. While none of the 1,079 households in the control group accessed a free savings account, not every one of the 1,080 households in the treatment group opened their free savings account. dupas/karlan/robinson/ubfal:2018 reveals that, out of the 1,080 treated households, only 54% actually opened up the bank account, while only 42% made at least one deposit during the RCT. As in dupas/karlan/robinson/ubfal:2018, we are interested in the effect of opening and {\it using} the free savings account. We define the binary variable $D_i = D_i(A_i) \in \{0,1\}$ to indicate if household $i$ has opened and used the free savings account in this RCT.\footnote{Following the paper, we consider an account used if the owner has made at least one deposit during the RCT. We have also explored alternative definitions of usage, and we obtained qualitatively similar findings.}

Given this setup, we use our IV regressions to estimate and conduct inference on the LATE for several outcomes of interest.\footnote{In this RCT, $D_i(0)=0$ and so there are no always takers. As a consequence, the LATE coincides with the TOT, i.e., $E[Y(1)-Y(0)|D=1]$.} dupas/karlan/robinson/ubfal:2018 collect information on two types of outcomes: saving stocks and downstream outcomes. For brevity, we select three outcomes: savings in formal financial institutions, savings in cash at home or in a secret place, and expenditures in the last month. These variables were reported by the households in the final survey of the RCT and measured in 2010 US dollars.

Table (ref) presents the results of the IV regressions computed using our Stata package.\footnote{Our Stata package is publicly available at \href{https://sites.northwestern.edu/federicobugni/carlate/}{https://sites.northwestern.edu/federicobugni/carlate/}.} For each outcome variable, we estimate the LATE of using the savings account based on the SAT, SFE, and 2S IV regressions. Given the setup of the RCT, all of these estimators are consistent. We can also consistently estimate the standard errors of these LATE estimators. Since the RCT uses SBR (i.e., $\tau(s)=0$ for all $s\in \mathcal{S}$), the standard errors for all of these estimators coincide. To illustrate our results, it is also relevant to compare these with the estimates of the standard errors under the incorrect assumption that the RCT used SRS (i.e., $\tau(s)=1$ for all $s\in \mathcal{S}$). (It is relevant to note that the standard Stata command {\it ivregress} would presume that the data were collected using SRS.) First, by Theorem (ref), the standard error of the SAT IV estimator does not depend on the details of the CAR mechanism, so the estimated standard error does not depend on $(\tau(s):s\in \mathcal{S})$. Second, since this RCT uses $\pi_A(s)=1/2$ for all $s\in\mathcal{S}$, Theorem (ref) implies that the standard error of the SFE IV estimator does not depend on $(\tau(s):s\in \mathcal{S})$. Finally, as predicted by Theorem (ref), the standard error of the 2S IV estimator under $\tau(s)=1$ for all $s\in \mathcal{S}$ is larger than necessary, resulting in a loss in statistical power. When measured in terms of effective sample size, using the larger standard errors in the 2S IV regression is analog to losing 2.52%, 5.52%, and 9.12% of the sample in the SAT IV regressions of the first, second, and third outcome variables, respectively.

We now briefly describe the quantitative findings. For brevity, we focus on the LATE estimators based on the SAT IV regression. For complier households, opening and using these savings accounts result in an average increase of savings in formal financial institutions of \$17.572, an average decrease in savings in cash at home or in a secret place of \$7.323, and an average decrease in total expenditures in the last month of \$2.427. The first estimator is significantly different from zero at all significance levels, while the other two are not statistically significantly different from zero. Finally, our estimates indicate that 44.15% of the households are compliers, while the remaining 55.85% are never takers.

For the sake of comparison, Table (ref) also includes the corresponding results from dupas/karlan/robinson/ubfal:2018. Their regressions are analogous to our SFE IV specification but with several differences. First, they run regressions using panel data from three periods, while we only use data from the first period. Second, their sample is affected by attrition in the second and third periods. Third, their regressions include additional controls such as the baseline outcome value and period indicator variables. None of these controls are available for our regression based on the data from the first period. Despite these differences, we note that the two sets of results are qualitatively similar.

Following the arguments in Section (ref), we could use the data to study the optimality of the RCT parameters. For example, we can use the results in Section (ref) to estimate the optimal treatment assignment probabilities. We find that the optimal constant treatment assignment probabilities $\pi_A^*$ for the three outcome variables are 0.549, 0.491, and 0.496, respectively. These probabilities are all very close to the one used in the RCT and consequently result in very minor efficiency gains.

Conclusions

This paper studies inference in an RCT with CAR and imperfect compliance of a binary treatment. By CAR, we refer to randomization schemes that first stratify according to baseline covariates and then assign treatment status to achieve “balance” within each stratum. In this context, we allow the RCT participants to endogenously decide whether to comply or not with the assigned treatment status. Given the possibility of imperfect compliance, we study inference on the LATE.

We study the asymptotic properties of three LATE estimators derived from IV regression. The first one is the “fully saturated” or SAT IV regression, i.e., a linear regression of the outcome on all indicators for all strata and their interaction with the treatment decision, with the latter instrumented with the treatment assignment. We show that the proposed LATE estimator is asymptotically normal, and we characterize its asymptotic variance in terms of primitives of the problem. We provide consistent estimators of the standard errors and asymptotically exact hypothesis tests. This LATE estimator is consistent under weak conditions regarding the CAR method used to implement the RCT (i.e., Assumptions (ref)-(ref)).

Our second LATE estimator is based on the “strata fixed effects” or SFE IV linear regression, i.e., a linear regression of the outcome on indicators for all strata and the treatment decision, with the latter instrumented with the treatment assignment. Our last LATE estimator is based on the “two-sample” or 2S IV linear regression, i.e., a linear regression of the outcome on a constant and the treatment decision, with the latter instrumented with the treatment assignment. The consistency of both of these LATE estimators requires additional conditions relative to the one based on the SAT IV linear regression (i.e., Assumptions (ref)-(ref)). In particular, they require that the target proportion of RCT participants assigned to each treatment cannot vary by strata (see Assumption (ref)). Under these conditions, we show that both LATE estimators are asymptotically normal, and we characterize their asymptotic variance in terms of primitives of the problem. We also provide consistent estimators of their standard errors and asymptotically exact hypothesis tests.

Our characterization of the asymptotic properties of the LATE estimators allows us to investigate the influence of the parameters of the RCT. We use this to propose strategies to minimize their asymptotic variance in a hypothetical RCT based on data from its pilot study. We also establish that the asymptotic variance of the proposed LATE estimators does not increase if the strata of the RCT becomes finer and all else remains equal. We determine the optimal treatment assignment probability vector and show how to estimate it consistently based on data from a pilot study.

We confirm our theoretical results in Monte Carlo simulations. We also illustrate the practical relevance of our findings by revisiting the RCT in dupas/karlan/robinson/ubfal:2018.