Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
38,990 characters · 7 sections · 82 citation commands
\ifnum1=1 \shortTitle{Manipulation-Robust Regression Discontinuity Designs} \else \shortTitle \fi
\pubMonth \pubYear \pubVolume \pubIssue \Keywords{Regression Discontinuity Design; Manipulation; Diagnostic Test}
\ifnum1=1 \else \fi
The credibility of regression discontinuity (RD) design, “one of the most credible non-experimental strategies for the analysis of causal effects” (cattaneo_idrobo_titiunik_2020), is questioned when individuals manipulate the running variable that determines the treatment assignment. Hence, assessing manipulation is a critical procedure in RD analyses. However, the definition for manipulation is absent in existing models.
As manipulation has not been defined, identification argument is silent about manipulation. Thus, one cannot effectively argue for identification against a particular manipulation in consideration. In other words, existing identification is based on high-level conditions.
In this study, we articulate identification under low-level conditions so that it can be explicitly justified under manipulation. We establish the conditions using a potential outcome framework for the running variable that determines treatment assignment. Specifically, we define manipulation as an indicator that assigns one of two potential running variables, with and without manipulation. Given this framework, we restate the continuity condition hahnIdentificationEstimationTreatment2001 for identification with simple and explicit low-level conditions.
For example, a test score is a running variable when passing the examination is treatment. Manipulation is an action that assigns a manipulated score for the manipulated student and a non-manipulated score for other students. With manipulation, only one of two scores is observed. For instance, a teacher manipulates a test score of a student who has a non-manipulated failing score to a manipulated passing score. For such a manipulated student, we observe only the manipulated score not their original non-manipulated score.
Our low-level conditions require that manipulation must be a randomization. Specifically, manipulation must not only randomly assign the manipulated score when it is around the passing cutoff but also randomly select students to manipulate when their original non-manipulated scores are around the passing cutoff. In other words, the teacher must not precisely assign the score to pass the examination nor precisely select students who have failing scores.
Formally, we introduce an indicator of manipulation, $M$, such that the realized running variable is given by $R = M R^*(1) + (1 - M) R^*(0)$, where $R^*(1)$ is the running variable with manipulation ($M=1$) and $R^*(0)$ is the running variable without it ($M=0$). Identification holds under two restrictions: The manipulation $M$ randomly assigns $R = R^*(1)$ when $R^*(1)$ is assigned around the cutoff and randomly selects units to manipulate when they have $R^*(0)$ around the cutoff. In other words, manipulation must not precisely assign $R^*(1)$ around the cutoff nor precisely select units whose $R^*(0)$ are around the cutoff. Those two restrictions accommodate most examples, as illustrated in previous studies such as leeRegressionDiscontinuityDesigns2010 and gerardBoundsTreatmentEffects2020. Our framework formalizes their examples as two simple and explicit restrictions on $(M, R^*(0), R^*(1))$.
In this framework, the continuous density function is not a by-stander of identification. We demonstrate that continuous density is critical for the continuity condition hahnIdentificationEstimationTreatment2001, which is decomposed into the continuous mean potential outcomes weighted by the continuous density functions. Furthermore, we establish a low-level condition for the density test because although “a running variable with a continuous density is neither necessary nor sufficient for identification except under auxiliary assumptions” mccraryManipulationRunningVariable2008, the auxiliary assumption was absent.
We provide the auxiliary assumption as two restrictions on precise manipulation: If manipulation precisely assigns $R^*(1)$ or precisely selects units who have $R^*(0)$ around the cutoff, it must not assign to and select from both sides of the cutoff. In other words, precise manipulation must be one-sided both in its assignment of $R^*(1)$ and selection in $R^*(0)$. This one-sided restriction is also a low-level condition for partial identification in gerardBoundsTreatmentEffects2020 where a special case of their result justifies the density test. \footnote{Their restriction directly relies on local randomization and hence is also a high-level condition. See Remark (ref) in Section (ref) for a detailed discussion.} In summary, our framework induces low-level conditions for identification, diagnostic tests, and partial identification under possible manipulation. In other words, researchers may assess whether their RD designs are manipulation-robust.
We emphasize that although the existing practices are valid, their usage must be updated based on new interpretations. Statistical packages for RD design are well established. For example, rdrobust (calonicoRobustNonparametricConfidence2014a, Calonico_Cattaneo_Farrell_Titiunik_2017) is the dominant option for estimation and balance or placebo tests, while the rddensity package (Cattaneo_Jansson_Ma_2018, cattaneoSimpleLocalPolynomial2019) is increasingly used for the density test. Bounds are available from rdbounds package gerardBoundsTreatmentEffects2020. Although all these devices remain functional, our framework clarifies when and how they are used under what assumptions.
RD is a powerful tool in various disciplines. For extensive surveys, see leeRegressionDiscontinuityDesigns2010, DiNardo_Lee_2011, cattaneo_idrobo_titiunik_2020 (cattaneo_idrobo_titiunik_2020,Cattaneo_Idrobo_Titiunik_2024). Among the growing body of RD studies, we contribute to two strands of the literature.
First, we contribute to the literature on point identification in RD designs. hahnIdentificationEstimationTreatment2001 formalize the idea of RD in Thistlethwaite_Campbell_1960 with the minimal but less intuitive continuity condition. leeRandomizedExperimentsNonrandom2008 replaces the continuity condition with an analogy of randomization: If the running variable is equally likely to be just below or above the cutoff then the treatment is as if randomized. leeRegressionDiscontinuityDesigns2010 illustrate a local randomization failure under manipulation that precisely assigns the running variable to the right of the cutoff. However, manipulation is implicit in their identification argument because it is not defined in their model. mccraryManipulationRunningVariable2008 introduces an explicit manipulation concept although its connection to identification remains implicit. In this study, we provide the first low-level identification condition with explicit restrictions on manipulation by formalizing the concepts introduced by mccraryManipulationRunningVariable2008, leeRandomizedExperimentsNonrandom2008, and leeRegressionDiscontinuityDesigns2010 via a potential outcome framework.
Second, we contribute to the literature on diagnostic tests for continuous density function (density test) and continuous conditional mean functions of covariates (balance or placebo test) as consequences of local randomization leeRandomizedExperimentsNonrandom2008. These tests have been updated. otsuEstimationInferenceDiscontinuity2013 propose an empirical likelihood test for the density test, cattaneoSimpleLocalPolynomial2019 propose a density test from local polynomial estimates, bugniTestingContinuityDensity2020a propose a density test with $g$-order statistics, and canayApproximatePermutationTests2018 propose randomization tests for covariate balancing. However, the null hypotheses of these tests do not imply identification. \footnote{mccraryManipulationRunningVariable2008 conjectures an assumption (monotonic manipulation) for the density test, but the conjectured condition is neither necessary nor sufficient. See the Supplementary Appendix for details.} For fuzzy designs, Arai_Hsu_Kitagawa_Mourifie_Wan_2021 propose a complementary diagnostic procedure that is free from restrictions on the density. \footnote{Bertanha_Imbens_2020 also discuss tests for exogeneity and external validity in fuzzy designs.} However, there is no testable restriction in their procedure for sharp designs and the reduced form of fuzzy designs. gerardBoundsTreatmentEffects2020's condition for partial identification may be used for the density test as a special case. Because their condition is a high-level assumption that can be challenging to justify for some designs, our auxiliary assumption is the first low-level condition to connect these diagnostic tests with identification.
In an RD design, we exploit treatment assignment $D \in \{0,1\}$ by a scalar running variable $R$ by exceeding the cutoff $c$. For example, students receive qualification $D$ when their test score $R$ exceeds the passing cutoff $c$. This treatment $D = 1\{R \geq c\}$ represents a sharp design. \footnote{We do not consider measurement error for $R$ that has been studied extensively. For example, see Yu_2011, Davezies_Le_Barbanchon_2017, Pei_Shen_2017, Yanagi_2017, Bartalotti_Brummet_Dieterle_2020 and Dong_Kolesar_2022.} For fuzzy designs with noncompliance, we consider their reduced form, essentially a sharp design. For a pair of potential outcomes, $\{Y(1),Y(0)\}$, $Y = DY(1) + (1 - D)Y(0)$ is observed. We aim to identify our target parameter, the average treatment effect (ATE) for students whose score is at the cutoff: $E[Y(1) - Y(0)|R=c]$.
The remainder of this paper introduces the following notation: For a random variable $Z$, let $E[Z|R=c_+] \equiv \lim_{r \downarrow c} E[Z|R=r]$ and $E[Z|R = c_-] \equiv \lim_{r \uparrow c} E[Z|R=r]$. For a density function $f$, let $f(c_+) \equiv \lim_{r \downarrow c} f(r)$ and $f(c_-) \equiv \lim_{r \uparrow c} f(r)$. Throughout this paper, we impose Assumption (ref) in Appendix, which imposes the existence of relevant moments, densities, and their limits.
Identification of the ATE follows from the continuity condition hahnIdentificationEstimationTreatment2001
This condition is the minimal restriction for identification under an ideal design with the same mean types $Y(d)$ for those who have $R$ near the cutoff $c$. A concrete mechanism for (ref) is local randomization of $R$ leeRandomizedExperimentsNonrandom2008. \footnote{This local randomization differs from a related recent concept in Cattaneo_Frandsen_Titiunik_2015 and Cattaneo_Titiunik_Vazquez-Bare_2017 who consider explicit randomization within a small range near the cutoff.} In leeRandomizedExperimentsNonrandom2008, if $R$ is locally randomized, the following restrictions hold
where $f_R(r)$ is the density for $R$. Nevertheless, local randomization is also a high-level condition for an ideal design. These restrictions are silent about manipulation of the running variable because manipulation is not defined in their models.
In this study, we provide simple low-level conditions for (ref) from the potential outcome framework for manipulation $(M, R^*(1), R^*(0))$. As in the Introduction, $M \in \{0,1\}$ is an unobserved indicator of manipulation, $R^*(1)$ is the counterfactual running variable with manipulation, and $R^*(0)$ is the counterfactual running variable without manipulation. Hence, $R = M R^*(1) + (1 - M)R^*(0)$. In this framework, identification is shown under the low-level condition where both manipulated $R^*(1)|M=1$ and non-manipulated $R^*(0)|M=0$ are locally randomized as in (ref). In other words, manipulation must be a random assignment of $R^*(1)$ when it assigns $R^*(1)$ around $c$, and also a random selection of units whose $R^*(0)$ are around $c$.
The decomposition (ref) has a critical implication in the relationship between continuous density functions and identification: Identification relies on the balanced mean types $E[Y(d)|R=c_-,M=m] = E[Y(d)|R=c_+,M=m]$ weighted by balanced densities $f_{R|M=m}(c_-) = f_{R|M=m}(c_+)$ for each manipulation status $m \in \{0,1\}$. Conversely, identification can fail because of systematic differences in the (non-)manipulator's mean types $E[Y(d)|M=m,R=r]$ or their densities $f_{R|M=m}(r)$ around the cutoff. We illustrate an implication for the latter in an example of a test score $R$ as the running variable for a qualification awarded via the examination as treatment.
From Proposition (ref), continuous density functions are critical for identification. Continuous density is neither necessary nor sufficient for identification in general. Sufficiency is particularly important for the density test mccraryManipulationRunningVariable2008 because it guarantees that the null of the density test implies identification.
In the following, we provide the conditions for the sufficiency, specifically, those under which manipulation is detectable and the design is manipulation-robust because we may detect if it exists.
We illustrate how detection works and fails using the following toy example of a bribed teacher, inspired by diamond2016long. The teacher receives a rebate of $\beta_i > 0$ if a student $i$ with $R_i^*(0)$ passes. Simultaneously, manipulating the score incurs a marginal cost of $\kappa > 0$ where $R_i^*(0)$ is ex-ante randomized locally at the cutoff. In this toy model, the teacher maximizes the net payoff $\beta_i 1\{R_i \geq c\} - \kappa |R_i - R^*_i(0)|$ for each student. For a student $i$ with the initial score $R_i^*(0)$, the teacher's optimal manipulation $M_i$ satisfies \[ M_i = 1\{R^*_i(0) < c \mbox{ and } \beta_i \geq \kappa \cdot (c - R^*_i(0))\} \] and the optimal $R^{*}_i(1)$ equals $c$. The induced potential outcome framework for manipulation ($M_i,R^*_i(1),R^*_i(0)$) violates both conditions (ref) and (ref). The condition (ref) fails because the teacher precisely selects those who failed $R^*_i(0) < c$ only if they are worth saving $\beta_i \geq \kappa \cdot (c - R^*_i(0))$. The condition (ref) fails because the teacher precisely assigns $R_i^*(1)$ just at the threshold $c$. Hence, neither $R_i^*(0)|M=0$ nor $R_i^*(1)|M=1$ is randomized locally at the cutoff.
Such manipulation is detectable. Figure (ref) illustrates the initial score $R^*(0)$ and the realized score $R$, deviating from the model by modifying $R^{*}(1)$ so that $R$ has a density. The dashed line represents the density of $R^*(0)$ and the solid line represents the density of $R$. These densities differ in two ways. First, the teacher always assigns $R^*(1)$ above $c$ and never assigns $R^*(1) = c_-$. Hence, a bunch (shaded area) in the observed density $f_R(r)$ appears. Second, the teacher always selects students to manipulate only from the failed students $R^*(0) < c$ and never manipulates passing students $R^*(0) \geq c$. Hence, a notch (dotted area) appears at the cutoff point. Consequently, the realized score density, $f_R(r)$, jumps by $f_R(c_+) - f_R(c_-)$, indicating the presence of manipulation.
However, detection fails if bunches or notches are present on the other side, as illustrated in Figure (ref). If a teacher assigns $R^*(1)$ for the control $R^*(1) < c$ to intentionally fail someone (Figure (ref) left), then the bunches may cancel out. Similarly, if another teacher selects $R^*(0)$ from the treated $R^*(0) \geq c$ to fail them (Figure (ref) right), then the notches may cancel out. In other words, the detection strategy can fail with two-sided incentives for manipulation; some favor treatment, while others are against it. An auxiliary assumption is necessary to prevent such coincidental failure.
Now we introduce the auxiliary assumption formally. We first assume that $R^*(0)$ is ex-ante randomized locally. This assumption is virtually a definition for $R^*(0)$ and $M$: $M$ indicates manipulation that violates either (ref) or (ref), and $R^*(0)$ is the running variable after incurring all other innocuous manipulations that satisfy both (ref) and (ref).
The auxiliary assumption imposes two restrictions: Manipulation $M$ always assigns treatment whenever (ref) fails and never selects units from the treated whenever (ref) fails. These are one-sided manipulations because they always favor the treatment and never the control side.
Under the auxiliary assumption, continuous density implies identification. Hence, passing the density test confirms the identification. \footnote{In general, any mixture of three types in Assumption (ref) is allowed with complex notations. See Appendix for the general cases.}
Theorem (ref) also justifies the balance or placebo test; namely, the conditional means of covariates are continuous at the cutoff under the null hypothesis of the density test:
We have provided two main results clarifying new identification conditions (Proposition (ref)) and conditions for the density and balance tests being valid (Theorem (ref) and Corollary (ref)). These conditions are missing or implicit in existing empirical studies. We revisit a few empirical studies to discuss their implications for practices.
From Proposition (ref), identification holds if manipulation randomly assigns $R^*(1)$ around the cutoff and randomly selects units whose $R^*(0)$ are around the cutoff. The former can be justified by the famous no precise control or imprecise control strategy that “individuals do not precisely manipulate $X$ around the threshold has the prediction that treatment is locally randomized.” leeRegressionDiscontinuityDesigns2010. A similar claim is seen in the latest textbook: “if units lack the ability to precisely manipulate the score value they receive, there should be no systematic differences between units with similar values of the score.” Cattaneo_Idrobo_Titiunik_2024.
This no precise control strategy is widely used for identification in empirical studies. For example, Dahl_Loken_Mogstad_2014 study the impact of eligibility for paternal parental leave on his peer fathers in the workplace. They state that “The key identifying assumption of our fuzzy RD design is that individuals are unable to precisely control the assignment variable, date of birth, near the cutoff date c, in which case the variation in treatment near c is random.” Pinotti_2017 studies the impact of a work permit on crime rates by exploiting the permit being offered when the application is submitted before an unknown threshold time. He claims that “These complexities provide a compelling argument for the fundamental identification assumption that applicants within an arbitrarily narrow bandwidth of the cutoff were unable to precisely determine their assignment to either side of it.”. Similar arguments appear in more recent articles such as dechezlepretreTaxIncentivesIncrease2023 and huangPoliticalInfluenceBank2024. However, they are insufficient because no precise control implies the inability to control the manipulated $R^*(1)|M=1$ but remains silent about the non-manipulated $R^*(0)|M=0$. For the continuity condition hahnIdentificationEstimationTreatment2001, we need to regulate who would be manipulated in terms of their non-manipulated values, $R^*(0)$. Hence, their identification discussions may be incomplete.
Failure in the latter condition on $R^*(0)|M=0$ is addressed in some empirical studies, although its mechanism is under-explored. jepsenLaborMarketReturns2016 document a discontinuous density function for a design with the test score as the running variable, blaming a selection due to examination retakes. In Example (ref), we detailed two mechanisms in their context. An obvious mechanism is the imbalance in mean outcomes by self-selection due to manipulation. The other is non-trivial: imbalanced densities can fail identification without self-selection in their outcomes.
Our framework is simple and applicable to most RD designs. For example, in Dahl_Loken_Mogstad_2014, if a father quits a job because he is ineligible for the program, he would have no workplace and associated dependent variable. Such an attrition of the dependent variable is a threat for the local randomization of $R^*(0)|M=0$ because ineligible fathers are more likely to have $R^*(0)$ missing than others. A similar consideration justifies the identification in Pinotti_2017. If Pinotti_2017 had the universe of the work permit application timestamps and no systematic selection is possible in terms of $R^*(0)|M=0$, the identification should hold. Thus, our procedure offers a sophisticated check for those identification concerns in any design.
When manipulation can violate either (ref) or (ref), diagnostic tests should be employed to detect such manipulation, for which the condition for Theorem (ref) must hold. From Theorem (ref), we must verify that any manipulation of $R^*(1)|M=1$ must favor $R^*(1) \geq c$ whenever it fails (ref) and any selection of $R^*(0)$ should be from $R^*(0) < c$ whenever it fails (ref). The former is straightforward: if someone is willing and able to assign their $R^*(1)$ just above the cutoff, no one must be willing or able to assign their $R^*(1)$ just below it.
The latter can be critical for some designs. bradley_unions_2017 study the impact of labor union (National Labor Relations Board) formation in U.S. firms on patent-related variables. Many studies use union voting as a device for an RD design. dinardoEconomicImpactsNew2004 describe a typical process for a U.S. union formation. Among their described procedures, the followign Step 5. suggests that a re-election can be proposed via objection to the initial election.
The re-election is manipulation, although it may sound as innocuous because no one can ensure the final election results via re-election. However, re-elections violate the condition for $R^*(0)|M=0$ (ref) because objections would be reasonably made only from the failing side that can be for or against the union.
Many designs should be able to detect manipulation. For example, Angrist_Lavy_Leder-Luis_Shany_2019 is a follow-up study of Angrist_Lavy_1999 that exploits the Maimonides' rule in the Israeli school system: a school with $41$ students must have two classes of $20$ and $21$. The treatment is the assignment to a smaller class, but schools may have manipulated the enrollment as “schools are warned not to move students between grades or to enroll those overseas so as to produce an additional class” Angrist_Lavy_Leder-Luis_Shany_2019 by the Israeli Ministry of Education (MOE). Such manipulation is costly and should occur only to exceed the cutoff because “School leaders might care to do this because educators and parents prefer smaller classes. MOE rules that set school budgets as an increasing function of the number of classes also reward manipulation” (Angrist_Lavy_Leder-Luis_Shany_2019). Hence, the density test is valid for their design. otsuEstimationInferenceDiscontinuity2013 and Angrist_Lavy_Leder-Luis_Shany_2019 report the discontinuity of the density in Angrist_Lavy_1999 data. While continuous density is sufficient, it is not necessary for identification. Angrist_Lavy_Leder-Luis_Shany_2019 verify that the index of socioeconomic status is unrelated to Maimonides's rule conditional on a few covariates, and such logic may justify identification. \footnote{As in (ref) and example (ref), identification failure is driven by a difference in manipulators and non-manipulators. If $E[Y(d)|R^*(m)=c_-,M=m] = E[Y(d)|R^*(m)=c_+,M=m], m \in \{0,1\}$ and these means are the same across $m \in \{0,1\}$, then identification holds with discontinuous densities.} It may also be consistent with Arai_Hsu_Kitagawa_Mourifie_Wan_2021 who report the passage of their fuzzy RD test, which does not involve any restrictions on the density but says nothing about its reduced-form sharp design. Continuous density confirms identification, but designs with discontinuous density may also be salvaged, despite being challenging.
RD identification is valid for an ideal design that assigns treatment as if it is randomized locally at the cutoff. However, individuals may manipulate the running variable. Because manipulation has never been defined in existing models, existing identification is based on high-level conditions that are silent about when a design is ideal under manipulation.
We provide the low-level condition for identification by introducing the potential outcome framework for manipulation. In Section (ref), we derive simple low-level conditions for identification as restrictions on manipulation via the framework. Low-level conditions require that manipulation to be randomization, which can be achieved by prohibiting two precise manipulations: the manipulated running variable must not be precisely assigned on a particular side of the cutoff and units for manipulation must not be precisely selected from a particular side of the cutoff. These restrictions arise from a decomposition of the continuity condition hahnIdentificationEstimationTreatment2001 for identification; the continuity condition follows from the balanced mean potential outcomes weighted by the balanced densities. This decomposition highlights the critical role of continuous density in our framework.
Furthermore, in Section (ref), we established the low-level auxiliary assumption that guarantees that diagnostic tests can detect manipulations. Under the proposed auxiliary assumption, we reveal that the null hypothesis of the popular density test mccraryManipulationRunningVariable2008 implies identification. The auxiliary assumption eliminates manipulation with two-sided incentives. Our restriction is explicit in manipulation as an action to influence the running variable. Hence, the auxiliary assumption can be justified from particular stories about manipulation in consideration.
In Section (ref), we discuss the consequences of our framework in published empirical studies, documenting that identification claims in previous studies may be incomplete and providing remedies against it. Furthermore, we highlight a study that may not satisfy the auxiliary assumption and hence may be incapable of detecting manipulation. The possible detection failure is suggested by a two-sided incentive for initiating manipulation that leads to precise selection from either side of the cutoff. We emphasize that the manipulation, which sounds innocuous in existing models, can be a threat to its detection and identification.
The study has some limitations. First, the distribution of $R^*(0)$ or its proxy variable may be available; however, this additional information may improve testing and identification. Second, the predetermined covariates may have alternative uses based on our analysis. Many studies propose estimating with covariates: Frolich_Huber_2019 propose a method with a multi-dimensional non-parametric estimation; Calonico_Cattaneo_Farrell_Titiunik_2019 develop an easy-to-implement augmentation; Noack_Olma_Rothe_2021 consider flexible and efficient estimation including machine-learning devices; Kreiss_Rothe_2022 and Arai_Otsu_Seo_2021 explore augmentation with high-dimensional covariates. Nevertheless, covariates are rarely used to adjust for a possible identification failure and our results may suggest an alternative use of covariates. Finally, our analysis may not be trivial with a multi-dimensional running variable $R$. It would also be promising to extend our analysis to RD designs with multiple cutoff values for which Cattaneo_Keele_Titiunik_Vazquez-Bare_2016 propose a pooling parameter and its implementation and Cattaneo_Keele_Titiunik_Vazquez-Bare_2021 consider an extrapolation method. Developing conceptual and practical recommendations for these designs is a future issue to be explored.