Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
21,220 characters · 11 sections · 19 citation commands
Causal Inference for Qualitative Outcomes
\doublespacing
Causal inference has become a cornerstone of applied econometrics, with methods such as instrumental variables, difference-in-differences, and regression discontinuity widely employed to identify and estimate the effects of policies or interventions imbens2022causality, angrist2022empirical.
These methodologies are primarily developed for numerical outcomes, whether continuous or discrete. Yet, many empirical applications across disciplines involve qualitative outcomes, which take the form of ordered or unordered categories.\footnote{\ In political science, voter ideology is often recorded on an ordinal scale ranging from “ very liberal" to “ very conservative" mason2015disrespectfully, and attitudes toward policy issues are commonly measured using Likert scales kuziemko2015elastic. In health, researchers analyze life satisfaction frey2002can, subjective well-being reisinger2022subjective, and self-reported health status peracchi2012heterogeneity, peracchi2013heterogeneous---all of which lack a natural cardinal interpretation. In consumer research, perceived product quality and satisfaction are often captured through consumer ratings and review scores mayzlin2014promotional, which are inherently ordinal.} In such cases, widely used causal estimands, such as the average treatment effect and its conditional versions, are ill-defined, as arithmetic operations on outcome categories lack meaningful interpretation.
This paper highlights the fundamental challenges that arise when conventional causal inference methods are applied to qualitative outcomes and introduces a principled approach to addressing these issues. A common approach among applied researchers is to dichotomize the outcome based on an arbitrary threshold. However, results may be highly sensitive to the choice of threshold, and the validity of identifying assumptions for one dichotomization does not necessarily imply identification under an alternative thresholding scheme yamauchi2020difference. We advocate for a fundamental shift toward alternative estimands that are meaningful, interpretable, and identifiable within standard research designs.
Specifically, we demonstrate that a naive application of widely used research designs to qualitative outcomes leads to the identification of quantities whose magnitude and sign depend on arbitrary coding, rendering them economically meaningless. We thus shift our focus to well-defined and interpratable estimands that characterize how treatment affects the probability distribution over outcome categories. We establish that conventional identification assumptions---or slight modifications thereof---suffice for identifying these new estimands, ensuring compatibility with standard research designs. For estimation, we rely on conventional econometric methods, but explicitly model category probabilities. We further provide an accompanying open-source R package to facilitate implementation---named causalQual.\footnote{\ Our framework, though introduced with basic examples, naturally extends to more complex settings, including instrumental variables with multiple instruments HLM2025,mogstad2018using,mogstad2018identification,goff2024vector, difference-in-differences with heterogeneous effects and staggered treatment adoption callaway2021difference,SunAbraham2021,roth2023s, deChaisemartin2023,Borusyak2024, goodman2021difference, and fuzzy regression discontinuity designs cattaneo2022regression. We plan to continuously update our R package to incorporate these extensions as they develop. Users interested in specific extensions are encouraged to contribute by submitting feature requests or discussions at \href{https://github.com/riccardo-df/causalQual/issues}{https://github.com/riccardo-df/causalQual/issues}.}
While the issues we highlight may seem straightforward, we note that causal inference with qualitative outcomes remains underdeveloped. Existing research proposes estimands---such as the probabilities that treatment is (strictly) beneficial---that depend on the joint distribution of potential outcomes, whose identification is pursed under restrictive conditions volfovsky2015causal, agresti2017ordinal, chiba2018bayesian.\footnote{\ lu2018treatment, lu2020sharp adopt a partial identification approach, deriving sharp bounds for these estimands using only the marginal distributions of potential outcomes.} In contrast, we introduce a class of estimands that apply to both multinomial and ordered outcomes and achieve identification without requiring additional assumptions beyond those conventionally made in empirical research.\footnote{\ yamauchi2020difference studies a related estimand within a difference-in-differences framework, imposing strong parametric and distributional assumptions on the outcome as well as a distributional parallel trends assumption athey2006identification.}
We study the causal effect of a binary treatment $D_i \in \{0, 1 \}$ on a qualitative---either multinomial or ordered---outcome $Y_{i} \in \{ 1, \dots, M \}$, with $\vector X_i$ as pre-treatment covariates. We adopt the potential outcomes framework neyman1923, rubin1974estimating and use $Y_{i} ( 1 )$ and $Y_{i} ( 0 )$ to denote the outcome that unit $i$ would experience under each treatment level, with $Y_{i} = D_i Y_{i} ( 1 ) + ( 1 - D_i ) Y_{i} ( 0 )$. We further define $p_{m} ( d, \vector x ) := \operatorname{\mathbb{P}} ( Y_{i} = m | D_i = d, \vector X_i = \vector x )$ as the conditional class probability and $\pi ( \vector x ) := \operatorname{\mathbb{P}} ( D_i = 1 | \vector X_i = \vector x )$ as the propensity score.
If $Y_{i}$ were a cardinal numeric variable, the individual treatment effect could be naturally defined as $Y_{i} ( 1 ) - Y_{i} ( 0 )$. In that case, standard causal estimands such as the Average Treatment Effect (ATE) $\tau := \operatorname{\mathbb{E}} [ Y_{i} ( 1 ) - Y_{i} ( 0 )]$ and the Average Treatment Effect on the Treated (ATT) $\tau_T := \operatorname{\mathbb{E}} [ Y_{i} ( 1 ) - Y_{i} ( 0 ) | D_i = 1]$ would be well-defined and directly identifiable under additional assumptions.
However, because $Y_{i}$ is a qualitative variable, such differences are not meaningful. Instead, we define
as the Probability of Shift (PS)---the change in the probability of belonging to category $m$ due to treatment.\footnote{\ For instance, $\delta_{m} > 0$ indicates that the treatment increases the likelihood of observing outcome category $m$. By construction, $\sum_{m = 1}^M \delta_{m} = 0$, reflecting the intutitive trade-off that an increase in the probability of some outcome categories must be offset by a decrease in others.} We further define two conditional versions of PS: the Probability of Shift on the Treated (PST)
and the Conditional Probability of Shift (CPS)
In this section, we discuss the identification of PS and conditional versions thereof across several commonly used research designs. We conclude with a brief discussion on estimation strategies. Identification proofs are provided in Appendix (ref), and simulation results are provided in Appendix (ref).
Selection-on-observables research designs are based on the following standard assumptions imbens2015causal:
Together, these assumptions allow for the identification of $\tau$ and conditional versions thereof. However, the qualitative nature of $Y_{i}$ renders $\tau$ ill-defined. Therefore, we shift our focus to the identification of PS and its conditional counterparts.
Specifically, under Assumptions (ref)--(ref),
This establishes that the standard selection-on-observables assumptions are sufficient to identify the CPS. Moreover, since $\delta_{m} = \operatorname{\mathbb{E}} [ \delta_{m} ( \vector X_i ) ]$ and $\delta_{m, T} = \operatorname{\mathbb{E}} [ \delta_{m} ( \vector X_i ) | D_i = 1 ]$, the PS and PST are also identified.
We focus on the simple case of a single binary instrument $Z_i \in \{ 0, 1 \}$ for ease of illustration. We use $D_i(1)$ and $D_i(0)$ to define the potential treatment statuses and $Y_{i} ( d, z )$ to index potential outcomes by both treatment status $d$ and instrument assignment $z$. We classify units into four distinct compliance types angrist1996identification: always-takers ($at$) ($D_i(1) = D_i(0) = 1$), never-takers ($nt$) ($D_i(1) = D_i(0) = 0$), compliers ($co$) ($D_i(1) = 1, D_i(0) = 0$), and defiers ($de$) ($D_i(1) = 0, D_i(0) = 1$). To shorten notation, we denote an unit's compliance type by $T_i \in \{at, nt, co, de\}$.
A standard approach in Instrumental Variables (IV) settings is to target identification and estimation of the Local Average Treatment Effect (LATE) $\tau_L := \operatorname{\mathbb{E}} [ Y_{i} ( 1 ) - Y_{i} ( 0 ) | T_i=co]$. Identification relies on the following standard assumptions imbens1994identification, angrist1996identification:
However, the qualitative nature of $Y_{i}$ renders $\tau_L$ ill-defined. We thus shift our focus toward the identification of the Local Probability Shift (LPS):
Specifically, Appendix (ref) shows that, under Assumptions (ref)--(ref),
This establishes that the standard IV assumptions are sufficient to identify the LPS.
We focus on the standard sharp Regression Discontinuity (RD) design for ease of illustration. In this setup, we consider a single observed covariate $X_i$---the “ running variable." Units receive treatment if their running variable exceeds some threshold $c$ ($D_i = \vmathbb{1} ( X_i \geq c )$).
A standard approach in RD settings is to rely on the following continuity assumption for identification of the Average Treatment Effect at the Cutoff (ATC) $\tau_C := \operatorname{\mathbb{E}} [ Y_{i} ( 1 ) - Y_{i} ( 0 ) | X_i = c ]$. \setcounter{assumption}{0}
However, the qualitative nature of $Y_{i}$ renders $\tau_C$ ill-defined. We thus shift our focus toward the identification of the Probability of Shift at the Cutoff (PSC):
For identification of PSC, we introduce an alternative continuity assumption.
Under Assumptions (ref) and (ref),
This establishes that a slight modification of the standard RD assumptions is sufficient to identify the PSC.
We focus on the standard two-period Difference-in-Differences (DiD) framework for ease of illustration. We observe units in two time periods $s \in \{ t, t - 1 \}$ and use $Y_{is} ( d )$ to index potential outcomes by both treatment status $d$ and time $s$. No units receive treatment in the pre-treatment period $t - 1$, while in the post-treatment period $t$ a subset of units is treated ($D_i = 1$). In this context, the PST is defined as
A standard approach in DiD settings is to impose the following parallel trends assumption for identification of the ATT, now defined as $\tau_T := \operatorname{\mathbb{E}} [ Y_{it} ( 1 ) - Y_{it} ( 0 ) | D_i = 1 ]$.
However, the qualitative nature of $Y_{i}$ renders $\tau_T$ meaningless, and Assumption (ref) does not enable the identification of the PST. We thus introduce an alternative parallel trends assumption.
Under Assumption (ref), we obtain
This establishes that a slight modification of the standard DiD assumption is sufficient to identify the PST.
For estimation, researchers can rely on conventional methods while explicitly modeling category probabilities rather than the outcome itself. For instance, note that $$ \delta_{m} = \operatorname{\mathbb{E}}[ \operatorname{\mathbb{E}} [ \vmathbb{1} ( Y_{i} = m ) | D_i = 1, \vector X_i ] - \operatorname{\mathbb{E}} [ \vmathbb{1} ( Y_{i} = m ) | D_i = 0, \vector X_i ]]. $$ This suggests that we can construct an estimate $\hat{\delta}_{m}$ by applying standard estimation strategies to the binary variable $\vmathbb{1} ( Y_{i} = m )$ and estimating the ATE on that variable.\footnote{\ Applying such estimation strategies to the subsample of treated units would yield an estimate $\hat{\delta}_{m, T}$ of $\delta_{m, T}$.} Similar reasoning applies to other parameters. This approach allows applied researchers to utilize familiar econometric tools while targeting causal parameters that are both meaningful and interpretable.
This paper highlights the fundamental challenges that arise when conventional causal inference methods are applied to qualitative outcomes and advocates for a fundamental shift toward alternative estimands that are meaningful, interpretable, and identifiable within standard research designs. Our approach ensures validity and interpretability without departing from widely used research designs.
During the preparation of this work the authors used ChatGPT only to improve language and readability. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.
We are grateful to Jana Mareckova, Michael Lechner, and Franco Peracchi for useful comments and discussions. The R package for implementing the methodologies developed in this paper is available on CRAN at \href{https://cran.r-project.org/web/packages/causalQual/index.html}{https://cran.r-project.org/web/packages/causalQual/index.html}. The associated website is at \href{https://riccardo-df.github.io/causalQual/}{https://riccardo-df.github.io/causalQual/}.
\doublespacing \newrefcontext[sorting = nty] \printbibliography