Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
36,238 characters · 16 sections · 37 citation commands
From What Ifs to Insights: Counterfactuals in Causal Inference vs. Explainable AI
The concept of \textcolor{black}{the counterfactual---the opposite of a fact---}has played a pivotal role in \textcolor{black}{two distinct data science areas: in causal inference (CI) and in explaining predictions (explainable AI, or XAI). While both areas have made important contributions to theory and practice, they focus on different types of modeling shmueli2010explain: CI focuses on causal explanatory modeling, while XAI arises in the context of predictive modeling.} \textcolor{black}{The concept of the counterfactual in CI and XAI} hinges upon the same fundamental premise---what if scenarios. Yet, these two \textcolor{black}{uses} leverage the concept distinctively along multiple dimensions, including purpose, causality vs. correlation, the estimated quantity of interest, aggregation level, the modified object, and assumptions. These differences offer valuable insights into the nuances of counterfactual \textcolor{black}{uses, highlighting potential avenues for cross fertilization between the distinct areas of CI and XAI. In this work we focus on the counterfactual-oriented causal inference approach (also known as the “potential outcomes framework”), and on the XAI approach of counterfactual explanations. We discuss how counterfactuals} are defined, used, evaluated, generated, and operationalized \textcolor{black}{in CI vs. XAI, highlighting conceptual and practical differences. By comparing and contrasting the two, we hope to identify opportunities for cross fertilization across CI and XAI.}
While the term counterfactual is heavily used by CI and XAI researchers, we find a gap in terms of a clear and comprehensive definition. Some authors introduce the counterfactual via an example glymour2016causal,pearl2018book, wachter2017counterfactual. CI researchers typically use “counterfactual outcome" to refer to an outcome that did not materialize, as in “counterfactual (or potential) outcomes” \citetext{e.g., sjolander2012language; pearl2018book; dickerman2020counterfactual}, while in XAI counterfactual refers to the inputs, such as: “[a counterfactual is a] statement of how the world would have to be different for a desirable outcome to occur.” wachter2017counterfactual. To be able to discuss, compare and contrast counterfactuals in CI and in XAI, we seek a definition that encompasses both the outcome and inputs (treatment). The definition by glymour2016causal is a good start:
We propose the following more comprehensive definition, which specifically refers to outcomes (outputs) and treatments (inputs):
Counterfactuals have a rich history in philosophy, statistics, econometrics, and computer science. The concept was first formally introduced by philosopher David Lewis in 1973, who described counterfactuals as subjunctive conditionals in the context of “possible worlds.” In Lewis's semantics, a counterfactual statement like “If A were the case, then B would be the case” is true if and only if B is true in the closest possible world in which A is true lewis1973counterfactuals. This concept is a cornerstone of philosophical debates, particularly in the philosophy of mind and metaphysics.
In 1974, statistician Donald Rubin introduced the “potential outcomes” framework, which is fundamentally a counterfactual interpretation of causality. In Rubin's framework, the causal effect of an intervention is defined as the difference between the potential outcomes with and without the intervention rubin1974estimating. This interpretation has been influential in econometrics, biostatistics, and social science in general, providing a basis for causal inference from observational data. \textcolor{black}{The counterfactual is also a key component of computer scientist Judea Pearl's structural causal modeling (SCM) approach for identifying causal effects, where counterfactuals are derived from a functional causal model pearl2009causality. The SCM counterfactual reasoning provides machinery for answering counterfactual questions from deterministic and probabilistic models pearl2009causality.}
\textcolor{black}{In the context of prediction, especially from “black box” models, counterfactual explanations provide} \textcolor{black}{an explanation for the predicted outcome of a single instance.} \textcolor{black}{The counterfactual explanation aims to} explain why a classifier made a certain prediction. \textcolor{black}{This predicted class typically leads to a decision regarding the predicted entity (e.g., accept/reject for a loan seeker).}
\textcolor{black}{The counterfactual explanation was first proposed in the context of document classification for explaining why a predictive model labeled a website as containing adult content or not martens2014explaining. Insight into the predicted class is provided by describing the words that, had they not appeared on the website, would change the predicted class. This was later generalized to tabular data wachter2017counterfactual. In this setting, a counterfactual explains what caused the predictive model to label a certain unit as belonging to a certain class in terms of changes in the feature-value combinations that would lead to a different predicted outcome. Note that other popular XAI methods, such as LIME and SHAP, that explain a \textcolor{black}{numerical} prediction score rather than a \textcolor{black}{categorical predicted} outcome \textcolor{black}{(predicted class)}, do not provide a counterfactual (see fernandez2020explaining for a more detailed discussion.)}
Empirical research \textcolor{black}{in numerous disciplines including} marketing, information systems (IS), and strategic management, frequently employs the concept of counterfactuals primarily within the realm of CI goldfarb2011search, flammer2016impact, tirunillai2017does, xu2017battle. Predominantly, these studies \textcolor{black}{focus} on identifying causal relationships, such as the effect of advertising on purchase conversion, using a mix of experimental and observational methodologies. \textcolor{black}{Similarly,} \textcolor{black}{counterfactual queries are common in public health and medicine---two fields frequently faced with urgent, high-stakes questions about the causal effects of various drugs, behavioral interventions, therapies, and surgical procedures. Medical researchers and practitioners wish to know the answers to practically and theoretically important counterfactual questions; they often turn to clinical trials as well as observational studies to answer such questions.} \textcolor{black}{Using our counterfactual definition, the focus in causal inference is on the THEN outcomes that result from pre-selected IF scenarios.} While \textcolor{black}{causal inference} methods have proved instrumental in establishing causality, there is a growing opportunity to further leverage counterfactual explanations, particularly in data-rich environments, to garner additional insights and actionable implications.
Counterfactual explanations, a concept extensively used in XAI, can help researchers and practitioners understand how changes in the input variables could lead to different outcomes, as predicted by the AI model. \textcolor{black}{Returning to our definition of a counterfactual, in counterfactual explanations the focus is on identifying the IF scenario that leads to a change in the THEN portion.} While CI primarily deals with \textcolor{black}{estimating} the overall effect of a treatment or an intervention (e.g., advertising) on an outcome (e.g., purchase conversion) \textcolor{black}{in aggregate}, counterfactual explanations \textcolor{black}{provide prescriptions for action by} exploring how different aspects of the treatment \textcolor{black}{and input values} could lead to different (predicted) outcomes, generally \textcolor{black}{for specific individuals.} \textcolor{black}{Merging methods and concepts from CI and XAI may thus provide useful bridge between descriptive, predictive, and prescriptive applications of data science.}
Consider the example of studying the causal effect of digital advertising on purchase conversion. Traditional CI might estimate the average effect of advertising on purchase conversion.\footnote{The customer's behavior of making an actual purchase of the advertised product--the advertising industry commonly refers to these as “purchase conversions” gordon2019comparison.} But this leaves unanswered questions such as:
Counterfactual explanations can \textcolor{black}{assist in} providing answers to these questions, offering nuanced insights into the complex relationships between features, interventions, and outcomes. \textcolor{black}{One example of a counterfactual explanation used in the domain of advertising leverages a learned classifier to suggest feature transformations that, if realized, \textcolor{black}{would} turn a \textcolor{black}{predicted} low quality advertisement into a high quality \textcolor{black}{prediction} tolomei2017interpretable. A hypothetical explanation might be something like: if your ad's text readability score \textcolor{black}{increased from medium to high}, then your ad's predicted class would change from low to high quality.}
\textcolor{black}{Another example that clearly contrasts the types of counterfactual queries in CI and XAI is job recruitment. Here are two counterfactual queries corresponding to CI vs. XAI: }
In the context of criminal justice, the recent controversy over the use of algorithmic risk assessment tools\footnote{\url{https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing}} again highlights differences in CI and XAI counterfactual queries. For example:
\textcolor{black}{These examples are meant to give an initial glimpse of how the counterfactual plays different roles in CI vs. XAI.}
\textcolor{black}{While the counterfactual has proven an exceptionally useful concept in both causal inference and XAI, to the best of our knowledge these developments have mostly been occurring in parallel without considering how they might differ, and importantly, how they can benefit each other: How might causal explanation benefit from counterfactual predictions? Likewise, how might counterfactual explanations make use of \textcolor{black}{CI} counterfactuals?}
Secondly, empirical research has traditionally focused on causal inference. Yet, there is significant potential in incorporating the use of counterfactual explanations to enrich the theoretical as well as managerial implications of causal empirical research. \textcolor{black}{By uncovering the underlying differences between CI and XAI, we hope that new research and applications will emerge that combine CI and XAI in order to} gain deeper insights, make more informed decisions, and provide more actionable recommendations.
The fields of \textcolor{black}{CI} and \textcolor{black}{XAI} use different terms and notation. For simplicity, we use the following:
Figure (ref) provides a schematic of the different terms and notations.
\textcolor{black}{In this section we examine key dimensions that distinguish between the CI counterfactual and the XAI counterfactual. These include the purpose of using the counterfactual, what is modified, what is estimated/computed, the aggregation level, and assumptions, \textcolor{black}{as summarized by Table (ref)}.}
Counterfactuals serve distinct but interrelated purposes in CI and XAI. In both domains, they fundamentally inform decision-making, albeit with different emphases and \textcolor{black}{in different contexts and for different audiences.}
In the realm of CI, which has its roots in applied econometrics \textcolor{black}{and statistics}, counterfactuals are primarily deployed to \textcolor{black}{estimate the causal effect of an intervention ($T$) on an outcome of interest ($Y$). Ideally, we would want to compare the effect of the intervention ($T=1$) to that of no intervention ($T=0$ \textcolor{black}{or alternative interventions), such as in the case of evaluating a new policy or medication}. The notion of the counterfactual is used to}is used to} tackle the fundamental problem of causal inference holland1986statistics, \textcolor{black}{which is} the challenge of simultaneously observing all outcomes for a given unit $i$ \textcolor{black}{under each intervention/non-intervention}. This gives rise to the missing data problem: if we observe \textcolor{black}{the} outcome \textcolor{black}{($y_i$) for unit $i$ who was given the intervention} ($t_i=1$), what would be the outcome $\tilde{y}_i$ \textcolor{black}{without intervention} ($\tilde{t}_i=0$)? \textcolor{black}{Or vice versa: if we observe the outcome under no intervention, what would be the outcome under intervention? These missing outcomes are called potential outcomes.} Various identification strategies have been proposed to circumvent this issue which essentially revolve around discerning the effect of an intervention or treatment under a hypothetical scenario, informing the decision of whether or not to implement a treatment.
\textcolor{black}{In contrast,} the field of XAI \textcolor{black}{uses} counterfactuals primarily to elucidate the inner workings of complex (often referred to as “black box”) \textcolor{black}{predictive} models. \textcolor{black}{Specifically, a counterfactual explanation martens2014explaining,wachter2017counterfactual,guidotti2022counterfactual describes how the “nearest possible world” ($\tilde{\underline{x}}_i$) has to be different for a desirable outcome ($\hat{\tilde{y}}_i$) to occur wachter2017counterfactual. \textcolor{black}{Counterfactual explanations can help data scientists “debug” predictive models by detecting spurious correlations and unwanted biases in the training data. But arguably their most valuable application is in helping individuals receiving a negative decision from an algorithmic predictive model---such as a loan rejection---understand what would have been needed to receive a positive decision.} Here, the focus is not on uncovering causal relationships per se, but rather on augmenting the interpretability/comprehensibility and transparency of the \textcolor{black}{predictive} model, describing what causes the model ($f$) to generate a certain prediction. The primary question asked might be: “\textcolor{black}{For instance $i$,} which set of features should we change, and by how much ($\tilde{\underline{x}}_i$), to achieve a certain model output ($\hat{\tilde{y}}_i$)?” \textcolor{black}{The answer to this question can} empower users to understand the decision-making process of the model and to make informed decisions based on these insights venkatasubramanian2020philosophical.}an2020philosophical}.}
The contrast between \textcolor{black}{the use} of counterfactuals \textcolor{black}{in CI vs. XAI} underscores an important distinction: While CI aims to comprehend \textcolor{black}{and estimate} causal effects \textcolor{black}{between interventions ($t$) and measurable outcomes ($y$) in the real world, counterfactual explanations in XAI are based on correlations or associations between the predicted outcome ($\hat{y}$) and the input variables ($\underline{x}$) \textcolor{black}{as found in the training data}. The counterfactual explanation is based on the correlation structure of the black box predictive model ($f$), linking the predictions and inputs of unit $i$.} \textcolor{black}{There is no guarantee, however, that observed correlations in the training data accurately reflect causal structures (i.e., data generating processes) operating in reality. Unlike CI-based empirical research, where typically a domain theory guides reasoning about the underlying causal mechanisms and relations underlying the observed data hofman2021integrating, in XAI it is not obvious what an analogous “domain theory” would look like for a model designed, for example, to predict loan default.\footnote{In some cases, however, legal restrictions might provide a basis for including or excluding model features related to race or gender.} For this reason, the predictive models used in XAI can become decoupled from reality and lead to dubious recommendations, e.g. that individuals make causally ineffective feature changes karimi2022survey.}
The quantity of interest in CI is a causal estimand, whereas in the XAI counterfactual explanation interest is in a hypothetical feature value combination. The quantities generated for these purposes end up being different in terms of outcomes vs. inputs as well as in terms of level of aggregation. We explain each next.
In CI the main quantity of interest, or estimand, is the difference between two outcomes. Ideally, for each unit $i$ we would want $y_i-\tilde{y}_i$ (the difference between the measured outcome and the counterfactual outcome\footnote{A causal effect is a comparison of potential outcomes, for the same unit, at the same moment in time, post treatment imbens2015causal}). Decisions by policymakers would then be based on this (ideal) quantity of interest. In contrast, in XAI, the quantity of interest is the “best" combination of feature values for unit $i$: $\tilde{\underline{x}}_i$, which we might call counterfactual input values. Decisions (by individual $i$) would then be based on $\tilde{\underline{x}}_i$. In other words, in CI the quantity of interest is the THEN portion of the counterfactual definition, whereas in XAI it is the IF portion.
As described above, counterfactual explanations remain at the individual level ($i$). However, while the ideal causal effect in CI is at the individual level, due to the FOCI, we cannot estimate the individual-level quantity of interest $y_i-\tilde{y}_i$. Instead, CI estimation resorts to group averages that rely only on observed outcomes ($y_i, i=1,\ldots n$). Examples include the Average Treatment Effect (ATE) that computes the difference between the average outcome of the treated and the average outcome of the untreated; and subgroup averages such as the Average Treatment Effect on the Treated (ATT), or Conditional Average Treatment Effect (CATE). More intricate averages are used when we have a mediator and want to study the mediation mechanism, or when we have multiple treatment variables acharya2016explaining, imai2011unpacking. In all these cases, estimands are based on averages, and therefore CI formulas are typically written using expected values (see Table (ref)).
While the above aggregation levels in XAI and CI are most common, we note that aggregation can also be on a continuum: counterfactual explanations can be aggregated across multiple units to learn more about the data and the model, and CI estimands try to get closer to individual level effects, e.g. by using “synthethic controls" abadie2003economic.
A further distinction lies in the targets of modification in these fields. In CI, the target of modification often involves changing the structural form of the model or adjusting the treatment variable in an attempt to simulate different hypothetical scenarios. This is consistent with the purpose of CI, which is to estimate the causal effect of a treatment or intervention on an outcome. In this context, counterfactuals are used to represent the outcome that would have occurred had the treatment been different. This often requires careful model selection and testing to ensure that the counterfactual scenarios are plausible and consistent across different models.
In contrast, in the field of XAI, the target of modification is the input data to a given, fixed predictive model. Counterfactuals in XAI are used to illustrate how changes in the input features would have led to different model output. Rather than changing the model itself, XAI focuses on understanding how the model responds to changes in the input data. This approach is useful for interpreting complex models, as it provides insights into how important each feature is and, in particular, how much these features should change to get a different model output (counterfactual explanation).
The fields differ in the assumptions they make and the emphasis they place on understanding the system they are working with. CI relies on a multitude of assumptions that represent beliefs or assertions about the underlying structure and processes of the system that cannot be directly tested with the observed data. Consequently, significant emphasis is placed on understanding the data generation process (DGP). Underlying Rubin's potential outcomes framework are several well known assumptions that enable identification and estimation of causal effects:
These are strong assumptions that may not hold in practice. Specifically, assumptions (3) and (4) are based on the premise that the researcher has complete knowledge of the causal structure between the treatment and the outcome. Thus, many of the widely used CI methods suggest relaxing these assumptions and instead proposing a set of more plausible assumptions (see Appendix A).
\textcolor{black}{In contrast, the assumptions of counterfactual explanation methods concern the predictive model. Specifically:
}
\textcolor{black}{Because the end goal in CI and XAI is different, namely to estimate a causal effect (CI) and to provide a counterfactual explanation (XAI), measuring the quality of the resulting object differs. In CI we are interested in evaluating the quality of the estimate (e.g. ATE), whereas in XAI we are interested in evaluating the quality of the explanation. The measures in both cases are significantly different.} In CI, counterfactuals are used to ensure the reliability of the causal estimates and to validate the assumptions underlying the CI methods. This is typically achieved \textcolor{black}{by estimating} the standard error and constructing confidence intervals. \textcolor{black}{In addition, to test the validity of the identification assumptions, a series of robustness checks is conducted}. For instance, in Regression Discontinuity Design (RDD), robustness checks such as the McCrary test and covariate smoothness test are employed dickerman2020counterfactual,flammer2015does.\footnote{Similarly, in the difference in differences approach, data scientists typically present empirical evidence of parallel trends between the treated and control groups xu2017battle.} These tests are designed to ensure that there is no manipulation around the cutoff and that the covariates are smooth around the threshold, respectively. These robustness checks serve as performance indicators for the CI methods, helping to ensure the reliability of the causal estimates.
On the other hand, in XAI, counterfactuals are used differently, and their performance is evaluated based on a variety of metrics. These metrics are designed to assess the quality of the counterfactual explanations generated by the XAI model. For instance, a variety of performance measures are considered, \textcolor{black}{including how realistic the counterfactual modifications are (feasibility), how close the counterfactual is to the instance's input data (proximity), how short the explanation is (sparsity), and how robust the counterfactual is to model changes (stability) guidotti2022counterfactual,karimi2022survey,verma2020counterfactual, sokol2020one}.
\textcolor{black}{The analysis we presented clarifies differences in the use of counterfactuals in both causal inference and in explainable AI. Our discussion centered on how counterfactuals are defined, used, evaluated, generated, and operationalized in these two domains. We hope our discussion shines light on several possible areas of synergy between the fields of causal inference and explanainable AI. Towards this end, we briefly mention a few areas of research that combine prediction with counterfactual explanations.}
\textcolor{black}{One salient example of fusing prediction with causal modeling comes from a stream of machine learning research studying issues of counterfactual fairness kusner2017counterfactual. Predictive counterfactual fairness techniques goethals2023precof could help to identify implicit biases in a learned model by revealing counterfactual explanations such as if you were not a woman, you would have received the loan. These counterfactual explanations can stimulate further research into the social and institutional mechanisms generating the data used to train the predictive model. Conversely, causal models can help ensure that returned counterfactual explanations are truly actionable and therefore useful to end users.}
\textcolor{black}{Another interesting use of the counterfactual concept is in evaluating, predicting, and explaining the effects of personalized interventions in both business schnabel2016recommendations, bottou2013counterfactual and high-stakes medical contexts derubeis2014personalized, oberst2019counterfactual. For instance, there is growing interest in predicting the counterfactual effects of individual healthcare interventions by leveraging causal domain knowledge prosperi2020causal. In some AI-based clinical decision-making systems komorowski2018artificial, the counterfactual of interest involves estimating how a policy (i.e., a decision rule for assigning treatments or recommendations on the basis of observed covariates) would have performed in an online experiment if it had been used instead of the (human) policy that actually generated the data joachims2021recommendations. One might also want to know why a policy performs better than the current one. To answer this kind of causal question, one technique counterfactually removes individual observations from medical intervention datasets and evaluates the resulting impact of these observations on the value of a proposed policy gottesman2020interpretable. These influential data points can then be presented to experts for closer analysis, ultimately fostering potential insights into how to better sequence medical interventions for a variety of patient populations and conditions.}
\textcolor{black}{In summary, the counterfactual is an increasingly popular and powerful concept for imagining novel answers to what if scenarios commonly encountered in science, medicine and public policy, business and beyond. With greater appreciation of the conceptual and methodological subtleties of counterfactual analysis, we see opportunities to both improve our understanding of the causal mechanisms in a given domain and explain the predictions of complex models.}
\singlespacing