Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
48,185 characters · 15 sections · 32 citation commands
Do covariates explain why these groups differ? The choice of reference group can reverse conclusions in the Oaxaca-Blinder decomposition
\twocolumn[ \icmltitle{Do covariates explain why these groups differ? The choice of reference group can reverse conclusions in the Oaxaca-Blinder decomposition}
\icmlsetsymbol{equal}{*}
\icmlaffiliation{inst1}{Massachusetts Institute of Technology, Cambridge, MA, USA} \icmlaffiliation{inst2}{University of California, Berkeley, Berkeley, CA, USA}
\icmlcorrespondingauthor{Manuel Quintero}{[email removed]}
\icmlkeywords{Machine Learning, ICML}
\vskip 0.3in ]
\printAffiliationsAndNotice{DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.}
As a motivating example, suppose a data scientist at Hospital $H$ notices that mortality rates are noticeably higher at Hospital $H$ than at Hospital $K$. She knows the medical literature links many risk factors ($X$), such as high blood pressure, to mortality ($Y$). The observed differences in mortality could be due to differences in these risk factors between the two groups of patients (distribution of $X$) or to differences in how well each hospital manages these risk factors (distribution of $Y|X$). If she can figure out how much each of these factors is driving the difference in mortality rates, her hospital may be able to change its policy to improve patient care. Before committing to a potentially costly causal analysis, can she use the available information to develop hypotheses for what drives the difference in mortality?
The modern explainability literature in machine learning (ML) provides many methods for attributing outcome behavior to covariates Dwivedi2023. These include Shapley values Shapley1953, Lundberg2017, chen2025shapley, Functional ANOVA stone1994, hooker2004discovering, hookerGeneralizedFunctionalANOVA2007, lengerich2020purifying, fumagalli2025fanova, Accumulated Local Effects apley2020, and partial dependence plots friedman2001greedy, liu2025trees. However, these methods all focus on decompositions within one group, not on attributing differences across groups. By contrast, the Oaxaca--Blinder decomposition (OBD) attributes disparities between two groups to $X$ and $Y|X$. The OBD was developed for linear models oaxaca1973,blinder1973 and remains widely used in this form --- e.g., to analyze gender wage gaps paradaContzen2025, educational attainment differences barhaim2023, and health disparities cartwright2021. Moreover, recent work suggests the OBD might be applied in conjunction with flexible ML decompositions bach2024heterogeneity,quintas2024multiply,quintero2025.
Since the original paper oaxaca1973, it has been well known that the OBD requires choosing one of the two groups (Hospital $K$ or $H$ in our example) as a reference, and that this choice affects the numerical result. This dependence on the reference choice is known as the “index number problem.” Several responses to this asymmetry occur in the OBD literature. Some argue that a single reference group makes the most sense to use in context, as in dinardoFortinLemieux1996's study of unionization, the minimum wage, and economic inequality. Others effectively construct a new reference group by aggregating the original two groups in various ways neumark1988,oaxacaRansom1994,reimers1983,cotton1988,kassenboehmer2014distributional,sharafRashad2016,rahimiNazari2021,Allen2022,bachan2022genderchancellor.
However, in many applications, there will be no clear choice of reference group. Moreover, there is no consensus on the appropriate resolution. Some papers report results for both references ONeill2006,fortinLemieuxFirpo2011,sharafRashad2016,rahimiNazari2021. Some report results for just one reference, with no argument for excluding the other reference zhang_2019, ayubiShahbaziKhazaei2024,paradaContzen2025,singleton2016,lamJamiesonMittinty2021,leythienne2021paygapeu,alHanawiNjagi2022,tao2024gender.
It is possible that, despite the index number problem, both references always yield the same substantive conclusions; e.g., in our example, both references might attribute the mortality differences partly to the distribution of risk factors, differing only in magnitude. In this case, all of the approaches above should yield similar qualitative conclusions. However, if the choice of reference could yield different substantive conclusions, the distinction in approaches would matter; e.g., suppose one reference attributes the mortality difference to the difference in risk factors and the other does not. We are not aware of an investigation into whether the choice of reference groups can yield different substantive conclusions. We fill this gap with a characterization of when the OBD's substantive conclusions are sensitive to the reference group and a theoretical and empirical investigation of how common such sensitivity is.
In what remains, we start by reviewing the index number problem and OBD in more detail in (ref), and we formalize a form of substantive conclusion. We give an existence proof that a real-data problem can yield two substantively different conclusions ((ref)). And we show that differences in substantive conclusions need not be driven solely by model misspecification or small sample sizes ((ref)). In (ref), we demonstrate that certain types of conclusion changes are common in parameter space but uncommon empirically. We resolve this seeming disparity by observing that typical data analyses are not uniformly distributed in parameter space --- and provide intuition for when this non-uniformity might occur ((ref)). We suggest best practices for data analysts in (ref).
We first review the index-number problem and the OBD. We start by considering the case where we have perfect population information about all variables of interest. Then we describe how to proceed with real data.
As a first step toward describing the OBD, we illustrate the index number problem by framing it as a counterfactual exercise as in fortinLemieuxFirpo2011.
We consider an observed outcome $Y \in \mathbb{R}$ and covariates $X$ in some space $\mathcal{X}$. We take two groups, $H$ and $K$. We assume that the tuple $(X,Y)$ is independently and identically distributed within each group $g$ with corresponding expectation $\mathbb{E}_g$. We suppose we observe a gap in the mean outcomes, $\mathbb{E}_H[Y]$ and $\mathbb{E}_K[Y]$, and would like to understand to what extent the gap is due to differences in the distribution of $X$ versus differences in the distribution of $Y\mid X$ between the two groups. For the moment, we assume we have access to population-level quantities, including all expectations.
To analyze the gap between $\mathbb{E}_H[Y]$ and $\mathbb{E}_K[Y]$, we might construct a counterfactual driven only by differences in covariates. Consider again our example, where $Y$ represents mortality and $X$ represents risk factors. In this example, we might ask: (A) How would Hospital $H$'s mortality rate change if its patients had the risk factor distribution of patients at Hospital $K$? But we notice this question starts from the patient group at Hospital $H$ as a reference. We could instead start from group $K$ and ask: (B) How would Hospital $K$'s mortality rate change if its patients had the risk factor distribution of patients at Hospital $H$?
More generally, we might ask: (A) How much would Group $H$'s mean outcome ($Y$) change if Group $H$ kept its outcome--covariate relationship ($Y|X$) but had the covariate ($X$) distribution of Group $K$? or (B) How much would Group $K$'s mean outcome ($Y$) change if Group $K$ kept its outcome--covariate relationship ($Y|X$) but had the covariate ($X$) distribution of Group $H$?
To address the general questions, it will be useful to compute the mean of a counterfactual $Y$ that has the outcome--covariate relationship ($Y|X$) of group $g$ and the covariate ($X$) distribution of group $g'$. By the law of total expectation, this mean equals $M_{g',g} := \mathbb{E}_{g'} [ \mathbb{E}_g [ Y | X ] ]$. Then the difference in mean outcomes between the two groups can be decomposed into differences between each group's mean and the chosen counterfactual mean. The decompositions corresponding to questions (A) and (B) above are, respectively, (ref) and (ref) below.
Traditionally the term corresponding to changing the covariates is called the “explained component” since this part of the difference is explained by the observed covariates fortinLemieuxFirpo2011. The remaining term, corresponding to changing the conditional $Y|X$ distribution, is called the “unexplained component” since it is not explained by the observed covariates. We use the notation $E$ and $U$ for these components, respectively, with superscript denoting the reference group choice fortinLemieuxFirpo2011.
We might interpret $E$ and $U$ as the respective contributions of the covariates and outcome--covariate relationship to the difference in outcome means between groups $H$ and $K$. The index number problem, then, represents the observation that in general, $E^{(K)} \neq E^{(H)}$ and $U^{(K)} \neq U^{(H)}$.
A conceptual resolution? Before proceeding, we observe that there is not an immediate and general conceptual resolution of the index number problem. It is not clear to us in the hospital example above that one direction is more natural --- or even that the two directions represent clearly distinguishable policy questions. There may be resolutions in particular areas of study. For instance, many papers on the gender wage gap seem to favor constructing a new reference group reimers1983,cotton1988,neumark1988,kassenboehmer2014distributional,bachan2022genderchancellor. This tendency might suggest a consensus; however, we also observe published work that takes either men or women as the reference group nielsen2000wage,piazzalunga2019increase,leythienne2021paygapeu,tao2024gender,paradaContzen2025.
Typically the OBD represents the special case of (ref) where the conditional expectation of the outcome given the covariate is linear, as we describe next.
We now restrict to $X \in \mathbb{R}^d$. For each group $g$, we assume that $\mathbb{E}_g[Y \mid X] = \alpha_g + X^\top \beta_g$ for some intercept $\alpha_g \in \mathbb{R}$ and slope $\beta_g \in \mathbb{R}^d$. Then, for groups $g,g'$, we observe $\mathbb{E}_{g'} [ \mathbb{E}_g [ Y | X ] ] = \alpha_g + \mathbb{E}_{g'}[X]^\top \beta_g$. We also introduce the notation $\mu_g := \mathbb{E}_g[X], \Delta \mu := \mu_H - \mu_K, \Delta \beta := \beta_H - \beta_K, \Delta \alpha := \alpha_H - \alpha_K. $ With these choices, (ref) respectively become
(ref) is the Oaxaca--Blinder decomposition (OBD) with reference group $H$, and (ref) is the OBD with reference group $K$.
Recent work suggests that the OBD can be layered with flexible machine learning models to provide an analogous decomposition beyond linear models bach2024heterogeneity,quintas2024multiply,quintero2025. In the present work, we maintain focus on linear models for the following reasons: (1) The linear model version of the OBD remains the one widely used in practice in the present day. (2) We identify substantive problems with correctly specified linear models; we expect practical issues only compound beyond this relatively simple case.
In general, data analysts will have access to real data for each of the two groups. We assume they fit a linear model to data in each group separately. Then they can apply (ref) with the fitted intercept and slope values for each $\alpha, \beta$ and the empirical mean covariate vectors for each $\mu$. In our own experiments below, we use ordinary least squares (OLS) linear regression in each group.
It is common to draw substantive conclusions based on the signs of the empirical explained and unexplained components, often coupled with statistical significance oaxacaRansom1998, jann2008,fortinLemieuxFirpo2011. For instance, consider the hospital example. Suppose now that Hospital $H$ has lower mortality ($\Delta_Y < 0$), and suppose we take $H$ as the reference. If the OBD explained component has a negative sign, we might conclude that group $H$ can attribute its lower mortality rate in part to its risk factors, such as lower blood pressure. In real data, we might additionally require that the observed sign be statistically significant in order to form a conclusion; without significance, we might instead hesitate to conclude that risk factors are driving any difference in mortality. Analogous to the explained component, if the OBD unexplained component has a negative sign, we might attribute group $H$'s lower mortality in part to the quality of hospital care.
The index number problem reflects that the two reference choices for the OBD could yield two different sets of values for the explained and unexplained components. We are not aware of an illustration in the literature that the two OBD directions can lead to substantively different conclusions. We supply such an illustration next for a real-data analysis in healthcare. In (ref), we provide a second illustration, which arises from analyzing differences in income as well as health insurance status using U.S.\ Census data.
Data. To construct an example, we examine data from the PhysioNet cohort of intensive care unit (ICU) patients collected for a study of in-hospital mortality goldberger2000physiobank. The dataset contains routine measurements taken at admission, including heart rate (HR), mean arterial pressure, temperature, and urine output. These values are recorded for every patient at the time of entry into the unit and provide a description of their initial physiological state.
Medical practitioners often look at moderate elevations in heart rate as a first indication of stress, infection, or concerning changes in blood flow and circulation. In the present example, we restrict our analysis to patients whose admission heart rate lies between the $50$th and $75$th percentiles. This patient set, which we refer to as HR quartile two, reflects a clinical presentation in which the heart rate is elevated but not extreme.
Analysis. Within the data set described above, we will compare two patient groups across hospitals (rather than hospital-specific patient groups as in our introductory example). In particular, we let the groups be $g \in \{\text{men}, \text{women}\}$. The mortality rate for men exceeds the mortality rate for women by about $3.5$ percentage points; see (ref). We would like to understand to what extent the observed mortality difference between men and women in HR quartile two might be attributable to observed characteristics upon admission (covariates) vs.\ medical care (outcome given covariates). To that end, let $Y$ be a binary indicator for in-hospital mortality, and let $X$ be the vector of admission covariates comprising heart rate, mean arterial pressure, temperature, urine output, age, and ICU type. Despite the binary outcome $Y$, it is common practice in applied health and policy research to fit separate linear models (via OLS) for the relationship between a binary outcome and covariates in each group, and then apply the OBD jann2008, edokaChangesCatastrophicHealth2017, sujin_LPM_OB, mweembaGapSelfRatedHealth2023.\footnote{In fact, this use of the linear model for a binary outcome is so common that it has a name, the “linear probability model.”} We provide a detailed description of our data analysis --- and our search procedure for a data analysis exhibiting a substantive conclusion change --- in (ref).
Two different conclusions. We report the OBD for each reference group in (ref). We next interpret the substantive conclusions we might draw from each direction. First, consider the case with women as the reference. The explained component is positive ($0.021$), with a statistically significant difference from 0. This result suggests that differences in admission covariates place men at higher mortality risk. A medical practitioner might therefore examine whether various factors before admission differ across gender; these factors include early differences in blood flow and circulation, underlying disease burden, or overall severity at admission.
Second, consider the case with men as the reference. The explained component is negative ($-0.007$), though the difference from 0 is not statistically significant. In this case, the observed covariate differences seem to predict lower (or equal) mortality for men relative to women rather than higher. In other words, the same covariates would now be interpreted as offering men a slight mortality advantage. This interpretation is substantively different than the interpretation that results from using women as the reference.
How does the sign flip arise? Since $\Delta\mu$ is fixed between $E^{(H)}$ and $E^{(K)}$, it must be that some components of $\beta_H$ and $\beta_K$ are changing sign to give $E^{(H)} := \Delta\mu^T \beta_H$ and $E^{(K)} := \Delta\mu^T \beta_K$ opposite signs. In (ref), we show the fitted linear models of in-hospital mortality on admission heart rate, holding all other covariates fixed at their group means. We see that the signs of the slopes differ. In Appendix (ref), we report values for all elements of each vector $\beta_{g}$ in the full analysis. We see that there are multiple covariates $c$ for which $\beta_{\textrm{men},c} < 0 < \beta_{\textrm{women},c}$. And as $\Delta\mu_c$ has the same sign for all of these covariates, each one pushes towards a sign flip. In aggregate, these changes add enough to cause a sign flip in the explained component.
Given the results in (ref), natural questions arise. For example, are such conclusion reversals specific to the explained component? Are these reversals always consequences of model misspecification or sampling variability? Here, we present a simulated example inspired by real-life healthcare data. Our example exhibits sign reversals in the unexplained component of the OBD under correct model specification and large sample sizes.
Motivation. While our example is purely simulated, it is motivated by the following real-life concerns and observations. Body mass index (BMI) is a widely recognized and routinely measured marker of wellness heymsfield2016 and is strongly linked to cardiometabolic outcomes, such as elevated blood pressure brown2000body. A researcher studying cardiac health may use commonly observed covariates like BMI to investigate whether differences in blood pressure between two groups are explainable by differences in basic markers of health between the groups, or differences in how these markers map to blood pressure.
Our example imagines two groups of adults in primary care being followed for elevated blood pressure, using systolic blood pressure (SBP) at a follow-up visit as the outcome $Y$. The BMI ($X$) of each adult is measured at an intake visit. The first, Group $H$, comes from a higher-resource setting, such as an urban neighborhood with structured care pathways, frequent follow up, and better access to healthy food options. The second, Group $K$, comes from a lower-resource setting, where patients face barriers such as longer travel to clinics, infrequent follow up, and limited access to nutritious food. These factors shape both the composition of each group (including average BMI) and the structure of the SBP--BMI relationship. We follow a clinical literature finding an approximately linear relationship between BMI and SBP kaufman1997, cappuccio2008, chen2023; in particular, we assume a separate linear relationship in each group.
While an OBD cannot establish causality without further assumptions, it can suggest where one might look for potential drivers of observed inter-group disparities. If the explained component is effectively zero and the unexplained component (differences in $Y|X$) dominates (with a sign in the direction of the observed outcome difference), it may highlight the need to investigate differences in medical care, such as adherence support or access to antihypertensives. If the unexplained component is effectively zero and the explained component (differences in $X$) dominates (with a sign in the direction of the observed outcome difference), it may be most impactful to target interventions on weight management, nutrition, and physical activity.
Simulated data. We generate the BMI ($X$, in kg/m$^2$) of an individual in group \( g \) from a group-specific normal distribution. We generate the SBP ($Y$, in mmHg) of an individual in group \( g \) as a noisy function of their BMI:
Here $\alpha_g \in \mathbb{R}$ is the group-specific intercept (in units of mmHg); $\beta_g \in \mathbb{R}$ is the group-specific slope, representing the average change in SBP (mmHg) per unit increase in BMI (kg/m$^2$); and $\varepsilon$ reflects individual variability in SBP that is uncorrelated with BMI. We take $\varepsilon$ independent and identically distributed across adult patients, with the same (mean-zero Gaussian) distribution in each group. We present key parameters in (ref) and provide further details about the data-generating process in (ref).
(ref) shows the population SBP--BMI relationships for both groups (two lines). The figure also shows data samples to illustrate the BMI distribution in each group --- as well as how the intercepts, slopes, and BMI distributions interact. Overall, Group $H$ has lower average SBP than Group $K$, owing both to lower average BMI and a shallower slope relating BMI to SBP, perhaps reflecting improved access and adherence to treatment, or availability of non-pharmacological support. However, Group $H$ has a higher baseline SBP ($\alpha_H$), perhaps reflecting genetic factors in Group $H$.
Analysis. Since we have access to all population quantities in this example, there is no need to fit models or check statistical significance. We can compute the OBD directly from the known population distributions.
Two different conclusions. (ref) summarizes the OBD results under each reference group. For both choices of reference group the explained components are negative, so both choices of reference imply that $H$'s lower average SBP is due in part to its lower average BMI. However, the sign of the unexplained OBD component flips depending on the choice of the reference group. When group $H$ is the reference, the negative unexplained component implies that institutional factors further reduce group $H$'s blood pressure relative to group $K$. However, when group $K$ is the reference, institutional factors appear to actually increase group $H$'s average blood pressure relative to group $K$. Intuitively, because blood pressure in group $K$ is so much more responsive to BMI, group $H$'s lower average BMI appears so good as to “over-explain” the difference in SBP; the unexplained component is then forced to be positive to compensate.
In this example, the sign change is in the unexplained component rather than the explained component (cf.\ (ref)). Since we simulate the data according to group-specific linear models, there is no misspecification. And since we work directly with population quantities, the difference in OBD conclusions is not a product of sampling error.
Our examples above serve as existence proofs that conclusion reversals are possible when using the OBD with different references. In what follows, we focus on conclusions based on sign flips. We show that sign reversals (of either the explained or unexplained component) represent a large proportion of the parameter space, in a sense we make precise below. Then we explain why these sign flips still might not be common in practical applications.
We start by concretizing the type of decision we will focus on. In particular, we will consider a conclusion based on the sign of either the explained or unexplained component of the OBD. We will assume we are working with full population information so that a sign flip alone is enough to change conclusions. In practice, flipping from a signed value to zero, or a statistically significant value to an insignificant one can also be considered to change conclusions. We focus here on the more extreme case where we are able to flip to the opposite sign by choosing a different reference.
Explained component. Recall from (ref) the two possible explained components, each corresponding to a different reference. With this notation in hand, we can define changing a signed conclusion at the population level.
From the definition, we can immediately see the following alternative characterization.
Since $\Delta\mu$ is constant across analyses, sign flips in the explained component are straightforwardly driven by differences in sign in the slope between groups.
Unexplained component. Unlike the explained component, sign reversals in the unexplained component arise from the interaction between slope and intercept differences.
The following alternative characterization for sign flips of the unexplained component is analogous to (ref).
See (ref) for the proof. From (ref), we see that group-specific slopes serve a different role in sign flips in the unexplained case relative to the explained case. Suppose we hold fixed the covariate means $\mu_H$ and $\mu_K$. Then scaling up the magnitude of the vector change-in-slopes ($\Delta \beta$) induces sign flips for a greater range of values of the scalar change-in-intercepts ($\Delta \alpha$). That is, larger magnitudes of \( \Delta \beta \) create “more room” for sign flips in the unexplained component.
After characterizing sign flips in (ref), we can ask how likely they are. One formalization of this question is to ask what proportion of the volume of parameter space corresponds to sign flips. An immediate challenge with talking about volume within a parameter space is that the parameter space inherits its units from the covariates and response; in particular, any individual unit choice is somewhat arbitrary (e.g., centimeters vs.\ millimeters), but also different covariates need not be comparable (e.g., temperature vs.\ weight). The usual way to alleviate this incompatibility in one group is to center the covariates and set their scale (e.g., standard deviation) to be 1. We can use similar normalization to center the covariates such that $\mu_K = \mathbf{0} \in \mathbb{R}^d$ and scale the covariates such that $\Delta\mu = \mathbf{1} \in \mathbb{R}^d$ (i.e., $\mu_H = \mathbf{1}$). We note that our standardization choice ($\mu_K= \mathbf{0},\mu_H=\mathbf{1}$) is possible as long as $\mu_K \neq \mu_H$ elementwise (see (ref)).
In the theory that follows, we will find it convenient to restrict our focus to a bounded range of parameters. Namely, we assume that, for some $M > 0$ and all groups $g$, the entries of $\beta_g$ and $\alpha_g$ lie in $[-M,M]$. Note, however, that we do not restrict the choice of $M$, and the proportions we compute in all cases end up having no dependence on $M$.
As the next two results demonstrate, under these choices, the fraction of parameters leading to sign flips is fairly large. First, for the explained component, we find that, under our conditions, half of the parameter space $(\beta_H, \beta_K, \alpha_H, \alpha_K)$ corresponds to sign flips.
See (ref) for the proof.
Second, for the unexplained component, we find that the proportion of parameter space corresponding to sign flips nears 1/2 as the dimension of the covariate vector grows.
See (ref) for the proof. The joint Irwin-Hall probability in (ref) is hard to evaluate analytically, so in (ref) we evaluate it numerically for various values of $d$. For even modest dimension $d$, the fraction of parameter space leading to sign flips exceeds 40%.
While our results above assume the standardization $\mu_K=\mathbf{0}, \mu_H=\mathbf{1}$, we confirm by simulations that we expect similar results when we take each coordinate of $\mu_g$ uniform in the range $[-M,M]$ as well. See (ref) for details.
In (ref), we show that a large fraction of parameter space corresponds to sign flips. However, we here observe that empirical evidence suggests sign flips in practice may be rare. We discuss resolutions to this apparent contradiction in (ref).
An exhaustive search of the literature using the OBD is beyond the scope of the present paper; a paper corresponding to a Stata implementation of the OBD has thousands of citations alone jann2008, and there are implementations in other languages as well Hlavac2022oaxaca. Without trying to form a representative sample, we examined a selection of papers performing the OBD. Of the papers we examined, many did not report both directions of the OBD. Of the papers that reported both directions ONeill2006, fortinLemieuxFirpo2011, sharafRashad2016, rahimiNazari2021, none exhibited a sign flip.
Publication or other biases could affect the published literature. To avoid these concerns and also offer a more systematic investigation, we examined collections of OBD analyses one might potentially run on two publicly available datasets. In each case, our systematic search for sign flips turned up few examples. First, we examined the PhysioNet ICU cohort collected for the study of in-hospital mortality goldberger2000physiobank. We started by evaluating the OBD across nine predefined data subsets: the full cohort, each ICU type category, two age groups (patients younger than $65$ and patients aged $65$ or older), and two groups defined by a median split of the SAPS--I score. For each of these subsets, we fit gender-specific linear models and computed the OBD. We found no sign flips. We then expanded our analysis to a larger and more heterogeneous collection of subpopulations. Across all $139$ data subsets, we found that $27$ exhibited at least one sign reversal --- a far cry from the number predicted by (ref). The HR quartile two example in (ref) was one such example. We describe the data and our search procedure fully in (ref).
We ran a similar search on U.S.\ labor force data; in particular, the raw data come from the 2016 American Community Survey 1-Year Public Use Microdata Sample uscensus_acs_pums_2016. We again purposely constructed a large set of data analyses to search for sign flips in different subsets of the data, comprising all combinations of the factors in Appendix (ref). Out of the 355 comparisons we examined, we found zero cases where the explained component flipped sign and one case where the unexplained component flipped sign. See (ref) for details of this data set and our search procedure. Again, these numbers are substantially lower than those predicted by (ref).
We conclude from these two searches that we do not expect sign flips to be more than empirically rare, but their existence is still possible.
How can it be that sign flips appear to be rare in practice ((ref)) but common in terms of parameter-space volume ((ref))? Our resolution is that regression parameters seen in practice need not be uniformly distributed over parameter space; in fact, we generally expect they are not.
Explained component. First, consider the explained component. Under the standardization from (ref), we can simplify the characterization of when a sign flip happens.
This observation follows immediately from (ref). In practice, we expect that the general direction of $\beta$'s largely agrees, at least in sign, in each of the two groups. For instance, consider the SBP--BMI example in (ref). We expect a positive relationship between BMI and SBP for both the higher-resource group $H$ and the lower-resource group $K$; i.e., we expect $\beta_H, \beta_K > 0$. So, by (ref), we do not expect to see a sign flip. Similarly, (ref) helps us see why few of the hospital mortality analyses ((ref) and (ref)) show explained component sign flips. While we may expect risk factors to have different effect sizes for men and women a priori, there is little reason to think that many factors that increase men's mortality would decrease women's (or vice versa). Indeed, this observation makes the explained component sign flip in (ref) more surprising, as it implies substantive differences in mortality risk in this subpopulation of men and women.
Unexplained component. Next, consider the unexplained component. In this case, a sign flip intuitively requires the intercept and slope effects to pull in opposite directions across groups. When both move in the same direction, so that the group with higher baseline outcomes also exhibits higher returns to covariates, the unexplained component preserves its sign regardless of the chosen reference group. We now formalize these ideas.
See (ref) for the proof. In some domains, intercept and slope advantages might often move in the same direction; (ref) rules out sign flips in such cases. As an illustration, consider an analysis of the gender wage gap that decomposes differences in the average earnings between men and women into a component based on covariates (e.g., work experience) and outcomes given covariates (i.e., how experience is remunerated). (ref) corresponds to the group with higher baseline wages ($\alpha_g$) also having higher returns to experience ($\beta_g$). (ref) is one explanation for why sign flips are rare in our analysis of U.S. labor force data ((ref)): (ref) holds in 230 out of the 355 comparisons we search (64.8%), ruling out unexplained component sign flips in those cases.
However, (ref) is not plausible in all domains. In the ICU setting from (ref), the assumption requires the higher mortality population to also be more sensitive to observed covariates. This relationship may fail to hold when important factors, such as environmental exposure, chronic conditions, or family history of disease, are not recorded in a researcher's dataset. If such omitted covariates are strong predictors of mortality, a population may have a high mortality rate that is less sensitive to observed covariates. Consistent with this intuition, (ref) never holds across the 139 comparisons in the ICU data exercise. In (ref), we explore how the presence of omitted covariates can generate parameter differences $\Delta\beta$ and $\Delta\alpha$ that do preclude sign flips, without invoking (ref).
It is well known that the OBD depends on the choice of reference group. We here demonstrate that this asymmetry can result in substantively different conclusions between the two reference choices. We present real and simulated examples as existence proofs. We show that sign reversals can occur in either the explained or unexplained component and are not solely products of model misspecification or a small sample size. We find that the fraction of parameter space representing sign reversals is large. But our empirical evidence suggests sign reversals may be rare in practice --- since the fitted linear regression parameters in real data analyses are not well modeled by a uniform distribution over parameter space. We recommend that data analysts using the OBD always report results using both reference groups --- to accurately convey the degree of support behind substantive conclusions arising from the OBD.
DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited. This material is based upon work supported by the Combatant Commands under Air Force Contract No. FA8702-15-D-0001 or FA8702-25-D-B002. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Combatant Commands.
Manuel Quintero was supported in part by the Fortunato and Catalina Brescia Fellowship at the Institute for Data, Systems, and Society (IDSS), Massachusetts Institute of Technology.
The code to reproduce all experiments, simulations, and figures in this paper is publicly available at: \url{https://github.com/manuelquinteroc/ReferenceOaxacaBlinder}