Detecting and Mitigating Group Bias in Heterogeneous Treatment Effects

Detecting and Mitigating Group Bias in Heterogeneous Treatment Effects

\RUNTITLE{Detecting and Mitigating Group Bias in Heterogeneous Treatment Effects}

\TITLE{Detecting and Mitigating Group Bias in Heterogeneous Treatment Effects}

\ARTICLEAUTHORS{ \AUTHOR{Joel Persson\footnote{This research was initiated during the author's PhD at ETH Zurich. It was completed independently of the author’s current employment at Spotify.}\footnote{Corresponding author, email: [email removed]} } \AFF{Spotify} \AUTHOR{Jurri{\"e}n Bakker} \AFF{Booking.com} \AUTHOR{Dennis Bohle} \AFF{Booking.com} \AUTHOR{Stefan Feuerriegel} \AFF{Munich Center for Machine Learning & LMU Munich} \AUTHOR{Florian von Wangenheim} \AFF{ETH Zurich} }

\ABSTRACT{ Heterogeneous treatment effects (HTEs) are increasingly estimated using machine learning models that produce highly personalized predictions of treatment effects. In practice, however, predicted treatment effects are rarely interpreted, reported, or audited at the individual level but, instead, are often aggregated to broader subgroups, such as demographic segments, risk strata, or markets. We show that such aggregation can induce systematic bias of the group-level causal effect: even when models for predicting the individual-level conditional average treatment effect (CATE) are correctly specified and trained on data from randomized experiments, aggregating the predicted CATEs up to the group level does not, in general, recover the corresponding group average treatment effect (GATE). We develop a unified statistical framework to detect and mitigate this form of group bias in randomized experiments. We first define group bias as the discrepancy between the model-implied and experimentally identified GATEs, derive an asymptotically normal estimator, and then provide a simple-to-implement statistical test. For mitigation, we propose a shrinkage-based bias-correction, and show that the theoretically optimal and empirically feasible solutions have closed-form expressions. The framework is fully general, imposes minimal assumptions, and only requires computing sample moments. We analyze the economic implications of mitigating detected group bias for profit-maximizing personalized targeting, thereby characterizing when bias correction alters targeting decisions and profits, and the trade-offs involved. Applications to large-scale experimental data at major digital platforms validate our theoretical results and demonstrate empirical performance. }

\KEYWORDS{heterogeneous treatment effects; causal machine learning; bias detection; bias mitigation; randomized experiments}

Introduction

Heterogeneous treatment effects (HTEs)---the presence of systematic variation in treatment effects across individuals or subgroups---have become central to how organizations learn about and target interventions across a wide range of domains, including marketing Lemmens2025, Hitsch2024 and healthcare Feuerriegel2024, Kraus2024. Recently developed methods based on machine learning (ML) Athey2016, Wager2018, Chernozhukov2018 have made it possible to estimate highly personalized HTEs by modeling conditional average treatment effects (CATEs) as flexible functions of rich, high-dimensional, and individual-level covariates.

While individualized CATE predictions have been adopted for automated decision-making related to personalization Lemmens2025, treatment effect estimates themselves are typically not interpreted, reported, or audited at the individual level. Instead, they are aggregated to subgroups of substantive interest to the business, organization, or research question at hand, such as demographic segments, risk strata, or markets. This practice reflects both an interest in more generalizable subgroup-level effects and the fact that ML-based CATEs are functions. For instance, digital platforms may personalize recommendations to individual users while inferring effects at the segment level Lemmens2025; medical professionals may deploy individualized treatment rules while evaluating effects by clinically relevant strata (such as young vs. old or smokers vs. non-smokers) Hernan2023; and firms and labor market programs may use ML systems for personalization while reporting outcomes for coarser groups defined by age, location, or other characteristics agrawal2025economics, athey2022effective. More generally, whenever CATE estimates are plotted, tabulated, or summarized over bins or discrete covariates, they are implicitly being summarized to treatment effects for groups.

Despite this widespread practice, it is less recognized that such subgroup-level summaries, in general, do not recover a causal effect. For this to hold, the appropriately weighted group-average of the CATE must equal the estimand of the corresponding group-average treatment effect (GATE). This is a strong requirement and may fail even when the CATE is point-identified, correctly specified, and unbiased and consistently estimated. As a result, subgroup-level aggregates of CATEs learned even under ideal conditions may lack a causal interpretation. More generally, this problem reflects the distinction between unbiasedness, confounding, and collapsibility in causal inference Greenland1999, Huitfeldt2019, Didelez2022, Colnet2023.\footnote{\SingleSpacedXI Collapsibility refers to the property that a conditional effect measure, when properly marginalized, recovers the corresponding marginal effect measure. An estimator may be unbiased and unconfounded yet not collapsible Greenland1999.}

In this paper, we study the problem of detecting and mitigating this group bias in personalized CATE predictions from experimental data. Using a stylized example (Section (ref)), we first show that even state-of-the-art CATE learners trained on experimental data can exhibit systematic group bias relative to the corresponding GATE. Perhaps surprisingly, including group indicators, estimating separate CATE models per group, or using more training data, does not necessarily remove the group bias and may even be practically infeasible. In many settings, groups of interest are defined ex post by different stakeholders, such as, managers, researchers, or auditors, and is often unavailable at training time due to privacy, legal, or operational constraints.\footnote{\SingleSpacedXI Even if group indicators are included, they can be dropped by regularization, effectively limiting the group-wise heterogeneity in treatment effects captured by the model. Moreover, re-training the model separately within each group amounts to re-estimating a nonparametric treatment response function at smaller and potentially noiser subsets of the data, thus requiring even stronger assumptions than for the original model that is presumed to be biased (such as sufficient overlap within each group), and may therefore introduce new forms of bias. Assuming regularization does not drop the indicators and that the stronger within-group identification conditions are met, estimation is still subject to differential sample sizes and signal-to-noise across groups, which may still introduce group bias.} Another key challenge of the problem is that only randomized experiments provide reliable benchmarks for model-implied estimates, yet they cannot produce evidence of treatment effects at the same level of granularity as CATEs from ML. Because potential outcomes are never jointly observed Holland1986 and personalized CATE estimates condition on many individual-level covariates, model-free experimental estimators such as difference-in-means can only be applied at more aggregate levels, such as those that identify GATEs. Thus, rather than measuring or correcting bias at the individual level, we aim to reliably detect and mitigate bias in CATE predictions at the group level---for which experiments provide nonparametric identification and model-free estimation.

We aim to make three contributions:

First, we introduce a general methodology for detecting group bias in CATE predictions from randomized experiments (Section (ref)). We define group bias as a causal estimand capturing the discrepancy between model-implied and experimentally identified GATEs, that is, whether aggregating predicted CATEs recovers the corresponding subgroup treatment effects. We introduce a general, asymptotically normal estimator of this group bias and provide a formal statistical test for detection. The resulting procedure is nonparametric, model-agnostic, and applies to both binary and continuous outcomes, different scales of treatment effects (i.e., additive and relative effect scales), and arbitrary CATE learners under minimal assumptions.

Second, we propose a shrinkage-based bias-correction approach for mitigation. A na{\"i}ve correction by simply subtracting the estimated bias per group, akin to standard bias-correction, removes group bias only in expectation. In finite samples, this may increase the dispersion of group bias among the groups, as it tends to overcorrect for smaller or noisier groups. As a solution, we formulate mitigation as choosing how much to debias (i.e., how much to shrink the correction) in order to minimize the expected loss (risk) of residual group bias. For natural loss functions, we derive the closed-form oracle minimizer as well as feasible estimators, both of which automatically adjust the debiasing to the signal-to-noise ratio of the estimated group bias. We apply our framework to large-scale A/B test data from a leading online travel platform to demonstrate empirical performance and validate our theoretical results (Sections (ref) and (ref)).

Third, we analyze the implications of mitigating detected group bias in personalized CATE predictions for decision-making. By focusing on profit-maximizing personalized targeting, we characterize how bias correction alters the optimal decision rule, targeting decisions, and expected profits, and the factors that contribute to this (Section (ref)). We illustrate our theoretical insights using counterfactual off-policy evaluation on the Criteo Uplift Prediction Dataset Diemert2018. We then discuss the resulting trade-offs and provide practical guidance for implementation (Section (ref)).

Related Work

Our study connects to three strands of research: (1) algorithmic bias in machine learning, (2) bias in causal inference, and (3) bias mitigation and its implications.

Algorithmic Bias in Machine Learning

Our work is related to, but distinct from, the literature on algorithmic bias and algorithmic fairness. In that literature, bias is typically operationalized as a prediction or decision problem: bias is said to exist if there are systematic disparities in predictive accuracy, error rates, or decision outcomes across groups defined by protected attributes Chouldechova2020, barocas2023, castelnovo2022clarification, Corbett2017, De2022. Related work on algorithmic fairness, often motivated by legal or ethical considerations, studies how such disparities should be mitigated, typically by modifying prediction models or downstream decision rules Kleinberg2018. Different from this literature, we do not study bias in individual-level CATE predictions or downstream decisions per se, but bias that arises when personalized treatment effect predictions are aggregated to recover group-level causal estimands.

While causal or counterfactual notions are sometimes used to define measures of bias or fairness Carey2022, nilforoshan2022causal, the object being predicted is typically not itself a causal effect, and the causal estimand obtained after mitigation is often left implicit. Our work differs in that we focus on causal inference, with both the individual-level prediction, the group-level estimand, and the bias obtained once aggregating the former to the latter, all relate to treatment effects.

Methodologically, several papers in the algorithmic bias literature frame bias detection as a statistical testing problem, focusing on bias in outcome predictions or classification decisions Taskesen2021, Diciccio2020, Yik2022. Although we also employ statistical testing, the bias we study is fundamentally different. Because individual-level treatment effects are unobserved Holland1986, bias assessment for CATE models necessarily proceeds through aggregation.\footnote{\SingleSpacedXI Here, we refer to settings in which the CATE model depends on many and/or continuous covariates, as this is precisely the regime that motivates ML-based estimation for personalization. To assess bias in such settings, training a new CATE model is not a viable solution, as the original model is presumed to be biased, and model-free estimates from experiments, such as difference-in-means, identify treatment effects only at more aggregate levels.} In the lens of this literature, our contribution is thus to show that ML models of personalized CATEs will tend to be biased when summarizing the CATES into the causal effects at broader groups of intrinsic interest.

Bias in Causal Inference

Our work also connects to research on bias in causal inference. Here, we distinguish three different issues: bias from identification failures, calibration, and bias from ambiguity in the causal estimand.

First, Gordon2019, Gordon2023 show that ML estimators for causal inference often fail to recover experimental estimates of average treatment effects, essentially revisiting the LaLonde critique Lalonde1986 with modern methods Imbens2024 and marketing data. In these works, the discrepancy reflects an identification problem: observational data do not support the assumptions required to recover the target estimand. Our work highlights a different issue. Even when CATE models are correctly identified, specified, and estimated, aggregating their predictions can fail to recover intended subgroup-level treatment effects. This can arise because models are trained to optimize global rather than group-specific objectives, or because regularization shrinks modeled heterogeneity Chernozhukov2018, Melnychuk2024.

Our work is also related to methods that summarize or evaluate estimated CATEs, but differs in both aims and approach. Chernozhukov2018b aim to summarize the predictive content of CATE for more aggregate groups. They do so by sorting predicted CATES into bins and then using regressions to perform inference on the average CATE per bin. Leng2024 aim at calibrating CATE across its distribution, by regressing the average CATE within calibration bins onto experimental subgroup effects. In these works, “groups” are technical devices endogenously defined by quantiles of the CATE distribution. In contrast, in our work, groups are defined by a manager, policy-maker, or researcher (e.g., markets, segments, or demographic attributes), and the question is whether aggregating individual-level CATE predictions within groups recovers the group-level causal effect, and how to correct for the potential bias. As such, our work is motivated by a practical use of CATE models, rather than by their properties. Moreover, while those work focuses on additive effect measures, our framework also applies to relative treatment effects, such as relative risk and lift factors, commonly used in health sciences and marketing.

In doing so, our research connects to recent econometric literature highlighting when commonly used estimation practices fail to recover the causal estimands researchers intend to estimate Goodman2021, Goldsmith2024. Among these works, the issue is that the average treatment effect is not identified by the estimation strategy in question when individual-level effects are heterogeneous. This issue is related to the concept of collapsibility from the causal inference literature, which characterizes when population-marginal causal effects (such as the ATE) can be recovered from weighted averages of conditional effects Huitfeldt2019, Didelez2022, Colnet2023. We extend this logic from population-level effects to subgroup-level effects, showing when aggregation of individual-level CATEs fails to recover group-average treatment effects (GATEs), and propose methods to detect and mitigate the resulting group bias.

Bias Mitigation and Its Implications

Finally, our work relates to research on bias mitigation and its implications. One line of work in marketing studies bias in ML models for CATE specifically. Ascarza2022 introduce fairness constraints in the training of causal trees to regulate downstream targeting across groups, while Huang2023 propose how to calibrate away bias in CATE estimates induced by privacy-preserving noise in the training data. While both papers focus on personalized CATEs, they target different bias objects and are tied to specific learners, constraints, or data-generating processes. By contrast, we focus on errors that arise when aggregating CATE into group-level treatment effects, while remaining agnostic to the model.

Another stream of research studies the consequences of constraining how predictive models are used. Rambachan2020 develop an economic model of how group-level constraints on predictive allocations affect optimal decision rules in hiring, while related work examines applications in business analytics De2022 and public policy Corbett2017, Chohlas2023. A recurring insight in this literature is that correcting group-level disparities in individual-level predictions does not necessarily yield the intended economic or allocative outcomes. For example, De2022 provide counterpoints to the personalization--fairness “trade-off”, while Chohlas2023 show that allocation rules designed to promote equity can be Pareto-dominated by purely efficiency-oriented policies.

Our analysis of targeting implications is conceptually related but differs in focus. Rather than studying fairness or equity, we examine the trade-off that arises when personalized treatment effect predictions are used both for aggregate causal inference and for individualized treatment decision-making. In this setting, correcting aggregated CATE predictions to recover GATEs improves group-level causal inference but can alter targeting decisions, resulting in profit loss relative to using the raw CATE predictions. Our analysis clarifies when such trade-offs arise and how they depend on the chosen bias-mitigation strategy. We distill these insights into practical guidance for how managers and firms can navigate this trade-off, helping to rationalize why coarser targeting policies may be preferred even when more granular personalization is technically feasible Lemmens2025.

Motivating Example

To build intuition for why group bias can arise in ML models of heterogeneous treatment effects, we consider a motivating example in which a personalized CATE model is trained on experimental data and then evaluated by aggregating predictions to ex post-defined groups. The example shows that systematic group bias can arise even under correct identification and specification, and is not specific to any particular estimation approach. The same logic extends to multiple groups and, as shown later, to relative effect measures.

Suppose we observe data on $N$ individuals indexed by $i=1,\ldots,N$. For each individual, let $X_i$ denote a vector of pre-treatment covariates, $T_i\in\{0,1\}$ a binary treatment assigned uniformly at random, and let potential outcomes be given by

where $\mu_t(x)=\mathbb{E}[Y(t)\mid X=x]$ and the errors $\varepsilon_{i0}, \varepsilon_{i1}$ have mean zero and finite variance. The CATE is

which we assume is a smooth, learnable function of $X$. In this data-generating process, the standard identification assumptions hold: There is no interference between units (stable unit treatment value assumption), and the treatment is randomly assigned, implying no confounding and overlap. As a result, $\tau(x)$ is point-identified, and we can use a model $f$ to obtain a CATE estimate $\widehat{\tau}^f(x)$ from the data $(X_i,T_i,Y_i)_{i=1}^N$.

One way to represent this estimation task is empirical risk minimization, where the objective is to solve

with finite-sample analogs implemented via meta-learners Kunzel2019, causal forests Wager2018, athey2019generalized, and related methods. Another representation is estimating equations: modern causal ML estimators solve moment conditions of the form

for some estimating function $\psi(\cdot)$, possibly after orthogonalization with respect to nuisance parameters $\eta_0$ Chernozhukov2018. This encompasses double/debiased ML, doubly robust (DR) learners Kennedy2023, R-loss minimization nie2021quasi, and other orthogonal learners Foster2023. In all cases, the CATE model is trained to optimize an objective over the data, under regularization and hyperparameter tuning to control model complexity.

Now suppose that, after the CATE model has been trained, a binary group variable $G\in\{1,2\}$ is introduced to analyze the data at different segments of managerial, organizational, or research relevance. Essentially, the variable $G$ partitions the population into two groups of unequal size, with $N_1>N_2$. The group indicator may be included among the covariates $X$ or correlated with them, but it does not affect treatment assignment per its randomization. The induced groups may therefore differ arbitrarily in their covariate distributions and outcome distributions, even though treatment remains randomized within each group. Our interest lies in using the trained CATE model to infer GATEs $\tau_g = \mathbb{E}[\tau(X)\mid G=g]$, $g\in\{1,2\}$ by aggregating the individual-level CATE predictions via subgroup sample averages, $\widehat{\tau}^f_g = \mathbb{E}_N[\widehat{\tau}_f(X)\mid G=g]$.

We illustrate the resulting tension using a simple simulation. We set sample sizes to $N_1=150$ and $N_2=100$, generate $T_i \sim \text{Bernoulli}(1/2)$, and draw two continuous covariates from group-specific bivariate normal distributions.\footnote{\SingleSpacedXI The sample sizes are chosen for visual clarity; increasing them narrows the precision of the group bias of CATE in GATE without changing its sign or magnitude.} The true CATE $\tau(X)$ is constructed as a nonlinear function of the covariates with different group means, and thus generating different GATEs $\tau_g$. Outcomes are generated as $Y_i = Y_i(0) + T_i\tau(X_i)$. We then construct a generic CATE predictor by perturbing the true CATE with group-specific mean shifts and mean-zero noise, chosen so that the predictor is unbiased in the population but systematically biased within groups. We also estimate CATEs using a range of representative and widely used causal ML models, namely, causal forests, metalearners (S-, T-, and X-learners), and the DR-learner, which we implemented using the grf, glmnet, and randomForest packages in R with default hyperparameter values.\footnote{\SingleSpacedXI We implement causal forests using grf, the metalearners with random forests for the outcome regressions, and the DR-learner via 5-fold cross-fitted estimation of the propensity score and outcome regressions using regularized generalized linear models (glmnet), followed by a second-stage regression of the resulting orthogonalized (AIPW) pseudo-outcome on covariates using a random forest.} We include the group indicator $G_i$ when training the models to allow them to capture the group-wise heterogeneity. All models then satisfy point-identification, and are correctly specified.

Figure (ref) summarizes the results. In the left panel, the generic CATE predictions are centered around the 45-degree line, indicating no overall bias. However, conditional on group membership, predictions systematically understate treatment effects for the larger group and overstate them for the smaller group. When these predictions are aggregated to the group level, the resulting GATE estimates differ systematically from the true GATEs (center panel). The bias is statistically significant at the 5% level for both groups, and the sign and magnitude vary. The same qualitative pattern appears for the causal ML estimators (right panel), with the sign and magnitude of the group bias now also varying across models. This demonstrates that even under ideal identification and correct specification, ML models that are unbiased for individual-level CATE can exhibit systematic and uncontrolled bias of group-level effects.\footnote{\SingleSpacedXI An underlying mechanism is that the models are trained to optimize global fit over the full distribution of the data, whether by empirical risk minimization, solving a moment condition, or some orthogonalized version thereof Chernozhukov2018, Foster2023, nie2021quasi. Under regularization and heterogeneous group sizes, covariate distributions, and signal-to-noise ratios, prediction errors then need not cancel symmetrically within the groups chosen ex post.}

Framework

This section provides our framework. We first introduce the relevant estimands and then presenting methods and theory for the detection and mitigation of group bias from experimental data.

Treatment Effect Estimands

We consider individual-level experimental data on pre-treatment covariates $X$, a randomly assigned treatment $T \in \{0,1\}$, and an outcome $Y$ that may be binary or continuous. The random variables $(X,T,Y)$ are drawn i.i.d. from a distribution $P$. The data are partitioned into groups indexed by $G \in \mathcal G$, where $G$ may be contained in $X$. Groups may represent demographic attributes, market segments, or some other one-dimensional partitioning of interest, or they may be defined as a function of observed variables.\footnote{\SingleSpacedXI Groups may also be defined or data-driven manner, provided they are defined independently of the bias detection and mitigation procedures. Examples of the latter include quantiles of predicted outcomes (risk or churn quantiles), or clusters constructed using pre-treatment information} For each group $g \in \mathcal G$, let $P_g$ denote the conditional distribution of $(X,T,Y)$ given $G=g$, which may differ arbitrarily across groups. In a sample of size $N=\sum_{g \in \mathcal G} N_g$ from $P$, group sizes $N_g$ may also vary.

There is a pre-existing ML model $f \colon X \mapsto \widehat \tau^f(X)$, where $\widehat \tau^f(X)$ is its prediction of the (true) CATE, defined on an additive or relative scale as

respectively. The model was trained on some prior data using a possibly unknown objective and procedure. No assumptions are imposed on it beyond standard regularity conditions, such as that its predictions have finite variance, and that we can obtain its CATE prediction for each observation.

We are interested in whether the model's CATE predictions recover the GATE, defined as

respectively. Either estimand $\tau_g$ can be estimated from the treated and controls observations per group, also via appropriate regression adjustment estimators (see Appendix (ref) for details).

Additive effects for continuous outcomes are standard in the statistics and econometrics literature based on the potential outcomes framework. They directly answer questions such as the dollar impact of a promotion on sales or the effect of a medical treatment on blood pressure. In applied research and practice, however, relative effects are also widely used due to their scale-free, percentage-change interpretation and are commonly used when binary outcomes are of primary interest.\footnote{\SingleSpacedXI For relative effects to be well-defined, denominators must be strictly positive. A common ad-hoc fix is to add a small constant to the numerator and denominator, analogous to practices for fitting regression models with zero-valued log-transformed outcomes. However, this can introduce bias. More principled approaches include modeling on appropriate scales (e.g., via log-link GLMs).} For example, in the health sciences, interest often lies in effects on adverse events such as disease onset or death, while in marketing and the tech sector, common binary outcome is conversion. In these settings, $\E[Y(t)] = \Pr(Y(t)=1)$, and so the treatment effect estimands correspond to a causal measure of the risk difference and relative risk, which are standard in the health sciences Greenland1999, Hernan2006, or magnitude and relative measures for “lift”, which are standard in advertising measurement and incrementality testing Gordon2023. It is worth noting that, although additive and relative effects are expressed on different scales, they are equivalent via simple transformations: relative effects are additive effects scaled by the mean baseline potential outcome or, equivalently, they are additive effects measured on the log scale and then transformed back.\footnote{\SingleSpacedXI Any ratio $\E[Y(1)]/\E[Y(0)]$ can be written as $\E[Y(1)-Y(0)]/\E[Y(0)] + 1$ or equivalently as $\exp(\log\E[Y(1)] - \log\E[Y( 0)])$.}

Our framework applies to both additive and relative effect measures. The distinction matters because aggregation from CATEs to GATEs behaves differently across the scales, as we show in the next section.

Group Bias

To formalize group bias of CATE predictions relative to GATEs, we first distinguish between two sources: estimation error in the CATE itself and bias induced by aggregation. This distinction is crucial, as only the former represents the underlying CATE error of interest, while the latter must be controlled to ensure that such errors propagate into an error in the GATE rather than an ill-defined or misaligned estimand. Separating the two allows detected group bias to be interpreted as error in the GATE and ensures that statistical inference targets precisely this object.

We begin by contrasting what holds at the population level with what occurs in finite samples for a given group. When treatment effects are defined on the additive scale, the population ATE satisfies

This equality explains why most ML models of the ATE first estimate $\tau(X)$ and then averages them over the covariate distribution used for training. In finite samples, estimation errors in CATE tend to cancel symmetrically when averaged over the same distribution, yielding an accurate estimate of the ATE.

However, this aggregation property does not generally hold over arbitrary subgroups. The reason is not a failure of identification, but of estimation combined with aggregation: CATE is either estimated via a single model trained to minimize error over the population, or separately for subgroups that vary in size and distribution. Then, estimation errors need not cancel within groups, nor be equal in magnitude across groups. This issue is exacerbated by that causal ML models typically rely on asymptotic guarantees and regularization, which can penalize heterogeneity unevenly when sample sizes or signal-to-noise ratios differ.

When treatment effects are instead measured on a relative scale via ratios, an additional complication is introduced. Because ratios aggregate nonlinearly, the expectation of relative CATEs generally does not point-identify the corresponding group-level estimand. By Jensen’s inequality, we have

Hence, the simple average of relative CATEs recovers an estimand that is not a GATE but instead (weakly) upward biased relative to it. Because Eq. (ref) is derived with respect to the true CATE, this issue arises even when $\tau(X)$ is known or estimated perfectly.

To address bias from aggregation, and ensure group-level errors reflects the CATE model itself, we draw on the concept of collapsibility Colnet2023.

Collapsibility formalizes when a conditional treatment effect estimand can be aggregated to its marginal estimand. It is a property of the estimand and not just of its estimate. Additive effects are collapsible with uniform weights ($W=1$), recovering the familiar linear aggregation result mentioned earlier. For relative effects, the appropriate weights are $W(X)=\E[Y(0)\mid X]/\E[Y(0)]$ Greenland1999,Colnet2023, reflecting that the marginal relative effect is a ratio of means rather than a mean of ratios (cf. Eq. (ref)), and therefore requires weighting conditional ratios by their baseline potential outcomes.\footnote{\SingleSpacedXI Not all effect estimands are collapsible; e.g., conditional (log) odds ratios are not.}

We can now define a causal estimand of group bias in CATE.

The expression mirrors the canonical definition of bias in statistics, but conditional on a group and appropriately weighted so error in CATE propagates to bias with respect to the causal estimand of GATE.

As shown in Section (ref), the group bias will generally differ across groups. To assess whether bias is unevenly distributed, one can consider contrasts in the group bias. A natural measure is the cross-group bias $b_g - b_{-g}$, where $b_{-g} = \sum_{k \in \mathcal{G} \setminus g} b_{k} \Pr[G=k]$ is the average bias across all other groups. This estimand captures whether the bias induced by a CATE model for group $g$ differs systematically from that of the remaining population, and provides a basis for across-group comparisons.\footnote{\SingleSpacedXI One may also consider pairwise contrasts $b_g - b_h$ for any $h \neq g$ in $\mathcal G$, scalar dispersion measures such as $\Var_G[\widehat B_g]$ for estimates $(\widehat B_g)$, or empirical densities over $(\widehat B_g)$. We later provide results based on the latter two (cf. Appendix (ref) to the empirical results in Section (ref) and Appendix (ref) on a simulation study.}

Two examples illustrate why detecting and mitigating the group bias can matter in practice.

Detection

Estimation via Decomposition of Group Bias.

We now describe a general approach for estimating and detecting group bias in CATE predictions. The main idea is that, although the CATE $\tau(X)$ is unobservable, the group bias of a CATE model can be decomposed as simply the difference between two group-level aggregates that are directly estimable:

where the first equality follows by applying linearity of expectations to Eq. (ref) and the second by Definition (ref). Here, $\tau^f_g = \E_{P_g}[W\widehat{\tau}^f(X)]$ is the potentially-biased GATE parameter implied by the CATE model using the true weights, and $\tau_g$ is the corresponding GATE estimand defined in Eq. (ref). This decomposition holds for any model of CATE used and irrespective of whether effects are defined as differences or ratios. Therefore, a general estimator can be written as

where $\widehat\tau^{f}_g = \E_{N_g}[\widehat W \widehat{\tau}^{f}(X)]$, and $\widehat W$ are estimates of the weights $W=\E[Y(0)\mid X, G]/\E[Y(0)\mid G]$ if effects are measured on a relative scale (cf. the discussion after Definition (ref)) and $\widehat W=1$ if not, and $\widehat\tau_g$ is an estimate of GATE for the same group. Hence, estimating the group bias in CATE does not require the actual CATE: it suffices to compare properly aggregated predictions to a direct estimate of GATE.

In randomized experiments, the weights required to collapse ratio CATEs are nonparametrically identified and easily estimated. In particular, randomization implies that $\E[Y(0)\mid X, G]/\E[Y(0)\mid G]$ is identified by $\E[Y\mid X, G, T=0]/\E[Y\mid G, T=0]$, which can be estimated by fitting regression models to the non-treated using a group indicator and covariates as controls, and then taking the ratio of fitted values per observation.\footnote{\SingleSpacedXI It is useful to consider when weighting for collapsing ratio CATEs is necessary. Because the weights are identified by $\E[Y\mid X, G, T=0]/\E[Y \mid G, T=0]$, their estimates approach a value of one when outcome heterogeneity with respect to covariates is negligible among the non-treated observations in a group. In that case, the weights may, in principle, be ignored for ratio CATEs.} The estimated GATE $\widehat\tau_g$, in turn, is obtained via the appropriate contrast-in-means (difference or ratio) or regression-adjustment estimator; see Appendix (ref).\footnote{\SingleSpacedXI This type of bias assessment of comparing a model-based treatment effect estimate to an experimental one parallels the classical evaluation in Lalonde1986 and its revisits for ML methods in marketing settings Gordon2019,Gordon2023. Here, however, we assess bias in subgroup treatment effects implied by individual-level predictions, i.e., across levels of aggregation in HTEs.}

Inference and Statistical Test.

In practice, a point estimate $\widehat B_g$ rarely suffices as evidence of the group bias; instead, we want to perform inference using a statistical test. The following proposition provides the sufficient conditions for this.

Proof. See Appendix (ref). $\square$

Proposition (ref) establishes valid inference for the group bias $b_g$ under standard regularity conditions.\footnote{\SingleSpacedXI Here and throughout, asymptotic arguments are with respect to the observations used to form the group-level aggregates $\widehat \tau^f_g$ and $\widehat \tau_g$, and we use $N_g$ to denote the effective sample size determining their joint convergence.} If the collapsibility weights are consistently estimated, then $\E_{N_g}[\widehat W\widehat\tau^f(X)]$ consistently estimates its parameter $\tau_g^f$ as the number of observations for the average grows, while the direct estimator $\widehat\tau_g$ is $\sqrt{N_g}$-consistent and asymptotically normal for $\tau_g$. As such, the difference $\widehat\tau_g^f-\widehat\tau_g$ isolates $b_g$ increasingly well with more data. In that sense, $\widehat\tau_g^f$ and $\widehat\tau_g$ are nuisance components: they are not of interest in themselves but must satisfy consistency and regularity conditions to enable inference on $b_g$.

The assumption of joint asymptotic normality is mild. Conditional on the fitted CATE model, both nuisance components are averages (or smooth functions thereof) of random variables with finite second moments, so standard central limit arguments apply. Sample splitting can be used to eliminate covariance between the two components if desired, at the cost of a reduced effective sample size.

Under the null hypothesis $H_0 \colon b_g = 0$ of no group bias, Proposition (ref) implies that $Z_g = (\widehat B_g - b_g)/\sigma_g$ is approximately distributed as a standard normal. Replacing the unknown standard error with an estimate thereof thereby allows us to use a standard Wald test. As such, we reject the null hypothesis against its two-sided alternative and say that we have detected group bias if

where $z_{1-\alpha/2}$ is the $1-\alpha/2$-quantile of a standard normal and $\widehat \sigma_g$ the standard error.

Closed-form expressions for the standard error exist but are cumbersome to derive, as they depend on the weights and the covariance between the model-implied and experimental GATEs (unless independent samples are used), all of which themselves depend on whether effects are measured as differences or ratios. As a simple and general solution, one can use an appropriate bootstrap scheme. Appendix (ref) provides an implementation with the usual nonparametric bootstrap.

For testing for cross-group bias, the null hypothesis of interest is $H_0 \colon b_g = b_{-g}$, which we evaluate with estimated difference $\widehat B_g - \widehat B_{-g}$. Here, $\widehat B_{-g}$ is the estimated bias on complement of group $g$, obtained analogously. Valid inference follows by an application of Slutsky's lemma to Proposition (ref), provided the focal group is not asymptotically small relative to the remainder. For either test, family-wise error across multiple groups can be controlled using a Bonferroni correction.

Mitigation

Problem and Objective.

Our next step is to mitigate the group bias out-of-sample. A natural but naïve strategy is to simply subtract the estimated group bias $\widehat B_g$ from each new prediction, yielding adjusted CATE predictions $\widehat\tau^{f}(X)-\widehat B_g$, analogous to classical bias correction. If $\widehat B_g$ estimated $b_g$ without error, this adjustment would eliminate group bias exactly.

However, in practice, $\widehat B_g$ is subject to estimation error that depends on the sample size and signal-to-noise ratio for the group. Here, the problem is not the estimation procedure itself, as $\widehat B_g$ is unbiased and consistent per Proposition (ref), but rather its sampling variability---we may get an “unlucky draw” of detection data that makes $\widehat B_g$ a poor approximation of $b_g$, even though the estimator has the desired properties. This issue will be more problematic for smaller or noisier groups, which will be subject to greater variance and sensitivity to outlier observations. As a result, the naïve strategy may latch onto noise and amplify the group bias unevenly across groups, leading to greater cross-group bias compared to before the mitigation.

As a solution, we introduce a group-specific shrinkage factor $\gamma_g\in[0,1]$ and instead correct new predictions as

This formulation nests the range of possible debiasing strategies: $\gamma_g=0$ implies no correction, $\gamma_g=1$ implies the naïve strategy of a full correction, and any value of $\gamma_g$ between zero and one implies an intermediate strategy with a different bias-variance trade-off.

To make the trade-off explicit, consider the residual group bias left after debiasing, $b_g - \gamma_g \widehat{B}_g$. By Proposition (ref) and the standard rules for linear combinations of random variables, this residual bias has an ex ante expected value of $(1-\gamma_g)b_g$ and variance $\gamma_g^2\sigma^2_{g}$. Less shrinkage (i.e., $\gamma_g$ closer to one, meaning more debiasing) therefore trades off greater bias reduction against variance inflation, while more shrinkage (i.e., $\gamma_g$ closer to zero, implying less debiasing) achieves the opposite. Since both bias and variance contribute to the finite-sample deviation of $\widehat B_g$ from $b_g$, any effective method for choosing $\gamma_g$ must balance this trade-off according to the accuracy and precision of the estimated group bias per group.

We balance this trade-off by framing mitigation as a statistical decision problem---incorporating a loss function and aiming for risk minimization under uncertainty. Let $L(b_g-\gamma_g\widehat B_g)$ be the loss from a debiasing error. We want to choose $\gamma_g$ to minimize the (Bayes) risk of such an error, that is,

where the expectation is over the sampling distribution of $\widehat B_g$ from the detection stage. For a given loss function $L(\cdot)$, the minimizer $\gamma_g^{L^*}$ determines how strongly to act on estimated group bias, thereby representing a particular debiasing strategy.

Optimal Debiasing.

We next derive optimal debiasing strategies for mitigating group bias in terms of the shrinkage parameter $\gamma_g$. We consider two natural loss functions: mean debiasing error (signed linear loss), and mean-squared debiasing error (corresponding to MSE). We obtain three debiasing strategies: an mean error strategy, which yields a binary decision of whether to debias, an MSE$-$ strategy, which yields a continuous correction that trades off bias reduction against estimation variance, and an MSE$+$ strategy, which is a simpler approximation of the latter. Our main result is that the oracle solutions, as well as their feasible estimators, admit closed-form solutions that automatically adapt to the statistical uncertainty in bias detection. We start with the optimal solution under mean-error loss.

Proof. This result is straightforward, and we therefore provide a proof sketch. By unbiasedness of $\widehat B_g$ per Proposition (ref), the mean debiasing error is

It follows that one should debias fully ($\gamma_g = 1$) if true bias is present and not debias otherwise ($\gamma_g = 0$). $\square$

Because $b_g$ is unknown, we implement this rule by substituting the test decision from the detection stage. The feasible estimator of $\gamma^{\mathrm{ME}}_g$, which we call the mean-error strategy, is thus

Optimal debiasing under mean-error loss is thus determined by the signal-to-noise ratio of the estimated bias. Smaller or noisier groups tend to exhibit larger estimated bias but also greater estimation variance; the rule accounts for this by debiasing only when evidence is sufficiently strong. In this sense, the significance level $\alpha$ reflects our risk tolerance: smaller values of $\alpha$ (e.g., 0.01) imply that we debias only when evidence is strong (risk-averse), while larger values of $\alpha$ (e.g., 0.10) imply we debias more liberally (risk-tolerant).

We now turn to optimal debiasing under squared loss. The mean-squared debiasing error is

which recovers the familiar bias-variance trade-off, but in terms of how much to debias as controlled by $\gamma_g$.

Proof. See Appendix (ref). $\square$

While the mean error strategy yields a binary decision of whether to debias, the MSE minimizer prescribes how much to debias. Holding $b_g^2$ fixed, $\gamma^{\mathrm{MSE}}_g \to 1$ as $\sigma_g^2 \to 0$, and $\gamma^{\mathrm{MSE}}_g \to 0$ as $\sigma_g^2 \to \infty$, meaning that more precise estimates during detection lead to stronger debiasing.\footnote{\SingleSpacedXI Equivalently, a rational decision-maker hedges against estimation noise and debiases less when evidence is imprecise. Because of this, the MSE-minimizer can also be interpreted as a form of classical empirical Bayes shrinkage.}

Note that $\E[\widehat B_g^2] = \sigma_g^2 + b_g^2$ for any $\widehat B_g$ satisfying Proposition (ref). Therefore, Eq. (ref) can be rewritten as

This representation suggests two feasible estimators of the MSE-optimal shrinkage, which we can call the MSE$-$ strategy and the MSE$+$ strategy:

The empirical moments $\widehat\E[\widehat B_g^2]$ and $\widehat\sigma_g^2$ can be obtained during the detection stage, for example via bootstrap resampling, as part of constructing the test statistic in Eq. (ref).

It is straightforward to see that $\widehat\gamma^{\mathrm{MSE}-}_g$ is unbiased for $\gamma^{\mathrm{MSE}}_g$. Therefore, it will shrink the debiasing optimally given estimated moments. The other estimator, $\widehat\gamma^{\mathrm{MSE}+}_g$, is easier to implement, as it involves one less moment, but will tend to overcorrect. This is because it replaces $b_g^2$ in the numerator by $\widehat B_g^2$. Since $b_g^2 = \E[\widehat B_g^2]-\sigma_g^2$, this substitution inflates the numerator and thus pulls $\widehat\gamma^{\mathrm{MSE}+}_g$ upward relative to $\gamma^{\mathrm{MSE}}_g$. Nonetheless, it remains useful as a simpler approximation.

Evaluation.

The mitigation procedure minimizes the expected risk of debiasing errors, but does not by itself show how well the correction worked. A natural and tempting approach to evaluate mitigation is to compare the debiased CATE predictions to the GATE estimate $\widehat \tau_g$ obtained during detection. We now explain why this reuse is invalid and, instead, how to obtain valid inference.

Reusing the detection-stage GATE to evaluate mitigation is inappropriate because the same estimate is then used both to select the amount of debiasing and to assess its effectiveness. This induces post-selection bias: the estimated residual bias is artificially shrunk toward zero because it effectively assesses in-sample performance, and inference is invalid since the estimate (and its test statistic) is derived from the data used to optimize the mitigation. As a remedy, one can use sample splitting or hold-out data. whereby, one re-estimates the GATE $\tau_g$ on an independent evaluation sample, for example, a held-out split of the original experimental or from a new experiment, and compare this estimate $\widetilde \tau_g$ to the properly collapsed, debiased CATE predictions. Specifically, the estimator for the residual group bias is

where, just like for the initial detection, the weights $\widehat W$ and GATE estimator $\widetilde \tau_g$ are selected for scale of the treatment effect (i.e., the appropriate contrast-in-means or regression adjustment; see Appendix (ref)).

We now show that this is an unbiased estimator of the residual group bias. If we ignore $\widehat\gamma_g\widehat B_g$ in Eq. (ref), then the remainder is simply a group bias estimate $\widetilde B_g$ on the new data, and Eq. (ref) can be written as

Because $\widehat\gamma_g \widehat B_g$ is already chosen, we are interested in the residual bias conditional on it. Hence,

and so $\widehat B^{\widehat\gamma}_g$ is an unbiased estimator of the residual bias $b_g - \widehat \gamma_g \widehat B_g$ introduced in Section (ref).

For inference, the statistics literature on post-selection inference tells us that, when the data for selection and inference are independent, normal approximations or bootstrap procedures yield valid inference under minimal assumptions Rinaldo2019,Kuchibhotla2022,Rasines2023. Accordingly, testing proceeds as prescribed by Proposition (ref) and the initial detection: testing $H_0\colon b_g - \widehat \gamma_g \widehat B_g=0$ evaluates whether mitigation removed bias for group $g$, and testing differences in this residual bias for a given group compared to that of the rest evaluates whether the mitigation equalized the bias across groups. Appendix (ref) provides a pseudo-algorithm for evaluating the detection and mitigation on historical data.

Empirical Application: Group Bias in a Large-Scale A/B Test

Setting

We apply our framework to data from a large-scale internal A/B test at Booking.com, a leading online travel platform, in which an ML model of CATE was used to infer the lift of a marketing intervention (described later). The company routinely uses ML models of CATE to understand how effects of interventions vary across individuals and subgroups of their customer base. While estimated CATEs are intended to capture meaningful behavioral heterogeneity, they should not exhibit systematic bias across stable user segments (e.g., by geography or demographics).

For this A/B test, an internal team selected country of origin as the group variable of interest for assessing GATEs. This choice reflects both managerial relevance (countries naturally define markets and segments in the travel industry) and internal evidence that treatment effects vary substantially across countries, whether due to underlying conditions or because other determinants of treatment effect heterogeneity correlate with country of origin.\footnote{\SingleSpacedXI Booking.com does not collect sensitive personal attributes such as race or nationality and does not use protected information in its ML systems. Country of origin is not a protected attribute under the General Act on Equal Treatment in the Netherlands, where the company is headquartered.} The CATE model evaluated in this application was an existing, general-purpose ML model trained prior to this analysis. This reflects a common organizational structure in platform companies, where models are developed and productionized by engineering teams, while evaluation, reporting, and decision-making are handled ex post by data science and product teams.

There are several reasons why a user-level model of CATE may fail to recover the GATE defined with respect to country of origin. In digital platform companies, CATE models are typically trained on pooled data, either because this is believed to optimize overall predictive performance or because the customer base in some countries has insufficient sample size.\footnote{\SingleSpacedXI More generally, group-level bias can arise for additional reasons, including limited availability of group attributes at training time due to legal, privacy, or operational constraints. For example, to obtain user-level data on country of origin, an online platform must either ask each user to share it or infer it algorithmically. Such data would then need to be stored and governed in a privacy-preserving and secure way for millions of users, which may not be feasible reputationally, technically, or economically. Importantly, as shown in Section (ref), group bias can arise even when group indicators are included in the CATE model and treatment assignment is randomized.} As a result, users from different countries are inevitably unevenly represented in the training data, both in terms of sample size and the information their covariates provide about treatment effects. Even when CATE predictions capture meaningful individual-level heterogeneity, aggregating these predictions to the country level can yield systematically biased GATE estimates that obscure inference about within-market and cross-market effectiveness.

Detecting such bias is therefore important. An operational advantage of the proposed framework is that it allows the CATE model to remain unchanged and continue operating in production: testing for group bias requires only the model’s CATE predictions and A/B test data, and any bias correction is applied ex post. This makes the approach particularly attractive for large-scale platforms, where models operate in real time and retraining or redeployment is costly and time-consuming.

Data

The A/B test ran between January and March 2020. The treatment was a free benefit to encourage users to complete their hotel bookings. The exact nature of the benefit cannot be disclosed, but similar incentives in the travel industry include free breakfast, late check-out, or room upgrades. The offer was shown to users who met eligibility criteria (described below), were randomly assigned to the treatment arm, and who navigated to hotels included in the campaign.

Incoming user sessions were randomly assigned to treatment (offer displayed) or control (no offer), until both arms had 18.5 million observations. Eligibility required that: (i) the session was on a computer; (ii) the user selected a hotel that was part of the campaign; (iii) their search met a minimum spend threshold; and (iv) the user's planned stay involved at most six people. User sessions that did not meet these criteria were excluded, ensuring that treatment and control groups were comparable on baseline factors.

The unit of observation is a user-session on the desktop website, identified by an anonymized session ID, country-of-origin indicator variable, and eight behavioral, pre-treatment covariates capturing browsing, search, and purchase history. Due to confidentiality, the covariates cannot be disclosed but are known to be predictive of CATE on the platform. Each observation also has the treatment status, a binary booking indicator, and a XGBoost prediction of ratio-CATE. The CATE model was trained on data from identical A/B test that ran one year earlier, using the eight covariates as features. Details on the model and data are provided in Goldenberg2020.

The company decided to restrict the data to countries with at least 10{,}000 observations to ensure reliable results. This is motivated by that the inference guarantees for detection are asymptotic (Proposition (ref)) and because simulations (Appendix (ref)) show better performance at this sample size. As a result, the empirical findings are conservative and less likely to be driven by estimation noise.

Application of Framework

We use ratio-of-means to estimate GATEs and the estimator in Goldenberg2020 for collapsing CATEs, which is standard at the company. See Appendix (ref) for details on the estimator, and Appendix (ref) for simulation evidence of its performance when used for detection and mitigation. We follow our procedure in Section (ref) to evaluate the mitigation. We run 10-fold cross-validation with random splits into detection and mitigation sets, each with fifty bootstrap resamples per group to obtain the sampling distribution statistics required to implement the debiasing strategies.

Empirical Results

Figure (ref) plots the experimentally estimated GATEs against model-predicted GATEs before and after mitigation, providing a direct visualization of group bias and its correction. Before mitigation, the predicted GATEs exhibit a substantially narrower range than the experimental GATEs, as seen by comparing the $x$-axis to the $y$-axis in Figure (ref). This reflects that the CATE model was trained on all data across groups to minimize empirical loss under regularization, thereby shrinking heterogeneity in treatment effects toward the global mean, particularly for smaller or less predictable groups. When CATE predictions are aggregated to the group level, this shrinkage carries over to the predicted GATEs, resulting in systematic bias relative to the experimental estimates.

The corresponding scatters after mitigation are shown in Figure (ref). After correction, the predicted GATEs span a range comparable to that of the experimental GATEs, reflecting the purpose of debiasing. What matters, however, is not only recovering the range, but how closely the corrected predicted and experimental GATE align, as indicated by their proximity to the 45-degree line of no error. The mitigation strategies differ markedly in how they achieve this. Naïve debiasing produces the widest spread around the diagonal, reflecting amplification of noise when bias estimates are imprecise. The risk-minimizing strategies account for estimation variance and can therefore address this. Among them, the MSE$-$ strategy produces the tightest overall alignment with the experimental GATEs, consistent with its optimality under mean-squared loss over the debiasing error. The mean error strategy applies full correction only when evidence is sufficiently strong, leading to more conservative adjustments for smaller groups, resulting in that the predicted GATEs still have a somewhat narrower range that the experimental estimates. The heuristic MSE$+$ strategy exhibits a somewhat intermediate behavior: it has a tighter spread than naïve but more outlier points than the mean error and MSE$-$ strategy.

We finally examine how group bias relates to group sample size. Intuitively, groups that make up a larger share of the data are expected to exhibit smaller bias, since they contribute more to the model’s training objective (assuming the user base is somewhat stable) and therefore tend to be predicted more accurately. Figure 3(a) supports this pattern. Most countries account for only about 1% of the total sample, while just five countries (fewer than one tenth of all groups) make up approximately 5, 8, 10, 22, and 33%---and over 75% collectively. Among the least represented groups, the estimated biases range from about $-2$ to over $+4$ standard deviations. For the five largest groups, by contrast, the estimates do not exceed $\pm 1$.

Figure 3(b) shows that all debiasing strategies improve this pattern, though in different ways. All methods substantially reduce the most extreme upward deviation, from more than four standard deviations away from zero to just above one. However, the naïve strategy also increases the most extreme downward deviation by nearly a full standard deviation, a behavior not observed for the risk-minimizing strategies. Moreover, the naïve strategy disproportionately corrects the single largest group: the right-most purple point in Fig. 3(b) lies exactly at zero. This can be explained by that naïve debiasing optimizes for bias reduction in expectation. The largest group will always have the most precise bias estimate, and so the correction is most effective for that one group. The same logic also explains why naïve debiasing increases bias for the smaller groups: it ignores heterogeneity in estimation variance and instead applies an overconfident, full correction uniformly. The risk-minimizing strategies do not have this weakness; a larger fraction of the smaller groups’ bias estimates fall close to zero under their debiasing.

Implications for Targeting

We now analyze the implications for targeting. We focus on the canonical application of using CATE models for profit-maximizing personalized targeting, where treatment carries a cost and positive binary outcomes yield revenue. Our analysis focuses on the empirically relevant case in which organizations correct personalized CATE models to recover GATEs using experimental evidence, for instance, for purposes of insights, reporting, or model auditing, while continuing to target based on the personalized predictions to exploit all heterogeneity. For this setting, we show that group debiasing induces trade-offs that depend on the accuracy of the CATE model, the uncertainty in estimated group bias, and the profit margins underlying the targeting problem. We discuss the implications of our results and how to navigate them in Section (ref).

Profit-Maximizing Personalized Targeting

Consider a firm that assigns a binary treatment $T \in \{0,1\}$ to users based on covariates $X \in \mathcal X$. Let $Y(T) \in \{0,1\}$ denote the potential conversion outcome under treatment $T$, and let $\Pr[Y(T)=1 \mid X] = \E[Y(T) \mid X]$ denote the corresponding conversion probability. A conversion yields revenue $R>0$, while assigning $T=1$ incurs cost $C \in (0,R)$ regardless of the outcome. The firm seeks to learn a targeting policy $\pi \colon \mathcal X \to \{0,1\}$ to maximize expected profit. The expected profit from assigning $T$ is

Assigning treatment is therefore optimal whenever

Hence, the optimal policy $\pi^* \in \argmax_{\pi\in\Pi} \mathbb{E}[\psi(\pi)]$ assigns treatment whenever the relative CATE exceeds the inverse profit-margin $M$, i.e., the incremental lift required for treatment to break even.

The firm can approximate the optimal policy by replacing $\tau(X)$ with an ML estimate $\widehat\tau^{f}(X)$. By the plug-in principle, this yields the empirical profit-maximizing policy

Suppose the firm applies one of the mitigation procedures. The profit-maximizing policy then becomes

The debiased policy continues to maximize expected profit, but using CATE estimates that are corrected to exhibit no statistically detectable group bias. Hence, Eq. (ref) can be interpreted as solving the program

where the constraint restricts the policy space to those for which all group-level bias tests fail to reject at the pre-specified significance level $\alpha$.\footnote{\SingleSpacedXI Eq. (ref) admits an alternative interpretation as a stochastic program with chance constraints. We focus on the statistical-testing interpretation, which directly corresponds to the framework's implementation.}

A key implication is that debiasing induces group-specific thresholds for targeting. This follows from that $\widehat\tau^{f}(X) - \gamma_g \widehat B_g > M$ is equivalent to $\widehat\tau^{f}(X) > M + \gamma_g \widehat B_g$, and thus debiasing shifts the threshold by an amount that varies across groups. As shown in Section (ref), the magnitude of this shift depends on the size and precision of the estimated group bias, so that groups with larger or more precisely estimated bias receive larger adjustments. By contrast, the unconstrained policy based on unadjusted CATE predictions applies a common threshold to all individuals, reflecting the principle that treatment allocations are based on a common standard for all individuals Corbett2017.

Expected Profit Loss and Probability of Altered Targeting

We now consider the consequences of debiasing on targeting decisions and profits. Let

which equals one when mitigation alters targeting. The expected per-group profit change from debiasing is

Eq.'s (ref)--(ref) show that profit is impacted only when debiasing alters targeting, weighted by profit lift from treatment. It is therefore useful to characterize the probability of this event, as done in the following.

Proof. See Appendix (ref). $\square$

Proposition (ref) characterizes the probability that debiasing alters a targeting decision---that is, the probability that the constrained and unconstrained policies disagree for a given user.\footnote{\SingleSpacedXI In the proposition, $b_g(X)$ represents systematic estimation error at covariate value $X$, $\varepsilon(X)$ captures random estimation noise, and $\sigma^2(X)$ denotes its conditional variance. These imply that the CATE prediction error is approximately normal, which holds asymptotically for many modern CATE learners under standard regularity conditions. Here, normality is invoked for analytical convenience, and the qualitative result does not rely on it: any continuous error distribution with no point mass at the decision threshold yields an analogous expression, that differs only CDF being evaluated.} The marginal probability for a given group, or for the population as a whole, is obtained by integrating this conditional probability over the corresponding covariate distribution.\footnote{\SingleSpacedXI The marginal probability for group $g$ is given by $\Pr(\widehat{\pi}(X;\widehat{\gamma}_g\widehat{B}_g)\neq \widehat{\pi}(X)\mid G=g) = \int \Pr(\widehat{\pi}(X;\widehat{\gamma}_g\widehat{B}_g)\neq \widehat{\pi}(X)\mid X)\, \diff P_g(X)$, whereas the for the population, it is given by $\Pr(\widehat{\pi}(X;\widehat{\gamma}_g\widehat{B}_g)\neq \widehat{\pi}(X)) = \int \Pr(\widehat{\pi}(X;\widehat{\gamma}_g\widehat{B}_g)\neq \widehat{\pi}(X)\mid X)\, \diff P(X)$.} Empirically, these marginal probabilities correspond to the share of individuals whose treatment assignment would change after debiasing.

Proposition (ref) shows that four factors determine whether debiasing changes targeting decisions: (a) the magnitude of the correction $|\widehat{\gamma}_g\widehat{B}_g|$, (b) the proximity of the predicted CATE $\widehat{\tau}^{f}(X)$ to the profit-lift threshold $M$, (c) the level of the threshold $M$ itself, and (d) the CATE residual variance $\sigma^2(X)$.

For (a), the correction $\widehat{\gamma}_g \widehat{B}_g$ shifts the effective decision boundary inside the cumulative distribution function (CDF) from $M$ to $M + \widehat{\gamma}_g \widehat{B}_g$. A larger and more precisely estimated group bias $\widehat B_g$, which implies that $\widehat \gamma_g$ approaches 1 (cf. Section (ref)), therefore increases the probability that a CATE prediction crosses the threshold; the opposite holds for smaller or noisier estimates. Hence, more precise detection (and therefore larger bias corrections) raise the chance of altered targeting with profit losses. For (b), units whose predicted CATEs lie close to the boundary $M$ under the unconstrained policy are most likely to be targeted differently under the constrained one, since even small corrections $\widehat{\gamma}_g \widehat{B}_g$ can change whether $\widehat{\tau}^{f}(X) - \gamma_g \widehat{B}_g > M$ holds. For (c), the level of $M$ determines how many units lie near the treatment threshold. Lower thresholds (e.g., when conversion profits are small or treatment costs are high) concentrate more units near $M$, making the policy globally more sensitive to debiasing. Environments with low break-even lifts therefore face inherently higher risk of profit loss from correcting group bias. Finally, for (d), the residual variance $\sigma^2(X)$ appears in the denominator of the CDF. Greater variance thereby flattens the normal CDF and reduces the chance that a CATE estimate crosses the decision boundary, whereas lower variance sharpens the distribution and makes targeting decisions more sensitive to small shifts induced by debiasing.

Illustration on the Criteo Uplift Prediction Dataset

Data and Methods.

We illustrate the targeting implications using the Criteo Uplift Prediction Dataset, also used in related work Leng2024.\footnote{\SingleSpacedXI A description of the dataset and a link to download it is available at \url{https://ailab.criteo.com/criteo-uplift-prediction-dataset/}.} The dataset is a large-scale benchmark for HTE estimation and policy learning, constructed by combining data from multiple advertising campaigns in which users were randomly assigned to receive or not receive display ads Diemert2018.

Each observation corresponds to a user--advertisement impression pair with twelve anonymized pre-treatment covariates $X$, a binary treatment $T$ indicating whether the ad was shown to a user, and a binary outcome $Y$ indicating whether a user converted.\footnote{\SingleSpacedXI All covariates are anonymized and numerically projected to preserve privacy while retaining predictive signal. Randomization checks confirm treatment assignment is independent of covariates. 84.6% of impressions are assigned to treatment, reflecting industry practice of maintaining a small control population to minimize opportunity costs from withholding ad impressions if they have an effect. See Section 3 of Diemert2018 for details on the data.} The dataset contains 14 million observations.

We randomly split the data into three independent subsets: (i) 40% as a training split for fitting the CATE model, (ii) 30% as a detection split for estimating and testing group bias, and (iii) 30% as a targeting split for estimating policy outcomes and profits via off-policy evaluation. Following Eq. (ref), we fit a relative-scale CATE model as a T-learner instantiated with XGBoost, which separately estimates $\E[Y(1)\mid X]$ and $\E[Y(0)\mid X]$ on the treated and control units in the training split. On the detection and targeting splits, we obtain predicted relative CATEs by evaluating the two outcome models per value of $X$ and taking the ratio.

Because the dataset lacks predefined groups $\mathcal G$, treatment costs $C$, and revenues $R$, we construct these to emulate a realistic empirical setting. We define five “baseline-conversion” groups by fitting a regularized logistic regression model on the control units to predict $\E[Y \mid X]$ and taking quintiles of its fitted values, thereby segmenting users by ex-ante purchase propensity. We set conversion revenue to $R=1$ and base treatment cost to $C=0.005$, implying a profit-lift threshold $M=R/(R-C)=1.005$, or conversely, a profit margin of $0.995$, consistent with small per-impression costs in digital advertising. To examine sensitivity to less favorable unit economics, we increase $C$ up to $0.75$, yielding $M \in [1.005, 4]$.

On the detection split, we estimate experimental GATEs via ratio-of-means and compare them to the properly collapsed predicted GATEs from the CATE model using the procedure in Section (ref). This yields estimated group biases $\widehat B_g$ and corresponding mitigation factors $\widehat\gamma_g$, which we use to debias CATE predictions per Eq. (ref). We evaluate four mitigation strategies: the naïve correction, the mean error strategy, the MSE$+$ strategy, and the MSE$-$ strategy (see Section (ref)).

To assess the impact on targeting decisions, we compute the share of observations whose targeting changes after applying a given mitigation strategy. Specifically, for each observation $i=1,\ldots,N_g$, group $g$, and mitigation strategy, we evaluate the disagreement indicator $\varphi$ in Eq. (ref) and take sample averages either within groups or over the full population. These averages correspond to marginal analogs of the conditional probability characterized in Proposition (ref). To study how this share (probability) varies with the determinants in that proposition, we additionally compute these averages across ranges of each determinant.

We then counterfactually evaluate the profit impact using the Horvitz--Thompson estimator horvitz1952, also known as inverse-propensity weighting (IPW). In randomized experiments, IPW is an unbiased and consistent estimator of counterfactual outcomes and thus allows us to quantify the expected profit effect of debiasing. In this setting, the IPW estimator for an arbitrary policy $\widehat{\pi}_0$ and group $g$ is

where the treatment propensity $p \defeq \Pr(T=1)$ is known and constant by the uniform random assignment, and is computed as the empirical treatment rate in the targeting split.\footnote{\SingleSpacedXI If treatment assignment is random conditional on covariates, $p$ may be replaced by an estimate of the conditional propensity score, for example from logistic regression.} By standard results from semiparametric theory, a heteroskedasticity-robust variance estimator is

Taking the difference in $\widehat \psi_g$ between the unconstrained policy $\widehat{\pi}(X)$ and a debiased policy $\widehat{\pi}_g(X;\widehat\gamma_g\widehat B_g)$ yields the estimated profit change $\Delta \psi_g$ in Eq. (ref). We aggregate these per-group point and variance estimates to the population using weights $N_g/N$, via standard rules for linear combinations of random variables. We compute these profit differences over the same ranges of the determinants as for our analysis on targeting decisions, thereby assessing the corresponding implications for profits. The asymptotic normality of the IPW estimator allows us to construct confidence intervals in the usual manner.

Empirical results.

Figure (ref) shows the magnitude of the bias corrections per group and their economic consequences. The naïve strategy yields the largest corrections and profit losses. Both the MSE$+$ and MSE$-$ strategies yield similar corrections yet only one of them produces a statistically significant profit loss, which can only be explained by that it assigns more units to suboptimal treatment status than the other strategy. The mean-error strategy concentrates its corrections almost entirely on the highest baseline-conversion group (group 5) and likewise does not incur a statistically significant profit loss.

We next examine implications for targeting decisions. Figure (ref) plots the share of users whose targeting assignment changes after debiasing as a function of each determinant entering Proposition (ref), following the procedure described in Section (ref). We observe that changes in targeting increase proportionally with the magnitude of the bias corrections. They are most likely for users whose predicted CATEs lie close to the profit-lift threshold, with the naïve strategy additionally altering decisions at larger distances. Increasing the profit-lift threshold (via higher treatment costs) reduces the share of targeting decisions that change substantially, consistent with that targeting decisions in environments with low break-even-thresholds are more sensitive to debiasing. Finally, greater noise in CATE predictions quickly attenuates the impact of debiasing. Overall, the empirical patterns align with the implications of determinants in Proposition (ref).\footnote{\SingleSpacedXI We also compute 95% Wilson score confidence intervals, but they are not visually distinguishable from the point estimates due to the large sample size. For reference, each point in panel (a) is based on approximately 840{,}000 observations, each point along the curves in panel (b) on about 210{,}000 observations, and each point along the curves in panels (c) and (d) on the full targeting split of 4.2 million observations.}

Finally, we examine profit implications of debiasing. Figure (ref) similarly shows how changes in profit vary with the four determinants of whether debiasing changes targeting decisions. The patterns largely mirror those in Figure (ref) but inversely. This follows from that the original empirical targeting is a better approximation of the optimal policy, so any changes induced by debiasing tend to reduce profit.\footnote{\SingleSpacedXI Because some profit differences are close to zero, results are shown in dollar units rather than percentages; these magnitudes are illustrative given the normalization of $R=1$ and $C\in[0.005,0.75]$. What matters is the relative ordering and shape of the curves.} Consistent with this logic, we see that profit losses are approx. linear in the bias correction and concentrate among users near the profit-lift threshold, and they diminish when the profit-lift threshold or the variance of the CATE estimates increases. Across debiasing strategies, the naïve one produces large but also more fluctuating profit reductions, whereas the other debiasing strategies have smaller and more stable impact.

Taken together, these results indicate that the economic impact of debiasing is second-order relative to the extent it alters targeting. When biases and margins are moderate, correcting CATE estimates to recover GATEs may not incur substantial profit losses if one follows a risk-minimizing shrinkage strategy, especially compared to naïve approaches that ignore the estimation uncertainty.

Practical Takeaways and Guidance

We now distill our theoretical and empirical results into main takeaways and practical guidance.

When does group bias arise? Our work point to conditions under which group bias is most pronounced. Group bias is most likely when CATE models are trained on pooled data with heterogeneous group representation, high-dimensional or continuous covariates, and strong regularization, and when groups of interest are defined ex post to model training. In such settings, CATE predictions can have systematic group bias even when correctly identified and estimated on randomized experimental data using a model that is consistent and unbiased of CATE; see the stylized example in Section (ref) and the empirical evidence in Section (ref). By contrast, group bias may be negligible when groups are large, well represented, and the definition of groups is closely aligned the covariate-strata within which CATEs are estimated.

When does debiasing matter? In our framework, the mitigation objective is not to correct downstream decisions, but to correct CATE predictions for group bias when that bias is estimated with heterogeneous precision. Whether this correction has economic consequences depends on how often it alters decisions together with the treatment effect on the economic outcome in question; see Eq. (ref). Our results show that the economic impact is determined by how large the corrections are relative to decision thresholds and how much mass of predicted CATEs lies near those thresholds (cf. Proposition (ref) and Section (ref)). Debiasing has only a second-order impact when biases are modest and profit margins are not tight, particularly when the bias correction is optimized using risk-minimizing shrinkage rather than naïvely applied (Figure (ref)). As such, shrinkage-based bias correction can be preferred when CATE predictions are used both for group-level causal inference and personalized targeting.

Choosing covariates versus choosing groups. As suggested by the discussion above, a central tension concerns the interaction between the richness of covariates used to estimate CATEs and the granularity of groups used to estimate GATEs. When CATE models rely on a small number of categorical covariates, experimental data may support estimation of GATEs at comparable levels of granularity to the CATE, so debiasing can improve the CATE models recovery of GATE with little economic cost when used for targeting. By contrast, when CATE models use high-dimensional or continuous covariates, overlap is sparse and experimentally estimating GATEs at the same granularity becomes infeasible. Practitioners must then choose between coarsening the covariates for CATE or re-defining the groups used to estimate GATE. Both choices improve statistical feasibility but entail costs: coarsening covariates reduces predictive accuracy, while broad groups attenuate meaningful heterogeneity. Either choice can diminish the profitability of personalized targeting, as implied by our characterization in Section (ref).

Coarse personalization has a place. Despite these trade-offs, organizations often report treatment effects and use them for targeting at coarse levels, even when richer data and models are available. Coarser policies can be easier to deploy, scale, and communicate Lemmens2020, and are typically more stable in practice, as flexible CATE models can be sensitive to hyperparameters, sample noise, and random seeds Wager2018. With this in mind, a pragmatic design principle supported by our analysis is to use the coarsest group partition that captures actionable heterogeneity for decision-making, and the richest covariate set that preserves overlap for estimating GATEs (cf. Section (ref) and Appendix (ref)). At the extreme, when targeting policies are based directly on estimated GATEs rather than individual CATEs, mitigating detected group bias improves targeting performance by construction relative to no debiasing. Whether such coarse targeting outperforms personalized policies based on CATE predictions, before or after bias correction, depends on the data-generating process in each particular application.

Within-group rankings are preserved. Because the bias correction is constant across individuals within groups (cf. Eq. (ref)), within-group rankings of individual CATE estimates are preserved. As a result, any policy based on within-group treatment prioritization (e.g., a top-$K$ rule for some $K < N_g$) is mathematically invariant to the bias correction. Therefore, for this class of policies, mitigating detected group bias carries no trade-off, in that group-level causal inferences from the model are improved without altering the economic returns from its personalized targeting decisions.

Conclusion

Recent advances in causal machine learning have enabled researchers and practitioners to estimate highly personalized heterogeneous treatment effects. Such estimates are often aggregated to broader groups with the objective to uncover more generalizable or stable insights. However, these aggregates need not recover a causal estimand and may instead be biased relative to the corresponding group-average treatment effect. This paper studies the detection, mitigation, and implications of such group bias.

We have developed a unified statistical framework for defining, estimating, and testing for the group bias, along with a shrinkage-based bias-correction methodology that adaptively optimizes the debiasing. We have provided practical guidance for implementing and evaluating these methods using historical data from randomized experiments. The framework is agnostic to the choice of treatment effect estimand and CATE learner, imposes minimal assumptions, and requires no additional model training, but only computing sample moments. Finally, we characterized the resulting trade-offs in the context of profit-maximizing targeting and validate our theoretical results using large-scale randomized experiments at digital platforms.

We conclude with limitations and future directions. First, our framework presumes that subgroup GATEs can be reliably estimated. This may be challenging in small or imbalanced samples but can be mitigated using regression-adjustment methods or by redefining groups. Second, while our framework is deliberately agnostic to the source of bias, understanding the underlying mechanisms remains important. Integrating in-processing methods may be useful when the source of bias is known, but risks exacerbating bias when it is not. Third, extending the framework to non-experimental settings with unmeasured confounding would be valuable but poses conceptual and technical challenges. Quasi-experimental designs such as instrumental variables, regression discontinuity, or difference-in-differences can be used to estimate the benchmark group effects, but these designs identify local average treatment effects. Consequently, the CATE estimates must be collapsed to the corresponding local estimand to enable a valid comparison. How to do so, and whether the resulting local bias estimand remains meaningful, are open questions.

Funding and Competing Interests

Authors 1 was an intern at Booking.com at the start of the project that led to this research. Authors 2 and 3 are employed at Booking.com. Authors 4 and 5 have no competing interests.

\ACKNOWLEDGMENT{ We would also like to thank participants of the following conferences and workshop for valuable feedback and comments: CODE@MIT 2025, Workshop on Platform Analytics 2023, ISMS Marketing Science Conference 2023, Annual Theory + Practice in Marketing Conference 2023, The 2023 American Causal Inference Conference, and the 2023 Marketing Science conference on Diversity, Equity And Inclusion. Finally, we would like to thank everyone at Booking.com that enabled the research collaboration. }

\interlinepenalty=10000 \SingleSpacedXI