Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
43,826 characters · 13 sections · 30 citation commands
Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction
\twocolumn[ \icmltitle{Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction}
\icmlsetsymbol{equal}{*}
\icmlaffiliation{yyy}{Department of Economics, Keio University, Tokyo, Japan} \icmlaffiliation{comp}{CyberAgent, Inc., Tokyo, Japan}
\icmlcorrespondingauthor{Undral Byambadalai}{undral\textunderscore [email removed]} \icmlcorrespondingauthor{Tatsushi Oka}{[email removed]} \icmlcorrespondingauthor{Yasui Shota}{yasui\textunderscore [email removed]}
\icmlkeywords{distributional treatment effects, field experiments, regression adjustment, machine learning}
\vskip 0.3in ]
\printAffiliationsAndNotice
Randomized experiments have played a crucial role in understanding the effects of interventions and guiding policy decisions, ever since the seminal work by fisher1935design. The estimation of causal effects through randomized experiments has found widespread application across various scientific disciplines rubin1974estimating, heckman1997making, imai2005get,imbens2015causal and has also become a standard practice within the technology sector tang2010overlapping, bakshy2014designing, xie2016improving, kohavi2020trustworthy.
When analyzing data from randomized experiments, one commonly used measure is the Average Treatment Effect (ATE). However, it is often the case that understanding the distributional treatment effects can provide a richer perspective than solely focusing on overall average effects. Furthermore, while randomized experiments simplify outcome-based analysis, pre-treatment auxiliary information is frequently available. This quest for a more comprehensive understanding of treatment effects, marked by supplementing auxiliary data, calls for new approaches to enhance precision through pre-treatment data incorporation.
In this work, we propose a novel regression-adjustment method to estimate a wide range of distributional parameters in the randomized experiment setup. Our approach draws inspiration from the generic Neyman-orthogonal moment condition chernozhukov2018debiased, which facilitates the decoupling of nuisance parameter and treatment effect estimation into two stages. The nuisance parameters of our interest are the conditional outcome distributions given pre-treatment covariates, and we propose the use of machine learning models (e.g., LASSO, random forests, neural networks, etc.), allowing for complex data and distributional structures. By integrating these sophisticated machine learning techniques with cross-fitting, we reduce the sensitivity of our treatment effect estimator to errors arising from nuisance parameter estimation.
Our paper makes several noteworthy contributions. First, our approach expands the scope of regression adjustment. While regression adjustment is commonly employed for variance reduction in the estimation of the ATE with mean regression deng2013improving, poyarkov2016boosted, guo2021machine, our method leverages pre-treatment information under the distributional regression framework, incorporating machine learning methods. This enables us to conduct more powerful statistical inference for distributional parameters, including the Distributional Treatment Effect (DTE) and the Quantile Treatment Effect (QTE). Second, we provide theoretical validation for the regression-adjusted method by demonstrating the variance reduction of the estimator of outcome distribution. Third, we establish asymptotic properties for the proposed treatment effect estimators and provide a uniformly valid inference method. Lastly, our simulation and real-data analysis highlight the significance and effectiveness of our method.
The rest of the paper is structured as follows. Section (ref) describes related literature. We setup the problem and introduce notations in Section (ref). Section (ref) then introduces the regression-adjusted estimators for distributional parameters. We derive the asymptotic results in Section (ref). Section (ref) reports empirical results based on simulated experiments and real datasets. Section (ref) concludes. The Appendix in the paper includes all proofs, as well as additional experimental details and results.
\paragraph{Regression Adjustment} There is an extensive literature investigating the use of pre-treatment covariates to reduce variance in estimating the ATE, dating back to fisher1932statistical, followed by cochran1977sampling, yang2001efficiency, rosenbaum2002covariance, freedman2008regression, freedman2008regression2, tsiatis2008covariate, rosenblum2010simple, lin2013agnostic, berk2013covariance, ding2019decomposing, negi2021revisiting, among others, in the case of low-dimensional asymptotics. In high-dimensional settings, this topic has been studied by bloniarz2016lasso, wager2016high, lei2021regression, chiang2023regression, among others. Recent work by list2022using has linked regression adjustment to the semiparametric problem of estimating a low-dimensional parameter when a high-dimensional but orthogonal nuisance parameter is present, focusing on estimating the ATE. Our work extends those existing studies to estimate distributional parameters of treatment effects.
\paragraph{Conditional Average Treatment Effects} To characterize the heterogeneity in treatment effects, an alternative approach is to condition on observed variables and estimate the Conditional Average Treatment Effect (CATE) imai2013estimating, athey2016recursive, johansson2016learning, shalit2017estimating, alaa2017bayesian, wager2018estimation, chernozhukov2018debiased, kunzel2019metalearners, shi2019adapting, nie2021quasi, guo2023estimating, sverdrup2023proximal, van2023causal. The CATE can be regarded as the ATE within subgroups defined by observed characteristics, such as gender, age, prior engagement with the platform, and more. Consequently, the CATE captures observed heterogeneity given the information available to the researchers, whereas our approach is valuable for quantifying unobserved heterogeneity and can be extended to estimate distributional parameters conditional on observed information.
\paragraph{Distributional Treatment Effects} Distributional and quantile treatment effects have long been recognized as important parameters to estimate beyond the mean effects. The quantile treatment effect was first introduced by doksum1974empirical and lehmann1975nonparametrics. Subsequently, estimation and inference methods for distributional and quantile treatment effects in various settings have been developed and applied in econometrics, statistics and machine learning community, including heckman1997making, imbens1997estimating, abadie2002bootstrap, abadie2002instrumental, chernozhukov2005iv, koenker2005quantile, bitler2006mean, athey2006identification, firpo2007efficient, chernozhukov2013inference, koenker2017handbook, callaway2018quantile, callaway2019quantile, chernozhukov2019generic, ge2020conditional, zhou2022estimating, kallus2024localized, among others. Some recent works, including park2021conditional and kallus2023robust, explore the Conditional Distributional Treatment Effects as distributional analysis is useful even after conditioning on observed variables. However, there has been limited research on regression adjustment for unconditional distributional treatment effects. One exception is by jiang2023regression, who consider quantile-regression adjustment for the QTE, but under covariate-adaptive randomization. Another exception is the study by oka2024regression, which investigates the distribution regression approach using finite-dimensional covariates. We bridge this gap by proposing regression-adjusted estimators for various distributional parameters when data are obtained from randomized experiments with possibly high-dimensional covariates. Furthermore, our approach accommodates both discrete and mixed discrete-and-continuous outcome distributions, whereas quantile regression adjustment is specifically designed for continuous outcomes.
\paragraph{Semiparametric Estimation} Our work is closely linked to the extensive literature on semiparametric estimation, which addresses the challenge of estimating low-dimensional parameters in the presence of high-dimensional nuisance parameters. This literature includes seminal contributions such as klaassen1987consistent, robinson1988root, bickel1993efficient, andrews1994asymptotics, newey1994asymptotic, robins1995semiparametric, chernozhukov2018debiased, ichimura2022influence, among others. Our setup can be framed as a semiparametric problem characterized by the Neyman-orthogonal moment condition, as outlined in neyman1959optimal, chernozhukov2018debiased, chernozhukov2022locally. Notably, cross-fitting is a commonly used technique in this literature. While our technical arguments share similarities with classical semiparametric methods, our research introduces a novel perspective by emphasizing the significance of flexible machine learning methods for estimating distributional treatment effects, within the framework of randomized experiments.
We assume that our data are generated from a randomized experiment with $K$ treatment arms. Let $Y \in \mathcal Y \subset \mathbb R$ denote the scalar-valued observed outcome, $W\in \mathcal{W}:= \{1, \dots, K\}$ denote the index of the treatment arm, and $X\in \mathcal X \subset \mathbb{R}^{d_x}$ denote pre-treatment covariates. We observe a size $n$ random sample $\{Z_i\}_{i=1}^{n} := \{(X_i, W_i, Y_i)\}_{i=1}^{n}$ from the distribution of $Z := (X, W, Y)$. The probability of assignment to treatment arm $w$ is denoted as $\pi_w:=P(W_i =w)$ satisfying $\sum_{w \in \mathcal{W}} \pi_{w} = 1$, while $n_w$ indicates the number of observations in treatment group $w$, satisfying $\sum_{w \in \mathcal{W}} n_{w} = n$.
We follow the potential outcome framework [e.g., rubin1974estimating, imbens2015causal] and let $Y(1), \dots, Y(K)$ denote the potential outcomes, which are hypothetical and represent what the outcome for an individual would be under each treatment scenario. These are unobserved variables and we only observe the outcome for the treatment that is actually administered to each individual. We assume no interference and impose Stable Unit Treatment Values Assumption (SUTVA), which gives us $Y = Y(W)$. In other words, treatment assigned to one unit does not affect the outcome for another unit, and so the potential outcome under any treatment is equal to its observed outcome. Throughout the paper, we also maintain the following two assumptions.
Assumption (ref) states that the treatment indicator is independent of the potential outcomes and the pre-treatment covariates. Assumption (ref) states that the treatment assignment probabilities are bounded away from 0 and 1. These assumptions are satisfied because we have a randomized experiment where the researcher assigns individuals to treatment groups randomly and have a control over the treatment assignment probabilities.
The parameters of our interest are based on (cumulative) distribution functions of potential outcomes, denoted by
for $y\in\mathcal Y \subset \mathbb R$ and $w\in\mathcal{W}$, where $\1_{\{\cdot\}}$ represents the indicator function. In general, the potential outcomes $\{Y(w)\}_{w \in \mathcal{W}}$ are unobserved variables. However, under Assumptions (ref) and (ref), the potential outcome distribution $F_{Y(w)}(y)$ is the same as the outcome distribution $F_{Y|W}(y|w)$ under each treatment $w$. Therefore, they are identifiable given the data from $Z=(X,W,Y)$.
The results of this paper can be applied to estimate a range of distributional parameters, provided that they rely on (Hadamard) differentiable transformations of potential outcome distributions. We provide a few illustrative examples below.
\paragraph{Example 1: Distributional Treatment Effect} Let $w, w' \in \mathcal{W}$ be two different treatment groups. We are interested in the Distributional Treatment Effect (DTE), which is defined as, for $y\in\mathcal Y,$
To contrast, the Average Treatment Effect (ATE) is defined as
The DTE is a parameter that is indexed by a continuum of $y\in\mathcal Y$ and measures the effect of treatment on the whole distribution, whereas the ATE only quantifies the mean effect. As a special case, one can also be interested in the DTE at a certain threshold; i.e., $\mathcal Y$ can be defined to be a singleton set. One advantage of this measure is that it is well-defined for any type of outcome, including discrete, continuous, and mixed discrete-continuous variables.
\paragraph{Example 2: Probability Treatment Effect} The DTE may not be straightforward to interpret since it measures the differences between two cumulative distributions. However, we can compute more intuitive measures based on these differences. Specifically, the DTE can be used to compute, what we will call, the Probability Treatment Effect (PTE), which is given by
for $y\in\mathcal Y$ and some $h>0.$ The PTE measures the changes in the probability that the outcome falls in interval $(y, y+h]$. The PTE is also well-defined for any type of outcome, including discrete, continuous, and mixed discrete-continuous variables.
\paragraph{Example 3: Quantile Treatment Effect} Another common measure used to characterize the entire distribution is the quantile function, defined as $F_{Y(w)}^{-1}(\tau):= \inf\{y: F_{Y(w)}(y)\geqslant \tau\}$ for $\tau \in (0,1)$. The Quantile Treatment Effect (QTE) for quantile $\tau\in(0,1)$ is then given by
The QTE quantifies the difference in quantiles between two potential outcome distributions across a continuum of $\tau \in (0, 1)$. For example, one might be interested in the difference between the medians (when $\tau = 0.5$) of two groups. It is important to note that the QTE is only well-defined for continuous outcomes and may not be appropriate for discrete or mixed discrete-continuous outcomes.
As explained in the previous section, the potential outcome distribution serves as the fundamental building block for a broad range of distributional parameters. A simple estimator for the distribution function $F_{Y(w)}(y)$ is the empirical distribution function, given by:
for each treatment $w\in\mathcal{W}$. While this estimator is an unbiased and consistent estimator, we aim to enhance its precision by leveraging pre-treatment covariates.
To incorporate pre-treatment covariates $X$, we consider the distribution regression framework, in which the conditional distribution function $F_{Y(w)|X}(y|x)$ is regarded as the mean regression for a binary dependent variable $\1_{ \{Y(w) \leqslant y\} }$. That is, for each $y \in \mathcal{Y}$ and $w \in \mathcal{W}$, we can write
For each location $y \in \mathcal{Y}$, the conditional mean function can be separetely estimated using various methods, such as linear regression, logistic regression, or other machine learning techniques (e.g., LASSO, random forests, boosted trees, deep neural networks, etc.). Additionally, the distribution regression is applicable for continuous, discrete, and mixed discrete-and-continuous outcome variable as explained in chernozhukov2013inference.
For the regression-adjusted estimator of $F_{Y(w)}(y)$, the conditional distribution functions are nuisance parameters. We represent them as $\gamma_{y}^{(w)}(x):= F_{Y(w)|X}(y|x)$ for each $w \in \mathcal{W}$ and let $\widehat \gamma_y^{(w)}(\cdot)$ be an estimator for $\gamma_y^{(w)}(\cdot)$. We will explain the necessary conditions for the estimator $\widehat{\gamma}_y^{(w)}(\cdot)$ in the following section. The regression-adjusted estimator of $F_{Y(w)}(y)$ for each $w\in\mathcal{W}$ is then defined as
The regression-adjusted estimator is obtained by adjusting the empirical distribution function by subtracting $\widehat\gamma_y^{(w)}(\cdot)$ that is averaged over observations in that treatment group and adding $\widehat\gamma_y^{(w)}(\cdot)$ that is averaged over all observations.
This characterization of regression adjustment aligns closely with the concept of the augmented inverse propensity weighted estimator, as explored in robins1994estimation, robins1995semiparametric. list2022using also consider a similar adjustment method for estimating the ATE. We extend this formulation to encompass distribution functions for any arbitrary outcome location $y \in \mathcal{Y}$. It is worth noting that this estimator also serves as an unbiased estimator for the distribution function, as the expected value of the adjustment terms cancels out to zero.
\paragraph{Moment condition problem} We rewrite our problem as a moment condition problem. In what follows, we will simply write $\gamma_y^{(w)}$ to denote $\gamma_y^{(w)}(\cdot)$ and let $\gamma_y:= (\gamma_y^{(1)}, \dots, \gamma_y^{(K)})^{\top}$. Let $\theta_y^{(w)}:= F_{Y(w)}(y)$ and $\theta_y := (\theta_y^{(1)}, \dots, \theta_y^{(K)})^{\top}$. Define moment functions
where, for each $w \in \mathcal{W}$,
The following lemma shows what moment conditions are implied by our setup with a randomized experiment. Later, we will show how the regression-adjusted estimator, given in (ref), can be seen as a method of moments estimator that solves the sample counterpart of these moment conditions.
Our parameter of interest $\theta_y$ for each $y\in\mathcal Y$ is identified as the solution to the moment condition in (ref), where $Z=(X, W, Y)$ is the data and $\gamma_y$ is the possibly infinite-dimensional nuisance parameter.
An important property of the moment conditions defined in (ref) is that they are Neyman orthogonal with respect to the nuisance parameters neyman1959optimal, chernozhukov2018debiased, chernozhukov2022locally, ichimura2022influence. More precisely, the derivative of its expectation with respect to the nuisance parameters vanishes when evaluated at the true parameter values. The following lemma states it formally.
Neyman orthogonality implies that the moment condition is first-order insensitive to the estimation errors of the nuisance parameters. This property, coupled with a form of sample-splitting called cross-fitting, allows us to derive the asymptotic distribution of the regression-adjusted estimator under mild conditions, even when the conditional distribution functions are estimated via machine learning (ML) methods.
The following lemma shows how the regression-adjusted estimator, given in (ref), can be seen as an estimator that solves the sample counterpart of the moment conditions defined earlier.
\paragraph{Estimation procedure} We now explain our algorithm, which is summarized in Algorithm (ref). Our estimation procedure involves a sample-splitting method called cross-fitting [e.g., chernozhukov2018debiased]. First, we split the data into $L$ roughly equal-sized folds, where $L>1$. Then, for every observation, we use a ML method and predict nuisance functions $\widehat \gamma_y^{(w)}(X_i)$ by training on data from treatment group $w$, excluding data points from the fold the observation belongs to. This ensures that the observation and the nuisance estimates are independent. Finally, we form a point estimate of $F_{Y(w)}(y)$ by plugging in estimates of the nuisances in (ref). We do this for all $y\in\mathcal Y$ and $w\in\mathcal{W}$. Then we stack the estimators together to get a regression-adjusted estimator $(\widehat\theta_y)_{y\in\mathcal Y}$. We discuss the statistical inference in the next section.
\paragraph{Efficiency Gain} To illustrate the potential efficiency gain from the proposed regression-adjusted method, consider the scenario where the true conditional distribution function, $\gamma_{y}$, is employed in ((ref)), leading to the idealized form of the regression-adjusted estimator, denoted by $ \widetilde{\theta}_{y}^{(w)}:= \widetilde{F}_{Y(w)}(y)$. Let $\widetilde{\theta}_{y}:= (\widetilde{\theta}_{y}^{(1)}, \dots,\widetilde{\theta}_{y}^{(K)})^\top$. The following theorem highlights the efficiency improvements of this regression-adjusted estimator in comparison to the empirical distribution function. As demonstrated in the next section, our estimator asymptotically possess the same efficiency property in terms of variance.
Theorem (ref)(a) shows the efficiency gains achieved by applying regression adjustment to distribution functions. Furthermore, Theorem (ref)(b) elaborates on these gains in terms of a vector of regression-adjusted estimators, indicating a marked improvement in the precision of the estimator for the DTE as a special case.
In this section, we derive the asymptotic distribution of the regression-adjusted estimator. These results are built upon the functional central limit theorem, functional delta method and other related results from belloni2017program. \paragraph{Additional Notation} We introduce some additional notations to state our results. For a vector $a=(a_1, \dots, a_p)^{\top}\in\mathbb R^p$, $\|a\|= \sqrt{a^{\top}a}$ denotes the Euclidean norm of $a$. Let $\ell^{\infty}(\mathcal Y)$ be the space of uniformly bounded functions mapping an arbitrary index set $\mathcal{Y}$ to the real line; $UC(\mathcal{Y})$ be the space of uniformly continuous functions mapping $\mathcal{Y}$ to the real line. $\mathbb G_{n,P} f$ denotes the empirical process $\sqrt{n}\sum_{i=1}^{n}(f(Z_i)-\int f(z)dP(z))$; but we will omit $P$ and simply write $\mathbb G_{n} f$. Let $\mathcal P_n$ denote the set of probability measures, that is weakly increasing in $n$, i.e., $\mathcal P_n \subseteq \mathcal P_{n+1}$. We use $\rightsquigarrow$ to denote the convergence in distribution or law. Lastly, let $\mathbb G_P$ denote the P-Brownian bridge, as defined in Appendix Section (ref).
The following theorem shows that under regularity conditions, stated fully in Appendix (ref), our estimator $(\widehat{\theta}_y)_{y\in\mathcal Y}$ is asymptotically Gaussian. Since we employ cross-fitting, the conditions required for the estimation of nuisance functions become much milder compared to not using any data-splitting. It is required that the estimators of nuisance functions attain sufficiently rapid rates of convergence $\tau_n$, in particular $\tau_n = o(n^{-1/4})$ in smooth problems.
Then, as a special case of the above theorem, for fixed $y\in\mathcal{Y}$, we have pointwise asymptotic normality, stated as $\sqrt{n} (\widehat\theta_y -\theta_y) \rightsquigarrow N(0, \mathrm{Var}(\psi_y))$. Note that, $\mathrm{Var}(\psi_y)$ can be consistently estimated via sample moment conditions using cross-fitting as well. The estimate of the asymptotic variance can then be used to construct the confidence intervals in a usual manner.
\paragraph{Functionals of $\theta$}
The parameters we are interested in are functionals of potential outcome distributions. The examples include the DTE, the PTE and the QTE we discussed in Section (ref). If, for instance, we are interested in the DTE between treatments $1$ and $2$, for each $y\in\mathcal Y$, we can calculate it as $(1,-1, 0, \dots, 0) \theta_y$. Let $\phi(\theta^0)=\phi((\theta_y)_{y\in\mathcal Y})$. The following theorem shows the large sample law of the plug-in estimator $\phi(\widehat \theta):= \phi((\widehat \theta_y)_{y\in\mathcal Y})$. For complex objects, the inference can be facilitated by bootstrap. The validity of a multiplier bootstrap method gine1984some is also shown in the theorem below.
Here $\rightsquigarrow_B$ denotes weak convergence of the bootstrap law in probability, as defined in Appendix (ref). $\phi(\widehat\theta^*)=\phi((\widehat \theta^*)_{y\in\mathcal Y})$ is the bootstrap version of $\phi(\widehat \theta) $, and $\widehat\theta_y^*=\widehat\theta_y + n^{-1} \sum_{i=1}^{n} \xi_i\psi_y(Z_i; \widehat\theta_y, \widehat\gamma_y)$ is the multiplier bootstrap version of $\widehat\theta_y$. More details about the multiplier bootstrap procedure to obtain pointwise and uniform confidence bands can be found in the Appendix (ref). The assumption of Hadamard differentiability is imposed so that we can use the delta method. The formal definition can be found in the Appendix (ref).
In this section, we compare our regression-adjusted estimators to simple estimators in two types of experiments. In the first experiment, we use a synthetic dataset to assess the performance of our proposed method in finite samples. For the second experiment, we reanalyze data from a randomized experiment, conducted by ferraro2013using, to compare the methods using real-world data.
We conduct Monte Carlo simulation study to evaluate the performance of our adjusted estimators in finite samples. We compare the simple estimator with two regression-adjusted estimators - linear adjustment and ML adjustment. The simple estimator is based on empirical distribution functions. The regression-adjusted estimators are calculated according to the procedure in Algorithm (ref) with 5 folds. For the linear adjustment, we use a linear regression for estimating $\widehat \gamma_y$. For the ML adjustment, we use logistic LASSO for estimating $\widehat \gamma_y$.
In this experiment, we generate i.i.d. sample of size $n \in \{500, 1000, 5000\}$ with covariates, a binary treatment and a continuous outcome. We design the data generating process such that the half of covariates are irrelevant to the outcome. Appendix (ref) contains more details about the data generating process and describes the evaluation metrics.
The top figure of Figure (ref) plots the bias of these estimators as a % of the true values of the DTE, across different quantiles under the sample sizes we consider. We confirm that the bias is small for all estimators across all quantiles. Even when sample size is small ($n=500$), the bias is at most $2\%$. This is as expected since all estimators are unbiased estimators of the distribution functions and hence the DTE.
Next, we turn to the RMSE. The bottom figure of Figure (ref) plots the RMSE reduction in $\%$ terms for the linear and ML adjustment, compared to the simple estimator. We see that the linear adjustment and ML adjustment estimators yield smaller RMSE compared to the simple estimator across all quantiles. Moreover, we see that ML adjustment outperforms linear adjustment in all cases. Specifically, for $n=5000$, RMSE reduction over the simple estimator is around 40-50% for the ML adjusted estimator, while it is between roughly 15-35% for the linearly adjusted estimator. This is as expected because our data generating process consists of variables that are irrelevant to the outcome, and so ML adjustment better captures the relationship between the outcome and covariates compared to the linear model. Improved prediction quality for $\widehat \gamma_y$ results in more variance reduction for the DTE.
We reanalyze data from a randomized experiment conducted by ferraro2013using in 2007, to examine the effect of norm-based messages or nudges on water usage in Cobb County, Atlanta, Georgia. Three different interventions to reduce water usage were implemented and compared to the control group (no nudge). list2022using re-estimate the regression-adjusted ATE for the intervention called “strong social norm” that combines prosocial appeal and social comparison (the strongest intervention) relative to the control group. We extend their analysis by estimating the regression-adjusted DTE and PTE of this intervention over the control group ($W\in\{0,1\}$) using the same pre-treatment covariates $X$, which is monthly water consumption in the year prior to the experiment. Thus, the dimension of the covariate space $\mathcal X$ is $d_x=12$. The outcome variable $Y$ is level of water consumption from June to September of 2007. The unit of our outcome variable is in thousands of gallons and is discretely distributed. Note that although the measure in gallons appears to be approximately continuous in practice, the presence of subtle discreteness can create problems for both theoretical and practical statistical inference. Thus, the QTE is not applicable here. The results for the other two treatments - “technical advice” and “weak social norm” - relative to the control group are summarized in Appendix (ref).
Figure (ref) plots the DTE and the PTE of the intervention compared to the control group. We compute the DTE for $y\in\{0,1,2,\dots, 200\}$. Figure (ref) top left represents the simple estimate of the DTE, whereas the top right figure depicts the regression-adjusted estimate of the DTE. For regression adjustment, we estimate the conditional distribution functions $\widehat \gamma_y$ via gradient boosting using 10-fold cross-fitting. The shaded areas represent the 95% pointwise confidence intervals. We can see that the regression-adjusted DTE has substantially smaller variance and hence tighter confidence intervals compared to the simple DTE, especially between 15 and 110. Based on the regression-adjusted DTE results, we see that the DTE is increasing up until 40-50 and starts declining after that. We can draw conclusions about how the outcome distributions differ under treatment and control, keeping in mind this is differences in cumulative distributions.
More intuitive measure of this distribution change is the PTE. Figure (ref) bottom left represents the simple PTE, while the bottom right represents the regression-adjusted PTE, with the 95% pointwise confidence intervals. We show the PTE in increments of $h=10$ for $y\in\{0, 10, 20,\dots, 200\}$. We see that the treatment is effective in that it reduces the probability of high water consumption and increases the probability of low water consumption. Specifically, the results from the regression-adjusted PTE indicate increase in water usage in the range of 20-40; decrease in water usage in the range of 60-110 (with a minor exception that within the range (80, 90], which is represented by point 80 in the graph, the confidence interval exceeds 0 by only a little). The variance reduction is especially helpful at the range (70, 80]. The probability change for the range (70,80] is not significant under simple estimates, while it is significantly negative under regression adjustment.
We provide a novel regression adjustment method to estimate various measures of distributional treatment effects to capture heterogeneity. Our framework accommodates high-dimensional setup with many pre-treatment covariates and offers flexible modeling by incorporating machine learning techniques for the regression adjustment.
Some limitations of our method are as follows. Firstly, we consider a setting where we have an experimental data with perfect compliance and no interference (no network or peer effects). While suitable for some applications, these assumptions may prove restrictive in other contexts. Secondly, our approach relies on the presence of pre-treatment covariates highly predictive of the outcome. While we enhance variance reduction compared to linear regression by employing flexible machine learning methods to improve prediction quality, substantial variance reduction may not occur if the covariates lack high-quality information. Thirdly, we focus on a setup where experimental data is already collected, neglecting opportunities to incorporate variance reduction strategies at the design stage of the experiment. These limitations suggest avenues for future research on distributional analysis that incorporates these concerns.
We extend our gratitude to the four anonymous reviewers and the program chairs for their insightful comments and discussions, which significantly enhanced the quality of this paper. We also appreciate the feedback provided by Hiroki Yanagisawa and Shuting Wu. Additionally, Oka acknowledges the financial support from JSPS KAKENHI Grant Number 24K04821.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
\onecolumn