Manuel Quintero, William T. Stephenson, Advik Shreekumar, Tamara Broderick
arXiv 23 Apr 2025 · Statistics — Methodology
arXiv:2504.16864 · PDF · DOI · OpenAlex · Extracted main text
In science and social science, we often wish to explain why an outcome is different in two populations. For instance, if a jobs program benefits members of one city more than another, is that due to differences in program participants (particular covariates) or the local labor markets (outcomes given covariates)? The Kitagawa-Oaxaca-Blinder (KOB) decomposition is a standard tool in econometrics that explains the difference in the mean outcome across two populations. However, the KOB decomposition assumes a linear relationship between covariates and outcomes, while the true relationship may be meaningfully nonlinear. Modern machine learning boasts a variety of nonlinear functional decompositions for the relationship between outcomes and covariates in one population. It seems natural to extend the KOB decomposition using these functional decompositions. We observe that a successful extension should not attribute the differences to covariates -- or, respectively, to outcomes given covariates -- if those are the same in the two populations. Unfortunately, we demonstrate that, even in simple examples, two common decompositions -- functional ANOVA and Accumulated Local Effects -- can attribute differences to outcomes given covariates, even when they are identical in two populations. We provide a characterization of when functional ANOVA misattributes, as well as a general property that any discrete decomposition must satisfy to avoid misattribution. We show that if the decomposition is independent of its input distribution, it does not misattribute. We further conjecture that misattribution arises in any reasonable additive decomposition that depends on the distribution of the covariates.
appendix boundary found by appendix_command · 40% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Daniel W. Apley and Jingyu Zhu (2020) Visualizing the effects of predictor variables in black box supervised learning models | 0.928 | 4 | 3 | 100% |
| 2 | Giles Hooker (2007) Generalized Functional ANOVA Diagnostics for High-Dimensional Functions of Dependent Variables | 0.843 | 4 | 4 | 75% |
| 3 | Giles Hooker (2004) Discovering additive structure in black box functions | 0.843 | 3 | 3 | 100% |
| 4 | Raj Agrawal and Tamara Broderick (2023) The skim-fa kernel: high-dimensional variable selection and nonlinear interaction discovery in linear time self | 0.405 | 1 | 1 | 100% |
| 5 | Anestis Antoniadis, Sophie Lambert-Lacroix, and Jean-Michel Poggi (2021) Random forests for global sensitivity analysis: A selective review | 0.405 | 1 | 1 | 100% |
| 6 | Philipp Bach, Victor Chernozhukov, and Martin Spindler (2024) Heterogeneity in the us gender wage gap | 0.405 | 1 | 1 | 100% |
| 7 | Amine Belhadi, Swapnil S. Kamble, V. Mani, et al (2021) An ensemble machine learning approach for forecasting credit risk of agricultural smes’ investments in agriculture 4.0 through s… | 0.405 | 1 | 1 | 100% |
| 8 | Alan S. Blinder (1973) Wage Discrimination: Reduced Form and Structural Estimates | 0.405 | 1 | 1 | 100% |
| 9 | Gaelle Chastaing, Fabrice Gamboa, and Clémentine Prieur (2012) Generalized Hoeffding-Sobol decomposition for dependent variables - application to sensitivity analysis | 0.405 | 1 | 1 | 100% |
| 10 | Nicole Fortin, Thomas Lemieux, and Sergio Firpo (2011) Decomposition Methods in Economics | 0.405 | 1 | 1 | 100% |
Showing the top 10 of 31 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | Do covariates explain why these groups differ? The choice of reference group can reverse conclusions in the Oaxaca-Blinder decomposition | 0.644 | 2 | 2 |