Patrick Rehill, Nicholas Biddle
arXiv 20 Oct 2023 · Machine Learning · publishedData & Policy (2024) · 12 citations (OpenAlex)
arXiv:2310.13240 · PDF · DOI · OpenAlex · Extracted main text
Causal machine learning tools are beginning to see use in real-world policy evaluation tasks to flexibly estimate treatment effects. One issue with these methods is that the machine learning models used are generally black boxes, i.e., there is no globally interpretable way to understand how a model makes estimates. This is a clear problem in policy evaluation applications, particularly in government, because it is difficult to understand whether such models are functioning in ways that are fair, based on the correct interpretation of evidence and transparent enough to allow for accountability if things go wrong. However, there has been little discussion of transparency problems in the causal machine learning literature and how these might be overcome. This paper explores why transparency issues are a problem for causal machine learning in public policy evaluation applications and considers ways these problems might be addressed through explainable AI tools and by simplifying models in line with interpretable AI principles. It then applies these ideas to a case-study using a causal forest model to estimate conditional average treatment effects for a hypothetical change in the school leaving age in Australia. It shows that existing tools for understanding black-box predictive models are poorly suited to causal machine learning and that simplifying the model to make it interpretable leads to an unacceptable increase in error (in this application). It concludes that new tools are needed to properly understand causal machine learning models and the algorithms that fit them.
appendix boundary found by appendix_titled_section at “Appendix A --- Variables used in application” · 92% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Rehill \ Biddle (2023) `Fairness Implications of Heterogeneous Treatment Effect Estimation with Machine Learning Methods in Policy-making' | 1.000 | 5 | 3 | 100% |
| 2 | Athey, Tibshirani \ Wager (2019) `Generalized random forests', The Annals of Statistics 47(2), 1148–1178 | 0.969 | 11 | 5 | 91% |
| 3 | Lundberg \ Lee (2017) `A Unified Approach to Interpreting Model Predictions' | 0.950 | 7 | 3 | 86% |
| 4 | Chernozhukov, Chetverikov, Demirer, Duflo, Hansen, Newey \ Robins (2018) `Double/debiased machine learning for treatment and structural parameters', The Econometrics Journal 21(1), C1–C68 | 0.928 | 10 | 5 | 80% |
| 5 | Wager \ Athey (2018) `Estimation and inference of heterogeneous treatment effects using random forests', Journal of the American Statistical Associat… | 0.928 | 5 | 3 | 80% |
| 6 | Imbens \ Rubin (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Cambridge University Press | 0.928 | 4 | 3 | 100% |
| 7 | Tiffin (2019) `Machine learning and causality: The impact of financial crises on growth', IMF Working Papers 19 | 0.928 | 4 | 3 | 100% |
| 8 | Nie \ Wager (2021) `Quasi-oracle estimation of heterogeneous treatment effects', Biometrika 108(2), 299–319 | 0.909 | 8 | 4 | 75% |
| 9 | Sharma, Syrgkanis, Zhang \ Kıcıman (2021) `Dowhy: Addressing challenges in expressing and validating causal assumptions', arXiv:2108.13518 [cs] | 0.874 | 7 | 2 | 100% |
| 10 | Künzel, Sekhon, Bickel \ Yu (2019) `Metalearners for estimating heterogeneous treatment effects using machine learning', Proceedings of the National Academy of Sci… | 0.843 | 4 | 3 | 75% |
Showing the top 10 of 90 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | Distilling interpretable causal trees from causal forests | 0.405 | 1 | 1 |