Improving the Finite Sample Estimation of Average Treatment Effects using Double/Debiased Machine Learning with Propensity Score Calibration

Improving the Finite Sample Estimation of Average Treatment Effects using Double/Debiased Machine Learning with Propensity Score Calibration

\jelcodes{C01 \and C14 \and C21}

Introduction

The increasing amount of observational data has led to a growing interest in causal inference methods that can handle high-dimensional data. As a result, machine learning techniques for estimating causal effects have gained popularity. A well-known example that can be used is the double/debiased machine learning (DML) estimator that relies on the prediction of nuisance functions with machine learning methods \citep*{Chernozhukov:2018}. One of the nuisance functions for estimating an average treatment effect (ATE) is the propensity score, which is the probability of receiving treatment conditional on a set of covariates. While an increasing number of covariates makes the unconfoundedness assumption in observational studies more plausible, propensity score estimates become more extreme as treatment assignment can be better predicted, and the overlap assumption is more difficult to satisfy DAmour:2021.\footnote{The unconfoundedness and overlap assumptions are two crucial assumptions in a selection-on-observables setting.}

Extreme propensity scores have been identified to be problematic for estimators relying on double-robust score functions or inverse probability weighting already before the use of machine learners \citep*[e.g.][]{Froelich:2004, Busso:2014}. Methods of re-weighting the propensity scores or trimming methods have been developed and analysed to overcome this problem. For example, Crump:2009 propose a rule of thumb of just discarding all propensity scores that are smaller than 0.1 and larger than 0.9. Yang:2018 provide asymptotic results for a smooth weighting approach approximating sample trimming. Huber:2013 propose to trim and re-weight extreme inverse propensity scores. In the context of unbalanced treatment assignment where extreme propensity score estimates are likely, Ballinari:2024 proposes an estimation procedure for the ATE relaying on undersampling the dataset.

Machine learners aggravate the problem of extreme propensity scores, as their increased flexibility allows for even better label classification. In finite samples, however, researchers have shown that machine learners tend to over- or underestimate the probability of a label, i.e. the treatment probability \citep*[e.g.,][]{Niculescu:2005}. In fact, machine learners are generally designed to perform well in classification tasks and less so in quantifying the likelihood of a class. Deep neural networks, for example, have been found to overestimate probabilities in different settings Clarte:2023. Errors in the predicted propensity scores can lead to biased finite sample estimates of the ATE. Lechner:2024 have demonstrated through a large simulation study that using DML results in a biased ATE when the selection into treatment is high.

In a prediction setting, calibration methods have been employed to improve the performance of machine learners for classification and regression problems \citep*[e.g.][]{Bella:2010}. Calibration approaches aim to adjust the predicted probabilities closer to the true probabilities. In other words, a well-calibrated binary classifier should assign a predicted probability value (propensity score), such as 0.8, to a group of samples in such a way that roughly 80% of those samples truly belong to this class (are treated). Popular approaches to calibrate predicted probabilities are Platt scaling Platt:1999, Beta scaling Kull:2017, isotonic regression Zadrozny:2002, Venn-Abers calibration Vovk:2012, temperature scaling Guo:2017 and expectation consistent calibration Clarte:2023.

This paper proposes to use calibration methods to improve the predictions of the propensity scores in the ATE estimation using the DML framework. The DML estimator is adjusted to include an additional estimation step for calibrating the propensity scores. We compare the performance of the ATE estimated with calibrated propensity scores to traditional DML and a re-weighting approach through a simulation study and show under what conditions calibration does not alter the asymptotic properties of DML. Our findings demonstrate that calibrating propensity scores may significantly reduce the root mean squared error (RMSE) of the ATE in finite samples, especially in scenarios with high treatment selectivity. The reduction in RMSE is primarily driven by a reduced bias in the ATE estimate, consequently improving the coverage of the confidence intervals. When there is almost no selectivity, traditional DML performs well, and calibration does not improve the performance but it does not harm the performance either. The higher the selection into treatment, the larger the improvement that can be achieved by calibrating the propensity score. Most of the calibration methods analysed in this paper perform well, with Venn-Abers calibration, Platt scaling, and Beta scaling being the best-performing calibration methods in the simulation study. We find that lower mean squared errors of the machine learner predicting the propensity score are associated with lower bias in the ATE. Practitioners are thus advised to choose the calibration method with the lowest Brier score, which measures the squared difference between the predicted probabilities and the treatment indicator (see Section (ref)). Last, we showcase the benefits of calibration in an empirical example evaluating the effect of language courses for unemployed individuals on their employment probability.

The contribution of this paper is twofold. First, we contribute to a small but growing literature that uses calibration methods in the context of treatment effect estimation. Gutman:2022 explore using calibrated propensity scores for the inverse probability weighting estimator. VanderLaan:2024a demonstrate the use of isotonic calibration for calibrating inverse propensity weights. Some papers combine calibration methods with the estimation of heterogeneous treatment effects. vanderLaan:2023 propose a new nonparametric and doubly-robust method for calibrating predictors of heterogeneous treatment effects, but they do not calibrate the propensity scores. Xu:2022 propose a method to assess if heterogeneous treatment effects are well calibrated. Shachi:2024 analyse the improvements in ATE estimation when using Platt scaling and isotonic regression for propensity scores predicted by a logistic regression. Second, we contribute to the small literature focusing on the finite sample performance of DML. Knaus:2021 compare different machine learning estimator for heterogeneous treatment effects in a large simulation study. Lechner:2024 show that DML is biased in finite samples when the selection into treatment is high. To the best of our knowledge, this is the first study to investigate the calibration of propensity scores in a DML framework and compare a comprehensive set of calibration methods and machine learners in a simulation study.

The remainder of the paper is structured as follows. Section (ref) defines the notation. Section (ref) discusses the different calibration methods used. In Section (ref), we explain the estimation procedure and present asymptotic results. Section (ref) describes the simulation study and its results. In Section (ref) we use the new procedure in an empirical application and last, Section (ref) concludes.

Notation

We observe a dataset of $N$ i.i.d. observations and denote the index set as $\mathcal{I} = \{1, \dots, N\}$ and the dataset as $\mathcal{S} = \{(D_i, Y_i, X_i) : i \in \mathcal{I}\}$. This paper focuses on the case where $D_i$ is binary, taking values $d \in \{0,1\}$. However, the approach can easily be extended to discrete $D_i$. The random vector $X_i = [X_{i,1}, \dots, X_{i,q}]$ contains $q$ covariates. We denote the conditional expectation of the observed outcome as $\mu(d,x):= \mathbb{E}[Y_i|X_i=x, D_i=d]$ and the propensity score as $p(x) := P(D_i = 1 | X_i = x)$ and will refer to them as nuisance functions. We are interested in the estimation of the expected difference in conditional expectations $\theta:=\mathbb{E}[\mu(1,X_i) - \mu(0,X_i)] = \mathbb{E}[\tau(D_i,X_i,Y_i)]$, where $\tau(d,x,y)$ is the efficient score function defined as:

Under suitable assumptions, the quantity of interest $\theta$ corresponds to the ATE: (i) conditional independence, (ii) common support, (iii) exogeneity of confounders, and (iv) stable unit treatment value assumption Imbens:2009. The results in this paper do however not dependent on the identification of a treatment effect. Indeed, when these assumptions are not fulfilled, the quantity $\theta$ corresponds to the average prediction effect as described in Chernozhukov:2018.

In what follows, we denote the $L_s$-norm as $\lVert\cdot\rVert_s$, for example $\lVertg(X)\rVert_s = \int\vert g(x)^s\vert^{1/s} dP(x)$. A sequence converging to zero is denoted by $o(1)$ and a bounded sequence by $O(1)$. Similarly, a sequence of random variable converging in probability to zero is denoted by $o_p(1)$ and a sequence bounded in probability by $O_p(1)$.

Calibration Methods

Definition

We are interested in the estimation of the propensity score defined as $p(x) = P(D_i = 1 | X_i = x)$. Let $\hat p(x)$ denote the predicted probability coming from a statistical/machine learning model fitted on a training dataset $\mathcal{S}_T = \{(D_i, Y_i, X_i) : i \in \mathcal{I}_T\}$, where $\mathcal{I}_T \subset \mathcal{I}$. We would hope that $\hat p(x)$ can be interpreted as the probability of receiving treatment:

Equation (ref) is satisfied if $\hat p(x)$ is (perfectly) calibrated, that is Gupta:2020:

a.s. for all $p$ in the range of $\hat p$, where the expectation is taken over $D_i$ for $i\notin \mathcal{I}_T$. In words, if the estimated propensity score is 0.8, we would expect that 80% of the individuals with the same propensity score $p$ are actually treated. In essence, if the model is perfectly calibrated, the predicted probability should match the true probability, ensuring that, on average, the model's predictions align with reality Wang:2023. Unfortunately, this is generally not the case without strong distributional assumptions, which may not hold in practice Gupta:2020. We follow vanderLaan:2023 and quantify the calibration of the propensity score by the $L_2$ calibration error:

Several approaches have been proposed to improve the calibration of probability predictions Bella:2010. A calibrator is a function $f: [0,1] \mapsto [0,1]$ mapping $\hat{p}$ to a predictor $\pi(x) := (f \circ \hat p) (x)$ with presumably improved calibration properties, that is $CAL(\pi) < CAL(\hat p)$. For the true propensity score $p(x)$, the calibrator corresponds to the identity function. In general, however, the calibrator function $f$ is not known and is typically estimated on a so called calibration dataset $\mathcal{S}_C = \{(D_i, Y_i, X_i) : i \in \mathcal{I}_C\}$, where $\mathcal{I}_C \subset \mathcal{I}$ and $\mathcal{I}_C \cap \mathcal{I}_T = \emptyset $ (see Section (ref) for more details). An estimated calibrator is denoted as $\tilde\pi(x)$, to emphasize that it is estimated on a different dataset than the propensity score $\hat p(x)$. In finite samples, perfect calibration is generally not achievable and many empirical calibration techniques restore to asymptotic calibration, where the calibration error vanishes as the sample size grows Gupta:2020. In the reminder of this section we give an overview of several popular approaches introduced by the machine learning literature to estimate the calibrator map.

Platt Scaling

In the context of support vector machines, Platt:1999 introduced an approach commonly referred to as Platt scaling or Platt calibration, where the calibration map is defined as a logistic transformation:

The scalar parameters $\alpha$ and $\beta$ are estimated by maximum likelihood on the calibration dataset $\mathcal{S}_C$. The logistic mapping between the predicted and calibrated probabilities can be derived from the assumption that the propensity scores within each class (treatment) are normally distributed with the same variance Kull:2017.\footnote{$P(\hat p(X_i) \leq c| D_i = d, \mathcal{S}_T)$ corresponds to the cumulative normal distribution function with $\mathcal{S}_T$ being the training set.} Due to the parametric assumption, the calibration error might not vanish asymptotically Gupta:2020. In particular, the calibration error does not vanish asymptotically if the propensity scores $\hat p(x)$ are already perfectly calibrated since the identity function is not a member of the calibration map defined in Equation (ref). Platt scaling is a simple and computationally efficient method, which is particularly useful when the sample size is small vanderLaan:2023.

Beta Scaling

The assumption that propensity scores within each class are normally distributed is unreasonable for many probabilistic classifiers that predict probabilities in the range $[0,1]$. Kull:2017 propose a more flexible form of Platt's scaling, called Beta scaling. Their approach can be derived by assuming that the predicted probabilities are beta-distributed within each class.\footnote{$P(\hat p(X_i) \leq c| D_i = d, \mathcal{S}_T)$ corresponds to the cumulative beta distribution function with $\mathcal{S}_T$ being the training set.} The calibration map is defined as:

with $\beta_0, \beta_1 \geq 0$. Beta scaling can be shown to be equivalent to a logistic regression on $\log(\hat p(x))$ and $\log(1-\hat p(x))$ Kull:2017. We thus estimate the parameters $\alpha, \beta_0, \beta_1$ by maximum likelihood on the calibration dataset $\mathcal{S}_C$, with the constraint $\beta_0, \beta_1 \geq 0$. At the cost of an additional parameter, Beta scaling is more flexible than Platt scaling. In particular, the calibrator map does contain the identity function, parametrized by $\alpha=0, \beta_0=1, \beta_1=1$.

Isotonic Calibration

Zadrozny:2002 relax the parametric assumptions of Platt and Beta scaling by calibrating probabilities with a nonparametric approach: isotonic regression, also known as monotonic regression \citep*{Barlow:1972}. The idea is to fit a piecewise constant function to the predicted probabilities that is nondecreasing. For a new observation with covariates $X_i=x$ and raw predicted propensity score $\hat p(x)$, the calibrated probability is given by:

where $\hat p_{(i)}$ is the $i$-th ordered value of the set $\{\hat p(X_i) : i \in \mathcal{I}_C\}$ and $N_C$ is the cardinality of $\mathcal{S}_C$. The parameters of the isotonic regression are estimated over the calibration dataset by the following optimization problem:

which can be computed using the pool adjacent violators algorithm. Isotonic calibration is a popular method in the machine learning literature Zadrozny:2002, Gupta:2021. vanderLaan:2023 show that the calibration error $CAL(\hat \pi(x))$ goes to zero as $N_C$ grows. The nonparametric nature of this calibrator comes at the cost of overfitting when the calibration dataset is small Caruana:2006. Moreover, it is crucial to note that calibrated propensity scores using isotonic regression can take values of 0 and 1. In particular, it can be shown that when in the calibration dataset $\mathcal{S}_C$ the observation with the smallest uncalibrated predicted propensity score is not treated, then $\tilde\pi_{(1)}=0$. This leads to significant problems in a DML setting, as the inverse of the propensity score is used to estimate the ATE.

Venn-Abers Calibration

Vovk:2012 and Vovk:2015 propose a calibration approach based on Venn predictors. Their approach relies on the nonparametric isotonic regression (see Section (ref)) but overcomes some of its shortcomings (e.g. overfitting). Given a calibration dataset $\mathcal{S}_C$ and a new observation $(D_i, Y_i, X_i)$, we proceed as follows. First, using an isotonic regression fit a calibrator $\pi_0(x)$ on the sample $\{(\hat p(X_i), D_i) : i \in \mathcal{I}_C\} \cup \{(\hat p(X_i), 0)\}$. A second calibrator $\pi_1(x)$ is fitted on the sample $\{(\hat p(X_i), D_i) : i \in \mathcal{I}_C\} \cup \{(\hat p(X_i), 1)\}$. Vovk:2015 provide guarantees that at least one of $\tilde\pi_0(x)$ and $\tilde\pi_1(x)$ is perfectly calibrated for the new observation. In other words, we fit an isotonic regression on the calibration dataset extended by a new observation. For this new observation, we once assume that it is treated and once that it is not treated. One of the two resulting probabilities will be perfectly calibrated for the new observation.\footnote{For computational efficiency, the isotonic regressions are not recomputed for each new observation $(D_i, Y_i, X_i)$. Instead, the above procedure is repeated on the sample $\{(\hat p(X_i), D_i) : i \in \mathcal{I}_C\} \cup \{(p, d)\}$ for a range of values $p$ and for $d\in\{0,1\}$. We obtain two vectors $\tilde\Pi_0$ and $\tilde\Pi_1$, storing the calibrated probabilities for the range of $p$ values. For a new observation $(D_i, Y_i, X_i)$, we select $\tilde\pi_0(x)$ and $\tilde\pi_1(x)$ from the vectors $\tilde\Pi_0$ and $\tilde\Pi_1$ according to its raw predicted probability $\hat p(x)$. This procedure is implemented in the Python package venn-abers 1.4.4.}

For the estimation of the ATE, we need, however, a single propensity score estimate. Following arguments of log loss minimization, Vovk:2012 show that a single calibrated probability can be obtained as:

While the calibration guarantees no longer hold for Equation (ref), $\tilde\pi(x)$ will be almost perfectly calibrated as long as $\tilde\pi_0(x)$ and $\tilde\pi_1(x)$ are close to each other. This directly follows from the fact that $\tilde\pi_0(x) \leq \tilde\pi(x) \leq \tilde\pi_1(x)$ and that either $\tilde\pi_0(x)$ or $\tilde\pi_1(x)$ are perfectly calibrated. In practice, the empirical findings of Vovk:2012 and Vovk:2015 show that Equation (ref) indeed leads to more calibrated probabilities compared to Platt scaling, isotonic regression or uncalibrated predictions.

Temperature Scaling

Guo:2017 propose a simple calibration method called temperature scaling. The idea is to rescale the log predicted probabilities by a scalar parameter $T$:

For $T=1$, the calibrated probabilities are equal to the predicted probabilities, as $T\to 0$ the propensity score collapses to a point mass $\pi(x)= 1$ whereas for $T\to \infty$ the propensity score converges to 0.5. Temperature scaling is closely related to Platt and Beta scaling but is more parsimonious as it only has one parameter. Guo:2017 propose to estimate the parameter $T$ by maximizing the log-likelihood over the calibration dataset $\mathcal{S}_C$. Empirical results show that temperature scaling applied to neural networks is competitive with more complex calibration methods Guo:2017.

Expectation Consistent Calibration

Expectation consistent calibration has been introduced by Clarte:2023 to calibrate predictions of neural networks. The idea is to rescale the probabilities in the same way as done by the temperature scaling approach (see Equation (ref)). Instead of maximizing the log-likelihood, Clarte:2023 fit the temperature parameter $T$ by enforcing that over the calibration dataset, the average calibrated propensity score coincides with the average proportion of correct treatment “classifications”. Let $\hat{d}_i = \mathds{1}(\hat p(X_i) \geq 0.5)$ where $\mathds{1}(\cdot)$ is the indicator function, that is, we predict treatment if the propensity score is larger than 0.5. Then $T$ is chosen such that the following equality holds:

where $N_C$ is again the cardinality of the calibration dataset $\mathcal{S}_C$. The intuition of this approach is that when the machine learning algorithm correctly classifies a certain share of observations (accuracy), its “confidence” should, on average, be equal to this share Clarte:2023.

Calibrated Double/Debiased Machine Learning Estimator

Estimation Procedure

Double/debiased machine learning is an estimation method that enables using machine learning methods for causal inference Chernozhukov:2018. One causal estimand of interest that is possible to estimate using DML is the ATE. The idea is to estimate the ATE by using the efficient score function defined in Equation (ref) first introduced by Robins:1995. The efficient score function is Neyman-orthogonal, meaning that small estimation errors in the nuisance parameters do not affect the estimation of the ATE Chernozhukov:2018. Therefore, machine learners can be used to estimate the nuisance parameters despite having a small regularization bias. Furthermore, the nuisance parameters are estimated using cross-fitting to avoid overfitting. Chernozhukov:2018 provide asymptotic theory for the DML estimator, demonstrating its consistency and asymptotic normality, even when machine learners are used to predict the nuisance functions (see Assumption (ref)).

Algorithm (ref) shows the procedure for estimating an ATE ($\theta$) with the DML approach introduced by Chernozhukov:2018 directly extended by the calibration procedure. The algorithm partitions the dataset $\mathcal{S}$ into $K$ folds, trains the nuisance functions on $K-1$ folds, and the resulting models are used to generate predictions for the remaining fold. This evaluation fold is further partitioned into $J$ sub-folds. The calibrator $\pi(x)$ is estimated on $J-1$ sub-folds using one of the approaches discussed in Section (ref), while the pseudo-outcomes $\tau_i$ are computed on the remaining sub-fold. This calibration process is repeated $J$ times to compute pseudo-outcomes for each sub-fold. The entire procedure is repeated across all $K$ folds, such that each fold is used once as the evaluation/calibration set. The ATE is then estimated by averaging the pseudo-outcomes. Other approaches to split the dataset into training $\mathcal{S}_T = \{(X_i, D_i, Y_i) : i\in \mathcal{I}_T\}$, calibration $\mathcal{S}_C = \{(X_i, D_i, Y_i) : i\in \mathcal{I}_C\}$ and evaluation sets $\mathcal{S}_E = \{(X_i, D_i, Y_i) : i\in \mathcal{I}_E\}$ are possible, as long as the sets are disjoint.

Asymptotic Results

Calibration of propensity scores is expected to improve the finite sample performance of the DML estimator. In the following, we provide assumptions under which a calibrator does not change the asymptotic properties of the estimator. In particular, we show that it has the same asymptotic distribution as the DML estimator. This asymptotic result may serve as a guideline for researchers to choose a suitable calibration method, which not only improves the finite sample performance but also does not deteriorate the asymptotic properties of the estimator.

Assumption (ref) imposes the usual conditions for the DML estimator to be consistent and asymptotic normally distributed Chernozhukov:2018. Converges rates satisfying Assumption (ref)-(vi) have been proven for a variety of machine learning algorithms, e.g. Lasso \citep*{Belloni:2013}, random forests \citep*{Wager:2015}, boosting \citep*{Luo:2016} or neural nets \citep*{Farrell:2021}. In addition, we require the calibrator to fullfil the following assumption.

Assumption (ref) requires the calibrator to converge to the true expected value of the treatment $D_i$ conditional on the estimated of propensity score $\hat{p}(X_i)$. In particular, Assumption (ref)-(iv) requires the calibrator to converge fast enough. For example, when the estimated uncalibrated propensity scores are beta distributed within the treated and nontreated observations, Beta scaling converges at the parametric rate $N^{1/2}$, fulfilling the requirement of Assumption (ref)-(iv). Rates for the nonparametric calibrators using isotonic regressions can be found in, e.g. Zhang:2002. Notice that the above assumption requires the calibrator to nest the identity function: the true propensity score $p \in \Xi_N$ and Assumption (ref) has to hold for all $p^* \in \Xi_N$. In fact, Platt scaling does not fulfill this requirement and the asymptotic result presented below do not hold for this calibrator. Assumption (ref)-(v) requires the calibrator to increase the $L_2$-error of the uncalibrated propensity score estimator by no more than a finite factor $C$. The theoretical result is then given by the following theorem.

The proof of Theorem (ref) is relegated to Appendix (ref). Theorem (ref) shows that the DML estimator with calibrated propensity scores has the same asymptotic distribution as the usual DML estimator. In particular, the estimator attains the parametric rate of convergence $\sqrt{N}$ and its variance achieves the semi-parametric efficiency bound Hahn:1998. While this asympotic result does not shed light on the finite sample properties, it provides assumptions under which a calibration method does not alter the asymptotic distribution of the ATE estimator.

Simulation Study

Data Generating Process (DGP)

We generate six different DGPs to test the finite sample performance of the calibration methods. In the spirit of Nie:2021, data is generated as follows:

We vary the difficulty of the baseline main effect and the propensity score function. Let $X_{i,j}$ denote the $j$th entry of the random vector $X_i$. An easy and a difficult baseline main effect $b(X_i)$ are generated as follows:

where the difficult baseline main effect is the scaled Friedman:1991 function. We consider three different propensity score functions:

where $\beta_{2,4}(x)$ is the beta cumulative distribution function with shape parameters 2 and 4. The first propensity score function has a logistic functional form and should be easy to estimate. Following Kunzel:2019, the second and third propensity score functions are highly nonlinear and pose greater challenges for accurate estimation. Furthermore, the third function produces extreme propensity scores. Figure (ref) shows the overlap of the propensity scores by plotting kernel densities of the propensity score distributions conditional on $D_i$.\footnote{Figure (ref) is based on a simulated sample of 2,000 observations. The kernel densities slightly overestimate (underestimate) the maximum (minimum) propensity score.} The first subplot has a nearly perfect overlap between the treated and untreated group, the second one shows already a strong selection into treatment, but the minimum and maximum probability of being treated lie at 10% and 70%, respectively. For the extreme propensity scores, the maximum probability of being treated lies at 95% and the minimum probability at 5%. Hence, the treatment selection is very strong.

We consider six different combinations of the baseline main effect and the propensity score function, resulting in six different DGPs as shown in Table (ref).\footnote{The Python code for the simulation study is available at \url{https://github.com/dballinari/Improving-the-Finite-Sample-Performance-of-DML-with-Propensity-Score-Calibration}.}

Simulation Design and Estimators

For each of the DGPs, we generate $R=1,000$ samples of sizes ($N$) 2,000, 4,000 and 8,000 and compare three different estimators for the nuisance functions, namely a random forest, gradient boosting, and a linear (for the conditional outcomes) and logistic (for the propensity scores) regression with $L_1$ penalty (Lasso). For the linear and logistic regressions, additional features are generated by including polynomials and interactions of order two of the raw covariates $X_i$. For each DGP and sample size, we tune the machine learner of choice on 20 simulation replications, choosing the hyperparameter combination that most frequently yields the best cross-validation performance. Table (ref) and table (ref) in Appendix (ref) shows the grid search and selected hyperparameters used for the different estimators.

The different calibrated versions of the DML estimator are compared with the DML estimator \citep*{Chernozhukov:2018} and the DML estimator with normalized and truncated weights \citep*{Huber:2013}. The calibration methods used are Platt scaling, Beta scaling, isotonic regression, Venn-Abers calibration, temperature scaling and expectation consistent calibration. The performance of the estimators is evaluated using the root mean squared error (RMSE), the bias, the standard deviation and the coverage of the 95% confidence interval of the ATE estimate $\hat\theta$. The performance of the propensity score estimation is evaluated using the Brier score Brier:1950 defined as:

and for calibrated propensity scores we replace $\hat p(X_i)$ with $\tilde{\pi}(X_i)$. Ideally we would measure the mean squared difference between the true propensity score and the estimated/calibrated propensity score. However, in empirical applications the true propensity score is not observed. The Brier score replaces the true unknown propensity score with the observed treatment indicator. At the population level, $p(x)$ minimizes the Brier score.\footnote{The solution to the optimization problem $\min_g \mathbb{E}[(g(X)-D)^2]$ is $g(X) = E[D|X] = p(X)$.} This score is a popular measure for assessing the performance of probability forecasts Dal:2015. Moreover, Equation (ref) is a natural candidate to evaluate the performance of the propensity score estimation, as the theoretical results of the DML framework rely on the mean squared convergence of the propensity score estimate Chernozhukov:2018.

Results

Table (ref) shows the results for samples of size $N = 2,000$ for the different calibration methods, DGPs and machine learners. The results for the other sample sizes can be found in the Online Appendix (ref). By comparing the different DGPs, we see that DML performs well if the propensity score has a nearly perfect overlap and is easy to estimate (DGP 1 and 3). In this situation, using calibration methods does not improve the results but does also not worsen them. Moreover, all three machine learners work in these cases, as a linear model can easily approximate the data.

When the propensity score is challenging to estimate (DGP 2 and 4), the RMSE of the DML estimator increases, primarily due to a higher bias. Calibration methods can considerably improve the ATE estimation in terms of RMSE, with Venn-Abers calibration, Platt, or Beta scaling showing the best performance. As long as the outcome regression is easy to estimate and linear (DGP 2), all estimators perform well, with Lasso being the most effective. This underscores the practical relevance of the double-robustness property of the DML estimator: despite the difficulty in estimating the propensity score, the simple baseline main effect can be well estimated by a linear model.

Lasso performs significantly worse than the other two machine learners for highly nonlinear baseline main effects (DGP 4). Calibration methods can greatly improve the results, reducing the RMSE for all three machine learners. In particular, for Lasso Venn-Abers calibration reduces the RMSE of DML by almost 60%, reaching nearly the same RMSE as more sophisticated machine learners. Calibration methods reduce the bias in the ATE estimation, which is the main reason for the RMSE improvements. After calibration, the coverage of the confidence interval is close to the nominal level of 95% for all calibration methods. These patterns become even more noticeable when the propensity scores become more extreme (DGP 5 and 6). Except for isotonic regression, all calibration methods reduce the RMSE of the DML estimator by at least 50% when the propensity scores are extreme and the outcome regression is difficult to estimate (DGP 6). The reduction stems again from a decrease in the bias.

In summary, calibration of the propensity scores can significantly reduce the RMSE of the DML estimator when the propensity scores are difficult to estimate, regardless of the machine learner used. The improvements are mainly related to a reduction in the bias of the ATE estimate, while the standard deviation remains relatively stable. The reduction in bias translates into better coverage of the 95% confidence interval. Reweighed DML improves only marginally the DML estimator's RMSE, indicating that the calibration of the propensity scores does not simply tackle the problem of extremely small or large propensity scores. Venn-Abers calibration, Platt scaling, and Beta scaling perform best among the calibration methods. Isotonic regression performs poorly because some propensity scores are pushed to zero or one, leading to a significant bias. Therefore, we exclude it from the discussion below.

Next, we will focus on the results for DGP 4 for two reasons. First, as it is a challenging DGP, DML exhibits some bias, indicating room for improvement. As seen earlier, if the DGP is easy to estimate, calibration is unnecessary. Second, DGP 4 is still realistic, as we still have overlap, and there are not too extreme values in the propensity scores. Figure (ref) shows the RMSE for DGP 4 for the different machine learners and different sample sizes. For random forest and gradient boosting, the RMSE is of similar magnitude across the different calibration approaches. The difference in RMSE to the DML estimator and the reweighed DML estimator decreases with increasing sample size, meaning that calibration is more important for smaller sample sizes. As discussed earlier, the linear model performs considerably worse than the other two machine learners for the highly nonlinear DGP 4. Nevertheless, using Venn-Abers calibrated propensity scores significantly improves the RMSE of the Lasso estimator, reducing it to a similar level as the other two machine learners. This result holds across all three sample sizes.

The calibration methods improve the RMSE by improving the estimation of the propensity scores. This can be seen from Figure (ref), which shows for each replication the relation between the Brier score of the propensity score and the bias of the ATE estimate for samples of size 4,000 drawn from DGP 4.\footnote{The same patterns can be seen for the relation between the bias of the ATE estimate and the mean squared difference between true and predicted propensity scores.} For all three machine learners, Platt scaling and Venn-Abers calibration reduces the Brier score of the propensity score. This reduction in the Brier score of the propensity score translates into a reduction of the ATE estimate's bias. However, the reduction in the ATE bias is not of the same proportion as the reduction in the Brier score of the propensity score since prediction errors of the conditional outcome also influence the ATE estimator. This can be seen from the results of the linear model, where despite a large reduction in the Brier score of the propensity score, we still have a considerable variation in the bias of the ATE estimate. In summary, the Brier score could guide researchers in their choice of the calibration method (if any). For example, if we use the Brier score as a criterion for choosing the calibration method in each replication sample generated from DGP 4 ($N=4,000$), the RMSE of estimates based on gradient boosting predictions would have amounted to 0.0016, a 60% decrease compared to the DML estimator.\footnote{In more detail, for each replication we calibrate the predicted propensity scores using one of the calibration methods presented in Section (ref) (excluding the isotonic regression). The calibration procedure follows the same cross-fitting approach outlined in Algorithm (ref). We then compute the Brier score for each calibration method and choose the calibration method with the lowest Brier score. The ATE estimate for the replication is computed using the propensity scores calibrated with the chosen calibration method.}

Empirical Application

We compare the different approaches in an empirical setting using the dataset from Knaus:2020. We analyse the effect of a language course on the employment status of Swiss unemployed in 2003.\footnote{An extensive dataset description can be found in Knaus:2022.} The dataset contains 80,706 observations with 2,412 treated and 78,294 control units and includes information on whether an individual participated in a language course ($d = 1$) or not ($d = 0$). The outcome ($Y_i$) measures the number of months an individual was employed during the first six months following the start of the unemployment spell. The ATE is identified in an unconfoundedness setting, assuming that the potential outcomes are independent of the treatment assignment conditional on the covariates. The covariates ($X_i$) include socio-economic characteristics and labour market history. From the literature, we know that in the short run, active labour market programs have a negative effect on the number of months employed due to a lock-in effect Lalive:2008,Knaus:2022.\footnote{Individuals taking part in a course are less likely to search for a job due to time constraints.} Hence, we expect a negative ATE.

The simulation study showed that calibration methods are particularly relevant in small samples, as they tackle a finite sample problem. The empirical application is thus conducted on random sub-samples of the full dataset to investigate the performance of the different calibration methods in small samples. We draw random sub-samples of 12.5%, 25%, and 50% of the treated and the control units separately.\footnote{The random sub-samples of different sizes are drawn incrementally. For example, to get a 25% sample, we draw the same observations as in the 12.5% sample and add 12.5% more observations to the sample. The same is done for the 50% sample.}

Table (ref) reports the results for a single draw of the sub-samples using a random forest to predict the nuisance functions. The table reports for each calibration method the ATE, its standard error and the Brier score. On the full sample, the ATE is approximately -0.59 across all calibration methods, indicating that visiting a language course leads to a decrease in employment duration of slightly more than half a month in the first six months after the start of the unemployment spell. The result confirms our previous finding, that for large sample sizes the calibration methods do not affect the results. In the following, we will evaluate the results for the different sub-samples in relation to the full sample estimate. Given the large sample size, the standard errors are relatively small and the ATE estimate is likely close to the true quantity of interest.

A researcher estimating the ATE using DML on a sub-sample of 12.5% would obtain an estimate of -0.73, concluding that the language course has a 23% more negative effect compared to the estimate she would obtain using the full sample. After calibrating the propensity scores using the calibration method with the lowest Brier score, the ATE estimate reduces to -0.614, bringing it considerably closer to the full sample estimate. Thus, when working with a small sub-sample, calibrating propensity scores may yield an estimate closer to that of the full sample. As the size of the random sub-samples increases, the DML estimate converges to the full sample estimate, and the differences between the calibrated and uncalibrated DML estimates decrease. This is consistent with theoretical expectations: with larger sample sizes, DML converges to the true effect, the accuracy of propensity score estimation improves, and the need for calibration diminishes.

The results reported in Table (ref) are confirmed across different random draws of the sub-samples and different machine learners. In Table (ref) in the Online Appendix (ref), we report average ATE estimates across ten random draws of the sub-samples for random forest, gradient boosting, and Lasso.

Conclusion

This paper investigates the impact of calibration methods on the performance of the double/debiased machine learning estimator in finite samples through a simulation study. We find that calibration methods can significantly improve the performance of the DML estimator when the propensity scores are difficult to estimate and there is high selection into treatment. The improvements are mainly related to a reduction in the bias of the ATE estimate, while the standard deviation remains relatively stable. Venn-Abers calibration, Platt scaling, and Beta scaling perform best among the calibration methods. Isotonic regression performs poorly because some propensity scores are pushed to zero or one, leading to a significant bias. Thus, the choice of the calibration method is crucial and some methods do not work well in the DML framework. However, if a suitable calibration method is chosen, even if there is no need to calibrate the propensity score, it does not hurt to do so. This is validated by our asymptotic result, providing conditions under which the calibrator does not alter the asymptotic properties of the DML estimator. Practitioners should consider the Brier score as a criterion for choosing the calibration method, and should avoid the use of isotonic regression. These results are confirmed in an empirical application, where we find that calibration methods can be particularly relevant in small samples and do not hurt in larger samples.

Future research could investigate the use of calibration methods on the estimation of other treatment effects, such as heterogeneous treatment effects, or in other causal settings, such as difference-in-difference or instrumental variable settings.