Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
47,463 characters · 15 sections · 62 citation commands
Sample Fit Reliability
\thispagestyle{empty}
\setcounter{page}{1}
Improving the reliability of empirical results is a core concern for all applied econometrics leamer1983, leamer1985. Researchers routinely assess the credibility of causal identification angrist2010, the selection of confounders wuthrich2020, the functional form of models athey2017, the properties of estimators busso2014, and the model uncertainty \parencites{lehrer2017}{steel2020}. Less attention, however, is dedicated to assessing and improving the data that is used to represent a phenomenon kuschnig2021. In other words, researchers often test model fit (by holding data constant and varying the model), but rarely test sample fit (by holding the model constant and varying the data).
Accepting data “as-is” can lead to studies that over-fit to noise hastie2009 and fail to replicate to new data \parencites{hubbard1996empirical}. Unfortunately, some have concluded that empirical research is not to be trusted \parencites[e.g.,][]{ioannidis2005}. In comparison, fields such as data science and machine learning deal with bad data (including errors in measurement, labeling, data entry, disambiguation, data conversion, record merging, censoring, and so forth) with computational methods that can test the sensitivity and rubustify the results with respect to the underlying data \parencites[see e.g.,][]{fischler1981}{liu2008}{zheng2016}{schouten2018}{cohen2020}. Much as the “credibility revolution” in applied econometrics angrist2010 shifted attention to the importance of research design, we argue that more attention in the field should be focused on data reliability. We believe computational methods offer promising tools and solutions to that end.
In this paper, we provide a data-focused perspective for assessing sample fit in econometric models. In particular, we suggest a re-sampling procedure to estimate the reliability of the data, check the sensitivity of the results, and improve the robustness of the analysis. Accordingly, we propose Sample Fit Reliability (SFR) as a computational approach with three aspects: Scoring, to estimate a reliability score for every observation in a sample based on the expected estimation loss over sub-samples; Annealing, to test the sensitivity of results to the sequential removal of unreliable data; and Fitting, to estimate a weighted regression that adjusts for the reliability of the sample. To demonstrate the utility of SFR, we provide an example based on real-world data, test the performance of SFR using synthetic data in simulations, and replicate renowned field experiments in economics. We find that, in each case, SFR methods improve understanding of sample fit and reveal valuable insights about the reliability of results.
Our contribution is threefold. First, we put forward the concept of sample fit as a complement to the widely used notion of model fit. Second, we combine ideas from classical robust statistics with modern machine learning to develop SFR. Third, we introduce a graphical approach to sensitivity analysis that uses annealing based on reliability scores to provide insights about the stability and heterogeneity of effects. Finally, we provide an implementation of SFR in a Python package called \href{https://pypi.org/project/samplefit/}{samplefit} that is available on the PyPI repository for future use by applied researchers.
Concerns about unreliable data traditionally have been addressed via robust estimation with a long tradition in statistics rousseeuw2005. In particular, M-estimation methods such as Huber regression huber1964 acknowledge the instability of OLS in the presence of outliers. M-estimation modifies the loss function and incorporates weighting based on residuals \parencites{fox2002}{yu2017}. Another stream of literature aims to identify sets of “influential observations” cook1979. Influential observations are data points that strongly impact quantities of interest, including model parameters, when those observations are omitted belsley2005. The measurement of influential data points generally relies on leverage and the magnitude of residuals chatterjee1986, or a similar assessment of influence \parencites{hampel1974}{hampel2011}.
Researchers also use re-sampling methods to test, diagnose, and improve results by perturbing the underlying data yu2013. Re-sampling methods include techniques such as the bootstrap, the jackknife, cross-validation, and other forms of sub-sampling \parencites{efron1982}{efron1994}. Bootstrapping, for example, is used in econometrics to quantify sampling uncertainty for estimated parameters \parencites{diciccio1996}{horowitz2001}{mackinnon2006}; bootstrap aggregation (“bagging”) is used in machine learning to reduce variance and improve predictions \parencites{breiman1996}{buhlmann2002}; bootstrapping and sub-sampling are used together in ensemble methods such as random forests breiman2001 and causal forests \parencites{lechner2018}{wager2018}{wager2019}; and sub-sampling is used in computer vision to separate inliers vs. outliers raguram2008.
Additionally, grandvalet2001, grandvalet2004 points out that bagging stabilizes estimation in the presence of a few influential points by indirectly down-weighting those points, similarly to robust regression methods. In computer vision, re-sampling techniques such as the Random Sample Consensus (RANSAC) algorithm fischler1981 drop outliers and influential points, instead of re-weighting. In particular, RANSAC repeatedly draws random sub-samples of the data and assesses the fit of data points excluded from the sub-sample. Random sub-sampling is repeated until the number of inliers reaches a given criteria, and a final model is fitted on a “consensus” sub-sample of inliers. Random sub-sampling is also used in the optimization of the Least Median of Squares rousseeuw1984b, S-estimators \parencites{rousseeuw1984}{koller2017}, and forward search regression models \parencites{atkinson2007}{atkinson2010}.
Our work relates to several recent studies that address aspects of sample fit. Works by broderick2020, kuschnig2021 and moitra2022 identify data points within a sample that distort the estimation of model parameters when they are removed from an analysis in an adversarial manner. For example, broderick2020 develop a computationally-efficient method to approximately identify the maximum influence perturbation (AMIP). As such, AMIP provides information about the worst-case scenario -- i.e., situations where the estimation of a parameter can be distorted by removing a minimum proportion of the sample. kuschnig2021 further adapt perturbation testing to cases where jointly influential data points are a concern rousseeuw1990, and moitra2022 provide theoretical guarantees for low dimensional cases.
We develop Sample Fit Reliability as a computational approach that builds on insights from both classical robust statistics and modern machine learning. We begin by focusing on residuals as the fundamental diagnostic tool for regression. However, as pointed out by belsley2005, OLS residuals tend to underestimate the true error of outliers due to the squared error criterion. Therefore, we adopt a re-sampling procedure similar to RANSAC and compute the absolute loss based on residuals from many minimal random sub-samples; doing so reduces dependency on the results from any one sub-sample, and helps to alleviate the underestimation of the errors of outliers. Furthermore, re-sampling explores a wide range of possible model fits and stabilizes the estimation of loss through averaging. Thus, in contrast to RANSAC, we do not automatically search for a “best” sub-sample (e.g., a sub-sample free of outliers). Instead, we efficiently use all of the information from each sub-sample to estimate the expected loss for each observation. We therein avoid arbitrary specifications of error thresholds. Finally, we define reliability scores to be reversely related to the expected losses.
Below, we define the target quantity of interest, the estimation algorithm, a new approach to sensitivity analysis, and a robust re-weighting procedure for estimation. We focus on the parametric family of models, and in particular on the linear regression models, of type:
$$Y_i = X_i'\beta + \epsilon_i$$
where $Y_i$ is the outcome, $X_i$ is a vector of covariates, $\beta$ is a vector of unknown coefficients, and $\epsilon_i$ is the error term.
There are several reasons why we focus on linear models. First, linear models are still the dominant models for causal inference \parencites{angrist2008}{imbens2015}, despite recent developments in causal machine learning athey2019. For example, selection-on-observables via covariate adjustment, instrumental variables via two-stage least squares, and difference-in-differences via fixed effects are typically all estimated in linear models. As such, current econometric research focuses on addressing problems with estimating treatment effects via OLS in the presence of effect heterogeneity \parencites{sloczynski2022}{pinkham2022}. Similarly, our research addresses problems with estimating effects -- including treatment effects -- in the presence of unreliable data. Second, and in comparison to flexible machine learning models, linear models provide a high degree of interpretability via marginal effects. Interpretability is valuable for communicating results to policy makers and is often required in regulated contexts. Our work therefore supports the advancement of simpler, more-interpretable models by restricting and/or re-weighting the sample according to the reliability of the data. Third, data in the social sciences is often scarce, and collecting additional data is generally expensive. As such, often there is insufficient data for highly-flexible, non-parametric models racine2007.
To address possible data issues, we are interested in estimating the reliability of each data point in a sample, conditional on a given model specification. We define our quantity of interest as follows. Suppose that we have access to $i=1,...,N$ training data $Z_i=(X_i, Y_i)$ and an input $x$. Consider an OLS estimator of the conditional mean $M(x;Z_i)=\mathbb{E}[Y_i \mid X_i=x]= X_i'\beta$. We are interested in the estimation error $\gamma(x)$ given a loss function as $\hat{\gamma}(x)=\mathcal{L}\big(Y_i, \hat{M}(x;Z_i)\big)$. With re-sampling we aim to stabilize the estimator of the loss as:
$$\hat{\Gamma}^{\infty}(x) = \mathbb{E}^*\big[\mathcal{L}\big(Y_{i}^*, M(x;Z_{i}^*)\big)\big]$$
where $Z_{i}^*$ are sub-samples drawn without replacement and the expectation $\mathbb{E}^*$ is taken with respect to the re-sampling measure. To evaluate the expectation, we rely on sub-sampling aggregation and form the Monte Carlo estimator as follows:
$$\hat{\Gamma}^{S}(x) = \frac{1}{S}\sum^S_{s=1} \gamma_s^*(x) \text{ where } \gamma_s^*(x)=\mathcal{L}\big(Y_{i,s}^*, M(x;Z_{i,s}^*)\big)$$
with $Z_{i,s}^*$ being the $s$-th sub-sample. Thus, for $S \rightarrow \infty$ we approximate $\hat{\Gamma}^{\infty}(x)$. Finally, we define the reliability score as a reverse transformation of the expected loss as follows:
$$\hat{\psi}(x) = \frac{\hat{\Gamma}^{S}(x) - \max\{\hat{\Gamma}^{S}(X_1,...,X_N)\}}{\min\{\hat{\Gamma}^{S}(X_1,...,X_N)\} - \max\{\hat{\Gamma}^{S}(X_1,...,X_N)\}}$$
to ensure boundedness of reliability scores as $0 \geq \hat{\psi}(x) \geq 1$. A reliability score of 0 corresponds to the largest expected loss. A reliability score of 1 corresponds to the smallest expected loss.
The SFR algorithm for Scoring the reliability of each data point in a sample is defined below. Consider an $iid$ random sample of size $i=1,...,N$ with random variables consisting of features $X_i$ of dimension $p$ and outcomes $Y_i$. SFR draws a random sub-sample $(Z^*_{i,s})_{i\in A_s}$ of size $\eta=p+1$ from the original data $Z_i=(X_i, Y_i)$ without replacement,\footnote{Such sub-sampling is close to the elemental sets sampling, where sub-sample size is exactly equal to $p$ olive2007. In practice, we find the almost elemental sets of size $p+1$ to have more stable performance.} where $A_s$ is the set of indices corresponding to the $s$-th sub-sample for $s=1,...,S$ sub-sampling iterations. For each sub-sample $(Z^*_{i,s})_{i\in A_s}$, SFR estimates the model $M_s\big(x;(Z^*_{i,s})_{i\in A_s}\big)$ via OLS and evaluates the corresponding loss, i.e. $\hat{\gamma}_{s}(X_{(-i)})$, for each out-of-bag data point $(Z_{i,s})_{i\notin A_s}=(X_{(-i)}, Y_{(-i)})$, given a loss function $\mathcal{L}\big(Y_{(-i),s}^*, \hat{M}(X_{(-i)};Z_{i,s}^*)\big)$,\footnote{We use an absolute loss similarly to RANSAC fischler1981.} where $\hat{M}_s(X_{(-i)};Z_{i,s}^*)$ is the model prediction for an out-of-bag observation $X_{(-i)}$. The losses for an individual data point $i$ are subsequently averaged over the $S$ sub-sampling iterations in which the data point $i$ was not used for estimation, i.e. out-of-bag losses, as $\Gamma^S(X_i) = \frac{1}{S}\sum^S_{s=1} \gamma_s(X_{(-i),s})$.\footnote{The number of possible sub-samples increases with order $N^{\eta}$ and it becomes prohibitive to explore all of them. Yet, as pointed out by koller2017 in the case of S-estimators, in practice setting S=1000 is sufficient.} A reliability score $\psi_i$ for data point $i$ is then reversely related to the average loss, such that the largest (smallest) average loss corresponds to a reliability score of zero (one). The reliability scores are effectively bounded between 0 and 1 through $\hat{\psi}_i = \frac{\hat{\Gamma}^{S}(X_i) - \max\{\hat{\Gamma}^{S}(X_1,...,X_N)\}}{\min\{\hat{\Gamma}^{S}(X_1,...,X_N)\} - \max\{\hat{\Gamma}^{S}(X_1,...,X_N)\}}$. Intuitively, if an observation exhibits a large (small) loss throughout many random sub-samples, relative to the other observations in a sample, we define such an observation as having a low (high) reliability.\\
\\
As can be seen in Algorithm (ref), the main output of SFR is a full complement of reliability scores $\hat{\psi}_{i}$ across all observations in the sample. The scores postulate a continuous, point-wise measure of data reliability based on the reverse relationship to the average losses, i.e. higher average losses imply lower reliability scores. Then, reliability scores can be inspected and investigated to generate insights about sources of error (e.g., a corrupt process for capturing data, censoring, measurement error, or a secondary data generating process).
The practical implications of scoring reliability are illustrated in Figure (ref), based on the popular Boston housing dataset \parencites{harrison1978}{belsley2005}.\footnote{Descriptive statistics are provided in Appendix (ref).} We consider the relationship between the per capita crime rate and the percentage of the population that is considered to be lower status in a neighborhood (e.g. lack of education, labor work, etc.). The scatterplot of the data suggests a slightly positive relationship, distorted by a few neighborhoods that have an unusually high level of crime, but a moderate level of lower status individuals. Figure (ref) shows that SFR naturally identifies unusual data points and assigns the reliability scores accordingly. In particular, we see that reliability scores are lower for observations that are further from the center of the distribution and/or reinforced by less evidence. Such diagnostics of sample fit can help to draw attention to conditions where the model does not generalize well.
Researchers can sort reliability scores into increasing order (i.e. $\hat{\psi}_{(1)} \leq \hat{\psi}_{(2)} \leq \cdots \leq \hat{\psi}_{(N)}$) for a continuous measure of reliability across the sample. Sorted reliability scores $\hat{\psi}_{(i)}$ then can be used to assess the sensitivity of sample fit by re-estimating the model, while incrementally dropping unreliable data points from the sample. Accordingly, the estimation of a quantity of interest can be visualized along the annealing path.\footnote{Previously, riani2014 advocated for graphical monitoring of robustness via related concept of forward plots atkinson2007.} More specifically, Annealing drops a specified share of the most unreliable observations in a sequential manner to capture the sensitivity of parameter estimates to unreliable data.\footnote{The appropriate level of annealing will depend on the particular dataset and the amount of evidence required by the researcher; however, we have found that a $10\%$ annealing share is sufficient for many practical applications.} To reflect the uncertainty of estimated parameters, we use bootstrap approximation to take the two steps of the estimation procedure (both the estimation of reliability scores and the subsequent annealing) into account for inference. Here we follow bodory2020 who show favourable performance of bootstrapping compared to asymptotic approximations for matching and weighting estimators that use estimated propensity scores and trim the sample accordingly \parencites(see for example)(for a discussion of trimming based on propensity scores)[][or]{crump2009}{lechner2019}. We illustrate the annealing based on the example from the Boston housing dataset in Figure (ref).
In Figure (ref), we see that the estimated effect decreases as we remove the most unreliable data points, and the estimated effect stabilizes once approximately $5\%$ of the sample has been annealed. After $5\%$, further annealing of the data does not substantially change the point estimate. The annealing graph shows that the original effect estimate might not provide an appropriate description of an underlying relationship. We also note that restricting the sample through annealing essentially estimates the effect for different sub-populations, as defined by reliability scores. Therefore, it is recommended to check the covariate balance between the restricted and unrestricted sample in order to prevent over-generalization of the results diamond2013. The annealing graph also draws the attention of researcher to the data points in a sample that may merit further inspection. For example, researchers might investigate the source of unreliability (i.e., measurement errors, different sub-populations, or a mis-specified model), and adapt accordingly (e.g., collect new data or specify new model).
In the sections above, we demonstrate how to use SFR to score the reliability of data and then perform a graphical, post-estimation sensitivity analysis of sample fit via annealing. In some cases, researchers may then identify problems with data that justify dropping a portion of the sample, and researchers may then continue their analysis with the cleaned sub-sample. In the following section, however, we test how well a regression weighted by reliability scores can make an automatic analysis more robust.
SFR can be used for Fitting a given model $M(x;Z_i)$ to the sample weighted by reliability scores $\hat{\psi}_i$, or their transformations $g(\hat{\psi}_i)$, wherein data points with higher reliability scores are up-weighted, and conversely data points with lower reliability scores are down-weighted. Similar forms of weighting are ubiquitous in most robust regression analysis gervini2002. We suggest a squared transformation of reliability scores, i.e. $g(\hat{\psi}_i)=\hat{\psi}_i^2$, that further accentuates the down-weighting of low reliable observations.\footnote{See for example nie2021 for a similar squared weighting scheme to down-weight influence of extreme propensity scores in a non-parametric treatment effect estimation.}
We thus apply the weighted least squares estimator as follows:
$$\hat{\beta}=\operatorname*{arg\,min}_{\beta}\sum_{i=1}^N g(\hat{\psi}_i)\cdot\big(Y_i - X'_i\beta\big)^2$$
to estimate the parameters of the model $M(x;Z_i)=\mathbb{E}[Y_i \mid X_i=x]=X'_i\beta$.\footnote{Given our definition of reliability, with the most unreliable data point having a score of zero, weighting will effectively discard one data point. The loss of information, however, is of order $(1/N)$ and will vanish asymptotically.} We apply non-parametric pairs bootstrap diciccio1996 to approximate standard errors due to the two-step estimation procedure described above (i.e., SFR first estimates reliability scores and then estimates a weighted regression using the estimated scores). diciccio2019 establish the consistency of the bootstrap approximation for the sampling distribution of the weighted least squares estimator and further show its better coverage in finite samples in comparison to asymptotic approximations.
Table (ref) presents results of a reliability-weighted fit for the Boston housing dataset examined earlier. We see that the reliability-weighted fit supports a better description of the underlying relationship by effectively down-weighting the influence of unreliable data points. Doing so results in a lower marginal effect, in line with conclusions drawn from the annealing exercise.
In this section, we investigate the performance of SFR with simulated data in four stylized scenarios, with a sample size of $N=1000$ and an outlier share of $\alpha = 5\%$. In Appendix (ref), we vary $N$ and $\alpha$ and obtain qualitatively similar results.
Scenario 1: No Outliers. In a baseline scenario, we simulate $i=1,...,N$ data points from a linear model $Y_i= X_i'\beta+\epsilon_i$, where $X_i \sim \mathcal{N}(0,1)$, $\beta=1$, and $\epsilon_i \sim \mathcal{N}(0,2/3)$. OLS is the best linear unbiased estimator in this scenario.
Scenario 2: Randomly Distributed Outliers. In the second scenario, we simulate data in the same manner as in the first scenario, but include a set $\mathcal{B}$ of size $\alpha\times N$ consisting of randomly distributed outliers via simulating data points $X_i \sim \mathcal{N}(0,2)$ with a corresponding outcome $Y_i \sim \mathcal{N}(0,2)$ for $i \in \mathcal{B}$. In this case, one would expect random distortions of the OLS estimator.
Scenario 3: Densely Distributed Outliers with Leverage. In the third scenario, we simulate the data again as in the first scenario, but in this case include a set $\mathcal{B}$ of size $\alpha\times N$ of densely distributed outliers with high leverage by simulating data points $X_i \sim \mathcal{N}(2,1/2)$ with a corresponding outcome $Y_i \sim \mathcal{N}(-1,2/3)$ for $i \in \mathcal{B}$. Scenario 3 is comparable to creating an influential set as in broderick2020, including the masking phenomenon as in kuschnig2021. The OLS estimator is expected to have a negative bias that would attenuate the main effect.
Scenario 4: Clusters of Densely Distributed Outliers with Leverage. In the fourth scenario, we simulate data in the same manner as in Scenario 3, except that we split the set $\mathcal{B}=\mathcal{B}_1 \cup \mathcal{B}_2$ into two equal halves and simulate data points $X_i \sim \mathcal{N}(2,1/2)$ with a corresponding outcome $Y_i \sim \mathcal{N}(-1,2/3)$ for $i \in \mathcal{B}_1$ and data points $X_i \sim \mathcal{N}(-2,1/2)$ with a corresponding outcome $Y_i \sim \mathcal{N}(1,2/3)$ for $i \in \mathcal{B}_2$, respectively. Hence, we create two distinct clusters of outliers with leverage that are expected to bias the OLS estimator in a similar way as in Scenario 3.
In Figure (ref), we present graphical results from estimating reliability scores based on the synthetic data described above. In Scenarios 2 through 4, we see that SFR accurately scores the reliability of observations corresponding to both clustered and randomly distributed outliers. Notice that reliability scores are both distance and density sensitive -- reliability scoring reflects the distance to the regression line and the amount of information around each point. As such, leverage points that comply with the imposed regression line are scored with lower reliability, even though they would not distort the estimation. Moreover, leverage points that stand off the imposed regression line and do distort the estimation, are scored with even lower reliability. As a consequence, even in Scenario 1 without outliers, SFR assigns lower reliability scores to data points with low density surrounding the imposed regression line such that the estimation would not be distorted. Hence, the simulation provides evidence that SFR scoring appropriately evaluates the reliability of the data.
To go beyond an assessment of the reliability of individual data points, and to gain substantive insights as to how unreliable data can affect the estimation of parameters for a given model, we apply the annealing procedure described above to assess the sensitivity of effects to the exclusion of unreliable data. We present graphical results from the annealing procedure for the synthetic datasets in Figure (ref). For Scenario 1 (no outliers), annealing has no effect; for Scenarios 2, 3 and 4 (with outliers), annealing converges to the true underlying effect $\beta=1$ after removing the first $5\%$ of unreliable data points (i.e., the same $5\%$ included in set $\mathcal{B}$ during simulation of the synthetic data).\footnote{Note, that the annealing converges faster to the true effect in Scenario 2 as some of the randomly distributed outliers overlay with the true underlying DGP.} We therefore conclude that annealing can help to detect unstable effects related to outliers, without distorting results that are free of outliers.
Finally, we evaluate the performance of a weighted regression based on reliability scores for 1,000 replications of each scenario. Table (ref) reports the results for the mean squared error, the mean absolute bias, the standard deviation and the Jarque Bera statistic of normality, for the estimation of the effect (slope) parameter. In column 1 (no outliers), OLS estimation is theoretically the optimal choice. Nevertheless, SFR exhibits low bias and slightly lower efficiency, while being normally distributed as well. In column 2 (randomly distributed outliers), the bias for OLS is considerably larger, and so is its variance. In contrast, SFR performs substantially better than OLS, both in terms of bias and MSE. In column 3 (single cluster of densely distributed outliers with high leverage), OLS is noticeably biased, and is again outperformed by SFR. Finally, in column 4 (two clusters of densely distributed outliers with high leverage), OLS again has high bias, while SFR correctly addresses the presence of outliers and returns bias and MSE that is an order of magnitude lower than that of OLS, while being comparably efficient.
Appendix (ref) reports additional results for comparisons with the Huber and RANSAC estimators. SFR is more robust than both Huber and RANSAC in the simulations, especially against high leverage points. A known drawback of the Huber estimator is sensitivity to high leverage points yu2017, and the simulations demonstrate high variance for the RANSAC estimator, although RANSAC's performance is expected to be better in datasets with higher outlier contamination raguram2008. Across the simulations, we conclude that SFR compares favourably to both OLS and robust regression methods, when an automated analysis is preferred to the researcher-driven methods for scoring and annealing covered in previous sections.
Replication and reproducibility is an important aspect of all empirical research \parencites{anderson2008}{azoulay2015}{berry2017}{hoffler2017}. In the following section, we replicate three renowned field experiments from economics. In each case, the average treatment effect is identified via randomization and estimated by OLS in the following linear model:
where $Y_i$ is the outcome, $\alpha$ is the intercept, $W_i$ denotes the binary treatment indicator, $\theta$ measures the corresponding average treatment effect (ATE) and $\epsilon_i$ is the error term.
In the first replication, we revisit the well-known field experiment in labor economics by lalonde1986. lalonde1986 evaluates the impact of job training on future yearly earnings in US dollars based on a dataset of 445 individuals (henceforth the “labor data”), where 185 were randomly assigned to join the National Supported Work (NSW) demonstration program in the US dehejia1999. Appendix (ref) reports descriptive statistics for the dataset.
In the second replication, we revisit the randomized control trial of angelucci2015. angelucci2015 follow a vast literature on microcredit (for an overview, see banerjee2015 or meager2019) and study the impact of access to microcredit on household welfare in Mexico. The sample includes data for more than 16,000 households (henceforth the “microcredit data”), out of which approximately half received access to microcredit through door-to-door loan promotion. We follow broderick2020 and kuschnig2021 and re-evaluate the impacts of the microcredit provision on household profit in US dollars (PPP per 2 weeks). Appendix (ref) reports descriptive statistics for the dataset.
In the third replication, we consider a field experiment concerned with the economics of charity list2007. list2007 study the effect of mail solicitation on charitable giving in the US based on a dataset of more than 50,000 prior donors (henceforth the “charity data”), where individuals were randomly assigned into treated and control with a treatment share of approximately $2/3$. The treated group received direct mail solicitations seeking contributions with an additional matching grant. We re-evaluate the impact of the matching grant on the donation amount in US dollars. Appendix (ref) reports descriptive statistics for the dataset.
We begin our replications by scoring data point reliability and plotting the results in Figure (ref). For the labor data, SFR suggests that there are four data points with low reliability, representing individuals with atypically high earnings above 30,000 dollars.\footnote{For a discussion of outliers in the non-experimental version of the dataset, see canavire2021.} For the microcredit data, SFR reveals that there is one data point that is highly unreliable, with very high leverage, representing a single household with an extremely large loss. For the charity data, SFR suggests that the dataset contains several unreliable data points with moderate leverage, and that the unreliable data is partly balanced between treated and control. These observations represent unusually high donations compared to the rest of the sample.
In reviewing Figure (ref), we note that low reliability scores do not automatically imply “bad data”. Instead, the scores and plots above may motivate researchers to inspect the data more carefully and assess whether the indicated observations are corrupted by measurement error, are related to a rare or unexpected condition, or otherwise are outliers that may distort what is represented by the estimated “average” effect of the treatment.
We continue our replications by annealing the samples and plotting the re-estimated treatment effect across the sequential removal of the most unreliable data points, along with a bootstrapped $95\%$ confidence interval for inference.
In Figure (ref), we observe that the estimated effect of job training on earnings gradually decreases from the initial estimate of almost 2,000 dollars to around 1,000 dollars after $2\%$ of the most unreliable points are removed from the sample. The annealed effect becomes statistically indifferent from zero after about $1\%$ of the most unreliable points are removed from the sample. As such, we believe that the annealing graph enhances the substantive conclusions one can draw from the study based on the reliability of the sample.
In Figure (ref), we observe that the effect of the provision of microcredit on profits changes rapidly and inconsistently during the annealing of the first half of $1\%$ of the sample. After dropping around $1\%$ of the sample, the effect changes from negative to positive and then stabilizes around zero. We also observe instability in the estimation of the treatment effect, represented by the wide confidence interval in the region with the lowest reliability scores. Given that overall the replication does not find evidence for an effect (as the original study did not), it might appear that SFR does not change the substantive conclusions of the study. However, SFR does allow one to conclude with higher confidence that the ATE for the vast majority of the sample is in fact close to zero.
In Figure (ref), we observe a clear pattern in the treatment effect of matching grants on donations. Dropping the first $2\%$ of the sample with the lowest reliability scores changes the point estimate of the ATE from a positive, to a negative, and back to a positive result (albeit with broad confidence intervals that appear to come in and out of statistical significance). Annealing the first $2\%$ stabilizes the effect close to zero, along with a tight bootstrapped confidence band. Together with scoring, annealing reveals that the average effect seems to be driven by a few donations that are exceptionally large. Moreover, there appears to be no evidence of an effect for over $98\%$ of the sample. Instead, alternative explanations might exist to shed light on the results that were annealed from the top $2\%$ of unreliable observations in the sample.\footnote{However, note the implicit assumption of effect homogeneity that is imposed by the linear model specification.} In turn, researchers might be motivated to test crump2008 and investigate treatment effect heterogeneity in a systematic way \parencites[see e.g.,][]{athey2019}[or][for such approaches]{kunzel2019}[as well as][for such heterogeneity analysis in the context of charitable giving.]{cagala2021}.\footnote{Note that list2007 also conduct additional, more refined analyses going beyond the difference-in-means evaluation for the aggregate effect and provide results on conditional analyses as well as effect heterogeneity analyses.}
In reviewing Figures (ref), (ref), and (ref), we note that restricting the sample via annealing results in the estimation of an ATE for different sub-populations, as stratified by sorted reliability scores. In other words, the annealing procedure translates the ATE to a type of a conditional average treatment effect, i.e. CATE \parencites{abrevaya2015}{jacob2021}. We therefore recommend researchers to examine the covariate distribution between removed and remaining sub-samples to assess whether underlying sub-populations differ in practice \parencites(see for example)(for a discussion of appropriate covariate balance measures)[][or]{iacus2011}{diamond2013}.
Finally, we complete our replications by executing an automatic analysis that estimates treatment effects based on reliability scores via a weighted regression with 1,000 bootstrap replications for inference. We summarize the results from the weighted regressions in Table (ref), and comparisons to Huber and RANSAC estimations are provided in Appendix (ref).
In the case of the evaluation of the job training program, an SFR regression weighted by reliability scores yields a point estimate that is substantially lower than an OLS estimate, with a similar statistical precision based on the bootstrap inference. SFR results for the microcredit study also suggest an effect that is markedly different from the original OLS estimate, although the estimate is noisy and we do not find evidence for an effect that is different from zero. Lastly, SFR estimation of the effect of charitable giving yields a smaller, but more precise estimate of the ATE, compared to the original estimate.
In this paper, we propose the concept of sample fit as a complement to the widely used notion of model fit and argue for a more data-focused perspective in econometric models. To assess sample fit, we combine ideas from classical robust statistics and modern machine learning to develop Sample Fit Reliability -- a computational re-sampling approach to estimate the reliability of the data, check the sensitivity of the results, and improve the robustness of the analysis. SFR entails three aspects: Scoring, to estimate reliability scores for every data point in a sample that reflect the average estimation loss over sub-samples; Annealing, to test the sensitivity of estimation results to the sequential removal of the most unreliable observations from the sample; and Fitting, to estimate a weighted regression that down-weights the unreliable observations in the sample. We provide a software implementation of SFR in a Python library called \href{https://pypi.org/project/samplefit/}{samplefit} available on PyPI to facilitate its usage for applied empirical research.\footnote{Data and replication files can be found at: \url{https://github.com/okasag/samplefit/tree/main/replication}.}
We illustrate the application of SFR using a real-world data example; test its performance based on synthetic data in simulations; and apply SFR in replications of renowned field experiments in economics. In general, we find that SFR methods improve understanding of sample fit and reveal valuable insights about the reliability of results. As such, we join a stream of research that aims to improve sensitivity analysis, model interpretation, and other forms of robust generalization by explicitly tackling data issues to increase credibility of empirical studies.
Given a substantial amount of data work (i.e., cleaning, filtering, imputing, scaling, etc.) involved in empirical research, we aspire to shift the attention in applied econometrics to the importance of data reliability, much as the “credibility revolution” angrist2010 shifted attention to the importance of research design. With the development of SFR, we provide one approach for assessing sample fit, but there also are complementary approaches, such as recently proposed methods by broderick2020, kuschnig2021 and moitra2022 that challenge results in an adversarial manner.
Finally, it would be valuable to extend the SFR framework to non-parametric models, including highly flexible machine learning models. We also see potential for future research that attempts to simultaneously optimize model fit, together with sample fit, for better prediction models, better analysis of effect heterogeneity, and a better understanding of algorithmic fairness.
\printbibliography