Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
125,854 characters · 20 sections · 63 citation commands
Assessing Utility of Differential Privacy for RCTs
\doublespacing
\acresetall
\Acp{RCT}, have become powerful tools for assessing the impact of interventions and policies in many contexts 10.1257/aer.110.7.1952. Today, they are considered the gold standard for inference in the biomedical fields and many social sciences. In economics, their use has expanded rapidly since the 1990s, with studies ranging from small-scale interventions at the individual or village level to national programs. Researchers have published an increasing number of studies that rely on RCTs for at least part of the inference. The AEA RCT Registry alone has over 11,500 registrations as of January 2026, with new ones added at a rate of 1748 per year in 2025. More than 12,700 unique researchers are associated with these registrations rct-report-2026.
Alongside the growth of RCTs, the push for research transparency has led journals to require public release of data and code as “replication packages.” The \ac{AER}, the flagship journal of the \ac{AEA}, has required analysis data and code since 2004 bernanke2004. Some AEA journals launched in 2009 implemented such a policy from creation; see vilhuber2020 and vlaeminck2021 for a review of reproducibility practices in economics. The broader availability of complete replication packages has enabled re-analyses and meta-analyses that strengthen understanding of both methods and findings. For example, meager_understanding_2019 re-analyzed multiple RCTs using Bayesian hierarchical analysis (BHA), while 10.1257/aeri.20210236 examined event studies for pre-treatment time trends. \footnote{It should be noted that 10.1257/aeri.20210236 still had to exclude nearly four times as many papers as they included because data were not readily available.}
A parallel development concerns novel privacy/ confidentiality\footnote{We will use privacy and confidentiality interchangeably to mean data confidentiality.} protection methodologies. While privacy protection has long been a standard part of the toolkit of statistical agencies hundepool2012statistical and ethical research guidelines, new formal privacy mechanisms --- such as various forms that satisfy \ac{DP}dwork_calibrating_2006,dwork_calibrating_2016,TheAlgorithmicFoundationsofDPDwork --- offer transparent and mathematically provable confidentiality. We briefly describe the basic idea behind DP in the next section. Nevertheless, the typical (recent) guidance followed by researchers who conduct \acp{RCT} primarily relies on much weaker methods, such as de-identification department_of_health_and_human_services_methods_2012,kopper_j-pal_2020,dime_-identification_2020, $k$-anonymity k-anonymity, $l$-diversity MachanavajjhalaGKV06, and other aggregation-based methods hundepool2012statistical. However, most data included within replication packages that allow for exactly reproducing the results in the papers are typically simply de-identified. We are not aware of the application of techniques from \ac{DP} or relaxations of DP in the context of the dissemination of data collected as part of \Acp{RCT}. \footnote{Randomized response RandomizedResponseWarner is shown to be a (distributed) DP mechanism, used at the collection stage, and has been applied in industry. A recent example explicitly referencing it in data collection within an RCT is kancharla2021robust; the earliest (central) DP work is of Vu2009.} This suggests that much of the current literature on \Acp{RCT} publishes replication packages that contain inadequately protected data. This is particularly concerning in the economic data setting we are exploring, because many of these studies have data from respondents in \acl{LMIC}, where there may be lower legal protections than in Europe or North America.
One of the possible reasons for the absence of strong privacy protection methods in this literature is that no tools are available to non-specialists that would allow for easy but efficient protection \textcolor{black}{consistent with the definition of \acl{DP}}.\footnote{We are aware of the Two Ravens tool tworavens, but have not seen usage of it in the space we surveyed. } Efficiency here is defined as “perturbing inference as little as possible compared to the unprotected inference.” We note that inference even in the “unprotected” case is already subject to uncertainty that is often not adequately taken into account meager_understanding_2019. This is even more important for the uncertainty and data modifications that are generated through \ac{SDL}. abowd_economic_2015 and slavkovic_seeman_2022 argue for the need to account for the privacy-preserving noise in analyses. slavkovic_seeman_2022, and references therein, discuss a general way to adjust for privacy-preserving noise in addition to other sources of uncertainty.
The present article contributes to the literature on privacy-aware analysis. Broadly, we aim to contribute on two separate dimensions. First, in the context of data collected for \acp{RCT}, we investigate the feasibility of preserving the generic quality of inference obtained using the confidential data even when the same inference is performed using data endowed with privacy protections that are stronger than the simple de-identification usually used. Second, we do so while maintaining the feasibility of application, here defined as \textcolor{black}{computational feasibility} on commodity hardware used by researchers in these fields (and in particular, by researchers in \acl{LMIC}, where many \acp{RCT} are conducted). We have created an R package\iftoggle{anonymous}{ available on Github (URL redacted)}{, DPrct DPrct}, which includes some basic differential privacy algorithms, including the algorithms discussed and proposed in this paper. The package allows researchers to easily implement privacy-preserving methods on their RCT-based data.
Our focus on \acp{RCT} is intentionally narrow.\footnote{A somewhat different approach is taken by rosenblatt2023b, who start with frequently-used published datasets and explore the (conceptual) reproducibility of analyses in articles that used such datasets. They, too, focus on the simpler methods and find mixed results.} We believe that exploring the impact of privacy-preserving technologies in the context of \acp{RCT} is useful for several reasons. First, statistical methods are, in general, straightforward: standard linear regression, difference-in-difference methods, and possibly even simple difference in means across treated and untreated populations. These are among the first analysis methods for which adaptations to DP protection have been studied awan2020structure, alabi,slavkovic2021perturbed, barrientos2018, bowen2020synthetic. If formal privacy-preserving methods cannot be made to work “out-of-the-box” and at scale in this context, then it will be much more difficult to argue for broader application. Second, most \acp{RCT} are small-scale, using samples of the overall population, allowing us to avoid computational constraints when algorithms scale with sample size $N$.\footnote{We note that sampling might also allow us to leverage privacy-amplifying methods balle2018privacy, though we do not exploit that in this paper. We also note that DP was originally designed with large rather than small samples in mind but the small-scale databases are as equally important in practice.} Third, \acp{RCT} are often accompanied by pre-analysis plans, with specific hypotheses in mind and with the intent to avoid false discovery. These areas have also been explored within the DP framework Vu2009, nixon2020, dwork_differentially_2021). Furthermore, it is already understood in the privacy community that the inherent noisiness of the sampling may affect inference slavkovic_seeman_2022. The analogy between adding noise for the purpose of BHA meager_understanding_2019, and adding noise for privacy protection may be convenient to improve the acceptance of such methods. A similar Bayesian framework can be used to adjust noisy inference due to privacy seeman2020private.
Specifically, we explore the impact of privacy-preserving methods in two ways. First, we propose three privacy-preserving mechanisms that use as building blocks some of the basic DP algorithms. Through simulations, we show their appropriateness for the stated goals. We also explain why they are not fully DP, and identify the underlying data issues that complicate an easy fully-DP implementation. We then use data and models from a published analysis of an RCT, apply the algorithms to the data prior to reproducing the study's original methods, and verify that inference is not unreasonably affected. In this paper, we use the data from Reducecrime [henceforth “BJS”] as published in their replication package ReducecrimePkg.\footnote{The article was published in the \ac{AER}. One of the authors is the Data Editor of the \ac{AEA} at the time of writing of this article, but the article in question predates his tenure at the \ac{AEA}, and was not subject to his review.} The protected data is constructed so as to be a drop-in replacement for the data originally published by BJS. Importantly, we explicitly do not change the analysis methods used by the authors, except where necessary to take into account the privacy-preserving mechanism. There are possibly many ways that other researchers might have analyzed the data collected by this (or other) sets of authors. We do not explore those, but note that it is precisely through the availability of replication packages that such differing approaches can be addressed -- through replications.
Randomized controlled trials are experiments conducted to evaluate the effectiveness of a policy, a drug, or some other intervention. They are particularly popular in biomedical fields and have become increasingly common in policy evaluations in socio-economic contexts. The key feature is that assignment to “treatment” is randomized within the eligible population -- whether it means receiving a novel drug or a cash grant. Random assignment can be at the level of individuals, institutions (e.g., schools, hospitals), villages, provinces, etc. Generally, a control group is also chosen at random from the same population, but does not receive the treatment. The popularity and strength of \acp{RCT} comes from the powerful notion of causality, due to the randomization procedure. As long as the randomization procedure actually worked as designed, it is often sufficient to compare means for the two samples (treatment and control) of relevant outcome variables to obtain a valid causal inference as to the effectiveness of the treatment. For the application in this paper, this simplicity also means that methods used to obtain causal inference are (in theory) statistically simple, and thus lend themselves, in theory, more easily to the application of strong privacy-preserving mechanisms.
The key definition of differential privacy, DP, relies on the concept of $\epsilon$-indistinguishability dwork_calibrating_2006,machanavajjhala2015. An algorithm $M$ satisfies $(\epsilon,\delta)$-differential privacy (\acl{appDP}) for some $\epsilon,\delta>0$ if for each of its possible outputs $\omega$ and for every pair of databases $D_1 , D_2$ that differ on the changing of a single record,
When $\delta=0$, an algorithm $M$ satisfies $\epsilon$-differential privacy (\acl{pureDP}). In other words, given the outputs from two databases that differ only in a single record are very “similar”, it is statistically very difficult (quantified by parameter $\epsilon$, also referred to as a privacy-loss, that is positive but ideally a small value greater than 0) to know if any particular record was included in the database or not. Various privacy-preserving methods/mechanisms can satisfy these and related definitions desfontaines2022sok. In general, mechanisms inject precisely controlled statistical noise into the data. Randomized response RandomizedResponseWarner is an example of input noise injection at the collection stage. We will be concerned with output noise injection, which involves perturbing data that have already been collected.\footnote{For examples of DP mechanism perturbations, and a much broader taxonomy, see desfontaines2022sok.} The noise is typically taken from well-known distributions, such as Laplace and Gaussian (normal).
In what follows, we aim to use terms that are comprehensible to a variety of disciplines. All of our mechanisms will start with original or confidential data $D$. Parameters estimated using $D$ are treated as confidential. A mechanism $M(\cdot)$ that transforms $D$ into privacy-preserving \textcolor{black}{or privacy-protected} output $\omega$ is said to protect confidential data. In this paper $\omega$ is schema-equivalent to $D$, and we will designate the privacy-preserving output as $\widetilde{D}$. We try to avoid the somewhat ambiguous term “privatized” for $\widetilde{D}$, as many social scientists associate the act of “making private” as making something non-public, which is the opposite of the intent of the application of $M(\cdot)$ here. In the context of our analysis of BJS, we will use the term \textbf{original}, because the authors have already transformed the \textbf{confidential} data into the \textcolor{black}{downloadable} and published data in ReducecrimePkg using a mechanism $M_{deidentify}$. We will treat those data as if they were “confidential” for the purpose of the experiments in this article. Finally, because we are in the context of \acp{RCT}, we also have randomized assignment (possibly stratified) to treatments (including the control group), which we will denote as $R$, an (usually public) algorithm that conditions on the data available to the researchers.
We pursue three goals motivated by a setting where the typical researcher wants to:
We focus on a particular key table in BJS, Table 2. It is the result of several independent regressions, each with a separate dependent (response) variable of interest, measured in the short term (Panel a) and long term (Panel b), controlling for a set of assignment indicators for the various treatments, other covariates, and stratifiers. \footnote{Reducecrime stratify the treatment across a few subgroups, but for expositional ease, we mostly ignore that aspect. In the empirical analysis, we implement the authors' stratified random assignment as per their methodological description.} These are “intent-to-treat” (ITT) regressions. They are typical of \acp{RCT}, where the experimenter is interested in determining whether a particular treatment has any effect on a response variable when the treatment is applied to an entity, individual, or treatment unit. The experimental treatment is randomly assigned to the treatment units according to some chosen experimental design, and response variables are recorded at various points after the treatment. As is typical for this type of analysis, BJS has both discrete and continuous covariates. The response variables are also a mix of continuous and discrete outcomes.
The various characteristics and attributes of the treatment unit (in this case a person participating in the study), including membership in different strata and the actual outcomes, may all be sensitive. In BJS, participation in therapy, receipt of cash (two treatments), and self-reported criminal activity (covariates) are all quite sensitive attributes and may pose privacy concerns. While the inclusion of many covariates in the regression analysis can improve the estimate of the treatment effect, these covariates also pose a privacy concern for the treatment units participating in the study. An attacker, who may access the database provided as part of the replication package, may be able to reidentify individuals, and learn new, sensitive information about them. This would constitute a confidentiality violation for the treatment units.
Consider a confidential dataset $D = \left [ \mathbf{y}, \mathbf{T}, \mathbf{X}\right ]$ with $n$ rows and $1+t+p$ columns, where $n$ is the total number of treatment units, a single response variables $\mathbf{y}$, $t$ mutually exclusive treatment assignments $\mathbf{T}$ (the control group being one of them), and $p$ covariates $\mathbf{X}$\footnote{Additionally, there could be $b$ blocking variables $\mathbf{V}$, in which case the confidential dataset is $D = \left [ \mathbf{y}, \mathbf{T}, \mathbf{X}, \mathbf{V}\right ]$ with $n$ rows and $1+t+p+b$ columns. The corresponding regression model is $\mathbf{y}=\mathbf{T}\boldsymbol{\tau}+\mathbf{X}\boldsymbol{\gamma}+\mathbf{V}\boldsymbol{\nu}+\mathbf{e}$ where $\mathbf{V}$ is $n\times b$ matrix and $e_i\overset{i.i.d.}{\sim}N(0,\sigma^2)$ for $i=1,\dots,n$. In this case, the randomized treatment assignment may depend on the blocking variables $\mathbf{V}$.}. $\mathbf{T}$ is assigned based on some \textcolor{black}{randomized treatment assignment} algorithm $R$.
The regression model of interest, in the absence of blocking variables , is given by
or
where $\mathbf{T}$ is \textcolor{black}{an} $n \times t$ matrix of exhaustive assignment to treatment arms, $\boldsymbol{\tau}$ is \textcolor{black}{a} $t$-dimensional vector of \textcolor{black}{(fixed)} treatment effects, $\mathbf{X}$ is $n \times p$ matrix of covariates, $\gamma$ is a $p$-dimensional vector of regression coefficients \textcolor{black}{(fixed effects) corresponding to the explanatory covariates}, and $\mathbf{e}$ is \textcolor{black}{an} $n$-dimensional error term with $e_{i} \stackrel{i.i.d}{\sim} N(0,\sigma^{2})$ for $i=1,\dots,n$\footnote{In BJS, there are many response variables, \textcolor{black}{but each regression is treated independently, and can be assumed to be based on a different dataset with the first column being the different response variables but with the same $t+p+b$ columns for the treatment assignment, explanatory covariates and blocking variables (if any)}.} .
In the context of \Acp{RCT}, the parameter(s) of interest to the experimenters are the treatment effects $\boldsymbol{\tau}$. From the experimenter's perspective, statistical utility is preserved if the inference concerning the treatment effects $\boldsymbol{\tau}$ (such as point estimates, uncertainty quantification of point estimates, interval estimates and $p$-values for significance tests) is minimally affected by the mechanism used to perserve privacy of the data and/or the relevant statistics in order to protect privacy.
We propose three methods to produce privacy-preserving data, $\widetilde{D}$ designed specifically to preserve the utility of the inference on the treatment effects. The first method samples a perturbed multivariate histogram and is model-agnostic (see, Algorithm (ref)). The second two methods are model-informed, but use the sampled perturb histogram to protect the model covariates. Of the two model-informed methods, one provides additional confidentiality protection to the original data by infusing noise to the estimated model parameters. All three methods provide much stronger privacy protections than de-identification and are built from techniques and ideas from differential privacy (DP). While we denote privacy budgets using $\epsilon$ and $\delta$, the methods themselves do not satisfy strict ($\epsilon,\delta$)-DP definitions. Like in DP, increasing $\epsilon$ or $\delta$ will decrease the privacy protections and increase utility of the privacy-preserving data.
Sampling from a perturbed multivariate histogram is one of the earliest proposed mechanisms dwork_calibrating_2006, wasserman_statistical_2010 that satisfies $\epsilon$-DP. The perturbed multivariate histogram mechanism can be extended to a stability-based histogram mechanism to satisfy ($\epsilon,\delta$)-DP by introducing a probability threshold $c$ bun16_multilearner,karwa2017finite. The probability threshold allows to only infuse noise to the observed combinations of column values rather than all possible combinations by reducing the probability that low frequency combinations are sampled. Our multivariate histogram method is based on these traditional mechanisms but does not satisfy strict $\epsilon$-DP or ($\epsilon,\delta$)-DP definitions due to how it discretizes the continuous variables and how it deals with possible combinations of discretized covariates that are not observed in the sample. The histogram counts are protected using the Laplace mechanism and when necessary, we choose the precision of binning\footnote{The precision of binning prescribes how many bins are created for each continuous variable dimension, as a function of the sample size (for e.g., if $\zeta = \frac{2}{3}$ and $p=2$, with both variables being continuous, the total number of bins in the multivariate histogram is $\left(n^{\frac{2}{3}}\right)^2 = n^{\frac{4}{3}}$).}, denoted by $\zeta$, and discretize continuous variables. It is important to notice the bounds on the discretized bins are informed by the original confidential data. This is the first reason why the Multivariate Histogram Method in Algorithm (ref) is not strictly differentially private. Some confidentiality protection comes from adding uniform noise to discretized column values to make them continuous again at the end of sampling from the perturbed histogram.
The deviation from DP has to do with our threshold $c$ and the unique valued rows $\mathbf{r}^*_1,\dots, \mathbf{r}^*_q$. Under $\epsilon$-DP, the unique values rows would have to include all possible combinations of the categories and discretized levels. If we had $m_{cat}=7$ binary categorical variables and $m_{cont}=7$ continuous variables which were discretized into $\eta=10$ bins each, then there would be $q=2^{m_{cat}}\times10^{m_{cont}}=$1.28 billion unique valued rows. However, in Windows the limit for a variable in the 32-bit build of R is 4Gb Rbase. Using the VGAM package, generating even $900$ million Laplace random variables to sanitize proportions exceeds the 4.0Gb limit VGAM. Since data from \Acp{RCT} studies can have a large number of covariates, we only use combinations of discretized covariates observed in our sample. This would not be a problem for the stability-based histogram which satisfies ($\epsilon,\delta$)-DP, except \Acp{RCT} data often has a small number of observations with its large number of covariates. This data structure can lead to each combination of discretized and categorical covariates observed in the sample to only be observed a once or twice (i.e., $q$ is close to $n$). Often, this results in no privacy-preserving proportions being above the threshold $c$. In the case where no privacy-preserving proportions are initially above the threshold, we set the threshold to $0$.
\SetKwComment{Comment}{/* }{ */}
The perturbed multivariate histogram method does not depend on the model used in the study. It can be generalized to any $D$ and does not matter which columns are response, treatment variables, covariates, etc. It does depend on what columns are continuous and which are discrete or categorical. To generalize, in Algorithm (ref) we allow $D$ to be a $n\times (m_{cat}+m_{cont})$ database, where the first $m_{cont}$ columns are continuous and the rest are categorical. We denote the element in row $i$ and column $j$ as $d_{ij}$. The row vectors are $\mathbf{d}_{i\cdot}$ for $i=1,\ldots,n$ and the column vectors are $\mathbf{d}_{\cdot j}$ for $j=1,\ldots,(m_{cat}+m_{cont})$. Lines 3 to 7 in Algorithm (ref) discretize the continuous variables. The multivariate histogram is created by lines 9 and 10. The histogram is perturbed in line 11 using Laplace noise. The privacy-preserving proportion becomes $0$ if it is below a threshold in lines 12 to 18. Then the unique rows are sampled in line 20 using normalized privacy-preserving proportions. Finally, the continuous covariates are converted back to continuous values using uniform noise in lines 21 to 24. We denote the application of Algorithm (ref) to $D$ with $\epsilon$ and $\zeta$ parameters as $M_{hist}(D; \epsilon, \delta, \zeta)$.
Within the proposed model-informed methods, the infused-noise can be targeted to prioritize accuracy in the model summary fitted to the privacy-preserving data. It focuses on maintaining the relationship between the variables that is described by the model.
We propose two model-informed algorithms for sanitizing the data $D$. Both methods use the MV Histogram Method (Algorithm (ref)) and post-processing to sanitize the explanatory variables of a model. They differ on how the response variable is generated, but have the same overall flow (Figure (ref)). The first method is based on approximate DP, but uses the model parameters that are estimated using the confidential data, $D$. This method, called the Hybrid Method (Algorithm (ref)), allows flexibility in what models are used but does not satisfy the strict approximate DP definition. The second method, called the GenModel Method (Algorithm (ref)), which preserves the confidentiality of model parameters, but relies heavily on the normality assumptions of multiple linear regression models with Gaussian residuals, such as (ref).
The Hybrid Method uses the estimated model parameters ($\hat{\boldsymbol{\tau}},~\hat{\boldsymbol{\gamma}},$ and $\hat{\sigma^2}$) from a prediction model like (ref) that depends on standard inferential procedures performed on the confidential data, $D$. The parameters of the model ( $\boldsymbol{\tau},\boldsymbol{\gamma},$ and $\sigma^2$) are estimated using a standard inferential procedure, such as ordinary least squares, and the original confidential data Dunn_Smyth_2018. Then the covariate variables are protected using Algorithm (ref), $\widetilde{\mathbf{X}}\gets M_{hist}(\mathbf{X}; \epsilon, \delta, \zeta)$. Using the same treatment assignment algorithm as the confidential data ($R(\cdot)$) we generate new treatment indicators based on the $\widetilde{\mathbf{X}}$, $\widetilde{\mathbf{T}}\gets R(\widetilde{\mathbf{X}})$. The generation of $\widetilde{\mathbf{T}}$ is a post-processing procedure, and thus does not require privacy budget allocation. Then, $\mathbf{y}$ is inputted from the model. Finally, the privacy-preserving inference concerning the treatment effects $\boldsymbol{\tau}$ is obtained using a post-processing procedure, which involves performing standard inferential procedures (the same procedures that one may use when privacy preservation is not a concern) on the privacy-preserving replication data [$\widetilde{\mathbf{y}}$, $\widetilde{\mathbf{T}}$, $\widetilde{\mathbf{X}}$]. By using the parameter estimates obtained from the confidential data $D$ in this algorithm, it is expected that inference validity is preserved by the sanitization procedure. The released $\widetilde{\boldsymbol{\tau}}$ is nevertheless perturbed and therefore conventionally protected because it is based on $\epsilon$-DP-protected $\widetilde{\mathbf{X}}$ and model-informed imputation (as outlined below), but is not formally DP-protected. Since the $\widetilde{\mathbf{y}}$ is imputed using the estimated model coefficients which are estimated with the confidential data, these values do not satisfy approximate DP.
Algorithm (ref) uses the ($\epsilon,\delta$)-DP protected treatment indicators $\widetilde{\mathbf{T}}$ and covariates $\widetilde{\mathbf{X}}$, the confidential parameter estimates $\hat{\boldsymbol{\tau}}$, and a suitable model for the privacy-protected response variable $\widetilde{\mathbf{y}}$. The confidential $\hat{\boldsymbol{\tau}}$ are not published; rather privacy-protected $\widetilde{D}(\epsilon,\zeta) = \left [ \widetilde{\mathbf{y}}, \widetilde{\mathbf{T}}, \widetilde{\mathbf{X}} \right ]$ is published as part of the replication package, and $\widetilde{\boldsymbol{\tau}}$, its associated standard errors and inferential statistics based on (ref), are estimated using $\widetilde{D}$, and published as part of the relevant article. It is possible that information about the confidential estimated effects and thus about the confidential data could leak through the release of $\widetilde{D}$. However, this risk is minimal compared to the risk from publishing the estimated effects derived from the confidential data.
We also consider a model-informed method that satisfies approximate DP to sanitize $D$ when the regression model is of the form (ref) or similar models with a normally distributed error term. We take advantage of the fact that the estimated coefficients and residuals are generated from a Gaussian distribution, and use two algorithms proposed by karwa2017finite. We refer to these as the DP Variance Algorithm and DP Range Algorithm which are detailed in Appendix (ref).
The privacy-preserving dataset outputted by the Hybrid method is denoted $\widetilde{D}\gets M_{hybrid}(D; \epsilon, \delta, \zeta, R).$
We also propose a model-informed method, GenModel , which infuses noise to the model parameters. The algorithm is detailed in Algorithm (ref) and visualized in Figure (ref). The process of infusing noise to the model parameters in this method would satisfy approximate DP if the privacy-preserving covariates where generated in a way that satisfied approximate that is ($\epsilon,\delta$)-DP, unlike our adaptation of the perturbed multivariate histogram in Algorithm (ref). The GenModel method requires the generating model to have Gaussian distributed error terms. Additionally, the number of rows in the data, $n$, needs to be large enough to use the central limit theorem such that the sample mean of $n$ random variables from a $\chi^2$ distribution with 1 degree of freedom is approximately Gaussian. When comparing results of datasets protected by the GenModel algorithm with different privacy budgets in Sections (ref) and (ref), we abbreviate the algorithm as GenM- followed by the privacy budget (e.g., GenM-1 ).
The first step of the GenModel algorithm is to sanitize the covariate dataset using $\epsilon_{\mathbf{X}},\delta_{\mathbf{X}}$ privacy parameters with the MV Histogram algorithm (Algorithm (ref)). Then privacy-preserving treatment levels are assigned using the randomized treatment assignment algorithm, $R$. Then we sanitize the estimates of the model parameters $\widetilde{\boldsymbol{\tau}},~\widetilde{\boldsymbol{\gamma}},$ and $\widetilde{\sigma^2}$. With the privacy-preserving parameter estimates, covariate data, and treatment assignments, we impute $\mathbf{y}$. The key idea behind sanitizing the model parameters utilizes two important statistical properties of inference in a regression model. We first construct $B$ independent proxies of the response variable $\mathbf{y}$ (denoted by $\hat{\mathbf{y}}_{b}$, $b=1,\dots,B$) using the prediction model with the privacy-preserving predictor variables
where $\mathbf{e}_{b} \overset{i.i.d}{\sim} N(0,\hat{\sigma}^2 I_n )$ for $b=1,\dots,B$. We then perform the regression of these $B$ proxy responses on the privacy-preserving data $\widetilde{\mathbf{W}} = [\widetilde{\mathbf{T}},\widetilde{\mathbf{X}}]$ separately for each $b=1,\dots,B$ to obtain $B$ independently and identically distributed sets of estimates. Denote the coefficient estimates as $\boldsymbol{\beta}^{*}_{b}=[\boldsymbol{\tau}^{*}_{b},\boldsymbol{\gamma}^{*}_{b}]$ for $b=1,\dots,B$ where $\boldsymbol{\beta}^{*}_b$ is a $t+p$-dimensional vector stacking the $t$-dimensional $\boldsymbol{\tau}^{*}_b$ and $p$-dimensional $\boldsymbol{\gamma}^{*}_b$. Similarly, denote the $\hat{\sigma^2}$ estimates as $\mathbf{s}^{*2}=[\xi_1,\dots, \xi_B]$.
The first key statistical property that we leverage is that $\beta^{*}_{b}$ follows a multivariate Gaussian distribution with mean $\hat{\boldsymbol{\beta}} = [\hat{\boldsymbol{\tau}},\hat{\boldsymbol{\gamma}}]$ and covariance matrix $\hat{\sigma}^2 (\widetilde{\mathbf{W}}^{T}\widetilde{\mathbf{W}})^{-1}$. Thus, $\boldsymbol{\beta}^{*}_{1},\dots,\boldsymbol{\beta}^{*}_{B}$ can be considered as an i.i.d sample from a multivariate Gaussian distribution\footnote{If the model includes blocking variables, $\mathbf{V}$, and is of the form $\mathbf{y}=\mathbf{T}\boldsymbol{\tau}+\mathbf{X}\boldsymbol{\gamma}+\mathbf{V}\boldsymbol{\nu}+\boldsymbol{\epsilon}$, then the data consisting of $[\mathbf{X}~\mathbf{V}]$ is protected with the MV Histogram algorithm. The treatments are randomly assigned. The privacy-preserving predictors are $\widetilde{\mathbf{W}}=[\widetilde{\mathbf{T}},~\widetilde{\mathbf{X}},~\widetilde{\mathbf{V}}]$. Additionally, $\boldsymbol{\beta}^{*}_{b}=[\boldsymbol{\tau}^{*}_{b},\boldsymbol{\gamma}^{*}_{b},\boldsymbol{\nu}^{*}_{b}]$ are used to sanitize the model coefficients.}. In particular, the $\boldsymbol{\tau}^{*}_{b,j}$'s constitute an i.i.d sample of size $B$ from an univariate Gaussian distribution with mean $\hat{\boldsymbol{\tau}}_{j}$ and variance $\hat{\sigma}^2 ((\widetilde{\mathbf{W}}^{T}\widetilde{\mathbf{W}})^{-1})_{jj}$ for $j=1,\dots,t+p$. For $j=1,\dots,t$, these are the point estimate and the estimated variance of the treatment effect $\boldsymbol{\tau}_{j}$. For $j=t+1,\dots,t+p$, these are the estimated covariate coefficient, $\widehat{\gamma_{j-t}}$, and its estimated variance. We use two \acl{appDP} algorithms by karwa2017finite to sanitize the point estimates of the estimated regression coefficients and the residual standard errors. Additionally, the residual mean squared error is $\xi_b\sim\frac{\sigma^2}{n}\sum_{i=1}^nZ_i$ where $Z_i\overset{i.i.d.}{\sim} \chi^2(1)$ conditional on $\widetilde{\mathbf{W}}$. For $n$ large enough, $\xi_b$ is approximately Gaussian with mean $\sigma^2$ and variance $2\sigma^2/n$. The residual mean squared errors are protected in a similar fashion to the coefficients. First we use the DP Variance algorithm proposed by karwa2017finite (Algorithm (ref) in Appendix) with privacy parameters $\epsilon_{var},\delta_{var}$ and standard deviation bound parameters $\sigma_{min}$ and $\sigma_{max}$. Then we use the DP Range algorithm (Algorithm (ref) in Appendix) with privacy parameters $\epsilon_{var},\delta_{var}$ and mean bound parameter $\upsilon_{\sigma^2}$. Then Laplace noise is added to the mean of the $B$ MSE values with privacy parameter $\epsilon_{\sigma^2}$. The privacy-preserving estimate of the residual variance is denoted $\widetilde{\sigma^2}$. This can be used to estimate the coefficient covariance matrix as $\widetilde{\sigma}^2 (\widetilde{\mathbf{W}}^{T}\widetilde{\mathbf{W}})^{-1}$ Dunn_Smyth_2018. A privacy-preserving mean of of the sample $\boldsymbol{\tau}^{*}_{b,j}$ for $b=1,\dots, B$ would make an appropriate privacy-preserving estimate of $\beta_{j}$. For the $k$ treatment effect, we get a privacy-preserving range of the mean based on the sample using the $k$th diagonal value of $\widetilde{\sigma}^2 (\widetilde{\mathbf{W}}^{T}\widetilde{\mathbf{W}})^{-1}$ as the estimated standard deviation in the DP Range algorithm proposed by karwa2017finite (Algorithm (ref) in Appendix) with privacy parameters $\epsilon_{range,\tau, k},\delta_{range,\tau, k}$, range confidence parameter $\alpha_{range,\tau, k}$, and mean bound parameter $\upsilon_{\tau,k}$. Then we generate Laplace noise with scale parameter equal to the output of DP Range over $\epsilon_{\tau,k}$, and add it to the sample mean. We repeat this with all the covariate coefficients $\ell=1,\dots,p$ (reindexing $\ell=j-t$) with privacy parameters $\epsilon_{range,\gamma, \ell},\delta_{range,\gamma, \ell}, \epsilon_{\gamma, \ell}$ and range algorithm parameters $\alpha_{range,\gamma,\ell}$ and $\upsilon_{\gamma,\ell}$. More generally, several parameters can be expressed in terms of $\beta$ such as $\epsilon_{range,\beta,j}=\epsilon_{range,\tau,j}$ when $j\leq t$ and $\epsilon_{range,\beta,j}=\epsilon_{range,\gamma,j-t}$ otherwise. In the end, the GenModel method has an overall privacy budget of $(\epsilon,\delta)$ where $\epsilon=\epsilon_{\mathbf{X}}+\epsilon_{var}+\sum_{j=1}^{t+p}\epsilon_{range,\beta,j}+\epsilon_{\beta,j}$ and $\delta=\delta_{\mathbf{X}}+\delta_{var}+\sum_{j=1}^{t+p}\delta_{range,\beta,j}$. We find it useful to denote the privacy budget for generating the response separate from the budget for sanitizing the covariates, $\epsilon_{\mathbf{y}}=\epsilon-\epsilon_{\mathbf{X}}$ and $\delta_{\mathbf{y}}=\delta-\delta_{\mathbf{X}}$. The application of the GenModel method on a dataset $D$ is denoted $M_{GenM}(D;~ \boldsymbol{\epsilon},~\boldsymbol{\delta},~B,~ \sigma_{min},~\sigma_{max},~\boldsymbol{\upsilon},~\boldsymbol{\alpha}_{range},~\zeta,~R)$. Note that $B$, the number of generated proxy response variables, is a user choice and can be made as large as required.\footnote{$B\geq 5000$ has been suggested, in personal communication with one of the authors of karwa2017finite, for the validity of the algorithms to be subsequently applied.}
The $\widetilde{\sigma^2}$ produced in line 17 of Algorithm (ref) is accurate to $\sigma^2$ up to a factor according to karwa2017finite. It may add too much noise to generate $\widetilde{\mathbf{y}}$ and utility will be lost. We discuss several potential variants. All the variants produce a $\widetilde{\mathbf{y}}$ that is based on some elements that are generated with methods that satisfy approximate DP. However, the variant methods use $\hat{\sigma^2}$ from the analysis with the confidential data. These would be a relaxation of the confidentiality protection. One variant uses the estimated $\hat{\sigma^2}$ that was fitted on the confidential data rather than $\widetilde{\sigma^2}$ in lines 24 to generate $\widetilde{\mathbf{y}}$. Similarly, $\hat{\sigma^2}$ could be used in line 17 and line 18. This would eliminate the need for line 15 and the inputs, $\epsilon_{var},\delta_{var},\sigma_{min},$ and $\sigma_{max}$.
The GenModel method could be extended to a generalized linear model using asymptotic properties of coefficient estimates. Given a generalized linear model, $\hat{\boldsymbol{\tau}}$ are asymptotically (as sample size $n$ increases) normally distributed. In the current work, we only consider linear regression model.
To compare the three proposed methods (Algorithms (ref), (ref), and (ref)), we will use a simulated dataset, $D_{sim}$. In section (ref), we discuss how the simulated dataset is generated. We will make comparisons across seven privacy budgets in Section (ref). Then in Section (ref), we compare across 11 models that differ by number of continuous and binary covariates and the type of continuous variables (finite-valued or unbounded). We focus on metrics for comparing the estimated treatment effects from the analysis on the sanitized and confidential datasets. These metrics are called utility measures and are described in Section (ref). The analysis methods remain the same across the confidential and sanitized data, but further improvements could be made by developing new analysis methods to account for the noise added into the sanitized datasets.
An existing utility metric relies on the overlap between the 95% confidence intervals (CI's) of the treatment effects from the sanitized and confidential estimated treatment effects for each treatment level doi:10.1198/000313006X124640. The sanitized treatment effects, $\widetilde{\tau}$, are estimated using the synthetic dataset produced by one of our three proposed algorithms. The confidential treatment effects, $\widehat{\tau}$, are estimated from the confidential dataset. For a treatment effect, $\tau$, let $(L,U)$ be the CI for the treatment effect computed based on the confidential dataset and $(\widetilde{L},\widetilde{U})$ be the CI computed based on the sanitized dataset. If the confidence intervals overlap, then their intersection is $(L^{over},U^{over})$, where $L^{over} = \max (L,\widetilde{L} )$ and $U^{over} = \min (U,\widetilde{U})$. In this case, the CI overlap metric is the ratio of the width of the intersection and the average CI width. When the CI's do not overlap, the overlap metric takes the value $0$. This overlap metric, denoted as $\widetilde{{O}}$, is
The confidence interval overlap can vary between $0$ and $1$, with values near $1$ indicating that there is a large degree of overlap. Thus, higher values (near 1) indicate that the deviation of the inference regarding the treatment effect based on the confidential dataset from the same inference based on the sanitized dataset is small. Lower values (near 0) can be a result of the centers of the intervals being far apart or unequal size of the intervals. If the confidence intervals are similarly sized, the further apart the centers are the smaller the intersection of the CI's is. Since the metric is scaled by the average interval width, if one CI is much wider than the other, then the overlap metric will be small even if the smaller interval is completely contained within the larger one.
Another utility metric we use is absolute difference. For our experiments, we can use the true treatment effect value, since it is known. The absolute difference is $|\Delta|=|\widetilde{\tau}-\tau|$. For real-world examples, we can approximate this by using the estimated treatment effect from analysis on confidential data, $\hat{\tau}$. The estimated absolute difference is $\widehat{|\Delta|}=|\widetilde{\tau}-\widehat{\tau}|$. This metric allows us to evaluate the sanitized point estimate, but does not take into account the standard error of the sanitized point estimate. Further research could be done on these algorithms to improve standard error estimates to be used in inference based on the sanitized data. Thus, absolute error can help show potential for a mechanism which needs the inference on the sanitized data to be adjusted.
Using the utility metrics discussed in Section (ref), we compare our three proposed sanitizing algorithms (Algorithms (ref), (ref), and (ref)) using simulated data sets in two experiments. The first experiment compares across privacy budgets ($\epsilon$ and $\delta$ values). The second experiment focuses on comparing the mechanism applications across models with different counts of covariate variables and different types of covariate variables. Each experiment uses a different simulated dataset with 1,000 observations. There is a control and treatment group assigned using complete random assignment. The data generating models are multiple regression models with the true treatment effect $\tau=5$ and true residual standard deviation $\sigma=2$. The model intercept is $1$ for both experiments. For each simulated dataset, the coefficients are generated from a geometric series so that there is a diminishing influence of each additional covariate in the model.
The coefficients for categorical variables are from a geometric series: $3(7/11)^{r-1}$ for $r=1,\dots, m_{cat}$ where $m_{cat}$ is the number of categorical covariates. Each categorical covariate is binary. To simulate each column, we randomly generate $c_{bin}$ from $\operatorname{Bin}(n=1000,~p=1/2)$. Then $c_{bin}$ observations are randomly sampled and given value $1$ in the column. Otherwise, the observations have value $0$.
The coefficients for continuous variables are from a geometric series: $0.99(2/3)^{r-1}$ for $r=1,\dots, m_{cont}$ where $m_{cont}$ is the number of continuous covariates. We use two alternating methods to generate the continuous covariate columns. The alternating types are unbounded continuous covariates and continuous columns that have a finite set of possible values ($0,0.01,0.02,\dots,0.2$). Starting with the first continuous column, all the unbounded covariate observations are generated from a Gaussian distribution with mean $0$ and standard deviation $2$. Finite-valued covariate observations are generated from a $\operatorname{Uniform}(0,0.2)$ and then rounded to two decimal places. Due to how the continuous variables are generated the scale of the unbounded values is likely larger than the finite-valued variables. Thus, the Gaussian generated variables will have a larger impact on the generated response variables than the finite-valued variables.
For our two experiments, we refer to the simulated data as the confidential data since that is what it represents in practice. Many algorithm parameters are fixed across these two experiments. When comparing the methods, we use the bin parameter $\zeta=2/3$ for the MV Histogram algorithm and the internal MV Histogram steps of the model-informed algorithms. In the GenModel method, 50% of the privacy budget is allocated to sanitizing the covariates. To sanitize the residual variance 5% of the budget is allocated to sanitizing the residual variance and split evenly among the steps. The standard error bound parameters are set as $\sigma_{min}=2^{-15}$ and $\sigma_{max}=2^{15}$. The treatment effect is sanitized using 15% of the privacy budget. The remaining privacy budget is split evenly to sanitize the remaining model coefficients. For each covariate coefficient, treatment effect, and variance parameter, the budget is split evenly between the DP Range algorithm of karwa2017finite (Algorithm (ref) in Appendix) and the infusion of Laplace noise. The range algorithm uses parameters $\upsilon=50$ and $\alpha_{range}=0.05$. There are $B=5000$ proxies generated in the GenModel Algorithm\footnote{The parameters selected are consistent with the real data example in Section (ref). A summary of the parameters for GenModel across the experiments and real data is in Appendix Table (ref).}.
To compare utility across privacy budgets, we simulate a dataset with 4 binary covariates and 4 continuous covariates. Let $T_i$ be an indicator that observation $i$ is assigned to the treatment group. We simulate one dataset using
, where $Z_i,X_{1,i},X_{3,i}\overset{iid}{\sim}N(0,1)$, $X_{4+k,i}$ for $k=1,\dots,4$ are binary variables, and $X_{2,i}$, $X_{4,i}$ are the rounded uniform variables.
We compare confidence interval overlap and absolute difference metrics across several privacy budgets. We consider $\epsilon\in \{0.5,~1,~2,~4,~5000\}$ under a pure DP framework $(\delta=0)$. With an extremely large privacy budget, such as $\epsilon=5000$, there are essentially no privacy guarantees. The algorithms simply become resampling and regenerating algorithms rather than sanitizing algorithms. When $\epsilon=4$, the GenModel privacy budget for the multivariate histogram step is $\epsilon_{\mathbf{X}}=2$. Similarly for $\epsilon=2$ the multivariate histogram step has a budget $\epsilon_{\mathbf{X}}=1$. By comparing the GenModel results with twice the privacy budget to the Hybrid results, we can see how much utility is lost when using sanitized model parameter estimates in the GenModel method rather than the confidential estimates in the Hybrid method. We perform 20 repetitions. Within each repetition, a sanitized dataset is generated using each privacy method, and metrics about the sanitized treatment effect are recorded. The original confidential data remains the same throughout all repetitions and privacy budgets. The CI overlaps and the treatment effect point estimates are then averaged across the 20 repetitions. The median absolute difference measure is reported; this gives one metric which is robust to outlier repetitions. From our resulting plots, we determineGenModel is prone to low-valued outliers for the sanitized treatment effect and thus the absolute difference metric.
The confidential estimated treatment effect is $\widehat{\tau}=5.2$ with estimated standard error $0.124$. The MV Histogram and Hybrid methods result in estimated standard errors between $0.112$ and $0.136$. However, the GenModel method results in estimated standard errors in a much wider range between $0.002$ and $0.484$. In Table (ref), we see that the Hybrid sanitized datasets generally have the highest average CI overlaps and the lowest median absolute difference for low privacy budgets. The MV Histogram sanitized datasets result in higher mean CI overlaps than the GenModel datasets, especially for larger privacy budgets. This is likely because some GenModel sanitized datasets have extremely small estimated standard errors produced by the original inference applied to the GenModel sanitized dataset. There are variants of GenModel we discussed in Section (ref) which could address the wide range of estimated standard errors but do not satisfy the DP framework. Future work could focus on adjusting the post-sanitization analysis to account for the sanitizing noise, or improving the sanitized estimate of residual variance. Since the widths of the confidence intervals make it difficult to compare the GenModel method to other methods, we can also consider the median absolute difference metric ($|\Delta|$).
The median absolute difference decreases as the privacy budget increases for the MV Histogram and GenModel methods. The mean absolute difference from the GenModel method is less than 0.25. The GenModel method with twice the privacy budget of the Hybrid method results in median $|\Delta|$ within $0.01$ of the the Hybrid method, with the exception of the $\epsilon_\mathbf{X}=0.5$ case. For $\epsilon=2$ and $\epsilon=4$, the GenModel method has a slightly lower median $|\Delta|$ than the MV Histogram. As shown in Figure (ref), the spread of the sanitized treatment effects is slightly larger for the GenModel method.
Overall, the three algorithms have mean estimated treatment effects near the confidential estimated treatment effect $\widehat{\tau}=5.20$. The GenModel method results in an average sanitized treatment effect closer to the true value of $\tau=5$ for $\epsilon\in \{5000,4,2\}$. Figure (ref) shows the distribution over the 20 repetitions of the sanitized treatment effects, $\widetilde{\tau}$, from the sanitized data from three methods for four $\epsilon$ values. The spread across the 20 repetitions of $\widetilde{\tau}$ from the GenModel data is much wider than the spread of $\widetilde{\tau}$ from the other two methods. The centers of the GenModel $\widetilde{\tau}$ for each privacy budget are closer to or equidistance from the true treatment effect, $\tau=5$ than the other two methods. The MV Histogram and Hybrid methods have sanitized treatment effects that are tightly focused around the confidential data's estimated treatment effect.
To compare utility across models with various number and types of covariate variables, we simulate a dataset with 10 binary covariates and 10 continuous covariates. We simulate one dataset using the following generating model:
where $Z_i,X_{2j-1,i}\overset{iid}{\sim}N(0,1)$ for $j=1,\ldots,5$; $X_{2j,i}$ for $j=1,\dots,5$ are the rounded uniform observations; and $X_{j,i}$ for $j=11,\dots,20$ are binary.
The models for analysis that are considered use the first $2$, $5$, or $10$ binary variables and continuous variables. From these, we consider nine models: one for each combination of possible number and types of variables. Specifically, the regression models are of the following form:
The model $j$ values for the number of binary covariates, $m_{cat}^{(j)}$, and the number of continuous covariates, $m_{cont}^{(j)}$, for $j=1,\dots,9$ can be found in Table (ref). We also consider two special case models. Both use five continuous covariates and the first two binary covariates. The Unbounded Model uses the five unbounded continuous covariates ($X_{1},X_{3},\dots,X_{9}$) and the Finite Set Model uses the five continuous covariates that take a finite set of values ($X_{2},X_{4},\dots,X_{10}$).
The simulated data is generated using (ref), but most of the regression models for the analysis omit variables. Thus, we risk an omitted variable bias in the models Dunn_Smyth_2018. However, we check the regression assumptions for the models fitted with the confidential data (Figure (ref) in Appendix (ref)). Based on the plots, the regression assumptions are met and there is not evidence that the models have omitted variable bias.
Using a pure DP framework ($\delta=0$), we fix the privacy budget to be $\epsilon=1$ for MV Histogram (Algorithm (ref)) and Hybrid (Algorithm (ref)). We also compare sanitized datasets from the GenModel algorithm (Algorithm (ref)) at two privacy levels. One method uses the same overall budget $\epsilon=1$ and is denoted GenM-1 . The other, denoted GenM-2 , uses $\epsilon=2$. For GenM-2 the privacy allocation $\epsilon_{\mathbf{X}}=1$ is equal to privacy budget for the Hybrid and MV Histogram methods.
Similar, to the comparisons made for the privacy budgets, we average CI overlap, and estimated/sanitized treatment effect across 20 repetitions. We take the median of the absolute difference measures. These results for models with two binary covariates are shown in Table (ref). The other models are in Table (ref). Again, the estimated standard errors from the GenModel sanitized data are from a much larger range ($0.002$ to $0.472$) than other algorithms. The estimated standard error of the treatment effects for MV Histogram and Hybrid method range from $0.123$ to $0.219$. For the confidential data, these range from $0.136$ to $0.203$.
Other than Model 1, the average CI overlaps for the GenModel data are quite small. In Model 1, which has only four covariates, the GenM-2 average CI overlap is only $0.1$ less than that of the MV Histogram. The CI overlaps for GenModel are often near $0$ or near $0.5$ as shown in Figure (ref). The MV Histogram has a large spread of CI overlap values, especially for models with 5 or less binary variables (Figure (ref). Over all the models, the Hybrid method has the highest average CI overlap compared to the other sanitization method and often has the lowest median absolute difference. In Table (ref)\footnote{Since each model is only fit once with the confidential data, the average $\hat{\tau}$, average $\widetilde{{O}}$, and median $|\Delta|$ reported in Tables (ref) and (ref) are the fitted model results rather than summarized results across repetitions.}, GenM-2 results in a median absolute difference better than or within $0.1$ of the MV Histogram. This is also true of Model 4 in Table (ref). As expected from higher privacy budgets, GenM-2 has better utility than GenM-1 when using the median $|\Delta|$ metric. It is better when using the average $\widetilde{{O}}$ metric for all models but Model 9. For Model 1, Unbounded Model, and Model 4, the median absolute difference of GenM-1 is better than the MV Histogram which has the same overall budget.
The GenModel performs particularly poorly in our utility metrics for models with ten binary covariates (Table (ref)). The privacy budget has to be split to sanitize each covariate coefficient, thus increasing the number of covariates increases the amount of noise infused to the sanitized coefficients. GenM-1 has its worst utility for Model 8 (average $\widetilde{{O}}=0.15$ and median $|\Delta|=0.74$). GenM-2 has its worst utility in Model 9 (average $\widetilde{{O}}=0.15$ and median $|\Delta|=0.44$) For a fixed number of total covariates, we find GenModel has lower median absolute difference and higher average CI overlap when the model has more continuous covariates than binary covariates. For example, GenM-1 has average $\widetilde{{O}}=0.21$ and median $|\Delta|=0.33$ and GenM-2 has average $\widetilde{{O}}=0.33$ and median $|\Delta|=0.18$ for Model 6 (10 continuous and 5 binary). For Model 8 (5 continuous and 10 binary), GenM-1 has average $\widetilde{{O}}=0.15$ and median $|\Delta|=0.74$ and GenM-2 has average $\widetilde{{O}}=0.28$ and median $|\Delta|=0.25$. The GenModel methods have better results in Model 3 than Model 7 and better results in Model 2 than Model 4. There is not a clear pattern when comparing models with the same number of covariates for the MV Histogram. This could be due to how the response variables were generated or something systematic in the algorithm. All methods did better on the Unbounded model than the Finite Set Model; this might be due to the Finite Set covariate having a smaller scale of impact of the response variable than the unbounded covariates in the generation of the response variables.
When we look at the distribution of the estimated treatment effects (Figure (ref)), the Hybrid method has the smallest spread of sanitized treatment effects. The spread of the $\widetilde{\tau}$ from the GenModel methods is much larger than the other sanitization methods. For GenM-1 the spread is so great, the plot should not share an axis with the other three methods (Figure (ref).
There are many outliers for the GenModel methods, GenM-1 even reports a negative sanitized treatment effect for one of its repetitions. Additionally, we see the increase in spread from Model 7 to 9 with the GenModel methods. The Hybrid method is the most concentrated around the confidential estimated treatment effects. For the first five models, the MV Histogram has a similar spread on the center 50% of repetitions of sanitized treatment effects as the GenM-2 method, but the MV Histogram doesn't have outliers.
Our two experiments described in Sections (ref) and (ref) compare the three proposed sanitization methods across various privacy budgets and regression models. The Hybrid method consistently out preforms the MV Histogram and the GenModel . However, if a researcher wants to provide confidentiality protection to the estimated model parameters, then the MV Histogram or GenModel methods are also reasonable options. The MV Histogram does well when there is a large number of covariates.
The GenModel method performs well with a slightly higher privacy budget and a low number of covariates. However, the estimated standard errors of treatment effects from inference on GenModel sanitized data can vary wildly. This makes GenModel results hard to evaluate with the CI overlap utility metric, but the utility can be seen in the absolute difference metric. There are potential variants discussed in (ref) which could address the standard error estimate issues. However, those variants use the residual standard error from the model fit on the confidential data and reduce the confidentiality protections. We did not implement these variants since the Hybrid method is a much simplified model-informed sanitization method than these variants. It could be argued these variants provide more privacy protection since they only depend on the confidential residual error rather than all of the fitted model coefficients and residual standard error from the confidential data. However, it is unclear how to measure the difference in privacy protections between the GenModel variants and the Hybrid method. Further work should be done to improve the implementation and analysis of the GenModel method. Even with these improvements, the GenModel method should be used for datasets with a limited number of covariates.
\color{red}
\color{black} In this section, we apply our proposed methodos (Algorithms (ref), (ref), and (ref)) on a real-world randomized control trial and evaluate their performance, focusing on the analyses as reported in [BJS] Reducecrime. The associated replication files, including the de-identified data, are available in ReducecrimePkg.
For our evaluation, we focus on the results reported in Table 2, Panel B of BJS. Specifically, BJS consider the long-term (12-13 months after the program) \footnote{Formally, we use Round 5 data, as the original code ReducecrimePkg specifies.} effect of therapy and cash grant on eight outcome variables related to antisocial behavior, individually and through one a summary index called {\it Antisocial behaviors z-score} (referred to as fam_asb_lt). The seven individual outcome variables shown in Table (ref) include indicators for anti-social behaviors like selling drugs, standardized specific behavior index values like an index of domestic verbal abuse, and the count of recent stealing activities. The sample is composed of $947$ high-risk youths in Monrovia, Liberia\footnote{One observation of the original data reported $0.1$ for the value for the number of women the respondent was supporting. This entry is changed to be $1$.}. A $2 \times 2$ factorial design is used with two stratification variables based on the groups the youths were in when they were randomly assigned treatments, once at the time of being assigned to therapy (there were 55 such groups), and once at the time of being assigned to receive a cash grant of $200$ USD (there were 20 such groups). BJS find that neither cash nor therapy alone has a lasting (12-13 month) effect, but that the combination of both treatments does reduce “antisocial behavior”.
The analysis data are obtained from the file named STYL_Final.dta as provided in the replication package ReducecrimePkg. The treatment assignments are encoded using three binary treatment variables tpassonly (indicating that only therapy was received), cashassonly (indicating that a cash-only grant was received), and tpcashass (indicating that both therapy and cash grant were received). The blocking variable based on therapy assignment is tp_strata_alt while the blocking variable based on cash grant is cg_strata. Fifty-five baseline covariates measured at the beginning of the study period are included in the analysis. The binary baseline covariates and the continuous ones are detailed in Tables (ref) and Table (ref) in the Appendix (ref).
Even though, fifteen baseline covariates and three response variables\footnote{The response variables are averaged over two months per respondent.} are indicator variables, the analysis preformed by BJS treat all variables as continuous. We mirror this choice so our results reflect what BJS could have done to release a privacy-preserving replication package. However, it is important to note that the original analysis reports results for fitted models which do not meet the necessary model assumptions. Without the model assumptions the validity of the model to capture and explain the sample and population data is questionable. Specifically, multiple regression models assume the residuals are normally distributed with equal variance (i.e., homoscedastic) and mean $0$. This is often checked with a QQ-Plot for normality and by plotting the residuals against the fitted values. We recreate the analysis as described in ReducecrimePkg. Using a QQ-Plot for normality and a plot of the residuals over the fitted values as diagnostic tools, the regressions assumptions are not met in seven out of eight models (Figure (ref) in Appendix). All response variables, except the standardized aggressive behavior index (abshostilstd_ltav), result in residuals that deviate from the normal distribution on the upper tail. The spread of the residuals increase as the fitted value increase for the models with fam_asb_it, stealnb_ltav, disputes_all_z_ltav, and domabus_z_ltav response variables. There are clear banded trends in the residuals over the fitted values for the models with drugssellever_ltav, carryweapon_ltav, arrested_ltav, and domabuse_z_ltav.The residuals for the model with the standardized index of domestic abuse response (\texttt{domabuse_z_ltav}) do follow the normal distribution, and the the residual variance appears reasonably constant over the fitted values. These concerns with the model fit for seven out of eight models may have adverse effects on the utility of our model-informed algorithms: the Hybrid (Algorithm (ref)) and GenModel (Algorithm (ref)) algorithms. The model-informed algorithms assume the model accurately represents the data generating model. The GenModel method also heavily relies on the residual distribution being Gaussian.
In addition to the analysis with the full 55 baseline covariates predictors, we considered analysis with a subset of ten covariates, and a reduced subset of seven variables to better assess computational feasibility. The reduced subset includes: age_b, asbhostil_b, drugssellever_b, drinkboozeself_b, druggrassself_b, harddrugsever_b, and \texttt{steals_b}. The first two covariates are age and baseline standardized antisocial behavior index for the individuals participating in the study. The remaining covariates record the baseline antisocial behavior of the youths in terms of whether they: have ever sold drugs; drink alcohol; smoke grass/opium; have ever consumed hard drugs; and have exhibited stealing behavior in the two weeks prior to their baseline interview, respectively. The covariate variables in the reduced subset are shown in the dark blue cells of Appendix Tables (ref) and (ref). In addition to the reduced subset variables, the subset of the covariates includes three other variables: $\texttt{school\_b}$, $\texttt{wealth\_indexstd\_b}$, and $\texttt{cognitive\_score\_b}$. These are an individual's years of schooling, baseline standardized index of wealth, and baseline standardized index of cognitive function, respectively (light blue cells in Appendix Table (ref)). In terms of the number of predictor variables, the full set has 60 since there are three treatment variables and two blocking variables. The subset has 15 and the reduced subset has 12. The models have 61, 16, and 13 coefficients including the intercept that the GenModel algorithm will have to sanitize.
The original analysis by BJS reports the estimated treatment effects, which they denote as ITT, robust standard errors, and adjusted p-values for the cash only, therapy only, and both cash and therapy treatment levels. With the original data, we find similar results with the full baseline covariate set, the 15 predictor subset, and the 12 predictor reduced subset as shown in Appendix Table (ref). At the significance level of $\alpha=0.05$, the therapy only and cash only treatments do not have any significant treatment effects for any response variables across the three covariate sets. The treatment with both therapy and cash is significant for the summary index (fam_asb_lt) and the aggressive behavior index (asbhostilstd_ltav). Additionally the estimated treatment effects for the subset and reduced subset are within $0.02$ of the treatment effect from the full baseline covariate set for both the fam_asb_lt ($\widehat{\tau}_{both}=-0.26$) and the asbhostilstd_ltav ($\widehat{\tau}_{both}=-0.31$) responses.
To compare the analyses from the original dataset and the privacy-preserving datasets, we use the confidence interval (CI) overlap (Equation (ref)) and absolute difference from Section (ref). We also look at the adjusted p-values from the privacy-preserving datasets to see if similar conclusions would be drawn compared to the original study by BJS\footnote{BJS use a Westfall-Young correction to adjust the p-values. For the anti-social summary index, the p-values are adjusted for nine comparisons to correspond to Figure 1 in their paper. For the individual responses, the p-values are adjusted for 21 comparisons (three treatment levels and seven individual response variables). Our analysis uses a Bonferroni correction and adjusts the summary index p-values for three comparisons (three treatment levels) and the individual p-values for 21 comparisons. We will use a significance level of $\alpha=0.05$ throughout our analysis.}.
For all three covariate sets, we compare the MV Histogram algorithm (Algorithm (ref)), the Hybrid algorithm (Algorithm (ref)), and GenModel algorithm ((ref)). We generate five privacy-preserving datasets: one from MV Histogram, one from Hybrid, and three from GenModel at three privacy levels.
All our privacy method applications set $\delta=0$ and the bin parameter $\zeta=2/3$. For MV Histogram and Hybrid methods the privacy parameter is $\epsilon=1$. We implement the GenModel method at three privacy levels $\epsilon\in \{1,2,9\}$. With privacy budget $\epsilon=1$, the privacy allocation to the covariate data is $\epsilon_{\boldsymbol{\gamma}}=0.5$ and the privacy-preserving dataset is denoted GenM-1 . The two synthetic datasets made with the GenModel algorithm with $\epsilon=2$ and $\epsilon=9$ are denoted GenM-2 and GenM-9 respectively. In both cases, the MV Histogram step to sanitize the covariate data has the same privacy budget allocated as the Hybrid and MV Histogram algorithms ($\epsilon_{\boldsymbol{\gamma}}=1$). For all three applications of the GenModel algorithm, after the multivariate histogram step, the remaining budget is split evenly over the eight response variables ($\epsilon_{\mathbf{y}}=\frac{{\epsilon}-\epsilon_{\boldsymbol{\gamma}}}{8}$). Within a response, each treatment coefficient and the standard error are allocated 10% of the response-level budget ($\epsilon_{var,\sigma^2}=\epsilon_{range,sigma^2}=\epsilon_{\sigma^1}=\frac{\epsilon-\epsilon_{\boldsymbol{\gamma}}}{240}$ and $\epsilon_{range,\tau,k}=\epsilon_{\tau,k}=\frac{\epsilon-\epsilon_{\boldsymbol{\gamma}}}{160}$ for $k=1,2,3$). The remaining budget is split evenly among the covariate coefficients ($\epsilon_{range,\gamma,\ell}=\epsilon_{\gamma,\ell}=\frac{3(\epsilon-\epsilon_{\boldsymbol{\gamma}})}{80*(p+1)}$ for $\ell=1,\ldots,p$). We use 5,000 iterations to generate proxy response variables. The standard deviation bounds for sanitizing the residual variance for each model are $(2^{-15},2^{15})$. All models use $\alpha=0.05$ and $\upsilon=50$ for the parameters to sanitize the range of each coefficient in the model (Algorithm (ref)) These parameter choices correspond to those in the simulated experiments in Section (ref). The allocation of the privacy budget and the specification of other parameters are summarized in Appendix Table (ref).
We report the privacy-preserving treatment effect statistics for each of the sanitizing algorithms, covariate sets, and model response and the original data results in the Appendix (Tables (ref) to (ref)).
In this section we will focus on only four response variables: fam_asb_lt, stealnb_ltav, carryweapon_ltav, and asbhostilstd_ltav. The anti-social behavior summary index is included because it is a main focus of the analysis done by BJS. The aggressive behavior index (asbhostilstd_ltav) is chosen since it satisfies the regression model assumptions with the original data. Then we select two responses which could be better modeled as discrete count data and a binary variable respectively: the count of stealing activities (stealnb_ltav) and the indicator of carrying a weapon (carryweapon_ltav). BJS treat these response variables as continuous, which is one reason why they do not satisfy the regression model assumptions.
From Section (ref), we expect Hybrid method to produce the highest utility. With 60 predictors, the GenModel is not likely to do well with the full baseline models. For the three responses where model assumptions are violated the utility of the privacy-preserving datasets from the Hybrid and GenModel model-informed algorithm's utility will suffer.
For the reduced subset of covariates and the fam_asb_lt response, the Hybrid synthetic data produce a privacy-preserving treatment effect close to the the original data treatment effect estimate, and both are significant at $\alpha=0.05$ level ($\widehat{\tau_{both}}=-0.25,~\widetilde{\tau}_{both,Hybrid}=-0.43$ from Table (ref)). The MV Histogram synthetic data results suggest that the only cash treatment increases anti-social behavior ($\widetilde{\tau}_{cash,MV}=0.34$ for fam_asb_lt and it is significant at $\alpha=0.05$). The Hybrid method also indicated the cash only treatment increases anti-social behavior ($\widetilde{\tau}_{cash,Hybrid}=0.24$ for fam_asb_lt and it is significant at $\alpha=0.05$). This contradicts the original data results. Almost all of the privacy-preserving treatment effects from GenModel synthetic data are significant with the reduced subset of covariates. This is likely due to the concerns about the estimated standard errors that we discussed in Section (ref). For GenM-1 and asbhostilstd_ltav, the privacy-preserving treatment effect for combined therapy and cash has the opposite sign of the original estimated treatment effects ( $\widehat{\tau_{both,GenM-1 }}=1.48$). The privacy budgets for this may be too low to handle eight responses and 12 predictors. For the models with stealnb_ltav and carryweapon_ltav as responses, the GenModel algorithm produces some extreme privacy-preserving treatment effects (e.g. $\widetilde{\tau}_{therapy,GenM-1 }=-45.28$ and $\widetilde{\tau}_{cash,GenM-9 }=-46.43$ for \texttt{carryweapon_ltav}). This is likely due to the poor fit of these models to the original data.
Much like our experimental results in Section (ref), the Hybrid synthetic dataset almost always has the highest CI overlap with the original estimated treatment effects (Figure (ref)). The MV Histogram generally performs better than the GenModel methods. The GenModel datasets have confidence interval overlaps that are $0$ or near $0.5$. The GenM-9 has similar or better results then the GenModel privacy-preserving datasets with lower privacy budgets. The Hybrid method once again low absolute differences between the privacy-preserving treatment effect and the original treatment effect (Figure (ref)). With a few exceptions, GenM-9 has similar or better absolute difference than the MV Histogram synthetic data. The absolute difference values tend to increase as the number of covariates increase for all of the methods because the MV histogram will discretize more variables which adds more noise.
Another concern with model-informed sanitization algorithms is how they influence diagnostic checks of the released data. For example, if BJS only released the synthetic data generated with the Hybrid or GenModel algorithms, a researcher replicating the study would be unaware of the issues with the model fit in the underlying original data (bottom plots in Figure (ref)). When a model is used to generate synthetic data and is fitted to the resulting synthetic data, the underlying assumptions of regression model will always meet by the synthetic data ones whether or not they were in the original data. The model-agnostic procedures such as the MV Histogram are more robust; they are likely to retain the data characteristics that result in the poor fit of the regression model. This can be seen in the diagnostic plots for the model of the indicator of carrying a weapon with the reduced covariate subset fitted to the original and three synthetic datasets in Figure (ref). The MV Histogram synthetic data captures the same issues about the model fit as our analysis of the original data, while the Hybrid and GenM-1 synthetic datasets suggest that the model fits well to the data.
To address Goal 3 outlined in Section (ref), we discuss the computational feasibility of our algorithms that are summarized in Table (ref). The simulations were run on personal laptop with a quadcore Intel Core i7-1165G7 and 16 GB of RAM. The Hybrid method has approximately consistent computation time across the covariate sets. With the removal of three covariates from the subset of covariates, generating the GenModel privacy-preserving data for the reduced subset of covariates is on average 2% faster. The MV Histogram takes twice as long to generate a synthetic dataset with the full covariate set with 60 predictors than the subset with 15 predictors. Comparing the full and subset times for the GenModel methods, each additional covariate takes on average $9.8$ additional seconds.
We can break down the privacy-preserving data generation times into smaller sub-steps for GenModel . As a model-informed algorithm, GenModel must be repeated to sanitize each response variable. However, the privacy-preserving covariates and treatments can be shared across these iterations. Table (ref) breaks down the computation times for sub-steps of the process of generating the privacy-preserving responses. In the GenModel algorithm, generating the proxy responses and then estimating the model parameters is the most time consuming sub-step. Sanitizing the model parameters is the second most time consuming step. Once the model parameters are protected, generating the privacy-preserving responses takes a couple seconds.
Overall, all three methods are computationally tractable on commodity computer hardware. The GenModel method is the most computationally taxing. This computational time will also likely increase with the number of iterations of proxies generated. Both the Hybrid and the MV Histogram algorithms are very quick to apply to the original dataset by BJS.
Much of the literature (in economics) publishes replication packages with either weakly protected (de-identified) data (contrary to our stated Goal 2), or withholds the data out of privacy concerns (impeding the ability for others to investigate inference, related to what we called Goal 1). The trigger for the analysis conducted here was the need for privacy protection in the presence of a requirement to publish data underlying an \ac{RCT}, while maintaining reasonably broad usability of the data. We start with a specific focus on economists and social scientists interested in replication of RCTs. We explore the use of one of the simplest DP mechanisms (histogram count perturbation with Laplace noise) for generation of the protected sensitive covariate data. We show that with very careful consideration of the type of data and the problem at hand, the original algorithm needs detailed tweaking, but that our proposed model-agnostic mechanism (MV Histogram) can produce a protected dataset (Goal 2) and that analyses from such data would sufficiently maintain precise inference (Goal 1). Even for low values of the privacy-loss budget (i.e., stronger privacy), we can obtain comparable estimates in our regression models of interest.
The GenModel method, one of the model-informed mechanisms, infuses noise to the estimated model parameters to provide additional confidentiality protections and has comparable utility to the multivariate histogram for larger privacy budgets and when the number of covariates is small for well-specified models like in Section (ref). This method uses privacy-preserving covariates from the MV Histogram method and then generates proxy fitted model coefficients in order to sanitize each model parameter. However, in Section (ref), we see when the model does not fit the data well or the model assumptions are not met, the GenModel preforms poorly. Additionally, the standard errors of the privacy-preserving treatment effect under the GenModel are so small that the $p$-values do not accurately reflect those from the original analysis. Further work should be done to adjust these for the infused noise.
The other model-informed mechanism, the Hybrid mechanism offers strong privacy protection at conventional levels of $\epsilon$ for covariate data, but does not infuse noise into the estimated model parameters. This yields much better utility than the GenModel mechanism. In the Hybrid mechanism, outcomes are imputed based on the privacy-preserving covariates and confidential parameters. In the real-world experiment, there are between 14 and 62 estimated parameters (point estimates and associated standard errors). The Hybrid mechanism works quite well to allow for reasonable inferences in both simulations and real-world examples, in general close to the original (unprotected) inferences on population (intent-to-treat) parameters. This is achieved by leveraging the targeted structure, focusing on a small number of parameters of interest. The mechanism allows for the release of the protected data and the perturbed parameters as part of a privacy-preserving replication package.
Publication of replication package is a requirement in an increasing number of social science journals. While we replicated the analysis from BJS, which includes a large number of covariates, there are mixed recommendations among the literature about whether covariates should be included, how they should be incorporated, and how many should be included, especially in clinical trial research covariate_nonlinear,covariate_increasepower,covariate_interactions,EU_covariate_guidelines. Our algorithms work best with a small number of covariates which aligns with EU_covariate_guidelines guidelines on the analysis of clinical trial data. However, the flexibility of our model allows for a variety of model structures. If interaction terms between treatments and covariates are included in the model as recommended by covariate_blocking and covariate_interactions, only a small adjustment to the model-informed algorithms is needed. In this case, the treatments should be sanitized with the covariates in the MV Histogram stage, instead of randomly assigned after the fact.
Some caveats apply. Firstly, both model-informed methods assume the model used to sanitize the data accurately portrays the underlying relationships between the variables. When the model fits poorly or the model assumptions are not met, this can lead to less utility in the privacy-preserving data. The GenModel is especially sensitive to violations of the normality assumption of the residuals since the theory behind the sanitization of the model parameters relies heavily on this assumption. When fitting the model used to produce the privacy-preserving data on the privacy-preserving output of the the Hybrid or GenModel mechanisms, the model assumptions will always be met by the fitted privacy-preserving regression. Thus those using the privacy-preserving data from a replication package must trust that model fit to the original data met the model assumptions. They will not be able to replicate checking those model assumptions. Additionally, the Hybrid mechanism can be extended to additional types of regression such as logistic regression and Poisson regression since it does not rely on the normality assumptions that the GenModel mechanism does.
Additionally, some major assumptions have been made in our implementation of the MV Histogram (Algorithm (ref)). Since the discretization used in Algorithm (ref) uses the observed range of the continuous variables, the range is not strictly protected under DP. We have treated the range of the observed values as not sensitive. When $\delta=0$, what combinations of categorical and discretized continuous values appear in the observed data is also treated as not sensitive. This is because Algorithm (ref) only adds noise to the proportions of unique values rows observed in the original dataset, rather than all possible combinations of the categorical and discretized continuous values. Thus the MV Histogram and by extension the GenModel method do not protect against all possible attacks. For an algorithm, $M$, to satisfy approximate differential privacy, $P(M(D_1)=\omega)\leq e^{\epsilon}P(M(D_2)=\omega)$ (Equation (ref)) for all possible outputs, $\omega$, and for all pairs of datasets, $D_1$,$D_2$ that differ on on record (in this case a row of participant information). Consider the case where there are 12 possible combinations of covariates and $D_1$ only has 10 of those unique combinations of covariates in its rows. If the output of the MV Histogram has 11 unique covariate combinations, an adversary knows it did not come from $D_1$. The MV Histogram algorithm satisfies Equation (ref) for some outputs $\omega$ and some pairs of datasets $D_1$ and $D_2$ given the following three conditions are met. First, we assume the bounds for the discretization of continuous variables as not sensitive. Second, $\delta$ must be greater than the ratio of two over the number of unique possible combinations of column values. This informs the choice of the binning parameter, $\zeta$. The third condition requires at least one privacy-preserving proportion to be above the initial threshold of $2\frac{\log(2/\delta)}{n\epsilon}+\frac{1}{n}$. This is dependent on parameters we set for the algorithm ($\epsilon$, $\delta$, $\zeta$), on the properties inherent to the study design and participant pool (e.g., $n$ and the distribution of participants across the covariate combinations), and the specific random draw when implementing the algorithm. With these three conditions, the MV Histogram and the model-informed algorithms satisfy the inequality that characterizes approximate DP, $P(M(D_1)=\omega)\leq e^\epsilon P(M(D_2)=\omega)+\delta$. However, since it only holds for specific pairs of neighboring datasets and depends on the randomly drawn noise of the mechanism, these methods do not strictly satisfy approximate DP.
Finally, how useful is having a privacy-protected replication package to address broader questions posed by other researchers? Does the privacy-protected replication package allow for robustness tests, for instance, through other specifications? The currently proposed mechanisms are tightly bound to the original authors' proposed specifications, and may not perform well when other researchers apply non-congenial specifications. In such cases, access to the original data and additional privacy budget may be necessary in order to create additional protected datasets that allow for such investigation. Transparency of sanitization, however, is increased, as the privacy-preserving mechanism would be released as part of the replication package. This could in turn better support adjustments of inference due to additional randomness present, and allowing for replication in other contexts, including informing new data collections.
We demonstrate that reasonably simple methods with strong privacy guarantees could become “out-of-the-box” methods. The ideas and results reported here are the first step towards better understanding of feasible privacy-preservation of \acp{RCT}-based data, ensuring that confidentiality of data contributors to \acp{RCT}, often from \ac{LMIC}, will be more strongly protected while maintaining the ability of researchers to draw meaningful inferences. While policy-oriented stakeholders are primarily interested in the latter, citizens that contribute their data to \acp{RCT} and companies, such as fin-tech providers, that provide key data to researchers are also heavily invested in protecting privacy. Consumer and citizen protection agencies, ethic review boards, and other regulators, should be interested in knowing of the existence of privacy-enhancing methods, possibly facilitating approval of studies in the presence of strong privacy guarantees.
This work was supported through Digital Credit Observatory (CEGA, University of California, Berkeley/Bill and Melinda Gates Foundation (MP)), and in-part by NSF awards SES-1853209 and CNS-1702760 to Penn State University. We would like to thank Luqi Emanuele and Shuhang Lou for their contributions towards this work and its extensions. We declare no known conflicts of interest.