Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
83,420 characters · 16 sections · 53 citation commands
Optimal Policy Choices Under Uncertainty
Policymakers often rely on empirical evidence to decide how to allocate resources across many different policies, including food stamp expansions, job training programs, tax rate changes, and more. However, empirical estimates of policy benefits and costs are often noisy, and it's unclear how resources should be allocated when faced with this statistical uncertainty. For example, one policy might have a larger estimated impact but also be estimated with substantial noise, while another might have a smaller estimated impact but be estimated more precisely. Should policymakers favor the policy with the larger estimated impact or the policy with less statistical uncertainty? Without a principled approach to managing this tradeoff, policymakers risk directing resources toward policies that only appear attractive due to estimation error. In this paper I address the question: How should policymakers make simultaneous changes to many different policies while accounting for statistical uncertainty about policy impacts?
This paper develops a framework for optimal policy choices that explicitly accounts for statistical uncertainty. I begin with a social planner who must choose upfront spending on a given menu of policies to maximize social welfare---defined as the weighted sum of individual money-metric utilities---subject to a future budget constraint. I incorporate statistical uncertainty by treating both true policy benefits and costs and their empirical estimates as random. To establish a welfare benchmark, I consider an oracle social planner who knows the distribution of true policy benefits and costs and maximizes expected social welfare. Standard arguments show that this problem is equivalent to one in which, for (almost) every realization of the empirical estimates, the oracle planner maximizes posterior expected social welfare using the distribution of true benefits and costs as the prior.
I then characterize the information required for optimal policy choices. Since empirical estimates are only informative about the welfare and budget impacts of changes close to the existing policy regime, I focus on a local analysis of the planner's problem, restricting attention to small changes in spending. I show that optimal local policy choices depend on the posterior expected gradient of the net welfare impact. This result delivers a simple sufficient statistic, in the spirit of the public economics literature, which a researcher can report to the planner: the posterior mean benefit and net cost of each policy. Given these posterior means, the planner can solve for optimal local policy without additional information about the underlying distribution of policy impacts.
In practice, the true distribution of benefits and costs across policies is unknown, making the oracle planner's decision infeasible. I propose an empirical Bayes approach to approximate the oracle planner by estimating the most likely prior from the data and using it to construct estimates of posterior mean benefits and net costs. The key idea is that the planner observes noisy estimates of benefits and costs for many policies. Taken together, these estimates are informative about the underlying distribution of true policy benefits and net costs. I use this estimated distribution to estimate posterior means and show that policy choices based on these posterior means approximate the oracle planner arbitrarily well as the number of policies grows.
Formally, I assume the observed estimates for policy benefits and costs are conditionally Gaussian and centered at their true values, consistent with widely-used hypothesis tests and standard error calculations. The true benefits and costs follow a multivariate location-scale model that depends on policy type, with residuals distributed according to a flexible prior unknown to the planner. I estimate this unknown prior nonparametrically from the data and combine it with location-scale parameter estimates to yield estimates of posterior mean benefits and net costs. I define the empirical Bayes local spending rule as the local spending rule that solves the planner's local problem using these estimated posterior means.
Existing results in the empirical Bayes literature show the estimated posterior mean benefits and costs converge in mean squared error to the true posterior mean benefits and costs. But for the planner, what matters is not mean squared error but the performance of the empirical Bayes local spending rule in the planner’s local problem. My main theoretical contribution is to derive finite-sample rates of convergence for two measures of the gap between the empirical Bayes approximation of the local problem and the oracle planner's local problem that are uniformly valid over a large class of data generating processes and policy environments. I also show that a sample plug-in approach, which solves the local problem with raw point estimates of benefit and net cost without adjusting for estimation uncertainty, fails to converge in important cases. Together, these results demonstrate that simply plugging in raw sample estimates leads to suboptimal decisions, while the empirical Bayes approach asymptotically matches the oracle planner's optimal performance.
As an intermediate step in proving these theoretical results, I derive upper bounds on mean squared error risk of the posterior mean estimates by extending the proof of Theorem 1 in chen2022empirical to the multivariate (two-dimensional) location-scale model, albeit for a discrete conditioning variable. This mean squared error result for the multivariate location-scale model is potentially of independent interest.
Finally, I illustrate the proposed method in an empirical application to 68 policies studied by hendren2020unified. The application demonstrates the implications of empirical Bayes shrinkage of benefit and net cost estimates for optimal policy choice. The central finding is that across several different local problem specifications, I find that the empirical Bayes local spending rule results in local increases to welfare, while the sample plug-in spending rule results in local decreases to welfare. These empirical results are consistent with the theoretical performance results established earlier in the paper and illustrate the practical relevance of carefully accounting for statistical uncertainty: properly incorporating estimation uncertainty into policymaking can qualitatively change the welfare implications of policy choices.
\paragraph{Related Literature} The decision problem in this paper builds upon the well-studied question in public economics of how to allocate spending across government policies to maximize welfare. I follow the large literature in public economics that models a planner who chooses local changes in spending to maximize welfare; recent examples include hendren2020unified, finkelstein2020welfare, and bergstrom2024optimal. I adopt a sufficient statistics approach to solving the local welfare maximization problem, in the spirit of an extensive literature in public economics that derives low-dimensional statistics that are sufficient to draw welfare conclusions gruber1997consumption, feldstein1999tax, saez2001using, chetty2008moral, chetty2009sufficient, schmieder2016effects, kleven2021sufficient. I depart from this literature by allowing for statistical uncertainty about policy impacts, then carefully setting up a tractable decision problem under statistical uncertainty that can be solved with sample estimates.
This paper also builds on a broad literature in statistical decision theory, which dates back to wald1949statistical and more recently the seminal paper of manski2004statistical, and the more recent literature on Empirical Welfare Maximization (EWM) kitagawa2018should, athey2021policy, mbakop2021model, sun2024empirical, opocher2025policy. The EWM literature typically studies how to optimally target a given policy using sample data (e.g., selecting the optimal group of individuals by their observable characteristics to receive a treatment). I instead study how to optimally make changes to many policies using noisy empirical estimates.
One paper in the EWM literature that considers statistical uncertainty when making many policy changes is chern2025policy. They propose a policy rule that explicitly trades off between the size of estimated welfare and the estimation uncertainty of welfare, which, as also discussed in andrews2025certified, is equivalent to maximizing worst-case performance over a confidence set for welfare. While I consider the same question of how to make changes to a set of policies based on noisy sample estimates of policy impacts, my approach is different and developed independently. In particular, I consider a planner who maximizes expected welfare, knowing policy impacts are drawn from some unknown distribution. I show that estimation uncertainty matters because it informs Bayesian updating even though the planner has no direct preference over estimation error (i.e., the planner is risk neutral in welfare space). Moreover, the empirical Bayes approach I propose allows for shrinkage by pooling together information across policies and attains bounds on expected regret as the number of policies grows, even if there is non-vanishing uncertainty for each policy estimate. In contrast, chern2025policy provide high probability bounds on regret that are attained as estimation error goes to zero.
The relative changes to upfront spending from the optimal local change to spending induce an implicit ranking of policies. Rankings with statistical uncertainty have been studied in the econometrics literature; mogstad2024inference propose a frequentist approach to inference on the ranks themselves, while andrews2024inference perform inference on the highest ranked outcome. Several papers have proposed empirical Bayes approaches to ranking under a decision-theoretic framework, including gu2023invidious and kline2024discrimination. Instead of focusing on discrete rankings, in my paper I allow for decisions to vary continuously in magnitude and direction, capturing the idea that policymakers choose not only whether to change a policy but by how much.
While empirical Bayes methods are typically used in economics for denoising or ranking, in this paper I apply them to a policymaking decision problem. Another paper that proposes empirical Bayes methods in a policymaking setting is yamin2025poverty, who studies the problem of how to allocate cash transfers to minimize poverty using noisy measures of income. That paper shows in theory and in simulations that a nonparametric empirical Bayes approach to allocating transfers outperforms a sample plug-in approach. Similarly, in this paper I prove that an empirical Bayes approach to decision-making can perform better than a sample plug-in approach in a related but distinct policymaking setting.
The literature in Bayesian statistical decision theory is deeply related to the large literature on empirical Bayes methods, which dates back to the seminal work of robbins1956empirical and has since been expanded by many researchers in various fields jiang2009general, efron2012large, koenker2014convex, jiang2020general, gu2023invidious, soloff2024multivariate, chen2022empirical. This paper specifically builds upon the nonparametric maximum likelihood approach, pioneered by kiefer1956consistency, for multivariate, heteroscedastic empirical Bayes, as developed by soloff2024multivariate. I extend results on mean squared error risk bounds in this literature to allow for a multivariate location-scale family of distributions and derive finite-sample rates of convergence directly for the social welfare measure of interest.
\paragraph{Outline} The rest of the paper proceeds as follows. In Section (ref), I characterize the optimal local change to spending for the oracle planner under statistical uncertainty, showing that posterior mean benefits and net costs are sufficient statistics. In Section (ref), I show that replacing these posterior means with raw sample estimates can lead to suboptimal decisions. I then propose an empirical Bayes approach that recovers the oracle planner’s performance asymptotically. Section (ref) applies the method to 68 policies studied by hendren2020unified and shows that accounting for statistical uncertainty through empirical Bayes shrinkage can reverse the welfare implications of policy changes. Section (ref) concludes.
Consider a social planner who can adjust a finite number of policies $j = 1, \dots, J$ through changes to upfront spending on each policy. Let $s_j$ denote the change in upfront spending on policy $j$ and let $s = (s_1, \dots, s_J)$ collect upfront spending changes on all policies. The planner wants to maximize social welfare subject to a budget constraint. Let $W(s)$ denote social welfare after upfront spending changes $s$ and $G(s)$ denote the present discounted value of the planner's long-run budget after upfront spending changes $s$. Then $W(0)$ and $G(0)$ are the welfare and budget, respectively, of the current policy regime. Note that $G(\cdot) > 0$ means that the planner is spending money in the long run, and $G(\cdot) < 0$ means the planner is bringing in money in the long run.
Without statistical uncertainty, the planner knows the exact welfare and the long-run budget after any given spending change and can therefore exactly enforce the budget constraint. In practice, only estimates of the welfare and budget impact of policy changes are available at the time of decision-making. I instead assume that the planner chooses $s$ knowing that, in some future period after the true welfare and budget impacts are realized, the budget constraint will be closed with certainty through a budget-closing policy. Let $\mu$, assumed to be known, denote the welfare impact per unit increase in budget from closing the budget. Then closing a budget shortfall of size $G$ yields a welfare impact of $-\mu G$. Note that by assuming $\mu$ to be known, I rule out the ability to close the budget constraint with any of the policies that are being learned about from sample estimates at the time of decision-making.
I first analyze the planner's decision problem without statistical uncertainty and defer the case with uncertainty to Section (ref). The planner chooses changes to spending $s$ to maximize the net welfare impact of $s$, that is, the sum of the direct welfare impact $W(s)$ and the indirect welfare impact from closing the budget $-\mu G(s)$:
Under this formulation of the planner's objective, an alternative interpretation for $\mu$ is the marginal value of relaxing the budget constraint from the Lagrangian formulation of maximizing welfare subject to closing the budget constraint.
Global maximization of welfare requires knowledge of the net welfare impact of any possible spending change, including those far from observed policies. This entails extrapolating empirical estimates beyond the observed policy regime, which in turn requires strong assumptions about how net welfare impact varies with $s$. To avoid such extrapolation, I consider the optimal local change to policy spending: that is, starting from the current policy regime, what is the best “small” change to policy spending? This local approach to welfare analysis is often taken in the sufficient statistics literature in public economics to address a similar tension between credibility of assumptions and ability to make welfare statements.\footnote{See chetty2009sufficient and kleven2021sufficient for recent reviews of the sufficient statistics approach in public economics.}
To formalize this idea, let $w(s) \equiv W(s) - \mu G(s)$ denote the net welfare impact of spending change $s$ and suppose $w(s)$ is continuously differentiable at $s = 0$. Let $V$ be a compact consideration set of feasible local directions of change in spending, $V \subseteq \mathbb{R}^J$, and for each scale $t>0$, define the scaled set $tV = \{tv:v \in V\}$. Restricting the global problem to small-scale changes in spending $s \in tV$ gives
where dividing by $t$ ensures net welfare is comparable across different spending scales $t$. The consideration set $V$ governs what is a “small-scale” change in spending; for example, a reasonable choice for $V$ could be all $v$ within a small distance of zero. The set may also encode real-world constraints faced by the planner, like political constraints to changing spending on certain policies.
As the scale $t \to 0$, the restricted global problem converges to
by definition of the directional derivative, where $\langle \cdot, \cdot \rangle$ denotes the dot product on $\mathbb{R}^J$ and $\nabla w$ denotes the gradient of $w$ at $0$.
I define the planner's local problem to be this $t \to 0$ limit of the restricted global problem. Also note that a first-order Taylor expansion of $w(s)$ around zero gives $w(s) \approx w(0) + \langle \nabla w, s \rangle$ for small $s$. This formalizes the idea that local problem approximates the global problem for small deviations from the status quo, that is, for consideration sets that are small neighborhoods around zero. The objective of the local problem is the instantaneous rate of increase in $w(s)$ along direction $v$,
The optimal local spending rule is the direction $v \in V$ that maximizes this rate of increase. The optimal local spending rule may not be unique if any of the components of $\nabla w$ are zero, but all optimal local spending rules result in the same maximal rate of increase in net welfare impact. Going forward, I restrict attention to local spending rules $v$ such that $v_j = 0$ whenever the $j$th component of $\nabla w$ is zero, which results in a unique optimal local spending rule.
Given any consideration set $V$, knowing the gradient $\nabla w$ is enough to characterize both the optimal local spending rule and the maximal rate of increase in net welfare impact. The sufficiency of the gradient to describe the optimal local spending rule is in the spirit of the sufficient statistics approach to welfare analysis, which provides low-dimensional statistics that are sufficient to make certain statements about welfare effects in various economic models.
Certain forms of the consideration set $V$ yield simple closed-form solutions. For example, if $V$ is equal to an $L^p$ unit ball for $p \in [1,\infty)$, $V = \mathcal B_p \equiv \{v \in \mathbb{R}^J: \Vert v \Vert_p \leq 1\}$, the dual norm gives
In Section (ref) I provide theoretical results for consideration sets $V \subseteq \mathcal B_p$.
I use the notation of hendren2020unified to re-express the gradient $\nabla w$ in terms of more familiar economic objects. Taking social welfare to be the weighted sum of individuals' money-metric utilities, the change in social welfare due to a marginal change in upfront spending on policy $j$ can be written as
Here $\eta_j$ is the average social marginal utility of income for policy $j$, which is the increase in social welfare from giving \$1 on average to the individuals impacted by policy $j$. I will also refer to $\eta_j$ as the welfare weight for policy $j$. The term $WTP_j$ is the sum of individuals' marginal willingness to pay for policy $j$, which I will call the benefit of policy $j$.
The change in long-run budget due to a marginal change in upfront spending on policy $j$ is
which I will call the net cost of policy $j$. With this notation, the gradient of the net welfare impact at zero spending is
As discussed in finkelstein2020welfare, estimates of $WTP_j$ and $G_j$ are available for many different policy changes. The Policy Impacts Library provides such estimates for over 149 policies in the United States. To ensure estimates of benefit and net cost are comparable across different policies, in this paper I normalize the size of a marginal change in upfront spending on policy $j$ to be one monetary unit of program cost. In practice this means that I divide estimates of the benefit and net cost of policies by the program cost.
In this paper I assume that $\eta_j$ is known ex-ante by the planner for each policy $j = 1,\dots,J$. This means that the uncertainty to be introduced in Section (ref) about the direct welfare impact of each policy comes from uncertainty about the benefit of each policy to its recipients, rather than uncertainty about welfare weights $\eta_j$. I make this assumption because empirical estimates in the literature primarily quantify the benefit and net cost of different policy changes, while $\eta_j$ captures in part the preferences of the planner, which are not as easily estimated.\footnote{There do exist methods to back out the social marginal utility of income across the income distribution, albeit without taking into account statistical uncertainty bourguignon2012tax, hendren2020measuring. However, for those approaches to be valid the planner must find the current tax schedule optimal, which is at odds with the premise of this paper that the planner wants to make changes to the current policy regime.}
In practice the true welfare and budget impacts from changes to spending are unknown to the planner. Instead, the planner observes sample estimates of the benefits and net costs, denote $\{(\widehat{WTP}_j, \widehat{G}_j)\}_{j=1}^J$, together with their covariance matrices $\{\Sigma_j\}_{j=1}^J$ from empirical studies of $J$ different policy changes. In this paper I model both the true welfare and budget impacts and the sample estimates as random. In particular, I first assume that the true impacts $\{(WTP_j, G_j)\}_{j=1}^J$ are jointly drawn from some distribution $F$. In this section I assume $F$ is known by the planner; in Section (ref) I present an empirical Bayes approach to proceed when $F$ is not known.
Motivated by the central limit theorem, I model the sample estimates as independent across policies and conditionally Gaussian with known covariance matrices:\footnote{As discussed in Section (ref), I normalize the sample estimates of benefit and net cost by program cost to ensure they are comparable across different policies. If program costs are observed without statistical uncertainty, the Gaussian distribution approximation motivated by the central limit theorem is reasonable.}
While unbiasedness and exact Gaussianity are not without loss, approximate normality and unbiasedness are already implicit in the reported standard errors and hypothesis tests in the literature. By conditioning on $\Sigma_j$ I take it to be fixed. In practice, following the empirical Bayes literature chen2022empirical, walters2024empirical, soloff2024multivariate, I will use consistent covariance matrix estimates from the empirical studies for $\Sigma_j$. I leave the problem of dealing with estimated covariance matrices and error in the normal approximation to future work.
I again analyze the planner's local problem. Consider the oracle social planner, who knows both the distribution of true policy impacts $F$ and the sample estimate model of (ref). In line with the aggregation theorem of harsanyi1955cardinal, the planner evaluates any candidate local direction of change in spending by the expectation of the planner's local objective. Here the expectation averages over both the randomness of the true policy impacts and the randomness of sample estimate noise. In this sense, the planner is risk neutral in welfare space, where here risk comes from uncertainty about the welfare and budget.
For notational simplicity let $Y_j \equiv (\widehat{WTP}_j, \widehat{G}_j)$. The planner solves for the optimal local spending rule with respect to consideration set $V$ by solving
where $\mathcal{Y} \subseteq \mathbb{R}^{2J}$ is the space of sample estimates and $Y_{1:J}$ collects sample estimates $Y_j$ for all $J$ policies. I write the spending rule $v$ as a map from $\mathcal{Y}$ to $V$ to emphasize that it can depend on the realization of the sample estimates because the planner observes sample estimates before choosing spending.
Note that by the law of iterated expectations,
Then the optimal local spending rule is the Bayes solution, which is the rule that maps each realization of the sample estimates $Y_{1:J} \in \mathcal{Y}$ to the vector $v \in V$ that (locally) maximizes the posterior expected net welfare impact given the realization of the sample estimates. The optimality of the Bayes solution for a decision-maker who maximizes such an expected objective is a standard result in Bayesian inference robert2007bayesian. Thus, using $\pi$ as shorthand for the posterior distribution of the true impacts $\{({WTP}_j, {G}_j)\}_{j=1}^J$ given sample estimates $Y_{1:J} = \{(\widehat{WTP}_j, \widehat{G}_j)\}_{j=1}^J$, the optimal local spending rule maps each $Y_{1:J}$ to the solution of the following problem:
where the final equality follows from linearity of the expectation operator.
\paragraph{Planner sufficient statistic} The sufficient statistic for the planner to describe the optimal local spending rule is now the posterior expected gradient,
in the sense that given $E_\pi[\nabla w]$, the planner who knows consideration set $V$ can solve for the optimal local spending rule. Because $\mu$ and the $\eta_j$'s are assumed known by the planner, the planner only additionally needs to know the posterior mean benefit and net cost of each policy to construct the posterior expected gradient. Importantly, thanks to the local focus, it suffices to know the welfare and budget impacts of individual marginal policy changes rather than the joint impacts of changing all policies simultaneously. Consequently, the local approach is directly compatible with the many empirical estimates of effects of individual policy changes.
\paragraph{Researcher sufficient statistic} A researcher who has compiled benefit and net cost estimates for a set of policies and would like to assist the planner in making optimal policy choices may not know the parameters of the oracle planner's local problem like $\mu$, welfare weights $\eta_j$, or the consideration set $V$. Crucially, the sufficiency of posterior mean benefits and net costs for the planner to construct the posterior expected gradient means that the researcher only needs to report to the planner the set of posterior mean benefits and net costs,
The planner---who knows $\mu$, the $\eta_j$'s, and $V$---can then use the posterior mean benefits and net costs to solve for the optimal local spending rule as described above.
I emphasize that posterior means depend on both sample estimates and uncertainty, where uncertainty arises from sample estimate noise and from the randomness of true policy impacts. Intuitively, posterior means shrink sample estimates towards the distribution of the true policy impacts, with more shrinkage for sample estimates that are noisier relative to the dispersion of the true policy impacts. This can be seen in, for example, Tweedie's formula for the posterior mean robbins1956empirical. Thus for the researcher to be able to calculate posterior means, at a minimum they must observe sample estimates together with some measure of sampling noise, like standard errors. In fact, even if the distribution of the true policy impacts were known, without observing standard errors, it is not possible to do shrinkage and thus optimal policy.
In practice, the researcher does not know the distribution of the true policy impacts and so cannot calculate posterior mean benefits and net costs. In Section (ref) I propose that the researcher instead report feasible empirical Bayes estimates of the posterior means to the planner, and show that the planner can approximate the optimal decision well using the empirical Bayes posterior mean estimates in place of posterior mean benefits and net costs.
\paragraph{Sufficient statistic for direction} For a wide range of policy-relevant consideration sets $V$, the researcher who does not know the planner's parameters $\mu$, $\{\eta_j\}$, or $V$ can report to the planner a simpler statistic, namely the set of ratios of posterior mean benefit to posterior mean net cost and the set of signs of the posterior mean net costs (or benefits). For the planner, who knows $\mu$, $\{\eta_j\}$, and $V$, those ratios and signs are sufficient to know whether the optimal local spending rule increases or decreases upfront spending on each policy for a subset of possible consideration sets $V$. In particular, suppose $V = c\mathcal{B}_p$ for any $p \geq 1$ and scalar $c > 0$, where recall $\mathcal{B}_p$ denotes the $L^p$ unit ball. Let $v^*$ denote the optimal local spending rule. Then if either $E_\pi[G_j] > 0$ and $\eta_j > 0$ or $E_\pi[G_j] < 0$ and $\eta_j < 0$,
If instead either $E_\pi[G_j] > 0$ and $\eta_j < 0$ or $E_\pi[G_j] < 0$ and $\eta_j > 0$, then
Because $\mu$ and the $\eta_j$s are known by the planner, given a consideration set $V$ that is an $L^p$ ball, the ratio $E_\pi[WTP_j]/E_\pi[G_j]$ together with the sign of $E_\pi[G_j]$ (or the sign of $E_\pi[WTP_j]$) is a sufficient statistic that a researcher can report to the planner for whether the optimal spending rule increases or decreases spending on policy $j$.
Throughout this paper I assume that the planner chooses changes to upfront spending $s_j$ on each policy $j$. One could instead imagine that the planner chooses net spending on policies, equivalently the change in budget due to policy changes, which takes into account fiscal externalities in addition to upfront spending. To understand how the problem with net spending as the choice variable is different, let $p_j$ denote the change in net spending on each policy $j$, which I collect into a vector $p = (p_1, \dots, p_J)$. For policy changes that are local to zero, I can approximate $p_j$ by $s_jG_j$ for each policy $j$.
The optimal local change to net spending can be summarized by the gradient of the net welfare impact with respect to the choice variable $p$ at zero. In the absence of statistical uncertainty, the problem with net spending as the choice variable is locally a reparameterization of the problem with upfront spending as the choice variable, using $p_j = s_jG_j$. So by the chain rule the gradient of net welfare impact with respect to $p$ at zero is
This formulation could be appealing because the gradient depends on the ratio of $WTP_j$ and $G_j$ for each policy $j$, which is exactly the marginal value of public funds (MVPF) for each policy $j$, as discussed in hendren2020unified.
With statistical uncertainty, there are several issues with taking net spending to be the choice variable. First, when net costs are noisily measured, choosing a target level of net spending is infeasible for the planner. This is because net spending $p_j$ depends in part on net cost $G_j$, which is unknown due to statistical uncertainty at the time of decision-making. Additionally, a small change in net spending may not correspond to a local policy change when the policy has net cost $G_j$ close to zero. Crucially, hendren2020unified estimate that some policies “pay for themselves” and thus have net costs that are close to or equal to zero. For such policies, a small change in net spending means a large change in upfront spending, that is, a large policy change. For such large changes, existing empirical estimates seem unlikely to provide useful guidance.
A further difficulty under uncertainty arises due to the irregular behavior of expectations of ratios. Locally the planner's optimal choice of net spending is summarized by the posterior expected gradient, which involves terms that are a posterior expectation of a ratio of noisy parameters, $E_\pi \left[ \frac{WTP_j}{G_j} \right]$. These expected ratios can be statistically ill-behaved, that is, the posterior expectation may be infinite or undefined. To provide intuition for why, note that if a random variable $X$ has positive and right-continuous density at 0, $E \left[\frac{1}{X} \right]$ is either infinite or does not exist. So when net spending is the choice variable for a set of policies, some of which pay for themselves, the expected gradient is likely to not be well-defined. In contrast, when the planner chooses upfront spending, the gradient is a linear combination of noisy parameters and so the expected gradient is well-defined.
These issues highlight that the formulation of the planner's problem becomes delicate in settings with statistical uncertainty and that working with upfront spending as the planner's choice variable yields better behavior than using net spending as the choice variable.
In the previous section I considered an oracle planner who forms a posterior over policy impacts $\{(WTP_j, G_j)\}_{j=1}^{J}$, taking as their prior the correctly specified distribution of true policy impacts. In practice, however, it may be the case that the planner does not know the true distribution of those policy impacts, and so is not able to construct posterior means and derive the optimal local spending rule as before. In this section I propose an empirical Bayes approach to approximate the optimal but infeasible local spending rule of the oracle planner. The empirical Bayes approach uses an estimate of the distribution of policy impacts, together with sample estimates and their standard errors, to produce shrunk posterior mean estimates.
To estimate the distribution of policy impacts, I assume $(WTP_j, G_j)$ are drawn independently from an unknown prior that varies with policy type, and observed sample estimates $\{(\widehat{WTP}_j, \widehat{G}_j)\}_{j=1}^{J}$ are conditionally Gaussian and unbiased. Despite the unbiasedness of the sample estimates for the true policy impacts, I show in Section (ref) that a sample plug-in approach---which solves for the optimal local spending rule using a sample plug-in version of the gradient---can perform poorly. This result is valid for any number of policies $J$, fixing the sampling noise of the sample estimates, and so is not resolved as the number of policies for which I observe sample estimates grows, unless the sample estimates grow arbitrarily precise. Thus, rather than simply plugging sample estimates into the planner's local problem, in this paper I develop an empirical Bayes approach to obtain asymptotically optimal decisions in the planner's local problem.
I use an adaptation of modern empirical Bayes methods to estimate the most-likely prior for $\{(WTP_j, G_j)\}_{j=1}^{J}$ under the proposed model for the observed sample estimates $\{(\widehat{WTP}_j, \widehat{G}_j)\}_{j=1}^{J}$. This estimated prior yields posterior mean estimates of benefit and net cost, which can be plugged into (ref) to obtain an estimate of the posterior expected gradient. The empirical Bayes local spending rule then solves the local problem of (ref) with this estimated posterior expected gradient. In Section (ref) I provide theoretical convergence results showing that the empirical Bayes approach approximates the oracle planner's local problem arbitrarily well as the number of policies grows.
I assume the true values of benefit and cost $(WTP_j, G_j)$ are independent random vectors across policies $j$. Benefits and net costs may systematically differ based on type of policy; for example, hendren2020unified find that policies targeting children have systematically higher returns than policies targeting adults. To account for this I assume that $(WTP_j, G_j)$ is drawn from a distribution that is shifted and scaled according to the type of policy $j$, similar to the conditional location-scale model proposed in chen2022empirical. I let $X_j$ denote the type of policy $j$ and assume there are $T$ different types of policies, $X_j \in \{1,\dots,T\}$.
The planner observes estimates $(\widehat{WTP}_j, \widehat{G}_j)$ of the benefit and net cost of each policy change $j$ from empirical studies, together with their covariance matrix $\Sigma_j$. Recall that I model estimates as independent and conditionally Gaussian in (ref). Going forward, I extend this assumption to hold conditional on policy type $X_j$.
For a distribution $F_0$ normalized to have zero mean and identity covariance matrix, I assume the following model for $(WTP_j, G_j)$ given policy type $X_j = t$:
Here the residuals $\tau_j \in \mathbb{R}^2$ are distributed according to common distribution $F_0$. The nonrandom vector $\alpha_t \in \mathbb{R}^2$ shifts this distribution and the nonrandom scale matrix $\Omega_t$ scales this distribution. By conditioning on $X_j$ and $\Sigma_j$ I take them to be known and fixed. In this paper I maintain that policies are independent and that policy types are correctly specified; I leave the problem of dealing with dependent policies and with misspecified policy types to future work.
For the theoretical results in this paper I impose assumptions on the data-generating process that I argue are economically reasonable. I first impose the following assumption, which uniformly bounds the residual term for policy benefit and net cost. This assumption is reasonable if one believes that, for example, no single policy change has an impact on welfare or budget per unit of upfront spending as large as GDP.
I additionally impose an assumption on the social planner's preferences, which uniformly bounds the marginal welfare impact of closing the budget constraint $\mu$ and the welfare weight $\eta_j$ for each policy $j$ away from infinity. This assumption is reasonable if one thinks the planner's preferences are represented by finite Pareto weights for each individual in society.
Finally, I impose the following regularity assumption:
Assumption (ref) requires that the eigenvalues of the scale parameters are uniformly bounded away from zero and infinity.
Under the model (ref) and (ref), the sample estimates $\widehat{WTP}_j$ and $\widehat{G}_j$ are unbiased for the true benefit and net cost $WTP_j$ and $G_j$ for each $j$. One might think that a sample plug-in approach---which solves the planner's local problem using a gradient constructed from using raw sample estimates in place of posterior means in (ref)---would perform well because of this unbiasedness. In fact, for a fixed number of policies $J$, the sample plug-in local objective is consistent if the empirical estimates $(\widehat{WTP}_j,\widehat{G}_j)$ are consistent for all policies $j$. Usually consistency of the empirical estimates holds when the sample size of each empirical study that produces the estimates grows to infinity.
In this paper I consider a different thought experiment where the sample size of each empirical study is fixed and the number of policies $J$ grows. This scenario better approximates the problem of a policymaker who has access to noisy empirical estimates for many policies, but limited precision for each policy. In Proposition (ref) below I derive lower bounds on two different measures of the gap between the oracle planner's local problem and the sample plug-in local problem. The lower bounds do not converge to zero as the number of policies $J$ increases, implying that the sample plug-in local spending rule can perform poorly relative to the oracle planner's optimal local spending rule.
To analyze the local spending rule across different numbers of policies $J$, I need a sequence of consideration sets $\{V_J\}_{J \in \mathbb{N}}$. I further require suitable normalization of the planner's local objective so that the local problem is comparable across different $J$. Because the bounds I provide will be for sequences such that for all $J$, $V_J \subseteq \mathcal{B}_p \subseteq \mathbb{R}^J$ for some given $p \geq 1$, I normalize by the order of the largest possible local problem objective among directions in $\mathcal{B}_p \subseteq \mathbb{R}^J$. The following lemma characterizes the normalization factor.
The proof of this and all subsequent results are available in Supplemental Appendix (ref).
Let $\widehat{\nabla w}$ denote the sample plug-in gradient, constructed by plugging the sample estimates into the posterior expected gradient of (ref). Let $\hat v_J : \mathcal{Y} \to V_J$ denote the sample plug-in local spending rule, which, for every possible value of the sample estimates $Y_{1:J} \in \mathcal{Y}$, solves $\max_{v \in V} \langle \widehat{\nabla w}, v \rangle$. Let $N_p$ denote the normalization factor given $p$ derived in Lemma (ref), that is, $N_p = J^{\frac{p-1}{p}}$ for $p \in [1,\infty)$ and $N_p = J$ for $p = \infty$. In what follows all expectation and probability statements are conditional on $\Sigma_{1:J}$ and $X_{1:J}$, which I omit when unambiguous.
In Proposition (ref) below I derive finite-sample lower bounds on two different expressions. The first object,
measures whether the objective of the sample plug-in local problem is uniformly close to the true local objective. The second object,
measures how close to optimal the sample plug-in local spending rule is, as given by the difference between the local welfare improvement from the sample plug-in local spending rule and the maximal local welfare improvement.
The first result holds because it is possible to construct a data-generating process such that each component of the sample plug-in gradient falls outside of the support of each component of the posterior expected gradient with sufficiently high probability. For the second result, note that if the $j$th component of the posterior expected gradient is strictly positive but the $j$th component of the sample plug-in gradient is strictly negative (or vice versa), the sample plug-in local spending rule $\hat v_J$ changes spending in the opposite direction as the optimal local spending rule on policy $j$. The result then follows because it is possible to construct a data-generating process such that these sign mistakes happen with sufficiently high probability, as long as $p > 1$.
This result shows that in settings with limited information about the effect of each policy, using the sample plug-in rule can lead to suboptimal decisions. In contrast, I next show that an empirical Bayes approach can asymptotically match the performance of the oracle planner as the number of policies grows.
Motivated by the lack of convergence of the sample plug-in approach as the number of policies $J$ grows large, I instead propose an empirical Bayes approach. The empirical Bayes approach first estimates the unknown distribution of the true policy impacts and then approximates the oracle planner's local problem with estimates of the shrunk posterior means, obtained using the estimated distribution.
The first step is to estimate the unknown parameters in the model, which are the prior $F_0$, the location parameters $\alpha_0 \equiv (\alpha_1, \dots, \alpha_T)$, and the scale parameters $\Omega_0 \equiv (\Omega_1, \dots, \Omega_T)$. For notational simplicity recall $Y_j \equiv (\widehat{WTP}_j, \widehat{G}_j)'$ and let $\theta_j \equiv (WTP_j, G_j)'$. Let $E_J[ \cdot | X_j = t]$ and $Var_J[\cdot | X_j = t]$ denote the sample mean and sample variance, respectively, conditional on $X_j = t$. To estimate the location and scale parameters, notice that for each policy type $t$,
Thus location estimates $\widehat\alpha = (\widehat\alpha_1, \dots, \widehat\alpha_T)$ and scale estimates $\widehat\Omega = (\widehat\Omega_1, \dots, \widehat\Omega_T)$ can be constructed with the sample analog of these formulas,
In practice $\widehat\Omega_t$ may not be positive semi-definite due to estimation error, although it will always be symmetric. In the empirical illustration I censor the negative eigenvalues of each $\widehat\Omega_t$ to zero to produce positive semi-definite scale matrix estimates.
To estimate the unknown prior $F_0$ and obtain posterior mean estimates, I will first transform the model to remove the location-scale transformation. For each policy $j$ of type $X_j = t$ the model of (ref) and (ref) is equivalent to
which is an example of the multivariate heteroscedastic empirical Bayes model studied by soloff2024multivariate. I therefore implement the nonparametric maximum likelihood estimation (NPMLE) method from soloff2024multivariate to estimate $F_0$, but replacing the unknown $\alpha_t$ and $\Omega_t$ with estimates $\widehat\alpha_t$ and $\widehat\Omega_t$. The NPMLE $\widehat{F}_J$ is the estimate of $F_0$ that maximizes the log-likelihood of the transformed data $\widehat{Z}_j$ under the following model for policy $j$ of type $X_j = t$:
Specifically, for $\mathcal{P}(\mathbb{R}^2)$ the set of all probability distributions supported on $\mathbb{R}^2$ and $\varphi_{A}(\cdot)$ the density of a Gaussian random vector with mean 0 and covariance matrix $A$,
In practice I approximate the above maximization problem by replacing $\mathcal{P}(\mathbb{R}^2)$ with the collection of distributions supported on a finite grid koenker2014convex. soloff2024multivariate provide a Python package npeb to implement the NPMLE estimation procedure.
Using the estimated parameters from the previous subsection, I construct the empirical Bayes local spending rule to approximate the oracle planner's local spending rule. The oracle posterior means of benefit and net cost for policy $j$ of type $X_j = t$, which I denote $WTP_j^*$ and $G_j^*$, are
The expectation with respect to $F_0, \alpha_0$, and $\Omega_0$ emphasizes that these are the posterior means under the true common prior $F_0$ and true location-scale parameters $\alpha_0, \Omega_0$. The empirical Bayes posterior mean estimates, which I denote $\widehat{WTP}_j^*$ and $\widehat{G}_j^*$, are plug-in versions of the oracle posterior means, using estimates $\widehat{F}_J$, $\widehat\alpha$, and $\widehat\Omega$:
Here the subscript emphasizes that these are posterior means computed as if the true parameters were the estimated prior $\widehat F_J$ and estimated location-scale parameters $\widehat{\alpha}, \widehat\Omega$. Posterior mean estimates $E_{\widehat F_J, \widehat\alpha, \widehat\Omega}\left[ \tau_j \middle\vert \widehat{WTP}_j, \widehat{G}_j, X_j, \Sigma_j \right]$ can easily be calculated with the Python package npeb.
The oracle planner solves their local problem given a consideration set $V$:
The empirical Bayes local spending rule solves the oracle planner's local problem but plugs in empirical Bayes posterior mean estimates for the oracle posterior means. Specifically, the empirical Bayes local spending rule $\hat v^*: \mathcal{Y} \to V$ solves, for every possible value of the sample estimates $Y_{1:J} \in \mathcal{Y}$,
As discussed at the end of Section (ref), the empirical Bayes posterior mean estimates are a sufficient statistic for the oracle planner who knows $\mu$, welfare weights $\eta_j$, and consideration set $V$ to solve for the empirical Bayes local spending rule. Moreover, by the reasoning at the end of Section (ref), if the planner's consideration set is an $L^p$ ball, reporting the ratios of shrunk benefits to shrunk net costs together with the signs of the shrunk net costs suffices for the planner to know whether the empirical Bayes local spending rule increases or decreases spending on each policy.
How well does the empirical Bayes local spending rule perform relative to the optimal but infeasible local spending rule of an oracle planner? Results in existing studies, like soloff2024multivariate, suggest that the posterior mean estimates $\widehat{WTP}_j^*$ and $\widehat{G}_j^*$ will approximate the oracle posterior means $WTP_j^*$ and $G_j^*$ well on average over all policies $j$ under the mean squared error criterion as the number of policies $J$ grows. However, the criterion for a well-performing local spending rule is not the minimization of mean squared error but the maximization of net welfare impact achieved by solving the planner's local problem.
In this section I derive finite-sample rates of convergence for the gap between the oracle planner's local problem and the empirical Bayes approximation of the local problem. These rates characterize the asymptotic performance of the empirical Bayes local spending rule relative to the oracle local spending rule as the number of policies grows large. They are finite-sample in that they depend on the number of policies in the sample $J$ and are valid over a large class of data-generating processes and consideration sets. To derive the finite-sample rates, I need to impose additional assumptions on the estimators used to obtain the empirical Bayes local spending rule.
Assumption (ref) requires that the scale estimators respect the uniform bounds of Assumption (ref), that the number of policies of each type is proportional to the total number of policies $J$, that the eigenvalues of the normalized sample estimate variances are uniformly bounded, and that the location and scale estimators perform well. Assumption (ref) requires that the prior estimate is an approximate maximizer of the log-likelihood of the residualized data $\widehat{Z}_j$. These are regularity assumptions that are similar to those used in the literature, with Assumption (ref) similar to assumptions in chen2022empirical and Assumption (ref) satisfied by the NPMLE estimator proposed by soloff2024multivariate with appropriate choice of discretization rate\footnote{See Proposition 6 and Section 4.1.1 of soloff2024multivariate for a discussion of how to choose the discretization rate for the support of an approximate NPMLE so that statistical requirements like Assumption (ref) are satisfied.}. In the statement of the theorem I will assume, among other things, that $J \geq 7$, which is sufficient for $\kappa_J$ to be positive. Note that the number of policy types $T$ is taken to be fixed as the number of policies $J$ grows.
The above assumptions specify a class of prior distributions, location and scale estimators, and planner preference parameters that are governed by a set of hyperparameters, \\$\mathcal H = (s_w, \bar{s}_w, s_g, \bar{s}_g, M, \mu, k, \overline{k}, c, \overline c, m, \overline{m})$. The following rates are uniform over data-generating processes for a given $\mathcal H$. In what follows, I use the notation $x \lesssim_{\mathcal H} y$ to mean there exists some positive constant $C_{\mathcal H}$ that depends only on $\mathcal H$ such that $x \leq C_{\mathcal H} y$.
Given a sequence $\{V_J\}_{J \in \mathbb{N}}$ I define for each $J$ the empirical Bayes local spending rule $\hat{v}_J^*: \mathcal{Y} \to V_J$ as the solution to, for every possible value of the sample estimates $Y_{1:J} \in \mathcal{Y}$, $\max_{v \in V_J} \langle \widehat{\nabla w}^*, v \rangle$.
The theorem shows that with a large number of policies, the empirical Bayes approach approximates the oracle planner well as long as the sequence of consideration sets $V_J$ can be written as a subset of the unit $L^p$ ball for some $p$ strictly greater than 1. The empirical Bayes approach approximates the oracle planner well in two different ways: the objective of the empirical Bayes local problem uniformly approaches the true local objective in expectation (result 1), and the local welfare improvement from the empirical Bayes local spending rule approaches the maximal local welfare improvement in expectation (result 2).
To prove this theorem, I bound the left-hand side of results 1 and 2 above by a function of the mean squared error of the empirical Bayes posterior mean estimates $\widehat{WTP}_j^*$ and $\widehat{G}_j^*$. I then derive a finite-sample upper bound on the mean squared error, extending the proof of Theorem 1 in chen2022empirical to the multivariate setting with a discrete conditioning variable for the location-scale model. The proof of this mean squared error result, available in Supplemental Appendix (ref), may be of independent interest.
Note that for $p=1$ the upper bound rates go to infinity with $J$, so the theorem does not speak to how well the empirical Bayes approach performs when $p=1$. The intuition for why empirical Bayes can perform poorly when $p=1$ is that when $V_J = \mathcal{B}_1$, the optimal local spending rule only spends on the single policy with the largest posterior expected rate of increase in net welfare impact, while the empirical Bayes local spending rule spends on the single policy with the largest empirical Bayes estimated rate of increase in net welfare impact. However, empirical Bayes ensures performance guarantees on average across all policies but not for any individual policy efron2012large.
In this section I apply the empirical Bayes method proposed in the previous section to estimate optimal local spending rules for making many policy changes at once. I find that empirical Bayes shrinkage can have major consequences for welfare relative to a sample plug-in approach, which solves the local problem with raw point estimates of benefit and net cost instead of empirical Bayes posterior mean estimates. In particular, I estimate that the empirical Bayes approach results in increases to welfare, while the sample plug-in approach results in decreases to welfare.
I use a sample of benefit and net cost estimates for 68 different policies compiled by hendren2020unified, which is the set of all policies for which they report confidence intervals on both benefit and net cost estimates. Given the finding in hendren2020unified that policies targeting children---especially child education policies---are systematically different from policies targeting adults, in the illustration I take the number of policy types to be $T = 3$: non-education, child education, and adult education. This yields 20 non-education policies, 23 child education policies, and 25 adult education policies.
As discussed at the end of Section (ref), a researcher who has compiled sample estimates of benefit and net cost can report to the planner the posterior mean benefits and net costs, which are sufficient for the planner to solve for the optimal local spending rule. In practice, the researcher does not know the distribution of true policy impacts and must estimate the posterior means. In Section (ref) I walk through how to calculate empirical Bayes posterior mean estimates following the method of Section (ref). In Section (ref) I compare posterior mean estimates to sample estimates of benefit and net cost and discuss the consequences for optimal policy choice. I then present numerical welfare estimates showing that in this illustration, the empirical Bayes approach yields welfare gains, while the sample plug-in approach results in welfare losses.
Further details about the data and the implementation are in Supplemental Appendix (ref).
Before displaying results for all policies in the sample, I first walk through how to obtain empirical Bayes estimates for two different policies. The first is the Michigan college scholarship program Kalamazoo Promise Scholarship, for which hendren2020unified estimate a program cost-normalized WTP of 2.008 with imputed variance 0.410, and a program-cost normalized net cost of 1.039 with imputed variance 0.015. The second policy is the Hope and Lifetime Learners Tax Credits, for which hendren2020unified estimate a program cost-normalized WTP of $-42.82$ with imputed variance 1243.36, and a program cost-normalized net cost of $4.86$ with imputed variance 37.64.
To obtain empirical Bayes estimates of benefits and net costs, I follow the procedure described in Section (ref). In particular, I obtain location and scale estimates for each policy type and residualize the sample estimates against the location and scale estimates. I then perform nonparametric empirical Bayes shrinkage on the residuals, using Python package npeb from soloff2024multivariate. Finally I scale and shift the shrunk residuals back to obtain empirical Bayes posterior mean estimates.
Intuitively, empirical Bayes posterior mean estimates shrink sample estimates towards the estimated distribution of the true policy impacts. The amount of shrinkage is decreasing in the precision of the sample estimate and increasing in the extremeness of the sample estimate relative to the distribution of the true policy impacts. This pattern is clear for these two policies. For the Kalamazoo Promise Scholarship, the sample estimates are relatively precise and the empirical Bayes posterior means (WTP of 2.012, net cost of 1.000) are very close to the original estmates. In contrast, for the noisily estimated Hope and Lifetime Learners Tax Credits, the empirical Bayes posterior means (WTP of 1.13, net cost of $0.36$) are quite different from the sample estimates.
Posterior mean benefits and net costs are sufficient for the planner to solve their local problem, as discussed at the end of Section (ref). A researcher who knows the set of $J$ policies but is unsure about the planner's parameters (consideration set $V$, average social welfare weights $\eta_j$, or marginal welfare impact of closing the budget constraint $\mu$) can report empirical Bayes posterior mean estimates to the planner. Given those estimates, the planner can solve for the empirical Bayes local spending rule.
Furthermore, as discussed at the end of Section (ref), reporting the ratio of posterior mean benefit to posterior mean net cost together with the sign of the posterior mean net cost is sufficient for the planner to know whether to locally increase or decrease spending on each policy for a range of consideration sets. For the sake of illustration, assume that the consideration set $V$ is an $L^p$ ball and that $\eta_j = 1$ for all policies $j$. Then the planner should increase spending on a policy if that policy's ratio of posterior mean benefit to posterior mean net cost is greater than $\mu$ and the posterior mean net cost is positive, or if the ratio is less than $\mu$ and the posterior mean net cost is negative.
To understand what this means in practice, recall that $\mu$ is the marginal welfare impact of the budget-closing policy, meaning that $-\mu$ is the marginal welfare impact of closing a budget deficit. It is reasonable to assume $\mu$ is positive, since closing a budget deficit, whether through tax increases or spending cuts, generally reduces welfare. Furthermore, $\mu = 1$ means the welfare impact of closing the budget is one to one with the size of the budget, while $\mu > 1$ indicates that closing the budget is especially distortionary or costly for welfare, which may be the case for many budget-closing policies.
For the Kalamazoo Promise Scholarship, the set of $\mu$ for which the planner increases spending is similar whether the researcher reports the ratio of sample benefit to sample net cost (any $\mu$ less than 1.933) or the ratio of empirical Bayes posterior mean benefit to empirical Bayes posterior mean net cost (any $\mu$ less than 2.012). In contrast, for the noisily estimated Hope and Lifetime Learners Tax Credits, the set of $\mu$ for which the planner increases spending differs drastically: for the ratio of sample estimates the range of $\mu$ may be unreasonable in many settings (any $\mu$ less than $-8.81$), while for the ratio of empirical Bayes estimates, the range of $\mu$ may be reasonable for many settings (any $\mu$ less than 3.14). This contrast highlights how accounting for statistical uncertainty through empirical Bayes shrinkage can overturn the spending recommendations implied by raw sample estimates.
I first obtain empirical Bayes posterior mean estimates of benefit and net cost as described above for all policies in the sample. In Figure (ref) I plot shrunk empirical Bayes estimates versus sample estimates for WTP in panel (a) and net cost in panel (b). As previously discussed, a researcher who does not know the parameters of the planner's decision problem, namely $\mu, \eta_1, \dots, \eta_J,$ or $V$, only needs to report the shrunk posterior mean estimates of Figure (ref) to the planner. The planner, who knows the parameter values, can solve for the empirical Bayes local spending rule using the posterior mean estimates. In each panel I label the eight policies with the greatest amount of empirical Bayes shrinkage for WTP in panel (a) and for net cost in panel (b). Generally, these are the policies with the largest estimation error, though this also includes policies with extreme estimates relative to the estimated prior, which mechanically shrink more.
To highlight the consequences of these differences, I next show that the empirical Bayes and sample plug-in rules can result in welfare effects that are locally of the opposite sign. In Table (ref) I present estimates of the planner's local objective along the empirical Bayes local spending rule and along the sample plug-in local spending rule, averaged over 1000 coupled bootstrap draws, following the coupled bootstrap method discussed in Supplemental Appendix (ref). I derive estimates of the local objective for $\mu \in \{0.5, 3\}$ and consideration sets $V = \mathcal{B}_2$ and $\mathcal{B}_\infty$, setting welfare weights $\eta_j = 1$ for all policies. I emphasize that the validity of these estimates do not rely on the specific location-scale structure of the prior I assume in this paper, only on exact Gaussianity and unbiasedness of the sample estimates.
For all considered combinations of $V$ and $\mu$, the empirical Bayes local spending rule leads to a local increase in social welfare, while the sample plug-in local spending rule leads to a local decrease in social welfare. These striking differences generally arise because of policies that look costly with low benefit, but are noisily estimated. In Figure (ref), these are the policies that are left of the 45 degree line in panel (a) and right of the 45 degree line in panel (b). The sample plug-in rule generally makes large spending cuts on these policies for these values of $\mu$. On the other hand, the empirical Bayes rule takes into account the statistical uncertainty about those policies, moderating those spending cuts and avoiding large welfare losses driven by noise. This empirical finding that the sample plug-in rule performs poorly in cases where the empirical Bayes rule performs well is consistent with the theoretical results of Section (ref) and illustrates their practical importance.
In this paper I study how to make optimal policy changes when policy impacts are estimated with statistical noise. I set up a statistically well-behaved decision problem where the planner makes local changes to upfront spending on a set of policies to maximize net welfare impact. Under the assumption that the planner knows the distribution of policy impact sample estimates and the prior distribution of the true policy impacts, I characterize the optimal local spending rule, which locally maximizes posterior expected net welfare impact. A sufficient statistic for the optimal local spending rule is the posterior expected gradient of net welfare impact, which in turn depends on posterior mean benefits and net costs. A key implication is that a researcher who is unsure about the planner's specific parameters can report posterior mean benefits and net costs to the planner, who can use the posterior means to solve for the optimal local spending rule.
When the prior distribution of true impacts is unknown, I propose estimating it via NPMLE and obtaining an empirical Bayes local spending rule by solving the local problem with an estimated gradient that plugs in estimates of posterior mean benefit and net cost. I show that this empirical Bayes approach performs well, unlike a sample plug-in approach, by deriving finite-sample rates of convergence for two objects. These rates show that the empirical Bayes estimate of the local objective converges uniformly and that the local welfare improvement from the empirical Bayes rule converges to the maximal local welfare improvement. Taken together, these results imply that the empirical Bayes approach asymptotically matches the performance of the oracle planner as the number of policies grows. Finally, in an application to a set of 68 policies studied by hendren2020unified, I demonstrate that the plug-in approach can lead to welfare losses, while, in those same situations, careful incorporation of statistical uncertainty through empirical Bayes shrinkage yields welfare gains.
\FloatBarrier
\singlespacing