Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
89,254 characters · 20 sections · 57 citation commands
Empirical Welfare Maximization with Constraints
JEL classification codes: C14, C44, C52
When a welfare program induces varying benefits across individuals, and when resources are scarce, policymakers naturally want to prioritize eligibility to individuals who will benefit the most. Based on experimental data, cost-benefit analysis can inform policymakers on which subpopulations to prioritize, but these subpopulations might not align with any available eligibility policy such as an income threshold. kitagawa_who_2018 propose a statistical rule, Empirical Welfare Maximization (EWM), that can directly select an eligibility policy from a set of available policies based on the experimental data. For example, if available policies take the form of income thresholds, EWM considers the problem of maximizing the expected benefits in the population \[ \max_{t\leq\overline{t}}\ \ensuremath{\mathbb{E}}[\text{benefit}\cdot\mathbf{1}\{\text{income}\leq t\}] \] and approximates the optimal threshold based on benefits estimated from experimental data. Recent work has demonstrated that the EWM approach performs well across a broad range of data distributions. As the sample size grows, the average of benefits obtained under the eligibility policy selected by EWM converges to the highest attainable level, a property I refer to as as uniform asymptotic welfare efficiency.
I follow this line of work in focusing on settings where the eligibility policy for welfare programs must be determined ex ante and cannot be easily adjusted during implementation, such as Food Stamp and Medicaid in the United States.\footnote{There are also settings in which eligibility is implemented sequentially until the budget is exhausted, such as anti-poverty programs in Chile as analyzed by carneiro2019tackling. My analysis does not extend to those cases, as it does not account for the possibility that the benefits and costs may vary depending on the order in which individuals enroll.} In practice policymakers often face budget constraints, but only have imperfect information about whether a given eligibility policy satisfies the budget constraint. First, there may be imperfect take-up: eligible individuals might not participate in the welfare program, resulting in zero cost to the government, e.g. finkelstein_take-up_2019. Second, costs incurred by eligible individuals who participate in the welfare program may vary considerably, largely driven by individuals' different needs but also many other factors, e.g. finkelstein_adjusting_2017. Both considerations are hard to predict ex ante, implying that the potential cost of providing eligibility to any given individual is unknown at the time of designing the eligibility policy. Unobservability of the potential cost requires estimation based on experimental data, contributing to uncertainty in the budget estimate of a given eligibility policy.
For this empirically relevant setting where the budget needed to implement an eligibility policy involves an unknown cost, this paper introduces a new property of statistical rules, namely asymptotic feasibility. Policies are feasible if they satisfy the budget constraint in the target population; otherwise, they are referred to as infeasible. A statistical rule is asymptotically feasible if given a large enough experimental sample, the statistical rule is very likely to select feasible eligibility policies. While budget overruns are generally tolerated in countercyclical programs like unemployment insurance and Medicaid in the United States, in other settings, such as subsidized health and education programs in developing countries, a potential budget overrun due to an underestimated budget can be highly undesirable for policymakers, as securing additional funding may be difficult.\footnote{For example, international and national funding for malaria control has fallen short of what is estimated to be needed in recent years. Motivated by the fact that Kenya can afford to distribute bed net subsidies to only 50% of its target population in 2007, bhattacharya_inferring_2012 analyzed the constrained optimal allocation to increase take-up.} In this context, ensuring asymptotic feasibility is particularly desirable, especially for policymakers who are highly risk-averse to budget overruns.
This paper answers three questions in the current setting with unknown cost: is there any statistical rule that achieves uniformly good performance for a wide range of data distributions in terms of both welfare efficiency and feasibility, whether the obvious extension of the existing EWM statistical rule remains attractive, and what are some alternative statistical rules.
Firstly as a novel theoretical contribution, I quantify a class of reasonable data distributions that is particularly challenging for statistical rules. Specifically, no statistical rule can be uniformly welfare-efficient and feasible simultaneously over this class of data distributions. A notable example within this class is the expansion of welfare programs, such as tax credits to incentivize labor force participation, which can “pay for themselves,” as demonstrated in hendren_unified_2020, because they have zero net cost to the government. This impossibility result provides theoretical characterization of constrained settings that need to be ruled out for any statistical rule to achieve uniformity.
Second, I show the direct extension of the existing EWM statistical rule is not appealing in the setting with unknown cost. The reason is that this sample-analog rule ignores the estimation error in the estimated budget needed to implement a given policy, which has non-negligible consequences even when the sample size is large. For data distributions mentioned earlier where the budget constraint is exactly binding, the welfare loss does not vanish with sample size. The probability of selecting infeasible policies also does not vanish with sample size. Intuitively, this latter issue can be mitigated by using a slightly downward-biased version of the budget constraint, effectively making the budget estimate more conservative.\footnote{Similar intuition arises in optimal prediction under asymmetric loss christoffersen1997optimal, where positive prediction errors are more costly than negative ones, making a downward-biased predictor optimal.} I formalize this idea by setting the degree of conservativeness proportional to the standard error of the budget estimate, effectively selecting only policies with an upper confidence bound below the budget constraint. This modification to the sample-analog rule ensures that feasible policies are selected with high probability.
So far, the discussion assumes that even minor budget violations are unacceptable. However, in practice, exceeding the budget constraint may be desirable if the welfare gains outweigh the borrowing costs.\footnote{What happens when a welfare program goes over budget is highly context-specific, and there are various ways to model it. In this paper, I focus specifically on the scenario where the government borrows money to cover the excess cost while other models-such as those involving rationing as in kitagawa_who_2018 and poorer services are certainly possible. } To account for this situation, I introduce a new objective function that maximizes welfare gains but imposes some penalty once spending exceeds the budget, thereby giving the new objective function the intuitive interpretation of a trade-off function. I propose the trade-off rule, which optimizes the sample-analog version of the trade-off function. I show the trade-off rule is uniformly asymptotically welfare efficient.
To illustrate the trade-off rule, I apply it to data from the Oregon Health Insurance Experiment (OHIE) to select a more flexible Medicaid expansion policy than the current one. Medicaid is a government-sponsored health insurance program intended for the low-income population in the United States. The current Medicaid expansion policy determines eligibility solely based on household income. I examine whether health can be improved by allowing the income threshold to vary by the number of children in the household, setting the budget constraint equal to the cost of the current policy. Imposing a reasonable penalty for exceeding the budget in this context, that any overrun needs to be repaid in full, the trade-off rule selects an eligibility policy that expands eligibility for many households above the current threshold, especially those with children. This occurs because, based on the OHIE data, the additional health benefit from extending eligibility to these households outweigh the penalty of having to repay the overrun in full.
The rest of the paper proceeds as follows. Section 1.1 discusses related work in more detail. Section 2 presents theoretical results. Sections 3 and 4 discuss properties of two statistical rules, illustrated by an empirical example of designing a more flexible Medicaid expansion policy for the low-income population in Oregon. Section 5 conducts a simulation study and Section 6 concludes. Proofs, supporting lemmas, additional results and computational details can be found in the Appendix.
This paper is related to the traditional literature on cost-benefit analysis, e.g. dhailiwal_comparative_2013, and to the recent literature on EWM, e.g. kitagawa_who_2018, rai_statistical_2019, athey_policy_2021 and mbakop_model_2021. More broadly, this paper contributes to a growing literature on statistical rules in econometrics, including manski_statistical_2004, dehejia_program_2005, hirano_asymptotics_2009, stoye_minimax_2009, chamberlain_bayesian_2011, bhattacharya_inferring_2012, demirer_semi-parametric_2019, yata_optimal_2021 and kitagawa_treatment_2022, among others.
The traditional cost-benefit analysis compares the cost and benefit of a given welfare program. The effect of program eligibility is first estimated based on a randomized control trial (RCT), and then converted to a monetary benefit for calculating the cost-benefit ratio. For example, gelber_effects_2016 and heller_thinking_2017 compare the efficiency of various crime prevention programs based on their cost-benefit ratios. However, the cost-benefit ratio is only informative for whether this welfare program should be implemented with the fixed eligibility policy as implemented in the RCT.
The literature on statistical rules in econometrics has also developed a definition for optimality of statistical rules. manski_statistical_2004 considers the regret, defined to be loss in expected welfare achieved by the statistical rule relative to the welfare achieved by the theoretically optimal eligibility policy. In the absence of any constraint, under the theoretically optimal eligibility policy, anyone with positive benefit from the welfare program would be assigned with eligibility. The minimax regret rule minimizes the upper bound on the regret that results from not knowing the data distribution. stoye_minimax_2009 shows that with continuous covariates and no functional form restrictions on the set of policies, minimax regret does not converge to zero with the sample size because the theoretically optimal policy can be too difficult to approximate by a statistical rule. kitagawa_who_2018 avoid this issue by imposing functional form restrictions. They propose the EWM rule, which starts with functional form restrictions on the class of available policies, and then selects the policy with the highest estimated benefit (empirical welfare) based on an RCT sample. They prove the optimality of EWM in the sense that its regret converges to zero at the minimax rate. Importantly, the regret is defined to be loss in expected welfare relative to the maximum achievable welfare in the constrained class, which avoids the negative results of stoye_minimax_2009. athey_policy_2021 propose doubly-robust estimation of the average benefit, which leads to an optimal rule even with quasi-experimental data. mbakop_model_2021 propose a Penalized Welfare Maximization rule which relaxes restrictions of the policy class.
The existing EWM literature has not addressed budget constraints with an unknown cost. kitagawa_who_2018 consider a capacity constraint, which they enforce using random rationing. Random rationing is not ideal as it uses the limited resource less efficiently than accounting for the cost of providing the welfare program to an individual. When there is no restriction on the functional form of the eligibility policy, bhattacharya_inferring_2012 demonstrate that given a capacity constraint, the optimal eligibility policy is based on a threshold on the benefit of the welfare program to an individual. When the cost of providing the welfare program to an individual is heterogeneous, however, budget constraints can be more complicated than capacity constraints, and require estimation. carneiro_optimal_2020 considers the optimal choice of covariate collection in order to maximize the precision of the estimation for the average treatment effect. While they also consider a constrained decision problem, the budget constraint can be verified directly. They also allow for a more complicate trade-off between additional covariates and additional observations. sun_treatment_2021 propose a framework for estimating the optimal rule under a budget constraint when there is no functional form restriction. The main contribution of this paper is to characterize theoretical properties of statistical rules when allowing both functional form restrictions and budget constraints with an unknown cost.
As explained in a prior version of this paper sun2021empiricalwelfaremaximizationconstraints, the current setting of EWM with constraint shares the same mathematical structure as another important setting: fairness constraints across sensitive subgroups. viviano2024fair cast welfare of sensitive subgroups as multiple objective functions for policymakers, and solve the constrained optimal policy via the Pareto frontier. kock2024regularizing consider a penalized objective function that penalizes violations to the constraint. The trade-off rule proposed in this paper takes a similar form by penalizing budget overrun, but is tailored to linear objectives and constraints.
I begin by setting up a general constrained optimization problem, which depends on the following attributes of an individual:
Here $\tau$ is benefit from the treatment for the individual, $C$ is the cost to the policymaker of providing the individual with the treatment, and $X\in\mathcal{X}\subset\mathbb{R}^{p}$ denotes their $p$-dimensional characteristics. The individual belongs to a population that can be characterized by the joint distribution $P$ on the attributes $A$. The unknown distribution $P\in\mathcal{P}$ is from a class of distributions $\mathcal{P}$.
A policy $g(X)\in\{0,1\}$ determines the treatment status for an individual with observed characteristics $X$, where 1 is treatment and 0 is no treatment. Let $\mathcal{G}$ denote the class of policies policymakers can choose from. The optimization problem is to find a policy with maximal benefit while subject to a constraint on its cost:\footnote{Following kitagawa_who_2018, I implicitly assume the maximizer exists in $\mathcal{G}$ with the notation in ((ref)). }
If the policymaker does not have a fixed budget but still wants to account for cost, the scalar $\tau$ can be the difference in benefit and cost. I impose a harsh constraint at known $k$ to model a fixed budget.
The benefit-cost attributes $(\tau,C)$ of any given individual may be unobserved in practice. The focus of this paper is the setting where policymakers can construct their estimates $(\tau_{i}^{\ast},C_{i}^{\ast})$ in a random sample of sample size $n$ along with the characteristics $X_{i}$ from an experiment or quasi-experiment that satisfy Assumption (ref), as discussed later.
Applying the Law of Iterated Expectation, the constrained optimization problem ((ref)) can be written as
When the eligibility policy can be based on any characteristics whatsoever, the class of available policies is unrestricted i.e. $\mathcal{G}=2^{\mathcal{X}}$. In this unrestricted class, when the cost is non-negative, the above expression makes clear that the optimal eligibility policy is based on thresholding by the benefit-cost ratio $\ensuremath{\mathbb{E}}_{P}[\tau\mid X]/\ensuremath{\mathbb{E}}_{P}[C\mid X]$ where the numerator and the denominator are respectively the average effects conditional on the observed characteristics (CATE) and the conditional average resource required. Appendix (ref) provides a formal statement.
Given a random sample, to approximate the optimal eligibility policy $g_{P}^{\ast}$, one can estimate the benefit-cost ratio based on the estimated CATE and the estimated conditional average resource required. The resulting statistical rule selects eligibility policies that are thresholds based on the estimated benefit-cost ratio. The challenge is that the selected eligibility policy can be hard to implement when the estimated benefit-cost ratio is a complicated function of $X$. Restrictions on the policy class $\mathcal{G}$ address this issue. A common restriction is to consider thresholds based directly on $X$, e.g. assigning eligibility when an individual's income is below a certain value.
Restrictions on the policy class $\mathcal{G}$ mean that there might not be closed-form solutions to the population problem ((ref)). In particular, the constrained optimal eligibility policy $g_{P}^{\ast}$ might not be an explicit function of the CATE and the conditional average resource required. Therefore it might be difficult to directly approximate $g_{P}^{\ast}$ based on the estimated CATE and the estimated conditional average resource required. However, this is not an obstacle to deriving guarantees for the statistical rules. As I demonstrate later, the derivation does not require the knowledge of the functional form of the constrained optimal eligibility policy $g_{P}^{\ast}$.
I next specialize the constrained optimization problem to selecting eligibility policy for welfare programs with the example of Medicaid expansion. In the example of implementing welfare programs, policies take the form of eligibility policies. I restrict attention to non-randomized policies as in the leading example of welfare programs, deterministic policies such as income thresholds are more relevant. Theoretically oriented readers may proceed directly to Section (ref).
Suppose the government wants to implement some welfare program. The treatment in this example is eligibility for such welfare program. Due to a limited budget, the government cannot make eligibility universal and can only provide eligibility to a subpopulation. To use the budget efficiently, policymakers consider the constrained optimization problem ((ref)). In this example, the policy $g(X)$ assigns an individual to eligibility based on their observed characteristics $X$, and is usually referred to as an eligibility policy. I denote $\tau$ to be the benefit experienced by an individual after receiving eligibility for the welfare program. Specifically, let $(Y_{1},Y_{0})$ denote the potential outcomes that would have been observed if an individual were assigned with and without eligibility, respectively. The benefit from eligibility policy is therefore defined as $\tau\coloneqq Y_{1}-Y_{0}$. Note that maximizing benefit is equivalent to maximizing the outcomes (welfare) under the utilitarian social welfare function: $\ensuremath{\mathbb{E}}_{P}[Y_{1}\cdot g(X)+Y_{0}(1-g(X))]$. I denote $C$ to be the potential cost from providing an individual with eligibility for the welfare program. Both $\tau$ and $C$ are unobserved at the time of assignment and will need to be estimated.
Policymakers might be interested in multiple outcomes for an in-kind transfer program. hendren_unified_2020 capture benefits by the willingness to pay (WTP). Assuming eligible individuals make optimal choices across multiple outcomes, the envelope theorem allows policymakers to focus on benefit in terms of one particular outcome.
\paragraph{Medicaid Expansion }
Medicaid is a government-sponsored health insurance program intended for the low-income population in the United States. Up till 2011, many states provided Medicaid eligibility to able-bodied adults with income up to 100% of the federal poverty level. The 2011 Affordable Care Act (ACA) provided resources for states to expand Medicaid eligibility for all adults with income up to 138% of the federal poverty level starting in 2014.
Suppose policymakers want to maximize the health benefit of Medicaid by adopting a more flexible expansion policy. Specifically, they relax the uniform income threshold of 138% and allow the income thresholds to vary with the number of children in the household. Once the income thresholds are set, they must be codified in state legislation and publicly announced.\footnote{The legislated eligibility policy is publicly available on federal websites such as \href{https://www.macpac.gov/medicaid-101/eligibility/}{MACPAC}.} Therefore, in this example, it is reasonable to assume that the eligibility policy must be determined ex ante.
The policy class in this example includes income thresholds that can vary with the number of children in the household:
for characteristics $x=(\text{income},\ \text{numchild})$ and $\ \beta_{j}\geq0$.
Convincing policymakers to adopt a more flexible expansion policy as in (ref) may still require specifying a clear budget target to ensure that the new policy does not exceed the expenditure level of the current expansion with the uniform income threshold of 138%. Given that Medicaid is a countercyclical program and has some flexibility to accommodate budget overruns, later in Section (ref), I develop a new rule to allow explicit trade-off between welfare gains from additional spending and a penalty for budget overruns.
Correspondingly, the constrained optimization problem ((ref)) sets $\tau$ to be the health benefit from Medicaid, $C$ to be the cost to Medicaid, and the appropriate threshold $k$ to be the average cost to Medicaid under the current expansion policy with the uniform income threshold of 138%. The characteristics $X$ include both income and number of children in the household.
To simplify the notation, I define the welfare function and the budget function:
and the constrained optimal policy $g_{P}^{\ast}$ is therefore the solution to
As explained in Example (ref) from Section (ref), under a utilitarian social welfare function, maximizing the benefit with respect to eligibility policy is equivalent to maximizing the welfare, which is why I refer to $W(g;P)$ as the welfare function. The welfare function and the budget function are both deterministic functions from $\mathcal{G}\rightarrow\mathbb{R}$. The index by the distribution $P$ highlights that welfare and budget of policy $g$ vary with $P$, and in particular, whether a policy $g$ satisfies the budget constraint depends on which distribution $P$ is of interest.
When the benefit-cost attributes $(\tau,C)$ are unobserved and the distribution $P$ is unknown, both the welfare function and the budget function are unknown functions. Denote by $\widehat{g}$ a statistical rule that selects an eligibility policy after observing some experimental data of sample size $n$ distributed according to $P^{n}$. This section provides formal definitions for two desirable properties of $\widehat{g}$.
The above two properties build on the existing EWM literature. For the first property, the current EWM literature evaluates statistical rules by whether they attain at least $W(g_{P}^{\ast};P)$ in expectation over repeated sample draws as $n\rightarrow\infty$. Instead, I focus on convergence in probability, where the probability of $\widehat{g}$ selecting eligibility policies that achieve strictly lower welfare than $g_{P}^{\ast}$ approaches zero as $n\rightarrow\infty$. In the setting of the existing EWM literature, the constrained optimal policy $g_{P}^{\ast}$ is also the unconstrained optimal policy $\mathcal{G}$. Therefore it is impossible for any statistical rule $\widehat{g}$ to select a policy that achieves higher value than $g_{P}^{\ast}$. In my setting, however, the constrained optimal policy $g_{P}^{\ast}$ is not necessarily the unconstrained optimal policy. Thus, I allow the statistical rule $\widehat{g}$ to select a policy that achieves higher welfare than $g_{P}^{\ast}$ for all data distributions, albeit at the cost of violating the budget constraint.
The second property is new to the EWM literature. It imposes that given a large enough sample size, the statistical rule $\widehat{g}$ is unlikely to select infeasible eligibility policies that violate the budget constraint, so that it is “asymptotically feasible”. Asymptotic feasibility of statistical rules is specific to the current setting where the budget constraint involves unknown cost. Exactly satisfying a fixed budget constraint without the smallest violation is the most conservative way to articulate policy makers' preferences.
While both are desirable properties, the next section shows a negative result that it is impossible for a statistical rule to satisfy both properties when the data distribution is unknown and belongs to a sufficiently rich class of distributions $\mathcal{P}$.
Uniformity is a desirable property for a statistical rule, ensuring that its performance guarantees hold uniformly over a class of DGPs, thereby providing robustness to uncertainty about the true DGP. However, in this section, I prove an impossibility result that no statistical rule can be both uniformly asymptotically welfare-efficient and uniformly asymptotically feasible in a sufficiently rich class of distributions $\mathcal{P}$ described in the following Assumptions (ref)-(ref). The intuition is that there exists a sequence $P_{h_n}$ that converges to $P_0\in \mathcal{P}$, but along the sequence, their corresponding constrained optimum $g^{\ast}_{h_n}$ does not converge to the constrained optimum $g^{\ast}_{P_0}$ under $P_0$. Assumptions (ref) and (ref) characterize such point of discontinuity $P_0$.
Assumption (ref) assumes there exists a sequence of data distributions $\{P_{h_{n}}\}$ that differ only marginally relative to $P_{0}$. Moreover, the budget function $B(g;P_{h_{n}})$, evaluated at polices that meet the budget constraint exactly under $P_{0}$, approaches $B(g;P_{0})$ from above. In Appendix (ref), I give more primitive assumptions under which Assumption (ref) is guaranteed to hold, requiring the sequence to be differentiable in quadratic mean at $P_{0}$, and along the sequence, the budget function $B(g;P_{h_n})$ is twice continuously differentiable at $P_{0}$ with positive derivatives. These primitive assumptions are relatively weak, and have also been considered in the literature to construct the local parametrization around $P_{0}$, e.g. hirano_asymptotics_2009.
Consider a one-dimensional policy class $\mathcal{G}$, e.g. income thresholds. Figure (ref) illustrates a distribution $P_{0}$ that satisfies both Assumptions (ref) and (ref), while both the welfare function $W_{}(g;P_{0})$ and the budget function $B(g;P_{0})$ are still continuous in $g$, satisfying Assumption (ref). Importantly, the constrained optimal policy $g_{P_{0}}^{\ast}$ satisfies the budget constraint exactly, i.e. $B(g_{P_{0}}^{\ast};P_{0})=k$, but is separated from the rest of feasible eligibility policies such that there exists a neighborhood around $g_{P_{0}}^{\ast}$ where feasible policies can achieve welfare gain without any effect on the budget.
Figure (ref) provides some intuition for Theorem (ref). Note that if a statistical rule $\widehat{g}$ is pointwise asymptotically welfare-efficient and pointwise asymptotically feasible under $P_{0}$, then it has to select eligibility policies close to $g_{P_{0}}^{\ast}$ with high probability over repeated sample draws distributed according to $P_{0}^{n}$ as $n\rightarrow\infty$.
Under Assumption (ref), the class of distributions $\mathcal{P}$ is sufficiently rich so that along a sequence of data distributions $\{P_{h_{n}}\}$ that is contiguous to $P_{0}$ as $n\rightarrow\infty$, the budget functions $B(g;P_{h_{n}})$ converge to $B(g;P_{0})$ while $B(g_{P_{0}}^{\ast};P_{h_{n}})>k$, i.e. $g_{P_{0}}^{\ast}$ is not feasible under $P_{h_{n}}$. Figure (ref) showcases $P_{1}$ as one distribution from this sequence. The contiguity between $\{P_{h_{n}}\}$ and $P_{0}$ implies that the statistical rule $\widehat{g}$ must select policies close to $g_{P_{0}}^{\ast}$ with high probability under $P_{h_{n}}^{n}$ as well. However, the policy $g_{P_{0}}^{\ast}$ is infeasible under $P_{h_{n}}$, and therefore the statistical rule $\widehat{g}$ cannot be asymptotically feasible under $P_{h_{n}}$.
Figure (ref) also highlights the importance of Assumptions (ref) and (ref) in driving the impossibility result. Importantly, under $P_0$, the budget constraint is binding at the constrained optimal threshold (Assumption (ref)), and increasing the eligibility threshold here has a strictly positive impact on welfare but zero impact on budget (Assumption (ref)). If, instead, increasing the threshold strictly raises both welfare and budget, then Assumption (ref) is violated. In Section (ref), I demonstrate that relaxing either Assumption (ref) or (ref) gives rise to statistical rules that achieve uniformity within certain subclasses of DGPs. Nonetheless, the impossibility result remains relevant for some real-world policy settings where such assumptions may approximately hold. While the nominal cost of welfare program eligibility may be positive, the Marginal Value of Public Funds (MVPF) framework emphasizes that the relevant cost is the net cost, which incorporates fiscal externalities, such as increased tax revenue if the program increases individuals' incomes. Specifically, hendren_unified_2020 estimate fourteen welfare programs (out of 133) to have negative or zero net cost to the government, which implies these programs “pay for themselves”, aligning with Assumptions (ref) and (ref). As a result, the impossibility result suggests that statistical rules designed for such cases may be highly sensitive to small changes in the underlying distributions even in large samples.\footnote{As estimated by hendren_unified_2020, expanding Medicaid, as in Example (ref), entails a strictly positive net cost. The impossibility result discussed in this section therefore does not apply to this example. Instead, I use this example to illustrate a new trade-off rule proposed later in Section (ref).}
The previous negative result implies that no statistical rule can be both uniformly asymptotically welfare-efficient and uniformly asymptotically feasible. Thus, policymakers might want to consider statistical rules that satisfy one of these two properties. A direct extension to the existing approach in the EWM literature, the sample-analog rule, might be a natural candidate. In this section, I show the direct extension is neither uniformly asymptotically welfare-efficient nor uniformly asymptotically feasible.
I first describe the EWM approach kitagawa_who_2018 and its direct extension. Since $(\tau,C)$ involves potential outcomes, they are often unobserved and require estimation based on RCT that introduces estimation errors in addition to sampling errors. Section (ref) describes how to construct individuals' benefit and cost estimates $(\tau_{i}^{\ast},C_{i}^{\ast})$. To highlight the drawback of the direct extension to EWM, I first consider settings where we observe an experimental data of sample size $n$ where $(\tau,C)$ is directly observable, i.e. $(\tau_{i}^{\ast},C_{i}^{\ast})=(\tau_{i},C_{i})$. The goal of the simplification is to highlight that the non-uniformity I show below can arise from sampling errors alone.
One can estimate the welfare function and the budget function using their sample-analog versions:
A direct extension to the existing approach in the EWM literature is a statistical rule that solves the sample version of the population constrained optimization problem ((ref)):
The subscript \textquotedblleft sample” emphasizes how this approach verifies whether a policy satisfies the constraint by comparing the sample analog $\widehat{B}_{n}(g)$ with $k$ directly, i.e. imposes a sample-analog constraint. If no policy satisfies the constraint, then I set $\widehat{g}_{\text{sample}}$ to not assign any eligibility, i.e. $\widehat{g}_{\text{sample}}(x)=0$ for all $x\in\mathcal{X}$.
A key insight from kitagawa_who_2018 is that without a constraint, the sample-analog rule is uniformly asymptotically welfare-efficient. Unfortunately this intuition does not extend to the current setting where the constraint involves an unknown cost. There are common data distributions under which the amount of welfare loss does not vanish even as the sample size gets larger. Consider a one-dimensional policy class $\mathcal{G}=\{g:g(x)=\mathbf{1}\{x\leq t\}\}$, which is based on thresholds of a one-dimensional continuous characteristic $X$. Suppose the policymakers know benefit is positive for everyone so that welfare function is strictly increasing, and only need to estimate whether a given threshold satisfies a capacity constraint due to imperfect take-up. Furthermore, suppose the experiment sample observes take-up $C_{i}$ so that the only uncertainty arises from sampling errors. Proposition (ref) shows settings with some zero take-ups satisfy Assumptions (ref) and (ref), and the sample-analog rule is neither pointwise asymptotically welfare efficient nor pointwise asymptotically feasible.
Figure (ref) illustrates the setup of Proposition (ref), where the sampling uncertainty can be particularly problematic for the sample-analog rule. Since policymakers know the benefit is positive for everyone, $\widehat{g}_{\text{sample}}$ takes a simple form of the highest threshold where the sample-analog constraint is satisfied exactly. The driving force behind the failure of $\widehat{g}_{\text{sample}}$ as described in Proposition (ref) is that due to sampling uncertainty, whether a policy satisfies the sample-analog constraint is an imperfect measure of whether it satisfies the constraint in the population.
Since the sample-analog rule $\widehat{g}_{\text{sample}}$ restricts attention to policies that satisfy the sample-analog constraint, there is no guarantee the selected policy is actually feasible. This is very likely to happen when there is welfare gain in exceeding the budget constraint as in the setup of Proposition (ref) where $W(g;P)$ is strictly increasing in $g$. Therefore, the sample-analog rule $\widehat{g}_{\text{sample}}$ is not asymptotically feasible under $P$. As illustrated in Figure (ref), after observing a sample depicted in panel (a), the sample-analog rule picks an infeasible threshold because the sample-analog constraint is still satisfied there. However, as argued later in Corollary (ref), the probability of large budget violations vanishes as the sample size increases.
The more problematic case is illustrated in Figure (ref) panel (b), where the sample-analog rule picks a suboptimal threshold because the sample-analog constraint is violated at the constrained optimum $g_{P}^{\ast}$. In the setup of Proposition (ref), when the sample-analog rule $\widehat{g}_{\text{sample}}$ misses $g_{P}^{\ast}$, it is guaranteed to select a suboptimal policy and therefore it is welfare-inefficient under $P$ even asymptotically. This result relies on the assumption that the welfare function strictly increases when the budget function remains constant in the neighborhood of the budget constraint. This aligns with Assumption (ref), which, as discussed in Section (ref), reflect real-world scenarios where some welfare programs have zero or negative net cost to the government. I demonstrate that the same issue extends to broader contexts by employing a simulation calibrated to a real-world DGP from the OHIE in Section (ref). If instead the budget function is strictly increasing and violates Assumption (ref), I show the sample-analog rule is asymptotically welfare-efficient in Proposition (ref) of Appendix (ref).
In the remainder of the paper, I present constructive results that modify the sample-analog rule and propose a new rule. To lay the groundwork, this section outlines the benefit and cost estimates. The appropriate expressions for these estimates depend on the type of observed data. Below I state the estimates formed based an RCT that randomly assigns the eligibility, which is the leading case of kitagawa_who_2018. The observed data $\{A_{i}^{\ast}\}_{i=1}^{n}$ consists of i.i.d. observations $A_{i}^{\ast}=(Y_{i},Z_{i},D_{i},X_{i})\in\mathcal{A}^{\ast}$. The distribution of $A_{i}^{\ast}$ is induced by the distribution of $(Y_1,Y_0,C,X)$ as in the population, as well as the sampling design of the RCT. Here $D_{i}$ is an indicator for being in the eligibility arm of the RCT, $Y_{i}$ is the observed outcome and $Z_{i}$ is the observed cost of providing eligibility to an individual participating in the RCT. The observed cost is mechanically zero if an individual is not randomized into the eligibility arm. The estimates for $(\tau,C)$ are
where $\alpha(X_{i},D_{i})=\frac{D_{i}}{p(X_{i})}-\frac{1-D_{i}}{1-p(X_{i})}$ and $p(X_{i})$ is the propensity score, the probability of receiving eligibility conditional on the observed characteristics. Since the sampling design of an RCT is known, the propensity score is a known function of the observed characteristics.
Appendix (ref) gives primitive assumptions under which Assumption (ref) is guaranteed to hold for $\widehat{W}_{n}(\cdot)$ and $\widehat{B}_{n}(\cdot)$ constructed using an RCT such as in ((ref)) or an observational study, assuming unconfoundedness and strong overlap. As standard in the literature, I need to restrict the complexity of the policy class $\mathcal{G}$. The policy class of income thresholds considered in this paper is in fact a rather simple class with VC-dimension $d+1$ where $d$ is the number of different thresholds as in ((ref)).
Let $\hat{\mathcal{G}}=\{g\in\mathcal{G}:\widehat{B}_{n}(g)\leq k\}$ denote the set of policies that $\widehat{g}_{\text{sample}}$ can choose from, which contains policies that do not violate the sample-analog of the budget constraint. Unsurprisingly, for any finite sample, $\hat{\mathcal{G}}$ can always contain infeasible policies, sometimes of sizable budget violations. The probability that $\hat{\mathcal{G}}$ contains a policy that violates the population budget constraint by a fixed amount $c$ is as follows:
The corollary stated below shows the chance that a large amount of budget violation of $c>0$ occurs is smaller when there is less variability in $\widehat{B}_{n}(g)$. The chance also vanishes to zero as the sample size gets larger.
A simple modification to the sample-analog rule can reduce the probability of selecting infeasible policies. Instead of $\hat{\mathcal{G}}$, let $\widehat{g}_{\alpha}$ choose policies from a subset $\hat{\mathcal{G}}_{\alpha}$ defined in Theorem (ref) and maximize $\widehat{W}_n(g)$ as before. Then as a direct consequence of Theorem (ref), with probability at least $1-\alpha$ this modified rule is guaranteed to not mistakenly choose infeasible eligibility policies. In practice, $\alpha$ may be set at the conventional level, e.g. 5%. Then effectively this modification first forms a uniform upper confidence band for the costs of all the policies with asymptotic coverage at least 95% and $\hat{\mathcal G}_{\alpha}$ collects only the policies for which the upper bound on cost is below the threshold.
Note that in (ref) the sample-analog constraint is tightened by $c_{\alpha}\cdot\frac{\widehat{\Sigma}^{B}(g,g)^{1/2}}{\sqrt{n}}$ where $c_{\alpha}$ is negative, which means the class $\hat{\mathcal{G}}_{\alpha}$ only includes eligibility policies where the constraint is slack in the sample. The sample-analog constraint is tightened proportionally to the standard error to reflect that $\widehat{B}_{n}(g)$ might be particularly noisy for some $g$. The tightening therefore shrinks inversely proportional to the (square root of) sample size because intuitively larger sample size reduces the sampling uncertainty.
Although $\widehat{g}_{\alpha}$ achieves asymptotic feasibility uniformly over a wide class of DGPs, it may lead to welfare inefficiencies under certain DGPs. Below I show that if one lets $\alpha_n\rightarrow0$ as sample size increases, at a rate such that $c_{\alpha_{n}}=o(n^{1/2})$, then the modified sample-analog rule $\widehat{g}_{\alpha_{n}}$ is both asymptotically welfare-efficient and asymptotically feasible uniformly over two subclasses of distributions.
Note that the above results do not contradict the impossibility result in Section (ref), which shows lack of uniformity over a broader class of distributions, including those satisfying Assumptions (ref) and (ref). Specifically, Corollary (ref) establishes uniformity within a subclass of distributions that violate Assumption (ref), while Corollary (ref) establishes uniformity within a subclass of distributions that violate Assumption (ref).
Up to this point, the discussion has assumed that any budget violation is undesirable. If policymakers are willing to borrow, potentially at some penalty, to exceed the budget constraint in order to achieve higher welfare, then alternative rules might be preferable. Section (ref) first formalizes this setting as a trade-off problem. Section (ref) derives a statistical rule that implements such trade-off in the sample, and is shown to be uniformly asymptotically welfare-efficient.
Exceeding the budget can have negative economic consequences, such as increased borrowing costs or reduced funding for other programs. However, it may not be severe enough to entirely outweigh the perceived welfare gains. Consider a new objective function in which the policymaker, while operating within the budget constraint, seeks to allocate resources efficiently to maximize welfare, but once the budget is exceeded, must trade off the welfare gains from additional spending against the penalty associated with the overrun. Let $\overline{\lambda}>0$ denote this penalty. Using the notation $(x)_{+}$ to represent the positive part of $x\in\mathbb{R}$, the objective function can be written as\footnote{This objective function can also be motivated as policymakers might be willing to trade off violations of the constraint against gains in welfare only to a certain extent, bounding the marginal gain of relaxing the constraint $\lambda\in[0,\overline{\lambda}]:$
}
Here $r>0$ denotes the rate at which monetary units are converted into welfare units, since welfare is often not measured in monetary terms whereas the budget is.
This objective function (ref) is a non-smooth but piecewise linear optimization problem. Denote its solution to be:
and note that the solution $\widetilde{g}_{P}$ can relax the budget constraint and achieves weakly higher welfare than the constrained optimal policy $g_{P}^{\ast}$ for any data distribution $P$. The next lemma formalizes this observation.
Given a new objective function ((ref)) that trades off the gain and the cost from violating the constraint, the goal is to derive a statistical rule that is likely to select eligibility policies that maximize the new objective function $V(g;P)$. Consider the trade-off statistical rule defined as
where the subscript “tradeoff” highlights that this statistical rule is able to relax the constraint by trading off the gain and the cost from violating the constraint. Since $\widehat{g}_{\text{tradeoff}}$ solves a sample-analog version of ((ref)), I show it consistently achieves the maximal value of $V(g;P)$ under weak conditions in Lemma (ref). Theorem (ref) further verifies that it consistently achieves welfare that is weakly higher than $W(g_{P}^{\ast};P)$. I leave to future research to verify whether the trade-off rule is minimax rate optimal.
To gain intuition for the above results, note that Lemma (ref) shows the trade-off rule $\widehat{g}_{\text{tradeoff}}$ uniformly consistently achieves $V(\widetilde{g}_{P};P),$ which by Lemma (ref) is weakly higher than the welfare achieved by the constrained optimal policy $g_{P}^{\ast}$ for any data distribution $P$. Therefore, the trade-off rule $\widehat{g}_{\text{tradeoff}}$ is asymptotically welfare-efficient uniformly over $\mathcal{P}$. At the same time, larger $\bar{\lambda}$ implies smaller violation to the budget constraint, relative to the welfare gain $W(\widehat{g}_{\text{tradeoff}};P)-W(g_{P}^{\ast};P)$.
Example (ref) of Section (ref) explains a more flexible Medicaid expansion policy that would allow the income thresholds to vary with the number of children. In this example, while the budget constraint is set equal to the cost of the current policy, policymakers would be willing to exceed the budget constraint, potentially at some penalty, in order to achieve higher welfare. This subsection uses this example, together with data from the Oregon Medicaid Health Insurance Experiment (OHIE), to illustrate the trade-off rule and compare it with the sample-analog rule and its modification.
I use the experimental data from the OHIE, where Medicaid eligibility ($D_{i}$) was randomized in 2007 among Oregon residents who were low-income adults, but previously ineligible for Medicaid, and who expressed interest in participating in the experiment. finkelstein_oregon_2012 include a detailed description of the experiment and an assessment of the average effects of Medicaid on health and health care utilization. I include a cursory explanation here for completeness.
The original OHIE sample consists of 74,922 individuals (representing 66,385 households). Of these, 26,423 individuals responded to the initial mail survey, which collects information on income as percentage of the federal poverty level and number of children, which are the characteristics of interest for targeting ($X_{i}$).\footnote{More accurately, I follow sacarny_out_2020 to approximate number of children by the number of family members under age 19 living in house as reported on the initial mail survey. I exclude individuals who did not respond to the initial survey from my sample, which differs from the sample analyzed in finkelstein_oregon_2012 as I focus on individuals who responded both to the initial and the main surveys from the OHIE. Due to this difference, the expansion policies selected using my sample do not directly carry their properties to the population underlying the original OHIE sample, as the distributions of $X$ differ. } After one year, the main survey collects data related to health ($Y_{i}$), health care utilization ($H_{i}$) and actual enrollment in Medicaid ($M_{i}$), which allows me to construct estimates for the benefit and cost of Medicaid eligibility $(\tau,C)$. Therefore I further exclude individuals who did not respond to the main survey from my sample.
For health ($Y_{i}$), I follow the binary measurement in finkelstein_oregon_2012 based on self-reported health, where an answer of “poor/fair” is coded as $Y_{i}=0$ and “excellent/very good/good” is coded as $Y_{i}=1$. For health care utilization ($H_{i}$), the study collected measures of utilization of prescription drugs, outpatient visits, ER visits, and inpatient hospital visits. finkelstein_oregon_2012 annualize these utilization measures to turn these into spending estimates, weighting each type by its average cost (expenditures inflated with the CPI-U to 2007 dollars) among low-income publicly insured non-elderly adults in the Medical Expenditure Survey (MEPS). Note that health and health care utilization are not measured at the same scale, which requires rescaling when I consider the trade-off between the two. I address this issue in Section (ref). Lastly, since the enrollment in Medicaid still requires an application, not everyone eligible in the OHIE eventually enrolled in Medicaid, which implies $M_{i}\leq D_{i}$.
Given the setup of the OHIE, Medicaid eligibility ($D_{i}$) is random conditional on household size (number of adults in the household) entered on the lottery sign-up form and survey wave. While the original experimental setup would ensure randomization given household size, the OHIE had to adjust randomization for later waves of survey respondents (see the Appendix of finkelstein_oregon_2012 for more details). Denote the confounders (household size and survey wave) with $V_{i}$, and define the propensity score as $p(V_{i})=\operatorname{Pr}\{D_{i}=1\mid V_{i}\}$. If the propensity score is known, then the construction of the estimates follows directly from the formula ((ref)). However, the adjustment for later survey waves means I need to estimate the propensity score, and I adapt the formula ((ref)) following athey_policy_2021 to account for the estimated propensity score.
Specifically, define the conditional expectation function (CEF) of a random variable $U_{i}$ as $\gamma^{U}=\ensuremath{\mathbb{E}}[U_{i}\mid V_{i},D_{i}${]}. Since $V_{i}$ in my case is discrete, I use a fully saturated model to estimate the propensity score $\widehat{p}(V_{i})$ and the CEF $\widehat{\gamma}^{U}(V_{i},D_{i})$. I then form the estimated Horvitz-Thompson weight with the estimated propensity score as $\widehat{\alpha}(V_{i},D_{i})=\frac{D_{i}}{\widehat{p}(V_{i})}-\frac{1-D_{i}}{1-\widehat{p}(V_{i})}$. For health benefit due to Medicaid eligibility, define the estimate $\tau_{i}^{\ast}=\widehat{\gamma}^{Y}(V_{i},1)-\widehat{\gamma}^{Y}(V_{i},0)+\widehat{\alpha}(V_{i},D_{i})\cdot\left(Y_{i}-\widehat{\gamma}^{Y}(V_{i},D_{i})\right)$. For the cost due to Medicaid eligibility, define the estimate $C_{i}^{\ast}=\widehat{\gamma}^{Z}(V_{i},1)+\frac{D_{i}}{\widehat{p}(W_{i})}\cdot\left(Z_{i}-\widehat{\gamma}^{Z}(V_{i},D_{i})\right)$ where $Z_{i}=M_{i} \cdot H_{i}$. Since an eligible individual only incurs cost to Medicaid if enrolled, I need to account for imperfect take-up in forming $C_{i}^{\ast}$.
Table (ref) presents the summary statistics. While Appendix (ref) argues the estimation errors in $\tau_{i}^{\ast}$ and $C_{i}^{\ast}$ are asymptotically negligible, in finite samples, the cost estimates are highly variable, resulting in noisy estimate $\widehat{B}_n(g)$.
To formalize the budget constraint requiring that the average cost of any proposed policy does not exceed that of the status quo 2014 Medicaid expansion, I calibrate the per capita cost under the 2014 policy. Following finkelstein_oregon_2012, who cite wallace2008effective, Medicaid spending among individuals comparable to the Oregon Health Insurance Experiment (OHIE) participants was approximately \$3,000 per enrollee in Oregon in 2004, which corresponds to about \$3,300 in 2007 dollars. Under the 2014 policy, I adjust for imperfect take-up by defining the per capita cost as
where $M(1)$ denotes enrollment status if offered Medicaid eligibility and $g_{2014}(x)=\mathbf{1}\{\text{income}\leq 138\%\}$ represents the status quo 2014 expansion policy of providing eligibility to all adults with income up to 138%. The enrollment rate under the status quo 2014 policy $\ensuremath{\mathbb{E}}_{P}[M(1)g_{2014}(X)]$ is based on point estimate from the OHIE.\footnote{The theoretical results in this paper are developed for a fixed $k$. Addressing the estimation error in $k$ is left to future work.}
Figure (ref) summarizes the selected expansion policies, which are income thresholds specific to the number of children. The sample-analog rule $\widehat{g}_{\text{sample}}$ chooses to restrict Medicaid eligibility, especially lowering the income threshold for childless individuals far below the current level, and the estimated welfare is 3.79% increase in reporting good subjective health. The budget estimate for the selected policy is \$1,311, slightly below the threshold of $k=\$1,377$, as $\widehat{g}_{\text{sample}}$ imposes the sample-analog version of the budget constraint. However, due to the large variation in the cost estimates as illustrated in Table (ref), meeting the sample budget constraint still involves uncertainty about whether the selected policy meets the budget constraint in the population as argued in Proposition (ref). After taking into account of estimation uncertainty, the modified sample-analog rule $\widehat{g}_{\alpha=5\%}$ is much more conservative than the sample-analog rule $\widehat{g}_{\text{sample}}$. The budget estimate of the selected policy \$1,174, with a standard error of 66, making it statistically significantly below the budget constraint at the conventional 5% level. The welfare estimate of the selected policy is lower at 3.43%. One can reduce the conservativeness by increasing $\alpha$ and therefore lowering the statistical guarantee that the selected policy meets the budget constraint in the population. I examine the results under higher values of $\alpha$ in Appendix (ref).
To construct the trade-off rule $\widehat{g}_{\text{tradeoff}}$ as proposed in Section (ref), I need to specify both the penalty parameter, $\overline{\lambda}$ and the conversion rate, $r$. For illustration, I assume that exceeding the budget incurs a full repayment of the overrun, implying $\overline{\lambda}=1.$ In my empirical illustration, the budget constraint is in terms of monetary value. The objective function, however, is measured based on self-reported health, which does not directly translate to a monetary value. finkelstein_value_2019 converts self-reported health into value of a statistical life year (VSLY) based on existing estimates. Specifically, a conservative measure for the increase in quality-adjusted life year (QALY) when self-reported health increases from “poor/fair” to “excellent/very good/good” is roughly 0.6. The \textquotedblleft consensus\textquotedblright estimate for the VSLY for one unit of QALY from cutler_your_2004 is \$100,000 for the general US population. I therefore follow finkelstein_value_2019 and set $r = (0.6\cdot\$100,000) = \$60,000.$ The trade-off objective function is therefore
The trade-off rule $\widehat{g}_{\text{tradeoff}}$ based on (ref) chooses to assign Medicaid eligibility to more individuals, and to raise the income thresholds above the current level for those with children. Therefore the welfare estimate for the selected policy is higher at 3.86% and budget estimate for the selected policy is \$1,402, slightly above the budget constraint. The higher level occurs because on average the benefit estimates are positive, and the trade-off rule finds that the additional health benefit from violating the budget constraint exceeds the cost of doing so, especially those with children. While a traditional cost-benefit analysis would also recommend prioritizing those with children based on Table (ref), the trade-off rule offers a more interpretable recommendation by directly selecting income thresholds.
However, setting $\overline{\lambda}=1$ assumes that the penalty is limited to repaying the budget overrun, potentially understating the true penalty for policymakers, as it does not account for additional economic costs associated with exceeding the budget, such as reduced funding for other programs or lower quality of Medicaid services. Accordingly, in Appendix (ref), I explore how the results change when $\overline{\lambda}$ is increased, which is equivalent to making budget overruns more costly to the policymaker. For example, with $\overline{\lambda}=1.68$, the welfare and budget estimates for the selected policy essentially drop to those of $\widehat{g}_{\text{sample}}$.
In this example, the policies chosen by different rules differ substantially. A natural question is how policymakers should choose the rules. In this Medicaid example, it is reasonable to assume that policymakers can borrow, potentially at some penalty, to exceed the budget constraint in order to achieve higher welfare. Therefore the trade-off rule $\widehat{g}_{\text{tradeoff}}$ is theoretically attractive. It selects policies that achieve at least the maximum feasible welfare, while accounting for the penalty $\overline{\lambda}$ from violating the budget constraint. Because the amount of budget violations depend on the choice of $\overline{\lambda}$ as shown in Theorem (ref), one can assess this trade-off for their particular settings by varying $\overline{\lambda}$. However, in other context where policymakers are financially conservative and even minor violation to the budget constraint is unacceptable, then the modified sample-analog rule $\widehat{g}_{\alpha}$, with a pre-specified significance level $\alpha$, is theoretically attractive. It offers a statistical guarantee that the budget constraint will be met in the population with high confidence. However, the selected eligibility policy may appear very conservative, as illustrated in the Medicaid expansion example above.
To ensure the practical relevance of the simulation, I calibrate to the distribution of the data from the OHIE, which is also used for empirical illustration in Section (ref), based on Example (ref) of Section (ref).
For the purpose of this simulation study, the OHIE represents the population $P$, and I take the estimates $(\tau_{i}^{\ast},C_{i}^{\ast})$ constructed in Section (ref) as the true benefit and cost $(\tau,C)$. Under this simulation design, I can solve for the constrained optimal policy as \[ g_{P}^{\ast}\in\arg\max_{g\in\mathcal{\widetilde{G}},\ B(g;P)\leq k}W_{}(g;P) \] where $\left(W_{}(g;P),B(g;P)\right)$ are the sample analogs in the OHIE sample. The policy class $\mathcal{\widetilde{G}}$ includes income thresholds that can vary with the number of children as described in Equation (ref) of Appendix (ref), which is slightly coarsened than the class used in Section (ref). The maximum feasible welfare is given by $W_{}(g_{P}^{\ast};P)=3.76\%$, an increase of 3.76% in reporting good subjective health. The cost associated with the constrained optimal policy is $B(g_{P}^{\ast};P)=\$1,340$, slightly below the constraint $k=\$1,377$ as defined in Equation (ref).
Table (ref) compares the performance of various statistical rules $\widehat{g}$ through 500 Monte Carlo iterations. At each iteration, I randomly draw observations from the OHIE sample to form a random sample. I simulate with the same sample size as the original sample to hold the amount of sampling uncertainty constant. Given the random sample, I collect eligibility policies chosen by each of the following statistical rules:
I evaluate the welfare function and the budget function $\left(W_{}(g;P),B(g;P)\right)$ for a given policy in the original OHIE sample. Averages over 500 iterations provide simulation evidence on the properties of the above statistical rules, as shown in Table (ref).
Row 1 of Table (ref) illustrates that it is possible for all three statistical rule $\widehat{g}$ to select infeasible policies. A lower probability of selecting infeasible policies suggests the rule is closer to achieving asymptotic feasibility. In the distribution calibrated to the OHIE sample, the original sample-analog rule $\widehat{g}_{\text{sample}}$ might not be asymptotically feasible as it can select infeasible eligibility policies in 10.2% of the draws. In contrast, Theorem (ref) guarantees that a simple modification $\widehat{g}_{\alpha=5\%}$ selects infeasible eligibility policies in less than 5% of the draws, regardless of the distribution. Simulation confirms such guarantee as the mistakes only happen 0.2% of the time.
Row 2 of Table (ref) illustrates that it is possible for all three statistical rule $\widehat{g}$ to achieve weakly higher welfare than the constrained optimal policy $g_{P}^{\ast}$. This can happen when $\widehat{g}$ selects an infeasible policy. A lower probability of selecting suboptimal policies suggests the rule is closer to achieving asymptotic welfare-efficiency. Theorem (ref) implies that the trade-off rule $\widehat{g}_{\text{tradeoff}}$ is uniformly asymptotically welfare efficient while there is no such guarantee for the sample-analog rule $\widehat{g}_{\text{sample}}$.
In the distribution calibrated to the OHIE, the trade-off rule $\widehat{g}_{\text{tradeoff}}$ on average achieves higher welfare than the sample-analog rule $\widehat{g}_{\text{sample}}$. As shown in row 3 of Table (ref), the welfare loss of $\widehat{g}_{\text{tradeoff}}$ is 1% of the maximum feasible welfare $W_{}(g_{P}^{\ast};P)$, compared to 6% for $\widehat{g}_{\text{sample}}$. However, its improvement can be at the cost of violating the budget constraint more often than $\widehat{g}_{\text{sample}}$, at a rate of 45.2%. Though as shown in row 4 of Table (ref), on average the violation is limited, which confirms Theorem (ref).
In this paper, I focus on properties of statistical rules when the cost of implementing any given policy needs to be estimated. The existing EWM rule selects an eligibility policy that maximizes a sample analog of the social welfare function, and only accounts for constraints that can be verified with certainty in the population. However, in some cases, the cost of providing eligibility to any given individual might be unknown ex-ante due to imperfect take-up and heterogeneity. Therefore, in addition to asymptotic welfare-efficiency that has been studied by the EWM literature, I introduce a new desirable property of statistical rules in the setting of unknown cost, namely asymptotic feasibility, which requires the selected policy to satisfy a budget constraint when the sample size is large enough. Unlike the setting of known cost, I prove an impossibility result that no statistical rule can be uniformly asymptotically welfare-efficient and feasible. The direct extension to the existing EWM approach is no longer asymptotically welfare efficient nor asymptotically feasible for certain real-world relevant data distributions. As an alternative, I propose the trade-off rule that guarantees asymptotic welfare efficiency while ensuring any budget violations are bounded above by welfare gains. I illustrate the theoretical results using experimental data from the OHIE. A promising avenue for future research is to verify whether the trade-off rule maintains minimax rate optimality in the constrained setting, just as how the EWM rule is in the unconstrained setting.