Certified Decisions

Certified Decisions

Introduction

Inferential tools, such as hypothesis tests and confidence intervals, are ubiquitous in empirical practice. At the same time many empirical papers aim, directly or indirectly, to improve decision-making by potential consumers of research, such as policymakers, firms, or households. As highlighted by manski2021econometrics, the connection between inference and decision-making is unclear: outside of special cases it is rare that optimal decisions, in the tradition of wald1949statistical, depend on the data through the results of conventional inference procedures.\footnote{Though see tetenov2016testing, batesetal2024testing,vivianoetal2025testing for some interesting exceptions.} There is thus an apparent tension between the goal of informing decision-making and conventional, inference-focused, methods for empirical research.

Despite this tension---or perhaps because of it---some procedures in the literature aim to bridge the gap between inference and decision-making. In particular, when researchers report confidence sets, one natural recipe for decisions is to proceed as if the confidence set enumerates all plausible parameter values, and choose actions that minimize the worst-case loss over the confidence set. For $\mathcal A$ an action space, $L(a, \theta)$ a loss function, and $\hat\Theta(Y) \subset \Theta$ a data-dependent confidence set with $\P\{\theta\in\hat\Theta(Y)\}\ge1-\alpha$, we may choose actions \[ \delta(Y) \in \argmin_{a\in \mathcal A} \sup_{\theta \in \hat\Theta(Y)} L(a, \theta). \numberthis \label{eq:asif_p} \] manski2021econometrics terms decision rules of this form “as-if decisions with set estimates.” Decision rules of this form are implicitly envisioned by Neyman, who writes neyman1977frequentist

More recently, ben2021safe and chernozhukov2025policylearningconfidence select policies by minimizing the worst-case loss over a confidence set for a social welfare function.

A number of recent papers have also considered the other direction, first making a statistical decision and then constructing confidence sets for the resulting loss or welfare. kitagawa2018ewm select a policy by maximizing an empirical welfare function, and in their supplementary materials discuss how one may construct a confidence set for the true welfare via simultaneous inference and projection ponomarev2024lower. The recent literature studying inference on winners benjamini2019winners,andrews2024inference,zrnic2024winners,zrnic2024aoswinners generalizes this approach, considering selection of the “best” of several options based on noisy estimates of loss or welfare, and then constructing bounds for the true loss associated with estimated “best” choice. For $\hat L(a;Y)$ an estimate of the loss associated with option $a$ based on data $Y,$ this corresponds to choosing \[\delta(Y)=\argmin_{a\in\mathcal A} \hat L(a;Y)\] and then (in the one-sided case) constructing a loss bound $R(Y)$ with

What, if anything, do these exercises---forming decisions based on confidence sets or confidence sets for the loss of recommended decisions---accomplish? This paper provides an answer, considering the value of providing certificates for the loss associated with a recommended decision. In particular, we consider a data analyst who provides a decision-maker with a recommended decision and estimated risk bound $(\delta(Y), R(Y)).$ If this pair satisfies (ref), we say that $R(Y)$ is a P-certificate for $L(\delta(Y),\theta),$ and say that $(\delta(Y), R(Y))$ is a P-certified decision. The loss bounds $R(Y)$ derived in the literature studying inference on winners are thus P-certificates by construction. For as-if decision-making, note that if we define

then $\theta\in\hat\Theta(Y)$ implies $R(Y)\ge L(\delta(Y),\theta),$ so \[ P\br{L(\delta(Y), \theta) \le R(Y)} \ge P(\theta \in \hat\Theta) \ge 1-\alpha.\] Hence, both procedures discussed above imply corresponding P-certificates.

We show that P-certified decisions provide useful guarantees for an ambiguity-averse decision-maker who has limited capacity for data analysis and delegates it to an analyst. Specifically, we consider a decision-maker who needs to decide whether to adopt an action recommended by the data analyst, or to instead implement a safe default action with known (but potentially large) loss. This decision-maker believes that the analyst's estimate $R(Y)$ contains information about the loss of the recommended action, but has limited knowledge of the data-generating process and wishes to protect themselves against a large increase in risk uniformly over a potentially large class of data generating processes.

P-certificates are useful to such a decision-maker. Specifically, for a decision-maker with nonnegative loss bounded by $1$ and a default action costing $C \in [0,1)$, any rule that adopts the recommended action $\delta(Y)$ only when $R(Y)\le C$ yields expected loss no greater than $C + \alpha (1-C)$, meaning that the decision-maker faces a risk increase of an most $\alpha (1-C).$ Moreover, over the class of non-randomized adoption rules, the optimal adoption decision is simply the threshold rule $\one(R (Y) \le C)$.

This result speaks only to the optimal action for a decision-maker presented with a given P-certified decision. We next study optimal provision of P-certificates, and show that it is closely connected to as-if optimization. We find that the class of as-if decisions \`a la (ref) and (ref) is essentially complete under a minimally weak preference ordering for P-certified decisions.\footnote{This result echoes---though it is developed independently from---a recent result by kiyani2025decision, in the context of conformal prediction.} That is, any P-certified decision $ (\tilde \delta, \tilde R)$ can be weakly improved by some as-if decision $(\delta, R)$ corresponding to some confidence set, in the sense that $R(\cdot) \le \tilde R(\cdot)$ almost surely and the loss estimate $R(\cdot)$ is thus weakly tighter than the loss estimate $\tilde R(\cdot)$. This establishes that all reasonable P-certified decisions can be cast as as-if optimization (ref), but it does not distinguish certain confidence sets from others. This result has some bite, and implies for instance that the studentized projection approach discussed in e.g. andrews2024inference is dominated by the approaches of chernozhukov2025policylearningconfidence.

With more structure on the decision problem, we further show that some optimality results for inference translate to optimality for P-certificates. In particular, if parameter is scalar and the loss function prefers high parameter values (e.g. implementing treatments with higher effects yields lower loss), then uniformly most accurate confidence lower bounds for $\theta$ are optimal so long as we prefer smaller values of $R(Y)$.

Finally, as an extension, we consider certified decisions that correspond to alternative approaches to frequentist inference. A rapidly growing literature proposes e-values as alternatives to p-values and confidence sets ramdas2024hypothesis. We show that e-variables generate a type of as-if decisions---called E-posterior decisions by grunwald2023posterior---that provide an E-certificate of the form $\E [L (\delta,\theta) / R(Y)] \le 1$. Similarly, we show that the class of such as-if decisions is essentially complete among E-certified decisions: for any E-certified $(\tilde\delta, \tilde R)$, there exists e-variables and corresponding E-posterior decisions $(\delta(\cdot), R(\cdot))$ where $R(\cdot) \le \tilde R(\cdot)$ almost surely. We show that E-certified decisions imply risk bounds for downstream decision-makers even in contexts with unbounded losses.

This paper is most similar to recent work by kiyani2025decision, who study decision-making in the context of conformal inference. Similar to our results in (ref) but preceding this paper, kiyani2025decision show that the problem of minimizing the expected value of an $1-\alpha$ upper bound for loss is equivalent to the problem of searching for a conformal set with coverage $1-\alpha$ and minimizing the expected minimax loss against the conformal prediction set (see (ref)). The key to both arguments is the observation that a probabilistic upper bound on loss can be inverted to form a confidence set.

(ref) sets up a stylized model of an ambiguity-averse decision-maker and discusses guarantees provided by P-certificates. (ref) discusses optimal P-certificates. (ref) extends our results to E-certificates.

Ambiguity-averse decision-makers and P-certified decisions

Consider a decision-maker facing a loss function $L(a,\theta) \in [0,1]$ for an action $a\in \mathcal A$ and an unknown parameter $\theta \in \Theta$. The decision-maker tasks an analyst to analyze the data $Y \in \mathcal Y$ and to provide a recommended action $\delta(Y) \in \mathcal A$. The data $Y$ is sampled from a distribution $P$. To model uncertainty, we assume that Nature may choose any $ (\theta, P)$ pairs from a set $\mathcal M$. For instance, if $(\theta, P)$ specifies a statistical model indexed by $\theta$, then $\mathcal M = \br{(\theta, P_\theta): \theta \in \Theta}$. More generally, we allow $\theta$ may fail to fully describe $P,$ for instance due to nuisance parameters or model incompleteness.

The decision-maker has access to a default action $a_0\not\in\mathcal{A}$, which yields a loss $C$ which is known to the decision-maker and does not depend on $\theta.$ To assess whether to adopt the analyst's recommendation instead, the decision-maker also asks for an assessment of the loss $L(\delta (Y),\theta)$. In this section and the next, we assume that this assessment takes the form of a high-probability upper bound on the loss, which we call a $(1-\alpha)$ P-certificate $R(Y)$ for $\delta(Y)$.\footnote{$\delta(\cdot), R (\cdot)$ are allowed to depend on external randomization devices. For compactness, we subsume such external random variables into $Y$.}

The guarantee (ref) is both familiar and convenient, since many familiar statistical procedures yield guarantees of this form. In particular, if the analyst computes a $1-\alpha$ confidence set $\hat\Theta(Y)$, as-if minimax optimization over the confidence set provides such a guarantee: For $\delta(Y) \in \argmin_{a\in \mathcal A} \sup_{\theta \in \hat\Theta(Y)} L (a, \theta)$ and $R(Y) = \min_{a\in \mathcal A} \sup_{\theta \in \hat\Theta(Y)} L(a, \theta)$,\footnote{If the $\argmin$ does not exist, then for any $\epsilon > 0$, we may choose $(\delta (Y), R (Y))$ such that \[ \sup_{\theta \in \hat\Theta(Y)} L(\delta(Y), \theta) \le \inf_{a\in \mathcal A} \sup_{\theta \in \hat\Theta(Y)} L(a, \theta) + \epsilon \] and $R(Y) = \sup_{\theta \in \hat\Theta(Y)} L(\delta(Y), \theta) $. } the guarantee (ref) is a consequence of coverage: \[ P\br{L(\delta(Y), \theta) \le R(Y)} \ge P\{\theta \in \hat\Theta(Y)\} \ge 1-\alpha. \numberthis \label{eq:confidence} \] We shall see in (ref) that the converse is also essentially true: Any reasonable $(\delta, R)$ pair may be derived from as-if optimization from some confidence set.

For now, we consider a decision-maker's adoption decision given a P-certified decision. We suppose that the decision-maker encounters considerable ambiguity, considering a model $\mathcal{M}_{\mathrm{DM}}$ that is potentially much larger than $\mathcal{M}$. Given a P-certified decision, however, the decision-maker confines attention to those $ (\theta, P)$ pairs under which (ref) holds \[ \mathcal{M}_{\mathrm{DM}}^{(P,\alpha)} = \br{(\theta, P) \in \mathcal{M}_{\mathrm{DM}}: P\bk{L(\delta(Y), \theta) \le R(Y)} \ge 1-\alpha}. \] Assume that the decision-maker has some prior $\pi$ over $\mathcal{M}_{\mathrm{DM}}^{(P,\alpha)}$, which could for instance be a prior on $\mathcal{M}_{\mathrm{DM}}$ truncated to $\mathcal{M}_{\mathrm{DM}}^{(P,\alpha)}$.

Suppose that after observing $(\delta(Y),R(Y))$ the decision-maker adopts the recommended action, $Q=1,$ with probability probability $q(a,r) \equiv \P(Q=1 \mid \delta(Y)=a, R (Y)=r) $ while otherwise, $Q=0$, they select the safe action $a_0.$ The decision-maker wishes to choose the option rule $q$ from some class $\mathcal{Q}$ to minimize minimize expected loss under her prior, subject to the constraint that the worst-case expected loss is not much worse than $C$. That is, let $\delta_Q(Y) = Q\delta(Y) + (1-Q)a_0$, where for $\tau \in (0, 1-C]$ the decision-maker chooses $q$ to solve:

The preference (ref) minimizes Bayes risk under $\pi$, subject to the constraint that the worst-case risk is bounded. The preference (ref) encodes that the decision-maker is conservative, ambiguity-averse, but persuadable. If the decision-maker were consumed by ambiguity aversion, then she would implement the default action and would have no use for the analyst. In contrast, under (ref), she is willing to hazard most an excess risk of $\tau$ for plausible rewards when the worst case over $\mathcal{M}_{\mathrm{DM}}^{(P,\alpha)}$ is overly paranoid.

The preference (ref) is also reminiscent of the ambiguity-averse preference (DP) considered in banerjee2020theory. banerjee2020theory consider a weighted average between a Bayes expected utility and the worst-case expected utility. One may interpret this weighted average as a Lagrange-multiplier form of (ref).

P-certified decisions let the decision-maker ensure that the worst-case risk constraint is respected, while still sometimes adopting the recommendation. Suppose the decision-maker never accepts a recommendation when $R(Y) > C$. If they accepts the recommendation with probability at most $u \in [0,1]$, $\sup_{a,r} q(a,r)\le u$, then they incur no more than $u\alpha (1-C)$ in excess risk, regardless of how large $\mathcal{M}_{\mathrm{DM}}$ is.

(ref) assures the decision-maker that she does not have to tolerate much excess risk if she adopts whenever $R(Y)$ indicates that $\delta(Y)$ is plausibly preferable to the default action.\footnote{Since $\P\{L(\delta_Q(Y),\theta)>R(Y)\}\le\alpha$ and $L(\delta_Q(Y), \theta)\le 1$, we also obtain the following $P$-specific bound, which is markedly better if $R (Y)$ can be much smaller than $C$: \[ \E_P\bk{ L(\delta_Q(Y), \theta) } \le \alpha + \E_P[QR(Y) + (1-Q)C]. \] If $Q = \one(R(Y) \le C)$, this bound simplifies to $\alpha + \E_P [\min(R(Y), C)]$.} To further control the excess loss, she may either adopt less, in the sense of choosing a smaller $u$, or ask that the analyst provide recommendations with higher confidence, requesting a larger $1-\alpha$.

Motivated by (ref), we may ask for what beliefs $\pi$ and sets of rules $\mathcal{Q}$ the problem (ref) yields the simple the simple threshold decision rule $q(a,r) = u\one(r \le C).$ To this end, we impose a few restrictions.

First, suppose the decision-maker's beliefs $\pi$, the expected loss is upper bounded by $R(Y)$: $\E_\pi[L(\delta (Y), \theta) \mid R (Y) =r ] \le r$. Moreover, suppose that the decision-maker's model $\mathcal{M}_{\mathrm{DM}}^{(P,\alpha)}$ is sufficiently rich so that any joint distribution over $(L,R)=(L (\delta(Y), \theta), R(Y)) \in [0,1]^2$ consistent with the P-certificate constraint (ref) is rationalized by some member of $\mathcal{M}_{\mathrm{DM}}^{(P,\alpha)}$.

Second, suppose that the decision-maker only considers simple adoption decisions based on $R(Y)$ and not on $\delta(Y)$: i.e., $q(a,r) = q (r)$. This restricts to only “simple” adoption decisions, for instance because the decision-maker either does not observe $\delta(Y)$ when deciding to adopt or has a difficult time assessing different recommendations $\delta(Y)$ on their own merits.

Third, suppose the decision-maker commits to adopting with at least probability $u$ for some assessment $R(Y) = r$: $\sup_{r \in [0,1]} q(r) \ge u$. For instance, it may be reasonable to impose $q(0) = 1$ so that decisions with the maximally favorable assessment is always adopted. If we further restrict decision rules to be non-randomized, then this condition (with $u=1$) is equivalent to the decision-maker committing to sometimes accept the analyst's recommendation.

Finally, under the preceding assumptions, one can show (see (ref) below) that the worst-case excess risk is bounded above by $u \alpha (1-C).$ Assume the decision-maker is sufficiently conservative that this constraint binds, $\tau=u \alpha(1-C).$

Under these conditions, we can show that $u \one(r \le C)$ is optimal for (ref).

The intuition for (ref) is simple. Since the decision-maker commits to accepting with probability at least $u$, this exhausts the excess risk budget $\tau$. Accepting any recommendation with $R(Y) > C$ strictly increases the worst-case risk over a rich $\mathcal{M}_{\mathrm{DM}}^{(P,\alpha)}$. On the other hand, accepting any $R (Y) = r \le C$ with probability less than $u$ worsens the objective in (ref) without relaxing the constraint. Thus, the optimal acceptance decision is $u\one(r \le C)$. In particular, if the decision-maker commits to non-randomized decision rules $q(r) \in \br{0,1}$ and to accept some recommendations ($\sup_{r} q(r) = 1$), then the optimal acceptance decision is immediately $q(r) = \one(r \le C)$. (ref) furthermore characterizes the form of optimal adoption rules when we do not restrict to $q\in\mathcal Q(u),$ though their form depends on the details of the prior.

Together, (ref) and (ref) show that P-certified decision rules are compatible with simple adoption decisions for an ambiguity-averse decision-maker willing to tolerate some increase in her worst-case risk relative to a safe option. Thus, P-certified decisions offer downstream decision-makers---who may be less familiar with the statistical environment---simple insights from data that guide decisions, without sacrificing (much) safety.

Provision of P-certified decisions requires that the analyst knows the decision-maker's loss function $L (a, \theta)$, but does not require knowledge of the prior $\pi$ nor the cost of default option $C$. To further reduce information requirements, via (ref) it is even possible for the analyst to simply communicate a confidence set $\hat \Theta(Y)$ without knowledge of the loss function, and ask the decision-maker to find $(\delta(Y), R(Y))$ via as-if optimization.

We make two remarks before illustrating with an example. First, we note that the results (ref) and (ref) apply to any P-certified decisions, including the trivial certificate where $R(Y) = 1$ with probability $1-\alpha,$ independent of $\delta(Y)$. Thus, these results do not require that the analyst be particularly skilled in data analysis, nor that they necessarily shares the same objective as the decision-maker, just that the analyst be able to credibly provide P-certificates.

Second, the validity of P-certificates is to some extent verifiable. To prove that $(\delta(\cdot), R(\cdot))$ is a P-certified decision, when the loss function $L$ is money-metric the analyst can in principle offer insurance policies that pay out $1$ unit of utility if it turns out that $L(\delta(Y), \theta) > R(Y)$, in exchange for $\alpha$ units of insurance premium. The decision-maker can purchase any amount of such policies before seeing the data. If $ (\delta (\cdot), R (\cdot))$ is indeed a P-certificate, offering such insurance has nonnegative expected value for the analyst.\footnote{As neyman1937outline writes, “The theoretical statistician constructing [a confidence interval] may be compared with the organizer of a game of chance in which the gambler has a certain range of possibilities to choose from while, whatever he actually chooses, the probability of his winning and thus the probability of the bank losing has permanently the same value, [$\alpha$].”}

Example: inference on winners

To illustrate our results, we consider an example based on the recent literature on inference on winners benjamini2019winners,andrews2024inference,zrnic2024winners,zrnic2024aoswinners. Consider a decision-maker interested in potentially implementing an action $a$ from a finite set $\mathcal{A}.$ For instance, the decision-maker could be a municipal policymaker deciding considering whether to implement a program, as studied in bergman2024cmto, that encourages low-income households with children to move to high-opportunity neighborhoods. In this context, each $a\in\mathcal{A}$ could index a potential set of “recommended” neighborhoods, while the default action $a_0$ corresponds to the status quo of not making any recommendation.

The decision-maker's goal is to maximize some bounded outcome, for instance the average household income rank in adulthood (relative to the adult income distribution) for children in the targeted households. Specifically, for $\theta(a)\in[0,1]$ the (unknown) true outcome associated with selection $a,$ the policymaker would like to choose actions $a$ that solve $\max_{a\in\mathcal{A}}\theta(a)$. The policymaker also faces a fixed cost $\kappa$ of making a recommendation at all, for instance because such recommendations have not been made in the past and this new program needs to be explained to the public. To represent this decision-maker's preferences we can consider the loss function $L(a,\theta)=1-\theta(a),$ and take $C=1-\theta(a_0)-\kappa$ where for simplicity we treat $\theta(a_0)$ as known.

Suppose the decision-maker is assisted by an analyst, who uses data (e.g. Census-tract level measures for economic opportunity together with a model for long-term outcomes as in bergman2024cmto) to form an estimate $X(a)$ for the outcome associated with each possible action $a,$ along with an associated standard error $\sigma(a)$, which we treat as fixed for simplicity. The analyst, but not the decision-maker, knows the joint distribution of estimation error, where for $a\in\mathcal{A}$, $\frac{X(a)-\theta(a)} {\sigma(a)}\sim Z(a)$. In this setting, we can take $Y=(X(\cdot),\sigma(\cdot))$ to collect the estimates and standard errors.

The analyst may select a recommended action via empirical welfare maximization, taking $\delta(Y)=\argmax_{a\in\mathcal{A}}{X(a)}$. To form a P-certificate, they could take $R (Y)=1-(X(\delta(Y))-c_{1-\alpha}),$ for $c_{1-\alpha}$ the $1-\alpha$ quantile of the maximum error $\sup_{a\in\mathcal{A}}Z(a)\sigma(a).$ This bound, corresponding to an unstudentized version of what andrews2024inference call the projection approach, was suggested in the supplementary materials of kitagawa2018ewm. Alternatively, the analyst could consider studentized projection and report $R^*(Y)=1-(X(\delta (Y))-c^*_{1-\alpha} \sigma(\delta(Y))),$ for $c^*_{1-\alpha}$ the $1-\alpha$ quantile of the maximum studentized error $\sup_{a\in\mathcal{A}}Z(a).$ Studentization ensures that the width of the interval reflects the standard error for the recommended policy $\delta (Y),$ and further ensures that e.g. the inclusion of a single, very nosily estimated option $a$ in the choice set does not greatly influence the critical value. For this reason, the theoretical discussion of projection in andrews2024inference focuses primarily on the studentized approach.

Given either $(\delta(Y),R(Y))$ or $(\delta(Y),R^*(Y)),$ the decision-maker must decide whether to implement to action $\delta(Y).$ The preference (ref) captures that they decisonmaker seek to maximize the expected outcome under some joint prior $\pi$ on $\theta$ and the distribution of $Y.$ However, since the decision-maker does not know the distribution of $Y$ (beyond their knowledge that the analyst provides a P-certificate), they limit attention to adoption rules $q$ such that risk is not greatly increased relative to the status quo, even in the worst case. Under the conditions of (ref), we have shown that it is then optimal for the decision-maker to adopt (perhaps with some probability) when $R(Y)\le C.$ By the duality between tests and confidence sets, however, $\one\{R(Y)\le C\}$ and $\one\{R^*(Y)\le C\}$ are size-$\alpha$ tests for the null $L(\delta(Y),\theta)\ge C,$ so (ref) implies that the decision-maker adopts the recommended action only when the analyst rejects the null that it increases loss relative to the status quo.

Optimal P-certificates

In settings where P-certificates are useful, it seems reasonable that both analysts and decision-makers will prefer tighter bounds on the loss. That is, given the choice between level-$1-\alpha$ P-certificates $R(Y)$ and $\tilde{R}(Y)$ such that $R(Y) \le \tilde R (Y)$ for all $Y,$ it is natural to prefer $R$ to $\tilde{R}$. For instance, if the decision-maker uses the non-randomized adoption rule $Q=\one(R(Y)\le C)$ derived in (ref), then \[ \E_P[L(\delta_Q,\theta)]\le\alpha+\min(R(Y),C), \] and using $R(Y)$ rather than $\tilde{R}(Y)$ yields a tighter risk bound on the loss. More generally, if we consider any loss function $B(R,\theta)$ that is increasing in $R$ for all $\theta,$ then $\E_P[B(R(Y),\theta)]\le\E_P[B(\tilde{R}(Y),\theta)]$ for all $(P,\theta),$ so $R$ yields a lower expected loss than $\tilde{R}$.

When $R$ and $\tilde{R}$ are ordered almost surely, or more generally in the sense of first-order stochastic dominance, we say $R$ weakly dominates $\tilde R$.

Our main result in this section shows that the class of P-certified decisions from as-if optimization against a confidence set is an essentially complete class with respect to dominance. That is, any P-certified decision rule $ (\tilde \delta, \tilde R)$ is weakly dominated by some decision rule $(\delta,R)$ formed by as-if optimization via (ref). As an implication, these decision rules are also essentially complete with respect to any risk function of the form $\E_ {P} [B(R, \theta)]$, for any weakly increasing $B(\cdot, \theta)$.

To state this result, denote by $\mathcal C_{P,\alpha, \epsilon}$ the class of decision rules formed by ($\epsilon$-approximate) as-if optimization against a confidence set $\hat \Theta (Y)$ in the following fashion: For $\epsilon > 0$, consider $(\delta(Y), R(Y))$ such that

where $\sup$ over an empty set is defined to be $0$. The constant $\epsilon$ accounts for cases where the minimax loss over $\hat\Theta(Y)$ is not achieved by any element of $\mathcal A$; our result holds for any $\epsilon > 0$.

(ref) shows a sense in which it is without loss to limit attention to P-certificates derived from as-if optimization with confidence sets. Thus, not only is as-if optimization a convenient way to generate P-certified decisions, but there is, in a sense, no need to look beyond this recipe.

In the context of conformal prediction, recent work by kiyani2025decision (their Theorem 2.3) shows a result similar to (ref).\footnote {We developed (ref) independently of their result. } To connect these results, let $\vartheta$ be a random prediction target and consider $H$ the joint distribution over $ (\vartheta, X)$. In a prediction context, $\vartheta$ is the unknown true label of a unit and $X$ is the observed features. A conformal prediction set $\hat \Theta (X)$ satisfies $H(\vartheta \in \hat\Theta(X)) \ge 1-\alpha$, where the probability is taken over $(\vartheta, X) \sim H$. kiyani2025decision show that the following optimization problems are equivalent, with essentially the same argument as that underlying (ref)

In contrast, we consider a parameter inference setting, where probabilities are solely taken over $Y$ fixing $\theta$, rather than jointly over $ (X, \vartheta)$. We also consider evaluations of the certificate $R(\cdot)$ beyond its expectation, using a weaker preference ordering ((ref)).

Example: inference on winners, continued

(ref) establishes a strong sense in which as-if optimization is reasonable. Dominance ordering is demanding, however, and there are many $(R,\delta)$, $(\tilde{R},\tilde{\delta})$ pairs that cannot be ordered, in the sense that neither $R(Y)$ nor $\tilde{R}(Y)$ (weakly) stochastically dominates the other. Consequently, weak dominance in the sense of (ref) provides only a partial order over the class of certified decisions, and does not generally provide clear recommendations for “optimal” procedures. This partial order is nevertheless sufficient to rule out some methods discussed in the literature.

To illustrate, let us return to the example, introduced in (ref), of an analyst who selects a recommended action via empirical welfare maximization, $\delta(Y)=\argmax_{a\in\mathcal{A}}X(a),$ and forms P-certificate $R(Y)$ or $R^*(Y)$ using unstudentized or studentized projection, respectively. The certified decision $(\delta,R)$ can be obtained via as-if optimization using the confidence set \[ \hat\Theta(Y)=\br{\theta:\theta(a)\ge X(a)-c_{1-\alpha} \text{ for all }a\in\mathcal{A}}, \] and so is easily cast into the essentially complete class constructed in (ref).

By contrast, $(\delta,R^*)$ is less naturally understood via as-if optimization, since $R^*$ corresponds to the studentized confidence set \[ \hat\Theta^*(Y)=\br{\theta:\theta(a)\ge X(a)-\sigma(a)c^*_{1-\alpha}\text{ for all }a\in\mathcal{A}}.\] If the analyst conducts as-if optimization over $\hat\Theta^* (Y)$, they recommend decision $\tilde\delta(Y)=\argmax_{a\in\mathcal{A}}\br{X(a)-\sigma (a)c^*_{1-\alpha}},$ which is a special case of the risk-aware empirical maximization approach proposed by chernozhukov2025policylearningconfidence and can differ from $\delta(Y)$ when $\sigma(\cdot)$ is non-constant. The corresponding risk bound $\tilde{R}(Y)=1-\max_{a\in\mathcal{A}}\br{X(a)-\sigma(a)c^*_{1-\alpha}}$ is no larger than $R^*(Y)$ in every realization of $Y$, so $(\tilde\delta,\tilde R)$ weakly dominates $(\delta,R^*)$.\footnote{Formally, however, if $\Theta=[0,1]^\mathcal{A}$ then $ (\delta,R^*)$ still belongs to the essentially complete class we study, since it results from as-if optimization against the confidence set $\hat\Theta^{**}(Y)=\br {\theta\in\Theta:L(\delta(Y),\theta)\le 1-(X(\delta(Y))-\sigma(\delta(Y))c^*_ {1-\alpha})}.$ Note, however, that $\hat\Theta^{**}(Y)$ is a super-set of $\hat\Theta^* (Y),$ which helps explain the superior performance of $(\tilde\delta,\tilde {R})$ relative to $(\delta,R^*)$.}

Optimality under monotonicity

The non-uniqueness of optimal certified decisions, in the sense of weak dominance, is intuitive: (ref) shows that P-certificates are closely linked to confidence sets, and in most problems the class of admissible (i.e. undominated) confidence sets is large. If we narrow attention to settings where optimal (uniformly most accurate) confidence sets exist, one might hope that optimal P-certified decisions will exist as well and are found by as-if optimization. The next theorem shows that hope is borne out, provided the loss is decreasing in the parameter and we limit our comparison of stochastic dominance to risk values exceeding $ \inf_{a\in\mathcal{A}}L(a,\theta)$.\footnote{Absent the latter restriction, it is impossible to dominate as-if optimization against the trivial confidence set which takes $\hat\Theta(Y)$ empty with probability $\alpha,$ and equal to $\Theta$ with probability $1-\alpha$.}

To state the result, we recall the definition uniformly most accurate confidence bounds:

Example: choosing a treatment proportion

(ref) does not apply to the inference on winners problem since the parameter of interest $\theta$ is vector-valued, and there does not in general exist a uniformly most accurate confidence set. Consider instead a setting where the decision-maker is considering a single treatment, for instance enrollment in a job training program, and must decide what fraction $a\in[\varepsilon,1]$ of a population randomize into treatment. Below $a=\varepsilon>0$ they treat no one, $a_0=0.$ The average outcome under treatment is $\theta,$ while the (known) average outcome under control is $\rho$. There is again a fixed cost $\kappa$ of treating anyone, and there is also a variable cost $\psi(a)$ of treating fraction $a>0$ of the population where we assume $\psi(a)$ is continuous and increasing, reflecting e.g. the increasing cost of hiring large numbers of qualified instructors. If the decision-maker would like to maximize the average outcome in the population, net of costs, then we can represent their preferences with loss function $L(a,\theta)=a(1-\theta)+\psi(a)$ for $a\in\mathcal{A},$ along with loss $C=(1-\rho)-\kappa$ for the default action $a_0=0.$ Note that the loss is decreasing in $\theta$ for all $a$ by construction, and $\inf_{a\in\mathcal{A}} L(a,\theta)$ is attained for all $\theta.$

Suppose the analyst observes a normally distributed estimate $X\sim N(\theta,\sigma^2)$ for the average outcome under the treatment, where $\sigma$ is fixed and known to the analyst and $Y=(X,\sigma)$. The uniformly most accurate level 1-$\alpha$ lower confidence bound for $\theta$ is $\hat\theta(Y)=X+\sigma z_{\alpha},$ for $z_\alpha$ the $\alpha$-quantile of a standard normal distribution. By (ref), the optimal P-certified decision in this example takes $\delta(Y)=\argmin_{a\in \mathcal{A}}L(a,\hat\theta(Y))$ and $R(Y)=(\delta(Y),\hat\theta(Y))$. If the decision-maker then implements the recommended treatment when $R(Y)\le C,$ this corresponds exactly to implementing treatment (for some portion of the population) when the analyst is able to reject that $\theta \le \bar\theta$ for a threshold $\bar\theta$ such that $\inf_{a}R(a,\bar\theta)=C.$

Extension: E-certificates

We have shown that P-certified decisions offer a natural route to combine (frequentist) statistical inference with decision-making. However, high-probability guarantees, as appearing in confidence intervals and hypothesis tests, are not the only type of frequentist guarantee available. A recent literature in statistics proposes e-values as an alternative and a complement to traditional statistical inference ramdas2024hypothesis. An e-variable against the parameter value $\theta_0$ is a nonnegative random variable $E(Y, \theta_0)$ such that \[ \E_P[E(Y, \theta_0)] \le 1 \text{ for all $P$ such that $(\theta_0, P) \in \mathcal M$}. \] One can interpret $E(\cdot, \theta_0)$ as the realized dollar payoff of a bet, priced at \$1, against the hypothesis $H_0: \theta = \theta_0$. Since the expected payoff under $H_0$ is less than \$1, this bet is (at most) fair. Correspondingly, a large realization of $E (Y, \theta_0)$ can be interpreted as evidence against $H_0$---in the sense that it is an unlikely event under $H_0$---much like a small p-value is evidence against $H_0$.

As with confidence sets, we can use e-variables to certify the quality of decisions. Throughout this section, we assume $L(a,\theta) > 0$, but no longer assume $L(a, \theta) \le 1.$

As (ref) highlights, $R(Y)$ is a stochastic upper bound for loss in the sense that the ratio $L(\delta(Y), \theta)/R(Y)$ has ex ante expectation bounded below some prespecified $\gamma$.

Similar to our previous results, one can derive E-certified decisions using a version of as-if optimization, now defined using e-variables. grunwald2023posterior proposes what he terms the E-posterior: if there exists an e-variable $E(\cdot, \theta)$ for every $\theta \in \Theta$, the E-posterior is defined as $1/E(Y, \theta)$, with the convention $1/0 = \infty$. grunwald2023posterior proposes computing the action that minimizes the worst-case E-posterior-weighted loss, along with the minimized value. Formally, since minimization over $\mathcal A$ might not be attained by any element in $\mathcal A$, let $\mathcal C_ {E,\epsilon}$ be the following class of $(\delta, R)$ such that, for some collection of e-variables $E(\cdot, \theta)$,

Intuitively, for choosing $\delta(\cdot)$, states $\theta$ deemed implausible by the data---those with large $E (Y,\theta)$---have their losses downweighted, while those with small $E (Y,\theta)$ have their loss upweighted. grunwald2023posterior shows that $(\delta(Y), R(Y))$ is an E-certified decision at multiple 1.\footnote{Here, our setup is slightly different since the minimization is up to a constant $\epsilon$. Nevertheless, observe that since $R(Y) \ge L(\delta(Y), \theta) / E(Y,\theta)$, \[ \E_P[L(\delta(Y), \theta) / R(Y)] \le E_P[E(Y,\theta)] \le 1. \]}

We next show that, as with P-certificates in (ref), it is without loss to limit attention to as-if optimization, since the set of E-certified decisions of the form (ref) is an essentially complete class with respect to the dominance order.

Like our results for P-certificates, using E-certificates also provide guarantees when combined with downstream decision-making. For a decision-maker who never adopts when $R(Y) > C$, the decision-maker at most doubles the cost of the default action $C$. Unlike our results for P-certificates, such a guarantee does not require that the loss function be bounded above.

Motivated by (ref), if we only sought certificates with $\E_P\left[\max\pr{\frac{L (\delta (Y),\theta)} {R (Y)}, 1}\right] \le 1+\gamma$ to start with, $R(Y)$ can often be improved by slightly modifying the E-posterior. We detail this improvement in (ref).

Conclusion

This paper considers combining statistical inference with statistical decisions. In two leading modes of frequentist inference, statistical inference translates to certified decisions---decisions paired with certificates of their performance. We show that these certified decisions provide ambiguity-averse downstream decision-makers with useful risk guarantees and simple adoption decision rules---which may be implemented and understood without statistical sophistication on part of the decision-maker. Moreover, we show that it is without loss to base certified decisions on statistical inference procedures, and thus that inferential and decision goals are, at the very least, partially aligned.