The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
104,666 characters
Conformal Policy Learning with Distribution-Free Safety Guarantees
\maketitle
\begin{abstract}
Policy learning aims to determine who should be treated based on individual characteristics. In high-stakes settings such as medicine and public policy where safety is a central concern, improving the average outcomes alone may not be sufficient: decision makers may also seek to protect individuals from harm, in line with the Hippocratic principle of ``do no harm.''
In this paper, we propose \textit{conformal policy learning} (CPL), a policy learning procedure with a new distribution-free safety guarantee that controls the probability of assigning treatment to an individual who would be harmed relative to control. CPL views each treatment decision as testing a hypothesis of counterfactual harm and assigns treatment by thresholding conformal p-values. These p-values use observable proxies and selective calibration to address the challenge that the potential outcomes under comparison are never simultaneously observed. For randomized experiments, under standard exchangeability conditions, CPL provides finite-sample safety guarantee at a user-specified level, without imposing any outcome modeling assumptions. Moreover, when the outcome model is consistently estimated, CPL achieves asymptotically optimal welfare subject to the safety constraint.
In observational studies, CPL with learn-then-balance weights achieves doubly robust safety guarantees. We evaluate CPL through extensive simulations and apply it to an empirical study of AI-powered interventions designed to reduce conspiracy beliefs.
\end{abstract}
\keywords{AI safety, Causal inference, Conformal prediction, Policy learning}
\section{Introduction}
Policy learning, also known as the treatment choice problem, aims to learn a rule that automatically assigns future treatment options based on individual characteristics~\citep{manski2004statistical,hirano2009asymptotics,kitagawa2018should,athey2021policy}.
It has been the foundation for data-driven decision-making in various domains spanning precision medicine~\citep{murphy2003optimal,qian2011performance,zhao2012estimating}, online advertising and recommendation systems~\citep{li2010contextual,dudik2011doubly}, political campaigns~\citep{imai2011estimation}, and criminal justice~\citep{kleinberg2018human}, among others.
Policy learning methods often aim to maximize the average welfare (expectation of the realized outcome) within a policy class~\citep{manski2004statistical,kitagawa2018should,athey2021policy}.
While welfare maximization is widely useful, it can be insufficient in high-stakes domains such as medicine and public policy where individual safety is of concern. In such applications, decision makers may seek not only on-average improvement of the outcomes, but also controlling the number of individuals harmed by the intervention~\citep{gadbury2004individual,kallus2022s, richens2022counterfactual, ben2025safe}, i.e., ``do no harm.''
As an example, suppose a new intervention benefits $60\%$ of a group while harming the remaining $40\%$ by the same magnitude.
A decision-maker who maximizes average welfare would decide to treat the group; this would harm 40\% of the population, which can be unacceptable if individual safety is of primary concern.
In this paper, we study policy learning with a distribution-free safety guarantee. Formally, assume access to observed data $\{(X_i, T_i, Y_i)\}_{i=1}^n$ where $Y_i\in \{0,1\}$ is a binary outcome, $T_i\in\{0,1\}$ is the binary treatment, and $X_i\in \cX$ is the observed features for each unit $i$. Under the potential outcome framework with SUTVA (formalized in Section~\ref{subsec:setup}), the outcome is $Y_i=Y_i(T_i)$, where $(Y_i(1),Y_i(0))$ are the potential outcomes under treatment and control, respectively.
For a new test point with observed features $X_{n+1}$ and unknown potential outcomes $(Y_{n+1}(0),Y_{n+1}(1))$, our goal is to learn a policy $\hat\pi: \mathcal{X} \rightarrow \{0, 1\}$ that maps features to treatment assignments with the following safety guarantee: for a pre-specified level $\alpha\in (0,1)$,
\begin{equation}\label{eq:def_safety_intro}
\PP\bigl(Y_{n+1}(\hat\pi(X_{n+1})) < Y_{n+1}(0) \bigr) \leq \alpha.
\end{equation}
This safety guarantee~\eqref{eq:def_safety_intro} means the probability of the realized outcome being worse than the ``status-quo'' outcome under control is no greater than $\alpha$, thereby limiting the risk of harming the new individual. When this policy is implemented on $m$ new individuals,~\eqref{eq:def_safety_intro} implies that the expected number of units harmed by the treatment is no greater than $m \alpha$.
Such guarantees are important for two related reasons. First, negative treatment effects are substantively costly in many settings. Second, treatment rules are increasingly embedded in (semi-)automated decision systems: once a learned rule is deployed, it may assign interventions repeatedly and at scale. In such settings, trust in the system requires explicit control of the probability that it harms the individuals it chooses to treat.
We highlight three applications that motivate such explicit harm-rate control.
\vspace{-0.5em}
\paragraph{Example 1 (AI-powered Interventions):} Generative AI is emerging as a new class of intervention in the social sciences, with applications designed to change attitudes and behaviors through scalable, personalized interactions \citep[e.g.,][]{costello2024durably, bai2025llm}. At the same time, recent empirical studies highlight an important risk: while such AI interventions may benefit many individuals and tasks, they may also harm others. For example, a randomized experiment in a global consulting firm \citep{dell2023navigating} found that access to AI can harm the productivity of high-skilled consultants when working on challenging intellectual tasks. Similarly, \cite{bastani2025generative} found that generative AI without guardrails can harm the learning of high school math students. As interventions are powered by black-box AI, controlling the harm is fundamental for safety, public trust, and efficient deployment of AI-powered treatment.
\vspace{-0.5em}
\paragraph{Example 2 (Precision Medicine):} Precision medicine---prevention and treatment strategies that account for individual variability---is central in modern medicine \citep{kosorok2019precision}. The problem of controlling individual risk while maximizing benefit has long been recognized.
It is especially relevant when efficacious medications may also lead to a higher risk for certain individuals, such as opioid treatment of chronic pain~\citep{laber2018identifying} and type-2 diabetes with insulin therapies~\citep{wang2018learning}.
\vspace{-0.5em}
\paragraph{Example 3 (Criminal Justice):} In the US criminal justice system, how to safely use risk scores to evaluate an individual's likelihood of reoffending and identify their criminogenic needs is a major topic of interest \citep{skeem2020impact}. For example, \cite{ben2025safe} analyze a field experiment to estimate the causal effect of algorithmic recommendations on judges' decisions at a criminal first appearance hearing. Here, a treatment rule with the safety guarantee can prevent arrestees from committing a new crime or failing to appear in court, while avoiding unnecessarily harsh decisions.
\vspace{0.5em}
We aim to achieve the safety guarantee in a model-agnostic fashion---meaning that it holds without strong modeling assumptions on the data distribution or the learning algorithms---and tightly in finite samples, so this framework is widely applicable to various high-stakes settings.
\subsection{Overview of contributions}
We develop \textit{conformal policy learning} (CPL) to achieve the safety guarantee~\eqref{eq:def_safety_intro}.
Distinct from standard policy learning methods that maximizes empirical welfare within a policy class, our starting point is to view each treatment decision as testing a counterfactual harm event $Y_{n+1}(1) < Y_{n+1}(0)$. A p-value \(p_{n+1}\) satisfying
\[
\mathbb P\bigl(Y_{n+1}(1)<Y_{n+1}(0),\,p_{n+1}\leq \alpha\bigr)\leq \alpha
\]
yields the desired safety guarantee by assigning treatment only when \(p_{n+1}\leq\alpha\).
This connects safe policy learning to conformal inference~\citep{vovk2005algorithmic, lei2021conformal, jin2023selection}.
The key challenge here---which necessitates novel constructions of powerful conformal p-values---is that the ``harm'' event involves both potential outcomes and is therefore unobserved even for the labeled data.
CPL addresses this difficulty through two techniques. First, it uses observable proxy labels to construct valid conformal p-values. Second, to reduce conservativeness, it selects the labeled observations used in p-value calibration according to which treatment arm provides a sharper proxy for harm. We call this strategy ``selective calibration.''
The resulting p-value compares the test conformity score with the selected calibration scores computed using the proxy labels. The CPL workflow is in Figure~\ref{fig:intro}.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figs/intro_v2.pdf}
\caption{\textbf{Workflow of Conformal Policy Learning (CPL).} Given labeled data $\{(X_i,T_i,Y_i)\}_{i=1}^n$, the goal is to assign safe treatment for a new test unit $X_{n+1}$ with harm rate target $\alpha\in(0,1)$. CPL constructs proxy label $Y_i^\dagger$ and selects a subset of data for calibration. It then computes a p-value by comparing a test conformity score to the calibration scores. The treatment is determined by whether the conformal p-value is below $\alpha$, which controls the harm rate below $\alpha$.}
\label{fig:intro}
\end{figure}
For randomized experiments, CPL provides finite-sample, distribution-free safety guarantees. Under the exchangeability conditions ensured by random assignment, we show that CPL controls the harm rate exactly below $\alpha$ in finite samples, without making any assumption about the outcome models used in the conformal p-values and the selective calibration rule.
We first establish this result for balanced randomized experiments and then extend it to stratified experiments using weighted conformal inference~\citep{TibshiraniBCR19}.
We next study the sharpness and optimality of CPL. Because individual-level harm, \(Y(1)<Y(0)\), is not identifiable from observed data, we formulate the optimality under partial identification~\citep{kallus2022s,li2023trustworthy}. We characterize the optimal policy that maximizes the power (probability of treatment) and average welfare subject to the safety constraint for every joint potential-outcome distribution compatible with the observed data.
Then, we show that as long as the score function used in the p-values and selective calibration rule converge to oracle ones, CPL attains these global optima asymptotically.
We further extend CPL to observational studies. In this setting, the treatment assignment mechanism is unknown, and the calibration weights must be estimated.
We develop a learn-then-balance procedure that estimates these weights and enforces balance on functions tailored to the thresholding decisions and the partially identified harm target. Under suitable regularity conditions, the resulting policy has a doubly robust asymptotic safety guarantee: the excess harm rate vanishes when either the propensity-score model or the outcome models, but not necessarily both, are consistently estimated.
Moreover, if both components converge at standard slow nonparametric rates, the excess harm rate is of a parametric order.
With consistent outcome models, CPL with observational data attains the same welfare and power optima as in the randomized case.
Our excess harm rate bound does not pay the price of policy class complexity common in policy learning.
Finally, we evaluate CPL through simulations and an empirical study. In simulations across randomized experiments, stratified experiments, and observational studies, CPL shows robust harm-rate control and high power and welfare. In the empirical study, CPL assigns an AI-powered intervention designed to reduce conspiracy beliefs while tightly controlling the harm rate.
The rest of the paper is organized as follows.
Section~\ref{sec:setup} introduces the problem setup and connects the safety guarantee to conformal hypothesis testing.
Section~\ref{sec:rct} develops CPL for balanced randomized experiments and studies its finite-sample validity and asymptotic optimality; the framework is then extended to stratified experiments in
Section~\ref{sec:stratified} and observational studies in Section~\ref{sec:obs}.
Section~\ref{sec:sim} presents simulation studies, and Section~\ref{sec:real} applies CPL to an empirical study of AI-powered interventions.
We close the paper with a discussion on extensions and future directions in Section~\ref{sec:ext}.
\subsection{Related Work}
This article lies in the intersection of causal inference, policy learning, and conformal inference. We summarize several important lines of related work below.
Our work is connected to the established literature on policy learning \citep[e.g.,][]{murphy2003optimal, manski2004statistical, hirano2009asymptotics, zhao2012estimating, kitagawa2018should, athey2021policy, jin2025policy}, which aims to select, among a given class of policies, the one that maximizes an objective (such as average welfare) while optionally respecting a constraint (such as harm rate).
Within this literature, this work is closely related to a small but emerging literature on policy learning with safety considerations (where the exact meaning of safety varies). One line of work develops safe policy learning algorithms in settings that necessitate extrapolating beyond the observed labeled data, and the safety refers to not performing worse than a ``status quo'' policy~\citep{zhang2022safe, ben2025safe, jia2025bayesian, wu2025safe}. Our safety notion of individual harm is conceptually related since it measures the harm relative to the status quo of no treatment, but distinct enough to yield completely different techniques.
In addition, \cite{ben2024policy} studies policy learning when the objective involves counterfactuals, providing doubly robust algorithms and regret bounds for the learned policy; the dual form of a special case in their framework (with an unknown Lagrange parameter) coincides with the welfare maximization problem subject to harm rate control; this connects with our setting and the method of~\cite{li2023trustworthy}.
As standard in policy learning, these methods select a policy that maximizes an empirical objective within a policy class, whose performance is often measured by the regret (the gap between the true objective of the learned policy from the optimal), which is typically bounded by a statistical error term that scales with the complexity of the policy class (such as the VC-dimension). In contrast, CPL leverages conformal inference to achieve finite-sample, distribution-free safety guarantee (without a high-probability excess error bound term) when propensity scores are known; moreover, when propensity scores are unknown and estimated so inexact harm control is inevitable, our excess harm rate bound does not pay the price of the policy class complexity.
Our safety notion follows from a literature in causal inference and policy learning that bounds, estimates, and controls the same harm rate notion as~\eqref{eq:safety_guarantee}. A line of work studies its bounds under various assumptions such as monotonicity~\citep{huang2012assessing} and certain conditional independence conditions~\citep{shen2013treatment,yin2018assessing}. Relatedly, due to the non-identifiability, several work focuses on establishing bounds on the conditional harm rate for binary outcomes, including~\cite{gadbury2004individual} without covariates, \cite{zhang2013assessing} with covariates, \cite{kallus2022s} on the sharp identification bounds, and~\cite{wu2024quantifying} using a sensitivity model for the correlation between the potential outcomes. In addition, a recent independent work of~\cite{scauda2026counterfactual} studies the population-level optimal welfare subject to harm rate control. Since we aim to control the harm rate without strong modeling assumptions, this work is implicitly tied to the Frech\'et--Hoeffding bound characterized in~\cite{zhang2013assessing,kallus2022s}.
We show CPL attains the optimal welfare subject to safety guarantee among the worst-case distributions in these work.
Our method builds on the conformal p-values proposed in~\cite{jin2023selection} for i.i.d.~data and~\cite{jin2023model} for covariate shift settings, which were extended to model selection~\citep{bai2024optimized} and online settings~\citep{xu2024online}. In that literature, the p-values quantify the confidence in a large, \emph{ordinary} outcome (i.e., no potential outcomes) exceeding a \emph{known} threshold.
While we borrow the high-level intuitions, the technical route in constructing such p-values for our problem---which is the key contribution here---is sharply distinct since the two outcomes under comparison are never simultaneously observed. The resulting optimality and robustness properties are likewise quite different.
The conformal inference approach also connects our work with a line of work on conformal inference for individual treatment effects (ITE) $Y_{n+1}(1)-Y_{n+1}(0)$~\citep{lei2021conformal,jin2023sensitivity,yin2024conformal}. These work typically focuses on constructing a prediction set for the counterfactual outcome for a unit who has already received treatment or control, where inference for the ITE of a new test point with two unknown outcomes appears particularly challenging. The latter case (with binary outcomes) is the setting we address, and we study the ``decision'' problem rather than prediction set construction. This involves a thresholding decision rule which necessitates distinct calibration and theoretical analysis techniques.
\section{Problem Setup and Conceptual Framework}
\label{sec:setup}
\subsection{Problem Setup}
\label{subsec:setup}
We assume access to a set of labeled data $\{(X_i,T_i,Y_i)\}_{i=1}^n$, where $X_i\in \cX$ is the feature, $T_i\in\{0,1\}$ is the treatment assignment, and $Y_i\in \{0,1\}$ is the binary outcome. We define the potential outcomes $Y_i(t)$ for $t \in \{0, 1\}$ and assume the triplets $\{(X_i,Y_i(1),Y_i(0))\}_{i=1}^n$ are independent and identically distributed (i.i.d.) from an unknown super-population $\PP_{X,Y(1),Y(0)}$.
Assuming the Stable Unit Treatment Value Assumption (SUTVA) \citep{rubin1980randomization}, the observed outcome is given by $Y_i=Y_i(T_i)$. The joint super-population $\PP_{X,Y(1),Y(0)}$
is unidentifiable from data because researchers observe only one potential outcome for each unit~\citep{holland1986statistics}.
Finally, the distribution of the labeled data $\{(X_i,T_i,Y_i)\}_{i=1}^n$ is induced by the unknown super-population and the treatment assignment mechanism and denoted as $\PP_{X,Y,T}$.
Throughout, we make the unconfoundedness assumption for the treatment assignment mechanism, which is standard in the causal inference literature~\citep{imbens2015causal}.
\begin{assumption}[Unconfoundedness]\label{assump:uncon}
The treatment assignments $\{T_i\}_{i=1}^n$ are mutually independent and obey $T_i\indep (Y_i(0),Y_i(1))\given X_i$, with $\PP(T_i=1\given X_i=x) \coloneq e(x)$, and we call $e(x)\in (0,1)$ the propensity score.
\end{assumption}
Our framework covers both randomized experiments (Sections~\ref{sec:rct} and~\ref{sec:stratified}) and observational studies (Section~\ref{sec:obs}). In randomized experiments, Assumption~\ref{assump:uncon} is satisfied by design and $e(x)$ is known. In observational studies, Assumption~\ref{assump:uncon} should be evaluated based on domain knowledge; even though it holds, the propensity score $e(x)$ is unknown and needs to be estimated.
Policy learning uses the labeled data to determine the treatment for a new individual (which we call the test unit/point) with observed feature $X_{n+1}$ and unobserved potential outcomes $Y_{n+1}(1)$ and $Y_{n+1}(0)$.
Following the literature~\citep[e.g.,][]{manski2004statistical, hirano2009asymptotics, zhao2012estimating, kitagawa2018should, athey2021policy}, we assume it is from the same super-population independently of the labeled data.
\begin{assumption}[IID]\label{assump:iid}
The test point is drawn from $(X_{n+1},Y_{n+1}(1),Y_{n+1}(0))\sim \PP_{X,Y(1),Y(0)}$ independently of the labeled data.
\end{assumption}
Our framework can be naturally extended to relax the i.i.d.~assumption and allow for covariate shift between the labeled data $i \in \{1, \ldots, n\}$ and the test point; see Section~\ref{sec:ext} for a discussion.
\subsection{Harm Rate}
\label{subsec:harm_rate_discuss}
Recall that our goal is to develop a policy learning algorithm producing a rule $\hat\pi: \cX \rightarrow \{0,1\}$ that maps features $X$ to treatment assignment $\hat\pi(X)$ with the \textit{counterfactual} safety guarantee:
\@\label{eq:safety_guarantee}
\PP\big(Y_{n+1}(\hat\pi(X_{n+1})) < Y_{n+1}(0)\big) \leq \alpha
\@
for a pre-specified confidence level $\alpha\in (0,1)$.
Here, the probability is over the labeled data (based on which $\hat\pi$ is built) and the test point. We call the left-handed side probability in~\eqref{eq:safety_guarantee} the ``harm rate'' following~\cite{zhang2013assessing}; the same quantity is also referred to as the ``fraction of negatively affected'' in the literature~\citep{kallus2022s,li2023trustworthy,wu2024quantifying}.
Throughout, we focus on binary outcomes; this setting is most extensively studied~\citep{zhang2013assessing,kallus2022s,wu2024quantifying}.
The safety guarantees can be rewritten as
\@
\PP\big(Y_{n+1}(\hat\pi(X_{n+1})) < Y_{n+1}(0)\big) \leq \alpha ~~ \Longleftrightarrow ~~ \PP\big(Y_{n+1}(1) < Y_{n+1}(0), ~\hat\pi(X_{n+1}) = 1 \big) \leq \alpha. \label{eq:eqivalence}
\@
Thus, harm occurs when the individual treatment effect on the test unit is negative and one decides to treat the unit. Thus, a trivial way to achieve this is to treat the test unit with probability $\alpha$; however, such a policy is clearly suboptimal due to limited welfare and power (whose meaning will be made precise later). Instead, we aim for a procedure that achieves this with reasonable, and sometimes optimal, welfare and power.
Meanwhile, the above result shows that to control the harm rate, we should detect and avoid treating units that can be harmed by the treatment. This is the intuition for CPL.
\subsection{Connecting Safety Guarantees to Hypothesis Testing}
Our technical route differs from standard policy learning approaches, so it is helpful to begin with the conceptual framework: connect the safety guarantees with hypothesis testing.
Equation~\eqref{eq:eqivalence} suggest that to make safe treatments, one needs to identify units that are likely \emph{not to} be harmed by the treatment, i.e., $Y_{n+1}(1)\geq Y_{n+1}(0)$.
We view this as testing a random null hypothesis $H_0\colon Y_{n+1}(1) < Y_{n+1}(0)$.
Suppose we can construct a p-value $p_{n+1}$ such that
\@\label{eq:p-value}
\PP\big(Y_{n+1}(1) < Y_{n+1}(0), ~ p_{n+1} \leq \alpha \big) \leq \alpha.
\@
This is a non-conventional definition of p-values because the truth of $H_0$ is itself random, but this notion is sufficient for the desired safety guarantee:
setting $\hat\pi(X_{n+1}) = \ind\{p_{n+1} \leq \alpha\}$, the validity~\eqref{eq:p-value} directly implies the safety guarantee~\eqref{eq:safety_guarantee} via the equivalence relationship in~\eqref{eq:eqivalence}. Intuitively,
a small p-value informs strong evidence against the null, i.e., the test unit is unlikely to be harmed and thus can safely receive the treatment.
The remaining of the paper focuses on addressing two questions. First, how can we construct p-values that satisfy~\eqref{eq:p-value} in finite sample without making strong modeling assumptions? We address this question by expanding the recent literature of conformal p-values \citep{vovk2005algorithmic, bates2021testing, jin2023selection} to causal inference contexts~\citep{imbens2015causal,lei2021conformal}, while respecting the partial-identification nature of
the individual-level causal effects~\citep[e.g.,][]{heckman1997making}. Second, is thresholding p-values optimal in any sense? This is important as it is not clear a priori why our approach may be desired, even if it might achieve the safety guarantee. We will show that with specifically designed p-values, our method achieves optimal power and welfare among all safe policies at the population level.
\section{Conformal Policy Learning with Balanced Randomized Experiments}
\label{sec:rct}
To fix ideas, in this section, we introduce our framework in the simple setting of balanced randomized experiments with $e(x)=1/2$; this will be gradually generalized in Sections~\ref{sec:stratified} and~\ref{sec:obs}.
In Section~\ref{subsec:pre}, we start with a single-arm construction that provides distribution-free safety guarantees. In Section~\ref{subsec:cpl_rct}, we introduce observable proxies and selective calibration that sharpens the procedure while preserving the same guarantee. In Section~\ref{subsec:rct_optimality}, we characterize the welfare-optimal policy under partial identification, followed by an explanation on the sharpness of selective calibration in Section~\ref{subsec:discuss_sharp}.
Throughout this section, all learned functions are fitted on a training sample independent of the labeled observations used for calibration and the test unit. In practice, this can be achieved by sample splitting~\citep{lei2018distribution}. For notational simplicity, we use \(\{(X_i,T_i,Y_i)\}_{i=1}^n\) to denote the labeled observations available for calibration; selective calibration introduced in Section~\ref{subsec:cpl_rct} may retain only a subset of these observations. Conditional on the training sample, the learned functions are treated as fixed.
\subsection{Warm-up: Direct Application of Existing Conformal p-values}
\label{subsec:pre}
We begin with a simple adaptation of the existing idea to show how conformal p-values can serve as the foundation for safe policy learning. Given that we focus on binary outcomes, we can rewrite~\eqref{eq:p-value} as finding a p-value $p_{n+1}$ obeying
\@\label{eq:safety_equiv_bin}
\PP\big( Y_{n+1}(1) = 0, Y_{n+1}(0) = 1,~ p_{n+1} \leq \alpha \big) \leq \alpha.
\@
That is, we would like to find p-values that quantify the confidence in a small treated outcome \emph{and} a large control outcome.
The conformal selection (CS) framework~\citep{jin2023selection} provides a natural starting point. Given labeled data $\{(X_i,Y_i)\}_{i=1}^n$ for an ordinary outcome $Y\in \RR$ (i.e., no potential-outcome framing) and a known threshold $c\in \RR$, CS proposes conformal p-values with a similar null property $\PP(Y_{n+1}\leq c, p_{n+1}\leq \alpha)\leq \alpha$.
Setting aside the issue that we now have two potential outcomes to deal with, one simple idea is to
develop a conservative p-value $p^{(1)}_{n+1}$ that satisfy
\@\label{eq:safety_equiv_bin_1}
\PP\big(Y_{n+1}(1) = 0, ~ p^{(1)}_{n+1} \leq \alpha \big) \leq \alpha,
\@
which directly implies~\eqref{eq:safety_equiv_bin}. For treated outcome, it means we will use the treated units for calibrating p-values.
The conformal p-value from CS takes the form
\@\label{eq:cs_pval}
p_{n+1}^{(1)} = \frac{1+\sum_{i=1}^n T_i \cdot \ind\{V(X_i,Y_i)\leq V(X_{n+1},0)\}}{1 + \sum_{i=1}^n T_i},
\@
where $V$ is any score function $V\colon \cX\times\cY\to \RR$ obeying $V(x,y)\leq V(x,y')$ whenever $y\leq y'$ for any $x\in \cX$.
\paragraph{Example (clipped score function).}
One example is the clipped score function~\citep{jin2023selection}, $V(x,y) = M\ind\{y>0\} + (1 - \hat{\mu}_1(x))$, where $\hat{\mu}_1(x)$ is an estimator for the conditional expectation function $\E[Y_i \mid T_i = 1, X_i = x]$ and $M>2\sup_x|\hat\mu_1(x)|$ is a sufficiently large constant so that~\eqref{eq:cs_pval} reduces to
\@\label{eq:cs_pval_clip}
p_{n+1}^{(1)} = \frac{1+\sum_{i=1}^n T_i \cdot \ind\{Y_i= 0\} \cdot \ind\{ \hat{\mu}_1(X_i)\geq \hat{\mu}_1(X_{n+1})\}}{1 + \sum_{i=1}^n T_i}.
\@
Intuitively, this p-value focuses on potentially unsafe cases (i.e., $Y_i(1) = 0$) among treated units and examines whether the test unit is extreme with respect to $\hat{\mu}_1(x)$. When this p-value is small, the test unit is unlikely to come from the distribution of potentially unsafe units, and thus is likely to be safe. \qed \\
To see why the conformal p-value~\eqref{eq:cs_pval} satisfies~\eqref{eq:safety_equiv_bin_1} for any score function,
we outline the theoretical arguments from~\cite{jin2023selection}.
Let $n_1 = |\cI_1|$ be the number of treated units, where $\cI_1 = \{i\in [n]\colon T_i=1\}$.
Consider the ``oracle'' p-value
\@\label{eq:cs_pval_oracle}
p_{n+1}^{\ast (1)} = \frac{1+\sum_{i=1}^n T_i \cdot \ind\{V(X_i,Y_i(1))\leq V(X_{n+1},Y_{n+1}(1))\}}{1 + n_1}
\@
which is not computable since $Y_{n+1}(1)$ is unobserved. The only difference between~\eqref{eq:cs_pval} and~\eqref{eq:cs_pval_oracle} is that $Y_{n+1}(1)$ in the oracle p-value is replaced by $0$, and we have $Y_i(1)=Y_i$ for $i\in \cI_1$ in~\eqref{eq:cs_pval_oracle}. First, conditional on $\{T_i\}_{i=1}^n$, in this randomized experiment, the data $\{(X_i,Y_i(1)\}_{i\in \cI_1} \cup\{X_{n+1},Y_{n+1}(1)\}$ are exchangeable, which implies $\PP(p_{n+1}^{*(1)}\leq \alpha)\leq \alpha$~\citep{vovk2005algorithmic}.
Second, on the event that $Y_{n+1}(1)=0$, the two p-values coincide.
The two facts imply $\PP(p_{n+1}^{(1)}\leq \alpha,~Y_{n+1}(1)=0) = \PP(p_{n+1}^{*(1)}\leq \alpha,~Y_{n+1}(1)=0)\leq \PP(p_{n+1}^{*(1)}\leq \alpha) \leq \alpha$.
Of course, one may also leverage the control samples and consider the null hypotheses $H_0^{(0)}\colon Y_{n+1}(0)=1$. We can similarly construct the p-value (using a monotone function $V$ as before)
\@\label{eq:cs_pval_0}
p_{n+1}^{(0)} = \frac{1+\sum_{i=1}^n \ind\{T_i = 0\} \cdot \ind\{V(X_i,1-Y_i)\leq V(X_{n+1},0)\}}{1 + \sum_{i=1}^n \ind\{T_i = 0\}}.
\@
Following exactly the same rationales as above, this p-value is valid in the same sense.
\begin{comment}
\begin{table}[t]
\centering
\footnotesize
\begin{minipage}[t]{0.48\textwidth}
\centering
\[
\begin{array}{c|c|cc|cc}
\text{Group} & T & Y(1) & Y(0) & Y^* & \hat{s}(X) \\[1ex]
\toprule
& 1 & 1 & ? & 1 & 0.05 \\
& \vdots & \vdots & \vdots & \vdots & \vdots \\
\multirow{-3}{*}{\text{Treated}, $Y(1)=1$} & 1 & 1 & ? & 1 & 0.23 \\[1ex]
\rowcolor{lightgray}
& 1 & 0 & ? & \tikzmarknode{boxA_start}{0?} & 0.85 \\
\rowcolor{lightgray} & \vdots & \vdots & \vdots & \vdots & \vdots \\
\rowcolor{lightgray} \multirow{-3}{*}{\text{Treated}, $Y(1)=0$} & 1 & 0 & ? & 0? & \tikzmarknode{boxA_end}{0.65} \\
\bottomrule
\text{Test point }X_{n+1} & - & \textcolor{blue}{0?} & \textcolor{blue}{1?} & \textcolor{blue}{0?} & \tikzmarknode{circle_target}{\textcolor{blue}{0.3}}
\end{array}
\]
\end{minipage}
\hfill
\begin{minipage}[t]{0.48\textwidth}
\centering
\[
\begin{array}{c|c|cc|cc}
\text{Group} & T & Y(1) & Y(0) & Y^* & \hat{s}(X) \\[1ex]
\toprule
\rowcolor{lightgray}
& 0 & ? & 1 & \tikzmarknode{boxC_start}{0?} & 0.95 \\
\rowcolor{lightgray}
& \vdots & \vdots & \vdots & \vdots & \vdots \\
\rowcolor{lightgray}
\multirow{-3}{*}{\text{Control}, $Y(0)=1$}
& 0 & ? & 1 & 0? & \tikzmarknode{boxC_end}{0.42} \\
\multirow{3}{*}{\text{Control}, $Y(0)=0$}
& 0 & ? & 0 & 1 & 0.03 \\
& \vdots & \vdots & \vdots & \vdots \\
& 0 & ? & 0 & 1 & 0.21 \\
\bottomrule
\text{Test point }X_{n+1} & - & \textcolor{blue}{0?} & \textcolor{blue}{1?} & \textcolor{blue}{0?} & \tikzmarknode{circle_target}{\textcolor{blue}{0.3}}
\end{array}
\]
\end{minipage}
\caption{P-values as rank tests against an incomplete science table, where the unknown label $Y^*:=\max\{Y(1),1-Y(0)\}$ indicates a harmed unit if $Y^*=0$. The left table corresponds to the p-value based on treated samples with $Y_i(1)=0$, whose unknown labels are conservatively imputed as $Y_i^*=0$ (grey shaded). The right table is based on control samples with $Y_i(0)=1$, whose unknown labels are conservatively imputed as $Y_i^*=0$ (grey shaded).}
\label{tab:science_tables}
\end{table}
\end{comment}
\subsection{Proposed Method: Conformal Policy Learning}
\label{subsec:cpl_rct}
While both p-values in the last section are feasible and valid, they are conservative since $H_0^{(1)}: Y_{n+1}(1) = 0$ and $H_0^{(0)}: Y_{n+1}(0) = 1$ are both strict implications of $H_0: Y_{n+1}(1) = 0, Y_{n+1}(0) = 1$.
In this section, we propose the general CPL procedure that improves and
subsumes the previous two options as special cases.
Following the conformal inference literature, we call the subset of the labeled data used to compute the p-values the ``\textit{calibration data}''. Then, both p-values in Section~\ref{subsec:pre} only use either the treatment or the control group as the calibration data, which can be substantially generalized.
First, consider any inclusion function $\hat{g} \colon \cX\times\{0,1\}\to [0,1]$ whose training process is independent of the labeled data and the test point.
Conditional on data, we draw independent inclusion indicators $G_i\sim \textrm{Bernoulli}(\hat{g}(X_i,T_i))$ for each $i\in [n]$, which defines the calibration set
\@\label{eq:def_Dcal}
\cD_{\textrm{calib}} = \{(X_i,T_i,Y_i)\}_{i\in \cI_\textrm{calib}},\quad \text{where} \quad \cI_{\textrm{calib}} = \{i\in [n]\colon G_i=1\}.
\@
The inclusion into $\cD_\textrm{calib}$ can depend both on $T_i$ and $X_i$, and is allowed to be random, although later we shall see a binary $\hat{g}$ suffices for optimality.
Since not all labeled data are used in calibration, we call this technique ``selective calibration''.
Second, to allow involving data from both arms in calibration, we combine information from the two treatment arms. Define the computable proxy outcome $Y_i^\dagger = T_iY_i+(1-T_i)(1-Y_i)$, so $Y_i^\dagger=0$ whenever $Y_i=0$ for a treated sample or $Y_i=1$ for a control sample.
Generalizing the previous ideas, $Y_i^\dagger=0$ captures potentially unsafe units (i.e., units for which $H_0$ might happen).
With the two techniques in hand, we construct the p-value
\@\label{eq:pval_general}
p_{n+1} = \frac{1+ \sum_{i=1}^n G_i \times \ind\{V(X_i,Y_i^\dagger)\leq V(X_{n+1},0)\}}{1+\sum_{i=1}^n G_i}.
\@
This covers the single-arm p-values in Section~\ref{subsec:pre}: it recovers $p_{n+1}^{(1)}$ (resp.~$p_{n+1}^{(0)}$)
when $G_i = T_i$ (resp.~$G_i = 1 - T_i$).
Another simple case is to take $G_i = 1$ which includes all the labeled data as the calibration set.
With this p-value, our conformal policy learning simply produces a thresholding rule $\hat\pi(X_{n+1}) \coloneqq \ind\{p_{n+1} \leq \alpha\}$. Algorithm~\ref{algo:rct} summarizes the procedure.
\begin{algorithm}
\small
\captionsetup{font=small}
\caption{Conformal Policy Learning (Randomized Experiments)}
\label{algo:rct}
\begin{algorithmic}[1]
\REQUIRE{Labeled data $\{(X_i, T_i,Y_i)\}_{i=1}^n$, test data $X_{n+1}$, target level $\alpha \in (0,1)$, score $V(\cdot, \cdot)$, inclusion function $\hat{g}(\cdot,\cdot)$.} \\[0.5ex]
\STATE Draw independent inclusion indicators $G_i \sim \text{Bern}(\hat{g}(X_i,T_i))$ for $i=1,\dots,n$.
\STATE Compute $Y_i^\dagger = T_iY_i+(1-T_i)(1-Y_i)$ for $i=1,\dots,n$.
\STATE Compute p-value $p_{n+1}$ via~\eqref{eq:pval_general}.
\STATE Compute $T_{n+1}=\ind\{p_{n+1}\leq \alpha\}$.
\ENSURE{Safe treatment $T_{n+1}$.}
\end{algorithmic}
\end{algorithm}
Theorem~\ref{thm:validity_rct} establishes the distribution-free safety guarantee for any score and inclusion functions whose inclusion probability in two groups sums up to a constant. The proof is in Appendix~\ref{app:subsec_validity_rct}.
\begin{theorem}\label{thm:validity_rct}
Assume Assumption~\ref{assump:uncon} with $e(x) \equiv 0.5$ and Assumption~\ref{assump:iid}.
Then, for any score function $V \colon \cX\times\cY \to \RR$ that is non-decreasing in the second argument and $\hat{g}\colon \cX\times\{0,1\}\to [0,1]$ whose training process is independent of $\{(X_i,T_i,Y_i)\}_{i=1}^n\cup \{X_{n+1}\}$, and obey $\hat{g}(x,1)+\hat{g}(x,0)=a$ for a constant $a>0$ for any $x\in \cX$, it holds that $\PP(Y_{n+1}(1) = 0, Y_{n+1}(0) = 1, p_{n+1} \leq \alpha) \leq \alpha$ for any $\alpha\in (0,1)$. That is, the finite-sample safety guarantee~\eqref{eq:safety_guarantee} holds for $\hat\pi(X_{n+1}) =\ind\{p_{n+1}\leq \alpha\}$.
\end{theorem}
Theorem~\ref{thm:validity_rct} states that, in principle, users can use any score function $V(\cdot, \cdot)$ and any inclusion function $\hat{g}(\cdot,\cdot)$, and CPL (Algorithm~\ref{algo:rct}) guarantees a controlled harm rate in finite samples. We emphasize that the two simplifying assumptions, namely $e(x) = 1/2$ and $\hat{g}(x,1) + \hat{g}(x,0) = a$, are used only to streamline presentation, and will be eliminated in Section~\ref{sec:stratified}.
This theorem covers the special cases we discussed in Section~\ref{subsec:pre}. The one based on the treated (resp.~control) samples correspond to $\hat{g}(x,t) = t$ (resp.~$\hat{g}(x,t) = 1-t$), both obeying the constant-sum condition with $a = 1$. As a third example, the conformal p-value using all the labeled data corresponds to $\hat{g}(x,t) \equiv 1$ and thus it also satisfies the constant-sum condition with $a = 2$.
Finally, while~\eqref{eq:pval_general} represents a restrictive class of policies, we shall see in the next subsection that specific choices of $V$ and $\hat{g}$ lead to asymptotically optimal safe policy learning algorithm among all safe policies (i.e., beyond the policy class specified by CPL).
The detailed proof of Theorem~\ref{thm:validity_rct} is in Appendix~\ref{app:subsec_validity_rct}, and we provide a sketch here to clarify the intuitions. We again consider the ``oracle'' p-value
\@\label{eq:cpl_pval_oracle}
p_{n+1}^{\ast} = \frac{1+\sum_{i=1}^n G_i \times \ind\{V(X_i,Y_i^\ast) \leq V(X_{n+1},Y_{n+1}^{\ast}\}}{1+\sum_{i=1}^n G_i},
\@
where $Y_{i}^\ast = \max\{Y_{i}(1),1-Y_{i}(0)\}$ so that $Y_{i}^\ast = 0$ if and only if $Y_{i}(1) = 0$ and $Y_{i}(0) = 1$, i.e., unit $i$ is harmed. This is again not computable because $Y_{i}^\ast$ is \emph{never} observed even for the labeled data.
The first key fact is that this oracle is valid as long as $\hat{g}$ satisfies the constant-sum condition. Indeed, we can show that in balanced randomized experiments, the data $\{(X_i, Y_{i}^\ast)\}_{G_i = 1}$ and $(X_{n+1}, Y_{n+1}^\ast)$ are exchangeable conditional on $\{G_i\}_{i=1}^n$, which implies $\PP(p_{n+1}^\ast\leq \alpha\given \{G_i\}_{i=1}^n)\leq \alpha$.
Second, the observed p-value~\eqref{eq:pval_general} replaces $Y_{n+1}^\ast$ in~\eqref{eq:cpl_pval_oracle} with the ``null'' value $0$ and replaces the calibration label $Y_i^\ast$ with $Y_i^\dagger$, thereby upper bounding the oracle one on the ``unsafe'' null event. Specifically, the proxy outcome is conservative in the sense that $Y_i^\ast\geq Y_i^\dagger$; thus, whenever $Y_{n+1}^*=0$, the monotonicity of $V$ implies $p_{n+1}\geq p_{n+1}^\ast$.
This gives $\PP(Y_{n+1}^\ast =0, p_{n+1}\leq \alpha\given \{G_i\}_{i=1}^n) \leq \PP(Y_{n+1}^\ast =0, p^\ast_{n+1}\leq \alpha\given \{G_i\}_{i=1}^n) \leq \PP(p^\ast_{n+1}\leq \alpha\given \{G_i\}_{i=1}^n) \leq \alpha$.
\begin{remark}[Sharpness of conformal p-values]\label{rmk:conservative}
Our conformal p-value is subject to two sources of conservativeness compared with $p_{n+1}^*$: (i) the ``oracle'' labels $Y_i^*$ are replaced by the proxy labels $Y_i^\dagger$, and (ii) the test label $Y_{n+1}^*$ is replaced by the threshold $0$. As in conformal selection~\citep{jin2023selection}, the issue (ii) will be addressed by a tailored ``clipped'' score function.
On the other hand, (i) is rooted in the partial identification nature of the harm rate, and specific choices of $\hat{g}$ makes our p-value sharp; we shall discuss this in more details once the optimality results in the next section are ready.
\end{remark}
\subsection{Optimality of Conformal Policy Learning}
\label{subsec:rct_optimality}
Having established the finite-sample safety guarantees for CPL, we proceed to study the optimality.
We first analyze the optimal policy among all safe policies (i.e., including policies beyond our framework) under partial identification. We then show that CPL with specific choices of the score and inclusion functions asymptotically achieves global optimality.
To formalize the discussion, we define some additional notations.
The unknown, true joint distribution over $(X,Y(1),Y(0))$ is denoted as $\PP_{X,Y(1),Y(0)}$, which induces the observable distribution $(X_i,Y_i,T_i)\sim \PP_{X,Y,T}$.
Also, we denote the collection of distributions $P$ over $(X,Y(1),Y(0))$ that are compatible with the observable distribution $\PP_{X,Y,T}$ as
\$
\cP = \big\{P_{X,Y(1),Y(0)}\colon P_{X,Y(T),T}=\PP_{X,Y(T),T} ~\text{for}~ P(T=1\given X,Y(1),Y(0))= e(X)\big\},
\$
that is, the induced distribution of $(X,Y,T)$ under $P$ coincides with $\PP_{X,Y,T}$ on the observables.
For any distribution $P$ and any treatment assignment rule $\pi\colon \cX\to \{0,1\}$, we write the harm rate as
\$
\text{Err}(\pi;P) := P( Y_{n+1}(\pi(X_{n+1}))< Y_{n+1}(0)) = P(Y_{n+1}(1)=0, Y_{n+1}(0)=1, \pi(X_{n+1})=1).
\$
By the tower property, letting $\EE_P$ denote the expectation under $P$, we know
\$
\text{Err}(\pi;P) = \EE_P\big[ \pi(X_{n+1}) \cdot f_P(X_{n+1})\big] \leq \alpha,\quad \text{where}\quad f_P(x) := P(Y_{n+1}(1) < Y_{n+1}(0)\given X_{n+1}=x).
\$
Here we call $f(x)$ the \emph{harm rate function}.
Similar to the scalar marginal harm rate $\text{Err}(\pi,P)$, the function $f_P(\cdot)$ is not identifiable from data because $Y(1)$ and $Y(0)$ are never simultaneously observed.
\cite{kallus2022s} derived the sharp partial-identification upper bound for $f_P(x)$, given by
\@\label{eq:def_bd_gamma}
\gamma(x):= \min\{1-\mu_1(x),\mu_0(x)\},
\@
where $\mu_t(x) = \EE[Y(t)\given X=x]$ for $t\in \{0,1\}$ is the conditional mean function of each potential outcome. Namely, for any super-population $P_{X,Y(1),Y(0)}$ whose observed distribution $P_{X,Y,T}$ (induced by the same treatment assignment mechanism) is equal to $\PP_{X,Y,T}$, its harm rate function obeys $f_P(x)\leq \gamma(x)$, and there exists one such distribution whose harm rate function coincides with $\gamma(x)$.
The most common objective of policy learning is to maximize the welfare. Following the policy learning literature~\citep[e.g.,][]{murphy2003optimal, manski2004statistical, hirano2009asymptotics, zhao2012estimating, kitagawa2018should, athey2021policy}, we define the welfare of a policy $\pi\colon \cX\to \{0,1\}$ as
\$
\text{Welfare}(\pi; \PP) \coloneqq \EE\big[Y_{n+1}(\pi(X_{n+1}))\big],
\$
where the larger value of outcome $Y$ corresponds to the larger welfare.
The welfare-optimal treatment rule subject to the safety guarantee can be defined as
\@\label{eq:def_opt_welfare_phi}
\pi_{\textnormal{welfare}}^* = \argmax_{\pi\colon \cX\to \{0,1\}} \quad & {\textnormal{Welfare}} (\pi; \PP) \\
\text{subject to}\quad & \max_{P\in \cP} ~\text{Err}(\pi;P) \leq \alpha. \notag
\@
Here, we optimize the welfare subject to the worst-case safety guarantee. The constraint, $\max_{P\in \cP} ~\text{Err}(\pi;P) \leq \alpha$, ensures that, regardless of the underlying data-generating process, the harm rate is upper bounded by $\alpha.$
We define the oracle welfare-optimal score function
\@\label{eq:s_welfare_optimal}
s_{{\textnormal{welfare}}}(x)=- (\mu_1(x)-\mu_0(x))/\gamma(x),
\@
where we use the convention $0/0=0$, $a/0=+\infty$ if $a>0$ and $a/0=-\infty$ if $a<0$ throughout.
The following theorem formally gives the optimal solution $\pi^*_{\text{welfare}}$ under the worst-case safety constraint. It is an implication of the more general Theorem~\ref{thm:welfare_optimal_general} in Appendix~\ref{app:subsec_general_opt}.
\begin{theorem}
\label{thm:welfare_optimal}
Assume $s_{\mathrm{welfare}}(X)$ has no point mass. Then an optimal solution to \eqref{eq:def_opt_welfare_phi} is
\[
\pi^*_{\mathrm{welfare}}(x) = \mathbf{1}\{s_{\mathrm{welfare}}(x)\leq r^*\}, \quad
\text{where}\quad r^* := \sup\left\{ \tilde r\leq 0: \mathbb{E}\!\left[ \gamma(X) \mathbf{1}\{s_{\mathrm{welfare}}(X)\leq \tilde r\} \right] \leq \alpha \right\}.
\]
Moreover, writing $\tau(x)=\mu_1(x)-\mu_0(x)$, if $\mathbb{E}[\gamma(X)\mathbf{1}\{\tau(X)>0\}]\leq\alpha$, then $r^*=0$ and $\pi^*_{\mathrm{welfare}}(x)=\mathbf{1}\{\tau(x)>0\}$ almost surely. Otherwise, $r^*<0$ and $\mathbb{E}[\gamma(X)\pi^*_{\mathrm{welfare}}(X)]=\alpha$.
\end{theorem}
Theorem~\ref{thm:welfare_optimal} shows that the optimal treatment rule for the welfare maximization is based on a cutoff on $s_{\text{welfare}}(X_{n+1})$ while staying in the region $\tau(X_{n+1})\geq 0$. The intuition is as follows.
We can show that the problem~\eqref{eq:def_opt_welfare_phi} that defines $\pi_{\text{welfare}}^*$ can be rewritten as
\@\label{eq:def_opt_welfare_phi_2}
\maximize_{\pi \colon \cX\to \{0,1\}} \quad & \E[\pi(X_{n+1})\tau(X_{n+1})] + \text{constant} \\
\text{subject to}\quad & \E[\pi(X_{n+1}) \gamma(X_{n+1})] \leq \alpha, \notag
\@
where $\tau(x) = \mu_1(x) - \mu_0(x)$ captures the conditional average treatment effect (CATE).
The optimal solution thus seeks the feature regions that yield the largest welfare gain $\tau(X)$ relative to each unit of ``cost'' $\gamma(X)$.
The next question is whether, and under what conditions, the conformal policy learning method can achieve this optimal welfare.
Following Theorem~\ref{thm:welfare_optimal}, we
define the conformal p-value
\begin{equation}
\label{eq:pval_welfare_optimal}
p^{\text{welfare}}_{n+1} = \frac{1+ \sum_{i=1}^n G_i \times \ind\{Y_i^\dagger = 0\} \times \ind\{\hat{s}(X_i) \leq \hat{s}(X_{n+1})\}}{1+\sum_{i=1}^n G_i},
\end{equation}
for the specific choices
\@
& G_i = T_i \one\{1 - \hat{\mu}_1(X_i) \leq \hat{\mu}_0(X_{i})\} + (1-T_i) \one\{1 - \hat{\mu}_1(X_i) > \hat{\mu}_0(X_{i})\}, \label{eq:def_G_welfare_optimal}\\
&\text{and}\qquad \hat{s}(x) = - (\hat{\mu}_1(x) - \hat{\mu}_0(x))/\hat{\gamma}(x).\label{eq:def_s_welfare_optimal}
\@
This corresponds to taking a clipped score $V(x,y) = M\ind\{y>0\} + \hat{s}(x)$ with a sufficiently large constant $M > 0$ in Algorithm~\ref{algo:rct}.
Note that $\hat{g}(x,1) + \hat{g}(x,0) = 1$ and therefore it also satisfies the condition for the inclusion function specified in Theorem~\ref{thm:validity_rct}. The intuition about the expression of the inclusion function $G_i$ is given in the next subsection, where we discuss the sharpness of our results.
Consider the conformal policy learning algorithm with estimated welfare cutoff (if necessary)
\$
\hat\pi_{\mathrm{welfare}}(X_{n+1}) := \ind\{p^{\mathrm{welfare}}_{n+1}\leq\alpha\} \ind\{\hat\mu_1(X_{n+1}) > \hat\mu_0(X_{n+1})\}.
\$
Theorem~\ref{thm:asymp_opt_welfare} shows that when conditional expectation functions of potential outcomes are correctly specified, CPL achieves the optimal welfare asymptotically. Its proof is included in Appendix~\ref{app:subsec_opt_welfare_ours}.
\begin{theorem}\label{thm:asymp_opt_welfare}
Suppose Assumption~\ref{assump:uncon} holds with $e(x) = 0.5$ and Assumption~\ref{assump:iid} holds. Suppose $\|\hat{\mu}_t(X) -\mu_t(X)\|_{L_2(\PP_{X})}=o_P(1)$ as $n\to \infty$ for $t \in \{0,1\}.$
Additionally, assume $s_{{\textnormal{welfare}}}(X)$ has no point mass.
Then, $\EE[Y(\hat\pi_{{\textnormal{welfare}}}(X_{n+1}))]\to {\textnormal{Welfare}}(\pi_{\textnormal{welfare}}^*;\PP)$ as $n\to \infty$ where $\pi_{{\textnormal{welfare}}}^*$ is the optimal solution in Theorem~\ref{thm:welfare_optimal}.
\end{theorem}
The consistency condition on the estimated conditional expectation functions could be achieved by flexible machine-learning-based methods.
Moreover, even if the consistency condition fails, the conformal policy learning algorithm based on $p^{\text{welfare}}_{n+1}$ still satisfies the finite-sample safety guarantee as per Algorithm~\ref{algo:rct}.
In Appendix~\ref{app:subsec_power_opt}, we present analogous optimality results when researchers are interested in maximizing the ``power'', the probability of treating the test unit. The only difference is that we should now use a power-oriented score $\hat{s}(x) = \hat{\gamma}(x)$.
\subsection{Optimality, Sharpness, and Selective Calibration}
\label{subsec:discuss_sharp}
The optimality result suggests that, in the non-trivial setting where the constraint is binding, conformal policy learning can \emph{exhaust} the harm rate budget under partial identification, matching the globally optimal rule whose worst-case harm rate is exactly $\alpha$.
We now continue on Remark~\ref{rmk:conservative} to explain why this sharpness can be achieved.
The use of clipped score essentially eliminated the conservativeness of using $0$ instead of $Y_i^*$. We now focus on the conservativeness arising from replacing the ``oracle'' outcome $Y_i^* = \max\{Y_i(1),1-Y_i(0)\}$ by the proxy label $Y_i^\dagger = Y_iT_i + (1-Y_i)(1-T_i)\leq Y_i^*$, and discuss how this is eliminated by selective calibration.
The use of $Y_i^\dagger$ is deeply rooted in the partial identification nature of the problem: even for the labeled data, the harm $Y_i^*$ is not observed.
As such, the sharpness hinges on whether, on the population level, calibration with $Y_i^\dagger$ matches the worst-case harm-rate bound that relies on $\gamma(X_i)$ in~\eqref{eq:def_bd_gamma}.
CPL achieves so through the selective calibration mechanism.
Assuming, for intuitions, that the conditional mean functions are perfect, the optimal conformal p-value uses the inclusion indicator $G_i = T_i \ind\{1 - {\mu}_1(X_i) \leq {\mu}_0(X_{i})\} + (1-T_i) \ind\{1 - {\mu}_1(X_i) > {\mu}_0(X_{i})\}$. Namely, a treated unit $X_i$ enters the calibration set if and only if $1 - \mu_1(X_i)\leq \mu_0(X_i)$, in which case
\$
\gamma(X_i) = 1-\mu_1(X_i) = \EE[ \ind\{Y_i^\dagger=0\}\given X_i,T_i=1]
\$
since $Y_i^\dagger=Y_i(1)$ for $T_i=1$. That is, the proxy label $Y_i^\dagger$ provides ``sharp'' calibration for the worst-case harm rate $\gamma(X_i)$. A similar observation applies to control units.
The above discussion shows that for sharp harm rate control, one should use a labeled unit for calibration if and only if the treatment status matches whether the outcome model is ``active'' in the worst-case harm rate function $\gamma(x)$, i.e., whether a treated unit obeys $\gamma(X_i)=1-\mu_1(X_i)$ or a control unit obeys $\gamma(X_i)=\mu_0(X_i)$.
In the simulations, we shall evaluate the ``informativeness'' of labeled data and the power of CPL.
Finally, we re-emphasize the model-free nature of conformal policy learning. Even though the arm informativeness---through the inclusion function $\hat{g}$---is estimated and therefore certainly imperfect, our method provides conservative harm rate control without any modeling assumptions.
\section{Conformal Policy Learning with Stratified Experiments}
\label{sec:stratified}
In this section, we use weighted conformal inference to generalize our previous results to allow for both arbitrary, known propensity score $e(x)$ and arbitrary inclusion function $\hat{g}$.
It also serves as the foundation for CPL in observational studies, which we cover in the next section.
The selective calibration process remains the same as Section~\ref{subsec:cpl_rct}. Consider any pre-trained monotone score function $V\colon \cX\times\{0,1\}\to \RR$ and inclusion function $\hat{g}\colon \cX\times\{0,1\}\to [0,1]$.
The calibration set is the same as~\eqref{eq:def_Dcal} where $G_i\sim \textnormal{Bern}(\hat{g}(X_i,T_i))$ are independently drawn inclusion indicators.
When computing conformal p-values for a new test point $X_{n+1}\sim \PP_X$, conditional on $G_i=1$, there is a covariate shift between $\cD_{\textrm{calib}}$ and $X_{n+1}$ that arises from both the treatment assignment and the exogenous random inclusion into the calibration set. To address this shift, we define the function $w\colon \cX\to \RR^+$ by
\@\label{eq:def_weight}
w(x) =
\bigl(e(x)\hat{g}(x,1) + (1-e(x))\hat{g}(x,0)\bigr)^{-1}.
\@
Because $e(x) \coloneqq \Pr(T_i = 1 \mid X_i=x)$ and $\hat{g}(x,t) \coloneqq \Pr(G_i = 1 \mid X_i = x, T_i = t)$,
one can show that $w(x) = \Pr(G_i = 1 \mid X_i = x)^{-1},$ the inverse probability of being included for calibration given $X_i = x$, regardless of the choice of $e(x)$ and $\hat{g}$.
Given the weights, we compute
\@\label{eq:def_pval_weight}
p^{\text{str}}_{n+1} = \frac{w(X_{n+1}) +\sum_{i = 1}^n G_i w(X_i) \ind\{V(X_i,Y_i^\dagger)\leq V(X_{n+1},0)\}}{w(X_{n+1}) + \sum_{i = 1}^n G_i w(X_i)}.
\@
The p-value~\eqref{eq:pval_general} in Section~\ref{sec:rct} is a special case of~\eqref{eq:def_pval_weight}, where the simplifying assumptions $e(x)=0.5$ and $\hat{g}(x,1) + \hat{g}(x,0)=a$ for a constant $a>0$ implied constant weights and no reweighting is needed.
Theorem~\ref{thm:valid_weighted} states that CPL with the above weighted conformal p-value
achieves the distribution-free safety guarantee, whose proof is in Appendix~\ref{app:subsec_weighted_validity}.
\begin{theorem}\label{thm:valid_weighted}
Suppose Assumption~\ref{assump:uncon} and Assumption~\ref{assump:iid} hold, and $e(x)$ is known.
Then, for any score function $V \colon \cX\times\cY \to \RR$ that is non-decreasing in the second argument and for any inclusion function $\hat{g}\colon \cX\times\{0,1\}\to [0,1]$ obeying $e(X)\hat{g}(X,1)+(1-e(X))\hat{g}(X,0)>0$, $P_X$-a.s., it holds that $\PP(Y_{n+1}(1) = 0, Y_{n+1}(0) = 1, p^{\text{str}}_{n+1} \leq \alpha) \leq \alpha$ for any $\alpha\in (0,1)$. That is,~\eqref{eq:safety_guarantee} holds for $\hat\pi^{\text{str}}(X_{n+1}) = \ind\{p^{\text{str}}_{n+1}\leq \alpha\}$.
\end{theorem}
As in Section~\ref{sec:rct}, given consistent outcome modeling, CPL achieves optimality with
\begin{equation}
\label{eq:pval_power_optimal_weights}
p^{\text{str-opt}}_{n+1} = \frac{w(X_{n+1}) + \sum_{i=1}^n w(X_i) G_i \ind\{Y_i^\dagger = 0\} \ind\{\hat{s}(X_i) \leq \hat{s}(X_{n+1})\}}{w(X_{n+1}) +\sum_{i=1}^n w(X_i) G_i},
\end{equation}
where $G_i = T_i \one\{1 - \hat{\mu}_1(X_i) \leq \hat{\mu}_0(X_{i})\} + (1-T_i) \one\{1 - \hat{\mu}_1(X_i) > \hat{\mu}_0(X_{i})\}$. The score functions remains the same as the preceding case: $\hat{s}(x) = - (\hat{\mu}_1(x) - \hat{\mu}_0(x))/\hat{\gamma}(x)$ for welfare maximization. This is the weighted version of equation~\eqref{eq:pval_welfare_optimal}.
The proof of Theorem~\ref{thm:opt_stratified_experiments} is in Appendix~\ref{app:subsec_opt_stratified}.
\begin{theorem}\label{thm:opt_stratified_experiments}
Suppose Assumption~\ref{assump:uncon} and Assumption~\ref{assump:iid} hold, and $e(x)$ is known. Suppose $\|\hat{\mu}_t(X) -\mu_t(X)\|_{L_2(\PP_{X})}\stackrel{P}{\to}0$ as $n\to \infty$ for $t \in \{0,1\}$ and $s_{{\textnormal{welfare}}}(X)$ has no point mass.
Let
\$
\hat\pi_{\text{str-opt}}(X_{n+1}) \coloneqq \ind\{p_{n+1}^{\text{str-opt}} \leq \alpha\} \ind\{ \hat\mu_1(X_{n+1})>\hat\mu_0(X_{n+1})\}
\$
where we take the same $G_i$ as~\eqref{eq:def_G_welfare_optimal} and $\hat{s}(\cdot)$ as~\eqref{eq:def_s_welfare_optimal}.
Then $\EE[Y(\hat\pi_{\text{str-opt}}(X_{n+1}))]\to {\textnormal{Welfare}}(\pi_{\textnormal{welfare}}^*;\PP)$ as $n\to \infty$, where $\pi_{\textnormal{welfare}}^*$ is the optimal solution in Theorem~\ref{thm:welfare_optimal}.
\end{theorem}
Analogously, CPL with the same selection indicators $\{G_i\}$ as~\eqref{eq:def_G_welfare_optimal} and the estimated power-optimal score yields asymptotically optimal power; this result is deferred to Theorem~\ref{thm:power_opt_str} in Appendix~\ref{app:subsec_power_opt}.
\section{Conformal Policy Learning with Observational Data}
\label{sec:obs}
In this section, we further generalize CPL to observational studies. With observational data, the main challenge is that the propensity score, hence the weight function~\eqref{eq:def_weight}, is unknown and needs to be estimated. While a natural idea is to estimate the weights and plug them into the conformal p-value, it remains unclear how the safety guarantee changes with the estimation quality.
We present a learn-then-balance procedure to obtain the estimated weights so the resulting policy learning algorithm enjoys a double robustness property.
\subsection{Conformal Policy Learning with Learn-then-Balance Weights}
We construct p-values of a similar form as~\eqref{eq:def_pval_weight} and~\eqref{eq:pval_power_optimal_weights}:
\@\label{eq:def_pval_weight_est}
p_{n+1}^{\text{obs}} = \frac{\hat{w}_{n+1} +\sum_{i=1}^n \hat{w}_i G_i \ind\{Y_i^\dagger = 0\} \ind\{\hat{s}(X_i)\leq \hat{s}(X_{n+1})\}}{\hat{w}_{n+1} + \sum_{i=1}^n \hat{w}_i G_i},
\@
where the inclusion indicators are $G_i \sim \text{Bern}(\hat{g}(X_i,T_i))$ for a function $\hat{g}\colon \cX\times\{0,1\}\to [0,1]$ whose training process is independent of the labeled and unlabeled data.
The only difference of~\eqref{eq:def_pval_weight_est} from~\eqref{eq:pval_power_optimal_weights} is that the weights are now estimated. Notably, here we work with a form analogous to~\eqref{eq:pval_power_optimal_weights} since our balancing weights are more easily stated in terms of the score function $\hat{s}(\cdot)$. It corresponds to~\eqref{eq:def_pval_weight} with the conformity score $V(x,y) = M\ind\{y>0\}+\hat{s}(x)$ for a sufficiently large constant $M>2\sup_x|\hat{s}(x)|$.
Denoting $\cI_{\textrm{calib}}=\{i\in[n]\colon G_i=1\}$, the estimated weights $\{\hat{w}_i\}_{i\in\cI_{\textrm{calib}}\cup\{n+1\}}$ in~\eqref{eq:def_pval_weight_est} are obtained by a learn-then-balance approach.
We begin with two learned functions, one preliminary weight function $\tilde{w}\colon \cX\to \RR^+$, and one estimated harm-rate function $\hat\gamma\colon \cX\to [0,1]$.
A natural choice is to define $\tilde{w}(x) = 1/[\hat{e}(x)\hat{g}(x,1)+(1-\hat{e}(x)\hat{g}(x,0)]$ where $\hat{e}(x)$ is an estimated propensity score function, and $\hat\gamma(x) = \min\{1-\hat\mu_1(x),\hat\mu_0(x)\}$ for estimated outcome models $\{\hat\mu_t(\cdot)\}_{t\in\{0,1\}}$.
For simplicity, we require the propensity score and outcome models to be trained independently of the calibration and test data (in practice, one can use sample splitting to fit the models~\citep{chernozhukov2018double} and the properties are analogously studied in~\cite{jin2025cross}).
Then, define the balancing feature vector
\$
\hat\phi(x) = \big(\hat{\gamma}(x)\ind\{ \hat{s}(x)\leq \hat{t}\}, \tilde{w}(x)\big)\in \RR^2,\quad \text{where}\quad \hat{t} = \sup \bigg\{t\in \RR\colon \frac{1}{n}\sum_{i=1}^n \hat{\gamma}(X_i) \ind\{\hat{s}(X_i)\leq t\} \leq \alpha \bigg\}.
\$
Finally, we let $\{\hat{w}_i\}_{i\in \cI_{\textrm{calib}}}$ be the optimal solution to the following optimization program with $\delta_n=O(n^{-1/2})$:
\@\label{eq:def_sbw_main}
\argmin_{ w\succeq 0} \Bigg\{ \|w\|^2 : \, \bigg| \frac{1}{|\mathcal{I}_{\textrm{calib}}|} \sum_{i\in \mathcal{I}_{\textrm{calib}}} w_i \hat\phi_k(X_i) - \frac{1}{n} \sum_{i=1}^n \hat\phi_k(X_i) \bigg|\leq \delta_n, k \in \{1,2\}; \frac{1}{|\mathcal{I}_{\textrm{calib}}|}\sum_{i\in \mathcal{I}_{\textrm{calib}}} w_i = 1 \Bigg\}.~~~~
\@
We shall see that the post-hoc processing to ensure the finite-sample approximate balancing~\eqref{eq:def_sbw_main} is essential for us to obtain the doubly robust safety guarantees even when the propensity model is misspecified.
The first condition in~\eqref{eq:def_sbw_main} follows the balancing weights~\citep{hainmueller2012entropy,zubizarreta2015stable}, by enforcing a finite-sample balancing condition inspired by the desired population-level property: for the correct weights $w(\cdot)$, it should hold that $\frac{1}{|\cI_{\textrm{calib}}|}\sum_{i\in \cI_{\textrm{calib}}} w(X_i) f(X_i)\approx \frac{1}{n}\sum_{i=1}^n f(X_i)$ for any fixed function $f\colon \cX\to \mathbb R$. Here, we balance specifically-designed functions to obtain favorable statistical properties under general conditions on the learned functions.
\subsection{Doubly Robust Safety Guarantees}
We now formalize the safety guarantees of CPL~\eqref{eq:def_pval_weight_est} with weights $\{\hat{w}_i\}_{i\in \cI_{\textrm{calib}}}$ from~\eqref{eq:def_sbw_main}.
These results will require mild regularity conditions for the learn-then-balance
optimization, which we defer to Assumption~\ref{assump:bal_reg}. These conditions
require overlap and boundedness, local stability of the population
balancing solution, and regularity of the population cutoff.
Under these
conditions, and given that either the preliminary weight function or the outcome models are consistent, we obtain asymptotic safety guarantee for CPL. The proof of Theorem~\ref{thm:dr_obs} is included in Appendix~\ref{app:subsec_dr_obs}, which follows from our general theory with estimated weights in Appendix~\ref{app:subsec_dr_obs_general}.
\begin{theorem}[Model double robustness]\label{thm:dr_obs}
Suppose Assumption~\ref{assump:bal_reg} holds, and $\delta_n = O(n^{-1/2})$. Then,
$$\limsup_{n\to \infty}\PP(Y_{n+1}(\hat{\pi}_{\textnormal{obs}}(X_{n+1}))<Y_{n+1}(0))\leq \alpha$$ under either of the following conditions:
\begin{enumerate}[label=(\roman*).]
\item The preliminary weight function is consistent: $\|\tilde{w}-w\|_{L_2(\PP_X)}=o_P(1)$ for the true weight $w(\cdot)$ in~\eqref{eq:def_weight}.
\item The outcome models are consistent: $\|\hat\gamma-\gamma\|_{L_2(\PP_X)}=o_P(1)$.
\end{enumerate}
\end{theorem}
\begin{remark}
We remark that the techniques and conditions here differ from existing model double-robustness results in conformal prediction (e.g.~\cite{lei2021conformal}) where correct outcome model alone is enough to ensure validity; in our setting, because of the thresholding nature of the CPL policy, explicit finite-sample balance turns out to be an important element of our theoretical analysis for double robustness.
\end{remark}
Taking a step further, we show that the learn-then-balance weights lead to a product error rate structure, which further yields the $O_P(n^{-1/2})$ convergence of the harm rate of CPL given slow, nonparametric convergence rates of the estimated models. The proof of Theorem~\ref{thm:dr_obs_rate} is in Appendix~\ref{app:subsec_dr_obs_rate}.
\begin{theorem}[Rate double robustness]\label{thm:dr_obs_rate}
Suppose Assumption~\ref{assump:bal_reg} holds and $\delta_n = O(n^{-1/2})$. If
$\|\tilde w-w\|_{L_\infty(\PP_X)}
=
O_P(n^{-1/4})$ and
$\|\hat\gamma-\gamma\|_{L_1(\PP_X)}
=
O_P(n^{-1/4})$,
then $\{\PP(Y_{n+1}(\hat{\pi}_{\text{obs}}(X_{n+1}))<Y_{n+1}(0)\given \cA_n) - \alpha\}_{+} = O_P(n^{-1/2})$, where $\cA_n$ is the $\sigma$-field for the randomness in calibration data, training, and selective inclusion.
\end{theorem}
\subsection{Optimal Conformal Policy Learning with Observational Studies}
Since the population optimization problem only depends on the super-population of the potential outcomes and observed features, the population-level optimal solution remains the same as Theorem~\ref{thm:welfare_optimal}.
The following theorem shows that under mild conditions, our method with the optimally chosen score and inclusion function achieves the optimal welfare with observational data. Its proof is in Appendix~\ref{app:subsec_optimality_obs}.
The results for power optimality are deferred to Appendix~\ref{app:subsec_power_opt}.
\begin{theorem}[Welfare optimality with observational data]
\label{thm:obs_welfare_opt}
Let $\hat\pi_{\textrm{obs}}(X_{n+1})=\ind\{p^{\textrm{obs}}_{n+1}\leq\alpha\}\ind\{\hat{s}(X_{n+1})<0\}$ where we take the same $G_i$ as~\eqref{eq:def_G_welfare_optimal} and $\hat{s}(\cdot)$ as~\eqref{eq:def_s_welfare_optimal}.
Suppose Assumption~\ref{assump:bal_reg} holds, and $\delta_n = O(n^{-1/2})$.
Furthermore, assume $\|\hat g-g^*\|_{L_2(\PP_{X,T})}+\|\hat\gamma - \gamma\|_{L_2(\PP_{X})}+\|\hat s-s_{{\textnormal{welfare}}}\|_{L_2(\PP_X)}=o_P(1)$ for $g^*(x,t)=t\ind\{1-\mu_1(x)\leq\mu_0(x)\}+(1-t)\ind\{1-\mu_1(x)>\mu_0(x)\}$, and
$s_{{\textnormal{welfare}}}(X)$ defined in~\eqref{eq:s_welfare_optimal} has no point mass.
Then
$
\EE[Y\{\hat\pi_{\textrm{obs}}(X_{n+1})\}]
\to
{\textnormal{Welfare}}(\pi_{{\textnormal{welfare}}}^*;\PP)$ as $n\to\infty$, where $\pi_{{\textnormal{welfare}}}^*$ is the optimal solution in Theorem~\ref{thm:welfare_optimal}.
\end{theorem}
As in randomized experiments, optimality requires two convergence conditions: $\hat g\to g^*$, which makes the proxy-label calibration sharp, and $\hat s\to s_{\mathrm{welfare}}$, which provides the optimal treatment ranking. When $\hat g$, $\hat s$, and $\hat\gamma$ are constructed from plug-in outcome model estimates, these
conditions follow from consistent outcome models under the usual regularity conditions. In this outcome-correct case, the balancing condition aligns the weighted calibration with the oracle harm-welfare curve, so the weight estimator need not be consistent. However, inheriting the model consistency of Theorem~\ref{thm:dr_obs}, one can show that the conclusion also continues to hold under consistent
weights, $\bar w\propto w$, without requiring
$\hat\gamma\to\gamma$, provided that $\hat g\to g^*$ and
$\hat s\to s_{\mathrm{opt}}$ still hold. Yet, we do not pursue it here since the latter two convergence conditions usually require consistency of the estimated outcome models which imply the consistency of the harm rate estimation.
\section{Simulation Studies}
\label{sec:sim}
\subsection{Balanced Randomized Experiments}
\label{subsec:simu_rct}
We begin by evaluating the methods in balanced randomized experiments (Section~\ref{sec:rct}).
The data in the randomized experiment settings are generated using the general framework below.
\vspace{-0.75em}
\paragraph{Data generating processes.}
Throughout, we generate the features $X\sim P_X$ for some distribution $P_X$ and i.i.d.~treatment indicators $T\sim \text{Bern}(1/2)$, independent of everything else. The potential outcomes follow $\PP(Y(1)=1\given X=x)=\mu_1(x)$ and $\PP(Y(0)=1\given X=x)=\mu_0(x)$ for some functions $\mu_1(\cdot)$ and $\mu_0(\cdot)$. The potential outcomes are coupled negatively, meaning that
$\PP(Y(1)=0,Y(0)=1\given X=x) = \min\{1-\mu_1(x), \mu_0(x)\}$.
Note that any coupling would lead to the same observed data distribution $P_{X,Y,T}$ (and thus the same results for any method), but the negative coupling leads to the worst-case harm rate.
The specific choice of $P_X$ and $\mu_1(\cdot)$, $\mu_0(\cdot)$ is introduced in each set of simulations below.
We first design four diverse settings to compare our methods and baselines, followed by two additional settings to investigate the performance of our methods, including (i) the power of selective calibration by varying the informativeness of arms, and (ii) decisions by heterogeneous subgroups.
\subsubsection{Harm rate control in diverse settings}
\label{subsubsec:simu_rct_main}
The first set of simulations evaluates the harm rate control across diverse settings and modeling choices. We set $X\in \RR^{20}$ with $P_X= \cN(0, I_d)$. The conditional mean functions are $\mu_t(x) = \exp(\eta_t(x))/(1+\exp(\eta_t(x))$, $t\in \{0,1\}$ for some functions $\eta_t(x)$. We design four settings (see Appendix~\ref{app:subsec_simu_rct_details} for detailed DGPs):
\begin{itemize}
\item Setting 1: approximately linear, where $\eta_t(x)=\beta_t^\top x$ for some $\beta_t\in \RR^{20}$ and fully-observed $X\in \RR^{20}$;
\item Setting 2: approximately linear DGP but the observed covariates are $X_{1:7}\in \RR^{7}$;
\item Setting 3: nonlinear $\eta_t(x)$ involves the first 5 entries of $x$, with fully observed $X\in \RR^{20}$;
\item Setting 4: nonlinear $\eta_t(x)$ involves the first 8 entries of $x$, but the observed covariates are $X_{1:5}\in \RR^{5}$.
\end{itemize}
In settings 2 and 4, missing some covariates typically reduces the accuracy of learned outcome models (finite-sample harm-rate control still holds as per Theorem~\ref{thm:validity_rct}). We vary the total labeled sample size $n_{\text{total}}\in\{500,1000,2000\}$, fixing the number of test data $m=1000$, and $\alpha=0.1$. We use $\cD_{\text{label}}$ to denote the labeled data and $\cD_{\textrm{test}}$ to denote the test data.
\vspace{-0.75em}
\paragraph{Methods and evaluation.}
We compare the following three baselines and two CPL methods:
\begin{enumerate}[label=(\arabic*)]\setlength\itemsep{-0.1em}
\item \texttt{Li et al.}, \cite{li2023trustworthy} with doubly-robust estimator and constrained optimization among depth-2 decision trees via \texttt{econml} python library. The policy is learned on $\cD_{\text{label}}$ and evaluated on $\cD_{\textrm{test}}$. This method requires the distributional assumption about the potential outcomes. In particular, it assumes that the potential outcomes are non-negatively correlated, which is violated in this setup.
\item \texttt{Policy-Tree}, which maximizes welfare $\EE[Y(\pi(X_{n+1}))]$ among policies represented by depth-2 decision trees with \texttt{econml}. The policy is learned on $\cD_{\text{label}}$ and evaluated on $\cD_{\textrm{test}}$. While this method is not designed for safety control, we include it as the baseline due to its popularity as a standard practice.
\item \texttt{Threshold}, which uses $\cD_{\text{label}}$ to obtain an estimator $\hat{\gamma}(x)=\min\{1-\hat\mu_1(x),\hat\mu_0(x)\}$ and treat $\{j\in[m]\colon \hat{\gamma}(X_{n+j})\leq \tilde\gamma\}$ where $\tilde\gamma = \max\{\gamma\colon \sum_{j=1}^m \hat\gamma(X_{n+j})\ind\{\hat{\gamma}(X_{n+j})\leq \gamma \} \leq 0.1\cdot m\}$. This heuristic calibration is valid only when the estimator $\hat{\gamma}$ is sufficiently accurate.
\item \texttt{CPL-sel-welfare}, the CPL algorithm that maximizes the welfare, i.e., Algorithm~\ref{algo:rct} with $g(x,t)=t \ind\{1-\hat\mu_1(x)\leq \hat\mu_0(x)\} +(1-t)\ind\{1-\hat\mu_1(x)>\hat\mu_0(x)\}$ and a clipped score function $V(x,y) = M\ind\{y>0\} - (\hat{\mu}_1(x) - \hat{\mu}_0(x))/\hat{\gamma}(x)$ with a sufficiently large constant $M > 0$. This method always satisfies the safety guarantee and asymptotically achieves the optimal welfare if $\mu_t(x)$ is consistently estimated.
\item \texttt{CPL-sel-power}, the CPL algorithm that maximizes the power, i.e., Algorithm~\ref{algo:rct} with $g(x,t)=t \ind\{1-\hat\mu_1(x)\leq \hat\mu_0(x)\} +(1-t)\ind\{1-\hat\mu_1(x)>\hat\mu_0(x)\}$ and a clipped score function $V(x,y) = M\ind\{y>0\} + \hat{\gamma}(x)$ with a sufficiently large constant $M > 0$. This method always satisfies the safety guarantee and asymptotically achieves the optimal power if $\mu_t(x)$ is consistently estimated (Appendix~\ref{app:subsec_power_opt}).
\end{enumerate}
\vspace{0.25em}
\begin{figure}[!t]
\centering
\includegraphics[width=\linewidth]{figs/RCT_main_all_with_oracle.pdf}
\caption{(a) Empirical harm rate, (b) power (probability of treatment in the test units), (c) average welfare of various methods at level $\alpha=0.1$ in the randomized experiment studies, averaged over $200$ independent simulation runs. Each row represents a different prediction model (logistic regression, random forest) for $(\hat\mu_1,\hat\mu_0)$, and each column represents a different data-generating process. The $x$-axis is the total labeled sample size $n$. The blue dashed lines in (b) and (c) show the optimal power and welfare of CPL based on oracle correct models $(\mu_1,\mu_0)$ without any estimation error.}
\label{fig:rct_fna}
\end{figure}
For the two CPL variants, the labeled data $\cD_{\text{label}}$ is randomly split into the training ($75\%$) and calibration ($25\%$) folds $\cD_\textrm{train}$ and $\cD_\textrm{calib}$. The training fold is used to fit two models $\hat\mu_1(x)$ and $\hat\mu_0(x)$ for $\mu_1(x)$ and $\mu_0(x)$, respectively.
The function $\hat{\gamma}(x)=\min\{1-\hat\mu_1(x),\hat\mu_0(x)\}$ then estimates the upper bound on the harm rate function.
We adopt two model classes for training the outcome models: logistic regression, and random forest, which are used in both \texttt{Threshold} and \textsc{CPL}\xspace methods.
\texttt{Li et al.}~and \texttt{Policy-Tree} use random forests for nuisance component estimation to be consistent with the tree-based policy class.
See Appendix~\ref{app:subsec_simu_rct_details} for additional details on method implementation.
Given the learned treatment $T_{n+j} \coloneqq \hat\pi(X_{n+j}) \in \{0,1\}$ for $j\in [m]$ produced by the methods, we evaluate three metrics: the harm rate by $\frac{1}{m}\sum_{j=1}^m T_{n+j}\ind\{Y_{n+j}(1)<Y_{n+j}(0)\}$, the welfare by $\frac{1}{m}\sum_{j=1}^m Y_{n+j}(T_{n+j})$, and the fraction of treatment by $\frac{1}{m}\sum_{j=1}^m T_{n+j}$.
All metrics are averaged over $200$ independent runs.
\vspace{-0.75em}
\paragraph{Simulation results.} Figure~\ref{fig:rct_fna} presents the results for the five methods across various settings. First, the three baselines lead to a drastic violation of the target harm rate. The \texttt{Threshold} method relies on accurate outcome models to ensure consistency of $\hat\gamma(\cdot)$, which is difficult to satisfy with limited labeled data. \texttt{Policy-tree}, which focuses on welfare maximization can incur large harm rates. Finally, \texttt{Li et al.}~achieves harm rate only when the two potential outcomes are non-negatively correlated given the features; as a result, it leads to exceedingly high harm rate due to (i) worst-case negative coupling and (ii) inconsistency due to missing covariates in settings 2 and 4.
On the other hand, the two variants of \textsc{CPL}\xspace control the harm rate tightly at the target level, showing both the validity and sharpness of CPL. The two variants based on different score functions demonstrate negligible difference: in these settings, the rank of instances based on the two optimal scores does not drastically change the decisions by \textsc{CPL}\xspace. In practice, this means that researchers can approximately maximize both power and welfare together via CPL without worrying about the tradeoff between the two objectives.
Finally, compared with the oracle power and welfare (blue dashed lines), the CPL methods achieve close-to-optimal performance, and the gap shows the impact of estimation error in $\hat\mu_t$ functions. Such gap seems to be moderate even for misspecified models (Logistic regression in settings 3 and 4).
\subsubsection{Effectiveness of selective calibration}
We now use another set of experiments to dive deeper into the selective calibration mechanism.
Following the discussion in Section~\ref{subsec:discuss_sharp}, the harm rate control by using the conservative proxy $Y_i^\dagger = T_iY_i +(1-T_i)(1-Y_i)$ is tight if a selected sample satisfies (i) $T_i=1$ and $\gamma(X_i)=1-\mu_1(X_i)$,
or (ii) $T_i=0$ and $\gamma(X_i)=\mu_0(X_i)$.
Our selective calibration method in Section~\ref{subsec:cpl_rct} aims to address this by adaptively selecting the informative arm.
In this part, we vary the magnitudes of $\mu_1(x)$ and $\mu_0(x)$ to how this strategy contributes to the statistical efficiency.
We vary the proportion of samples obeying $\gamma(X) = 1-\mu_1(X)$ (treated arm is informative) and those obeying $\gamma(X)=\mu_0(X)$ (control arm is informative); see Appendix~\ref{app:subsec_rct_aux} for the detailed data-generating process.
In addition to the two CPL variants evaluated in Section~\ref{subsubsec:simu_rct_main}, we evaluate two more procedures:
\begin{enumerate}[label=(\arabic*)]\setlength\itemsep{-0.1em}
\item[(6)] \texttt{CPL-treat}, our method in Section~\ref{subsec:pre} with treated samples in $\cD_\textrm{calib}$ and $V(x,y)=My-\hat\mu_1(x)$ with a sufficiently large constant $M > 0$.
\item[(7)] \texttt{CPL-control}, our method in Section~\ref{subsec:pre} with control samples in $\cD_\textrm{calib}$ and $V(x,y) = My +\hat\mu_0(x)$ with a sufficiently large constant $M > 0$.
\end{enumerate}
\vspace{0.25em}
The procedures are evaluated in terms of harm rate, power (fraction of treatment), and welfare, with the metrics averaged over $200$ independent runs.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figs/RCT_arm_aux_fna_power.pdf}
\caption{(a) Empirical harm rate, (b) power (probability of treatment) and (c) average welfare of the seven procedures in the arm-informativeness experiments at level $\alpha=0.1$ over $200$ independent runs. Within each panel, each column represents a prediction model (logistic regression, random forest) for $(\hat\mu_1,\hat\mu_0).$}
\label{fig:rct_aux_arm}
\end{figure}
The results are summarized in Figure~\ref{fig:rct_aux_arm}, where the $x$-axis is the fraction of treated samples being informative (i.e., $\gamma(X)=1-\mu_1(X)$).
As before, all variants of \textsc{CPL}\xspace~control the harm rate below $\alpha=0.1$.
\texttt{CPL-treat} and \texttt{CPL-control} demonstrate clear power tradeoffs: \texttt{CPL-treat} is more powerful when the treated arm is more informative ($x$-axis above $0.5$), and the opposite happens otherwise.
Importantly, \texttt{CPL-sel-power} and \texttt{CPL-sel-welfare} are often comparable to the more powerful single-arm variants, showing the effectiveness of selective calibration. This justifies our recommendations for the asymptotically optimal variants (Section~\ref{subsec:rct_optimality}), since in practice it is often unknown which arm might be more powerful.
\subsubsection{Performance under subgroup heterogeneity}
To further inspect the behavior of \textsc{CPL}\xspace~under treatment effect heterogeneity, we design a setting with three subgroups driven by the first two features in $X\in \RR^{20}$ (Figure~\ref{fig:rct_aux_subgroup} panel (a)). There is strong cross-group heterogeneity, but the units in the same group are largely similar. Studying the decisions in each group offers a zoom-in observation of ``who gets treated'' with different scoring functions $s(\cdot)$ in \textsc{CPL}\xspace.
The first group consists of half of the population, whose worst-case harm rate $\gamma(X)$ is relatively large while the conditional average treatment effect $\tau(X)=\mu_1(X)-\mu_0(X)$ is also large. Groups 2 and 3 consist of $1/4$ of the population each, whose worst-case harm rate $\gamma(X)$ and conditional average treatment effect $\tau(X)$ are both small; the main difference is that the treated samples are more ``informative'' in Group 2 (i.e., $1-\mu_1(X)<\mu_0(X)$), while the control samples are more informative in Group 3. We follow the same procedures as before to evaluate the four variants of \textsc{CPL}\xspace.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figs/RCT_subgroup_aux_2panel.pdf}
\caption{(a) Subgroup setup, (b) Per-group fraction of $T_{n+1}=1$ for variants of \textsc{CPL}\xspace in the subgroup DGP at level $\alpha=0.01$. Panels (c-d) show random forests (RF) as the outcome model only.}
\label{fig:rct_aux_subgroup}
\end{figure}
The results averaged over $200$ independent simulation runs are summarized in Figure~\ref{fig:rct_aux_subgroup}.
In panel (b), we show the fraction of $T_{n+j}$ within each group based on random forests predictions. Different score functions lead to different treatment prioritization patterns. \texttt{CPL-treat} concentrates the safe treatment budgets on group 2 (small $\gamma(X)$ with treatment arm being informative) since it ranks instances based on $\hat\mu_1(x)$, while \texttt{CPL-control} concentrates the budgets on group 3. In contrast, \texttt{CPL-sel-power} distributes the budgets relatively uniformly across groups 2 and 3 (since their harm rates are comparably small) by using the score function $\hat\gamma(x)$. \texttt{CPL-sel-welfare}, which ranks instances by balancing harm rate and treatment effect size, puts most budgets on Group 1 (large treatment effects) but assigns fewer treatments in general.
\subsection{Stratified Experiments and Observational Studies}
\label{subsec:simu_obs}
In this part, we proceed to evaluate \textsc{CPL}\xspace in stratified experiments and observational studies. The stratified experiments induce a \emph{known} covariate shift between the calibration and test data, while in observational studies this covariate shift needs to be estimated.
\vspace{-0.75em}
\paragraph{Simulation settings.} We first sample the triplets $\{(X_i,Y_i(1),Y_i(0)\}_{i=1}^{n+m}$ using data generating processes to be specified later.
In the labeled data, the treatment assignments are sampled by $T_i \mid X_i \sim \mathrm{Bernoulli}(e(X_i))$ independently with propensity score function $e(x)$ to be specified later. The observed outcome is $Y_i=Y_i(T_i)$.
\vspace{-0.5em}
\paragraph{Methods and evaluation.}
Fixing the confidence level at $\alpha=0.1$, we compare the following procedures:
\begin{enumerate}[label=(\arabic*)]\setlength\itemsep{-0.1em}
\item \texttt{Li et al.}, \cite{li2023trustworthy} with a doubly-robust estimator and constrained optimization among depth-2 decision trees via \texttt{econml} python library, where the outcome models and propensity scores are estimated under two-fold cross-fitting. The policy is learned on $\cD_{\text{label}}$ and evaluated on $\cD_{\textrm{test}}$. This method requires that the potential outcomes are non-negatively correlated, which is violated here.
\item \texttt{Policy-Tree}, which maximizes welfare $\EE[Y(\pi(X_{n+1}))]$ among depth-2 decision trees with \texttt{econml}. The policy is learned with $\cD_{\text{label}}$ and evaluated on $\cD_{\textrm{test}}$ with similar cross-fitting estimation of outcome and propensity score models. While this method is not designed for safety control, we include it as the baseline because of its popularity as a standard practice.
\item \texttt{Threshold}, the same as in Section~\ref{subsec:simu_rct} which calibrates the threshold using cumulative estimated $\hat\gamma(X)$ on the test data. This method is valid only when the estimator $\hat{\gamma}$ is sufficiently accurate.
\item \texttt{CPL-sel-welfare}, the method in Section~\ref{sec:obs} using the same $g(x,t)$ and $s(x)$ functions as in the randomized experiment case. The weights are estimated using learn-then-balance with features $\hat\phi(x) = (\hat{\gamma}(x)\ind\{ {s}(x)\leq \hat{t}\}, \hat{w}(x))$, where $\hat{\gamma}$ and $\hat{w}$ are estimated using logistic regression or random forests.
\item \texttt{CPL-sel-power}, the method in Section~\ref{sec:obs} using the same $g(x,t)$ and $s(x)$ functions as in the randomized experiment case. and the weights are estimated in the same way as \texttt{CPL-sel-welfare}.
\item \texttt{Plugin-oracle-sel-welfare}, the method in Section~\ref{sec:stratified} using the correct weights and the same $g(x,t)$ and $s(x)$ functions as \texttt{CPL-sel-welfare}.
\item \texttt{Plugin-oracle-sel-power}, the method in Section~\ref{sec:stratified} using the correct weights and the same $g(x,t)$ and $s(x)$ functions as \texttt{CPL-sel-power}.
\end{enumerate}
\vspace{0.25em}
Here, the last two plugin-oracle methods essentially evaluate \textsc{CPL}\xspace in the stratified experiment setting where the propensity score $e(x)$, and hence the covariate shift weights, are known. Our theory implies their finite-sample harm rate control due to weighted exchangeability, no matter the accuracy of the scores and selective calibration functions.
On the other hand, \texttt{CPL-sel-power} and \texttt{CPL-sel-welfare} evaluate the robustness of CPL in observational studies with estimated weights.
\subsubsection{Harm rate control with known or estimated weights}
We first evaluate the harm rate control using
the same data-generating process as in Section~\ref{subsec:simu_rct}, together with a linear or nonlinear propensity score.
The empirical harm rate, fraction of treatment, and average welfare, averaged over $200$ independent simulation runs, are summarized in Figure~\ref{fig:obs_fna}.
First, the three baselines drastically violate the harm rate control due to similar reasons as in Section~\ref{sec:rct}, now with additional estimation challenges in observational settings.
The two plug-in oracles, \texttt{Plugin-oracle-sel-power} and \texttt{Plugin-oracle-sel-welfare}, confirm the finite-sample harm-rate control of Theorem~\ref{thm:valid_weighted}.
Finally, the two methods with estimated weights, \texttt{CPL-sel-power} and \texttt{CPL-sel-welfare}, also demonstrate robust harm-rate control below the target level.
Although the four CPL methods rely on distinct scoring functions, they achieve similar performance in terms of fraction of treatment and average welfare, showing that the two objectives can align.
\begin{figure}
\centering
\includegraphics[width=\linewidth]{figs/OBS_main_all.pdf}
\caption{(a) Empirical harm rate, (b) power (probability of treatment in the test units), (c) average welfare at level $\alpha=0.1$ in observational studies, averaged over $N=200$ runs. Each row is a prediction model (logistic regression, random forest) for $(\hat\mu_1,\hat\mu_0)$, and each column is a data-generating process. The $x$-axis is the total labeled sample size $n$. The \texttt{Plugin} methods use the true weights.}
\label{fig:obs_fna}
\end{figure}
\subsubsection{Double robustness}
We now additionally examine the double robustness property established in Section~\ref{sec:obs}.
We design a data-generating process where the outcome models and propensity score model are all logistic in some nonlinear transformation of the raw features $X\in \RR^{8}$; see Appendix~\ref{app:subsec_simu_obs} for details.
We sample observational data $\{(X_i,T_i,Y_i)\}_{i=1}^n$ as the labeled dataset, where a random subset of 75\% is used as the training fold for training the models $\mu_1(\cdot)$, $\mu_0(\cdot)$, and $e(\cdot)$, while the remaining is used as the calibration data in \textsc{CPL}\xspace.
We vary a parameter in the propensity score model to control the strength of confounding (the $x$-axis of Figure~\ref{fig:obs_aux}).
The methods evaluated include \texttt{CPL-sel-power/welfare} based on estimated propensity scores, and two known-propensity baselines \texttt{Plugin-oracle-sel-power/welfare} used as oracle comparison only. The four procedures are implemented in the same way as in the preceding parts.
Within each procedure, we vary the model classes of the outcome and propensity score models to demonstrate the double robustness property.
A correct logistic model runs logistic regression over the nonlinearly-transformed features,
while a misspecified logistic model runs logistic regression over the raw features.
By Theorem~\ref{thm:dr_obs}, we expect \texttt{CPL-sel-power/welfare} to control the harm rate when either of them is correctly specified.
Finally, we also consider that both outcome models and propensity scores are trained via random forests, which typically have slower-than-parametric convergence rates yet are less prone to model misspecification; we expect it to control the harm rate, especially when the labeled sample size is sufficiently large.
The empirical harm rate averaged over $200$ independent simulation runs at nominal level $\alpha=0.05$ is summarized in Figure~\ref{fig:obs_aux}. In the first three columns, when either the outcome models or the propensity score model is well-specified, we observe tight harm-rate control by the two CPL variants, which is also close to the plugin-oracle ones. This validates the double robustness theory.
When the outcome models are misspecified, we observe a larger slack in the harm rate than the other two configurations when sample size is small.
On the other hand, when both models are misspecified (the fourth column), the two CPL variants can violate the harm rate control, yet the violation is moderate.
We note that we design the settings deliberately to fail the both-misspecified procedures. In many other settings, misspecification can create a slack in the conservative calibration through the proxy outcome $Y_i^\dagger$, which often compensates the misspecification in the weights and keeps the harm rate below the budget even though both models are wrong.
Finally, the nonparametric models in the fifth column yield tight harm rate control, showing the robustness to model misspecification and the quick convergence of the harm rate in \textsc{CPL}\xspace.
\begin{figure}
\centering
\includegraphics[width=0.8\linewidth]{figs/OBS_aux_fna.pdf}
\caption{Empirical harm rate in the double robustness experiments; results are averaged over $N=200$ independent runs. Each column corresponds to one specification of outcome and propensity score models. Each row corresponds to a total labeled sample size. The $x$-axis is the strength of confounding.}
\label{fig:obs_aux}
\end{figure}
\section{Empirical Application to AI-Powered Interventions}
\label{sec:real}
In this section, we apply \textsc{CPL}\xspace to a real-world dataset in the social sciences, where the AI model, ChatGPT, is used as an intervention to persuade participants out of some conspiracy beliefs, in which events are understood as being caused by secret, malevolent plots involving powerful conspirators \citep{costello2024durably}. The original study found that brief conversations with AI could reduce conspiracy beliefs by 20 percent on average, and the effect was durable for at least 2 months. Given the societal concern over widespread conspiracy theories, an increasing number of researchers and policymakers are evaluating similar AI interventions as a scalable solution. If policymakers scale up such AI interventions, it is of significant importance to consider safety, as the treatment effects of AI interventions are likely to be highly heterogeneous for several reasons. First, ``people believe a wide range of
conspiracies, and the specific evidence brought to bear in support of even a particular conspiracy
theory may differ substantially from believer to believer'' \citep[page 1,][]{costello2024durably}. Second, as AI chatbots treat people with natural texts as the intervention, the content of the treatment itself is heterogeneous and unpredictable. Here, to safely scale up these types of AI-powered interventions, we use CPL to decide who should be treated by AI by controlling the harm rate with the safety guarantee.
In this study, before the experiment, the participants stated a conspiracy theory they believed in and reported a numerical score quantifying their belief in it. They are then randomly assigned to treated and control groups, where treated participants engage in a live conversation with a GPT model that is instructed to talk them out of the conspiracy, while the participants in the control condition engage with a neutral conversation with the GPT model. After the experiment, they again report a numerical score quantifying their belief in the same conspiracy theory.
We take all participants in the raw dataset as the analysis population. We binarize the outcome $Y$ to indicate whether the post-experiment belief score is below $50$, a cutoff the authors originally used to define their analysis population. The participants are randomly split into training (40\%), calibration (40\%), and test (20\%) folds. There are 416 participants originally treated out of 667 participants in the test fold. The fraction of $Y=1$ in the treated group is $0.274$, while the fraction of $Y=1$ in the control group is $0.100$. We consider a stringent harm rate of $\alpha=0.025$.
We build features based on participants' demographic covariates (education, age, gender, race, religion), political and psychological covariates, AI-related covariates (familiarity and trust in generative AI), baseline belief state variables and textual embedding for the stated conspiracy.
The training fold is used to fit the outcome models and conditional treatment effect models, which are used in a similar way as in the simulations to build welfare-maximizing score functions (except that we truncate on extremely small estimated harm rate for stability).
See Appendix~\ref{app:econml-welfare} for the detailed implementation.
\vspace{-0.5em}
\paragraph{Empirical welfare and power.}
We report power (the fraction of treated units in the test data) and (estimated) empirical welfare of the welfare-maximizing and power-maximizing variants of \textsc{CPL}\xspace. We also compare the results against three baselines as references: the first is to treat everyone in the test data (All treat), the second is to treat no one in the test data (All control), and the third is to treat $2.5$\% of test units, which trivially satisfies the safety constraint.
Because we observe the realized outcome in the test data, we can estimate the average welfare and harm rate of the policy as follows. Let $T_{n+j}^{\text{real}}\in \{0,1\}$ be the actual treatment assigned by the experiment, and $T_{n+j} = \hat\pi(X_{n+j})$ be the decision produced by \textsc{CPL}\xspace.
We are interested in the average welfare $\text{Welfare}(\hat\pi):=\EE[Y_{n+j}(\hat\pi(X_{n+j}))]
= \EE[Y_{n+j}(1)\cdot\hat\pi(X_{n+j}) +Y_{n+j}(0)\cdot(1-\hat\pi(X_{n+j}))]$, for which an unbiased estimate is
\@\label{eq:real_welfare_est}
\hat{\text{Welfare}} = \hat\EE_{\textrm{test}}\big[Y_{n+j}\cdot \hat\pi(X_{n+j})\given T_{n+j}^\text{real}=1\big]
+
\hat\EE_{\textrm{test}}\big[Y_{n+j}\cdot (1-\hat\pi(X_{n+j}))\given T_{n+j}^\text{real}=0\big],
\@
where $\hat\EE_{\textrm{test}}$ denotes the empirical average in the test fold.
We also estimate the harm rate
$
\PP(Y_{n+j}(1)<Y_{n+j}(0),\hat\pi(X_{n+j})=1)
$
by the following conservative estimator.
\@\label{eq:est_fna}
\hat{\text{Harm}} =& \hat\EE_{\textrm{test}}[(1-Y_{n+j})\ind\{1-\hat\mu_1(X_{n+j})\leq \hat\mu_0(X_{n+j})\}\hat\pi(X_{n+j})\given T_{n+j}^{\text{real}}=1\big] \notag \\
&\qquad + \hat\EE_{\textrm{test}}[ Y_{n+j} \ind\{1-\hat\mu_1(X_{n+j}) > \hat\mu_0(X_{n+j})\}\hat\pi(X_{n+j})\given T_{n+j}^{\text{real}}=0\big] .
\@
Note that $\hat{\text{Harm}}$ is unbiased and asymptotically normal for the population quantity $ \EE\big[ \{ (1- \mu_1(X_{n+j}))\ind\{1-\hat\mu_1(X_{n+j})\leq \hat\mu_0(X_{n+j})\} + \mu_0(X_{n+j})\ind\{1-\hat\mu_1(X_{n+j})> \hat\mu_0(X_{n+j})\} \} \cdot \hat\pi(X_{n+j}) \big]$, which upper bounds the harm rate. This is a valid conservative estimator of the harm rate even if $\hat\mu_t(x)$ is misspecified, and this is a consistent estimator for the sharp upper bound of the harm rate when $\hat\mu_t(x)$ is consistently estimated.
\paragraph{Results.} The main results are summarized in Table~\ref{tab:real_welfare}. Several points are worth noting. First, treating everyone (All treat) violates the safety constraint as its harm rate is $3.92$\% and exceeds $2.5$\%. In contrast, our proposed CPL methods achieve the tight control of harm rates at 2.5\% and 2.1\%, respectively. Second, while treating no one (All control) and treating only $2.5$\% of test units (Trivially-safe), of course, satisfy the safety constraint, they have extremely low power and welfare. In contrast, our proposed methods can treat more than 90\% of test units and achieve the high average welfare. While maintaining the safety constraint, our method simultaneously achieves high power and welfare. Finally, it is important to note that the difference between our welfare-maximizing and power-maximizing variants are minimal in practice. It is true that, consistent with our theory, our power-maximizing variant has a slightly higher fraction of treated test units, and our welfare-maximizing variant has a slightly higher average welfare. However, overall, their actual policy decision on who gets treated is similar, which implies that researchers can use either variant in practice and expect to approximately optimize both power and welfare in various applications.
\begin{table}
\centering
\begin{tabular}{c|c|c|c|c|c}
\midrule
& All treat & All control & Trivially-safe & \textsc{CPL}\xspace-welfare & \textsc{CPL}\xspace-power \\
\hline
Est.~harm rate & 0.0392 (0.0111) & 0 & 0.001 (0.0002) & 0.0247 (0.0094) & 0.0207 (0.0086) \\
\hline
Num.~treatment & 667 & 0 & 16 & 605 & 615 \\
\hline
Est.~welfare & 0.274 (0.0218) & 0.0996 (0.0189) & 0.104 (0.0184) & 0.254 (0.0246) & 0.250 (0.0231) \\
\midrule
\end{tabular}
\caption{Estimated harm rate, number of treatment, and estimated welfare (standard deviation) on the test fold using different methods. ``All treat'' treats all test units; ``All control'' treats no test units; ``Trivially-safe'' randomly treats $\alpha$-fraction of test units. The welfare/harm rate estimates are based on~\eqref{eq:real_welfare_est} and~\eqref{eq:est_fna}.}
\label{tab:real_welfare}
\end{table}
\vspace{-0.5em}
\paragraph{Safe treatments by \textsc{CPL}\xspace.}
We now take a closer look at the decisions produced by the welfare-maximizing varinat of \textsc{CPL}\xspace.
Figure~\ref{fig:real_rf_sel_scatterplot} visualizes the treatment decisions, where panel (a) plots the test points based on the predicted harm rate $\hat{\gamma}(X)$ and predicted treatment effect $\hat\tau(X)$, while panel (b) plots the test points based on the predicted outcomes $\hat\mu_0(X)$ and $\hat\mu_1(X)$.
\textsc{CPL}\xspace treats the test units with the largest values of $\hat\tau(X)/\hat{\gamma}(X)$. The decision boundary is plotted in both panels, and the region not treated is indicated in light grey. While the actual harm $\ind\{Y_{n+j}(1)<Y_{n+j}(0)\}$ is not observed, we conservatively estimate it by checking the label $Y_{n+j}^\dagger = T_{n+j}^\text{real}Y_{n+j}+(1-T_{n+j}^\text{real})(1-Y_{n+j})$, where $T_{n+j}^\text{real}$ is the actual treatment received by the $j$-th test unit, among those with $\hat{g}(X_{n+j},T_{n+j}^\text{real})=1$, i.e., either $\hat\mu_1(X_{n+j})\leq \hat\mu_0(X_{n+j})$ and $T_{n+j}^\text{real}=1$ or $\hat\mu_1(X_{n+j}) > \hat\mu_0(X_{n+j})$ and $T_{n+j}^\text{real}=0$. The colored dots are those with $\hat{g}(X_{n+j},T_{n+j}^\text{real})=1$, among which the blue dots are those with $Y_{n+j}^\dagger=1$ and the red dots are those with $Y_{n+j}^\dagger=0$.
We observe that very few red dots in the treated region can possibly be harmed.
\begin{figure}[htbp]
\centering
\includegraphics[width=0.8\linewidth]{figs/grf_wfsel_0025.pdf}
\caption{Visualization of treatment decisions by \textsc{CPL}\xspace using the welfare-maximizing variant, where outcome models are estimated by causal forests. Colored dots are those with $\hat{g}(X,T)=1$, which provide a conservative bound for harm; red dots are those who can possibly be harmed, with $Y^\dagger=0$.}
\label{fig:real_rf_sel_scatterplot}
\end{figure}
We further examine how the treatment decision by \textsc{CPL}\xspace varies with participants. Figure~\ref{fig:real_rf_sel_by_pre_belief_wf} plots the moving average of predicted treated and control outcomes, as well as the fraction of safe treatments, for every value of pre-experiment belief scores (smoothed by a moving window of size $10$).
In general, the two outcome curves indicate that a stronger pre-treatment belief makes it less likely to be persuaded out of the conspiracy in both conditions, but the effect of the AI intervention seems strong for participants with a strong pre-treatment belief.
The welfare-maximizing variant of \textsc{CPL}\xspace prioritizes a treatment-effect-versus-harm-rate tradeoff. It mainly treats units with firm pre-treatment belief for whom the treatment is likely to make a huge difference (the gap between the two blue curves) while the estimated harm rate is relatively low.
\begin{figure}[!t]
\centering
\includegraphics[width=0.7\linewidth]{figs/grf_pre_belief_wf_0025.pdf}
\caption{Fraction of safe treatment (red), average predicted treated (dark blue) and control (light blue) outcomes among test participants within a moving window of self-reported pre-experiment conspiracy belief.}
\label{fig:real_rf_sel_by_pre_belief_wf}
\end{figure}
Figure~\ref{fig:real_rf_sel_by_fam_trust} similarly reports this information among participants with a specific GenAI familiarity score (panel a) and GenAI trust score (panel b). In this case, however, the outcomes and treatment decisions do not change significantly based on these features, indicating that AI intervention may be similarly safe for users with different familiarity with or trust in GenAI.
\begin{figure}[!t]
\centering
\includegraphics[width=0.8\linewidth]{figs/grf_by_fam_trust_wf_0025.pdf}
\caption{Fraction of treatment by the welfare-maximizing variant (red), average predicted treated (dark blue) and control (light blue) outcomes among test units stratified by self-reported GenAI-related variables.}
\label{fig:real_rf_sel_by_fam_trust}
\end{figure}
\section{Discussion}
\label{sec:ext}
In this article, we developed the CPL approach that allows for individualized treatment assignment with a safety guarantee. We prove that when learning a treatment assignment rule from randomized experiments, CPL provides the safety guarantee in a finite sample without imposing any modeling assumption about potential outcomes. Additionally, when the outcome regression model is consistently estimated as assumed in many existing methods, CPL also asymptotically achieves the optimal welfare and power with appropriate choices of score $V$ and inclusion $g$ functions. We then extended our results to observational studies and derived novel asymptotic doubly robust safety guarantees for CPL.
We now briefly discuss several natural extensions of our proposed conformal policy learning. The first concerns external validity settings where the test units may come from a different superpopulation than the labeled data and Assumption~\ref{assump:iid} is violated \citep[e.g.,][]{egami2023elements,jin2025beyond}. The most common and natural strategy is to relax the i.i.d. assumption to the covariate shift assumption, i.e., the distributions of the labeled data and the test data differ only in the covariate distribution. Under this setting, we can generalize CPL by simply multiplying the current conformal weights by additional weights $w_{Q/P}(x):=\ud \QQ_X/\ud \PP_X(x)$ that account for the covariate shift density ratio where $\PP$ denotes the distribution of the labeled data and $\QQ$ represents the distribution of the test data. Assuming both $e(x)$ and $w_{Q/P}(x)$ are known, CPL then proceeds in exactly the same way as in Section~\ref{sec:stratified}. When either of them is unknown, one may use similar strategies as in Section~\ref{sec:obs} to estimate and plug in these quantities.
The second natural extension concerns the control of harm rate among subgroups. In many problems where fairness and equity are stressed, it is desirable to maintain the harm rate control within each subgroup, such as those defined by demographic features~\citep{romano2020malice}. Following the setup in the main framework, our goal is to find the treatment assignment rule $\hat\pi(X_{n+1})\in \{0,1\}$ such that, for a group indicator $\cG(\cdot)$,
$\PP( Y_{n+1}(\pi(X_{n+1})) <Y_{n+1}(0)\given \cG(X_{n+1})=1)$.
Given a new sample with $\cG(X_{n+1})=1$, this can be achieved by taking the calibration data from the same subgroup, i.e., $\{(X_i,T_i,Y_i)\colon \cG(X_i)=1\}$.
These data are induced by the full data in the subgroup $\{(X_i,Y_i(1),Y_i(0))\colon \cG(X_i)=1\}$ which is exchangeable with the new test sample conditional on the group indicator. The same \textsc{CPL}\xspace procedures can then be performed within the subgroup for randomized experiments and observational studies.
\section*{Acknowledgments}
The authors thank Eli Ben-Michael and Molly Offer-Westort for helpful discussions at the Online Causal Inference Seminar and the Political Methodology summer meeting, respectively. We also thank seminar participants at Yale Economics, Harvard Applied Stats Workshop, University of Tokyo Applied Stats Seminar, and the Political Methodology summer meeting. Y.J.~is partially supported by NSF DMS-2610282.
\newpage
\bibliographystyle{apalike}
\bibliography{reference}