Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
153,192 characters · 24 sections · 90 citation commands
Personalized Policy Learning through Discrete Experimentation: Theory and Empirical Evidence
\TITLE{Personalized Policy Learning through Discrete Experimentation: Theory and Empirical Evidence} \RUNTITLE{Personalized Policy Learning through Discrete Experimentation} \RUNAUTHOR{Zhang et al.}
\ARTICLEAUTHORS{ \AUTHOR{Zhiqi Zhang$^1$, Zhiyu Zeng$^2$, Ruohan Zhan$^3$, Dennis J. Zhang$^1$} \AFF{ $^1$ Washington University in St. Louis, St. Louis, MO\\ $^2$ Shanghai Jiao Tong University, Shanghai, Shanghai, China\\ $^3$ University College London, London,UK\\ [email removed], [email removed], [email removed], [email removed] } }
\ABSTRACT{ Randomized Controlled Trials (RCTs), or A/B testing, have become the gold standard for optimizing various operational policies on online platforms. However, RCTs on these platforms typically cover a limited number of discrete treatment levels, while the platforms increasingly face complex operational challenges involving optimizing continuous variables, such as pricing and incentive programs. The current industry practice involves discretizing these continuous decision variables into several treatment levels and selecting the optimal discrete treatment level. This approach, however, often leads to suboptimal decisions as it cannot accurately extrapolate performance for untested treatment levels and fails to account for heterogeneity in treatment effects across user characteristics. This study addresses these limitations by developing a theoretically solid and empirically verified framework to learn personalized continuous policies based on high-dimensional user characteristics, using observations from an RCT with only a discrete set of treatment levels. Specifically, we introduce a deep learning for policy targeting (DLPT) framework that includes both personalized policy value estimation and personalized policy learning. We prove that our policy value estimators are asymptotically unbiased and consistent, and the learned policy achieves a $\sqrt{n}$-regret bound. We empirically validate our methods in collaboration with a leading social media platform to optimize incentive levels for content creation. Results demonstrate that our DLPT framework significantly outperforms existing benchmarks, achieving substantial improvements in both evaluating the value of policies for each user group and identifying the optimal personalized policy.
} \KEYWORDS{Deep Learning, Policy Learning, Experimentation, Platform Operations}
In the rapid evolution of digital platforms, it has become increasingly common for firms to make decisions by running randomized controlled trials (RCTs), also known as experimentation or A/B testing. This method randomly assigns users to either treatment groups (which receive new candidate decision variables as interventions) or a control group (which maintains the status quo), compares their outcomes, and selects the optimal levels for each user group. Tech giants frequently run thousands of online experiments simultaneously to fine-tune algorithms, optimize user interface designs, and tailor marketing messages kohavi2017surprising. Recent advances in AI and machine learning have further enabled platforms to personalize interventions rafieian2023ai.
One major limitation of running RCTs on these platforms is limited sample sizes: due to the finite number of users, these platforms cannot test too many treatment levels in a single RCT, nor can they accurately estimate personalized treatment effects for very small subsets of users. This limitation is less critical when platforms focus on comparing a small number of discrete policies, such as a newly updated recommender system versus the original one feldman2022customer. However, challenges arise when decision variables are continuous and policies need to be personalized, a scenario that is increasingly common in practice. For instance, e-commerce platforms like Alibaba frequently adjust their pricing strategies to optimize sales and profits zhang2020long.
When decision variables are continuous or have too many levels, current industry practice typically involves discretizing the treatment into several distinct levels, randomly assigning each level to a user group, and selecting the best-performing one. For example, ride-sharing platforms may experiment with different incentive levels for drivers, selecting the level that delivers the highest return on investment christensen2023demand, while also testing various price levels during peak demand periods to assess customer price elasticity cohen2016using. Similarly, streaming platforms may experiment with different discount durations—offering lower per-period costs for longer subscription commitments—and choose the option that maximizes user engagement tian2020optimizing.
While such industry practices can help identify the best treatment levels among those tested, discretization limits the range of decision variables evaluated and prevents extrapolation to treatment levels outside the experimented set. As a result, the selected treatment from the experiment may be significantly outperformed by the optimal decision within the full continuous space. Moreover, relying solely on simple comparisons from A/B test results overlooks complex interactions between treatments and high-dimensional user characteristics, often resulting in suboptimal and non-personalized policies. This highlights the need for developing methods to identify optimal personalized policies using observations from only a discrete set of treatment levels in an RCT.
Therefore, in this paper, we aim to address the following research questions. First, how can we accurately estimate policy value over a continuous treatment space, conditioned on high-dimensional features, when observing only a discrete number of experimental arms on this continuous variable? Furthermore, how can we derive the optimal personalized policy for each feature combination? Our goal is to develop a theoretical framework with provable guarantees and rigorously validate our methods through real-world RCTs and applications.
Theoretically, we propose a Deep Learning framework for Policy Targeting, referred to as {\sf DLPT}, to estimate policy values conditional on high-dimensional personalized features and to identify optimal personalized policies using only experimental data with discrete treatment levels. Specifically, our model assumes a semi-parametric data generation process, where the outcome variable depends parametrically on the continuous treatment variables but non-parametrically on the features of each data point (i.e., the context). The model first translates each data point's features into a set of nuisance parameters, which subsequently determines the parameters in the specified functional form of the continuous treatment variables. As a result, the model accommodates a wide range of continuous functions for treatments, making it particularly powerful for extrapolating treatment effects beyond the observed discrete treatment levels.
With this model, we implement a three-stage method for continuous policy evaluation and optimization. In the first stage, we estimate the nuisance parameters in our model via nonparametric methods like neural networks. In the second stage, we estimate the policy value using universal orthogonal score functions, leveraging the nuisance estimates from the first stage. This ensures that our estimator is asymptotically unbiased and semi-parametrically efficient. In the third stage, we learn the optimal personalized policy by optimizing the estimated policy values and provide theoretical guarantees for the optimality of the learned policy. Particularly, our policy achieves a regret bound of $O\left(r\left(1+\sqrt{\log(1/r)}\right)\sqrt{d/n} + r\sqrt{\log(1/\delta)/n}\right)$, where $r$ specifies the distance between any two policies in the policy class $\Pi$ in terms of the estimated policy value, and $d$ represents the complexity of the policy class $\Pi$. Notably, our results demonstrate that the learned policy achieves a minimax optimal regret rate of $\sqrt{n}$ within a fixed policy class.
We then validate our framework empirically in collaboration with one of the leading social media platforms worldwide, referred to as “Platform O." Similar to Instagram, users on Platform O can both create and consume content, and the platform’s revenue is primarily driven by advertisements viewed by users. The platform is particularly interested in improving user-generated content production, which in turn enhances user engagement and increases future ad revenue. In partnership with this platform, we conducted a large-scale RCT involving 12.3% of its users from August 23 to August 29, 2021. The experiment aimed to encourage users to create new videos by offering monetary rewards. Four different levels of monetary rewards were tested, resulting in four treatment groups and one control group (which didn't receive the monetary incentives). Using this experiment, we aim to evaluate our framework's performance on both policy value estimation and policy evaluation, benchmarked with a set of frameworks from the literature and industry practices.
Our empirical evaluation comprises two components. First, we assess the performance of DLPT against a set of benchmark methods using experimental data. We begin by measuring the accuracy of Average Treatment Effect (ATE) estimates: for each of four discrete monetary‐reward levels, we withhold that level from training, estimate its effect on users’ video‐upload behavior, and compare to the true ATE. DLPT achieves a mean absolute percentage error (MAPE) of 2.25%, whereas benchmark MAPEs span from 7.18% to 43.59%, demonstrating substantially greater precision in recovering out‐of‐sample ATEs. Next, we evaluate policy‐learning regret within the finite observed policy class (the five experimental reward levels). By concealing half of the user subgroups during training and computing regret on these held‐out groups, we find that DLPT attains the lowest mean percentage regret (MPR) of 2.22%, compared with benchmark MPRs ranging from 6.32% to 10.05%.
Second, we evaluate the ability of these frameworks to identify the personalized optimal incentive policies beyond the finite observed policy class. We construct semi–synthetic data by fitting a fully connected neural network (FNN) to the observed treatments and user covariates; this FNN then serves as the “ground truth” data‐generating process. Policy performance is evaluated across four classes: (i) the Finite Observed class (restricted to the five experimental levels), (ii) a Discretized Continuous class, (iii) a Linear class, and (iv) a DNN class. Across all four classes, DLPT consistently outperforms the benchmarks, achieving MPRs between 0.01% and 19.69%, whereas benchmark MPRs lie between 0.66% and 34.75%. Within each policy class, the policy learned by DLPT yields the lowest regret, underscoring its flexibility and superior capacity to recover optimal incentive policies.
Our study makes several pivotal contributions to both academic research and practical applications in experimentation and platform analytics. From a methodological perspective, we are the first, to our best knowledge, to propose a framework for estimating policy values and learning personalized policy over a continuous treatment variable when only observing discrete sets of treatment levels. The proposed framework and estimation procedure guarantee a $\sqrt{n}$ regret bound for the learned policy. In real-world settings, we have validated our method in collaboration with a major online platform. The results confirm that our framework not only upholds strong theoretical promises but also excels in practical deployments, effectively handling extensive user features and nuanced treatment effects. More broadly, we develop our method for post-experiment analysis, allowing for seamless integration into existing experimental platforms. This facilitates its adoption by practitioners who can readily apply our approach to enhance decision-making processes without modifying their existing experimentation infrastructure.
The remainder of the paper is organized as follows. We review the related literature in Section (ref). Section (ref) introduces the {\sf DLPT} framework, outlining our proposed three-stage method along with theoretical guarantees. In Section (ref), we describe the empirical setting of a large-scale field experiment. Section (ref) presents the empirical evaluation of the {\sf DLPT} framework using the experimental data. We further assess the performance of {\sf DLPT} through semi-synthetic studies in Section (ref). Finally, we conclude the paper in Section (ref). All codes for synthetic studies can be accessed in the Github Repository.\footnote{See \url{https://anonymous.4open.science/r/Policy-Learning-through-Discrete-Experimentation-Theory-and-Empirical-Evidence-0B0B}}
Our research builds on three key streams of literature: personalized pricing, policy learning and causal Machine Learning (ML), and experimentation on platforms.
Personalized Pricing. Our paper connects to personalized pricing, a critically important and well-established topic that has garnered significant attention across the fields of operations, economics, and marketing. This research can broadly be categorized into two main types: (1) pure modeling studies, which develop sophisticated mathematical models to analyze broad market dynamics, such as the effects of personalized pricing on social welfare, market efficiency, consumer surplus, and producer profits aguirre2010monopoly,chen2020competitive,elmachtoub2021value,ke2024algorithmic; and (2) applied economics studies, which rely on empirical analyses using observational or experimental data and to which our research contributes.
Applied economics studies in this stream focus on the consumer side of the market and aim to develop and implement personalized pricing strategies by empirically estimating demand curves at various price points. smith2023optimal implement several model specifications such as the Bayesian choice model and multinomial logit model (MNL) and use an inversed propensity weighted (IPW) estimator for profits. hitsch2024heterogeneous use conditional average treatment effect (CATE) estimators to find optimal policies. Various models, including linear regression models aryal2024price, random coefficient models rossi1996value,aribarg2021price, and nonparametric regression models hausman1995nonparametric,chen2021nonparametric,wang2024nonparametric, are also wildly used in demand estimation in applied research. In the meanwhile, ML techniques have increasingly been integrated into demand estimation within the field of personalized pricing, primarily due to the availability of richer and more comprehensive data sets bajari2015machine,qi2022offline,dube2023personalized,yoganarasimhan2023design,wei2024neural.
Table (ref) summarizes representative literature in applied research of personalized pricing with data used in the paper (whether experimental data or observational data), personalization capability, causal inference capability, optimized policy type (discrete or continuous), and validation of the methods proposed (using cross-validation, simulation data, or another experiment data). We contribute to this literature by proposing a double machine learning-based framework for personalized policy targeting that is capable of causal inference for discrete experiments, policy personalization over continuous policy range, and methods validation through real-world data as well as synthetic studies.
Policy Learning and Causal ML. Our research speaks to policy learning, a dynamic field at the intersection of economics, medical research, operations, and ML, which has evolved significantly over the past few years. Conventionally, policy learning can be distinguished based on the nature of the treatment variables and divides into two categories: discrete and continuous variables. Discrete variables deal with decisions that have a set number of distinct options. athey2021policy use doubly robust score to find optimal treatment allocation as a binary policy. zhou2023offline propose a method for multi-action policy learning with historical data. xiong2024staggered propose optimal experimental design for staggered rollouts. zhan2024policy extend the policy learning to adaptively collected data settings. In medical research, there is a substantial body of literature dedicated to identifying optimal individualized treatment rule (ITR) using observational data zhao2012estimating,zhao2015new,pan2021improved. Continuous variables optimization involves treatments that can vary without limitation. chen2016personalized introduce a non-convex loss function designed to refine the individualized dose rule (IDR) problem, and the method requires randomized trials where candidate dose levels assigned to study subjects are randomly chosen from a continuous distribution within a safe range. kallus2018policy propose a continuous-treatment evaluator with a kernel function to incorporate localized data from comparable treatments.Both chen2016personalized and kallus2018policy validate their methods through warfarin dataset international2009estimation with multiple levels of warfarin dosage. Our work builds on chernozhukov2019semi where the authors focus on observational study and propose a semi-parametric efficient framework for off-policy evaluation and optimization with continuous action spaces. We underscore our contribution in providing our framework {\sf DLPT}, an easily implementable approach that can extrapolate from a limited number of discretized experiments to learn optimal personalized continuous treatments. Simultaneously, our method adeptly manages causal inference, enhancing its applicability and effectiveness in practical settings.
Our work also connects to the theory and application of double machine learning (DML) chernozhukov2018double. By integrating machine learning with Neyman orthogonality, DML excels in estimating parameters of interest, even when faced with challenges such as regularization and overfitting biases in nuisance parameter estimation. farrell2020deep,farrell2021deep have developed innovative nonasymptotic high-probability bounds for estimating nuisance parameters using deep neural networks (DNN). knaus2022double utilizes DML to assess the effectiveness of four labor programs in Switzerland. ye2023deep propose a deep learning-based causal inference framework for estimating treatment effects for combinatorial experiments, showcasing the integration of advanced machine learning techniques with causal analysis methodologies. We extend the existing DML framework to accommodate continuous policy learning settings. This extension leverages the strengths of machine learning in managing high-dimensional individual data. Our enhanced framework has been validated through the use of real-world experimental data alongside extensive synthetic studies, demonstrating its effectiveness and robustness in practical applications.
Experimentation on Platforms. Large-scale experiment analysis of platforms is typically segmented into pre-experiment studies and post hoc experiment analysis. Pre-experiment studies primarily concentrate on the design of the experiment. johari2022experimental analyze the bias of estimators on different sides (customer-side and listing-side) of randomization for experiments on online platforms. ye2023cold propose a two-sided randomization experiment to evaluate the effectiveness of new algorithms. Additionally, the field is seeing advances in adaptive sequential experiments, where user information is incorporated as it becomes available gur2022adaptive,zhao2024pigeonhole, leading to updated treatment assignments xiong2023optimal and optimized experimental strategies such as multiarm-bandit experimental design simchi2023multi. There is also growing interest in addressing experimental interference, which can skew results in large-scale platforms. wager2021experimenting tackle this by assuming structured interference and using mean-field models to account for it effectively. bojinov2023design derives the optimal design of switchback experiments considering the carry-over effect. holtz2024reducing use cluster randomization to deal with possible interference issues. zhan2024estimating apply structured choice model to address seller-side competition in recommendation phase. Recent methodological advances have also introduced augmented difference-in-differences methods for causal inference li2023augmented.
In contrast, post hoc experiment analysis offers a wide range of applications, enabling platforms to perform causal inference based on observed outcomes of discrete treatments, thereby supporting more informed and effective decision-making. For instance,yu2022delay document the impact of waiting time on users abandonment on a ride-sharing platform. zeng2023impact investigate the causal effects of social nudges on content creation through a large-scale experiment. Building upon this body of work, our contribution extends beyond traditional causal inference by offering a novel extrapolation from existing experiments. Our framework {\sf DLPT} is tailored for discrete experiment settings and requires only mild assumptions for the identification of nuisance parameters. It enhances operational flexibility significantly by enabling not only causal inference for current treatment arms but also personalized continuous policy learning. This approach allows for valid extrapolation without the need for conducting complex, multi-arm A/B tests, simplifying the experimental process while broadening the scope of actionable insights.
In this section, we introduce our theoretical framework, {\sf DLPT}, for personalized policy learning. The framework first extrapolates the causal effects of discrete experimented treatments to the entire continuous space, and then find the optimal personalized continuous-treatment policy based on the extrapolation. We start with formalizing the problem and then propose our method accompanied with its theoretical guarantees.
Problem Setup.\footnote{ On notations: Throughout the paper, vectors and matrices are denoted in boldface. Vectors are treated as column vectors, and $\bm{v}'$ denotes the transpose of vector $\bm{v}$. Random variables are represented by capital letters, and their realizations by lowercase letters.} A platform conducts a randomized experiment to determine the optimal treatment level within a continuous treatment space \( \mathcal{T} = [0, \bar{T}] \subset \mathbb{R} \), where \( \bar{T} < \infty \), with the goal of maximizing an outcome of interest (e.g., revenue). Due to operational constraints, the platform can only implement \( m \) discrete treatment levels, \( \{ t_1, t_2, \ldots, t_m \} \subset \mathcal{T} \), during the experiment. Let \( T \) denote the treatment level assigned to a user. The platform observes the assigned treatment level \( T \), as well as high-dimensional and bounded pretreatment covariates \( \bm{X} \in \mathcal{X} \in \mathbb{R}^{d_{\bm X}} \). Conditional on covariates, each treatment level is assigned with equal probability,\footnote{Our method does not require uniform treatment assignment; this assumption is made purely for clarity of exposition.} i.e., $ \mathbb{P}(T = t_j \mid \bm{X} = \bm{x}) = \frac{1}{m}, \quad \forall j \in \{1, \ldots, m\}.$ After treatment assignment, the user reveals an outcome \( Y \in \mathcal{Y} \subset \mathbb{R} \), which the platform can observe. Without loss of generality, we focus on binary outcomes \( Y \in \{0,1\} \); our results extend directly to general bounded outcomes.
Following farrell2020deep, we assume that the underlying data generation process (DGP) has a semi-parametric specification such that
Above, the link function $G(\cdot)$ is parametrically specified a priori, which can vary from linear to non-linear forms, such as sigmoid, depending on the specific domain knowledge and hypotheses stipulated by the researchers. The nonparametric part, $\bm{\theta}^*(\cdot)$, is typically known as “nuisance parameter” in causal inference literature newey1994asymptotic, which is designed to be highly adaptable---without parametric restrictions---to encapsulate a summary of user characteristics and their interactions with the treatment level $t$. This $\bm{\theta}^*(\cdot)$ shall be learned using deep neural networks (DNNs) in our empirical settings, which afford the flexibility needed to capture complex, non-linear relationships that may exist between user attributes and treatment effects.
In our analysis, we assume a specific sigmoid-polynomial specification for $G(\cdot)$ as the DGP. Accordingly, specification (ref) is detailed as follows:
where \( \tilde{\bm{T}}_K(t) = (1, t, t^2, \ldots, t^K)' \in \mathbb{R}^{K+1} \) denotes the degree-\( K \) polynomial feature vector constructed from the treatment variable \( t \). The coefficient vector \( \bm{\theta}_K^*(\bm{x}) \in \mathbb{R}^{K+1} \) varies with covariates \( \bm{x} \), and governs the influence of each polynomial term in the transformed treatment input. As to be discussed in Remark 1, the optimal $K$ is chosen as the number of discrete treatments minus $1$. Note that this functional form assumption for $G(\cdot)$ may be misspecified. However, even if the true data generation process is arbitrary, our specification above can offer strong approximation power. We provide further theoretical analysis in Section (ref).
Policy. A policy $\pi$ is a treatment assignment rule that maps from the covariate space $\mathcal{X}$ to the treatment space $\mathcal{T}$. For a given user with covariates \( \bm{x} \), the value \( \pi(\bm{x}) \) denotes the treatment level that the policy prescribes for that user based on her characteristics. Let $H(\cdot)$ denote the value function chosen by researchers or practitioners. For example, $H(\bm{\theta}_K^*(\bm{x}),\pi(\bm x)) = wG(\bm{\theta}_K^*(\bm{x}),\pi(\bm x))-c\pi(\bm x)$ represents net benefit as the profit gained from users outcome with weight $w$ minus policy cost with unit cost $c$. For a given policy $\pi$, we use $V_K(\pi)$ to denote its policy value, which represents the expected value when the treatment assignment follows policy $\pi$:
where the expectation is taken over the covariate distribution.
Goal. Given a pre-specified policy class $\Pi$ (such as linear policies or decision-tree policies) and the current semi-parametric DGP assumption $G(\bm \theta^*_K(\bm x),t)$, we aim to learn a policy $\hat{\pi}\in\Pi$ from the data, such that its policy value is maximized, or equivalently, we aim to minimize its regret $R_K(\hat{\pi})$, which measures the policy's suboptimality as defined below:
where $\pi_K^*=\arg\max_{\pi\in\Pi}V_K(\pi)$ denotes the optimal in-class policy.
Three-Stage Policy Learning Method: {\sf DLPT}. We propose a three-stage method to learn optimal policies, which we term as Deep Learning Policy Targeting ({\sf DLPT}). The key is to causally extrapolate the policy value of continuous treatment $T$ from discrete observed experiments. Our method unfolds in the following stages: (i) estimating the nuisance parameter $\bm{\theta}_K^*(\bm{x})$ via structured ML models; (ii) estimating counterfactual policy values using Neyman orthogonal scores; and (iii) learning the optimal personalized policy within the specified policy class. We detail each stage in sections (ref), (ref), and (ref) respectively and complement these procedures with theoretical guarantees for policy value estimation and policy learning.
In later sections, we shall instantiate our {\sf DLPT} method in the field. Specifically, our empirical setting considers outcome $Y$ as a binary variable indicating user video uploads, the treatment $T$ as a continuous variable denoting monetary incentive offered to encourage user creation, and $\bm X$ as users' features (See Section (ref) for detailed empirical setting). Our goal is to learn a cost-effective policy for offering monetary incentives. Although this illustration is specific to our empirical application, our {\sf DLPT} method is designed to be flexible and can be customized to accommodate various continuous outcome variables. Let \( n \) denote the number of users involved in the experiment, indexed by \( i \in \{1, \ldots, n\} \). For each user \( i \), let \( \bm{X}_i \), \( T_i \), and \( Y_i \) denote her covariates, treatment level, and observed outcome, respectively. At the conclusion of the experiment, the platform collects \( n \) i.i.d. observations: $\{ \bm{Z}_i = (\bm{X}_i', T_i, Y_i)' \}_{i=1}^n.$
In the first stage, we approximate the link function \( G(\cdot) \) using a structured DNN. Specifically, we employ a flexible feed-forward DNN, denoted by \( \hat{\bm{\theta}}(\bm{x}) \), to approximate the nuisance parameter \( \bm{\theta}_K^*(\bm{x}) \). Following our sigmoid-polynomial specification introduced in Equation (ref), the interaction between the treatment variable \( t \) and the estimated covariate-dependent coefficients \( \hat{\bm{\theta}}(\bm{x}) \) is modeled in a structured and explicit form:
where \( \hat{\theta}_k(\bm{x}) \) denotes the coefficient corresponding to the \( k \)-th polynomial term of the treatment variable, for \( k = 0, \ldots, K \), as a function of user covariates \( \bm{x} \). The nuisance parameter estimator \( \hat{\bm{\theta}}(\cdot) \) is learned by solving the following empirical risk minimization (ERM) problem:
where \( \ell(\cdot) \) is an appropriate loss function (e.g., squared loss for continuous outcomes or cross-entropy loss for binary outcomes), and \( \bm{\Theta} \) denotes the functional space of all possible feed-forward DNN under a chosen set of hyperparameters.
To achieve a sufficiently rapid convergence rate of the nuisance estimates---which is crucial for Stage 2 (policy value estimation) and Stage 3 (policy learning)---we establish certain regularity conditions. These conditions are commonly assumed in the literature with feed-forward neural neowrks farrell2020deep, pertain to the continuity and cuvation of loss function $l(\cdot)$ (Assumption (ref) in Appendix (ref)), the joint distribution of $(\bm X',T,Y)'$ (Assumption (ref)(a) in Appendix (ref)), and DGP nuisance parameter $\bm \theta^*(\bm x)$ (Assumption (ref)(b) in Appendix (ref)). Given the structure of \( \tilde{\bm{T}}_K(t) = (1, t, t^2, \ldots, t^K)' \), we analogously define the policy-induced polynomial vector as \( \tilde{\bm{\pi}}_K(\bm{x}) = (1, \pi(\bm{x}), \pi(\bm{x})^2, \ldots, \pi(\bm{x})^K)' \). Proposition (ref) presents the convergence rate for estimating nuisance parameters during Stage 1, with its proof deferred to Appendix (ref). This result is foundational to ensuring the reliability and efficiency of subsequent stages of {\sf DLPT} implementation.
In the second stage of our framework, we estimate the policy value of each counterfactual policy $\pi$. Recall that the policy value is defined as:
where $w$ represents the unit profit gained from users' engagement outcomes, and $c$ denotes the unit cost of the policy. This formulation captures the expected net benefit of implementing policy $\pi$.
With the nuisance estimate $\hat{\bm \theta}(\cdot)$ obtained from Stage 1, a natural policy value estimator is to plug $\hat{\bm \theta}(\cdot)$ into Equation (ref). However, the estimation error of $\hat{\bm \theta}(\cdot)$ will propagate to the policy value estimation, and the error diminishing rate guaranteed in Proposition (ref) will be not fast enough to guarantee valid inference and prepare for efficient policy learning.
In light of this, we follow farrell2020deep and propose a semi-parametric policy value estimator by using orthogonalized score functions of $\hat{\bm \theta}(\cdot)$. Score functions with this orthogonality condition, or more precisely, universal (Neyman) orthogonality condition, are typically referred to as influence functions in econometrics literature newey1994asymptotic,ichimura2022influence. These influence functions exhibit first-order robustness to small variations in nuisance parameter estimation, by having error decay rate faster than $\sqrt{n}$. This robustness allows for semi-parametrically inference and efficient policy learning, even if the convergence speed of the nuisance estimates is not sufficient to achieve semi-parametric efficiency itself.
With the score function in (ref), the below result shows that its derivative with respect to our nuisance parameters evaluated at the true parameter values is 0. In other words, it satisfies Neyman orthogonality.
Now we are ready to introduce our policy value estimator $\hat{V_K}^{\sf DLPT}(\pi)$, based on the score function in Proposition (ref), composed with the sample-splitting estimation procedure. Specifically, we randomly partition the dataset \( S \), consisting of \( n \) i.i.d. samples, evenly into two subsets \( S_1 \) and \( S_2 \). We first estimate the nuisance parameters \(\hat{\bm{\theta}}\) using subset \( S_1 \) following the procedure outlined in Stage 1. Subsequently, we utilize subset \( S_2 \) for policy evaluation in Stage 2.
Proposition (ref), adapted from Theorem 3 in farrell2020deep, presents the $\sqrt{n}$-asymptotic normality of our policy value estimator $\hat{V}_K^{\sf DLPT}(\pi)$ with the estimated variance $\hat{\Psi}_K^{\sf DLPT}(\pi)$, allowing for policy value inference and hypothesis testing. It also shows that, with the Neyman orthogonality score function, our estimator is consistent and asymptotically normal.
Proposition (ref) shows that we can construct an semi-parametric estimator for evaluating any personalized policy $\pi$ from the data and conduct valid inference. Note that this estimator can be used to evaluate any policies, not limited to the observed levels in the experiment. With this estimator, we next discuss how to learn the optimal personalized policy from the data.
We denote the optimal policy derived from our framework as \(\hat{\pi}^{\Pi}_{DLPT}\), defined as the policy within a specified class \(\Pi\) that maximizes the expected policy value. Formally, this policy \(\hat{\pi}^{\Pi}_{DLPT}\) is obtained by solving the constrained empirical risk minimization (ERM) problem:
where $\mathbb{E}_{S_2}[\cdot]$ denotes the empirical average over sample $S_2$. The below result provides the regret guarantee of the learned policy.
Theorem (ref) shows that the learned policy achieves minimax optimality, with regret scaling at the rate $1/\sqrt{n}$. This rate is consistent with results in policy learning literature for discrete treatment athey2021policy,zhou2023offline. In other words, our model permits far more flexible functional forms and continuous optimization variables, yet its policy‑learning regret remains as efficient as in the original discrete‑experiment setting. The proof of Theorem (ref) is provided in Appendix (ref).
In particular, we present four distinct policy classes, spanning from industry standard to the most flexible option. All of them shall be evaluated in our empirical application:
Note that our method relies on identifying the optimal continuous policy within a flexible (semi-parametric) class of DGPs. However, despite the flexibility of our DGP specification in Equation (ref), misspecification remains possible. Therefore, in this section, we evaluate the approximation capability of the sigmoid-polynomial function assumed in Equation (ref) relative to any DGP satisfying $s_t$-H\"older continuity (see Assumption (ref) in Appendix (ref)) defined as $f(\bm{x},t) := \mathbb{E}[Y \mid \bm{X} = \bm{x},\, T = t] $.
Assumption (ref) imposes a smoothness condition on the ground-truth DGP function, which requires that $f(\bm x,t)$ is sufficiently smooth in the treatment variable $t$ uniformly over the context space $\mathcal{X}$. Under this condition, Theorem (ref) establishes the approximation properties of the sigmoid-polynomial function $G(\bm{\theta}_K^*(\bm{x}),t)$, yielding the key result that the sigmoid-polynomial form can approximate the oracle DGP with a provable finite-sample error bound. The proof of Theorem (ref) is provided in Appendix (ref).
The polynomial component captures the variation of the function with respect to $t$, and under smoothness assumptions on the ground-truth DGP, standard approximation theory guarantees that a polynomial of degree $K$ can approximate the DGP with an error rate of $O(K^{-s_t})$.
We conclude this section by emphasizing the trade-off involved in selecting the approximation degree $K$ and experiment implementation costs. A large $K$ can reduce the bias in value approximation, but its choice is constrained by identifiability, $K$ must not exceed the number of discrete treatment levels used in the experimental design (see Remark (ref)). In practice, the number of treatment levels cannot be made arbitrarily large--especially in high-dimensional covariate settings--because this requires a sufficiently large sample to ensure adequate coverage of each treatment level across the covariate space. Otherwise, the estimation variance of the nuisance parameters increases, leading to higher optimization error and degraded policy performance. Thus, selecting $K$ involves balancing model complexity with available data.
In this section, we introduce our collaborating platform, describe the large-scale randomized field experiment we conducted on the platform in order to evaluate our methodologies, and present the model-free empirical analysis of the experiment.
To empirically validate our proposed {\sf DLPT} framework, we partnered with Platform O, a leading global short-video platform. On this platform, users act as both content creators and viewers, with most videos being user-generated. Creators independently decide when and what content to produce, while viewers can access videos for free. Platform O primarily earns revenue through advertisements supplied by third-party advertisers. Creators do not receive a direct share of this ad revenue; instead, prominent creators monetize their content by selling products or securing direct advertising deals as influencers.
While larger creators can independently secure brand deals, smaller creators—especially newcomers—often lack the resources to do so. Yet, their participation is crucial for the platform's long-term success. To incentivize these creators, the platform has introduced targeted financial incentive programs. Platform O uses a proprietary digital token called “Points,” which users can earn through video gifts, monetization programs, and other engagement-based initiatives. These Points can be redeemed for cash-equivalent rewards. To incentivize small or new creators to upload videos, Platform O typically uses the “Points Earning Task” feature to offer financial incentives, paid in points, for each uploaded video. This feature serves as the basis for our experimental intervention.
To examine how user-generated content production responds to financial incentives (specifically, reward Points) and to identify the optimal reward level for maximizing the platform’s return on investment, we conducted a randomized experiment from August 23 to 29, 2021, testing four distinct reward levels. The experiment targeted creators classified as low-activity producers, defined as users who had produced videos on fewer than three days during the four weeks preceding the experiment. These low-activity producers accounted for $12.3\%$ of all producers on the platform. All low-activity producers participated in the experiment and were randomly assigned to either a control group or one of the four treatment groups.
Users in the four treatment groups were offered a reward task that could be completed for varying levels of incentives during the experiment. This task, titled “Post a Video," allowed users to earn a reward by posting at least one video during the experiment period (August 23 to 29, 2021). Users who posted multiple videos could earn the reward only once during the experiment. The reward levels randomly assigned to the four treatment groups were 178, 358, 538, or 718 Points, with each 100 Points approximately equivalent to \$0.01 USD. Users in the control group did not see this task and thus could not complete it. The specific user interface for this experiment is shown in Figure (ref). As illustrated, users accessed their profile page (Panel (a)) and clicked on “My Wallet" (Panel (b)) to view available Points Earning Tasks. Treatment group users saw the “Post a Video" task with different Point values depending on their assigned group (Panel (c)), while control group users did not see this task at all.
This feature potentially involves a trade-off: offering higher rewards increases the program’s cost but could also lead to more video production. The platform’s primary objective in conducting this experiment was to identify the optimal cost-effective level of incentives to encourage content production. While the incentive (i.e., Points awarded for video uploads) is conceptually a continuous variable, the platform could only implement and test a discrete set--in this case four levels--of treatments. These features of the experiment makes it a perfect test bed for our {\sf DLPT} framework. Last but not least, based on the experimental results, the platform selected the optimal discrete level and subsequently rolled out the feature to all users.
Due to data confidentiality, we only have access to a subsample of $7,349,648$ users in this experiment. Let $t$ denote the treatment levels: the control group ($t$ = 0) included $1,478,472$ users, the 178 Points group ($t$ = 178) included $1,478,666$ users, the 358 Points group ($t$ = 358) included $1,479,277$ users, the 538 Points group ($t$ = 538) included $1,478,819$ users, and the 718 Points group ($t$ = 718) included $1,479,414$ users. In our primary analyses, each user $i$ is associated with an observed outcome $Y_i$, indicating whether the user posted at least one video during the experimental period (August 23 to 29, 2021). Specifically, $Y_i = 1$ if the user posted at least one video, and $Y_i = 0$ otherwise.\footnote{Technically, $Y_i$ is a discrete variable taking values greater than or equal to zero. However, after discussions with the platform, it became clear that they are more concerned with whether a user publishes at least one video, rather than the exact number of videos posted. This preference is motivated by two reasons. First, only $9.53\%$ of low-activity users post more than one video within a week. Second, the platform's goal is to encourage users to develop a habit of content creation, for which posting a single video per week is deemed sufficient. Based on these considerations, and in alignment with the platform’s priorities, we follow their recommendation and define the outcome variable as binary.}
For each user, we also collected a set of pre-treatment covariates, denoted $\bm{X_i}$, which include 12 discrete variables (covering aspects such as gender, the user’s primary region of activity, the presence of other video-sharing platforms on the user’s device, and the user’s activity level) and 46 continuous variables (e.g., the number of videos uploaded by the user in the 30 days preceding the treatment). After applying one-hot encoding to the categorical variables, the dimensionality of the user pre-treatment covariates used in our analysis expanded to 77. Table (ref) in the Appendix (ref) provides a comprehensive description of all the covariates incorporated into our analysis.
To ensure the validity of the randomization process, we compared user demographics, content consumption, and content creation behavior during the 10 days prior to the experiment across all groups. As shown in Table (ref), the five groups exhibited similar distributions in key demographic variables, such as gender and city tier. Furthermore, content consumption (measured by app usage) and content creation (measured by whether users uploaded at least one video) were also comparable across groups during the pre-experiment period. These results confirm that the randomization was successfully implemented, as there were no statistically significant differences in these characteristics across groups (all p-values $> 0.05$).
For this experiment to serve as a suitable testbed for our {\sf DLPT} framework, it must satisfy two key conditions beyond its experimental design. First, the treatment—in this case, the financial reward—must have a statistically significant impact on the outcome variable, i.e., whether to upload a video during the experiment. Second, the effect of the treatment should vary across different reward levels and user subgroups, which creates the room for personalization. Accordingly, in this subsection, we present the average treatment effects (ATEs) and heterogeneous treatment effects (HTEs) from the original experiment prior to implementing our framework.
We first provide empirical evidence that examines the ATEs of receiving Points on the recipient’s content production. For each of the four treated Point levels, we compared user outcomes against the control group ($t = 0$) to calculate the ATE for each level. We use the following ordinary least squares regression specification:
where $Y_i$ is the binary outcome variable indicating whether user $i$ posted at least one public video during the experiment week, and $D_i$ is the binary treatment variable indicating whether user $i$ is assigned to the treated group. For each tested Point level, we estimate specification (ref) using data from control and treated users at that specific Point level.
As shown in Table (ref), the probability of users posting a video during the experimental period was significantly higher in all four treatment groups compared to the control group (all $p$-values $< 0.0001$), with relative effect sizes ranging from $13.01\%$ to $19.69\%$. Moreover, the upload probability increased with higher reward amounts, and significant differences were observed across different reward levels. In the last column of Table (ref), we report the maximum pairwise $p$-value from comparisons between adjacent reward levels (e.g., 178 vs. 358, 358 vs. 538, and 538 vs. 718); this $p$-value is $< 1.189 \times 10^{-7}$, indicating that the treatment effects significantly vary across reward levels and that larger rewards lead to greater increases in upload probability. Consistent with prior literature kaynar2023estimating, we also observe diminishing marginal returns to financial incentives in promoting content creation: although the incentive roughly doubles at each level, the marginal increase in user participation decreases from 3.19% to 2.10%, and eventually to 1.39%.
We further investigate the HTEs of reward levels across user subgroups. Specifically, we identify two user characteristics that may moderate the effect of Points rewards on content production. First, users’ historical video uploading behavior may influence their responsiveness to reward incentives. Second, inspired by zeng2024impact, we consider users’ content consumption behavior, which may also affect their likelihood of content production. To examine the two potential moderators, we construct two binary variables. The first, $LowP_i$, captures historical content production behavior and equals one if user $i$ uploaded fewer videos in the 30 days prior to the experiment than the median user in the sample. The second, $LowC_i$, reflects historical content consumption behavior and equals one if user $i$ completed watching fewer videos in the 30 days prior to the experiment than the sample median.
We rely on the following OLS regression specifications to test the interaction between the treated Point rewards and two potential moderators, using data from control and treated users at that specific Point level:
For data confidentiality, we normalize the outcome variable. Table (ref) presents the HTEs of different Point reward levels on users' content production. Across all reward levels, we observe significant positive treatment effects, with effect sizes increasing as the reward amount rises. Importantly, users' historical production and consumption behaviors significantly moderate these effects. Specifically, users classified as low producers ($LowP$) exhibit a consistently lower responsiveness to rewards compared to high producers, with negative and statistically significant interaction terms across all reward levels (all $p$-values$<0.0001$). Conversely, users classified as low consumers ($LowC$) demonstrate a significantly higher responsiveness to rewards, as indicated by positive and significant interaction terms (all $p$-values$<0.01$). These patterns suggest that user heterogeneity plays a meaningful role in shaping the effectiveness of reward interventions. Consequently, these results provide strong motivation for personalized reward targeting strategies.
In this section, we empirically evaluate the {\sf DLPT} framework using the field experiment data described in Section (ref). Section (ref) outlines the implementation details and benchmarks. Section (ref) presents ATE estimation results for the four tested Point levels. Section (ref) reports policy learning regret for the evaluation user groups under the discrete policy class.
To implement the {\sf DLPT} framework in our empirical setting, we follow the three-stage procedure outlined in Section (ref). We denote the dataset of $7,349,648$ users as $\mathcal{S}$ and partition it into a training set $\mathcal{S}_{\text{train}}$ and an inference set $\mathcal{S}_{\text{inference}}$. The specific splitting strategy is designed to align with our evaluation objectives, which we will elaborate on in the following sections. Nevertheless, the implementation procedure itself is general and remains applicable across different evaluation setups.
Stage 1: Nuisance Parameters Estimation. In Stage 1, we assume that the outcome $Y$, user features $\bm{X}$, and continuous treatment level $T$ (rescaled by dividing by 1000 for training stability) are generated according to the DGP specified in Equation ((ref)), with the polynomial degree set to $K=3$:
The policy value for a given policy $\pi$ is specified as follows:
where $w=0.5$ and $c=0.1$ are parameters set by Platform O. These represent the per-unit revenue (in Point) from users uploading videos and the per-unit cost (also in Point) of providing monetary incentives, respectively. Thus, $V(\pi)$ captures the net value, as the revenue from user activity minus the cost of incentives, measured entirely in Points. Figure (ref) illustrates the structured DNN that approximates the DGP in Equation (ref), implemented via TensorFlow. The training dataset $\mathcal{S}_{\text{train}}$ is used to fit the structured DNN, which is trained under a binary cross-entropy loss function through the ADAM optimizer kingma2014adam.
Stage 2: Policy Value Estimation. Next, we construct estimators for two quantities: (i) the ATE of treatment level $t$ relative to the control, denoted as $\mu(t)$; and (ii) the policy value of a given policy $\pi$, denoted $V_K(\pi)$. We estimate the policy value for policy learning in the next stage, whereas we estimate the ATE of observed treatment primarily for evaluation purposes.
We then construct Neyman Orthogonal score function for the ATE estimator using testing data $\mathcal{S}_{\text{inference}}$ following Proposition (ref) (since the non-parametric representation in Stage 1 is trained on $\mathcal{S}_{\text{train}}$):
Similarly, the Neyman Orthogonal score function for the policy value estimator is given by:
Here $\check{t}_i$ denotes the treated level of points of the evaluation data.
Stage 3: Policy Learning. We derive the personalized optimal policies by solving the ERM problem $ \hat{\pi}_K^{DLPT} = \inf_{\pi\in{\Pi}}~\{-\hat{V}_K^{\sf DLPT}(\pi)\}$ for inference data $\mathcal{S}_{\text{inference}}$ under four different policy classes $\Pi$:
For the first two discrete policy classes, we compute the value of $ \psi_V (\bm z_i,\pi; \hat{\bm\theta},\bm \Lambda)$ for each candidate policy $\pi \in \Pi$, given user covariates $\bm x_i$, and assign to user $i$ the policy that yields the highest estimated value. For the last two policy classes, we optimize the objective $\hat{V}_K^{\sf DLPT}(\pi)$ directly using TensorFlow's automatic differentiation baydin2018automatic, treating it as the loss function to train the policy classes. See Appendix (ref) for implementation details.
Benchmarks. To evaluate the effectiveness of our framework in estimating the ATE and, more importantly, in enabling personalized policy learning, we compare it against five established benchmarks: (a) linear regression ({\sf LR}); (b) logistic regression ({\sf LogR}); (c) the logit choice model ({\sf LCM}); (d) a pure deep learning approach ({\sf PDL}); and (e) a uniform level policy ({\sf UNI}).
Under the {\sf LR} approach, it assumes a linear relationship between the expected outcome and the covariates and treatment such that $\mathbb{E}[Y|\bm X=\bm{x}, T = t]=\bm\alpha\bm{x}+\beta t$. This linear functional form is a common and interpretable baseline for estimating treatment effects in empirical settings, particularly in market intervention effectiveness and pricing studies (e.g., lodish1995tv,besbes2015surprising,simester2020efficiently,cui2020reducing).
When the outcome $Y$ is binary (uploading videos or not), {\sf LogR} is a more appropriate choice. {\sf LogR} models the probability of the outcome as a sigmoid function of a linear combination of covariates and treatment: $\mathbb{E}[Y|\bm X=\bm{x}, T = t]=\sigma(\bm\alpha\bm{x}+\beta t)$, where $\sigma(x)=\frac{1}{1+\exp(-x)}$. We remark that while {\sf LR} inherently assumes the linear boosting effect of point policy, {\sf LogR} accounts for the pattern of diminishing returns. This is widely used for modeling customer response to interventions (e.g., guadagni1983logit,mela1997long,bleier2015personalized,ascarza2017beyond).
The LCM approach assumes that the utility a user derives from uploading videos, given their features $\bm x$ and point level $t$, is $u = \bm\alpha \bm{x} +\bm\beta \bm x t+\gamma$, where $\gamma$ represents a Type-I extreme value error term. This leads to the probability of uploading as $\mathbb{P}[Y=1|\bm X=\bm x,T=t]=\frac{\exp(u)}{1+\exp(u)}=\sigma(u)$. This model, rooted in discrete choice theory, is a widely adopted benchmark for understanding customer behavior (e.g., tian2020optimizing,dube2023personalized,el2023joint). While LogR and LCM share the same underlying logistic functional form, a crucial distinction is LCM's ability to model a heterogeneous effect of treatment on the outcome through the interaction term $\bm\beta \bm x t$.
The {\sf PDL} use the similar DNN as shown in Figure (ref) to approximate $\mathbb{E}[Y|\bm X=\bm{x}, T = t]=f_{\sf PDL}(\bm{x},t)$. Unlike the parametric assumptions of the previous models, PDL offers a non-parametric approach to capture complex, non-linear relationships between inputs and outcomes. With the increasing abundance of data, the deep learning approach has been widely adopted (e.g., simester2020targeting,garcia2022demand,yoganarasimhan2023design).
The {\sf UNI} policy, as the current industry practice, assigns a uniform treatment level to all individuals, typically selected by maximizing the estimated policy value derived from the ATE calculated across the observed sample.
The first four methods---{\sf LR}, {\sf LogR}, {\sf LCM}, and {\sf PDL}---are used for both ATE estimation and policy learning. The final benchmark, {\sf UNI}, is limited to policy learning and assigns the same treatment level to all individuals. We include {\sf UNI} as a baseline to assess the added value of personalized, data-driven policy optimization. The implementations of all benchmarks use the same training dataset $\mathcal{S}_{\text{train}}$ and inference dataset $\mathcal{S}_{\text{inference}}$. See Appendix (ref) for detailed benchmark descriptions and implementations.
In this section, we present the ATE estimation result in Stage 2, which serves as an indicator of the accuracy of the estimated nuisance parameters in Stage 1. Specifically, we adopt a cross-evaluation strategy: for each Point Reward level $t \in \{178, 358, 538, 718\}$, we designate users assigned to treatment level $t$ as the masked evaluation set $\mathcal{S}_{\text{masked}}$, which outcomes are used for computing the ground truth ATE of the masked level. The remaining users are then randomly split into a training set $\mathcal{S}_{\text{train}}$ (for estimating nuisances in Stage 1) and an inference set $\mathcal{S}_{\text{inference}}$ (for constructing orthogonalized ATE estimator in Stage 2). The ATE estimation results for each level $t$ are reported in Table (ref). Column (2) presents the ground-truth normalized ATE for the four levels of Points. Columns (3)–(12) report the estimated ATEs and their corresponding absolute percentage errors (APE) against the ground-truth ATEs shown in Column (2). To evaluate the accuracy of the estimates, we use three performance metrics to aggregate the performance of different methods across the $4$ test cases: Mean Absolute Percentage Error (MAPE), Mean Squared Error (MSE), and Mean Absolute Error (MAE).
As shown in Table (ref), across all levels, {\sf DLPT} consistently achieves the lowest APEs, with values ranging from 1.32% to 3.30%, significantly outperforming baseline methods such as {\sf LR} (18.66%–61.63%) and {\sf LCM} (27.80%–63.50%). The summary metrics at the bottom of the table further highlight {\sf DLPT}'s superior accuracy, yielding the lowest MAPE (2.25%), MSE (3.46), and MAE (1.70). Among baseline methods, {\sf PDL} also performs competitively, achieving 7.18% in MAPE and outperforming traditional models such as {\sf LR}, {\sf LogR}, and {\sf LCM} by a large margin. These results demonstrate the effectiveness of {\sf DLPT} in accurately estimating treatment effects, which is critical for downstream policy learning.
Unlike {\sf LR} and {\sf LogR}, which rely on linear assumptions or simple parametric transformations, {\sf DLPT} leverages the representational power of deep learning combined with bias correction techniques to achieve more accurate ATE estimates. While {\sf PDL} also benefits from the flexibility of neural networks and performs considerably better than parametric models, it does not incorporate explicit debiasing procedures. As a result, its estimation errors, though relatively low, remain consistently higher than those of {\sf DLPT}. This highlights the importance of correcting for estimation bias in high-capacity models when recovering treatment effects.
Finally, it is noteworthy that both {\sf DLPT} and {\sf PDL} exhibit slightly higher estimation errors at the boundary treatment levels ($t=178$ and $t=718$) compared to the intermediate levels ($t=358$ and $t=538$), a pattern not observed with more parametric methods. We interpret this as suggestive evidence that these two deep learning methods tend to overfit within the observed parameter range, implying that extrapolation for DL-based methods is inherently more challenging than interpolation. Consequently, when employing these methods in practice, experimenters should design initial experiments to span a sufficiently large parameter space, thereby mitigating the need for extrapolation.
We now present the results from Stage 3 on personalized policy learning, focusing on the discrete policy class, specifically, the Finite Observed Policy Class. To ensure minimal approximation error, we set the polynomial degree $K=4$ and change the structured DNN accordingly. We will extend the analysis to the four policy classes, including continuous policies, in Section (ref). We adopt a similar cross-evaluation logic as in Section (ref) in the implementation but also add a random partitioning of the user subgroups into training and inference datasets. To enable this, we discretize user covariates and partition users into $2,287$ subgroups based on six demographic and behavioral characteristics: gender, developmental level of the user’s primary city of residence (categorized as highly developed, moderately developed, or less developed), user activity level (low, medium, or high), platform usage (iPhone or other), life stage (e.g., mature phase), and number of friends. Users within each subgroup share identical values across these dimensions. We then split all users into the training and inference datasets at the group level randomly. The resulting training and inference datasets are approximately equal in size and randomly composed, denoted as $\mathcal{S}_{\text{train}}$ (with 1,112 subgroups containing 3,681,464 users) and $\mathcal{S}_{\text{inference}}$ (with 1,175 subgroups containing 3,668,184 users).
To evaluate our method against benchmark approaches, we assume that as experimenters we observe the outcome variables in $\mathcal{S}_{\text{train}}$ but not those in $\mathcal{S}_{\text{inference}}$. Consequently, to implement our proposed method, we first use $\mathcal{S}_{\text{train}}$ in Stage 1 to obtain $\hat{\theta}(\cdot)$ and then apply Stages 2 and 3 using features from $\mathcal{S}_{\text{inference}}$. For each subgroup \( j \) in $\mathcal{S}_{\text{inference}}$, we compute the estimated optimal policy—that is, the optimal point level among $\{0,178,358,538,718\}$—by selecting the point level that yields the highest estimated value according to our Stage 3 estimator. Similarly, for the benchmark methods, we use $\mathcal{S}_{\text{train}}$ to estimate parametric policy values based on features and then determine the optimal policy using features from $\mathcal{S}_{\text{inference}}$. Finally, for each subgroup \( j \) in $\mathcal{S}_{\text{inference}}$, we calculate the “ground-truth” policy value for each point level using empirical average outcomes, and we determine the “ground-truth” optimal policy as the point level that yields the highest average outcome within that subgroup. We then compare the values of each method's estimated optimal policy and the ground-truth optimal policy for each subgroup.
Table (ref) compares the performance of the proposed {\sf DLPT} framework with all benchmark methods. Performance is evaluated based on aggregating results through all subgroups in $\mathcal{S}_{\text{inference}}$ using three metrics: (1) Mean Percentage Regret (MPR), which measures regret relative to the optimal in-class policy value; (2) accuracy, defined as the percentage of subgroups for which the learned policy correctly identifies the optimal Point level; and (3) group-size-weighted accuracy, which similarly compares the learned best Point level to the true best Point level, but weights each subgroup’s contribution by its user size.
As shown in Table (ref), the proposed {\sf DLPT} framework achieves the best performance by a significant margin. It obtains the lowest MPR of $0.48\%$, indicating that its obtained policy is only $0.48\%$ lower than the ground-truth optimal policy. {\sf DLPT} also substantially outperforms all baselines in both accuracy (73.08%) and group-size-weighted accuracy (79.83%), reflecting its ability to correctly identify the best policy for both small and large user subgroups. The {\sf UNI} method, which selects the optimal point level with the highest average outcome across all users (thus ignoring heterogeneity), performs moderately well with an MPR of 6.41% and accuracy around 25--34% (note that random guessing among five treatment levels yields an accuracy of 20%). Surprisingly, traditional models such as {\sf LR}, {\sf LogR}, and {\sf LCM} perform worse than {\sf UNI} across all metrics. This suggests that the rigid functional forms of these traditional models may be overwhelmed by the large number of user features included. Even {\sf PDL}, a more flexible baseline, does not match the performance of {\sf DLPT}, particularly in accuracy. These results collectively underscore the effectiveness of our {\sf DLPT} framework in policy learning, highlighting the importance of modeling heterogeneity flexibly and correcting for biases in the estimation of heterogeneous treatment effects.
In this section, we extend our evaluation of the {\sf DLPT} framework through semi-synthetic studies grounded in real-world field experiment data. It allows us to evaluate policy learning performances when considering a broader range of policy classes beyond the experimental setting. In Section (ref), we describe the training of a fully connected deep neural network (FNN) on the complete dataset of $7,349,648$ users; this FNN serves as the semi-synthetic DGP we use to approximate the ground-truth DGP. Section (ref) presents the results on personalized policy learning under continuous policy classes. Finally, Section (ref) provides a series of robustness checks to assess the performance of our framework.
To approximate the ground-truth DGP, we train a FNN on the complete dataset of $7,349,648$ users such that $\mathbb{E}[Y|\bm X, T]=f^{\text{FNN}}(\bm X,T)$. Both user feature $\bm X$ and assigned treatment level $T$ are the input, and binary outcome $Y$ is the output. The FNN architecture consists of one input layer with 78 nodes (with $\bm X$ in 77 dimensions and $T$ in 1 dimension), two fully connected layers with $4,096$ nodes per layer, and one output layer with 1 node and sigmoid as activation function, resulting in a total of $17,108,993$ parameters. We use binary cross-entropy as the loss function, adopt the Nadam optimizer dozat2016incorporating, and set the batch size to $1,024$. The model is trained for 100 epochs, achieving a final accuracy of 99.80%. While this high-capacity configuration carries a risk of overfitting, it enables flexible simulation across the entire continuous treatment space and closely approximates the ground truth DGP for all users in the dataset, facilitating robust evaluation across various continuous policy classes.
It is important to note that the FNN does not need to perfectly recover the ground-truth DGP or attain 100% predictive accuracy for the evaluation to be valid. The evaluation validity holds because we use the FNN both to generate training data for policy learning methods and to evaluate the learned policies based on the data from the FNN. However, achieving a high-fidelity approximation of the FNN is still desirable since the closer the FNN matches the ground-truth DGP, the more accurately we can assess the approximation power of the {\sf DLPT} framework in practice. This, in turn, increases our confidence in the framework’s performance under realistic misspecification in real-world deployment, as discussed in Section (ref).
To generate semi-synthetic data $\mathcal{S}^{\text{FNN}}$, we use the trained $f^{\text{FNN}}(\cdot)$ to obtain the probability of outcome variable $Y$ given $\bm X$ and $T$. For all $7,349,648$ users, we generate $Y^{\text{FNN}}_i = B(1,f^{\text{FNN}}(\bm X_i,T_i))$ following the binomial distribution with $\mathbb{P}(Y^{\text{FNN}}_i=1)=f^{\text{FNN}}(\bm X_i,T_i)$. The $\mathcal{S}^{\text{FNN}}$ is randomly half split into a training set \( \mathcal{S}^{\text{FNN}}_{\text{train}} \) and a testing set \( \mathcal{S}^{\text{FNN}}_{\text{testing}} \). We then randomly half-split testing set \( \mathcal{S}^{\text{FNN}}_{\text{testing}} \) into inference set \( \mathcal{S}^{\text{FNN}}_{\text{inference}} \) and evaluation set \( \mathcal{S}^{\text{FNN}}_{\text{evaluation}} \). We use the training data \( \mathcal{S}^{\text{FNN}}_{\text{train}} \) to for stage 1 estimation, inference data \( \mathcal{S}^{\text{FNN}}_{\text{inference}} \) for stages 2 and 3; we then evaluate the obtained policies on the evaluation data \( \mathcal{S}^{\text{FNN}}_{\text{evaluation}} \).
In this section, we present the results of policy learning using semi‐synthetic data. To evaluate the robustness of each method’s performance, we hold the training set $\mathcal{S}^{\mathrm{FNN}}_{\mathrm{train}}$ fixed and draw 30 bootstrap resamples (with replacement) from the inference set $\mathcal{S}^{\mathrm{FNN}}_{\mathrm{inference}}$ and evaluation set $\mathcal{S}^{\mathrm{FNN}}_{\mathrm{evaluation}}$. As in Table (ref), Panel A of Table (ref) displays the mean of MPR over the 30 resamples for each benchmark method across policy classes. The primary distinction is that, in this setting, the policy classes include not only discrete treatment levels but also continuous policy classes. In the rightmost column of Table (ref), we report the largest $p$-value obtained from pairwise comparisons of DLPT regret against each benchmark. Consistent with our findings on real data under discrete policies, the proposed DLPT framework significantly outperforms all benchmarks across all four policy classes (with the largest $p$-value $<0.05$), underscoring its superiority even in settings with continuous treatment regimes.
Now, we would like to provide some intermediate steps for the policy learning to investigate why our method could outperform other methods. Specifically, similar to Section (ref), we divide users into four segments based on their production and consumption values prior to the experiment: low production combined with low consumption (LL), low production combined with high consumption (LH), high production combined with low consumption (HL), and high production combined with high consumption (HH). In Panels B and C of Table (ref), we report the results of policy learning under the DNN policy class across methods. For each group, we compute the ground-truth optimal point level $t^*$ under the DNN policy class and the corresponding ground-truth optimal value $V^*$.
As shown in Panels B and C of Table (ref), the optimal treatment levels and the corresponding optimal values vary considerably across user groups. For users in the LL group, the optimal ground-truth treatment level is $443.31$ with $0.0432$ as the average value, while for users in the HH group, it is only $230.28$ with $0.1742$ as the average value. Moreover, the optimal treatment level increases from HH user to LL user monotonically, while the optimal value decreases from HH user to LL user monotonically. This aligns with intuition, as more engaged users typically require fewer incentives to produce content, resulting in a lower optimal treatment level and higher optimal value. All methods capture this monotonic trend to some extent, but most fail to accurately estimate the true optimal levels, especially for the HL and HH groups. These two groups, although requiring lower incentives, still need meaningful treatment, yet many benchmark methods recommend treatment levels that are too low. For instance, {\sf UNI} always selects the minimum level $178$ regardless of the user group, resulting in a completely non-personalized policy. In contrast, our proposed {\sf DLPT} framework not only recovers the correct trend but also closely matches the true optimal policies across all groups, achieving significantly lower APEs and lower PRs (with the largest $p$-value$<0.05$). While {\sf PDL} performs reasonably well in distinguishing between groups, it struggles with HL and HH users, likely due to its limited ability to handle policies near treatment boundaries. We believe that the debiasing procedure used in {\sf DLPT} provides robustness when estimating optimal policies close to boundary levels (in our case, at levels $178$ or $738$), enabling more accurate personalization for these challenging cases.
Next, we validate the regret bound derived from Theorem (ref) in Section (ref) using our data. Specifically, we first draw and fix a single validation set of $20,000$ observations from the user population for evaluation across all experiments. For each sample size $n \in $ \{$1,000$, $5,000$, $10,000$, $20,000$\}, we independently draw \( n \) observations to estimate nuisance parameters and another \( n \) observations to train the policy. We repeat this procedure 30 times for each \( n \), computing policy regret on the fixed validation set for each replication under DNN policy class. Figure (ref) presents the policy regret of {\sf UNI}, {\sf PDL}, and {\sf DLPT} methods with the increase of $n$.
We highlight that as the training dataset size increases, the average policy regret of our {\sf DLPT} decreases linearly with respect to the square root of the sample size. This empirical pattern aligns precisely with our theoretical result in Theorem (ref), which predicts that policy regret scales as $1/\sqrt{n}$. Thus, our theoretical guarantees are not only rigorous but also empirically relevant in practical settings. This finding underscores that experimenters can reliably employ our method across experiments of varying sample sizes.
We also perform robustness checks to assess the resilience of the {\sf DLPT} framework under two practical challenges. Owing to space constraints, full details are deferred to Appendix (ref), and here we provide a high‐level summary. First, we vary treatment coverage by restricting the span of treatment levels within the bounded treatment space. These experiments demonstrate that wider treatment coverage consistently enhances policy learning performance. Second, we evaluate the performance under different assumed polynomial degrees \( K \in \{1, 2, 3, 4\} \). The resulting regret values align with the theoretical insights discussed in Section (ref), confirming that the regret scales with \( K^{-s_t} \). Figure (ref) presents the policy regret with treatment level coverage and polynomial degree assumption.
In this paper, we introduce the {\sf DLPT} framework, a novel approach for estimating continuous policy values and learning personalized continuous policies with observations collected at discrete treatment levels. We provide theoretical guarantees for {\sf DLPT}, including the asymptotic unbiasedness of the policy value estimator and $\sqrt{n}$-regret bound for the learned policies. The practical efficacy of {\sf DLPT} was rigorously tested through a large-scale field experiment on a leading video-sharing platform, where it demonstrated substantial improvements in policy value estimation and policy regret minimization over conventional methods.
We conclude by outlining two avenues for future research. First, our analysis implicitly assumes that the Stage-3 empirical risk-minimization problem can be solved exactly, but we do not provide theoretical conditions that guarantee its computational tractability. Characterizing when---and under what assumptions---this optimization problem is solvable remains a significant open question, with broader implications for the policy learning literature beyond the scope of this work. Second, while our framework focuses on identifying the optimal policy given a predefined set of discrete treatments, it also has the potential to inform optimal experimental design --- specifically, the selection of treatment levels to include in the experiment. Addressing this challenge would allow practitioners to better integrate experimental design with downstream policy learning. Although we offer heuristic guidance in this paper, such as increasing the parameter span tested, a more systematic investigation remains a promising future direction.