Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
29,118 characters · 9 sections · 19 citation commands
Simulation-Based Benchmarking of Reinforcement Learning Agents for Personalized Retail Promotions
With AI surpassing human-level performance on various benchmarks, such as image classification, basic reading comprehension, and board games, there is an increasing focus on creating autonomous AI agents for specific environments Ray_Perrault_Jack_Clark_2024. In retail and e-commerce, advanced reasoning and an understanding of causal relationships are required to make effective product assortment, promotion, and pricing decisions katsov_introduction_2017. The combination of these requirements, marketplace dynamics, and the sparsity of customer purchase events across large product catalogs makes it a challenging domain to apply autonomous AI agents. To advance the development of AI in retail, open datasets, and simulation platforms are needed that capture the end-to-end customer experience Bernardi_Batra_Bruscantini_2021. Public retail datasets are quite limited, and existing simulation platforms focus on specific problem domains, such as dynamic pricing serth_interactive_2017 and product recommendations santana_mars-gym_2020, ie_recsim_2019. An ideal simulation platform can enable the evaluation of a wide range of marketing agents that optimize customer experiences.
Targeting promotions is already one of the most impactful applications of AI agents; large e-commerce companies, such as Wayfair Fei_2021, Booking.com Kangas_Schwoerer_Bernardi_2021, Stitch Fix Glynn_2018, and Amazon Kanase2022 have found success using contextual bandits and RL approaches, to decide who gets what offer, when, and over what channel. Enabling this use case has traditionally required large-scale online experimentation programs to collect exploration data and prove the uplift over less sophisticated approaches Treybig_2022. Simulation platforms can lower the barrier to adopting RL by enabling the offline development of advanced agents and providing estimates of the potential uplift from deploying them.
We previously introduced RetailSynth, an interpretable multi-stage retail data synthesizer, and showed that it faithfully captures the complex nature of the retail customer decision-making process over the full journey from choosing to visit a storefront to deciding exactly which product and how much to purchase xia_retailsynth_2023. In this work, we extend RetailSynth to enable the training and evaluation of RL agents that target promotions (coupons) to individual customers. We propose an environment where the agent is trained using offline batch data to target store-wide coupons to customers at discrete time steps. In alignment with industry practice, coupons are set up as discrete actions across a range of discount levels and evaluated based on their impact on customer revenue over the evaluation period, while monitoring secondary metrics such as the profit margin impact, the number of categories a customer purchases, and the fraction of customers active at the end of the evaluation period. We characterize the environment using static baseline policies where all customers receive the same coupon and then compare the performance of the baseline policies to personalized policies learned by the RL agents. We segment the customers based on their latent price sensitivity and show that personalized policies typically target less aggressive discounts to less price-sensitive customers. Based on our observation that price-insensitive customers still often receive large discounts, there do appear to be opportunities to improve agent performance on this benchmark.
To our knowledge, our work is the first to benchmark RL agents on simulated retail customer shopping trajectories. It provides much needed guidance to practitioners on the potential uplift of deploying coupon-targeting agents in a multi-category retail environment. We also provide insights into which customer features effectively summarize the sparse transaction data and a deep dive into metrics to consider prior to deployment. We intend this paper to serve as a blueprint for how to simulate AI agents that optimize the end-to-end retail customer journey. The remainder of our paper is organized as follows: Section (ref) gives a detailed overview of the simulation environment; Section (ref) describes the agent training and evaluation experiments; and Section (ref) describes challenges and directions for future work.
We leverage the RetailSynth framework to simulate the shopping behavior of a cohort of customers choosing from a large product catalog covering multiple categories. This model was previously calibrated on a public grocery dataset dunnhumby_complete_2014 and shown to generate realistic, synthetic data (with the KS-statistic < 0.2 for each of the decision stage choice distributions). As shown in Figure (ref), we integrate the customer decision model with a simulation environment that facilitates training and evaluating offline reinforcement learning agents. The customer decision model is sensitive to changes in the customer state that evolve in response to marketing and pricing decisions. Based on the context of these decisions and the customer purchasing activity, the environment history is summarized at each time step in the simulation to form the batch dataset that is used for agent training. Learned policies are then deployed in the simulation environment, and their performance is measured based on accumulated revenue and other secondary metrics such as category penetration and customer retention.
The customer decision model is a four-stage decision model that covers the decision to visit the store, to make a purchase within a specific category, which product to buy from a chosen category, and how much of that product to purchase. At each time step in the simulation, the probability of customer $u \in U$ purchasing quantity $Q$ of product $i \in I$ in category $j \in J$ is given by:
where $S_u$, $C_{uj}$, $B_{ui}$, and $Q_{ui}$ indicate the binary outcome for each of the listed decision stages. The latent customer state is defined by the product, category, and store visit utilities and is responsive to pricing and marketing decisions. Customer state transitions are captured in the following equations by the inclusion of lagged variables (specifically in the store visit model). The product utility that underlies all decision stages is defined as
$\mathbf{X_{uit}}$ represents observable, time-varying features that are customer- and product-specific (e.g. digital display advertising). $Z_{i}$ is an unobserved factor, such as brand strength, that affects both prices and demand. Note the price sensitivity coefficient $\beta^w_{ui} \sim \beta_u^w \beta_i^w$ where $\beta_u^w$ and $\beta_i^w$ are customer-specific and product-specific factors.
Each product has a shelf price that is either the base price or a discounted price. The final purchase price $P_{uit}$ is the shelf price, adjusted to account for any personalized coupons the customer chooses to redeem, and is given by $P_{uit} = (1 -D^{coupon}_{uit}*\mathbb{I}^{coupon}_{uit})P^{shelf}_i$. Shelf pricing is assumed to follow a high-low strategy and is simulated using a two-state hidden Markov model as described in xia_retailsynth_2023.
We model a cohort of customers, acquired at the same time period and evolve the probability to revisit the firm based on both outbound marketing and in-store browsing activity. The store visit probability model is an auto-regressive model given by
The store visit utility, $\mu^{store}$ depends on the customer's latent propensity $\gamma_0^{store}$ to revisit the store, outbound marketing activity $X^{store}$, and the effect of browsing activity in the prior time step, $SV$. We make the simplifying assumption here that all products are in the customer's consideration set for each visit but would recommend relaxing this assumption for larger assortments.
Once the customer has decided to visit the store, category purchase decisions are assumed to be independent Bernoulli trials, where the probability of purchase is given by $\exp(\gamma_{0j}^{cate} + \gamma_{1j}^{cate} CV_{ujt}) / (1 + \exp(\gamma_{0j}^{cate} + \gamma_{1j}^{cate} CV_{ujt}))$. Product choice decisions follow a multinomial logit choice model given by $\exp{(\mu^{prod}_{uit})}/ \sum_{k \in J_j} \exp{(\mu^{prod}_{ukt}})$. For a realized product choice, the purchase quantity is given by a shifted Poisson distribution with the form $\lambda_{uit} ^{q-1}\frac{exp(-\lambda_{uit})}{(q-1)!}$ where $\lambda_{uit} = \exp(\gamma^{prod}_{0i} + \gamma^{prod}_{ui}\ \mu^{prod}_{uit})$.
Agents integrated with the RetailSynth environment can optimize product assortment, pricing, or marketing decisions. Here, we focus on personalized marketing agents, leveraging contextual bandit and deep RL algorithms and train those agents off-policy on summarized histories of customer purchases zhu_pearl_2023 to target customer-specific coupons.
The offline batch training dataset $\mathcal{D}$ comprises tuples $(H_{u, t-1}, A_{u, t-1}, R_{t})$ and is of length $dim(U) \times T$. Summarized customer purchase histories, $H_t$ are obtained by applying a feature engineering function $f$ to raw observations $(O_{t=0} \ldots O_{t})$. An observation $O_t$ is defined as
where $\mathbf{Q}_{t-1} = (Q_{i(t-1)}\ for\ i\in I)$ represents the purchase activity in the previous time period, $\mathbf{P}^{shelf}_t = ((1 - D_{it})P_{i}^{base}\ for\ i \in I)$ the product shelf price, $X^{store}_t$ the store marketing features, and $\mathbf{X}_t = (X_{it}\ for\ i\in I)$ the product marketing features.
The contextual features used to summarize the customer purchase history, $H_t$, and their relative importance are shown in Appendix (ref). The action space consists of discrete coupons with discount values, $D^{coupon}$, in the range of $[0, 1)$ that apply to all products $i \in I$. The bandit agents are configured to use the revenue as the reward, $R_{t} = \sum_{i \in I} R_{it} = \sum_{i \in I} P_{it} Q_{it}$ where $P_{it}$ and $Q_{it}$ refer to the product price and quantity sold. Deep RL agents optimize the cumulative revenue over the full trajectory, $V_{t} = \sum_{\tau=1}^t \delta^{t - \tau} R_{\tau}$ where $\delta$ is the discount factor.
In our previous work xia_retailsynth_2023, we showed that the RetailSynth customer decision model produces highly variable short-term customer purchasing behavior and long-term store loyalty under different pricing scenarios due to the customers having different price sensitivities. We hypothesized that personalized promotions would provide revenue uplift in this environment by targeting discounts to customers based on their latent willingness to pay. In this study, we designed experiments to both validate the environment's suitability for its intended purpose and size the potential revenue uplift from using RL agents to learn personalized promotion policies.
We built our simulation workflows in Python 3.10, with key dependencies on NumPyro phan2019composable for customer choice modeling and TensorFlow Agents TFAgents for reinforcement learning. To ensure that we collected sufficient agent training data, we set up a cloud workflow on AWS for parallel computation, leveraging Batch for auto-scaling compute with R4 memory-optimized EC2 instances and S3 for storage awsbatch, awsec2. To keep compute costs reasonable, we capped the number of customers in the simulation to 100,000. We also reconfigured simulation parameters to increase the average store visit probability to \ 90% and decrease the size of the product catalog to 2,514. See Appendix (ref) for more details. We did profile the agent training workflows on G5 GPU compute instances; however, the observed speed increase was <50%, and we did not find it cost-effective to move our workload to the GPU. The entire workflow for the subsequent experiments consumed approximately 1,950 CPU-hours in total.
To verify that the simulation environment presented non-trivial trade-offs between short-term revenue and long-term customer loyalty, we first conducted simulations of static benchmark policies. In these simulations, we gave all customers the same coupon at each time step and then measured the accumulated revenue and customer retention rate after 70 time steps (Figure (ref)). We did observe a trade-off, with revenue and retention rates showing opposing trends for lower coupon discounts from 0% to 40%. The customer retention rate continued to increase for coupon discounts up to 50%, while the trend in accumulated revenue reversed. Based on these observations, we expected that an optimal policy would assign a range of coupon values to specific customers to increase customer retention and maximize overall revenue. For benchmarking learned policies, we used the static policy of 0% coupon value since it yielded the highest overall revenue.
We first collected an offline training dataset, leveraging a random collection policy, where the probability of choosing different coupon levels for each customer is uniformly distributed (Algorithm (ref)). The offline dataset comprised data from 100,000 customers over $T=50$ time steps, equivalent to roughly one year of purchasing history. In our distributed computing environment, we optimized memory usage by setting the batch size $B=100$ customers, requiring $N_{batch}=1000$ parallel simulations. To train the agents, we followed Algorithm (ref), training the agent on the offline dataset covering $t= 1, \dots, T$ and then evaluated their performance by resuming the simulation at time step $t = T +1$ and collecting $T_{eval}=20$ additional observations.
Our overall goal was to determine which agents could give the maximum performance with optimal hyper-parameters. For hyper-parameter tuning, we sampled $N_{tune}=20$ different hyper-parameter configurations using the Tree-structured Parzen Estimator in Optuna akiba_optuna_2019 and then selected the best configuration based on the average accumulated revenue from $N_{agent}=3$ independent training and evaluation runs. For final benchmarking with optimal hyper-parameters (Table (ref)), we followed the same agent training procedure, while increasing $N_{agent}$ to $10$ and setting $N_{eval}$ to $10$. See Appendix (ref) for more details.
We trained and evaluated a wide range of agents: linear contextual bandits (Linear Thompson Sampling (LinTS), Linear Upper Confidence Bound (LinUCB)); a neural contextual bandit (Neural Boltzmann (NB)); and deep reinforcement learning methods (Proximal Policy Optimization (PPO) and Deep Q-Network (DQN)). The performance metrics we computed included accumulated revenue, accumulated demand, customer retention, category penetration, and redeemed coupon discount value (Table (ref)). Reflecting common industry practice, we report accumulated revenue as the primary objective and consider the other objectives secondary. In the case where multiple agents yield similar revenues, we would recommend selecting an agent for deployment that performs best along the secondary metrics most relevant to the firm. For example, if the firm is trying to grow, then increasing customer retention and category penetration might be considered more important than minimizing the coupon discount rate.
Our results indicate that all the agents, except DQN, effectively learn to target coupons more effectively than a random policy. In this scenario, it is important not only to beat a random policy but also to outperform static benchmarks. Comparing against the best baseline policy of offering everyone a 0% coupon discount, only LinTS, LinUCB, and PPO agents show improved revenue performance. Looking at the secondary metrics, we find that the PPO agent generated the highest level of revenue, while minimizing the average coupon discounts. In a real-world setting, we would recommend deploying this policy based on the observed performance.
The relative performance of the different agents reflects important characteristics of the environment and training workflow that we have built here. First, the strong performance of the linear contextual bandit agents implies a simple linear structure in terms of rewards and state-action relationships. In addition, the strong performance of the PPO agent and poor performance of the DQN agent may be explained by two factors. First, the random collection policy is a relatively strong policy, and so a policy gradient method like PPO can reliably learn an effective policy. Second, the reward distribution is sparse (due to the customers purchasing only a small number of items in the catalog) and difficult to model accurately without overfitting, which may explain the poor performance of the NB and DQN agents that leverage neural reward models.
To provide additional guidance to practitioners, we performed a sensitivity study where we decreased the size of the training dataset and evaluated the performance of the top 3 agents, LinUCB, LinTS, and PPO. As shown in Appendix (ref), we observed only about a 1% decrease in revenue performance for the contextual bandit agents (LinTS and LinUCB) when decreasing the number of customers in the training set by 10x, while the PPO agent performance decreased about 7%. It is noteworthy that all of the agents provide revenue uplift, even with just 10,000 active customers---a fact that might surprise practitioners accustomed to case studies involving larger firms with millions of customers. It also reinforces the importance of using simpler algorithms over more complex ones early in the algorithm development process, where achieving lift over existing baselines as quickly as possible is the objective versus at a later point in time where the focus may shift to maximizing the performance of a proven decisioning system.
We conduct an in-depth analysis of the driving factors of agent performance and estimate how much potential improvement more advanced algorithms might achieve on this benchmark. Leveraging the interpretable nature of the simulations, we divide customers into two equal-sized segments based on their latent price sensitivity (Figure (ref)a). From a business point-of-view, the price-insensitive segment is more valuable because the customers, on average, generate nearly 50% more revenue, and so there is a real cost to offering them larger coupons than required to maximize long-term revenue.
We chose to carefully analyze the policies of LinUCB and PPO agents based on their strong performance and differences in algorithmic complexity. We observe in Figure (ref)b that both agents target the coupons as expected the majority of the time. The agents primarily allocated 50% off coupons to price-sensitive customers and 0% off to the price-insensitive segment. However, seemingly non-optimal coupons are also frequently offered to customers in both segments.
In Appendix (ref), we show the mean reward distribution of each offer for each segment. There is a clear separation between the best and worst actions for the price-insensitive segment, yet the bandit agent, which should choose actions with the highest reward, still chooses the 50% coupon discount almost 10% of the time for this segment. We observe a similar action allocation distribution for the PPO agent, which implies that the shortcomings of the policies are more a result of the input data rather than formulation of the optimization objective or model architecture. Improved performance then is likely to come from increasing the volume of training data and the richness of the customer features used to train the agents.
Adoption of AI agents in traditional industries like retail will be greatly accelerated by developing open datasets, benchmarking suites, and simulation platforms that enable reproducible evaluation and iterative development of new methods. In this work, we demonstrate how to leverage RetailSynth simulations of customer shopping behavior to benchmark the performance of coupon-targeting agents. This approach can scale to large customer bases and product assortments and be extended for a variety of important use cases from marketing message optimization to assortment selection. We also recognize that there are a number of important challenges to overcome:
Addressing these challenges will help drive reinforcement learning forward and make it more practical to deploy AI agents in traditional industries such as retail. Our ultimate goal is to enable retailers to offer more deeply personalized and satisfying shopping experiences.