Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
93,050 characters · 48 sections · 59 citation commands
Multi-Agent Reinforcement Learning for Dynamic Pricing in Supply Chains: Benchmarking Strategic Agent Behaviours under Realistically Simulated Market Conditions
\email{[email removed], [email removed],[email removed], [email removed]}
\pagestyle{plain}
\setcounter{page}{0}
As customers demand more advanced functionalities from enterprise resource planning (ERP) systems and their vendors to manage inventories, manufacturers, suppliers, distributors, and retailers, the need for an intelligent dynamic pricing mechanism becomes more critical tarn2002exploring, emma2024enterprise, smith2024dynamic. ERP systems efficiently integrate and automate core supply chain functions such as procurement, logistics, and financial planning, but their capacity to optimize dynamic pricing stays underutilized. Modern supply chains are highly interconnected and complex, with multiple stakeholders each making pricing and inventory decisions jain2008managing. Most of the entities in these chains still rely on static pricing models. These are based on fixed rules, historical data, or cost-plus formulas, which fail to reflect real-time changes in demand, supply, or competitor behaviour hajji2012dynamic, li2015pricing. As a result, these static pricing strategies could lead to revenue losses and missed market opportunities. For instance, Cachon and Feldman (2010) show that static or naïve dynamic pricing strategies can underperform by up to 22.6% in revenue compared to dynamic pricing, with an average loss of 7%, showing the real cost of failing adaptive pricing mechanisms cachon2010dynamic. Unlocking this would allow businesses to set prices dynamically through demand, inventory, and other factors, thus enhancing profitability and competitiveness.
Dynamic pricing offers a data-driven alternative to static pricing by adjusting prices in response to market conditions. Integrating such strategies within ERP systems comes with several challenges. It requires seamless data exchange, real-time decision-making across actors, and scalable models. Current pricing solutions lack these capabilities, especially the ability to learn autonomously from evolving market dynamics menon2024erp. Machine learning techniques have been increasingly used in ERP systems for tasks like forecasting, but there remains a lack of solutions that realistically model decentralized and interactive pricing decisions across multiple supply chain actors in a learning manner.
Recent research has explored Reinforcement Learning (RL) as a method to optimize pricing strategies wong2023deep. Many RL-based models, such as Bayesian approaches, feature-based learning, and Q-learning, act in a single-agent setting, where pricing decisions are made isolated. This overlooks the interdependent nature of pricing decisions across supply chain entities jain2008managing, neto2005single. When a retailer adjusts prices, it affects demand, thus influencing supplier production levels and inventory. Ignoring these dependencies could potentially lead to suboptimal pricing strategies.
To address this gap, we benchmark and evaluate different Multi-Agent Reinforcement Learning (MARL) algorithms for optimizing dynamic pricing in supply chains, built upon existing and extending MARL models. Unlike traditional RL methods, MARL involves multiple autonomous agents, where each represents a distinct entity with a product catalogue and pricing decisions to learn collaboratively or competitively. These agents then interact with a shared environment that is shaped by market conditions, competitor actions, and changes in supply and demand. They dynamically adjust their pricing strategies through reward-based learning, thereby potentially improving revenue generation and market adaptability.
In this context, MARL could be particularly suited for capturing the interdependent nature of pricing decisions in supply chains, which traditional single-agent RL methods fail to model busoniu2008comprehensive. This study investigates the potential of MARL to enhance dynamic pricing strategies within ERP-integrated supply chains. Existing pricing mechanisms, particularly those based on static rules or heuristics, often lack the flexibility to adapt to volatile market environments and the strategic interactions between multiple pricing agents. By contrast, MARL offers a data-driven and adaptive alternative capable of capturing complex interdependencies and evolving agent behaviors in competitive and cooperative settings.
The research explores how various MARL algorithms, each with different learning dynamics and coordination mechanisms, can improve the adaptability, stability, and responsiveness of pricing. Comparative evaluation is conducted against traditional rule-based approaches to assess performance in environments characterized by fluctuating demand, customer segmentation, and supply-side variability.
In addition, the study examines how the efficacy of MARL-based strategies is influenced by underlying market dynamics, such as volatility, agent heterogeneity, and feedback latency. Through a series of controlled experiments, this work highlights the strengths and limitations of reinforcement learning in operational pricing tasks and contributes a practical framework for the deployment of MARL in enterprise-level decision support systems.
The remainder of this research is organized as follows. Section (ref) reviews relevant literature on pricing algorithms, reinforcement learning approaches to pricing, and challenges in applying MARL to supply chain contexts. Section (ref) details the methodological framework, including pre-processing of the data set, the demand model, the simulation environment, agent architectures, and the evaluation setup. Section (ref) presents the experimental results, comparing MARL strategies to rule-based baselines. Section (ref) interprets the findings in light of prior work, explores trade-offs, and discusses limitations. Finally, Section (ref) concludes the study and outlines directions for future research.
Prior studies denote the importance of dynamic pricing in supply chain revenue optimization and market adaptability, but its implementation to ERP systems remains fairly limited kim2015dynamic, han2019multi. Recent work has started exploring RL for pricing optimization, but it mostly focuses on single-agent configurations. This research aims to cross this gap by using MARL to enhance pricing strategies. To position this study in the existing literature, this section focuses on three areas: (1) Pricing Algorithms, covering both static and dynamic pricing; (2) Reinforcement Learning for Pricing Optimization, distinguishing between single-agent RL and MARL; and (3) Challenges in MARL, discussing computational complexity, scalability, and data limitations in applying MARL to pricing environments.
To build upon existing pricing methodologies, we explore static pricing, which relies on fixed or rule-based methods, and dynamic pricing, which uses machine learning to adjust prices. These pricing approaches are fundamental in supply chain management, providing a basis for more advanced RL-based pricing strategies.
Traditional pricing relies on rule-based methods, cost-plus pricing, and historical trends. These models typically set fixed prices based on predefined formulas rather than adapting dynamically to market conditions. Rule-based pricing, often used in retail, adjusts prices by, for instance, undercutting the lowest competitor price or following markup rules. In spite of their simplicity, rule-based pricing remains widely used, particularly among Amazon, Walmart, and eBay wang2023algorithms, chen2021multiobjective. Still, their lack of adaptability could lead to suboptimal revenue and potential algorithmic complicity in generally competitive markets. Another common approach, cost-plus pricing, determines prices by adding a fixed margin to costs. Even though this method covers the cost, it fails to account for demand elasticity, competitive actions, or supply fluctuations, making it unsuitable for dynamic market environments shafiee2010overview, li2015pricing. Static pricing strategies have also been applied in energy and financial markets, where fixed prices are used for long-term contracts and auction-based transactions weron2014electricity. While effective in stable markets, these models struggle in environments with high demand uncertainty or stochastic supply features. Recent research suggests that static pricing can sometimes achieve performance levels similar to dynamic pricing under specific conditions sun2024static.
Dynamic pricing models are being increasingly used across industries by adjusting prices in real-time based on demand fluctuations, inventory levels, and competitive behaviour hwang2006dynamic, el2023machine. Methods such as Linear Regression, Random Forests, and Gradient Boosting are commonly used to forecast sales or estimate demand das2024optimizing, gupta2014machine. Unsupervised learning methods, like clustering, support segmentation-based pricing strategies. However, these approaches usually fall short in adaptability to changes and may optimize for short-term gains without thinking about long-term customer behaviour. Real-world applications for this include airline ticket pricing, hotel bookings, and e-commerce platforms, where prices change dynamically. Despite their effectiveness, dynamic pricing strategies face challenges related to customer trust, market volatility, and complexity. As a result, there is growing interest in RL techniques that can learn pricing policies through interaction and adapt over time by explicitly modelling long-term objectives.
Reinforcement learning has been heavily applied in decision-making tasks, including those of pricing optimization. Traditional RL methods, such as Q-learning, Deep Q-Networks (DQN), and Actor-Critic (A2C) algorithms, extend the Markov Decision Process (MDP) framework and have shown promising results in adaptive decision-making watkins1992q, roderick2017implementing, konda2003actor. This section reviews two main approaches in RL for pricing: (1) Single-Agent Reinforcement Learning, where a single decision-maker optimizes its pricing strategy independently, and (2) Multi-Agent Reinforcement Learning (MARL), which extends RL to multiple interacting agents.
Single-agent RL (SARL) has been extensively used in various decision-making and optimization tasks watkins1992q. In SARL, an individual agent learns an optimal policy by interacting with an environment and receiving feedback in the form of rewards. The agent's goal is to maximize its expected reward over time. Traditional SARL methods rely on MDPs, assuming a stationary environment where the transition probabilities between states stay fixed hilton2023scaling. The most fundamental RL algorithm, Q-learning, is a value-based method where an agent updates a Q-table to estimate the expected future reward for each state-action pair. However, this method struggles with high-dimensional state spaces, leading to the invention of DQNs. These use neural networks to approximate Q-values, enabling better generalization watkins1992q. In addition to value-based approaches, policy gradient methods optimize the policy function by calculating gradients of the expected reward for the policy parameters. A2C methods combine value and policy-based learning, where the actor learns the policy and the critic evaluates the value function to improve learning stability georgila2014single. These methods are particularly useful for continuous action spaces. Q-learning and DQN models struggle here because of discretization challenges. Recent research has explored the scalability of SARL models, showing that RL algorithms' performance improvements follow power-law scaling with model size hilton2023scaling. This suggests that larger and more complex RL models can achieve better results. However, SARL faces significant challenges in dynamic and multi-agent environments by assuming a stationary and predictable environment. This is unrealistic in competitive markets and supply chain systems, where multiple decision-makers interact and all influence outcomes georgila2014single.
MARL extends regular RL by enabling multiple agents (such as manufacturers, suppliers, retailers, or robotic entities) to learn collaborative or competitive pricing strategies busoniu2008comprehensive. MARL has been applied successfully to logistics in the supply chain, inventory management, vehicle routing, and demand forecasting. This is not well explored in dynamic pricing ren2022multi, kwon2001multi. While single-agent RL assumes an isolated solitary decision-making entity to optimize its own strategy, MARL enables multiple agents to learn and make adaptive pricing decisions simultaneously with shared or competing objectives. This leads each agent to optimize its policy considering other evolving strategies, and hence a more dynamic and interactive environment for pricing. The interaction is most important in supply chains since the pricing, demand, inventory levels, and supply constraints are at different points of several interdependent entities blos2015modeling. MARL's ability to capture strategic interactions makes it well-suited for dynamic pricing scenarios where traditional methods fail to account for these multi-agent dependencies.
Despite its promise, MARL faces several challenges that have limited its application in complex practical settings. The decentralized interaction of agents leads to high computational overhead, problems with scalability, and instability of the training process. Other challenges in applications from supply chain management arise where the unavailability of quality training data and effective communication mechanisms among the agents are critical issues. A combination of algorithmic advancements and computational optimizations is needed to overcome these challenges. The following subsections highlight two major obstacles: Computational Complexity and Scalability and Data Limitations, together with potential solutions that mitigate these issues.
MARL significantly increases computational demands due to the exponential growth of the joint state-action space as agents are added. Unlike SARL, where policy updates are independent, MARL agents must adapt to each others changing strategies, complicating optimization. This problem in dimensionality clogs exploration and leads to increasing training times zhou2023multi, wong2023deep. A common solution is centralized training with decentralized execution (CTDE), where agents share information during training but act independently during execution. CTDE improves learning, but it provokes a high computational cost. Alternative strategies include mean-field approximations and graph-based learning. These reduce complexity by modelling only the local interactions wong2023deep. To ease constraints of resources, parallel and distributed learning could help, though synchronizing policies in non-stationary settings remains a burden.
Scaling multi-agent systems worsens state-action growth, making convergence harder and instability better, particularly in dynamic supply chains where agents must learn competing agents' behaviour. Graph-based MARL and hierarchical learning are approaches to structuring interactions more efficiently wong2023deep. Data scarcity is another challenge: ERP pricing data is often private or hidden, which makes research lean toward synthetic or simulated environments that may not capture real-world complexity. As a result, generalization across market conditions remains a key challenge zhou2023multi. Also, inter-agent communication creates overhead, thus hindering training speed. Solutions include attention mechanisms and selective message filtering, which can improve communication efficiency without slowing down performance wong2023deep.
This section describes the methodological framework developed to investigate how autonomous pricing agents can learn competitive strategies in a supply chain. The main task is to simulate a market environment in which pricing decisions from agents influence customer demand and the market. The core goal is to evaluate how different rule-based and MARL pricing strategies perform in terms of revenue, price stability, fairness, and market efficiency.
To do this, a real-world dataset was explored, preprocessed, and used to extract useful features in pricing and demand through feature selection and engineering. These features were used to train a predictive model that estimates customer demand with respect to price changes. This model functioned as the backbone for a custom simulation environment where agents set prices and receive feedback in a weekly cycle. The environment supported learning agents along with rule-based baselines. This was done by comparing strategic behaviours. The remainder of this section outlines the dataset selection and preparation, the construction of the demand model, the simulation setup, the agent architectures, and the evaluation protocol.
To simulate realistic supply chain pricing and demand scenarios, this research made use of the publicly available Online Retail II dataset, released through the UCI Machine Learning Repository chen2012online. The dataset contains over one million transactions recorded by a UK-based online retailer, specialized in giftware, serving both individual consumers and wholesalers, between December 2009 and December 2011.
Several studies have used this dataset to model customer segmentation and profitability dynamics. For instance, Chen et al. chen2012data employed a Recency, Frequency, and Monetary (RFM) model through k-means clustering and decision tree induction to segment customers based on behaviour patterns. This confirms the suitability of the dataset for consumer-centric business intelligence implementations. In more recent work, Chen et al. chen2015predicting continued to use this dataset to demonstrate the effectiveness of using RFM time series to model and predict customer profitability using a multilayer feed-forward neural network (MFNN). Also, the dataset's application extends to pattern mining. Singh et al. singh2018prefix referenced real-world retail datasets, including the Online Retail II dataset, to validate algorithms to extract behavioural sequences that begin or end with specific events using prefix/suffix sequential pattern mining.
The dataset's granularity and scale enabled detailed pricing and demand modelling across time, products, and customers. It covers 5,243 unique products and 5,876 customers. Each transaction included key variables such as product ID, timestamp, price, quantity, and customer ID, allowing for temporal, behavioural, and price-level analysis. An overview of the entire dataset structure, including column names, data types, and definitions, is provided in Appendix (ref).
EDA revealed several patterns relevant to pricing and demand modelling. B2B transactions were selected due to the presence of customer identifiers and notable behavioural differences; as non-registered customers paid on average 55.7% more. Transactions were highly concentrated in the UK (92.1%) and showed strong temporal effects, such as end-of-year seasonal peaks, midday spikes, and higher activity on Thursdays. Product pricing was highly variable, with some items showing volume discounts of over 50%. Price volatility (as measured by coefficient of variation) averaged around 0.39, with a small subset exceeding 2.0 (as seen in Figure (ref)) hendricks1936sampling. Cancellations and returns were rare but were removed during cleaning. Complete descriptive statistics, volatility figures, and formulas are detailed in Appendix (ref).
This section outlines the preprocessing pipeline applied to the dataset. This includes the removal of abnormal records, the extraction of customer and time-based features, and the construction of domain-specific variables. Additionally, product categories, week numbers, lagged and smoothed demand, price and trend indicators served as the foundational inputs for a demand prediction model as well as input states for agent models.
To ensure relevance and data quality, preprocessing started with the removal of invalid entries. Transactions with negative values for quantity or price were excluded, as well as records without a customer ID. These filters removed returns, errors, or overall incomplete purchases.
To enhance predictive power and help align with the temporal structure of demand modelling, a variety of domain-relevant features were engineered from the dataset.
\paragraph{Datetime and Country Features} Multiple datetime indicators were extracted from each timestamp, including year, month, day, week, weekday, hour, and two binary flags: one for weekends (to capture behavioural differences between weekdays and weekends) and one for the holiday season, which is set to true during calendar weeks 47 to 52 to denote end-of-year peak activity. To support geographic segmentation, the categorical country column was also encoded into a numerical code label variable.
\paragraph {Semantic product clusters} To capture semantic similarities between products, clustering was applied to the description field. Each unique product description was embedded using Sentence-BERT (MiniLM-L6-v2), a model designed to produce semantically meaningful sentence embeddings suitable for tasks such as clustering and semantic search reimers2019sentencebert. The resulting vector representations were grouped using K-Means clustering into 20 product categories, generating a new categorical feature. This number was chosen to balance category differentiation with avoiding fine-grained or overlapping product categories. This allowed for the representation of similar products to be grouped on semantic meaning instead of sparse identifiers, aligning with evidence that clustering based on extracted feature representations can enhance modelling performance abdalgader2024experimental, bandara2020forecasting.
\paragraph{Temporal demand signals} Weekly aggregations were computed per product to align the dataset with the temporal structure of demand modelling. To capture short- and long term trends, several time-based features such as previous-week sales and over 2- and 4-week window rolling averages were derived bentaieb2012review.
\paragraph{Transformations and interaction features} To improve model behaviour and reduce skewness, demand was log-transformed. To also capture non-linear effects, price received both logarithmic and quadratic transformations. Seasonality was approximated using sine and cosine decompositions of calendar week and month values. Interaction terms (such as price-holiday, and price-category) allowed for conditional effects. Finally, features like momentum, trajectory, volatility, and price vs. category average were included to enhance sensitivity to recent temporal market behaviour. Definitions and equations for all indicators and features are in Table (ref), Appendix (ref).
Forecasting weekly product demand is essential for simulating market behaviour and informing agent pricing. This section describes the supervised model used to simulate demand.
To forecast weekly demand, a LightGBM (Light Gradient Boosting Machine) model ke2017lightgbm was used. It was selected for its computational efficiency, scalability, and ability to handle high-dimensional tabular data with minimal preprocessing shwartz2022tabular. Unlike deep learning models, LightGBM offers interpretability via feature importance scores, aligning with ERP requirements for explainability. The model was trained on a product-week aggregated dataset (8,777 observations, 21 features) with log-transformed demand as the target.
As shown in Table (ref), the model generalized well, achieving $R^2 = 0.74$ on the test set. The gap with cross-validation ($R^2 = 0.591 \pm 0.093$) suggests the test set contains more predictable patterns. This performance is sufficient to simulate realistic agent interactions.
Temporal and pricing features, including lag, rolling means, trend, and volatility, ranked highest in feature importance (Appendix (ref)), confirming their relevance. To assess price responsiveness, counterfactual analysis was performed by scaling price in the test set (0.5$\times$ to 2.5$\times$) while holding other inputs constant. The resulting demand curve (Figure (ref)) was smoothed using a 5-point centered rolling mean.
Price elasticity of demand ($\varepsilon$) was calculated as:
The resulting $\varepsilon = -0.072$ indicates inelastic demand, which is typical for giftware products wang2018modeling. Although trained on log-transformed demand, elasticity was computed in the original scale using inverse-transformed demand predictions.
A custom simulation environment was developed to simulate market interactions and evaluate dynamic pricing strategies in a competitive setting. This environment provided a weekly time-stepped framework in which multiple pricing agents interact in a shared market and observe demand feedback influenced by their own and competitors' pricing decisions.
The environment was initialized with a list of agents and a pre-trained LightGBM-based demand prediction model (see Section (ref)). Each simulation step corresponds to one calendar week and updates the internal state variables such as the current date, year, week number, and holiday indicator. Agents submit product prices, and the environment uses the demand model to simulate weekly sales based on product-level features and competitive market conditions.
After predicting demand and calculating profits for each agent, the environment updates product-level demand histories and aggregates outcomes into a centralized history log. In addition, a structured market observation dictionary with week-number, holiday, and category clusters is compiled and passed to agents helping inform future pricing decisions.
Each agent maintains a portfolio of products, each with its own cost structure and demand history. At each simulation step, agents observe the environment and select product pricing actions. For each step in the loop, it predicts demand, computes revenue, updates histories, and invokes an act on each agent to determine the next pricing action, allowing agents to refine their policies based on observed outcomes and evolving market conditions.
Three distinct Multi-Agent Reinforcement Learning (MARL) frameworks were implemented and evaluated for optimal pricing strategies: Multi-Agent Deep Deterministic Policy Gradient (MADDPG), Multi-Agent Deep Q-Network (MADQN), and QMIX. Each offers different capabilities for the dynamic pricing domain, including continuous vs. discrete action spaces, degree of centralization, and agent coordination mechanisms.
Multi-Agent Deep Deterministic Policy Gradient (MADDPG) extends DDPG to multi-agent settings, enabling stable learning in non-stationary environments through centralized training and decentralized execution lowe2017multi. Adapted from OpenAI's reference code for TensorFlow 2.x and dynamic pricing openai_maddpg, each agent uses a local actor network to map observations to continuous pricing actions. A centralized critic evaluates joint actions using the global state. The full architecture of the MADDPG agent, including actor-critic structure, smoothing, and reward design, is illustrated in Appendix (ref). This highlights the actor networks, action execution, centralized critics, and experience replay buffers, including custom contributions like recency-biased sampling and price stability penalties. The MADDPG implementation consists of:
While action decisions are decentralized, centralized critics leverage joint state-action information during training to improve learning stability across agents, encoding competitive market dynamics via price ratios, demand trends, seasonality, and market share metrics in agent state representations. During execution, each agent independently observes the market and adjusts prices by applying its actor network, but during training, the centralized critics leverage global information to compute more accurate gradients, facilitating stable policy learning across all agents.
The Multi-Agent Deep Q-Network (MADQN) adapts DQN for multi-agent environments using discrete pricing actions ($-10\%$ to $+10\%$) mnih2015dqn, foerster2016learning. Unlike MADDPG’s continuous control and centralized critics, MADQN agents learn independently, relying solely on local observations and rewards. This decentralization encourages self-interested behaviour, suitable for competitive pricing without coordination. MADQN tends to produce more aggressive strategies, leading to higher average prices and sharper differentiation between agents. Its architecture (Appendix (ref)) includes a fully-connected Q-network (128, 64, 32 neurons), a target network for stability, $\varepsilon$-greedy exploration, and a recency-biased replay buffer to decorrelate updates. Custom elements such as price stability penalties and prioritized sampling distinguish this implementation from standard DQN li2021revisiting, liu2019dynamic.
QMIX represents a value-based MARL approach that overcomes the limitation of independent Q-learning (like MADQN) by enabling coordinated strategies. It combines per-agent utility functions (Q-values) into a single joint action-value function through a mixing network that enforces monotonicity constraints rashid2018qmix.
Designed for decentralized execution with coordinated learning, each agent maintains its own Q-network for local action evaluation, while a centralized mixing network aggregates these per-agent utilities into a global action-value estimate. This mixing network, conditioned on a global state, ensures interpretability of local Q-values while enabling coordinated strategies. A coordinator pattern manages updates for efficient joint optimization during training.
QMIX's advantage lies in representing joint action-values which is not just a sum of individual functionalities, allowing it to learn policies where inter-product pricing coordination gives better returns than independent price optimization. As shown in Appendix (ref), its architecture includes agent-level Q-networks and a centralized mixing network. In practice, QMIX fosters a more balanced price-competition dynamic, discovering strategies that maintain competitive positioning and avoid destructive spirals, ensuring pricing decisions are locally informed yet globally coordinated.
To contextualize MARL dynamic pricing, this study implements rule-based agents. Unlike learning agents, these do not adapt strategies, but follow fixed decision rules from common retail pricing practices. These rule-based agents serve as realistic, interpretable benchmarks, representing strategies typically embedded in ERP and pricing management systems.
Each Rule-Based agent follows a specific strategy that defines how prices are adjusted weekly. These strategies are deterministic but context-sensitive, reacting to competitor prices, historical data, demand signals, or seasonal effects. The following strategies were implemented:
The experimental framework was designed to evaluate the effectiveness of different pricing strategies in a competitive environment. A structured simulation was implemented, which allowed for controlled comparisons between MARL algorithms and baseline strategies.
The simulations were configured to model market dynamics over a span of two years, where each episode represented 104 weeks. At the start of each episode, four competing agents were initialized with an identical product portfolio. Each consisting of five products with category clusters (1, 2, 3, 5, and 10), where products were all initialized with identical starting prices. Then, the simulation was run for 30 episodes per experiment to allow sufficient time for MARL algorithms to converge.
Each agent-type was carefully tuned and optimized for the retail pricing domain (see Table (ref) for an overview).
Exploration parameters were designed to decay exponentially with increasing episodes, facilitating thorough environment exploration in early training while progressively shifting toward exploitation. For MADDPG, exploration noise started at 0.2 and decayed multiplicatively by a factor of 0.9995 per episode, with a minimum threshold of 0.05. MADQN and QMIX employed an $\epsilon$-greedy exploration strategy, starting at 1.0 and decaying by 0.995 per episode, also with a minimum exploration rate of 0.05.
To assess the effectiveness of the implemented MARL algorithms and benchmark strategies, multiple evaluation metrics were employed, capturing both economic performance and market dynamics. In particular, this study emphasizes emergent fairness, and market stability, reflecting the research objective to explore not only revenue optimization but also the broader implications of agent behaviour on market outcomes.
The metrics include revenue per agent, price stability, Nash Equilibrium Proximity nash1950equilibrium, optimality gap, welfare fairness ceriani2012origins, atkinson1999contributions, Jain's fairness Index jain1984quantitative, market share evolution, and price volatility. Together, these metrics provide a comprehensive view of agent adaptation, competitive behaviour, and overall market outcomes. Formal definitions and computational formulas for each metric are detailed in Appendix (ref) (Table (ref)).
All experiments were executed on the Snellius National Supercomputer, operated by SURF in the Netherlands surf_snellius. The experiments were configured to run on CPU-only mode. Each simulation utilized 8 CPU cores and was allocated 16 GB of RAM. The models were implemented using TensorFlow 2.x, optimized for CPU operations. Each full simulation was allocated up to 24 hours of computation time. The most compute-intensive components were the neural network training operations for MADDPG and QMIX.
Experiments ran under fixed computational conditions (as detailed in Section (ref)), controlled by set scripts to minimize variability between runs. All experiment configurations (agent types, hyperparameters, simulation setups) were defined in dictionaries, serialized as JSON, and executed in isolated, and timestamped directories to prevent cross-contamination. Models, key simulation outputs, and performance metrics were systematically stored in standardized CSV files, logs, and visualizations. Methodological consistency was ensured through fixed time horizons (104 weeks per episode, 30 episodes) and unified evaluation windows. Finally, all simulations used a single, pre-trained demand model, and agents were initialized with identical product portfolios, pricing, and cost structures. The simulation environment maintained consistency with fixed weekly progressions, uniform agent action protocols, and centralized demand modelling.
Pricing adaptability refers to the ability for an agent to adjust prices to market conditions over time, which are expressed as changes in pricing magnitude and frequency. While high adaptability can enable quick responses to demand or competition fluctuations, if it is unregulated, it could cause severe instability. We evaluate it using two custom metrics (defined in Appendix (ref)): adjustment magnitude (average absolute percentage price change between time steps) and adjustment frequency (proportion of changes exceeding a 1% threshold).
As shown in Table (ref), MADQN agents demonstrate the most aggressive pricing, exhibiting the highest adjustment magnitude (0.0358) and frequency (0.763). This indicates a reactive, and exploratory strategy. QMIX and mixed MADQN configurations also show increased responsiveness to these metrics. Alternatively, MADDPG agents are more conservative (mean magnitude 0.0152), with smoother and less volatile policy updates. As expected, rule-based agents have the lowest adjustment metrics due to their static design. Crucially, this means that high adaptability does not always lead to instability. Mixed strategies (such as MADDPG + Rule, MADDPG + MADQN) maintain high price stability (>0.98) despite frequent adjustments. This suggests that hybrid agents can balance dynamic pricing with coordinated stability. However, MADQN's high volatility (0.085) together with high adaptability implies that aggressive pricing can cause instability. Rule-based agents are the most stable (0.944-1.000) and least volatile (\textit{0.024}), but mostly lack adaptability.
These findings highlight a trade-off: adaptability improves market responsiveness but it can generate instability unless balanced by conservative agents. Hybrid agent configurations are promising for achieving dynamic pricing while maintaining stable.
This subsection compares the performance of MARL agents with rule-based pricing agents across all configurations. The evaluation focuses on mean revenue generation, and market share.
Figure (ref) illustrates the market share captured at the final episode for each agent across experimental configurations, including 95% confidence intervals over each independent simulation run. This offers insight into long-term competitive dominance and strategic effectiveness.
To compare configurations statistically and isolate individual agent performance, Table (ref) reports the mean return per agent, averaged across all episodes and eight independent simulation runs. This allows for a normalized comparison and highlights significant performance improvements over rule-based agents.
To assess performance differences in per-agent reutrns, the Wilcoxon signed-rank test was applied. MARL-only configurations (B, C, F) were tested against the rule-based baseline (A). Even though consistent improvements were observed, statistical significance was not achieved $(p = 0.125)$. This was due to the small sample size (four agents). Future experiments with more agents could improve statistical power. Configurations with mixed types (D, E, G, H) were excluded from direct statistical testing.
\paragraph{Rule-based agents} achieve equitable market shares and highly stable pricing but generate substantially lower total revenue due to static pricing and limited responsiveness. Interestingly, in mixed configuration G (one MADDPG agent vs. three rule-based), total revenue modestly rises to \pounds628,016, suggesting rule-based agents can constrain isolated agents and that a single MARL agent is not enough to disrupt a stable rule-based market.
\paragraph{MADQN} agents exhibit the most pronounced revenue dominance, generating just under \pounds1 million in mean revenue through aggressive pricing. However, this comes with high price volatility and substantial market share fluctuations. This indicates significant short-term revenue generation but unstable competitive outcomes. This is probably due to a lack of coordinated pricing behaviour.
\paragraph{QMIX} agents consistently perform well in cooperative settings. They effectively coordinate pricing for strong revenue and sustained competitive positioning. Their performance deteriorates in heterogeneous configurations with learning assumptions that are conflicting. This highlights their strength in homogeneous or collaborative markets.
\paragraph{MADDPG} agents adopt a more cautious strategy. They achieve modest revenue and market share by prioritizing pricing stability over aggressive profit maximization. When combined with other agents (Configurations D and H), MADDPG contributes to dynamic outcomes but rarely dominates, which suggests conservative and risk-averse learning dynamics.
Overall, MARL agents outperform rule-based strategies in revenue generation but introduce trade-offs in volatility and coordination complexity. MADQN effectively brings value but can destabilize the market. QMIX offers a strong middle ground for cooperative agents, while MADDPG provides (suboptimal) stable pricing. Even though less adaptive, rule-based agents remain competitive in mixed environments. This proves that they are valuable in systems that prioritize transparency and stability.
To assess how market dynamics influence agent effectiveness, we analyse three key categories of outcomes. First, we examine emergent market dynamics, including fairness, and market volatility, which reflect structural conditions in the simulated environment. Second, we evaluate coordination behaviour through metrics such as Nash equilibrium proximity, and price convergence. Finally, we assess agent performance in economic terms via efficiency indicators, such as revenue optimality gap, and welfare fairness. These categories together allow us to analyse how different configurations influence total performance in terms of competitiveness, coordination, and responsiveness.
Table (ref) presents key indicators of market-level dynamics, coordination and performance. Fully rule-based environments (4x Rule) exhibit near-perfect fairness (0.9896 $\pm$ 0.0000), zero market share volatility, and low price instability. This forms a clear baseline for predictable dynamics. In contrast, configurations involving MADQN agents, exhibit significant volatility and lower fairness scores (such as 0.5844 $\pm$ 0.1625 for 4x MADQN). These patterns highlight a tendency for highly adaptive agents to disrupt equitable competition, introducing pricing instability. Hybrid configurations can either moderate or increase instability. For example, 1x MADQN + 3x Rule maintains high fairness (0.9991) and low volatility (0.7pp), indicating that the presence of a rule-based agent can constrain destabilizing behaviour. Alternatively, MADDPG + MADQN or MADDPG + QMIX configurations show lowered fairness and increased volatility. This suggests that hybridization does not necessarily stabilize dynamics without balance.
As shown in Table (ref), strategic coordination varies considerably across configurations. Nash equilibrium proximity is highest in 1x MADDPG + 3x Rule (0.9999) and 2x MADDPG + 2x MADQN (0.8782). This indicates that hybrid configurations may encourage behaviour that seeks an equilibrium. This is potentially due to tension between adaptive and fixed strategies. Interestingly, 4x MADQN shows the lowest Nash equilibrium proximity (0.5788). This strengthens earlier findings that these agents focus on aggressive and divergent strategies that usually disrupt stability. Price convergence follows similar trends: 4x Rule and 1x MADDPG + 3x Rule reach the highest convergence levels (0.9761 and 0.9324), while 4x MADQN yields almost no convergence at all.
Table (ref) examines whether dynamic strategies deliver economically meaningful improvements. Unsurprisingly, the lowest revenue optimality gap is observed in the rule-based configuration (0.0000). Note that this does not come with high absolute performance. In contrast, 4x MADQN reaches a high gap (0.7060), which suggests significant room between the actual and optimal pricing performance, even as it achieves the highest mean revenue overall. This confirms that aggressive pricing may exploit short-term gains without long-term efficiency. Finally, welfare fairness scores comply with earlier fairness metrics, with rule-based and hybrid configurations (such as 1x MADDPG + 3x Rule) maintaining high fairness. Conversely, MADQN and MADDPG + MADQN configurations rank lowest here, which suggests that price aggressiveness compromises not only fairness but also equitable welfare distribution.
These results highlight a fundamental feature in multi-agent learning environments. While adaptability can increase revenue, it often lowers market stability, fairness, and overall coordination. Rule-based and MADDPG agents deliver stable but less efficient outcomes. Conversely, MADQN excels in profit maximization at the cost of volatility and inequity. Hybrid configurations can strike a balance, but only when carefully configured and tuned. Agent interactions therefore shape not just outcomes but the very nature of the market itself. This makes configuration design a critical consideration in deploying such learning based pricing systems.
This section reflects on the key findings, examines how they relate to existing work, and considers the implications of the results. It also addresses limitations of the current study and touches on relevant ethical concerns.
Earlier studies on RL for pricing tend to focus on single-agent environments georgila2014single, neto2005single. These approaches fall short when it comes to capturing the interactions between different supply chain actors. This study expands on this by testing several MARL methods: MADDPG, MADQN, and QMIX inside a market simulation based on actual retail data. It further advances recent MARL pricing research wong2023deep, zhou2023multi through two contributions. First, it introduces hybrid agent populations, combining learning agents with rule-based strategies to reflect realistic market heterogeneity. Following, it evaluates performance using structural metrics such as fairness, coordination, and market stability, which remain under-explored in literature but are critical for practical deployment.
Among the tested models, MADQN stood out in terms of revenue generation. Its performance exceeded that of both static and single-agent baselines by a large margin. However, this level of gain came with costs which reflect similar patterns found in adversarial RL research lowe2017multi. High revenues were often paired with instability and uneven market outcomes. In contrast, QMIX was better at promoting coordination among agents. Its behaviour aligns with findings in the cooperative MARL literature liang2025review. MADDPG took a more stable path, often finding a trade-off between stability and adaptability. This study is set apart from purely simulated work due to the integration of a real-world demand model, which was built using LightGBM and enriched with price elasticity estimates.
The experiments highlight a trade-off between adaptability, fairness, and stability. MADQN agents adjusted prices most frequently and captured the highest revenue, but introduced volatility and inequity. Rule-based agents, by contrast, provided stability and fairness but lacked responsiveness, leading to lower revenue. Mixed agent configurations performed best overall. For instance, MADDPG paired with QMIX or rule-based agents achieved moderate adaptability while maintaining fairness and coordination. This suggests that combining exploratory and conservative agents balances market responsiveness and stability. Finally, MARL agents did not always dominate in mixed-agent settings. When surrounded by rule-based agents, even highly adaptive agents were limited in influence. This underscores the need to consider market composition and real-world constraints when deploying MARL in practice.
While the results provide valuable insights into the behaviour of MARL agents in dynamic pricing scenarios, several limitations must be acknowledged to contextualize the findings and guide future research. The following sections outline key areas where these constraints were most notable.
The demand model produced a near-zero elasticity ($\varepsilon = -0.072$), which denotes inelastic behaviour typical of giftware sales, where demand is relatively insensitive to price changes. However, this is a limit in generalizability. In elastic markets like travel, digital services, or groceries, consumers respond more strongly to price shifts. Several MARL agents, including MADQN and QMIX, exploited this inelasticity by raising prices across episodes, knowing demand would remain fairly stable. While optimal in this context, such behaviour would likely be restricted in markets where elasticity burdens pricing power. This highlights the need for future simulations to use more elastic datasets or model heterogeneous consumer responses to avoid unrealistic, and loophole-driven strategies, ensuring that insights better reflect real-world dynamics.
Each agent handled a small, fixed group of products chosen from different clusters. While this setup allowed for controlled experiments, it does not reflect the complexity of actual supply chain environments. Real systems often involve large and nested product structures, cross-selling, and bundled pricing. Future work could explore agents that work over multiple levels of a catalogue and make combined decisions across related items.
The dataset used for modelling comes from a single UK-based retailer, collected between 2009 and 2011. Even though it has a large transaction volume and structured format, making it suitable for simulation, several limitations affect generalizability. Customer behaviours, such as response to discounting, average order size, and purchasing frequency, could reflect UK consumer behaviour in the early 2010s. This feature heavily is shaped by local economic conditions and retail habits of that period. These patterns may be significantly different in other regions (such as North America or Asia) or in today’s e-commerce environments, where consumer expectations, promotional sensitivity, and platform dynamics have significantly changed. Similarly, seasonality patterns such as holiday spikes in demand may not align with global retail volatility and changes. As a result, the dataset supports controlled experimentation, but future studies should consider more recent and regionally diverse data sources, which would enhance realism and generalizability.
Reproducibility was a core principle in this study. This was ensured through deterministic data splits and consistent training procedures, but some variability in outcomes was observed due to inherent stochasticity of RL. Certain agents, such as MADQN and QMIX, occasionally exhibited different convergence behaviours across independent runs even with identical initial settings. Although this makes exact replication of individual trajectories challenging, average performance across runs was stable. These same agent types were particularly resource intensive, with training times of up to 12 hours per configuration on optimized 8-core CPU infrastructure (Snellius), reflecting the computational complexity discussed in Section (ref). This limits scalability in larger markets or real-time ERP contexts. The current environment did not yet include other factors such as supply limits, delayed fulfilment, or inventory-sensitive pricing. Future work should explore how such constraints shape agent behaviour and influence pricing decisions.
This research explored how MARL can enhance dynamic pricing strategies in supply chains. This was done by addressing the gap between static pricing logic together with the need for adaptive, decentralized decision-making under market uncertainty. By benchmarking MADDPG, MADQN, and QMIX against rule-based pricing agents in a realistic market simulation, the study aimed to analyse and understand the trade-off between adaptability, revenue performance, fairness, and stability.
The main research question asked how MARL can improve dynamic pricing compared to traditional models while accounting for interactions between agents. The results showed that MARL, and MADQN in particular, significantly outperformed rule-based agents in terms of revenue, confirming the advantage of data-driven adaptation. However, this gain comes with a cost: higher volatility, reduced fairness, and coordination difficulties. This study showed that:
Concluding, while MARL has clear advantages, its effectiveness depends on the design of agent configurations and the underlying market conditions. Its benefits depend on how and where it is used, including algorithm setup and system limitations. The inelastic nature of demand in the dataset also limits generalizability to other markets. Nonetheless, the observed agent behaviours and trade-offs offer insights into how MARL systems behave under stable demand conditions. In particular, this study suggests that MARL is most beneficial when demand is predictable, competition is strategic, and price setting is flexible.
This study demonstrates that MARL can be practically integrated with ERP-relevant demand modelling. It also shows the importance of design considerations for dynamic pricing agents; particularly in low-elastic or regulated market segments. It also highlights that the environment and strategies of others strongly shape the performance of adaptive agents; an insight that is often missing from prior single-agent literature.
Future research should focus on two promising directions. First, it should incorporate products with elastic demand, which would help evaluate pricing responsiveness in more sensitive markets. Second, it should explore scalable MARL architectures, such as graph-based or hierarchical methods. These could improve coordination and training efficiency; especially in larger supply chains. These steps would help move MARL from experimental validation towards deployment readiness in business enterprise systems.
\onecolumn