Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
31,053 characters · 30 sections · 16 citation commands
Incorporating Cognitive Biases into Reinforcement Learning for Financial Decision-Making
Financial markets are complex adaptive systems in which human decision-making plays a central role. Traditional economic theory assumes rational agents who maximize expected utility; however, empirical evidence consistently demonstrates that real-world financial decisions deviate significantly from this rational model kara2025thero, kanapickiene2024acr. Cognitive biases, such as loss aversion, overconfidence, and anchoring, systematically influence investor behavior, creating observable patterns in market dynamics that cannot be explained by purely rational models.
Reinforcement learning (RL) has emerged as a powerful framework for automated trading systems, capable of learning optimal strategies from market data without explicit programming pricope2021deeprl, xu2023deeprl. However, most RL-based trading agents assume rational decision-making, potentially overlooking the psychological factors that drive real market participation. This disconnect raises a fundamental question: can incorporating cognitive biases into RL frameworks lead to more realistic and potentially more robust financial decision-making systems?
This study explores the integration of cognitive biases into reinforcement learning for financial decision-making. We propose a framework that explicitly models behavioral biases, particularly loss aversion and overconfidence, within a Q-learning trading agent. Our approach modifies the reward structures and action-selection mechanisms to reflect the psychological phenomena observed in human traders. We conducted comprehensive experiments, including hyperparameter tuning, ablation studies on bias parameters, and analysis of training dynamics.
Our experimental results reveal significant challenges in achieving the hypothesized benefits of this approach. Although some configurations demonstrate interesting behavioral patterns, most fail to outperform rational agents at the baseline. These findings provide valuable insights into the complexities of bridging behavioral finance and machine learning, highlighting critical pitfalls and providing guidance for future research directions. This study serves as both a cautionary tale and a foundation for developing more nuanced approaches to incorporating human psychology into algorithmic trading systems.
Reinforcement learning formulates trading as a sequential decision-making problem in which an agent interacts with a financial market environment. At each time step $t$, the agent observes a state $s_t \in \mathcal{S}$ representing market conditions, selects an action $a_t \in \mathcal{A}$ (e.g., buy, sell, or hold), receives a reward $r_t$, and transitions to a new state $s_{t+1}$. The goal is to learn a policy $\pi: \mathcal{S} \rightarrow \mathcal{A}$ that maximizes the cumulative expected return.
In Q-learning, we learn an action-value function $Q(s, a)$ that represents the expected return of taking action $a$ in state $s$ and following the optimal policy thereafter. The Q-function is updated using temporal difference learning as follows:
where $\alpha \in (0,1]$ is the learning rate and $\gamma \in [0,1)$ is the discount factor.
For financial applications, the state space $\mathcal{S}$ typically includes price features, technical indicators, or discretized market conditions. Actions commonly represent trading operations (buy, sell, and hold), and rewards reflect portfolio performance metrics, such as returns or risk-adjusted measures.
Previous work on RL-based trading has focused primarily on optimizing Sharpe ratios pricope2021deeprl or maximizing cumulative returns xu2023deeprl, typically assuming rational agents seeking optimal strategies to maximize their returns. However, real financial markets are populated by human traders whose behavior systematically deviates from rationality.
Behavioral finance research has identified numerous cognitive biases that systematically influence financial decisions kara2025thero, kanapickiene2024acr, wang2023theio. We focus on two key biases relevant to trading.
Loss Aversion: Loss aversion, a central concept in prospect theory turgay2025improvingdm, describes the tendency for losses to be perceived as more significant than equivalent gains. Empirically, the psychological impact of a loss is approximately twice that of an equivalent gain, leading to risk-averse behavior when facing potential losses and risk-seeking behavior when facing certain losses.
Overconfidence: Overconfidence bias manifests as traders overestimating their ability to predict market movements, leading to excessive trading frequency and underestimation of risk. After a sequence of successful trades, overconfident traders may increase their risk tolerance, believing that their success validates their predictive abilities.
These biases create observable patterns in market behavior, including momentum effects, mean reversion, and volatility clustering. De Bondt and ( Understanding these phenomena is crucial for developing robust trading systems that can operate effectively in real markets dominated by human participants.
Incorporating cognitive biases into RL frameworks presents several theoretical and practical challenges. First, biases are typically studied through static decision scenarios, whereas trading requires sequential decision-making under uncertainty. Second, the optimal method for parameterizing biases within RL reward structures is not well established. Third, bias incorporation may introduce training instability and lead to suboptimal policies. Our study empirically explores these challenges through systematic experimentation.
We formulate financial trading as a Markov Decision Process (MDP) $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$
The agent maintains a portfolio with a cash balance $b_t$ and asset holdings $\pi_t$ at time $t$. Given the current price $p_t$, the portfolio value is $v_t = b_t + \pi_t \cdot p_t$.
We incorporate loss aversion by modifying the reward function to asymmetrically penalize losses, relative to gains. The base reward is the change in the portfolio value:
The loss-averse reward function amplifies the negative rewards by a factor $\lambda \geq 1$:
This formulation reflects the empirical finding that losses are psychologically weighted approximately twice as heavily as gains ($\lambda \approx 2$) turgay2025improvingdm. In our ablation studies, we explore $\lambda \in \{1, 1.5, 2, 2.5, 3\}$ to examine the sensitivity to the loss aversion parameter.
We model overconfidence through dynamic risk adjustment based on the recent performance. Let $\bar{r}_k$ denote the average reward over the past $k$ time steps. The exploration rate $\epsilon$ is adjusted as follows:
where $\epsilon_0$ is the base exploration rate, and $\beta > 0$ controls the sensitivity to recent performance. This formulation decreases exploration (increases exploitation) after successful periods, modeling the overconfident trader's reduced perceived uncertainty.
We employ tabular Q-learning with $\epsilon$-greedy exploration to achieve this. The Q-table is initialized to $Q(s, a) = 0$ for all $(s, a)$ pairs. During training, actions are selected according to the following:
The Q-function is updated using a modified reward as follows:
where $r_{\text{biased}}$ incorporates loss aversion and potentially other bias modifications.
We discretized the continuous price data into $n$ states using equal-width binning. Given the price sequence $\{p_t\}_{t=1}^T$ with minimum $p_{\min}$ and maximum $p_{\max}$, we create $n-1$ bin
The state at time $t$ is determined by $s_t = \text{digitize}(p_t, \{b_i\})$, where digitize returns the bin index containing $p_t$. We explore $n \in \{5, 10, 15, 20\}$ to understand the impact of state-space granularity on learning dynamics.
We evaluated the performance using two primary metrics:
Sharpe Ratio: The risk-adjusted return measure:
where $r$ is the return sequence, $r_f$ is the risk-free rate (assumed to be zero), and $\sigma(r)$ is the standard deviation of returns.
Cumulative Returns: The total portfolio return over the evaluation period:
Higher Sharpe ratios indicate better risk-adjusted performance, whereas cumulative returns measure absolute profitability.
We generated synthetic financial data using a random walk model:
where $p_0 = 100$ and $\sigma = 1$. This generates $T = 200$ daily price observations, simulating a realistic trading environment with inherent uncertainty. The synthetic approach allows for controlled experimentation while avoiding market microstructure effects and data snooping biases present in real financial data.
Our baseline Q-learning agent uses a standard rational reward structure ($\lambda = 1$), fixed exploration rate $\epsilon = 0.1$, learning rate $\alpha = 0.1$, and discount factor $\gamma = 0.9$. The baseline served as a comparison point to assess the impact of bias incorporation.
We conducted three categories of experiments.
Hyperparameter Tuning: We systematically vary the state space size $n \in \{5, 10, 15, 20\}$ while keeping other parameters fixed, training for 50 epochs per configuration.
Bias Parameter Ablation: We explore loss aversion multipliers $\lambda \in \{1, 1.5, 2, 2.5, 3\}$ and discount factors $\gamma \in \{0.5, 0.7, 0.9, 0.99\}$ to understand sensitivity to bias strength and temporal discounting.
Architecture Ablations: We investigate the impact of action space reduction (removing "hold"), Q-table initialization strategies (zero, random, small positive), initial portfolio values, reward scaling for positive gains, and temporal continuity in reward calculation.
All experiments used fixed random seeds for reproducibility and reported results averaged over multiple runs, where applicable.
Figure (ref) presents the hyperparameter tuning results for state space discretization. Across all configurations ($n \in \{5, 10, 15, 20\}$), we observe consistently negative cumulative returns, indicating that none of the tested configurations achieved a profitable trading strategy. The rate of decline varies: configurations with $n=5$ and $n=20$ show steeper declines than those with $n=10$ and $n=15$, suggesting that both overly coarse and overly fine discretizations may impair learning.
The Sharpe ratio evolution (Figure (ref), left panel) exhibits high variability across epochs for all the configurations. None consistently maintained positive Sharpe ratios, indicating poor risk-adjusted returns. The variability suggests training instability, potentially due to the challenging nature of the synthetic random walk environment, where profitable strategies are inherently difficult to discover.
These results highlight a fundamental challenge: even without bias incorporation, finding profitable strategies in a random walk market is nontrivial. The efficient market hypothesis suggests that in truly random markets, no strategy should consistently outperform, and our results are consistent with this expectation.
Figure (ref) shows the impact of varying the loss aversion multipliers. Surprisingly, moderate loss aversion ($\lambda = 1.5, 2.0$) does not consistently improve performance compared to the baseline ($\lambda = 1$). Higher multipliers ($\lambda = 2.5, 3.0$) lead to worse outcomes, with cumulative returns declining more rapidly.
The Sharpe ratio trajectories reveal high volatility, regardless of the multiplier value. This suggests that simply amplifying negative rewards may not capture the nuanced effects of loss aversion on sequential decision-making. The increased penalty for losses may lead to overly conservative policies that avoid trading altogether, or conversely, may trigger risk-seeking behavior when facing certain losses, a phenomenon consistent with prospect theory but challenging to model in RL.
These findings indicate that incorporating naive loss aversion via reward scaling may be insufficient. More sophisticated modeling, perhaps incorporating reference points or framing effects, may be necessary to capture the psychological mechanisms underlying loss-averse behaviors.
Figure (ref) examines the effect of varying the discount factor $\gamma$. Higher values ($\gamma = 0.9, 0.99$) give more weight to future rewards, whereas lower values ($\gamma = 0.5, 0.7$) emphasize immediate returns. Our results show no clear superiority of any particular discount factor, with all configurations exhibiting similar and volatile performances.
This suggests that in our relatively short-horizon trading environment (200 time steps), the temporal structure of the rewards may not be the primary limiting factor. The role of the discount factor may be more critical in longer-horizon scenarios or when learning value functions that depend on strategic long-term positioning.
Figure (ref) compares the performance with and without the "hold" action. Removing "hold" forces the agent to always take a directional position (buy or sell), which may align better with human trading behavior, where pure cash positions are uncommon. The results show mixed outcomes: some state space configurations (particularly $n=20$) achieve slightly higher Sharpe ratios with a reduced action space, whereas others perform worse.
This suggests that the action space design significantly impacts the learning dynamics; however, the optimal configuration depends on the state representation. The relationship between state granularity and action-space complexity requires further investigation.
Figure (ref) compares different reward formulations: proportional gains (normalized by the initial balance) versus volatility-penalized rewards. Proportional gains show more stable but near-zero returns, whereas volatility-penalized rewards exhibit higher volatility with consistently negative outcomes. This indicates that the specific form of reward shaping strongly influences the learning dynamics, and careful design is necessary to achieve the desired behaviors.
Figure (ref) examines initialization strategies: zero, random (uniform $[0,1]$), and small positive ($0.1$). All strategies showed similar volatile performance with no clear winner, suggesting that initialization has a limited impact in our setting. This may be because the Q-table quickly overwrites the initial values during training or because the exploration strategy ($\epsilon$-greedy) ensures sufficient state-action coverage regardless of initialization.
Figure (ref) presents an interesting finding: agents initialized with positive portfolio holdings ($\pi_0 \in \{5, 10\}$) significantly outperform those starting from zero holdings ($\pi_0 = 0$). This suggests that having an initial position may help bootstrap learning by providing immediate feedback on portfolio value changes rather than requiring the agent to first learn to take positions.
This finding has practical implications: pre-training agents with initial positions or using transfer learning from pre-positioned agents may improve the training efficiency in RL trading applications.
Figures (ref) and (ref) explore advanced reward modifications. Scaling positive rewards (Figure (ref)) showed variability patterns similar to those of other experiments, with no clear benefit. Temporal continuity smoothing (Figure (ref)) demonstrates that intermediate state complexities ($n=15$) achieve better performance than extreme values but still fail to consistently achieve profitability.
Our comprehensive experimental analysis revealed several critical insights.
These findings contribute valuable negative results to the literature, helping guide future research toward more promising directions for integrating behavioral finance and reinforcement learning.
Our experimental results highlight several limitations of this approach. First, the synthetic random walk environment may not capture the nuanced dynamics of real financial markets, where biases may have different effects. Second, our simple tabular Q-learning approach may be insufficient to model the complexity of cognitive biases, which often involve reference points, framing effects, and context-dependent evaluations turgay2025improvingdm.
Third, training instability remains a significant challenge, suggesting that more sophisticated learning algorithms (e.g., actor-critic methods and policy gradient algorithms) or regularization techniques may be necessary. Fourth, our single-asset setting ignores portfolio effects and diversification benefits that may interact with bias modeling.
Ethical considerations also arise from incorporating biases into trading systems afjal2024evolvingtl, garg2025artificialii. The design of agents that exploit human irrationality raises questions about market manipulation and fairness. However, understanding and modeling biases may also help develop more robust systems that can operate effectively in markets dominated by biased agents. [insert here]
All experimental results are visualized in the following figures, which provide comprehensive visualizations of the findings.
Reinforcement learning applications in finance have focused primarily on optimizing trading strategies under the assumption of rational agents pricope2021deeprl, xu2023deeprl, vetrin2024reinforcementli. These studies demonstrate the potential of RL for algorithmic trading but largely ignore the psychological factors that influence real market participants. Our study extends the literature by explicitly incorporating cognitive biases into the RL framework.
Behavioral finance has extensively documented cognitive biases in financial decision-making kara2025thero, kanapickiene2024acr, wang2023theio, establishing that human traders systematically deviate from rationality in their decisions. However, most behavioral finance research studies static decision scenarios rather than sequential decision making under uncertainty. Our study bridges this gap by adapting behavioral insights to dynamic RL environments.
Recent studies have begun incorporating human-like behaviors into RL, primarily in non-financial domains. hong2023learningti use offline RL to learn policies that influence human behavior, while zhong2025agentstt demonstrate that RL-trained agents can exhibit human-like decision-making flexibility. banerjee2025estimatingcb develop methods for estimating cognitive biases using attention-aware inverse planning, which could inform bias parameterization in future work.
In finance-specific contexts, hu2025aidrivenam used agent-based modeling with RL to simulate investor behavior but focused on market-level emergent properties rather than individual agent optimization. cheridito2025abidesmarlam develop multi-agent RL environments for limit order books, providing infrastructure for studying market microstructure but not explicitly modeling biases.
Our study uniquely combines RL optimization with explicit cognitive bias modeling for individual trading agents, providing a systematic empirical investigation of the challenges and opportunities of this integration. Our negative results complement the literature by identifying critical pitfalls and guiding future research.
This study provides a comprehensive empirical investigation of the incorporation of cognitive biases into reinforcement learning for financial decision-making. Our experiments systematically explored hyperparameter tuning, bias parameter ablation, and architectural variations, revealing significant challenges in achieving the hypothesized benefits.
The key findings include the following: (1) simple bias incorporation via reward scaling is insufficient; (2) training instability is a major obstacle; (3) initial conditions significantly impact outcomes; and (4) no configuration achieves consistent profitability in our random walk environment. These negative results are valuable contributions that help identify critical pitfalls and guide future research directions.
Our study establishes several promising directions for future research. First, more sophisticated bias modeling that incorporates reference points, framing effects, and prospect-theoretic value functions may better capture human psychology. Second, advanced RL algorithms (deep Q-networks and actor-critic methods) may handle the increased complexity introduced by bias modeling. Third, testing on real market data or more sophisticated synthetic environments may reveal benefits that are not apparent in random walk markets. Fourth, multi-agent settings with heterogeneous biased agents may create market dynamics in which bias incorporation becomes beneficial.
This study contributes to the growing literature at the intersection of behavioral finance and machine learning by providing empirical insights and theoretical foundations for future research. While incorporating cognitive biases into RL trading systems presents significant challenges, understanding these challenges is crucial for developing robust and realistic automated trading systems that can operate effectively in markets dominated by human participants.