The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
31,053 characters
Incorporating Cognitive Biases into Reinforcement Learning for Financial Decision-Making
\maketitle
\begin{abstract}
Financial markets are influenced by human behavior that deviates from rationality due to cognitive biases. Traditional reinforcement learning (RL) models for financial decision-making assume rational agents, potentially overlooking the impact of psychological factors. This study integrates cognitive biases into RL frameworks for financial trading, hypothesizing that such models can exhibit human-like trading behavior and achieve better risk-adjusted returns than standard RL agents. We introduce biases, such as overconfidence and loss aversion, into reward structures and decision-making processes and evaluate their performance in simulated and real-world trading environments. Despite its inconclusive or negative results, this study provides insights into the challenges of incorporating human-like biases into RL, offering valuable lessons for developing robust financial AI systems.
\end{abstract}
\section{Introduction}
Financial markets are complex adaptive systems in which human decision-making plays a central role. Traditional economic theory assumes rational agents who maximize expected utility; however, empirical evidence consistently demonstrates that real-world financial decisions deviate significantly from this rational model \citep{kara2025thero, kanapickiene2024acr}. Cognitive biases, such as loss aversion, overconfidence, and anchoring, systematically influence investor behavior, creating observable patterns in market dynamics that cannot be explained by purely rational models.
Reinforcement learning (RL) has emerged as a powerful framework for automated trading systems, capable of learning optimal strategies from market data without explicit programming \citep{pricope2021deeprl, xu2023deeprl}. However, most RL-based trading agents assume rational decision-making, potentially overlooking the psychological factors that drive real market participation. This disconnect raises a fundamental question: can incorporating cognitive biases into RL frameworks lead to more realistic and potentially more robust financial decision-making systems?
This study explores the integration of cognitive biases into reinforcement learning for financial decision-making. We propose a framework that explicitly models behavioral biases, particularly loss aversion and overconfidence, within a Q-learning trading agent. Our approach modifies the reward structures and action-selection mechanisms to reflect the psychological phenomena observed in human traders. We conducted comprehensive experiments, including hyperparameter tuning, ablation studies on bias parameters, and analysis of training dynamics.
Our experimental results reveal significant challenges in achieving the hypothesized benefits of this approach. Although some configurations demonstrate interesting behavioral patterns, most fail to outperform rational agents at the baseline. These findings provide valuable insights into the complexities of bridging behavioral finance and machine learning, highlighting critical pitfalls and providing guidance for future research directions. This study serves as both a cautionary tale and a foundation for developing more nuanced approaches to incorporating human psychology into algorithmic trading systems.
\section{Background}
\label{sec:background}
\subsection{Reinforcement Learning for Financial Trading}
Reinforcement learning formulates trading as a sequential decision-making problem in which an agent interacts with a financial market environment. At each time step $t$, the agent observes a state $s_t \in \mathcal{S}$ representing market conditions, selects an action $a_t \in \mathcal{A}$ (e.g., buy, sell, or hold), receives a reward $r_t$, and transitions to a new state $s_{t+1}$. The goal is to learn a policy $\pi: \mathcal{S} \rightarrow \mathcal{A}$ that maximizes the cumulative expected return.
In Q-learning, we learn an action-value function $Q(s, a)$ that represents the expected return of taking action $a$ in state $s$ and following the optimal policy thereafter. The Q-function is updated using temporal difference learning as follows:
\begin{equation}
Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha \left[ r_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t) \right]
\end{equation}
where $\alpha \in (0,1]$ is the learning rate and $\gamma \in [0,1)$ is the discount factor.
For financial applications, the state space $\mathcal{S}$ typically includes price features, technical indicators, or discretized market conditions. Actions commonly represent trading operations (buy, sell, and hold), and rewards reflect portfolio performance metrics, such as returns or risk-adjusted measures.
Previous work on RL-based trading has focused primarily on optimizing Sharpe ratios \citep{pricope2021deeprl} or maximizing cumulative returns \citep{xu2023deeprl}, typically assuming rational agents seeking optimal strategies to maximize their returns. However, real financial markets are populated by human traders whose behavior systematically deviates from rationality.
\subsection{Cognitive Biases in Financial Decision-Making}
Behavioral finance research has identified numerous cognitive biases that systematically influence financial decisions \citep{kara2025thero, kanapickiene2024acr, wang2023theio}. We focus on two key biases relevant to trading.
\textbf{Loss Aversion:} Loss aversion, a central concept in prospect theory \citep{turgay2025improvingdm}, describes the tendency for losses to be perceived as more significant than equivalent gains. Empirically, the psychological impact of a loss is approximately twice that of an equivalent gain, leading to risk-averse behavior when facing potential losses and risk-seeking behavior when facing certain losses.
\textbf{Overconfidence:} Overconfidence bias manifests as traders overestimating their ability to predict market movements, leading to excessive trading frequency and underestimation of risk. After a sequence of successful trades, overconfident traders may increase their risk tolerance, believing that their success validates their predictive abilities.
These biases create observable patterns in market behavior, including momentum effects, mean reversion, and volatility clustering. De Bondt and ( Understanding these phenomena is crucial for developing robust trading systems that can operate effectively in real markets dominated by human participants.
\subsection{Integration Challenges}
Incorporating cognitive biases into RL frameworks presents several theoretical and practical challenges. First, biases are typically studied through static decision scenarios, whereas trading requires sequential decision-making under uncertainty. Second, the optimal method for parameterizing biases within RL reward structures is not well established. Third, bias incorporation may introduce training instability and lead to suboptimal policies. Our study empirically explores these challenges through systematic experimentation.
\section{Method}
\label{sec:method}
\subsection{Problem Formulation}
We formulate financial trading as a Markov Decision Process (MDP) $\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$
\begin{itemize}
\item $\mathcal{S}$ is a discrete state space obtained by discretizing price data into $n$ states
\item $\mathcal{A} = \{\text{buy}, \text{sell}, \text{hold}\}$ is the action space
\item $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0,1]$ is the state transition probability
\item $\mathcal{R}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ is a reward function incorporating cognitive biases
\item $\gamma \in [0,1)$ is the discount factor
\end{itemize}
The agent maintains a portfolio with a cash balance $b_t$ and asset holdings $\pi_t$ at time $t$. Given the current price $p_t$, the portfolio value is $v_t = b_t + \pi_t \cdot p_t$.
\subsection{Loss Aversion Integration}
We incorporate loss aversion by modifying the reward function to asymmetrically penalize losses, relative to gains. The base reward is the change in the portfolio value:
\begin{equation}
r_{\text{base}}(s_t, a_t) = v_{t+1} - v_t
\end{equation}
The loss-averse reward function amplifies the negative rewards by a factor $\lambda \geq 1$:
\begin{equation}
r_{\text{LA}}(s_t, a_t) = \begin{cases}
r_{\text{base}}(s_t, a_t) & \text{if } r_{\text{base}} \geq 0 \\
\lambda \cdot r_{\text{base}}(s_t, a_t) & \text{if } r_{\text{base}} < 0
\end{cases}
\end{equation}
This formulation reflects the empirical finding that losses are psychologically weighted approximately twice as heavily as gains ($\lambda \approx 2$) \citep{turgay2025improvingdm}. In our ablation studies, we explore $\lambda \in \{1, 1.5, 2, 2.5, 3\}$ to examine the sensitivity to the loss aversion parameter.
\subsection{Overconfidence Modeling}
We model overconfidence through dynamic risk adjustment based on the recent performance. Let $\bar{r}_k$ denote the average reward over the past $k$ time steps. The exploration rate $\epsilon$ is adjusted as follows:
\begin{equation}
\epsilon_{\text{overconf}}(t) = \epsilon_0 \cdot \exp(-\beta \cdot \max(0, \bar{r}_k))
\end{equation}
where $\epsilon_0$ is the base exploration rate, and $\beta > 0$ controls the sensitivity to recent performance. This formulation decreases exploration (increases exploitation) after successful periods, modeling the overconfident trader's reduced perceived uncertainty.
\subsection{Q-Learning with Biased Rewards}
We employ tabular Q-learning with $\epsilon$-greedy exploration to achieve this. The Q-table is initialized to $Q(s, a) = 0$ for all $(s, a)$ pairs. During training, actions are selected according to the following:
\begin{equation}
a_t = \begin{cases}
\text{random action} & \text{with probability } \epsilon(t) \\
\arg\max_a Q(s_t, a) & \text{otherwise}
\end{cases}
\end{equation}
The Q-function is updated using a modified reward as follows:
\begin{equation}
Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha \left[ r_{\text{biased}}(s_t, a_t) + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t) \right]
\end{equation}
where $r_{\text{biased}}$ incorporates loss aversion and potentially other bias modifications.
\subsection{State Space Discretization}
We discretized the continuous price data into $n$ states using equal-width binning. Given the price sequence $\{p_t\}_{t=1}^T$ with minimum $p_{\min}$ and maximum $p_{\max}$, we create $n-1$ bin
\begin{equation}
b_i = p_{\min} + \frac{i}{n-1}(p_{\max} - p_{\min}), \quad i = 1, \ldots, n-1
\end{equation}
The state at time $t$ is determined by $s_t = \text{digitize}(p_t, \{b_i\})$, where digitize returns the bin index containing $p_t$. We explore $n \in \{5, 10, 15, 20\}$ to understand the impact of state-space granularity on learning dynamics.
\subsection{Evaluation Metrics}
We evaluated the performance using two primary metrics:
\textbf{Sharpe Ratio:} The risk-adjusted return measure:
\begin{equation}
\text{SR} = \frac{\mathbb{E}[r] - r_f}{\sigma(r)}
\end{equation}
where $r$ is the return sequence, $r_f$ is the risk-free rate (assumed to be zero), and $\sigma(r)$ is the standard deviation of returns.
\textbf{Cumulative Returns:} The total portfolio return over the evaluation period:
\begin{equation}
R_{\text{cum}} = \sum_{t=1}^T r_t
\end{equation}
Higher Sharpe ratios indicate better risk-adjusted performance, whereas cumulative returns measure absolute profitability.
\section{Experimental Setup}
\label{sec:experimental_setup}
\subsection{Dataset}
We generated synthetic financial data using a random walk model:
\begin{equation}
p_{t+1} = p_t + \epsilon_t, \quad \epsilon_t \sim \mathcal{N}(0, \sigma^2)
\end{equation}
where $p_0 = 100$ and $\sigma = 1$. This generates $T = 200$ daily price observations, simulating a realistic trading environment with inherent uncertainty. The synthetic approach allows for controlled experimentation while avoiding market microstructure effects and data snooping biases present in real financial data.
\subsection{Baseline Configuration}
Our baseline Q-learning agent uses a standard rational reward structure ($\lambda = 1$), fixed exploration rate $\epsilon = 0.1$, learning rate $\alpha = 0.1$, and discount factor $\gamma = 0.9$. The baseline served as a comparison point to assess the impact of bias incorporation.
\subsection{Experimental Design}
We conducted three categories of experiments.
\textbf{Hyperparameter Tuning:} We systematically vary the state space size $n \in \{5, 10, 15, 20\}$ while keeping other parameters fixed, training for 50 epochs per configuration.
\textbf{Bias Parameter Ablation:} We explore loss aversion multipliers $\lambda \in \{1, 1.5, 2, 2.5, 3\}$ and discount factors $\gamma \in \{0.5, 0.7, 0.9, 0.99\}$ to understand sensitivity to bias strength and temporal discounting.
\textbf{Architecture Ablations:} We investigate the impact of action space reduction (removing "hold"), Q-table initialization strategies (zero, random, small positive), initial portfolio values, reward scaling for positive gains, and temporal continuity in reward calculation.
All experiments used fixed random seeds for reproducibility and reported results averaged over multiple runs, where applicable.
\section{Results}
\label{sec:results}
\subsection{Hyperparameter Tuning Results}
Figure~\ref{fig:hyperparam_tuning} presents the hyperparameter tuning results for state space discretization. Across all configurations ($n \in \{5, 10, 15, 20\}$), we observe consistently negative cumulative returns, indicating that none of the tested configurations achieved a profitable trading strategy. The rate of decline varies: configurations with $n=5$ and $n=20$ show steeper declines than those with $n=10$ and $n=15$, suggesting that both overly coarse and overly fine discretizations may impair learning.
The Sharpe ratio evolution (Figure~\ref{fig:hyperparam_tuning}, left panel) exhibits high variability across epochs for all the configurations. None consistently maintained positive Sharpe ratios, indicating poor risk-adjusted returns. The variability suggests training instability, potentially due to the challenging nature of the synthetic random walk environment, where profitable strategies are inherently difficult to discover.
These results highlight a fundamental challenge: even without bias incorporation, finding profitable strategies in a random walk market is nontrivial. The efficient market hypothesis suggests that in truly random markets, no strategy should consistently outperform, and our results are consistent with this expectation.
\subsection{Loss Aversion Effects}
Figure~\ref{fig:loss_aversion} shows the impact of varying the loss aversion multipliers. Surprisingly, moderate loss aversion ($\lambda = 1.5, 2.0$) does not consistently improve performance compared to the baseline ($\lambda = 1$). Higher multipliers ($\lambda = 2.5, 3.0$) lead to worse outcomes, with cumulative returns declining more rapidly.
The Sharpe ratio trajectories reveal high volatility, regardless of the multiplier value. This suggests that simply amplifying negative rewards may not capture the nuanced effects of loss aversion on sequential decision-making. The increased penalty for losses may lead to overly conservative policies that avoid trading altogether, or conversely, may trigger risk-seeking behavior when facing certain losses, a phenomenon consistent with prospect theory but challenging to model in RL.
These findings indicate that incorporating naive loss aversion via reward scaling may be insufficient. More sophisticated modeling, perhaps incorporating reference points or framing effects, may be necessary to capture the psychological mechanisms underlying loss-averse behaviors.
\subsection{Discount Factor Ablation}
Figure~\ref{fig:gamma_ablation} examines the effect of varying the discount factor $\gamma$. Higher values ($\gamma = 0.9, 0.99$) give more weight to future rewards, whereas lower values ($\gamma = 0.5, 0.7$) emphasize immediate returns. Our results show no clear superiority of any particular discount factor, with all configurations exhibiting similar and volatile performances.
This suggests that in our relatively short-horizon trading environment (200 time steps), the temporal structure of the rewards may not be the primary limiting factor. The role of the discount factor may be more critical in longer-horizon scenarios or when learning value functions that depend on strategic long-term positioning.
\subsection{Action Space Reduction}
Figure~\ref{fig:action_space_reduction} compares the performance with and without the "hold" action. Removing "hold" forces the agent to always take a directional position (buy or sell), which may align better with human trading behavior, where pure cash positions are uncommon. The results show mixed outcomes: some state space configurations (particularly $n=20$) achieve slightly higher Sharpe ratios with a reduced action space, whereas others perform worse.
This suggests that the action space design significantly impacts the learning dynamics; however, the optimal configuration depends on the state representation. The relationship between state granularity and action-space complexity requires further investigation.
\subsection{Reward Structure Variants}
Figure~\ref{fig:reward_structure} compares different reward formulations: proportional gains (normalized by the initial balance) versus volatility-penalized rewards. Proportional gains show more stable but near-zero returns, whereas volatility-penalized rewards exhibit higher volatility with consistently negative outcomes. This indicates that the specific form of reward shaping strongly influences the learning dynamics, and careful design is necessary to achieve the desired behaviors.
\subsection{Q-Table Initialization}
Figure~\ref{fig:qtable_initialization} examines initialization strategies: zero, random (uniform $[0,1]$), and small positive ($0.1$). All strategies showed similar volatile performance with no clear winner, suggesting that initialization has a limited impact in our setting. This may be because the Q-table quickly overwrites the initial values during training or because the exploration strategy ($\epsilon$-greedy) ensures sufficient state-action coverage regardless of initialization.
\subsection{Portfolio Initialization}
Figure~\ref{fig:portfolio_initialization} presents an interesting finding: agents initialized with positive portfolio holdings ($\pi_0 \in \{5, 10\}$) significantly outperform those starting from zero holdings ($\pi_0 = 0$). This suggests that having an initial position may help bootstrap learning by providing immediate feedback on portfolio value changes rather than requiring the agent to first learn to take positions.
This finding has practical implications: pre-training agents with initial positions or using transfer learning from pre-positioned agents may improve the training efficiency in RL trading applications.
\subsection{Reward Scaling and Temporal Continuity}
Figures~\ref{fig:reward_scaling} and~\ref{fig:reward_temporal_continuity} explore advanced reward modifications. Scaling positive rewards (Figure~\ref{fig:reward_scaling}) showed variability patterns similar to those of other experiments, with no clear benefit. Temporal continuity smoothing (Figure~\ref{fig:reward_temporal_continuity}) demonstrates that intermediate state complexities ($n=15$) achieve better performance than extreme values but still fail to consistently achieve profitability.
\subsection{Summary and Insights}
Our comprehensive experimental analysis revealed several critical insights.
\begin{enumerate}
\item \textbf{Bias incorporation alone is insufficient:} Simply modifying reward structures to reflect cognitive biases does not guarantee improved performance. The interaction between bias modeling and learning dynamics is complex and nontrivial.
\item \textbf{Training instability is a major challenge:} High variance in Sharpe ratios and returns across epochs suggests that bias incorporation may amplify existing training instabilities. More sophisticated learning algorithms or regularization techniques may be required.
\item \textbf{Environment matters:} The random walk market may be too challenging or too simple to reveal benefits of bias incorporation. Testing on real market data or more sophisticated synthetic environments may yield different results.
\item \textbf{Initial conditions matter:} Portfolio initialization significantly impacts outcomes, suggesting that careful attention to training setup is crucial.
\item \textbf{No configuration achieves consistent profitability:} This aligns with theoretical expectations for random walk markets, but also suggests that more sophisticated bias modeling may be necessary to achieve hypothesized benefits.
\end{enumerate}
These findings contribute valuable negative results to the literature, helping guide future research toward more promising directions for integrating behavioral finance and reinforcement learning.
\subsection{Limitations and Ethical Considerations}
Our experimental results highlight several limitations of this approach. First, the synthetic random walk environment may not capture the nuanced dynamics of real financial markets, where biases may have different effects. Second, our simple tabular Q-learning approach may be insufficient to model the complexity of cognitive biases, which often involve reference points, framing effects, and context-dependent evaluations \citep{turgay2025improvingdm}.
Third, training instability remains a significant challenge, suggesting that more sophisticated learning algorithms (e.g., actor-critic methods and policy gradient algorithms) or regularization techniques may be necessary. Fourth, our single-asset setting ignores portfolio effects and diversification benefits that may interact with bias modeling.
Ethical considerations also arise from incorporating biases into trading systems \citep{afjal2024evolvingtl, garg2025artificialii}. The design of agents that exploit human irrationality raises questions about market manipulation and fairness. However, understanding and modeling biases may also help develop more robust systems that can operate effectively in markets dominated by biased agents. [insert here]
\subsection{Figures}
All experimental results are visualized in the following figures, which provide comprehensive visualizations of the findings.
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure1_hyperparam_tuning.png}
\caption{Hyperparameter tuning results for state space discretization ($n \in \{5, 10, 15, 20\}$). {\em (Left)}{} Sharpe Ratio evolution over epochs. {\em (Right)}{} Cumulative Returns over training epochs. All configurations show negative cumulative returns with high variability, indicating difficulty in learning profitable strategies.}
\label{fig:hyperparam_tuning}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure2_loss_aversion.png}
\caption{Loss aversion ablation study with multipliers $\lambda \in \{1, 1.5, 2, 2.5, 3\}$. {\em (Left)}{} Sharpe Ratio trajectories over epochs. {\em (Right)}{} Cumulative Returns evolution. Higher multipliers ($\lambda \geq 2.5$) show worse performance, suggesting that excessive loss aversion amplification degrades learning dynamics.}
\label{fig:loss_aversion}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure3_gamma_ablation.png}
\caption{Discount factor ablation with $\gamma \in \{0.5, 0.7, 0.9, 0.99\}$. {\em (Left)}{} Sharpe Ratios over epochs. {\em (Right)}{} Cumulative Returns. No clear superiority of any discount factor, with all configurations exhibiting similar volatile performance patterns.}
\label{fig:gamma_ablation}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure4_action_space_reduction.png}
\caption{Action space reduction study (removing "hold" action). {\em (Left)}{} Sharpe Ratios for different state space sizes. {\em (Right)}{} Returns over epochs. Mixed outcomes suggest that action space design significantly impacts learning, with optimal configuration depending on state representation.}
\label{fig:action_space_reduction}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure5_reward_structure.png}
\caption{Reward structure variants comparison. {\em (Left)}{} Sharpe Ratio comparison. {\em (Right)}{} Cumulative Returns. Proportional gains (Reward Struct 1) show stable but near-zero returns, while volatility-penalized rewards (Reward Struct 2) exhibit high volatility with negative outcomes.}
\label{fig:reward_structure}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure6_qtable_initialization.png}
\caption{Q-table initialization strategies (zeros, random, small positive). {\em (Left)}{} Sharpe Ratios during training. {\em (Right)}{} Returns over epochs. All initialization strategies show similar volatile performance, suggesting limited impact of initialization in this setting.}
\label{fig:qtable_initialization}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure7_portfolio_initialization.png}
\caption{Portfolio initialization ablation with initial holdings $\pi_0 \in \{0, 5, 10\}$. {\em (Left)}{} Sharpe Ratios showing improved performance for positive initial portfolios. {\em (Right)}{} Cumulative Returns demonstrating significant benefits of initial positioning. Agents with $\pi_0 > 0$ achieve positive Sharpe ratios and cumulative returns, validating the importance of initial conditions.}
\label{fig:portfolio_initialization}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure8_reward_scaling.png}
\caption{Reward scaling for positive gains (scaling factor 1.5) across different state space sizes. {\em (Left)}{} Sharpe Ratios over epochs. {\em (Right)}{} Cumulative Rewards. Similar variability patterns as other experiments, with no clear benefit from positive reward scaling.}
\label{fig:reward_scaling}
\end{figure}
\begin{figure}[h!]
\centering
\includegraphics[width=0.9\textwidth]{figures/figure9_reward_temporal_continuity.png}
\caption{Temporal continuity in reward calculation with smoothing factor. {\em (Left)}{} Sharpe Ratios across epochs for different state space sizes. {\em (Right)}{} Cumulative Returns. Intermediate state complexities ($n=15$) achieve relatively better performance, though all configurations still show significant volatility.}
\label{fig:reward_temporal_continuity}
\end{figure}
\section{Related Work}
\label{sec:related_work}
Reinforcement learning applications in finance have focused primarily on optimizing trading strategies under the assumption of rational agents \citep{pricope2021deeprl, xu2023deeprl, vetrin2024reinforcementli}. These studies demonstrate the potential of RL for algorithmic trading but largely ignore the psychological factors that influence real market participants. Our study extends the literature by explicitly incorporating cognitive biases into the RL framework.
Behavioral finance has extensively documented cognitive biases in financial decision-making \citep{kara2025thero, kanapickiene2024acr, wang2023theio}, establishing that human traders systematically deviate from rationality in their decisions. However, most behavioral finance research studies static decision scenarios rather than sequential decision making under uncertainty. Our study bridges this gap by adapting behavioral insights to dynamic RL environments.
Recent studies have begun incorporating human-like behaviors into RL, primarily in non-financial domains. \citet{hong2023learningti} use offline RL to learn policies that influence human behavior, while \citet{zhong2025agentstt} demonstrate that RL-trained agents can exhibit human-like decision-making flexibility. \citet{banerjee2025estimatingcb} develop methods for estimating cognitive biases using attention-aware inverse planning, which could inform bias parameterization in future work.
In finance-specific contexts, \citet{hu2025aidrivenam} used agent-based modeling with RL to simulate investor behavior but focused on market-level emergent properties rather than individual agent optimization. \citet{cheridito2025abidesmarlam} develop multi-agent RL environments for limit order books, providing infrastructure for studying market microstructure but not explicitly modeling biases.
Our study uniquely combines RL optimization with explicit cognitive bias modeling for individual trading agents, providing a systematic empirical investigation of the challenges and opportunities of this integration. Our negative results complement the literature by identifying critical pitfalls and guiding future research.
\section{Conclusion}
\label{sec:conclusion}
This study provides a comprehensive empirical investigation of the incorporation of cognitive biases into reinforcement learning for financial decision-making. Our experiments systematically explored hyperparameter tuning, bias parameter ablation, and architectural variations, revealing significant challenges in achieving the hypothesized benefits.
The key findings include the following: (1) simple bias incorporation via reward scaling is insufficient; (2) training instability is a major obstacle; (3) initial conditions significantly impact outcomes; and (4) no configuration achieves consistent profitability in our random walk environment. These negative results are valuable contributions that help identify critical pitfalls and guide future research directions.
Our study establishes several promising directions for future research. First, more sophisticated bias modeling that incorporates reference points, framing effects, and prospect-theoretic value functions may better capture human psychology. Second, advanced RL algorithms (deep Q-networks and actor-critic methods) may handle the increased complexity introduced by bias modeling. Third, testing on real market data or more sophisticated synthetic environments may reveal benefits that are not apparent in random walk markets. Fourth, multi-agent settings with heterogeneous biased agents may create market dynamics in which bias incorporation becomes beneficial.
This study contributes to the growing literature at the intersection of behavioral finance and machine learning by providing empirical insights and theoretical foundations for future research. While incorporating cognitive biases into RL trading systems presents significant challenges, understanding these challenges is crucial for developing robust and realistic automated trading systems that can operate effectively in markets dominated by human participants.
\bibliography{iclr2025}
\bibliographystyle{iclr2025}