Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
79,457 characters · 22 sections · 63 citation commands
Bayesian Bilinear Neural Network for Predicting the Mid-price Dynamics in Limit-Order Book Markets
Bayesian inference is known to be a difficult task outside a relatively small class of well-studied models, generally involving conjugate priors for the likelihood. The analytical Bayesian treatment of general, even small-dimensional, problems is widely unfeasible. The increased computational capacity available these days, as much as the availability of powerful algorithms such as Monte Carlo Markov Chain or Metropolis-Hastings, opened the possibility for a simulation-based approach to Bayesian inference. However, Bayesian methods in typical large-scale complex Machine Learning (ML) problems have long been impractical. Though ML generally operates under a frequentist perspective, the first steps into a probabilistic approach to Deep Learning (DL) are relatively recent, see e.g., murphy_machine_2012,gal_dropout_2016 and references therein. Only recently, we are witnessing a growing interest in Bayesian DL, boosted by its demand across multiple disciplines. Indeed, even in a simulation-aided setting, Bayesian inference on the potentially thousands of parameters over highly non-linear models like Neural Networks (NN) is certainly not a simple task.
Yet, the interest in probabilistic modeling and Bayesian methods in other disciplines has a much longer history. Especially in econometrics and finance, the probabilistic dimension is an innate and essential element in modeling. Indeed econometric research is at the cross-edge between applied statistics, probability theory, stochastics, and the study of economic phenomena frisch1933. As such, the econometric practice is that of developing well-reasoned and economically motivated, essential, mostly parametric, probabilistic models that are thoroughly tested, validated and back-tested following the principles of statistical inference. For example, the concepts of significance testing, confidence intervals, asymptotic analysis, and stationarity are typical in the econometric literature. On the other hand, such an approach in DL is currently inapplicable.
At the same time, researchers in economics and practitioners in finance acknowledge the flexibility, scalability, and gains in predictive tasks that ML can bring when applied to economic problems, e.g., mullainathan2017machine,dixon2020machine. Especially in modern, electronically-driven financial markets, operating at ultra-high frequencies and generating massive complex and multidimensional datasets underlying the complex dynamics of market variables arising from the interactions of multiple players and forces at different levels, ML methods have gained much attention, see, e.g., varian2014big. Business and financial applications are part of those high-risk domains where quantifying the uncertainty underlying models' estimates and predictions is of utmost importance salinas2020deepar. A probabilistic dimension reflecting uncertainties related to model estimation and perhaps accounting for the typical elements of business activity that are difficult to predict makridakis2009forecasting, would be beneficial.
The recent advances in Bayesian DL have the potential of bringing this element into play, narrowing the gap between the highly-probabilistic yet parsimonious modeling of the econometric practice and the flexible non-linear and non-parametric ML rationale. Bayesian inference for neural networks has recently been shown to be challenging yet feasible kingma_auto_2014,osawa_practical_2019,blundell_weight_2015. Bayesian Neural Networks (BNNs) are engaged with the typical elements of Bayesian inference, in particular with a trainable distribution over its parameters and a consequent predictive distribution that enables classical statistical tools, econometric methods, and relevant risk-related and uncertainly-related analyses, e.g., based on predictive distributions geweke2010comparing. Nevertheless, much research in this direction is still needed.
This paper aims to introduce the use of BNNs in economic problems in light of the above discussion, boosting further research and interest in this research direction. We propose a Bayesian version of the Temporal Attention-augmented Bilinear network as a lightweight DL model for a financial times-series classification task. We propose a first Bayesian DL econometric application in the challenging task of predicting mid-price movements in Limit Order Book (LOB) markets. Our results explore the feasibility of such an approach, compare its forecasting performance against non-Bayesian specifications based on different optimization algorithms, and address the advantages of adopting BNNs in financial applications.
Neural Networks (NNs) have been successfully applied in several ML problems, such as image classification krizhevsky_imagenet_2012,simonyan_very_2015 computer vision girshick_rich_2014,ren_faster_2015, natural language processing collobert_unified_2008,goldberg_neural_2017 or speech recognition mohamed_acoustic_2012,dahl_context-dependent_2012. Despite their undeniable use and performance in delivering leading results on different predictive tasks, the decisions are achieved in a rather uninterpretable manner.
NNs correspond to statistical black-box models that achieve feasible point estimate predictions by adapting their high-dimensional parameters on a try-and-error basis. Based on the nature of the problem, the user defines a cost function and a network architecture that allows to approximate complex non-linear functions to tackle a prediction or classification task. By allowing for a sufficiently large number of trainable parameters, under mild assumptions, NNs can approximate any arbitrary function cybenko_approximation_1989,Lu2017expressive,Hanin2019universal. It is with little surprise that NNs found significant use in financial and econometrics applications, where complex and interacting latent structures in the data drive the behavior of different economic variables. A review of different NN applications in finance is provided in mcnelis_neural_2005, an early discussion on econometric applications can be found in kuan_artificial_1994 and within time-series analysis in terasvirta_linear_2005,m_qi_trend_2008,hewamalage2021recurrent. cenesizoglu_effects_2014 analyzed the relationship between LOB variables and mid-price movements showing that it is possible to obtain economical gain from these variables and the mid-price return. Further, their causality analysis supports the use of lagged LOB variables for forecasting purposes. The design of a set of features, extending the LOB feature set in the high-frequency forecasting application of kercheval_modelling_2015, can be found in ntakaris_feature_2019, while ntakaris_benchmark_2018,ntakaris_feature_2019 tackle the mid-price movement prediction with the rich LOB data under different ML perspectives, including NNs. For a similar prediction task, dixon_sequence_2018 addressed the use of recurrent NN. The use of Long-Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNNs) is discussed in tsantekidis_forecasting_2017,zhang2019deeplob,passalis2019adaptive,tsantekidis_using_2020 and the use of Neural Bag of Features in passalis2017time,passalis2018temporal. Taking advantage of the spatial structure in the LOB, sirignano_deep_2019 provides an extensive analysis of over 500 stocks for prediction price movements, while tran2017tensor encodes the LOB data as two-order tensors. An attention mechanism capable of exploiting and retaining the temporal mode of the order flow is introduced in tran_temporal_2019 and extended to accommodate multiple attentions in shabani_multi-head_2022.
By fine-tuning the network and increasing the number of parameters, one possibly achieves functions with higher complexity and improved forecasting ability. The lack of interpretation and the impossibility of condensing the decision process to a simple decision rule, along with overfitting issues and their native non-probabilistic setup, create challenges in the use of NNs in high-risk domains and for all those applications where uncertainties in predictions are of relevance goan_bayesian_2020. Moreover, the lack of a well-defined building protocol (e.g., the absence of an Akaike Information Criterion -like statistic for features' relevance determination) makes their adoption by experts and practitioners from such domains difficult caruana_intelligible_2015,holzinger_what_2017,vu_shared_2018,holzinger_causability_2019.
A Bayesian perspective on NNs provides a natural way to reason around uncertainties. At the same time, it provides tools for model regularization and offers insights into how decisions are made. Indeed, the Bayesian paradigm offers a perspective on NNs that can address many of the issues currently faced by NNs. Recent research investigated how Bayesian principles can adapt to large NNs. To this end, a learnable distribution is placed over the parameters, resulting in BNNs. A survey on early developments in BNNs can be found in mackay_probable_1995, and recent introductions to BNNs are those of, e.g.,lampinen_bayesian_2001,jospin_hands_2020,goan_bayesian_2020. For a specialized survey on algorithms for training BNNs see magris2023. In BNNs, parameters are treated as random variables, and the learning focuses on the distribution of these parameters conditional on the observed training data sample. In the learning phase, the latent distribution of the parameters is inferred based on the current knowledge and the observed data by use of the Bayes theorem resulting in a distribution of the model parameters conditional on the data, the posterior distribution. Further details on BNNs and their training appear in Section (ref).
Financial applications involving BNNs are somewhat limited. An application for automatic relevance determination in option pricing is that of mbuvha_automatic_2019. A recent example of stock-price prediction is found in chandra_bayesian_2021, where exploitative MCMC-based learning is used to forecast daily closing prices of four stocks, showing that in terms of RMSE performance metric, their BNN outperforms non-Bayesian counterparts. A forecasting study based on electricity prices is provided in vahidinasab_bayesian_2008,ghayekhloo_combination_2019, while Bitcoin data is used in jang_empirical_2018. Sign-changes in returns have been analyzed under a Multilayer Perceptron (MLP) BNN in skabar_direction-change_2009. This study uses low-frequency daily closing prices and lagged moving averages as features, showing a slight 52% accuracy over a random classifier and no gains with respect to a standard MLP. There are, however, no applications involving tick-by-tick data generated from typical modern financial markets running over the LOB systems. Nevertheless, the bridging potential that BNNs could provide between the fields of econometrics and ML has not been recognized.
A BNN is any stochastic Artificial Neural Network (ANN) trained using Bayesian inference. ANNs aim at approximating arbitrary functions $\bm{y} \,{=}\, NN_{\theta}\autobracket*{\bm{x}}$, whose parameters are denoted by $\theta$. Over a training dataset $\mathcal{D}$, the standard estimation approach is to determine a minimal-cost point estimate $\hat{\theta}$ using backpropagation. In BNNs, parameters are treated as latent random variables, and the goal is to learn the distribution of the parameters conditional on $\mathcal{D}$. The first step is that of defining the joint distribution of the data and the parameters $p\autobracket*{\theta,\mathcal{D}}=p\autobracket*{\mathcal{D}|\theta}p\autobracket*{\theta}$, which depends on our prior belief over the latent variables $p\autobracket*{\theta}$ and the chosen form of likelihood $p\autobracket*{\mathcal{D}|\theta}$. Under independence between the model parameters and the inputs, the Bayesian posterior is written as $p\autobracket*{\theta|\mathcal{D}} = p\autobracket*{\mathcal{D}, \theta} p\autobracket*{\theta} / p\autobracket*{\mathcal{D}}$. Computing the weight-independent term known as marginal likelihood (or evidence) is perhaps one of the most difficult tasks in Bayesian inference: the prior-to-posterior update is usually intractable. From the posterior distribution, the model uncertainty is quantified as the marginal probability distribution of the output $\bm{y}_i$ for a certain input $\bm{x}_i$, through the predictive distribution
When performing classification, the average model prediction approximates the relative probability of each class,
where $\theta^{\autobracket*{n}} {\sim}\, p\autobracket*{\theta | \mathcal{D}}$. If the cost of giving a false positive is equal across all classes, the final classification is taken according to the most likely class, i.e.,
The Temporal Attention-augmented Bilinear Network (TABL) architecture tran_temporal_2019 is a lightweight DL model which has been shown to be particularly suited for multidimensional time-series forecasting. It augments the bilinear projection with an attention mechanism exploiting the temporal dimension across the features. This enables it to compare favorably with alternative architectures such as Bilinear networks, CNNs, LSTM networks, and several other ML algorithms tran_temporal_2019.
Figure (ref) illustrates the architecture of the TABL layer. It maps a $D\times T$-dimensional input matrix $\mathbf{X}$ onto a $D' \times T'$-dimensional output $\mathbf{Y}$, where $D$ and $D'$ correspond to the number of features and $T$, $T'$ correspond to the number of temporal instances. The network initially operates a projection of the temporal dimension of the input matrix to a $D'\times T$ -dimensional feature space modeling the dependence on the first mode while preserving the temporal order of the features. It further learns the relative importance of the temporal instances with respect to each other, producing an attention mask where only the most relevant instances are preserved. A learnable scalar drives the mixture of the temporal and non-temporal features passed to a final mapping that returns the final representation adjusted for bias. This is achieved by:
where $a_{ij}$ and $e_{ij}$ denote the element $(i,j)$ of $\mathbf{A}$ and $\mathbf{E}$, respectively, $\phi(\cdot)$ is a predefined activation function, $\mathbf{W}_1 \in \mathbb{R}^{D'\times D}$, $\mathbf{W} \in \mathbb{R}^{T\times T}$, $\mathbf{W}_2 \in \mathbb{R}^{T\times T'}$ and $\mathbf{B} \in \mathbb{R}^{D' \times T'}$ are the parameters of the layer, and ${0 \leq \lambda \leq 1}$ is a learnable mixing coefficient which determines the importance of using the temporal attention in the mapping. Experiments in shabani_multi-head_2022 further show that the inclusion of additional temporal attention heads opens toward richer structures in the temporal dependence across lagged features, relevant for forecasting purposes.
To formulate the Bayesian network formed by one TABL layer (B-TABL), we define the parameter vector $\bm{\theta}$ formed by the parameters of TABL, i.e., $\bm{\theta} = \{\mathbf{W}, \mathbf{W}_1, \mathbf{W}_2, \mathbf{B}, \lambda\}$. Variational Inference (VI) is a well-established methodology for complex statistical inference circumventing the typical intractable integration problem arising in Bayesian inference by approximating the true posterior $p\autobracket*{\bm{\theta} | \mathcal{D}}$ with a distribution $q\autobracket*{\bm{\theta}}$ whose normalization constant is easier to compute (variational distribution). A review on VI from a statistician's perspective is that of blei2017variational, from the ML perspective, that of tran_practical_2021, while very recent applications in multidimensional econometric models are, e.g., those of gunawan2021variational,gefang2022. Fixed form variational Bayes assumes a fixed parametric form for the density in some class of distributions $\mathcal{Q}$, indexed by a variational parameter vector. A perspective on the problem with general non-conjugate likelihoods for priors in the exponential family can be found in khan_2018_fast. We chose both $p\autobracket*{\bm{\theta}}$ and $q\autobracket*{\bm{\theta}}$ to be Gaussian distributions with diagonal covariance matrices:
where ${\alpha>0}$ is a known precision parameter and $\bm{\mu} \in \mathbb{R}^{P}$, $\bm{\sigma}^2 \in \mathbb{R}^{P}$. $P$ is the number of the parameters in the network, i.e., the number of parameters in $\bm{\theta}$. $q\autobracket*{\bm{\theta}}$ implies a factorization of the join in the product of the marginals, known as mean-field approximation. In VI, the variational parameters $\autobracket*{\bm{\mu},\bm{\sigma}^2}$ are obtained by maximizing the following objective:
Eq. (ref) can be maximized with the gradient-based optimization, that is, with the following update:
where $t$ is the iteration index, $\hat{\nabla}_{x} \mathcal{L}_t$ denotes an unbiased estimate of the gradient of $\mathcal{L}$ at $(\bm{\mu}_t,\bm{\sigma}^2_t)$ with respect to $x$, and $\rho_t,\delta_t$ are adaptable learning rates. The natural gradient VI method of khan_2017_conjugate tackles the update (ref) in terms of the natural parameter $\bm{\alpha}$ of $q\autobracket*{\bm{\theta}}$, rather than its mean and covariance matrix, and scales the gradient of the corresponding SGD update for $\bm{\alpha}_t$ with the inverse of the Fisher information matrix (FIM) of $q\autobracket*{\bm{\theta}}$. khan_2017_conjugate show that the direct computation of the FIM can be avoided by computing {\it natural} gradients in the natural parameter space using the gradient with respect to the expectation parameters of the exponential-family posterior. For the Gaussian mean-field VI under consideration, this leads to the natural-gradient variational inference (NGVI) update:
with ${\beta_t>0}$ being a scalar learning rate. By expressing (ref) in terms of the standard MLE objective $f\autobracket*{\bm{\theta}} = -1/N\sum_{i=1}^N \log p\autobracket*{\mathcal{D}_i|\bm{\theta}}$ and expressing the gradients of its expectation with respect to $\bm{\mu}$ and $\bm{\sigma}$ in terms of the gradient $g\autobracket*{\bm{\theta}}$ and Hessian $\mathbf{H}\autobracket*{\bm{\theta}}$ of $f\autobracket*{\bm{\theta}}$, the update results in:
where the division is element-wise, $\tilde{\alpha} {=} \alpha/N$ and $\bm{\theta}_t {\sim} \mathcal{N}\autobracket*{\bm{\theta}|\bm{\mu}_t,\text{diag}\autobracket*{\bm{\sigma}^2_t}}\bm{\sigma}^2_t}}$, with $\bm{\sigma}^2_t {=} [{N(\mathbf{s}_t +\tilde{\alpha})}]^{-1}$. The scaling vector $\mathbf{s}_t$ involves the gradient and the diagonal of the Hessian, which can be replaced by their stochastic estimates $\hat{g}\autobracket*{\bm{\theta}}$ and $\hat{\nabla}^2_{\bm{\theta} \bm{\theta}} f\autobracket*{\bm{\theta}}$. The former can be computed using backpropagation. Due to the general non-convexity of $f$, the latter can be negative, which might lead to negative variances. Non-negativity is granted by the following approximation:
with $i$ denoting the $i$-th data sample in the mini-batch $\mathcal{M}$ of size $M$, and $\theta_j$ the $j$-th element of $\bm{\theta}$. By writing for $\hat{\mathbf{h}}\autobracket*{\bm{\theta}_t}$ the vector of all $\hat{h}_j$, under this approximation the update for $\mathbf{s}_t$ reads:
The algorithm involving updates (ref) and (ref) is referred to as the Variational Online Gauss-Newton (VOGN) khan_2018_fast. Opposed to SGD and related algorithms such as RMSprop, Adam, and AdaGrad, which use the gradient magnitude $[\frac{1}{M} \textstyle\sum_{i\in \mathcal{M}}\hat{\nabla}_{\theta_j} f_i\autobracket*{\theta} ]^2$ for approximating the $j$-th entry of the diagonal Hessian, in (ref), VOGN uses averages of the squared gradients, avoiding explicit constraints on $\bm{\sigma}^2$. As shown in osawa_practical_2019, it leads to good empirical performance and practical feasibility of the updates (ref) and (ref) on large datasets compared to alternatives, e.g. Bayes by Backprop (BBB) blundell2015weight.
Existing automatic-differentiation libraries can be used to retrieve the gradients; however, current codebases directly return sums of the gradients over mini-batches, whereas individual gradients are required in (ref). Thus via chain rule, we derive the individual gradients for a TABL layer and adapt current second-order optimization routines osawa_2018_torch to accommodate them. Our B-TABL implementation adopts a log-softmax activation function at the last layer such that the output vector of the network interprets as logs of class-probabilities, with a one-to-one mapping between classes and indexes of vectors' elements. That is, for an input time-series $\mathbf{X}_i$, the corresponding output of the network is a vector
where $p_{ic} =\exp{\autobracket*{\bm{l}_i[c]}}/{\sum_c \exp\autobracket*{\bm{l}_i[c]}}$, $c$ being an index running along the number of network outputs (the number of classes $C$). $\bm{l}_i$ denotes the output of the last layer, corresponding to the input $\mathbf{X}_i$, passed to the softmax activation, and $\bm{l}_i[c]$ its $c$-th element. The loss used is the negative log-likelihood, i.e., for a sample in class $c$ the loss is computed as $-\log\bm{p}_i[c]$. We shall refer to $\bm{p}_i$ as class-probabilities or scores. Losses are averaged across samples for each mini-batch. For a trained model and for each input vector, predictions are provided by the class of the maximum log-score (or maximum class-probability), i.e., $\max_{c} \log \bm{p}_i[c]$. We shall refer to this criterion as the classification rule. Therefore, due to posterior sampling, log-scores are stochastic (as for the class-probabilities and the index of their maximum element), leading to stochastic class labels.
Trading in modern financial markets is organized through an order-driven mechanism that collects and matches inflowing limit and market orders through a time-priority rule. Trades participate in the market by submitting orders or cancellations over previously submitted orders. Each message (order or cancellation) submitted to the exchange comes with an associated timestamp, price, and quantity (along with a unique identifier). By submitting a limit order, a trader expresses his/her willingness to buy or sell a certain amount of the security at a specified price, i.e., the trader specifies the buy/sell price and the number (or fractions) of stocks he/she wants to trade. Limit orders are collected and stored in what is known as the Limit Order Book (LOB). At a time instance $t$, the cross-section of the LOB provides a snapshot of the number of outstanding limit orders, their prices, and quantities.
In particular, buy (sell) limit orders define the bid (ask) side of the book. The highest buy and lowest ask prices represent the best prices to sell or buy a certain amount of a security. These best-prices are known respectively as bid and ask prices ($p^B_t, p^A_t$). Market orders are immediately executed on the bid or ask side at the current best price, leading to trades. Limit orders at the current bid/ask prices are filled according to a time-priority rule (first submitted, first traded). A market order decreases the quantity available at the best price, and, if the market order quantity is equal to or greater than the outstanding quantity of the limit orders at the current best price, it reduces the total depth of the market (i.e., the number of different price levels on which the limit orders are arranged). As the limit orders on the top of the book are filled, the actual best price moves to that of the next LOB level until a new incoming limit order (on the same side of the book) re-fills the gap between the bid and ask prices, or a new market order erodes the top of the book causing a further update in the best bid or ask price. We refer to, e.g., ntakaris_benchmark_2018 for further details on the LOB mechanism.
It is clear that the order inflow (along with order cancellations) is governed by a highly-stochastic mechanism that leads to a rich, multidimensional dataset consisting of order types, prices, and quantities, whose instances reflect the dynamics of the bid and ask prices as well as of deeper LOB levels. Although broad stylized facts on the LOB dynamics lead to some analytic tools for modeling the LOB, e.g., cont_stochastic_2010, tackling its dynamics is very challenging, and ML methods can provide a useful alternative for a number of forecasting goals. While the first level of the LOB has been commonly used in econometric research, tran2021levels showed that the information in multiple levels increases the performance of ML models. Both ML and econometrics research focused on the dynamics of the synthetic price measure across the two sides of the book known as mid-price: $p_t = {\small \frac{1}{2}} (p^A_t-p^B_t)$.
We focus on the task of forecasting mid-price changes at the future (tick-by-tick) updates of the LOB. This implies a complex classification problem over three classes: mid-price increases, mid-price decreases, or remains stationary. We used the publicly available FI-2010 dataset ntakaris_benchmark_2018, which collects the LOB states for five stocks traded at the NASDAQ Nordic Helsinki exchange from June 1 to June 14, 2010 (collecting approximately 4.5 million events across ten trading days). At each epoch (i.e., LOB update), the data consists of 144-dimensional feature vectors. In total, there are 453,975 features extracted over non-overlapping blocks of ten events and normalized using the $z$-score. The dataset provides labels corresponding to the direction of the price movement on five different horizons, corresponding to the price movements in the next 10, 20, 30, 50, and 100 events. In our experiments, we utilize the 10-events horizon and adopt the experimental setup of tsantekidis_forecasting_2017,tsantekidis_using_2020 where the last three days are taken as the test set (corresponding to 150,418 samples). For the first seven days, the initial 75% of instances constitute the training set, and the last 15% the validation set.
As in tran_temporal_2019, we use the first 40 dimensions consisting of raw prices and quantities. For the BNN implementation, we employ the VOGN algorithm. Networks' weights are initialized under a multivariate Gaussian prior with parameters $\bm{\mu} \,{=}\, \bm{0}$ and $\bm{\Sigma} \,{=}\, \bm{I}$. The learning rate, momentum factor, and decay rate of the L2 norm regularization are respectively set to 0.01, 0.999, and 0.85. Its performance on the validation set is evaluated over 10 MC draws from the posterior at each epoch, while the predictive distribution for each input in the test set is approximated by the collection of $N_{s} = 50$ forecasts following $N_{s}$ feed-forward passes for $N_{s}$ independent samples from the variational posterior. Prior's means and variances, respectively set to one and zero, are initially warmed up following the method described in osawa_practical_2019.
The Bayesian training of the network is evaluated with respect to two non-Bayesian alternatives: the ADAM kingma_adam_2015 optimizer and Stochastic Gradient Descent (SGD). For SGD, the momentum is set to 0.99; for ADAM, the first and second moments are fixed to 0.9 and 0.999. For both algorithms, the initial learning rate is set to 0.01 and dynamically updated until the validation loss reaches a plateau. For all the optimizers, the training is set to 1000 epochs with a mini-batch of size 256. When training with ADAM, we also employ MC dropout gal_dropout_2016 in the testing phase. MC dropout is not a Bayesian method but has a connection with Bayesian theory and serves as an approach to predictive distribution approximation gal_dropout_2016. A random deletion of the NN connections allows for posterior sampling. Sampling from the approximate posterior enables MC integration of the likelihood, uncovering an approximation to the predictive distribution. By repeated forward passes for the same input sample, the randomized dropout yields samples from the predictive distribution. gal_dropout_2016 find that even a small number of forward passes can suffice. Similar to the B-TABL, we apply $N_{s} = 50$, and set the dropout rate to 10%. As in osawa_practical_2019, we do not compare VOGN with Bayes by Backprop blundell2015weight since it is very slow to converge for larger-scale experiments like the one targeted in this paper.
In \figurename (ref) we compare f1-scores and accuracy metrics across training epochs for both training and validation sets. For both VOGN and ADAM, it takes as little as 15 epochs to stabilize and smooth the learning rate of the curves. The initial values of the parameters are randomly initialized. To avoid local minima and to boost the search, the learning rate in ADAM is regularly perturbed, resulting in the step-wise behavior observed of the curves observed across the panels.
For VOGN, curves referring to the training set show a steeper rate at initial epochs and up to about epoch 500, reporting a remarkably higher f1-score and accuracy than for ADAM. This stands for a general superiority in the performance of VOGN with respect to ADAM at early epochs up to moderate ones, indicating that despite the random initialization and the stochastic components embedded in VOGN, perhaps due to its higher number of parameters due to the existence of the parameters' variance, the algorithm converges rather quickly. Only around epoch 500 ADAM metrics are comparable to those of VOGN. At higher epochs, we do not observe a relevant difference in f1-scores, while, in terms of accuracy, ADAM slightly outperforms VOGN on the training set. That is, the learning in ADAM is shown to be slower but, on average, steady, with certainly lower rates compared to VOGN, but constantly improving across the epochs. On the other hand, after steep improvements in initial phases, VOGN's training is quite achieved already at epoch 500, leaving only a slight 5% improvement of the metrics in the following 500 epochs.
Also on the validation set, we observe that at initial epochs, metrics for VOGN greatly outperform those for ADAM; perhaps the prior variance adds a randomization effect that allows a wider sampling of the space around the local parameters to access large gradients that readily adjust the step direction towards the minimum. It is only around epoch 800 that we observe a comparable performance. This could be interpreted as a better generalization ability of VOGN on unseen data, especially if noticing that for VOGN the f1-score and accuracy curves on the validation set are slightly higher than for training. The homogeneity of the data and the same complexity across the two sets, further motivate the conclusion that VOGN embeds a more general classification rule, along with the metrics provided in (ref). Also the different levels in curves' smoothness underline that while VOGN quickly approaches the minimization objective, ADAM appears to repeatedly overshoot the objective, leading to segmented curves up to epoch 500. At higher training epochs, validation curves' rates of growth for VOGN and ADAM appear quite flat, indicating that the training is overflown and the performance metrics are comparable. In this light, we might expect a comparable performance of the two algorithms on the test set, perhaps without a strong winner. This is indeed the case, see Subsection (ref) and Subsection (ref). As it also emerges from Subsection (ref), the performances for MCD and SGD are quite poor compared to VOGN and ADAM, making the former two optimizers quite unsuitable for our classification task, and therefore omitted from \figurename (ref).
Following the updates (ref) and (ref), VOGN learns variational posterior's mean and updates the prior precision to the posterior diagonal covariance matrix $\bm{\sigma}^2 I$, with $\bm{\sigma}^2_t = 1/\autobracket*{N\autobracket*{\bm{s}_t+\tilde{\alpha}}}{\alpha}}}$. For illustration purposes, \figurename (ref) depicts the learning of B-TABL's $\lambda$, a characterizing parameter for the network architecture.
In general, for all the parameters in TABL we observe a similar pattern where both parameters' means and variances converge to certain (different) levels. The posterior distribution is representative of the parameters' uncertainty after observing the data, that in VOGN's variational setting is forced to be a Gaussian distribution. We make use of the posterior's valuable information on the parameters' relevance by conducting individual t-tests on their significance. Very low p-values support the relevance of all the TABL parameters and implicitly that the network architecture is well-scaled for the problem under consideration. Following ((ref)), the posterior distribution builds the predictive one, a major focus in this paper.
We approximate VOGN's predictive distribution with $N_s=50$ draws from the posterior distribution. I.e., according to (ref), we approximate the posterior distribution in (ref) for a given input sample by fifty samples drawn from it to capture the uncertainly associated with a forecast $\hat{y}_{ic}$ given an unseen input $\bm{x}_i$ and the data used in the training phase. According to the decision criterion in (ref), the forecast's class is given by the predicted class of maximum class-probability. Note that, aligned with (ref), the predictive distribution is a distribution on class-probabilities $p\autobracket*{\bm{y}_i|\bm{x}_i,\mathcal{D}}$ and not on the forecasts $\hat{y}_{ic}$. \figurename (ref) provides insights into interpreting predictive probabilities and explains pitfalls in uncertainty interpretation.
The bottom-right panel in Figure (ref) depicts the box-plot of the class-probabilities corresponding to $N_s$ forward passes for a certain input in the test set. For this example, the distribution of the predicted classes is unarguably leaning towards class 3, while across the 50 samples, the class probabilities for classes 1 and 2 are similar. The bottom-left panel depicts the corresponding kernel probability density functions of the class probabilities, along with their per-class average. I.e., predicted probabilities for the sample input in \figurename (ref) are $p\autobracket*{\bm{y}_i|\bm{x}_i,\mathcal{D}} = \left[0.323,0.324,0.352\right]$ for classes 1, 2 and 3 respectively. A complete analysis of the joint distribution of $p\autobracket*{\bm{y}_i|\bm{x}_i,\mathcal{D}}$, here out of scope as it would likely involve a dependence modeling step through copulas that perhaps is not general. Yet the extracted mean values are quite informative. In particular, we see that the probability of class 3 is certainly higher than those of other classes, even though not overwhelming. That is, the predictive distribution underlines a scenario of rather high uncertainty. Indeed, an investor who, e.g., invests in an asset expecting its price to increase would, on average, observe an actual increase in the asset value with a 35% probability. This is the correct interpretation upon the predictive probability $p\autobracket*{\bm{y}_i|\bm{x}_i,\mathcal{D}}$. However, upon applying the classification criterion (ref) the conclusions might be quite misleading. The top-right panel in \figurename (ref) depicts per-sample class-probabilities, where big-sized points represent classes of maximum probability. For 43 samples out of 50 the class of maximum probability is 3. For three samples is class 1, and for three samples is class 2. By applying (ref) we would classify 86% of the samples in class 3, as depicted by the histogram in panel 4 of \figurename (ref). The histogram misleadingly covers a situation of high uncertainty with a quite overwhelming frequency observed for class 3: if we were to draw the forecasts according to the joint distribution in panel 2 we would observe, on average, only about 35% of the samples in class 3.
The availability of the predictive distribution in Bayesian DL frameworks is certainly the most remarkable aspect with respect to non-Bayesian approaches such as ADAM. Indeed, continuing with the above example, ADAM would not capture any uncertainty in the predicted label. An investor following ADAM's forecasts (whose performance is comparable with that of VOGN, see Section (ref)) would not be able to capture the high degree of uncertainty that VOGN unveils. Needless to say that the impact of uncertainties on whatever trading strategy an investor adopts is quite significant. E.g., an investor might choose to trade based only on predictions associated with relatively low uncertainty, or take well-informed actions to account for the actual possibility that the direction of the price movement is opposite to the predicted one. On the other hand, ADAM's forecasts are incapable of addressing the low 35% probability chances of a price increase, leaving the investor completely blind about the actual probability of a price increase and the perhaps adverse downward 32%-likely movement.
By inspecting a large number of examples and the overall statistics on class-probabilities we observe that the case studied in \figurename (ref) is somewhat atypical, in the sense that 35% of the predicted probabilities for the maximum-probability class are in the lowest quantiles of predicted probabilities for the prediction's class, see (ref). Typical values are about 50%: this still results in a wildly uncertain general scenario. Such relevant uncertainty for operational scenarios and real-life decision processes is entirely left unaddressed by ADAM and non-Bayesian methods. Whereas the decision criterion (ref) does provide a feasible and practical way to construct forecasts, average class-probabilities on repeated forward passes (i.e., predictive distributions) are the truly informative element about forecasts' uncertainty, inaccessible to non-Bayesian DL approaches.
or the sample input in \figurename (ref) the true label corresponds to class 1. In general, for wrong and correct classifications, the score variation in the class of maximum-probability class can be large. Table (ref) reports some statistics on the class of maximum probability (rank 1, denoted by for an input $\bm{x}_i$ with $\hat{p}^{(1)}_i$ ), class of second-highest maximum probability (rank 2, $\hat{p}^{(2)}_i$) and on the remaining class (rank 3, $\hat{p}^{(3)}_i$), along with the difference between the first two ranks, for correctly and miss-classified labels. For correct classifications, the average (median) predictive probability on rank-1 classes is 55% (51%), while for the miss-classified ones 51% (50%), the average distance between the predictive probabilities on classes of rank 1 and 2 is respectively 30% (25%) for correct classifications and 22% (20%) for miss-classifications. In both cases, we do observe samples of probability rank 1 with a corresponding predictive probability as high as 100% and as low as 33%. Ideally, we would like to observe (i) high rank-1 predictive probabilities for correct classifications, (ii) quite lower values for miss-classified samples, and (iii) a neat separation between $\hat{p}^{(1)}_i$ and $\hat{p}^{(2)}_i$ for correctly classified samples. This is the case, but the magnitudes of the differences are small. Table (ref) underlines that the predictive uncertainty in the forecasts is consistent and homogeneous whether the labels are eventually correct or wrong, with $\hat{p}^{(1)}_i$ consistently being of about twice $\hat{p}^{(2)}_i$ and $\hat{p}^{(2)}_i$. The observed differences in rank-1 and rank-2 predictive probabilities are, on average, as little as 7.5% between correctly and miss-correctly classified labels, while $\hat{p}^{(1)}_i$ differs by only 3.4%.
Accordingly, the empirical survivor function (ESF) for correctly classified labels in \figurename (ref) slightly dominates the one for miss-classified labels. Importantly, \figurename (ref) unveils that the probability of observing $\hat{p}^{(1)}_i> 0.6$ is only about 10-15%. This means that mild-to-low uncertainties on the maximum probability class are quite rare, and high-confidence is even rarer (7%-9% for $\hat{p}^{(1)}_i> 0.9$). However, the ESFs do not cross each other, and the difference is positive. For the same (or greater) level of confidence, the number of correctly classified samples is, on average, 5% (but up to 9%) higher for the correctly classified samples than the miss-classified ones (difference curve in \figurename (ref)).
To better understand the uncertainties arising from the predictive distribution, \figurename (ref) depicts the (kernel) density estimates of the scores across correctly and miss-classified labels. The top row in \figurename (ref) refers to correctly classified samples (TPs). For TPs, scores are well-separated in the sense that the distribution of class one is well-detached and distinguishable from the others. Interestingly, when the model is correct, the uncertainty in classes 2 and 3 is much lower than in the stationary-price case. This stands for the existence of clear patterns in the features that are truly indicative of the direction of the price movement, driving predictive probabilities close to 1 (i.e., uncertainty close to zero). When the model is correct about class-1 assignments, its confidence is somewhat lower and the densities of the scores for whatever change in price direction generally overlap. Confidence in stationary prices is about 0.5, while the remaining 0.5 is equally spread across classes 2 and 3.
Indeed, by only focusing on the miss-classified labels (FPs) in the bottom row in \figurename (ref), we find further evidence that when the model does not correctly classify a stationary mid-price, its predictions are about equally spread among a price-increase and a price-decrease, showing that in this case there is no intrinsic bias in the model parameters leaning towards a certain class; the model is simply wrong, and forecasts are flip-coins on classes 2 and 3. On the other hand, the bias towards the majority class is consistent for FPs in classes 2 or 3, and the scores for the true-label are always those of lowest means. The same distribution on class-1 TPs almost identically replicates on class 2 and 3 FPs: the model interprets certain patterns in the features as remarkably non-indicative of the true class 2 and 3 labels, causing an overflow of low scores for both of them. The relevant probability mass, excluded from classes 2 and 3, is transferred to class 1 following a distribution being very close to that observed on class-1 TPs. This suggests that the model well-distinguishes patterns indicative of classes 2 and 3, and when these are absent, class-1 classification is enforced. In this regard, see (ref).
Tails in FPs for classes 2 and 3 constitute interesting cases of very high class-3 and class-2 predictive probabilities corresponding to wrong assignments in classes 2 and 3. Patterns indicative of classes 3 and 2 are causing false positives in classes 2 and 3: (rarely) typical features for classes 3 and 2 are observed for mid-prices, eventually moving in the opposite direction. These real-surprise elements in the order flow are perhaps aligned with its stochastic nature.
There is a conclusive important insight on the model's learning within figure \figurename (ref). In particular, the B-TABL (and TABL, as the following applies to the scores from all the optimizers) architecture is learning how to classify increases and decreases in mid-prices but not stationary prices. The distribution of class-1 scores is the very same for the TPs in class 1 and the FPs in classes 2 and 3. That is, a distribution is placed over the scores in class 1 and is updated only if relevant features for class-2 or class-3 decisions are detected. Indeed when the model is not capable of correctly classifying classes 2 and 3, the distribution of the scores on classes 2 and 3 is roughly the same. In fact, the three plots in \figurename (ref) are almost identical at a visual inspection. I.e., unless there is robust evidence of an upward or downward price movement very likely corresponding to correct classification, the distribution of class-1 scores as FPs for classes 2 and 3 is the same. When the model truly does not detect any relevant information to discern whether the price is moving, movements are classified as stationary, and the very same distribution observed in class-1 TPs is adopted. This means that the density for TPs in class 1 is not actually learned from features that characterize this class. The density that is observed for class-1 labels is to be interpreted as the one that best improves the validation loss when the model is not detecting clear signals of future price movements. This is further supported by the analyses in Section (ref), (ref) and (ref).
Non-Bayesian methods such as ADAM and SGD provide single forecasts for a trained model. VOGN and MCD, due to the random sampling from the posterior and the dropout layer, lead to different foretasted labels at each forward pass. As noted in Section (ref), the distribution of the scores provides a misleading interpretation of forecasts' uncertainties. However, scores embed rich information for understanding the behavior of the labels' forecast and the underlining label-assignment mechanism. Again, the distribution of labels' forecasts analyzed in this section is based on $Ns=50$ draws. For VOGN, 96% of drawn labels are all assigned in the same class, 4% to two classes, 0.3% to three. Among the inputs whose labels' forecasts are across two classes, the predictive probability for the maximum class is 42%, and the difference between the predictive probability on the two classes is, on average, 3% (max. 36%, min. 0%). 45% of these inputs correspond to true labels in class 1, and for 98%, class 1 represents one of the two classes where the labels distribute, while the others are classes 2 and 3 with 50% frequency. I.e., the model appears inconsistent in labeling stationary prices over positive or negative movements but is very consistent in labeling the latest two. 61% of the samples with forecasts across two labels show, however, at least 80% of the 50 draws in the same class, while very ambiguous inputs account for only 6%, with a difference in the number of samples in the two classes not exceeding three. We could provide analogous information for the 0.3% of the samples whose forecasts' labels are observed over three classes; rather, we point out that the predictive probability on the three classes is on average 31%, 35%, and 35%, corresponding indeed to the most uncertain classifications. The above numbers suggest that typically the consistency in the foretasted labels is remarkable, i.e., that the modal value is overwhelming. This means that a single draw from the network would be very likely to equal the modal value. Indeed by randomly selecting with replacement 500 output vectors of labels from the 50 draws available for each test example, on average, 99.36% of the labels correspond to theirs distribution modal value.
Therefore in the following analyses, we include performance metrics based on modal forecasts as representative of the typical performance observed over a single forward pass. For completeness, we also consider metrics based on means and medians of the $N_s$ labels, rounded to the closest class' integer index. We omit the above details for MCD but underline that 72% of the labels' forecasts are observed over three classes: a remarkable difference. This is due to the fact that on a single TABL layer, the regularization usually provided by dropout causes a random-like assignment of the output classes.
Table (ref) reports the performance of different Bayesian and non-Bayesian optimizers for the (B)-TABL architecture. With respect to the test set, Table (ref) includes micro-, macro-, and weighted macro- averages as synthetic measures for evaluating the overall performance of the different classifiers across multiple classes. Micro-averages are constructed by summing the true/false-positive/negative rates individually for each class, before applying the definition of the specific performance measure under consideration. On the other hand, macro-averages refer to simple averages of the individual performance measures computed for each class. By accounting for the relative sample frequency of each class in taking averages, we construct weighted macro-averages. Note that accuracy and micro-averages for precision, recall, and F1-score are all equal and reported under a single column. Although macro-averages are the performance measures usually reported, as our sample is highly imbalanced (67% of the test samples in the stationary class and equally distributed across the remaining two classes), alternative multi-class statistics are here relevant. Macro-averages weight each class equally by computing the average of the metrics computed independently for each class. As a consequence, non-dominant-class' metrics might mislead the conclusions on the overall performance of the classifier. By accounting for class weights, weighted macro-averages naturally alleviate this issue. On the other hand, micro-averages, by summing the true/false-positive/negative rates individually for each class, aggregate the contributions of all classes to compute average metrics. By weighting each sample equally, micro-averages apply well to imbalanced problems where, from a qualitative standpoint, there are no differences in the importance of each class. In our context of imbalanced classes and multi-class task, the preferred metrics are the f1-score, which embeds both precision and recall, and micro-averages.
For the VOGN optimizer, results are divided into three panels. The upper one reports summary statistics for individual metrics computed for each of the $N_s=50$ simulated outputs, i.e., in a sample-by-sample fashion. Following the discussion in Section (ref), the second panel addresses the possibility of constructing labels' forecasts based on group statistics extracted from the $N_s$ labels. These correspond to forecasts' labels sample mean, median, and mode. The former statistics requires rounding to the nearest integer to be feasible, yet in our sample rounding applies to only 3.5% of the per-example labels' means, to 0.26% of medians, and never to modes. The third panel reports the metrics corresponding to predictive distribution, i.e., by considering the class of maximum predictive probability, computed under the standard Bayesian averaging approach and, alternatively, by considering median probabilities as a robust alternative to possible severe outliers.
For VOGN, predictive's distribution results are consistently the highest ones. However, up to three decimals, there are generally no differences between the three panels. Performance measures for median and modal forecasts largely overlap and equal predictive's distribution metrics. Slightly worse results are obtained by considering (rounded) forecasts' averages. The former aligns with the sample-by-sample centrality measures and predictive distributions' ones. This also suggests that for forecasting purposes, a single draw from the posteriors (whose corresponding label would approximate the forecasts' median label very closely) would lead to results perfectly aligned with the predictive's ones (implying a considerable computational advantage).
Among the other optimizers, ADAM stands out as the most valid alternative. Expect on precision, it does not perform better than any VOGN's metrics. Interestingly, metrics' minima in the top panel are always higher than ADAM's metrics (except for precision, where neither the maximum reaches ADAM's performance). This provides significance to the results in favor of VOGN as even the most unfortunate posterior sampling shows superior performance than ADAM, up to 1.8%. Concerning VOGN's predictive distribution, the observed improvements in performance with respect to ADAM are slight yet significant: the Bayesian optimizer does not provide worse results than the widely-adopted ADAM (except for precision), and it enables the predictive analysis of forecasts' uncertainty described in Section (ref). Lastly, MCD and SGD do not seem to be competitive for the prediction task under analysis. In (ref) we provide analogous stock-specific results.
Our following considerations concern the single-class problem classification. Despite the above multi-class task where each label is classified across three classes, by the single-class task we mean a binary classification problem where the true class is specified in advance, and the other classes constitute the negative class. Remind that the model is calibrated for the multi-class task: single-class metrics could be improved by re-calibrating the model specifically for forecasting a specific price-change direction.
A first useful analysis is that of inspecting the distribution of labels assigned to the true class, see \figurename (ref). The plot suggests a positive bias towards class 1, and a negative bias in the labels frequencies in other classes. As confirmed later, the first is due to the large number of FPs for class one, the latter is due to low TP rates for classes 2 and 3. Note that the differences between the frequencies based on VOGN's modal prediction and predictive distribution are irrelevant, while for MCD these are minor and favor predictions based on the predictive density. In the following, we will focus only on results resulting from predictive probabilities. From the analyses in (ref) we find that MCD alignment with the sample frequencies is not indicative of a genuine satisfactory performance: for class 1 (classes 2 or 3) this arises from a lower (comparable) TPR and comparable (lower) FPR with respect to the other optimizers. See e.g. \figurename (ref) therein.
From Table (ref) it appears that VOGN and ADAM have quite heterogeneous performances based on the measure and class under consideration. In particular, a conclusion on whether it is advisable to use VOGN or ADAM, in general, cannot be made. Overall we observe a tendency for ADAM to perform better in terms of precision and recall, thus on TPs therein involved. Yet when the two are considered jointly (harmonic mean) the f1-score favors VOGN. VOGN furthermore improves the detection of TNs involved in computing accuracy and of course enables the uncertainty analyses based on the predictive distribution.
For our multi-class classification problem, we consider Receiver Operating Characteristic (ROC) curves for the predicted classes. In cases where there are no disparities in the cost of false negatives as opposed to false positives, the ROC is a synthetic measure of the quality of models' prediction, irrespective of the chosen classification threshold. To construct ROC curves we discard ambiguous examples by thresholding each validation input's soft-max output and mark the remaining test examples as correctly or incorrectly classified, from which TRP and FPR rates are computed. We apply following thresholds $\{0.05,0.1,0.15,\dots,1\}$.
\figurename (ref) depicts ROC curves computed from micro and macro FPR and TRP rates for both VOGN and ADAM. For VOGN the figure includes the 95% interval extracted from the TRP variation across the forecast samples along with the main solid line based on the predictive distribution. The multi-class micro and macro averages for VOGN's curves are dominating. This indicates that larger predicted scores are increasingly more tightly associated with TP than FP, for VOGN more than for ADAM, and that across the whole FPR domain scores implied by VOGN are more conclusive (in terms of TPs) for the true label.
A commonly reported measure is the FPR at 95% TPR, which can be interpreted as the probability that a negative example is misclassified as positive when the true positive rate (TPR) is as high as 95%: for macro-averages we compute 88% and 90%, and for micro-averages 76% and 77%, for VOGN's forecasts based on the predictive distribution and ADAM respectively. To assess how well-calibrated a model is, CC compare how well the true class frequency determined by a classifier are calibrated to the true frequency of the positive class, for binned predictions (we take 20 bins). The CC curve of a perfectly calibrated model would lie on the diagonal curve, while over-confident predictions would generally result in CC above the diagonal.
CCs in the left panel of \figurename (ref) underline a comparable performance on micro- and macro- averages. A remarkable S-shape occurs at lower predicted probabilities. This means that the overall satisfactory statistics in Table (ref) arise from a balance between the non-ideal scenario where the models are either too much overconfident (predicted probabilities around 20%) and too much underconfident (around 50%). That is, the underlying scores shift from associating too little probability to the true label to way too much. At high scores, both VOGN's and ADAM's micro-average is quite aligned to the diagonal, yet macro-averages are over-confident suggesting high FPs for the dominant class 1.
ROC and CCs plots for the single-class task are found in \figurename (ref). From the ROC panel, we observe that VOGN outperforms ADAM in classifying labels of classes 1 and 2, and it has a slightly lower performance on class 3. As for Table (ref) ADAM's metrics are higher than VOGN's for class 3, determining improved TPRs. CCs for classes 1 and 2 are quite satisfactory, and the same comment applies as for the CCs in \figurename (ref). Remarkable is however the U-shape of the curves for class 1: high class-1 probabilities are overconfident and misleading as there are no samples in class 1 at all when models' probabilities for class 1 are about 1 (confirming the inference from micro- and macro- CCs in \figurename (ref)). Aligned with the discussion in Section (ref), models are truly learning the classification of classes 2 and 3. For samples in classes 2 and 3 which however do not display typical class 2 or 3 features, scores associated with classes 2 and 3 are about zero, and all the probability mass is allocated on class 1. In fact, out of the (only) 20 class-1 probabilities higher than 0.75, the 75% of them correspond to FNs for classes 2 or 3. This might be indicative of inadequacy in networks' architecture in uncovering deeper patterns in the data that could address class 2 and 3 classification, or non-stationarity elements of true and atypical surprise not observed in the training set or perhaps not learnable at all due to their randomness.
Table (ref) reports the Area Under the ROC (AUROC), the Expected Calibration error (ECE), and the L2-norm Distance (ECD) between the CCs and the diagonal line and the CCs. High AUROC, small ECD, and small (in absolute value) ECE are preferred. Results are aligned with the earlier plots and confirm the above comments. The low ECEs for MCD are not to be interpreted as evidence of improved calibration, as they arise from rather symmetric S-shaped CCs, that however largely deviate from the diagonal (see ECDs).
We proposed a first econometric time-series application with Bayesian Neural Networks. Our task focuses on predicting the direction of mid-price changes in modern limit-order book markets. By utilizing a state-of-the-art optimizer for Bayesian learning and adopting the suitable TABL capable of fully exploiting the ultra-high frequency and complex multidimensional nature of the data, we obtain promising results showing that Bayesian methods in deep learning are feasible, attractive, and valuable for economic applications.
With a number of detailed analyses, we compare several optimizers on the same forecasting exercise and unveil that the Bayesian VOGN optimizer provides, on a general level, the best performance metrics on both multi-class and single-class classification tasks. Yet, VOGN's performance is comparable with the well-known and reliable optimization scheme provided by ADAM. At the same time, Monte Carlo dropout and Stochastic gradient descent methods do not seem to be suitable for the task under analysis. Furthermore, we extensively interpret and discuss the results, grasping important insights into the model's learning and decision process. The unique feature of Bayesian methods is that of providing posterior and predictive distributions, leading to estimates of the uncertainties associated with the forecasts. The paper discusses how to use and interpret predictive probabilities, providing insights into their implication in the decision process. Following our analysis, and besides promoting further research and applications involving Bayesian deep learning methods, future research might explore to which extent posterior probabilities lead to better uncertainty-informed trades, e.g., by applying and comparing Bayesian and non-Bayesian models for constructing actionable trading strategies, verified with robust back-testing procedures.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sk\l odowska-Curie grant agreement No. 890690, and the Independent Research Fund Denmark project DISPA (project No. 9041-00004).