Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
109,756 characters · 32 sections · 137 citation commands
Transformers versus LSTMs for electronic trading
{3pt} {3pt}
Financial time series prediction is a significant task in investing and market-making activities. The Efficient Market Hypothesis proposed by Eugene EMH states that all the information of the asset’s inner value has already precisely and completely reflected on the asset price, and it is impossible to beat the market by financial prediction. However, whether the market is efficient is questionable because technical analysis TA believes the financial market is the physical movement of price (or features derived from prices). The price information can be interpreted by waves and patterns that can repeat themselves, where it is possible to make profitable buy or sell decisions in advance alma991000618293301591,boroden2008fibonacci. During the prediction, challenging factors are noise and volatile features because price information is generally non-linear and non-stationary alma991000193490801591. Lots of models are proposed to solve the financial time series problem. A typical linear model for regression is Auto-Regressive Integrated Moving average (ARIMA) 7046047 and its variations, which requires domain expertise to handcraft features. With the development of machine learning, Artificial Neural Networks (ANN) raises great interest because of their capability to extract more abstract features from data and find a hidden non-linear relationship without assumptions or human expertise by adding more parameters. Long short-term memory (LSTM), which is a special recurrent neural network (RNN) architecture that has been proven successful in the application of sequential data, is widely applied to handwriting recognition DBLP:journals/corr/abs-1902-10525 and speech recognition https://doi.org/10.48550/arxiv.1610.09975. Like RNN, the Transformer attention is also used to handle the sequential data. Compared to LSTM, the Transformer does not need to handle the sequence data in order, which instead confers the meaning of the sequence by the Self-attention mechanism. \\ Applying LSTM and Transformer for financial time series prediction is a popular trend nowadays. Depending on the historical financial data, researchers usually make predictions for the future numerical prices, price difference, return or future price movement (rise, stationary, fall). Although LSTM and Transformer are applied in different aspects for this problem. There are mainly two research directions:
Since 2017, the Transformer has been increasingly used for Natural Language Processing (NLP) problems. It produces more impressive results than RNN, such as machine translation https://doi.org/10.48550/arxiv.1806.06957 and speech applications Karita_2019, replacing RNN models such as LSTM in NLP tasks. Recently, a surge of Transformer-based solutions for less explored long time series forecasting problem has appeared SURVEY. However, as for the financial time series prediction, LSTM remains the dominant architecture.\\ Whether Transformer-based methods can be the right solution for financial time series forecasting is a problem worth investigating. Therefore, this paper is going to compare the performance of different Transformer-based and LSTM-based methods on financial time series prediction problems based on LOB data and attempt to adapt new models based on Transformer and LSTM. The contributions of this study are summed up as follows:
The later parts of this study are structured as follows: The background and related work are introduced in Section (ref). Section (ref) describes the formulation of three financial prediction tasks. Section (ref) explains the details of previous Transformer-based and LSTM-based methods used for comparison. Section (ref) is the analysis of experiment results. Please note that the details of the newly developed DLSTM model and the architecture changes of Transformer models are explained in Section (ref) in Section (ref). The study is organized in this way to provide readers with a better understanding of the relationship among three financial prediction tasks.\\ The Source Code of this study is available at: https://github.com/772435284/transformers_versus_lstms_for_electronic_trading \paragraph{Acknowledgements} We would like to express our gratitude to Zhipeng Wang for constructive comments and suggestions.
Time series prediction has been applied in different financial activities. For example, the trader or market maker predicts the future price or the price movement of the assets so that he/she can design trading/market making strategies based on the prediction results to make profit from it. This part examines papers related to time series prediction using LSTM and Transformer. Most of them are related to financial time series prediction.
LSTM has been widely utilized in financial time series prediction depending on OHLC data and its derived financial indices. Many works prdictLSTM,CAO2019127,10.1371/journal.pone.0180944,8126078,FISCHER2018654 present predicting stock prices as successful by LSTM. Bidirectional LSTM (BiLSTM) is applied to increase the prediction performance DBLP:journals/corr/abs-1911-09512. With the rise of NLP, Sequence-to-Sequence Model (S2S) S2S is proposed and applied in machine translation and question answering. It is now also applied in financial time series prediction by combining LSTM structure and attention mechanisms contributing to higher performance AttenLSTM,Zhang_2019.\\ LSTM has successfully forecasted high-frequency data depending on the large datasets extracted from the limit order book (LOB). In terms of making predictions upon order book data, Convolution Neural Network (CNN) and LSTM are both in favour of research and sometimes they are combined. Sirignano et al. LSTMLOB trained a universal model using LSTM to predict the LOB price movement by data from all stocks, which outperforms linear and non-linear models trained on a specific asset. Zhang et al. DeepLOB combine CNN and LSTM to form a new deep neural network architecture called DeepLOB to predict future stock price movements in LOB data outperforming architecture only containing LSTM. Zhang et al. LOBs2s also combine the DeepLOB architecture with Seq2Seq and Attention model to forecast multi-horizon future price movements in one forward procedure, which reduces the training effort and achieves better performance in long horizon prediction than DeepLOB. According to Tsantekidis et al. https://doi.org/10.48550/arxiv.1810.09965, the structure of CNN-LSTM architecture is able to outperform other models on LOB price movement prediction because of its more stable behaviour. Kolm et al. Kolm2021DeepOF use the OFI feature derived from the LOB to predict the future min-price return, where the CNN-LSTM structure still achieves the best performance.
As Transformer makes a great contribution to NLP https://doi.org/10.48550/arxiv.2005.14165, many advanced models are proposed, such as BERT and GPT-3, which now have more influence in time series prediction. As the canonical Self-attention mechanism has $O(L^2)$ time and memory complexity, many modifications have been made to the Transformer in order to adapt to the time series prediction problem to process the long sequence efficiently. There are many alternative Transformer models for long time series forecasting problem have been developed recently SURVEY: LogTransLogTrans, Reformer Reformer, Informer Informer, Autoformer Autoformer, Pyraformer liu2022pyraformer and the recent FEDformer FEDFormer.\\
Theses Transformer models mentioned above are mainly evaluated on non-financial datasets, such as electricity consumption, traffic usage, and solar energy dataset. They achieve a considerable performance increase in accuracy over the LSTM model. Some works start applying Transformers for financial time series prediction. Hu 9731073 uses a Temporal Fusion Transformer with support vector regression (SVR) and LSTM to predict stock price. Sridhar et al. 9538640 predict the Dogecoin price through Transformer, which has superior performance compared to LSTM. Sonkiya et al. predictBERT do sentiment analysis by BERT to generate the sentiment score, which is then combined with other financial indices and passed into the Generative Adversarial Network (GAN) for stock price prediction. These works 9731073,9538640,predictBERT use Transformer-based methods to make predictions based on OHLC data and the research on applying Transformer for LOB prediction is limited. \\ Overall, LSTM is applied broadly in financial time series prediction and has been tested on various datasets, while the Transformer is limited. Therefore, this study wants to apply these Transformer-based models to a wider region in financial time series prediction to compare their performance to LSTM.
This study compares LSTM-based and Transformer-based methods among three financial prediction tasks based on LOB data. In this section, the basic concept of LOB will be first introduced, and then the formulation of three financial prediction tasks will be explained in detail. Three tasks are listed below:
With computers and the Internet, most financial products such as stocks, forex and cryptocurrency are traded on the electronic market nowadays. Two types of orders exist in the electronic market: limit order and market order. According to Gould et al. LOB, a limit order is the order to execute buy or sell direction at a specific price, where the orders can be succeeded, overdue, or cancelled and are recorded by the LOB. There are bid limit orders and ask limit orders used to buy and sell products by the trader or sell and buy products by the market marker. The highest bid price the buyers are ready to buy is referred to as the best bid price, and the lowest ask price the sellers are ready to sell is called the best ask price. The average of these two prices is called the mid-price, which reflects the current value of the financial product. The difference between them is the spread, which the market marker can usually make a profit. The illustration of the LOB is shown in Fig (ref).
Another type of order is the market order, where the trader can immediately buy/sell the product at the best price. There exists a matching mechanism in the LOB. Most exchanges adopt the price and time priority matching mechanism. The limit orders will be first executed in order with a better price. If two orders have the same execution price, then the order that comes first in time will be executed first, following the first in first out (FIFO) principle.
The first task is to predict the LOB Mid-Price Prediction, which is to compare the ability to predict absolute price values similar to non-financial datasets in previous works LogTrans, Informer, Autoformer, FEDFormer,liu2022pyraformer. The definition of time series prediction is given below and shown in Figure (ref):
\\ First, define a sliding window size $L_{x}$ for the past data. The input data at each time step $t$ is defined as:
Then define a prediction window size $k$, where the goal is to predict the information in future $L_x+k$ steps. It will be the single-step prediction when $k=1$ and be multi-horizon prediction when $k>1$. Then the output at time step t is defined as:
The next step is to define the $x_t$ and $y_t$ in the input and output for mid-price prediction. Assume the market depth is 10. For a limit bid order at time t, the bid price is denoted as $p_{i, t}^{b i d}$ and the volume is $v_{i, t}^{b i d}$, where $i$ is the market depth. Same for the limit ask order, ask price is $p_{i, t}^{ask}$ and volume is $v_{i, t}^{ask}$. Then the LOB data at time t is defined as:
The past mid-price will be added to LOB data as input, and the mid-price is represented as:
Finally, the $x_t$ will be:
The target is to predict the future mid-price, so $y_t = p_{t}^{\text {mid }}$.
The second task is to predict the mid-price change, which is the the difference of two mid-prices in different time step. Trading strategies can be designed if the price change becomes negative or positive. The input of this task is the same as the mid-price prediction, as described in Equation (ref). The target is to regress the future difference between current mid-price $p_{t}^{\text {mid }}$ and the future mid-price $p_{t+\tau}^{\text {mid}}$:
Like the mid-price prediction, a prediction window size is defined as $k$, then the output of this task in each timestamp $t$ is represented as:
According to Ruppert alma991000193490801591, the absolute price information is generally non-stationary, while the price change information, such as price difference and return and approximately stationary. The the mid-price difference is a difficult target for deep learning methods to predict because it is hard to extract meaningful pattern from it, although it helps design trading strategies. An example of non-stationary and stationary sequence is shown in Figure (ref).
For this reason, an easier classification task for predicting mid-price movement is introduced here. To train a model to predict mid-price movement, the first step is to create price movement labels for each timestamp. This study follows the smoothing labelling method from Tsantekidis et al. 8010701 and Zhang et al. DeepLOB: Use $m^-$ to represent the average of the last $k$ mid-price and $m^+$ to represent the average of the next $k$ mid-price:
$k$ is set to $20, 30, 50, 100$ in this study following previous work of Zhang et al. DeepLOB.\\ And then, define a percentage change $l_t$ to decide the price change direction.
The label is dependent on the value of $l_t$. A threshold $\delta$ is set to decide the corresponding label. There are three labels for the price movement:
An example of labelling for horizon 100 is shown in Figure (ref). Assume there is an input in Equation (ref) at timestamp $t$, predicting mid-price movement is a one-step ahead prediction, which is to predict the mid-price movement in timestamp $t+1$.
LSTM was introduced by Hochreiter et al. lstm is one of the RNNs with structural adaptability in time series data input. Although the traditional RNN has the capacity to store data, but it suffers from the exploding gradient problem and vanishing gradient problem. Exploding/vanishing gradient means the gradient that is used to update the neural networks increases/decreases exponentially, which makes the neural network untrainable dlbook. Therefore, RNN is not successful in studying long-time series relations Rumelhart1986LearningRB. LSTM neural network utilizes the coordination of three gates to keep long-term dependency and short-term memory. According to Gers et al. 818041, the three gates that LSTM utilizes are 1) forget gate, 2) input gate 3) output gate. The structure of an LSTM cell is shown in Figure (ref).
The calculations of the LSTM are as follows lstm:\\ First, the LSTM need to decide to forget some information from the cell state, which is done by the forget gate. The forget gate has its sigmoid function $\sigma$. Then the function of forget gate is:
Where the $W_f$ is the weight of the last hidden state and input, $b_f$ is the bias of the hidden state.\\ The next step is to design what information the neural cell should remember. To update the information, the input gate should coordinate with a $tanh$ layer containing a vector of new candidate values $\tilde{c}^{\langle t\rangle}$. The calculation is as follows:
\useshortskip
And then, with the previous steps of calculations, the cell state can be updated:
Lastly, calculate the result from the output gate to get the new hidden state:
The output of LSTM can be every hidden state or the final hidden state, which depends on the application. In the implementation, this hidden state will be fed into a multi-layer perceptron (MLP), also known as feedforward-backwards propagation neural network (FFBPN). The output of this layer will pass through an activation function to generate the final output. Usually, the final hidden state will be utilized in financial time series prediction, which produces absolute price prediction or price movement prediction.
Besides the canonical LSTM, three more LSTM based-models are chosen for comparison to Transformer-based models. They are DeepLOB DeepLOB, DeepLOB-Seq2Seq LOBs2s and DeepLOB-Attention LOBs2s created by Zhang et al. The architecture of these three models are shown in Figure (ref) and (ref). Here the structures of these three models are briefly explained:\\
DeepLOB DeepLOB DeepLOB’s architecture consists of three main components: Convolutional Blocks, an Inception Module and an LSTM layer. \\ A. Convolutional Blocks The LOB inputs mentioned in Equation (ref) are fed into the convolutional blocks that contain multiple convolutional layers, where the first and second convolutional blocks are more important than the third one. The first convolutional block has a layer with filter size of ($1\times2$) and stride of ($1\times2$). At each order book level, this layer summarise the price and volume information $\left\{p^{(i)}, v^{(i)}\right\}$. For the second convolutional block, it has a layer with the same filter size and stride as the first one, but it is a feature mapping for the micro-price defined by micro:
$I$ is called the imbalance. Then the last convolutional block integrates the feature information from the previous two layers. The whole convolutional blocks work as a feature extractor.\\ B. Inception Module the Inception Module employs the time series decomposition method. The input is decomposed by two $1\times1$ convolutions and one max-pooling layer into three lower-dimensional representations. Then these representations pass through convolution layers with 32 channels to be merged together. This decomposition method improves the prediction accuracy.\\
C. LSTM Layer Finally, the extracted features are inputted into one LSTM layer to capture the underlying pattern and dependencies. The last output layer can be a SoftMax layer or a linear layer, which depends on the specific tasks.\\
DeepLOB-Seq2Seq LOBs2s To generate multi-horizon predictions, Zhang et al. developed DeepLOB-Seq2Seq. The main idea is to feed the output of the Inception Module into a Seq2Seq architecture to do iterated multi-step (IMS) prediction. Seq2Seq seq2seq architecture contains encoder and decoder constructed by recurrent neural network (RNN). Assume the sequence input is $X_{T}=\left(x_{1}, x_{2}, \cdots, x_{T}\right)$, the encoder will output a hidden state at each timestamp t:
After obtaining the hidden states from the encoder, a context vector $c$ has to be constructed from these hidden states. The last hidden state or the mean of all hidden states can be taken as a context vector. Context vector work as a "bridge" between the encoder and decoder, where the context vector is utilized to initialize the decoder, and the hidden state output of the decoder at each timestamp is:
Then the distribution of the output $y_t$ is:
where the output of the decoder not only depends on the previous true value as input but is also conditioned on the context vector.\\ DeepLOB-Attention LOBs2s With the same idea as the DeepLOB-Seq2Seq model, the difference of the DeepLOB-Attention model is changing the Seq2Seq architecture into Attention. The attention model DBLP:journals/corr/LuongPM15 constructs the context vector differently instead of using the last hidden state or the mean of all hidden states. Same as Seq2Seq model, the encoder outputs hidden state $h_t$ and decoder outputs hidden state $d_t$. The first step is to compute a similarity score between the hidden state $d_t$ of the decoder and each encoder state $h_i$, where the similarity score is usually calculated by dot product:
And then, normalize the similarity scores to obtain a weight distribution by softmax:
Finally, generate the context vector from the attention weights:
After that, the process of producing the output $y_t$ is the same as the Seq2Seq model.\\
The Transformer attention has an encoder-decoder structure that relies on the Self-attention mechanism without relying on CNN and RNN. This architecture allows the Transformer to process the long sequence and has no problem with the vanishing gradient, which means it can model the dependency regardless of the length of the input/output. A series of components forms the Transformer: Multi-head Self-attention, positional-wise feed-forward neural network, layer-normalization, and residual connection. According to Vaswani et al. attention and Farsani et al. TSMTSF, the architecture of the Transformer for financial time series prediction is shown in Fig.(ref).
There is a slight difference between this transformer and the vanilla one used for NLP tasks. The word embedding process is omitted, and the financial time series is fed into the transformer using time-stamp encoding. The details of time-stamp encoding will be explained in Section (ref).
Self-attention is a mechanism for finding the relevant vector in a sequence. The target of Self-attention is to compute the attention score and then extract information based on the score. According to Vaswani et al. attention, the calculations of scaled dot-production Self-attention are as follows: First, to compute the attention score, multiply the input vector $I$ with different learnable weight matrices $W^q$ and $W^k$ to obtain the query and key:
The next step is to calculate the attention matrix, where each element in it is an attention score:
Where $d_k$ is the dimension of query and key.\\ Lastly, multiply the attention matrix with the values, and the final output can be obtained:
The output is the weighted sum of the relevance of different vectors in the sequence. In the implementation of the transformer, Multi-head Self-attention is used, which is beneficial for finding different types of relevance.
Different from the fixed sinusoid encoding used in the vanilla transformer attention, timestamp encoding from the Informer Informer is more informative for time series data and a similar method is applied in Autoformer Autoformer and FEDformer FEDFormer. The timestamp encoding method uses learnable embedding layers to produce positional information to add to the sequence, where timestamp information like a minute, hour, week, year and extra time stamps like event or holiday can be incorporated. To obtain the time-step encoding, the first step is to calculate fixed sinusoid encoding. Assume the input is $X_{t}=\left\{x_{1}, x_{2}, \ldots, x_{L_{x}} \mid x_{i} \in R^{d_{x}}\right\}_{t}$ at timestamp $t$ , where $L_x$ is the sliding window size and $d_{x}$ is the model dimensionality, then the encoding is calculated as follows:
And then, project the original input $x_i^t$ into the model dimensionality vector $u_i^t$ using convolutional filters. The next step is to use a learnable embedding layer $\text {SE}_{pos}$ to incorporate the timestamp information. The structure of the timestamp embedding is shown in Fig.(ref).\\
Finally, add up all the calculation results above to get the final encoding:
Where $i \in\left\{1, \ldots, L_{x}\right\}$ and $\alpha$ is the parameter balancing the ratio of scalar projection and embeddings.
As mentioned in Section (ref), several new Transformer-based models Reformer, LogTrans, Informer, Autoformer, FEDFormer are dedicated for long time series forecasting. However, they have not been tested on financial time series data. In this study, they are chosen as the alternative models to compare with the vanilla Transformer and LSTM. These models and their relationships are briefly summarized as follows (see Figure (ref)):\\
LogTrans Li et al. LogTrans put forward LogTrans with convolutional Self-attention generating queries and keys in the Self-attention layer. This work makes the convolutional layer widely used in the attention module in the later studies Informer, Autoformer, FEDFormer. It uses a Logsparse mask to reduce the time complexity from $O(L^2)$ to $O(L\log(L))$.\\ Reformer Reformer Reformer changes the time complexity of the Transformer to $O(L\log(L))$ as well by using sensitive hashing instead of dot-product in the calculate of attention. It also replaces the residual connection in the vanilla Transformer with a reversible residual connection, which makes the Transformer more efficient.\\ Informer Although Reformer and LogTrans reduced the time complexity to $O(L\log(L))$, the memory consumption is still the same, so the efficiency gain is not high. Also, LogTrans and Reformer use iterated multi-step prediction (IMS), generating a single prediction in each timestamp and using that prediction iteratively to obtain multi-step prediction IMS, which suffers from error accumulation. Informer Informer proposed by Zhou et al. employs a ProbSparse Self-attention mechanism to achieve $O(L\log(L))$ time and memory complexity. They propose an innovative generative style decoder to make direct multi-step(DMS) prediction, which is to generate the multi-step prediction at once in one forward procedure DMS. This method speed up the long-term forecast compared to the LogTrans and Reformer. The DMS forecast method and learnable timestamp encoding are applied in the Autoformer Autoformer and FEDformer FEDFormer.\\ Autoformer The optimization of Transformer on time series prediction is a trade-off between efficiency and information utilization. Depending on the structure of Informer, Autoformer Autoformer reduced the time complexity with an Auto-Correlation mechanism rather than making Self-attention sparse in LogTrans and Informer, which can preserve the information well and measure the sub-series dependency. Time series decomposition is a method commonly used in time series analysis to deconstruct the time series into several components STL,TS. The underlying temporal pattern can be revealed from these components to make the time series more predictable alma991000567577201591. Autoformer first embeds the time series decomposition as an inner neural block to derive the trend-cyclical component from the input sequence and the seasonal component from the difference between the trend-cyclical component and the input sequence. This new decomposition architecture can deconstruct time series to use the series periodicity to update attention.\\ FEDformer Based on the decomposition architecture of Autoformer, Zhou et al. FEDFormer builds the FEDformer to use Fourier transform and Wavelet transform, which are in the frequency domain as a new decomposition method to reach linear complexity $O(L)$. .\\
The first task to compare transformer versus LSTM is the mid-price prediction. In this task, predicting the absolute value of the mid-price is similar to the previous work's LogTrans, Informer, Autoformer, FEDFormer,liu2022pyraformer experiment on non-financial datasets.
Dataset All the experiments are based on cryptocurrency LOB data, which are collected in real-time from Binance Exchange using cryptofeed cryptofeed WebSocket API and saved to the database using kdb+tick triplet novotny2019machine. In this experiment, one-day LOB data of product BTC-USDT (Bitcoin-U.S. dollar tether) on $2022.07.15$. containing $863397$ ticks is utilized. The time interval between each ticks is not evenly spaced. The time interval is $0.1$ second on average. The first $70\%$ data is used to construct the training set, and the rest $10\%$ and $20\%$ of data are used for validation and testing. The reason why only using one day of LOB data is that it is enough to train for a task prediction for absolute value without overfitting according to previous works’ experiments on non-financial datasets.\\ Models For the comparison purpose, I choose canonical LSTM and vanilla Transformers along with four Transformer-based models: FEDformer FEDFormer, Autoformer Autoformer, Informer Informer and Reformer Reformer. For the implementation of Transformer-based models, they are taken from open-source code repositories Autoformer_repo, FEDformer_repo. The implementation of vanilla Transformer attention, Reformer Reformer, Informer Informer and Autoformer Autoformer are from the Autoformer repository Autoformer_repo. The implementation of FEDformer FEDFormer is from its own repository FEDformer_repo. \\ Training setting The dataset is normalized by the z-score normalization method. The validation set and the test set are normalized by the mean and standard deviation of the training set. All the models are trained for 10 epochs using the Adaptive Momentum Estimation optimizer and L2 loss with early stopping. The batch size is 32, and the initial learning rate is 1e-4. All models are implemented by Pytorch pytorch and trained on a single NVIDIA RTX A5000 GPU with 24 GB memory.
Quantitative result To evaluate the performance of different models, following the previous works LogTrans, Informer, Autoformer, FEDFormer,liu2022pyraformer, the performance metrics consist of Mean Square Error (MSE) and Mean Absolute Error (MAE), representing the prediction error. Lower MSE or MAE indicates the model has less prediction error. MSE and MAE are calculated by:
where $Y_{i}$ is the true value and $\widehat{Y}_{i}$ is the predicted value. $n$ is the number of ticks. \\ The results of all models are shown in Table (ref). From the table, these outcomes can be summarized:\\ 1. Both FEDformer and Autoformer outperform LSTM and FEDformer has the best performance in all the prediction lengths. FEDformer and Autoformer give a large increase in performance in terms of MSE and MAE compared to LSTM. For FEDformer, it gives $24\% (0.104 \rightarrow 0.0793)$ MSE reduction on 96 prediction length and $21\%(0.771 \rightarrow 0.608)$ on 336 prediction length. For Autoformer, it gives $11\%(0.104 \rightarrow 0.0926)$ MSE reduction on 96 prediction length and $16\%(0.771 \rightarrow 0.643)$ MSE reduction on 336 prediction length. These results show that both Autoformer and FEDformer perform well in terms of MSE and MAE because of their low error and long-term robustness. \\ 2. Although FEDformer and Autoformer's MSE and MAE are low on this task, LSTM is relatively not bad on mid-price prediction. LSTM outperforms the other three models: Informer, Reformer, and vanilla Transformer, which indicates that LSTM is robust in handling LOB data, while transformer-based models require lots of modification to perform well. \\ 3. Vanilla Transformer model has worse performance on prediction lengths $96$ and $192$ and Reformer has worse performance on prediction lengths $336$ and $720$ because they suffered from error accumulation during the IMS prediction process. Informer’s worse performance than LSTM is mainly due to its sparse version of attention, leading to information loss on the time series.
\\ Qualitative results The prediction results of compared models on all the prediction horizons are shown in Figure (ref). When the prediction horizon is $96$, Autoformer and Reformer are able to generate a proper trend for the future mid-price, while other models generate almost a flat line as predictions. On the prediction horizon of $192$, almost all the models’ predictions plateau except the Reformer, but Reformer’s result becomes more stochastic than the prediction horizon of $96$. For larger prediction horizons $336$ and $720$, all the models can hardly predict a proper trend and Reformer’s result in not plotted because it becomes too stochastic. \\ Based on the qualitative results above, although Autoformer and FEDformer outperform LSTM in terms of MSE and MAE, their actual prediction performance is far more inadequate for high-frequency trading. The analysis from Figure (ref) is just “eyeballing” whether the model is good. In this case, another formal metric out of sample $R^2$ is added here to judge the prediction quality. According to Lewis-Beck r_square, $R^2$ determine how well the prediction result $Y$ can be explained by the input $X$, and higher $R^2$ indicates the model has a better fit for the predicted value. Out of sample $R^2$ is defined by:
where $\bar{Y}=\frac{1}{n} \sum_{i=1}^{n} Y_{i}$; $\widehat{Y}_{i}$ is the predicted value and $Y_{i}$ is the ground truth.\\ Here the out-of-sample $R^2$ is calculated based on the price difference. Please note that the price difference here differs from the one mentioned in Section (ref). The price difference here is calculated from the absolute price prediction result, while the one in Section (ref) is the direct prediction target. The result of $R^2$ is shown in Table (ref).
\\ From the table, all the out-of-sample $R^2$ values are negative for all models. However, according to Lewis-Beck r_square, $R^2$ at least needs to be larger than zero to indicate that input $X$ can explain output $Y$. This indicates that predictions for absolute mid-price are useless for trading purposes. The metrics of MSE and MAE obscure the real quality of prediction results, highlighting the importance of $R^2$ calculated based on the price difference. \\
To sum up, although Autoformer and FEDformer have lower MSE and MAE than LSTM, their prediction result is not helpful and practical for trading. The more sensible way is to directly use the price difference as the prediction target.
Both transformer-based models and LSTM are unable to generate the satisfactory result on the mid-price prediction, so this task turns to the mid-price difference prediction. The mid-price difference is a useful alpha-term structure in trading for traders and market makers Kolm2021DeepOF.
Dataset The dataset for this experiment is collected the same way as the last experiment, but the dataset size becomes larger to avoid overfitting. In this experiment, four days of LOB data for the product BTC-USDT from 2022.07.03 (inclusive) to 2022.07.06 (inclusive) is used, containing 3432211 ticks. The first $80\%$ of data is used as a training set, and the rest $20\%$ is split in half for validation and testing. \\ Models and Limitations Five models are being compared in this experiment: canonical LSTM lstm, vanilla transformer attention, CNN-LSTM (DeepLOB DeepLOB model used for regression), Informer Informer and Reformer Reformer. State-of-the-art FEDformer and Autoformer are not compared in this task because their time decomposition structure is limited in handling price difference series. As mentioned in Section (ref), the price difference is approximately stationary. The time decomposition method is useful for the non-stationary time series, such as the mid-price series, where it can extract meaningful patterns. In contrast, the price difference series is approximately stationary, and little meaningful information can be extracted from it. Therefore, FEDformer and Autoformer can only produce a poor result in this task and will not be compared below. \\ Training settings the training setting is the same as the last experiment.
Following the previous works Kolm2021DeepOF, out of sample $R^2$ is the evaluation metric for this task. The performance of all the models is shown in Figure (ref). The canonical LSTM achieves the best performance among all models, which reaches the highest $R^2$ around $11.5\%$ in forecast length $5$ to $15$. For CNN-LSTM, it has comparable performance to LSTM. On the other hand, Informer, Reformer and Transformer have worse $R^2$ than LSTM, but their $ R^2$ trend is similar. In short, for the price difference prediction task, LSTM-based models is more stable and more robust than Transformer-based models. This result is in expectation because Reformer, Informer and Transformer already have worse performance than LSTM in mid-price prediction task because of their shortcomings. At the same time, the state-of-art FEDformer and Autoformer cannot be applied because of their limitation. In order to let these state-of-the-art transformer-based models make a meaningful prediction, a new structure is designed in the next part, and it is applied to the price movement prediction task.
The alternative Transformer-based models mentioned in Section (ref) mainly focus on the long time series forecasting problem, which is a regression task. For the LOB data, it is easy to adapt these models for the mid-price prediction and mid-price difference prediction because both are regression problems. For the mid-price movement prediction task, the model needs to produce a classification result for the future, and there are few existing Transformer models specialized available for this task. In contrast, most of the Transformer-based models are designed for classification tasks without forecasting, such as sentiment analysis, spam detection and pos-tagging in NLP. In this case, adapting the existing Transformer-based models to do price movement forecasting is necessary. I first adapt Transformer-based models in price movement forecasting and want to facilitate transformer development in this specific task. The new architecture of the transformer-based model is shown in Figure (ref). The details are explained as follows:\\ Predicting the next mid-price movement based on the past price and volume information is an one step ahead of prediction.
A straightforward method to adapt the transformer-based model is to pass the next predicted mid-price into a softmax activation. However, this method performs poorly because it only considers the past mid-price information and ignores future ones. It is worth noting that in the labelling process in Section (ref), previous and next $k$ mid-price information are utilized. In this case, I adapt the existing transformer-based models to feed the whole predicted mid-price sequence into a linear layer and finally pass through a softmax activation function to generate price movement output. This adaptation will benefit those transformer-based models using the DMS forecasting method because they have fewer errors in the long-time series prediction process.
Inspired by the Dlinear model dlinear and Autoformer, combining the merits of time decomposition with the LSTM, a new model named DLSTM is designed. DLSTM is designed based on these three observations: Firstly, the time series decomposition method is capable of increasing the performance, especially embedding this process by neural blocks in previous works DeepLOB, Autoformer, FEDFormer. Secondly, LSTM is a robust and simple model for multiple forecasting tasks. Thirdly, Dlinear beats other Transformer-based models in some long time series forecasting tasks thanks to the time series decomposition method and DMS prediction. However, predicting price movement is one step ahead prediction, where the model will not suffer from the error accumulation effect. In this case, it is sensible to replace linear with LSTM, because LSTM is a model well-known better than linear in handing time series.
\\ The architecture of DLSTM is shown in Figure (ref). The main difference between Dlinear and DLSTM is that the LSTM layers replace the Linear layer. According to the time decomposition method introduced in Autoformer Autoformer, in the prediction process, assume there is a time series $X_{T}=\left(x_{1}, x_{2}, \ldots, x_{T}\right)$, first decompose it into Trend series by the moving average:
where $AvgPool(\cdot)$ is the average pooling operation and the $Padding(\cdot)$ is to fix the input length.\\ And then the Remainder series is calculated by $X_{r}=X_{T}-X_{t}$. After that, these two series are inputted into two LSTM layers. Finally, the hidden states $H_{t}$ and $H_{r}$ produced by two LSTM layers will be added together and then pass through a linear and softmax activation to generate the final price movement result.
Dataset In this experiment, the largest dataset among three tasks is utilized to avoid over-fitting and test the model’s robustness. The whole dataset contains 12 days of LOB data of product ETH-USDT (Ethereum-U.S. dollar tether) from 2022.07.03 (inclusive) to 2022.07.14 (inclusive), containing 10255144 ticks. The training and testing data are taken from the first six days and the last three days, and the left data are used for validation. The test set is also used for the simple trading simulation. \\ Models Thanks to the innovative structure mentioned in Section (ref), most of the transformer-based models can be adapted and applied in this task for comparison, which are: Vanilla Transformer attention, Reformer Reformer, Informer Informer, Autoformer Autoformer, FEDformer FEDFormer. On the other hand, all the LSTM-based models are compared in this task as well, which are: canonical LSTM lstm, DLSTM, DeepLOB DeepLOB, DeepLOB-Seq2Seq LOBs2s, DeepLOB-Attention LOBs2s. Besides these models, a simple MLP model is built as a baseline. The implementation of the Transformer-based models are based on the code repository mentioned above. The implementation of DeepLOB DeepLOB, DeepLOB-Seq2Seq LOBs2s, DeepLOB-Attention LOBs2s are based on two repositories DeepLOB_repo, s2sLOB_repo. For the DLSTM, it is inspired by code of Dlineardlinear model from its repository dlinear_repo. \\ Training settings The batch size for training is set to 64 and the loss function is changed to Crossentropy loss. Other training settings are the same as the last experiment.
The performance of models is evaluated by classification metrics: accuracy and the mean of precision, recall and F1-score. Result are shown in Table (ref) and Table (ref).
A few outcomes can be observed from the result: \\ 1. DLSTM outperforms all the previous LSTM-based and Transformer-based models. It achieves the highest accuracy, precision, recall and F1 score in all the prediction horizons. This result shows that the time series decomposition structure originating from Autoformer can effectively handle time series, especially when combined with a simple LSTM model. DLSTM is making one step ahead prediction for the mid-price movement, so it will not suffer from error accumulation from the DMS prediction process. \\ 2. DeepLOB-Attention model has the second best result in horizon 50 and 100 (excluding accuracy). DeepLOB-Seq2Seq has the second best result for prediction horizon 20. This result indicates that the encode-decoder structure and attention mechanism can contribute to the prediction performance because the autoregressive process can correlate the mid-price movement from different prediction horizons.\\ 3. The DeepLOB-Attention and DeepLOB-Seq2Seq performance is comparable to DeepLOB but better than DeepLOB, especially in the long prediction horizon. This result accords with the result in the previous paper LOBs2s, which proves the correctness of the result.\\ 4. The Autoformer gets the second-best result in prediction horizon 30. Although it is not the best model, it still means that Autoformer is usable for the time series prediction, and its time decomposition structure is adequate. The shortcoming of Autoformer and the latest FEDformer is that they are huge models compared to LSTM, and they need to be fine-tuned to work well in a specific task. In contrast, LSTM-based models’ sizes are much smaller and do not need much hyper-parameters tuning. More analysis of the efficiency will be discussed in the Section (ref).\\ To summarize, combing the results of this task and previous tasks, LSTM-based models generally have their advantage in financial time series for their robustness and good compatibility. Although the Transformer-based model is large and complicated to tune and requires a long training time, they are still usable for the forecasting task. Furthermore, the research of Transformer-based method in time series prediction is meaningful because the time decomposition method from Autoformer contributes back to the original LSTM model.
In order to show the models and their predictions are practical and useful in trading, a simple trading simulation (backtesting) is designed. Three models with good classification metrics performance are chosen for comparison: DLSTM, DeepLOB DeepLOB, Autoformer Autoformer. Canonical LSTM lstm and Vanilla Transformer attention are used as baselines. The three-day test set is used for this trading simulation. To make a fair comparison among models, the trading simulation follows the simple setup in the previous work DeepLOB: The number of shares pre-trade $\mu$ (volume) is set to one. At each timestamp, the model will predict the price movement ($0$: fall, $1$: stationary, $2$: rise) as a trading signal. When the prediction is $2$, enter the long position, and the position is held until it encounters $0$. The same rule is applied to the short position when the prediction is $0$, and only one direction of position can exist in this simulation trading. A delay is set between the prediction and the order execution to simulate the high-frequency trading latency. For example, assume the model generates a prediction $2$ at time $t$, $\mu$ shares will be bought at time $t+5$.
\\ Several assumptions are made for the simulation trading:\\ 1) Since the trading product is in cryptocurrency exchange, the trading volume is considerable sufficient in the market, which means the simulated trades will not have a market impact.\\ 2) The focus of this part of the experiment is to show the practicality of the prediction result and make relative comparisons among models instead of inventing a fully developed high-frequency trading strategy. Industrial HFT trading strategies usually require the combination of different prediction signals and precise entry exit rules DeepLOB. For simplicity, the order is assumed to be executed at the mid-price without transaction cost.
\\ As displayed in Table (ref) and Figure (ref), each model’s profitability is presented. The performance of simulated trading is evaluated by cumulative price return (CPR) and the Annualized Sharpe Ratio (SR). The CPR is formulated by:
where $s$ is the trading position, which is $1$ for long position and $-1$ for short position. $\mu$ is the number of shares.\\ And the Sharpe Ratio is calculated by:
The value of annualized SR is enormous because the assumptions mentioned above are not realistic for practical trading.\\ Based on the results, LSTM based-model’s performance in simulated trading is generally better than Transformer-based model. The canonical LSTM model achieves highest CPR and SR in prediction horizon 20 and 30 and DeepLOB has the best performance in prediction horizon 50. For DLSTM, it has comparable performance to canonical LSTM and DeepLOB model. This result shows that the prediction result from LSTM-based models are robust and practical for trading. Autoformer’s CPR is the lowest in prediction horizon 20 and 30. The state-of-the-art Autoformer sometimes has even worse performance than the vanilla Transformer in simulated trading, although it obtains a better classification metrics. To summarize, LSTM-based models are relatively the better models for electronic trading.
In the real-world market, all operations, including buying or selling, need a commission fee, and sometimes the transaction cost might outweight the return. This section will introduce a hypothetical transaction cost of $0.002\%$ to further compare the robustness among different models. The results are shown in Table (ref) and Figure (ref).
From the table, DLSTM has the highest CPRs and SRs for all the prediction horizons, outperforming all other models. This shows DLSTM's strong profitability and robustness against the risk brought by the transaction cost. LSTM-based methods' performance is generally better than Transformer-based methods. Canonical LSTM and DeepLOB achieve the second-best CPRs and SRs in different prediction horizons. This indicates that the LSTM-based model's prediction results are more practical and effective in electronic trading. Interestingly, Transformer-based models' performance drops significantly under the transaction cost. The state-of-the-art Autoformer produces even less profit than vanilla Transformer, yielding negative CPRs and SRs in prediction horizon 50 and 100, although its prediction classification metrics is better than Transformer.\\
To further investigate the impact of transaction cost on simulated trading. An experiment is extended to see how the CPR and SR change as the transaction cost increase. The experiment on CPR is done on the three-day test set from $2022.07.12$ (inclusive) to $2022.07.14$ (inclusive) as mentioned in Section (ref). The experiment on SR has a longer backtesting period ranging from $2022.07.13$ (inclusive) to $2022.07.24$ (inclusive) because annualized Sharpe Ratio is calculated based on daily CPR, so using a longer backtesting period can produce a more accurate SR. The experiment results are shown in Figure (ref) and Figure (ref).
\\In terms of the CPR, all models’ CPR decreases as the transaction cost increases. For the prediction horizon $20$ and $30$, DLSTM still outperforms other models when the transaction cost increases. All the models generate comparable CPR except the Autoformer in the prediction horizon $50$ and $100$. Autoformer’s CPR is the lowest in all prediction horizons for most transaction cost settings. Regarding the SR, all models’ SR decrease as the transaction cost increases. However, DLSTM maintains a higher SR than other models as the transaction cost increases. At the same time, for the Autoformer, its SR drops significantly and even becomes the lowest as the transaction cost increases. Overall, DLSTM keeps its profitability and robustness for different transaction costs, while the Transformer-based method’s performance can be largely affected by the transaction cost. This result further indicates that the LSTM-based method is superior for electronic trading.
The efficiency comparison of Transformer-based and LSTM-based models on price movement prediction is shown in Table (ref). The comparison is separated into two parts, the left panel is the practical efficiency, and the right is the theoretical efficiency. The latest transformer-based models have a focus on lowering the time and memory complexity. Autoformer and FEDformer claim to achieve $O(L)$ time and memory complexity in theory. However, their actual inference time and memory consumption are higher than the vanilla transformer models because of their complex design. For the training process, it usually takes more than $12$ hours to train an Autoformer and FEDformer model, even with the cutting-edge GPU device (e.g., 24GB NVIDIA RTX 3090 GPU is used here), which is not efficient to retrain the model on new data. The researchers should reconsider the focus of the Transformer-based model on time series application. The time and memory complexity is not a big threshold for the vanilla Transformer, where its inference speed and memory consumption is acceptable depending on today’s computing power. \\
The LSTM-based model has higher efficiency than the Transformer-based model for its low inference time and small model size, where its theoretical efficiency corresponds to its practical efficiency. This gives the LSTM-based model advantage in high-frequency trading, which requires fast execution speed. This also again emphasizes that LSTM-based models are the better model in electronic trading.
This study systematically compares LSTM-based and Transformer-based models among three financial time series prediction tasks based on cryptocurrency LOB data. The first task is to predict the LOB mid-price. FEDformer and Autoformer have less error than other models, and LSTM is still a strong model that surpasses Informer, Reformer and vanilla Transformer. Although the mid-price prediction error is low, the quality of the mid-price prediction result is far from sufficient for practical use in high-frequency trading. The second task is to predict LOB mid-price difference. LSTM-based methods show their robustness in time series prediction and perform better than Transformer-based models, which reach the highest $11.5\%$ $R^{2}$ in around $10$ prediction steps. State-of-the-art Autoformer and FEDformer are limited in this task because their time decomposition architecture can not handle the difference sequence. However, in a separate study Barez2023, it was shown that custom transformer configurations can outperform the standard transformers.\\
The last task is to predict the LOB mid-price movement. New architecture for the Transformer-based model is designed for adapting the classification task. A new DLSTM model is proposed combining the merits of LSTM and time decomposition architecture from Autoformer. DLSTM outperforms all other models in classification metrics, and Autoformer shows comparable performance to LSTM-based models. A simple trading simulation is done to verify the practicality of the prediction. LSTM-based models have overall better performance than Transformer-based models and DLSTM model beats all other models under the transaction cost.\\
In conclusion, based on all the experiments on three different tasks, the Transformer-based model can only outperform LSTM-based models by a large margin in terms of the limited metrics for mid-price prediction. In comparison, the LSTM-based model is still dominant in the later two tasks, so LSTM-based models are generally the better model in financial time series prediction for electronic trading.\\
For future research, applying LSTM-based and Transformer-based models in Deep Reinforcement Learning (DRL) can be a proper direction. A complete high-frequency trading strategy usually requires the combination of different prediction signals and needs an experienced trader to control the take-profit and stop-loss. In this case, using DRL to generate the optimal trading strategy directly can get us one step closer to the actual trading.