Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
16,246 characters · 19 sections · 20 citation commands
My Publication Title --- Multiple Authors
Time series forecasting is challenging, especially in the financial industry pedersen2019efficiently. It involves statistically understanding complex linear and non-linear interactions within historical data to predict the future. In the financial industry, common applications for forecasting include predicting buy/sell or positive/negative price changes for company stocks traded on the market. Traditional statistical approaches commonly adapt linear regression, exponential smoothing holt2004forecasting, winters1960forecasting, gardner1985forecasting and autoregression models makridakis2020m4. With the advances in deep learning, recent works are heavily invested in ensemble models and sequence-to-sequence modeling such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) hochreiter1997long. In Computer Vision domain, Convolutional Neural Networks (CNN) ren2015faster, ronneberger2015u have shown prominence in learning local patterns which are suitable for modeling short-term dependencies, although not suitable for modeling long-term dependencies due to limited receptive field. Most recently, Transformers vaswani2017attention, have shown great success in Natural Language Processing (NLP) devlin2018bert, brown2020language, smith2022using domain, achieving superior performance on long-term dependencies modeling compared to LSTM.
Our contributions is the following: we leverage the advantages of CNNs and Transformers to model short-term and long-term dependencies in financial time series, as shown in Figure (ref). In our experiments, we show the advantage of the proposed approach on intraday stock price prediction of S&P 500 constituents, outperforming statistical methods including Autoregressive Integrated Moving Average (ARIMA) and Exponential Moving Average (EMA) and a state-of-the-art deep learning-based autoregressive model DeepAR salinas2020deepar.
Typical forecasting techniques in the literature utilize statistical tools, such as, exponential smoothing (ETS) holt2004forecasting, winters1960forecasting, gardner1985forecasting and autoregressive integrated moving average (ARIMA) makridakis2020m4, on numerical time series data for making one-step-ahead predictions. These predictions are then recursively fed into the future inputs to obtain multi-step forecasts. Multi-horizon forecasting methods such as taieb2010multiple, marcellino2006comparison directly generate simultaneous predictions for multiple pre-defined future time steps.
Machine learning (ML) approaches have shown to improve performance by addressing high-dimensional and non-linear feature interactions in a model-free way. These methods include tree-based algorithms, ensemble methods, neural network, autoregression and recurrent neural networks hastie2001elements. More recent works have applied Deep learning (DL) methods on numeric time series data bao2017deep, gensler2016deep, romeu2015stacked, sagheer2019time, Sutskever2014sequence. DL automates the process of feature extraction and eliminates the need for domain expertise.
Since the introduction of transformers vaswani2017attention, they have become the state of the art model to improve the performance of NLP applications. The commonly used approach is to pre-train on a large dataset and then fine-tune on a smaller task-specific dataset devlin2018bert. Transformers leverage from multi-headed self-attention and replace the recurrent layers most commonly used in encoder-decoder architectures. In contrast to RNNs and LSTMs where the input data is sequentially processed, transformers bypass the recursion to ingest all inputs at once; thus, transformers allow for parallel computations to reduce training time and do not suffer from long-term memory dependency issues.
The remainder of the paper is organized as follows. We discuss our proposed methodology for time series modeling Further, we discuss our benchmark baseline models along with the performance evaluation metrics and report the experimental results. Finally, we highlight some concluding remarks and future directions for this study.
Our proposed method is called CNN and Transformer based time \underline{\textbf{s}}eries modeling (CTTS) as shown in overview Figure (ref).
We used standard min-max scaling to standardize each input time series within $[0, 1]$. Given a raw stock prices time series $\textbf{x}$, the standardized time series were calculated as
This $\textbf{x}_{standardized}$ is then passed into our model to learn the sign of the change in the very next time step, which is defined as our prediction target.
As shown in Figure (ref), we use 1D CNN kernels to convolute through a time series, projecting each local window into an embedding vector that we call a token. Each token carries the short-term patterns of the time series. we then add positional embedding vaswani2017attention to the token and pass that through a Transformer model to learn the long-term dependencies between these tokens. The transformer model outputs a latent embedding vector of the time series, which is then passed through a Multilayer Perceptron (MLP) with softmax activation in the end to generate sign classification outputs. The output is in the form of probability for each class (up, down, flat), where the probability for all 3 classes sum up to 1.
In our experiments, we benchmarked CTTS - our proposed method against 4 methods. 1) DeepAR - a state-of-the-art autoregressive recurrent neural networks-based time series forecasting method, 2) AutoRegressive Integrated Moving Average (ARIMA), 3) Exponential Moving Average (EMA) and 4) naive constant class predictions. We benchmarked the performance using sign prediction accuracy and thresholded version of the sign prediction accuracy (discussed later) using our model prediction probabilities. We ran our experiments on a Linux machine with 8 16GB NVIDIA T4 Tensor Core GPUs, and using PyTorch v1.0.0 DL platform in Python 3.6. In all models, we set a fixed random seed for reproducible results.
We used the intraday stock prices of S&P 500 constituent stocks obtained from licensed Bloomberg data service bloomberg. The data was sampled at 1 minute interval for the year 2019 (52 weeks, each week has 5 trading days). For every stock, we sampled 7 time series for each week. Data from the first three quarters (weeks 1 to 39) were used for training and validation. Data was randomly split and 80% was used for training and the remaining 20% for validation. We had around 507K training and 117K validation samples. Data from the last quarter (weeks 40 to 52) was used for testing, totaling 209K time series. For each time series, the first 80 time steps (input) were used to forecast the sign of price change at the 81st time step (target). The overall test set performance is denoted by the aggregate for the evaluation metrics over the test set.
In our experiments, we used cross entropy loss, and AdamW loshchilov2017decoupled optimizer with batch size of 64 and max epoch of 100. we used CNN kernel size of 16 and stride 8. Transformer with depth 4 and 4 self-attention heads, with an embedding dimension of 128, and a drop rate of 0.3 to prevent overfitting.
DeepAR forecasting algorithm is a supervised learning algorithm for forecasting 1D time series using autoregressive recurrent neural networks. DeepAR benefits from training a single model jointly over all of the time series. We used a batch size of 128, Adam optimizer, and the normal distribution loss. The model was trained for a maximum of 300 epochs, with early stopping mechanism set to a patience of 15. The base learning rate was 1e-3, adjusted by learning rate scheduler with decay factor 0.1 and a patience of 5. We also used dropout with probability 0.1. DeepAR generated multiple (200) samples of the prediction target for each time series and we defined the prediction probability over the three classes as the proportion of samples predicted per class.
Autoregressive Integrated Moving Average (ARIMA) models capture autocorrelations in the data using a combination approach of autoregressive model, moving average model, and differencing wilks2011statistical. We compared the continuous valued ARIMA forecasts with the last known price in the input data to define the predicted sign, and the corresponding prediction probability was defined as the percentage of the absolute delta between the predicted and the last known price with respect to the standard deviation of the past 80 data points, capped by 1.
An exponential moving average (EMA) is a type of moving average that places a greater weight and significance on the most recent data points. We used the "estimated" initialization method, which treats the initial values like parameters, and chooses them to minimize the sum of squared errors. Similar to ARIMA, we delta between the predicted and the last known price to define the predicted sign, and the corresponding prediction probability.
We tested the three naive baselines where we always predict a constant sign i.e., either the predicted price always goes up, down, or remains flat. The prediction probabilities for these were set to 1 for the winning class, and 0 for the remaining two classes.
We evaluated all our models on 3-class [class 1-price goes up, class 2-price goes down, class 3-price remains flat] and 2-class [class 1-price goes up or remains flat, class 2-price goes down]. We used the averaged sign prediction accuracy over all test samples to evaluate our models. Higher accuracy implies more accurate forecasts.
Further, we evaluate a thresholded version of the sign accuracy, where we defined our threshold as the 75$^{th}$ percentile of all predicted probabilities of dominating classes. We retain only the samples that exceed the threshold and compute the sign prediction accuracy over these retained samples.
We summarize the quantitative benchmark results from the sign accuracy in Table (ref). We show the sign accuracy for both 2-class and 3-class tasks as explained above. Note that random guess in 2-class task leads to 50% of accuracy, and random guess in 3-class task leads to 33% of accuracy. As shown in the 2-class and 3-class columns of Table (ref), CTTS outperformed all baseline methods in both cases. This demonstrates the benefit of combining CNNs and transformers for time series forecasting.
Further, we evaluate a thresholded version of the sign accuracy, where we defined our threshold as the 75th percentile of all predicted probabilities of dominating classes. We retain only the samples that exceed the threshold and compute the sign prediction accuracy over these retained samples. As shown in 2-class$^{*}$ and 3-class$^*$ columns of Table (ref), the accuracy after thresholding over the prediction probability has increased for all methods. In addition, the gain of accuracy increase is the most for our proposed method CTTS. This shows that the class probabilities that CTTS outputs are reliable. Specifically, high-confidence predictions from CTTS are often correct, thus thresholding filters out erroneous low-confidence predictions, leading to a boost in the sign accuracy.
Another highlight is the significant gap that CTTS has achieved for thresholded 3-class$^*$ accuracy compared to other baselines. This can be harnessed in the future for trading decision-making. For example, a straightforward trading decision can be buy/sell/hold stocks when the predicted class is up/down/flat, respectively. Given that CTTS's predicted probablitlies are reliable as discussed earlier, the amount of stock shares to buy/sell/hold can depend on the predicted probability, the higher the probability, the more shares to consider.
In this paper, we tackle the challenging problem of time series forecasting of stock prices in the financial domain. In this paper, we demonstrated the combined power of CNN and Transformer to model both short-term and long-term dependencies within a time series. In our experiments over intraday stock price of S&P 500 constituents in year 2019, we demonstrated the success of the proposed method CTTS in comparison to ARIMA, EMA, and DeepAR, as well as the potential for using this method for downstream trading decisions in the future.