Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
36,595 characters · 13 sections · 22 citation commands
Modeling and Forecasting Intraday Market Returns: a Machine Learning Approach
\doublespacing
The recent advances in high-frequency data estimation are associated not only to the technological development and growing processing capacity of big data, but also to the interest in understanding and predicting the behavior of variables in shorter time spaces. Machine Learning methods have been developed in parallel as increasingly accurate tools for estimating and predicting high-dimensional data. Both these fields of study can be easily accommodated into the economic and financial data environment. The adaptation of these methods to incorporate and update information over time allowed the development of robust predictive methods, progressively more relevant in time series analysis.
This paper examines the intrinsic relation between market returns and volatility measures, besides lagged returns themselves. Using minute-by-minute intraday data, we find that the CBOE Volatility Index (VIX) can be a strong predictor for the S&P 500 ETF (SPY) in high-frequency, especially through machine learning models. For each day in the sample, we implement a 30 minute rolling-window estimation procedure to forecast the subsequent minute, totaling 340 estimated market returns in a day. This framework is based on the estimation scheme of Chinco2019 for a cross-section of stock returns. We use different approaches to estimate these returns, from ordinary least squares (OLS) regression benchmarks to more sophisticated methods, focusing on nonlinear models. To build minute-by-minute machine learning models, we relied on the work of MMM2021.
Particularly, we apply Long-Short-Term Memory (LSTM) neural networks sHjS1997 and Random Forest (RF) models breiman2001random to estimate multiple intraday rolling windows, using a range of predictive variables in different settings. LSTM models are a variant of Recurrent Neural Networks (RNNs), that differ from the standard neural networks on the ability to remember the previous states in time. This method is broadly used in applications related to weather data, and speech and writing recognition. For economic and financial data in particular, we want this time dependence to be present, what makes the LSTM such an attractive model, specially due to its ability to remember what matters and forget what is irrelevant to the model. For a nice application of LSTMs in asset pricing see pelger19.
Introduced by breiman2001random, the Random Forest is an ensemble method, which means it combines several simpler models, producing an optimal improved aggregate version in the end. In this case, the base models are classification or regression trees, a nonparametric method based on the mechanism of recursive partitioning of the space of covariates. The average of the estimates generated by each tree are used to build the final forecast, known as Random Forest. RF models have been shown to be a very competitive forecasting tool. See, for example, mcMgVaVeZ2019.
The choice to analyze volatility measures as potential predictors is an important issue in the economic literature. Among others, Corsi2009 and PattonSheppard2015 established a whole literature on realized volatility ($RV$), defined as the daily sum of the cumulative squared returns during business hours of a trading day. The forecasting methodology in this field of research uses predominantly autoregressive structures endowed with long memory to predict realized volatilities. McAleerMedeiros2008 provide a extensive review of theoretical developments and empirical applications concerning realized volatility.
Time-series properties of the VIX index are addressed by works like FMS2014, as well as its positive contemporaneous link with the volume of the S&P 500 index. Martin2017 and MartinWagner2019 explore the predictability of market and stock returns defining an associated volatility index named SVIX that provides a bound on the equity premium perceived by the investor. By associating implied variance (VIX$^2$) and realized variances measures, BTZ2009 introduce the variance risk premium (VRP) as the difference between those two variables, that can be used to explain the aggregate stock market returns. HoerovaBekaert2014 extend these results by decomposing VIX$^2$ into a conditional variance of the stock market and the equity variance premium. The authors conclude that the variance premium is a significant predictor of stock returns.
This plethora of applications for volatility measures and machine learning methods raises our interest in combining these fields of study in a high frequency environment to predict future market returns. As will be discussed throughout the paper, our outputs indicate that machine learning models as the LSTM may improve the performance of benchmark linear regression models in a minute-by-minute framework. Furthermore, if we choose the right regressors, particularly the VIX in our study, the predictive ability becomes even higher. On the other hand, Random Forests did not bring improvements to our estimations, when compared to the established benchmarks.
The paper is organized as follows. Section (ref) reviews the methodology to be implemented in the machine learning framework. The high-frequency data details are presented in Section (ref). Section (ref) discusses the employment of different predictive models, as well as the benchmarks to our exercise. The results of intraday estimations and aggregate outputs are described in Section (ref). Finally, we conclude the paper in Section (ref).
Define the following high-frequency forecasting model, where $t$ is the day, and $m$ the intraday minute:
where $\boldsymbol{X}_{t,m} := (Y_{t,m-1},\dots,Y_{t,m-p},\boldsymbol{Z}'_{t,m},\dots,\boldsymbol{Z}'_{t,m-r})'$ is a $n$-dimensional vector of predictors, with $p \geq 1$ and $r \geq 0$. $\mathcal{F}_h: \mathbb{R}^n \rightarrow \mathbb{R}$ is an unknown measurable function and $U_{t,m+h} := Y_{t,m+h} - \mathcal{F}_h(\boldsymbol{X}_{t,m})$ has zero mean and finite variance.
For any chosen model framework and forecasting horizon $h=1,\dots,H$, we want to define the target function $\mathcal{F}_h$, to be estimated from the available data set. This function can be an ensemble of multiple models, and it can vary according to the horizon $h$.
MMM2021 expatiate in detail about the different machine learning methods, as well as their advances and applications in time series data environment. The authors discuss both linear and nonlinear models, of which we will focus on the last. The reason for this is that sometimes the linearity hypothesis may not encompass all the characteristics of variables such as volatility measures, specially in a high-frequency environment. In this case, we look for alternatives in the universe of models in statistical learning literature.
Recurrent Neural Networks (RNNs) are neural networks that allow for feedback among the hidden layers. RNNs can use their internal state (memory) to process sequences of inputs. A generic RNN can be written as \[
\] where $\widehat{Y}_{t,m+h|m}$ is the prediction of $Y_{t,m+h}$ given observations only up to minute $m$ at day $t$, $\boldsymbol f$ and $g$ are functions to be defined and $\boldsymbol H_{t,m}$ is what we call the (hidden) state. From a time-series perspective, RNNs can be see as a kind of nonlinear state-space model.
RNNs can remember the order that the inputs appear through its hidden state (memory) and they can also model sequences of data so that each sample can be assumed to be dependent on previous ones. However, RNNs are hard to be estimated as they suffer from the vanishing/exploding gradient problem. Fortunately, there is a solution to the problem proposed by sHjS1997: the Long-Short-Term Memory (LSTM) network . Figure (ref) shows the architecture of a typical LSTM layer. A LSTM network can be composed of several layers. In the figure, red circles indicate logistic activation functions, while blue circles represent hyperbolic tangent activation. The symbols “X” and “+” represent, respectively, the element-wise multiplication and sum operations. The RNN layer is composed of several blocks: the cell state and the forget, input, and ouput gates. The cell state introduces a bit of memory to the LSTM so it can “remember” the past. LSTM learns to keep only relevant information to make predictions, and forget non relevant data. The forget gate tells which information to throw away from the cell state. The output gate provides the activation to the final output of the LSTM block at day $t$ and minute $m$. Usually, the dimension of the hidden state ($\boldsymbol{H}_{t,m}$) is associated with the number of hidden neurons.
Algorithm (ref) describes how the LSTM cell works. $\boldsymbol f_{t,m}$ represents the output of the forget gate. It is a combination of the previous hidden-state ($\boldsymbol H_{t,m-1}$) with the new information ($\boldsymbol X_{t,m}$). Note that $\boldsymbol f_{t,m}\in[0,1]$ and it attenuates the signal coming com $\boldsymbol c_{t,m-1}$. The input and output gates have similar structure. Their goal is to filter the “relevant” information from the previous minute as well as from the new input. $\boldsymbol p_{t,m}$ scales the combination of inputs and previous information. This signal will be then combined with the output of the input gate ($\boldsymbol i_{t,m}$). The new hidden state will be an attenuation of the signal coming from the output gate. The prediction is a linear combination of hidden states. Figure (ref) illustrates how the information flows in a LSTM cell.
A regression tree is a nonparametric model that approximates an unknown nonlinear function with local predictions using recursive partitioning of the space of the explanatory variables (predictors).
The idea of regression trees is to approximate $\mathcal{F}_h(\boldsymbol{X}_{t,m})$ in (ref) by \[ h_D(\boldsymbol{X}_{t,m}) = \sum_{j=1}^{J} \beta_j I_j(\boldsymbol{X}_{t,m}),\quadwhere\quad I_k(\boldsymbol{X}_{t,m})=
\] From the above expression, it becomes clear that the approximation of $\mathcal{F}_h(\cdot)$ is equivalent to a linear regression on $J$ dummy variables, where $I_j(\boldsymbol{X}_{t,m})$ is a product of indicator functions.
Let $J$ and $N$ be, respectively, the number of terminal nodes (regions, leaves) and parent nodes. Different regions are denoted as $\mathcal{R}_1,\ldots,\mathcal{R}_J$. The root node at position $0$. The parent node at position $j$ has two split (child) nodes at positions $2j+1$ and $2j+2$. Each parent node has a threshold (split) variable associated, $X_{s_jt}$, where $s_j\in\mathbb{S}=\{1,2,\ldots,p\}$. Define $\mathbb{J}$ and $\mathbb{T}$ as the sets of parent and terminal nodes, respectively. Figure (ref) gives an example. In the example, the parent nodes are $\mathbb{J}=\{0,2,5\}$ and the terminal nodes are $\mathbb{T}=\{1,6,11,12\}$.
Therefore, we can write the approximating model as
where
\[ I(X_{s_j,t,m};c_j)=
\] \[ n_{i,j}=
\] $\mathbb{J}_i$: indexes of parent nodes included in the path to leaf $i$. $\boldsymbol{\theta}_i=\{c_k\}$ such that $k\in\mathbb{J}_i$, $i\in\mathbb{T}$ and $\sum_{j\in\mathbb{J}}B_{\mathbb{J}i}\left(\boldsymbol{X}_{t,m};\boldsymbol{\theta}_j\right)=1$.
Random Forest (RF) is a collection of regression trees, each specified in a bootstrap sample of the original data. The method was originally proposed by breiman2001random. Since we are dealing with time series, we use a block bootstrap. Suppose there are $B$ bootstrap samples. For each sample $b$, $b=1,\ldots,B$, a tree with $K_b$ regions is estimated for a randomly selected subset of the original regressors. $K_b$ is determined in order to leave a minimum number of observations in each region. The final forecast is the average of the forecasts of each tree applied to the original data: \[ \widehat{Y}_{t,m+h|m}=\frac{1}{B}\sum_{b=1}^B\left[\sum_{i=1}^{\mathbb{T}_b} \widehat{\beta}_{i,b} B_{\mathbb{J}i,b}(\boldsymbol{X}_{t,m}; \widehat{\boldsymbol{\theta}}_{i,b})\right]. \]
Our dataset consists of 1 minute frequency data for both the S&P 500 ETF (SPY) and the CBOE Volatility Index (VIX), gathering a total of 3,000 business days between January 2005 and December 2016. We filter the data within each day to the observations between 09:40 and 15:50 (inclusive).
We calculate log-returns of SPY for each minute of day $t$, using a rolling five-minute return scheme, as follows:
where $r_{t,m_2:m_1}$ represents the log difference in prices computed between minutes $m_1$ and $m_2$, and $P_{t,m}$ is the price of SPY at day $t$ and minute $m$. We explore the VIX index variable in level, and implement a transformation of the annualized VIX to an intraday minute variable, in order to facilitate the interpretation of our results\footnote{$VIX_{\text{Intraday}} = VIX_{\text{Annual}} \times (1/\sqrt{1440}) \times (1/\sqrt{252})$: 252 business days in a year and 1440 minutes in a day.}. Using this data set, we also compute the one-minute squared version VIX$_m^2$ and first difference $\Delta$VIX$_{m}$.
Inspired by BTZ2009, we generate a measure of what could be a high-frequency analogous to the variance risk premium, as the difference between the squared one-minute SPY return and the VIX$^2$ at minute $m$:
The following estimations will be centered on models that use volatility measures as inputs, particularly the VIX. Despite presenting additional configurations and benchmark models, we focus on understanding how the minute-by-minute VIX relates to very high-frequency market returns, as well as its overall predictive capability across the models.
To illustrate the common behavior of market return and the volatility measure over the years, Figure (ref) displays the contemporary relation between the average minute-by-minute SPY log-return and VIX for each day of the sample. The darker the dots in the figure, the more recent the analyzed sample. It is noticeable that the relationship between both variables becomes less disperse over time, more concentrated in recent years, exhibiting lower returns for lower levels of volatility. Additionally, we present the summary statistics for minute-by-minute SPY log-returns and the annualized VIX in Table (ref), which shows that both variables are skewed right and leptokurtic. As expected, market returns and the VIX are negatively correlated.
Despite the clear intrinsic relationship between market returns and near-term volatility implied by stock index option prices, we want to investigate the forecasting potential of SPY returns in high-frequency using volatility measures. The subsequent section presents the fundamental estimation framework adopted in this paper, based on a minute-by-minute rolling window mechanism. This approach is similar to the one developed by Chinco2019, and allows the researcher to understand very short-term effects over intraday periodicity variables.
To estimate our models, we implement a 30-minute rolling window estimation scheme for very high-frequency predictions. For each minute $m$ and day $t$, we use the previous non-overlapping thirty minute window to forecast the one-minute-ahead five-minute log-return of SPY. At the end of each day $t$, we run a total of 340 estimations between 09:40 and 15:50\footnote{We chose to avoid the first and last ten minutes of each business day in our estimations because these are the moments that usually contain a greater amount of missing values and/or repeated values.}.
It is important to point out that our high frequency analysis uses non-overlapping input and output variables. This means that for each minute-by-minute log return $r_{t,m_2:m_1}$, we use lagged regressors that do not overlap minutes $m_2$ to $m_1$. For example, if we want to run a regression of the five-minute log return ($r_{t,m-4:m}$) on the lagged $\Delta$VIX, we compute the non-overlapping variable as $\Delta$VIX$_{t,m-6:m-5} =$ VIX$_{t,m-5} -$ VIX$_{t,m-6}$. This strategy prevents from using future information to forecast minute $m+h$ when observations are available only up to minute $m$.
Machine learning algorithms perform better when numerical input variables are scaled. In each window, we perform a standard MinMax scaling to the data, by using the parameters from the train observations to scale both train and test data\footnote{For each window $\omega$ rescale: $\tilde{x}^{train}_{\omega,m} = \frac{x^{train}_{\omega,m} - \min(x^{train}_{\omega})}{\max(x^{train}_{\omega}) - \min(x^{train}_{\omega})}$ and $\tilde{x}^{test}_{\omega,m} = \frac{x^{test}_{\omega,m} - \min(x^{train}_{\omega})}{\max(x^{train}_{\omega}) - \min(x^{train}_{\omega})}$}. This transformation rescales variables into the range $[0,1]$. We implement the standardization to the entire estimation data set, explored in both machine learning and benchmark models, as described in sequence.
The architecture of the LSTM network allows the model to learn and forecast long sequences of data, a particularly useful attribute for time-series analysis. This approach can be put into action by running rolling window regressions in a one-shot multi-step framework for each day $t$. There are many options available to the researcher in terms of network structure and hyper-parameters to be chosen, including the number of layers, hidden states, loss function and optimizer. In our paper, we optimize the hyper-parameter by tuning it according the models performance on random subsamples.
As discussed in Section (ref), Random Forest models (which we will also refer as RF from now on) fit classifying decision trees on data subsamples and employ an averaging approach to enhance the predictive power of the data. This ensemble learning method bootstraps the observations in blocks, using randomly sampled training sets (estimation intraday windows, in our case), controlling for potential overfit of the model. The choice of the number of trees to be used is a critical decision in a Random Forest, once it represents the main tuning engine in the model framework.
For each machine learning model (LSTM and RF), we establish three groups of predictors $\boldsymbol{X}_{t,m}$. The first one is composed exclusively by the lagged volatility variable $\boldsymbol{X}_{t,m} = VIX_{t,m-5}$, while the second contains only the lagged five-minute return $\boldsymbol{X}_{t,m} = r_{t,m-9:m-5}$. We call these models VIX and AR(1), respectively. Finally, the last group gathers all the variables described in Section (ref): \[ \boldsymbol{X}_{t,m} = \left(r_{t,m-9:m-5}, r_{t,m-9:m-5}^2, VIX_{t,m-5}, \Delta VIX_{t,m-6:m-5}, VRP_{t,m-5}\right)', \] which we labeled as the aggregate model.
As benchmarks, we chose different setups for $\boldsymbol{X}_{t,m}$, implementing the same rolling-window approach. In addition to applying a traditional autoregressive model of order 1, we use lagged squared return, VIX, $\Delta$VIX, and $VRP$ as predictors. We estimate the coefficients in each window through standard ordinary least squares (OLS) regressions.
We define the benchmark models as follows:
We use the logic presented in the beginning of this section in order to regress the market returns on non-overlapping lagged variables. The benchmarks OLS-AR(1) and OLS-VIX are particularly important to our analysis, once it can be directly compared to the first two groups of predictors defined in the previous section. From both machine learning and benchmark models, we expect to understand the importance of the lagged log-return, as well as the VIX, in high-frequency forecasting.
This estimation practice was executed in a cloud environment, applying \colorbox{light-gray}{scikit-learn}, \colorbox{light-gray}{tensorflow}, and \colorbox{light-gray}{keras} Python libraries to implement machine learning methods. We employ \colorbox{light-gray}{Python 3.9.4 version} to perform the estimations and \colorbox{light-gray}{R 4.1.2 version} for output analysis. In the next section, we compare the performance metrics for both machine learning and benchmark models. We contrast specifically the regressions over different groups of predictors presented in Section (ref) to the single input OLS models in the set of equations (ref).
Due to the high dimension of the estimated intraday outputs in each model (described in Section (ref)), we summarise our results through two performance metrics widely used in the literature: the daily Root-mean-square error (RMSE) and out-of-sample $R^2$. To obtain the mean and median RMSE, as well as the standard deviation, we calculate the daily measure
and then compute its overall metric throughout the sample. Then, we calculate the same out-of-sample performance metric implemented by WelchGoyal2008 and CampbellThompson2008 for each day $t$
to generate the average $R^2_{OOS}$, its median, and standard deviation across the days. This metric is relevant to introduce a relation between the mean squared error of our model estimate and the naive projection, represented by the in-sample historical mean within each estimation window. If the error of the model's prediction is lower than the error of the naive forecast, the $R^2_{OOS}$ is necessarily positive. The same holds for the opposite case, in which a negative $R^2_{OOS}$ represents a better performance of the historical average model.
To exemplify how each predictive model works throughout a day, Figures (ref) and (ref) illustrate the intraday rolling prediction for a randomly selected date $t$ (January 8, 2007). Each plot represents a machine learning model (LSTM and Random Forest) compared to the actual series, the historical mean based prediction, and the respective VIX or AR(1) benchmark. At each day $t$ calculate the daily performance metrics for each one of those series, and then compute the average, mean, and standard deviation for the entire sample.
The top plot in Figure (ref) presents the LSTM-VIX model (that uses VIX as predictive variable) minute-by-minute predictions, while the bottom one displays the same scenario for the RF-VIX model. Comparing both to the OLS-VIX benchmark, and the naive projection, one can see that the LSTM method tracks the observed series more smoothly, but also more precisely, while the Random Forest follows pretty much the same pattern established by the benchmark model. For this specific date, the $R^2_{OOS}$ for the LSTM, Random Forest, and benchmark models were $21.15\%$, $-37.10\%$, and $6.33\%$, respectively. We can see a clear superiority of the LSTM model, followed by the benchmark and, finally, the Random Forest model.
This pattern is repeated in Figure (ref), in which we perform the same analysis comparing autoregression based models. The minute-by-minute series forecasts follow an akin behavior to that observed in the previous figure. The curves are smoother, but also more precise for LSTM-AR(1) predictions. Even more interesting, the model now presents a positive $R^2_{OOS}$, compared to the negative ones obtained through the RF-AR(1) and OLS-AR(1) models on the same date: $6.93\%$, $-21.87\%$, and $-14.03\%$, respectively. Once again, the Random Forest output suggests a poor performance for this model. Obviously, these results are restricted to the analysis of a random day in the sample. The subsequent tables present and summarize the general outputs obtained by the estimation of the full sample, leading to similar conclusions.
Tables (ref) and (ref) present the average, median, and standard deviation of $R^2_{OOS}$ and RMSE, respectively. These outputs are calculated over the performance metrics, using out-of-sample predictions for each day $t$, as described above\footnote{We trim the outputs in each model to restrict it to be between the 1$^{st}$ and 99$^{th}$ percentiles, excluding outliers in both directions.}. Looking only at the bottom panel of each table, which contains the benchmarks presented in the set of equations (ref), there is a clear hegemony of the OLS-VIX in terms of performance. The mean and median values of the out-of-sample $R^2$ outperform the naive prediction based on the historical mean ($R^2_{OOS}>0$). As expected, this same conclusion can be drawn from the RMSE table, in which the VIX excels all the other variables as a predictor of market returns among the OLS equations.
When we include the LSTM and Random Forest machine learning models in the analysis, we can see critical improvements (and some downgrades) in terms of performance. As one can see, the LSTM-VIX is the best performing model, followed by the OLS-VIX model. Even though the RF-VIX model performs well when compared to other models, it is not able to surpass the benchmark's performance. The LSTM approach for the autoregressive component also improves its performance when compared to the OLS-AR(1) model. Differently from the benchmark outputs, the mean and median $R^2_{OOS}$ values for the LSTM-AR(1) are strictly positive, outperforming the historical mean projection. On the other hand, the RF-AR(1) model presents a drastic deterioration in performance when we use the lagged market return as predictor.
The LSTM and RF aggregate models (including all variables) do not present any improvement in its statistics when compared to the OLS-VIX benchmark, signaling that inserting other volatility measures and lagged market returns on the same model do not provide a better performance, compared to the models presented previously. To illustrate the different distributions of $R^2_{OOS}$ metrics across the days in our sample, Figure (ref) presents the density for both machine learning (in the top row), and benchmark models (in the bottom). As one can see, the LSTM $R^2_{OOS}$ distribution is predominantly left-skewed for every group of predictors, in accordance with the results presented in Tables (ref) and (ref). The RF models distributions present a reasonable performance in general, apart from the RF-AR(1). Still, these models are not able to overcome the OLS-VIX, which stands out as the best performing benchmark.
In a complementary way, Figure (ref) pursues to unravel the relationship between the results obtained by each machine learning model, compared to their respective benchmarks over the years. The vertical axis of each plot represent the LSTM and Random Forest models estimations using as regressor the AR(1) (in the first column of plots) and the VIX (in the second column). The horizontal axis displays the predicted values for the benchmark models. This visualization allows one to identify when machine learning models were able to overcome its benchmarks (the points below the \ang{45} dotted line). The last years in our sample seem to present lower and less disperse estimation errors for all the models combinations. Moreover, the VIX-based models got even better recently, when using both LSTM and RF models. These highlights synthesize the objective of this paper: contribute to the short-term forecasting literature, as well as the implementation of modern forecasting techniques through nonlinear rolling window models in a high-frequency setting.
There is growing interest in understanding how to predict high-frequency stock returns. Our paper introduces an innovative approach to estimate minute-by-minute market return (SPY) forecasts using volatility measures and lagged returns as predictors. In addition to standard benchmark models, we implement nonlinear machine learning methods as an attempt to capture idiosyncrasies of this kind of data, whose properties can differ substantially from variables at lower frequencies.
The outputs obtained in our estimations indicate a preliminary, but also encouraging path to better understand how to predict high-frequency market returns. We focus on neural networks (Long-Short-Term Memory) and tree-based (Random Forests) models to estimate multiple intraday rolling windows, using different regressors configurations. Models that use the Cboe Volatility Index (VIX) as predictor stand out, indicating that the VIX is a strong candidate predictor, when compared to other variables. The precision of the forecasts obtained using the OLS benchmark model gets even higher when we apply the LSTM to predict market returns using exclusively the VIX. Although the random forest model is not able to outperform the results obtained by the previous models (eventually even worsening it in some cases), the LSTM proved to be very promising in terms of predictive power, for both VIX and past market returns as regressors.