Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
126,531 characters · 21 sections · 75 citation commands
Recurrent Conditional Heteroskedasticity
Financial time series, e.g. currency exchange rates or stock returns, exhibit stylized facts such as volatility clustering and leverage effects. The volatility clustering phenomenon of financial time series refers to the observation that “large changes tend to be followed by large changes, of either sign, and small changes tend to be followed by small changes" Benoit1967. This behavior implies that the volatilities, i.e. the conditional standard deviations, of financial returns are observed to be highly autocorrelated and exhibit periods of both low and high volatility. The leverage effects exhibited in financial time series, on the other hand, relate to the observation that the negative and positive past returns have asymmetric effects on the volatility Black:1976. More specifically, the current volatility tends to be larger following a previous negative shock, i.e. a return below its expected value, than a positive one of the same absolute value. The volatility clustering and leverage imply that the volatility of financial assets changes over time, i.e., being heteroskedastic.
Time-varying volatility is a key assumption in the volatility modeling literature. A large number of volatility models have been developed since Engle:1982 proposed the Autoregressive Conditional Heteroskedastic (ARCH) model, which allows the conditional variance, i.e, the squared volatility, to change over time as a {\it deterministic} function of the historical shocks while leaving the unconditional variance unchanged. The most successful extension of the ARCH model is the Generalized Autoregressive Conditional Heteroskedasticity (GARCH) model of Bollerslev1986; it models the current conditional variance as a linear function of the past conditional variances and squared returns. The GARCH model together with the ARCH model and their variants define a class of models, often referred to as the conditional heteroskedastic or GARCH-type models, which use deterministic functions of historical information, e.g. the past returns and past conditional variances, to model the current conditional variance and are able to capture the stylized facts of financial time series. Another notable line of research in the volatility modeling literature focuses on the stochastic volatility (SV) model Taylor:1986 and its variants, which formulate the conditional variance using latent {\it stochastic} processes that do not directly involve the past returns. Our article is, however, interested in GARCH-type models which are probably more popular in the volatility modeling literature because it is much easier to estimate GARCH-type models than SV-type alternatives. See Koopman:2016 for a comprehensive comparison between the GARCH-type and SV-type models.
Recurrent neural networks (RNNs) in the Deep Learning literature are successfully used in a large number of industrial-level applications; e.g. language translation, image captioning, speech synthesis. RNNs are well-known for their ability to efficiently capture the long-range memory and non-linear serial dependence existing within various types of sequential data, and are considered as the state-of-the-art models for many sequence learning problems Lipton:2018. See Goodfellow2016 for a comprehensive discussion of various types of neural network models (NNs) and their broad range of applications. The recent success of RNNs has motivated econometricians to incorporate RNNs and other deep learning models into their econometric models. There is a large amount of research along this line, with a focus on modelling the mean rather than the variance of financial asset returns; see, e.g., ZHANG:1998,ZHANG:2003. Leveraging the power of deep learning in volatility modelling is still somewhat overlooked in the econometrics literature. Donaldson:1997 are one of the first to propose a NN-GARCH model that adds a feedforward NN component into the conditional variance of the GJR model Glosten:1993. Their in-sample and out-of-sample results on four stock markets suggest that the NN-GARCH model is preferred to several benchmark GARCH-type models. Roh:2007 proposes NN-based volatility models that first estimate the conditional variance by an econometric volatility model, then use these estimates as inputs to a feedforward neural network, which then non-linearly transforms these inputs to output the final estimate of the conditional variance. Kim:2018 extend this idea by using the outputs from several GARCH-type models rather than a single one as the inputs to a recurrent neural network; see also Luo:2018. LIU201999 uses the Long Short-term Memory, a sophisticated RNN technique, for volatility modelling and reports its improved prediction over GARCH in two datasets. In general, these hybrid models that combine neural networks and financial econometric models are empirically superior to several econometric models and plain neural network models, in terms of predictive performance. However, these models are often engineering-oriented and ignore the interpretable aspects of econometrics volatility models and the important stylized facts of financial time series. Some of these existing models use feedforward NNs rather than RNNs and thus might ignore the time effects in time series data. Their design is also rather inflexible in the sense that they combine a particular econometric model with a particular NN. It is important to design a more flexible framework that is easy to adapt to advances in both the deep learning and volatility modelling literature.
GARCH-type models are generally simple yet highly interpretable in the sense that they are designed to explain the distinct behaviors of financial time series. Any new volatility model should not overlook this interpretability of the traditional econometric models. This paper proposes a new class of models, called the REcurrent Conditional Heteroskedastic models (RECH), that not only improve the forecast performance of GARCH-type models, by leveraging the capability of learning non-linearity and long-range dependence of RNNs, but also place significant emphasis on the interpretation of the estimated volatility, by inheriting the well-established features of GARCH-type models. We now briefly explain why RECH models fit well within the volatility modeling literature. First, similarly to GARCH-type models, the conditional variances in RECH models are a deterministic function of the past values; hence it is easy to estimate RECH models as their likelihood functions can be evaluated analytically. Second, RECH models are still able to explain the stylized facts of the underlying volatility dynamics. Third, by inheriting the predictive power from deep learning techniques, RECH models often forecast better. Fourth, the highly flexible design of RECH models makes it easy to adopt advances in both the deep learning and volatility modeling literatures, allowing it to be used in a wide range of applications in financial time series analysis. A Matlab software package implementing Bayesian estimation and inference for RECH models together with the examples reported in this paper is available at \url{https://github.com/vbayeslab}.
The rest of the article is organized as follows. Section (ref) briefly reviews the GARCH model and its variants. Section (ref) briefly reviews different types of neural networks and proposes RECH models. Section (ref) discusses Bayesian estimation and inference for RECH models. Section (ref) presents the simulation study and applies RECH models to analyze four benchmark financial datasets. Section (ref) concludes. The Appendix gives implementation details and further empirical results.
Let $y=\{y_t,\ t=1,...,T\}$ be a time series of demeaned returns and $\mathcal F_t$ be the $\sigma$-field of the information up to time $t$. Conditional heteroskedastic models represent the conditional variance $\sigma_t^2:=\text{\rm Var}(y_t|\mathcal F_{t-1})$ of the observation $y_t$ as a deterministic function of the observations and the conditional variances in the previous time steps. Mathematically, these models are expressed as:
$f(\cdot)$ is a positive deterministic function parameterized by the vector of unknown parameters $\theta$; $p,q \ge 0$ are the number of lags of $\sigma_t^2$ and $y_t$ respectively; and $\omega$ is a non-negative constant ensuring that the conditional variance $\sigma^2_t$ is positive. The shocks $\epsilon_t$ are i.i.d. with zero mean and unit variance.
The GARCH model Bollerslev1986 formulates the conditional variance $\sigma^2_t$ as a linear combination of the previous returns and conditional variances in a ARMA($p,q$) form as:
$\omega>0, \alpha_i,\beta_j \ge 0$, $i=1,...,p$, $j=1,...,q$ and $\sum_{i=1}^{p}\alpha_i+\sum^{q}_{j=1}\beta_j <1$ to ensure the stationarity of the GARCH process.
The structure of the conditional variance in (ref) is symmetric in the sense that the conditional variance $\sigma^2_t$ does not depend on the sign of the $y_t$, implying that the conventional GARCH($p,q$) model cannot capture the important leverage effect, i.e., $\sigma_t^2$ depends asymmetrically on the previous returns, in financial time series. Glosten:1993 propose a variant of the GARCH model, often called the GJR model, of the form
$\omega>0, \alpha_i,\beta_j \ge 0, \alpha_i+\gamma_i \ge 0$, $i=1,...,p$, $j=1,...,q$ and $\sum_{i=1}^{p}\alpha_i+\sum^{q}_{j=1}\beta_j +\sum^{p}_{i=1}\gamma_i <1$ to ensure the stationarity of the $y_t$ process and the positivity of the conditional variance. The indicator function $1\!\!1[y_t<0]$ in (ref) equals $1$ if $y_t<0$ and is $0$ otherwise. In the GJR model, if $\gamma_i>0$, negative returns are more influential than positive returns.
Another popular GARCH-type model is the Exponential Garch (EGARCH) model of Nelson:1991
where the roots of the polynomial $(1-\beta_1 L - ... -\beta_p L^p)$ must lie outside the unit circle to ensure the stationarity of the $y_t$ process. By working on the log-scale, the EGARCH model removes the positivity constraints on the model parameters and states the leverage terms in Eq. (ref) to capture the asymmetry in volatility clustering. See Poon:2003 and Bollerslev:2008 for a comprehensive discussion of the family of GARCH models and their properties. We will use GARCH, GJR and EGARCH as the benchmark econometric models to compare against RECH models, because they are widely used in the volatility modelling literature.
This section denotes the time series data as $\{D_t=(x_t,z_t),t=1,2,..\}$, where $x_t=(x_{t,1},...,x_{t,K})^\top$ is the vector of inputs and $z_t$ the scalar output. For the sequence $\{x_t\}$, $x_{i:j}$ denotes $(x_i,...,x_j)$ for $i\leq j$. The goal of recurrent neural network models is to model the conditional distribution $p(z_t|x_t,D_{1:t-1})$.
There are several standard time series models. One approach is to represent time effects {\it explicitly} via some simple functions, often a linear function, of the lagged values of the time series. This is the mainstream time series data analysis approach with the well-known ARIMA method Box:1976. This section considers an alternative approach representing time effects {\it implicitly} via latent variables that are designed to store the memory of the dynamics in the data. These latent variables, also called hidden states, are updated recurrently using the information carried over by their values from the past and the information from the data at the current time. Recurrent neural networks (RNNs), belonging to the second category, were first developed in cognitive science Elman:1990 and successfully used in machine learning.
If the serial dependence structure is ignored, then a feedforward neural network (FNN) can be used to transform the raw input data $x_t$ into a set of hidden units $h_t$, often called {\it learned features}, for the purpose of explaining or predicting $z_t$. Figure (ref) is a graphical representation of a FNN model with one hidden layer containing $L$ hidden units.
Given the FNN model in Figure (ref), the output $z_t$ is calculated as:
$V$ is a $L \times K$ matrix of weights connecting the input layer to the hidden layer; $\beta =(\beta_1,...,\beta_L)^\top$ is a vector of weights connecting the hidden layer to the output layer; $\beta_0$ is a scalar; $b=(b_1,...,b_L)^\top$ is a bias vector and $\phi(\cdot)$ is a non-linear scalar function, called the activation function. The scalar function $\phi$ is applied component-wise to a vector. In modern neural network modelling, the default recommendation for $\phi(\cdot)$ is to use the rectified linear unit Nair:2010,Le:2015, or ReLU, having the form $\phi(z) = max\{0,z\}$. The density $p(z_t|\eta_t)$ depends on the learning task. For example, if $z_t$ is continuous, then typically $p(z_t|\eta_t)$ is a normal distribution with mean $\eta_t$ and variance $\sigma^2$; if $z_t$ is binary, then $z_t|\eta_t$ follows a Bernoulli distribution with probability $\text{logit}^{-1}(\eta_t)$.
FNNs provide a powerful way to approximate the true function that maps the input $x_t$ to the mean ${\rm E}(z_t|x_t)$ or to transform the raw data $x_t$ into summary statistics $h_t$ having some desirable properties. However, FNNs are unsuitable for time series data analysis as the time effects and the serial correlations are ignored. The main idea behind RNNs is to let the set of hidden units $h_t$ feed itself on its lagged value $h_{t-1}$. Hence, an RNN can be best thought of as a FNN that allows a connection of the hidden units to their value from the previous time step, enabling the network to possess memory. This basic RNN model Elman:1990 can be written as:
the parameters are the bias vector $b$, the bias scalar $\beta_0$, the weight matrices $V$, $W$, and $\beta$ for input-to-hidden, hidden-to-hidden and hidden-to-output connections, respectively. Similarly to FNNs, $\phi(\cdot)$ is a non-linear activation function; common choices are the ReLU or the sigmoid $\phi(z)=1/(1+e^{-z})$. Usually, we can set $h_1=0$, i.e. the neural network initially memoryless.
Figure (ref) Nguyen:2019 graphically illustrates the RNN model (ref)-(ref). The circuit diagram (Left) can be interpreted as an unfolded computational graph (Right), where each node is associated with a particular time step. The calculation of $h_t$ can be represented as a Simple Recurrent Neuron (SRN) unit, as Figure (ref) shows, and we refer to (ref) as $h_t=\text{SRN}(x_t,h_{t-1})$, taking data $x_t$ at time $t$ and the previous state $h_{t-1}$ as the inputs. Using the SRN structure, the unfolded graph of the RNN model of Elman:1990, which is normally referred to as the Simple RNN model, can be reinterpreted as the unfolded graph in Figure (ref)(right).
There are more sophisticated recurrent neuron unit structures to compute $h_t$, such as the memory cell in the Long Short-Term Memory (LSTM) model of Hochreiter1997 and the Statistical Recurrent Unit (SRU) of Oliva:2017. Below, we sometimes write $h_t=\text{RNN}(x_t,h_{t-1})$ to represent the way the hidden state $h_t$ is computed in an RNN model. Our article, however, only considers the SRN.
The key motivation of RECH models is to allow the constant term $\omega$ in the general formulation of GARCH-type models in (ref)-(ref) to be driven by an auxiliary deterministic process governed by an RNN, in order to capture complex dynamics such as non-linearity and long-term dependence that might not be captured efficiently by the GARCH-type component $f(\sigma_{t-1}^2,....,y_{t-q})$. Our general RECH model is written as:
$g(\cdot)$ is a non-negative activation function; $p$ and $q$ are lag orders of $\sigma^2_t$ and $y_t$ respectively; $\beta_0$ is a scalar; $\beta=(\beta_1,...,\beta_L)$ is the weight vector with $L$ the number of hidden states. The reason an activation function is applied to $\omega_t$ is to ensure the conditional variance $\sigma^2_t$ is positive. We refer to $\omega_t$ in (ref) as the {\it recurrent} component, as it is driven by a RNN, and $f(\cdot)$ as the {\it GARCH} component as this is formed based on the GARCH-type structures without the constant term. Hence, we shall refer to the parameters of the recurrent component as the recurrent parameters, and refer to the parameters in $f(\cdot)$ as the GARCH parameters. The recurrent state $h_t=\text{RNN}(x_t,h_{t-1})$ takes as its inputs the previous state $h_{t-1}$ and a vector of additional information $x_t$ whose choice is discussed shortly.
The conditional variance in (ref) is a sum of the recurrent and the GARCH components. This flexible design allows RECH models to enjoy many advances from both worlds of deep learning and volatility modelling. Similarly to deep learning models, RECH models can use highly sophisticated neural network structures to capture complicated dynamics, e.g., long-range dependence and non-linearity, of the volatility dynamics and hence improve the forecasting of traditional GARCH-type models in applications where the underlying volatility dynamics exhibits long memory and nonlinearity. Similarly to GARCH-type models, RECH models use simple yet interpretable structures to simulate important stylized facts in financial time series such as volatility clustering and leverage effects. RECH models are well suited to modeling volatility because they inherit many properties from the GARCH-type models and distinguish themselves from the existing NN-based volatility models that often overlook the interpretability of the mainstream econometric models. As the recurrent and GARCH components are additive, increasing the complexity of the recurrent component $\omega_t$ will not decrease the interpretability of the GARCH component, and hence of RECH models. The general formulation in (ref)-(ref) implies that most (if not all) variants of the GARCH models are nested in the RECH framework, because RECH models reduce to the corresponding GARCH-type models if $\beta=0$.
In previous work that combined an NN-based component with a GARCH-type model, Donaldson:1997 added a FNN-based component to a GJR model and reported some improvement of their FNN-GJR model compared to several benchmark GARCH-type models. However, as discussed above, it might be inefficient to use a FNN to analyze time series data as FNNs are typically designed for cross-sectional data. Also, the estimation method of Donaldson:1997 uses randomized weights for the FNN component rather than optimizing them; this technique is not recommended in the modern deep learning literature Goodfellow2016. Nguyen:2019 incorporate LSTM into the stochastic volatility models and name their model LSTM-SV. LSTM-SV belongs to the class of parameter-driven models while RECH is an observation-driven model; see Koopman:2016 for a comprehensive comparison between these two classes of models. More specifically, the volatility dynamics in RECH is a {\it deterministic} process rather than a stochastic latent process as in LSTM-SV, making it much easier to estimate RECH models than the LSTM-SV model as the likelihood of RECH models can be calculated analytically. The RNN component of the LSTM-SV model takes only its past values as inputs while the recurrent component $\omega_t$ of RECH models allows any information including past observations as inputs. We show below that this flexibility enables RECH models with SRN to capture complicated dynamics and the leverage effect in financial time series without needing to use complicated RNN structures such as LSTM. Finally, like GARCH and SV, RECH and LSTM-SV complement each other and offer different perspectives towards the volatility modelling problem. The Appendix compares the performance of the RECH and LSTM-SV models.
The RECH framework is highly flexible because it can easily incorporate advances from both the deep learning and volatility modeling literatures to design the recurrent and GARCH components, respectively. For example, by using the SRN structure for the recurrent component $\omega_t$ and the conditional variance structure of the GARCH(1,1) model for the GARCH component, we obtain the SRN-GARCH specification of RECH model as:
$x_t$ is the input vector of the RNN at time $t$. Figure (ref) graphically represents the SRN-GARCH model. For simplicity, we consider the standard normal distribution ${\cal N}(0,1)$ for the errors $\epsilon_t$. Here, we have used a linear activation function for $g(\cdot)$ in Eq. (ref), and set $\beta_0,\beta_1 \geq 0$ to ensure the positivity of the conditional variance. Alternatively, one can use a positive activation function for $g(\cdot)$, such as the ReLu or sigmoid, and relax the positivity constraints for $\beta_0$ and $\beta_1$. We follow the GARCH literature and put the stationarity and positivity constraints on the GARCH parameters $\alpha$ and $\beta$, i.e., $\alpha,\beta \ge 0$ and $\alpha+\beta<1$. These constraints do not imply that the SRN-GARCH model is stationary, but might improve numerical stability in estimation.
The recurrent function SRN is \[\text{SRN}(x_t,h_{t-1}):=\phi(v^\top x_t + w h_{t-1}+b),\] with $\phi(\cdot)$ a non-linear activation function. We use the bounded ReLU activation for $\phi(\cdot)$ as it is easier to train and often performs better than other alternatives in the deep learning literature LIEW2016718. The bounded activation also guarantees a finite unconditional volatility; see Theorem (ref). The recurrent weight $w$ and offset term $b$ are scalars. By default, we use only one recurrent state, i.e. $h_t$ is a scalar; however, it is possible to extend the specification in (ref)-(ref) to the case where $h_t$ is a vector of hidden states. There is no restriction on the choice of the input vector $x_t$; typically, $x_t$ should include variables such as covariates and past returns that are deemed useful for predicting volatility $\sigma_t^2$. If there are no covariates, as in the applications below, our choice for $x_t$ is the vector of the past return $y_{t-1}$ and the past conditional variance $\sigma_{t-1}^2$. We also found it useful to include in $x_t$ the past recurrent component $\omega_{t-1}$. Hence, $x_t=(w_{t-1},y_{t-1},\sigma_{t-1}^2)^\top$ and the input weights are $v=(v_0,v_1,v_2)^\top$.
While a non-zero $\beta$ in (ref) quantifies the linear dependence of the current conditional variance $\sigma_t^2$ on its past value $\sigma_{t-1}^2$, a non-zero weight $v_2$ quantifies the non-linear dependence of $\sigma_t^2$ on $\sigma_{t-1}^2$. That is, the recurrent component allows non-linear dependence of $\sigma_t^2$ on $\sigma_{t-1}^2$. It is also well perceived in the deep learning literature that RNNs are able to capture long-range dependence pmlr-v97-greaves-tunnell19a, therefore the recurrent component can allow long-range dependence that $\{\sigma_{s}^2,s<t\}$ have on $\sigma_t^2$.
We note that the SRN-GARCH specification uses the conditional variance structure of the GARCH model, but is still able to capture the leverage effects of the volatility dynamics as the input vector $x_t$ includes the asymmetric leverage term $y_{t-1}$ itself, not $y_{t-1}^2$. Section (ref) shows that this choice of the input vector $x_t$ makes the volatility estimated by RECH models less sensitive to the choice of the structure for the GARCH component. For selecting the lags $p$ and $q$ in the GARCH component, we find that SRN-GARCH(1,1) often works well in almost all cases. This is probably because the recurrent component $\omega_t$ is able to capture the long-range dependence pmlr-v97-greaves-tunnell19a, hence larger lags are unnecessary. This is also consistent with the observation in the financial econometrics literature that the GARCH(1,1) model often works the best among other GARCH models Hansen:2005. Below, by SRN-GARCH without mentioning the lags we mean SRN-GARCH(1,1). We note that the SRN-GARCH specification simplifies to the GARCH(1,1) model if $\beta_0 >0$ and $\beta_1=0$.
Many other specifications of RECH models can be constructed. For example, the SRN-GJR specification is obtained by using the SRN structure for the recurrent component and the conditional variance structure of the GJR(1,1) for the GARCH component. Table (ref) presents several RECH model specifications. It is also possible to use other RNN structures such as LSTM Hochreiter1997 or SRU Oliva:2017 for the recurrent component. It is worth noting that the GJR and EGARCH models accommodate the leverage effects as linear terms in the conditional variance equation and hence can only capture the linear dependence of the leverage effects. RECH models, on the other hand, are able to capture other leverage dependence rather than the linearity, e.g. non-linearity or temporal dependence, of the leverage effects by allowing the leverage term $y_{t-1}$ to be an input of the RNN.
Given the general formulation of RECH models in (ref)-(ref), its $\sigma^2_t$ process, and thus its $y_t$ process, is not guaranteed to be stationary unless $\beta_1=0$ and the GARCH parameters satisfy the stationary constraints of the corresponding GARCH components $f(\cdot)$. For example, the SRN-GARCH specification in (ref)-(ref) is stationary if $\beta_0, \alpha, \beta >0, \beta_1 = 0, \alpha+\beta<1$. Although non-stationarity for volatility may be mathematically less appealing, it is often argued to be more realistic in practice, e.g. Bellegem:2012. Theorem (ref) below guarantees that the variance of $y_t$ is bounded.
The proof can be found in the Appendix. The conditions in the theorem prevent the volatility $\sigma_t^2$ from exploding and not cause numerical issues in training. The condition that $\alpha>0,\beta>0$, $\alpha+\beta<1$ is standard in the GARCH literature and easy to impose. The second condition, i.e. the finite recurrent component condition, is imposed by using the bounded ReLU activation function (to bound $h_t$), and assuming a bounded support for $\beta_0$ and $\beta_1$ (we used uniform U(0,0.5) for these two parameters).
This section discusses Bayesian estimation and inference for RECH models. We are interested in sampling from the posterior distribution \thinmuskip= 3mu plus 0mu minus 2mu \medmuskip= 4mu plus 2mu minus 2mu \thickmuskip=5mu plus 5mu minus 2mu
\thinmuskip= 0mu \medmuskip= 1mu plus 1mu minus 1mu \thickmuskip=2mu plus 3mu minus 1mu where $p(y_{1:T}|\theta)$ is the likelihood function, $p(\theta)$ is the prior and $p(y_{1:T}) = \int_{\Theta}p(y_{1:T}|\theta)p(\theta)d\theta$ is the marginal likelihood. Recall that the vector of model parameters $\theta$ consists of the recurrent and GARCH parameters. For example, the SRN-GARCH specification in (ref)-(ref) has the nine parameters $\theta=(\beta_0,\beta_1,\alpha,\beta,v_0,v_1,v_2,w,b)$.
The SMC method is an attractive approach for Bayesian inference and forecasting in volatility modelling li:2019. SMC can sample efficiently from non-standard posteriors, provides the marginal likelihood estimate as a by-product, and is a convenient way for computing one-step-ahead forecasts. In order to sample from the posterior $\pi(\theta)$, the SMC method Neal:2001,DelMoral:2006,Chopin:2002 first samples a set of $M$ weighted particles $\{W^j_0,\theta_0^j\}^M_{j=1}$ from an easy-to-sample distribution $\pi_0(\theta)$, such as the prior $p(\theta)$, and then traverses these particles through intermediate distributions $\pi_t(\theta), \;\; t=1,...,K$, which become the posterior distribution $\pi(\theta)$, i.e. $\pi_K(\theta)=\pi(\theta)$. In our article, we set $\pi_0(\theta)=p(\theta)$ as the prior $p(\theta)$ if it is possible to sample from $p(\theta)$. There are two common ways to design such a sequence of intermediate distributions: likelihood annealing Neal:2001 and data annealing Chopin:2002. The SMC with likelihood annealing uses the following intermediate distributions \thinmuskip= 3mu plus 0mu minus 2mu \medmuskip= 4mu plus 2mu minus 2mu \thickmuskip=5mu plus 5mu minus 2mu
\thinmuskip= 0mu \medmuskip= 1mu plus 1mu minus 1mu \thickmuskip=2mu plus 3mu minus 1mu where $\gamma_t$ is referred to as the temperature level and $0 = \gamma_0 < \gamma_1 < \gamma_2 < ... < \gamma_K=1$.
The SMC method consists of three main steps: reweighting, resampling and a Markov move. There are various ways to implement SMC in practice; here we briefly present one of these. At the begining of iteration $t$, the set of weighted particles $\{W_{t-1}^j,\theta_{t-1}^j\}^M_{j=1}$ that approximate the intermediate distribution $\pi_{t-1}(\theta)$ is reweighted to approximate the target $\pi_{t}(\theta)$. The efficiency of these weighted particles as a representation of $\pi_{t}(\theta)$ is often measured by the effective sample size (ESS) Robert:1998,LiuChen:1998 defined in (ref). If the ESS is below a prespecified threshold, the particles are resampled; the resulting equally-weighted samples are then refreshed by a Markov kernel whose invariant distribution is $\pi_{t}(\theta)$. Algorithm (ref) summarizes this SMC using the likelihood annealing method. We follow Dang:2018 and choose the tempering sequence $\gamma_t$ adaptively to ensure a sufficient level of particle efficiency by selecting the next value of $\gamma_t$ such that ESS stays above a threshold.
SMC with likelihood annealing sampler is suitable for in-sample analysis, as it uses the sequence of distributions in (ref) which requires the full training data $y_{1:T}$ to be available. For out-of-sample rolling forecasts where the model parameters $\theta$ are updated once new data arrive, it is necessary to use SMC with the data annealing Chopin:2002. This SMC sampler generates weighted particles from the following sequence of distributions \thinmuskip= 3mu plus 0mu minus 2mu \medmuskip= 4mu plus 2mu minus 2mu \thickmuskip=5mu plus 5mu minus 2mu
\thinmuskip= 0mu \medmuskip= 1mu plus 1mu minus 1mu \thickmuskip=2mu plus 3mu minus 1mu with $y_{1:t}$ the data available up to time $t$. The unnormalized weights at the SMC step $t$ in (ref) become
Algorithm (ref) in the Appendix summarizes SMC with data annealing. For RECH models, we use SMC with likelihood annealing for in-sample Bayesian analysis, and SMC with data annealing for out-of-sample analysis and forecasting.
The marginal likelihood is often used to choose between models using the Bayes factor jeffreys:1935,Kass:1995. In order to compare the relative performance between two models $M_1$ and $M_2$ on data $y_{1:T}$, we can use the Bayes factor \thinmuskip= 3mu plus 0mu minus 2mu \medmuskip= 4mu plus 2mu minus 2mu \thickmuskip=5mu plus 5mu minus 2mu
\thinmuskip= 0mu \medmuskip= 1mu plus 1mu minus 1mu \thickmuskip=2mu plus 3mu minus 1mu The larger the Bayes factor $BF_{M_1,M2}$, the stronger evidence that $M_1$ is more strongly supported by the data than $M_2$. We note that the SMC with likelihood annealing sampler in the previous section provides an efficient way to compute the marginal likelihood.
As SMC is parallelizable, the running time depends on how the algorithm is parallelized. For example, we can run the algorithm on a single multi-core machine or multiple multi-core cluster machines. Table (ref) shows the runtime of the SMC sampler, with $M=1000$ and $M=10000$ particles, when sampling the GARCH and SRN-GARCH models using only one core and six cores (numbers in parentheses). We run all examples on a standard laptop with moderate specification: Intel Core i7, 16GB RAM, 2.2GHz and 6 cores. We use $M=1000$ in this paper as this value is sufficient to obtain consistent estimation results.
For comparison, the table also shows the running time of the MCMC sampler for the GARCH model by the R package bayesGARCH of Ardia:2010 with two different numbers of iterations $N=20,000$ and $N=200,000$, and the runtime of the MCMC sampler for the SV model using the R package stochvol of Hosszejni:2020 with $N=50,000$ and $N=500,000$. These values of $M$ and $N$ are selected such that the standard errors of the SMC estimators (characterized by the effective sample size) are similar to that of the MCMC estimators (measured by the Integrated Autocorrelation Time, IACT). Here, we use the CODA R package of Plummer:2006 to compute the IACT of the MCMC chains obtained from the bayesGARCH and stochvol packages.
It is important to note that, unlike the MCMC sampler, SMC is parallelizable and hence its runtime can be reduced significantly when running on a computing cluster. As shown in Table (ref), the SMC sampler for SRN-GARCH is quite computationally expensive when using a single CPU core. However, its runtime is significantly reduced when running in parallel with six cores. In all of our examples in Section (ref), SMC was run on a high performance computing cluster.
This section evaluates the in-sample and out-of-sample performance of RECH models on simulation and stock return datasets. We use the SMC with likelihood annealing to perform in-sample Bayesian inference and the SMC with data annealing to obtain one-step-ahead forecasts. Table (ref) lists our implementation settings of the SMC samplers.
We now discuss the selection of the prior distributions for RECH models. We use a normal prior with a zero mean and variance $0.1$ for the recurrence parameters $(v_0,v_1,v_2,w,b)$; as empirical results from the deep learning literature show that the values of the weights of neural networks are often small. As a linear activation function for $g(\cdot)$ is used, we put the uniform prior $U(0,0.5)$ on $\beta_0$ and $\beta_1$ to impose the positivity of $\omega_t$. For the GARCH parameters, we choose the same priors as we use for the corresponding GARCH-type models, as summarized in Table (ref).
Next, we discuss the score metrics used to evaluate the out-of-sample performance. Denote by $D_{\text{test}}$ a test dataset, $T_{\text{test}}$ the number of observations in $D_{\text{test}}$ and $ \widehat\theta$ the posterior mean estimate of $\theta$; we use four predictive scores to measure out of sample performance: the partial predictive score (PPS), the number of violations (\#Vio.), the quantile score (QS) and the hit percentage (%Hit) to measure the out-of-sample performance. The PPS Gneiting2007 is evaluated on the test dataset $D_{test}$ as \[PPS:=-\frac{1}{T_{test}}\sum_{D_{test}} \log \; p(y_t|y_{1:t-1},\widehat\theta).\] The model with smallest PPS is preferred. The \#Vio. is defined as the number of times over the test data $D_{test}$ that the observation $y_t$ is outside its 99% one-step-ahead forecast interval. One of the main applications of volatility modelling is to forecast the Value at Risk (VaR). The $\alpha$-VaR is defined as the $\alpha$-quantile of the one-step-ahead forecast distribution $p(y_t|y_{1:t-1},\widehat\theta)$. The performance of a method producing VaR forecasts is often measured by the quantile score Taylor:2017 defined as \[QS:=\frac{1}{T_{test}}\sum_{D_{test}} (\alpha-I_{y_t\leq q_{t,\alpha}})(y_t-q_{t,\alpha}),\] where $q_{t,\alpha}$ is the $\alpha$-VaR forecast of $y_t$, conditional on $y_{1:t-1}$. The smaller the quantile score, the better the VaR forecast. The %Hit Taylor:2017 is defined as the percentage of the $y_t$ in the test data that is below its $\alpha$-VaR forecast. The %Hit is expected to be close to $\alpha$, if the model predicts well. We note that these predictive performance measures complement each other. For example, it is possible to make the number of violations small by increasing the forecast volatility, but the PPS and QS scores then increase. A volatility model minimizing all three predictive scores, and having a hit percentage close to $\alpha$, is arguably the preferred one.
For the simulation data, given the true volatility $\sigma_t$, we follow Hansen:2005 and use six additional predictive scores as summarized in Table (ref).
We generated a time series of $2000$ observations from the GARCH(1,1) model
The first $1000$ observations are used for model estimation and the last $1000$ observations for out-of-sample analysis. Table (ref) shows the posterior means and the posterior standard deviations of the GARCH(1,1) and SRN-GARCH model parameters, obtained from the SMC using likelihood annealing. Figure (ref) plots the estimated volatility together with the true volatility of the simulated data, i.e. the true values $\sigma_t^2$ generated from equation (ref).
The estimation results in Table (ref) and the volatility plots in Figure (ref) suggest some important implications. First, the GARCH parameters $\alpha$ and $\beta$ of the SRN-GARCH model in Table (ref) are close to those of the GARCH model and the volatility estimated from the RECH model in Figure (ref) are close to the true volatility, implying that the estimated RECH model is close to the GARCH(1,1) model if the GARCH(1,1) is the true model. The almost identical estimates of the marginal likelihood in the last column of Table (ref) also indicate that the GARCH(1,1) and RECH models provide an equally good fit to the data. Second, the coefficient $\beta_1$ is statistically insignificant, i.e. the posterior mean is less than two standard deviations from zero, and the values of the recurrent component $\omega_t$ are consistently small at all time points as Figure (ref) shows, implying that there are no volatility effects other than linearity that are captured by the recurrent component $\omega_t$ and that the recurrent component $\omega_t$ contributes very little to the conditional variance at all time steps. Additionally, the weights $v_1$ and $v_2$ of the inputs of the recurrent component are statistically insignificant, suggesting that there is no evidence of non-linearity, long range dependence and leverage effects within the data generating process GARCH(1,1).
We generated a time series of $2000$ observations from the following non-linear GARCH-type model
The model in (ref)-(ref) modifies the GJR(1,1) model by adding non-linear transformations of the past observation, conditional variance and leverage term to the equation of the conditional variance. The volatility evolution in (ref) suggests that the simulated volatility exhibits highly non-linear effects. The parameters of the model in (ref)-(ref) are set so that the simulated data somewhat resembles real financial time series data exhibiting both volatility clustering and leverage effects. The first $T = 1000$ observations are used for model estimation and the last $1000$ for out-of-sample analysis. Table (ref) shows the posterior means and standard deviations of the parameters from the GARCH(1,1), GJR(1,1), EGARCH(1,1) and three RECH counterparts.
The estimation results from Table (ref) suggest the following. First, the posterior means of $\alpha$ and $\beta$ from the GARCH and GJR models are close to their true values, suggesting that the GARCH and GJR models can capture the linear serial dependence within the volatility dynamics of the data generating process. The constants $\omega$ of both the GARCH and GJR model are significantly inflated compared to the true value, possibly caused by the non-linear effects that cannot be captured by the GARCH and GJR models. The leverage parameter $\gamma$ of the GJR model is close to zero and statistically insignificant, implying that the GJR model cannot capture the leverage effect in the data generating process. The leverage parameter $\gamma$ of the EGARCH model is more than three standard deviations from zero, implying that the EGARCH model is the only benchmark GARCH-type model that can capture the simulated leverage effect.
Second, the coefficients $\beta_1$ of the SRN-GARCH and SRN-GJR models are more than three standard deviations from zero, implying that there is strong evidence of non-linearity in the volatility dynamics, and that the RNN structure within the recurrent component $\omega_t$ of the SRN-GARCH and SRN-GJR model is able to capture such dependence. The weight $v_1$ with respect to the leverage input $y_{t-1}$ of the RNN in the SRN-GARCH model is more than two standard deviations from zero and the leverage parameters $\gamma$ of the SRN-GJR and SRN-EGARCH are more than three standard deviation from zero, all suggesting that the three RECH specifications can capture the leverage effects exhibited within the simulated volatility. The recurrent component $\omega_t$ of RECH models is useful in capturing the leverage effects overlooked by the GARCH and GJR models. Interestingly, the coefficient $v_2$ with respect to the input $\sigma_{t-1}^2$ in the SRN-GJR is more than two standard deviations from zero, indicating that the SRN-GJR is able to detect the non-linearity dependence of the past conditional variance on the current conditional variance $\sigma_t^2$, exhibited within the simulated data generating process. The analysis above suggests that observing the recurrent parameters of RECH models, e.g. $\beta_1, v_1$ and $v_2$, helps to detect the possible non-linearity effects within the underlying volatility.
Third, the estimates of marginal likelihood in the last column of Table (ref) show that RECH models consistently have higher marginal likelihood than their GARCH-type counterparts and that the SRN-GJR model provides the best fit to the data. The difference between the log marginal likelihood estimates is equivalent to the Bayes factors of the SRN-GARCH, SRN-GJR and SRN-EGARCH models compared to the GARCH, GJR and EGARCH models of roughly $e^3\approx 20.1$, $e^{8}\approx 3000$ and $e^{3}$, respectively, strongly supporting the RECH models. We note that among the benchmark GARCH-type models, the EGARCH model best fits to the SIM II data.
Figure (ref) plots the values of the recurrent component and estimated volatility of the SRN-GJR model, together with the true volatility, at all time points. Figure (ref) in the Appendix plots the volatility estimated by the GJR model and the true volatility. During the low volatility periods, i.e. time $t$ is between 0-100 or 900-1000, the volatility of the SRN-GJR model are very close to the true volatility, which is also the case for the GJR model in Figure (ref). During the periods when the volatility changed dramatically and oscillated highly, i.e. $t$ is between 200-250, 350-400 or 500-700, the volatility produced by the SRN-GJR model still tracks the true volatility well, but this is not the case for the GJR model. The GJR model often produces overly-large and overly-small volatility during these periods of abruptly changed volatility. The SRN-GJR model, on the other hand, appears to be able to capture well these changes. The plot of the recurrent component $\omega_t$ at all time points in Figure (ref) shows that it is highly responsive to the changes in the true volatility.
Table (ref) reports the forecast performance of the benchmark GARCH-type and the RECH models. For the SIM I data, the forecast performance of the SRN-GARCH model is very close to that of the GARCH model, in all predictive scores, which again strongly supports the earlier in-sample conclusion that the SRN-GARCH model closely approximates the GARCH model if the GARCH is the true model. For the SIM II data, Table (ref) suggests some important results. First, the RECH models outperform their GARCH-type counterparts in most of the predictive scores, which is consistent with the in-sample analysis showing that the RECH models fit better than their benchmark GARCH-type counterparts. Second, the SRN-GJR model has the best forecast performance for all predictive measures and the SRN-EGARCH model also performs well on the SIM II data, which is consistent with the in-sample analysis showing that these two RECH specifications have the highest marginal likelihood estimates. Third, amongst the benchmark GARCH-type models, the EGARCH model has the best predictive performance, compared to the GARCH and GJR models.
This simulation study examines if RECH models are able to simulate the long-memory volatility, by fitting the FIGARCH$(1,d,1)$ model of Baillie:1996 to data generated from RECH models. The FIGARCH$(1,d,1)$ model is defined as:
where $ d \in (0,1)$ is the fractional integrated parameter and $L$ is the backshift operator. The parameter $\Theta = (\omega,\psi,d,\beta)$. When $d=0$, the FIGARCH becomes a GARCH model. When $d>0$ and is close to 1, the persistence of the past shocks in the FIGARCH process decays at a slow hyperbolic rate Baillie:1996; hence the FIGARCH process exhibits long-memory effects in its volatility dynamics.
We use the SRN-GARCH model as the true data generating process (DGP) with the four different parameter sets $\theta_i, \; i=1,..,4$, listed in Table (ref). These are the estimated parameters obtained in Section (ref) when the SRN-GARCH model is fitted to four real datasets. For each parameter set $\theta_i$, $500$ datasets of $T=3000$ observations are generated from each of two different specifications of the SRN-GARCH model: $\beta_1 = 0$ and $\beta_1$ equals to the true values in Table (ref), i.e. $\beta_1 \neq 0$. We note that if $\beta_1 = 0$, the DGP is the GARCH$(1,1)$ model. To generate each time series, we generate $10,000$ observations and use the last $3,000$ for the simulation data. We then use the Matlab MFE toolbox \footnote{https://github.com/bashtage/mfe-toolbox/}, with the default settings, to produce the Quasi-Maximum Likelihood Estimate (QMLE) of the parameter $\widehat\Theta_i, \;i=1,..,4$, of the FIGARCH$(1,d,1)$ model. Table (ref) shows the means and standard deviations averaged over $500$ QMLE estimates of the FIGARCH$(1,d,1)$ parameters.
The important conclusion from Table (ref) is that the short-memory and long-memory properties of the GARCH$(1,1)$ and SRN-GARCH models, respectively, are distinguishable. When the DGP is the GARCH$(1,1)$, the estimates of the fractional integrated parameter $\widehat{d}$ are insignificant in all cases, suggesting that there is no evidence of the long-memory effects in the volatility of the GARCH$(1,1)$ model. When the DGP is the SRN-GARCH model, i.e. $\beta_1 \neq 0$, the estimates of the fractional integrated parameter $\widehat{d}$ are close to $1$ in all cases, implying the existence of long-memory in the volatility dynamics of simulated time series, which are generated from SRN-GARCH models. The difference in the QMLE estimates of the parameter $d$ between the two DGPs in Table (ref) implies that the SRN-GARCH model is able to simulate long-memory volatility effects. We observe similar results for the SRN-GJR and SRN-EGARCH models.
We demonstrate the performance of RECH models using four stock index datasets: the Standard and Poor's $500$ Index (SP500), the Japanese Nikkei 225 Index (N225), the Russell 2000 Index (RUT) and the German stock index (DAX). The datasets were downloaded from the Realized Library of The Oxford-Man Institute\footnote{https://realized.oxford-man.ox.ac.uk/}. We used the daily closing prices $\{P_t, \ t=1,...,T_P\}$ and calculated the demeaned return process as \thinmuskip= 3mu plus 0mu minus 2mu \medmuskip= 4mu plus 2mu minus 2mu \thickmuskip=5mu plus 5mu minus 2mu
\thinmuskip= 0mu \medmuskip= 1mu plus 1mu minus 1mu \thickmuskip=2mu plus 3mu minus 1mu The length of the four return series is fixed to be $T=4000$, with $T=T_P-1$, and each series is divided into an in-sample period of the first $T_{\text{in}}=2000$ observations and an out-of-sample period of the last $T_{\text{out}}=2000$ observations. Table (ref) summarizes the datasets.
Table (ref) reports some descriptive statistics for these four datasets together with the modified R/S test Lo:1991 for long-range memory in the logarithm of the squared returns. Lo's modified R/S test is widely used in the financial time series literature; see, e.g., Lo:1991, Giraitis:2003, Breidt:1998. All the index data exhibit some negative skewness, a high excess kurtosis and high variation. The N225 returns are more skewed and leptokurtic than those of the SP500, RUT and DAX data. The result of Lo's modified R/S test for long-memory dependence with several different lags $q$ indicates that there is significant evidence of long-memory dependence in the SP500, RUT and DAX stock indices. For the N225 data, however, the evidence of long memory is less clear as the null hypothesis of short memory for the squared returns is not rejected at the 5% level of significance when $q=20$ and $q=30$.
The Realized Library provides different realized measures\footnote{See https://realized.oxford-man.ox.ac.uk/documentation/estimators for the list of the available realized measures} that can be used in financial econometrics as a proxy to the latent $\sigma_t^2$. We use the following six common realized measures including Realized Variance (RV) Andersen:1998, Bipower Variation (BV) Nielsen:2004, Median Realized Volatility (MedRV) Andersen:2012, Realized Kernel Variance Barndorff&Nielsen:2008 with the Non-Flat Parzen kernal ($\text{RKV}_1$), the Tukey-Hanning kernal ($\text{RKV}_2$) and the Two-Scale/Bartlett kernal ($\text{RKV}_3$), to evaluate the forecast performance of the volatility models using the predictive scores in Table (ref). Shephard:2010 give more details about the Realized Library.
Denote by $RV_t$ the realized measure of $\sigma_t^2$ at time $t$. As the realized measures ignore the variation of the prices overnight and sometimes the variation in the first few minutes of the trading day when recorded prices may contain large errors Shephard:2010, we follow Hansen:2005 to scale the realized measure $RV_t$ as \thinmuskip= 3mu plus 0mu minus 2mu \medmuskip= 4mu plus 2mu minus 2mu \thickmuskip=5mu plus 5mu minus 2mu
\thinmuskip= 0mu \medmuskip= 1mu plus 1mu minus 1mu \thickmuskip=2mu plus 3mu minus 1mu with ${\rm E}(y_t|\mathcal F_{t-1}) = 0$, and use $\widetilde\sigma^2_t$ as the estimate of the latent conditional variance $\sigma_t^2$; see Table (ref) for a definition of $T_\text{in}$ and $T_\text{out}$ used in our datasets. See Martens:2002 and Fleming:2003 for a similar scaling estimator of the daily volatility.
Table (ref) and (ref) summarize the estimation results of fitting the benchmark GARCH-type models and their RECH counterparts to the SP500, N225, RUT and DAX datasets. The posterior mean estimates and posterior standard deviation estimates are obtained using the SMC with likelihood annealing sampler. We draw the following conclusions from the estimation results.
First, the marginal likelihood estimates show that the RECH models fit the index datasets better than the GARCH-type models, except for the SRN-EGARCH model for the N225 data. For example, for the SP500 data, the Bayes factors of the SRN-GARCH, SRN-GJR and SRN-EGARCH models compared to the GARCH, GJR and EGARCH models are roughly $e^{36}$, $e^{28}$ and $e^{10}$, respectively, which, according to Jeffrey’s scale for interpreting the Bayes factor Jeffreys:1961, decisively support the RECH models. Among the benchmark GARCH-type models, the EGARCH model constantly has the highest marginal likelihood.
Second, the estimated posterior means of the parameter $\beta_1$ of the RECH models are more than two standard deviations from zero in all cases, providing evidence of the volatility effects rather than linearity, e.g. probably non-linearity and long-memory effects, in the volatility dynamics and also suggesting that the recurrent component of the RECH models is able to effectively detect these effects. Additionally, the coefficients $v_2$ of the RECH models are statistically significant, indicating that the RECH models are able to detect the serial dependence rather than linearity that the previous conditional variance $\sigma_{t-1}^2$ has on $\sigma_t^2$.
Third, the existence of the leverage effects in the volatility is clear across all four stock markets. The leverage parameters $\gamma$ of the GJR and EGARCH models are statistically significant, implying that these models can detect the asymmetric volatility. All the leverage effect-related parameters $\gamma$ and $v_1$ in the RECH models are statistically significant, except the parameter $v_1$ of the SRN-EGARCH model. In particular, the {\it linear} leverage coefficient $\gamma$ of the SRN-GRJ and SRN-EGARCH models are significant, similarly to those of the GRJ and EGARCH models. Interestingly, the {\it non-linear} leverage coefficient $v_1$ of the RECH models is significant in almost all cases, suggesting that the spillover effect of asymmetric volatility can be non-linear. In particular, the leverage coefficient $v_1$ of the SRN-GARCH model is also statistically significant across all markets; i.e., unlike the conventional GARCH model, SRN-GARCH can detect the leverage effects in volatility.
Finally, as pointed out by a reviewer, in the EGARCH case the $\alpha$ and $\beta$ appear to be less affected by adding the recurrent component. This is because the EGARCH structure is very different from GARCH and GJR (see Table (ref)); the EGARCH part in SRN-EGARCH is already non-linear in $\sigma_{t-1}$ and hence it can accommodate non-linearity. This doesn't mean that SRN-EGARCH cannot improve EGARCH - $\beta_1$ is still far away from zero and the improvement is evident in better marginal likelihood and better out-of-sample prediction as confirmed in Section (ref).
Figure (ref) shows the volatility estimated by the GARCH and SRN-GARCH models for the SP500 index data, together with the values of the recurrent component $\omega_t$ at all time points. Figure (ref) and (ref) in the Appendix are similar plots for the SRN-GJR and SRN-EGARCH models. Clearly, the recurrent component $\omega_t$ is responsive to the changes in the volatility dynamics: it is small during the low volatility periods and large in the high volatility periods. This distinct behavior of financial volatility is well-captured by the recurrent neural network structure of the recurrent component $\omega_t$.
Figure (ref) plots the standardized residuals $\widehat\epsilon_t$ from the GARCH and SRN-GARCH models together with their QQ-plots.
We observe similar results for the SRN-GRJ and GRJ models, SRN-EGARCH and EGARCH models. Generally the RECH residuals appear to lie closer to the expected straight line than those of the counterpart GARCH-type models.
Table (ref) provides the skewness and kurtosis statistics together with the $p$-values of the Ljung-Box (LB) autocorrelation test of the residuals and squared residuals estimated by the RECH and the benchmark GARCH-type models. The $p$-value of the LB test, together with the sample ACF plots, of the standardized and squared standardized residuals suggest that there is no evidence of autocorrelation. The residuals produced by all models in Table (ref) exhibit some negative skewness and have kurtosis values higher than 3 (the kurtosis of the standard normal distribution). In general, the residuals of the RECH models seem closer to normality than those of the corresponding GARCH-type models. Similarly to the GARCH-type models, it is straightforward to use Student's $t$ distribution for the innovation in the RECH models to improve the residual diagnostics; however, this extension is not considered here.
Figure (ref) plots the one-step-ahead forecast conditional variance of the GARCH and SRN-GARCH models, together with the out-of-sample realized variance of the S&P500 data, obtained by data annealing SMC. Figure (ref) and (ref) in the Appendix are similar plots for the case of the SRN-GJR and SRN-EGARCH models, respectively. To save space, we do not report the plots for the N225, RUT and DAX datasets as similar behaviors of the forecast variance are observed for these datasets. We note from Figures (ref), (ref) and (ref) that in general, the RECH models and their GARCH-type counterparts produce forecast variances that adequately track the movement of the realized variance. The forecast variance of the RECH and benchmark models are similar in the low volatility regions while the RECH models have higher variance forecasts during high volatility periods. The variance forecasts of the RECH models seem to track the realized variance better than those of the benchmark GARCH-type models.
The in-sample analysis suggests that the RECH models fit the in-sample data of the four index datasets better than the counterpart GARCH-type models. However, it is possible that the superior in-sample performance is the result of overfitting Pagan:1990,Donaldson:1997. Table (ref) provides summary statistics on the one-step-ahead forecasts of conditional variance and standardized residuals. The most important conclusion from Table (ref) is that the RECH models do not overfit the data, as the forecast conditional variance of the RECH models are not excessively variable and the forecast residuals of the RECH models are very close to those of the GARCH-type benchmark models. The RECH models occasionally produce one-step-ahead forecast residuals with lower kurtosis than the counterpart GARCH-type models.
Table (ref) shows the forecast performance of the models using the four predictive scores PPS, \#Vio, QS and %Hit. As these four predictive scores complement each other, for each pair of the RECH and GARCH-type models, we compare their forecast performance by counting the number of times one model has a better predictive score than the other and report this count in the last column of Table (ref). The model with the higher count is preferred. Table (ref) shows that the RECH models consistently outperform their counterpart GARCH-type models for the S&P500, RUT and DAX data. For the N225 index, the predictive improvement of the RECH models over the benchmark counterparts is less clear, especially for the SRN-GARCH and GARCH models.
Tables (ref) to (ref) summarize the forecast performance measured by the predictive scores defined in Table (ref). Each table contains six panels, corresponding to the six realized measures mentioned earlier. For each pair of the RECH and GARCH-type models, their forecast performance are also compared in the same way as in Table (ref). Additionally, in each panel, bold numbers are used to indicate the lowest forecast errors. For each type of realized measure, the model with the highest number of lowest forecast errors is preferred. The table shows that the RECH models in general outperform their counterpart GARCH-type models.
In particular, for the SP500, N225 and RUT datasets, the RECH models consistently perform the best across all panels. For example, the forecast results in Table (ref) show that the SRN-GARCH model has the highest numbers of lowest forecast errors in all panels, implying that the SRN-GARCH model forecasts volatility the best for the SP500 data. For the N225 and RUT datasets, the SRN-EGARCH model clearly outperforms the other RECH and benchmark models in all realized measures. The superior predictive performance of the RECH models over the GARCH-type counterparts provides further evidence to support the conclusion that the RECH models do not overfit the index datasets.
As mentioned in Section (ref), we now discuss an useful feature of the RECH models; that is, the volatility estimates and volatility forecasts of the RECH specifications are less sensitive to the choice of the structure for the GARCH component than a single GARCH-type model. For example, for each dataset in Table (ref) and (ref), we compute the difference between the highest and lowest marginal likelihood estimates among the RECH specifications and calculate the same value for the GARCH-type benchmark models. Table (ref) shows that these discrepancies of in-sample performance among the RECH models are much smaller than those of the GARCH-type models, across all datasets. For each panel in Tables (ref) to (ref), we compute the difference between the highest and lowest forecast scores among the RECH models and do the same for the benchmark models; Figure (ref) plots the results.
The comparison results in Figure (ref) indicate that the discrepancies of out-of-sample performance among the RECH models are consistently lower than those of the GARCH-type models, except for the $\text{MSE}_2$ score for the N225 data. The result that model performance is less sensitive to the RECH specification is useful in practice as users do not need to worry about which specification should be used for their financial dataset.
This section reports on the application of the RECH model to analyse the USD/GBP daily exchange rates observed from 21/03/2001 to 01/03/2009\footnote{The dataset was also downloaded from the Realized Library of The Oxford-Man Institute}. We use the first $1000$ observations for model estimation and the last $1000$ observations for evaluating predictive performance.
The in-sample results (not shown) suggest that, unlike for the stock data, for exchange rate data GARCH performs the best compared to GJR and EGARCH. This is consistent with the study of Hansen:2005 who show that there is no evidence that GARCH is outperformed by more sophisticated models on the DM/USD exchange rates. The in-sample result also shows that SRN-GARCH outperforms GARCH, thus being the best model, in terms of marginal likelihood.
Table (ref) summarizes the forecast performance measured by the predictive scores discussed in Section (ref), which suggests that RECH models are able to improve on their counterpart GARCH-type models in terms of volatility forecasts.
We propose a new class of conditional heteroskedastic models, which we call RECH models, by incorporating a RNN structure into the conditional variance of the GARCH-type models, and study in detail three RECH specifications: SRN-GARCH, SRN-GJR and SRN-EGARCH. We use Sequential Monte Carlo with likelihood annealing and data annealing for in-sample Bayesian inference and out-of-sample forecasting. We also use the estimate of marginal likelihood as a by-product of the SMC for model choice. The extensive simulation and empirical studies suggest that the RECH models not only have both attractive in-sample performance and accurate out-of-sample forecasts, but can also explain the volatility movement. In addition to the empirical study reported in Section (ref), we tested the RECH models on all of the other datasets included in the Realized Library which contains 31 major stock markets around the world. In all cases, with the adding of the RNN component, the RECH model is not worse than its GARCH counterpart, whereas in some of these cases, significant predictive improvement is achieved by RECH.
An attractive feature of the proposed hybrid framework is that it is easy to use advances in both the deep learning and volatility modeling literatures to extend the current RECH models. This opens up many interesting future applications and areas of research. For example, one can use the Fourier Recurrent Unit (FRU) of zhang:2018 to construct the recurrent component of the RECH framework; FRU is currently considered as the state-of-the-art RNN architecture in deep learning. For the GARCH component, one can use the Bad Environment - Good Environment (BEGE) model of Bekaert:2015, which can efficiently simulate the heavy tailed behavior of financial returns. Another interesting research direction is extending the univariate RECH models to the multivariate case. We conjecture that the recurrent neural network architectures will be more powerful for multivariate inputs as they can naturally capture the interaction between the inputs. This research is in progress.