Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
96,007 characters · 17 sections · 146 citation commands
Forecast Encompassing Tests for the Expected Shortfall
\def\spacingset#1{ {#1}} \spacingset{1}
\if00 \fi
\if10 {
} \fi
{\it Keywords:} evaluating forecasts, combining forecasts, loss function, model selection, statistical tests
\spacingset{1.45}
Through the recent introduction of Expected Shortfall (ES) as the primary market risk measure for the international banking regulation in the Basel III Accords Basel2016, Basel2017, there is a great demand for reliable methods for evaluating and comparing the predictive ability of competing ES forecasts. The ES at probability level $\alpha \in (0,1)$ is defined as the expectation of the returns smaller than the respective $\alpha$-quantile (the Value at Risk, VaR), where $\alpha$ is usually chosen to be 2.5% as proposed by the Basel Accords. The ES is replacing the VaR in the banking regulation as it overcomes several shortcomings of the latter such as being not coherent and its inability to capture tail risks beyond the $\alpha$-quantile Artzner1999,Danielsson2001,Basel2013. While the empirical properties favor the ES over the VaR as a risk measure, the ES lacks elicitability, which implies that no strictly consistent loss functions exist. The non-elicitability of the ES is overcome by considering the pair VaR and ES which are jointly elicitable, i.e.\ there exist joint loss functions for the VaR and the ES Fissler2016. This discovery triggered a rapidly growing branch of literature in developing forecasting methods and forecast evaluation techniques for the ES, see Patton2019, DimiBayer2019, BayerDimi2019, Taylor2019, Barendse2020, FisslerZiegelGneiting2016 and Nolde2017 among others.
A desirable tool for the comparison of ES forecasts are encompassing tests, which however build upon the existence of strictly consistent loss functions. Given two competing forecasts A and B, forecast encompassing tests the null hypothesis that forecast A performs not worse than any (linear) combination of these forecasts. This is carried out by testing whether the optimal combination weight of forecast B deviates significantly from zero.\footnote{For the classical theory on forecast encompassing see HendryRichard1982, MizonRichard1986, Diebold1989, Ericsson1993, harvey1998tests, clark2001tests, GiacominiKomunjer2005, NewboldHarvey2007 and ClementsHarvey2009 among others.} This null hypothesis allows for the convenient interpretation that forecast B does not add any information to forecast A and thus, forecast A is superior to forecast B. The existence of appropriate loss functions is inevitable for encompassing tests for two reasons. First, the superior performance of competing forecasts is defined in the statistical sense by using strictly consistent loss functions. Second, loss and identification functions are crucial for M- or GMM-estimation of the optimal forecast combination weights through an appropriate regression framework for the risk measure under consideration.
In this paper, we introduce novel encompassing tests for the ES based on the joint loss functions for the VaR and ES developed in Fissler2016. We introduce the following three test variants for the ES. First, we propose to jointly test forecast encompassing for the VaR and ES, henceforth denoted the joint VaR and ES encompassing test. We introduce a second test variant, denoted the auxiliary ES encompassing test, which estimates the optimal combination weights for the vector of the VaR and ES, however, only tests the parameters associated with the ES. While incorporating both, VaR and ES forecasts, this variant only tests encompassing of the ES forecasts. The third variant overcomes the tests' dependence on VaR forecasts and tests encompassing of competing ES forecasts stand-alone, which comes at the cost of a potential model misspecification. We henceforth call this test the strict ES encompassing test. This variant is particularly relevant due to the current set of rules established by the Basel Committee of Banking Supervision, which only imposes the financial institutions to report ES forecasts Basel2016, Basel2017. Only this test variant can be applied in situations where the person evaluating the forecasts merely has forecasts for the ES at hand. However, in situations where both, the VaR and ES forecasts (stemming from the same model or forecasting procedure) are available, application of the joint or auxiliary tests is generally recommended.
We implement the encompassing tests through M-estimation of the optimal combination weights Patton2019, DimiBayer2019 and in an environment with asymptotically non-vanishing estimation uncertainty of the forecasting procedures GiacominiKomunjer2005, GiacominiWhite2006. As the strict ES encompassing test is potentially subject to model misspecification, we derive the asymptotic distribution of the test statistics in a general setting which allows for misspecified models. This generalizes the asymptotic theory of Patton2019, DimiBayer2019 and BayerDimi2019 to potentially misspecified (and nonlinear) models. We base the Wald test statistics of the encompassing tests on a misspecification-robust covariance estimator. Our implementation further introduces a link or combination function which captures the different linear and nonlinear forecast combination methods in the existing encompassing testing literature, see ClementsHarvey2009 and ClementsHarvey2010 among others.
We analyze the finite sample behavior of our encompassing tests and the effect of the potential model misspecification in an extensive simulation study using models from various model classes associated with the ES. For this, we consider classical GARCH models, the GAS (generalized auto-regressive score) models with time-varying higher moments of Creal2013, the GAS models for the VaR and ES of Patton2019 and the ES-CAViaR models of Taylor2019. Data stemming from the latter three model classes induces some model misspecification for the strict ES encompassing test, which allows us to evaluate the effect the misspecification has on our tests. We find that all tests exhibit approximately correct size and good power properties for all considered simulations. This also holds for the strict ES encompassing test which demonstrates that this test is robust to the degree of model misspecification we usually encounter in financial applications.
Tests for forecast encompassing are commonly used to establish a theoretical basis for forecast combinations in cases when encompassing is rejected for both forecasts ClementsHarvey2009, NewboldHarvey2007, GiacominiKomunjer2005. This implies that neither of the forecasts stand-alone performs as good as an optimal forecast combination, which indicates that a forecast combination incorporates more information than the individual forecasts. GiacominiKomunjer2005, Timmermann2006, Halbleib2012 and Taylor2019FCcomb advocate general forecast combination methods for multiple reasons and particularly for risk measures with small probability levels, as it is customary for the VaR and the ES.
We apply our encompassing tests to ES forecasts from classical GARCH and GAS models, but also from the recently developed dynamic ES models of Taylor2019 and Patton2019 for daily returns of the IBM stock, the S$\&$P 500 and the DAX 30 indices. The test results imply that for the IBM stock, forecast combination methods outperform the stand-alone forecasting models in many instances. In comparison, this pattern seems to be less pronounced for the S$\&$P 500 and the DAX 30 indices, which are already well diversified through their versatile composition. Thus, classical diversification gains Timmermann2006 of forecast combination methods might be less pronounced for stock indices. The two ES based test variants exhibits very similar results, which further indicates that the strict ES test is robust against potential misspecifications in financial settings.
The classical idea of forecast encompassing goes back to HendryRichard1982, ChongHendry1986 and MizonRichard1986 and is developed for mean forecasts under the squared loss function. Broad reviews on encompassing testing are provided e.g.\ by NewboldHarvey2007 and ClementsHarvey2009. HarveyNewbold2000 extend the encompassing technique which classically focuses on two competing forecasts to encompassing of multiple forecasts. GiacominiKomunjer2005 develop (conditional) encompassing of quantile forecasts and focus on encompassing tests for methods instead of models. ClementsHarvey2010 generalize encompassing tests to probabilistic forecasts by relying on strictly consistent scoring rules. GiacominiKomunjer2005 and ClementsHarvey2010 investigate extensions of encompassing to more complicated functionals of the conditional distribution. Our work pursues this path by developing encompassing tests for the ES as a prominent example of higher-order elicitable functionals where only joint loss functions for vector-valued functionals are available. Our testing approach can be adapted to further higher-order elicitable functionals such as the pair mean, variance and the Range Value at Risk Cont2010, Embrechts2018, FisslerZiegel2019.
The rest of the paper is organized as follows. In Section (ref), we introduce encompassing tests for the ES and derive the asymptotic distribution of the associated test statistics under model misspecification. Section (ref) presents an extensive simulation study analyzing the size and power properties of our tests. In Section (ref), we apply the testing procedure to daily financial returns of the IBM stock and the S&P 500 and DAX 30 indices and Section (ref) concludes. All proofs are deferred to Appendix (ref). Technical details of the proofs and additional results are provided in the supplementary material.
We consider a stochastic process $Z = \left\{ Z_t: \Omega \to \mathbb{R}^{l+1}, l \in \mathbb{N}, t= 1,\dots,T \right\}$, which is defined on some common and complete probability space $(\Omega, \mathcal{F}, \mathbb{P})$, where $\mathcal{F} = \left\{ \mathcal{F}_t, t = 1, \dots, T \right\}$ and $\mathcal{F}_t = \sigma \left\{ Z_s, s\le t \right\}$. We partition the stochastic process as $Z_t = (Y_t,X_t)$, where $Y_t: \Omega \to \mathbb{R}$ is an absolutely continuous random variable of interest and $X_t: \Omega \to \mathbb{R}^{l}$ is a vector of explanatory variables. We denote the conditional distribution of $Y_{t+1}$ given the information set $\mathcal{F}_t$ by $F_t$. Accordingly, $\mathbb{E}_t$, $\operatorname{Var}_t$ and $h_t$ denote the expectation, variance and density corresponding to $F_t$. Following GiacominiKomunjer2005, we consider ($\mathcal{F}_t$-measurable) one-step ahead forecasts, henceforth denoted by $\hat f_{t}, \hat q_{t}$ and $\hat e_{t}$, which are generated by a function $f \big( \gamma_{t,m}, Z_t,Z_{t-1},\dots \big)$, which is fixed over time. For this, $\gamma_{t,m}$ denotes the (estimated) model parameters at time $t$ or alternatively the semi- or non-parametric estimator used in the construction of the forecasts. This construction allows for both, fixed forecasting schemes, where the model parameters $\gamma_{t,m}$ are only estimated once, and rolling window forecasting schemes, where the parameters $\gamma_{t,m}$ are re-estimated in each step. We denote general competing forecasts by $\hat{\boldsymbol{f}_t} = (\hat f_{1,t}, \hat f_{2,t})$, specific VaR (quantile) forecasts by $\hat{\boldsymbol{q}}_t = (\hat q_{1,t}, \hat q_{2,t})$ and ES forecasts by $\hat{\boldsymbol{e}}_t = (\hat e_{1,t}, \hat e_{2,t})$.
In the context of evaluating point forecasts, an important property of risk measures (or more general statistical functionals) is elicitability Gneiting2011. Elicitability means that there exist strictly consistent loss functions, i.e.\ loss functions $\rho(Y,f)$ depending on the random variable $Y \sim F$ and the issued forecast $f$, whose expectation $\mathbb{E} \left[ \rho(Y, \cdot) \right]$ is uniquely minimized by the true risk measure $\Gamma(F)$. Using such a loss function, one can assess the quality of issued forecasts by comparing their average losses induced by the realizations of the predicted variable. Evaluating forecasts through strictly consistent loss functions has the desired impact that it incentivizes financial institutions to truthfully report their correct forecasts Gneiting2011, FisslerZiegelGneiting2016. As a direct consequence, the literature on tests for forecast comparison and forecast rationality evolves around the associated loss functions, see MizonRichard1986, DieboldMariano1995, ElliottKomunjerTimmermann2005, GiacominiKomunjer2005, GiacominiWhite2006, PattonTimmermann2007, ClementsHarvey2010, Gneiting2011 and Patton2011 among many others.
Many important statistical functionals such as the variance, the ES, the minimum, the maximum and the mode are not elicitable, i.e.\ no strictly consistent loss functions exist Gneiting2011, Heinrich2014, Fissler2016. This deficiency calls for generalized approaches in many academic disciplines. We built our test procedure for the ES on such an approach, which considers multiple functionals stacked as vectors and considers joint elicitability. Fissler2016 show that the ES is jointly elicitable with the VaR by constructing strictly consistent joint loss functions for this pair, which we utilize in our encompassing approach.
In the following section, we formally introduce the concept of forecast encompassing in the classical case of one-dimensional, real-valued and elicitable functionals. Subsequently, we make use of the higher-order elicitability of the ES and generalize the encompassing approach to ES forecasts in Section (ref).
Following e.g. HendryRichard1982, MizonRichard1986, Diebold1989 and GiacominiKomunjer2005, we formally introduce the classical concept of linear forecast encompassing for one-dimensional, real-valued and elicitable functionals. We assume that two competing forecasters predict the variable of interest $Y_{t+1}$ and issue one-step ahead point forecasts $\hat{\boldsymbol{f}_t} = \big( \hat f_{1,t}, \hat f_{2,t} \big)$ for a given functional $\Gamma(F_t)$.\footnote{While we focus our approach on one-step ahead forecasts, extensions to multi-step ahead forecasts are straight-forward by employing a HAC-type estimator for the asymptotic covariance.} In order to conduct the forecast evaluation in an out-of-sample fashion, we divide the sample size $T$ in an in-sample part of size $m$ and an out-of-sample part of size $n$ such that $T = m+n$. The in-sample period is used to generate the forecasts $\hat f_{1,t}$ and $\hat f_{2,t}$ as described in the beginning of Section (ref), while the out-of-sample period is used for the evaluation of the forecasts. This procedure poses little restrictions on how to generate the forecasts and allows for parametric, semiparametric or nonparametric techniques and for nested and non-nested forecasting procedures GiacominiKomunjer2005.
Let $\rho \big(Y_{t+1}, \hat f_t \big)$ be a strictly consistent loss function for $\Gamma(\cdot)$. Then, we say that forecast $\hat f_{1,t}$ encompasses $\hat f_{2,t}$ at time $t$, if
for all $\big(\theta_{1},\theta_{2} \big) \in \Theta \subseteq{\mathbb{R}^2}$. Equation ((ref)) implies that, in terms of the loss induced by $\rho$, the forecast $\hat f_{1,t}$ is at least as good as any (linear) combination of $\hat f_{1,t}$ and $\hat f_{2,t}$. Hence, forecast $\hat f_{2,t}$ does not add any information on $Y_{t+1}$ which is not already incorporated in $\hat f_{1,t}$. We define $\big( \theta^\ast_{1}, \theta^\ast_{2} \big)$ as the optimal combination parameters which minimize the expected loss,
By definition, it holds that $\mathbb{E} \left[ \rho \big(Y_{t+1}, \theta_{1} \hat f_{1,t} + \theta_{2} \hat f_{2,t} \big) \right] \ge \mathbb{E} \left[ \rho \big(Y_{t+1}, \theta^\ast_{1} \hat f_{1,t} + \theta^\ast_{2} \hat f_{2,t} \big) \right]$ for all $\big(\theta_{1},\theta_{2} \big) \in \Theta$. In particular, this implies that
Combining ((ref)) and ((ref)) yields the following definition of forecast encompassing.
Tests for forecast encompassing are carried out through the following steps. First, we regress the realizations $Y_{t+1}$ onto the forecasts $\hat f_{1,t}$ and $\hat f_{2,t}$ using an appropriate regression technique for the functional under consideration in order to obtain the estimated combination (or encompassing) parameters $\hat \theta_n$ and their asymptotic distribution. Then, we test whether these parameters equal one and zero respectively.
As discussed e.g.\ in ClementsHarvey2009 and ClementsHarvey2010, there exist several different testing specifications available for the encompassing principle, which differ in terms of the admissible specifications of the linear (or nonlinear) forecast combination formula. We generalize and unify these approaches by introducing a general link or combination function,
which maps the forecasts and the respective parameters onto a linear or nonlinear forecast combination and where $\mathfrak{F}$ denotes the random space of the issued forecasts. For this, the function $g$ and the parameter space $\Theta$ have to be chosen such that there exists a $\theta_0 \in \Theta$, such that $g(\hat{\boldsymbol{f}_t},\theta_0) = \hat f_{1,t}$ almost surely, which enables testing whether $\hat f_{1,t}$ alone captures the full information provided by any forecast combination through testing the parametric restriction $\theta^\ast = \theta_0$.
This general definition unifies the following existing specifications of forecast encompassing, but also allows for more general linear and nonlinear specifications, see e.g.\ Ericsson1993, ClementsHarvey2009 and ClementsHarvey2010.
In this section, we consider encompassing tests for the ES. For absolutely continuous distributions $F_t$, the ES is formally defined as
where $Q_{t,\alpha}(Y_{t+1})$ denotes the conditional $\alpha$-quantile of $Y_{t+1}$ given $\mathcal{F}_t$. As discussed in the previous section, the main ingredient of forecast encompassing tests is the specification of the underlying loss function, which has to be associated with the risk measures we consider forecasts for. As such loss functions do not exist for the ES stand-alone, we utilize a strictly consistent joint loss function for the pair consisting of the ES and the VaR, given by Fissler2016 as
where the arguments $Y$, $q_\alpha$ and $e_\alpha$ denote the return realization, the quantile and the ES respectively. As this loss function exhibits the desirable property of having loss differences which are homogeneous of order zero, it is often denoted as the FZ0-loss function, see e.g. Patton2019. While there exist infinitely many strictly consistent loss functions for the pair VaR and ES, the recent literature seems to agree upon this choice: DimiBayer2019 find that it exhibits a stable numerical performance in M-estimation and empirically yields relatively efficient parameter estimates. Nolde2017 discuss the desirable property of homogeneity of these loss functions and Patton2019, BayerDimi2019 and Taylor2019 use this loss function to estimate dynamic ES models.
Following the specification of a link function in ((ref)), we introduce the quantile- and ES-specific link functions
where $ \mathfrak{Q}$ and $\mathfrak{E}$ denote the random spaces of the VaR and ES forecasts, $\Theta^\beta \subseteq \mathbb{R}^{k_\beta}$ and $\Theta^\eta \subseteq \mathbb{R}^{k_\eta}$ such that $\Theta = \Theta^\beta \times \Theta^\eta$ and $k_\beta + k_\eta = k \in \mathbb{N}$. We assume that the functions $g^q$, $g^e$ and the parameter space $\Theta$ are chosen such that there exist values $\beta_0 \in \Theta^\beta$ and $\eta_0 \in \Theta^\eta$, such that $g^q(\hat{\boldsymbol{q}}_t, \beta_0) = \hat q_{1,t}$ and $g^e(\hat{\boldsymbol{e}}_t, \eta_0) = \hat e_{1,t}$ almost surely.
In the following, we introduce the concept of joint forecast encompassing for the pair consisting of the VaR and the ES. Analogously to ((ref)), we define the optimal combination parameters for the VaR and ES as
We test whether the sequence of joint quantile and ES forecasts $\big( \hat q_{1,t}, \hat e_{1,t}\big)$ encompasses the sequence $\big( \hat q_{2,t}, \hat e_{2,t}\big)$ for all $t=m,\dots,T-1$ by estimating the parameters of the following semiparametric regression,
where $Q_\alpha(u_{t+1}^q|\mathcal{F}_t) = 0$ and $ES_\alpha(u_{t+1}^e|\mathcal{F}_t) = 0$ almost surely for all $t=m,\dots,T-1$ by using the M-estimation technique introduced in Patton2019 and DimiBayer2019. We then test for $\big( \beta^\ast, \eta^\ast \big) = \big( \beta_0, \eta_0 \big)$ using a Wald type test statistic.
Definition (ref) develops a joint encompassing test for the VaR and ES, which is reasonable given the joint elicitability property of the VaR and ES. However, a further objective of this paper is to construct encompassing tests for the ES stand-alone, which we do in the following.
This parameter restriction is tested using a Wald type test statistic based on the estimates of the regression setup given in ((ref)). As we do not test the quantile specific parameters $\beta^\ast$, we do not impose that the underlying quantile forecast also encompasses its competitor under this null hypothesis. Hence, even though this test is based on the joint regression, it only tests encompassing of the ES forecasts. We call this test auxiliary ES encompassing test as it still depends on the auxiliary quantile forecasts which are used for the estimation of the optimal combination parameters.
Given that both, the VaR and ES forecasts are available, application of either the joint or auxiliary test is the most plausible approach given their joint elicitability. However, even though the emphasis of the auxiliary encompassing test is on the ES, it still requires quantile forecasts for the implementation of the parameter estimation. This can be problematic for two reasons. First, the quantile forecasts are still used in the estimation procedure and thus have an indirect effect on the parameter estimates of the ES specific parameters. E.g., the previous tests are not applicable for ES forecasts which are based on the same VaR forecasts, as this implies perfect collinearity of the quantile regressors. Second, the auxiliary test is only applicable in the setup where the person applying the test has access to the quantile forecasts. In the current implementation of the regulatory framework of the Basel Committee Basel2016, Basel2017, the banks are only obligated to report their ES forecasts (at probability level $2.5\%$), but not the corresponding VaR forecasts. Thus, the accompanying VaR forecasts, which the ES forecasts are internally based on, are in general not available to the regulator who has to decide on an adequate risk management of the financial institution at hand.
In order to account for these scenarios, we further introduce the strict ES encompassing test, which only requires ES forecasts in the following. For this, we slightly modify the definition of ((ref)) by replacing $g^q(\hat{\boldsymbol{q}}_t, \beta)$ through $g^q(\hat{\boldsymbol{e}}_t, \beta)$,\footnote{Note that the parameters denoted $\theta^\ast$ in ((ref)) and in ((ref)) can generally differ.}
We test whether $\hat e_{1,t}$ strictly encompasses $\hat e_{2,t}$ for all $t=m,\dots,T-1$ by setting up the slightly transformed regression
where $Q_\alpha(u_{t+1}^q|\mathcal{F}_t) = 0$ and $ES_\alpha(u_{t+1}^e|\mathcal{F}_t) = 0$ almost surely for all $t=m,\dots,T-1$. The crucial difference between this test and the joint and auxiliary encompassing tests is that instead of using the quantile forecasts $\hat{\boldsymbol{q}}_t$ in the quantile link function $g^q$, we use the ES forecasts $\hat{\boldsymbol{e}}_t$ for both, the quantile and ES link functions $g^q$ and $g^e$. We argue that this can be seen as a best feasible solution due to the lack of loss functions for the ES stand-alone together with the necessity of developing forecast evaluation methods for the ES stand-alone due to the current setup of the Basel III regulatory framework Basel2016, Basel2017.
The underlying idea of this test is mainly motivated by pure scale models, i.e.\ $Y_{t+1} = \sigma_{t} u_{t+1}$, $u_{t+1} \sim F(0,1)$, which is still the most frequently used class of models for risk management with the GARCH and stochastic volatility models as prime examples. For this model class, the VaR and ES forecasts are perfectly colinear, $\hat e_t = \frac{\xi_\alpha}{z_\alpha} \hat q_t$, where $z_\alpha$ and $\xi_\alpha$ are the $\alpha$-quantile and $\alpha$-ES of the distribution $F(0,1)$. Hence, the quantile model $g^q(\hat{\boldsymbol{e}}_t, \beta) = g^q(\hat{\boldsymbol{q}}_t \xi_\alpha / z_\alpha, \beta) = g^q(\hat{\boldsymbol{q}}_t, \tilde \beta)$ is correctly specified, but with transformed quantile parameters $\tilde \beta$.\footnote{For the prominent case of linear encompassing link formulas $g^q(\cdot)$, it holds that $\tilde \beta = \beta z_\alpha / \xi_\alpha $.} As we only test on the ES-specific parameters $\eta$ as described in Definition (ref), our test is invariant to this (often linear) transformation of the parameter $\beta$ and thus, it is correctly specified for pure scale models.
In the general case, the quantile equation can possibly be misspecified. Thus, we provide asymptotic theory under general model misspecification for the M-estimator in the following section. The potential model misspecification might bias the pseudo-true parameters and challenge the interpretability of the test decision, but we argue that this effect is negligible for this setup. First, the misspecification is only slight in the sense that daily financial return data is approximated well by pure scale processes. Second, the misspecification is indirect in the sense that while the quantile parameters are potentially misspecified, we only test the ES parameters, which are influenced by the misspecification only indirectly through the joint estimation. Furthermore, we illustrate that the performance of our strict ES encompassing test is not negatively influenced by more general data generating processes in the simulation study in Section (ref) by considering GAS models with time-varying higher moments of Creal2013 and the dynamic ES models of Patton2019 and Taylor2019.
Tests for equal (superior) predictive ability in the sense of DieboldMariano1995, clark2001tests, GiacominiWhite2006, West2006 and the model confidence set approach of Hansen2011 can be seen as a general alternative to encompassing tests. As these tests are directly based on the average loss difference, they can only test the predictive ability of the VaR and ES jointly. In contrast, encompassing tests are based on the regression coefficients of the semiparametric quantile and ES models and hence, only indirectly on the respective loss function. This fundamental difference allows for stand-alone encompassing tests for ES forecasts, which constitutes a great advantage for ES encompassing tests.
Strictly speaking, strict consistency of loss functions only implies that the optimal forecast exhibits the smallest possible loss in expectation. In reality however, competing forecasts are often misspecified due to estimation error or misspecified forecasting models. Patton2019MisspecFC shows that then, the ranking induced by the loss functions can be sensitive towards the choice of (strictly consistent) loss functions or even misleading. Holzmann2014 show that for competing forecasts which are based on nested information sets and which are correctly specified given their underlying (but usually incomplete) information set (auto-calibrated), applying any strictly consistent loss function results in a correct ranking of the forecasts. In our case of testing forecast encompassing, we indeed build on nested information sets as it obviously holds that $\sigma \big\{ \hat f_{1,t}, \hat f_{2,t} \big\} \supseteq \sigma \big\{ \hat f_{1,t} \big\}$. Thus, by further assuming that the issued forecasts are auto-calibrated given the forecaster's information set, we can conclude that the ranking implied by ((ref)) is indeed the correct one and invariant towards the choice of strictly consistent loss functions.
In the following, we use the short notation $g^e_t(\eta) = g^e(\hat{\boldsymbol{e}}_t, \eta)$ and $g^q_t(\beta) = g^q(\hat{\boldsymbol{q}}_t, \beta)$ (or $g^q_t(\beta) = g^q(\hat{\boldsymbol{e}}_t, \beta)$ in the case of the strict test). We define the M-estimator as
and the pseudo-true parameter as\footnote{ The pseudo-true parameter can generally depend on the issued loss function, i.e. in this case on the zero homogeneous choice in ((ref)).}
When the link (regression) functions $g^q(\cdot)$ and $g^e(\cdot)$ are correctly specified, we get that the pseudo-true parameter $\theta^\ast_n$ equals the classical true regression parameter and it is independent of the sample size $n$. We further define the corresponding identification functions, which are almost surely the derivative of the loss function $\rho$ with respect to $\theta$,
We restrict our attention to processes which satisfy the following conditions.
The following propositions show consistency and asymptotic normality of the M-estimator under potential model misspecification.
The two preceding propositions extend the asymptotic theory of Patton2019 to the case of possibly misspecified models, and the misspecification theory for linear models of BayerDimi2019 to nonlinear models. The proofs in Appendix (ref) combine, extend and go along the lines of the ideas of Engle2004 and Patton2019. The conditions closely resemble the regularity conditions of Patton2019. As we further allow for model misspecification, we impose the unique minimization condition (ref) and slightly strengthen the moment conditions (ref). In the baseline case of linear encompassing link functions $g^q$ and $g^e$, the required moment conditions simplify to those given in BayerDimi2019.
For the estimation of the asymptotic covariance matrix $\widehat{\Omega}_{n}$ under possible model misspecification, we follow the approach of DimiBayer2019 and BayerDimi2019. We deal with the three nuisance quantities in $\widehat{\Omega}_{n}$ as follows. In order to estimate the density quantile function $h_t(g_t^q(\beta^\ast_n))$, we follow the nid-estimator of Hendricks1992. As the degree of misspecification in the investigated financial time series is small BayerDimi2019, we approximate $F_t(g_t^q(\beta^\ast_n)) \approx \alpha$. For the conditional truncated variance, $\operatorname{Var}_t \big( g^q_t( \beta^\ast_n)- Y_{t+1} \big| Y_{t+1} \le g^q_t( \beta^\ast_n) \big)$, we employ the scl-sp estimator of DimiBayer2019.
We now consider the asymptotic distributions of our three ES encompassing tests proposed in Section (ref) under the null hypotheses and for general link functions, where we test certain $s$-dimensional ($s \in \mathbb{N}, s \le k$) sub-vectors of $\theta$. For this, let $R \in \mathbb{R}^{k \times s}$ be a selection matrix whose columns consist of $k$-dimensional Cartesian unit (column) vectors $e_j \in \mathbb{R}^k$, which are zero apart from a one in dimension $j$. E.g., when $g_t^q(\beta)$ and $g_t^e(\eta)$ equal the linear link functions with intercept, given in the first point of Example (ref), $\theta = (\beta_1, \beta_2, \beta_3, \eta_1, \eta_2, \eta_3)$. Then, for the strict and auxiliary ES encompassing tests, $R = (e_5, e_6)$ and for the joint test $R = (e_2,e_3,e_5, e_6)$. These choices pick the respective parameters from $\theta$. Then, we define the respective test statistics by
For linear link functions, this theorem implies that the limiting $\chi^2$ distribution of the joint test has four degrees of freedom, while the one of the strict and auxiliary tests has two degrees of freedom.
An important application of these ES encompassing tests is in the context of selecting the best-performing forecast, i.e.\ selecting at time $T$ a superior forecasting method for the future. This is particularly relevant as the ES is recently introduced into the Basel regulations without having proper forecast selection procedures at hand. Following GiacominiKomunjer2005, we propose the following decision rule. We test the two encompassing hypotheses $\mathbb{H}_{0}^{(1)}$: $\hat e_{1,t}$ encompasses $\hat e_{2,t}$ and $\mathbb{H}_{0}^{(2)}$: $\hat e_{2,t}$ encompasses $\hat e_{1,t}$ for $t = m,\dots,T-1$. Then, there are four possible scenarios: (1) if neither $\mathbb{H}_{0}^{(1)}$ nor $\mathbb{H}_{0}^{(2)}$ are rejected, the test is not helpful for forecast selection. (2) If $\mathbb{H}_{0}^{(1)}$ is rejected while $\mathbb{H}_{0}^{(2)}$ is not rejected, we can conclude that forecast $\hat e_{2,t}$ does add information to forecast $\hat e_{1,t}$, while we cannot conclude the reverse. Thus, we decide to use the forecasting method of $\hat e_{2,t}$. (3) If $\mathbb{H}_{0}^{(2)}$ is rejected while $\mathbb{H}_{0}^{(1)}$ is not rejected, the same logic applies inversely and we use the forecasting method of $\hat e_{1,t}$. (4) If both, $\mathbb{H}_{0}^{(1)}$and $\mathbb{H}_{0}^{(2)}$are rejected, the test delivers statistical evidence that both forecasts contain exclusive information and that a forecast combination outperforms the stand-alone forecasts. Consequently, we use a combined forecast $\hat e_{c,t} = g^e(\hat{\boldsymbol{e}}_t, \hat \eta_n)$ where the estimated combination parameters $\hat \eta_{n}$ are obtained from the M-estimator proposed here.
Testing forecast encompassing conditional on some information set $\tilde{\mathcal{G}}_t = \sigma\{ \boldsymbol{W}_t \}$ based on some $\mathcal{F}_t$-measurable vector of instruments $\boldsymbol{W}_t$ in the sense of GiacominiKomunjer2005 can be facilitated through estimating the regression parameters through (overidentified) GMM-estimation instead of M-estimation. However, for the strict ES test, this approach requires asymptotic theory under model misspecification for the overidentified GMM estimator based on nonsmooth objective functions. While such theory is available for smooth moment conditions (see e.g. HallInoue2003 and HansenLee2019), its generalization to nonsmooth objective functions is not straight-forward and thus, we leave conditional ES encompassing tests based on misspecified GMM-estimation for future research. The moment conditions of our unconditional approach can be interpreted as conditional encompassing with respect to the instruments $\nabla g^q_t(\beta)$ and $\nabla g^e_t(\eta)$. In the classical baseline case of linear forecast encompassing, these instruments simplify to $\hat{\boldsymbol{q}}_t$ and $\hat{\boldsymbol{e}}_t$ and thus, our approach tests conditional encompassing with respect to the information set $\mathcal{G}_t = \sigma\{ 1, \hat{\boldsymbol{q}}_t, \hat{\boldsymbol{e}}_t \} \subseteq \mathcal{F}_t$, which in most cases already captures the most relevant information which is available.
In this section, we evaluate the size and power properties of our three proposed ES encompassing tests and compare them to the VaR encompassing test of GiacominiKomunjer2005. For this, we describe the simulation setup in Section (ref) and we report and discuss the simulation results in Section (ref). Section (ref) considers three extensions of the simulation setup with respect to additional data generating processes (DGPs), loss and link functions.
We employ the three encompassing tests based on the linear link functions $g^q( \hat{\boldsymbol{f}_t}, \beta) = \beta_1 + \beta_2 \hat f_{1,t} + \beta_3 \hat f_{2,t}$ and $g^e( \hat{\boldsymbol{e}}_t, \eta) = \eta_1 + \eta_2 \hat e_{1,t} + \eta_3 \hat e_{2,t}$, where $\hat{\boldsymbol{f}_t} = \hat{\boldsymbol{q}}_t$ for the joint and auxiliary tests and $\hat{\boldsymbol{f}_t} = \hat{\boldsymbol{e}}_t$ for the strict test, together with the parameter space $\Theta = \{ \theta = (\beta, \eta) \in \mathbb{R}^6: ||\theta|| \le K \}$.\footnote{ We choose the constant $K$ large enough such that the parameter estimation is not restricted in realistic settings but the parameter space $\Theta$ is indeed convex.} For the respective encompassing tests, in each case we test the following two opposing hypotheses:
In the following, we describe two DGPs where for the first, both forecasting models stem from classical GARCH models while the second considers two joint GAS models for the VaR and ES of Patton2019. For both model classes, we simulate data as a convex combination of two distinct models with a flexible convex combination weight $\pi \in [0,1]$. This implies that for $\pi = 0$, the first model encompasses the second, while for $\pi=1$, the inverse holds. For all intermediate parameters $\pi \in (0,1)$, the data stems from a linear combination and both forecast encompassing null hypotheses should be rejected which indicates that a forecast combination method is preferred.
The two GARCH models, which are calibrated to daily IBM returns, are given by $\tilde Y_{j,t+1} = \hat \sigma_{j,t} u_{t+1}$, for $j=1,2$, where $u_{t+1} \stackrel{iid}{\sim} \mathcal{N}(0,1)$ and the two distinct volatility specifications are given by
For both models, we obtain VaR and ES forecasts by $\hat q_{j,t} = z_\alpha \hat \sigma_{j,t}$ and $\hat e_{j,t} = \xi_\alpha \hat \sigma_{j,t}$, for $j=1,2$, where $z_\alpha$ and $\xi_\alpha$ are the $\alpha$-quantile and $\alpha$-ES of the standard normal distribution. Notice that the time index $t$ on $\hat \sigma_{j,t}$ indicates that it is a $\mathcal{F}_t$-measurable forecast for time $t+1$. While the first specification in ((ref)) is a classical GARCH(1,1) model Bollerslev1986, the second specification in ((ref)) follows the GJR-GARCH model of Glosten1993, which allows for a leverage effect. We simulate data from the convex combination of these processes, $Y_{t+1} = \big( (1-\pi) \hat \sigma_{1,t} + \pi \hat \sigma_{2,t} \big) u_{t+1}$ for 21 equally spaced values of $\pi \in [0,1]$, where $u_{t+1} \stackrel{iid}{\sim} \mathcal{N}(0,1)$.
In the second simulation setup, we implement the one-factor (1F) and two-factor (2F) GAS models for the VaR and ES of Patton2019. The 1F-GAS model evolves as
The 2F-GAS model follows the specification
where the forcing variable is given by $\lambda_t = \big( \hat q_{2,t-1} ( \alpha - \mathds{1}_{\{\tilde Y_{2,t} \le \hat q_{2,t-1} \}} ) , \, \mathds{1}_{\{\tilde Y_{2,t} \le \hat q_{2,t-1} \}} \tilde Y_{2,t} / \alpha - \hat e_{2,t-1} \big)^\top$. For both models, $j=1,2$, we simulate $\tilde Y_{j,t+1} \sim \mathcal{N} \big( \hat \mu_{j,t}, \hat \sigma_{j,t}^2 \big)$, where the conditional mean and standard deviations are given by $\hat \mu_{j,t} = \hat q_{j,t} - z_\alpha \frac{\hat e_{j,t} - \hat q_{j,t}}{\xi_\alpha - z_\alpha}$ and $\hat \sigma_{j,t} = \frac{\hat e_{j,t} - \hat q_{j,t}}{\xi_\alpha - z_\alpha}$, such that $Q_\alpha(\tilde Y_{j,t+1}|\mathcal{F}_t) = \hat q_{j,t}$ and $ES_\alpha(\tilde Y_{j,t+1}|\mathcal{F}_t) = \hat e_{j,t}$ almost surely. The parameter values for this model are obtained from Table 8 of Patton2019 and correspond to calibrated parameters to daily S&P 500 returns.
In order to simulate returns which follow a convex combination of these two conditional distributions, we simulate Bernoulli draws $\pi_{t+1} \sim \operatorname{Bern}(\pi)$ for 21 equally spaced values of $\pi \in [0,1]$, and let $Y_{t+1} = (1-\pi_{t+1}) \tilde Y_{1,t+1} + \pi_{t+1} \tilde Y_{2,t+1}$. Thus, for $\pi=0$, $Y_{t+1}$ follows the 1F-GAS model, for $\pi = 1$, $Y_{t+1}$ follows the 2F-GAS model and for $\pi \in (0,1)$, $Y_{t+1}$ follows some convex combination of these two models.\footnote{ While generating returns stemming from convex combinations of GARCH-type volatility models is straight-forward by using convex combinations of the conditional volatilities, this is not as simple for the more general GAS models considered in this section. Consequently, we use this more involved approach based on Bernoulli draws in order to generate these convex model combinations.}
Both models in the GARCH DGP generate data from a pure scale (volatility) process resulting in perfectly colinear VaR and ES forecasts. In contrast, the more general VaR/ES GAS models in the second DGP generate VaR and ES forecasts which are not colinear and consequently introduce misspecification in the quantile model of the strict ES encompassing test. As the utilized parameters are calibrated to daily financial returns, these models reflect a realistic degree of misspecification encountered in practical risk management.
Table (ref) reports the empirical sizes of the three different ES encompassing tests introduced in Section (ref) together with the VaR encompassing test of GiacominiKomunjer2005 at a 10$\%$ nominal significance level based on $2000$ Monte Carlo replications. Table S.1 and S.2 in the supplementary material present equivalent results for nominal sizes of 1$\%$ and 5$\%$. The column panel $\mathbb{H}_0^{(1)}$ indicates that we test whether model 1 encompasses model 2, while the panel $\mathbb{H}_0^{(2)}$ indicates the reverse.
We find that the two ES encompassing tests (the strict and auxiliary test) are well-sized, especially in large samples for both DGPs and for both null hypotheses. While the joint VaR and ES test is slightly oversized, the VaR test exhibits even larger sizes. This behavior is especially remarkable as the ES is considerably further in the tail than the VaR at the same probability level and hence, harder to estimate and test. This pattern can be explained by the fact that the asymptotic covariance of the two tests involving the VaR is subject to estimation of the density quantile function $h_t(g_t^q( \beta^\ast_n))$, which is hard to estimate for small probability levels Koenker1978, GiacominiKomunjer2005, DimiBayer2019.
We further find that the strict and the auxiliary tests behave almost identically. This also holds for the VaR/ES GAS DGP for which the regression model of the strict ES encompassing test is potentially misspecified.\footnote{ Figure S.1 in the supplementary material plots the ratio of the VaR and ES forecasts for a simulated return series of the 2F-GAS model (and further misspecified models described in Section S.1 in the supplementary material). Following the discussion after Definition (ref), this ratio mainly governs the degree of misspecification the regression model of the strict ES encompassing test is subject to. For the 2F-GAS model, the ratio of the VaR and ES forecasts fluctuates approximately between 0.7 and 0.85, while it equals 0.84 for the location-scale approaches under normality. This demonstrates that the VaR/ES GAS simulation designs, which are calibrated to real financial data, generate some moderate degree of model misspecification in the regression model.} This suggests that the approximation error induced by the misspecification in the strict ES test is negligible for realistic financial settings. Remarkably, in the vast majority of cases, the strict ES test exhibits better size properties than the correctly specified joint VaR and ES and the VaR encompassing tests.
We present power curves (empirical rejection rates) for both DGPs and different sample sizes in the individual plot panels in Figure (ref). In each plot, we depict the respective power curves for our three ES encompassing tests and the VaR encompassing test of GiacominiKomunjer2005 for both null hypotheses and for a nominal significance level of $10\%$ based on $2000$ Monte Carlo replications. We observe increasing power for both DGPs, both tested null hypotheses and for all four encompassing tests for increasing (decreasing) values of the combination parameter $\pi$. We find that while the VaR and joint VaR and ES tests are considerably oversized, they produce a similar test power compared to the strict and auxiliary ES encompassing tests, especially for larger (smaller) values of $\pi$. Again, the strict and auxiliary ES encompassing tests are almost indistinguishable, which implies that the strict test is robust against the misspecification induced by the VaR/ES GAS models. Interestingly, we find that the power curves for the VaR/ES GAS specifications are slightly asymmetric implying that the tests react differently to the specifications of different numbers of driving factors in the GAS models.
In this section, we consider three extensions of our simulation setup. First, in Section (ref), we present two additional DGPs. Second, in Section (ref), we analyze the behavior of our tests under different strictly consistent loss functions for the pair VaR and ES. Third, in Section (ref), we employ two additional link functions by testing for forecast encompassing for affine and nonlinear forecast combinations.
In this subsection, we consider two additional DGPs, namely the GAS-$t$ model of Creal2013 and the ES-CAViaR models of Taylor2019, which are described in detail in Section S.1 in the supplementary material. Both models go beyond the class of pure scale models and consequently generate model misspecification in the quantile regression equation of the strict ES encompassing test. Figure (ref) presents power curves for these two DGPs and Table S.3 in the supplementary material reports the corresponding test sizes.
The results of these two DGPs qualitatively confirm the simulation results of Section (ref). The three ES specific tests exhibit accurate empirical test sizes, especially in large samples and the strict and the auxiliary tests generally exhibit better size properties than the joint VaR and ES and the stand-alone VaR encompassing tests. Increasing the sample size results in increasing power for both DGPs and for all considered encompassing tests. Noteworthy, all tests show considerably lower power for the ES-CAViaR DGP compared to the other DGPs. This result is comparable with the power results of GiacominiKomunjer2005 as this DGP is a slightly modified version of their DGP. In summary, these two additional DGPs demonstrate that the new ES encompassing tests perform well for a variety of realistic data generating processes.
The three encompassing test specifications for the ES presented in Section (ref) are built on the zero-homogeneous joint loss function for the VaR and ES given in ((ref)). While this loss function is the most popular choice in the recent literature on semiparametric ES models Patton2019, BayerDimi2019, Taylor2019, there exists an entire class of joint loss functions for the VaR and ES, proposed by Fissler2016 as
where the function $\mathfrak{g}$ is twice continuously differentiable and increasing, $\phi$ is three times continuously differentiable, strictly increasing and strictly convex, and $a$ and $\mathfrak{g}$ are integrable functions Fissler2016. The loss function in ((ref)) is a special case of ((ref)) for the choices $\mathfrak{g}(z) = 0$, $\phi(z) = -\log(-z)$ and $a(z) = 0$.
The ES encompassing tests can generally be set up by using any choice of ((ref)) (fulfilling certain further weak regularity conditions). We consider two additional specifications in the following. Following the theory of homogeneous loss functions Nolde2017 and the numerical performance in linear regression settings DimiBayer2019, we fix $\mathfrak{g}(z) = 0$ and in addition to $\phi(z) = -\log(-z)$, we employ the choices $\phi(z) = 1/\sqrt{-z}$ and $\phi(z) = - 1/z$.\footnote{Strictly speaking, the asymptotic theory in Theorem (ref) only covers the M-estimator based on the loss function in ((ref)). However, the proofs and the resulting asymptotic covariance matrices are easily extended to the general case by combining the methods of this paper with the extension of DimiBayer2019 to the case of general loss functions as given in ((ref)).}
Figure (ref) shows rejection rates for the VaR/ES GAS DGP for different sample sizes and the three encompassing test specifications for the ES, where the different line colors represent the three different loss specifications. Table S.4 in the supplementary material reports the corresponding test sizes. We find that the tests based on the three different loss functions perform almost indistinguishably, especially in large samples. This result is not unexpected (especially in large samples where the expectation is well approximated by the sample mean) as Fissler2016 show that all loss functions in the class in ((ref)) are uniquely minimized by the true VaR and ES.
For the potentially misspecified strict ES encompassing test, the pseudo-true parameter defined in ((ref)) may theoretically depend on the underlying loss function and notice that the VaR/ES GAS DGP used in Figure (ref) allows for such a model misspecification. However, we find that neither the rejection rates of the strict ES encompassing test are affected by employing different loss functions, nor are the rejection rates of the two correctly specified encompassing tests in Figure (ref). This result implies that the potentially different pseudo-true parameters are almost entirely unaffected by the misspecification.
In this section, we employ two additional link function specifications. First, we consider an affine combination including an intercept\footnote{We include an intercept as this stabilizes the performance of the associated quantile regression. We do not include classical convex combinations (where $0 \le \beta_2 \le 1$) as our theoretical framework does not allow for testing on the boundary (see e.g. Andrews1999 for details).}
where $\hat{\boldsymbol{f}_t} = \hat{\boldsymbol{q}}_t$ for the joint and auxiliary tests and $\hat{\boldsymbol{f}_t} = \hat{\boldsymbol{e}}_t$ for the strict test. For the joint test, the first null hypothesis is given by $\mathbb{H}_0^{(1)}: (\beta_2^\ast, \eta_2^\ast) = (1,1)$ while for the strict and auxiliary tests, it is given by $\mathbb{H}_0^{(1)}: \eta_2^\ast = 1$. The second, opposing null hypotheses, $\mathbb{H}_0^{(2)}$ are obtained by replacing the ones by zeros. For the affine link functions, we employ the same DGPs as for the encompassing tests based on linear link functions.
Furthermore we employ the nonlinear link functions, where $\hat{\boldsymbol{f}_t}$ is given as in ((ref)) and
and we test the same null hypotheses as for the linear link functions described in ((ref)). For the nonlinear link function, we employ a slightly modified GARCH DGP. As in Section (ref), let $\hat \sigma_{1,t}$ and $\hat \sigma_{2,t}$ denote the conditional volatilities of the GARCH and GJR-GARCH models. Then, we simulate data according to
for an equally spaced grid of 21 values for $\pi \in [0,1]$ and where $u_{t+1} \stackrel{iid}{\sim} \mathcal{N}(0,1)$. This ensures that for $\pi=0$, $\hat q_{1,t}$ and $\hat e_{1,t}$ are the correct VaR and ES forecasts, and vice versa for $\pi=1$. For any $\pi \in (0,1)$, the true VaR and ES are given by a combination according to the nonlinear link functions in ((ref)).
Figure (ref) presents power curves for these two additional link functions, where for the affine combination, we consider both, the GARCH and the VaR/ES GAS DGPs. Table S.5 in the supplementary material presents the associated test sizes. We find that the power curves for the affine combinations in Figure (ref) are comparable to the linear specifications, as given in Figure (ref) for both considered DGPs. Moreover, the encompassing tests based on the nonlinear link function in the third row of plots perform similar to the linear encompassing tests in the GARCH setting. This extension of the simulation setup shows that our ES encompassing tests can be applied based on a variety of different link functions.
We use close-to-close returns from the IBM stock, the S$\&$P 500 index and the DAX 30 index from June 1st, 2000 until May 31st, 2019, which amounts to a total of $T = 4779$ daily observations. We use a fixed forecasting scheme, i.e.\ the model parameters are estimated once on the first $m=2000$ in-sample observations. These parameter estimates are used to generate the VaR and ES forecasts in a rolling-window fashion for the remaining out-of-sample period of $n = 2779$ days. Following the suggestion of the Basel III Accords, we use the probability level $\alpha = 2.5\%$ for the VaR and the ES.
For the analysis, we consider the following competing forecasting models. First, we employ the Historical Simulation (HS) model which generates VaR and ES forecasts by computing the empirical quantile and ES at level $\alpha$ of the past 250 trading days. The second model is the RiskMetrics (RM) model, which models the conditional volatility as an IGARCH equation with fixed parameter values, $\hat \sigma_{t}^2 = 0.94 \hat \sigma_{t-1}^2 + 0.06 Y_{t}^2$ and Gaussian residuals. Third, we use the GJR-GARCH(1,1)-$t$ model of Glosten1993 with Student-$t$ residuals. The fourth model is given by the Student-$t$-GAS model with time-varying variance and degrees of freedom introduced in Section S.1 in the supplementary material. The fifth and sixth model are the one and two factor GAS models for the VaR and ES of Patton2019 set out in Section (ref) and estimated by minimizing the strictly consistent loss function for the VaR and ES given in ((ref)). The last two models are the two dynamic ES-CAViaR models of Taylor2019 described in Section S.1 in the supplementary material. Table S.6 in the supplementary material shows the correlations of the respective VaR and ES forecasts of these models. We find that no pair of forecasts is perfectly correlated, which is crucial for the applicability of the encompassing tests as implied by condition (ref) of Assumption (ref).
We run pair-wise encompassing tests comparing all eight forecasting methods. Hence, for each model pair, we run encompassing tests for both hypotheses, i.e.\ that the first forecast encompasses the second, denoted by $\mathbb{H}_0^{(1)}$ and the inverse, denoted by $\mathbb{H}_0^{(2)}$. This results in four possible outcomes of these two tests: (1) non-rejection (NR) indicates that none of the null hypotheses is rejected and the tests are not helpful. (2) encompassed (E1) denotes the setting where the first model is encompassed by the competitor model but does not encompass it, i.e.\ $\mathbb{H}_0^{(1)}$ is rejected but $\mathbb{H}_0^{(2)}$ is not, which results in choosing the competitor model. (3) encompassing (E2) indicates that the first model encompasses the other but is not encompassed by it, i.e.\ $\mathbb{H}_0^{(1)}$ is not rejected but $\mathbb{H}_0^{(2)}$ is, which implies that we choose the first model. Finally, (4) combination (C) refers to a setting where both null hypotheses are rejected and we consequently opt for a forecast combination.
In Table (ref), we report relative frequencies of test outcomes at the $10\%$ significance level for the different encompassing tests for all three return time series. Tables S.7, S.8, S.9 in the supplementary material report the individual $p$-values of the encompassing tests. The results can be summarized as follows: first, for the IBM stock returns we find many cases of double rejections and hence empirical evidence for using forecast combinations. This implies that the individual models provide additional and exclusive information and hence, a forecast combination is often superior to the stand-alone forecasting models. This finding supports the theoretical advantages of forecast combinations, presented in a general setting e.g., by GiacominiKomunjer2005, Timmermann2006 and Halbleib2012, and specifically for the pair VaR and ES by Taylor2019FCcomb. Second, for the S$\&$P 500 index, and especially for the DAX 30 index, we overall observe less instances of double rejections of the ES encompassing tests. While the decrease in cases where the VaR encompassing test opts for a forecast combination is smaller, these rejections have to be considered carefully given that the VaR encompassing test is oversized in all simulation setups in Section (ref), even in large samples. This result can be explained by the fact that the S$\&$P 500 and DAX indices are well diversified and the returns fluctuate to a lesser extent and exhibit less extreme outliers than single stock return series. Furthermore, the considered VaR and ES forecasts show larger correlations for the indices than for the single stock in Table S.6 in the supplementary material, which negatively influences the tests' power. Third, in terms of the frequencies of the cases E1 and E2, we observe recurring patterns over the different models for both time series. Especially the ES-specific GAS and CAViaR type models seem to exhibit a superior performance, while the HS, RM, GARCH and GAS-$t$ models tend to be encompassed more often. Lastly, the two tests which only focus on testing encompassing of ES forecasts perform almost identically, which supports the conclusion from the simulation study that the potential misspecification does not negatively influence the performance of the strict ES test in realistic financial settings. This is encouraging as the strict ES encompassing test can be applied in cases where one does not have VaR forecasts at hand, such as it is currently imposed by the Basel Committee of Banking Supervision Basel2016, Basel2017.
Tables S.10, S.11 and S.12 in the supplementary material report the joint VaR and ES losses for the zero-homogeneous loss function in ((ref)) for forecasts stemming from the stand-alone models and the respective forecast combinations with estimated combination weights from the underlying regressions. These results qualitatively confirm the results of the encompassing tests: e.g., for the IBM stock in the first panel of Table (ref), we find that forecast combinations are particularly preferred for the first four models (in the model ordering of the table). The average losses in Table S.10 show a similar pattern throughout all three panels. We observe that the optimal forecast combinations exhibit substantially smaller losses compared to the stand-alone models for the first four models while this decrease is of a considerably smaller magnitude for the last four models.
\FloatBarrier
With the implementation of the third Basel Accords Basel2016,Basel2017, risk managers and regulators currently shift attention towards the risk measure Expected Shortfall (ES), which demonstrates the necessity of forecast evaluation and comparison tools for the ES. In this paper, we introduce new forecast encompassing tests for the ES, which are based on a joint loss function and an associated joint regression framework for the ES together with the Value at Risk Fissler2016, Patton2019, DimiBayer2019. We propose three variants of the ES encompassing test, which allow for testing forecast encompassing for flexible parametric forecast combination methods. As one test variant is potentially subject to model misspecification, we extend the existing asymptotic theory of Patton2019, DimiBayer2019 and BayerDimi2019 to cases of potential model misspecification for flexible parametric models. Potential future research on extending the setting presented in this paper includes encompassing tests for convex forecast combinations and forecast combinations which theoretically prevent crossings of the VaR and the ES. Both approaches require non-standard asymptotic theory for tests on the boundary of the parameter space.
Tests for forecast encompassing establish a theoretical foundation for forecast combinations of two competing forecasts when both opposing hypotheses of forecast encompassing are rejected. This situation corresponds to the case when neither forecast encompasses its competitor. Generally, applying forecast combinations can be highly beneficial through the diversification gains stemming from combining different model specifications and underlying information sets. This benefit can be particularly pronounced for extreme risk measures such as the ES Taylor2019FCcomb, as the stand-alone models are very sensitive to the very little observations in the tails of the return distributions. Thus, combining forecasts can be seen as a robustification of the forecasts. Our application empirically validates this conjecture, especially pronounced for daily returns from the IBM stock.
While we propose encompassing tests suitable for the cases that the ES is reported jointly with and without its accompanying VaR forecast, testing encompassing for the VaR and ES jointly is most natural given their joint elicitability. Furthermore, this joint elicitability property clearly hints towards reporting ES forecasts jointly with their corresponding VaR forecasts by default. In contrast, the auxiliary test can be seen as the first forecast comparison procedure for the ES, which focuses almost entirely on the ES. Theoretically, its application is most reasonable in cases where the primary focus of the researcher is on the ES, even though VaR forecasts are available. Eventually, the strict ES test is specific to scenarios where only competing ES forecasts are available. Furthermore, in cases where competing ES forecasts are built on the same VaR model (forecasts), the strict ES test is applicable while the application of the joint and auxiliary tests is infeasible due to the collinearity of the (identical) VaR forecasts.
\if00 {
For helpful comments, we thank the editor (Michael McCracken), the associate editor and two referees, as well as Sander Barendse, Sebastian Bayer, Ralf Brüggemann, Joachim Grammig, Alastair Hall, Andrew Patton, James Taylor and the seminar participants at Universität Konstanz, Duke University, Universität Hohenheim, the 2019 QFFE conference in Marseille, the 2019 IAAE conference in Nicosia, the 2019 ESEM conference in Manchester, the 2019 Statistische Woche in Trier and the 2019 Financial Econometrics Workshop in Örebro. Financial support by the Klaus Tschira Foundation, the Universität Hohenheim, the Graduate School of Decision Sciences (GSDS) and a travel fund of the IAAE is gratefully acknowledged.
} \fi
\FloatBarrier \singlespacing {6pt}