Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
77,156 characters · 8 sections · 54 citation commands
Proper scoring rules for evaluating asymmetry in density forecasting
Macroeconomic forecasting has always been of pivotal importance for central bankers, policymakers and researchers. Nowadays, the vast majority of the research in macroeconomics and finance mainly focuses on the development and implementation of forecasting techniques that minimize the expected squared forecast error (gneiting2011making). This approach is grounded on the implicit assumption of using a symmetric loss function in evaluating the accuracy of a forecast.
Despite being common practice, the use of symmetric loss functions in forecasting is unrealistic especially in policy institutions, where the policymakers could have a specific aversion to positive or negative deviations of a forecast from the target. Consider a policymaker who is interested in forecasting employment. Suppose that, if the predicted employment rate drops below a given threshold, she will be forced to adopt new expansionary economic policy. It is highly likely that the policymaker is more averse to forecasts that give too high probability mass to the right part of the distribution of the employment rate (positive growth of employment), while she may be more relaxed with respect to forecasts that give too high probability mass to the left part of the distribution (negative or low growth of employment). Other examples relate to energy markets that have recently experienced negative prices. WTI oil prices collapsed to -37.63 US dollar for barrel in April 2020; German electricity prices have measured several negative prices with the introduction of renewable energy resources (RES). Producers would be more sensitive to prices below a threshold, up to zero if the marginal cost of production is zero, as it is the case of RES, than higher prices. These examples call for the design of a more general class of loss functions and scoring rules that account for asymmetry, in order to guide the process of making and assessing forecasts. To the best of our knowledge, a measure that properly incorporates asymmetry in density forecasting evaluation does not exist in the literature.
The main goal of this paper is the proposal of novel and practical forecasting evaluation tools that can fill in this gap and answer the increasing demand from policymakers and central bankers. We plan to achieve this result by introducing an innovative asymmetric scoring rule that is able to measure and evaluate heterogeneous aversion to different deviations of a density forecast from the target. We derive some properties of the new scoring rule and in particular demonstrate that it is a proper scoring rule. Moreover, we provide threshold- and quantile-weighted versions that allow to emphasize the performance of the forecast in regions of interest to the policymaker.
Within the literature on point forecasting, Christoffersen1996,Christoffersen1997 proposed some asymmetric loss functions. In the former paper, they studied the optimal prediction problem under general loss structures and characterized the optimal predictor under an asymmetric loss function, focusing on the LinEx and the LinLin asymmetric functions. In the latter paper, the authors provided an illustration of an asymmetric loss in the context of GARCH processes.
More recently, scholars have begun to empirically investigate the degree of loss function asymmetry of central banks and other international institutions. Among others, Elliott2005 and Patton2007 proposed formal methods to infer the degree of asymmetry of the loss function and to test the rationality of forecasts. Within this stream of literature, Artis2001 found that IMF and OECD forecasts of the deficit of G7 countries are biased towards over or under-prediction relative to mean square error (MSE) forecasts. Regarding European institutions forecasts, Christodoulakis2008,Christodoulakis2009 found evidence of asymmetric loss. In another study, Dovern2017 documented that the GDP growth forecasts made by professional forecasters tend to exhibit systematic errors, and tend to overestimate GDP growth. Moreover, Boero2008 interpreted the tendency to over-predict GDP growth as a signal that policymakers exhibit greater fear of under-prediction than over-prediction, thus suggesting that their judgements are based on an asymmetric loss. More recently, Tsuchiya2016 examined the asymmetry of the loss functions of the Japanese government, the IMF and private forecasters for Japanese growth and inflation forecasts.
In the framework of forecast combination, elliott2004optimal showed that the optimal combination weights significantly differ under asymmetric loss functions and skewed error distributions as compared to those obtained with mean squared error loss. Finally, demetrescu2019predictive studied factor-augmented forecasting under asymmetric point loss function.
An alternative and more universal approach to forecasting is the provision of a predictive density, known as probabilistic or density forecasting (see elliott2016economic). Two key aspects of density forecasts are the statistical compatibility between the forecasts and the realized observations (calibration) and the concentration of predictive distributions (sharpness). The aim of probabilistic forecasts is to maximize their sharpness, subject to calibration (gneiting2013combining). Density forecasting is more complex than point forecasting since the estimation problem requires to construct the whole predictive distribution, rather than a specific function thereof (e.g., mean or quantile). Several reasons have been suggested for preferring density over point forecasts (e.g., elliott2016forecasting). First, point forecasting is often associated to the mean of a distribution and it is optimal for highly restricted loss functions, such as quadratic loss function, but inadequate for any prospective user having a different loss. Moreover, the value of a point forecast can be increased by supplementing it with some measures of uncertainty and complete probability distributions over the outcomes provide useful information for making economic decisions; see, for example, Anscombe1968 and Zarnowitz1969 for early works and the discussions in gp00, Timmermann2006 and gneiting2011making. CCM2020 extends the application to tail risk nowcasts of economic activity. Finally, in recursive forecasting with nonlinear models the full predictive density matters since the nonlinear effects typically depend not only on the conditional mean, but also on where future values occur in the set of possible outcomes.
A natural way to evaluate and compare competing density forecasts is the use of proper scoring rules, which assess calibration and sharpness simultaneously and encourage honest and careful forecasting. Despite the wide literature on the class of proper scoring rules for probabilistic forecasts of categorical and binary variables (e.g., see savage1971elicitation, schervish1989general) the advances for continuous variables are more limited. Motivated by these facts, we aim at designing a novel asymmetric proper scoring rule to be used for evaluating density forecasts of continuous variables, which is the typical case in macroeconomics and finance exercises (e.g., predicting variables such as unemployment, inflation, log-returns, GDP growth, and realized volatility).
gneiting2007strictly proposed the continuous rank probability score (CRPS) as a proper scoring rule for probabilistic forecasts of continuous variables, and more recently, gneiting2011comparing extended the CRPS by introducing a threshold- and a quantile-weighted version (tCRPS and qCRPS, respectively). These scoring rules give more emphasis to the performance of the density forecast in a selected region of the domain, $B$, by assigning more weight to the deviations from the observations made in $B$. The major drawback of both the CRPS and its weighted versions is the symmetry of the underlying reward scheme, meaning that they assign equal reward to positive and negative deviations of a probabilistic forecast from the target. This comes from the fact that the CRPS is built on the Brier score and inherits some of its properties, such as properness and symmetry. Similarly, since both the weighted versions of the CRPS essentially consist in re-weighting the CRPS over the domain of the variable of interest, they inherit the symmetry of the latter.
winkler1994evaluating did a first effort towards asymmetric scoring rules and proposed a general method for constructing asymmetric proper scoring rules starting from symmetric ones. However, this approach is limited to forecasting binary variables, and continuous variables were not investigated.
We address this issue and contribute to the literature on proper scoring rules for evaluating density forecasts by proposing a novel asymmetric proper scoring rule which assigns different penalties to positive and negative deviations from the true density. The main contribution of this paper is twofold. First, we define a new proper scoring rule which assigns an asymmetric penalty to deviations from the target density. Moreover, we provide a threshold- and quantile-weighted version of it and develop an adaptation of the Diebold-Mariano test to statistically compare the predictive ability of different forecasts. Then, we compare the performance of the scores with the CRPS and its weighted versions. Second, we use the proposed score to evaluate density forecasts in three relevant applications in macroeconomics (US employment growth) and commodity prices (oil and electricity prices) with data updated to the COVID-19 crisis period. Variables have experienced large volatilities, with sizeable spikes and negative energy prices. As we discussed above, players might be more sensitive to some specific parts of the distribution of these series and we shed light on how to evaluate this asymmetry.
The key result of this paper is the provision of a tool able to account for the decision maker's preferences in the evaluation of density forecasts, both in terms of domain- and error-weighting schemes. Domain-weighting gives heterogeneous emphasis to the performance on different regions, while the error-weighting asymmetrically rewards negative and positive deviations from the target value. The proposed weighted asymmetric scoring rule combines the two schemes and allows to evaluate the performance of the forecasting density from both perspectives.
The rest of the paper is organized as follows. Section (ref) presents a novel asymmetric scoring rule for density forecasts, its extension to threshold- and quantile-weighted versions and a test to compare the predictive accuracy of different forecasts. Then Section (ref) discusses its main properties. It also illustrates a comparison with the (weighted) CRPS in simulated experiments. Finally, Section (ref) provides different applications on forecasting US macroeconomic variables (employment rate) and commodity prices (oil and electricity prices). The article closes with a discussion in Section (ref).
The MATLAB code for implementing the proposed scoring rules is available at:
The evaluation and comparison of probabilistic forecasts typically relies on proper scoring rules. Informally, a scoring rule is a measure that summarises the goodness of a probabilistic forecast by combining the predictive distribution and the value that actually materializes. One can think of it as a measure of distance between the probabilistic forecast and the actual value. We consider positively oriented scoring rules, therefore if probabilistic forecast $P_1$ obtains a higher score than $P_2$, this means that $P_1$ yields a more accurate forecast than $P_2$. Therefore, the score can be interpreted as a reward to be maximized.
In more formal terms, following the notation of gneiting2007strictly, consider the problem of making probabilistic forecasts on a general sample space $\Omega$. Let $\mathcal{A}$ be a $\sigma$-algebra of subsets of $\Omega$, and let $\mathcal{P}$ be a convex class of probability measures on $(\Omega,\mathcal{A})$. A probabilistic forecast is any probability measure $P \in \mathcal{P}$, such that $P : \Omega \to \bar{\mathds{R}}$, where $\bar{\mathds{R}} = [-\infty,+\infty]$ denotes the extended real line, is said to be $\mathcal{P}$-quasi-integrable if it is measurable with respect to $\mathcal{A}$ and is quasi-integrable with respect to all $P \in \mathcal{P}$ (see bauer2011measure). A scoring rule is any extended real-valued function $S : \mathcal{P} \times \Omega \to \bar{\mathds{R}}$ such that $S(P,\cdot)$ is $\mathcal{P}$-quasi-integrable for all $P \in \mathcal{P}$. In practice, if $P$ is the forecast density and the event $\omega$ materializes, then the forecaster's reward is $S(P,\omega)$.
In order to be effectively used in scientific forecasts evaluation, scoring rules have to be proper, meaning that they have to reward accurate forecasts. Suppose the true density of the observations is $Q$ and denote the expected value of $S(P,\omega)$ under $Q(\omega)$ with
then the scoring rule $S$ is strictly proper if $S(Q,Q) \geq S(P,Q)$. The equality holds if and only if $P = Q$, thus implying that the forecaster has higher reward if she predicts $P=Q$. If instead $S(Q,Q) \geq S(P,Q)$ for all $P$ and $Q$, then the scoring rule is said to be proper.
The vast majority of the proper scoring rules proposed in the literature are symmetric, that is, they reward in the same way positive and negative deviations from the target. For example, suppose a forecast $P_1$ assigns too high probability mass to the right part of the domain (as compared to the true density) and a forecast $P_2$ assigns too high probability mass to the left part, by the same amount. If these forecasts are evaluated under a symmetric scoring rule, then they receive the same score.
A symmetric loss is unsatisfactory for many real world situations where the decision maker has a preference or aversion towards a particular kind of error. We aim at filling in this gap by defining a new asymmetric proper scoring rule for continuous variables, which is suited for evaluation and comparison of density forecasts and penalises more either side of the deviation from the target.
The following result shows the properness of our new score for every level of asymmetry.
The integrals in eq. (ref) can be numerically approximated by truncating the domain to $[u_{min},y]$ and $[y,u_{max}]$ such that
where $(w_{1,i}^y,u_{1,i}^y)_i$ and $(w_{2,i}^y,u_{2,i}^y)_i$, for $i=1,\ldots,N$, are the weights and locations of two Gaussian quadratures of $N$ points on $[y,u_{max}]$ and $[u_{min},y]$, respectively.
To get an insight of the shape of the ACPS\, for varying levels of asymmetry, $c$, we consider two examples: one with several probabilistic forecasts and the other with a fixed forecast.
In addition to asymmetric preferences towards under- or overestimation, a decision maker is usually concerned with a precise forecast in a specific range of all possible values. Therefore, it is important to have a tool that allows to assign heterogeneous weights to various regions of the set of possible values of the variable. This calls for a scoring rule able to account for both error-weighting, i.e. asymmetric preferences and domain-weighting of density forecasts.
gneiting2011comparing modified the CRPS by re-weighting the loss according to a user-specified weight function, which allows to select the regions where the decision-maker has greater concern. By exploiting the representation of the CRPS in terms of quantile functions, they define a threshold-weighted (tCRPS) and quantile-weighted (qCRPS) score functions as follows
where $w(z) \geq 0$ and $v(\alpha) \geq 0$ are the weight functions and level $\alpha \in (0,1)$. Table (ref) reports some examples of weighting functions for the case of real-valued variables of interest; notice that the uniform weight, $w(z)=1$ and $v(\alpha)=1$, leads to the standard CRPS. See ltrg2017 for discussion and applications of these scoring rules.
The definition of ACPS\, in (ref) can be modified to address this issue and obtain a threshold-weighted and a quantile-weighted asymmetric scoring rule, as follows.
As stated for ACPS, we can provide evidence of the properness of the two novel scores defined in eq. (ref) and eq. (ref).
Both tACPS\, and qACPS\, can be computed by approximating eq. (ref) and eq. (ref) in a way analogous to eq. (ref). The main advantage of the tACPS\, and qACPS\, consists in the ability to consider two levels of asymmetry: in terms of the loss at each point, and over different regions of the domain. This is fundamental to answer the need of the decision maker who is concerned with the performance of the forecast in a given interval of possible values (e.g., the right tail) and who has an aversion to particular deviations from the target (e.g., averse to underestimation).
Tab. (ref) provides a summary of some key differences between the CRPS and ACPS, and the corresponding weighted versions.
In the literature on asymmetric point forecasting measures, the choice of the shape parameter(s) of the loss function has been of interest especially over the last decade. Some works, such as Christoffersen1996 and demetrescu2019predictive performed an empirical exercise to rank competing forecasting models under asymmetric point forecast measure using a grid of asymmetry values.
Instead, Elliott2005,elliott2008biases and Patton2007 introduced procedures for inferring the value of the shape parameters of the forecaster's loss function. Assuming a collection of time series of forecasts of a given quantity of interest is available, they treat the loss function parameters as variable to be estimated and look for the values that would be most consistent with forecast rationality. This approach is appealing since the loss function parameters may provide information about the forecaster's objectives. However, the main drawback of these approaches is that they rely on the availability of a time series of observed forecasts from relevant decision-makers (e.g., the IMF, the OECD, or central bankers). Therefore, the application is case specific for those data.
When forecasts from multiple models are available, there is the need for statistical tools, such as tests, for assessing whether different forecasts are equally good. In the context of point forecasts, the Diebold-Mariano (DM) test is the most frequently used test for equal forecast performance. Essentially, it is based on the loss differential, defined as
where $e_{j,t} = \hat{y}_{j,t} - y_t$ is the forecast error of model $j=1,2$ at time $t=1,\ldots,T$, $\hat{y}_{j,t}$ is the point forecast of model $j$, $y_t$ is the true value, and $L(\cdot)$ is a given loss function. The null hypothesis of equal accuracy in forecasting is $H_0: \mathbb{E}[d_t] = 0$ for all $t$, versus the alternative $H_1: \mathbb{E}[d_t] \neq 0$. It can be shown that, if the loss differential series is (i) covariance stationary, and (ii) has short memory (see e.g. mccracken2020diverging), then under the null hypothesis
where $\bar{d}$ and $f_d(0)$ are the sample mean and the spectral density (at frequency 0) of the loss differential. Recently, mccracken2020diverging found that the slow decay of the loss differential series is the most frequent problem that hampers the use of the Diebold-Mariano test in real data economic applications.
The density forecasting approach requires some adaptations of the Diebold-Mariano test, since the forecast is an infinite dimensional object $P$.
This section investigates the performance of the proposed asymmetric scoring rule and compares it with the CRPS. In order to assess the good performance of our measure, we consider different target densities: (i) Gaussian, (ii) Student-t, (iii) Gamma, (iv) Beta. This range includes families of distributions with different support ($\mathds{R}$, $\mathds{R}_+$ and $[0,1]$), skewed and with fat tails. For the asymmetric scoring rule ACPS \, we use varying levels of asymmetry, corresponding to $c \in \{ 0.05, 0.275, 0.50, 0.725, 0.95 \}$. Recall that $c=0.50$ implies a symmetric loss.
Fig. (ref) and Fig. (ref) provide graphical evidence of the properness of the ACPS \, in two cases, with a Gaussian and a Student-t target, respectively. Both figures show that the ACPS \, rewards the forecast density which corresponds to the ground truth, for all levels of asymmetry. In addition, we find that the ranking of the competing probabilistic forecasts changes according to the value of $c$, due to the different penalty assigned to asymmetric deviations from the target.
To investigate further this aspect, Fig. (ref) presents the ranking of forecasts when none of the candidates corresponds to the true density, which is $\mathcal{N}(2,4)$. The CRPS indicates $\mathcal{N}(3,1)$ as the best forecast, as does the ACPS \, for values of $c$ around $0.5$. However, when the ACPS \, assigns more weight to the asymmetric loss, that is for $c$ close to the boundary of $(0,1)$, the ranking significantly changes. For $c=0.05$, that is when great importance is given to underestimation of the target, the $\mathcal{N}(0,1)$ is preferred, while $\mathcal{N}(0,16)$ is the best for the opposite case, when $c=0.95$.
Many economic and financial variables in levels are inherently positive (e.g. GDP, volatility) or take values on a bounded interval (e.g., interest rate, unemployment rate). To account for these cases, we investigate the performance of the ACPS \, in simulated experiments where the target density is either Gamma or Beta.
Fig. (ref) presents the results for a $\mathcal{G}a(2,1)$ target density. By looking at the worst performing densities according to ACPS, we find that $\mathcal{G}a(1,1)$ is assigned the highest penalty for values $c \leq 0.725$, while $\mathcal{G}a(1,2)$ becomes the worst for $c = 0.95$. This reflects that for $c \leq 0.725$, the asymmetric score penalizes more the underestimation, while for $c = 0.95$ it gives more weight to overestimation. Similar results are found in Fig. (ref) with a positively skewed Beta target density, $\mathcal{B}e(1,2)$.
We deep further the properties of the proposed asymmetric scoring rule by considering a threshold-weighted version and comparing it with the threshold-weighted CRPS. The goal is to disentangle the different role of the domain-weighting scheme, which reflects the interest of the decision-maker in having good forecasts within a specific interval of values, and of the error-weighting scheme, which corresponds to the decision-maker's loss in case of under or overestimation.
Consider a simulated experiment where $N=100$ observations are drawn from a Normal distribution $\mathcal{N}(1,4)$ and several forecasting densities are approximated using $M=500$ draws. We consider the domain-weighting schemes in Tab. (ref), using $5$ alternative asymmetry levels $c \in\{ 0.05, 0.275, 0.50, 0.725, 0.95 \}$.
In Tab. (ref) we find that the asymmetric penalty imposed by ACPS \, plays a significant role for all domain-weighting schemes considered. For an uniform weight, the ACPS \, agrees with the CRPS for $c=0.5$, i.e. the symmetric case, but rewards differently the density forecasts for alternative values of the asymmetry level $c$. When the interest is focused on the right tail of the distribution, both threshold-weighted CRPS and ACPS \, agree, but when the attention is on the left tail, the two scoring rules perform remarkably different. The CRPS favours the standard Normal over the $\mathcal{N}(3,1)$, while the ACPS \, rewards the latter for all $c \geq 0.275$.
The key insight obtained from this simulated exercise concerns the importance of domain- and error-weighting schemes. The first assigns an heterogeneous weight to the performance on different intervals, while the latter asymmetrically rewards negative and positive deviations from the true value. The threshold-weighted asymmetric scoring rule, tACPS, combines the two schemes and allows to evaluate the performance of the forecasting density from both perspectives. This is important to the decision makers, who are usually interested in a specific range of all possible values, thus calling for heterogeneous domain-weighting, and have asymmetric preferences towards under or overestimation, which motivates an asymmetric score.
In the empirical applications, we adopt a similar framework to amisano2007comparing and gneiting2011comparing, and consider the task of comparing density forecasts in a time series context. We use a fixed-length rolling window to provide a density forecast for $h$ step ahead future observations. We focus on three different applications related to macroeconomics (employment growth rate) and to commodity prices (oil prices and electricity prices). We compare several univariate models, such as the autoregressive (AR) model, the Markov-switching (MS) AR model and the time-varying parameter (TVP) AR model.
We use the AR(1) as benchmark model, then we specify $12$ lags for the employment growth rate (i.e., $1$ year of monthly observations) and $20$ lags for the oil (i.e., $1$ month of daily observations). Regarding the electricity prices, we include $7$ lags (i.e., $1$ week of daily observations) and by following common practice in the literature, we restrict lags to $t-1$, $t-2$, and $t-7$, which correspond to the previous day, two days before, and one week before the delivery time, recalling first similar conditions that may have characterized the market over the same hours and similar days (such as congestions and blackouts) and secondly the demand level during the days of the week. For the MS-AR model we consider only $1$ lag, while for the TVP-AR model we use $1$ and $2$ lags. For both AR and TVP-AR, we consider three specifications of the variance: constant volatility and time-varying volatility in the form of stochastic volatility with Gaussian and Student-t error. For the MS-AR, we impose an identification constraint on the error variance.
In the first application, we aim at forecasting monthly US total nonfarm seasonally adjusted employment growth rate downloaded from the FRED database. We consider the growth rate of the monthly employment rate in US from January 1980 to April 2020. We see evidence of some spikes, in particular with a strong fall in April 2020 due to present COVID-19 situation (see Figure (ref) in the supplementary material). We use a rolling window approach of $20$ years (thus $240$ observations) and we forecast $h=1$ and $h=12$ (thus $1$ year ahead) month ahead by using a recursive forecasting exercise.
For oil prices, we analyze daily West Texas Index (WTI) data (no weekends) from 02 January 2012 to 07 May 2020 in order to include in the analysis the recent turmoil. Indeed, large drops in demand that suddenly occurred and storage scarcity have resulted in negative WTI oil prices at the end of April 2020. As for the employment rate, we have used a rolling window of $4$ years and we forecast $h=1$ and $h=5$ days ahead by applying recursive techniques.
In the third application, we consider the problem of forecasting the day-ahead electricity prices in Germany, one of the largest and leading energy market. In the electricity markets, the phenomenon of negative prices -- when allowed to occur, such as in Germany where there is no floor price -- has become more frequent due to the increasing share of electricity generated from renewable energy sources (RES) and the current impossibility to store it (see Figure 2 in the supplementary material). We analyze daily data (with weekends) from 01 January 2014 to 08 May 2020. For the forecasting analysis, we have considered a rolling window of $3$ years and a recursive techniques for predicting $h=1$ and $h=7$ days ahead.
As we discussed in the introduction, policymakers or energy producers may be more concerned with forecasting values below a given threshold than the full distribution, since they require different measures, including in the case of energy variables to stop the production.\footnote{Unfortunately, we have not precise data to compute (i) the value of this threshold, excluding the case of RES producers of electricity prices, that could be still profitable even when prices are marginally above zero, and (ii) the level of asymmetry of the loss function. Therefore, we investigate several values of $c$, the parameter that drives the asymmetry of our measure.} This supports the application of the ACPS. For the oil series we perform a case study around the collapse of WTI prices and discuss how the ACPS \, results can be applied to identify the true unknown density.
Before evaluating the relative performance of all models, we check the calibration of the density forecasts. Calibration of density forecasts is based on properties of a density and refers to absolute accuracy (see BCR2019 for further details). The absolute accuracy can be studied by testing forecast accuracy relative to the “true”, unobserved density. Dawid82 introduced the criterion of calibration for comparing prequential probabilities with binary random outcomes and exploited the concept of probability integral transform (PIT), that is the value that a predictive CDF attains at the observations, for continuous random variables. The PITs summarize the properties of the densities and may help us to judge whether the densities are biased in a particular direction and whether the width of the densities has been roughly correct on average, see DieboldGunterTay1998. The PITs can provide an indication of whether a density is wrong in predicting higher moments or specific parts of the distribution, such as the tails; however they cannot distinguish among models that are also correctly calibrated. We apply the test of Knuppel2015 and refer to RossiSekhposyan2013 for evaluation of PITs in presence of instabilities, RossiSekhposyan2014 for application with large database and RossiSekhposyan2016 for a comparison of alternative tests for correct specification of density forecasts.
The PIT tests in Tab. (ref) indicate that all densities are correctly calibrated for the employment growth rate at 5% significance level, excluding the one given by the TVP-AR(2) model at 12-month horizon, for which the p-value is marginally lower at 4.9%. Density forecasts from models TVP-AR(2)-SV\footnote{Notice that the TVP-AR(2)-SV is always preferred in terms of relative accuracy.} and TVP-AR(2)-tSV are calibrated at 1-day ahead horizon; no density is correctly calibrated at 5-days ahead horizons. All densities are not correctly calibrated when predicting EEX electricity prices at both horizons. So, the PITs analysis suggests there is not a stochastically dominating model, but more specifications can provide (absolute) accurate forecasts suggesting the use of relative metrics such as the ACPS \, to discriminate among them. In the case of EEX prices, all models are wrong and a possible explanation is that the models considered in this text are based only on econometric properties of the series, hence they may be labelled as “purely econometric” models. GRR2020a and GRR2020b document how important is to extend these models with economically relevant variables, such as variables related to the demand and the production of electricity, including renewable energy sources, to increase accuracy. We leave this extension for further research and apply our metrics to an example where models in terms of calibration are all wrong.
Tab. (ref) shows the ranking of the probability forecasts over vintages and across models for all the three datasets for $c=0.05,\,0.5,\,0.95$.\footnote{See Table IV in the supplementary material for results for a higher range of $c$.} The ACPS test presented in Section (ref) is also reported.\footnote{In order to perform the test, we checked the stationarity and short memory of the loss differential series using the ADF test and the autocorrelation function, respectively.}
Regarding the employment growth rate, we can see at horizon 1-month ahead that the best model for $c=0.05$ is the AR(12)-tSV, for $c=0.5$ it is the TVP-AR with $2$ lags (the same for the CRPS measure), and for $c=0.95$ it is the AR(12)-SV, showing differences across different levels of asymmetry. The test indicates that most of the models provide superior forecasts than the AR(1) benchmark and only the AR(12) model does not provide gains. The difference in model performance for various levels of $c$ is confirmed for $h=12$ and interesting for $c=0.05$ only the AR(1)-MS is statistically superior. Therefore, our evidence supports the large literature on the use of time-varying and nonlinear models in modelling and forecasting (un)employment data. Moreover, the best model for $h=12$ and $c=0.5$ is the same when applying the CRPS. In Fig. (ref), we report the best model in each vintage for the two horizon ahead, where the black line refers to the CRPS, the red and the yellow for the ACPS \, for $c=0.05$ and for $c=0.95$, respectively. The graph shows large instability in the best model, in particular when using the CRPS. The ACPS \, rules seem to prefer one of the alternative models for more consecutive vintages. For example, by looking at the relative frequency of occurrence of each model as the best model, we find that for $c=0.05$, $31\%$ times the AR(12)-tSV is considered the best model for $h=1$. Similar percentages are found for other levels of $c$ and $h$, despite model order varies substantially across measures.
Moving to the oil prices, in the middle panel of Tab. (ref), we find that across vintages, for $1$ day ahead the TVP-AR(2) is the best model whereas the TVP-AR(2)-SV is the second best for the asymmetric levels $c=0.5,\,0.95$ and the CRPS. For $c=0.05$ the best model is the TVP-AR(2)-SV model, supporting PITS evidence that this model is among the few ones correctly calibrated. The TVP-AR(2)-SV model is again the best model for $1$ week ahead of forecasting and for $c=0.05$ and it is one of the two models to be statistically superior to the AR benchmark. For the same weekly horizon and other levels of $c$, again only few models are superior to the benchmark. Fig. (ref) confirms that the ACPS \ is less variable in this selection than the CRPS.
Fig. (ref) illustrates the ACPS \ for one step ahead density forecasts of the OIL prices, according to an AR(20) model, for each vintage of the rolling estimation and various levels of asymmetry. Despite showing the results for a single model, this figure presents some interesting insights. By looking at the scores between April 17 and April 21, we find that the forecast is worst performing for $c=0.05$ and best for $c=0.95$, indicating that the density forecast assigns more mass on the right part of the support as compared to the density of the observations. This situation is similar to the yellow line in Fig. (ref). Surprisingly, the ranking is reversed between April 21 and April 24, where the forecast receives a higher score under $c=0.05$. This suggests that the density forecast is likely to be a right-shifted version of the observation density, similarly to the blue line in Fig. (ref).
These results highlight how accounting for asymmetry in forecast evaluation may lead to dramatically different implications. By looking at the period until April 21, a decision maker averse to overestimation of oil price is likely to discard the AR(20) in favor of alternative models for making forecasts. Conversely, another agent facing the same decision problem, equipped with the same data and models, but averse to underestimation, is likely to agree to the AR(20).
Moreover, these insights provide an important value added of the ACPS \, as compared to symmetric scores. By looking at variation of the ranking according to the ACPS \, over time, it is possible to infer the relative dynamics of the forecasting and observation densities. In the case previously mentioned, between April 17 and April 21 the forecast tends to overestimate (i.e., its CDF is to the right of the observations CDF), while it tends to underestimate between April 21 and April 24 (i.e., its CDF is to the left of the observations CDF). Under a symmetric score it is not possible to grasp these insights since negative and positive deviations from the target are equally penalized.
The bottom panel of Tab. (ref) reports the results for the electricity prices. As in the previous cases, there is large uncertainty on the model ranking. In line with PIT evidence, the high volatility, spikes and negative prices of the electricity prices drive different results depending on the level of asymmetry of the user. At $h=1$ and $c=0.05$, the AR(20)-tSV is the best model, for higher values of $c$, the TVP-AR(2)-SV and TVP-AR(2)-tSV are the preferred ones. Many models with time-varying volatility outperform the constant volatility models, confirming evidence in GRR2020b. At $h=12$ the AR(20) for $c=0.05$, the TVP-AR(2)-tSV for $c=0.5$, and the TVP-AR(1) for $c =0.95$ give the highest ACPS. Fig. (ref) again indicates more stable performance of some models when accounting for asymmetry relative to use the symmetric CRPS.
This paper has introduced a novel asymmetric proper score for probabilistic forecasts of continuous variables, the ACPS. Its main application is the evaluation and comparison of density forecasts. In addition, we have proposed a threshold- and quantile-weighted version of the asymmetric score, which, by reweighing the domain, allows for a further level of asymmetry in the evaluation of forecasts. We also derive a test to compare the statistical accuracy of different forecasts. The definition of ACPS \, is sufficiently flexible to be used in a variety of univariate contexts, and carries over to the multivariate case. The latter deserves further investigation and is an open field for future research.
We provide a tool able to account for the decision maker's preferences in the evaluation of density forecasts both in terms of domain- and error-weighting schemes.
In an artificial data exercise, we have shown the good performance of our proposed asymmetric score for different continuous target distributions. In relevant macroeconomic and energy applications, we evaluate our score across different models and for different horizons, and we improve on the quality of the forecasts by providing an effective tool for density forecast evaluation.
The proposed score, ACPS, is of general use in any situation where the decision maker has asymmetric preferences in the evaluation of forecasts and thus it can be applied to a much wide range of applications. Further extensions could cover the area of forecast instability, see GiacominiRossi2010, and the case of state-dependent function of economic variables such as in ORS2020.
\setcounter{section}{0} \setcounter{table}{0} \setcounter{figure}{0} \setcounter{equation}{0} \setcounter{page}{1}