Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
52,833 characters · 12 sections · 68 citation commands
Comparing predictive ability in presence of instability over a very short time
\doublespacing
Instabilities during periods of crisis are common in time series and in particular in the last years when the COVID-19 pandemic caused a huge shock and changed the economic perspective. Consequently, forecasting becomes more challenging during periods of crisis and sudden recoveries, but these periods are often more important in the forecasting task since they usually carry more risk of catastrophic errors.
An exceptional example is the methodology proposed by bok2018macroeconomic, which was implemented by the New York FED but was suspended in September 2021 due to the challenges posed by the COVID-19 pandemic and recently was reintroduced as in almuzara2023new. In particular, the period related to COVID-19 and the sudden recovery Ng2021, Lenza2022 has gained attention among researchers and policymakers since for the first time we have had the opportunity to study some unpredictable shocks in the economy.
In detail, COVID-19 caused a shock to GDP that was in many ways unprecedented in recent history and tested the capacity of current methodologies in the presence of extraordinary situations. The main big challenges are both in how the forecasting methods behave but also how we should evaluate the forecasting that common models provide the policymakers. The first challenge has been strongly studied in the literature, where huber2023nowcasting argue that nonlinear methods may better accommodate extreme situations. Differently, foroni2022forecasting and Schor2021 stress the resilience of popular models, such as mixed frequency or dynamic factor models to accommodate the severity of recessions.
Indeed, the second task has not been strongly discussed in the literature, but it reveals the importance of evaluating and accounting forecasting instabilities. In such situations, average tests for forecast evaluation, such as the Diebold and Mariano test of equal unconditional predictive ability dieboldComparingPredictiveAccuracy2002, GiacominiWhite2006 may not be informative, as they do not have much power to detect instances in which one forecast outperforms the competitor only on a fraction of the sample. To account for forecasting instability, rossi2021forecasting recommends the application of local procedures, like the one-time reversal or the fluctuation tests from giacomini2010forecast. In comparison with diagnostics based on the evaluation of forecasts over an average, the fluctuation test is indeed local, as the statistic is only computed over a fraction of the out-of-sample evaluation period.
However, as the test is derived assuming that the fluctuation induced by the instability spans a relevant fraction of the sample, then it replicates on a smaller scale the difficulties incurred by the average, global dieboldComparingPredictiveAccuracy2002 test. Indeed, the Monte Carlo study in giacomini2010forecast shows that the attempt to run the test for a very short fluctuation period is frustrated by a relevant size distortion.
In this paper, we, therefore, discuss the difficulties associated with using global diagnostics for evaluation in times of crises, and we consider other diagnostics, based on the predictive instability tests in andrews2003end and Taylor_etal_2021_Pockets, to account for this situation. Indeed, situations of crises like the one induced by COVID-19 or by an economic recession do not typically conform well with the assumption of a large, even if local, evaluation period, as crises often span only a very small number of observations. As an example, in the COVID-19-induced recession, the instability mostly affected just two or three quarters.
One key finding of our paper is that in the presence of high, very localised instability, global tests may have no power at all, thus leading to the incorrect conclusion, and that this may also obfuscate a signal that would otherwise be clear, had the analysis not included the brief period of instability. In particular, we investigate in different Monte Carlo exercises the importance of considering the S test and the MAX procedure instead of the usual global tests when we have a short deviation. Moreover, we emphasize the importance of applying a correction for the dependence in the S test statistic by using the estimated value using the pre-changed sample and not the recommended in andrews2003end or a simple identity matrix.
In an empirical application to the US nominal GDP growth, we compare the nowcast from the Survey of Professional Forecasters (SPF) and the last available observation during the COVID-19 recession and the subsequent recovery. When COVID-19 is not considered in the analysis, DM and Fluctuation tests suggest that the SPF nowcast is more precise than the naive model, which does not happen when we include the COVID-19 period. However, if we consider the $S$ test and the MAX procedure, it provides more appropriate results. Given these results, we recommend that the forecaster should not pool the sample, but exclude the short periods of high local instability from the evaluation exercise.
The rest of the paper is organised as follows. In Section (ref), we introduce the Diebold and Mariano (DM) and Fluctuation test statistics, and we investigate their performance in case of a very large, localised shock. We also introduce some diagnostics to detect the presence of those shocks. In Section (ref) we study the performance of these diagnostics in two different Monte Carlo exercises; while in Section (ref) we evaluate the ability of the Survey of Professional Forecasters (SPF) to outperform a naive benchmark when a period of high but localised instability due the Covid-19 shock is included in the study. Section (ref) concludes the paper.
We consider the classical GiacominiWhite2006 framework, also see for example giacomini2010forecast, or Coroneo2020Comparing. To fix some notation, we denote the variable of interest by $y_t$, for which we want to compare two $h$-step ahead forecasts obtained from two alternative forecasting methods, based on some predictor variables $x_{t}$. We denote the observed vector by $w_t \equiv (y_t, x'_t)'$, defined on a complete probability space $(\Omega,\mathcal{F}, P)$, and the information set at time $t$ by $\mathcal{F}_t=\sigma(w_1',\hdots, w'_t)$. The two $h$-step ahead forecasts for time $t$ are based on the information set $\mathcal{F}_{t-h}$ and are denoted by $\widehat{y}^{(i)}_{t}\left(\widehat{\delta}^{(i)}_{t-h, R_i}\right)\equiv f^{(i)} \left(w_{t-h}, w_{t-h-1}, \hdots, w_{t-h-R_i+1};\widehat{\delta}^{(i)}_{t-h, R_i}\right)$ for $i=1,2$, where the forecasts are measurable functions of a sample of size $R_1$ for $f^{(1)}$ and $R_2$ for $f^{(2)}$. If a forecast is based on parametric models, the vector $\widehat{\delta}^{(i)}_{t-h, R_i}$ includes the estimates from the model. Otherwise, $\widehat{\delta}^{(i)}_{t-h, R_i}$ represents the semiparametric or nonparametric estimator used to construct the forecast. Notice, however, that in this framework the estimates $\widehat{\delta}^{(i)}_{t-h, R_i}$ are based on a rolling window of dimension $R_i<\infty$, so they are inconsistent.
For the two forecasts $\widehat{y}^{(i)}_{t}\left(\widehat{\delta}^{(i)}_{t-h,R_i}\right)$, denote the forecast error by $e^{(i)}_{t}\left(\widehat{\delta}^{(i)}_{t-h, R_i}\right)= y_t - \widehat{y}^{(i)}_{t}\left(\widehat{\delta}^{(i)}_{t-h, R_i}\right)$ and, for a real function $L(\cdot)$, that we interpret as a loss function, the loss associated with the forecast error is $L\left(e^{(i)}_{t}\left(\widehat{\delta}^{(i)}_{t-h, R_i}\right)\right)$. Finally, the loss differential at time $t$ between the two forecasts is
and the null hypothesis of equal predictive ability of the two forecasting methods is
at each point $t$. Denoting $R \equiv \max(R_1,R_2)$, we assume that we have a sample of dimension $R+h+T-1$ and we can therefore evaluate the hypothesis (ref) in the $\{R+h,\ldots,R+h+T-1 \}$ evaluation period. To abbreviate notation, denote $s=t-(R+h)+1$ and
where notice that the evaluation period with respect to $s$ is $\{1, \ldots, T\}$. \\ Let us denote $\mu_s$ by
Then, the null hypothesis of equal predictive ability of the two forecasting methods is
at each point $s \in \{1, \ldots, T\}$.\\ When $H_0$ is not met, then there is a $\mu_s \neq 0$ for some $s$ in the evaluation period.
dieboldComparingPredictiveAccuracy2002 propose to test the hypothesis in (ref) using the sample average
Denoting the long-run variance by
and by $\widehat{\sigma}_T^2$ an estimate of $\sigma^2_T$, the Diebold and Mariano test (hereafter DM test) uses the statistic
When $\widehat{\sigma}^2_T-\sigma^2_T=o_p(1)$ and given other regularity conditions (see for example Assumption GW in Subsection (ref)), GiacominiWhite2006 show that, under $H_0$,
where $Z$ is a standard normal distributed variable. {When the null hypothesis is not met, the DM test has non-trivial power in presence of local alternatives $\mu_s=\delta T^{-1/2}$ for all $s$.}
giacomini2010forecast describe the DM test as an average or global test, as it primarily detects deviations of the null hypothesis that are constant over the whole evaluation period. The DM test is less effective in detecting deviations from $H_0$ when they occur only on a fraction of the sample, and this loss of local power is stronger the shorter the fraction where the predictive ability of the two forecasts is different. Hence, the DM test might even have no power at all when $\mu_s$ changes sign over the evaluation sample so that $\sum_s \mu_s =0$ is possible.
Therefore, giacomini2010forecast propose to consider a local statistic, called the fluctuation statistic
where $k=\lfloor \kappa T \rfloor$ and we assume $k/T \to \kappa \in (0,\infty)$ as $k \to \infty$ and $T \to \infty$ as in Assumption 1(c) in giacomini2010forecast. They show that, under $H_0$ and regularity conditions,
where $B(\cdot)$ is a standard univariate Brownian motion and $\rho \in [0,1]$ is such that $s=\lfloor \rho T \rfloor$. The fluctuation test statistic is defined as
hence giacomini2010forecast characterise the convergence to the limit of the test statistic and provide simulated critical values for the test. In comparison with the DM test, the fluctuation test should have less power when $\mu_s$ is constant, but more when the predictive ability is different only on a subsample.
It is noteworthy that the critical values for the fluctuation test depend on the length of the fraction $\kappa$: the Monte Carlo study in giacomini2010forecast suggests a certain size-power trade-off in the choice of $\kappa$, in the sense that very small values ($\kappa=0.1$) are associated to size distortion in finite samples, whereas larger values are associated to lower power in presence of instability.
giacomini2010forecast demonstrate the value of the fluctuation test in the forecasting of exchange rate macroeconomic models. As their example makes clear, the natural application is in situations where the economic dynamics are slowly changing over time, and we can use the test to study the evolution, as in Rossi2010Application and Galvao2021JudgementalAdjustment. In other words, the test detects forecast differential instability across periods, or regimes, as it happens when forecasts from a macroeconomic model are compared against a benchmark in the presence of changes to the fundamental economic relations.
The fluctuation test is also effective in detecting the existence of what Timmermann2008Elusive, emphasising the local nature of the predictability of returns, refers to as pockets of predictability, that only appear in some periods in time corresponding to fractions in the sample, as in Urga2023_Pockets_Application. This situation, however, does not cover well the differential forecasting instability that occurs over only a small period of time, as is sometimes the case for economic recessions or other short-lived events.
In this case, the expected value of the differential predictive ability is rather described as
where $I_s(\tau)$ is an indicator function, taking value $1$ if $s=\lfloor \tau T \rfloor$ and $0$ otherwise; the factor $\delta_2 T^a$ characterises the dimension of the change in the prediction differential in relation to the sample size.
We then assume for the loss differential $d_s$ the data generating process
where $u_s$ is a zero-mean process. In this case, the loss differential $d_s$ is constant, but for a point in time $s=\lfloor \tau T \rfloor$ the situation of equal predictive ability corresponds to $\delta_1=0$ and $\delta_2=0$.
We first study the limit properties of the DM statistic assuming that the long-run variance is estimated using the Bartlett kernel,
where $c_l$ is the $l$-th sample covariance of $d_s$ (with $l=0$ the sample variance) and $M$ is a user-chosen bandwidth such that $M/T \rightarrow 0$ as $T \rightarrow \infty$. To establish the limit properties of the DM statistic, we introduce the following assumptions
Assumption GW
}
To characterise the local power of the test, it is also convenient to assume that there is $\sigma^2 = \lim_{T\rightarrow \infty}Var \left( \frac{\sqrt{T}}{T}\sum _{s=1}^{T} u_{s} \right)$.
We refer to Appendix (ref) for a complete proof.
A similar argument of course holds when the instability affects more than one observation, provided that the number is very small relative to the sample. The COVID-19-induced instability seems the typical example of this situation, but it suggests that including in the evaluation exercise the recession induced by the financial crisis may also generate a power loss, although this should be more subdued as the size of the shock is less.
Detecting forecasting superiority is therefore more difficult in cases of events that are limited in time and large in scale, and yet it is also usually more important for its policy implications. When the location of the event is known in advance, as it may be in the case of the recession induced by COVID-19, we propose to apply the predictive instability test of andrews2003end, and we show that its application to evaluate forecasts is justified. When the location of the potential predictive instability is not known, the approach is rather similar to detecting outliers or extreme values, as in Leadbetter1983.
The test defined in andrews2003end is based on testing the null hypothesis $H_0: \delta=0$ in
where $I_s(\tau)$ is treated as a dummy variable taking a non-zero value only when $s=\lfloor \tau T\rfloor$, and $u_s$ is a zero-mean process. \\ To simplify notation, we assume $\tau=1$, as in andrews2003end, where the generic $\tau$ situation is also briefly discussed.
In general, comparing residuals sum of squares from an unrestricted and restricted regression is done using an $F$ test or, if the data are not normally distributed, using a $\chi^2$ limit distribution. However, in this case, the usual asymptotic convergence in distribution of the $F$ statistics to a $\chi_1^2$ limit does not hold: intuitively, this is because $\delta$ is estimated using only one observation, so it is not possible to invoke a central limit theorem to establish the limit distribution of this estimate.
Instead, denoting $\widetilde{\mu}=\frac{1}{T} \sum_{s=1}^T d_s$, the estimate of $\mu$ in the restricted model, and $\widehat{\mu}=\frac{1}{T-1}\sum_{s=1}^{T-1} d_s$ the estimate in the unrestricted model, so that the restricted and unrestricted residuals are $\widetilde{u}_s=d_s-\widetilde{\mu}$ and $\widehat{u}_s=d_s-\widehat{\mu}$, respectively. Then the idea is to estimate the distribution of $\widetilde{u}_{T}^2$ with the sample distribution of $\widehat{u}_s^2$ for $s=1,\ldots, T-1$.
To improve the empirical size performance in finite sample, andrews2003end proposes a slight modification of this procedure, where the critical distribution is estimated from $\widehat{u}_{2(s)}=d_s-\widehat{\mu}_{2(s)}$, where $\widehat{\mu}_{2(s)}=\frac{1}{T-2} \sum_{j=1, j\neq s}^{T-1} y_j$. The null hypothesis is rejected at $\alpha$ asymptotic significance level if $\widetilde{u}_{T}^2$ exceeds the $(1-\alpha)$ sample quantile of $\widehat{u}_{2(s)}^2$ for $s=1,\ldots, T-1$.
To state the properties of the test, we introduce some additional notation, adapting the one presented in andrews2003end. Denote the test statistic by $S$, so $S=\widetilde{u}_{T}^2$, and, for any $s \neq T$, let $S_s(\mu)=d_s-\mu=u_s$: under strict stationarity, all these variables have the same distribution, denoted as $F_S(x)$. Also, let $S_s=\widehat{u}_{2(s)}^2$, with empirical distribution $\widehat{F}_{S,T}(x)=\frac{1}{T-1}\sum_{s=1}^{T-1} 1(S_s \leq x)$, and let $q_{S,1-\alpha}$ denote the $(1-\alpha)$ quantile of $F_S(x)$, and $\widehat{q}_{S,1-\alpha}$ denote the $(1-\alpha)$ sample quantile of $S_{s,s\neq T}$. Finally, let $S_{\infty}$ be a random variable with the same distribution as $d_{T}-\mu$.
Then we introduce the following assumptions:
Assumption A
} In Assumptions A.3 and A.4, we refer to $u_1$ as a generic $u_s$ as the strict stationarity of $w_t$ and the nature of the forecasting functions ensure the strict stationarity of $u_s$.
We refer to Appendix (ref) for a complete proof.
In the interest of simplicity, we present Theorem (ref) assuming that the instability only affects one point. Of course, it is possible to consider a longer span of observations, as long as this remains finite and, in practice, also small concerning the sample size. When instability affects $k$ observations ($T-k,\ldots,T$), andrews2003end shows that the procedure can be easily applied to quadratic forms of $\widetilde{U}_s=(\widetilde{u}_{s},\ldots,\widetilde{u}_{s+k-1})'$, $\widehat{U}_{2(s)}=(\widehat{u}_{2(s)},\ldots,\widehat{u}_{2(s+k-1)})'$, adjusting the definition of $\mu_{2(s)}$ to account for the fact more residuals are considered jointly.
One could compute quadratic forms directly from $\iota_k'\widetilde{U}_s$ and $\iota_k'\widehat{U}_{2(s)}$, where $\iota_k$ is a $k\times1$ vector of ones. andrews2003end also proposes an alternative statistic, that would account for the autocorrelation in $u_s$: denoting $\Sigma$ the variance-covariance matrix of ${U}_s=({u}_{s},\ldots,{u}_{s+k-1})'$, this alternative quadratic form is computed from $\iota_k'\Sigma^{-1}\widetilde{U}_s$ and $\iota_k'\Sigma^{-1}\widehat{U}_{2(s)}$.
The test in andrews2003end is designed for situations in which the exact location of the suspected instability is known. In many cases, however, the exact location of the occurrence of a brief differential in predictive is not known in advance. This situation is rather more similar to the problem of detecting one of the "pockets of predictability" discussed in Timmermann2008Elusive, as the forecast of a model is measured against a benchmark of no predictive ability.
A recent procedure to detect such pockets is in Taylor_etal_2021_Pockets, where it is characterised as the occurrence of an outlier at an unknown point in time in a series. As they are interested in detecting the location of a pocket, as well as in testing its statistical significance, Taylor_etal_2021_Pockets divide their sample into many short intervals and compute a $t$ statistic for each one. Our problem of interest is somewhat simpler, as we can just look at the statistic $d_s^2$.
The procedure consists of splitting the sample in a training period, $s=1,\ldots, T^*$, and a monitoring period $s=T^*+1,\ldots, E$ where $E \leq T$, and $T^*$ and $E$ are fractions of the sample period $T^*=\lfloor \lambda_1 T \rfloor$, $E=\lfloor \lambda_2 T \rfloor$, for $0 < \lambda_1 < \lambda_2 \leq 1$. We assume that no instability occurs during the training period, but instability may take place during the monitoring period. We then compare the maximum of the statistic $d_s^2$ during the training period, $\max_{s=1,\ldots,T^*} d_s^2$, and during the monitoring period, $\max_{s=T^*+1,\ldots,E} d_s^2$. In practice, we use the training period to estimate the probability that $\max_{s=T^*+1,\ldots, E} d_s^2> \max_{s=1,\ldots, T^*} d_s^2 $ if no instability has occurred.
Notice that, heuristically, if the statistic of interest $d_s^2$ was independently and identically distributed, and the training and monitoring period constituted fractions $\lambda_1$ and $\left( 1-\lambda_1 \right)$ of the sample, respectively, then in large sample that probability should be $\left( 1-\lambda_1 \right)$. Taylor_etal_2021_Pockets establish this result formally, and under conditions that allow for dependence in $d_s$.
Recall $w_t=\left(y_t,x_t'\right)'$ and denote the element in position $\iota$ as $\{ w_{\iota,t} \}$.
Assumption B Let $ \{ w_{\iota,t} \}_{t \geq 1} $ be a strictly stationary sequence of random variables and $\{ v_t(\xi) \}_{t \geq 1} $, $\{ v_t(\zeta) \}_{t \geq 1} $ with $\xi, \ \zeta \in \mathds{R}$, sequences of real numbers. For each $1 \leq i \leq j$, set $ \mathcal{F}_i^j \left( v_t(\cdot) \right) $ as the $\sigma-$algebra generated by the events $\{ w_{\iota,r} \leq v_t(\cdot)\}$, $i \leq r \leq j$, and, for $1 \leq l \leq t-1$, denote
then there exists a sequence $l_t(\xi; \zeta)=o(t)$, as $t \rightarrow \infty$, such that $\lim_{t_\rightarrow \infty}\alpha_{t, l_t}(\xi,\zeta) = 0$. }
Assumption B is discussed in Ferreira2002 and is a very mild mixing condition, which relaxes regularity condition $D(u_n)$ in Leadbetter1983, page 53, which in turn is already a relaxation of strong mixing. Notice that the stationarity and mixing condition in Ferreira2002 is referred to the series $d_s$: differently, we formulate this requirement in terms of the observables $w_t$: the validity of this stationarity and mixing conditions for $d_s$ follows from Theorems 3.35 and 2.49 of white2000asymptotic, recalling that $R$ is finite so the application of this result is justified.
Assumption B is sufficient to establish Proposition 1 of Taylor_etal_2021_Pockets. We then obtain
Thus, following Taylor_etal_2021_Pockets we suggest to consider $\max_{s=T^*+1,\ldots,E} d_s^2$ and $\max_{s=1,\ldots,T^*} d_s^2 $ and conclude that there is instability if $\max_{s=T^*+1,\ldots,E} d_s^2> \max_{s=1,\ldots,T^*} d_s^2$. Taylor_etal_2021_Pockets refer to this as the MAX procedure, which is a test with size $\frac{\lambda_2-\lambda_1}{\lambda_2}$.
Notice that although the monitoring interval $[\lfloor \lambda_1 T \rfloor+1,\lfloor \lambda_2 T \rfloor]$ is assumed to be proportional to the sample size, we still consider the MAX a diagnostic to detect instability even at a single point in time, or for a very short period, as that could be sufficient to cause $\max_{s=T^*+1,\ldots, E} d_s^2$ to exceed the threshold from $\max_{s=1,\ldots, T^*} d_s^2$.
In a nutshell, consider a sequence $\{\upsilon_s\}$ of independent standard normal distributions, observed at $s=1,\ldots,T$, with a possible point of instability $\upsilon_{s^*}$, with $E(\upsilon_{s^*})=\mu_s^*$ and ${s^*}>{T^*}$. The MAX procedure compares $\max_{s=T^*+1,\ldots, E} \upsilon_s$ against $\max_{s=1,\ldots, T^*} \upsilon_s$, and it is well-known that the latter has stochastic order $O_p((\ln(T^*))^{1/2})$, see Theorem 1.5.3 in Leadbetter1983. Thus, the MAX procedure would have non-trivial power against cases with $\mu_s^*=c (\ln(T^*)^{1/2})$ when $c>0$. Interestingly, this is slightly less than the local power for the test based on andrews2003end (which has non-trivial power if $\mu_s^*>0$), but much more than the local power of the Fluctuation test or of the Diebold and Mariano test (as we have seen these have no power in this situation). Thus, we state that the andrews2003end test and the MAX procedure may complement the information from the Fluctuation and DM tests.
This section is dedicated to investigating the size and properties of the tests previously described in a complete Monte Carlo study. As a second experiment, we compare the different versions of the Andrews ($S$) test when different choices of $\Sigma$ are considered. In particular, we consider the Data Generating Process (DGP) as
We assume that we do not observe $x_t$, but
where $\xi_t$, $\varepsilon_{t'}$, $v_{t''}^{(1)}$ and $v_{t'''}^{(2)}$ are independent for each $t$, $t'$, $t''$ and $t'''$.
Let us denote
where $R$ is the rolling window size. Then, we consider the forecasting rule as
Hence, this could be a situation in which forecaster $i$ is trying to predict an unobserved factor $x_t$ but she/he does it with an error, as it may happen when unobservable factors are estimated by using only a small information set.
Thus, the forecast errors are defined as
and notice that, for $i=1, 2$, $\widehat{\beta}_t^{(i)}$ is independent from $v_t^{(i)}$, as the latter is serially independent, and $v_t^{(2)}$ is independent from $x_t$. Then, the equal predictive ability useful for the DM test is
where the elements in Eq. (ref) are equal to $0$.
If $\sigma_1^2=\sigma_2^2$, then $\left\{ v_{t-R}^{(1)},\ldots, v_{t-1}^{(1)} \right\}'$ and $\left\{ v_{t-R}^{(2)},\ldots, v_{t-1}^{(2)}\right\}'$ are identically distributed, and $\widehat{\beta}_t^{(1)}$, $\widehat{\beta}_t^{(2)}$, as a finite function of identically distributed variables, are also identically distributed, so $E \left(\widehat{\beta}_t^{(1)}\right)^2 \sigma_1^2 = E \left(\widehat{\beta}_t^{(2)}\right)^2 \sigma_2^2$. Similarly, it follows that $E \left\{ \left(\beta-\widehat{\beta}_t^{(1)}\right)^2 x_t^2 \right\} = E \left\{ \left(\beta-\widehat{\beta}_t^{(2)}\right)^2 x_t^2 \right\}$ and $E \left\{ \left({\beta}-\widehat{\beta}_t^{(1)}\right) x_t \eta_t \right\}= E \left\{ \left({\beta}-\widehat{\beta}_t^{(2)}\right) x_t \eta_t \right\}$. It is straightforward to verify that $d_t$ is strictly stationary, strong mixing, and ergodic.
To evaluate the power of these tests it is sufficient to modify the value of $\sigma_2^2$ for one or more points $t$. In our baseline design, we set
where $\delta_s=1$ yields $E(d_s)= 0$.
We simulate the DGP in Eq. (ref) with parameters as in (ref), with sample size $T$ equal to $80$ and rolling window size $R$ equal to $20$ as in our real data application. Moreover, we set $\kappa=0.3$ for the Fluctuation test, and $T^*=76$ and $E=80$ for the $MAX$ diagnostic. For the DM and fluctuation tests, we estimate the long-run variance using the Bartlett kernel with bandwidth $M=\lfloor T^{2/9} \rfloor$, also we use fixed smoothing critical value for the DM test.
Subsequently, we compute the $DM$, fluctuation ($Fl$), and $S$ statistics and perform the corresponding tests, alongside the $MAX$ procedure. In detail, we set $\delta_s=1$ to study the size, and three different specifications for the power:
For each experiment, we run $10.000$ replications. Notice that $\delta_s = \delta \neq 1$ also affects the variance of $d_s$: this is particularly relevant for the $S$ test and $MAX$ procedure. Regarding the theoretical size, we set $5\%$. The summary of the results is provided in Table (ref).
All the tests are well-sized under the null hypothesis. The local power depends on the nature of the violation of the $E(d_s)=0$ hypothesis. The DM test (third column) has the best power performance when the deviation $E(d_s) \neq 0$ holds for all $s$, whereas the fluctuation test (forth column) has the best power performance in the case of a local but extended deviation. The $S$ test and $MAX$ diagnostic (fifth and sixth column) have no power when the deviation from the null hypothesis occurs at every point, and not much power (compared to the fluctuation test) in case of local but extended deviations. On the other hand, the $S$ test and $MAX$ diagnostic have the best power in the case of a very short deviation: interestingly, the $S$ has marginally more power compared to the $MAX$, reflecting the fact the $S$ test assumes exact knowledge of the location of the shift, whereas the $MAX$ procedure is more agnostic about this piece of information. On the other hand, the DM and fluctuation tests have no effective power in this situation of very short deviation.
Finally, we notice that, for the case of global and local deviations, the power of the DM and fluctuation tests increase with the distance from the null hypothesis $\delta=1$; the same holds for the $S$ test and $MAX$ diagnostic in the presence of very short deviation, but only when $\delta>1$: we conjecture that the lack of power for the $S$ test and $MAX$ procedure when $\delta<1$ depends on the reduction in the variance in $d_t$.
In the second experiment, we compare three versions of the $S$ test when more than one period is considered: in the first case (Column I), we do not apply a correction for the dependence in the test statistic, corresponding to using the identity matrix; in the second case (Column $\widetilde{\Sigma}$), we use $\widetilde{\Sigma}$ as recommended in andrews2003end, and in the third case (Column $\widehat{\Sigma}$), we employ $\widehat{\Sigma}$, where $\Sigma$ is estimated only using the pre-change sample.
The outcome of the experiment is provided in Table (ref): as andrews2003end anticipated using $\widehat{\Sigma}$ yields slightly better empirical size, since a few more observations are used; however, this size correction results in much less power when the dimension of the instability is large. On the other hand, perhaps because the dependence in the Monte Carlo exercise is not very large, we find that the power gain from using the estimate of $\Sigma$ instead of the Identity matrix is negligible.
In this section, we illustrate the results obtained from the Monte Carlo exercise with an empirical example dedicated to evaluating the nowcast of US nominal GDP growth by using the GDP nowcast from the Survey of Professional Forecasters (SPF) over the period 2000:Q1 to 2020:Q3. In particular, we consider for the SPF the nowcast of nominal GDP, denoted $\widehat{y}_t$, that is made when only information on nominal GDP of the previous quarter is available, ${y}_{t-1}$. As the survey covers many individuals, we use as $\widehat{y}_t$ the median of the responses for each point in time.
The nowcast of the quarterly growth rate is then $\frac{\widehat{y}_t-{y}_{t-1}}{{y}_{t-1}}$ and the error associated with the SPF nowcast is denoted as $e_t^{(1)}=\frac{{y}_{t}-\widehat{y}_t}{{y}_{t-1}}$. As a benchmark, we consider nowcasting nominal GDP growth as 0, which corresponds to nowcasting the GDP as the last available observation, $\widetilde{y}_t={y}_{t-1}$, so the error associated with the benchmark nowcast, $e_t^{(2)}$, is therefore $e_t^{(2)}=\frac{{y}_{t}-\widetilde{y}_t}{{y}_{t-1}}$. Using the quadratic loss function, the loss differential is then $d_t=e_t^{(1)2}-e_t^{(2)2}$.
Clearly, the benchmark in this exercise is not a very effective nowcast, for example, it even neglects the long-run growth in nominal GDP due both to economic growth and inflation. However, it is a convenient one, in the sense that we expect that $E(d_t)<0$, and therefore the null hypothesis should be rejected. This is then a fitting benchmark to check the predictions from Theorem (ref).
Figure (ref) provides the plot of the GDP growth, along with the SPF nowcast and the naive benchmark, while Figure (ref) shows the errors.
The SPF nowcast is always close to the target, except for an unanticipated and temporary surge of the growth rate in 2013; as anticipated, the naive benchmark does not take into account long-run real economic growth and inflation and the errors associated with it are then always positive, except for the 2008-2009 recession. Crucially, the SPF tracked well even the 2020 shock, whereas the naive errors are much higher for that period (in absolute value).
As a result (see Table (ref)), the performance in root mean square error terms (RMSE) of the naive benchmark worsens compared to the SPF nowcast, the ratio of the two RMSEs passing from 0.22 over the period up to 2019, to 0.16 when the three quarters in 2020 are added, even though the RMSE increased for both sources. We then turn to the DM and the Fl tests, again evaluated with the RMSE loss function: in both cases, we estimated the long-run variance using the Bartlett kernel and bandwidth $\lfloor T^{2/9}\rfloor$. For the Fluctuation test, we set $k$ so that the $k/T$ ratio, $\kappa$, is $0.3$, so the critical value for $FL_\kappa$ is $3.012$.
A summary of the results is available in Table (ref). A negative entry for the DM statistic means that the average RMSE for the SPF nowcast is lower than the average RMSE of the benchmark; $Fl_l$ and $FL_u$ are the minimum and maximum of the statistic $Fl_{s,k}$, respectively, where again a negative entry refers to the situation in which the RMSE of the benchmark exceeds the RMSE of the SPF nowcast, so the latter is more precise. In this example, both the DM and the Fluctuation tests suggest that the SPF nowcast is more precise than the benchmark, and significantly so if the sample is limited to up to 2019. As we anticipated the SPF does not seem to outperform the naive benchmark using the DM and the Fl tests when the observations for 2020 are included in the sample, although the RMSE ratio is even more favorable.
In conclusion, we analyse the performance of the two nowcasting models using the $S$ Andrews test and the $MAX$ diagnostic over the COVID-19 period (2020:Q1 - 2020:Q3). As the location of the COVID-19 recession and recovery can be treated as a shock at a known date, the application of the $S$ test seems appropriate, and in our case we focus on just three observations.
Results for the two diagnostics are in Table (ref): these both suggest an increase of the forecasting differential in the second period, but for the $S$ statistic weighing the errors with the restricted residuals we fail to reject the null, consistently with the finding in the Monte Carlo experience that this has less power, especially in presence of large shocks. Let us also add that the last column $q_{\text{MAX}}$ is the critical value which coincides with the MAX procedure over the pre-covid period (2000:Q1 - 2019:Q4).
COVID-19 was an exceptionally challenging event for forecasting and evaluation since it was a moment of extreme instability spanning over a very short period. In particular, tests like the dieboldComparingPredictiveAccuracy2002 test for equal forecasting ability or the giacomini2010forecast test have less power, if the time span of the instability is very short, even when the local discontinuity is very large relative to the sample size.
In this paper, we show that in these situations and subsequent recovery, using non-parametric diagnostics (such as the $S$ test or the MAX procedure) for local breaks or extreme values leads to better conclusions. We illustrate these results in two Monte Carlo exercises, and we provide evidence of the importance of selecting the correct test in a nowcasting exercise for the nominal US GDP, where we compare the SPF and a naive benchmark. Given these results, we recommend that the forecaster should not pool the sample, but exclude the short periods of high local instability from the evaluation exercise.