Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
141,317 characters · 16 sections · 62 citation commands
Equal Predictive Ability Tests Based on Panel Data with Applications to OECD and IMF Forecasts
Formal tests of the null hypothesis of no difference in forecast accuracy using two time series of forecast errors have been widely considered in the literature: see vuong89, diebold95, west96, clark01, clark15, giacomini06, clark07, {mariano12}, among others. On the contrary, the literature on such tests using panel data is scarce with {a few} exceptions: keane90, davies95 and qu22.
The main aim of this paper is to develop testing procedures for equal predictive ability (EPA) hypotheses based on panel data, taking into account the cross-sectional dependence (CD) and the temporal dependence in the data set. {Let $\widehat{y}_{l,it}$ be the forecast of agent $l=1,2$ made for the target value $y_{it}$ of time $t=1,2,\dots,T$ for unit (e.g., country, firm) $i=1,2,\dots,n$. We propose tests for comparing the predictive ability of two forecasting agents, using $n$ time series of loss differentials, $\Delta L_{it} = L(y_{it};\widehat{y}_{1,it})-L(y_{it};\widehat{y}_{1,it})$, of length $T$, where $L\left(\cdot\right)$ is a generic loss function gneiting11.} Our setting differs from that of {keane90} and davies95 in that we consider forecasts made by two forecasters on multiple economic units over time, whereas they consider forecasts made by multiple forecasters on a single economic unit over time. Our paper fills an important gap in the literature by allowing for multiple target values for each point in time {with respect to these two papers}.
We develop two types of tests corresponding to two EPA hypotheses. The first type focuses on the overall EPA hypothesis which states that the EPA holds on average over all units and over time. The statistics of this type, which we call $S$-statistics, are useful when a researcher is not interested in the differences of predictive ability for a specific unit {or clusters of units} but in the overall differences. The statistics of the second type, namely $C$-statistics, focus on the clustered EPA hypothesis which states that the EPA holds jointly for a fixed number of clusters of units in the panel. QTZ consider the same set of null hypotheses. Our work differs from that of QTZ in the way the test statistics are constructed, most notably the statistics for testing the clustered EPA hypothesis. While both works set up a joint null hypothesis as “all cluster means are zero" (see, (ref) in our paper and (11) in QTZ), we formulate multivariate test statistics in line with the joint hypotheses (the $C$-statistics in Section (ref)) with limiting null distribution being $\chi_G^2$, where $G$ is the number of clusters. On the other hand, QTZ formulate a univariate test statistic (the $J_n^D$ given in (12) in QTZ) based on the “average of the cluster averages of the loss differentials." If cluster sizes are equal, the numerator of $J_n^D$ is equivalent to these of $J_{n,T}^{DM}$ of QTZ and $S$-statistics in our paper, tests for the overall EPA. As a result, in general $J_n^D$ fails to detect departure from the clustered EPA hypothesis if the overall EPA hypothesis holds. See the detailed discussion given at the end of Section (ref).
The applied literature in comparing the accuracy of two or more forecasts suggests that the forecast errors of units, such as countries, are affected by common shocks such as the global financial crisis. For instance, pain14 show that the economic growth projections of the OECD for the period 2007-2012 are systematically upward biased. A similar tendency exists for other forecasters, such as the IMF. Moreover, these effects are carried into the loss differentials such that they follow a similar pattern, as we highlight later in this paper. The results of pain14 indicate also that the effect of these common shocks is heterogeneous across economies and some country clusters exists.
Following these insights, we build our testing framework around the loss differentials which follow an approximate factor model where some common factors affect all units in the panel with heterogeneous loadings. In addition, the error terms are allowed to be cross-sectionally weakly correlated. We therefore simultaneously allow the loss differentials to contain weak cross-sectional dependence (WCD) arising from, e.g., spatial error correlation, and strong cross-sectional dependence (SCD) due to the existence of common factors, using the terminology of chudik11. To develop our tests under WCD, we use non-parametric methods of variance estimation, based on geographic or economic distances between panel units kelejian07,kim13. In addition, we propose a novel partial sample variance estimator for large panels which deals with the case of unknown distances while being robust to arbitrary WCD and temporal dependence. To deal with SCD, we use the principal components estimator (PCE) built for large dimensional approximate factor models bai02,bai03. Following {the insights of} driscoll98, we also propose tests for the case of unknown number of common factors contrary to the ones based on the PCE. This last approach is flexible in the sense that it is robust under different types of CD. Moreover, they do not rely on a linear factor model contrary to the PCE based tests. They are also very easy to calculate as in the case of overall EPA testing, the proposed test is identical to applying the diebold95 test to cross-sectional averages of the loss differentials.
We analyze the asymptotic properties of the proposed test statistics using joint limits. Under mild conditions, the overall EPA test statistics and the clustered EPA test statistics are shown to converge in distribution to standard normal and chi-square with $G$ degrees of freedom under the null of interest, respectively. The finite sample properties of the tests are examined via Monte Carlo simulations. The results show in general that our tests have very good finite sample performance. Specifically, we find that driscoll98 based overall and clustered EPA tests are robust to arbitrary CD in the loss differentials. However, they have lower power than the tests developed specifically for the case of WCD if loss differentials are only weakly cross-sectionally correlated. Moreover, if the linear approximate factor model assumption is satisfied by the loss differentials, they have lower power compared to the tests based on PCE.
The proposed tests are used in two applications. In the first one, we compare the economic growth forecast errors of the OECD and the IMF using data for 29 countries over the period between 1998 and 2016. In the second application, the quality of the IMF consumer price inflation (CPI) forecasts is challenged by comparing them with random walk forecasts. This data set contains 127 countries over the period 1991-2019. In both applications, we find strong evidence of SCD in loss differentials of forecast errors. {The results of the first application show that neither overall EPA hypothesis nor clustered EPA hypothesis can be rejected for the economic growth forecasts of the OECD and the IMF. In contrast, in the second application, we find significant evidence against the overall and clustered EPA hypotheses on the comparison of the IMF and random walk forecasts, especially in the pre-crisis period.}
The remainder of the paper is as follows: The testing framework and the hypotheses of interest are presented in Section (ref). The panel EPA test statistics and their asymptotic properties are reported in Section (ref). Section (ref) is on the small sample properties of the proposed tests. {Section (ref) gives guidelines to practitioners willing to test the EPA hypotheses using panels.} In Section (ref) the economic growth forecast errors of the OECD and the IMF are compared using the proposed tests. Section (ref) concludes. Three appendices contain additional justification of our testing framework, proofs of results and {the details of the} second application comparing the IMF CPI forecasts with random walk forecasts.
Notation. Let $\mathbf{A} = [a_{ij}]$ be an $n \times n$ matrix. The column and row norms of $\mathbf{A}$ are $||\mathbf{A}||_1 = \max_{1\leq j\leq n}\sum_{i=1}^{n}|a_{ij}|$ and $||\mathbf{A}||_{\infty} = \max_{1\leq i\leq n}\sum_{j=1}^{n}|a_{ij}|$, respectively. The Euclidean norm of an $n \times m$ matrix $\mathbf{B}$ is $||\mathbf{B}|| = [\mathrm{Tr}(\mathbf{B}'\mathbf{B})]^{1/2}$. $M$ is a finite positive constant. $\overset{d}{\rightarrow}$ and $\overset{p}{\rightarrow}$ denote convergence in distribution and probability, respectively. Joint passage to infinity of $T$ and $n$ is denoted by $(T,n) \rightarrow \infty$.
We are interested in comparing the errors of forecasts made by two forecasters on an economic variable observed for units $i=1,2, \dots, n$ at time $t=1,2, \dots, T$. The loss differentials, $\Delta L_{it} = L(y_{it};\widehat{y}_{1,it})-L(y_{it};\widehat{y}_{1,it})$, are {potentially cross-sectionally correlated. The CD may be of strong or weak types or may even be generated by two distinct components of these two different types. To simplify the distinction, we assume the following structure on the loss differentials:}
where $\mathbf{f}_{t} = (f_{1t},f_{2t},\dots,f_{mt})'$ is an $m\times 1$ vector of unobservable common factors and $\boldsymbol{\lambda }_{i}= (\lambda_{1i},\lambda_{2i},\dots,\lambda_{mi})'$ is an $m\times 1$ vector of {fixed} factor loadings. The coefficients $r_{ij}$ are fixed, unknown elements of an $n\times n$ matrix $\mathbf{R}_{n}$. {We impose $r_{ii}=1$ for normalization of the variances.} $\mathbf{f}_{t}$ and ${\epsilon}_{it}$ are assumed to be zero mean weakly stationary time series allowed to be autocorrelated through time. In addition, ${\epsilon}_{it}$ is cross-sectionally uncorrelated. The unit specific means satisfy $|\mu_i|<\infty$.
We assume that the error terms $\varepsilon_{it}$ carry WCD, meaning that the variance of their cross-sectional average vanishes asymptotically:
where $\mathbf{r}_{i.}=(r_{i1},r_{i2},\dots,r_{in})^{\prime}$, $\boldsymbol{\gamma}_{n,0} = \mathrm{diag}(\gamma_{11,0},\dots,\gamma_{nn,0})$ and $\gamma_{ii,0} = \mathrm{E}({\epsilon}^2_{it})$. For this to be true, it requires the row and column sum norms of $\mathbf{R}_{n}$ to be bounded; see Assumption (ref) below. The common component in (ref) is assumed to induce SCD such that the variance of its cross-sectional average is bounded away from zero:
where $\boldsymbol{\Gamma}_{0} = \mathrm{E} (\mathbf{f}_{t} \mathbf{f}_{t}^{\prime })$. The conditions under which this holds true are given below, in Assumption (ref). The WCD can be modelled by a spatial process, such as spatial autoregression, spatial moving average as well as their higher order versions, or a factor model with a possibly infinite number of weak common factors chudik11.
{In this paper,} we follow closely the methodological implications of DM. Firstly, we adopt the view that the forecast errors may be related to forecasts made using some econometric models or simply by expert knowledge. As in DM, our approach is built on a model-free environment, meaning that it is agnostic about the process generating the forecasts. This is contrary to the extensions of DM in papers such as west96 and giacomini06 which focus, among other things, on problems associated with testing using forecast errors generated by nested models. Secondly, following DM, our assumptions concern directly the loss differentials. Forecast errors can be non-stationary and induce CD. What is important is the properties which are carried into the loss differentials, not the properties of the forecast errors specifically. As an example, in Appendix (ref) we show how {the specification in (ref)} can be obtained for two different loss functions, namely absolute and quadratic loss, starting with a pure factor model for the forecast errors. However, our approach is not necessarily limited to these loss functions. As in {the} DM approach, any other (symmetric or asymmetric) function can be used. Eventually, our methodology relies on the empirical analysis of time series and CD properties of the loss differentials, using CD tests pesaran15 and information criteria bai02.
The null hypotheses of EPA and the alternatives. The first hypothesis of interest is
where $\bar{\mu} = \lim_{n\to\infty} \bar{\mu}_n$ with $\bar{\mu}_n = n^{-1} \sum^n_{i = 1} \mu_i$. This hypothesis{, which was also considered by QTZ} states that the forecasts generated by the two agents are equally accurate on average over all $i$ and $t$. If a researcher is not interested in the difference in predictive power for a particular unit but the average difference over units, this hypothesis should be considered. It is particularly plausible to consider this in a micro forecasting study where the units are random draws from a population. {Throughout the paper, this hypothesis is called the overall EPA hypothesis.} {The alternative hypothesis in this case is that overall EPA does not hold:
Of course it is also possible to consider one-sided alternatives.
In a macro forecasting study, differences between clusters of units can have a specific economic importance and may be of interest from a policy perspective. For instance, a question of interest is whether the forecasts made by agents are more accurate for a particular cluster of countries in the sample. In this case, the null hypothesis can be formulated such that the predictive equality holds for $G$ clusters of units:
where $\bar{\mu}_{g} = \lim_{n_g\to\infty}\bar{\mu}_{g,n_g}$ with $\bar{\mu}_{g,n_g} = n_g^{-1} \sum_{i \in G_g} \mu_i$ and $G_g$ is the set of indexes of $n_g$ cross-sectional units which belong to cluster $g$. In this paper, this hypothesis is referred to as the clustered EPA hypothesis. The alternative hypothesis associated is the following:
QTZ consider the null hypothesis $H^{ca}_{0}$ without stating an alternative hypothesis. Their test against $H^{ca}_{0}$ is inconsistent in particular cases, as for instance, their clustered EPA test fails to reject $H^{ca}_{0}$ if $H^a_{0}$ and $H^{ca}_{1}$ hold simultaneously. This issue is discussed theoretically below in Section (ref). The difference between the two hypotheses is {therefore} important and the choice depends on the specific interest. The overall EPA may hold even if the two forecasters have different predictive ability for different clusters. This occurs if the average loss differentials of different country clusters are different from zero {(that is $H^{ca}_{1}$ holds)} but they add up to zero when pooled {(that is $H^a_{0}$ holds)}. If these clusters have an economic meaning, it is important to test the clustered EPA hypothesis.
In what follows, we propose test statistics for these two null hypotheses, under different assumptions on the CD structure in the loss differentials. These are No CD, WCD or SCD. The main difference between the proposed test statistics is the methods of calculation of the asymptotic variance depending on this structure. A summary of these test statistics is given in Table (ref). Most of our test statistics do not require the approximate factor model given in (ref) with the exceptions being $\underline{S}_{nT}^{(3)}$ and $\underline{C}_{nT}^{(3)}$. Nevertheless, we retain the factor model to simplify the discussion of different types of CD.
Consider the sample mean loss differential over time and units:
We provide testing procedures for the overall EPA implied in (ref) based on $\Delta\bar{L}_{nT}$ under different assumptions about the structure of CD in the loss differentials. Let $k_T(\cdot)$ be a kernel function and $d_T$ a sequence of {positive,} non-random bandwidth parameters. In all cases, the limiting null distribution of the test statistics is obtained under Assumptions (ref) and (ref).
Assumption (ref) is sufficient to obtain a CLT for the sample mean of ${\epsilon}_{it}$ for each $i$. Assumption (ref)(ref) is standard in time series literature. Assumption (ref)(ref) is a high level assumption that is used for consistent estimation of long-run variance for each $i$. The conditions under which this assumption holds are given by jansson02. Consistency also requires Assumption (ref)(ref) which controls the expansion of $d_T$ relative to $T$.
Overall EPA tests under cross-sectional independence. We first consider the simplest case where $\boldsymbol{\lambda }_{i}^{\prime }\mathbf{f}_{t} = 0$ for all $i,t$ and $r_{ij} = 0$ for all $i \neq j$. In order to test the null hypothesis $H^a_{0}$, we propose to use the following statistic:
where ${\hat{\sigma}}_{1,nT}^{2}= (nT)^{-1}\sum_{i=1}^{n} \sum_{t,s=1}^{T} k_{T}\left( {d_{ts}/d_T}\right) \Delta \tilde{L}_{it}\Delta \tilde{L}_{is}$, with $\Delta \tilde{L}_{it}=\Delta L_{it}-\Delta \bar{L}_{i,T}$, where $\Delta \bar{L}_{i,T} = T^{-1}\sum_{t=1}^{T} \Delta {L}_{it}$ and $d_{ts} = |t-s|$.
To prove this result, it is sufficient to show that $\sqrt{nT}(\Delta \bar{L}_{nT}-\bar{\mu}_{n})/\sigma_{1,nT} \overset{d}{\rightarrow} N(0,1)$ where $\sigma^2_{1,nT} = n^{-1}\sum_{i=1}^{n} \bar{\gamma}_{i,T}$ with $\bar{\gamma}_{i,T} = T^{-1} \sum_{t,s=1}^{T} {\gamma}_{i,d_{ts}}$, ${\gamma}_{i,d_{ts}} = \mathrm{E}(\epsilon_{it} \epsilon_{is})$, and $\hat{\sigma}_{1,nT}^{2} - \sigma^2_{1,nT} \overset{p}{\rightarrow} 0$. {It is easy to prove that the test statistic is divergent, hence consistent, under $H^a_{1}$. To see this, note that in this case $\Delta \bar{L}_{nT} \overset{p}{\rightarrow} \bar{\mu} \neq 0$, $\hat{\sigma}_{1,nT}^{2} \overset{p}{\rightarrow} \sigma^2_{1} > 0$ where $\sigma^2_{1} = \lim_{n,T\to\infty} \sigma^2_{1,nT}$. It follows that $P[|S_{nT}^{(1)}|>c]{\rightarrow}1$ for any constant $c \in \mathbb{R}$ as $(T,n) \rightarrow \infty$. }
Overall EPA tests under WCD. Suppose $\boldsymbol{\lambda }_{i}^{\prime }\mathbf{f}_{t} = 0$ for all $i,t$ but $r_{ij}$ is not necessarily zero for all $i \neq j$. In this case, the loss differentials $\Delta L_{it}$ are no longer independent across $i$. Define $d_{ij}=d_{ji}\geq 0$ as the distance between units $i$ and $j$. We make Assumptions (ref) and (ref) on the coefficients $r_{ij}$.
Assumption (ref) is standard in spatial econometrics literature and implies the WCD defined in (ref). The role of Assumption (ref) is to restrict the spatial correlation among panel units in relation to the distances between them. As noted by kelejian07, the corresponding condition in the time series context is fading memory over time. Under this assumption, as the distance between two panel units increase, the correlation between them decreases. This in turn sets a basis for using the spatial kernel function $k_{S}(\cdot)$ which gives smaller weights to the covariance between units which are more distant from each other.
To deal with the WCD when $T=1$, kelejian07 proposed a spatial heteroskedasticity and autocorrelation consistent estimator of the variance-covariance matrix. This estimator can be seen as a spatial version of the kernel estimators for time series such that it uses a spatial kernel based on the distance between units. The estimator is generalized by kim13 to panel data regression, by combining the spatial and time kernels. Define $k_{q} = \lim_{x\to0} [1-k_S(x)]/|x|^{q}$ and let $\rho_s = \max \{ q : k_{q} < \infty \}$ be the Parzen exponent of $k_S(\cdot)$ andrews91{, and $d_n$ a sequence of positive, non-random bandwidth parameters.} Assumption (ref) is placed on the spatial kernel function.
The condition in (ref)(ref) is satisfied by all kernel functions used in practice, such as Bartlett, Parzen, Tukey–Hanning and quadratic spectral andrews91. Conditions similar to that in (ref)(ref) are used by kelejian07, moscone12 and kim13. All these studies allow solely for kernels which truncate, i.e. those which equal zero after a certain value of the bandwidth parameter. The first two papers place assumptions on the relative expansion of $l_n = \max_{1 \leq i \leq n} l_{i,n}$ where $l_{i,n} = \sum^n_{j=1} 1\{d_{ij}\leq d_n\}$. Assumption (ref)(ref) controls the expansion of $d_n$ relative to $n$. This condition, as well as (ref)(ref), is not necessary for the consistent estimation of the variance-covariance matrices in our study. Nevertheless, these conditions are retained to compare our results with the existing literature.
We propose the following test statistic in this case of WCD:
where
The theoretical properties of the estimator $\hat{\sigma}_{2,nT}^{2}$ are explored by kim13. moscone12 use a similar estimator with the difference being that they set $k_{T}\left( \cdot \right) =1$.
The disadvantage of this variance estimator is that for its implementation the distances between all pairs of units, $d_{ij}$ have to be known to the researcher. Furthermore, the cut-off distance $d_{n}$ has to be chosen. In practice there may be many possible distance metrics and economic theory does not always help to choose between them. When a distance metric is not available we can use a partial sample estimator given by
where $\underline{n}$, an increasing function of $n$, is the number of observations used to calculate the variance. It is strictly smaller than $n$. Similar variance estimators are used by bai06 and moscone15. The first study focuses on the factor models whereas the second one deals with panel regression models with small $T$. Our variance estimator generalizes that of the moscone15 by allowing for a large $T$ with the help of a time kernel. The corresponding test statistic is given by
Under Assumptions (ref)-(ref), $\sqrt{nT}(\Delta \bar{L}_{nT}-\bar{\mu}_{n})/\sigma_{2,nT} \overset{d}{\rightarrow} N(0,1)$ as $(T,n) \rightarrow \infty$, where $\sigma_{2,nT}^{2} = n^{-1} \sum_{i,j=1}^{n}\mathbf{r}_{i.}' \bar{\boldsymbol{\gamma}}_{nT} \mathbf{r}_{j.}$ with $\bar{\boldsymbol{\gamma}}_{nT} = T^{-1} \sum_{t,s=1}^{T} \boldsymbol{\gamma}_{n,d_{ts}}$ and $\boldsymbol{\gamma}_{n,d_{ts}} = \mathrm{diag}(\gamma_{1,d_{ts}},\gamma_{2,d_{ts}},\dots,\gamma_{n,d_{ts}})$. Hence, the stated results follow by the consistency of the variance estimators. {Similar arguments made in the case of no CD lead to the consistency of the test statistics in the case which the alternative $H^a_{1}$ holds.}
{The test statistic $\underline{S}_{nT}^{(2)}$ is an asymptotic one which, in practice, can be hard to implement. Despite its advantage of not relying on a distance metric, it needs special care as there is not clear indications on which cross-sectional units to use in the calculation. A solution in practice is to fix $\underline{n}$, calculate the test statistic for a large number of subsamples of cross-sectional units of size $\underline{n}$, and to take the smallest test statistic. This provides a conservative test statistic which would be correctly sized but potentially with low power. Nevertheless, below, we consider another test statistic, ${S}_{nT}^{(3)}$, which is robust to WCD and does not pose similar problems. }
Overall EPA tests under SCD. This is the most general case with no specific restriction imposed on the CD of the loss differentials. To obtain a CLT for the means, we make Assumption (ref) on the common factors, their loadings and the error terms.
It is easy to see that, ${\boldsymbol{\lambda }}_{i}'\mathbf{f}_{t}$ satisfying Assumption (ref) lead to SCD as defined in (ref). We require Assumption (ref)(ref) for the consistent estimation of the long-run variance of the common factors. Assumption (ref)(ref) is standard for factor models. Assumption (ref)(ref) ensures that at least one of the factors contribute to the asymptotic variance of the cross-sectional averages. Although nonstandard, this assumption is not restrictive as it does not affect the validity of our testing procedures.
In this case of SCD, the variance estimator given in ((ref)) can be modified by setting $k_{S}(\cdot)=1$. This variance estimator does not require any knowledge of a distance measure between the units. Moreover, it assigns weights equal to one for all covariances from the same time period, hence {it is} robust to SCD as well as WCD. The test statistic takes the form:
where
The variance estimator (ref) is valid when $T$ is large, regardless of $n$ being finite or infinite driscoll98. {The special case of this test statistic where $k_{T}$ is the Bartlett kernel corresponds to the test statistic $J_{n,T}^{DM}$ of QTZ. As the authors note, an important advantage of this test statistic is that it does not rely on the linear approximate factor model in (ref). It is robust even under DGPs such as $\Delta L_{it} = \mu_i + g(\boldsymbol{\lambda }_{i},\mathbf{f}_{t})+\varepsilon_{it}$ where $g(\cdot,\cdot)$ is a non-parametric function. Furthermore, the test statistic $S_{nT}^{(3)}$ is very easy to calculate as it is identical to the DM test statistic for the cross-sectional averages of loss differentials. To see this, it suffices to write $\Delta \bar{L}_{nT} = T^{-1} \sum_{t=1}^{T} \bar{L}_{n,t}$ and $\hat{\sigma}_{3,nT}^{2}=T^{-1} \sum_{t,s=1}^{T} k_{T}\left(d_{ts}/d_{T}\right) \Delta \tilde{L}_{n,t}\Delta \tilde{L}_{n,s}$, where $\bar{L}_{n,t} = n^{-1} \sum_{i=1}^{n} \Delta L_{it}$ and $\Delta \tilde{L}_{n,t} = \bar{L}_{n,t} - \Delta \bar{L}_{nT}$.}
An alternative way to estimate the covariance matrix is to exploit the factor structure of the DGP. The PCE of large panels is investigated by stock02, bai02, bai03, among others. This is a flexible estimator which is robust to WCD and autocorrelation in the error terms. It is obtained by minimizing the average squared residuals computed for $m$ common factors:
subject to $\mathrm{Var}(\mathbf{f}_{t})=\mathbf{I}_{m}$ and $\boldsymbol{\Lambda}_n'\boldsymbol{\Lambda}_n$ being diagonal where $\boldsymbol{\Lambda}_n = (\boldsymbol{\lambda}_1,\boldsymbol{\lambda}_2,\dots,\boldsymbol{\lambda}_n)'$. Then the solution for the estimates of the common factors, $\widehat{\mathbf{f}}_{t}$, are given by $\sqrt{T}$ times the first $m$ eigenvectors of the matrix $\sum_{i=1}^{n} \Delta \tilde{\mathbf{L}}_{i.} \Delta \tilde{\mathbf{L}}_{i.}^{\prime}$ with $\Delta \tilde{\mathbf{L}}_{i.}=(\Delta \tilde{L}_{i1},\Delta \tilde{L}_{i2},\dots ,\Delta \tilde{L}_{iT})^{\prime}$ and the factor loadings can be estimated as $\widehat{\boldsymbol{\lambda }}_{i}=\frac{1}{T} \sum_{t=1}^{T}\widehat{\mathbf{f}}_{t}\Delta \tilde{L}_{it}$. {We make the following high level assumption on the asymptotic properties of the PC estimates.
This is a standard result in the literature on the approximate factor models. The conditions under which it holds are given in bai03 (see their Theorem 3).} Then the overall EPA hypothesis can be tested using
where
with $\widehat{\varepsilon}_{it} = \Delta \tilde{L}_{it} - \widehat{\boldsymbol{\lambda }}'_{i} \widehat{\mathbf{f}}_{t}$.
Under Assumptions (ref)-(ref) and (ref), $\sqrt{T}(\Delta \bar{L}_{nT}-\bar{\mu}_{n})/\sigma_{3,nT} \overset{d}{\rightarrow} N(0,1)$, where
with $\bar{\boldsymbol{\Gamma}}_{T} = T^{-1} \sum_{t,s=1}^{T} \boldsymbol{\Gamma}_{d_{ts}}$. The rate of convergence in the CLT is $T^{1/2}$ instead of the usual rate of $(nT)^{1/2}$ in the cases of no CD and WCD. This follows from the SCD characterized in (ref). {We can show the consistency of the test statistics using similar arguments to the ones made earlier, after Propositions (ref) and (ref). It can be easily shown that the variance estimator $\hat{\sigma}_{3,nT}^{2}$ is consistent under no CD as well as WCD. Hence the test statistic $S_{nT}^{(3)}$ is consistent in these cases as well. However, the rate of divergence is smaller than those of Propositions (ref) and (ref). Thus, the test is expected to have a lower power against the null $H^a_{0}$. }
In (ref), the first term in parentheses dominate{s} the second one. This is because, under Assumptions (ref), (ref) and (ref), $n^{-2} \sum_{i,j=1}^{n} \boldsymbol{\lambda}'_i \bar{\boldsymbol{\Gamma}}_{T} \boldsymbol{\lambda}_j = O(1)$ but $n^{-2} \sum_{i,j=1}^{n} \mathbf{r}_{i.}' \bar{\boldsymbol{\gamma}}_{nT} \mathbf{r}_{j.} = O(1/n)$. Hence, the latter is asymptotically negligible. This means that under SCD, one can use a simpler variance estimator without the second term in (ref).
{ When $T$ is fixed but $n$ is large, the test statistic $S_{nT}^{(3)}$ is not valid as it relies on the $T$-asymptotics. However, under additional assumptions on the DGP of the loss differentials, we can obtain a test statistic for fixed $T$. Specifically, assume that the loss differentials are based on optimal one-step ahead forecasts in the sense that they are serially uncorrelated. Consider the following test statistic:
where
We have the following result:
Here $t(T-1)$ represents the Student's $t$-distribution with $T-1$ degrees of freedom. The result shows that, under additional assumptions to those of Propositions (ref) and (ref), we can use the Student's $t$ critical values after adjusting the degrees of freedom in the test statistic ${S}_{nT}^{(3)}$. If the loss differentials carry SCD, we need normality of the error terms for the result to hold. If they are weakly cross-sectionally correlated, the normality assumption is not required as under WCD assumption cross-sectional averages of the loss differentials admit a CLT, i.e. they are normally distributed for each $t=1,2,\dots,T$. }
Define $G_g$, $g=1,\dots,G$, as the set of indexes of $n_g$ cross-sectional units which belong to cluster $g$ such that $G_{g} \cap G_{g'} = \emptyset$, $\forall g \neq g'$. In this subsection, we are interested in testing the null hypothesis $H^{ca}_{0}$ which can be written as
where $\bar{\boldsymbol{\mu}} =(\bar{\mu}_{1},\bar{\mu}_{2},\dots,\bar{\mu}_{G})^{\prime }$. Our tests are based on the empirical counterpart of this quantity: $\Delta\bar{\mathbf{L}}_{nT}=(\Delta \bar{L}_{1,n_1,T},\Delta \bar{L}_{2,n_2,T}, \ldots ,\Delta \bar{L}_{G,n_G,T})^{\prime }$ where $\Delta \bar{L}_{g,n_g,T} = (n_gT)^{-1} \sum_{i \in G_g} \sum_{t = 1}^T \Delta L_{it}$. We assume that the sets of indexes $G_g$, $g=1,\dots,G$ are known. Assumption (ref) is placed to control the asymptotic number of units per cluster.
With this assumption we do not rule out the possibility of having $G = 1$ in which case we have $n_1 = n$. This particular case corresponds to the overall EPA tests of the previous subsection. In what follows, these test statistics are generalized for $G > 1$ for each case of CD.
Clustered EPA tests under cross-sectional independence. In this case $\boldsymbol{\lambda }_{i}^{\prime }\mathbf{f}_{t} = 0$ for all $i,t$ and $r_{ij} = 0$ for all $i \neq j$. We propose the following statistic to test the hypothesis in (ref):
where
with $g_i \in \{1,2,\dots,G\}$ being a variable which states the cluster which $i$th unit belongs to and $\boldsymbol{\iota}_{g_i}$ being the $g_i$th column of $\mathbf{I}_{G}$. Notice that $\widehat{\boldsymbol{\Omega}}_{1,nT}$ is a diagonal matrix which contains an estimate of the average long-run variances of each cluster as diagonal elements {up to a factor which is asymptotically equal to $\tau^{-1}_{g}$}.
Under Assumptions (ref), (ref) and (ref), we have $\sqrt{nT}\boldsymbol{\Omega}_{1,nT}^{-1/2}(\Delta\bar{\mathbf{L}}_{nT} - \bar{\boldsymbol{\mu}}_{n})\overset{d}{\rightarrow} N(\mathbf{0},\mathbf{I}_{G})$ as $(T,n) \rightarrow \infty$, where $\bar{\boldsymbol{\mu}}_{n} =(\bar{\mu}_{1,n_1},\bar{\mu}_{2,n_2},\dots,\bar{\mu}_{G,n_G})^{\prime }$ and $\boldsymbol{\Omega}_{1,nT} = \sum_{i=1}^{n} \frac{n}{n^2_{g_i}} \boldsymbol{\iota}_{g_i}\boldsymbol{\iota}_{g_i}^{\prime} \bar{\gamma}_{i,T}$. This is a generalization of the CLT following Proposition (ref) which is obtained when $G=1$. {It is easy to see that the test is consistent under the alternative hypothesis $H^{ca}_{1}$. This is because under the assumptions of the proposition, the matrix $\widehat{\boldsymbol{\Omega}}_{1,nT}$ tends to a positive definite matrix as $(T,n) \rightarrow \infty$ whereas at least one entry of the vector $\Delta\bar{\mathbf{L}}_{nT}$ tends to a non-zero constant provided that there exists a $g \in \{1,\dots,G \}$ for which $\bar{\mu}_{g} \neq 0$. Furthermore, the test is consistent against $H^a_{0}$ as this hypothesis fails if any cluster has a non-zero mean. However, we expect this test to have lower power against $H^a_{0}$ compared to $S^{(1)}_{nT}$.}
Clustered EPA tests under WCD. Suppose $\boldsymbol{\lambda }_{i}^{\prime }\mathbf{f}_{t} = 0$ for all $i,t$ but $r_{ij}$ is not necessarily zero for all $i \neq j$. We can use the following statistic in order to test $H^{ca}_{0}$:
where
As discussed in the previous section, the estimator has the disadvantage of relying on a known distance between each unit in the panel. An alternative variance estimator for this case of cross-sectional clusters can be constructed as in (ref). Such an estimator is
where $\underline{n}_{g}$ is the number of observations taken into account in the calculation of variance in cluster $g$ hence we have $\underline{n} = \sum_{g=1}^{G} \underline{n}_{g}$. The corresponding test statistic is
Under the assumptions of the theorem, we have $\sqrt{nT}\boldsymbol{\Omega}_{2,nT}^{-1/2}(\Delta\bar{\mathbf{L}}_{nT} - \bar{\boldsymbol{\mu}}_{n})\overset{d}{\rightarrow} N(\mathbf{0},\mathbf{I}_{G})$ as $(T,n) \rightarrow \infty$, where $\boldsymbol{\Omega}_{2,nT} = \sum_{i,j=1}^{n} \frac{n}{n_{g_i} n_{g_j}} \boldsymbol{\iota}_{g_i}\boldsymbol{\iota}_{g_j}^{\prime} \mathbf{r}_{i.}' \bar{\boldsymbol{\gamma}}_{nT} \mathbf{r}_{j.}$. Then the stated results follow from the consistency of the variance estimators. This theorem nests Propositions (ref), (ref) and (ref). The first is obtained when $\mathbf{R}_n = \mathbf{I}_n$ and $G=1$, the second is obtained with only $G=1$ and the last is when only $\mathbf{R}_n = \mathbf{I}_n$. {The consistency of the tests follow from similar arguments to the ones above, those under Proposition (ref).}
Clustered EPA tests under SCD. We now consider the case of no specific restrictions on the CD in the loss differentials. With SCD, the clustered EPA hypothesis $H^{ca}_{0}$ can be tested using:
As discussed previously, this type of variance estimator is robust to arbitrary CD and it has the advantage of not requiring known distances between units. {Furthermore, like $S^{(3)}_{nT}$, it is very easy to calculate as it is identical to a Wald test applied to within-cluster cross-sectional averages where between cluster covariances as well as serial correlations are taken into account.} However, its performance may be poor in cases of $n$ large relative to $T$. Hence, once more the factor structure of the loss differentials can be used to form variance estimators. A test statistic with such an estimate is
Under the assumptions of the theorem, we have $\sqrt{T}\boldsymbol{\Omega}_{3,nT}^{-1/2}(\Delta\bar{\mathbf{L}}_{nT} - \bar{\boldsymbol{\mu}}_{n})\overset{d}{\rightarrow} N(\mathbf{0},\mathbf{I}_{G})$, where $ \boldsymbol{\Omega}_{3,nT} = \sum_{i,j=1}^{n} \frac{1}{n_{g_i} n_{g_j}} \boldsymbol{\iota}_{g_i}\boldsymbol{\iota}_{g_j}^{\prime} \left( \boldsymbol{\lambda}'_i \bar{\boldsymbol{\Gamma}}_{T} \boldsymbol{\lambda}_j + \mathbf{r}_{i.}' \bar{\boldsymbol{\gamma}}_{nT} \mathbf{r}_{j.} \right) $. Proposition (ref) is a special case of this theorem with $G=1$. It is easy to prove the consistency of the tests following arguments similar to the ones under Proposition (ref) and Theorem (ref). Similar to the discussion following Proposition (ref), $C_{nT}^{(3)}$ is expected to be consistent under WCD but have lower power than $C_{nT}^{(2)}$ in this case.
Furthermore, it is easy to see that the test is consistent against $H^a_{0}$, following similar arguments of the discussion of Proposition (ref). However, it is important to note that none {of} the overall EPA test statistics is consistent under general alternatives of $H^{ca}_{0}$. To see this, suppose that $G=2$ with $\bar{\mu}_{1,n_1} \to \mu^0 \neq 0$, $\bar{\mu}_{2,n_2} \to -\mu^0$ as $n\to\infty$. Then $\bar{\mu}_n \to 0$ as $n\to\infty$, hence, the overall EPA hypothesis holds asymptotically while the clustered EPA hypothesis fails. Therefore, the $S$-statistics follow their asymptotic null distributions under their respective assumptions, as stated in Propositions (ref)-(ref) and in Corollary 1.
Comparison of $C$-statistics and $J_n^D$ of QTZ. QTZ propose a test statistic for the clustered EPA hypothesis of the form
where $D_g = \frac{1}{\sqrt{n_gT}} \sum_{i \in G_g} \sum_{t = 1}^T \Delta L_{it}$ and $\bar{D} = G^{-1}\sum_{g=1}^G D_g$ and they suggest to reject the null $H^{ca}_{0}$ if $|J_n^D|>t_{G-1,1-\alpha/2}$ where $t_{G-1,1-\alpha}$ is the $1-\alpha$ quantile of the $t(G-1)$ distribution. To see how $J_n^D$ is related to the test statistics that we propose, let us first suppose that $n_g=n/G$ for all $g=1,\dots,G$, that is, the cluster sizes are identical. Then, the numerator of $J_n^D$ is equal to $\sqrt{nT}\Delta \bar{L}_{nT}$, the numerator of our $S$-statistics, and that of $J_{n,T}^{DM}$ of QTZ:
Hence, the test statistic is closer to the $S$-statistics rather than the $C$-statistics.
To have a clear idea on the asymptotic properties of $J_n^D$, let us suppose now that $G=2$ and the cluster sizes are asymptotically identical: $\tau_1=\tau_2=1/2$. Suppose also that $\bar{\mu}_{1,n_1} \to \mu^0 \neq 0$, $\bar{\mu}_{2,n_2} \to -\mu^0$ as $n\to\infty$, therefore $\bar{\mu}_n \to 0$. In this case $H^a_{0}$ holds and the asymptotic distribution of $\sqrt{G}\bar{D}$ is identical to that of $\sqrt{nT}\Delta \bar{L}_{nT}$ under WCD. Hence, even if the clustered EPA hypothesis fails, the test cannot detect the deviations from it.
Tests for joint equal predictive ability. In macroeconomic applications, the differences in the predictive ability for each cross-sectional unit can have a specific economic importance. When this is the case, one may be interested in the following hypothesis:
This hypothesis, namely the joint EPA hypothesis, states that the EPA hypothesis holds for each unit in the sample. This hypothesis can be seen as a special case of the clustered EPA hypothesis $H^{ca}_{0}$ with the number of clusters being equal to the number of units in the panel, that is $G=n$. However, in this case Assumption (ref) is violated as for each $g=1,\dots,G$, $n_g = 1$ and therefore $n_g/n \rightarrow 0$. Nevertheless, the hypothesis can still be tested using the above test statistics if $n$ is fixed, after suitable modification of the convergence rates. For instance, a test statistic which is robust to arbitrary CD is
with ${\Delta }\bar{\mathbf{L}} _{nT}=(\Delta \bar{L}_{1T},\Delta \bar{L}_{2T}, \ldots ,\Delta \bar{L} _{nT})^{\prime }$, and $\boldsymbol{\iota}_{i}$ being the $i$th column of $\mathbf{I}_{n}$. Different cases of CD can be covered by modifications on the variance estimator. These cases are discussed at length in a previous version of this paper akgun20, where we also consider the case of large $n$.
Tests for individual cross-sections. The overall test statistics $S_{nT}^{(1)}$, $S_{nT}^{(2)}$ and $\underline{S}_{nT}^{(2)}$, and clustered test statistics $C_{nT}^{(1)}$, $C_{nT}^{(2)}$ and $\underline{C}_{nT}^{(2)}$ are applicable to a single cross-section{, i.e. when $T=1$} {after minor modifications. As an example, let us take $S_{nT}^{(1)}$. The modified statistic will take the following form:
where ${\hat{\sigma}}_{1,n}^{2}= n^{-1}\sum_{i=1}^{n} \Delta \tilde{L}_{it}\Delta \tilde{L}_{it}$, with $\Delta \tilde{L}_{it}=\Delta L_{it}-\Delta \bar{L}_{n,t}$, where $\Delta \bar{L}_{n,t} = n^{-1}\sum_{i=1}^{n} \Delta {L}_{it}$. The modification is on the calculation of the variance, where we calculate the deviations from the overall mean, instead of the individual means.} For a single cross-section, a CLT can be easily obtained as a basis for these statistics. For instance, we can apply the CLT for independent but heterogeneous sequence white01 to show the normality of tests with no CD, and the CLT for spatially correlated triangular arrays of kelejian98 to show the normality of tests with WCD, respectively.
To investigate the finite sample properties of the test statistics given above, a set of Monte Carlo experiments are conducted. 2000 samples are generated from each DGP described below for the dimensions of $T\in \{10,20,30,50,100\}$ and $n\in \{10,20,30,50,100\}$. All tests are applied for the nominal size of 5%.
Two different DGPs are considered to explore the effect of WCD and SCD on the performance of the tests. DGP1 contains only WCD {which is controlled using a stationary spatial AR(1) process which satisfies Assumption (ref)}. {In this case, for each unit $i$, two independent forecast error series $(e_{1,it},e_{2,it})$ are generated using spatial AR(1) processes. Define first
where $w_{ij}$ is the element of the spatial matrix $\mathbf{W}_{n}$ in row $i$ and column $j$ with $w_{ii} = 0$ for all $i=1,2,\dots,n$. Then the forecast error series $e_{l,it}$, $l=1,2$ are generated as the $i$th element of the $n$-vector
where $\mathbf{u}_{l,n,t}=(\mathbf{u}_{l,1t},\mathbf{u}_{l,2t},\dots ,\mathbf{u}_{l,nt})^{\prime }$, $\mathbf{S}_n=(\mathbf{I}_{n}-\rho \mathbf{W}_{n})^{-1}$ and $\bar{s}_{2} = n^{-1}\mathrm{tr}(\mathbf{S}_n\mathbf{S}_n^{\prime })$. To explore the size of various tests, we use these two forecast error series $e_{l,it}$, $l=1,2$. We set $\rho = 0.5$.\footnote{We also tried $\rho = 0$ and $\rho = 0.9$. The findings are similar to those obtained with $\rho = 0.5$ and the results omitted to save space.} In this DGP a quadratic loss function is used.}
DGP2 contains SCD and WCD. In this case, following giacomini06, we generate the loss differential directly, so the tests do not rely on a specific loss function. This is given by
To investigate the size we set $\mu _{i}=0$ for each $ i=1,2,\dots ,n$ and generate factor loadings as $ \lambda _{1i},\lambda _{2i}\sim iid.N(1,0.2) $. The common factors are formed by $ f_{1t},f_{2t}\sim iid.N(0,1) $, The error series $\varepsilon_{it}$ are generated in the same spirit as in ((ref)){, precisely we set $\varepsilon_{it} = e_{1,it}$}. We finally set $\xi=\sqrt{1/3.4}$ to control for the variance of the loss differential series.
{We also undertake a robustness check to study the small sample properties of the tests $S_{n}^{(3)}$ and $\tilde{S}_{n}^{(3)}$ by considering a heavy-tailed error distribution. We generate errors from a $t(6)$ distribution for half of the panel and from a standard normal for the other half. Specifically, in (ref) we set, for each $l=1,2$, $u_{l,it}\sim iid.t(6)$ if $i=1,\dots,N/2$ and $u_{l,it}\sim iid.N(0,1)$ otherwise.}
{We explore the power of various tests under two different alternative hypotheses. For this purpose, in the case of DGP1, we generate a third series $e_{3i,t}$ as two different re-parametrization{s} of the series $e_{2,it}$.} The first one corresponds to the homogeneous alternative and the second one to the heterogeneous alternative. We generate $e_{3i,t}=\sqrt{1.2}e_{2,it}$ and report the results from testing the equality of forecast accuracy of $e_{1,it}$ and $e_{3i,t}$. In the heterogeneous scenario, we generate $e_{3i,t}=\sqrt{\theta_{i}}e_{2,it}$ where $\theta _{i} = 0.8$ for $i=1,\dots,n/2$ and $\theta _{i} = 1.2$ for $i=n/2+1,\dots,n$. Similarly, in the case of DGP2, we set $\mu _{i}=1.2$ for each $i$ in the case of homogeneous alternative and $\mu _{i} = -0.2$ for $i=1,\dots,n/2$ and $\mu _{i} = 0.2$ for $i=n/2+1,\dots,n$ in the case of heterogeneous alternative.
As the error series and the common factors are serially uncorrelated for each unit, it is implicitly assumed that we are dealing with one-step ahead forecasts. Hence, we set the time series kernel $k_{T}(\cdot)=1$ if $t=s$ and $k_{T}(\cdot)=0$ otherwise. Spatial interactions between units are created with a row-normalized rook-type weight matrix.\footnote{The units are assumed to lie on a $p_1 \times p_2$ rectangular grid such that the first $p_1$ units are located in the first column of the grid, the second $p_1$ units are located in the second column and so on. We therefore have $n = p_1 p_2$. For each $n\in \{10,20,30,50,100\}$, we choose $p_1$ as 2, 4, 6, 10 and 50, respectively.} The distance between two units is the given by the Euclidean distance. In the computation of the spatial kernel $k_{S}(\cdot)$, we use these distances. In addition, we use distances based on the wrong assumption that the units are located on a line. We use Bartlett kernel for all experiments and following kelejian07, we set the spatial kernel bandwidth to $d_n = \lceil n^{1/4} \rceil+1$ where $\lceil \cdot \rceil$ stands for the smallest integer bigger than its argument. Similarly, the overall and clustered EPA test statistics using partial sample variance estimators are calculated by setting $\underline{n} = \lceil n^{1/2} \rceil+1$ and $\underline{n}_{1} = \underline{n}_{2} = \lceil n^{1/2}/2 \rceil+1$, respectively.
For the tests using common factors, we consider three possibilities for the selection of the number of factors. First, we calculate them assuming $m=2$, meaning that for DGP2 the number of common factors is correctly specified. Second, we set $m=1$ for the case of under-specification. Third, we select the number of common factors by minimizing the $IC_{p1}$ information criterion\footnote{The authors consider several information criteria. In their simulations $IC_{p1}$ appears to be the best performing criterion under WCD.} of bai02 given by
where $V(m)$ is defined by (ref). The case of over-specification of the number of common factors is not separately considered because $IC_{p1}$ almost always over-estimates the number of common factors in small samples (see Table (ref) below).
Before the discussion of the size and power of the robust tests, as a benchmark we refer to the results on the non-robust tests $S^{(1)}_{nT}$ and $C^{(1)}_{nT}$. The size and power of these tests are reported in Table (ref). As is expected, all tests are incorrectly sized.
The results on the size of CD-robust overall EPA tests with DGP1 are given in Table (ref). The size of the kernel robust test $S^{(2)}_{nT}$ of the overall EPA hypothesis improves with either $T$ or $n$. First, we focus on the results when the distance metric is correctly specified. In the smallest samples with $T = 10$ and $n = 10$, this particular setting provides an empirical size of 9.9%. For $T = 100$ with $n = 10$ corresponding value equals 8.65% whereas for $T = 10$ with $n = 100$ it is 8.4%. In the largest sample its size is 6.3% which is close to the nominal value of 5%. When the distance between the panel units is incorrectly specified, the size of the test still improves with either dimension. However, as expected the size distortions are slightly larger in this case. In the largest smallest and largest sample sizes its size equals 12.5% and 8.3%, respectively. The test which uses the partial sample estimator of the variance has similar size values. However, its size improves only with $T$. When $T = n = 10$ its size is slightly larger than that of the kernel robust test with a misspecified distance. When $(T,n) = (10,100)$ the size distortion increases (13.4%). In the case of large $T$, however, it performs better than the kernel robust test with incorrect distance. For instance, when $(T,n) = (100,100)$ its size equals 7.7%.
The test $S^{(3)}_{nT}$ performs very well especially when $T$ is large and $n$ is small. In most of the combinations of $T$ and $n$ it shows better properties than $S^{(2)}_{nT}$. When $T$ is greater than 50, it is correctly sized {and} even when $T=30$ and $n=100$ its size is 6.9% which makes it the preferred test over any version of $S^{(2)}_{nT}$.
The test $\underline{S}_{nT}^{(3)}$ shows good properties even though it wrongly assumes that the loss series contain common factors. The test with $m=2$ performs similarly to $S^{(3)}_{nT}$ when $n$ is small but its performance is less good as $n$ gets large. In general, the size distortion of the test is bigger when $m=1$.
The case where the number of common factors is chosen by IC requires some special attention. The test performs well for small $n$ and $T$ but its performance drops as $n$ or $T$ gets large. To understand this behavior of the test, we check the small sample properties of $IC_{p1}$. The average number of common factors chosen by this criterion over simulations is reported in Table (ref). It is seen that, when either of the dimension of the panel grows, the performance of the IC increases. However, when one of the dimensions is small, this improvement is very slow. In DGP1, only when one of the dimensions is greater than 50, the performance is in acceptable levels. Once either $T$ or $n$ is greater than 50, the average number of factors selected over replications is either zero or very close to zero. This means, in fact in these cases the test converges to the non-robust test. Hence, the empirical size and power of the test equals the size and power of the non-robust test given in Table (ref). Of course, in practice the number of common factors is rarely known to the researcher and the most realistic application of this test is this case which is based on IC. However, it should not be understood that our testing procedure is best described by the performance in this case. In practice, if the no CD hypothesis is rejected by a suitable test, and the researcher decides with the help of an IC that the loss differentials do not contain common factors, an EPA test which is robust to WCD has to be used; for instance $S^{(2)}_{nT}$ or $S^{(3)}_{nT}$. As in the case of zero common factors $\underline{S}_{nT}^{(3)}$ is identical to $S^{(1)}_{nT}$, these tests have to be used only if the no CD hypothesis cannot be rejected before the application of EPA tests.
The results for the clustered EPA tests are reported in the right block of Table (ref). The kernel robust test $C^{(2)}_{nT}$ performs slightly worse than the overall test $S^{(2)}_{nT}$ and its performance improves rapidly with increases in the number of observations in either dimensions of the panel. In the case of large samples the empirical sizes of the two tests are comparable. Similar to the findings on the overall test, when the distance is misspecified, the size distortion in the test is only slightly higher and its performance gets better with increases of number of observations. The performance of the test based on the partial sample estimator, namely $\underline{C}^{(2)}_{nT}$ is unsatisfactory in small samples, especially for small $n$. For $T=10$, even for the largest $n$ we have, the empirical size of the test equals 19.4%. However its performance improves with $T$, and it reaches 5.5% in the largest sample considered.
The test $C^{(3)}_{nT}$ is a very viable alternative to $C^{(2)}_{nT}$. For small $T$, its size distortion is superior to that of $C^{(2)}_{nT}$ but even for $n=20$ it has good size properties. However, contrary to $C^{(2)}_{nT}$, its performance improves only with $T$. For instance, when $n=10$ and $T=20$, its size equals 9.9% which is better than that of the kernel robust test (10.7%). When $n=100$ and $T=20$ the performance of the latter improves dramatically and reaches 6.3% whereas that of $C^{(3)}_{nT}$ is 8.6%.
Finally we focus on the size of the test using estimated common factors, $\underline{C}_{nT}^{(3)}$. Irrespective of the choice of the number of common factors, when $T$ is small, the test suffers from more size distortions compared to $C^{(2)}_{nT}$ and $C^{(3)}_{nT}$. However for large $T$ and small $n$ the test with $m=2$ has a similar performance to the others. In fact for the smallest $n$ and largest $T$ that we consider it has an empirical size of 6% which is better than that of $C^{(2)}_{nT}$ (9.7%) and $C^{(3)}_{nT}$ (6.9%). When $m=2$ the improvement of the size of the test is much slower with the increase in $T$ and when the number of common factors is chosen by IC, it approaches the non-robust test, as expected.
The size results for DGP2 are reported in Table (ref). As expected, for this DGP the overall tests $S^{(2)}_{nT}$ and $\underline{S}^{(2)}_{nT}$ are grossly over-sized and their performance does not improve with increases in the sample size in any dimension. The test $S^{(3)}_{nT}$ shows very good properties except when $T$ is very small, in particular when $T>30$. Conclusions are similar for the factor-robust tests $\underline{S}_{nT}^{(3)}$. In fact, this test performs better than $S^{(3)}_{nT}$ for all samples sizes considered. A very important finding is that, even when the number of common factors is underspecified, the test performs very well. We also see that the three versions of the test are equivalent in large samples. For $T>10$ and $n>30$ these three tests have equal empirical size.
The findings concerning the clustered EPA tests robust to SCD are similar to those in the case of DGP1 with a few points worth mentioning. The test $C^{(3)}_{nT}$ behaves in line with theoretical expectations such that it has lower size distortions for large $T$ and small $n$ and it performs slightly worse than the overall test $S^{(3)}_{nT}$. The tests based on estimated common factors are found to be oversized in small samples when the number of common factors are chosen by the researcher. The IC based version shows less size distortions. For small $T$ it outperforms $C^{(3)}_{nT}$ overall. Finally, in largest samples the IC based test and the test with $m=2$ have identical size. This is expected because the information criterion $IC_{p1}$ consistently chooses the number of common factors when $n$ and $T$ are large, as seen in Table (ref).
{We conclude this section reporting the following two simulation exercises. In the first, we evaluate the performance of the test statistics $S^{(3)}_{nT}$ and $\tilde{S}^{(3)}_{nT}$. In the second exercise, we study the properties of the $J_n^D$ test statistic of QTZ. We note that the modified test $\tilde{S}^{(3)}_{nT}$ is valid only for one step ahead forecasts. Furthermore, it requires normality under SCD. Table (ref) reports the finite sample properties of $S^{(3)}_{nT}$ and $\tilde{S}^{(3)}_{nT}$ under the failure of normality assumption. As can be seen, the test statistic $\tilde{S}^{(3)}_{nT}$ is perfectly sized irrespective of the sample size even when normality fails, under both DGPs. Turning to the performance of $J_n^D$, we note that, the test is valid only under WCD (DGP1). This is confirmed by the results reported in Table (ref), where the test's size is very close to the nominal size irrespective of the sample size for DGP1, while it is grossly over-sized under DGP2. }
To summarize, in the case of both DGPs the overall EPA hypothesis can be tested with a size close to the nominal value for almost all sample sizes. In particular, it is found that the test $S^{(3)}_{nT}$ has very good properties in both DGPs. {In addition, $\tilde{S}^{(3)}_{nT}$ can be used for one step ahead forecasts with size practically equal to the nominal size, irrespective of the type of CD in the loss differentials.} For DGP1, for small $T$ and large $n$, the kernel robust test is preferred over the test based on the partial sample estimator given that the distance metric is correctly specified. Finally, the test $C^{(3)}_{nT}$ is preferred over others, however, $\underline{C}_{nT}^{(3)}$ has also good properties when the number of common factors is overspecified.
The power results of the tests for DGP1 under the homogeneous alternative hypothesis are given in Table (ref). In the previous subsection, we have seen that the size of the overall EPA tests $S^{(2)}_{nT}$, $\underline{S}^{(2)}_{nT}$ and $S^{(3)}_{nT}$ approach to the nominal level for this DGP. Here, it is seen that the power of the tests $\underline{S}^{(2)}_{nT}$ and $S^{(2)}_{nT}$ converge to 100%, for the latter the distance metric being unimportant. Hence the test is consistent in all cases. For moderate to large $T$, the test $S^{(3)}_{nT}$ is correctly sized. We observe that its power is slightly lower compared to that of $S^{(2)}_{nT}$ in these sample sizes. {For instance, when $T = 100$ and $n = 10$ the power of $S^{(2)}_{nT}$ equals 70.8% whereas those of $S^{(3)}_{nT}$ and $\underline{S}^{(3)}_{nT}$ are 63.5% and 57.2%, respectively.} {Hence,} even though they wrongly assume that there are common factors in the DGP, the power of the factor-robust tests $\underline{S}_{nT}^{(3)}$ are close to that of $S^{(3)}_{nT}$. {To conclude, we see that, the factor robust tests have lower power under WCD, as expected.}
The previous results of the clustered EPA tests showed that in general they are correctly sized only for large $T$. Here we see that their power is only slightly lower than the overall tests. For instance, the power of the asymptotically correctly sized tests $S^{(2)}_{nT}$ and $C^{(2)}_{nT}$ are 19.1% and 17.3%, respectively and their power reaches 100% in the largest sample.
Table (ref) reports the power results of the tests for DGP2 under the homogeneous alternative hypothesis. For this DGP, we have seen that the tests $S^{(2)}_{nT}$ are over-sized even asymptotically. Hence, we focus on the tests $S^{(3)}_{nT}$ and $\underline{S}_{nT}^{(3)}$. It is seen that the power of both tests are very similar for all sample sizes. For instance, for most of the sample sizes that we consider, the power of $C^{(3)}_{nT}$ equals the power of $\underline{C}_{nT}^{(3)}$ when the number of common factors is chosen by the IC.
{The power of the tests $S^{(3)}_{nT}$ and $\tilde{S}^{(3)}_{nT}$ under non-normality are reported in the second panel of Table (ref). As is stated in the previous subsection, $\tilde{S}^{(3)}_{nT}$ improves greatly over $S^{(3)}_{nT}$ in terms of size. We now see $\tilde{S}^{(3)}_{nT}$ has less power for any sample size compared to $S^{(3)}_{nT}$. It has to be noted however that $\tilde{S}^{(3)}_{nT}$ performs still very well and its power equals 100% as sample size increases. }
To save space, we do not report the power results under the heterogeneous alternative hypothesis. The main findings are summarized in Figure (ref) where the power of $S^{(3)}_{nT}$, $C^{(3)}_{nT}$ and {$J_n^D$ are shown under DGP1 so that the assumptions of QTZ are satisfied.} It can be seen that under the homogeneous alternative, the power of our tests $S^{(3)}_{nT}$ and $C^{(3)}_{nT}$ rapidly approach 100%. {Whereas the power of $J_n^D$ is very low compared to our proposed tests, although it increases with sample size.} Under the heterogeneous alternative {on the other hand}, the power of the overall test $S^{(3)}_{nT}$ approaches the nominal size. This is because the overall EPA hypothesis holds under this alternative hypothesis. On the other hand, the expected value of the loss differentials is different from zero for clusters of panel units, hence the clustered EPA test $C^{(3)}_{nT}$ has power under this scenario. {Under this alternative hypothesis, we see that the power of $J_n^D$ approaches zero. Hence, it is inconsistent contrary to our proposed test $C^{(3)}_{nT}$.} {
} The application of the panel EPA tests require some preliminary information on the cross-sectional and temporal dependence properties of the loss differentials. First, the researcher has to determine whether the loss series contain CD, to choose between the non-robust tests $S^{(1)}_{nT}$ and $C^{(1)}_{nT}$, and the other tests which deal with CD. If the data displays CD, one has to have some information on the type of the CD in loss differentials as WCD and SCD may require the use of different tests. Exceptions to this are $S^{(3)}_{nT}$ and $C^{(3)}_{nT}$ which are shown to be performing very well under any type of CD in our simulations. Second, to determine the time series kernel bandwidth parameter, one has to determine whether the loss differentials are autocorrelated or not. This panel EPA testing approach is based on the following three steps.
Step A-- Analysis of CD: Test the no CD hypothesis using a test such as that of breusch80 or the modified and standardized version of it pesaran08. If the no CD hypothesis is not rejected, proceed to Step B.1, otherwise calculate $IC_{p1}$ and proceed to Step B.2.
Step B-- Testing for no autocorrelation: Consider the following empirical model for loss differentials:
and the hypothesis $H^{ac}_0: \pi_1 = \pi_2 = \dots = \pi_p$.
If the no autocorrelation hypothesis is not rejected, set $d_T=1$, otherwise set $d_T>1$. If the no CD hypothesis is not rejected in Step A, proceed to Step C.1, otherwise proceed to Step C.2.
Step C-- Testing for EPA:
In Step A, we suggest two tests of CD. The null hypothesis of BP test is the joint absence of CD between all pairs in the panel. The statistic is distributed as $\chi^2_q$ with $q = n(n-1)/2$ under the null. Hence, the test is more suitable for the cases of fixed and small $n$. The Modified BP statistic is a bias corrected and standardized version of the LM test. It is asymptotically normal under the null of no CD as $n \to \infty$ and is more suitable for large panels. These CD tests can be used to test the hypothesis of no CD in a data set but they do not help to identify the type of the CD. To see if the loss differentials contain common factors, we suggest the information criterion $IC_{p1}$ in (ref). As seen in our simulations (Table (ref)), this IC performs quite well to choose between WCD and SCD.
In Step B, we analyze the serial correlation in loss differentials. The test of the no autocorrelation hypothesis follows the analysis of the CD properties of the loss differentials because the variance computed for the Wald test of the no autocorrelation hypothesis depends on whether the loss series contain CD or not. We suggest using a variance estimator calculated by clustering on the time index if the no CD hypothesis is not rejected in Step A. This estimation corresponds to the driscoll98 variance estimator with a time series kernel bandwidth chosen such that the error terms in (ref) are assumed to be serially uncorrelated. For simplicity, here we do not distinguish between WCD and SCD, although this is possible by considering other variance estimators such as that of kim13. However, as noted by driscoll98, their variance estimator is valid under WCD. One important point in Step B is the determination of the lag length $p$. han17 found that the BIC is inconsistent in panel autoregressions even in the absence of fixed effects. We suggest to use the general-to-specific methodology that they propose. The method starts with $p_{max}$ chosen by the researcher, and it continues by eliminating the biggest insignificant lag until a significant lag is found. If the significance level of these tests is fixed by the researcher, the probability of overestimation of the lag length is nonzero. To avoid this, we determine the significance level as $\alpha_{nT}= \exp\{\ln(0.25) \sqrt{nT}/10\}$, as suggested by han17.
Finally, in Step C we test the EPA hypotheses $H^a_{0}$ or $H^{ca}_{0}$ on the basis of the outcome of the two previous steps. {Of course, pre-testing as documented in this methodology may affect the properties of our proposed tests. This is an interesting issue worth exploring but which goes beyond the scope of this paper.} In the next section, we follow this methodology.
We compare the OECD and IMF GDP growth forecasts using the EPA methodology described above. {Our application can be seen as complementary to the application of QTZ comparing the IMF forecasts to Consensus Economics forecasts.} The data for the IMF forecasts come from their Historical WEO Forecasts Database. The database includes historical $h$-years ahead forecast values, $h=1,2,\dots,5$, for the GDP growth rate and covers up to 192 countries and starts from early 1990's. We collected similar data from the past vintages of the Economic Outlook of the OECD. The Economic Outlook contains only 1-year ahead forecasts. Both organizations publish their forecasts twice a year. In our application we focus on their summer forecasts made for the following year. These forecasts are published in June every year by the OECD whereas IMF forecasts are published in April. Publishing dates are close, hence the forecast errors are comparable. To compute the forecast errors, we use {the GDP growth outturns by the IMF. We also tried using the GDP growth published by the OECD and the results are unchanged (the Pearson correlation coefficient between the two outturn series equal 0.995 hence the difference is negligible).} Eventually we constructed a balanced panel data set of {annual} GDP growth forecast errors of 29 OECD countries from the two organizations between 1998 and 2016.
We use the quadratic loss which is defined as
where, as is throughout this application, first organization is the OECD. This loss function is arguably the most frequently used one {and is robust to measurement errors contrary to other loss functions such as absolute loss hoga22.}
We begin the analysis by the DM tests applied to each country. In the computations of the DM test statistics, we use a bandwidth parameter $d_T=1$ because we have 1-step ahead forecasts. Note that below an autocorrelation test is used to confirm that the loss differentials are serially uncorrelated.\footnote{{It is worth noticing that we use the summer forecasts of the two organizations and they are expected to be autocorrelated by construction. Nevertheless, as the autocorrelation test we use indicates the absence of autocorrelation, its effect must be negligible. Below, for the panel tests we also use a different bandwidth for robustness and this expectation is confirmed.}} The results of the DM tests are given in Table (ref) {where we report average loss differentials, DM test statistics and their $p$-values for each country over the period 1998-2016}.
{First, in terms of the sign of the average loss differentials, a considerable amount of heterogeneity can be observed in the sample. We see that roughly half of the statistics are negative. Second, most of these statistics are statistically insignificant with exceptions being Belgium, Spain, Hungary and Luxembourg. For Belgium which is a country where the predictive ability of the IMF is superior, the EPA hypothesis can be rejected at 10%. For Spain OECD predictions, for Hungary IMF predictions outperform the other. For Luxembourg we can reject the EPA hypothesis at the 10% level and the predictive ability of the IMF is found to be superior for this country as well.}
{Finally, at the bottom right of Table (ref), we report average loss differentials over clusters of countries and the period 1998-2016. The average quadratic loss differential over all 29 OECD countries in the sample is 0.009. This shows that, when we average over all countries, the difference between the {predictive ability} of the two institutions is positive but very small.
An interesting question is whether there are clusters of countries for which the forecast performance of the two institutions differ dramatically. dreher08 test the hypothesis that the forecast performance of the IMF differs with respect to the direct influence of a country on the institution. They use GDP of a country as a proxy of the political influence and find evidence that the forecast bias of the institution declines with GDP. To see if we can find similar evidence in terms of the differences between the bias and efficiency of the forecasts made by the two institutions, we divide our country sample into the G7 and non-G7 countries. The GDP of the G7 countries account for almost 70% of the total GDP of all 29 countries in our sample in 2016, the last year of our data set. Hence, if the OECD's forecast performance does not vary with a country's GDP but the performance of the IMF does, as found by dreher08, we expect to have heterogeneity in average losses of the G7 and non-G7 countries.
The table shows that the average quadratic loss for G7 countries is 0.079. For non-G7 countries this average is -0.014. This shows that for G7 countries the IMF has a superior performance whereas for non-G7 countries the OECD does better in terms of their growth forecasts. Hence, the forecast ability of the two institutions indeed varies with country clusters. Below, we test if these averages are statistically different from zero using our tests.}
Cross-sectional and temporal dependence in loss differentials. As found in our Monte Carlo simulations, the increase in the number of cross-sections increases the power of EPA tests. However, the gain from the usage of panels depends on the degree and the nature of CD. Furthermore, the application of the correct EPA tests require some information on the cross-sectional and temporal dependence of the loss differentials, as we summarized in the previous subsection. Hence, before proceeding to panel tests of EPA, we analyze the CD and autocorrelation properties of our dataset.
The results of BP and Modified BP tests {show that} the null hypothesis of no CD is rejected using any test in conventional significance levels. {The results are omitted to save space but available from the authors.} This means that the tests which allow for CD are more reliable for our data set. Next, $IC_{p1}$ indicates existence of 6 common factors in both loss differential series. Hence, we conclude that both series display SCD. To determine the time series kernel bandwidth parameter, we estimate (ref) with $p=1$ which was chosen by the general-to-specific methodology and check the significance of the common autoregressive parameter by clustering in the time index. This autocorrelation test confirms that the loss differentials are serially uncorrelated, hence, {first} we set $d_T=1$. {To see the impact of the change in the time series bandwidth, we also try $d_T=T^{1/3}$.}
To see the time series profile of the common factors in the loss differential series, we report the plot of the first 6 PCs {of the} loss differential series in panels (a) and (c) of Figure (ref). The PCs are numbered in decreasing order with respect to their eigenvalues. For better interpretation of the common factor estimates, we report a focus on the first three PCs of the quadratic loss differential. Associated factor loadings estimates are reported in Table (ref). To save space, estimated loadings for other PCs are dropped but they are available from the authors upon request. The estimates of the common factors in loss differentials show the effect of the financial crisis. The first {three} common factors in quadratic loss fall in 2009. As can be seen in Table (ref), for 20 countries in the sample, the factor loading estimates are negative for the first common factor. Hence, the OECD, had a superior predictive ability compared to the IMF in this year. The second PC shows a similar pattern whereas the third PC has a movement in the opposite direction in the recovery period. This PC therefore compensates the effect of the first two common factors for some countries.
Panel tests for the EPA hypotheses. Here we apply the panel EPA tests to compare the performance of the two institutions. Following the insights of the Monte Carlo results and the CD analysis, we apply the factor-robust tests $S^{(3)}_{nT}$, $C^{(3)}_{nT}$, $\underline{S}_{nT}^{(3)}$ and $\underline{C}_{nT}^{(3)}$. As a benchmark, we also report the results from the tests assuming no CD, namely $S^{(1)}_{nT}$ and $C^{(1)}_{nT}$.
The results are given in Table (ref). The first part of the table reports the tests of the overall EPA hypothesis. Hence, they are the significance tests of the overall difference reported in Table (ref) which is 0.009. The test statistics for the non-robust test $S^{(1)}_{nT}$ are equal to 0.06 for both bandwidth parameters considered. The conclusion does not change using the robust tests for which the test statistics are slightly weaker. {Our results are in line with those obtained by QTZ on the comparison of IMF forecasts with Consensus Economic forecasts, in the sense that, they do not find a significant difference between the predictive ability of the IMF and Consensus Economics using EPA tests based on the summer forecasts of the two institutions.}
In the second part of the table the results for the clustered EPA tests are reported. The clusters we consider are G7 countries and the non-G7 countries for which the average loss differentials are reported in Table (ref). It is seen that for these clusters, the clustered EPA hypothesis cannot be rejected using any test and any bandwidth parameter. Hence, the predictive ability differences of the two institutions are insignificant.
In a second application, we challenge the quality of the IMF CPI forecasts by comparing them with random walk (RW) forecasts. {In a similar exercise, QTZ compare IMF CPI inflation forecasts with those obtained from the AR(1) model.} In this subsection, we summarize the application and its results. The details can be found in Appendix (ref). The data on the IMF CPI forecasts used in this application come from the Fund's Historical WEO Forecasts Database. The sample contains 127 countries over the period 1991-2019. We restrict our attention to the period considered also by QTZ but we use the data from all countries available, contrary to their sample which covers 86 countries. In this application, in addition to the quadratic loss function, we use the absolute loss function defined as
where $e_{1,it}$ is the forecast error made by the IMF for country $i$ and year $t$, and $e_{2,it}$ is the RW forecast errors calculated using the CPI data published by the IMF. We consider 2 different country clustering schemes. The first one divides the sample of countries into OECD and non-OECD countries, and the second one divides them into G7 countries, non-G7 OECD countries and non-OECD countries. In addition to the full sample analysis, we split the sample into pre- and post-global financial crises periods.
The results show that there is a small but statistically significant difference between the average absolute losses of the IMF and the RW forecasts in favor of the latter in the full sample. When we divide the sample into pre- and post-crises periods, we see that these differences are driven by the differences in the pre-crisis period. For the absolute loss, the clustered EPA hypothesis can be rejected in both pre- and post-crises periods.
{Using quadratic loss, QTZ find that the IMF CPI forecasts are significantly more accurate than the AR(1) forecasts. Our results show that, overall, IMF CPI forecasts are less accurate than the RW forecasts. However, for this loss function, neither overall EPA hypothesis nor clustered EPA hypothesis can be rejected in conventional levels using CD-robust tests.}
In this paper, we proposed novel predictive ability test for panels, corresponding to two different equal predictive ability hypotheses. Several overall EPA tests were developed to evaluate the hypothesis that the predictive ability of two forecasters is equal on average over all periods and units. Consistent clustered EPA tests were also built which are able to test the hypothesis of whether two forecasters have equal predictive power for all clusters of units. Our proposed tests are robust to different forms of cross-sectional dependence in the loss differentials, arising from weak and strong cross-sectional dependence. The proposed tests are found to have appropriate size and power in a set of Monte Carlo simulations. In particular, the overall EPA tests robust to arbitrary cross-sectional dependence are correctly sized. Finally, we provided some useful three-step guideline on how to run the tests, which is then implemented in two applications. We compare the prediction performance of two major organizations, the OECD and the IMF, on their historical economic growth forecasts. In a second application, we evaluate the CPI forecasts of the IMF by comparing them with random walk forecasts. We found evidence of strong cross-sectional dependence in loss differentials of forecast errors in both applications. The results showed that there are only minor differences between the predictive ability of the OECD and the IMF in terms of their economic growth forecasts and they are statistically insignificant. We found some evidence that the RW forecasts of the consumer price inflation are significantly more accurate than those made by the IMF.
We thank participants in the 18\textsuperscript{th} International Workshop on Spatial Econometrics and Statistics (AgroParisTech, Paris, France, 23-24 May 2019), in particular the discussant Paul Elhorst and Davide Fiaschi, the 39\textsuperscript{th} International Symposium on Forecasting (Thessaloniki, Greece, 16-19 June 2019), the 25\textsuperscript{th} International Panel Data Conference (Vilnius University, Vilnius, Lithuania, 4-5 July 2019), the 22\textsuperscript{nd} Dynamic Econometrics Conference (Nuffield College, University of Oxford, Oxford, United Kingdom, 9-10 September 2019), {the 20\textsuperscript{th} International Workshop on Spatial Econometrics and Statistics (Lille, France, 19-20 May 2022), the 27\textsuperscript{th} International Panel Data Conference (Bertinoro, Italy, 16-19 June 2022), and the 16\textsuperscript{th} World Conference of Spatial Econometrics Association (Warsaw, Poland, 23-24 June 2022), in particular the discussant Ioanna Tziolas.} The usual disclaimer applies. This paper is a part of Oguzhan Akgun's PhD research project and it was developed while Giovanni Urga and Zhenlin Yang were visiting the University Paris Panth\'{e}on-Assas - CRED (France) in May and June 2019 and 2022, and during Oguzhan Akgun and Alain Pirotte's several visits at the Centre for Econometric Analysis (CEA) of Bayes Business School (formerly Cass). We thank both institutions for financial support.
\newgeometry{right=1.8cm, left=1.8cm, top=2.7cm, bottom=2.7cm}