Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
61,817 characters · 23 sections · 66 citation commands
Flexible global forecast combinations
Forecast combinations---aggregations of multiple individual forecasts---are one of the most persistently reported empirical successes in forecasting. As a key economic institution, the European Central Bank elicits economic forecasts every quarter for the Eurozone from more than one hundred forecasters, an exercise known as the Survey of Professional Forecasters (SPF). Each forecaster has unique expertise, and some possess private information, so combining is a means to a more accurate and robust projection of the economy than any one forecaster could alone produce. For this reason, the Federal Reserve Bank of Philadelphia runs a similar survey by the same name for the United States. Exactly how to combine forecasts from these surveys is a long-standing problem.
Bates1969 and later Newbold1974 and Granger1984 linearly combined forecasts using variance-minimising weights constrained to sum to one---so-called optimal weights. When the forecasts are unbiased, these weights are optimal in a mean square error sense. In practice, however, they are often beaten by equal weights, a curious phenomenon Stock2004 called the `forecast combination puzzle'. This puzzle, explained theoretically by Claeskens2016, has spurred a formidable research effort to devise improved weighting schemes. Hansen2008 studied weights that minimise Mallow's criterion, which adds a penalty for complexity. To guarantee a convex combination, Conflitti2015 added a restriction to prevent negative weights. Matsypura2018 performed a combinatorial search for the best subset of forecasts to equally weight, and a similar scheme was proposed in Diebold2019 using an $l_1$-norm penalty. To handle highly correlated forecasts, Radchenko2023 allowed negative weights but subjected them to a trimming threshold. For other examples in this line of work, see Yang2004, Aiolfi2006, Capistran2009, Poncela2011, Genre2013, Burgi2017, Kourentzes2019, and Qian2022.
Common to all of the above papers is a focus on using local information to fit the weights, i.e., information that only concerns the forecast target. When just one variable needs forecasting, this approach is sensible. However, it is rare in economics to forecast only a single variable. Instead, forecasts of multiple variables are needed to paint a detailed picture of the economy, core examples being growth, inflation, and unemployment. In addition, policymakers often require forecasts of the economy at different time horizons to facilitate planning. The European Central Bank SPF indeed captures forecasts of multiple variables at multiple horizons, and each variable-horizon pair constitutes an individual forecasting task. Yet, these tasks are not independent; rather, they are highly related. For instance, Okun's law stipulates a strong negative correlation between growth and unemployment Okun1962. The Phillips curve sets forth a similar relationship between unemployment and inflation Phillips1958. It is not unrealistic to expect then that a forecaster's competence in predicting one variable might contain some signal about their competence in predicting another. This possibility motivates us to consider forecast combinations derived from global information shared across related tasks.
The idea of sharing information between prediction tasks emerged during the 1990s in the machine learning community, where it is known as multi-task learning Caruana1997. A vast literature now exists on multi-task learning owing to its success; the interested reader is referred to Zhang2022 for a comprehensive survey. Research in the forecasting community itself has lately trended towards multi-task learning Laptev2017,Salinas2020,Godahewa2021,Montero-Manso2021. In the 2018 M4 Competition, where time series were drawn independently from a large pool, global methods that shared information across forecasting tasks took out the top-three places Makridakis2020. Of these three, the second-place method by Montero-Manso2020 bears some relation to this work. Their method combined forecasts from a handful of classic time series models using weights from gradient boosted trees. The trees were grown on thousands of time series, enabling weights to be learned across tasks. Though similar, their problem is distinct from the economic forecast combination problem that is the main focus of this paper. Whereas Montero-Manso2020 combined a small number of forecasts for a large number of tasks drawn independently from a large pool, we combine a large number of forecasts for a small number of related tasks. Elaborate approaches involving boosted trees are not feasible in our setting.
In light of the preceding discussion, this paper proposes a new framework for globally combining forecasts. Our framework minimises a global loss function comprised of individual forecasting tasks. The framework is flexible to the level of relatedness among the different tasks. Specifically, using a task-coupling penalty, we interpolate between fully local combination, where all tasks are heterogeneous, and fully global combination, where all tasks are homogeneous. The best interpolation is determined in a data-driven fashion. Via this framework, we `globalise' the weighting schemes of Bates1969, Conflitti2015, and Matsypura2018. We then evaluate the new global combinations in both simulation and an application to expert forecasts from the European Central Bank SPF.\footnote{The literature on the European Central Bank and Federal Reserve Bank of Philadelphia SPFs typically refers to individual forecasts as `expert forecasts'; see footnotes 6 and 7 in Magnus2023 for papers that use those surveys. Expert forecasts often include judgement that is now recognised as an important element in forecasting and can be used to adjust individual model output Lawrence2006 or model selection/combination Petropoulos2018. In many areas, `judgemental forecasts' is a more common term; see Lawrence2006. Our methodology is also applicable to those areas.} The results indicate neither fully local nor fully global combination uniformly performs best. Instead, combinations that lie somewhere between these extremes typically lead to the best out-of-sample performance. We also show the benefits of our framework on model-based forecasts of economic and financial time series from the M4 Competition in (ref).
The paper is organised into six sections. Section (ref) introduces the proposed framework for globally combining forecasts. Section (ref) addresses computation of the new combinations. Section (ref) presents numerical experiments that gauge the benefits of globalisation. Section (ref) describes empirical comparisons of the new methods in application. Section (ref) closes the paper. Proofs are in (ref), additional synthetic data experiments in (ref), and additional empirical results in (ref).
To set the scene for our framework, we first describe the traditional single-task forecast combination problem. Let $y\in\mathbb{R}$ be the forecast target and $\bm{f}=(f_1,\ldots,f_p)^\top\in\mathbb{R}^p$ be forecasts of $y$. Denote by $\bm{e}=y\bm{1}-\bm{f}$ the forecast errors. It is customary to assume the errors satisfy $\operatorname{E}(\bm{e})=\bm{0}$ and $\operatorname{Var}(\bm{e})=\bm{\Sigma}$, where $\bm{\Sigma}$ is a $p\times p$ positive-definite matrix. Consider the linear combination forecast $\tilde{f}=\bm{f}^\top\bm{w}$, where $\bm{w}=(w_1,\ldots,w_p)^\top\in\mathbb{R}^p$ are unit sum weights controlling the contribution of individual forecasts to the combination forecast.
Since the forecasts are unbiased and the weights sum to one, the mean square error minimising forecast combination is that which minimises the combination forecast error variance $\operatorname{Var}(\bm{e}^\top\bm{w})=\bm{w}^\top\bm{\Sigma}\bm{w}$. This minimisation is performed with respect to a constraint set $\mathcal{W}$:
The simplest configuration of the constraint set is $\mathcal{W}^\mathrm{eql}=\{\bm{1}/p\}$, yielding equal weights. Using $\mathcal{W}^\mathrm{opt}=\{\bm{w}\in\mathbb{R}^p:\bm{1}^\top\bm{w}=1\}$ leads to optimal weights as proposed by Bates1969. The constraint set $\mathcal{W}^\mathrm{optcvx}=\{\bm{w}\in\mathbb{R}^p:\bm{1}^\top\bm{w}=1,\bm{w}\geq\bm{0}\}$, as studied by Conflitti2015, adds a nonnegativity condition to guarantee a convex combination. The resulting weights are referred to hereafter as optimal convex weights. A more elaborate configuration, $\mathcal{W}^\mathrm{opteql}=\{\bm{w}\in\mathbb{R}^p:\bm{1}^\top\bm{w}=1,\bm{w}=\bm{z}/(\bm{1}^\top\bm{z}),\bm{z}\in\{0,1\}^p\}$, produces equal weights restricted to an optimal subset of forecasts. These weights were investigated by Matsypura2018 and are referred to hereafter as optimal equal weights. Here, $\bm{z}$ is a vector of $p$ binary variables $z_j$ ($j=1,\ldots,p$) where $z_j$ assumes the value one if forecast $j$ is selected for inclusion in the combination and zero otherwise. The constraint $\bm{w}=\bm{z}/(\bm{1}^\top\bm{z})$ guarantees the selected forecasts are equally-weighted. Other weighting schemes can also be cast in this setup by appropriately choosing $\mathcal{W}$.
When the covariance matrix $\bm{\Sigma}$ is large-dimensional and estimated from data, it can be helpful to include a shrinkage penalty in the objective function Roccazzella2022:
where $\lambda\geq0$. Setting $q=2$ yields a ridge penalty Hoerl1970, while $q=1$ yields a lasso penalty Tibshirani1996. When $q=2$, the objective can be rearranged as $\bm{w}^\top(\bm{\Sigma}+\lambda\bm{I})\bm{w}$, so the ridge penalty has the effect of shrinking the covariance matrix towards the identity matrix $\bm{I}$, thereby stabilising the objective. The lasso penalty has a similar stabilising effect. Though there exist numerous covariance estimators that explicitly perform shrinkage Ledoit2004,Schafer2005,Touloumis2015, these do not accommodate missing data. Missing data is an important empirical consideration, discussed further in Section (ref). On the other hand, it is straightforward to mimic the effect of shrinkage by plugging a standard missing-data covariance estimator into (ref). Under all the aforementioned configurations of $\mathcal{W}$, the limiting shrinkage case ($\lambda\to\infty$) leads to equal weights as the optimal solution when $q=2$.
The problem described above concerns one forecasting task $y$. Suppose now we have multiple tasks $\bm{y}=(y^{(1)},\ldots,y^{(m)})^\top\in\mathbb{R}^m$. The $m$ tasks may comprise, e.g., different variables or different forecast horizons. We index all quantities relating to the $k$th component by superscript $(k)$. Hence, the combination forecast $\tilde{\bm{f}}=(\tilde{f}^{(1)},\ldots,\tilde{f}^{(m)})^\top\in\mathbb{R}^m$ has elements $\tilde{f}^{(k)}=\bm{f}^{(k)\top}\bm{w}^{(k)}$, where $\bm{f}^{(k)}=(f_1^{(k)},\ldots,f_p^{(k)})^\top$ and $\bm{w}^{(k)}=(w_1^{(k)},\ldots,w_p^{(k)})^\top$. The errors are $\bm{e}^{(k)}=y^{(k)}\bm{1}-\bm{f}^{(k)}$ with $\operatorname{Var}(\bm{e}^{(k)})=\bm{\Sigma}^{(k)}$.
Though the multi-task setup is typical of economics, research to date has treated the tasks in isolation, using weights fit on a per-task basis:
This combination is local because the individual tasks are in no way linked, i.e., solving optimisation problem (ref) for each task individually leads to the same weights as solving optimisation problem (ref). Information from one task that might be relevant to other tasks is neglected. Instead, one can consider a single vector of weights that is a minimiser of the total loss across all tasks:
This combination is global insofar as the resulting weights take into account information contained in all tasks. Since the loss term in the objective can be expressed equivalently as $\bm{w}^\top(\sum_{k=1}^m\bm{\Sigma}^{(k)})\bm{w}$, this approach can be interpreted as averaging over the task-specific covariance matrices. When the covariance matrices are estimated by the sample covariance matrix, averaging is the same as estimating a single covariance matrix after aggregating data from different tasks. Unfortunately, an implicit assumption underlies this approach that the tasks are completely homogeneous. This assumption might be unreasonably strong in practice and could harm forecast performance.
Rather than committing to a fully local or fully global approach, one can consider bridging the two approaches using per-task weights that are globally regularised:
Here, the penalty $\gamma\sum_{k=1}^m\|\bar{\bm{w}}-\bm{w}^{(k)}\|_q^q$ with $\gamma\geq0$ is a device to incorporate global information into the per-task weights. It achieves this goal by penalising departures from an auxiliary weight vector $\bar{\bm{w}}$ common to all tasks, where the departures are measured as squared deviations ($q=2$) or absolute deviations ($q=1$). Regardless of $q$, taking $\gamma\to\infty$ yields global combination (ref), while taking $\gamma\to0$ yields local combination (ref). Hereafter, we refer to the limiting case $\gamma\to\infty$ as `hard' global combination, and the case with finite nonzero values of $\gamma$ as `soft' global combination. These different cases are depicted in Figure (ref).
The value of $\gamma$ should reflect the level of relatedness among tasks---larger values encourage homogeneity, while smaller values promote heterogeneity. The best value in terms of out-of-sample forecast performance is usually unknown in application but is estimable from data.
The optimisation problem (ref) can be cast solely in terms of the per-task weights $\bm{w}^{(1)},\ldots,\bm{w}^{(m)}$:
where
Penalties like $\Omega_{\gamma,q}$, which penalise departures from a common parameter vector, first appeared in the context of multi-task kernel learning Evgeniou2004,Evgeniou2005.
When the departures are measured as squared deviations (i.e., $q=2$), it is not difficult to obtain an analytical solution:
That is, the optimal value of the common parameter vector $\bar{\bm{w}}$ is the average of the individual parameter vectors $\bm{w}^{(1)},\ldots,\bm{w}^{(m)}$. One can thus interpret our approach as finding per-task weights within a certain distance of the average weight vector. Some additional algebra gives an alternative expression for $\Omega_{\gamma,2}$:
This expression highlights that our approach explicitly penalises mutual distances between local weight vectors. Our experience is that formulating soft global combination using either of the above closed-form solutions yield computational performance similar to that of (ref), provided the number of tasks $m$ is not large. When $m$ is large, these solutions involve many more quadratic terms in the objective, which can impede computation. For instance, under the simulation design of Section (ref) when $m=10$ and $p=50$, it takes roughly six times longer to solve for optimal weights when using the second of the above closed-form solutions.
Form (ref) reveals that $\gamma$ plays a dual role. It shrinks towards equal weights when it appears in front of $\bm{I}$, similar to $\lambda$, but it also pushes towards a corner solution via the last term. While the full explicit solution cannot be derived for $\gamma\neq0$, it is possible to prove the following proposition.
For $\gamma=0$, we get an explicit solution $\bm{w}^{(k)} = {A^{(k)} \bm{1}}/(\bm{1}^\top A^{(k)} \bm{1})$ which is optimal weights of Bates1969 shrunk towards equal weights by $\lambda$. When $\gamma\ne 0$, it helps $\lambda$ with shrinkage, as expected, but also enters in a highly nonlinear way via $B$ and $D$, so the total effect of $\gamma$ is difficult to discern.
Sometimes it can be useful to limit the flow of information between certain tasks, e.g., when one or more tasks are unrelated. For this purpose, denote by $\mathcal{G}:=\{\mathcal{G}_1,\ldots,\mathcal{G}_g\}$ a collection of $g$ groups of tasks, where $\mathcal{G}_l\subseteq\{1,\ldots,m\}$, $\mathcal{G}_1\cup\cdots\cup\mathcal{G}_g=\{1,\ldots,m\}$, and $\mathcal{G}_l\cap\mathcal{G}_k=\emptyset$ for all $l\neq k$. Using this notation, one can modify $\Omega_{\gamma,q}$ to impose the restriction that only tasks within the same group share information:
where $\bar{\bm{w}}^{(l)}$ is an auxiliary weight vector for the $l$th group. When $\mathcal{G}$ consists of just one group, this grouped version of the penalty reduces to (ref). Conversely, when $\mathcal{G}$ consists of $m$ groups, the grouped penalty has no globalisation effect, i.e., it leads to local combination. The grouped version is helpful in our application to the SPF data in Section (ref) where we study different groups of variables and forecast horizons.
If the tasks under consideration vary in difficulty, one or more tasks might dominate the loss component of the objective function. To prevent this behaviour, we consider a scaled version of global combination:
where $\tau_q^{(1)},\ldots,\tau_q^{(m)}>0$ are fixed scaling parameters. If the tasks are to be evenly balanced, a suitable value of $\tau_q^{(k)}$ is the optimal objective value from local combination:
This configuration of $\tau_q^{(k)}$ places all tasks on equal footing, and we use it in all subsequent experiments.
Computation of forecast combinations in our framework varies in complexity according to the weighting scheme, i.e., the specific configuration of $\mathcal{W}$. We begin by describing methods for computation for optimal weights of Bates1969 and optimal convex weights of Conflitti2015, both natural candidates for our framework. The constraint sets $\mathcal{W}^\mathrm{opt}=\{\bm{w}\in\mathbb{R}^p:\bm{1}^\top\bm{w}=1\}$ and $\mathcal{W}^\mathrm{optcvx}=\{\bm{w}\in\mathbb{R}^p:\bm{1}^\top\bm{w}=1,\bm{w}\geq\bm{0}\}$ defining these combinations are convex. All the objective functions described in Section (ref) are convex. The resulting convex optimisation problems are efficiently solvable using most mathematical programming solvers; we use Gurobi gurobi2023.
Optimal equal weights of Matsypura2018 are another natural candidate for our framework. The constraint set defining these weights is less tractable than that for optimal weights or optimal convex weights. Recall the set is defined by a mix of continuous and discrete variables:
The integrality constraint $\bm{z}\in\{0,1\}^p$ is nonconvex but is amenable to a mixed-integer programming solver such as Gurobi. The constraint $\bm{w}=\bm{z}/(\bm{1}^\top\bm{z})$ is also nonconvex but cannot be handled directly by Gurobi. Matsypura2018 used the decomposition $\mathcal{W}^\mathrm{opteql}=\cup_{s=1,\ldots,p}\mathcal{W}_s^\mathrm{opteql}$, where $\mathcal{W}_s^\mathrm{opteql}=\{\bm{w}\in\mathbb{R}^p:\bm{1}^\top\bm{w}=1,\bm{w}=\bm{z}/s,\bm{z}\in\{0,1\}^p\}$ is the set of all vectors that equally weight $s$ forecasts. Since $s$ is fixed for $\mathcal{W}_s^\mathrm{opteql}$, the constraint $\bm{w}=\bm{z}/s$ is linear. The authors sequentially optimise over $\mathcal{W}_1^\mathrm{opteql},\ldots,\mathcal{W}_p^\mathrm{opteql}$ and retain a solution with minimal objective value. This decomposition approach is, however, infeasible in our framework, because different tasks need not combine the same number of forecasts. To this end, we use a new one-step approach which directly optimises over $\mathcal{W}^\mathrm{opteql}$. Though this new approach is proposed for the purpose of globally combining forecasts, it may be of independent interest for local forecast combination. We have found it to be to be uniformly faster than the approach in Matsypura2018 in the single-task setting, sometimes by an order of magnitude.
First, we rewrite the constraint $\bm{w}=\bm{z}/(\bm{1}^\top\bm{z})$ as the pair of constraints $\bm{w}s=\bm{z}$ and $s=\bm{1}^\top\bm{z}$, where $s\in\{1,\ldots,p\}$. The new constraint $\bm{w}s=\bm{z}$ is bilinear in $\bm{w}$ and $s$, meaning it is linear for fixed $\bm{w}$ or fixed $s$. Though this bilinear constraint remains nonconvex, it is amenable to spatial branch-and-bound techniques Liberti2008 which are similar to classic branch-and-bound techniques used for handling integrality constraints. As of version 9, released in 2020, Gurobi can solve optimisation problems with bilinear constraints to global optimality. We now rewrite the constraint set (ref) using the new bilinear constraint representation:
The constraint $s=\bm{1}^\top\bm{z}$ is, in fact, redundant in the above characterisation of $\mathcal{W}^\mathrm{opteql}$ since it is implied by the remaining constraints. Our experience is that Gurobi benefits from excluding it.
We evaluate the possible gains from global forecast combination in simulation. We work directly with the forecast errors which are sampled from a $p$-dimensional Gaussian $\bm{e}_t^{(k)}\sim N(\bm{0},\bm{\Sigma}^{(k)})$ for $t=1,\ldots,T$ and $k=1,\ldots,m$. We fix $p=T=50$, so the number of forecasters is of the same order as the number of samples. Different sample sizes are considered in (ref), though the main findings are robust to sample size. The number of tasks $m\in\{2,5,10\}$. The covariance matrices $\bm{\Sigma}^{(1)},\ldots,\bm{\Sigma}^{(m)}$ are constructed element-wise as $\Sigma_{ij}^{(k)}=\sigma_i^{(k)}\sigma_j^{(k)}\rho^{|i-j|}$. The correlation parameter $\rho=0.75$ to induce high correlations between forecasters, typical of forecaster surveys. For forecaster $j=1,\ldots,p$, the standard deviations $\sigma_j^{(k)}$ are generated by drawing random variables uniformly distributed on $[a,b]$ and correlating them with correlation coefficient $\alpha\in\{0,1/3,2/3,1\}$. The parameter $\alpha$ dictates the level of task-relatedness. As $\alpha$ approaches one, a forecaster's performance on one task is strongly indicative of their performance on other tasks. The converse is true as $\alpha$ approaches zero---a forecaster's performance on one task is weakly indicative of their performance on other tasks. The bounds $a=1$ and $b=3$ so the accuracy of the worst forecaster is up to three times poorer than that of the best forecaster. A visualisation of data from this simulation design is given in (ref).
As a measure of out-of-sample accuracy, we report the mean square forecast error on an infinitely large testing set relative to that from an oracle:
where $\hat{\bm{w}}^{(1)}$ denotes estimated weights for task one fit using an estimate $\hat{\bm{\Sigma}}^{(1)}$ of the true covariance matrix $\bm{\Sigma}^{(1)}$, and $\bm{w}^{(1)}$ denotes oracle weights fit using $\bm{\Sigma}^{(1)}$. We restrict our attention to the relative forecast error of the first task only to measure the marginal effect of adding additional tasks. The covariance matrices are estimated using the sample covariances $\hat{\Sigma}_{ij}^{(k)}=T^{-1}\sum_{t=1}^Te_{it}^{(k)}e_{jt}^{(k)}$ for all $(i,j)\in\{1,\ldots,p\}^2$.
The shrinkage parameter $\lambda$ is swept over a grid of ten values evenly spaced on a logarithmic scale between $0.001$ and $1000$. For every value of $\lambda$, the globalisation parameter $\gamma$ of soft global combination is swept over the same grid. The best values of $\lambda$ and $\gamma$ are chosen on a validation set constructed independently and identically to the training set, which we remark approximates the precision of leave-one-out cross-validation.
The simulations are run parallel in R R2023 with Gurobi given a single core of an AMD Ryzen Threadripper 3970x and a 300 second time limit for each value of $\gamma$ and $\lambda$.
Figure (ref) reports the relative forecast errors from 30 simulations.
The first row of plots is where the estimate $\hat{\bm{w}}$ and oracle $\bm{w}$ are fit under the sum to one constraint that defines optimal weights. The second and third rows correspond to the cases where $\hat{\bm{w}}$ and $\bm{w}$ are fit under the constraints that define optimal convex weights and optimal equal weights, respectively. The relative forecast error reported is not comparable across these three weighting schemes since the oracle is different in each case. Our goal is not to compare weighting schemes but rather to measure the benefits of globalisation. The interested reader is referred to (ref) for forecasts errors reported relative to equal weights---all key findings below remain the same.
Since local combination ignores information in additional tasks, its performance stays fixed as both the number of tasks and task-relatedness increase. In contrast, the relative forecast error of hard global combination decreases roughly linearly with task-relatedness, providing for substantial improvements when task-relatedness is high. Yet, when task-relatedness is low, hard global combination can underperform relative to local combination. This poor performance is made worse by adding additional tasks.
Soft global combination ameliorates the poor performance of hard global combination when the tasks are unrelated and nearly performs as well as hard global combination when the tasks are identical. There is, of course, a statistical cost to estimating the best level of globalisation. Between the extremes, soft global combination successfully adapts to the level of task-relatedness to improve over both local and hard global combination. The greater the number of tasks, the greater the possibility for improvement.
Among the three weighting schemes, optimal weights benefit most from globalisation. The constraint set that defines optimal weights is unbounded, and thus its relative forecast error can be arbitrarily bad. Optimal convex weights and optimal equal weights are defined by bounded constraint sets, so there exist finite upper bounds on their relative forecast errors. Thus, the opportunity to improve these weights is somewhat less than for optimal weights, yet often still substantial.
In (ref), we provide additional results and extended discussion for $T\in\{25,100,150\}$. For shorter series ($T=25$), soft global combination performs well even when the tasks are unrelated and significantly improves when they are related. Even though the justification is different for longer series, the conclusion is the same: soft global combination is preferred. It gets the best of both worlds regardless of whether the tasks are related.
The soft global combination results in this section correspond to the globalisation penalty configured with squared deviations ($q=2$). Further comparisons in (ref) indicate no material improvement by the absolute deviation penalty ($q=1$), so we restrict our attention to squared deviations hereafter.
The findings from these experiments suggest several recommendations for practitioners. First, consider globalising any forecast combinations when tackling multiple forecasting tasks. The potential gains from globalisation can be significant, even for moderate levels of task-relatedness. Second, unless domain knowledge indicates the tasks are unrelated or strongly related, use soft global combination with cross-validation. Soft global combination with $\gamma$ cross-validated is reasonably robust to task-relatedness, while the downside of applying hard global (local) combination to unrelated (strongly related) tasks is large. Last, when using optimal weights, employ global combination whenever possible since that weighting scheme benefits most from globalisation. The benefits persist even when globalising in tandem with shrinkage.
The European Central Bank SPF is an ongoing survey eliciting predictions for rates of growth, inflation, and unemployment from forecasters for the Eurozone. The survey has been conducted quarterly since 1999 Q1. In each round, the survey participants are asked to provide predictions of the three variables at several time horizons. We focus on the two rolling horizons in this paper, which are one and two years ahead of the latest available observation of the respective variable. For instance, in the 1999 Q1 survey, one-year forecasts corresponded to 1999 Q3 for growth, December 1999 for inflation, and November 1999 for unemployment.\footnote{To simplify exposition, forecasts of inflation and unemployment are referred to by the quarter they belong to, e.g., December 1999 inflation and November 1999 unemployment are called forecasts of 1999 Q4.} The total number of forecasting tasks $m=6$.
The SPF data is publicly available at the European Central Bank Statistical Data Warehouse (SDW). Actual values of inflation and unemployment are also available at the SDW. Actual values of growth are available from Eurostat. We access data at the SDW using the R package ecb Persson2022, and data from Eurostat using the R package eurostat Lahti2017. The data used in this paper was retrieved on 17 April 2022. After merging the forecasts and actual values, between $T=85$ and $T=90$ observations are available. The first observations are 1999 Q3 (one-year growth), 1999 Q4 (one-year inflation and unemployment), 2000 Q3 (two-year growth), and 2000 Q4 (two-year inflation and unemployment). The last observation is 2021 Q4.
A notable feature of the SPF is that forecasters enter and exit the survey at different times. This aspect of the survey, coupled with periodic nonresponse, gives rise to a sizeable portion of missing data. To deal with this issue, we follow previous works Matsypura2018,Radchenko2023 and filter the data to only include forecasters who respond for a reasonable number of periods. Specifically, the forecasters who provide a minimum of 40 forecasts (10 years) for every task over the full training set (1999 Q3 to 2019 Q4) are retained. This filtering criterion leads to a dataset comprising $p=34$ forecasters. Figure (ref) plots the filtered forecasts alongside actual values of the forecast targets.
To handle missing values that remain after filtering, the covariance matrices of forecast errors are estimated using all complete pairs of observations: $\hat{\Sigma}_{ij}^{(k)}=|\mathcal{T}_i^{(k)}\cap\mathcal{T}_j^{(k)}|^{-1}\sum_{t\in\mathcal{T}_i^{(k)}\cap\mathcal{T}_j^{(k)}}e_{it}^{(k)}e_{jt}^{(k)}$ for all $(i,j)\in\{1,\ldots,p\}^2$. Here, $\mathcal{T}_i^{(k)}$ denotes the periods in the training set where forecaster $i$ provided a forecast for task $k$. Covariance matrices constructed in this manner are not guaranteed positive-definite. For this reason, we take the positive-definite matrix nearest to $\hat{\bm{\Sigma}}^{(k)}$ using nearPD from the R package Matrix Bates2022. The forecast errors are standardised by the standard deviation of the forecast targets as estimated on the training set prior to estimating the covariance matrices.
The first set of experiments study the evolution of out-of-sample forecast performance as the globalisation parameter $\gamma$ is swept over its support (the `globalisation path'). Here, we take 30 values of $\gamma$ logarithmically spaced between $0.001$ and $1000$. As a measure of out-of-sample accuracy, we report the mean square forecast error on a testing set relative to that from local combination:
where, for a given weighting scheme, $\tilde{f}_{t+h\mid t}^{(k)(\gamma)}$ is a global combination forecast of task $k$ at time $t+h$ produced using a training set up to time $t$ with $\gamma\in[0,\infty)$, and $\underaccent{\bar}{T}$ and $\bar{T}$ are the first and last periods in the testing set. The denominator is the mean square forecast error from setting $\gamma=0$, so this metric is the percentage improvement due to globalisation. We pick $\underaccent{\bar}{T}$ and $\bar{T}$ so the testing set is the last five years to 2019 Q4. The period after 2019 Q4, covering the COVID-19 recession and 2021--2022 inflation surge, is considered in separate experiments in Section (ref).
Figures (ref), (ref), and (ref) report the globalisation paths of optimal weights, optimal convex weights, and optimal equal weights for fixed shrinkage parameter $\lambda=0.1$.
The globalisation paths of optimal weights are smooth because the fitted weights are a smooth function of $\gamma$ as Proposition (ref) implies, while those of optimal convex weights and optimal equal weights are nonsmooth. In the case of optimal convex weights, the convexity constraint makes the fitted weights nonsmooth in $\gamma$ when it is binding. The path for optimal equal weights is a step function in $\gamma$ due to the weights being discrete. Three ways of grouping the tasks are considered: grouping variable tasks (group 1: one-year growth, inflation, and unemployment; group 2: two-year growth, inflation, and unemployment); grouping forecast horizon tasks (group 1: one- and two-year growth; group 2: one- and two-year inflation; group 3: one- and two-year unemployment); and grouping all tasks (group 1: one- and two-year growth, inflation, and unemployment). The reader is reminded information flows only between tasks belonging to the same group.
Across all weighting schemes and tasks, there is always a globalisation path that attains its minimum at some positive amount of globalisation. The limiting case $\gamma\to\infty$, hard global combination, is sometimes helpful and sometimes harmful. For instance, growth and inflation realise roughly 15% improvement from hard global combination (optimal weights, grouped variables) at the two-year horizon while unemployment deteriorates by about 40% at the same horizon. This behaviour might be attributable to growth and inflation being difficult tasks at the two-year horizon (e.g., expert forecasts of those tasks are not responsive to the COVID effects in 2020 and 2021 as Figure (ref) shows), thus providing a noisy signal to unemployment. However, even in the cases where hard global combination on its own is not useful (such as one- and two-year unemployment forecasts), the optimal choice of $\gamma$ is still positive, and soft global combination can extract benefits.
The results lead us to the following practical suggestions regarding the groupings. For a one-year growth forecast, using all available information (i.e., the `grouped all' version) is beneficial as it is the best or close to the best performer across the different weights. For the same reason, we also recommend this grouping for two-year inflation and two-year unemployment forecasts. For one-year unemployment, one should group variables as this grouping is the best or close to the best across the different weights. For one-year inflation, grouped horizons deliver stable improvement across different weights (even though grouping variables works best for optimal weights). Finally, for a two-year growth forecast, we recommend grouping horizons but avoiding optimal weights. The convexity of the weights seems to be critical to avoid instabilities of negative weights, which was recently documented by Radchenko2023.
If one is working with only a single forecasting horizon, the selection of grouping becomes redundant. Also, in other applications, grouping all tasks seems a sensible default provided $\gamma$ is chosen judiciously on a task-by-task basis. This default option can be improved by using additional cross-validation to help determine which grouping performs best.
The second set of experiments are broader comparisons that acknowledge the level of globalisation requires tuning in practice. For this purpose, we use leave-one-out cross-validation---a valid procedure provided the combination forecast errors are uncorrelated Bergmeir2018. The value of $\gamma$ is tuned over ten values logarithmically spaced between $0.001$ and $1000$ on a per-task basis, so different tasks need not use the same value. To allow for comparisons of forecast accuracy across weighting schemes, we report the mean square forecast error relative to that from equal weights, a common benchmark in practice:
where $\tilde{f}_{t+h\mid t}^{(k)}$ is an arbitrary combination forecast and $\bar{f}_{t+h\mid t}^{(k)}$ is the equally-weighted combination forecast. Values of this metric less than one indicate superior performance to equal weights.
Table (ref) reports the average value of the performance metric across the six tasks, with the minimal and maximal values among the tasks in brackets. The shrinkage parameter $\lambda=0.1$. We study tuned $\lambda$ next.
The last five years of the data is again studied, but we now include the period 2020 Q1 to 2021 Q4 to evaluate recent performance during the COVID-19 recession and 2021--2022 inflation surge. Figure (ref) highlights how the quarters on and after 2020 Q1 contain several outliers. To prevent these outliers dominating the performance metric, the testing set is split before and after 2020 Q1. Likewise, to avoid the outliers contaminating the estimated covariance matrices and thus the estimated weights, the training set is stopped at 2019 Q4.
With few exceptions, soft global combination improves on local combination. The improvements are generally greatest pre-2020. The more minor improvements post-2020 are possibly a consequence of the recent period of deteriorated economic conditions during which task-relatedness could be less stable. In some instances, hard global combination outperforms both soft global combination and local combination. However, as in the previous section, it also sometimes underperforms. On the other hand, the data-driven determination of the globalisation level for soft global combination produces good combinations that consistently forecast well.
Optimal weights realise the most significant gains from globalisation among the three weighting schemes---soft global combination (grouped all) places first in terms of average performance across tasks (pre-2020) compared with local combination, which places last. Moreover, globalisation leads to smaller maximal loss for optimal weights. Though not always beating optimal weights according to average performance, optimal convex weights and optimal equal weights have more consistent performance across tasks, especially pre-2020. With a suitable amount of globalisation, each weighting scheme can beat the notoriously difficult benchmark of equal weights for one or more task groupings.
The results of Table (ref) are from tuning the globalisation parameter $\gamma$ while holding the shrinkage parameter $\lambda$ fixed. It is insightful to evaluate whether there are further benefits from tuning $\lambda$ in addition to $\gamma$. To this end, we cross-validate both parameters here. We focus on optimal (convex) weights to keep computation time reasonably low. Table (ref) reports the results.
Optimal weights witness an improvement across the board relative to the results of Table (ref) (for local, hard global, and soft global combinations). Though it is known Roccazzella2022 that optimal weights benefit from (carefully tuned) shrinkage, our result is the first documentation of similar behaviour for global combination. The results for optimal convex weights---whose nonnegativity constraint already imparts a form of shrinkage---are similar to Table (ref). Our core finding remains the same in both cases: globalisation via soft global combination is typically beneficial.
In more than 50 years of forecast combination literature spanning a myriad of weighting schemes, `forecasters still have little guidance on how to solve the forecast combination puzzle' Wang2023. Our empirical results show that before the COVID-19 recession (2017--2019), soft global combination not only improves upon local and hard global combinations but also upon equal weights. Our synthetic experiments also demonstrate improvement in stable conditions; see (ref). However, results post-2019 are mixed with around half cases where soft global combination performs better than equal weights.
One possible explanation is a structural break produced by the COVID-19 recession. Rossi2001 showed that the 2007--2008 global financial crisis (GFC) affected forecasting performance significantly. Currently, no similar research is available for the COVID period. However, Figure (ref) leaves no doubt that for inflation and growth, the effects of COVID far exceed those observed during the GFC. Simple tests for a structural break in Table (ref) support this claim.
The variance of the average forecast error is significantly different post-COVID at the 1% level (except for the unemployment forecasts). As there is little data available post-COVID, reestimated weights are likely to have large variability that negates the benefits of soft global combination. If weights from the pre-COVID period are used, they will not necessarily be optimal and may not provide the benefits observed under stable conditions. Wang2023 recommend equal weights in such cases. Until more data is available, equal weights are probably suitable for the post-COVID period. With more post-COVID data, soft global combination should quickly catch up as a strong competitor and a potential solution to the forecast combination puzzle; see also Frazier2023.
The benefits of equal weights centre around the substantial reduction of the variance at the cost of introducing a small bias; see Claeskens2016. A more recent approach by Blanc2020 requires an explicit solution to analyse the bias-variance trade-off. With the absence of an explicit solution in our case, a practitioner needs to empirically validate whether soft global combination beats the equally-weighted combination weights in their setting. Our findings suggest the likelihood of improving over the equal weights is high.
To date, the problem of combining economic forecasts has been handled on a per-task basis, with the combination for each variable and forecast horizon learned independently of other variables and horizons. When the forecasting tasks are related, as economic theory and evidence suggest, this approach of learning the combinations using only local information is potentially suboptimal. This paper investigates the value of a global approach, where task-relatedness is directly exploited to improve the quality of combinations. At the heart of our approach is a principled framework that accounts for the level of homogeneity across tasks by flexibly interpolating between fully local and fully global combinations. In addition to unifying local and global approaches under one umbrella, the new framework accommodates many existing weighting schemes. Empirical evidence from the European Central Bank SPF suggests combinations of expert forecasts for rates of growth, inflation, and unemployment in the Eurozone benefit from some degree of globalisation, as do combinations of these same variables across one- and two-year horizons. Further empirical evidence on economic and financial data from the M4 Competition in (ref) indicates similar benefits for combinations of model-based forecasts.
Our approach is not limited to point forecasts and can be extended to probabilistic forecasts. Consider, e.g., the optimal weights of Hall2007 and Geweke2011 that, in the case of only one task, maximise the log-score when combining $p$ individual predictive densities $p_{j} (\cdot; \bm{\theta}_j)$ using the historical observations $y_1,\dots,y_T$:
These weights readily extend to soft global combination for $m$ tasks:
The density parameter $\bm{\theta}_{j}^{(k)}$ can be different across tasks. This problem can be further enhanced using a shrinkage penalty or additional constraints, e.g., the high moment constraints of Pauwels2023.
Furthermore, our approach is based on the intuitive idea that a forecaster’s competence in predicting one variable might contain some signal about their competence in predicting another. One can examine the connection between the accuracy of the combined forecast and individual forecaster characteristics. We leave this direction for future research.
An R implementation of the global forecast combinations in this paper is publicly available at
We are especially grateful to the editor, associate editor, and two referees for their thoughtful, constructive, and encouraging reviews that greatly improved the manuscript. We are also grateful for valuable comments from the participants of the 43rd International Symposium on Forecasting, 6th International Conference on Econometrics and Statistics, and 2023 Econometric Society Australasian Meeting. Ryan Thompson acknowledges financial support by an Australian Government Research Training Program (RTP) Scholarship through Monash University, where he worked on the first draft of this paper as a PhD student.
The authors declare no conflict of interest.