Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
93,456 characters · 30 sections · 99 citation commands
A Kernel Score Perspective on Forecast Disagreement and the Linear Pool
Forecast combination is a widely popular method that applies to various types of forecasts, including point forecasts and forecast distributions. Empirically, combinations have repeatedly been found to perform well relative to using any individual forecasting method WangEtAl2023.
Combinations of forecast distributions often take an appealingly simple linear form proposed by Stone1961. This form is also known as the linear (prediction) pool. A strand of literature including genest1986combining, HallMitchell2007, GewekeAmisano2011, GneitingRanjan2013, LichtendahlEtAl2013, BerrischZiel2023 and Taylor2025 analyzes the linear pool, and compares it to other combination methods such as quantile-based approaches.
The present paper contributes to this literature by studying the linear pool for kernel scores Dawid2007,GneitingRaftery2007, a rich class of proper scoring rules that covers point or distribution forecast of discrete or continuous, univariate or multivariate outcomes. Table (ref) lists some prominent examples, including squared error for point forecasts and the Continuous Ranked Probability Score MathesonWinkler1976 for distributions. Our analysis shows that several well-known results on linear pooling under squared error generalize to all kernel scores. This is potentially surprising since kernel scores cover settings that seem distinct from an applied perspective.
Our results can be grouped into two categories. The first group describes mechanical properties of forecast combination under any kernel score. In particular, Proposition (ref) shows that the linear pool maximizes centrality among the $n$ component distributions when distance is measured in terms of kernel scores. This motivation distinguishes the linear pool from other forms of combination such as quantile averaging. We further argue that the linear pool's expected score (i.e., its entropy) can usefully be decomposed as the sum of two components capturing (i) the average entropy of the component distributions, and (ii) disagreement between the component distributions. This disagreement term coincides with the performance difference between the pool and the component distributions (see Proposition (ref)). For kernel scores, disagreement hence quantifies the gains from forecast combination.
Our second group of results derives implications for the performance of linear pools. In Proposition (ref), we present sufficient conditions for optimality of equal weights, nesting existing conditions for point forecasts. Motivated by the specific structure of kernel scores, we consider dynamic weights that reflect information on forecast disagreement, instead of being constant over time (Remark (ref)). We further show that a linear pool of calibrated componenents is underconfident, with the amount of underconfidence depending on forecast disagreement (Proposition (ref)).
In macroeconomics and finance, many studies consider notions of disagreement to either measure economic conditions (see ClarkMertens2024 and the references therein), or to test economic theories (see e.g. CoibionEtAl2012). As a side effect, the results of this paper suggest principled measures of forecast disagreement for various applied settings. In particular, we cover the case of disagreement among probability distributions (rather than point forecasts) recently studied by CumingsEtAl2021 and MitchellEtAl2024. In Section (ref), we illustrate a new measure of disagreement among bivariate forecast distributions that are based on either surveys or a Bayesian vector autoregressive model.
The present paper complements recent independent work by AllenEtAl2024 on linear pooling for kernel scores. They consider computational aspects (see their Proposition 2), a linear pool variant based on order statistics (their Section 4.2) and empirical analysis of weather forecasts (their Section 5). By contrast, our focus is on generalizing forecast combination results obtained under squared error loss, and the themes we study -- in particular, forecast disagreement -- are motivated by economic applications.
The paper is structured as follows: Section (ref) describes our formal setup. Section (ref) presents a disagreement-based motivation of the linear pool. Section (ref) presents a decomposition of the linear pool's entropy. Section (ref) relates the linear pool's ex-post performance to its disagreement component. Section (ref) studies statistical implications of the aforementioned results. Section (ref) presents a Gaussian example. Section (ref) provides empirical illustrations using bivariate forecast distributions for inflation and univariate forecast distributions for exchange rates. Section (ref) concludes with a discussion. The appendix contains specific formulas for various scoring rules, proofs, computational details and additional figures. R code accompanying the paper is available at \url{https://gitlab.kit.edu/fabian.krueger/kernel_pool_replication}.
A scoring rule $S: \mathcal{F} \times \Omega \rightarrow \mathbb{R} \cup \{\infty\}$ assigns a numerical score, given a forecast distribution $F \in \mathcal{F}$ and an outcome $y \in \Omega$. We use scoring rules in negative orientation, such that a smaller score indicates a better forecast. Throughout this paper, we assume that the class $\mathcal{F}$ of distributions is convex, such that it contains any linear combination $w F_1 + (1-w)F_2$ of two members $F_1, F_2$, where $0 \le w \le 1$.
Suppose that the predictand is distributed according to $F \in \mathcal{F}$. Then the expected score when stating the forecast $H$ is given by $$\mathbb{E}_F[S(H, X)] = \int_\Omega S(H, x)~\text{d}F(x),$$ where $\mathbb{E}_F$ denotes expectation with respect to a random variable (in this case, $X$) that is distributed according to $F$. Throughout much of this paper, we will treat distributions such as $F$ and $H$ as well as combination weights $w_1, w_2, \ldots, w_n$ as fixed, non-random objects. Exceptions occur in Sections (ref) and (ref), where we consider a `prediction space' framework in which forecasts and outcomes follow a joint distribution.
The score divergence $d_S(H,F)$ measures the difference in expected scores when stating $H$, as opposed to the actual distribution $F$. It is given by
A proper scoring rule $S$ satisfies $d_S(H, F) \ge 0$ for all $F, H \in \mathcal{F}$. Proper scoring rules thus incentivize truthful and accurate forecasting: A forecaster whose beliefs are represented by $F$ cannot do (strictly) better by reporting a forecast distribution other than $F$. ThorarinsdottirEtAl2013 argue that divergence functions $d_S(H,F)$ based on proper scoring rules are useful for measuring the discrepancy between two distributions $F$ and $H$ in applications. Below we will argue that divergence functions based on kernel scores, a subfamily of proper scores, are particularly useful due to their symmetry property that $d_S(H,F) = d_S(F,H)$.
The entropy $\mathbb{E}_F[S(F,X)]$ represents a forecaster's expected score when reporting $F$. Given that $F$ is an optimal forecast, this is the best (i.e., smallest) expected score that can be attained. Entropy hence measures the uncertainty implicit in $F$. In the following, we will occasionally use the shorter notation $\text{ent}(F) := \mathbb{E}_F[S(F,X)]$ to denote the entropy of $F$.
We next describe kernel scores, a rich family of scoring rules proposed by GneitingRaftery2007 and Dawid2007 that relates to statistical concepts of energy SzekelyRizzo2017 and to kernel methods in machine learning AllenEtAl2024. Appealingly, kernel scores can accommodate very general (Hausdorff) outcome spaces $\Omega$, containing e.g. univariate or multivariate, discrete or continuous outcome variables.
Our setup mostly follows Gneiting2012. Let $L: \Omega \times \Omega \rightarrow [0, \infty)$ be a nonnegative function that is symmetric in its two arguments, with $L(z, z) = 0$ for all $z \in \Omega$, and the property that $$\sum_{i=1}^n\sum_{j=1}^n c_ic_j L(x_i, x_j) \le 0$$ for all $n \in \mathbb{N}, x_1, x_2, \ldots, x_n \in \Omega$ and $c_1, c_2, \ldots, c_n \in \mathbb{R}$ such that $\sum_{i=1}^n c_i = 0$. The function $L$ is called a negative definite kernel. Based on $L$, one can construct the scoring rule
where $X$ and $\widetilde{X}$ are understood to be two independent draws from $F$. Scoring rules from the family in ((ref)) are proper with respect to the class $\mathcal{F}$ of Radon probability measures on $\Omega$ for which the expectation $\mathbb{E}_F[L(X, \widetilde X)]$ is finite Gneiting2012. Under these conditions, kernel scores satisfy $S_L(F, y) \ge 0$ for all $F \in \mathcal{F}$ and $y \in \Omega$, with $S_L(F,y) = 0$ if $F = F_\delta$ has point mass at $y$ Gneiting2012.
As detailed in Appendix (ref), the family at ((ref)) includes popular scoring rules such as the squared error, Brier score, Continuous Ranked Probability Score (CRPS) or Energy Score, which correpond to specific choices of $\Omega$ and $L$. For a scoring rule $S_L$ as in ((ref)), the entropy function is given by
The special case where $F = F_\delta$ places point mass on a single element $y \in \Omega$ results in the minimal entropy of zero.
For a kernel scoring rule, the divergence between two distributions $H, F$ is given by
here $E_{F, H}[L(X, \widetilde{X})]$ indicates that the expected value is with respect to two independent random variables $X \sim F$ and $\widetilde{X} \sim H$. Importantly, this divergence function is symmetric with respect to the two distributions $F$ and $H$.
While rich, the family of kernel scores at ((ref)) does not include all proper scoring rules that are popular in practice. In particular, the tick (or pinball, or piecewise linear) loss function underlying quantile regression KoenkerBassett1978 features an asymmetric divergence function and is thus not a kernel score. Interestingly, the divergence function of the quantile score is asymmetric even in the case of the median, so that absolute error is not a kernel score.\footnote{To see this, consider an example where $F$ is a standard normal distribution and $H$ is a normal distribution with mean $1$ and variance $2$. Then $d_S(H,F) = \int_{\Omega} \left(|x-1|-|x|\right) \text{d}F(x) \neq d_S(F,H) = \int_{\Omega} \left(|x|-|x-1|\right) \text{d}H(x),$ i.e. the divergence function is asymmetric. BentzienFriederichs2014 provide an expression of the divergence function for general quantile levels, which is also asymmetric in general.} Furthermore, the logarithmic score is not a kernel score AllenEtAl2023. This can again be deduced from the fact that its (Kullback-Leibler) divergence function is asymmetric.
Our definition of kernel scores at ((ref)) in terms of a negative definite kernel $L$ differs from AllenEtAl2024's definition in terms of a positive definite kernel (see their Equation 4). Both definitions can yield equivalent expressions of a given kernel score like the CRPS, but the respective kernel functions differ. Using their definition, empirical weight optimization problems directly takes the form of a convex quadratic program (see their Proposition 2), which is convenient for software implementation. At the same time, our definition yields more easily interpretable connections to forecast disagreement, a key theme of this paper. In Appendix (ref), we show how to translate our weight optimization problem into a convex quadratic format.
This section uses the notion of divergence to motivate linear pooling, as opposed to other forms of forecast combination. Here we initially consider a finite outcome space $\Omega$, with $|\Omega| = n_\Omega$. The elements of $\Omega$ could be univariate or multivariate, quantitative or categorical. As noted by AllenEtAl2024, a finite outcome space considerably simplifies the expected value expressions relevant for kernel scores, and aligns well with the fact that many forecasting models (such as meteorological ensembles or models estimated via Bayesian techniques) take the form of simulated or empirically observed samples. Therefore, and since $n_\Omega$ can be arbitrarily large, the assumption of a finite outcome space is not very restrictive from an applied perspective.
We identify a forecast distribution $F$ with an $n_\Omega \times 1$ vector $p$ containing predicted probabilities of all outcomes. Accordingly, the family $\mathcal{F}$ of forecast distributions consists of the $n_\Omega$-dimensional probability simplex $\text{PS}^{n_\Omega}$, i.e. the set of all $n_\Omega$ dimensional vectors whose elements are nonnegative and sum to one. Let $F_w = \sum_{i=1}^n w_i F_i$ be a linear pool of $n$ forecast distributions $(F_i)_{i=1}^n$, with $F_i \in \mathcal{F}$ and weight $w_i$ placed on the $i$th component, such that $w = (w_1, w_2, \ldots, w_n)' \in \text{PS}^n$. Clearly, $F_w \in \mathcal{F}$.
We further define the matrix $\underline{L}$ whose $[j,l]$ element is given by $L(x_{(j)},x_{(l)})$, where $x_{(j)}$ and $x_{(l)}$ are the $j$th and $l$th unique elements of $\Omega$. In this setup, we have
where $h$ is the $n_\Omega$-vector of probabilities corresponding to some forecast distribution $H$. Based on ((ref)) and ((ref)), the average divergence between the component distributions $F_1, F_2, \ldots, F_n$ and $H$ is given by
where the vectors $p_1, \ldots, p_n$ correspond to the component distributions, and the notation $D_\text{gen}(h)$ indicates disagreement around a generic vector $h$ of probabilities.
We next ask which vector $h$ of probabilities minimizes $D_\text{gen}(h)$. In the context of proper (but not necessarily kernel) scoring rules, NeymanRoughgarden2023 call this minimizer the `quasi-arithmetic pool'. The latter is the most central choice of probabilities, a desirable feature if the combination aims to represent a consensus of the individual forecasts $F_1, F_2, \ldots, F_n$. If $S_L$ is the Brier score and the predictand is univariate and categorical, the quasi-arithmetic pool is known to coincide with the linear pool (Pettigrew2019; NeymanRoughgarden2023). The following result shows that the equivalence between the quasi-arithmetic pool and the linear pool generalizes to all kernel scores.
In the special case where $S_L$ is squared error loss and the predictand is univariate and quantitative, Proposition (ref) recovers the textbook result that the arithmetic mean minimizes the sum of squared errors.\footnote{Specifically, suppose that the $i$th component distribution has point mass at $x_i \in \Omega$, such that its cumulative distribution function is given by $F_i(z) = \mathbf{1}(z \ge x_i).$ Then the score divergence between $F_w$ and $F_i$ is given by $(\mu_w - x_i)^2$, with $\mu_w = \sum_{i=1}^n w_i x_i$ being the mean implied by $F_w$.} Interestingly, the proposition also covers multivariate quantitative outcomes for which several kernel scores $S_L$ are available (most popularly, the Energy Score). The equivalence between the quasi-arithmetic and linear pools need not hold for proper scoring rules $S$ that are not kernel scores. In particular, if $S$ is the logarithmic scoring rule, the quasi-arithmetic pool is given by the logarithmic pool NeymanRoughgarden2023. However, the latter result hinges on the order of the two arguments of the relevant (Kullback-Leibler) divergence function. When reversing the order of the arguments, the linear pool minimizes the average Kullback-Leibler divergence to the component distributions (Abbas2009, Proposition 1; Pettigrew2019, Proposition 3). This type of sensitivity is a drawback of using an asymmetric divergence function. Conversely, the symmetry of a kernel score's divergence function is appealing.
We now consider linear pooling for a generic (not necessarily discrete) outcome space $\Omega$. As before, a linear pool is given by $F_w = \sum_{i=1}^n w_i F_i$, with $F_i \in \mathcal{F}$ and a weight vector $w \in \text{PS}^n$.
Let $S$ be a scoring rule, and let $\mathcal{F}$ be such that $S$ is proper with respect to $\mathcal{F}$. Then the entropy of $F_w$ can be written as
where $D \ge 0$ since the weights $w_i$ are nonnegative and $S$ is proper.
We demonstrate in Appendix (ref) that in the case of squared error, Equation ((ref)) recovers the famous decomposition of the linear pool's variance as discussed by wallis2005 and others. Furthermore, ShojaSoofi2017 derive the decomposition at ((ref)) for the case of the logarithmic score. In this case, $S(F, y) = -\log f(y),$ where $f$ is the density associated with $F$, and the corresponding divergence function is known as Kullback-Leibler divergence. Apart from these two special cases for $S$, we are not aware that the decomposition at ((ref)) has appeared in the literature. The decomposition is consistent with the better known inequality
which holds for any proper scoring rule $S$ GneitingRaftery2007.\footnote{ GneitingRaftery2007 define scoring rules in positive orientation, whereas we define them in negative orientation. Hence the word `convex' in their statement (on p. 362) that `a regular scoring rule $S$ is proper [..] if and only if the expected score function [..] is convex [..]' must be replaced by `concave' in our setting.} Equation ((ref)) shows that the difference between the left and right sides of ((ref)) is given by $D$. The following result describes some properties of the decomposition at ((ref)) for the family of kernel scores.
The properties of kernel scores as noted in part (a) are appealing for interpreting the pool's entropy. In particular, the symmetry property is useful since the order of the arguments $F_w$ and $F_i$ to the divergence function $d$ seems arbitrary. WaghmareZiegel2025 show that under an additional technical assumption, kernel scores are the only proper scoring rules with a symmetric divergence function. Nonnegativity of entropy is intuitively appealing if one aims to interpret entropy as uncertainty, all common measures of which are nonnegative.\footnote{In principle, one could enforce positivity of any scoring rule by adding a large constant $C \in \mathbb{R}_+$, which would not affect propriety of the scoring rule. However, this transformation would be at odds with empirical practice and possibly cumbersome to implement.} The logarithmic score, which is not a kernel score as noted above, does not share the advantages mentioned in (a): Its (Kullback-Leibler) divergence function is asymmetric, and its entropy function can be negative. Part (b) of the proposition shows that the average divergence term $D$ can be written in terms of pairwise divergences between component distributions. This result is potentially helpful for computation, e.g. when the components are of a simpler functional form than their linear pool, as in the Gaussian case.
For kernel scores $S = S_L$, we thus argue that the term $D$ at ((ref)) and ((ref)) defines a useful measure of average disagreement within the linear pool.
In the case of a univariate, continuous predictand, quantile combinations have become a popular alternative to the linear pool. LichtendahlEtAl2013 show that forecast distributions obtained via quantile combination tend to be less dispersed than the linear pool in terms of convex order and (even) moments. We next relate their results to entropy as an alternative measure of uncertainty. To facilitate the comparison, we momentarily assume that the $n$ forecast distributions $F_1,\ldots,F_n$ are strictly increasing, so that the $u$-quantile of $F_w^q$ is unique and equals $\sum_{i=1}^n w_iF_i^{-1}(u),$ where $u \in (0,1)$.
If $S$ is squared error, the proposition's convexity assumption holds without any restriction on $F_w$, and the proposition recovers LichtendahlEtAl2013's result that quantile combination yields a forecast distribution with smaller variance. For the CRPS, the convexity assumption also holds without any restriction on $F_w$. Note that the convexity assumption in Proposition (ref) is sufficient but not necessary for higher entropy of the linear pool. Furthermore, while untypical in practice, it is possible to construct kernel score examples in which the linear pool's entropy is strictly smaller than that of quantile-based combination (see Appendix (ref)).
Sections (ref) considers the linear pool's entropy function, which captures the pool's assessment of its own forecasting performance (ex ante, that is, before the outcome has realized). The following result instead consider the pool's ex post performance (that is, after observing the outcome $Y = y$).
For the special case where $S_L$ is squared error, the statement of the proposition is well known, dating back at least to Engle1983. See KnueppelKrueger2022 for details and discussion. For the special cases where $S_L$ corresponds to the CRPS, the statement of Proposition (ref) has been noted as Corollary 3.3.1 by Krueger2013. Furthermore, Proposition (ref) sharpens Proposition 1 of AllenEtAl2024 which states that $S_L(F_w, y) \le \sum_{i=1}^n S_L(F_i, y).$ Finally, NeymanRoughgarden2023 consider the difference $\sum_{i=1}^n w_i S(F_i, y) - S(F_c, y)$ where $F_c$ is some (not necessarily linear) combination of the $n$ forecast distributions $F_1,F_2, \ldots, F_n$, and $S$ is a proper scoring rule. They establish a specific form of $F_c$ (`quasi-arithmetic pooling') that optimizes the difference in a worst-case sense (see their Theorem 4.1).\footnote{Since they define scoring rules in positive orientation, their expression for the difference in question must be multiplied by minus one in our context.} By contrast, our Proposition (ref) provides the specific form of the difference for the case that $F_c = F_w$ is the linear pool, and the scoring rule $S = S_L$ is a kernel score.
Proposition (ref) implies that the linear pool improves upon the average performance of its components. The amount of improvement is given by disagreement, $D$. Hence, for a given average performance of the pool's components, and for given combination weights, it is desirable that the components be as diverse as possible.
According to Proposition (ref), the gains from linear pooling are constant, in that $S(F_w, y) - \sum_{i=1}^n w_i S(F_i, y) = D,$ where $D$ does not depend on $y$. Furthermore, our Proposition (ref) states that when using a kernel score $S_L$, $F_w$ coincides with the quasi-arithmetic pool. Taken together, the two results hence imply that the gains from quasi-arithmetic pooling are constant in $y$. The latter statement is derived by NeymanRoughgarden2023 in a formal setup that is slightly different from ours,\footnote{They consider general proper scoring rules for categorical outcomes, whereas we consider kernel scores for general outcomes on a finite outcome space.} and using different proof techniques. NeymanRoughgarden2023 also provide an appealing economic motivation for considering the gains from combination, in terms of the profit of an agent who subcontracts a group of expert forecasters.
In this section we discuss some implications of the previous results. For this purpose it will be necessary to take a stance on the properties of the forecast distributions $F_i$. We will describe these properties by means of a prediction space, a joint distribution of forecasts and realizations. GneitingRanjan2013 provide a general prediction space framework, formalizing ideas that date back to Bates-Granger-69 and MurphyWinkler1987. Our following definition is a slightly adapted version of GneitingRanjan2013's Definition 2.1.
Let $\mathcal{L}(Y|\mathcal{A})$ denote the distribution of $Y$ given some sub-$\sigma$-algebra $\mathcal{A}$, and let $\sigma(F_i)$ denote the $\sigma$-algebra generated by the forecast $F_i$ itself. We then consider the following notion of forecast calibration Tsyplakov2013, GneitingRanjan2013:
Intuitively, the condition implies that $F_i$ is free of any systematic biases. A user who learns about the forecast $F_i$ (but has no access to other relevant information about $Y$) should thus use the forecast `as is', i.e. there is no need to correct or post-process the forecast in any way. See KnueppelEtAl2022 for further discussion.
We first consider implications of Proposition (ref) on the choice of combination weights, a problem that has been widely discussed in the literature. In the context of mean forecasting under squared error loss, the empirical result that equal weights are often hard to beat (the `forecast combination puzzle') has motivated theoretical interest in the subject, see e.g. ClaeskensEtAl2016 and Elliott2025.
We initially assume that the weights $(w_i)_{i=1}^n$ are constant, i.e., they do not depend on conditioning information. Proposition (ref) implies that the population version of the weight minimization problem is given by
where we use the simplified notation $S_i = S_L(F_i, Y)$ and $d_{ij} = d(F_i, F_j)$. The following result presents a sufficient condition under which equal weights are optimal, i.e. $w^* = (1/n, \ldots, 1/n)'$ at ((ref)).
If the forecasts are exchangeable in the sense that the joint distribution of $(F_1, \ldots, F_n, Y)$ is invariant to permutations of $F_1, \ldots, F_n$, then Assumptions A and B are satisfied. However, Assumptions A and B are weaker than exchangeability, and depend on the scoring rule $S_L$ being used. In particular, for mean forecasting under squared error loss, Assumptions A and B depend only on the means $\mu_1, \ldots, \mu_n$ of the forecast distributions, so the distributions are allowed to differ regarding other facets such as variance or skewness. For other scoring rules such as CRPS, however, such differences would typically violate Assumptions A and B.
Assumption A is strong in principle but can well be plausible in practice. In particular, there is often uncertainty about the relative forecasting performance of various methods. This can be due to a short history of forecasts and observations (e.g. when forecasting European macroeconomic data, which are available only quarterly since 2002), due to entry and exit of forecasters in surveys or collaborative forecasting efforts CramerEtAl2022, or due to temporal instability of the methods' relative performance GiacominiRossi2010. In the presence of such uncertainty, the equal-performance hypothesis from Assumption A is arguably a plausible default. Assumption B imposes another symmetry-type condition on the methods. The plausibility of this condition hinges on the set of methods being combined. When $n = 2$, the condition is trivially satisfied. Remark (ref) links Assumptions A and B to a well-known set of assumptions that ensures optimality of equal weights in the case of mean forecasts Timmermann2006.
Elliott2025 and DieboldEtAl2025 argue that the assumptions of Remark (ref) are empirically plausible for survey forecasts of US macroeconomic variables.
We next consider weights that are not constant but reflect conditioning information used in the forecasting process. For example, the information set $\mathcal{A}_F = \bigcup_{i=1}^n \sigma(F_i)$ is generated by knowledge of the forecasts $F_1, \ldots, F_n$. As another example, let $\mathcal{A}_i$ denote the information set underlying forecast $i$. Then $\mathcal{A}_{1:n} = \bigcup_{i=1}^n \sigma(\mathcal{A}_i)$ is the union of the information sets underying the individual forecasts. Clearly, $\mathcal{A}_F \subseteq \mathcal{A}_{1:n}$.
Let $w_\mathcal{A}$ denote the combination weights that optimize expected performance conditional on $\mathcal{A}$, i.e.
Constant weights as considered at ((ref)) arise as a special case when $\mathcal{A} = \emptyset$ is the empty information set.
The proposition states that using a larger information set is necessarily beneficial when constructing forecast combination weights. Importantly, this population-level result assumes that $w_\mathcal{A}^*$ and $w_\mathcal{B}^*$ make optimal use of their respective information sets. The proposition is similar in spirit to results on nested information sets in forecasting (see e.g. HolzmannEulert2014 or KruegerZiegel2021).
Apart from the empty information set, the information set $\mathcal{A}_{F}$ seems particularly relevant from a practical perspective: Since the individual forecast distributions must be known for constructing any combination, they might as well be used for constructing the combination weights. Given that knowledge of the forecasts implies knowledge of pairwise divergences (i.e., $d_{ij} \in \mathcal{A}_{F}$ for all $i, j$), the optimal weights based on this information set are
The expression $\mathbb{E}_\mathbb{Q}(S_i|\mathcal{A}_{F})$ is a rather complicated object, as it implicitly depends on the distribution of $(F_1, F_2, \ldots, F_n, Y)$. Thus, any attempt to use $w_{\mathcal{A}_F}^*$ in practice requires to either estimate the term (e.g. via regression methods) or to restrict it by means of assumptions.
The term $\Delta_S$ from Definition (ref) considers the difference between a forecast's actual performance and its entropy (i.e., the estimate of its own performance). This difference has been studied by KnueppelEtAl2022 and others. Definition (ref) requires the forecast $F_i$ to be realistic in an `average' sense that is weaker than auto-calibration Krueger2024.
Proposition (ref) generalizes KnueppelKrueger2022 from squared error and a univariate predictand to all kernel scores and a possibly multivariate predictand. Intuitively, its result is due to a dual role of disagreement $D$: On the one hand, the pool's entropy becomes more pessimistic as $D$ increases (Proposition (ref)). On the other hand, ceteris paribus $D$ improves the pool's realized performance (Proposition (ref)). The proposition's broader implication that the combination of realistic components is underconfident has been noted by Hora2004 for a continuous univariate predictand, by RanjanGneiting2010 for a binary univariate predictand, and by GneitingRanjan2013 for a general univariate predictand, using various notions of calibrations. We are not aware of results for a general multivariate predictand as covered by Proposition (ref).
Proposition (ref) presents conditions under which quantile pooling is more effective than linear pooling, in the sense that it attains smaller entropy (i.e., is less dispersed) without introducing overconfidence. An interesting open question is whether analogous statements hold for kernel scores other than CRPS (in particular, for squared error).
Here we present a Gaussian example that illustrates the results on weighting schemes and underonfidence discussed in Section (ref).
Let $Y = X_1 + X_2 + X_3 + \varepsilon,$ with $$X =
' \sim \mathcal{N}\left(\mathbf{0},
\right),$$ where $\sigma_3 > 0$ and $\rho \in (-\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})$ are parameters and $Z$ is a Bernoulli random variable with success probability $1/2$ that is independent of $X$. The restriction on $\rho$ ensures that the covariance matrix of $X$ is positive definite. Conditional on $Z = z$, $\varepsilon \sim \mathcal{N}(0, 1 + z).$
We consider a setting with three forecasters, where forecaster $i \in \{1, 2, 3\}$ issues the correct conditional distribution of $Y$ given $Z = z$ and $X_i = x_i$. Since $
$ is bivariate normal given $Z = z$, the conditional distributions of $Y|Z,X_i$ is also normal. For $i = 1,2$, we have
Furthermore,
The setting implies that Forecasts 1 and 2 are exchangeable. If $\sigma_3$ and $|\rho|$ are sufficiently large, then $X_3$ is a more informative forecast than $X_1$ or $X_2$, performing better in terms of expected squared error or CRPS: If $\sigma_3$ is large, then $X_3$ is an important component of $Y$, so having access to it is particularly helpful. If $|\rho|$ is large, then $X_3$ is helpful for predicting $X_1$, $X_2$ and thus $Y$. The setting further implies that all individual forecast distributions are realistic (see Definition (ref)). Hence by Proposition (ref), any non-trivial linear pool will be underconfident.
We consider the following weighting schemes. First, equal weights of $1/3$ for each forecast. Second, static weights that are the finite sample analogue of $w^*$ defined at ((ref)). Third, weights $w_{\mathcal{A}_F}^*$ as defined at ((ref)). As discussed there, the weights are infeasible in practice, which is why we refer to them as `dynamic oracle' weights. The weights can be computed analytically in the current setup, and are informative as a hypothetical performance benchmark. We next derive an expression for $\mathbb{E}_{\mathbb{Q}}(S_i|\mathcal{A}_F)$ which is required to compute the weights. Let $\mu_\mathbf{X}$ and $\sigma^2_\mathbf{X}$ denote the mean and variance of $Y|X_1,X_2,X_3,Z$, and note that $\mathbb{E}_\mathbb{Q}(Y|\mathcal{A}_F) = \mu_\mathbf{X} = X_1 + X_2 + X_3$ and $\mathbb{V}_\mathbb{Q}(Y|\mathcal{A}_F) = \sigma^2_\mathbf{X} = 1 + Z$. Then $$\mathbb{E}_{\mathbb{Q}}(S_i|\mathcal{A}_F) = d_\mathcal{N}(\mu_\mathbf{X},\sigma^2_\mathbf{X}, M_i, V_i) + \text{ent}_\mathcal{N}(\mu_\mathbf{X},\sigma^2_\mathbf{X}),$$ where $d_\mathcal{N}$ and $\text{ent}_\mathcal{N}$ are the CRPS divergence and entropy functions in the Gaussian case (see Appendix (ref)). Fourth, a feasible counterpart of $w_{\mathcal{A}_F}$ using complete subset regressions, as decribed in Remark (ref). We refer to this method as `dynamic'.
For comparison to the various linear pooling methods, we also consider an equally weighted quantile combination. As an infeasible theoretical benchmark, we further consider the true forecast distribution $\mathcal{L}(Y|\mathcal{A}_F)$ which combines the information sets underlying the individual forecasts (i.e., $X_1, X_2$ and $X_3$) instead of the forecasts themselves. In all three scenarios, this approach yields a Gaussian distribution with mean $X_1 + X_2 + X_3$ and variance $1+Z$. Since $Z$ is a Bernoulli random variable with success probability one half, the expected CRPS of $\mathcal{L}(Y|\mathcal{A}_F)$ is $0.5 \times (1 + \sqrt{2})/\sqrt{\pi} \approx 0.68$ (see Appendix (ref)).\footnote{While also infeasible, the `dynamic oracle' method is forced to use a linear pool of the individual forecasts. This form is misspecified since the true distribution $\mathcal{L}(Y|\mathcal{A}_F)$ is Gaussian, so that the `dynamic oracle' performs worse than $\mathcal{L}(Y|\mathcal{A}_F)$ .}
Here we present simulation results for $1\,000$ Monte Carlo iterations. In each iteration, we use a training sample of $n_{\text{train}}$ observations (forecasts and outcomes) for estimating the weights, followed by a test sample of $100$ observations. We thus obtain a total of $100\,000$ test sample observations. For each weighting scheme, we compute the average CRPS across these observations. We set $n_{\text{train}}$ to either $16$ or $125$, corresponding to the values used in our empirical case studies below. We consider three scenarios:
We first set $\rho = 0$ and $\sigma_3 = 1$, so that all three forecasts are exchangeable, and the conditions of Proposition (ref) are satisfied. Hence equal weights are optimal.
Here we set $\rho = 0.2$ and $\sigma_3 = 1.2,$ so that the expected CRPS of forecast $3$ is about $12\%$ smaller (i.e., better) than the expected CRPS of forecasts 1 and 2.
Here we set $\rho = 0.25$ and $\sigma_3 = 1.8$, so that forecast $3$ is about $30\%$ more accurate than the other forecasts in terms of expected CRPS.
Table (ref) presents the simulation results. While the equal and `dynamic oracle' methods use fixed and known parameters, the `static' and `dynamic' methods feature finite sample estimation uncertainty. Comparisons of their performance across sample sizes ($n = 16$ versus $n = 125$) reveal the expected pattern that a larger sample size is beneficial. That said, this effect is quantitatively rather small, relative to variation of CRPS scores across methods (rows). Similarly, the performance gap between static and equal weights is small in Scenario 1 where equal weights are optimal.
Static and dynamic weights generally perform similarly, and dynamic oracle weights clearly outperform both of them. This indicates a potential for further benefits of the dynamic approach when using the regressor pool defined by $\mathcal{A}_F$ more effectively. Empirical evidence seems necessary to clarify how much of these potential gains can realistically be realized in practice.
The combinations mostly outperform forecast 3, which is the best individual forecast in all settings. This is particularly interesting in the case of static and equal weights, which generally yield underconfident forecasts (Proposition (ref)), whereas forecast 3 is realistic. In Scenario 1, equal weights are even maximally underconfident (as they maximize $\mathbb{E}_\mathbb{Q}(D)$) but are nevertheless optimal in terms of expected CRPS.
We further find small but consistent gains of equally weighted quantile combination as compared to equally weighted linear combination. These gains can be explained by the fact that quantile combination yields a sharper forecast distribution, which is desirable in the present case (see Proposition (ref)).
This section provides empirical illustrations on inflation and exchange rate forecasting.
We first consider bivariate forecasting of two popular US inflation measures, based on the consumer price index (CPI) and the price index of GDP (PGDP). For each measure, we consider quarterly annualized growth rates of the underlying index. Figure (ref) in the appendix plots the two time series. The series are highly correlated, so that either series is of potential help for predicting the other. We construct bivariate forecast distributions using two methods. First, we use average point forecasts from the PhiladelphiaFed2024a's Survey of Professional Forecasters (SPF). We use the bivariate empirical distribution of the SPF's historical forecast errors to construct a forecast distribution. This is a simple bivariate `postprocessing' method based on the principle of using past forecast errors in order to estimate future forecast uncertainty; see SchefzikEtAl2013 for further discussion. As a second forecasting method, we use a Bayesian vector autoregressive (BVAR) model with stochastic volatility. Specifically, we use the model proposed by Primiceri2005 and delNegro2015, as implemented in the R package bvarsv Krueger2015. We employ the default setting of the latter implementation -- in particular, using a single autoregressive lag, and priors that allow for time variation in both the mean and variance equations of the BVAR. For constructing the forecast distributions, we employ real-time data on CPI and PGDP, as provided by PhiladelphiaFed2024b. We use an expanding estimation window, with data ranging back until 1980. We consider both current-quarter forecasts and one-year-ahead forecasts. Since the current quarter's observation is not yet available to SPF participants, these two horizons correspond to $h = 1$ and $h = 5$ quarter ahead forecasts. We consider forecasts made between 1994:Q4 (the earliest quarter for which the Philadelphia Fed's real-time data is available) and 2024:Q2.
We use the Energy Score for evaluating the bivariate probabilistic forecasts; see Appendix (ref) for details. At any given date and forecast horizon, the two component distributions $F_1$ (SPF) and $F_2$ (BVAR) are equally weighted empirical distributions of $40$ and $5\,000$ observations respectively. For linear pooling, we consider equal, static or dynamic combination weights, where the latter are constructed as described in Remark (ref). We use a rolling window of $16$ observations for estimating the static and dynamic weights.\footnote{The label `static' refers to the fact that these weights are a finite sample analogue of the unconditional weights at ((ref)). In practice, the use of a rolling estimation window introduces time variation into these weights.}
Figure (ref) in the appendix plots the equally weighted pool's expected ES (i.e., its entropy) and its components for current-quarter forecasts. In most quarters, disagreement accounts for a modest share of the expected ES, with an average share of $7\%$. Two notable exceptions with large disagreement arise in 2008:Q4 and 2020:Q2. These two quarters are associated with the global financial crisis and the Covid-19 pandemic respectively. In these quarters, disagreement peaks both in absolute terms and regarding its share among the linear pool's expected score ($46.6\%$ in 2008:Q4, and $52.4\%$ in 2020:Q2). The top row of Figure (ref) shows the bivariate forecast distributions for these quarters. In both instances, the SPF distribution is located to the southwest of the BVAR distribution, indicating lower inflation rates according to both measures (CPI and PGDP). The SPF's assessment is in line with the eventual realizations, and can be explained by the survey's access to more timely intra-quarter information that is not available to the BVAR. In particular, the SPF point forecasts correctly anticipate the dis-inflationary short-term impact of the economic shocks of 2008:Q4 and 2020:Q2. Thus, the SPF's access to recent and possibly judgmental information is beneficial in these examples of short-term forecasts in turbulent periods.
Interestingly, the effect just described is not present for one-year-ahead forecasts ($h = 5$). Information about the current state of the economy seems to matter less at this longer horizon, where it is dominated by shocks occurring between the forecast date and the target date KruegerEtAl2017. To illustrate this point, the bottom row of Figure (ref) shows the one-year-ahead forecast distributions in 2008:Q4 and 2020:Q2. Disagreement between the SPF and BVAR distributions is small even in these turbulent periods. More broadly, at $h = 5$ the share of disagreement among the linear pool's entropy is $3\%$ on average, with a maximal share of $16\%$ attained in 2001:Q2. Figure (ref) in the appendix shows details for $h = 5$.
Table (ref) summarizes the forecast performance of the SPF and BVAR forecasts as well as the three linear pools. Among the two individual forecasts, the SPF attains a better score at $h = 1$ while the BVAR prevails at $h = 5$. Among the three pools, equal weights perform worst at $h = 1$ but best at $h = 5$. Static and dynamic weights generally perform similarly.
In Table (ref), we consider DieboldMariano1995 tests of equal predictive ability between any pair of pooling methods, using a heteroskedasticity-robust $t$ statistic and variance estimation via the function \verb|NeweyWest| in the \verb|R| package sandwich Zeileis2004,ZeileisEtAl2020. Using a two-sided test at the 5% level, pairwise differences in performance are statistically insignificant. The test results under squared error loss, which we report in Table (ref), are qualitatively identical.
As discussed in Section (ref), comparing a method's expected and actual scores provides evidence on calibration. To test the null that a method is realistic about its own performance (see Definition (ref)), we use a HAC robust $t$-test. KnueppelEtAl2022 refer to this method as `score-based calibration testing'. Using the same variance estimator as for Diebold-Mariano testing, we do not reject the null in any of the six cases considered (three methods times two horizons), yielding no evidence against calibration. See Table (ref) for details.
Our second case study considers distributional forecasting of exchange rates. Making point forecasts of exchange rates is notoriously difficult in the sense that simple benchmarks are hard to beat (see Rossi2013 and the references therein). Predicting other facets of the conditional distribution of exchange rates, such as their conditional variance, seems more promising and is also practically relevant for risk management purposes. A large number of studies, including HansenLunde2005, consider predicting exchange rate volatility, often via generalized autoregressive heteroscedasticity (GARCH) type models. A smaller number of papers including AbbateMarcellino2018 and Huber2019 consider forecasting the full distribution of exchange rates.
In order to forecast the exchange rate at date $t$, we simply use the $M$ most recent observations $y_{t-1}, y_{t-2}, \ldots, y_{t-M}$ available at date $t-1$. The window length $M$ is an important tuning parameter: A small choice of $M$ yields a flexible distribution that adapts to changing conditions, while a large choice of $M$ offers more stability and less estimation noise. The choice of $M = M^*$ that best governs this trade-off is largely an empirical question. Rather than selecting a single choice of $M$, we consider forecast combination over a grid of ten plausible choices ranging from $M = 250$ (representing roughly one year of data for each business day) to $M = 2500$ (representing roughly ten years). Similar approaches to averaging over forecasts constructed from different data windows have been studied by Pesaran2011, among others.
More specifically, let $M_i$ denote the $i$th choice of window length, where $i = 1, 2, \ldots, n = 10$. A combined forecast distribution with weights $w_1, w_2, \ldots, w_{10}$ then places weight\\ $\sum_{i=1}^{10} w_i~\mathbf{1}(M_i \le j)/M_i$ on observation $y_{t-j}$, where $\mathbf{1}(A)$ is the indicator function of the event $A$. Thus, the weights produced by the combined forecasts are necessarily decreasing in the distance $j$, which seems plausible. Figure (ref) illustrates this setup. As for inflation, we use either equal, static or dynamic combination weights. Static weights are based on a rolling window of 125 observations (i.e., roughly half a year of data for each business day).
We consider the exchange rate of the US Dollar against the Canadian Dollar, the Euro, the Japanese Yen and the British Pound.\footnote{The data is available from fred2026, corresponding to the series codes DEXCAUS, DEXUSEU, DEXJPUS and DEXUSUK.} Table (ref) summarizes the empirical results in terms of CRPS, whereas Table (ref) reports Diebold-Mariano comparisons between pooling methods. Dynamic weights tend to perform best overall, with the difference to equal and dynamic weights being statistically significant for Canada. That said, the results for squared error reported in Table (ref) indicate that equal weights tend to outperform the static and dynamic methods.
As regards calibration, the combined forecasts tend to be underconfident (see Table (ref)). This effect is statistically significant at the 5% level for two series (Canada and UK) and all three weighting schemes.
Proposition (ref) offers sufficient conditions under which equal weights are optimal in population. In practice, the relevant question is whether any violations of these conditions are severe enough to justify the costs of weight estimation Elliott2025. These costs are both statistical (in terms of weight estimation error) and conceptual (in terms of additional tuning parameters such as the sample used for weight estimation). In the simulation study of Section (ref), estimated weights outperform equal weights even when few observations are available for estimation. In the empirical examples of Section (ref), however, estimated weights do not yield clear or consistent benefits over equal weights. While similar results are well documented for point forecast combination under squared error WangEtAl2023, further empirical results seem necessary to assess the case of distributional forecasting under scoring rules such as the CRPS or energy score.
There is often a prohibitively large set of potential forecast methods, and it is necessary to choose a specific subset of methods that enters a linear pool. This selection is particularly relevant when using equal combination weights, since there is no weight estimation step that can fine-tune the initial selection. Suppose one currently uses an equally weighted pool of $n$ methods, and asks whether to add a further, $n+1$st, method. Based on Equation ((ref)), the additional method will improve the combination if its individual performance is sufficiently good (i.e., $\mathbb{E}_\mathbb{Q}(S_{n+1})$ is sufficiently small), and the method is sufficiently different from the other methods (i.e., $\sum_{i=1}^n \mathbb{E}_\mathbb{Q}(d_{i,n+1})$ is sufficiently large). While these considerations may provide some guidance, principled techniques for selecting the members of the linear pool in practice remain an important challenge.
As discussed in Section (ref), quantile combination produces a forecast distribution that is typically less dispersed than the linear pool. Under certain conditions, this is a desirable feature (see LichtendahlEtAl2013 as well as Proposition (ref) above). Conversely, if the individual forecasts are overconfident, the linear pool's increased dispersion is desirable. The empirical results on the relative performance of linear and quantile based combinations are mixed, which motivates Taylor2025 to propose a variant that encompasses both options.
As discussed throughout this paper, several key properties of the linear pool apply to discrete and continuous, univariate and multivariate settings alike. The linear pool's versatility is appealing, and differs from quantile-based combinations which apply mostly to the univariate continuous case.
Divergences of proper scoring rules are useful for measuring differences between distributions ThorarinsdottirEtAl2013. Here we argue to use kernel scores (i.e. a subclass of proper scoring rules) whose divergence functions are necessarily symmetric. Appendix (ref) provides details on the divergence functions for various kernel scores, covering both well-known measures (such as cross-sectional forecast variance for point forecasts and Cram\'er divergence for distributions) and less common ones (such as energy score divergence). Combined with Proposition (ref), these formulas yield broadly applicable measures of forecast disagreement as studied in economics ClarkMertens2024.
The CRPS is an increasingly popular scoring rule in statistics and machine learning. Our finding that various results on squared error generalize to the CRPS is potentially surprising: The CRPS is often viewed as a generalization of absolute error, to which it reduces when the forecast distribution is of Dirac type, with point mass at a single value GneitingRaftery2007. However, as discussed in Section (ref), absolute error is not a kernel score except in very special cases such as the Dirac case. Our results thus challenge the common intuition of the CRPS as being similar to absolute error, and instead highlights similarities between the CRPS and squared error.
The methods discussed in this paper require to compute entropy and divergence functions associated with kernel scores (see Table (ref) in the appendix). These computations are relatively straightforward for discrete samples that are popular in practice. Our replication material includes relevant software for \verb|R|, covering the CPRS and energy score in particular. The CRPS entropy and divergence for parametric distributions is more elusive, except for the Gaussian case which we cover in Appendix (ref). AllenEtAl2024 show that estimating the weights under a kernel score can be cast as a convex quadratic programming problem. While our optimization problem is not convex quadratic in its original form considered in Section (ref), Appendix (ref) shows how to translate it into such a form. All of our computations use this latter form, together with the function \verb|ipop| of the \verb|R| package \verb|kernlab| due to KaratzoglouEtAl2004.