Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
129,793 characters · 10 sections · 95 citation commands
-1.4cm bred From Reactive to Proactive Volatility Modeling with Hemisphere Neural Networks
\sloppy \\ \color{dpd} \fontfamily{phv}\selectfont \quad \quad \quad Universit\'{e} du Qu\'{e}bec \`{a} Montr\'{e}al \quad \quad \quad \newline \and Mikael Frenette \\ \color{dpd} \fontfamily{phv}\selectfont \quad \quad \quad Universit\'{e} du Qu\'{e}bec \`{a} Montr\'{e}al \quad \quad \quad \newline \and Karin Klieber \\ \color{dpd} \fontfamily{phv}\selectfont \quad \quad \quad Oesterreichische Nationalbank \enskip \quad \quad \quad }
\setstretch{0.98}
\center \setstretch{1.2}
\thispagestyle{empty}
\setcounter{page}{1}
\newgeometry{left=1.7 cm, right= 1.7 cm, top=2.3 cm, bottom=2.3 cm}
Unlike traditional deep learning strongholds such as speech recognition and computer vision, applications in social sciences are typically nowhere near perfect prediction accuracy. In other words, signal-to-noise ratio is low for most economic applications, and in the vicinity of 0 for finance applications. Still, the recent literature shows that deep learning methods can do surprising yet informative predictions in economics smalter2017macroeconomic,medeiros2019,andreini2020deep,hauzenberger2020real,barbaglia2022testing,HNN. Thus, it is particularly pertinent to estimate heterogeneous prediction uncertainty -- in order to determine when to trust or distrust a neural network's forecast.
In this paper, we provide a principled and effective way to do so, which comes in the form of a novel standalone density forecasting tool. Its design also allows for it to be a building block that can be merged with elements of other macroeconometric deep learning models. This is of independent interest given that deep neural networks (NN) and their associated software environments are fertile ground to build more structured models (either for the sake of interpretability, increased performance, or both; see farrell2021deep,HNN), or to incorporate the ever-growing sources of non-traditional data.
\vskip 0.2cm
{ \sc A {Hemisphere} Neural Network (Redux).} HNN introduces the concept of a Hemisphere Neural Network (HNN) where a NN is restricted so that its prediction is the sum of latent time series corresponding to the outputs of subnetworks. Those are constructed from groups of predictors separated at the entrance of the network into different hemispheres. The structure allows the understanding of the final layer's cells output as latent states in a linear equation. There, the motivation was interpretability of the conditional mean through separability. Here, the point is to go beyond the conditional mean.
This paper treats the mean and the variance of a predictive regression as two separate hemispheres in one neural network where the loss function is the negative log-likelihood. The model features a common core at the entrance of the network which accommodates for various interactions between the conditional mean and variance structures. This resembles the autoregressive conditional heteroskedasticity (ARCH) behavior where mean parameters enter the volatility equation or volatility-in-means with the reverse operation. But going straight for maximum likelihood estimation of the new architecture will fail, for old and new reasons. The most prominent of those is that the double descent phenomenon -- the modus operandi of modern deep learning -- will result in the usual benign overfitting of the conditional mean belkin2019reconciling,hastie2019surprises,bartlett2020benign and malign underfitting of the conditional variance. A key observation is that, in vastly overparameterized models aiming for the first two moments, in-sample overfitting of the first leads to underfitting of the second, and vice versa. {\color{black} Then, what will happen out-of-sample is anybody's guess. } Accordingly, left unchecked, HNN could completely overfit the training data with either a perfect conditional mean path or an equally perfect conditional variance process -- the allocation between the two very disparate models left to random initialization choices.
We overcome this particularly daunting roadblock by designing three main algorithmic modifications: a volatility emphasis constraint in estimation, a blocked out-of-bag recalibration, and blocked subsampling. The resulting HNN will prove highly competitive in our (point and density) forecasting exercise and provide more reliable coverage than currently available machine learning (ML) based alternatives. This desirable consistency in performance is a direct byproduct of the three aforementioned "modifications" bringing what could be called "conformal restrictions" in estimation and prediction. As the name suggests, such operations are related to the rapidly growing ML literature on conformal prediction where a pseudo-out-of-sample metric is used as raw material to construct prediction intervals with coverage guarantees vovk2005algorithmic.
\vskip 0.2cm
{ \sc Proactivity, Reactivity, and Related Literature.} We neither restrict the mean nor the variance to follow a specific law of motion. They are both neural (sub)networks taking a large panel of macroeconomic series as common input. Neural networks successfully deal with high-dimensional input spaces and are implemented in highly optimized software environments providing fast computations. We refer to proactive volatility forecasts as those leveraging leading indicators to predict heightened volatility before the model delivers a large forecast error. Conversely, reactive forecasts propagate shocks that already occurred, resulting in increased expected variance in the following periods--after the occurrence of an initial major shock. HNN provides proactive volatility forecasts based on observed indicators when it can, and reactive volatility based on the magnitude of previous prediction errors when it must.
The "reactive" class of models has a very long and distinguished history in econometrics, with (G)ARCH engle1982autoregressive,bollerslev1986generalized and stochastic volatility (SV) models Taylor1982stochvol,hull1987pricing,jacquier2002bayesian. The popularity of SV for macroeconomic forecasting is mostly unrivaled. It is the workhorse volatility process to close a Bayesian model and accounts for (slow) structural change in innovations' variance stock2007has,AAG2013,clark2015macroeconomic,carriero2019large. SV and GARCH models can be augmented with indicators that may have proactive qualities guidolin2021boosting, but this faces various important challenges, like that of high-dimensionality, and therefore, traditional reactive specifications have nearly always dominated the landscape. We find neural network adaptions for SV and GARCH to model time-varying volatility in financial time series in, e.g., luo2018neural,yin2022neuralgarch, yin2022varvol. However, estimating the predictive variance when applying deep learning models to estimate the predictive mean has turned out to be a very challenging task. Neural networks tend to be overconfident in making predictions guo2017calibration,amodei2016concrete and deliver residuals close to 0 belkin2019reconciling that are a rather elusive target in a secondary conditional variance regression. Furthermore, implementing GARCH or SV-like methods in the highly nonlinear structure of deep learning models implies a significant deviation from the very software environments making their computations feasible and efficient. Recent contributions apply SV in nonlinear or nonparametric models such as Bayesian additive regression trees huber2022inference or Bayesian neural networks hauzenberger2022bnn. However, these models rely on Bayesian estimation which often turns out to be computationally costly, and the volatility prediction remains solely reactive by construction.
Quantile and distributional regressions enjoy increasing popularity in the macroeconomic literature and have seldom been found to have proactive qualities adrian2019,adams2021forecasting,caldara2021understanding,delle2021modeling,guidolin2021boosting. Early propositions to overcome the normality assumption when modeling densities include the seminonparametric (SNP) model of gallant1987semi. Recent contributions extend the concepts of quantile and density regressions to nonlinear nonparametric models. clark2022tail do so in the context of Bayesian additive regression trees (BART) whereas barunik2022learning and chronopoulos2023forecasting do related things with neural networks.
From the deep learning literature side of the aisle, we find extensions of traditional methods which allow for the estimation of high-order moments, mixtures of distributions, as well as quantile regressions. nix1994estimating and bishop1994mixture proposed estimating the first and second moments of the predictive distribution with two separate neural networks. Building on this idea, the recent literature proposes different variants of mean-variance neural networks dybowski2001pinn,khosravi2014optimized as well as mixture density networks graves2013generating,lakshminarayanan2017simple. In that vein, the DeepAR model of salinas2020deepar is getting increasing attention. Amazon's DeepAR is a sequence-to-sequence probabilistic forecasting model which estimates the parameters of a distribution with Recurrent Neural Networks (RNNs) based on maximum likelihood. However, as documented in gasthaus2019probabilistic, DeepAR tends to underestimate variance, likely for the aforementioned double descent reasons. We will also find in our experiments that the quality of DeepAR's density forecasts is erratic. Lastly, the estimation of quantile regressions using neural networks dates back to taylor2000qrnn and has since been the subject of a copious amount of research feng2010robust,wen2017multi,cannon2018mcqrnn,moon2021learning.
\vskip 0.2cm
{ \sc Intended Use.} A relevant question is where HNN stands in this deluge of works. It is an economical yet not any less sophisticated solution to quantify time-varying uncertainty surrounding deep learning-based macroeconomic forecasts. It is fast, malleable, and easily understood -- through the use of only two (nonlinear) conditional moments. We will see that it works well for many targets without any particular tuning, and that both point and density forecasts are highly competitive and reliable. Lastly, it will be easily merged with more structured models, like that of HNN, giving a "complete" model of inflation \textcolor{black}{based on a nonlinear Phillips curve specification}.
\vskip 0.2cm
{\sc Summary of Forecasting Results.} In a thorough forecasting exercise using macroeconomic data for the US, we find that HNN has a great capacity for adaptation in the face of a heterogeneous pool of series. Adaptability is a recurring finding when applying machine learning tools to macroeconometric problems MRF and can be linked back to the carefully crafted semi-nonparametric structure of the model. Specifically, it captures the Great Moderation pattern in real activity variables (i.e., long-run change) and yet, without changing the specification nor hyperparameters, can deliver a more "spiky" volatility process for the S&P 500. The estimated volatility path for longer-run forecasts of macroeconomic targets ($s=4$ quarters ahead) displays a behavior that at times resembles more that of a (smoothly) switching process, in contrast to the slowly evolving SV process which dominates the literature. Those higher volatility regions are proactive in the sense that they begin before the advent of a major prediction error, a behavior that is observed both in-sample (with out-of-bag estimates) and out-of-sample for many targets (e.g., GDP growth, Unemployment Rate, Inflation).
In terms of performance, HNN always ranks among the top models in terms of RMSE, log score, coverage rates, and other metrics of calibration and probabilistic forecast evaluation. More interestingly, it never suffers "catastrophic failures" (like massive undercoverage) that seldom occur on some targets for the other sophisticated competing models. For instance, it is not infrequent to see BART and DeepAR substantially undercovering -- a phenomenon we delve into and attribute to harmless overfitting of the conditional mean leading to quite harmful underestimation {\color{black} and underfitting} of the conditional variance. This implies that while conditional means can be used as per the model’s estimation (and perform well as such), conditional variances cannot, and often fail in ways that basic manual quality control is unable to flag nor fix ex-ante. HNN, in contrast, appears to have a level of reliability mostly on par with that of AR competitors. The use of out-of-bag (and presumably non-overfitted) errors to calibrate or estimate the volatility process helps HNN a lot in being reliable out-of-the-box. Our much simpler NN$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$, a reduction of HNN which forfeits proactive volatility but keeps a sophisticated conditional mean function and fits a SV model on OOB residuals in a second step, is equally reliable in terms of coverage.
The forecasting section also includes a series of vignettes. First, we compare our approach to quantile regressions of various kinds. A striking observation is that for real activity targets -- which asymmetry and non-normality have been heavily documented following adrian2019 -- the normal likelihood-based HNN usually performs better or as well as the best (linear or nonlinear) quantile model. This is true for both tails of the distribution and short and long forecasting horizons. Second, we investigate whether the use of Long Short-Term Memory Networks hochreiter1997lstm could further improve HNN results in any material way---they do not. \textcolor{black}{In addition, we present results on monthly data for the US and a euro area forecasting exercise in the appendix. We find that HNN fares well with time series that are noisier and shorter. However, as one could expect from this more hostile terrain, gains with respect to autoregressive SV models are more punctual and modest in size.}
\vskip 0.2cm
{\sc Fusion With a Structured Deep Learning Model.} HNN and the overall apparatus developed in this paper can also be joined in a modular fashion with more structured deep learning models to obtain interpretable forecasts with reliable uncertainty quantification. We construct such a model for inflation by embedding HNN's Neural Phillips Curve (NPC) within this paper's arsenal. We find that the customized (yet restricted) HNN-NPC improves point and density forecasts over the plain density HNN. We also compare with a simpler Bayesian model also providing interpretability and uncertainty estimates via SV chan2016. In line with HNN's findings, we see that the neural model better captures inflation dynamics during important episodes like in the outset of the Great Recession and the post-Pandemic inflation surge. This is attributable to a very different reading of the contribution of real activity and expectations in less quiet economic times. We enrich such results by showing that HNN-NPC was rather confident when predicting high inflation in early 2020, and nearly dismisses its own deflationary forecast in 2020. Moreover, its proactive volatility qualities are also apparent in 2008 where the volatility estimate climbs out of its bed well before SV does so (following the 2008 crash in oil prices). All in all, HNN-NPC demonstrates how deep learning can improve over classic approaches while retaining essential qualities of the latter -- interpretability and uncertainty quantification.
\vskip 0.2cm
{\sc Outline.} Section (ref) introduces the mean-variance HNN by describing and motivating the network architecture, and presenting the key algorithmic modifications to plain MLE. Section (ref) conducts an extensive empirical analysis using macroeconomic data for the US. \textcolor{black}{In Section (ref) we extend HNN to a nonlinear Phillips curve model for inflation.} Finally, Section (ref) concludes.
This section describes our proposed neural network architecture to estimate the mean and variance of the predictive distribution of our target variable $y_{t+1}$ (or $y_{t+s}$ in the case of direct $s$ steps ahead forecasts). We assume that $y_{t+1}$ follows a Gaussian distribution and depends on a (potentially very large) number of $K$ covariates denoted by $\boldsymbol{X}_t$:
In this very general setup, the functions $f$ and $g$ are unknown and may be highly nonlinear. To remain agnostic on the functional form of $f$ and $g$, both will be approximated through a neural network (NN) structure hornik1989multilayer. Rather than estimating a standard deep NN model and hoping to make something out of the residuals, we design a specific architecture derived from HNN's original HNN that will obtain $g$ and $f$ jointly. In our application, hemisphere 1 ($h_m = f$) is the conditional mean and hemisphere 2 ($h_v = g$) is the conditional variance. Both hemispheres are fully nonlinear, nonparametric functions of the input space $\boldsymbol{X}_t$ which ultimately output two time series: conditional mean ($\hat{y}_t$) and conditional variance ($\hat{\sigma}^2_t$). Importantly, they get their assigned roles from how they enter the loss function, which is now proportional to the log-likelihood. Accordingly, the first building block of our approach is to have a neural network with objective function
where $\theta_{m}$ and $\theta_{v}$ are the network parameters consisting of the weights $w_j$ and the bias term $b_j$ (i.e., $\theta_m = (w_m, b_m)$ and $\theta_v = (w_v, b_v)$). The next questions are (i) what is the structure of $h_m$ and $h_v$, and (ii) how do we successfully solve (ref). The next paragraphs set out to answer those through a series of "ingredients" that reinvigorate what is otherwise a rather plain-looking MLE problem.
\vskip 0.25cm
{ \sc Ingredient 1: Two Hemispheres and A Common Core.} Figure (ref) summarizes the network's architecture. As can be seen, both hemispheres share the same input data as well as a few common layers before estimating the parameters of each hemisphere. The outcome of both hemispheres enter the loss function and thereby complete the model setup. Going backward from the loss towards the original inputs, the "yellow" hemisphere is
and the "blue" one is
where $\phi$ denotes a nonlinear activation function, $L_m$ and $L_v$ are number of hidden layers for each hemisphere, and $L_c$ is that of the common (red) core. The Softplus activation function in (ref) constrains $\hat{h}_v(\mathbf{X}_t;\theta_v)$ to be positive at all times. Clearly, the definitions of $h_m (\boldsymbol{X}_t;\theta_m)$ and $h_v (\boldsymbol{X}_t;\theta_v)$ are still incomplete given that $\boldsymbol{X}_t$ has yet to make an appearance on the right-hand side of (ref) and (ref). The common core at the entrance of the networks brings such completion via
with ${\bm Z}^{(0)}_t = \bm X_t$. Thus, the first layer in each hemisphere uses the outputs of neurons from the last layer of the common block ${\bm Z}^{(L_c)}_t$.
Having dedicated mean and variance hemispheres requires little additional ink to motivate, but the virtues of the common core, while numerous, are more subtle. Consider the following simple linear data generating process (DGP) with ARCH errors
{In this model, the corresponding hemisphere outputs and parametrizations would be}
As gourieroux1997arch puts it "{Even in the simple case, we cannot estimate separately the parameters of the conditional mean and those appearing in the conditional variance.}" Obviously, this does not mean all volatility models need to be estimated jointly. What it suggests, however, is that successful models of time series volatility often have some {cross-equation restrictions} between $h_m$ and $h_v$.
Rather than introducing cross-equation restrictions, which are likely both unfeasible and undesirable in a neural network setup, we discipline $h_m$ and $h_v$ with soft constraints, i.e., cross-equation regularization. We achieve this by estimating common layers at the entrance of the network, which can be interpreted as hemispheres sharing weights. As emphasized in Figure (ref), we estimate a few common layers for both hemispheres before separating the mean from the variance hemisphere, where hemisphere-specific neurons are presented in yellow for the former and in blue for the latter. While this example details how the conditional mean parameters enter that of the variance, the opposite sharing direction is also possible. For instance, latent structures driving volatility can flow in the mean hemispheres, which conveniently allow for GARCH- or SV- or any volatility-in-means effect which have been popular in finance to study the time-varying risk premium EngleLilienRobins1987 and now in macroeconomics to quantify the real effects of uncertainty carriero2018measuring,shin2020new.
By adding time trends to our set of covariates we approach a classical SV specification through a residuals trend-filtering perspective. GARCH dynamics would suggest making $h_v$ a recurrent neural network. As we will see, HNN results will be quite competitive without this additional complication -- RNNs and LSTMs are notoriously harder and longer to train. We nevertheless explore this possibility in Section (ref).
\vskip 0.25cm
{ \sc Ingredient 2: Volatility Emphasis.} As we know from the double descent phenomenon belkin2019reconciling,hastie2019surprises,bartlett2020benign, a mildly deep and large network will yield a near perfect in-sample fit even in the presence of large amounts of noise and yet produce stellar out-of-sample results. When focusing on out-of-sample point forecasts, we can safely embrace double descent and its associated benefits. However, this creates trouble for the historical (in-sample) analysis of conditional mean estimates and double trouble for the conditional variance, with the latter being unreliable both in- and out-of-sample. Our volatility emphasis constraint, coupled with the next two ingredients, will make MLE work in the context of densely parameterized models.
A first observation is that a model in the double descent region eradicates residuals, yet MLE is supposed to obtain the parameters of their non-degenerated distribution. A second is that the reverse solution is also possible: a perfect volatility model with no conditional mean. In other words, when solving (ref) without further adjustments, HNN can completely overfit the data with either $h_m$ or $h_v$, giving rise to vastly different models. This suggests that the overall prevalence of $h_m$ versus $h_v$, when those are left completely unconstrained, is not identified and cannot be obtained from in-sample estimation (in a similar spirit to the regularization parameter $\lambda$ in a ridge regression, GC2019). Note that early stopping can help in regularizing $h_m$ or $h_v$, but will do symmetrically which is highly suboptimal in many applications where it is clear that one hemisphere should be more expressive than the other.
As a solution, we bring in a constraint. We fix the average conditional predictive variance to a constant (i.e., $\texttt{mean}(h_v(\boldsymbol{X}_t;\theta_v))/ \texttt{var} (y_{t+1}) =\nu$) during estimation and let HNN learn deviations from it. We refer to $\nu$ as the volatility emphasis parameter, because it guides how much of the network fitting capacities \textcolor{black}{should go} to the volatility versus the mean. Why does this work? First, it serves as a solution to optimization cycling through near-perfect conditional mean versus conditional variance optima and the general indeterminate nature of the problem in overfitting situations. Fixing the expressivity of $h_m$ allows us to let $h_v$ benignly overfit (if need be) the way it is typically done for the conditional mean estimation in plain squared error minimization. As a result, the conditional mean is tied to deliver estimates that will look like a plausible OOB fit for every run (as set by $\nu$), and conditional variance can be projected OOB and compared to OOB squared errors (Ingredient 3 & 4) to obtain a non-overfitted volatility path in- and out-of-sample.
While the final unconditional variance is readjusted and not imposed (see Ingredient 4 below), we should not choose $\nu$ lightly, as it will influence the relative flexibility of $h_m$ and $h_v$ and estimated paths. Clearly, from experience, we expect $\nu$ to be close to 1 for stock returns and lower for other macroeconomic targets, especially those exhibiting persistence. In theory, $\nu$ could be cross-validated, but to avoid the obvious practical cost of doing so, we rather set $\nu$ through a very well-informed guess. We estimate a standard NN with an analogous architecture, calculate the mean of the squared blocked OOB residuals, and set $\nu = \texttt{mean}(\hat{\varepsilon}_{t,\text{NN}}^2)/ \texttt{var} (y_{t+1})$ where $\hat{\varepsilon}_{t,\text{NN}}$ denotes the blocked OOB residuals. In effect, if one were to conduct basic conformal prediction-based inference for a plain NN in a macroeconomic time series context and assume homoscedasticity, $\texttt{mean}(\hat{\varepsilon}_{t,\text{NN}}^2)$ and $\hat{\varepsilon}_{t,\text{NN}}$ in general would be natural inputs to obtain coverage-guaranteed (out-of-sample) prediction intervals chernozhukov2018exact. The presence of the denominator $\texttt{var} (y_{t+1})$ brings $\nu$ in universal units (i.e., between 0 and 1)\footnote{In practice, the original estimate can go marginally above 1 since the inputs are OOB rather than training residuals, and the plain NN model may do worse out-of-bag than simply taking the sample average when facing extremely low signal-to-noise ratios. We enforce an upper bound at 0.99, effectively forcing $\nu$ to always deliver a $R^2> 1 \%$. } and is implicit in our calculations because all the data will be standardized at the entrance of the network and scaled back to original units at the exit. A possibility for future work is to cross-validate $\nu$ in the neighborhood of the informed guess or update it through iterative HNN estimations, but our current empirical results suggest this extra legwork may not be necessary.
The inevitable failing of an unchecked HNN (and DeepAR later on) as well as the usefulness of the volatility emphasis constraint can also be intuitively understood from basic MLE econometrics for linear regression. Even when fitting the simplest linear model without shrinkage, the MLE estimate of the error variance is always biased downward: it yields $\frac{\sigma^2}{T}<\frac{\sigma^2}{T-K}$ where $K$ is the number of regressors and the second expression is the OLS version. This potentially major discrepancy is straightforward to correct when the number of degrees of freedom is known, which it is not in the deep learning context. The best course of action when the analytical calculation of degrees of freedom is impossible is the use of pseudo-out-of-sample metrics (of which cross-validation is the better known). Thus, curbing many problems at once, we fix $\nu$ to a plausibly unbiased value ex-ante calculated from out-of-bag sampling and an approximated $h_m$.
Two outstanding issues remain. If the originally imposed $\nu$ is not exactly in tune with $h_m$'s final performance, we may want to adjust the average conditional variance accordingly. Another observation is that the volatility emphasis constraint fixes the expressivity of the conditional mean, but not that of the variance. Thus, it does not prevent $h_v$ from overfitting what is left free in the likelihood, and may offer implausibly accurate conditional variance forecasts in-sample that will not be matched out-of-sample. We will get back to this when covering the fourth and final ingredient.
\vskip 0.25cm
{ \sc Ingredient 3: Blocked Subsampling.} We now turn to the important ingredient that has been implicit throughout. Bagging in our context entails two major benefits. First, the use of quantities which are immune to extreme overfitting. Second, it helps with optimization itself. There is no guarantee that a single run of stochastic gradient descent initiated randomly will yield the "true parameters". In that sense, our approach does not aim to succeed where traditional maximum likelihood estimation (MLE) would likely fail. This is less concerning when considering an ensemble of multiple runs, similar to what is common practice for point prediction with NNs anatomy. In this case, we employ \textcolor{black}{$B=1000$ runs}, which may seem excessive for out-of-sample predictions but is suitable for OOB "time series" that utilize an average of $(1-\texttt{subsampling.rate}) \times 1000$ runs at each point in time. \footnote{Considering 300-something runs can be sufficient but results may change in a very marginal way depending on the seed.}
More precisely, the calculations proceed as follows. Assume we have a sample of size 100 \textcolor{black}{and choose a subsampling rate of 0.80}. We estimate HNN using data points from 1 to 85, and project it "out-of-bag" on the 20 observations not used in training. This gives us $h_j(\boldsymbol{X}_{80:100};\hat{\theta}_{j,b})$ for a single allocation $b$ (for $b = 1, \dots, B$) while $h_j(\boldsymbol{X}_{1:80};\hat{\theta}_{j,b})$ are still NAs. By considering many such random (non-overlapping blocked) allocations where "bag" and "out-of-bag" roles are interchanged, we obtain the final $h_{t,m}$ and intermediary (see next ingredient) $h_{t,v}$ paths by averaging over $B$ at each $t$ such that
Interestingly, this procedure fits within the framework of newton1994approximate's Weighted Bayesian Bootstrap \textcolor{black}{and, in particular, of} newton2021weighted's extension of it for generic ML losses. In short, randomly weighted optimization of the loss provides an approximate Bayesian posterior.
\vskip 0.25cm
{ \sc Ingredient 4: Blocked Out-of-Bag Reality Check.} To obtain a proper estimate of the unconditional variance we introduce a recalibration step based on the blocked out-of-bag residuals. This is done by our reality check, which scales back the $h_v(\boldsymbol{X}_t;\theta_v)$ path making use of the OOB residuals.\footnote{In a sense, this step takes the concept of conformal prediction -- a method to form prediction intervals without making distributional assumptions vovk2005algorithmic,lei2013distribution,linusson2020efficient -- to a conditionally heteroskedastic environment. Uncertainty in future predictions is based on the residuals of a held-out validation set, which is used to recalibrate the prediction intervals. chernozhukov2018exact extends the applicability of such methods to dependent data using a block approach. } {\color{black} In our setup, the initial guess for $\nu$ -- coming from a plain NN and not the dual estimation of $h_v$ and $h_m$ -- might not exactly match the resulting average volatility of HNN's OOB residuals.} Moreover, even after early stopping, the raw $h_v$ may be overly wiggly and reflect conditional variance overfitting. Hence, we recalibrate $h_v$ using HNN's blocked OOB residuals by running
and then update the in-sample volatility such that
where $\hat{\varsigma}$ is the estimate of the scaling object $\varsigma= \mathrm{E}[\exp(\xi_t)]$. If there is a mismatch between $\nu$ and the new OOB residuals coming from HNN, $\zeta_0$ can adjust for it. $\zeta_1$'s role is to move accordingly and damper the overall variation in the final $\hat{h}_v$ in the event that the raw ${h}_v$ overfits. This is because $\hat{\varepsilon}_{t, \text{HNN}}^2$, coming from blocked subsampling, is a suitable approximation to the kind of prediction errors HNN will encounter in the real out-of-sample. Thus, if necessary, $\zeta_1$ acts as a raccord between $h_v$ and "reality".
The above operations can be seen as a direct neural translation of wooldridge2015introductory's Section 8.4 on weighted least squares (WLS). Note that the constant $\hat{\varsigma}$ is not part of Wooldridge's textbook because only relative (observation) weights are needed for the WLS application. In our case, we need an absolute metric and $\mathrm{E}[\exp(\xi_t)]$ is not merely equal to 1 as a result of $\exp()$ being a nonlinear function and $\xi_t$ likely being non-normal. We sample with replacement from the vector of $\hat{\xi}_t = \log \left(\hat{\varepsilon}_{t, \text{HNN}}^2\right)-\hat{\delta}_t$ to estimate the expectation. For out-of-sample volatility predictions, we thus use $\hat{h}_v(\mathbf{X}_t^{\text{test}}; \left[ \hat{\theta}_v, \hat{\zeta_0},\hat{\zeta_1}, \hat{\varsigma} \right])$.
\vskip 0.15cm
{ \sc Hyperparameters.} For each hemisphere we estimate a standard feed-forward fully connected network, which features two hidden layers ($\texttt{layers} = 2$). The same holds for the common block at the entrance of the network. Moreover, each layer (common or not) is given $\texttt{neurons} = 400$. We choose the ReLU activation function ($\operatorname{ReLU}(x)=\max \{0, x\}$) throughout the hidden layers and define a linear activation function for the output of $h_m$. To prevent the error variance from being negative, a natural choice for the output activation function for $h_v$ is the Softplus function ($\operatorname{Softplus}(x)=\log \left(1+\exp \left(x\right)\right)$), which imposes these bounds (${h}_v(\mathbf{X}_t;\theta_v) > 0 \enskip \forall t$) and is, in effect, a soft ReLU.
Hyperparameters for the optimization of the algorithm are set as follows. The maximum number of epochs is 100 and the learning rate is 0.001. Similar to HNN, we perform early stopping by using only a subset (80%) of the training sample for the estimation of the parameters and determine with the remaining set (i.e., 20%) when to stop the optimization. We set $B=1000$. The patience parameter in early stopping is 15 epochs. As a form of ridge regularization on network weights, early stopping may improve the efficiency of the algorithm and prevents the network from overfitting raskutti2014early. In addition, we apply dropout with a dropout rate of 0.2. We use the Adam optimizer and choose the whole sample for the batch size. Network weights $w_m$ and $w_v$ are initialized using $\mathcal{N}\left(0, {\small \sfrac{3}{100}} \right)$. Those choices are common to all target variables.
We test our proposed approach by modeling and forecasting key macroeconomic and financial variables of the US economy. We base our analysis on the FRED-QD database of mccracken2020fred, which is available on a quarterly frequency and features 248 US macroeconomic and financial aggregates. Our sample ranges from 1960Q1 to 2022Q4. All variables but prices are transformed according to mccracken2020fred to achieve approximate stationarity.\footnote{\color{black} NONBORRES (Reserves of Depository institutions (Nonborrowed)), TOTRESNS (Reserves of Depository institutions (total)), GFDEBTNx (total public debt), and BOGMBASEREALx (real monetary base) have been dropped because of their large shift in scale between in- and out-of-sample. While estimation and predictions were robust to their inclusion (by putting a very small weight on those variable), out-of-sample variable importance metrics were affected (see anatomy for further discussion on this issue in the context of Shapley Values). } Prices are in log first differences (inflation rate) rather than second differences (acceleration rate). All predictors are standardized to have zero mean and unit variance which is necessary for NN-based models and redundant for the others. We include two lags for each variable $X_{t,k}$ and add 100 linear trends to the set of covariates allowing for exogenous slow time variation in the parameters, and approximate trend filtering of the residuals à la stochastic volatility if the DGP requires so. Missing values at the beginning of the training sample are imputed using the EM algorithm of stock1999forecasting.
The target variables are GDP growth, change in the unemployment rate, headline CPI inflation, housing starts growth as well as S&P 500 stock returns. For each of them, we compute the one-step and four steps ahead predictive mean and variance for our hold-out sample starting in 2007Q1 and ending 2022Q4. NN-based models are re-estimated every two years whereas standard models are updated every quarter, all on an expanding window basis. Our forecasting exercise is based on a pseudo-out-of-sample analysis, which does not account for ragged edges or revisions in the underlying data set. Since we deal with large, dense models and none of the NN-based models put a disproportionate weight on a few indicators not available in real time, extending to a real-time exercise will not entail significant deviations from the results presented.
We explore the performance of the HNN by comparing the results to a set of competing models. This set is comprised of simple linear benchmarks including AR processes with SV and GARCH (AR$_\text{SV}$ and AR$_\text{G}$) as well as a high-dimensional Bayesian linear regression endowed with shrinkage and SV (BLR). In terms of nonlinear modeling choices, we consider standard neural network specifications (NN$_\text{SV}$ and NN$_\text{G}$), Bayesian additive regression trees (BART, see BART) as well as Amazon's DeepAR salinas2020deepar. Details on the implementation of the benchmark models can be found in Appendix (ref). NN$_\text{SV}$ and NN$_\text{G}$ use the same architecture as HNN's conditional mean, but are trained by minimizing the usual squared errors and the volatility processes are fitted in a second step on the resulting out-of-bag residuals. Those plain NNs allow to directly assess the relevance of a data-rich and densely parameterized nonlinear volatility function, and document HNN's proactivity versus standard approaches for often similar conditional means. Lastly, those two NN benchmarks allow to quantify the various merits of modeling jointly the first two conditional moments. As discussed in Section (ref), it is not difficult to think of DGPs where this could make a sizable difference, but knowing in what terrain we are standing is inevitably an empirical question.
The rest of this rather rich set of competitors allows us to span the space of relevant one-shot deviations from our framework. First, comparing the results to linear models sheds light on whether modeling nonlinearities pays off for macroeconomic point and density forecasting. Second, BART and DeepAR are the natural go-to nonlinear, nonparametric ML tools providing density forecasts. Tree ensembles are always very stubborn benchmarks for learning tasks with tabular data, and BART provides a probabilistic extension of boosting that produces natively density forecasts. DeepAR's architecture resembles that of a very crude HNN where there is only a common (LSTM) core, no hemispheres, and all remaining ingredients of Section (ref) have been dropped. Accordingly, performance differentials with HNN will procure a rough estimate of the (non-)marginal benefits of those propositions.
For each of our six target variables, we evaluate compactly the resulting point forecasts using the root mean square error (RMSE), the probabilistic forecasting accuracy by means of the log score ($\mathcal{L}$) and the share of variation explained in residuals' magnitude via the $R^2_{|\varepsilon_t|}$ of absolute residuals. For the out-of-sample (OOS) forecasted values at time $t$ for $s \in \{1,4\}$ we compute:
where $\varepsilon_{t,s} = y_{t+s}-\hat{y}_{t,s}$, $\eta$ is the standard deviation of the in-sample residuals, and $\varphi (. \phantom{.}; \hat{\sigma}_{t,s})$ is a normal density with zero mean and standard deviation $\hat{\sigma}_{t,s}$. While exotic in appearance, this last metric is only the out-of-sample goodness of fit \textcolor{black}{ of what would be the second stage regression in a weighted least squares problem.} Surely, it does not have all the qualities of other scoring rules and $|\varepsilon_t|$ is not exactly realized volatility, yet it arguably provides a metric that is much easier to interpret quantitatively. Lastly, it must be interpreted with care: a model can reach a high $R^2_{|\varepsilon_t|}$ because it has a failing conditional mean and the unexploited predictability flows in $|\varepsilon_t|$. Thus, a sufficient condition for safely gazing at the $R^2_{|\varepsilon_t|}$ of a particular model is for it to also have a low RMSE. Intuitively, a high-performing model with a fine $\mathcal{L}$ will have both a low RMSE and a high $R^2_{|\varepsilon_t|}$. Moreover, in Section (ref) we present additional density forecasting measures, which are the continuous ranked probabilitiy score (CRPS) and the coverage rate (68%), and assess model calibration using probability integral transforms (PITs).
We report evaluation metrics for the full test sample, as well as a subsample that ends prior to the Pandemic Recession. Given the unpredictable and unprecedented wild swings of 2020, those observations are always discarded for the real activity targets. In the interest of space and since AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$ and AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt G}$ often yield very similar results out-of-sample, we only report the best of the two according to $\mathcal{L}$ for each target/out-of-sample pair. All reported RMSEs are ratios with respect to that of the OLS-based AR(2). \textcolor{black}{Preferred models are those with low values in terms of RMSE and $\mathcal{L}$ and high values for $R^2_{|\varepsilon_t|}$.}
We report the main forecasting results through a series of dashboards for selected targets featuring essential statistics and visualizations. \textcolor{black}{This detailed analysis is conducted target by target to assess the individual performance and to provide some basic economic reasoning.} We compare HNN's conditional mean and volatility paths to that of selected benchmarks in the upper left and upper right panel of Figure (ref) to Figure (ref). We plot in-sample estimates up to the start of our hold-out (i.e., up to 2007Q1) and the recursively re-estimated out-of-sample ones thereafter (i.e., from 2007Q1 to 2022Q4). Our analysis is complemented by investigating the main drivers of the mean and the variance hemisphere. For this purpose, we measure Variable Importance (VI) as in MRF and HNN, which is itself inspired from what is traditionally done to interpret Random Forests breiman2001. Details are given in Appendix (ref). Additional results for leftover quarterly targets ($s=4$ cases and unemployment) can be found in Appendix (ref).
In general, HNN exhibits remarkable adaptability when faced with a diverse range of series. It adeptly captures the Great Moderation pattern in real activity variables, manages to produce a more "spiky" volatility pattern for the S&P 500, and sometimes exhibits behavior more akin to a smoothly switching process than to SV for predicting macroeconomic variables at longer horizons. These higher volatility periods demonstrate a proactive nature by often preceding significant prediction errors. This behavior is observable both in-sample, with out-of-bag estimates, and out-of-sample. In terms of performance, HNN consistently ranks among the top models across all metrics. Moreover, it does not experience substantial undercoverage, which occasionally plague other sophisticated competing models.
For the one-step ahead predictions of GDP growth HNN clearly outperforms all benchmarks. This finding holds for point and density forecasts as well as for both samples (see Figure (ref)). Considering the sample ending in 2019Q4 we find that all nonlinear techniques yield high predictive power with HNN giving the lowest RMSE and the lowest log score ($\mathcal{L}$). Moreover, HNN gives the highest $R^2_{|\varepsilon_t|}$ at 30%, distancing the nearest competitors NN$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$ and NN$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt G}$ by about 10 percentage points, and BART/DeepAR by more than 20. Similar findings are obtained from including Covid-19 pandemic observations. Coupled with the good forecasting performance, this implies that our model captures a substantial part of the "realized volatility". Extending the hold-out to the end of 2022 reveals that HNN also yields a good performance after the Covid-19 pandemic whereas other models (especially, BART and BLR) lose ground against the linear benchmark.
The right panel of Figure (ref) nicely demonstrates the reactive and proactive behavior of HNN's volatility hemisphere. The variance path increases at early stages of turmoil despite accurate predictions in previous periods, and is thus better prepared to receive larger errors than the green line corresponding to the SV specification. In fact, HNN shoots up at about the same time as AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$, for which higher predictions errors have already been accumulating at that point. In line with adrian2019, we find that the conditional variance is affected by developments in financial markets whereas the predictive mean is driven by labor market variables as well as real activity measures such as imports, exports and manufacturers' new orders (see Figure (ref) in Appendix (ref)).
Figure (ref) shows the results for the one-year ahead prediction of GDP growth. Again, high-dimensional models (except for DeepAR) yield high predictive accuracy when focusing on the hold-out ending in 2019. BART gives the best point forecasting performance, closely followed by BLR and NN specifications. In terms of log scores, HNN and BART clearly outperfom the other models. Moreover, $R^2_{|\varepsilon_t|}$ shows that HNN explains nearly a third of the realized volatility similar to NN$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt G}$. When including the periods after 2020, HNN beats all its linear and nonlinear competitors with respect to density forecasting performance. While alternative NN specifications perform rather poorly at the end of 2021 and the beginning of 2022, HNN yields highly competitive predictions and acknowledges the elevated uncertainty until the end of the sample.
Similar to the one-step ahead case, the volatility hemisphere shows proactive tendencies. The conditional variance picks up the uncertainty in the underlying data set early, resulting in superior density predictions. During the Great Moderation and in the periods after the Global Financial Crisis the variance is low and stable, narrowing the predictive distribution to rather certain estimates. Note that this also holds for the one-step ahead case. Again, variables measuring financial conditions are important drivers of the mean and the variance hemisphere. These include debt-to-income ratios of several sectors in the US economy as well as real disposable business income (see Figure (ref) in the appendix). As shown by adrian2019, the impact of financial conditions on GDP growth seems to be robust at multiple horizons. Similarly, de2017forecasting and amburgey2023real, amongst others, emphasize the importance of financial conditions on real activity, especially in the tails.
When interest centers on one-step ahead inflation predictions (see Figure (ref)) we find that our neural network models yield high forecasting performance for both point and density predictions as well as both samples. HNN outperforms all other models with respect to point forecasting performance in terms of RMSE and NN$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt G}$ gives the lowest log score for density performance, closely followed by HNN. A similar pattern is observed when extending the sample to the end of 2020. Noteworthy, we see that the HNN overestimates the effects of the Covid-19 pandemic in its mean estimate which is, however, accompanied by a large variance, implying that the model acknowledges the unprecedentedly high uncertainty involved. This nearly completely discounts the dramatic deflation forecast and results in a highly competitive $\mathcal{L}$. During this time, the variance hemisphere is mainly driven by employment variables and money stock which were heavily affected by the Covid-19 shock and exhibited major fluctuations in 2020 (see Figure (ref) in the appendix). BART gives the worst log scores compared to the other models as it tends to underestimate the variance during most periods.
Given the policy needs for interpretable inflation forecasts, we extend the HNN to a more structural approach proposed in HNN. Relying on a nonlinear Phillips curve specification, the architecture of the neural network is designed to provide among other things a measurement of economic slack and inflation expectations. As will be shown in Section (ref), the Neural Phillips Curve (NPC) model equipped with a mean and a variance hemisphere predicts inflation reasonably well and substantially outperforms its competitors.
Results for the quarterly forecasts of the S&P 500 presented in Figure (ref) show that our HNN outperforms all competitors in terms of density predictions and yields highly competitive point forecasts following BART and AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt G}$. Moreover, HNN explains almost a third of the variance of absolute residuals, similar to the DeepAR, but with allegedly much less "leftover conditional mean predictability" in it given DeepAR's higher RMSE.
When comparing the conditional volatility estimated by the set of models, we see that the predictive variance of HNN follows a different pattern than those of the other models. Our proposed approach attaches higher weight to macro uncertainty and gives a countercyclical variance path (see the upper left panel of Figure (ref)). Since the architecture of HNN allows for proactive and reactive volatility, the predictive variance takes into account signals from the input data set while at the same time accommodates for various forms of time variation. This way, it offers great flexibility going beyond the reactive structure of GARCH and SV processes and accounts for nonlinear relations between the covariates and the target. It relates to the strand of literature exploring the predictive power of exogenous variables for forecasting stock market volatility campbell2009stock,paye2012deja,engle2013stock,guidolin2021boosting,ma2022stock and sheds light on the economic sources of the volatility process. We find that the variables shaping the variance hemisphere are related to both, financial and economic conditions. Main drivers are variables closely moving with the economic business cycle, such as new housing permits, imports and employment, but also variables measuring financial conditions and stock market variables (see Figure (ref) in the appendix).
Turning to the short-term predictions of housing starts, which is presented in Figure (ref), we see a remarkable performance of the neural network models for the periods during and after the Covid-19 pandemic. In terms of density forecasts, HNN's predictive accuracy is only challenged by the highly competitive performance of AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$. While BLR outperforms all competitors for point and density predictions before 2020, controlling for nonlinearities gains in importance thereafter. All nonlinear models catch up on the AR benchmark with HNN yielding the lowest RMSE and a $\mathcal{L}$, which is comparable to AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$. Moreover, our proposed model explains about 10 % of the realized volatility measured by the $R^2_{|\varepsilon_t|}$ of absolute residuals, following AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$ which, however, gives a substantially higher RMSE.
Visual inspection of the conditional mean of the HNN (see the upper right panel of Figure (ref)) reveals some noteworthy patterns for the observations during the Covid-19 pandemic in 2020. Even though HNN underestimates the unprecedented downturn in the first quarter of 2020, it manages to take advantage of the signals provided by the unconventional behavior of various variables in the set of covariates for more accurate point and density forecasts than its competitors in the following periods. This raises the question: what drives this? For both hemispheres, we find that financial conditions have high predictive power. Moreover, new housing permits as well as commodity price developments (in particular, metals and fuels) play an important role (see Figure (ref) in the appendix).
Given the remarkably consistent performance of the HNN's density predictions, we challenge these results by adding evaluation metrics including the continuous ranked probability score (CRPS), the 68 % coverage rate and a PIT-based test for auto-calibration knuppel2022score. First, we compute the CRPS introduced by gneiting2007strictly, which is a proper scoring rule for predictive distributions and enjoys the advantage of being less sensitive to outliers. Let $F$ denote the cumulative distribution function and $\mathfrak{f}$ the predictive density with $\hat{y}_{t,s}$ and $\hat{y}'_{t,s}$ being independent random draws from the predictive density. The CRPS is then defined as $$\text{CRPS}_{t,s}(y_{t,s}) = \int_{-\infty}^{\infty} (F(z) - \mathbbm{1}\{y_{t,s} \leq z \})^2 dz = E_{\mathfrak{f}} \lvert \hat{y}_{t,s} - y_{t,s} \rvert - 0.5 E_{\mathfrak{f}} \lvert \hat{y}_{t,s} - \hat{y}'_{t,s} \rvert,$$ where $\mathbbm{1}\{y_{t,s} \leq z \}$ defines an indicator function, which returns the value 1 if $y_{t,s} \leq z$ and 0 otherwise. In the figure below we report the CRPS averaged across the hold-out relative to our AR benchmark.
Second, we consider the nominal coverage rate, which measures the frequency of the forecasts falling inside a specific interval. The predictive density is considered too wide (narrow), if the realized frequency exceeds (drops below) the nominal level chosen for the interval. Formally, this boils down to $$I_{t,s}^{\gamma} =
$$ where $L_{t,s}^{\gamma}$ and $H_{t,s}^{\gamma}$ are the lower and upper limits of the interval. We compare the relative frequency of interval hits, $\hat{\gamma}_{s} = \frac{1}{\#OOS}\sum_{t \in OOS} I_{t,s}^{\gamma}$, to the pre-specified coverage rate $\gamma$, which we set to 68 %.
The last metrics tests for auto-calibration based on probability integral transforms (PITs). Following knuppel2022score we use the PIT of the implied forecast distribution based on the energy score, which is given by $${U}_{ES,t,s} = \mathcal{P}_{\mathfrak{f}} \left( E_{\mathfrak{f}} || \hat{y}_{t,s}-\hat{y}'_{t,s} || \leq E_{\mathfrak{f}} || \hat{y}_{t,s}-y_{t,s} || \right),$$ where $|| \cdot ||$ gives the Eucledian distance and $\mathcal{P}_{\mathfrak{f}}$ and $E_{\mathfrak{f}}$ the probability and the expected value under the forecast distribution, respectively. We then test for standard uniformity of $\{U_{ES,t}\}_{t=1}^T$. A model is said to be well calibrated if we do not reject the null hypothesis of auto-calibration. This implies that regardless of any further transformations the forecast distribution will not improve.
Figure (ref) presents the additional scores for both evaluation samples. The left panels focus on the sample ending before the Covid-19 pandemic (i.e., 2007Q1 to 2019Q4) whereas the right panels present the full sample results (i.e., 2007Q1 to 2022Q4). Overall, the results confirm the promising performance of HNN. The relative CRPS shows substantial improvements of our approach against the AR model for most targets. The coverage rate is close to the selected level (which is 68 %) and shows no tendency of structurally underestimating the variance. Unlike other models, the HNN shows no evidence against auto-calibration.
Analyzing each metric in more details gives some interesting insights. Comparing the relative CRPS of the HNN with other nonlinear models underlines its remarkable density forecasting performance. HNN outperforms the linear benchmark in all cases except for high-order forecasts of housing starts and one-step ahead inflation forecasts when considering the full sample. Largest gains compared to the AR benchmark can be found for the unemployment rate. This holds for all models. In the case of HNN, this is attributable to excellent point and density forecasts (almost 30% reduction in RMSE for both horizons combined with a very high $R^2_{|\varepsilon_t|}$ for $h=4$ in Table (ref)). BART often yields competitive results except for inflation. DeepAR and BLR often show similar density forecasting performance to the AR benchmark, yielding scores close to 1. The NN models with time-varying volatility are often worse and only sometimes better (e.g., for unemployment) than linear models.
The coverage rate shows substantial underestimation of the variance for BART and DeepAR. The predictive densities of BART and DeepAR are too narrow for most targets and both evaluation samples. AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$ gives a rather mixed picture with density predictions of GPD growth and inflation one-step ahead being too wide while higher-order densities being too narrow. HNN, on the other hand, gives ratios close to the 68% level. For two target variables (i.e., GDP growth and four steps ahead unemployment), we get densities that are rather wide whereas the predictive distributions for housing starts and short-term inflation show slight underestimation of the variance. AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt G}$ as well as BLR tend to give wide predictive distributions with coverage ratios close to 80 % or above. Similarly, the NNs with SV or GARCH yield rather wide densities.
The test results for auto-calibration, presented in the last two panels in Figure (ref), suggest that the models are well calibrated for most targets with the exception of DeepAR. In this case, the test clearly rejects auto-calibration for inflation and housing starts regardless of the forecast horizon and GDP growth for four steps ahead predictions. When considering the full sample, this also holds for the unemployment rate. The other competitors give low p-values for at least one or two of the estimated target variables and even more when focusing on the full sample. HNN, on the other hand, shows auto-calibrated results for all targets when evaluating the full sample and all but one target (i.e., GDP four steps ahead at the 10% level) when considering the periods before the Covid-19 pandemic.
Some results from the previous section warrant a digression from our main thread of investigation. As is particularly apparent from coverage results in Section (ref), but also from log scores throughout Section (ref) and additional results in Table (ref), the quality of BART and DeepAR's probabilistic forecasts is rather uneven. In both cases, point forecasts often rank very highly but $\mathcal{L}$ and other density evaluation metrics show clear signs of distress. While BART's problems are frequently contained to in-sample historical estimates (quite visible in Figures (ref) and (ref)), that of DeepAR are rather generalized. This section first describes the facts, and then provides suggestive explanations for the phenomena. Finally, we discuss what can be done in both cases to alleviate such substantial problems so that, hopefully, the operation produces some wisdom to draw from for future applications of such methods.
First, let us stress that it is certainly not excluded that an extensive amount of tuning for both could non-trivially improve the probabilistic performance of both approaches, but this is not what is typically seen in the literature BART,green2012modeling,linero2018bayesian. There are practical reasons, of course, but also statistical ones, like the instability of cross-validation in such environments, or that tuning hyperparameters is not exactly Bayesian. Lastly, we typically expect proper calibration (whether the conditional mean is very proficient or not) to be obtained independently of tuning. For instance, this is what we get from AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$, HNN, and NN$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$.
A particularly telling example is the following. HNN turns in similar outperformance for both GDP and unemployment at the two horizons under study---as one would rightfully expect from two strongly cross-correlated targets. While HNN is decisively superior for both targets at $s=1$, BART and HNN yield a nearly identical (and stellar) log score for GDP ($s=4$). Yet, BART ranks last among all models in terms of log score for Unemployment Rate ($s=4$) whereas HNN reaches gains similar to GDP ($s=4$). Rarely does it hurt to look at the data underlying summary statistics, and we use the case of unemployment at $s=4$ to guide the following discussion. Moreover, it is a target for which the inherent "true" uncertainty is manifest.
Figure (ref) reports times series corresponding to the second panel of Table (ref) in Appendix (ref) where BART and DeepAR are reported to have fine RMSEs with dismal log scores. We compare conditional means of HNN, BART and DeepAR to the realized value in the left panel and show conditional volatility in the right panel. In-sample, BART's and DeepAR's fitted values nearly perfectly overlap with the realized ones, and are suggestive of an unrealistically good predictive ability. Accordingly, BART estimates a very low volatility path with little fluctuations throughout these periods. DeepAR inconsistently estimates the general volatility level to be at roughly the same level as HNN (for this specific case) and thus reports massive over-coverage in-sample. While BART's volatility estimates seem to be reasonable for the hold-out sample, those of DeepAR follow a strange path. As mentioned above, this behavior is not exceptional to this case, but is recurrent. For instance, a similar behavior is observed for S&P 500 (Figure (ref)) as well as housing starts (Figure (ref)) where BART reports noticeably lower estimates of average volatility than either AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$, AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt G}$ or HNN. This suggests probabilistic forecasts of these models often lack appropriate levels of uncertainty and historical analysis based on in-sample results, which are frequently conducted in macroeconometrics, may provoke misleading implications and, hence, should be interpreted with care.
What is causing this? In short, benign overfitting is, as the name suggests, benign for the conditional mean (out-of-sample). In our results, BART and DeepAR exhibit this phenomenon that has been described for neural networks belkin2019reconciling and tree ensembles MSoRF. Without further precautions, this overfitting is, however, malign for the conditional variance. The discussion of Ingredients 2 and 3 in Section (ref) already alluded that such problems would arise if not addressed directly, which we do for HNN by introducing the volatility emphasis parameter $\nu$ and using blocked subsampling to recalibrate the variance hemisphere. In practical terms, if one is only interested in minimizing out-of-sample RMSEs, the near-perfect in-sample fits attained by BART and DeepAR can safely be ignored. However, when it comes to deeper investigations, such as uncertainty quantification or in-sample analysis, the best course of action is to tread lightly. At this juncture, it becomes preferable to inspect separately the two models.
The overfitting issues of BART are more blatant in-sample. Hence, our results suggest that one should be careful using BART estimates in-sample to draw any economic conclusion even if BART yields the best (point or density) forecasts results out-of-sample. Note that the phenomenon is even more pronounced at the monthly frequency where pure noise is prevalent. Given that BART provides a reasonably convenient environment to go beyond mere conditional mean modeling and think about more structural objects (like some kind of time-varying parameters or any latent states), it is important to have reliable historical estimates. In the literature, we find possible solutions tailored to Random Forests, for which MSoRF finds a similar pattern, such as using blocked out-of-bag quantities HNN,MACE. However, out-of-bag sampling appears computationally unfeasible and would change the meaning of the Bayesian setup. More promising is some extensive case-by-case empirical tuning of the level of volatility prior. Again, some careful thinking is necessary about how such tuning should be conducted as BART will also have a preference for overly parameterized models in a pseudo-out-of-sample setup similar to what we see for out-of-sample. A possibility is to use an auxiliary model immune to such complication, like an autoregressive process. Or, when a preferred BART specification is chosen, to increase such priors as long as the out-of-sample fit is mostly intact -- analogous to reducing the unnecessary depth of trees in a Random Forest MSoRF. Note that, while not directly addressed here, some of BART's in-sample overconfidence spills out on the test sample mostly for calibration metrics, making it not as reliable as HNN or AR$_\text{\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\@setfontsize\halftiny\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt\@vipt\@viipt SV}$. Some of the aforementioned solutions could likely help in that regard.
Regarding DeepAR, one could legitimately presume that early stopping and dropout could help avoiding the perfect in-sample fit seen in our results. Yet, the double descent phenomenon and associated neural networks oddities often make it difficult to use traditional regularization intuition as guidance. Moreover, as we have noted in Section (ref), unconstrained fully nonparametric models of mean and variance can overfit using either moment and the allocation between the two will depend on a mostly unknown mapping between obscure architecture choices and final results. In further (unreported) experiments, adjusting the number of layers and neurons can sometimes help or hurt, and in a mostly unpredictable way, i.e., there is no clear mapping between the total number of neurons and density forecasting underperformance. The regularities are rather that (i) RMSEs are only remotely affected by such choices, (ii) in-sample nominal coverage is often extremely high, and (iii) out-of-sample coverage varies greatly but primarily on the low end. The most promising way of proceeding seems to be cross-validation based on blocked pseudo-out-of-sample density forecasting evaluation metrics. However, this would entail an unfeasible computational burden and, given the small sample size, probably places excessively high expectations on the power of cross-validation.
Recurrent neural networks are specifically designed to process sequential data rumelhart1986learning,qin2017dual. By keeping an internal memory state, which is used as additional input at each time step, RNNs capture patterns and dependencies over time. That is, RNNs receive information from two sources: external shocks to the system and the internal state from previous periods and as such, mimic the structure of a (G)ARCH process.
While RNNs have become popular in various domains such as natural language processing or speech recognition young2018rnn, they also come with limitations. Due to their recurrent nature and sequential processing, training can get computationally expensive, especially for long time series. Even more troublesome, RNNs are susceptible to vanishing or exploding gradients. To address these challenges, we implement a LSTM network hochreiter1997lstm, which uses a gating mechanism to filter pertinent information, and restrict each hemisphere to use only one recurrent layer, effectively reducing the depth by half compared to the original architecture. All other hyperparameter choices are unchanged (see Section (ref)).
As is evident from Table (ref) in Appendix (ref), there is no need for taking on the burden of endowing HNN with a recurrent (LSTM) structure. Across all targets we find that gains from HNN-LSTM are either small or nonexistent. For point forecasts we get very similar results from both model specifications. HNN yields lower RMSEs in most cases, regardless of the forecast horizon. In cases where HNN-LSTM beats our standard specification, it is by very small margins. The only exception is the one-step ahead prediction of inflation and housing starts when considering the full sample. This difference in performance for inflation, however, can be diminished when considering the structural approach presented in Section (ref). When focusing on density predictions, HNN yields better forecasting accuracy as its recurrent counterpart. We conclude that we can easily extend our proposed model to more complex types of neural networks, which yield, however, very similar results at the cost of higher computational burden.
Since all benchmarks considered so far estimate the variance process reactively, we expand our set of competitors by quantile regressions, which feature proactivity. We include a linear Bayesian quantile regression (BQR) with shrinkage, a quantile version of BART (QBART) as well as of the AR(2) model (QAR). By estimating different quantiles of the predictive distribution based on the input matrix $\bm X_t$, quantile regressions directly and proactively model the uncertainty surrounding the response variable. This makes them a fair but hard-to-beat benchmark. However, estimating multiple quantiles for each target and horizon adds complexity and computational burden to our exercise. Besides rather statistical phenomena such as quantile-crossing bassett1982quantile, which describes the lack of monotonicity when estimating conditional quantile functions, results may not have a straightforward interpretation in some cases. Consider, for example, the Neural Phillips Curve model with proactive volatility. Its extension to quantile regression would entail the estimation of an output gap measure for each quantile and, thus, raise the question of how to interpret the meaning of the resulting slack variables.
We evaluate the (tail) forecasting accuracy of all models using log scores and quantile-weighted continuous ranked probability score (CRPS$_{\omega}$). The weights are set such that the metrics allows for analyzing downside risks via the left tail and upside risks via the right tail of the distribution. To complete our analysis we also check the forecasting performance with respect to the center of the distribution.\footnote{Unweighted CRPS for HNN and the main benchmark models can be found in Figure (ref).} For details on model specification and implementation of the quantile regression approaches as well as the additional evaluation metrics we refer to Appendix (ref). Results are presented in Table (ref) and Table (ref) in the appendix.
We find that HNN remains highly competitive when investigating the predictive distribution in more detail and comparing its performance to quantile regression approaches. For real activity targets HNN either ranks first or is very close to the best performing model for both horizons, both samples and in each part of the distribution. This is remarkable since non-normality and, in particular, asymmetry of conditional distribution have become a major focal point of applied macroeconometric research. In a well-known article, adrian2019 show that when it comes to estimating the conditional distribution of GDP, quantile regressions perform well because the resulting distributions are left-skewed in recessionary periods and closer to symmetry during expansion. We find that, even though HNN builds upon the usual normality assumption, its sophisticated mean and variance functions provide the necessary flexibility to adjust and capture dynamics in the tails. HNN yields results very close to the Bayesian quantile regression for one-step ahead GDP growth and even outperforms it when considering the full sample. For four steps ahead, QBART tops the list of competing models but is again closely followed by HNN, especially when including the observations of the Covid-19 pandemic. Therefore, HNN's results are always in the ballpark of the (ex-post) best quantile regression model.
Turning to the results for the remaining targets, HNN outperforms all competitors for the unemployment rate one-step ahead and for four steps ahead when evaluating the sample up to the Covid-19 pandemic. Here, we find substantial gains in the tails as well as the center of the distribution. For higher-order forecasts of the unemployment rate (when considering the full sample) HNN yields highly competitive results compared to QBART. Similarly, QBART turns out to be the main competitor for higher-order density predictions of inflation. For the one-step ahead case we often find strong performance of BQR (see, e.g., GDP growth, housing starts, S&P 500), except for inflation where HNN yields the lowest CRPS. For S&P 500, especially for higher-order forecasts, BART remains the best performing model. Even though the Bayesian models, either plain or as quantile extensions, turn out to be very difficult to beat, HNN follows closely and thus, captures upside and downside risks to a similar extent.
Given their high importance for economic policy decisions in central banks and governmental institutions, inflation forecasts should be decent and preferably interpretable through some basic macroeconomic reasoning. No less important is to be aware of the level of uncertainty associated with a specific inflation forecast. As we have seen, HNN is a promising tool that manages to incorporate large amounts of data and captures nonlinearities via its sophisticated mean and variance specification. However, the anatomy of $h_m$ remains mostly unknown. In this section, we achieve both goals by bringing back interpretability of the conditional mean for this particular target and, at the same time, modeling the time-varying level of decency with the variance hemisphere. The resulting model, of appreciable architectural complexity, is HNN's Neural Phillips Curve embedded within this paper's probabilistic forecasting methodology (henceforth, HNN-NPC). Precisely, we impose a Phillips Curve structure on the fully nonparametric $h_m$ in (ref), resulting in
where $\boldsymbol{X}_t^{i}$ with $i \in \{ \mathcal{E}_\text{LR}, \mathcal{E}_{\text{SR}}, g, c \}$ being the subsets of columns that correspond, respectively, to long-run expectations, short-run expectations, "output gap", and commodity prices hemispheres. Their definitions in terms of FRED-QD are identical to that of the original paper. The exact list of mnemonics can be found in Appendix (ref). The new hemispheric structure of the conditional mean and its goal to gain interpretability implies the need to drop the common core at the entrance of the network. Regarding the volatility process, this results in a more structured one-way directional flow of $h_m$ into $h_v$. As there is no overwhelming theoretical reason to constrain the volatility process to solely rely on PC-inspired inputs we modify $h_v$ in (ref) for
where $h_{\tilde{v}}(\boldsymbol{X}_t;\theta_{\tilde{v}})$ is a subnetwork that serves in processing all variables of our high-dimensional data set to extract a time series for $h_v^{\text{NPC}}$ that carries relevant signals for the conditional variance that are not captured by the four series summing up to the conditional mean. All subnetworks (i.e., $h_{\tilde{v}}$ and the four subnetworks of the conditional mean) precede $h_v^{\text{NPC}}$. Figure (ref) summarizes the structure of HNN-NPC.
Since conditional mean hemispheres contain an unequal number of predictors, leading to unequal a priori importance assigned to each one, we rescale data as proposed in HNN. Precisely, we divide the scaled predictors of $\boldsymbol{X}^{i}$ by $\sqrt{\frac{{\#\text{\phantom{.}columns}(\boldsymbol{X}^{i})}}{\#\text{\phantom{.}columns}(\boldsymbol{X})}}$. This scheme is not necessary for the $h_{\tilde{v}}$ subnetwork because it takes the whole $\boldsymbol{X}$ and its relative influence is only guided by its total number of neurons and the volatility emphasis parameter $\nu$. Since the number of effective parameters is at least four times superior in the conditional mean subnetwork, the $\boldsymbol{X}$'s entering $h_{\tilde{v}}$ are multiplied by a factor of 5. Other hyperparameters are the same as described in Section (ref) except that each hemisphere now consists of 200 neurons and $\nu$ is now obtained from the blocked out-of-bag errors of HNN's plain NPC model.
First, we can compare whether the incoming restrictions hurt or improve predictive ability both in terms of RMSE and probabilistic forecasting metrics. Table (ref) in the appendix shows promising results with the more "specialized" HNN-NPC improving in a non-trivial fashion nearly all performance metrics including or excluding post-2020 data. This is not completely jarring because (i) relevant restrictions will bring down variance more than they incur bias and (ii) distilling FRED-QD to include only relevant variables (according to loose macroeconomic theory) can significantly improve the performance of an otherwise dense conditional mean model.
The first row of Figure (ref) reports the usual conditional mean and volatility, and now compares with a small Bayesian model of similar aim. Indeed, chan2016 provide a model of trend inflation with a Phillips Curve (with unemployment as forcing variable), drifting coefficients, and stochastic volatility. The first panel highlights that HNN-NPC fares particularly well during two turbulent eras by (i) avoiding the missing disinflation following the Great Recession and (ii) capturing a non-trivial part of the 2021-2023 surge in inflation. In contrast, CKP exhibit the typical observation that traditional PC-based forecasts were not only reactive rather than proactive during recent years, but also consistently "biased" downwards up until 2023. Regarding volatility estimates, HNN-NPC's proactive behavior is apparent during the Great Recession, with $h_{v,t}^{\text{NPC}}$ rising from its bed a few quarters before the SV process embedded in CKP---the latter erupts following the 2008 oil price crash and then apparently diffuses its effects for almost three years. Both volatility estimates spike following the initial Covid-19 crash. The HNN-NPC spike is particularly pronounced and explains why it yields the best log score despite a highly inaccurate deflation forecast in 2020Q2: the confidence interval for that particular point in time is so large it also includes positive inflation. As such, the erroneous point forecast is discounted in the probabilistic performance metrics since HNN-NPC practically "knew" it was turning in a forecast of extremely low reliability. The massive uncertainty quickly dissipates to a more reasonable level (comparable to that of post-Great Recession), then hits a low point before picking up again following the invasion of Ukraine.
For additional comparison, we also report contributions from a canonical PC regression in the second row of Figure (ref). In the case of "PC", those are constructed from a traditional PC specification -- including two lags of inflation and the Congressional Budget Office (CBO) estimate of the output gap -- with time-varying coefficients obtained from GC2019's two-steps ridge regression approach. As discussed in HNN, the key statistical distinction between HNN-based inflation modeling and the two models included for reference purposes is the nonlinear processing of a rich real activity data set. The use of neural networks serves as a convenient way to achieve that goal within a "generalized" PC environment.
Looking at contributions in the second row, we reaffirm some key findings from HNN. Among other things, we get a rapid closing of $g_t$'s contribution to inflation post-Great Recession. This contrasts with CKP and PC which report more lasting downward pressures following 2008. The next observation is the strikingly different behavior of $h_{g,t}$ starting from last quarter of 2020. HNN-NPC sees $h_{g,t}$ contributing strongly to the highest quarterly inflation forecasts in a generation. The two benchmarks are not nearly as agitated in 2021 and 2022, and report forecasts and contributions that fit within the popular PC-based narrative in 2021 that inflation would be transitory.
From Figure (ref), we also get some additional insights from updating the data up to 2023Q2 HNN. First, we see that $g_t$ is still pushing forecasts above the target range, but its effect has massively shrunk from the highs of 2021, mostly starting from 2022. As of 2023Q2, the contribution of real activity to inflation as estimated by HNN-NPC is about twice as much as that from CKP, yet it is the closest they have been to agreement in the last three years. The contribution of expectations seems to slowly decrease to pre-pandemic levels, but is appreciably higher than that of CKP, which has already settled to rather low levels. HNN-NPC's and CKP's point forecasts for 2023Q2 are roughly similar (with CKP gaining in performance in 2023), and so is their latest assessment of volatility. This agreement aligns with the observation that their predictions mainly differ during a few localized episodes, and 2023Q2 falls outside of that. Usual disagreement can be traced back to their differing evaluation of economic slack and expectations, especially when those are far from their mean. Key disagreeing segments for those components are the missing disinflation post 2008 and non-transitory inflation surge of 2021-2023. However, as of 2023Q2 the unemployment gap of CKP has mostly caught up with mounting real activity pressures expressed by HNN, resulting in the recent relative concordance between the two models.
We provide a way to conduct density forecasting where both conditional mean and variance are the outputs from neural networks. Results show that in many cases, HNN picks up early signals of increased future volatility before the occurrence of large prediction errors. This proactive behavior often gives a significant advantage over stochastic volatility specifications that are frequently used to close linear and nonlinear macroeconomic forecasting models. Moreover, its nominal coverage and overall probabilistic forecasting performance is much more consistent across targets and experiments than what we found for two leading nonlinear nonparametric machine learning alternatives. Therefore, HNN is an effective new tool for density forecasting in itself and a convenient building block for deep-learning based macroeconomic models when timely uncertainty quantification is needed. Concerning the latter aspect, we provided such an application by merging this paper's architecture with that of HNN's Neural Phillips Curve. There are many possible extensions. A conceptually obvious yet very relevant one is that of a multivariate normal predictive density, providing a MLE-based alternative for the estimation of (possible large) nonlinear/time-varying vector autoregressions.
\setlength\bibsep{5pt}
\setstretch{0.75}