Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
91,072 characters · 15 sections · 74 citation commands
Adaptive Dynamic Model Averaging with an Application to House Price Forecasting
\def\spacingset#1{ {#1}} \spacingset{1}
\if00 \fi
\if10 {
} \fi
{\it Keywords:} Adaptive forgetting; Stochastic optimisation; Prediction with Expert Advice; Dynamic linear model; Housing market
\spacingset{1.45}
A growing empirical literature provides strong evidence in favour of structural instability in many macroeconomic relations StockW1996,KoopP2007,NgW2013. Structural instability is crucial because, if left unaccounted for, it can have detrimental consequences for statistical inference and forecasting ClementsH1998,PesaranPT2006,GiacominiR2009,rossi2013advances.
Dynamic Model Averaging (DMA) is an econometric methodology that can accommodate time variations in both model parameters and model specification RafteryDMA2010. This methodology has gained increasing popularity in recent years for predicting various economic variables, such as inflation KoopKorobilis_2012,CataniaNonejad_2018, carbon prices KoopTole_2013, exchange rates ByrneKR2018, equity returns DanglH2012, and property price growth BorkMoller_2015. DMA creates a Dynamic Linear Model (DLM) for every possible subset of predictors and combines the forecasts of these models using weights that adapt over time RafteryDMA2010. To adapt to changes in the data generating distribution, DMA involves two parameters, called forgetting factors. The first forgetting factor is part of the DLM formulation, while the second is involved in the model averaging phase. These parameters allow a continuous trade-off between estimation in a static environment, and re-initialising the estimation process by discarding all past information, which is appropriate after a structural break. It is therefore not surprising that the choice of forgetting factors is critical to the forecast performance of DMA.
Initial work by RafteryDMA2010 and KoopKorobilis_2012 considered both forgetting factors to be user-defined and constant. However, in general, the type of structural change in economic relationships is unknown and may vary considerably over time ChenH2012. Structural breaks due to changes in regulatory conditions, in the behaviour of consumers and firms or in the preferences of policy makers constitute prominent examples where periods during which the data generating process is static are interrupted by episodes of abrupt change PesaranPT2006,KAPETANIOS2010. In this setting, and more generally whenever the speed or type of change in the data generating process is not constant, there does not exist a single choice of forgetting factors that is optimal for the entire length of a time series.
The DMA formulation of DanglH2012, adopted by a number of more recent works CataniaNonejad_2018,ByrneKR2018 involves no forgetting in the model averaging stage, and treats the choice of the DLM forgetting factor as an additional dimension of model uncertainty. In particular, the user specifies a grid of forgetting factor values, and the posterior distribution of this parameter is updated at every time step by marginalising over all DLM specifications. The Bayesian approach of DanglH2012 effectively assumes that the appropriate choice of the DLM forgetting factor is constant over time, and identical across models. McCormick2012 propose a similar approach that involves a grid of values for both forgetting factors. They propose to use the forgetting factors that maximise the predictive likelihood for each DLM specification to avoid the computational cost of Bayesian updating.
In this paper, we develop an adaptive dynamic model averaging (ADMA) methodology that consists of two components. The first involves the use of stochastic optimisation to identify the forgetting factor that minimises the expected one-step-ahead squared forecast error of each DLM. This leads to a fully online and data-driven algorithm which we call Adaptive Forgetting DLM (AF-DLM). As we show in the experimental results section AF-DLM is effective under different types of change in the data generating process, including cases in which the speed or type of change is variable over time. AF-DLM is also computationally less expensive than previous approaches since it does not involve a grid of forgetting factor values.
The second component of ADMA deals with model averaging. We show that the speed with which DMA weights respond to more recent observations is determined not only by the choice of the corresponding forgetting factor, but also by the mechanism that prevents underflow (weights becoming equal to machine zero). Therefore any approach to control the adaptability of DMA weights by tuning only the second forgetting factor is inherently limited. We propose to replace the current model averaging approach with the ConfHedge model combination algorithm from the field of machine learning known as prediction with expert advice VyuginTrunov_2019. In addition to being parameter-free, ConfHedge is currently the only model combination algorithm whose one-step-ahead squared forecast error over a finite number of time steps is within a known bound of the one-step-ahead squared forecast error of the optimal sequence of forecasting models.
To assess the effectiveness of the proposed methodology, we provide an in-depth empirical evaluation of ADMA on the task of forecasting UK house prices. The motivation for this application is twofold. First, although the latest boom-bust episode in real estate markets and its decisive role in the Great Recession has generated a vast interest in the behavior of international housing markets, the academic literature on house price forecastability is relatively small (especially when compared to the literature on other assets such as stock prices and exchange rates), and mainly concentrates on the US market RapachStrauss_2009,GHYSELS20135,BorkMoller_2015. In the UK, similarly to the US, housing activities account for a large fraction of GDP and of households' expenditures, real estate property comprises the largest component of private wealth (excluding private pensions), and mortgage debt constitutes the main liability of households ONS_2018. Thus, accurate forecasts of UK real estate prices are crucial for private investors and policy makers. Second, recent empirical evidence suggests that the relationship between real estate valuations and conditioning macro and financial variables displays time-varying patterns Aizenman2014,Anundsen2015,Paul2018. This makes housing markets an ideal setting for the application of dynamic econometric models.
In summary, the results of our empirical application suggest that ADMA offers significant forecasting gains relative to a linear autoregressive (AR) benchmark, as well as a battery of competing dynamic and static forecasting models. They also indicate that the best house predictors substantially differ over time and across regions. In-sample stability tests also support the conclusion that the data generating process of regional UK house prices is subject to structural instability.
The remaining paper is organised as follows. Section (ref) presents the ADMA methodology. In Section (ref) we assess the proposed AF-DLM on simulated time series exhibiting different types of dynamics. Section (ref) is devoted to the comparative evaluation of ADMA against alternative forecasting models on the task of predicting UK regional house prices. The paper ends with concluding remarks in Section (ref).
This section is divided into three parts. The first part provides a brief outline of the DMA methodology. The second part deals with the development of the stochastic optimisation approach to sequentially adapt the forgetting factor in a single DLM. The last part discusses limitations of existing model combination methods and presents the ConfHedge algorithm.
For a set of $D$ covariates, DMA creates a DLM for each possible subset (excluding the empty set), giving rise to $K=2^D-1$ models, $M_1,\ldots,M_K$. Model $M_k$ is defined by,
where $\theta^{(k)}_t \in \mathbb{R}^d$ denotes the coefficient vector and $x^{(k)}_t \in \mathbb{R}^d$ is the covariate vector (including a constant) of $M_k$ at time $t$. Eq. ((ref)), known as the state-transition equation, determines the dynamics of the unobserved coefficient vector. Eq. ((ref)), called the observation or measurement equation, links the response, $y_{t+1}$, to the coefficients and the covariates. The errors, or noise terms, $\omega^{(k)}_{t+1}$ and $\varepsilon^{(k)}_{t+1}$ in Eqs. ((ref)) and ((ref)), respectively, are assumed to be independent normally distributed random variables. The state transition covariance matrix, $W^{(k)}_{t+1}$, and the observational variance, $V^{(k)}_{t+1}$, are typically unknown.
The DMA forecast, $\hat{y}_{t+1}$, is obtained through a convex combination of the forecasts of the $K$ DLMs,
where $\hat{y}^{(k)}_{t+1}$ is the prediction by model $M_k$, and $p(M_k|\mathcal{F}^{(k)}_{t})$ is the probability (weight) assigned to $M_k$ conditional on the information available at time $t$, $\mathcal{F}^{(k)}_{t}$. The information set is defined as, $\mathcal{F}^{(k)}_{t}= \{y_{t},\ldots,y_1, x^{(k)}_{t+1},x^{(k)}_{t},\ldots,x^{(k)}_1,\mathrm{Priors}_{t=0}\}$, and contains the choice of priors, the realisations of the covariate vector and of the response up to time $t$, as well as the covariate vector at time $t+1$, $x^{(k)}_{t+1}$, required to predict $y_{t+1}$.
Because we consider a single DLM with forgetting throughout this subsection, we drop the superscript $(k)$ to simplify notation. We adopt the DLM formulation proposed of DanglH2012, in which the prior distribution for the coefficients vector, $\theta_0$, is Gaussian; the measurement variance, $V_t=V$, is constant over time; and an inverse-gamma distributed prior for $V$ is used. Consequently,
which enables a conjugate Bayesian analysis. The model specification and the assumptions about the priors imply that,
where $S_{t}$ is the mean of the estimate of $V$ at time $t$, and $n_{t}$ stands for the associated degrees of freedom. Conditional on $V$ the posterior distribution of the coefficient vector is Gaussian,
where $\hat{\theta}_{t}$ is the point estimate of $\theta_{t}$, and $C_{t}$ is the estimator for the conditional covariance matrix for $\theta_{t}$. Integrating out $V$, the posterior becomes a multivariate $t$-distribution,
The prior distribution for the coefficient vector at the next time-step is,
For a generic DLM, $W_{t+1}$ can be sequentially estimated, but the associated computational cost is prohibitive for a method like DMA which uses $2^D-1$ DLMs. The distinctive feature of the approach of RafteryDMA2010 is that it avoids altogether the estimation of $W_{t+1}$ by setting,
where $\lambda \in (0,1]$ is the {\em forgetting factor} parameter. This simplifies Eq. (ref) to,
The forgetting factor in Eq. (ref) allows a continuous range between estimation in a static environment and completely discarding all past data which is appropriate in response to a structural break. Specifically, setting $\lambda=1$ corresponds to $W_t=\mathbf{0}$ and thus reduces the DLM to a static linear model. On the other hand, as $\lambda$ tends to zero $W_t$ tends to infinity. This inflates the uncertainty about $\theta_{t+1}$ (see Eq. (ref)), which effectively re-initialises the estimation process. Intermediate values of $\lambda$ correspond to a random walk process for $\theta$ in which periods of high estimation error in the coefficients coincide with periods of high variability, and vice versa.
The prior distribution (predictive density) of $y_{t+1}$ conditional on $\mathcal{F}_{t}$ is,
Once the actual value $y_{t+1}$ is observed, we can compute the forecast error,
and update the prior distributions of the coefficient vector and the measurement variance through Eqs (ref)--(ref). The degrees of freedom and the estimator of the observational variance are updated according to,
The point estimate and the estimator of the covariance matrix of $\theta_{t+1}$ are obtained by,
where
is a coefficient vector (known as the Kalman gain), which measures the information content of $y_{t+1}$ in relation to the precision of the estimated regression coefficient.
We are now in a position to describe the adaptive forgetting DLM (AF-DLM). The approach we propose is motivated by the adaptive recursive least squares algorithm Haykin_2002. The central idea underlying this approach is that the optimal forgetting factor minimises the expectation of the one-step-ahead squared forecast error,
Since the expectation in the above equation is not available in analytical form it is not feasible to directly optimise it. However, the one-step-ahead squared forecast error,
is an unbiased estimator of the expectation in Eq. (ref). Stochastic optimisation algorithms are designed to optimise the expected value of a function which depends on a set of random variables. The most widely used stochastic optimisation algorithms involve first-order information and are hence variations of stochastic gradient descent. The term stochastic gradient in this context refers to the fact that the gradient of $J_{t+1}$ (which is an unbiased estimate of the gradient of $\mathbb{E}_{X,Y}[J_{t+1}]$) is used.
To use stochastic gradient descent to minimise the expected one-step-ahead squared forecast error we need an expression for the derivative of $J_{t+1}$ with respect to $\lambda$. Applying the chain rule produces,
To obtain $\frac{ \partial \hat{\theta}_{t}}{ \partial \lambda} $ we differentiate the update equation for $\hat{\theta}_{t}$, Eq. (ref), with respect to $\lambda$. Such an approach is utilised in a number of adaptive linear filters Haykin_2002, in online neural network training Almeida1999,Schraudolph99b,Baydin2018, as well as in streaming data classifiers PavlidisTAH2011,Anagn12. The outcome of this differentiation is,
The derivation of all the necessary quantities to estimate $\frac{ \partial \hat{\theta}_{t}}{ \partial \lambda} $ is lengthy and is hence provided in Appendix (ref).
Using the gradient $\frac{ \partial J_{t+1} }{ \partial \lambda} $, AF-DLM updates the value of $\lambda$ at each time-step through the highly influential adaptive moment estimation (ADAM) stochastic gradient descent algorithm Diederik2015. ADAM uses adaptive estimates of the first two moments of the stochastic gradient to tune the crucial step-size parameter of the gradient descent algorithm. ADAM has been shown to be effective in a wide range of challenging optimisation problems; it is straightforward to implement; and is capable of handling non-stationary objective functions Baydin2018. The latter aspect is particularly important for our purposes since the optimal value of $\lambda$ is itself time-varying for time series that exhibit structural breaks, or more generally non-constant dynamics.
In this section we discuss the model averaging component of DMA. Recall that according to Eq. (ref) the DMA forecast, $\hat{y}_{t+1}$, is a convex combination of the forecasts produced by the $K$ DLMs. Let $p(M_k|\mathcal{F}_t)$ denote the probability (weight) of model $M_k$ conditional on the information set $\mathcal{F}_t$. The following definition of $p(M_k|\mathcal{F}_t)$ accommodates all DMA variants,
Ignoring the constant $c$ and expanding Eq. (ref) gives,
where typically, $p(M_k|\mathcal{F}_0)= 1/K$. The last equation highlights that $\alpha$ controls the rate at which past information is discounted.
The recommendations in the literature are to set the forgetting factor $\alpha$ either equal to one (which corresponds to Bayesian Model Averaging), or very close to unity. In particular, DanglH2012 and ByrneKR2018 use $\alpha=1$, while in the eDMA {\tt R} package of CataniaNonejad_2018 the default is $\alpha=0.99$. RafteryDMA2010 and KoopKorobilis_2012 recommend using $\lambda=\alpha$. Specifically, RafteryDMA2010 recommend $\lambda=\alpha=0.99$, while KoopKorobilis_2012 consider two values $\lambda,\alpha \in \{0.95, 0.99\}$. Only RafteryDMA2010 mention the small positive constant, $c$, in Eq. (ref) which is included to avoid the weight of any model becoming equal to machine zero. Such a constant is present in the DMA implementation of KoopKorobilis_2012, but not in the {\tt R} package eDMA CataniaNonejad_2018. The example we discuss next aims to illustrate that the speed with which model probabilities adapt in response to changes in the optimal model specification depends critically on $c$.
For simplicity we consider a problem involving only three models whose coefficients at every time-step are known. The observational variance of each model is also known and constant over time. The response at each time-step is generated from one of the three models, but the identity of this model is unknown. In this setting no generality is lost if we assume that for every model, $y_t^{(k)} | \theta_t^{(k)}, x_t \sim \mathcal{N}(\mu_k, V_k)$, for all $t=1,\ldots,T$, and $k=1,2,3$. Thus we assume $y^{(1)}|\theta_t^{(1)},x_t \sim \mathcal{N}(1, 4)$, $y^{(2)}|\theta_t^{(2)},x_t \sim \mathcal{N}(0, 0.64)$, and $y^{(3)}|\theta_t^{(3)},x_t \sim \mathcal{N}(-2.5, 0.09)$. We construct a time series of length $T=300$ by sampling $y_t$ from $M_1$ for $t=1,\ldots,100$, from $M_2$ for $t=101,\ldots,200$, and from $M_3$ for $t=201,\ldots,300$. The three models are assigned equal prior probabilities.
Figure (ref) depicts the evolution of the weights of the three models using red, blue and green colour respectively. The three subfigures correspond to $c = 0,10^{-20},10^{-3}/3$, as recommended by CataniaNonejad_2018,KoopKorobilis_2012 and RafteryDMA2010, respectively. Within each subfigure a different type of dashed line is used to distinguish between the three values of the forgetting factor, $\alpha \in \{0.99, 0.95,0.9\}$. Note that $\alpha=0.9$ is much lower than any recommendation in the literature, and is only included to explore the extent to which this forgetting factor enables adaptation. Finally, the two grey vertical lines depict the timing of the change points at $t=100, 200$.
The top subfigure corresponds to $c=0$. For this setting using $\alpha=0.99$ causes the probability of $M_1$ to be effectively equal to one throughout the simulation despite the two change points. This occurs because the weights of models $M_2$ and $M_3$ are equal to machine zero by time-step $t=100$. A lower value of $\alpha$ allows the weights to adapt correctly to the first change point, but the response is slow even when $\alpha=0.9$. However, by time-step $t=200$ the weight assigned to $M_3$ is equal to zero for both $\alpha = 0.95,0.9$. Therefore when $c=0$ not even very aggressive forgetting, $\alpha=0.9$, is sufficient for Eq. (ref) to correctly identify the optimal model after the second change point. Instead $M_1$ is assigned a weight of one. Introducing a small constant, $c=10^{-20}$, (middle subfigure) enables the weights to adapt correctly in response to both change points for all values of $\alpha$. We see however that for $\alpha=0.99$ approximately 50 time-steps are required after the first change point for the weight assigned to $M_1$ to become noticeably lower than one. Although the response to the second change point is faster, notice that for $\alpha=0.95$ there is a short period immediately after the change point during which the weight of $M_1$ (rather than $M_3$) increases abruptly. The final subfigure corresponds to the case $c=10^{-3}/3$. A larger constant enables the weights to adapt much faster in response to both change points. Notice for instance that for $c=10^{-3}/3$ and $\alpha=0.99$ model probabilities adjust faster after the first change point, compared to the case when $\alpha=0.9$ but $c=10^{-20}$. However, a larger value of $c$ also induces much higher variability during periods in which the data generating process is static.
The above example demonstrates that the choice of $c$ in Eq. (ref) is at least as important as that of $\alpha$. Therefore any approach focused only on tuning $\alpha$ is not sufficient to fully control the speed with which model probabilities in DMA adapt. We propose to use the ConfHedge algorithm from the field of machine learning known as {\em prediction with expert advice} VyuginTrunov_2019. As we discuss next ConfHedge has very appealing theoretical properties and is parameter-free.
Prediction with expert advice studies the following online learning problem CBianchiL2006. At time-step, $t+1$, each of the $K$ forecasting models, called {\em experts}\/, provides a forecast, $\hat{y}^{(k)}_{t+1}$. An {\em aggregating algorithm} predicts $\hat{y}_{t+1}$ through a convex combination of the experts' forecasts. After observing $y_{t+1}$, the weight of every expert is updated based on a measure of forecast error, called {\em loss}\/. In a static environment, the goal is to design weight updates that guarantee that the loss of the aggregating algorithm is never much larger than the cumulative loss of the best expert, or the best convex combination of the losses of the experts. In a dynamic environment the expert (or convex combination of experts) that achieve the lowest loss may differ across segments of the time series, and comparing against the best expert over the entire time series length can result in algorithms with poor forecast performance. To address this issue consider a partition of the time series into at most $L+1$ segments, $1<t_{(1)}<t_{(2)} < \cdots <t_{(L)}<T$, and allow the best expert (or convex combination of experts) to differ across elements of this partition. The best partition into at most $L+1$ segments is the one for which the optimal sequence of experts (or convex combination of experts) achieves the lowest cumulative loss. The learning problem in this setting is considerably harder. The ideal aggregating algorithm must achieve a loss that is as close as possible to that of the sequence of experts (or convex combinations of experts) that form the best partition of the time series into at most $L+1$ segments. Note that neither the maximum number of change points, $L$, nor the length of each segment are known.
A number of algorithms have been proposed that achieve optimal upper bounds for this problem, but these typically assume that the loss function is uniformly bounded HerbsterW1998. This assumption is not satisfied in our case since each DLM expert in ADMA includes a Gaussian error term. ConfHedge is the first (and to the best of our knowledge the only) method that upper bounds the loss of the aggregating algorithm against an arbitrary sequence of experts (or convex combinations of experts), when the loss function is unbounded VyuginTrunov_2019.
We next briefly describe the ConfHedge algorithm for our problem. To distinguish from the DMA probabilities we denote as $w_{k,t+1}$ the weight assigned by ConfHedge to model $M_k$ at the stage of predicting the response at time $t+1$. Initially the weights of all models are equal, $w_{1,t} = 1/K$. At time-step $t$ the ConfHedge prediction $\hat{y}^{\text{CH}}_{t+1}$ is obtained through,
After observing $y_{t+1}$ the loss of every expert is estimated as the squared forecast error,
and the loss of the aggregating algorithm is defined as,
where $l_{t+1} = (l_{1,t+1},\ldots,l_{K,t+1})$. The weights for time-step $t+2$ are updated according to,\footnote{We present the update for the Fixed Share mixing scheme in VyuginTrunov_2019.}
At the first time-step, $\eta_1=\infty$, which implies $w^{\mu}_{k,1} = 1$ if model $M_k$ achieves the minimum loss at time $t=1$, and zero otherwise Rooij2014. In subsequent time-steps the step-size parameter $\eta$ is updated according to,
ConfHedge involves no user-defined parameters. Eq. (ref) shows that the algorithm explicitly uses a weighted combination of two terms. The first term distributes a weight of $1/(t+2)$ equally among the $K$ experts. Note that in the original DMA formulation, the constant $c$ in Eq. (ref) plays a similar role although in that case it is not possible to control the proportion of the overall weight that is equally allocated among all models. The second term assigns a progressively larger proportion of the total weight to experts that achieve lower loss.
Denote as $q_1,q_2,\ldots,q_T$ a sequence of convex combinations of experts, where $q_t \in \left\{x \in \mathbb{R}^K_{+} \,|\, \sum_{k=1}^K x_{k} = 1 \right\}$. The goal of the aggregating algorithm is to minimise the {\em shifting regret},
where $q_t$ changes at most $L$ times, at unknown time-points, $1<t_{(1)}<t_{(2)} < \cdots <t_{(L)}<T$. VyuginTrunov_2019 prove a number of results that upper bound the shifting regret of ConfHedge for finite $T$. These bounds depend on the maximum number of change points, $L$, as well as on the range of actual losses incurred by the individual forecasters. The description of these results is beyond the scope of this paper. The following simple proposition establishes that when the loss function is the one-step-ahead squared forecast error an upper bound on $R_T$ translates to an upper bound on the mean squared forecast error.
In this section, we investigate the behaviour of the proposed AF-DLM on simulated data originating from static, gradually drifting and abruptly changing data generating processes. We set the dimensionality of the covariate vector to five to enable the visualisation of the path of the estimated coefficients. In all cases, the covariates are sampled from a Gaussian distribution $x_t \sim \mathcal{N}(0,I)$, $t=1,\ldots,1000$, while the noise term in the measurement equation has unit variance, $\sigma^2_t=1$ in Eq. ((ref)).
A static environment is characterised by a constant coefficient vector, $\theta_t = \theta_0$. We set $\theta_0=(-2,-1,1,2,3)^\top$ and obtain 100 time series of $y_t$ by randomly sampling from the measurement equation, Eq. ((ref)). In Figure (ref) solid lines correspond to the actual values of each coefficient, while dotted lines and shaded regions of the same colour represent the median and the interquartile range of the estimated coefficient, respectively. The figure shows that $\hat{\theta}_t$ converges rapidly to $\theta_0$, and the estimates exhibit very little variability across the 100 simulations.
Figure (ref) illustrates the evolution of the forgetting factor, $\lambda_t$, through the ADAM stochastic gradient descent algorithm. The value of $\lambda_t$ is high throughout the simulation, and as $\hat{\theta}_t$ converges to $\theta_0$ the median value of $\lambda_t$ increases and the variability across simulations decreases. As the forgetting factor tends to unity, past and present examples become equally weighted and consequently parameter estimates become more accurate and less variable.
Next, we consider dynamic environments in which the coefficient vector changes at distinct time points, and remains constant in-between consecutive change points. We consider again time series of length 1000, and introduce change points at $t=100,400,700$. As in the static environment, we simulate data from a single time series of $\theta_t$ and create 100 time series of $y_t$ by different realisations of the noise term in the measurement equation, Eq ((ref)). The time series of the coefficient vector $\theta_t$ is specified by $\theta_0 = (3,2,1,-1,-2)^\top$ and, \[ \theta_t = \left\{
\right. \] The specific trajectory for $\theta_t$ is selected because it allows a clear visualisation of the evolution of $\hat{\theta}_t$ at each time-step. Solid lines in Figure (ref) depict the evolution of $\theta_{t}$ while dotted lines and shaded areas of the same colour correspond to the median estimated parameter and the associated interquartile range, respectively.
At the first change point, $t=100$, the magnitude of the change in every element of $\theta_t$ is the largest. Figure (ref) shows that $\lambda_t$ decreases very rapidly in response to this, and by the time step $t=200$ it assumes the smallest values observed during these simulations. The minimum value of $\lambda_t$ observed is lower than 0.9 which implies a very aggressive forgetting of past information, or equivalently a very small effective window size. As $\hat{\theta}_t$ approaches $\theta_t$, the forgetting factor steadily increases and approaches its maximum value when the two almost coincide. This occurs right before the second change point at $t=400$. The change in $\theta_t$ at $t=400$ is much smaller than that at the first change point, and this is reflected in the evolution of the forgetting factor. As Figure (ref) shows $\lambda_t$ decreases rapidly following the second change point but the lowest median value, which is close to 0.95, is much higher than the corresponding minimum following the first change point. Subsequently, the forgetting factor increases steadily as the estimated coefficients converge to the true values. The third change point at $t=700$ reverts $\theta_t$ to its value prior to the second change point. As Figure (ref) shows the effect of this change point on $\lambda_t$ is very similar to the pattern observed after the second change point.
Overall, the results depicted in Figure (ref) indicate that the proposed adaptive forgetting algorithm is capable of tuning $\lambda_t$ effectively in the presence of change points. Following each change point AF-DLM induces a sharp decline in $\lambda_t$, which enables the estimated coefficients to adjust rapidly. As $\hat{\theta}_t$ converges to the true coefficients, which are static in-between consecutive change points, $\lambda_t$ increases and, if the interval between consecutive change points is sufficiently long, it approaches unity.
Finally, we consider time series in which the coefficient vector changes gradually over time. For this purpose, we simulate from the state-space model assumed by the DMA algorithm, namely Eqs. ((ref)) to ((ref)), with $\theta_0 = 0$. Our objective in this case is to evaluate whether the proposed adaptive forgetting method can identify the true value of $\lambda$. Note that in this case we are not able to simulate different realisations of $y_t$ for a single time series of $\theta_t$, since the state transition covariance matrix, $W_t$, depends on the previous realisations of the forecast error.
We consider three values of the forgetting factor, $\lambda \in \{0.99, 0.97, 0.95\}$, and for each value simulate 100 time series. In Figure (ref) the dashed horizontal lines correspond to the true values of $\lambda$, while the solid lines and the shaded regions of the same colour depict the median and interquartile range of $\lambda_t$, respectively. As the figure shows the proposed method rapidly adjusts $\lambda_t$ towards the true value. After the initial adjustment period $\lambda_t$ fluctuates around the true value. This fluctuation is more variable for smaller values of $\lambda$, which is consistent with the model since a smaller forgetting factor implies a higher variability in the trajectory of $\theta_t$. Note that beyond the initial adjustment period, the median value of $\lambda_t$ is never more than 0.01 away from the true value. Furthermore, the interquartile range of $\lambda_t$ contains the true value in the vast majority of time-steps (the only exceptions occur for $\lambda=0.95$ and their duration is short).
Overall, the results on simulated time series illustrate that AF-DLM can effectively tune the value of the forgetting factor under different types of variation in the data generating process. In response to abrupt changes AF-DLM decreases $\lambda$ sharply thereby enabling the estimated coefficients to adjust rapidly. When the coefficients are static the adaptive forgetting factor tends to unity hence improving the accuracy and stability of the estimation process. In cases where the coefficients change gradually the forgetting factor fluctuates around a value that reflects the speed of drift. This concludes our empirical evaluation of the AF-DLM and in the next section we focus on the comparative evaluation of ADMA on the problem of forecasting UK regional house prices.
For our empirical application, we employ quarterly seasonally adjusted regional house price indices for the period 1982:Q1 to 2017:Q4. The data is provided by Nationwide, the largest building society in the world and one of the largest mortgage providers in the UK. Following Nationwide's classification, we consider 13 regional housing markets: the North, Yorkshire and Humberside, North West, East Midlands, West Midlands, East Anglia, Outer South East, Outer Metropolitan, Greater London, South West, Wales, Scotland and Northern Ireland. To transform nominal into real prices, we divide by the consumer price index (all items), obtained from the OECD Database of Main Economic Indicators, and then compute the annualised log transformation of real property price inflation as,
where $P_{r,t}$ stands for the level of the real house price index of market $r$ at time $t$.
For each region in our sample, we consider eleven economic variables as potential predictors of future house price movements: four regional-level and seven national-level predictors. The variables measured at the regional level include the price-to-income ratio (which proxies for affordability), income growth, the unemployment rate, and the growth in labour force. National-level predictors consist of the real mortgage rate, the spread between yields on long-term and short-term government securities, growth in industrial production, the number of housing starts, growth in real consumption, the Credit Conditions Index (CCI) proposed by Corugedo_2006, and a new measure of House Price Uncertainty (HPU) which we construct using the news based methodology of BakerScott_2016. For a description of the variables, the data sources and the transformations undertaken we refer the reader to Appendix (ref).
From the above predictors, the first nine have been used by BorkMoller_2015 to forecast house price movements in US metropolitan states. The last two have not been employed in a forecasting context before but may well have predictive content for future house price inflation. With regard to CCI, credit supply conditions in the UK economy, especially in the mortgage market, have changed dramatically since the 1970s. As argued by several authors, such changes were at the heart of the housing boom that preceded the Great Recession. It therefore seems natural to investigate whether an index of credit conditions may contain valuable information for forecasting.\footnote{A deficiency of simple proxies for credit conditions, such as interest-rate spreads and unsecured credit to income ratios, is that they fail to control for the economic environment, and are thus subject to an endogeneity problem. The methodology of Corugedo_2006 mitigates this problem by making use of a large number of economic and demographic controls.} Similarly, changes in house price uncertainty impact on housing investment and real estate construction decisions Cunningham2006,BanksBOS2015,OhY2019, and thus may lead to future house price movements.
In addition to macro and financial variables, there is a substantial empirical literature that documents the existence of strong spatial linkages between UK regional markets in sample Drake_1995, Meen_1999, CookThomas_2003, Holly_2010,Antonakakis_CFG2018. To accommodate this, we incorporate in the set of potential predictors lagged property price growth in contiguous regions. The number of neighbouring regions for each of the 13 real estate markets under consideration lies in the range of one to five.
Before proceeding to the forecasting exercise, we examine whether there is evidence of structural instability in the relationship between real house price inflation and individual house price predictors in sample. To do so, we employ two tests proposed by ChenH2012. The first is a Hausman-type ($H$) test that compares time-varying parameter estimates obtained by local linear regression to constant estimates obtain by ordinary least squares. The second is a Chow-type ($C$) test which compares the sum of squared residuals between the constant parameter and local linear regression models. The null hypothesis in both tests is that of time-invariant regression coefficients.
The $H$ and $C$ tests have a number of attractive features. First, because they impose minimal restrictions on the functional form of the time-varying parameters, they are consistent with both smooth and abrupt structural change and, from this perspective, correspond well to the AF-DLM of Section (ref). Second, they require no prior information regarding the timing and the number of breaks. Third, they are asymptotically pivotal and, fourth, they do not involve trimming of the boundary region near the end points of the sample period.
Tables (ref) and (ref) report wild-bootstrap $p$-values of the $H$ and $C$ tests for each of the 11 house price predictors considered and for each of the 13 regions. We observe that, out of the 286 $p$-values, none exceeds 10 percent, four exceed five percent, and the vast majority lie below the one percent threshold. This strong evidence of structural instability motivates the use of dynamic econometric models for forecasting house price inflation.
We begin our out-of-sample analysis by comparing the forecast accuracy of a battery of econometric models relative to the AR(1) benchmark as well as the performance of ADMA relative to each of the remaining models in the pool. This set of models consists of the DMA formulation of DanglH2012 (abbreviated as eDMA due to the {\tt R} implementation of CataniaNonejad_2018), two versions of the DMA of KoopKorobilis_2012 (one with relatively slow forgetting, $\lambda=\alpha=0.99$, and another with fast forgetting $\lambda=\alpha=0.95$), a single DLM with $\lambda=0.99$ that includes all available predictors and, finally, Bayesian Model Averaging (BMA). Table (ref) provides an overview of all models.
Table (ref) summarizes the forecasting performance of each model relative to the AR(1) benchmark over the out-of-sample evaluation period, which runs from 1995:Q1 to 2017:Q4. The second column of the table provides the realised Mean Squared Forecast Error (MSFE) of the AR(1) model for each region, while the remaining columns report the ratio of the MSFE of the competing models to that of the AR(1). Tables (ref) and (ref) report MFSE ratios when ADMA is set as the alternative and the benchmark model, respectively. In all three tables, a $\dagger$ indicates cases when the test of ClarkWest_2007 rejects the null hypothesis of equal predictive accuracy in favour of the one-sided alternative that the competing model outperforms the benchmark at the 5% significance level.
It is evident from Tables (ref), (ref) and (ref) that ADMA performs better than all other methods. First, it is the only method that achieves a statistically significant improvement over the benchmark in all regional markets and, second, it produces on average the most accurate forecasts with a mean MSFE 15% lower than the AR(1). The second best forecasting method is eDMA. This method generates significantly more accurate forecasts than the AR(1) model in nine regional markets and achieves a 10% average improvement in MSFE relative to the benchmark. A comparison of ADMA and eDMA suggests that ADMA is more accurate in eight of the 13 regions, with this improvement being statistically significant in six cases. In contrast, eDMA performs significantly better than ADMA in only three regions. DMA with fixed forgetting also outperforms the AR(1) benchmark in the majority of cases but its performance depends critically on the choice of $\lambda$ and $\alpha$, and no choice appears to be uniformly better. ADMA generates more accurate forecasts than DMA$_{0.99}$ and DMA$_{0.95}$ in all regional property markets but one. This forecast improvement is statistically significant in 11 regions for DMA$_{0.99}$ and in nine regions for DMA$_{0.95}$. BMA achieves a lower MSFE than ADMA in four markets but in all cases the difference is very small, and only once it is found to be statistically significant. In contrast, ADMA achieves a significant improvement over BMA in five regional markets. Finally, the performance of DLM$_{0.99}$ is uniformly worse than that of ADMA, and this model outperforms the benchmark only in six regional markets. This outcome is consistent with KoopKorobilis_2012 and BorkMoller_2015, who argue that the use of a large number of explanatory variables can cause model over-fitting which leads to inaccurate predictions.
Having discussed forecast accuracy, we employ the estimated ADMA weights, $w_{k,t+1}$, to identify important variables for predicting future property price movements, and to investigate how the best house predictors vary over time and across regional markets. Following KoopKorobilis_2012, for each predictor in our dataset, we scan through the set of DLMs and select those which contain the variable under consideration in their specification. The probability that ADMA assigns to this subset of models, called the posterior inclusion probability, reflects the importance of the variable in forecasting.
Figure (ref) displays the estimated posterior inclusion probabilities. For presentation purposes, we report results for the three most important predictors - classifying a predictor as important on the basis of its ADMA weights over the entire evaluation period- and focus on the three most volatile (Northern Ireland, East Anglia, the North) and the three most stable regional markets (Scotland, North West, West Midlands). Overall, the results in Figure (ref) suggest that the best predictors differ over time and across regions.
For volatile regions, we observe that in two out of the three regions (Northern Ireland and the North), the key house price predictor during the recent boom is CCI. In Northern Ireland, which is the most volatile region in our sample, the posterior inclusion probability attached to CCI is consistently high throughout the boom phase but drops in the last part of the sample period. In the North, the CCI posterior inclusion probability increases from around 40% in 1995 to 80% in 2004 and then falls back to its original level. These findings are in line with the widely held view that changes in credit conditions were at the heart of the house price surge prior to the financial crisis.
On the other hand, house price uncertainty plays an important role in predicting future house price movements in volatile markets ahead of the house price collapse of 2008:Q3. The probability of including HPU in the forecasting models of Northern Ireland and East Anglia rises to around 90% and 70%, respectively, in 2008:Q3 and then drops following the downturn in property prices. The mortgage rate and spread are important predictors of house price inflation in East Anglia and the North, though their ADMA weights are only marginally above 0.5. With regard to the other predictive variables, we note that these are important in some volatile regions, but not in others. Perhaps the most striking example is the unemployment rate. For Northern Ireland, this variable is one of the key determinants of future property price movements in the aftermath of the house price collapse, with an ADMA weight of around 0.7 from 20011:Q1 until the end of the sample. On the contrary, for East Anglia and the North, the probability of including the unemployment rate in the predictive model is never above 0.5.
Moving on to the stable housing markets of Scotland, North West and West Midlands, we notice that credit availability and house price uncertainty are again included in the set of important predictors. In all three regions, HPU becomes the best predictor ahead of the house price collapse of 2008:Q3 and during the bust phase. While, CCI is the key determinant of property price inflation at the start of the boom phase, from the first quarter of 2004 until the end of 2005. Similarly to volatile property markets, the remaining predictors show mixed predictive ability. Overall the results of the empirical application suggest that allowing for structural instability and regional heterogeneity is crucial for forecasting UK house prices.
Dynamic model averaging (DMA) is gaining increasing attention in macroeconomic time series forecasting due to its ability to accommodate time-variation in both the parameters as well as the specification of the optimal forecasting model. In this paper we introduced a novel adaptive methodology for DMA which aims to overcome limitations of existing DMA specifications with respect to both the sequential estimation of the optimal forgetting factor for each dynamic linear model (DLM), as well as the model averaging process. Motivated by work in adaptive filtering, we proposed to optimise the forgetting factor of each DLM through a state-of-the-art stochastic gradient descent algorithm. Our simulation study illustrated that this approach can effectively approximate the optimal forgetting factor under different types of change in the data generating process, including cases in which the speed or type of change is variable over time. A further advantage of our approach is that it is computationally less demanding compared to competing DMA specifications that sequentially update the DLM forgetting factor by considering a grid of values for this parameter. Our adaptive methodology also involves a parameter-free forecast aggregation algorithm from the literature on prediction with expert advice. This allows us to obtain finite-time performance guarantees about the forecast accuracy of the DMA forecast. To the best of our knowledge no other DMA specification has this property.
We conducted an in-depth empirical evaluation of the proposed methodology on the task of forecasting UK regional house prices. Our results indicate that the adaptive DMA produces overall more accurate forecasts than competing DMA specifications. They also reveal that no single predictor is consistently chosen as the key determinant of future property price movements. Credit availability was found to be an important predictor of house price inflation for several regional markets during the boom phase of the 2000s, while house price uncertainty appeared to play an important role in predicting house price movements on the eve of the price collapse of 2008:Q3.