Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
27,665 characters · 9 sections · 43 citation commands
The Aligned Economic Index & the State-Switching Model
A large body of work studies whether stock returns are predictable using valuation ratios and macroeconomic indicators (e.g., FamaFrench1988,CampbellShiller1988,KandelStambaugh1996,Guo2006,Lewellen2004,PolkThompsonVuolteenaho2006). WelchGoyal2008 show, however, that many prominent predictive regressions fail to deliver reliable out-of-sample gains relative to simple benchmarks. Two recurrent explanations are model uncertainty, the correct specification is unknown ex ante. and parameter instability, which makes estimates sensitive to the sample period.
Over the past decades, several approaches have been proposed to mitigate these issues, including economically motivated restrictions PanPettenuzzoWang2020, combinations of multiple predictors Huang2015, regime-shift specifications HammerschmidLohre2018, and new predictors Jiang2019. While these methods often improve forecasting performance, their gains are frequently concentrated in recessionary episodes, with weaker results in expansions.
A key mechanism behind this pattern is that investors may rely on different information sets and forecasting rules across states. CujeanHasler2017 provide an equilibrium framework in which agents switch attention and forecasting models over time, and Devpura2018 document predictor-dependent time variation in return predictability. Motivated by these findings, I use a simple state-dependent predictive regression in which slope coefficients are allowed to change across market states. The resulting state-switching model delivers stronger and more stable predictive performance across both expansionary and recessionary periods than the conventional one-state specification.
The aggregate stock market return is the excess return on the S&P 500 index: the continuously compounded log return (including dividends) minus the risk-free rate, proxied by the three-month Treasury bill. Working with excess returns focuses attention on the predictability of the (real) equity risk premium rather than on variation driven mechanically by the level of short-term interest rates.
I use updated data from WelchGoyal2008 covering January 1950 to December 2017. The predictor set includes the 14 popular variables in WelchGoyal2008, supplemented with the corporate bond premium and the lagged equity premium, for a total of 16 series.\footnote{The predictors are: dividend--price ratio (log), dividend yield (log), earnings--price ratio (log), dividend--payout ratio (log), equity premium volatility, book-to-market ratio, net equity expansion, Treasury bill rate, long-term yield, long-term return, term spread, default yield spread, default return spread, inflation, lagged equity premium, and the corporate bond premium. The data can be retrieved from Amit Goyal’s web page (www.hec.unil.ch/agoyal/); variable definitions are provided in WelchGoyal2008.} Following common practice, I focus on the post-war period. From an investor-relevance perspective, this emphasizes modern market dynamics. From a statistical perspective, Lewellen2004 argues that pre-1945 data have markedly different properties (e.g., unusually high volatility and persistence in some predictors), which can distort inference. I also exclude 1945--1949 because dividend policies during and immediately after World War II were unusually volatile, potentially affecting the behavior of dividend-based predictors Frankfurter1997. Finally, based on preliminary analysis on the first years of data, the baseline sample used throughout is January 1960 to December 2017 (696 monthly observations).
All data are monthly and the baseline forecasting horizon is one month ahead. This horizon is well suited for studying state dependence: business-cycle states typically last several months, so longer-horizon regressions may mechanically average over multiple states and blur regime effects. Moreover, short-horizon predictability can translate into longer-horizon predictability under standard present-value logic Cochrane2011,Huang2015,RapachRinggenbergZhou2016.
To define market states in real time, I construct an ex-ante indicator based on the slope of the yield curve (the term spread), computed as the 10-year Treasury yield minus the 3-month Treasury bill rate. In contrast to NBER recession dates, which are assigned ex post,\footnote{NBER recession dates can be retrieved from the Federal Reserve Bank of St.\ Louis (FRED). Because these dates are determined ex post, they are not directly usable for real-time forecasting exercises.} the yield-curve slope is observable at the time forecasts are formed and is widely viewed as a leading indicator.
Under the expectations hypothesis, the term spread reflects the difference between the current short rate and expected future short rates over a longer horizon. When the spread narrows, investors may be more uncertain about future economic conditions and require higher compensation for risk. The yield curve therefore summarizes aspects of macroeconomic sentiment and risk compensation Wright2006.
The slope of the Treasury yield curve is often treated as an early-warning indicator for downturns: inversions have historically preceded several U.S.\ recessions. Motivated by this, I define the market state at month $t$ as a down state if the yield curve was flat or inverted at any point in the preceding $\tau$ months:
where $S_t$ is the state indicator and $\text{tms}_{t-i}$ is the term spread (10-year minus 3-month), as in, e.g., FamaFrench1989,WelchGoyal2008. The parameter $\tau$ controls how long a down state persists after a (near-)inversion. To reduce the risk of in-sample overfitting and data-snooping concerns emphasized by WelchGoyal2008, I fix $\tau$ rather than optimizing it. In the baseline results, I set $\tau=9$ months.\footnote{Using $\tau$ between 3 and 12 months leaves the main findings unchanged.} This choice is short enough to avoid mechanically locking the economy into long down states, while remaining consistent with the average duration of recessionary episodes observed in the data.
I extend the standard predictive regression by allowing slope coefficients to differ across states, as in Boyd2005,Sander2018:
where $R_{t+1}$ is the excess return on the S&P 500 at month $t+1$, $x_t$ is a lagged predictor observed at month $t$, and $\varepsilon_{t+1}$ is the disturbance term. When $\beta_1=\gamma_1$, the model collapses to the conventional one-state predictive regression.
Out-of-sample forecasts are formed recursively:
where $(b_0,d_0,b_1,g_1)$ are OLS estimates of $(\beta_0,\delta_0,\beta_1,\gamma_1)$ computed using only information available up to time $t$. Specifically, for each forecast origin $t$, I regress $\{R_{s+1}\}_{s=M}^{t-1}$ on $\{1, S_s, S_s x_s, (1-S_s)x_s\}_{s=M}^{t-1}$, where $M$ is the start of the out-of-sample period. This recursive procedure avoids look-ahead bias by construction.
Combining complementary information from multiple predictors can yield strong forecasting performance across different market states. However, as shown by WelchGoyal2008, estimating a multivariate “kitchen-sink” regression typically leads to poor out-of-sample performance due to parameter instability and overfitting. To address this, I aggregate information from the 16 predictors in a more parsimonious way by constructing an aligned economic index.
The economic intuition is that investors may shift attention across fundamentals and macro indicators depending on market conditions. Consistent with this view, changes in investor attention and information processing have been linked to return dynamics BarberOdeanZhu2009,DaEngelbergGao2011.
A standard approach to summarize many predictors is principal component analysis (PCA), which reduces dimensionality by extracting common factors that explain maximal predictor variance StockWatson2002,Jolliffe2002. When predictors are noisy proxies and idiosyncratic components are correlated across series, however, PCA factors may be contaminated by common noise and become less useful for forecasting BoivinNg2006. Partial least squares (PLS) provides an alternative: it constructs components that explicitly maximize covariance with the forecast target (here, future excess returns), rather than explaining total predictor variance GeladiKowalski1986. This targeted dimension reduction is well suited to forecasting settings in which much predictor variation may be irrelevant for prediction KellyPruitt2015. Appendix (ref) describes the construction of the Aligned Economic Index in detail.
Figure (ref) plots the Aligned Economic Index (upper panel) and the normalized equity premium (lower panel) over January 1960 to December 2017. The gray bars indicate NBER recessions. If the index contains forecasting information about the risk premium, the two series should comove at business-cycle frequencies.
The figure suggests substantial comovement. Both series drop sharply early in the sample, rise during subsequent recoveries, and display heightened volatility around major market stress episodes (e.g., the late-1987 crash and the 2007--2009 financial crisis). The index is also noticeably less persistent than several classical valuation ratios, suggesting that small-sample bias concerns emphasized by Stambaugh1999 are less acute for this predictor.
Table (ref) reports predictive regression results using the Aligned Economic Index. Panel A estimates the conventional one-state model using $E^{PLS}$. Panel B estimates the state-switching model in Equation (ref). For comparison, I also report results for two alternative predictor-combination methods: (i) a PCA-based factor $E^{PCA}$,\footnote{I retain only the first principal component. In unreported results, including additional factors consistently reduces out-of-sample performance in both the one-state and state-switching specifications.} and (ii) the forecast-combination (FC) approach of Rapach2010, denoted $E^{FC}$.
If return predictability is time varying and predictor dependent Devpura2018, one would expect that allowing coefficients to switch across states should improve performance relative to the one-state specification, and that the magnitude of the gains may differ across combination methods.
Panel A shows that both $E^{PLS}$ and $E^{PCA}$ have significant in-sample predictive power. The adjusted $\bar R^2$ of $E^{PLS}$ (4.65%) is notably larger than that of $E^{PCA}$ (0.61%), indicating that $E^{PLS}$ explains a larger share of one-month-ahead variation in the equity premium. The forecast-combination predictor $E^{FC}$ has weak in-sample performance in this specification. Out of sample, $E^{PCA}$ yields a positive but small $R^2_{\text{oos}}$ (0.31%) that is not statistically significant and falls below the 0.50% economic threshold discussed by CampbellThompson2008. In contrast, $E^{FC}$ achieves a higher $R^2_{\text{oos}}$ (1.18%), consistent with the forecasting gains reported by Rapach2010, while $E^{PLS}$ performs best with $R^2_{\text{oos}}=2.60\%$.
Panel B turns to the state-switching model. Two patterns stand out. First, the state intercept shift $\delta_0$ is statistically significant (Newey--West t-statistics around 2.5) across all three predictors, supporting the relevance of yield-curve-based regimes. Second, both $\bar R^2$ and $R^2_{\text{oos}}$ increase relative to the one-state model for all three predictor-combination methods. All three predictors now produce statistically significant $R^2_{\text{oos}}$ values that comfortably exceed the CampbellThompson2008 0.50% threshold. Among them, $E^{PLS}$ remains the strongest performer, with $\bar R^2=5.90\%$ and $R^2_{\text{oos}}=4.12\%$.
Although $E^{PLS}$ is a strong predictor of excess monthly returns when looking at the entire sample period, it is interesting to further evaluate its performance during certain market states. Table (ref) reports the results across different market states. Panel A shows the forecasting performance using the conventional one-state univariate model, while panel B reports findings using the state switching model. The left part of Panel A shows the in-sample and out-of-sample performance of the one-state model across expansions and recessions. Both $E^{PLS}$ and $E^{PCA}$ can forecast, in-sample, the equity premium during recessions, however only $E^{PLS}$ is able to generate a positive adjusted $\bar R^2$ during expansions. Looking at the out-of-sample performance, both $E^{PCA}$ and $E^{FC}$ produce significant positive $R^2_{\text{oos}}$ values during recessions, but fail to do so during expansions, with $E^{PCA}$ even failing to outperform the benchmark model of the historical mean. These findings are in line with the conclusions of Henkel2011 and DanglHalling2012: return predictability only exists during recessionary periods. In contrast, the newly formed aligned economic index can significantly predict the equity premium during recessions and expansions. $E^{PLS}$ has an $R^2_{\text{oos}}$ of 5.99% during recessions and an $R^2_{\text{oos,exp}}$ of 1.45% during expansions, both significant at the 5% level or stronger. This confirms that using combined information of the predictors in an efficient way leads to improvement across different states. The right part of Panel A shows the same statistics across up and down states as defined by the state indicator based on the inversion of the yield curve (as defined in Equation (ref)). All three models produce positive in-sample $\bar R^2$ across both states, with $E^{PLS}$ outperforming the models substantially across both states. The out-of-sample results are less clear. First, $E^{PCA}$ fails to significantly outperform the benchmark model during an upstate and during a downstate. Thus, $E^{PCA}$ is only able to capture predictability during parts of the recessionary periods that do not overlap with the downstate periods. Secondly, both $E^{PLS}$ and $E^{FC}$ outperform, significantly on the 5% level or stronger, the naïve benchmark model during an upstate but fail to do so during a downstate with $E^{PLS}$ even generating a negative $R^2_{\text{oos,down}}$ of -7.86%. A downstate is initiated whenever the yield curve inverts, which generally happens right before the start of a recession. Thus, with the one-state model $E^{PLS}$ fails to predict the equity premium right before the start of a recession. These findings are in line with Sander2018 who shows that estimating the start of a recession inaccurately can lead to large economic consequences. In short, even though $E^{PLS}$ outperforms the other two combination methods across different states, it still fails to outperform the naïve benchmark model around the start of a recession in the one-state model.
Moving on to the state switching model in panel B of Table (ref), we see substantial performance gains across different states. First the in-sample performance of all three combination methods are further enhanced within the state switching model, however in line with panel A only $E^{PLS}$ is able to generate performance gains exceeding 1% across all states. More noteworthy, the out-of-sample performance is substantially affected by the state switching model. $E^{PCA}$ is able to significantly forecast the equity premium during recessions, up states and down states. During expansions however both $E^{PCA}$ and the benchmark model predict, on average, the equity premium with the same capacity producing an $R^2_{\text{oos}}$ of zero. In contrast, both $E^{PLS}$ and $E^{FC}$ can significantly, on the 5% level or stronger, predict the equity premium during expansions, recessions, up states and down states. $E^{PLS}$ generates $R^2_{\text{oos}}$ values of respectively 1.71% and 1.68% during expansions and down states, while producing $R^2_{\text{oos}}$ values of respectively 11.21% and 4.51% during recessions and up states. All of the values thus exceed the economic threshold of 0.50% of CampbellThompson2008 meaning that an investor using the state switching model with $E^{PLS}$ would make consistent substantial gains across the different market states. The main difference with the one-state model is the fact that $E^{PLS}$ is now able to predict the equity premium during the recessionary turning points as well, with the inversion of the yield curve (i.e. a down state) helping to time the right moment to change slope coefficients. Interestingly, the forecast combination method $E^{FC}$ exhibits the same overall behavior as $E^{PLS}$. $E^{FC}$ generates $R^2_{\text{oos}}$ values of respectively 2.19% and 6.36% during expansions and down states, while producing $R^2_{\text{oos}}$ values of respectively 3.95% and 2.03% during recessions and up states. Thus both $E^{PLS}$ and $E^{FC}$ are able to consistently outperform the naïve benchmark model across all market states, with $E^{PLS}$ yielding the overall greatest performance. In the next section I further analyze the relationship between $E^{PLS}$ and $E^{FC}$.
Over the out-of-sample period (January 1980 to December 2017), the forecasts implied by $E^{PLS}$ and $E^{FC}$ are strongly correlated (about 65%). The correlation rises further within down states, suggesting that both methods capture a similar component of risk-premium variation when turning-point risk is elevated.
Figure (ref) compares the two forecast series to realized returns. Forecasts based on $E^{PLS}$ are more volatile than those based on $E^{FC}$, while realized excess returns are most volatile. This difference is intuitive: PLS is a targeted method that loads more heavily on predictor components that covary with returns, whereas $E^{FC}$ averages across many individual predictive regressions, which mechanically smooths forecasts Rapach2010. The higher responsiveness of $E^{PLS}$ can be beneficial in environments where predictability is time varying.
Figure (ref) shows that $E^{PLS}$ places time-varying weight on the underlying predictors, with relatively large average weights on equity premium volatility, the Treasury bill rate, and the long-term yield. Weight variation is more pronounced around regime transitions, consistent with state-dependent and time-varying predictability Devpura2018. In contrast, $E^{FC}$ uses constant equal weights, which helps explain its smoother forecast path.
This paper studies return predictability using a real-time state-switching predictive regression and a new PLS-based aggregate predictor. The state-switching model uses the slope of the yield curve to define market regimes and generally improves the predictive performance of widely used predictors in sample and out of sample. The Aligned Economic Index $E^{PLS}$ aggregates information from the WelchGoyal2008 predictor set (augmented with two additional series) using a targeted dimension-reduction approach. Empirically, $E^{PLS}$ delivers strong and robust forecasting performance, and its gains are especially pronounced when combined with the state-switching specification. Importantly, the resulting framework yields statistically significant and economically meaningful out-of-sample improvements across both expansionary and recessionary regimes, making it a practical alternative to one-state predictive regressions whose performance is often concentrated in recessions.
\printbibliography[heading=bibintoc, title=\ebibname]