Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
102,088 characters · 40 sections · 39 citation commands
Predicting Market Troughs: A Machine Learning Approach with Causal Interpretation
\affil[1]{Department of Economics, University of California, Los Angeles, US}
\begingroup\footnote{Corresponding author: Peilin Rao.}\endgroup
Understanding the triggers of stock market troughs is of great economic significance: policymakers can conduct interventions to alleviate market capitulation and panick, and investors can make better informed asset allocation decisions. However, being able to move from pure prediction to credible causal inference for market troughs is none-trivial. The complex, nonlinear, and high dimensional nature of financial markets make causal inference susceptible to spurious conclusions with simplified models. This paper tackles the challenge by asking: What are the robust, causal drivers of market troughs, and how do our conclusions depend on the assumptions of the econometric models we use?
The rise of the "credibility revolution" in financial econometrics, empowered by methods such as Double/Debiased Machine Learning (DML) Chernozhukov2018, provides a path to this challenge. DML establishes a framework of robust methods to obtain statistically significant causal estimates even with high-dimensional confounding. While these methods begin to gain traction in finance on research topics like asset-pricing factors Feng2020, the use of DML for macro-finance questions like market timing is still nascent, with most recent contributions primarily focusing on prediction rather than formal causal inference Gu2020. It is also important to highlight that any conclusions drawn from these complex causal inference methods are highly sensitive to econometric model specifications, an issue exacerbated by the high dimensional unobserved confounding Chernozhukov2022.
Our primary contribution is a novel, comparative causal framework designed to address this challenge by testing the robustness of economic conclusions to model specification. We achieve this objective in two stages. To illustrate the model specification sensitivity and build our case for a flexible approach, we first establish a baseline by using DML to the canonical partially linear model (PLR), which is widely understood and would serve as a benchmark. It allowed us to examine and isolate the impact of its inherent linearity assumption. We understand that PLR's linearity assumption is ill-posed for our binary, interactive market capitulation problem; therefore, we implement a more complex DML framework to estimate Average Partial Effect (APE), which explicitly model non-linear interactions. Comparing the causal interpretation of the two models is our core contribution, and we demonstrates that the more flexible APE model is necessary for credibility inferences. It corrects spurious causal interpretation from the linear model, sometimes providing estimates with reversed signs, and unveils new causal channels for market troughs, particularly the role of volatility in risk appetite and liquidity.
This flexible DML causal analysis is made possible through our robust and high-performing predictive pipeline, developed to nowcast the probability of a market trough. A central methodological challenge of identifying market capitulation is that the label for trough always depends on future data. Algorithms like Bry1971 inevitably rely on observing data from the future, thereby creating data leakage. Our solution to this paradox is to frame our prediction objective in the form of nowcasting: estimating in real-time the probability that the current period eventually be identified as a trough. This nowcast method generates timely market trough signal before the event is confirmed. Our main predictive model, a Support Vector Machine (SVM), is trained on a full set of over 200 features, constructed from options, futures and macroeconomic data to capture market microstructure, dealer positioning and sentiment. Our model has remarkable out-of-sample performance (ROC AUC of 0.89), and we demonstrate its economic significance through a stylized backtest, utilizing the model signal as a capitulation detector for future trading.
This paper brings together the strengths of three streams of literature. It addresses the problems in the traditional linear prediction literature Goyal2008, the subsequent attempts to restore predictability through economic restrictions Campbell2008 and theoretical driven variables Lettau2001, and extends the nonlinear approaches in the machine learning literature Gu2020 by applying the rigor of modern causal inference with high dimensional confounders. Our contributions are:
By bridging from prediction to modern, robust causality, we are providing not only a useful nowcasting tool, but also a rigorous economic understanding of the forces that shape market bottoms. The rest of the paper is structured as follows. Section 2 describes the data and feature engineering. Section 3 describes the predictive modelling framework. Section 4 provides the predictive results and interpretation. Section 5 provides a robustness analysis. Section 6 assesses economic significance. Section 7 describes our comparative causal methodology and provides the robust causal estimates. Section 8 concludes.
We begin by incorporating diverse raw financial data, which is used to define our target variable and engineer a comprehensive set of features for market trough prediction.
We gather data from several high-quality sources from April 2013 to June 2025. Table (ref) lists the main data feeds, the series identifiers, and the time periods.
We label significant market turning points using a variation of the Bry1971 Algorithm. The Bry-Boschan (BB) algorithm is a rule-based process to date business cycles. We adapt its methodology to identify significant peaks and troughs in the daily S&P 500 log adjusted closed daily price series ($P_t$) by systematiclly removing minor price movements. The overall high-level procedure is outlined in Algorithm (ref). The main procedure consists of first identifying all potential turning points of the price series, and then applying censoring rules. The complete implementation, including all helper functions, are provided in Algorithm (ref) in (ref).
The BB algorithm has a well-known property of being "backward looking", which means confirming a turning point at time $t$ requires future price series for a window after $t$. This presents a problem for identifying turning points near the end of the price series. To address the issue rigorously and make sure the labeled troughs are both complete and determined algorithmically, we apply the BB algorithm to the S&P 500 price series extending to August 12, 2025. This lengthened data window is designed to provide the BB algorithm enough data so it can algorithmically identify all turning points till the end of the primary sample period of our study (June 2025). Any data after June 2025 is only used for labeling trough and is excluded from the feature engineering, model training, or performance evaluation of our analysis. Through that, we are able to develop a complete and objective set of trough labels that is rule based and without the reliance of any subjective judgement.
Utilizing the modified BB algorithm, We identify turning points with economic significance. We show in Table (ref) the peaks and troughs that are identified in the sample period. The troughs in this table form the positive class labels for the market trough prediction model. In Figure (ref), we illustrated these troughs compared against the S&P 500 price series, and show how the algorithms marks distinct main market troughs effectively.
An important methodological concern is that the backward looking nature of BB Algorithm creates data-leakage paradox for pure prediction tasks. To prevent leakage, we explicitly framed our objective as a nowcasting problem, analogous to the real-time detection of economic recessions, whose official labels from bodies like the NBER are also confirmed with a long ex-post lag. Thus, our model's goal is not to forecast trough probability, but to estimate, with only the features available up to day t ($\mathbf{x}_t$), the probability that that day $t$ eventually be labelled as a trough in the future. That distinction is critical in preventing data leakage: while the labels ($\mathbf{y}_t$) are defined in hindsight, the predictive features are inherently historical. Thus, our approach obeys the cardinal rule of time series analysis, providing a timely signal while maintaining the intended goal of detecting market capitulation.
It is necessary to highlight the nature of this nowcasting objective. The predictive features $\mathbf{x}_t$ are strictly historical, but the target label $\mathbf{y}_t$, by construction, originates by applying the BB algorithm, which needs to be matched to future price data. Thus, we must think of our model as a device to detect real-time, contemporaneously signature of the market state that is ex-post labelled as trough, and not treated as a direct forecast of future prices.
Using raw financial data, we construct a large and diverse set of over 200 possible predictors. To assist our analysis and economic interpretation, we group the indicators into two broad categories. The first group, which we call physical or structural indicators ($\mathbf{z}_t$), are intended to identify the underlying mechanics of the market (e.g., dealer positioning, monetary condition, liquidity). They represent actual flow and constraints in the financial system. Table (ref) details the definitions, mathematical expressions and economic rationale for the most relevant structural indicators. In addition to the indicators established from the literature, we also formulate a number of new metrics to capture economic characteristics not fully explained by standard measures. For instance, to measure the persistence and uni-directional nature of recent order flow, which is a potential sign of capitulation, we define a Flow Concentration measure. Similarly, we construct a measure for Unrealized Profits to measure the financial stress of recent market participants. The complete definitions can be found in Table (ref). The second group of indicators, psychological or sentiment indicators ($\mathbf{u}_t$), is designed to quantify market fear, risk-seeking, and panic, that frequently reach extremes before market troughs. We present the complete definitions, mathematical expressions, and economic rationale for each of these indicators in Table (ref).
To enhance the robustness of our study, we conduct a systematic treatment of outliers. We remove any value of Gamma Exposure indicators ($GEX_{OI}$ and $GEX_{V}$) exceeding the 99.9th percentile threshold. In the case of the open interest-based Put/Call Ratio ($PCR_{OI}$), we treat any observation of exactly zero as data artifacts and remove it from the analysis.
Table (ref) highlights summary statistics for the primary engineered parent indicators, and they provide insights that are important in our model design. Many series reveal non-normality; for example, the kurtosis of Gamma Exposure ({$gex_{oi}$}) is 1790.7, and the kurtosis for Realized Volatility ($RV$) is 32.4. Additionally, many series, like the credit spread and the VIX, are persistent: the first-order autocorrelation coefficients ($\rho(1)$) are very high, at 0.998 and 0.970 respectively. The fat tails and strong persistence properties of raw financial series motivate us to apply non-parametric scaling and non-linear machine models later.
The features generated are subject to further transformation. For any input time series $\mathbf{X}_t$, we calculate the Rate-of-Change (ROC)\footnote{The acronym ROC is used throughout this paper in our feature names (e.g., `_roc63_`) to indicate Rate-of-Change. It should be noted that this acronym should not be confused with the Receiver Operating Characteristic (ROC) curve for evaluating models, which we refer to as the ROC AUC.}, Trend Z-Score, and Wavelet Decomposition components. Lastly, we apply a rolling percentile rank transformation across a 252-day window to all features, which converts the value of each feature into the interval $[-1,1]$. Thus, we have a robust, non-parametric representation of each feature's value with respect to its recent history.
We formulate the task as a binary classification objective, and we illustrate the mechanism in Figure (ref). For any trough date $T$, a positive label ($\mathbf{y}_t=1$) is assigned to all time steps $t$ in the $W_L$-day labeling window immediately before the trough, such that $t \in [T-W_L, T]$. All other days are assigned a negative label ($\mathbf{y}_t=0$). Thus, the objective for the model is to predict $P(\mathbf{y}_t=1 | \mathcal{X}_t)$, i.e. the probability that day $t$ is an ex-post confirmed trough timeperiod, using only the historical feature data that exists in $\mathcal{X}_t$. In order to incorporate the temporal dynamics, the input for a prediction is a tensor $\mathcal{X}_t \in \mathbb{R}^{L \times D}$, corresponding to the $D$ feature vectors from the last $L$ time steps, which assembles to the lookback window.
The sequence tensor $\mathcal{X}_t$ is aggregated to a feature vector $\mathbf{x}_t \in \mathbb{R}^{4D}$ by calculating four statistics for each of the $D$ features over the lookback window $L$: Mean, Standard Deviation, Trend (slope of linear regression), and Last Value. This is important as it converts non-stationary indicators into stationary features. An Augmented Dickey-Fuller (ADF) test demonstrated that the percentage of stationary features increased from 90.6% in the parent set to 100% in the final aggregated set, increasing the stability of the model.
We used a nested cross-validation pipeline on a `TimeSeriesSplit` of the data to select the best model while avoiding data leakage. The inner loop is for hyperparameter tuning, while the outer loop provides an unbiased estimate of generalization performance. The pipeline within each fold is:
Table (ref) presents the search space and final values determined through these steps. We selected hyperparameters using the one-standard-error rule\footnote{The one-standard-error rule is a heuristic for choosing a parsimonius model, which has statistically similar predictive performance to the most optimal model determined through cross-validation. The procedures for the one-standard-error rule are as follows. The candidate model with the maximum mean score is identified and its standard error is computed. From all candidate models whose mean score is within one standard error of the best mean score, we choose the simplest model. In our case, the prefered SVM model is specifically the one with the least features (linear kernel, and lowest C, or regularization parameter.)} to balance predictive performance against model complexity. For the final production model (trained on the entire main dataset for evaluation using the hold-out test set), the pipeline resulted in identifying the 15 features listed in Table (ref).
SVM raw scores are not well calibrated probabilities. To tackle this, we calibrate the scores into reliable probabilities using an IsotonicRegression model, which is trained on the out-of-sample predictions and true labels from each of the cross-validation folds, so as to prevent any data leakage during the calibration step. The upper panel in Figure (ref) demonstrates the good calibration of the resulting nowcast on the hold-out test set.
We evaluate our primary model on the hold-out test set, present its performance against a number of benchmarks, and provide a detailed interpretation of the model predictions and drivers.
Due to the extreme class imbalance in predicting rare market troughs, standard classification metrics are ill-posed for this problem. Precision, Recall, and F1-Score are all contingent on assigning a fixed decision threshold (e.g. 0.5) to a model's probabilistic output. When the positive class (a trough) is incredibly rare, we would expect a very well-calibrated model to assign a very low probability to this event on most days. That means the predicted probability is almost never going to cross the 0.5 mark, resulting in zero positive predictions. If the number of true positives is zero, Precision, Recall, and F1 all go to zero and the default threshold is not measuring the predictive power of the model itself, but the inappropriateness of using a default threshold. Thus, our assessment is based on two metrics that are threshold insensitive and directly assess the quality of the nowcasting market capitulation warning system:
We choose ROC AUC to measure discrimination, and Brier score to measure probabilistic reliability, because they provided the best representation of the practical value of the model.
Our primary model shows strong capability with out-of-sample predictability and reliability on the hold-out test set. It attained a ROC AUC of 0.8905, showing strong discriminatory power, and a Brier score of 0.0170, indicating reliable probabilities. A visualized summary of the results is shown in Figure (ref). The top portion has the calibration curve, and the bottom shows practical utility of the model, showing that the green spikes in predicted trough probability act as the timely and accurate signal to actual market troughs. In non-capitulating stable market regimes, the model's green probability line remain around zero, demonstrating its ability to largely avoid false alarms.
\FloatBarrier
We compare our nowcast model with a range of benchmarks in Table (ref), which provides important context for our model's performance. The LassoCV model had the highest ROC AUC (0.9495), but the extremely poor Brier score (0.2528) underscores that the model's raw outputs are completely uncalibrated/invalid as probabilities; this highlights the merit of the post-hoc calibration done in our primary pipeline. The naive heuristic (VIX > 40) had low discriminatory power (AUC of 0.6656), confirming that our framework provides better utility. Lastly, the Gaussian Naive Bayes model performed worse than random guessing (AUC < 0.5), indicating that the core idea of conditional independence that governs the Naive Bayes model is violated. Thusly, our primary SVM model provided the best convergence of high discriminatory power combined with trustworthy probability nowcasting.
To understand which factors drive the model's predictions, we employ SHAP (SHapley Additive exPlanations) Lundberg2017. Figure (ref) provides a global summary by ranking features based on their mean absolute SHAP value, which represents their average impact on the model's output magnitude. The plot clearly identifies \seqsplit{gex_oi_roc63_scaled_std} (the standard deviation of the 63-day rate-of-change in Gamma Exposure) and \seqsplit{credit_spread_roc63_scaled_std} as the two most influential predictors.
To provide further insight into the characteristics of these key drivers, Table (ref) presents descriptive statistics for the five most important features identified by our SHAP analysis. The statistics show that our feature engineering and scaling process has successfully created well-behaved inputs for the model; compared to the raw parent indicators in Table (ref), these final features have much lower skewness and kurtosis. Their high first-order autocorrelation, with $\rho(1)$ values exceeding 0.9 for most, confirms the persistent, trend-like nature of the signals the model has learned to rely on.
To understand the directionality and heterogeneity of these impacts, Figure (ref) visualizes the SHAP value for every individual prediction in our hold-out set. The interpretation reveals nuanced relationships. Examining the top feature, \seqsplit{gex_oi_roc63_scaled_std}, we observe a pattern that is consistent with the figure's caption: high values of this feature (red dots) are associated with negative SHAP values, meaning they push the prediction toward a lower probability of a trough. Conversely, low values of this feature (blue dots) have a neutral or positive impact. This suggests the model has learned that a trough is more probable not when GEX is changing chaotically, but when its rate-of-change is smoother and more persistent. For the second feature, \seqsplit{credit_spread_roc63_scaled_std}, high values (red dots) have positive SHAP values, confirming that rising volatility in credit spreads is a key indicator of market stress that contributes to the model's trough predictions.
To move beyond global importance and explore nonlinear relationships, we examine SHAP dependence plots. These plots show how a feature's marginal contribution to the prediction (its SHAP value) changes across the range of its values. Figure (ref) illustrates these relationships for our most influential predictors, revealing key nonlinearities and interaction effects learned by the model.
One of the primary difficulties for any predictive model in finance is dealing with structural breaks: a fundamental change in the data-generating process in the market, which can disrupt relationships that are learned from historical data. Conventional econometric models (e.g., OLS, VAR) with fixed parameters are particularly susceptible to structural breaks, whereas our machine learning pipeline is more robust in being designed for flexibility. Our SVM model is non-parametric and implements adaptive feature engineering over rolling windows, further enhanced by the robust time-series cross-validation protocol, so it should be robust to evolving market environments.
Nonetheless, we conduct a series of diagnostic evaluations on the hold-out sample in order to justify the robustness of our model. These evaluations are used to identify common failure modes in machine learning models, such as degraded performance, covariate shift (when the distributions of input data change), and concept drift (when the relationship between inputs and outcome change).
While the statistics of Section 4 establish the predictive validity of the model, an important test is whether its forecasts have economically important signals that are robust to specific parameter choices. To go beyond a single point nowcast of market trough probability and investigate the economic properties of the signal provided by our model, we conduct a stylized backtest simulation on the hold-out test set, including a sensitivity analysis of the strategy holding period. We have no intention to propose a final, production-ready trading strategy. We use this backtest to diagnose and characterize the nature, strengths and weaknesses of the signal generated by the market capitulation nowcast model.
We simulate a simple trading strategy from the model's daily, out-of-sample calibrated trough probabilities. We perform our backtest using the E-mini S&P 500 (ticker: ES), the most liquid equity index futures contract.
The actual simulations have the following rules:
To explore the performance characteristics of the model signals under different leverage and position-sizing rules, we evaluate two separate cases:
The sensitivity analysis presented in Table (ref) provides a nuanced account of the economic value of the model. The results illustrate that the canonical signal is robust and economically valuable, while the pyramiding leverage presents limitation, providing a deep diagnosis of the nature of the signals it generate.
First, the performance on the Fixed-size strategy illustrates the strong economic edge of the raw signal. From the sensitivity analysis, there is a clear "sweet spot" for performance, as the annualized Sharpe Ratio peaks 2.01--2.03 for holding periods of 10 to 12 days. This is a strong finding because it indicates that the model's predictive power isn't simply some artifact of the first 5-day parameter selection, but pinpoints an actual market dynamic that plays out over a two- to three-week window following a capitulation signal.
Second, the Pyramiding strategy provides deep but cautionary insights. While the headline metrics look impressive, with a Sharpe Ratio of 2.18 at 10 days, they're completely swamped by the maximum drawdown numbers. As shown in Table (ref), a maximum drawdown over 100%, witnessed in the 5-day and 7-day periods, shows the "risk of ruin" event with a total loss of initial capital and all profits. Even the very large maximum drawdown for the other periods (e.g., 40.40% at 12 days and 80.06% at 20 days)—would represent a calamity for any type of real-world investment strategy. That makes the leveraged mechanical strategy completely uninvestable for its current design.
The drawdown failure does not suggest the model is problematic, as the backtest acts as a compelling diagnostic tool. The results show that our model is a good capitulation detector and a poor bear-to-bull trend-switching validator. The model detects moments of extreme panic that can lead to a sharp, V-shaped reversal, where the pyramiding strategy magnifies the return remarkably. However, it cannot consistently distinguish a prolonged market bottom from a "bear market rally," within a longer-duration downtrend. If the model falsely indicates a bottom in time that a market cannot recover from, the pyramiding logic creates a dangerously oversized position that loses its value when the market rolls again to a new low.
The economic significance of our model does not live simply in a straightforward trading rule, but in its predictive capability as a panic and capitulation detector. The sensitivity analysis has evaluated the model's signal robustness while effectively exposing the model's predictable failure mode. This understanding is essential because it implies for practical use cases, the model signal must not be used in isolation, but as a major signal component in the overall risk management analysis, likely in conjunction with additional longer-term regime filters to prevent prematurely market entries in prolonged downtrend.
Because our trough labels by the BB algorithm are retrospective, we must be cautious when interpreting the causal parameter $\theta$. $\theta$ does not represent a causal effect of a treatment on a future price path, but the effect on the contemporaneous state of the market, or more accurately, the effect on the probability that the market is in a state that would ultimately be determined as a trough in the future. This is an important distinction in order to interpret the policy implications of our findings (see Section 7.5).
Even though the predictive model in Section 4 establish strong out-of-sample nowcasting ability, the model does not give any causal analysis between the chosen features we identified and the market troughs. To advance towards robust causal interpretation rather than just statistical correlation, we implement the Double/Debiased Machine Learning (DML) framework. We conduct our analysis in two steps. As a foundational step, we first conduct and estimate the DML based Partially Linear Model (DML-PLR), which is a standard approach in the literature. Acknowledging its limitation, we conduct a more flexible and appropriate DML specification by estimating the Average Partial Effect (APE). APE accounts for binary outcomes, feature interactions, and non-linear treatment effects. With this comparative approach, we can identify robust causal drivers and demonstrate how model specification can impact our economic conclusions.
Our causal analysis starts with a base line: the DML approach for Partially Linear Regression (PLR) models, as outlined by Chernozhukov2018. This specification provides a point of reference, with the assumption that the treatment effect is constant and additively separable. We write the structural form as:
$$\mathbf{Y} = \theta \mathbf{D} + g(\mathbf{X}) + \epsilon$$
where $\mathbf{Y}$ is the trough outcome, $\mathbf{D}$ is the treatment variable (a single indicator of interest), $\mathbf{X}$ is a high-dimensional vector of all other features that are potentially confounding, and $g(\cdot)$ is an unknown nonlinear function. The DML2 algorithm with cross-fitting provides a $\sqrt{N}$-consistent and asymptotically normal estimate for the constant treatment effect $\theta$ by flexibly modeling two nuisance functions: the outcome model $\hat{l}_0 (\mathbf{X}) = \mathbb{E}[\mathbf{Y} | \mathbf{X}]$, and the treatment model $\hat{m}_0 (\mathbf{X}) = \mathbb{E}[\mathbf{D} | \mathbf{X}]$.
In order to avoid the arbitrary choice of a single machine learning model for the nuisance functions, we use a data-driven selection process in each cross-fitting fold. In estimating the conditional mean of the treatment, $\hat{m}_0(\mathbf{X})$, we conduct a 'horse race'. A `GradientBoostingRegressor` and a `LassoCV` model are trained on the training portion of the fold. The model that exhibits better predictive performance, as assessed by out-of-sample R-squared on the validation portion of the fold, is dynamically selected for the predictions. This automatic selection improves the robustness of the DML procedure.
While the PLR specification is a standard benchmark, it has two important limitations for this setting. First, for our binary outcome $\mathbf{Y} \in \{0, 1\}$, PLR is a Linear Probability Model (LPM), which could generate predictive probabilities outside the logical [0, 1] range. Second, it imposes the restrictive assumption that the causal effect of the treatment $\theta$ is constant and additively separable from the effects of confounders. That assumption is not likely to hold in financial markets, where the signaling power of an indicator often depends on the market context.
To circumvent the limitations of the PLR framework, we employ a more flexible DML estimator based on an interactive model. We define the conditional probability of a trough as a non-linear interactive function:
$$P(\textbf{Y}=1 | \textbf{D}=d, \textbf{X}=x) = l(d,x)$$
This specification is theoretically valid for a binary outcome and enables the treatment effect to vary with the state of the high dimensional confounders. The causal parameter that we are interested in is the Average Partial Effect (APE), $\theta_0$, defined as the expected gradient of the conditional probability function with respect to the treatment:
$$ \theta_0 = \mathbb{E}_{\textbf{D},\textbf{X}}\left[\frac{\partial l(\textbf{D},\textbf{X})}{\partial \textbf{D}}\right]$$
APE measures the average change in probability of a market trough for a one unit increase in treatment, averaged across the entire data distribution. In order to estimate APE reliably, this framework requires learning these three nuisance functions:
Similar to the PLR approach, we run both `GradientBoostingRegressor` and `LassoCV` models in each cross-fitting fold as a "horse race" to determine the best performing estimator of the conditional mean $\hat{m}_0(\mathbf{X})$ and conditional variance $\hat{v}_0(\mathbf{X})$, based on out-of-sample R-squared. It ensure that nuisance parameters are estimated from the most appropriate functional form for that slice of data.
Given a standard and flexible assumption that the treatment is a heteroskedastic Gaussian process conditional on confounders, the Neyman-orthogonal score function for APE is:
$$\psi(\mathbf{W}; \theta, \eta) = \underbrace{\frac{\partial l(\mathbf{D}, \mathbf{X})}{\partial \mathbf{D}} - \theta}_{\text{Naive Score}} + \underbrace{\frac{\mathbf{D} - m(\mathbf{X})}{v(\mathbf{X})} \left( \mathbf{Y} - l(\mathbf{D}, \mathbf{X}) \right)}_{\text{Bias Correction}}$$
where $\mathbf{W} = (\mathbf{Y}, \mathbf{D}, \mathbf{X})$ and $\eta = (l, m, v)$. A complete derivation of this score function is provided in (ref). For practical estimation of the score we need to compute each of its constituents. The partial derivative term $\partial l(\mathbf{D}, \mathbf{X}) / \partial \mathbf{D}$ is numerically computed with a standard finite difference approach in the fitted outcome model $\hat{l}$. The bias correction term, which contains the three nuisance functions, is important because it ensures the final estimate of $\theta$ is robust to first order estimation error in the machine learning models. In order to be robust against the estimation "noise" from the nuisance models, especially possible outliers when $\hat{v}(\mathbf{X})$ is close to zero, our final point estimate $\hat{\theta}$ is the median of the scores computed on the out-of-sample fold, and inference proceeds with the non-parametric bootstrap of those scores. A full treatment of justification for this approach can be found in (ref). The nuisance functions are estimated by the same horse race approach with GradientBoostingRegressor and LassoCV learners as specified in the DML-PLR analysis (section 7.1).
Any credible causal estimate depend on good model specification in order to limit endogeneity. To keep comparisons between DML-PLR and DML-APE fair, the methods adopted to mitigate bad controls and sensitivity to unobserved confounding are applied equally to both framework, as explained below.
In order to eliminate spurious results from multicollinearity or "bad controls", we have an explicit exclusion map. When an aggregated variable (e.g. \seqsplit{vrp_scaled_mean}) is selected to be the treatment variable $\mathbf{D}$; we exclude all other aggregated features with the same parent indicator (i.e. \seqsplit{vrp_scaled_std}, \seqsplit{vrp_scaled_trend} from the set of potential confounders $\mathbf{X}$. We also used an exclusion map to remove any features that would be mechanistic components of the treatment. For example, because Variance Risk Premium (VRP) is defined by VIX and Realized Volatility (RV), we eliminated all features based on VIX or RV from $\mathbf{X}$ when VRP related features are the treatment $\mathbf{D}$. This procedure is essential to estimate the total causal effect of VRP, rather than an effect partially-out by its own constituent parts, which would be misleading.
Even though the DML framework accounts for the observed confounders in $\mathbf{X}$, its estimate could be biased by unobserved confounders. To address this, we subject all statistical significant DML estimates to a formal sensitivity analysis based on Cinelli2020. It quantifies how large an unobserved confounder must be (expressed by its partial $R^2$ with the treatment and the outcome) to reject the causal claim. The sensitivity analysis enable formal elimination of hypotheses that are plausible under DML but not robust under unobserved confounding. Note that while our DML estimators are non-parametric, the sensitivity analysis framework is based on a linear model. We use it as a pragmatic and conservative guide to validate our causal claim against unobserved "worst-case" linear confounders.
Our comparative causal analysis shows that robust economic insight depends on model specification. While the DML-PLR framework provides a baseline, its linear assumption masks complex non-linear and interactive relationships, sometimes even misinterpreting causal effect. On the other hand, the DML-APE framework, powered by its flexibility and non-linear interactions, yields a richer and more plausible set of drivers, and in some cases reversing the sign of the estimates from the DML-PLR framework. Table (ref) displayed an organized comparison of the results of both frameworks. The full 27 robust estimates from the DML-PLR model and 48 robust estimates for the DML-APE model are listed in (ref). Overall, the comparisons yield three core insights.
First, a small group of core causal drivers are robust to model specification. For example, both models establish that trend in the Fed Funds futures slope (\seqsplit{ffr_slope_scaled_trend}) has a statistically significant, negative causal effect on trough probability. The agreement affirms the hypothesis that market perceptions of future monetary easing establish an important stabilizing force, whether the force is linear or non-linear.
Second, the shift to a more flexible APE framework allows elimination of findings that might be spurious due to the linearity assumption in PLR. The standard deviation of the credit spread (\seqsplit{credit_spread_scaled_std}) is a clear example. The PLR model estimated a significant negative coefficient on this variable, suggesting that spread volatility is stabilizing. However, its effect is statistically indistinguishable against zero in the APE specification. It shows that the negative PLR model estimate is likely an artifact of linearity, and APE successfully captures how credit spread volatility interacts with wider market context.
Third, and most importantly, the main contribution of the DML-APE model is its identification of new causal pathways and their plausible economic implications. The APE model discovers that the volatility of measures of options based risk appetite, (e.g., \seqsplit{gex_oi_trend_z_scaled_std}, \seqsplit{risk_neutral_skewness_scaled_std}, and the VRP measure \seqsplit{vrp_roc63_scaled_std}) are causal drivers of troughs, a group of causal drivers missed by the PLR framework. This indicates a more sophisticated market mechanism, where it is not only the level of fear, but also its rate of change and persistence, that causally drives market capitulation.
The DML-APE framework also reverses the sign of several causal estimates, which resolves counter-intuitives economic interpretation from the linear PLR. The volatility of the Amihud illiquidity trend (\seqsplit{amihud_illiquidity_trend_z_scaled_std}) is a prominent example. The PLR model inferences a negative effect ($\hat{\theta}=-0.0608$), as shown in Table (ref) suggests that volatility in the illiquidity trend is a stabilizing driver. The APE model then reverses the sign of the estimate, producing a robust positive effect ($\hat{\theta}=0.0160$ in Table (ref)). This illustrates the risk of overly restrictive assumption made by the PLR model. The APE model correctly captures the market mechanism that, on average, rising instability in market liquidity is a causal precursor to a trough. Similarly, the effect of the volatility of Put / Call ratio (\seqsplit{pcr_oi_roc63_scaled_std}) flips from negative in PLR to the more intuitive positive in the APE framework, aligning with the economic intuition that a volatile and increasing demand for puts is a destabilizing force for the market.
To sum up, the comparative analysis supports the choice of DML-APE as the primary causal framework. We have moved beyond unrealistic linear assumptions to reveal that the causal drivers of market troughs are rooted firmly in the non-linear dynamic and interactions of market volatility, options implied risk appetite, and liquidity.
The formal DML and sensitivity analysis allows us to formally discard plausible but non-robust causal hypotheses. A prime example is the trend of VIX. In the initial DML procedure, \seqsplit{vix_roc63_scaled_trend} produced a statistically significant coefficient. However, the nuisance model $R^2$ is greater than 0.7, meaning that much of its variation is already explained by other controls like credit spreads. The formal sensitivity analysis shows that the finding is highly sensitive to unobserved confounding. Therefore, we cannot make robust causal claim about the VIX trend and discard it from our primary findings. This filtering procedure by sensitivity analysis is important for credible economic interpretations in the presence of unobserved confounding and high dimensional confounders.
Before bridging our causal estimates by DML with structural models, we need to revisit how we interpret causal estimates. Because our estimand represents the effect on the contemporaneously measured market state, as opposed to the future market outcome, its interpretation and policy implications are nuanced. An intervention on a causal driver of market trough should not be interpreted as a tool to prevent future capitulation, but as a way to cure the current market dynamic underlying the market capitulation. Even though an effective policy that causally reduce market illiquidity might not halt a bear market, it can fix the "fire-sale" phenomenon of the market trough. This perspective reminds us that our findings identify causal triggers for the characteristics of a market bottom, not necessarily the trough event itself.
The DML framework provides strong, reduced-form evidence about the high-frequency causal drivers of market troughs, but the broad question is how these short term treatment effects, on a timescale of a few days, bridge with canonical structural macro-finance models of financial crisis, which describes the accumulation of financial market vulnerabilities over the long term. We frame this connection as "state and trigger". Structural models like Bernanke1999, Kiyotaki1997, He2013 describe the gradual buildup of systemic fragility using state variables such as intermediary capital constraints or aggregate balance sheet health. Our empirical DML framework then uncovers the acute, observable market signals that "trigger" market trough in the fragile state. In this section, we utilize the comparative analyses between the baseline DML-PLR and DML-APE framework to achieve a more granular, structural-macrofinance model-based understanding of these triggers.
Our DML model provides a high-frequency snapshot of the financial accelerator framework of Bernanke1999. The main mechanism is the external finance premium, which is the wedge between the cost of external finance and risk free rate. It rises following the deterioration of a borrower's balance sheet. The model expresses this with an equilibrium condition:
$$\frac{\mathbb{E}_t[\mathbf{R}_{t+1}^k]}{R_{t+1}} = s\left(\frac{\mathbf{N}_{t+1}}{\mathbf{Q}_t \mathbf{K}_{t+1}}\right), \quad \text{with} \quad s'(\cdot) < 0$$
where the left-side of the equation represents the ratio of expected return to a firm’s capital ($\mathbb{E}_t[\mathbf{R}_{t+1}^k]$) to the risk-free rate ($R_{t+1}$), and the right-side represents the external finance premium. The premium is the function $s(\cdot)$ that relies on the ratio of the firm's net worth ($\mathbf{N}_{t+1}$) to the value of its capital assets ($\mathbf{Q}_{t} \mathbf{K}_{t+1}$), which resembles the firm's underlying balance sheet condition. The important property is $s'(\cdot) < 0$: as a firm's net worth declines relative to its asset value, the firm's balance-sheet degrades and the external finance premium $s(\cdot)$ increases. Our \seqsplit{credit_spread} indicator is a market-based measure of this external finance premium.
The comparison between DML-PLR and DML-APE reveal a subtle but important difference. The DML-PLR model finds a strong, negative causal effect of the standard deviation of the credit spread (\seqsplit{credit_spread_scaled_std}, $\hat{\theta}=-0.0524$), while that strong effect disappears in the less restricted DML-APE model. This suggests that the true relationship is non-linear; the causal effect of credit spread volatility is conditional on the market state, and the DML-APE model correctly captures this interaction that confounds the linear estimate in the DML-PLR model.
In contrast, the causal pathway via monetary policy expectations demonstrates strong coherence across both models. The trend in the Fed Funds futures slope (\seqsplit{ffr_slope_scaled_trend}) has a robust and statistically significant negative causal effect on market capitulation for both DML-PLR ($\hat{\theta}=-0.1436$) and DML-APE ($\hat{\theta}=-0.0073$) models. This specification-invariant finding shows that the expectation of future policy easing has an immediate, causal stabilising influence, likely due to better expectation of future corporate net worth $\mathbb{E}_t[\mathbf{N}_{t+k}]$, which prevents an increase in the external finance premium.
We have also empirically captured triggers for leverage cycles Geanakoplos2010 and liquidity spirals Brunnermeier2009. In their structural frameworks, a damaging "scary news" can cause collateral values to drop below regulatory thresholds, forcing leveraged agents into fire sales. The fire sales are further aggravated by a feedback loop between market illiquidity and tightening funding conditions. The mechanics of a fire sale are described by the liquidity spiral in Brunnermeier's model:
$$|\Lambda^{j}_{1}|=m^{j}_{1}(\phi_{1}-1)$$
where $|\Lambda^{j}_{1}|$ represents market illiquidity—specifically how much the price moves for an order size. The illiquidity is the product of two mutually-reinforcing mechanisms: tightening margin requirements $m^{j}_{1}$, which shows a leveraged collapse, and funding illiquidity $\phi_{1}-1$, which demonstrates the difficulty of financing asset purchases when there is a fire sale. The equation is a vicious cycle, as the tightening of the margin forces sales, which are amplified by funding illiquidity, which then tightens market liquidity, creating more margin calls.
Our DML comparative analysis shows the empirical proxies for the triggers of the theoretical structural models. The most significant evidence is the causal channel for market liquidity. For the volatility of the Amihud illiquidity trend (\seqsplit{amihud_illiquidity_trend_z_scaled_std}), which is a direct proxy for the instability of the market illiquidity $|\Lambda^{j}_{1}|$, the DML-PLR framework produces a robust but counter-intuitive negative coefficient ($\hat{\theta}=-0.0608$). The DML-APE framework then reversed the sign and found a robust and theoretically consistent positive coefficient ($\hat{\theta}=0.0160$). This sign flip is critical, as the APE model's finding aligns with the structural theory: as instability in market iliquidity increases, it is a causal precursor to a trough, confirmation the illiquidity spiral market dynamic.
We can unify these findings under the modern structural framework of intermediary asset pricing He2013. The framework constructs that the financial system's risk-bearing capacity comes from the wealth of the economy's specialized, leveraged intermediary sector. If this sector is financially constrained, the market price for risk rises non-linearly. The relationship is expressed by the intermediary pricing kernel:
$$\mathbb{E}_t[d\mathbf{R}_t] - r_t dt = \alpha_t^I \text{Var}_t[d\mathbf{R}_t]$$
Here, the market's excess return on average (the equity risk premium on the left) is equal to the market variance times the intermediary sector's risk aversion ($\alpha_t^I$). The key is that $\alpha_t^I$ is not constant; it increase non-linearly as intermediaries' capital is depleted. A market capitulation is the outcome of an economy being entrenched in this period of financially constraint, with a high and volatile $\alpha_t^I$. Our comparative DML analysis provides a rich description of the trigger of troughs in this region.
To sum up, our nowcasting model's successful prediction capabilities stem from its ability to learn the short-time empirical signatures of the core triggers and mechanisms, corroborated by the long-term structural macro-finance theories on financial crises. Our comparative causal analysis reveals that, to correctly understand the triggers for market troughs, the causal frameworks require specifications that account for non-linearity and interaction. DML-APE offers a more theoretically coherent account of how the long-term latent risks, delineated by the structural macro-finance models, manifest short-term observable market capitulation, providing a rich empirical validation of modern asset pricing theory.
In this paper, we tackled the market trough prediction challenge by a dual-track approach. First, we established a high-performance machine learning pipeline, culminating in a SVM classifier, which produced an ROC AUC of 0.89 out-of-sample, and whose predictions we rendered interpretable via SHAP analysis. A stylized backtest of a simple E-mini S&P 500 futures strategy confirmed the economic significance of the nowcast signal as a market capitulation detector, which performed well in its ability to detect V-shaped price reversals, though it struggles in grinding bear markets. Second, and more importantly, we moved beyond statistical correlation and did a comparative causal analysis. We argued that while the DML-PLR model provided a good baseline, the more flexible DML-APE specification properly accounted for the binary outcome and non-linear interactions problem and provided more credible economic inferences that aligned with structural macrofinance theories. The DML-APE framework not only corrected the sign of some key causal effects, but also discovered new causal pathways related to market volatility.
Our primary causal findings, robust and statistically significant under DML-APE specification and formal sensitivity analysis, suggest that the triggers for market capitulations are fundamentally grounded in the non-linear dynamics of market instability. We found that it is not simply the level of fear, but the volatility of options-implied risk appetite (e.g., GEX, risk-neutral skewness) and the instability of market illiquidity (Amihud illiquidity) that causally drive market troughs. Our results provide high-frequency empirical validation for structural long term intermediary asset pricing theories, which conceptualizes market troughs as representing a phase transition into a constrained, non-linear regime whereby the market price of risk is erratic.
This research provides several channels for future work. Firstly, an extension of this research is to tackle the symmetric but distinct challenge of predicting market peaks. It likely requires a very different feature set to capture market euphoria before the peaks compared to the environment of capitulation. Secondly, a more extended dataset going back to pre-2013, including the 2008 global financial crisis (GFC), would be an important out-of-sample test of the model's performance and robustness across distinct macroeconomic regimes, though it is hard to obtain high frequency option market data before 2013. Methodologically, while our framework enhances DML-PLR by estimating the Average Partial Effects, future research could extend to Conditional Average Partial Effects (CAPE), which would provide further insights on how these causal impacts differ contingent on the market states. Lastly, a promising frontier is to try to use Physics-Informed Machine Learning (PIML) to connect high frequency causal trigger of market trough revealed by DML-APE with long-term market dynamics explained by established structural macrofinance model through Hamilton–Jacobi–Bellman (HJB) equation, and then use deep reinforcement learning to make the framework tractable and create a generalized causal discovery for financial capitulation