Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
51,071 characters · 14 sections · 34 citation commands
The Virtue of Sparsity in Complexity
\thispagestyle{empty}
\setcounter{page}{1}
Recent work KellyMalamudZhou2024a, DidisheimKeKellyMalamud2025a, KellyMalamud2025a highlights a virtue of complexity in empirical asset pricing: large nonlinear factor sets, when combined with shrinkage, can outperform traditional parsimonious models. As high-dimensional methods become central to machine-learning-based SDF estimation, sparsity---long a guiding principle in empirical asset pricing and econometrics---appears increasingly marginalized. The prevailing message is that richer models dominate simpler ones.
DidisheimKeKellyMalamud2025a illustrate this shift by comparing models built from large factor sets with (i) traditional low-dimensional factor models and (ii) models that compress a large set of factors into a few principal components KozakNagelSantosh2020a. They show that high-dimensional specifications outperform both alternatives and argue that principal-component compression may discard low-variance directions that still contain economically meaningful pricing information. Yet the idea that “more is better" remains difficult to reconcile with the long-standing economic discipline of parsimony in asset pricing. While large factor sets may improve performance, it remains unclear what economic structure an ever-expanding set of factors actually captures.
We reconcile this tension by distinguishing two notions of sparsity: capacity sparsity and factor sparsity. The former concerns the dimensionality of the candidate feature space, whereas the latter refers to the parsimonious economic structure of priced risks. In DidisheimKeKellyMalamud2025a, the sparsity at issue pertains primarily to capacity sparsity, as their critique is directed at the underperformance of approaches that restrict the candidate factor space, whether through classical low-dimensional asset-pricing structures, dimension reduction such as PCA, or strong shrinkage imposed on a rich set of candidate factors. By contrast, factor sparsity concerns whether the economic structure of priced risks itself remains sparse. Our key insight is that complexity and factor sparsity are complements. Complexity expands the menu of candidate components, while sparsity determines which components are selected. The economic value of complexity therefore does not arise from holding more factors; rather, it arises because a richer candidate space permits sharper selection. When sparsity is imposed not as compression but as selection within a rich nonlinear expansion, its role changes fundamentally. Rather than competing with complexity, sparsity governs how complexity translates into economic performance.
To operationalize this idea, we estimate conditional stochastic discount factors (SDFs) using a rich nonlinear feature expansion together with basis pursuit, the minimum-$\ell_1$ interpolating solution. The feature expansion enlarges the candidate factor span, while basis pursuit selects a parsimonious pricing kernel within that span. In contrast to approaches that discipline complexity by compressing the candidate space itself, our approach retains high capacity and imposes parsimony at the level of the selected pricing kernel.
To test for the presence of factor sparsity, we adopt the “apples-to-apples” empirical design of DidisheimKeKellyMalamud2025a. Holding the information set fixed, we vary model complexity by expanding the number of nonlinear factor expansions $P$. A key property of basis pursuit is that it identifies at most $T$ active factors, where $T$ denotes the number of in-sample observations. Consequently, even as the candidate feature space becomes highly overparameterized-particularly in regimes where $P \gg T$---the resulting SDF remains sparse relative to the large factor models considered in DidisheimKeKellyMalamud2025a.
Our empirical findings are striking. While both the “dense” SDFs of DidisheimKeKellyMalamud2025a and the sparse SDFs from our approach improve Sharpe ratios as model complexity increases, the gains from dense SDFs plateau beyond a certain level of complexity, whereas the sparse SDFs continue to improve as complexity grows. This pattern provides strong support for our central claim: the value of complexity lies not in retaining many factors, but in expanding the search space from which a sparse economic structure of priced risks can be identified. To the best of our knowledge, this virtue of sparsity within complexity has not previously been documented and may therefore appear somewhat surprising. As a crude analogy, the phenomenon resembles the double-descent pattern documented in machine learning, which becomes visible only when model complexity is sufficiently large. Similarly, the performance advantage of our sparse SDFs emerges only at very high levels of complexity---beyond those considered in DidisheimKeKellyMalamud2025a. This reconciles the apparent tension between complexity and parsimony in empirical asset pricing: complexity expands the search space, while sparsity governs the selection of economically meaningful pricing factors.
Our results reveal that dense and sparse SDFs improve performance through fundamentally different mechanisms. Dense SDFs primarily increase Sharpe ratios through volatility reduction: as model complexity grows, diversification across many weakly predictive components lowers portfolio volatility. By contrast, sparse SDFs achieve higher Sharpe ratios through stronger mean returns. More interestingly, although sparse SDFs typically exhibit slightly higher volatility than dense SDFs, they are {\it not} riskier in economic terms. In fact, they nearly dominate dense SDFs throughout the evaluation period, exhibiting almost first-order stochastic dominance and even near-pathwise dominance. Consistent with this evidence, certainty-equivalent returns are uniformly higher for sparse SDFs across a wide range of risk-aversion levels. The magnitude of these gains relative to DidisheimKeKellyMalamud2025a suggests that sparse SDFs are more effective at extracting economically meaningful signals from a rich candidate space, whereas dense SDFs improve performance primarily through diversification across many weak predictors.
These results reframe the virtue-of-complexity debate. More is better---not because more factors are ultimately retained, but because a richer feature space improves the chance of uncovering a sparse pricing structure. The economic gains from complexity depend critically on the inductive structure that governs how signals are extracted. The dense benchmark converts complexity mainly into broad diversification across many weak components, whereas sparse selection converts complexity into targeted signal extraction. Recognizing this distinction restores sparsity as a central principle in high-dimensional asset pricing.
Our paper contributes to the growing literature that integrates high-capacity machine learning methods into empirical asset pricing. Within this literature, two closely related streams have emerged. The first focuses on improving return prediction from large sets of firm characteristics using flexible nonlinear prediction rules and extensive model comparisons. This work primarily addresses the forecasting problem at the level of individual assets or characteristic-sorted portfolios (e.g., GuKellyXiu2020a; see also FengGiglioXiu2020a for evidence on disciplined factor selection in large factor collections). A second stream applies machine learning methods to estimate conditional stochastic discount factors (SDFs) or conditional factor structures that price the cross-section of assets. Representative examples include deep-learning approaches that impose no-arbitrage moment restrictions in the estimation of the pricing kernel (e.g., ChenPelgerZhu2024a) and latent-factor models with conditioning information embedded through modern dimension-reduction architectures (e.g., KellyPruittSu2019a,GuKellyXiu2021a). Our paper falls within this second strand. We study conditional SDF estimation in high-capacity settings and examine how model complexity translates into economic performance.
Our work also contributes to the recent debate on the virtue of complexity in empirical asset pricing. Recent evidence suggests that increasingly complex models---including overparameterized feature expansions paired with shrinkage---can deliver large gains in out-of-sample performance in return prediction and in factor pricing models (e.g., KellyMalamudZhou2024a,DidisheimKeKellyMalamud2025a). At the same time, recent critiques argue that some apparent gains from complexity in return prediction can reflect statistical artifacts or unstable extrapolation in highly flexible specifications (e.g., Nagel2025a). Related work clarifies why such critiques need not overturn the core virtue-of-complexity evidence, while also highlighting that complex models can exhibit nuanced behavior that depends on design and estimator choice (e.g., KellyMalamud2025a). Our results complement this debate by sharpening the mechanism: when the hypothesis class is rich, economic performance depends not only on the availability of many candidate components, but also on the selection principle that maps complexity into factor exposures.
More broadly, our paper relates to the long-standing role of sparsity and parsimony in empirical asset pricing. Classical practice emphasizes small factor representations, while modern high-dimensional approaches often discipline estimation through shrinkage and eigen-structure-aware regularization. For example, KozakNagelSantosh2020a construct robust SDFs in large candidate sets by shrinking exposures associated with low-variance principal-component directions. In contrast, recent virtue-of-complexity evidence argues that aggressively compressing large factor sets through dimension reduction can discard economically meaningful low-variance directions and thereby degrade pricing performance (e.g., DidisheimKeKellyMalamud2025a). Our analysis clarifies that these perspectives concern different notions of sparsity. We distinguish capacity sparsity, which limits the dimensionality of the candidate feature space, from factor sparsity, which concerns whether the underlying economic structure of priced risks remains parsimonious. This distinction allows complexity and sparsity to coexist: complexity expands the hypothesis class, while sparsity governs selection of economically meaningful components within it.
This perspective also connects to insights from statistical learning theory that emphasize the role of inductive bias in overparameterized regimes. When many solutions fit the data similarly well, generalization is governed not only by model capacity but also by the estimator's implicit preferences over the solution set. Wilson2025a develops this idea through the lens of soft inductive bias, and related work on interpolation, double descent, and benign overfitting shows how selection rules within high-capacity classes can produce counterintuitive out-of-sample behavior even when training error is driven to zero (e.g., HastieMontanariRossetTibshirani2022a,BartlettLongLugosiTsigler2020a). Applied to asset pricing, the relevant question is therefore not only whether the SDF is estimated in a high-dimensional space, but which pricing kernel is selected from a large set of near-equivalent high-dimensional solutions.
Finally, our findings provide a complementary perspective on the “factor zoo” in empirical asset pricing. The proliferation of proposed factors has long been noted (e.g., Cochrane2011a) and is closely tied to concerns about multiple testing, replication, and post-publication decay in characteristic-based predictors (e.g., HarveyLiuZhu2016a,HouXueZhang2020a,McLeanPontiff2016a). At the same time, modern open-data efforts document broad reproducibility of many published predictors when construction choices are standardized (e.g., ChenZimmermann2022a, JensenKellyPedersen2023a). Our results suggest an additional interpretation that is especially salient in high-capacity conditional SDF settings: the challenge is not merely that many candidate factors exist, but that identifying the economically relevant ones requires a powerful search space combined with an effective selection mechanism.
This section develops the empirical framework for estimating conditional stochastic discount factors (SDFs) and introduces the two notions of sparsity that arise in high-capacity asset pricing models. We begin with the conditional SDF representation used in modern empirical asset pricing. We then distinguish capacity sparsity from factor sparsity and explain how increasing model complexity can help reveal sparse economic structure. Next, we present our estimation approach, which combines scalable nonlinear feature expansions with basis pursuit to identify sparse pricing kernels within a large factor span. Finally, we characterize sparse and dense interpolating SDFs, clarifying when and why sparse selection can outperform ridgeless estimation.
Let $R_{t+1} \in \mathbb{R}^{N_t}$ denote the vector of excess returns on the $N_t$ traded assets available at time $t$, and let $\mathcal F_t$ denote the information set available at time $t$. Absence of arbitrage implies the existence of a stochastic discount factor (SDF) $M_{t+1}$ satisfying
where $0_{N_t}$ denotes the zero vector in $\mathbb{R}^{N_t}$.
We focus on tradable SDFs. Following HansenRichard1987a, any such SDF can be represented as the gross payoff of a managed portfolio,
where $w_t \in \mathbb{R}^{N_t}$ is $\mathcal F_t$-measurable.
Let $Z_{i,t} \in \mathbb{R}^D$ denote the characteristic vector of asset $i$ at time $t$, and let $s_1,...,s_P$ be $P$ basis functions of these characteristics. Define the feature matrix $$S_t \in \mathbb{R}^{N_t \times P},\;\; (S_t)_{i,p} = s_p(Z_{i,t}). $$ We parameterize portfolio weights linearly in these features:
where $\lambda \in \mathbb{R}^P$ is a coefficient vector.
This induces the $P$-dimensional vector of characteristic-managed factor returns
Substituting (ref) into (ref) gives
Thus conditional SDF estimation can be viewed as selecting a portfolio within the span of characteristic-managed factors $F_{t+1}$.
Taking unconditional expectations in the pricing condition yields the necessary moment restriction
Let $\Sigma := E[F_{t+1}F_{t+1}']$ and $\mu := E[F_{t+1}]$. When $\Sigma$ is nonsingular, the population coefficient vector satisfies
Equivalently, $\lambda^*$ is the common optimizer of the mean-variance problem, the Sharpe-ratio problem, and the pricing-error problem:
Empirically, conditional SDFs can be estimated through the mean-variance formulation, with performance typically evaluated through Sharpe ratios or pricing errors.
Recent work on the “virtue of complexity” DidisheimKeKellyMalamud2025a studies regimes in which the number of candidate factors \(P\) is large relative to the time-series sample size \(T\). In such settings, estimation is often stabilized through ridge regularization,
For \(\gamma>0\), this formulation introduces \(\ell_2\)-shrinkage into the estimator. For non-negligible values of \(\gamma\), the shrinkage acts as a form of capacity control by disciplining estimation in a large candidate factor space. We refer to this broader idea as capacity sparsity. It can also arise through ex ante restrictions on the factor span (for example, selecting a small number of economically motivated factors), through dimension-reduction methods such as principal components, or through shrinkage more broadly.
By contrast, our focus is on a different notion of sparsity, which we call factor sparsity. Rather than restricting the size of the factor span, factor sparsity concerns the structure of the pricing kernel itself. Under this view, the true SDF may depend on only a small subset of economically meaningful directions, even when the candidate factor space is large. The key difficulty, however, is that this sparse structure is unknown ex ante. Precisely for that reason, increasing model complexity can play a constructive role. By enlarging the space of candidate factors, richer feature representations expand the set of directions in which the true sparse structure may lie. Complexity therefore facilitates discovery, while sparsity governs selection within the enlarged space of candidate pricing kernels. Figure (ref) provides a schematic illustration of these two notions of sparsity and their interplay. In the figure, the sparse region can be understood as the set of pricing kernels with at most \(T\) active loadings, consistent with the later role of basis pursuit when the true SDF is representable in this way.
The entire region in Figure (ref) represents the candidate space of pricing kernels at a given complexity level. The high-pricing-error region contains kernels that fail to price assets well. The low-pricing-error, overfitting region contains kernels that fit the data in sample but generalize poorly, whereas the low-pricing-error, good out-of-sample region contains kernels that both fit well and generalize well. The black dot denotes the true SDF \(\lambda^\ast\). In the right panel, overparameterization enlarges the candidate space, but the different regions need not expand proportionally, consistent with the intuition in Wilson2025a. The figure is intended only as intuition, but it conveys the key idea: with limited capacity, the sparse structure is harder to uncover, whereas a richer candidate space increases the chance that economically meaningful sparse directions become identifiable. Basis pursuit, introduced later, is then designed to steer selection toward such sparse, well-generalizing kernels and, when the true sparse direction is represented in the enlarged span, toward \(\lambda^\ast\).
To operationalize factor sparsity, we retain a large factor span and impose sparse selection within it. It is convenient to work with the regression form of conditional SDF estimation, because in overparameterized regimes it makes the selection problem especially transparent. Let \(F_P\in\mathbb R^{T\times P}\) denote the in-sample matrix of managed-factor returns at complexity level \(P\), whose \(t\)-th row is \(F_{t+1}'\), where \(F_{t+1}\) is defined in (ref). Let \(\mathbf 1\in\mathbb R^T\) denote the unit payoff vector. Let $\mathcal{P}(\lambda)$ denote a regularization term. Then SDF estimation can be written as
When $P$ is large relative to $T$, the system may admit exact interpolating solutions---many pricing kernels price the managed factors equally well in sample. The central question becomes which SDF is selected from the set of exact interpolating pricing kernels \[ \mathcal I_P:=\{\lambda\in\mathbb R^P:F_P\lambda=\mathbf 1\}. \]
A natural selection principle under factor sparsity is the basis-pursuit limit of $\ell_1$ regularization.
\paragraph{Basis Pursuit} With $\ell_1$ regularization, \[ \mathcal{P}(\lambda)=\alpha\|\lambda\|_1, \] the estimator of (ref) converges as $\alpha\downarrow0$ to the basis pursuit solution
This formulation selects the minimum-$\ell_1$ pricing kernel among those that exactly price the managed factors in sample.
To enrich the candidate factor span without changing the underlying information set, we generate nonlinear transformations of the same $D$ base firm characteristics using random Fourier features (RFF) RahimiRecht2007a. Let \(Z_{i,t} \in \mathbb{R}^{D}\) denote the standardized characteristic vector for stock \(i\) at time \(t\). We generate \(P\) random features as:
where \(\boldsymbol{\omega}_p \sim \mathcal{N}(0, \sigma^2 \mathbf{I}_{D})\) and \(b_p \sim \text{Uniform}[0, 2\pi]\). The bandwidth parameter \(\sigma\) controls the smoothness of the implied kernel.
In support of our proposed procedure, we characterize when and why the sparse SDF selected by basis pursuit can outperform its dense counterpart given by the ridgeless $\ell_2$ criterion. Consider the limit of the ridge-regularized problem (ref) with $\mathcal P(\lambda)=\alpha\|\lambda\|_2^2$ as $\alpha\downarrow0$. In the overparameterized regime $P>T$, this selects the minimum-$\ell_2$ interpolator
commonly referred to as ridgeless.
The distinction between sparse and dense SDFs is not one of in-sample fit: both interpolate exactly. It is entirely about selection from the same interpolation set. We begin with a basic structural property of basis pursuit: even in the overparameterized regime, it admits an optimal solution whose support size is bounded by \(T\).
For our purposes, the key implication of Lemma (ref) is that enlarging the candidate factor span need not lead to a commensurate increase in the effective dimension of the selected SDF. Thus, basis pursuit can search over a much richer nonlinear span while remaining low-dimensional relative to the ambient feature space.
We next characterize how sparse and dense SDFs can differ in out-of-sample mean return. Let \[ \Pi_P:=F_P'(F_PF_P')^{-1}F_P \] denote the orthogonal projector onto \(\operatorname{row}(F_P)\), and let \[ M_P:=I-\Pi_P \] denote the orthogonal projector onto \(\ker(F_P)\). Here \(\operatorname{row}(F_P)\) is the subspace spanned by the rows of the in-sample managed-factor matrix \(F_P\), and \(\ker(F_P)\) is its orthogonal complement.
Let \(F_{P,t+1}^{oos}\in\mathbb R^P\) denote the out-of-sample managed-factor return vector associated with the \(P\)-dimensional feature expansion, and define \[ \mu_P:=\mathbb E[F_{P,t+1}^{oos}], \] where the expectation is taken with respect to the data-generating distribution of firm-characteristic and next-period-return pairs. Write \[ \mu_P^\parallel:=\Pi_P\mu_P,\qquad \mu_P^\perp:=M_P\mu_P, \] so that \[ \mu_P=\mu_P^\parallel+\mu_P^\perp, \qquad \mu_P^\parallel\in\operatorname{row}(F_P),\qquad \mu_P^\perp\in\ker(F_P). \] This decomposition is natural for comparing sparse and dense SDFs. As shown below, the distinction between \(\operatorname{row}(F_P)\) and \(\ker(F_P)\) governs exactly how sparse and dense selection can differ in mean return.
Define \[ \bar r_P(\lambda):=\mu_P'\lambda. \]
Proposition (ref) makes the mechanism transparent. The ridgeless interpolator is confined to $\operatorname{row}(F_P)$, whereas any feasible deviation from ridgeless must lie in $\ker(F_P)$. From the perspective of expected returns, this means that the minimum-$\ell_2$ rule can exploit only the component of population mean structure that projects onto the subspace generated by historical in-sample pricing relations; any mean-bearing structure orthogonal to that subspace is necessarily left unused. Basis pursuit, by contrast, is not confined in this way: its kernel displacement can access such directions while preserving exact in-sample fit. The mean-return advantage of basis pursuit is therefore governed by three quantities: the magnitude of omitted mean-bearing structure $\|\mu_P^\perp\|_2$, the size of the sparse kernel displacement $\|h_P^{BP}\|_2$, and its alignment with that structure through $\cos\theta_P$. In particular, basis pursuit exceeds ridgeless in mean exactly when its kernel displacement is positively aligned with the omitted component.
More importantly, the proposition clarifies how complexity can help. As the candidate span becomes richer, the component of mean returns omitted by the ridgeless row space may grow, creating additional scope for sparse selection to exploit economically relevant structure. If basis pursuit continues to make a nontrivial and favorably aligned kernel move, that scope translates into a widening mean-return advantage. Among the three conditions in Proposition (ref), condition $(ii)$ is the mildest in overparameterized settings, since basis pursuit and ridgeless will typically differ by a nontrivial kernel component. Condition $(i)$ is also plausible as complexity grows, because a fixed-\(T\) row space occupies a shrinking fraction of the ambient space as \(P\) increases. The most substantive condition is $(iii)$: sparse selection must not merely differ from ridgeless, but differ in a direction positively aligned with omitted mean-bearing structure. This is precisely where our hypothesis of the virtue of sparsity in complexity enters.
We next examine how the volatility of sparse SDFs behaves as complexity increases. A useful benchmark observation is that the scale of the sparse SDF remains controlled. Let \[ v_P:=\min_{\lambda\in\mathcal I_P}\|\lambda\|_1. \] Then \[ \|\hat\lambda_P^{BP}\|_2\le \|\hat\lambda_P^{BP}\|_1=v_P. \] If the feature expansions are nested in $P$, then $v_P$ is nonincreasing in $P$, since the optimizer at a lower complexity level remains feasible after padding with zeros. Thus, under nested expansions, the $\ell_2$ norm of the sparse SDF is bounded above by a nonincreasing sequence. If, in addition, the managed-factor covariance matrix $\Sigma_P$ satisfies \[ \lambda_{\max}(\Sigma_P)\le \bar\sigma^2 \] over the range of interest, then \[ \sqrt{\hat\lambda_P^{BP\prime}\Sigma_P\hat\lambda_P^{BP}} \le \bar\sigma\,\|\hat\lambda_P^{BP}\|_2 \le \bar\sigma\,v_P. \]
This benchmark argument complements Proposition (ref): the proposition explains how sparse selection can generate a mean advantage, while the scale-control argument shows why the volatility of the sparse SDF may also decline as complexity increases and, more generally, need not increase uncontrollably. Taken together, they provide a coherent explanation for our empirical finding that the Sharpe ratio of sparse SDFs ultimately continues to improve in rich feature spaces: complexity expands the scope for sparse selection to exploit omitted mean-bearing structure, while scale remains sufficiently controlled for those mean gains to translate into stronger risk-adjusted performance.
More broadly, this mechanism gives theoretical support to our hypothesis of the virtue of sparsity in complexity. When mean performance improves, that advantage strengthens with complexity, and the selected support remains highly stable and parsimonious, the evidence points to factor sparsity: the economically relevant pricing structure is concentrated in a small subset of directions within the expanded nonlinear span.
This section presents the empirical design and out-of-sample evidence on how sparse and dense SDFs evolve as complexity $P$ increases.
\paragraph{Data} We use the same data construction and source as DidisheimKeKellyMalamud2025a. The underlying dataset is the comprehensive sample of monthly US stock returns and firm characteristics compiled by JensenKellyPedersen2023a (JKP); stock-level data and documentation are available at \url{https://jkpfactors.com}. The universe includes NYSE, AMEX, and NASDAQ common stocks (CRSP share codes 10, 11, or 12) excluding nano caps (market cap below the first percentile of NYSE market cap). We reduce the 153 JKP characteristics to \(D=130\) with the fewest missing values, drop stock-months with more than 30% missing among these 130, and cross-sectionally rank-standardize each characteristic to \([-0.5,0.5]\) following GuKellyXiu2020a.
\paragraph{Model setup}
We summarize model complexity by \[ c := P/T, \] where \(T\) denotes the training-window length. In our implementation, we fix \(T = 60\) months (five years of training data) and vary \(P \in \{6, \ldots, 30000\}\), which yields complexity ratios
This grid spans the underparameterized regime (\(c < 1\)), the interpolation threshold (\(c = 1\)), and the overparameterized regime (\(c > 1\)). For each value of \(c\), we generate 20 independent RFF draws, corresponding to different realizations of \(\{\boldsymbol{\omega}_p, b_p\}\), and average the results to reduce Monte Carlo variation due to any particular feature realization.\footnote{Following KellyMalamudZhou2024a, we independently draw the bandwidth parameter $\sigma$, which controls the smoothness of the implied kernel, uniformly from the grid $\{0.5,0.6,0.7,0.8,0.9,1.0\}$ for each independent draw of the RFF frequencies $\{\boldsymbol{\omega}_p,b_p\}_{p=1}^P$ in (ref). This embeds varying degrees of nonlinearity in the resulting random features.}
\paragraph{Evaluation protocol and performance metrics} Our evaluation design follows the rolling-window framework of DidisheimKeKellyMalamud2025a. Fixing a training-window length of \(T=60\) months (five years), we roll the estimation window forward one month at a time. At each formation month \(t\), we estimate \(\hat{\boldsymbol{\lambda}}_t\) using the most recent \(T\) months of training data. Using the characteristics observed at month \(t\) and the asset returns realized in month \(t+1\), we form the corresponding managed-factor return vector \(F_{t+1}\). The realized out-of-sample SDF portfolio return is then \[ \hat r_{t+1}=\hat{\boldsymbol{\lambda}}_t'F_{t+1}. \] Repeating this procedure yields a sequence of \(N=360\) monthly out-of-sample returns, denoted by \(\{\hat r_\tau\}_{\tau=1}^N\).
The primary performance metrics are the monthly out-of-sample Sharpe ratio,
and the out-of-sample Hansen--Jagannathan distance (HJD),
where \[ \hat M_\tau = 1-\hat{\boldsymbol{\lambda}}_{\tau-1}'F_\tau, \qquad \hat{\mathbb E}_{OS}[\,\cdot\,]=\frac{1}{N}\sum_{\tau=1}^{N}[\,\cdot\,], \] and \((\cdot)^+\) denotes the Moore--Penrose pseudoinverse. Here \(F_\tau\) denotes the managed-factor excess-return vector in the \(\tau\)-th out-of-sample month.
To characterize the performance comparison more fully, we move beyond volatility-based summaries. This is particularly important in our setting, where, as shown later, higher volatility does not imply greater economic risk. We therefore examine the return distribution directly through stochastic-dominance comparisons, using Value-at-Risk (VaR), Expected Shortfall (ES), and the analogous upper-tail mean across a range of percentiles. Let \(\{\hat r_\tau(c)\}_{\tau=1}^N\) denote the monthly out-of-sample SDF returns at complexity level \(c\), and let \(Q_q(c)\) denote the empirical \(q\)-quantile of this sample. For any \(q\in(0,1)\), we define
and, for \(q>0.5\), define the upper-tail mean by
Thus, \(\mathrm{VaR}_q(c)\) records the empirical lower-tail quantile, \(\mathrm{ES}_q(c)\) averages returns below that quantile, and \(\mathrm{UTM}_q(c)\) averages returns above the corresponding upper-tail quantile. In the empirical analysis, we trace these quantities across a spectrum of percentile levels to assess how the distributional comparison between sparse and dense SDFs evolves across both downside and upside tails.
Finally, to translate statistical outperformance into investor welfare, we compute certainty-equivalent (CE) returns for a CRRA investor with relative risk aversion parameter \(\gamma\). Let \(R_{\tau}(c)=1+\hat{r}_{\tau}(c)\) denote the gross monthly out-of-sample return at complexity level \(c\). The CE gross return \(R^{\mathrm{CE}}(c)\) is defined as the constant gross return that delivers the same average power utility as the realized return sequence:
which implies
Figure (ref) presents the main results: virtue-of-complexity curves for out-of-sample mean returns, volatility, Sharpe ratios, and pricing error as functions of complexity \(c\).\footnote{Because the full complexity grid compresses the low-\(c\) region, we report virtue-of-complexity curves for \(c<10\) separately in \hyperref[app:low_c]{Appendix B}.} In the overparameterized regime, the dense SDF (ridgeless) plateaus in mean return, whereas the sparse SDF (basis pursuit) continues to improve as complexity rises. Volatility declines for both specifications, with the dense SDF remaining less volatile, consistent with its more diversified structure. Pricing error also falls for both models, although the sparse SDF ultimately achieves the lower pricing error. As a result, the sparse SDF eventually overtakes the dense SDF in Sharpe ratio. This pattern is consistent with our central mechanism: as complexity rises, the richer feature span gives sparse selection access to a broader set of candidate directions, while the sparse estimator continues to select a parsimonious pricing kernel. The gains therefore appear to come not from retaining more factors, but from allowing a sparse structure to be identified within a much richer nonlinear span. Most importantly, this outperformance emerges only in the very high-complexity regime, \(c>2000\), well beyond the range examined in prior work, where the upper limit of the complexity grid in DidisheimKeKellyMalamud2025a is \(1{,}000\). That is precisely where our central claim becomes visible: complexity and factor sparsity are complements, and the value of sparsity fully reveals itself only when the candidate span is sufficiently rich.
The fact that the sparse SDF often exhibits slightly higher volatility than the dense benchmark may initially suggest a standard risk--return tradeoff. As we show in this section, that interpretation can be misleading. A closer look reveals that volatility is not the most informative notion of risk in our setting. Once we examine return quantiles, Value-at-Risk, and Expected Shortfall across a wide range of percentiles, we find that the sparse SDF’s advantage is not confined to the mean. In the high-complexity regime, it extends across nearly the entire return distribution (see Figure (ref)), indicating that the sparse SDF nearly stochastically dominates the dense SDF. A more nuanced picture emerges when we trace the distribution across complexity. The effect of complexity is not symmetric across the upper and lower tails. As complexity increases, downside performance improves markedly, whereas upper-tail performance becomes flatter and may weaken somewhat. At low complexity, the sparse SDF already tends to outperform the dense benchmark in the upper tail, but can lag in parts of the downside tail. As complexity rises, however, this downside gap closes and eventually reverses. The complementarity between complexity and factor sparsity is therefore especially visible in downside protection and mean performance. By the highest complexity levels, the sparse SDF dominates the dense benchmark across almost the full return distribution.
To push the comparison to its most granular level, Figure (ref) plots the realized monthly returns of the two SDF portfolios side by side in the highest-complexity regime. Strikingly, the sparse SDF exceeds the dense benchmark in nearly every month. This near-pathwise dominance provides especially compelling evidence that the advantage of sparse selection is not a fragile artifact driven by a few exceptional periods.
To quantify the welfare gains delivered by sparse SDFs, we compute certainty-equivalent (CE) returns for a constant relative risk aversion (CRRA) investor. Unlike the Sharpe ratio, the CE maps the full distribution of SDF portfolio returns into investor utility and thus provides a direct measure of economic value. As shown in Figure (ref), the sparse SDF delivers higher investor welfare than the dense SDF at every level of risk aversion we consider, although the magnitude of this advantage varies with \(\gamma\).
The welfare advantage of the sparse SDF is largest at low risk aversion (\(\gamma=1\)), where CE comparisons are driven primarily by mean returns—the main channel through which basis pursuit outperforms ridgeless portfolios. As risk aversion rises, the CE gap narrows, reflecting the greater sensitivity of certainty-equivalent comparisons to finer differences across the full return distribution. The welfare gains from the sparse portfolio therefore become smaller for more risk-averse investors, but remain positive. Even at \(\gamma=5\), the sparse SDF continues to deliver higher CE returns in the high-complexity regime. Taken together with the distributional-dominance evidence, these results show that the advantage of the sparse SDF is not merely statistical: it survives translation into economically meaningful welfare criteria.
Our central claim is that basis pursuit selects a genuinely sparse pricing kernel from a very large candidate span. Figure (ref) provides direct evidence for this claim. Although the theory permits up to \(T=60\) active coefficients, the empirical support size is far smaller—about 31--33 across the entire high-complexity regime. Strikingly, this number remains highly stable, well below the theoretical ceiling of 60, even as complexity rises from \(c=500\) to \(c=5{,}000\). This pattern shows that basis pursuit is not merely enforcing a numerical cap on the number of selected factors; rather, as complexity increases, it continues to identify a similarly small and stable set of directions of progressively higher quality, thereby delivering steady improvements across all out-of-sample performance metrics.
This paper studies how complexity and parsimony interact in empirical asset pricing. We argue that the apparent tension between them reflects a distinction between two different notions of sparsity: capacity sparsity, which restricts the size of the candidate factor space, and factor sparsity, which concerns the parsimonious structure of priced risks within that space. Our central point is that the economic value of complexity arises not because large factor spaces should be used densely, but because richer spaces create more scope for informative sparse selection.
Empirically, we show that when the information set is held fixed and model complexity is increased through richer nonlinear feature expansions, sparse and dense selection lead to sharply different outcomes. The sparse SDF selected by basis pursuit continues to improve in the high-complexity regime, whereas the dense ridgeless benchmark plateaus. The advantage of sparse selection is not confined to average returns: it also appears in the return distribution and in investor welfare, indicating that factor sparsity is an economically meaningful property of the pricing kernel rather than a purely statistical convenience.
Our analysis helps clarify why this pattern can arise. In overparameterized settings where many pricing kernels fit the data equally well in sample, performance depends not only on the richness of the factor space but also on the selection rule used to choose among admissible kernels. Sparse selection can outperform dense interpolation when it moves toward economically relevant directions that the minimum-\(\ell_2\) rule does not exploit, and that advantage can become larger as complexity increases.
Taken together, our results reframe the virtue-of-complexity debate. Complexity is valuable not because more factors are ultimately retained, but because a richer candidate space enlarges the set of directions from which sparse selection can recover economically meaningful signals.