Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
51,970 characters · 17 sections · 61 citation commands
Scale-Insensitive Neural Network Significance Tests
\thispagestyle{empty}
Keywords: Neural Networks; Variable Significance Test; Rademacher complexity\\ JEL classification: C1; C5
\setstretch{1}
Deep neural networks have emerged as powerful tools for nonparametric estimation, consistently demonstrating superior predictive performance across many domains goodfellow2016deep,anthony1999neural. However, their effectiveness comes at the cost of interpretability - while these models can capture complex nonlinear relationships, understanding which input variables significantly influence their predictions remains challenging. Recent work has begun to address these challenges through both scale-insensitive analysis and variable significance testing.
A significant methodological advance came from farrell2021deep, who established a scale-insensitive framework for analyzing neural networks. Their approach avoids restrictive weight constraints through careful localization arguments, enabling rigorous statistical analysis of modern deep learning practices. In parallel, horel2020significance developed the first theoretically-based statistical test for assessing variable impact in regression using a one-layer sigmoid network. fallahgoul2024asset extended horel2020significance's work to multi-layer perceptrons (MLPs), which are generally preferred over one-layer networks due to their superior approximative power. However, their framework, based on metric entropy analysis, still requires several restrictive assumptions including bounded weights and specific architectural constraints.
This paper bridges these two lines of research by extending the framework of fallahgoul2024asset using scale-insensitive measures inspired by farrell2021deep. Specifically, we replace metric entropy calculations with Rademacher complexity bounds, drawing on results from KoltchinskiiPancheko2000 and Bartlett2005. Our methodology closely aligns with farrell2021deep, but addresses a different challenge: while they focus on causal inference through partial effects, we develop rigorous significance tests for neural network inputs.
Our theoretical framework rests on three key innovations. First, following farrell2021deep, we use scale-insensitive complexity measures that enable the analysis of neural networks without requiring bounded weights or specific architectural constraints. Second, we weaken the regularity conditions on the target function to require only Sobolev space membership $H^s([-1,1]^d)$ with $s > d/2$, substantially relaxing the conditions in fallahgoul2024asset. Third, we introduce a modified sieve space construction based on moment bounds rather than weight constraints, providing a more natural theoretical framework for contemporary deep learning practices.
Our analysis proceeds through careful localization arguments similar to those in farrell2021deep. We first establish consistency of the neural network estimator and derive non-asymptotic bounds on its convergence rate. Using tools from empirical process theory vaart1996weak, we then show that the test statistic has a well-defined limiting distribution under the null hypothesis. The key technical challenge lies in controlling the complexity of the function class without imposing artificial constraints on the network architecture, which we address through Rademacher complexity bounds.
Our theoretical results build upon and complement existing approaches in deep learning theory. In particular, our work and farrell2021deep both achieve optimal rates through distinct technical approaches that illuminate different aspects of neural network estimation. farrell2021deep develop a precise architecture-specific analysis, building on yarotsky2017error,yarotsky2018optimal's explicit ReLU constructions to provide sharp bounds and detailed insights into network expressivity. Their careful control of network parameters yields tight constants and explicit optimization guidance, revealing intricate connections between architecture and statistical complexity.
Our framework takes a complementary path, employing general functional analytic methods and scale-insensitive measures through Rademacher complexity. This approach accommodates broader network architectures and activation functions under moment conditions, trading some constant-specificity for increased generality. Where farrell2021deep leverage explicit ReLU constructions for precise parameter efficiency bounds, we use general Sobolev space approximation properties to establish optimal rates across a broader function class. This contrast reflects a fundamental tension in deep learning theory between architecture-specific precision and architecture-agnostic flexibility.
Both approaches utilize localization techniques to different effect. farrell2021deep's detailed shell-by-shell analysis provides sharp control of the approximation-estimation trade-off through careful handling of unbounded weights. Our more streamlined approach focuses on concentration inequalities, yielding simpler proofs while maintaining optimal rates. These complementary perspectives—farrell2021deep's architectural precision and our functional generality—contribute to a deeper understanding of the relationship between network structure and statistical estimation.
This paper contributes to several strands of literature. First, we advance the theoretical understanding of deep neural networks by providing rigorous statistical inference tools with weaker assumptions than previous work horel2020significance, fallahgoul2024asset. Second, we extend the scale-insensitive analysis of farrell2021deep to the problem of variable significance testing. Third, we contribute to the broader literature on significance testing in nonparametric models by developing methods applicable to modern machine learning architectures.
The remainder of the paper is organized as follows. Section (ref) establishes the foundational elements of our scale-insensitive framework, including the regression setting, modified regularity conditions, hypothesis testing framework, neural network architecture, Rademacher complexity measures, and estimation methodology. Section (ref) develops the main theoretical results for scale-insensitive neural network tests. Specifically, Subsection (ref) establishes consistency results for neural networks under our relaxed conditions, Subsection (ref) introduces scale-insensitive sieve spaces that replace traditional metric entropy calculations, and Subsection (ref) presents the asymptotic distribution theory for our test statistics. Section (ref) provides the conclusions of this study. All proofs are relegated to the appendix.
In this section, we establish the foundational elements of our scale-insensitive framework for neural network significance testing. We begin by introducing the regression framework and necessary assumptions, followed by the hypothesis testing framework and the architecture of deep neural networks under consideration. Readers who are interested in the proofs may refer to appendixes.
Let $(\Omega, \mathcal{F}, P)$ be a probability space where we observe the relationship between a response variable $Y$ and covariates $X \in \mathcal{X} := [-1,1]^d$ through the regression model
Unlike previous frameworks, fallahgoul2024asset, horel2020significance that rely on bounded weights and strict architectural constraints, we introduce scale-insensitive conditions through moment bounds. For the target function $f_\star$, we require
where $M$ is a positive constant and $\alpha$ represents multi-index notation for derivatives.
We now introduce weaker regularity conditions that replace the classical smoothness and structural assumptions in fallahgoul2024asset, horel2020significance.
Our framework substantially modifies the original structural assumptions from fallahgoul2024asset, building on the scale-insensitive approach pioneered by farrell2021deep. In particular, let us examine the original Assumptions 2.2 and 2.3, which require:
Instead, motivated by the scale-insensitive localization theory of farrell2021deep, KoltchinskiiPancheko2000, Bartlett2005, we propose requiring $f_\star \in H^s([-1,1]^d)$ for $s > \max\{1, d/2\}$. This modification has several important technical implications worth discussing.
First, the intersection of $\mathcal{C}^{\lfloor d/2\rfloor + 2}([-1,1]^d)$ and $\mathbb{H}^{\lfloor d/2\rfloor + 2,2}([-1,1]^d)$ in the original framework imposed a strong dual requirement: both classical and weak differentiability of high order. Our framework eliminates the need for classical differentiability entirely, requiring only Sobolev regularity. This shift from scale-sensitive to scale-insensitive measures provides two tangible benefits: we do not restrict the class of network architectures to have bounded weights for each unit, and we allow for a richer set of approximating possibilities.
Second, our framework eliminates the binary tree visual structure requirement from the original Assumption 2.3. While such structure provided a convenient way to analyze the approximation capabilities of neural networks in the original paper, we show that this architectural constraint is unnecessary when working with scale-insensitive complexity measures. This relaxation, in line with standard practice where optimization is not constrained zhang2021understanding, allows our framework to encompass a broader class of neural network architectures.
Our framework assesses variable significance in neural networks by examining partial derivatives, building on fallahgoul2024asset and horel2020significance. The key insight is that if a variable $X_j$ has no influence on the response $Y$, then $\frac{\partial f_\star(x)}{\partial x_j} = 0$ for all $x \in \mathcal{X}$.
We quantify this influence through a weighted $L^2$ norm of the partial derivative:
This leads to the natural hypothesis testing problem:
The test statistic in (ref) has several desirable properties:
The measure $P$ in (ref), representing the distribution of $X$, provides natural weighting that emphasizes effects in high-density regions of the feature space.
For linear functions $f_\star(x) = \sum_{k=1}^d \beta_k x_k$, our framework in (ref) reduces to classical significance testing
Thus, $H_0: \zeta_j = 0$ becomes equivalent to the familiar linear hypothesis $H_0: \beta_j = 0$. However, our framework extends naturally to nonlinear settings where $\frac{\partial f_\star(x)}{\partial x_j}$ varies with $x$, providing a more comprehensive measure of variable influence through the weighted integration in (ref).
This section covers the mathematical foundation of deep neural networks. For a more in-depth understanding, we recommend consulting the works of anthony1999neural and goodfellow2016deep. farrell2021deep also offer a thorough discussion on deep neural networks and their relationship to nonparametric estimation.
Multi-layer perceptrons (MLPs) represent a fundamental class of neural networks where information flows from the input layer through one or more hidden layers to an output layer. Each layer consists of computational units (nodes) that process incoming signals through an activation function, with connections between layers characterized by learnable weights. Figure (ref) illustrates a typical MLP architecture.
We consider MLPs with depth $L_d$ and width $H_n$ per layer, where $H_n$ increases with sample size $n$. Our framework differs from previous approaches fallahgoul2024asset,horel2020significance by replacing metric entropy calculations with scale-insensitive measures, enabling the analysis of networks without strict weight constraints.
When $l = L_d + 1$, the output function takes the form
Our framework introduces key innovations in the treatment of neural networks, advancing the theoretical analysis of their behavior and performance. First, we replace the traditional bounded weight assumption used in prior work, such as fallahgoul2024asset, with a weaker moment condition. Specifically, we require only that $\mathbb{E}[f(X)^2] \leq M$ for $f \in \mathcal{F}_{DNN,H_n}$. This condition aligns more closely with modern training practices, which often involve unconstrained optimization, while still providing sufficient control for rigorous theoretical analysis.
Second, we relax the activation function requirements by imposing only Lipschitz continuity on $\psi$, in contrast to the smooth, non-polynomial conditions of earlier frameworks. This generalization accommodates widely used activation functions like ReLU and its variants, which are central to modern deep learning. Importantly, this relaxation ensures three critical properties: (i) well-defined gradients for significance testing, (ii) stability of network outputs, and (iii) preservation of universal approximation capabilities. Together, these innovations provide a more flexible and practical foundation for analyzing neural networks.
The network class $\mathcal{F}_{DNN,H_n}$ can be viewed as a type of sieve estimation where basis functions are learned from data, see farrell2021deep. Following fallahgoul2024asset, we set the width $H_n$ to increase with sample size $n$, allowing the bias of least squares estimation to decrease asymptotically. This aligns with the classical theory of series estimation while accommodating the specific characteristics of neural networks.
A central innovation in our framework is replacing traditional metric entropy calculations with Rademacher complexity as a measure of function class complexity farrell2021deep,Bartlett2005,KoltchinskiiPancheko2000. This shift is motivated by both theoretical advantages and practical considerations in analyzing modern neural networks.
The key insight of our approach is using localization theory to analyze neural network complexity. For a neural network class with moment bound $\mathbb{E}[f(X)^2] \leq M$, we consider the localized class:
This framework yields three crucial theoretical results. First, for the localized class, we establish (Lemma (ref)):
This bound is derived through careful analysis of the empirical process without requiring bounded weights. Second, for the full neural network class with Lipschitz activations, we prove (Lemma (ref)):
The proof strategy for bounding the Rademacher complexity of deep neural networks (ref) combines localization techniques with sharp covering number bounds from farrell2021deep. Rather than trying to bound the global complexity directly, we first decompose the function class into localized subsets based on their $L_2$ norms. For these localized classes, we employ farrell2021deep's Lemma 4, which provides covering number bounds based on pseudo-dimension without requiring explicit bounds on network weights. This is crucial because traditional approaches often rely on weight-based bounds, which can be too restrictive for modern deep learning. The localization approach allows us to use Dudley's entropy integral more effectively, as we can obtain tighter bounds for each localized class. We then use a careful peeling argument to extend these local bounds to the full function class, handling the trade-off between the size of each localized class and its complexity. The crucial insight from farrell2021deep's approach is that we can control the complexity through function-space properties (pseudo-dimension and moment bounds) rather than parameter-space constraints, leading to bounds that better reflect the practical behavior of deep neural networks.
Third, we obtain sharp concentration inequalities (Lemma (ref)):
for $t > C\sqrt{\frac{r}{n}}$ and $Mt < r$, providing exponential tail bounds for the empirical process.
These results demonstrate several key advantages over traditional metric entropy approaches. Our framework exhibits scale insensitivity, as our bounds hold without requiring bounded weights, thus aligning with modern training practices. Furthermore, it offers architectural flexibility by naturally accommodating deep architectures through the Lipschitz constant $L$ and depth $L_d$. Perhaps most importantly, the localization technique enables sharp concentration results, providing tight control of the empirical process through exponential tail bounds.
Combined with our approximation rate result (Lemma (ref)) of $O(H_n^{-s/d})$ for target functions in $H^s([-1,1]^d)$, this framework provides a complete theoretical foundation for analyzing neural networks under minimal assumptions. The shift from metric entropy to Rademacher complexity thus represents a fundamental realignment of theory with modern deep learning practice.
Before presenting our main theoretical results on consistency and asymptotic distribution of neural network test statistics, we establish several foundational properties. These properties ensure that our weaker assumptions still provide sufficient structure for valid statistical inference.
These conditions play crucial roles in our subsequent analysis. The existence of weak derivatives (Condition 1) is essential for our gradient-based test statistic. While we have relaxed classical differentiability requirements, weak derivatives in $L^2$ provide sufficient regularity for analyzing partial effects. This condition will be particularly important in Section (ref) when we establish the asymptotic distribution of our test statistic.
The well-definedness of $\lambda_j^n$ (Condition 2) ensures our test statistic is mathematically meaningful. This is not immediate under our relaxed assumptions, as we no longer require bounded weights or classical derivatives. This property is fundamental to all our asymptotic results and will be repeatedly used in Section (ref).
The preservation of approximation rates (Condition 3) is critical for establishing consistency of our neural network estimator. In Section (ref), we will show that our estimator achieves the optimal convergence rate of $O(H_n^{-s/d})$, matching the best possible rate for $s$-smooth functions. This is remarkable as it demonstrates that our weaker conditions do not sacrifice statistical efficiency.
The Rademacher complexity bounds (Condition 4) serve as the technical cornerstone of our analysis. These bounds replace metric entropy calculations throughout our proofs and enable us to establish sharp concentration inequalities, control the asymptotic behavior of empirical processes, and obtain finite-sample guarantees. Specifically, in Section (ref), we will show how these bounds allow for the derivation of tight concentration inequalities. In Section (ref), they will help us control the asymptotic behavior of empirical processes, while in Theorem (ref), they will provide finite-sample guarantees.
Together, these conditions form a bridge between our relaxed assumptions and the strong theoretical guarantees we establish in Section (ref). They demonstrate that our framework maintains the necessary technical machinery for rigorous statistical inference while accommodating the practical realities of modern neural networks.
The proof of this proposition, given in Appendix (ref), introduces several technical tools that will be reused throughout our main theoretical developments. In particular, the localization techniques and concentration inequalities developed in the proof will be fundamental to our analysis of the test statistic's asymptotic behavior.
For estimation, we employ the method of least squares in conjunction with neural networks. This combination provides both theoretical tractability and practical implementability. The least squares estimator $\hat{f}_n$ is obtained by minimizing:
where the loss function takes the form:
Several aspects of this estimation framework merit discussion. First, the optimization structure (ref) plays a crucial role. The quadratic nature of the loss function, combined with our Lipschitz activation functions, ensures that the optimization problem is well-behaved. While the problem may not be convex due to the neural network architecture, it maintains sufficient structure for theoretical analysis. This structure is key to ensuring the feasibility of the optimization process despite the non-convexity of the underlying model.
Second, the network architecture plays a crucial role in balancing approximation error and complexity. From Lemma (ref), we know the approximation error scales as $O(H_n^{-s/d})$, while Lemma (ref) shows the complexity grows as $O(H_nL^{L_d}/\sqrt{n})$. To balance these rates, the width $H_n$ must increase with sample size $n$, while the depth $L_d$ remains fixed. Specifically, we require:
which ensures that the network's capacity grows at an appropriate rate relative to the sample size while maintaining the optimal approximation properties for functions in $H^s([-1,1]^d)$.
The subsequent analysis will show that $\hat{f}_n$ achieves optimal convergence rates and that our test statistics, based on its partial derivatives, have well-behaved asymptotic distributions. These theoretical properties, coupled with the practical implementability of our framework, make it a valuable tool for understanding variable importance in modern machine learning applications.
Several aspects of these nonasymptotic bounds warrant discussion. First, our consistency result and that of farrell2021deep achieve optimal rates through complementary technical approaches, each with distinct advantages. Our approach emphasizes general functional analytic methods, while farrell2021deep develops precise architecture-specific analysis.
A key methodological distinction lies in handling network architecture. farrell2021deep's approach, building on yarotsky2017error,yarotsky2018optimal, provides sharp bounds by carefully analyzing specific ReLU architectures. Their precise control of network parameters yields tight constants and explicit optimization guidance. Our approach trades this architectural specificity for broader applicability: by using scale-insensitive measures through Rademacher complexity, we accommodate general neural networks under moment conditions. While potentially yielding looser constants, this generality allows our results to extend naturally to various network architectures and activation functions.
The foundations in approximation theory also reflect this trade-off. farrell2021deep's use of yarotsky2017error,yarotsky2018optimal's explicit ReLU constructions provides detailed insights into network expressivity and parameter efficiency. Our reliance on general Sobolev space approximation properties, while less specific, establishes optimal rates for a broader function class. This trade-off between specificity and generality mirrors a fundamental question in deep learning theory: whether to pursue architecture-specific or architecture-agnostic analysis.
Both approaches employ localization, but to different effect. farrell2021deep's careful shell-by-shell analysis of unbounded weights provides sharp control of the approximation-estimation trade-off. Our more streamlined use of localization, focused on concentration inequalities, yields simpler proofs while maintaining optimal rates. Each approach offers valuable insights: farrell2021deep's analysis reveals intricate connections between network architecture and statistical complexity, while our method highlights fundamental relationships between functional complexity measures and estimation error.
The central innovation of our framework is replacing traditional metric entropy calculations with Rademacher complexity measures in the analysis of neural network sieve spaces. This shift enables us to handle unbounded weights, general activation functions, and weaker function space assumptions while maintaining optimal rates.
For a neural network with depth $L_d$ and width $H_n$, we define our sieve space:
where the localization radius $r_n$ is chosen to balance Rademacher complexity bounds with approximation errors.
Traditional metric entropy approaches require explicit bounds on network weights and specific architectural constraints to control complexity. These bounds often take the form $\log N_{[]}(\epsilon, \mathcal{F}, L^2(P)) \leq C(\frac{1}{\epsilon})^V$, where $V$ is the VC dimension or pseudo-dimension of the network class. Such bounds, while powerful for classical analysis, become restrictive for modern deep learning where weight constraints may be impractical or undesirable.
Our Rademacher complexity approach provides several fundamental advantages. First, it allows direct control of the empirical process through localization:
without requiring explicit weight bounds. This scale-insensitive measure adapts naturally to the function class complexity, as shown in Lemma (ref). Second, it enables sharper concentration inequalities - see Lemma (ref) - by exploiting the geometry of the function space rather than just its covering numbers.
The implications of this shift extend beyond technical convenience. As shown in Proposition (ref), where (S2) replaces traditional entropy conditions with Rademacher bounds while (S3) maintains minimal metric entropy control needed for technical lemmas.
This construction enables new theoretical developments in three key ways. First, in Theorem (ref), the asymptotic distribution result uses Rademacher complexity to handle unbounded weights through moment conditions rather than explicit bounds. Second, for the test statistic in Theorem (ref), the scale-insensitive approach allows general Lipschitz activation functions beyond the smooth, non-polynomial requirements of previous work. Third, as shown in Lemma (ref), optimal approximation rates are maintained under weaker Sobolev space assumptions on the target function.
The power of Rademacher complexity in our framework becomes particularly apparent in the localization theory developed through Lemmas (ref)-(ref). These results show how scale-insensitive measures can provide sharp concentration while accommodating modern neural network practices.
This reformulation through Rademacher complexity represents a fundamental realignment of sieve space theory with contemporary deep learning practice in three key aspects. First, while classical sieve theory typically requires explicit bounds on parameter norms, our Rademacher complexity approach accommodates the unbounded weights commonly seen in modern neural networks training. Specifically, by working with moment bounds $\mathbb{E}[f(X)^2] \leq M$ rather than parameter constraints $\|\theta\| \leq B$, we better reflect the implicit regularization observed in practice where networks achieve good generalization despite large weights. Second, our scale-insensitive bounds through Rademacher complexity naturally handle the homogeneity property of ReLU networks, where the same function can be represented by different scalings of weights across layers. This aligns with empirical observations where networks with varying weight magnitudes exhibit similar generalization behavior. Third, by analyzing complexity through function space properties rather than architecture-specific parameters, our framework better captures the role of overparameterization in modern deep learning, where networks may have far more parameters than classical theory would suggest is optimal, yet still generalize well.
Theorem (ref) provides a more general asymptotic distribution than previous work by establishing convergence under weaker conditions, i.e., Assumption (ref) and Definition (ref).
However, while $U(C',\epsilon_n)^{-1}(\hat{f}_n - f_*)$ converges to $h^{(max)}$, this result does not directly enable variable significance testing since it does not involve relationships to specific covariates $x_j$. We need a statistic that:
Following fallahgoul2024asset and horel2020significance, and equation (ref), but leveraging our scale-insensitive framework, we consider the functional:
The statistic $\mathcal{T}_j[\hat{f}_n]$ quantifies the significance of $x_j$ by measuring its squared partial derivative effect through $\hat{f}_n$.
Since $\hat{f}_n$ consistently estimates $f_*$ by Theorem (ref), $\mathcal{T}_j[\hat{f}_n]$ serves as a scale-insensitive "representative" of $\mathcal{T}_j[f_*]$. Note that direct calculation of $\mathcal{T}_j[f_*]$ is impossible as $f_*$ is unknown. In the next theorem, we derive the asymptotic distribution of $\mathcal{T}_j[\hat{f}_n]$ under our weaker conditions.
The asymptotic distribution we derived has several notable features. Under $H_0$, we showed that $U(C',\epsilon_n)^{-2}\mathcal{T}_j[\hat{f}_n] \rightsquigarrow \mathcal{T}_j[h^{(max)}]$, where $h^{(max)}$ is the unique maximizer of the limiting Gaussian process. Note that the randomness in our test statistic stems from two sources: the estimation of $\hat{f}_n$ and the empirical functional $\mathcal{T}_j$. Since we work with samples, and $\mathcal{T}_j[\hat{f}_n] = \frac{1}{n}\sum_{i=1}^n \left(\frac{\partial \hat{f}_n}{\partial x_j}(X_i)\right)^2$ can be seen as an empirical functional, it is natural to analyze its asymptotic behavior through the lens of empirical process theory.
The scale-insensitive normalization $U(C',\epsilon_n)^{-2}$ plays a crucial role in obtaining the proper limiting distribution. This normalization accounts for both the complexity of the neural network class and the sample size, ensuring that our test statistic converges to a well-defined limit regardless of network architecture. The limiting distribution $\mathcal{T}_j[h^{(max)}]$ inherits its properties from the unique maximizer $h^{(max)}$ of the Gaussian process, making our test both theoretically tractable and practically implementable.
We close this section by providing the asymptotic distribution of $\mathcal{T}_j[\hat{f}_n]$.
The computation of asymptotic distributions for test statistics can be challenging in practice. We extend and improve upon the three-step discretization approach from fallahgoul2024asset and horel2020significance to provide a more comprehensive framework for computing the asymptotic distribution of the test statistic $U(C', \epsilon_n)^{-2}\rho_n^2(\partial\hat{f}_n/\partial x_j(X))$. Our improvements include a more efficient discretization scheme that reduces computational complexity while maintaining accuracy, and a robust method for handling the scale-insensitive normalization factor.
The discretization process consists of three primary steps:
The basic framework of fallahgoul2024asset,horel2020significance can be enhanced through several computational and statistical improvements. First, we introduce an adaptive sampling framework that dynamically adjusts the number of neural networks $m$ based on convergence metrics:
where $\alpha$ is a learning rate and $\|\cdot\|_F$ is the Frobenius norm. This adaptive approach ensures efficient exploration of the function space while maintaining computational feasibility.
For robust covariance estimation, particularly in high-dimensional settings, we employ a structured approach with shrinkage estimation:
where $\lambda$ is a shrinkage parameter.
The practical implementation requires careful consideration of computational resources. The choice of parameters $(m, n_p)$ should balance accuracy with computational cost. We recommend leveraging GPU acceleration for large-scale computations and implementing parallel processing strategies for multiple test statistics.
The combination of these enhancements provides a robust and computationally feasible approach to computing asymptotic distributions while maintaining the theoretical guarantees established in fallahgoul2024asset,horel2020significance. The framework is particularly well-suited for analyzing complex neural network architectures in financial applications, as demonstrated in the empirical analysis of equity premium prediction.
This paper has developed a scale-insensitive framework for neural network significance testing that substantially generalizes existing approaches while maintaining mathematical rigor. Our framework makes three primary contributions to the literature. First, by replacing metric entropy calculations with Rademacher complexity bounds, we enable the analysis of neural networks without requiring bounded weights or specific architectural constraints. This better aligns theoretical guarantees with contemporary deep learning practices where optimization is typically unconstrained. Second, we significantly weaken the regularity conditions on target functions, requiring only Sobolev space membership $H^s([-1,1]^d)$ with $s > d/2$ rather than classical differentiability assumptions. This relaxation encompasses a broader class of functions while preserving optimal approximation rates. Third, our modified sieve space construction based on moment bounds rather than weight constraints provides a more natural theoretical framework for analyzing modern neural networks.
The theoretical implications of our results extend beyond significance testing. Our approach demonstrates that scale-insensitive measures can effectively control neural network complexity without imposing artificial constraints on architecture or training. The framework's ability to handle general Lipschitz activation functions and unbounded weights while maintaining optimal convergence rates suggests that Rademacher complexity may provide a more suitable foundation for analyzing deep learning methods than traditional metric entropy approaches.
From a practical perspective, our framework provides several advantages. The relaxed assumptions better match how neural networks are actually implemented and trained, making the theoretical guarantees more relevant to practitioners. The framework accommodates popular activation functions like ReLU and its variants, and supports standard training practices without requiring artificial constraints. Additionally, our test statistics remain computationally tractable while providing valid inference under weaker conditions.
Future research could extend this framework in several directions. The scale-insensitive approach could be adapted to analyze other deep learning architectures beyond feedforward networks, such as convolutional or recurrent networks. The framework might also be extended to handle dependent data structures or to develop tests for more complex hypotheses about feature interactions. Additionally, the connection between Rademacher complexity and generalization in deep learning merits further theoretical investigation.
Our results suggest that replacing rigid structural assumptions with more flexible functional analytic conditions may be a promising direction for developing statistical theory that better reflects the realities of modern machine learning practice. This alignment between theory and practice is crucial for advancing our understanding of deep learning methods while maintaining mathematical rigor.