Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
67,310 characters · 16 sections · 20 citation commands
0.5mm A Powerful Chi-Square Specification Test with Support Vectors 0.25mm
Keywords: Classification; Conditional Moment Restrictions; Specification Test; Support Vector Machine.
\pagenumbering{arabic} \setcounter{page}{1} \setcounter{footnote}{0}
Consider the following parametric regression model:
where $\mathcal{M}_{\theta_0}(X)$ is a parametric specification indexed by an unknown parameter vector $\theta_0 \in \Theta$, with $\Theta \subset \mathbb{R}^q$. Here, $X \in \mathcal{X} \subset \mathbb{R}^q$, $Y \in \mathcal{Y} \subset \mathbb{R}$, and $\varepsilon_{\theta_0}$ represents the parametric error. This framework encompasses many important models, including linear and nonlinear conditional mean regression, quantile regression, treatment effect models, and instrumental variables regressions, among others. Accurate specification of these parametric models is crucial for subsequent statistical inferences.
We aim to test the null hypothesis: \[ H_0: \mathbb{P}\left(\mathbb{E}[\varepsilon_{\theta_0} \mid X] = 0\right)=1 \text{ for some } \theta_0\in \Theta \] against the alternative hypothesis: \[ H_1: \mathbb{P}\left(\mathbb{E}[\varepsilon_{\theta} \mid X] \neq 0\right) >0 \text{ for all } \theta\in \Theta. \] The Integrated Conditional Moment (ICM) framework, introduced by bierens1982consistent, provides a classical approach to test the correct specification of the parametric model.
The corresponding test statistic is defined as: \[ n\hat{T}_{ICM} = \frac{1}{n}\sum_{i,j = 1}^{n} \varepsilon_{\hat\theta,i} \exp\left(-\frac{\lVert x_i - x_j \rVert_2^2}{2}\right)\varepsilon_{\hat\theta,j}, \] where $\varepsilon_{\hat\theta,i} = y_i - \mathcal{M}_{\hat{\theta}}(x_i)$, and $\hat{\theta}$ is a consistent estimator for $\theta_0$. Here, $\lVert \cdot \rVert_2$ denotes the Euclidean norm. muandet2020kernel extended the ICM test to the Kernel Conditional Moment (KCM) test. By replacing the kernel $\exp(-\lVert x_i - x_j \rVert_2^2 /2)$ in $n\hat{T}_{ICM}$ with any integrally strictly positive definite (ISPD) reproducing kernel $k(x_i,x_j)$, the KCM test statistic is given by: \[ n\hat{T}_{KCM} = \frac{1}{n}\sum_{i,j = 1}^{n} \varepsilon_{\hat\theta,i} k(x_i,x_j)\varepsilon_{\hat\theta,j}. \] escanciano2024gaussian proposed similar test statistics using a Gaussian Process approach, with a focus on addressing the estimation effects arising from $\hat{\theta}$.
Despite the different frameworks used to derive the test statistics mentioned earlier, their shared V-statistic structure suggests a close relationship among them. To unify these statistics under a single framework, we adopt the language of Reproducing Kernel Hilbert Spaces (RKHS). This unified perspective lays the foundation for developing our novel, powerful test statistic.
Given a dataset $\{\varepsilon_{\theta_0,i}, x_i\}_{i=1}^n$, we map the data into an RKHS $\mathcal{H}_k$ with reproducing kernel $k(x, x')$, resulting in points $\varepsilon_{\theta_0,i}k(x_i, \cdot) \in \mathcal{H}_k$. Under mild assumptions on the kernel, the null hypothesis holds if and only if the mean element \[ \mu_{\theta_0,k} = \mathbb{E}\left[\varepsilon_{\theta_0,1}k(X_1, \cdot)\right] = \boldsymbol{0} \in \mathcal{H}_k \quad \text{(see \citealp{muandet2020kernel})}. \] However, directly testing $\mu_{\theta_0,k} = \boldsymbol{0}$ is impractical due to its infinite-dimensional nature. Instead, we project $\mu_{\theta_0,k}$ onto a direction $w \in \mathcal{H}_k$ and analyze the normalized projection: \[ S_{0,k} = \frac{\left\langle \mu_{\theta_0,k}, w \right\rangle_{\mathcal{H}_k}}{\|w\|_{\mathcal{H}_k}}, \] where normalization by $\|w\|_{\mathcal{H}_k}$ ensures that the projection values are comparable across different directions. The hypotheses can then be expressed as: \[ H_0: S_{0,k} = 0 \quad \text{versus} \quad H_1: S_{0,k} \neq 0. \]
Under the alternative hypothesis $H_1$, the optimal direction $w$ that maximizes $S_{0,k}$ aligns with $\mu_{\theta_0,k}$, i.e., $w^* = \mu_{\theta_0,k}$, leading to $S_{0,k} = \|\mu_{\theta_0,k}\|_{\mathcal{H}_k}$. Consequently, all KCM-type test statistics estimate the squared norm: \[ S_{0,k}^2 = \left\langle \mu_{\theta_0,k}, \mu_{\theta_0,k} \right\rangle_{\mathcal{H}_k} = \mathbb{E}\left[\varepsilon_{\theta_0} k(X, X') \varepsilon_{\theta_0}'\right], \] where random variable $Z'$ is an independent copy of $Z$. Note that the above equality is the direct consequence of the reproducing property: \[ \left \langle k(x,\cdot), k(x',\cdot) \right \rangle_{\mathcal{H}_k} = k(x,x'). \]
However, maximizing \(S_{0,k}\) does not necessarily maximize the power of the test, as the power also depends on the variance of the asymptotic null distribution. In the case of KCM-type test statistics, it is \[ \frac{1}{n}\sum_{i,j = 1}^{n} \varepsilon_{\theta_0,i} k(x_i,x_j)\varepsilon_{\theta_0,j} \overset{d}{\to} \sum_{j=1}^{\infty} \tau_j W_j^2, \] where \(W_j \sim \mathcal{N}(0,1)\) and \(\{\tau_j\}\) are the eigenvalues of \(\varepsilon_{\theta_0} k(x,x')\varepsilon_{\theta_0}'\), i.e., they are the solutions of \[ \tau_j f_j(\varepsilon_{\theta_0},x) = \int \varepsilon_{\theta_0} k(x,x')\varepsilon_{\theta_0}' f_j(\varepsilon_{\theta_0}',x') \, d P(\varepsilon_{\theta_0}',x'). \] See muandet2020kernel for the details. The variance of the asymptotic null distribution is \(V^2 = 2 \sum_{j=1}^{\infty}\tau_j\), and a KCM-type test statistic is powerful only if the signal-to-noise ratio (SNR) \(S^2_{0,k} / V\) is large.
For a given RKHS (i.e., the kernel \(k(\cdot,\cdot)\) is fixed), selecting the optimal direction \(w\) through SNR can be formidable due to the infinite-dimensional nature of the problem. In this paper, we address this power-boosting issue from a novel perspective: we frame the testing problem as a classification problem, and our goal is to learn a direction \(w\) that exhibits desirable separation properties.
Specifically, we interpret the testing problem as two types of classification problems. In the first interpretation, we treat the nonparametric class \(\{\varepsilon_{\theta_0,i} k(x_i,\cdot)\}_{i=1}^n\) as one class and the parametric class \(\{\mathcal{M}_{\theta_0}(x_i) k(x_i,\cdot)\}_{i=1}^n\) as another class. The objective is to find a projection direction \(w\) that maximizes the discrepancy between these two classes. Geometrically, this is equivalent to achieving a large mean difference between the projected two classes while maintaining a relatively small overlap between them. Situations corresponding to a large projected mean difference with significant overlap (resulting in poor power performance) and a moderate projected mean difference with minimal overlap (resulting in good power performance) are illustrated in Figure (ref).
In the second interpretation, we may consider the residuals \(\{\varepsilon_{\theta_0,i} k(x_i,\cdot)\}_{i=1}^n\) as a single class rather than analyzing the separation between two classes. For simplicity, assume the residuals have a positive mean element: \(\mathbb{E}\left(\langle \varepsilon_{\theta_0,1} k(X_1,\cdot), w \rangle_{\mathcal{H}_k}\right) > 0.\) An effective projection direction \(w\) should ensure that most projected residuals are greater than zero. That is, even when the mean projection is near zero, only a few residuals should project to negative values. In contrast, a poor projection direction \(w\) (resulting in lower test power) may show many residuals that project below zero, even if their mean projection is far from zero. This reasoning is visualized in Figure (ref).
The main contribution of this paper is the application of two Support Vector Machine (SVM) algorithms to a training dataset (that is independent from the dataset used for testing) to learn a good direction: \( w = \sum_{j \in \mathbb{S}} \eta_j k(x_j, \cdot), \) which achieves effective “separation.” Here, $\mathbb{S}$ is an index set identifying the support vectors selected by the SVM algorithm, and $\{\eta_j\}_{j \in \mathbb{S}}$ are the corresponding weights.
The first algorithm aligns with the first perspective, aiming to separate the discrepancy between the nonparametric and parametric classes. The second algorithm incorporates the second perspective, aiming to separate the residuals from the origin point.
In practice, the parameter $\theta_0$ is unknown and must be estimated as $\hat{\theta}$. To mitigate estimation effects, we propose using a projection kernel $k_p(\cdot, \cdot)$, following the approach in escanciano2024gaussian.
To operationalize our approach, we introduce a $t$-type test statistic based on: \[ \mu_{\theta_0,\mathbb{S},k} = \mathbb{E}\left(\left \langle \varepsilon^\dagger_{\theta_0}k_p(X^\dagger,\cdot),\sum_{j \in \mathbb{S}} \eta_j k_p(x_j,\cdot)\right \rangle_{\mathcal{H}_k} \right) = \sum_{j \in \mathbb{S}}\eta_j \mathbb{E}\left[\varepsilon_{\theta_0}^\dagger k(X^\dagger, x_j)\right]. \] Its empirical counterpart is given by \[ \hat \mu^\dagger_{\hat{\theta}, \mathbb{S},k_p} = \frac{1}{n} \sum_{i=1}^{n} \left \langle \varepsilon^\dagger_{\hat{\theta},i} k_p(x_i^\dagger,\cdot), \sum_{j \in \mathbb{S}} \eta_j k_p(x_j,\cdot) \right \rangle_{\mathcal{H}_k} = \frac{1}{n} \sum_{i=1}^{n} \sum_{j \in \mathbb{S}} \eta_j \varepsilon^\dagger_{\hat{\theta},i} k_p(x_i^\dagger, x_j), \] where variables with the dagger superscript ($\dagger$) denote test data, and those without denote training data. The sample size $n$ refers to the number of test observations. The expectation is taken with respect to the test data distribution.
This test statistic offers several advantages. First, it is computationally efficient, with linear time complexity, making it suitable for large datasets. Second, it is omnibus, meaning it is capable of detecting any arbitrary alternative hypothesis. Third, it admits a pivotal asymptotic distribution under the null hypothesis, which simplifies inference. To the best of our knowledge, no existing test statistic in the literature simultaneously satisfies all these criteria.
The remainder of the paper is organized as follows. Section (ref) provides an equivalent characterization of the null hypothesis using $\mu_{\theta_0,\mathbb{S},k}$ and demonstrates its omnibus property, ensuring that the test can detect any deviation from the null. Section (ref) introduces the proposed test statistic and derives its asymptotic properties under the assumption that $\theta_0$ is known. Section (ref) addresses the challenges that arise when $\theta_0$ is unknown and must be estimated, proposing a correction through the use of a projection kernel $k_p(\cdot, \cdot)$. Section (ref) formalizes the application of the Support Vector Machine (SVM) algorithm to learn the direction: \( w = \sum_{j \in \mathbb{S}} \eta_j k(x_j, \cdot), \). Section (ref) presents simulation studies that validate the finite-sample performance of our method, along with real-data applications of the proposed approach. Finally, Section (ref) concludes the paper.
As introduced before, throughout the paper, we distinguish between training data and test data through the dagger superscript ($\dagger$): variables with the dagger superscript are testing points, while training points do not have any superscript. Both the training and testing points are sampled independently from the same distribution. Throughout the paper, we assume the following standard conditions hold: (i) the random variables $S=(Y,X)$ forms a strictly stationary process with probability measure $\mathbb{P}_{S}$; (ii) Reularity Conditions. (1) the residual function $\varepsilon: \mathcal{S} \times \Theta \longrightarrow \mathbb{R}$ is continuous on $\Theta$ for each $s \in \mathcal{S}$; (2) $\mathbb{E}(\varepsilon(S;\theta)|X=x)$ exists and is finite for every $\theta \in \Theta$ and $x \in \mathcal{X}$ for which $\mathbb{P}_X(x) >0$; (3) $\mathbb{E}(\varepsilon(S;\theta)|X=x)$ is continuous on $\Theta$ for all $x \in \mathcal{X}$ for which $\mathbb{P}_X(x) >0$. We will write $\varepsilon(s_i;\theta)$ as $\varepsilon_{\theta,i}$ if there is no confusion.
In this section, we demonstrate that $\mu_{\theta_0,\mathbb{S},k}^\dagger$, as defined in the Introduction, can be interpreted as a distance metric that quantifies the degree to which a parametric model fits the data. Throughout this section, we assume the availability of a training dataset $\{\varepsilon_{\theta_0,i} k(x_i,\cdot)\}_{i=1}^n$, and that the reproducing kernel $k(\cdot,\cdot)$ is integrally strictly positive definite: \[ \int_{\mathcal{X}} \int_{\mathcal{X}} f(x) k(x,x^\prime) f(x^\prime) dP(x) dP(x^\prime) > 0, \quad \text{for any non-zero } f \in L^2(P). \] where \(L^2(P)\) denotes the Hilbert space of square-integrable functions with respect to the measure \(P\). Additionally, we assume that $\theta_0$ is known.
As a running example throughout this paper, we adopt the widely used Gaussian kernel $k(x, x') = \exp\left(-\|x - x'\|_2^2 / \sigma\right)$, where $\sigma > 0$. For other commonly used reproducing kernels and their properties, see muandet2017kernel.
We now argue that the finite location mean difference metric $\mu_{\theta_0,\mathbb{S},k}^\dagger$ is omnibus, meaning it can detect any arbitrary alternative.
By Mercer's theorem, the reproducing kernel $k(x_j, x^\dagger)$ admits the decomposition: \[ k(x_j, x^\dagger) = \sum_{i=1}^\infty \lambda_i \phi_i(x_j) \phi_i(x^\dagger), \] where $\lambda_i$ and $\phi_i$ are the eigenvalues and eigenfunctions of the integral operator $T$, defined as: \[ Tf(x) = \int_{\Omega} k(x, x') f(x') dP(x'), \] with $P$ being a measure on the domain of $x'$, and $\Omega$ denoting the support. Mercer's theorem also ensures that the eigenfunctions $\phi_i(\cdot)$ form an orthonormal basis of $L^2(P)$, and the eigenvalues $\lambda_i$ are non-negative and decreasing. Consequently, we can express $\mu_{\theta_0,\mathbb{S},k}^\dagger$ as: \[ \mu_{\theta_0,\mathbb{S},k}^\dagger = \sum_{i=1}^\infty \left( \lambda_i \sum_{j \in \mathbb{S}} \eta_j \phi_i(x_j) \right) \mathbb{E}[\varepsilon_{\theta_0}^\dagger \phi_i(X^\dagger)]. \] Equivalently, this can be written as: \[ \mu_{\theta_0,\mathbb{S},k}^\dagger = \sum_{i=1}^\infty \gamma_i \mathbb{E}[\varepsilon_{\theta_0}^\dagger \phi_i(X^\dagger)], \] where $\gamma_i = \lambda_i \sum_{j \in \mathbb{S}} \eta_j \phi_i(x_j)$.
The omnibus property of $\mu_{\theta_0,\mathbb{S},k}^\dagger$ arises from the fact that it is a linear combination of infinitely many orthonormal basis functions. This structure ensures that the metric captures all possible deviations from the null hypothesis, enabling it to detect any arbitrary alternative.
In this section, we study the asymptotic properties of the test statistic under the null hypothesis, fixed alternatives, and local alternatives, assuming $\theta_0$ is known and $\{\eta_j\}_{j \in \mathbb{S}}$ are given (have been learned from data points independent from the ones used in testing). In Section 4, we address the estimation effect when $\theta_0$ is replaced by a consistent estimator $\hat{\theta}$. In Section 5, we discuss the selection of $\{\eta_j\}_{j \in \mathbb{S}}$ via the SVM algorithms.
Our test statistic is based on the metric $\mu_{\theta_0,\mathbb{S},k}^\dagger$ defined in the previous section. Given $\{\eta_j\}_{j \in \mathbb{S}}$, we propose the following test statistic: \[ \hat{T}_{\theta_0,\mathbb{S},k} = \frac{\hat{\mu}^\dagger_{\theta_0,\mathbb{S},k}}{\hat{\sigma}^\dagger_{\theta_0,\mathbb{S},k}}, \] where \[ \hat{\mu}^\dagger_{\theta,\mathbb{S},k} =\frac{1}{n} \sum_{j\in \mathbb{S}} \eta_j \sum_{i=1}^n \varepsilon_{\theta,i}^\dagger k(x_i^\dagger,x_j) = \frac{1}{n} \sum_{j\in \mathbb{S}} \eta_j (\boldsymbol{\varepsilon}_{\theta}^\dagger)^\top \mathbf{K}(\mathbf{X}^\dagger,x_j), \] where both $\boldsymbol{\varepsilon}_{\theta}^\dagger = (\varepsilon_{\theta,1}^\dagger,\ldots,\varepsilon_{\theta,n}^\dagger)$, and $\mathbf{K}(\mathbf{X}^\dagger,x_j) = (k(x_1^\dagger,x_j),\ldots,k(x_n^\dagger,x_j))^\top$ are $n \times 1$ vectors.
The empirical variance is \[ \left(\hat{\sigma}^\dagger_{\theta,\mathbb{S},k}\right)^2 = \frac{1}{n-1} \sum_{i=1}^{n}\left(\sum_{j\in \mathbb{S}} \eta_j \varepsilon_{\theta,i} k(x_j,x_i^\dagger) - \hat{\mu}^\dagger_{\theta,\mathbb{S},k}\right)^2. \]
The proofs for the above theorems can be obtained trivially by the Central Limit Theorem, Law of Large Numbers, and Slutsky's Theorem, hence omitted.
So far, we have assumed that the value of $\theta_0$ is known. In practice, $\theta_0$ is estimated by a consistent estimator $\hat{\theta}$. In this section, we discuss how to deal with the estimation effect when $\theta_0$ is estimated.
The following assumptions are required for this purpose.
Assumption 1 is weaker than the related conditions in the literature. We only impose that $\hat{\theta}$ converges in probability at a slower rate than usual. Additionally, we do not require it to admit an asymptotically linear representation. This could be useful in the context of non-standard estimation procedures, such as the LASSO. Assumption 2 is standard in the literature and imposes regularity conditions on the smoothness of the residual function.
We now introduce a projection operator $\boldsymbol{\Pi}: \mathcal{H}_k \longrightarrow \mathcal{H}_k$ defined as: \[ (\boldsymbol{\Pi}\omega)(x) = \omega(x) - \mathbb{E}\left[\omega(X) (g_{\theta_0}(X))^\top \right] \Gamma_{\theta_0}^{-1} g_{\theta_0}(x), \forall \omega \in \mathcal{H}_k, x \in \mathcal{X}, \] Applying this projection operator to a kernel function $k(x,x^\dagger)$ yields the projection kernel: \[ k_p(x,x^\dagger)=\boldsymbol{\Pi}k(x,x^\dagger) = k(x,x^\dagger) - \mathbb{E}\left[k(x,X^\dagger) (g_{\theta_0}(X^\dagger))^\top\right] \Gamma_{\theta_0}^{-1} g_{\theta_0}(x^\dagger). \]
To analyze the local behavior of the projected mean embedding $\mathbb{E}(\varepsilon_{\theta_0}k_p(x,X^\dagger))$ in a neighborhood of $\theta_0$, consider their derivatives with respect to $\theta$ evaluated at $\theta_0$:
This establishes that the projected mean embedding are locally robust to small perturbations in $\theta$ around $\theta_0$. The vanishing derivatives follow from the dominated convergence theorem.
The matrix estimator (using the testing data) to this projection operator is given by: \[ \hat{\boldsymbol{\Pi}}^\dagger = \boldsymbol{I}_n - \hat{\mathbf{g}}\left(\hat{\mathbf{g}}^\top \hat{\mathbf{g}}\right)^{-1} \hat{\mathbf{g}}^\top \] where $\hat{\mathbf{g}}$ is a $n \times d$ matrix of scores whose $i$th row is given by $(\hat g_{i}^\dagger)^\top = (\nabla_{\theta} \varepsilon_{\theta}^\dagger|_{\theta = \hat \theta})^\top$, and $\boldsymbol{I}_n$ is the $n \times n$ identity matrix. The projected version of the kernel vector ($\boldsymbol{K}(\boldsymbol{X}^\dagger,x_j)$) is given by: \[ \boldsymbol{K}_p(\boldsymbol{X}^\dagger,x_j) = (\hat{\boldsymbol{\Pi}}^\dagger)^\top \boldsymbol{K}(\boldsymbol{X}^\dagger,x_j) \]
The following theorem states how this projection kernel eliminates the estimation effect when vector multiplication is performed.
As long as the convergence speed satisfies $\alpha>1/4$, we have for any weight $\eta_j$, \[
\] The corresponding empirical variance and test statistic are \[ \left(\hat{\sigma}^\dagger_{\hat \theta,\mathbb{S},k_p} \right)^2= \frac{1}{n-1} \sum_{i=1}^{n}\left(\sum_{j=1}^{J}\eta_j \varepsilon_{p,\hat\theta,i}^\dagger k(x_j,x_i^\dagger) - \hat{\mu}_{\hat\theta,\mathbb{S},k_p}\right)^2, \]
\[ \hat T_{\hat \theta, \mathbb{S},k_p} = \frac{\hat{\mu}^\dagger_{\hat \theta,\mathbb{S},k_p}}{\hat{\sigma}^\dagger_{\hat \theta,\mathbb{S},k_p}}. \]
The asymptotic results for the $t$-statistic (Theorems 2-4) hold under the projection kernel $k_p(\cdot,\cdot)$ free from the estimation effect.
For our theoretical analysis, we use asymptotic critical values. However, since the weights $\{\eta_j\}_{j \in \mathbb{S}}$ and location points $\{x_j\}_{j \in \mathbb{S}}$ are chosen in a data-dependent manner, we generally recommend obtaining critical values via a multiplier bootstrap procedure to ensure proper control of the type I error at finite sample sizes, particularly when the sample size is relatively small.
For simplicity and ease of implementation, we compute the $t$-statistic without normalization and use $\hat{\mu}_{p, \hat{\theta}, J}$ instead of $\hat T_{p,\hat \theta, J}$ to construct test statistic. A detailed bootstrap procedure is provided in the Online Appendix B.
We approximate the asymptotic null distribution of $\sqrt{n}\hat{\mu}_{\hat \theta,\mathbb{S},k_p}^\dagger$ by that of $\sqrt{n}\hat{\mu}_{\hat{\theta}, \mathbb{S},k_p}^*$, where \[ \sqrt{n}\hat{\mu}_{\hat{\theta}, \mathbb{S},k_p}^* = \frac{1}{\sqrt{n}} \sum_{j \in \mathbb{S}} \eta_j (\boldsymbol{\hat \varepsilon}_{p}^*)^\top \mathbf{K}(\mathbf{X}^\dagger,x_j) \] and \[ \boldsymbol{\hat \varepsilon}_{p}^* = \hat{\boldsymbol{\Pi}}^\dagger (\boldsymbol{\hat \varepsilon}^\dagger \odot \mathbf{V}) . \] with $\mathbf{a} \odot \mathbf{b}$ being element-wise multiplication (Hadamard product) of vectors of the same size. $\mathbf{V}$ is a random vector of size $n$ with i.i.d random variables satisfying $\mathbb{E}(v_1) = 0$ and $\mathrm{Var}(v_1) = 1$. Notable examples include the Rademacher, standard normal, and Bernoulli random variables with \[ P(v_1 = 0.5(1-\sqrt{5})) = b, \quad P(v_1 = 0.5(1+\sqrt{5})) =1- b \] where $b = (1+\sqrt{5})/2\sqrt{5}$, see mammen1993bootstrap.
To theoretically justify the bootstrap approximation, no additional assumptions are needed. In contrast, other related bootstrap procedures often require extra conditions, such as restrictions on the bootstrap version of the estimator. These conditions may not hold in certain cases, such as when using the LASSO method.
Finally, we highlight that the time complexity of the proposed bootstrap procedure is $O(B n)$, where $B$ is the bootstrap size and $n$ is the test sample size. In contrast, for other KCM-type tests, the time complexity is $O(B n^2)$.
Theorems 2 and 3 establish that the test power increases with $\left|\hat{T}_{\hat{\theta}, \mathbb{S}, k_p}\right|$. A natural approach to enhancing test power is to maximize this signal-to-noise ratio. In the context of classification, this objective aligns with Fisher's discriminant analysis, which seeks to maximize the between-class variance while minimizing the within-class variance.
However, in an infinite-dimensional space such as a RKHS, directly inverting the covariance operator is ill-posed. Although Tikhonov regularization can stabilize the inversion process, it is computationally expensive.
In this section, we advocate and formalize the application of two Support Vector Machine (SVM) algorithms to learn the direction: \( w = \sum_{j \in \mathbb{S}} \eta_j k(x_j, \cdot). \) The first algorithm is the classical $\nu$-SVM, which aims to find a hyperplane orthogonal to $w$ that maximizes the margin between the nonparametric and parametric classes. The second algorithm treats the residual points (differences between the nonparametric and parametric parts) as a one-class classification problem, aiming to maximize the distance between the origin and a hyperplane that encloses the residual points. We demonstrate that these two algorithms increase lower bounds of the signal-to-noise ratio, thereby improving the power of the test.
We provide comprehensive details on the test statistic construction algorithms in the Online Appendix B.
The $\nu$-SVM is a supervised learning algorithm designed for class separation. It learns a hyperplane that maximizes the distance to the nearest training data point of any class, effectively maximizing the margin between classes.
In our framework, the hyperplane is defined as: \[ \left\{z_p k(x, \cdot) = (y_{p} k(x, \cdot), \mathcal{M}_{\hat{\theta},p}(x)k(x,\cdot)) \in \mathcal{H}_k : \left\langle w, z_p \right\rangle_{\mathcal{H}_k} + b = 0 \right\}, \] where $w$ is the hyperplane's normal vector and $b$ is the offset parameter. Here, $y_p$ and $\mathcal{M}_{\hat{\theta},p}(x)$ are generated using the same projection procedure as $\hat{\varepsilon}_p$. The dataset contains $2n$ training points, with half belonging to the nonparametric class and the other half to the parametric class.
Standard SVM typically assumes that data points lie in the same orthant, often the positive orthant, in the feature space. This assumption is often satisfied using a Gaussian kernel, which ensures that $\{k(x_i,\cdot)\}_{i=1}^n$ reside in the same orthant with unit length (see Section 2.3 of scholkopf1999support).
However, in our setting, the training data points $\{z_{p,i} k(x_i, \cdot)\}_{i=1}^{2n}$ do not lie in the same orthant, even when using a Gaussian kernel. To address this issue, we introduce shifted data points: \[ \{\tilde{z}_{p,i} k(x_i, \cdot)\}, \quad \text{where} \quad \tilde{z}_{p,i} = \{y_{p,i}, \mathcal{M}_{\hat{\theta},p,i}\} + e > 0 \; \forall i, \] with $e > 0$ being a constant shift. This transformation does not affect the properties of the test statistic because the shift is applied uniformly to both classes, and the test statistic is based on the difference between the two classes: \[ \hat{\varepsilon}_{p} k(x,\cdot) = (y_p + e - \mathcal{M}_{\hat{\theta},p}(x) - e) k(x,\cdot). \]
We then use the shifted data points to select $w$ via SVM. The primal optimization problem is given by \[ \max_{w, b, \xi, \rho} \quad \nu \rho - \frac{1}{n} \sum_{i=1}^n \xi_i - \frac{1}{2} \|w\|^2, \] subject to \[ l_i (\langle w, \tilde{z}_{p,i} k(x,\cdot)\rangle_{\mathcal{H}_k} + b) \geq \rho - \xi_i, \quad \forall i = 1, \dots, 2n, \] \[ \xi_i \geq 0, \quad \forall i = 1, \dots, 2n, \quad \rho \geq 0. \] Here, $\{l_i\}_{i=1}^{2n}$ are label variables that take the value $1$ for the nonparametric class and $-1$ for the parametric class. The parameter $\nu$ is a hyperparameter that balances the trade-off between maximizing the margin ($\rho$) and minimizing the number of support vectors. The slack variables $\{\xi_i\}_{i=1}^{2n}$ are introduced to allow for misclassification in the training data, thereby avoiding overfitting.
The margin parameter $\rho$ can be interpreted as the deviation signal, while $\|w\|_{\mathcal{H}_k}$ measures the noise level in our context. Specifically, for each class, if the data points are correctly classified by the hyperplane, their projections onto the direction $w$ should lie at least $\rho / \|w\|_{\mathcal{H}_k}$ away from the separating hyperplane. Consequently, the distance between the projected sample averages of the nonparametric and parametric parts should be at least $2 \rho / \|w\|_{\mathcal{H}_k}$, representing a signal-to-noise ratio that the SVM algorithm aims to maximize.
Solving the primal problem is challenging due to the infinite-dimensional nature of the RKHS. However, leveraging duality theory, we reformulate the primal problem in terms of dual variables $\{\alpha_j\}_{j=1}^n$. The resulting dual problem is \[ \min_{\alpha} \frac{1}{2} \sum_{i,j=1}^{2n} \alpha_i \alpha_j \, \tilde{z}_{p,i} \tilde{z}_{p,j} \, l_i l_j \, k(x_i,x_j), \] subject to \[ 0 \leq \alpha_i \leq 1/(2n), \quad \sum_{i=1}^{2n} \alpha_i \tilde{z}_{p,i} = 0, \quad \text{and} \quad \sum_{i=1}^{2n} \alpha_i \geq \nu, \] where $\{x_i = x_{n+i}\}_{i=1}^n$.
It can be shown that the training points associated with $\alpha_j > 0$ are support vectors, while the remaining points have $\alpha_j = 0$. Consequently, the index set $\mathbb{S}$ is determined by the indices of the support vectors. The weights $\{\eta_j\}_{j \in \mathbb{S}}$ are then given by: \[ \eta_j = \alpha_j \tilde{z}_{p,j} l_j. \]
Another approach to selecting $\mathbb{S}$ and $\{\eta_j\}$ is to use the One-Class Support Vector Machine (OCSVM) algorithm. OCSVM is an unsupervised algorithm designed for anomaly detection. It learns a hyperplane that separates the majority of data points, considered normal, from the origin, identifying points near the origin as anomalies. Geometrically, OCSVM maximizes the margin between this hyperplane and the origin while minimizing the number of data points within the margin.
In our context, we treat the differences between the nonparametric and parametric parts, i.e., $\{\hat{\varepsilon}_{p,i} k(x_i,\cdot)\}_{i=1}^n$, as `normal' data points, while the origin represents the null hypothesis and is considered an anomaly.
Similar to the two-class SVM, we need to transform the data points to lie in the same orthant: \[ \{\tilde{\varepsilon}_{p,i} k(x_i, \cdot)\}, \quad \text{where} \quad \tilde{\varepsilon}_{p,i} = \hat{\varepsilon}_{p,i} + e > 0 \; \forall i, \] with $e > 0$ being a constant shift. This transformation ensures all data points are positive and avoids issues caused by mixed signs.
We then use the shifted data points $\{\tilde{\varepsilon}_{p,i} k(x_i, \cdot)\}_{i=1}^n$ to select $w$ via OCSVM. The hyperplane is characterized as: \[ \left\{\tilde{\varepsilon}_{p} k(x, \cdot) \in \mathcal{H}_k : \left\langle w, \tilde{\varepsilon}_{p} k(x, \cdot) \right\rangle_{\mathcal{H}_k} = \rho \right\}. \]
To separate the majority of data points from the origin, the parameter $\rho$, which serves as a signal of deviation from the null hypothesis, should be maximized, while the norm of $w$ should be minimized. This objective is achieved by solving the following OCSVM optimization problem: \[
\]
The dual problem of OCSVM is given by \[ \min_{\alpha} \frac{1}{2} \sum_{i,j=1}^n \alpha_i \alpha_j \, \tilde{\varepsilon}_{p,i} \, k(x_i,x_j) \, \tilde{\varepsilon}_{p,j}, \] subject to \[ 0 \leq \alpha_i \leq 1/(\nu n), \quad \sum_{i=1}^n \alpha_i = 1, \quad \forall i=1,\ldots,n. \]
As with the two-class SVM, the training points associated with $\alpha_j > 0$ are support vectors, while the remaining points have $\alpha_j = 0$. Consequently, $\mathbb{S}$ is determined by the indices of the support vectors. The weights $\{\eta_j\}_{j \in \mathbb{S}}$ are then given by: \[ \eta_j = \alpha_j \tilde{\varepsilon}_{p,j}. \]
It is clear that the test power is determined by the value of \[ \left|T_{\hat{\theta}, \mathbb{S}, k_p}\right| = \left| \frac{\mu_{\hat{\theta}, \mathbb{S}, k_p}}{\sigma_{\hat{\theta}, \mathbb{S}, k_p}} \right|, \] where \[
\]
In this subsection, we investigate how the generalization error of the SVM algorithms affects the test power.
We begin with the two-class SVM algorithm and its generalization margin error bound (Theorem 7.3 in scholkopf2002learning). This result states that with probability at least $1-\delta$, we have: \[ \mathrm{Pr}\left(l^\dagger \langle \tilde{z}^\dagger_p k(x^\dagger, \cdot), w^* \rangle_{\mathcal{H}_k} < \rho \right) \leq \underbrace{\omega + \sqrt{\frac{c}{n} \left( \frac{R^2 \Lambda^2}{\rho^2} \ln^2(n) + \ln(1/\delta) \right)}}_{d}, \] where $w^*$ denotes the solution of the SVM, characterized by the index set $\mathbb{S}^*$ of support vectors and corresponding dual coefficients. Here $c$ is a universal constant, $\omega$ is the fraction of training points with margin smaller than $\rho / \|w^*\|_{\mathcal{H}_k}$, $R$ and $\Lambda$ are bounds on the norm of the data points and the norm of the weight vector, respectively: $\| z^\dagger k(x^\dagger, \cdot)\|_{\mathcal{H}_k} \leq R$ and $\| w^*\|_{\mathcal{H}_k} \leq \Lambda$, and $n$ is the sample size of the training dataset.
This generalization error bound implies that for a new test point, the probability of misclassifications by the learned separating hyperplane is bounded by $d$, which decreases as the margin parameter $\rho$ increases.
Without loss of generality, assume that the nonparametric part, after projecting onto $w^*$, has positive values for correctly classified points, while the parametric part has negative values for correctly classified points. Then, with probability at least $1-\delta$, we have:
Let $R_w$ be the bound on the projected value of the data points, i.e., $|\langle \tilde{z}^\dagger_p k(x^\dagger, \cdot), w^* \rangle_{\mathcal{H}_k}| \leq R_w$. It follows that: \[ \langle \tilde{Y}_p^\dagger k(x^\dagger, \cdot), w^* \rangle_{\mathcal{H}_k} \in
\] and \[ \langle \widetilde{\mathcal{M}}_{\hat{\theta}, p}^\dagger(x^\dagger) k(x^\dagger, \cdot), w^* \rangle_{\mathcal{H}_k} \in
\]
Subsequently, we can bound $\left|T_{\hat{\theta}, \mathbb{S}^*, k_p}\right|$ as: \[ \left|T_{\hat{\theta}, \mathbb{S}^*, k_p}\right| = \frac{\mu_{\hat{\theta}, \mathbb{S}^*, k_p}}{\sigma_{\hat{\theta}, \mathbb{S}^*, k_p}} \geq \frac{(1-d)^2 2\rho}{\sigma_{\hat{\theta}, \mathbb{S}^*, k_p}} + \frac{2(\rho - R_w)(1-d)d}{\sigma_{\hat{\theta}, \mathbb{S}^*, k_p}} - \frac{2R_w d^2}{\sigma_{\hat{\theta}, \mathbb{S}^*, k_p}}. \]
The last two terms are negative and negligible if $d$ is small. The first term is positive by construction and increases with $\rho$. Furthermore, the standard deviation $\sigma_{\hat{\theta}, \mathbb{S}^*, k_p}$ is positively related to $\|w^*\|_{\mathcal{H}_k}$. Combining these observations, we conclude that the test power is positively related to the margin distance $\rho / \|w^*\|_{\mathcal{H}_k}$, which is maximized by the SVM algorithm.
The generalization error bound for OCSVM (Theorem 8.6 in scholkopf2002learning) is more complex than that of two-class SVM. However, one definitive conclusion can be drawn: the distance between the origin and the hyperplane, $\rho / \|w^*\|_{\mathcal{H}_k}$, is inversely related to the probability bound $d$ of the generalization error.
To investigate how the OCSVM generalization error affects the test power, assume without loss of generality that $\mu_{\hat{\theta}, \mathbb{S}^*, k_p} < 0$. Then, with probability at least $1-\delta$, we have: \[ \mathrm{Pr}\left(\langle \tilde{\varepsilon}^\dagger_p k(x^\dagger, \cdot), w^* \rangle_{\mathcal{H}_k} < \rho\right) = F\left(\frac{\rho + \mu_{\hat{\theta}, \mathbb{S}^*, k_p}}{\sigma_{\hat{\theta}, \mathbb{S}^*, k_p}}\right) \leq d, \] where $F$ is the cumulative distribution function of the random variable: \[ \frac{\langle \tilde{\varepsilon}^\dagger_p k(x^\dagger, \cdot), w^* \rangle_{\mathcal{H}_k} + \mu_{\hat{\theta}, \mathbb{S}^*, k_p}}{\sigma_{\hat{\theta}, \mathbb{S}^*, k_p}}. \]
By the properties of $F$, we obtain: \[
\]
As $\rho / \|w^*\|_{\mathcal{H}_k}$ increases (and consequently $d$ decreases), the second term $-F^{-1}(d)$ also increases. The first term is positively related to the margin distance $\rho / \|w^*\|_{\mathcal{H}_k}$. Therefore, the test power is positively related to the margin distance $\rho / \|w^*\|_{\mathcal{H}_k}$, which is maximized by the OCSVM algorithm.
The above analysis also suggests that artificially shifting the data points to have large values of $\tilde{\varepsilon}_p$ would have little effect on the test power. This is because such shifting also moves the location of $F(\cdot)$ to the right. Hence, holding everything else constant, although $\rho$ becomes larger, so does $F^{-1}(d)$. Consequently, the increase in $\rho$ is counterbalanced by the corresponding increase in $F^{-1}(d)$, leaving the overall test power largely unaffected.
We consider the following simulation designs. The null model is given by \[ DGP_1: Y = \theta_0^\top X + \varepsilon. \] The alternative models are given by \[
\] where $\theta_0$ are $q$-dimensional vectors with first $p$ and the last $q-p$ elements being 1 and 0, respectively. Here, we set $c=0.25$, $p = \lfloor 0.1 q \rfloor$ and $q = \{10,20\}$. The covariates $X$ follow a $q$-dimensional standard normal distribution. The error term $\varepsilon$ is a standard normal random variable. Similar DGPs have been considered in escanciano2006consistent,tan2022integrated,escanciano2024gaussian.
Beside the proposed SVM based t-type statistics ($\hat T_{\nu-SVM}$ and $\hat{T}_{OCSVM}$), we consider three alternative test statistics: the ICM test ($\hat{T}_{ICM}$) by bierens1982consistent, the KCM test ($\hat{T}_{KCM}$) by muandet2020kernel, and the Gaussian Process test ($\hat{T}_{GP}$) of escanciano2024gaussian. All three alternative statistics take the form of \[ n \hat{T} = \frac{1}{n}\sum_{i,j = 1}^{n} \varepsilon_{\hat \theta,i} k(x_i,x_j)\varepsilon_{\hat \theta,j}. \] All tests employ the Gaussian kernel $k(x, y) = \exp(-\|x - y\|_2^2 / \sigma)$. For the ICM test, we fix $\sigma = 2$. For the $\nu$-SVM, OCSVM, KCM, and GP tests, we follow escanciano2024gaussian and select $\sigma$ using the median heuristic, i.e., $\sigma = \text{median}(\{\|x_i - x_j\|_2\}_{i \neq j})$. Both the proposed SVM-based tests and the GP test utilize the projection method described in Section 4 to mitigate estimation effects. For the KCM and GP tests, critical values are obtained via a multiplier bootstrap procedure as outlined in Section 4, while the ICM test uses the wild bootstrap procedure of delgado2006consistent. For the SVM-based tests, we report results using both analytic and multiplier bootstrap critical values. The number of bootstrap replications is set to $B = 500$, and each simulation scenario is repeated $R = 1000$ times. Empirical sizes and powers are computed as the proportion of rejections over the $R$ replications.
For the SVM-based methods, we allocate $10\%$ of the data for training the one-class SVM and use the remaining $90\%$ for testing. SVM implementations are carried out using the scikit-learn Python package (OneClassSVM for OCSVM and NuSVC for $\nu$-SVM). Due to space constraints, additional simulation designs and results are provided in Online Appendix C.
Regarding the simulation results, we observe that the performance of the SVM-based test statistics is similar under analytic and multiplier bootstrapped critical values. However, it should be emphasized that, in general, the finite-sample performance using bootstrap critical values is superior to that using analytic critical values. This is demonstrated by the additional simulation studies documented in Online Appendix C.
Both SVM-based test statistics demonstrate strong finite-sample performance compared to the other tests (Tables (ref) and (ref)). When the covariate dimension is moderate ($q = 10$), the SVM-based and GP tests exhibit accurate size control, whereas the ICM and KCM tests suffer from size distortion. In terms of power, the SVM-based tests perform comparably to the GP test, while the ICM and KCM tests show lower power. When the covariate dimension is high ($q = 20$), SVM-based tests maintain accurate size control and high power, whereas the other tests experience significant size distortion and reduced power.
Since our DGPs are sparse, we also investigate the finite-sample performance of the SVM-based test statistics when model estimators are obtained using LASSO. The results for $q = 10$ are presented in Tables (ref) and (ref), while the results for $q = 20$ are provided in Online Appendix C. Overall, we observe that the proposed SVM-based test statistics maintain accurate size control and high power under both analytic and multiplier bootstrapped critical values. \newlength{\colwidth} {0.104\textwidth}
Reproducing kernel-based test statistics are known to be highly sensitive to the choice of kernel parameters, a phenomenon extensively documented in the nonparametric two-sample testing literature (see gretton2012optimal, sutherland2021generativemodelsmodelcriticism, liu2021learningdeepkernelsnonparametric). To investigate the sensitivity of our proposed test statistics to the kernel parameter $\sigma$, we conduct a series of simulation studies using varying values of $\sigma$: $\sigma \in \{1, 2, 3, 4\}$. Figures (ref)--(ref) present the sizes and powers of the SVM-based, GP, and KCM tests under these different $\sigma$ values.
From these results, we draw two key observations. First, all specification test statistics exhibit sensitivity to the choice of kernel, consistent with findings in the existing literature on kernel-based testing. Second, SVM-based test statistics (with multiplier bootstrapped critical values) demonstrate robust performance across different $\sigma$ values. This robustness is particularly pronounced when the covariate dimension is high ($q = 20$).
In addition to the finite sample performance, the computational complexity is also noteworthy. Here we document the run times in \(\mathrm{DGP}_2\) as an illustration. We focus on the case where the sample size is \(n = 400\) and the covariate dimension is \(q = 20\). The records are collected by running tests in Python 3.13.2 on a modern desktop computer (AMD Ryzen 9 9950X CPU, 128GB RAM), with parallel computing enabled (using the joblib.Parallel routine with \(32\) CPU cores). In Table (ref), we report the time costs for the \(1000\) repeated experiments for each test with \(500\) bootstrap samples when required. Clearly, the proposed SVM-based tests are more computationally efficient than the other bootstrap-based tests. The analytic implementation of the SVM-based tests is even faster, as it does not require bootstrapping.
We apply our proposed SVM-based specification tests to two datasets from the UCI Machine Learning Repository lichman2017uci. The first dataset, “Wine Quality”, comprises 11 covariates and 4,898 observations. The response variable is wine quality, scored on a scale of 3 to 9. We use linear regression to model wine quality based on the covariates.
The second dataset, “Students' Dropout”, includes 36 covariates and 4,424 observations. The binary response variable indicates whether a student has dropped out of school. We employ the Probit model to investigate the probability of dropout given the covariates.
Given the relatively large sample sizes, we randomly split each dataset into 60% training data and 40% testing data. Table (ref) presents $p$-values from different testing procedures.
For the Wine Quality dataset, all bootstrap-based tests ($\hat T_{\nu-SVM}$ and $\hat{T}_{OCSVM}$ with bootstrap, $\hat{T}_{GP}$, $\hat{T}_{KCM}$, and $\hat{T}_{ICM}$) unanimously reject the null hypothesis (p-value = 0.000), indicating that the linear regression model is misspecified for this dataset. Interestingly, the SVM-based tests with analytic critical values fail to reject the null hypothesis at the 5% significance level, showing a notable difference from its bootstrap counterpart.
For the Students' Dropout dataset, Most tests ($\hat T_{\nu-SVM}$ and $\hat{T}_{OCSVM}$ with bootstrap, $\hat{T}_{GP}$, and $\hat{T}_{KCM}$) reject the null hypothesis at the 5% significance level, suggesting that the logistic regression model is inadequate for this dataset. The ICM ($\hat{T}_{ICM}$) and $\nu-$SVM ($\hat T_{\nu-SVM}$) tests are the only two that fail to reject the null hypothesis, contradicting the other test results.
The discrepancy between bootstrap and analytic implementations of the SVM-based tests confirms the simulation study findings that bootstrap methods generally provide better finite-sample performance. The ICM test's inconsistency with other tests for the Students' Dropout dataset might be due to its lower power in detecting certain types of misspecification, especially in higher-dimensional settings (this dataset has 36 covariates).
Overall, these results suggest that both datasets likely require more complex models than simple linear or logistic regression to accurately capture the underlying relationships between predictors and response variables.
This paper proposes a novel framework for enhancing the power of specification tests in parametric models by strategically integrating Support Vector Machines (SVMs) for direction learning. By addressing limitations in traditional methods, such as ICM and KCM tests, our approach focuses on identifying good projection directions in Reproducing Kernel Hilbert Spaces (RKHS) to enhance the test's ability to detect deviations from the null hypothesis. The key innovation lies in two SVM-based mechanisms: (1) maximizing the discrepancy between nonparametric and parametric classes via a margin-maximizing hyperplane, and (2) maximizing the separation between residuals and the origin using one-class SVM. These mechanisms target the signal-to-noise ratio in the test statistic, ensuring superior power against arbitrary alternatives.
Theoretical analysis establishes consistency under the alternative hypothesis, while simulations demonstrate significant power gains over existing methods, particularly in high-dimensional settings. The use of multiplier bootstrap for critical value computation ensures reliable inference even with a small sample size. Empirical studies on real datasets further validate the proposed methodology, highlighting its effectiveness in detecting model misspecification.
By leveraging SVMs to learn optimal projection directions, this work advances the toolkit for model validation, offering a computationally efficient, omnibus solution with a pivotal chi-square asymptotic distribution. Future research could explore adaptive kernel selection methods (e.g., the AdaBoost algorithm) to further enhance test power. Another interesting direction would be to construct test statistics based on the classification accuracy of the learned SVM, an approach gaining traction in the nonparametric two-sample testing literature kim2021classification,hediger2022use,lopez2016revisiting. These extensions could further capitalize on the connection between classification methodology and specification testing established in this work.