Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
63,838 characters · 17 sections · 36 citation commands
Using SVM to Estimate and Predict Binary Choice Models
\singlespacing
\singlespacing JEL Classification: C13, C22 \\ Keywords and phrases: machine learning, binary choice model, support vector machine, maximum likelihood estimation, estimation and prediction, asymptotics
\setcounter{footnote}{0} \onehalfspacing
Machine learning has become increasingly popular in economics. See, e.g., mullainathan-spiess-17 and athey-imbens-19, among many others. In machine learning, a variety of algorithms have been developed to analyze data within a flexible framework without an underlying statistical model. Although they were initially proposed as purely computational algorithms, it has recently been shown that some of them can be used to estimate a well-defined class of statistical and econometric models. For instance, it is now widely recognized that random forests and neural networks can be used to consistently estimate nonparametric regression functions under suitable regularity conditions. For the related asymptotic theories of random forests and neural networks, the reader is referred to, e.g., chi-vossler-fan-lv-22 and farrell-liang-misra-19, respectively, and the references therein.
In this paper, we show that the support vector machine (SVM), another widely used algorithm for classification in machine learning, can be used under appropriate conditions to consistently estimate the binary choice model (BCM). The SVM was originally developed by cortes-vapnik-95 as an algorithm to classify binary outcomes observed with other related features. It is based on a simple idea of using a hyperplane to partition the feature space into two subsets associated with binary outcomes and has thus been regarded as an algorithm entirely independent of any statistical or econometric model. However, it is natural to consider the SVM as a method for analyzing the BCM, since the way the SVM classifies binary observations is essentially the same as that of a BCM with a zero median error term. Indeed, the SVM and the traditional approaches to BCMs use the same type of classifier, the one given by a separating hyperplane, to predict binary outcomes from covariates. For observations generated by the BCM, the SVM classifier will therefore not be asymptotically optimal unless it consistently estimates the underlying BCM.
The main finding of this paper is that the slope coefficient of the hyperplane provided by the SVM is a consistent estimator for the slope parameter in the BCM under the same conditions as those required for consistency of the quasi-maximum likelihood estimator (QMLE). The estimator provided by the SVM, which is referred to as the SVM estimator in the paper, is thus asymptotically equivalent to the QMLE in this sense.\footnote{The SVM estimator and the QMLE generally do not have the same limit distribution, so they are not fully equivalent even asymptotically. Also, our asymptotic setup does not allow for the number $m$ of features or covariates going to infinity.} This is rather surprising, since the SVM has been considered as an algorithm which does not rely on any statistical or econometric model. Given that the SVM is used widely to analyze binary outcomes in machine learning, this asymptotic equivalence of the SVM estimator to the QMLE adds an interesting and important interpretation of the SVM. We establish consistency of the SVM estimator for the slope parameter without assuming that a pseudo-true value exists. This is essential because, for the SVM estimator, a pseudo-true value may not exist if the two classes defined by a binary outcome are severely imbalanced, in which case the SVM fails and simply predicts a majority class.\footnote{This is well known. As shown in the paper, the problem disappears if we use class weight, an option for the SVM adopted widely to deal with imbalanced data.}
The QMLE for the parameter in the BCM is generally inconsistent. It is, however, expected to yield a consistent estimator for the slope parameter in the BCM if the conditional expectation of covariates on its systematic component is given as a linear function. This condition is not as stringent as it may sound. It holds if covariates jointly have an elliptical distribution, and we may give appropriate weights to the observations on covariates so that they may be regarded as being drawn from such a distribution. See, e.g., ruud-86, newey-ruud-94. Consistent estimation of the slope parameter is of primary interest for the BCM, especially when its dimension is large with many covariates. This is because we may easily estimate the remaining one-dimensional intercept parameter once a consistent estimator for the multi-dimensional slope parameter is obtained. Under the required linearity in conditional expectation, consistency of the slope parameter in the BCM may be easily established if we assume the existence of a pseudo-true value as in ruud-83,ruud-86.\footnote{The existence of a pseudo-true value is not guaranteed. For a rigorous proof of his result, see chang-park-yan-25.}
The SVM estimator is comparable in many respects to the logit estimator, which is most commonly used QMLE especially for the large dimensional BCM, except that it breaks down in the extreme case where severe class imbalance exists. By simulation, we compare the SVM estimator with the logit estimator in terms of their finite sample performance. In finite samples, the performances of the two estimators could be quite distinct depending upon the distributions of covariates and errors. It seems clear that no one dominates the other. Though their performances are largely comparable, the SVM estimator performs significantly better than the logit estimator in some cases and vice versa in others.
Once a consistent estimator for the slope parameter is obtained, we may consistently estimate the intercept parameter in various ways. We may estimate the intercept parameter using, for example, the maximum score estimator of manski-85 or its smoothed version given by horowitz-92, by replacing the slope parameter with its consistent estimate. In fact, we show in the paper that replacing the true slope parameter by its SVM estimator does not affect the limit distribution of the maximum score estimator for the intercept parameter.
The rest of the paper is organized as follows. In Section 2, we introduce the SVM and BCM, which is followed by a framework to analyze the classification of the BCM outcomes by the SVM. Section 3 provides the asymptotic properties of the SVM estimator, and discusses the conditions under which it is consistent for the slope parameter in the BCM. It also explains how to obtain a consistent estimator for the intercept parameter, once a consistent estimator for the slope parameter is available, in the BCM. In Section 4, we contrast the SVM with the QMLE. We show that the SVM cannot be defined as a QMLE, even though the SVM and QMLE are largely comparable. Moreover, the effect of severe imbalance in binary outcomes is fatal to the SVM unless it is applied with class weights, while its effect is not so detrimental for the QMLE. Section 5 concludes the paper. Mathematical proofs are provided in the Appendix.
A word on notation. We let $\mathcal L(Z)$ signify the distribution of $Z$ for a random vector $Z$, and $\mathcal L(Z\mid W=w)$ signify the distribution of $Z$ conditional on another random vector $W$ taking the value $w$. Other standard notations include $1\{\cdot\}$ for the indicator function, $\|\cdot\|$ for the Euclidean norm, $\to_p$ for convergence in probability, and $o_p(1)$ as a generic notation for a term converging in probability to zero, among others, which will also be used. We also use the notation $f(\infty) := \lim_{t\to \infty} f(t)$ or $f(-\infty):=\lim_{t\to -\infty} f(t)$ when the limit is well-defined. These notations are used throughout the paper without any further reference.
Let $Y$ be a binary random variable taking values $1$ and $-1$, and $X$ be an $m$-dimensional random covariate which has the distribution $\mathcal L(X)$ over the support $\mathcal{X}$. We define $(Y_i,X_i)$, $i=1,\ldots,n$, to be an i.i.d.\ random sample from $(Y,X)$.
The support vector machine (SVM) is used widely in machine learning for classification problems.\footnote{We only consider the prototypical SVM without kernel trick throughout this paper; this restriction is mainly intended to relate} the SVM to the binary choice model. The methodology does not assume any underlying model, and it can be used to analyze any binary-valued $(Y_i)$, $Y_i=1$ or $-1$, associated with an $m$-dimensional set of features $(X_i)$ for $i=1,\ldots,n$. More precisely, the SVM finds a hyperplane
defined by $\alpha\in\mathbb R$ and $\beta\in\mathbb R^m$, which most effectively separates the two subsets of $(X_i)$, one associated with $Y_i = 1$ and the other with $Y_i = -1$, depending upon which side of the hyperplane each $X_i$ falls on.
The SVM was first introduced in the case where $(X_i)$ coupled with $Y_i=1$ and $(X_i)$ coupled with $Y_i=-1$ can be separated in the space $\mathbb R^m$ by a hyperplane. In this case, the SVM method, which will be referred to as the hard-margin SVM, chooses a hyperplane defined by $\alpha\in\mathbb R$ and $\beta\in\mathbb R^m$ given by the maximization problem
for all $i=1,\ldots,n$. Provided that $Y_i = \operatorname*{sgn}(\alpha + X_i'\beta)$, $Y_i(\alpha + X_i'\beta)/\|\beta\| = |\alpha + X_i'\beta|/\|\beta\|$ is the distance of $X_i$ to the hyperplane. We call $M$ in (ref) the margin between the hyperplane and $(X_i)$. Clearly, we may positively scale $\alpha$ and $\beta$ arbitrarily in (ref), and thus, we may set $\|\beta\| = 1/M$ so that (ref) reduces to
for all $i=1,\ldots,n$. The hard-margin SVM is usually defined by (ref).
The two subsets of $(X_i)$, which are associated with $Y_i = 1$ and $Y_i=-1$, respectively, are rarely separable in practical applications. In this case, we use the soft-margin SVM, which introduces slack variables $(\tau_i)$ and solves
for all $i=1,\ldots,n$ in the presence of an additional constraint $\sum_{i=1}^n\tau_i < \lambda_n$ with some constant $\lambda_n>0$ replacing (ref). As before, we may rewrite (ref) as
for $i=1,\ldots,n$, by taking the additional constraint into consideration and redefining $\lambda_n$ appropriately. Once again, we may redefine $\lambda_n$ appropriately and reformulate the minimization in (ref) as
where $[\,\cdot\,]_+ = \max\,\{\cdot,0\}$. Note that the two constraints appearing in (ref) may be combined and rewritten as \[ \tau_i \ge \big[1-Y_i(\alpha + X_i'\beta)\big]_+ \] and we set \[ \tau_i = \big[1-Y_i(\alpha + X_i'\beta)\big]_+ \] to derive (ref) from (ref). For the analysis of the SVM in this paper, we will focus on the soft-margin case, for which the optimization problem is given by (ref).
A natural setting to generate a binary outcome is to use the binary choice model (BCM), which is given by
where $\operatorname*{sgn}$ is the sign function defined as $\operatorname*{sgn}\,(z) = \pm 1$ for $z\ge 0$ and $z<0$, respectively, $X$ is an $m$-dimensional vector of covariates, $\theta_0 = (\alpha_0,\beta_0')'$ is the true value of the $(1+m)$-dimensional parameter vector $\theta = (\alpha,\beta')'$, and $U$ is the error term. Throughout the paper, we let $\theta\in\Theta$, where $\Theta$ is a convex subset of $\mathbb R^{1+m}$ that is arbitrarily large.\footnote{This is necessary because we identify $\theta_0$ only up to scalar multiplication in the derivation of our main result. Our definition of $\Theta$ is essentially equivalent to setting $\Theta = \mathbb R^{1+m}$. In this paper, however, we define $\Theta$ separately from $\mathbb R^{1+m}$ to make our subsequent assumptions more directly comparable to those existing in the literature.}
Since we focus on the consistency of the slope coefficient up to a positive scalar—referred to as slope consistency for simplicity—we impose conditions to ensure that $\theta_0$ is identified up to a positive scalar.
Assumptions (ref) and (ref) are imposed for the BCM in horowitz-92, modified from manski-75,manski-85, which ensure that $\theta_0 = (\alpha_0,\beta_0')'$ is identified up to multiplication by a positive scalar. Both of them are assumed to hold throughout what follows.
To predict a binary outcome $Y$, the SVM uses a classifier
with parameter $\theta = (\alpha,\beta')'$, which is formally defined as a function on $\mathcal X$ associating any realization of $X$ with a value of $Y$ taking values $\pm 1$. To evaluate a classifier $C$, we define its conditional risk as \[ R(x,C) = \mathbb P\big\{Y\ne C(X)\big|X=x\big\} \] for $x\in \mathcal X$, and its risk as $R(C) = \int R(x,C)\,P(dx)$, where $P$ is the distribution of $X$. An optimal classifier $C_0$ is defined as a classifier that has the smallest risk.
We can readily see that an optimal classifier is given by \[ C_0(x) = \operatorname*{sgn}\big[\Pi(x)-1/2\big] \] for $x\in\mathcal X$, where $\Pi$ is the conditional choice probability (CCP) defined as $\Pi(x) = \mathbb P\big\{Y=1\big|X=x\big\}$ for $x\in\mathcal X$. This is because \[ R(x,C) = \Pi(x)1\big\{C(x) = -1\big\} + \big(1-\Pi(x)\big)1\big\{C(x) = 1\big\} \] is minimized if we set $C(x) = \pm 1$ for $x\in\mathcal X$ such that $\Pi(x)\gtrless 1/2$.
For the BCM, the CCP is given by \[ \Pi(x) = \mathbb P\big\{U<\alpha_0 + X'\beta_0\big|X=x\big\} = F_x\big(\alpha_0 + x'\beta_0\big), \] where $F_x$ is the distribution function of $U$ conditional on $X=x$ and $\theta_0 = (\alpha_0,\beta_0')'$ is the true value of the parameter $(\alpha,\beta')'$. However, under Assumption (ref), we have $F_x(u) \geq 1/2$ if and only if $u \geq 0$ and $F_x(u) < 1/2$ if and only if $u < 0$, which yields \[ \Pi(x)
1/2 \quadif and only if\quad \alpha_0 + x'\beta_0
0 \] with $u = \alpha_0 + x'\beta_0$. The optimal classifier $C_0$ for the BCM is therefore given by
which belongs to the same class of classifiers as the SVM introduced in (ref).
Suppose that observations are drawn from the BCM so that the optimal classifier is given by (ref), and that the SVM applied to those observations yields the classifier in (ref) with
where $\hat\alpha$ and $\hat\beta$ define the separating hyperplane in the SVM. For the SVM classifier to be asymptotically optimal, $\hat\theta$ should converge in probability to $\theta_0$ up to a positive scale factor so that the separating hyperplane given by the SVM approaches the true hyperplane defined by $\theta_0$.\footnote{Here, asymptotic optimality of $\hat C$ means $\hat C(x) \to_p C_0(x)$ for $P$-almost every $x$.} Accordingly, we may use $\hat\theta$ as an estimator for $\theta_0$, recognizing that $\theta_0$ is identified only up to multiplication by a positive scalar.
For the rest of the paper, we assume that $(Y,X)$ is generated by the BCM unless stated otherwise explicitly, and consider $\hat\theta$ introduced in (ref) as an estimator, called the SVM estimator, of the parameter $\theta_0$ in the underlying BCM. As will be shown, $\hat\theta$ is consistent for $\theta_0$ under suitable regularity conditions. The required conditions for consistency of the SVM estimator $\hat\theta$, including both $\hat\alpha$ and $\hat\beta$ for the intercept and slope parameters $\alpha_0$ and $\beta_0$, respectively, are quite stringent. Consistency of the slope estimator $\hat\beta$ for the slope parameter $\beta_0$ is, however, much milder and only requires conditions that we typically impose for consistency of the quasi-maximum likelihood estimator (QMLE).
In this section, we develop the asymptotic properties of the SVM estimator $\hat\theta = (\hat\alpha,\hat\beta')'$ for the underlying BCM model.
Before establishing the consistency of the SVM estimator $\hat\theta = (\hat\alpha,\hat\beta')'$ for the BCM parameter $\theta_0 = (\alpha_0,\beta_0')'$ under suitable conditions, we start by defining the probability limit $\theta_\ast = (\alpha_\ast,\beta_\ast')'$ of the SVM estimator without assuming that $(Y,X)$ is generated by the BCM.
The SVM yields the solution $\hat\theta = (\hat\alpha,\hat\beta')'$ defined as a minimizer of
which is assumed to exist and be unique. For our asymptotics in the paper, we follow koo-lee-kim-park-08, zhang-wu-wang-li-16 and wang-yang-chen-liu-19, among others, and assume that $\lambda_n = o(n)$.\footnote{By setting $\lambda_n$ to be large, we make it less costly to increase the margin and allow for the observations in the margin and even on the other side. In practice, it is common to set $\lambda_n$ to a fixed value. For instance, when using existing packages in Python (sklearn.svm.LinearSVC) or MATLAB (fitcsvm), the default choice is $\lambda_n = 1/2$, under which conditions $\lambda_n = o(n)$ and $\lambda_n = o(\sqrt{n})$ used in this paper are satisfied. } Under this and other suitable regularity conditions, we may then expect that $Q_n(\theta)\to_p Q(\theta)$, where
uniformly in $\theta$ over some parameter set $\Theta$, from which it follows immediately that $\hat\theta \to_p \theta_\ast$ if $Q(\theta)$ has a unique minimum $\theta_\ast$.
Lemma (ref) follows immediately from Theorem 2.7 in newey-mcfadden-94, since all conditions required in their theorem are trivially satisfied under our Assumption (ref). In particular, note that $Q_n(\theta)$ and $Q(\theta)$ are convex in $\theta\in \Theta$.
Lemma (ref) assumes that $\theta_\ast$ is the unique minimizer of $Q(\theta)$ over $\Theta$. This assumption will not be imposed but verified under the conditions for the slope consistency introduced below. In particular, provided the slope consistency, i.e., $\beta_\ast = c_\ast \beta_0$ for some $c_\ast > 0$, Lemma (ref) below shows that the Hessian matrix of $Q$ at $\theta_\ast$ is positive definite. It then follows that $\theta_\ast$ is the unique minimizer of $Q$.
\paragraph{Gradient and Hessian for SVM when well-defined } For the later analysis of SVM slope consistency, we will make use of the gradient of $Q$. Define
and
where $\delta(\cdot)$ denotes the Dirac delta function with $\delta(z) = (d/dz) 1\{z>0\}$ in the distributional sense. For any $\theta = (\alpha,\beta')'\in \mathbb R^{1+m}$ such that $\beta_m\neq 0$, $\alpha+X'\beta$ has a Lebesgue density $f_{\alpha + X'\beta}$ by Assumption (ref) (a), and we may thus define $\ddot Q(\theta)$ more explicitly as
Under some regularity conditions, which will be provided below, $\dot{Q}(\theta)$ and $\ddot{Q}(\theta)$ actually become the gradient vector and Hessian matrix of $Q(\theta)$, respectively, for $\theta\in\mathbb R^{1+m}$ such that $\beta_m \neq 0$.
If $(Y,X)$ is generated by the BCM, Assumption (ref) (c) requires the continuity of the CCP $\Pi(x) = F_{x}(\alpha_0+x'\beta_0)$ in $x_m$, where $F_{x}$ is the distribution function of $U$ conditional on $X=x$ and $x=(x_{-m}',x_m)'$.
Lemma (ref) was established earlier by koo-lee-kim-park-08 and has been used in later work on SVMs, such as zhang-wu-wang-li-16. Their results, however, require that $\mathcal{L}(X|Y=1)$ and $\mathcal{L}(X|Y=-1)$ have Lebesgue densities, which is rather restrictive since it does not allow for the presence of discrete variables in $X$. In contrast, our assumptions do not impose such restrictions on $(Y,X)$, requiring only that the scalar index admit a Lebesgue density.
Subsequently, we assume that $(Y,X)$ is generated by the BCM with the true parameter value $\theta_0$. The SVM slope $\hat\beta$ is consistent for $\beta_0$ if and only if the probability limit $\beta_\ast$ of $\hat\beta$ is a positive multiple of $\beta_0$, and $\hat\theta$ is consistent for $\theta_0$ if and only if $\theta_\ast$ is a positive multiple of $\theta_0$. In this section, we consider the conditions for slope consistency and consistency of the SVM estimator, with a main focus on the slope consistency.
Let \[ V = \alpha_0 + X'\beta_0 \] and \[ \Pi(v) = \mathbb P\{Y=1 |V=v\} = \mathbb P\{U\leq V|V=v\}. \] We introduce two important assumptions that are crucial for the slope consistency of the SVM, which are also imposed for the slope consistency of the QMLE as in chang-park-yan-25.
Assumption (ref) requires that the error distribution depends on $X$ only through the index $V$. Such an index dependence condition for the error distribution is often used in the literature. It is used in, e.g., klein-spady-93 for the BCM, and ichimura-93 for a class of single-index models, among many others. This condition is satisfied trivially if $U$ is independent of $X$, as is often assumed for the BCM.
Assumption (ref) requires that the conditional mean of $X$ given $V$ is a linear function of $V$, which will be referred to simply as the linearity in expectation condition. This condition is certainly restrictive, but it holds, for instance, if $X$ has an elliptical distribution. We may also weight observations of $X$ appropriately so that the distribution of the weighted observations can be regarded as satisfying this condition; see, e.g., ruud-86 and newey-ruud-94.\footnote{ Specifically, in place of the original observations $\{x_i\}_{i=1}^n$ of $X$, they use the observations weighted by $w_i = \sigma(x_i)/\hat\tau(x_i)$ for $i=1,\ldots,n$, where $\sigma$ is the standard multivariate normal density which satisfies Assumption (ref), and $\hat\tau$ is a local kernel density estimator of $X$. The weighted observations may be approximately regarded drawn from a distribution with density $\sigma$. }
To analyze the SVM slope consistency, we consider the SVM over a restricted set of parameters $\theta = (\alpha,\beta')'$ specified as
with $c,r\in \mathbb R$. The resulting estimator will be referred to as the restricted SVM, whose population objective function is
Note that by Lemma (ref), for any $c\neq 0, r\in \mathbb R$, the gradient of $Q_\bullet(c,r)$ is
Let
Provided that $c_\ast > 0$, $(c_\ast,r_\ast)$ solves the FOCs $\dot Q_\bullet(c,r)=0$, and thus,
where the second equality follows from the law of iterated expectation and Assumption (ref), the third equality is by Assumption (ref), and the last line is by $\dot Q_\bullet(c,r) =0$. Since $Q(\theta)$ is convex and has a positive definite Hessian matrix at any $\theta$ with $\beta_m \neq 0$ by Lemma (ref), it holds that
and thus, the SVM slope $\hat\beta$ is consistent for $\beta_0$.
It is clear that the key to establishing the slope consistency of the SVM is to ensure $c_\ast>0$. In contrast to the slope consistency of the QMLE analyzed in chang-park-yan-25, we need an additional condition to ensure $c_\ast >0$.
In what follows, we first introduce some notations, after which Lemma (ref) provides the necessary and sufficient conditions for $c_\ast>0$. Let $\mathbb P\{Y=1\} \ge 1/2$. This causes no loss of generality, since we may redefine $Y$ by $-Y$ if necessary. Define \[ p(v) = \mathbb P\big\{U \leq V<v\big\} \quad\mbox{and}\quad q(v) = \mathbb P\big\{U>V>v\big\},\vspace{-0.084in} \] \[ \tau(v) = \mathbb E\,V1\big\{U \leq V<v\big\} \quad\mbox{and}\quad \sigma(v) = \mathbb E\,V1\big\{U>V>v\big\} \] for $v\in\mathbb R$. Under Assumption (ref) (a)-(b), $p$ is strictly increasing with $p(-\infty) = 0$ and $p(\infty) = \mathbb P\{Y = 1\} \ge 1/2$, whereas $q$ is strictly decreasing with $q(-\infty) = \mathbb P\{Y=-1\} \le 1/2$. Thus, $(p^{-1}\circ q)(-\infty)$ is well-defined in $\mathbb R \cup\{\infty\}$.
The following lemma shows that Assumption (ref) is indeed the necessary and sufficient condition for $c_\ast>0$, under some regularity conditions. We discuss the interpretation of Assumption (ref) in Section (ref).
Theorem (ref) establishes the consistency of the SVM slope estimator $\hat\beta$ for $\beta_0$ in the BCM. Corollary (ref) further shows that, under an additional symmetry condition, the intercept estimator $\hat\alpha$ is also consistent, together with $\hat\beta$.
Provided that the SVM slope estimator $\hat \beta$ is consistent up to a scalar $c_\ast >0$, we consider the estimation of the $\alpha_0$ up to the same scalar in this subsection.
Define a second-step estimator $\hat \alpha_{ms}$ by
where
and $\mathcal A$ is a compact and convex set on $\mathbb R$. Note that $\hat\alpha_{ms}$ is a maximum score estimator with a univariate generated regressor $\hat \beta'X_i$ using the SVM slope estimator $\hat\beta$. In particular, manski-75,manski-85 show that the maximum of \[ Q_{ms}(\alpha,\beta) = \mathbb E Y 1\{ \alpha+ \beta'X \geq 0 \} \] is given by $c\alpha_0, c\beta_0$ for any constant $c>0$. Therefore, if we fix $\beta$ at $\beta_\ast = c_\ast\beta_0$, then $Q_{ms}(\alpha,\beta_\ast)$ has a unique maximum $\alpha_\ast =c_\ast \alpha_0$ for $\alpha$.
Since $\hat \beta$ is consistent for $\beta_0$ up to a scalar $c_\ast >0$, it is expected that $\hat \alpha_{ms}$ has probability limit $c_\ast \alpha_0$. Consequently, $(\hat \alpha_{ms}, \hat \beta')'$ is consistent for $c_\ast \theta_0$. The following proposition establishes the consistency of $\hat \alpha_{ms}$.
To establish the convergence rate and limit distribution of $\hat\alpha_{ms}$, we introduce some assumptions. Assume $\beta_{m,0}>0$ without loss of generality, since we can use $-X_m$ in place of $X_m$.
Conditions in Assumption (ref) are also assumed in chen-lee-sung-14, which studies maximum score (MS henceforth) estimation when some regressors are conditional expectations and need to be estimated nonparametrically in the first stage. chen-lee-sung-14 show that the asymptotic distribution of the second stage MS estimator is the same as if the conditional expectations were used, provided that the conditional expectations are estimated with a convergence rate faster than $n^{1/3}$ in the first stage, along with some other regularity conditions.
If $\beta_\ast$ were known, $\alpha$ could be estimated by \[ \hat \alpha_\ast = \arg \min_{\alpha\in \mathcal A} \hat Q_{ms}(\alpha,\beta_\ast), \] whose cube root convergence rate and limit distribution have been established in kim-pollard-90. Now we give a result which states that the second stage MS estimator $\hat \alpha_{ms}$ using the SVM's slope has the same limit distribution as the infeasible estimator $\hat \alpha_\ast.$
Theorem (ref) uses $\|\hat \beta-\beta_\ast\| = O_p(n^{-1/2})$, which will be derived in the next section under suitable regularity conditions with other asymptotic properties of the SVM estimator.
In this section, we present the convergence rate and the asymptotic distribution of the SVM estimator. We first provide a technical lemma.
Lemma (ref) shows that the expected modulus of continuity of the process $Q_n-Q$ around $\theta_\ast$, over a neighborhood of radius $\delta$, is of order $n^{-1/2} \delta$. This result relies on a Lipschitz condition of the mapping $\theta \mapsto \ell_{\theta}(y,x) := (1-y(\alpha+x'\beta))_{+}$, in the sense that $\big| \ell_{\theta_1}(y,x) - \ell_{\theta_2}(y,x) \big| \leq \sqrt{1+\|x\|^2} \|\theta_1 -\theta_2\|$ for any $(y,x)\in \{-1,1\}\times \mathbb R^{1+m}$ and any $\theta_1,\theta_2 \in \mathbb R^{1+m}$.\footnote{ For a class of functions Lipschitz in the index parameter in Euclidean space, its bracketing number can be upper bounded by the diameter of the index parameter set to the power of the dimension. See, e.g., Example 19.7 in vandervaart-00, or Theorem 2.7.11 in vandervaart-wellner-96 for a more general statement. } Assumptions (ref) and (ref) guarantee that $Q(\theta)$ is twice continuously differentiable at $\theta_\ast$ with a positive definite Hessian matrix, as shown in Lemma (ref). These assumptions are also used to ensure that $\hat\theta \to_p \theta_\ast$ as in Lemma (ref). The only difference is that $\lambda_n = o(\sqrt{n})$ is assumed here, which is stronger than $\lambda_n = o(n)$ in Lemma (ref). Note that $\lambda_n = o(\sqrt{n})$ holds when using default choices in existing packages; see Footnote (ref) for details.
Now we are ready to establish the $\sqrt{n}$ convergence rate of the SVM estimator $\hat \theta$ to its probability limit $\theta_\ast$.
The limit distribution is the same as that in koo-lee-kim-park-08. However, one key assumption in koo-lee-kim-park-08 is that $\mathcal{L}(X|Y=\pm 1) $ have Lebesgue density functions which are continuous, whereas our results allow for discrete components of $X$.
In the same BCM context, chang-park-yan-25 show that the QMLE, which specifies that $U$ is independent of $X$ and has distribution function $G$ such that $\log G, \log (1-G)$ are strictly concave, is slope consistent under Assumptions (ref), (ref). However, the slope consistency of the QMLE does not require Assumption (ref).
This section contains four parts. First, we show that the SVM estimator is not a QMLE. Second, we interpret Assumption (ref), which ensures the slope consistency of the SVM estimator, as a condition ruling out severe class imbalance. We provide an illustration to show that Assumption (ref) may fail under some specifications when, roughly speaking, two classes are imbalanced in the sense that $Y$ takes on one value more frequently than another. Third, we show that the failure of the SVM under severe class imbalance can be effectively addressed by using class-weighted SVM. Indeed, the class-weighted SVM is shown to be slope consistent under Assumptions (ref) and (ref), without requiring the non-severe imbalance condition in Assumption (ref). Lastly, we provide simulation studies to show that, when both SVM and QMLE are slope consistent, their finite sample performance may differ across specifications, with one potentially outperforming the other.
In this section, we show that SVM's optimization problem is not equivalent to that of the quasi-maximum likelihood estimation (QMLE) for any specified distribution function. In this sense, the SVM cannot be viewed as a QMLE, even though both estimators are slope consistent under appropriate conditions.
The QMLE for BCM (ref), which assumes that $U$ is independent of $X$ and has distribution function $G$, maximizes \[ Q_{qml}(\theta) = \mathbb{E}\big[ 1\{Y=1\} \log G(\alpha+X'\beta) + 1\{Y=-1\} \log(1-G(\alpha+X'\beta)) \big] \] in the limit, whereas the SVM minimizes
For the SVM to be interpreted as the QMLE with a certain $G$, it is required that there exists some constant $\gamma >0$ such that
for all $z$. The first equality yields that $G(z) = e^{ - \gamma (1-z)_{+} }$ and the second yields $1-G(z) = e^{ - \gamma (1+z)_{+} }$. Unfortunately, there does not exist a constant $\gamma $ such that $e^{ - \gamma (1-z)_{+} } + e^{ - \gamma (1+z)_{+} } = 1$ for all $z$, which gives a contradiction.
The condition in Assumption (ref) plays a very important role in the consistency of the SVM estimator. In this section, we discuss this condition and show that it can be understood as a non-severe imbalance condition.
We introduce a notion. For two probability measures $P$ and $Q$ on $\mathbb R$, $Q$ is said to dominate $P$ for $\Pi$, written as $P\prec_\Pi Q$, if $\int \Pi(v)P(dv) < \int \Pi(v)Q(dv)$.\footnote{If $Q$ stochastically dominates $P$ in the first order and $\Pi(v) = \mathbb P\{U\leq V|V=v\}$ is strictly increasing in $v$, then $Q$ always dominates $P$ for $\Pi$ in our sense.} We also say that $P$ decreases for $\Pi$ if $\int \Pi(v)P(dv)$ decreases as $P$ changes.
In this lemma, we show that $\overline v$ decreases on $\mathbb R_+$ as the classes get more imbalanced, from which it follows immediately that Assumption (ref) becomes harder to be satisfied since $\tau$ is monotone increasing and $\tau(\overline v)$ decreases monotonically on $\mathbb R_+$. In fact, as we will demonstrate later, Assumption (ref) is violated if the class imbalance becomes severe. If the classes are balanced, we have $\overline v=\infty$, and Assumption (ref) always holds.
In what follows, we provide an example in which Assumptions (ref), (ref) are satisfied, so that any QMLE—which specifies that $U$ is independent of $X$ and has distribution function $G$ such that $\log G, \log (1-G)$ are strictly concave—is slope consistent. However, depending on parameter values, Assumption (ref) may fail under some specifications when, roughly speaking, two classes are severely imbalanced in the sense that $Y$ takes on one value much more frequently than another. Furthermore, we show by Monte Carlo simulations that the slope consistency of the SVM is significantly influenced by the validity of Assumption (ref).
\paragraph{Illustration: Imbalance and the SVM Slope Consistency} Let \[ V =_d \mathbb N(\mu,1), \quad \mbox{and} \quad U=_d \mathbb N(0,1) \] The underlying BCM can be that
where $X_1,X_2,U =_d \mathbb N(0,1)$ are independent, and $\alpha=\mu, \beta_1 = \beta_2 = 1/\sqrt{2}$. Regardless of the value of $\mu$, any QMLE—which specifies that $U$ is independent of $X$ and has distribution function $G$ such that $\log G, \log (1-G)$ are strictly concave—is slope consistent, since Assumptions (ref), (ref) are satisfied; see Theorem 3.2 in chang-park-yan-25. However, in what follows, we show that the SVM is consistent only when $\mu>0$ is not too large so that Assumption (ref) is satisfied.
We vary $\mu$ and compute numerically $\tau(\overline{v}), \sigma(-\infty)$ for each $\mu$. The upper graph in Figure (ref) plots $\tau(\overline{v}), \sigma(-\infty)$ for each $\mu\in [0,3]$. When $\mu$ is small, $\tau(\overline{v}) > \sigma(-\infty)$, i.e., Assumption (ref) is satisfied. However, when $\mu$ is greater than a certain value, Assumption (ref) fails, i.e., $\tau(\overline{v}) \leq \sigma(-\infty)$. The broken line plots this threshold value, which is around 1.453. The lower graph in Figure (ref) plots $\mathbb P\{Y=1\}$ to indicate how imbalanced the two classes are.
We further show by Monte Carlo simulations that, the slope consistency of the SVM depends on whether Assumption (ref) holds or not. More specifically, for each $\alpha\in \{0,1,1.5,2,3\}$, we generate $n\in\{1000,2000,5000\}$ observations according to the DGP given by (ref) with $Nsim=2000$ replications. Note that Assumption (ref) holds for $\alpha\in\{0,1\}$, whereas it fails when $\alpha\in\{1.5,2,3\}$.
Panel A in Table (ref) presents the bias and root mean square error of the rescaled slope estimator $\tilde \beta := \hat \beta_1/\hat \beta_2$ given by the SVM, along with $\mathbb P \{Y=1\}$ for each $\alpha$. Panel A in Table (ref) shows that, when $\alpha\in\{1.5,2,3\}$, the SVM slope is inconsistent since Assumption (ref) fails, whereas it is consistent when $\alpha\in\{0,1\}$ and Assumption (ref) holds.
In practice, applying the SVM is often coupled with the use of class weights, especially when there are more observations with $Y$ taking one value than another. More specifically, one may assign higher weights to the observations in the minority class and lower weights to those in the majority class, so that the number of observations in the two classes appears balanced. In this section, we show that the class-weighted SVM is slope consistent under Assumptions (ref) and (ref), without requiring the non-severe imbalance condition in Assumption (ref).
A class-weighted SVM corresponds to minimizing \[ Q_n^{w}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \left[ 1\{Y_i=1\} \big( 1-(\alpha+X_i'\beta) \big)_{+} + \hat w 1\{Y_i=-1\} \big( 1+(\alpha+X_i'\beta) \big)_{+} \right] + \frac{\lambda_n}{n}\|\beta\|^2 \] where $\hat w$ is a weight assigned to the observations in the class with $Y=-1$. Let the class-weighted SVM estimator $\hat \theta^{w} =(\hat \alpha^{w}, \hat \beta^{w})'$ be a minimum of $Q_n^{w}(\theta)$. Define \[ Q^{w}(\theta) = \mathbb E \left[ 1\{Y=1\} \big( 1-(\alpha+X'\beta) \big)_{+} + w 1\{Y=-1\} \big( 1+(\alpha+X'\beta) \big)_{+} \right]. \] where $w = \text{plim } \hat w$.
In what follows, we set $\hat w = \hat{\mathbb P}\{Y=1\} /\hat{\mathbb P}\{Y=-1\}$, the ratio of sample proportions of observations coupled with $Y=1$ and $Y=-1$, since this is the most commonly used choice in practice. Thus, \[ w = \frac{\mathbb{P}\{Y=1\} }{\mathbb{P}\{Y=-1\} }. \]
Theorem (ref) shows that the class-weighted SVM effectively addresses the inconsistency of the SVM that arises purely due to severe imbalance, i.e., when $\mathcal L(U|X) = \mathcal L(U|V)$ and $\mathbb E(X|V) = aV+ b$ are satisfied, but the non-severe imbalance condition in Assumption (ref) fails.
Panel B in Table (ref) demonstrates the consistency of the class-weighted SVM. In particular, when $\alpha\in \{1.5,2,3\}$ and Assumption (ref) is violated, the class-weighted SVM is still consistent, whereas the SVM is inconsistent as shown in Panel A.
In this section, we illustrate by Monte Carlo experiments that, when both the SVM and logistic regression (as a QMLE) are slope consistent, their finite-sample performance may differ across specifications, with one potentially outperforming the other.
For the data-generating processes, we assume that \[ Y = \mbox{sgn}( \alpha+ \beta_1 X_1 + \beta_2 X_2 - U) \] where $\beta_1 = 0, \beta_2 =1$, $\alpha=1/2$. We let $X_1, U =_d \mathbb{N}(0,1)$, and $X_1,X_2,U$ be independent of each other. The distribution of $X_2$ is specified based on the following normal mixture distribution standardized to have variance one,
for some $\mu$, where $\mbox{std}(\cdot)$ denotes standardization to mean zero and variance one. Note that Assumption (ref) is satisfied since $V=\alpha+X_2$ and $\mathbb E(X_1|V) = 0$. Assumption (ref) is verified numerically for $\mu\in\{0,1,2,3,4\}$, and thus, the SVM slope estimator is consistent.
For each $\mu\in\{0,1,2,3,4\}$, we generate $n\in\{250,500,2000\}$ observations with $Nsim=2000$ replications. In each replication, we obtain the estimates $(\hat \alpha, \hat \beta_1, \hat \beta_2)'$ of the parameter $(\alpha,\beta_1,\beta_2)$ given by the SVM and QMLE specifying the logistic distribution for the error term, referred to as logistic regression (LR). To compare the finite sample performance of estimators given by the SVM and LR, we rescale the estimators so that the last component of the rescaled estimator is set to 1. We divide each of the estimators by its last component, and, as before, focus on the rescaled slope estimator $\widetilde{\beta} = \hat \beta_1/\hat \beta_2$. Table (ref) presents the mean bias and RMSE of the rescaled slope estimator given by the SVM and LR. This table shows that, when $\mu\in\{0,1\}$, the LR slope estimator is more precise than that of the SVM, whereas the SVM performs better when $\mu\in\{2,3,4\}$.
In this paper, we show that the slope of the separating hyperplane given by the SVM consistently estimates the BCM slope parameter under the linear conditional mean condition for covariates given the systematic component used in the QMLE slope consistency literature. The SVM slope estimator is asymptotically equivalent to that of logistic regression in this sense. When binary outcomes are not severely imbalanced, this asymptotic equivalence holds as stated. Otherwise, the SVM should be used with class weights for this asymptotic equivalence to hold. We establish consistency of the SVM estimator for the slope parameter without assuming that a pseudo-true value exists. This makes our proof nonstandard and nontrivial. It is, however, essential because a pseudo-true value may not exist for the SVM estimator when there is severe imbalance in binary outcomes.
The asymptotic equivalence of the SVM estimator with the QMLE is quite surprising, since the SVM was originally developed as an algorithm that does not rely on any statistical or econometric model. Their asymptotics are indeed largely comparable, although the SVM estimator cannot be defined as a QMLE. The conditions required for slope consistency of the SVM estimator and QMLE are not overly restrictive, and can be made to hold by giving appropriate weights to the observations. For the BCM, once a consistent estimator for the slope parameter is available, we may readily obtain a consistent estimator for the intercept parameter as well. The asymptotic equivalence we established in the paper relates the SVM and logistic regression, the two most commonly used machine learning tools, more closely to each other. However, their finite-sample performances can be quite distinct depending on the distributions of covariates and errors. It seems that they will both survive, since neither dominates the other.