Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
48,698 characters · 14 sections · 61 citation commands
\pagenumbering{gobble}
{\textcolor{MidnightBlue}{Abstract}}
For highly skewed or fat-tailed distributions, mean or median-based methods often fail to capture the central tendencies in the data. Despite being a viable alternative, estimating the conditional mode given certain covariates (or mode regression) presents significant challenges. Nonparametric approaches suffer from the "curse of dimensionality", while semiparametric strategies often lead to non-convex optimization problems. In order to avoid these issues, we propose a novel mode regression estimator that relies on an intermediate step of inverting the conditional quantile density. In contrast to existing approaches, we employ a convolution-type smoothed variant of the quantile regression. Our estimator converges uniformly over the design points of the covariates and, unlike previous quantile-based mode regressions, is uniform with respect to the smoothing bandwidth. Additionally, the Convolution Mode Regression is dimension-free, carries no issues regarding optimization and preliminary simulations suggest the estimator is normally distributed in finite samples.
\textcolor{MidnightBlue}{Keywords:} Mode Regression; Convolution-based Smoothing; Conditional Quantile; Asymptotic Theory; Uniform Convergence.
\pagenumbering{arabic} \setcounter{page}{3}
Conventional econometric methods are generally mean-(or median)-based; such methods may fail to express the central tendency if distributions are highly skewed or long-tailed Kemp_Silva_2012, Chen_et_al_2016. The conditional mode emerges as a robust alternative, conveying the desirable interpretation of being the most likely value of a dataset chacon_2020. When data is discrete, the conditional sample mode is more straightforward, however, in a continuous variable setting, it can be estimated as the point that maximizes the conditional probability density function (pdf). Since Lee_1989, the estimation of conditional mode, called mode regression, has demonstrated its utility across various domains, specially in applications with asymmetric data, such as wages \citep*{Zhang_Kato_Ruppert_2021}; electrical energy consumption \citep*{ota_kato_hara_2019}; medical sciences wang_et_al_2017, traffic data einbeck_tutz_2006 and wildfire burned areas yao_li_2014.
In reviewing the conditional mode literature, two fundamental considerations emerge in the discussion: firstly, the assumption of whether the mode is global (unique) or local; secondly, if the estimation strategy employed is semiparametric (linear) or fully nonparametric. Linear approaches require the condition that the mode is a unique maximizer of the conditional density, while nonparametric techniques in turn are usually used for multimodal models.\footnote{Nonparametric estimation is not exclusive to multimodal specifications, since in few cases the global mode is also estimated nonparametrically sager_thisted_1982, feng_et_al_2020.} The first semiparametric estimator considering a unique mode was developed by Lee_1989 and establishes a linear relationship between the mode of the response and its covariates; however, despite being elegant, this model is impractical;\footnote{Has restrictive assumptions on the conditional density of the response and, due to the objective function, the estimator lacks a tractable distribution Kemp_Silva_2012.} subsequent models, such as the ones of Kemp_Silva_2012 and yao_li_2014, yield out non-convex optimization problems, resulting in functions that may have multiple maxima; also, the related algorithms are sensible to the starting points. On the other hand, although nonparametric estimation tends to avoid misspecification yao_et_al_2012, Chen_et_al_2016, these methods suffer from slow convergence even in moderate dimension sizes.
Two recent developments \citep*{ota_kato_hara_2019, Zhang_Kato_Ruppert_2021} have addressed the optimization and dimension issues by estimating the mode via a quantile regression (QR). This strategy relies on retrieving the conditional density (whose maximizer is the mode) from the derivative of the conditional quantile function, referred to as the conditional quantile density function (qdf). Similarly to what is done in \citet*{ota_kato_hara_2019} and \citet*{Zhang_Kato_Ruppert_2021}, we propose a novel approach which relies on working with the (convolution-based) smoothed version of the QR estimator of \citet*{fernandes_guerre_horta_2021}. This alternative provides some advantages: it is continuous, unbiased and less variable than the traditional QR estimator.
The main goal of this work is to derive the asymptotic properties for the estimator of the conditional mode via smoothed quantile regression, hereby denominated Convolution Mode Regression. Our procedure differs from \citet*{Zhang_Kato_Ruppert_2021}, as they opt for an "estimate then smooth" process, using the traditional QR framework;\footnote{Parallel to parzen_1979's parzen_1979 proposition of quantile estimation by smoothing the sample quantile function.} whereas, we prefer to "smooth then estimate", stemming from the smoothed quantile regression estimator.\footnote{Resembling the method of Nadaraya_1964, in which the unconditional quantile is estimated via inverting a smoothed estimator of the cdf (cumulative distribution function).} The convergence rate of the \citet*{Zhang_Kato_Ruppert_2021} estimator is $O_P( n^{-1/2}h^{-3/2} \sqrt{\log n} + h^2)$, whilst ours is $O_P( (\frac{\log n} {nh})^{1/4}) + o(h^{1/2})$; both rates dimension-free. Despite \citeauthor*{Zhang_Kato_Ruppert_2021}'s rate, their convergence is uniform solely in the design points of the covariates. In contrast to our estimator, which holds uniformity with respect to the design points, but also for the smoothing bandwidth. In light of this, our estimator can be more adequate in cases where the choice of the bandwidth is data-driven.
Following the introduction of global mode estimators by parzen_1962 and chernoff_1964,\footnote{sager_1978 categorized estimators as: "naive" when the density was not estimated (Chernoff), and as "indirect", since an intermediate step, such as Kernel Density Estimation, is required (Parzen). These classifications may blur, as the majority of direct estimators reveal some form of linkage with a type of density estimation sager_1978, chacon_2020.} sager_thisted_1982 generalized the framework of the latter and developed the first mode regression, establishing the global mode of the dependent variable as a monotone function of the covariate. Albeit its application only to ordinal data, this model showed that the conditional mode estimator could be formulated by applying a plug-in from a density estimator, such as a Kernel Density Estimator (KDE). Subsequently, mode regression literature has evolved and can be labeled according to two major factors: firstly, the assumptions of the model, as in a unique global mode vs. multiple local modes; secondly, the type of estimation, semiparametric (which postulates a linear relationship between response variable and covariates) vs. nonparametric (which allows for multiple local modes). Semiparametric estimation only works with the assumption of the mode being unique, however, a unique mode can be estimated nonparametrically. In light of this, we can divide the mode regression literature into 4 different strands:\footnote{Some papers may not fit precisely in this categorization, since they mix parametric and nonparametric traits liu_etal_2013, wang_2024, or use a Bayesian approach yu_arist_2012.} (1) Unique Mode with Semiparametric Estimation; (2) Unique Mode with Nonparametric Estimation, and (3) Multimodal with Nonparametric Estimation; afterward, special attention is given to (4) Conditional Quantile Approaches towards the mode, since this is more related to our contribution.\footnote{For a general survey of the role of the mode in statistics, we recommend chacon_2020; especially for mode regression, we indicate the review of chen_2018.}
(1) Unique Mode & Semiparametric Estimation: considering a linear framework, Lee_1989 (Lee_1989, Lee_1993) proposed to use a kernel smoothed loss function to estimate the conditional mode of truncated variables. The main drawback of this line of action lies in the underlying assumptions on the conditional density of the dependent variable, as well as homogeneity and symmetry of the error terms, which led to the conditional mode coinciding with the conditional mean chen_2018. Further work that sought to eliminate the error's assumptions, such as: Kemp_Silva_2012, where the loss function allows for smooth kernels; as well as yao_li_2014, who focused on high-dimensional data. Both methods have algorithmic issues, leading to a nonconvex optimization problems, with no guarantee of convergence to the global maximum, and high sensitivity to the selected starting point.\footnote{Some further exploration of this literature strand can be found in variable selection zhang_2013, time series analysis kemp_etal_2020 and in panel data ullah_etal_2021.}
(2) Unique Mode & Nonparametric Estimation: linearity can be too restrictive depending on the type of data; thus, nonparametric regression can model the components of the conditional mode as smooth functions of the covariates Chen_et_al_2016. A generalization of local polynomial regression is provided by yao_et_al_2012, denoted local modal regression, which presents better efficiency when errors are heavy-tailed. Nonetheless, the model suffers from the “curse of dimensionality” and error's symmetry is imposed.\footnote{According to \citet*{Zhang_Kato_Ruppert_2021}, their sixth assumption (symmetry of the error term) leads to the problem corresponding to conditional mean estimation.} A more recent nonparametric regression for single mode that mitigates the “curse of dimensionality” is found in feng_et_al_2020. This estimation of the conditional mode is achieved via an empirical risk minimization approach, which turns the problem into non-dependable on dimension.
(3) Multimodal & Nonparametric Estimation: as it is not always the case that data structures can be interpreted as unimodal, scott_1992 proposed to consider various modes as points of local maxima of the conditional density of the response. Both matzner_etal_1998 and einbeck_tutz_2006 solved prediction problems using the conditional mode. Using less restrictive assumption on the kernel density function, Chen_et_al_2016 developed a KDE-based mode regression with strong asymptotic properties and no specification issues. This model is more general than ours, since it allows for multiple modes, while we assume a unique mode at each design point of interest. Despite this, the model is susceptible to the "curse of dimensionality", given that the convergence is slow even with a modest number of regressors \citep*{Zhang_Kato_Ruppert_2021}.
(4) Conditional Quantile Approach: motivated by the challenges of nonconvex optimization and misspecification in linear models, as well as the susceptibility of nonparametric estimators to the "curse of dimensionality", \citet*{ota_kato_hara_2019} developed a novel semiparametric approach with an intermediate step based on quantile regression. The traditional quantile function of Koenker_Bassett_1978 is estimated, following, a conditional quantile density estimator is obtained via numerical differentiation. The underlying model does not impose linearity in the mode function, even when the quantile regression model is linear-in-covariates. Furthermore, it is computationally appealing, since quantile regression can be written as a linear programming problem. Still, the Koenker_Bassett_1978's (Koenker_Bassett_1978) empirical conditional quantile function is not continuous---hence the mentioned numerical differentiation, which leads to an approximate optimizer for the mode. In order to surpass this problem, \citet*{Zhang_Kato_Ruppert_2021} propose to post-smooth the quantile regression estimator by a kernel function. This strategy not only circumvents the approximate optimization, but also yields faster convergence rates. Both models take off from a key identity that we also explore in this paper, namely, that the quantile density is the reciprocal of the density function, evaluated at the quantile of interest, which summarizes how the conditional mode can be retrieved from the quantile density.
The rest of this paper is organized as follows. In Section (ref) the model is presented, we introduce the estimator and explore its relationship with the smoothed quantile regression. In Section (ref) we enunciate the main mathematical results of the paper, as well as commentary regarding convergence rates and different bandwidth scenarios. Section (ref) is a preliminary Monte Carlo study in which we simulate the possible limiting distribution for the Convolution Mode Regression Estimator. Finally, in Section (ref) we state our concluding remarks along with possibilities for future work. The Appendix contains additional mathematical material, such as derivations and convergence rates calculations.
Let $Y \in \mathbb{R}$ represent a target random variable for which we are interested in estimating the conditional mode, given a $d$-dimensional vector $X$ of covariates, and write $\mathcal{X}:=\mathrm{support}(X)$. Assume that $Y|X=x$ is continuous and unimodal, having conditional cdf $F(\cdot|x)$ and conditional pdf $f(\cdot|x)$. Then, the conditional mode of $Y$ given $X=x$, denoted by $m(x)$, is defined as:
Thus, $m(x)$ corresponds to the point in the covariate space at which the (conditional) density of the response attains its maximum value. Additionally, define the $\tau$-th conditional quantile of $Y$ given $X=x$ as the scalar $Q(\tau|x)$ given by
and the conditional quantile function as the mapping $\tau \mapsto Q(\tau|x)$.
It is important to point out that the quantile function is entirely retrievable from the cdf, since it is just the generalized inverse of the function $y\mapsto F(y|x)$, and in the case of the distribution being a continuous function we have that $Q(\cdot|x):=F^{-1}(\cdot|x)$ for each $x$ vdVaart_1998, koenker_2005. Furthermore, the conditional quantile density function (qdf) is defined through
A key identity explored by \citet*{ota_kato_hara_2019} and \citet*{Zhang_Kato_Ruppert_2021} is that, as a consequence of the Inverse Function Theorem, the identity
holds for every allowable $x$ and $\tau$. In this sense, given some regularity conditions which we introduce below, we can minimize the inverse of the density (which is the qdf), as in equation (ref), and retrieve the maximizer of $y \mapsto f(y|x)$ from equation (ref); thus,
Regarding the quantile function, both \citet*{ota_kato_hara_2019} and \citet*{Zhang_Kato_Ruppert_2021} consider the quantile regression model developed by Koenker_Bassett_1978, which stipulates a linear-in-covariates representation of the conditional quantile function:
where $\beta : (0,1) \mapsto \mathbb{R}^d$ is a functional parameter. For each fixed $\tau$ in the interval $(0,1)$, the vector $\beta(\tau)$ in (ref) solves a similar minimization problem as the one found in classic linear regression. For this end, the following population objective function is proposed:
with $\rho_\tau (u) := u[\tau - \mathbb{I}(u<0)]$ known as the check function. The true parameter $\beta(\tau)$ minimizes $R(b;t)$ with respect to $b\in\mathbb{R}^d$. The sample equivalent proposed by Koenker_Bassett_1978 is defined as:
where $\widehat{F}(\cdot;b)$ is the empirical distribution function of $\varepsilon_i(b):=Y_i - X_i^\intercal b$, for $i=1,...,n$, with the traditional quantile regression estimator as the minimizer of $\widehat{R}(b;\tau)$, with respect to $b\in\mathbb{R}^d$, that is:
According to Theorem 2.1 in Bassett_Koenker_1982, the empirical conditional quantile function $\tau\mapsto x^\intercal \widehat{\beta}(\tau)$ exhibits jumps, in particular it is not differentiable. To overcome this issue, \citet*{fernandes_guerre_horta_2021} proposed using a kernel-type cdf estimator, similar to Nadaraya_1964, instead of the empirical distribution function. The resulting smoothed version of the sample objective function in (ref) is:
where the symbol $\ast$ denotes the convolution operator, and where $\widehat{f}_h(\cdot;b)$ is the kernel estimator of the density of $Y_i - X_i^\intercal b$. Here, $k_h(u) = k(u/h)/h$, where $k\colon \mathbb{R}\to\mathbb{R}_+$ is a smooth kernel function and $h>0$. The new estimator is the minimizer of the objective function (ref), called the smoothed quantile regression estimator (SQRE) and defined by:
The mapping $ \tau \mapsto \widehat{\beta}_h(\tau)$ is continuously differentiable over the interval $(0, 1)$, unlike $\widehat{\beta}$. Differentiability offers notable advantages, and the reasons are twofold: (i) the smoothness of the objective function ensures the regularity of the resulting estimator; (ii) the asymptotic covariance matrix of $\widehat{\beta}_h(\tau)$ can be estimated in a standard fashion, as in Newey_McFadden_1994. Regarding differentiability, writing $\widehat{R}^{(1)}_h(b;\tau):= {\partial \widehat{R}_h(b;\tau)}/{\partial b}$, the SQRE satisfies the first-order condition $\widehat{R}^{(1)}_h(\widehat{\beta}_h(\tau);\tau) = 0$. Accordingly, following the Implicit Function Theorem, we obtain:
Explicit formulas for the first and second order derivatives of $\widehat{R}_h(b;\tau)$ with respect to $b$ (respectively, $\widehat{R}^{(1)}_h(b;\tau)$ and $\widehat{R}^{(2)}_h(b;\tau)$) are provided in equation (ref) in (ref).
Consider the following objective (or "sparsity"\footnote{This is the nomenclature used by \citet*{ota_kato_hara_2019} and \citet*{Zhang_Kato_Ruppert_2021}.}) function:
It is not difficult to show that
where
Under some regularity assumptions that will be introduced below, the function $\tau\mapsto s_x(\tau)$ has a unique maximizer, denoted $\tau_x$, which we call the conditional quantile mode of $Y$ given $X=x$. If we plug in this optimizer in the quantile function $Q(\cdot|x)$ we get the expression $m(x) = Q(\tau_x|x)$. Consequently, the estimation of $m(x)$ boils down to estimating the conditional quantile function, and $\tau_x$. In view of (ref), we define, for conformable $\tau$, $x$ and $h$, the sample conditional sparsity function as:
where
see \citet*{fernandes_guerre_horta_2021}.
The optimizers for the sparsity functions, both population and sample, as defined in (ref) and (ref), are given by:
where $0<\alpha<1/2$ is a constant.
Our proposed smoothed conditional mode estimator is then given by
for all $x\in\mathcal{X}$ and every allowable $h$.
Before providing consistency results of the proposed estimator, we state the conditions for which our results are derived.
Assumptions A1-A5 are taken directly from \citet*{fernandes_guerre_horta_2021}, with minor modifications. Due to A1-A3, the Hessian $D(\tau)$ as defined in (ref), is positive definite for all possible values of $\tau \in (0,1)$, therefore, $D(\tau)$ is invertible. Additionally, A2 ensures the function $\tau\mapsto Q(\tau|x)$ is increasing over the interval $(0,1)$, and, together with A3, that its derivative with respect to $\tau$ is strictly positive. Also, A3 expresses some ordinary regularity conditions which guarantee smoothness of $f(\cdot | \cdot)$ koenker_2005. Similar conditions can be found in Chen_et_al_2016; \citet*{ota_kato_hara_2019, Zhang_Kato_Ruppert_2021}; however, each of these estimates requires four-times continuous differentiability of the density. Assumptions A4 and A5 concern the kernel function $k$ and the bandwidth parameter $h$ and are necessary for the previously stated benefits of the SQRE over the traditional QR estimator. A6 ensures uniqueness of the conditional mode and is also commonly used in deriving consistency of M-estimators, see Theorem 5.7 in vdVaart_1998. Finally, Assumption A7 limits the possible values for the optimizer $\tau_x$, ensuring that the conditional modes are bounded away from the tails of the conditional distributions, uniformly on the covariate space.
The following lemma is a reinstatement of an inequality in \citet*{fernandes_guerre_horta_2021}.
Our next result is regarding the sample sparsity function and the fact that it converges to the population counterpart.
After showing that our sparsity functions are consistent, we prove that its maximizer $\widehat{\tau}_{x,h}$, in equation (ref), is also consistent for $\tau_x$.
Now that the consistency for the quantile modes is proved and the rates of convergence are defined, we proceed to state the consistency for the estimator of the mode, $\widehat{m}_h(x)$. Our second theorem is constructed using previous results from this paper and from \citet*{fernandes_guerre_horta_2021}.
In this section we employ a preliminary\footnote{For a more comprehensive Monte Carlo Study, considering different types of error terms, we refer the reader to the work of ongaratto_horta_2021.} version of a Monte Carlo study in order to illustrate if our estimator converges to a normal distribution in finite samples. All the analyses in this section were developed using R language, version 4.3.1 R_software via the software RStudio R_studio. The estimation of $\widehat{\beta}_h(\tau)$, as in (ref), was done via the package conquer from conquer.
In our simulation, we consider a type of heteroscedastic error distribution, similar to \citet*{fernandes_guerre_horta_2021}. Our pseudorandom data is generated from the model $Y = X^\intercal \beta + \epsilon$, where $X = (1, \Tilde{X})^\intercal$, with $\Tilde{X} \sim U(1,5)$, $\beta = (1,1)$ and the conditional heteroscedastic error term $\epsilon = \frac{1}{2} (1+\Tilde{X})Z$, with $Z$ following a Skew Normal distribution with zero mean, unit variance and parameter $\alpha=2$. As for the design point of the covariates, we condition $X=x$ with $x=(1, 3)^\intercal$. The simulations require $\tau$ values, so we consider a grid, $\mathcal{T} = \{ 0.01, 0.02, ..., 0.98, 0.99\}$. We contemplated 3 different samples sizes, $n = \{ 100, 250, 500\}$, each for the number of replications, $it=\{100, 1000\}$.
It is necessary to choose the bandwidth parameter $h$ in order to estimate $\widehat{m}_h(x)$ as in (ref). In line with \citet*{fernandes_guerre_horta_2021}, we employ the Silverman's rule-of-thumb bandwidth silverman_1986. We consider a $\tau$-dependent, plug-in rule-of-thumb bandwidth which can be expressed as:
where, for each quantile level $\tau \in \mathcal{T}$, there is a different value of $\widehat{S}(\tau)$, which is computed according to the following algorithm:
Since the bandwidth $h_{ROT}$ is data-driven, it depends on the generated sample, thus it can vary with each replication. Also, the bandwidth is $\tau$-dependent, therefore, there is a different smoothing parameter for each value of $\tau \in \mathcal{T}$.
Since no distribution is achieved in our asymptotic theory, we rely on simulations in finite sample sizes to analyze the behavior of the Convolution Mode Regression Estimator, specifically, its limiting distribution. In order to deduce if our proposed estimator is normally distributed, we analyze the quantile-quantile (Q-Q) plots for the estimated $\widehat{m}(x)$ against the theoretical quantiles of a Normal Distribution.
According to Figures (ref)-(ref) there is some evidence that our estimator is normally distributed, for the referred sample sizes. The generated sample points show some adherence to the 45° degree line, which represents the theoretical quantiles of a Normal Distribution. The “stickiness” of the points to the line seems to be weaker on the lower tail for all sample sizes with 100 replications. When we increase the number of replications of the simulation for $it=1000$ (Figures (ref) and (ref)), we get similar results to the previous ones, the generated quantile points of $\widehat{m}_{h_{ROT}}(x)$ overlap the red line, except in the lower limits of the samples. The last simulated example, in Figure (ref) ($n=500$ and $it=1000$), shows a better correspondence of the simulated points of the lower tail to the theoretical pathway. In short, the different simulated scenarios suggest that for finite samples, our estimator gets closer to being normally distributed.
In the present paper we developed a novel estimator for the conditional mode $\widehat{m}_h(x)$, called Convolution Mode Regression, which is based on inverting a convolution-type smoothed quantile regression. The idea of achieving the conditional mode via quantile regression is not groundbreaking, since it has been done previously \citep*{ota_kato_hara_2019, Zhang_Kato_Ruppert_2021}. Despite that, it presents advantages regarding the two main problems with mode regression, slow convergence in nonparametric settings and nonconvex optimization in linear environments. Since the mode is the maximizer of the density and the density can be retrieved from the qdf, our estimation strategy relies on an intermediate step. Specifically, to estimate $\widehat{m}_h(x)$, we first estimate the conditional quantile function $Q(\cdot|x)$, followed by the conditional quantile mode of $Y$ given $X=x$, denoted $\tau_x$. The qdf estimator we rely on, from \citet*{fernandes_guerre_horta_2021}, makes use of the linear structure of QR and also converges to the real pdf curve.
Differently from the existing work of \citet*{Zhang_Kato_Ruppert_2021}, who initially estimate the quantile regression then smooth it through a kernel, our approach relies on "smooth then estimate". We develop asymptotic consistency for our estimator, obtaining some differences in convergence rates, as compared to \citet*{Zhang_Kato_Ruppert_2021}. Apart from the initial smoothing, the main differentiation of our model from the authors' is in the bandwidth selection premise, since our assumption for the choice of $h$ is less restrictive. Furthermore, the uniformity of $\widehat{m}_h(x)$ with respect to $h$, something not obtained in \citet*{Zhang_Kato_Ruppert_2021}, makes our model an interesting choice when the bandwidth selection is data-driven or adaptive. Our preliminary simulations show, that for finite sample sizes, our estimator behaves similar to a Normal Distribution.
Further work related to present research can take many directions. In what we assess as more important, the continuation of the asymptotic properties, namely, the limiting distribution of the estimator. Furthermore, to refine the Monte Carlo Study, more in line with ongaratto_horta_2021, in order to compare the performance of the Convolution Mode Regression Estimator against \citeauthor*{Zhang_Kato_Ruppert_2021}'s. In accordance to that, implementations of our estimator in previous applied econometric and predictive problems can be carried out; besides to a generalization of the present framework for time series, focused on forecasting models for asymmetric data.