Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
51,257 characters · 20 sections · 30 citation commands
Adaptive Discrete Smoothing for High-Dimensional and Nonlinear Panel Data
Panel or longitudinal data are very important tool for empirical research in economics, social sciences, biostatistics and many other fields. Most of the literature on non- and semi-parametric panel models is based on the assumption that the regression function is the same across individuals. But this assumption, however, could be unrealistic in many applications, in particular when the number of observed individuals is large or there is unobserved heterogeneity across the individuals. In modern internet data with hundreds of thousands of users and a relatively short longitudinal time frame, it is important to study complicated and non-linear heterogeneity among different individuals.
A classical example is the prediction of demand and supply. More specifically, we will consider data from didichuxing.com (the Chinese counterpart of Uber in the U.S.) and estimate gaps between real time supply and demand in a metropolitan area in China being partitioned into 66 districts and the 24 hours day time being divided into 144 intervals, each containing 10 minutes. For companies like didichuxing.com it is crucial to develop a good model for estimating supply and demand gaps to provide better dispatching and services and to develop a dynamic pricing strategy.
Obviously, the model for predicting the demand and supply of taxi service at a given location and a specified time spot should not be uniform over all time-location combinations. Even at the same time spot, e.g., 8:00pm - 8:10pm, the supply, the demand, and the gap between them, as well as traffic conditions, vary dramatically across different locations in the city. Also, for the same location, the supply and demand are quite different in different time spots.
For modeling this kind of data sets, panel data models are widely used both in the literature but also in many empirical applications. However, due to significant heterogeneity across time and locations, the linear panel data model works poorly in terms of predicting the supply and demands, both in and out of sample when the number of observations per unit is relatively small. From a theoretical point of view, nonparametric panel data models are still a challenge as usually there are only a limited number of observations per individual available which makes nonparametric estimation of the individual regression functions quite imprecise or even impossible. Despite these challenges, one can often observe a certain grouping structure among models for different individuals in real data sets. For example, it is possible that district number one, at 8am, may share the same or a similar model for the gap between supply and demand, with district two at 6pm. One one hand, such a grouping structure among individual models can be used to improve the prediction for each model by borrowing observations from other similar models. On the other hand, these latent group structure among models is unknown and can hardly be figured out using common knowledge.
In this paper we propose a new adaptive discrete-smoothing (ADS) method which is designed to utilize the latent grouping structure among a large number of nonparametric panel data models. More precisely, the proposed ADS method simultaneously “clusters” the cross-sectional individuals and conducts model estimation.
In a non-parametric panel data model with the fixed effect setting, we assume that there are individual fixed effects $\alpha_i$ which determine the nonparametric regression function of each individual. As we explain later in the model specification part, these individual effects can be interpreted as the indices of a class of functions that determine the shape of the individual regression functions. To characterize the grouping structure among regression functions, we make an essential continuity/smoothness assumption: individuals with close values of $\alpha$s have similar regression functions. This assumption enables us to effectively address the curse of dimensionality (i.e., the insufficiency of samples for each individual model) in estimating nonparametric panel data models. The main idea behind the construction of our estimator is to use all the observations for estimating each individual model, including the observations from other individuals, but these observations receive a lower weight in estimation. More specifically, we develop a general two-stage estimator, where the first stage estimates the weights that characterize the “similarity” of the underlying regression functions. Given the estimated weight matrix between pairs of individuals, the second stage estimates the regression function for each individual using all the weighted observations.
Our idea shares a similar spirit as the “discrete smoothing” idea from LiRacine. In particular, LiRacine studied the nonparametric regression with discrete and continuous variables and used fixed weights for observations falling in other \textquotedblleft cells\textquotedblright, where each cell corresponds to a combination of categorical variables (\textquotedblleft discrete smoothing\textquotedblright). They showed that the mean squared error can be effectively reduced by using the weighted observations across cells. Instead of using the fixed weights, our adaptive smoothing approach over discrete cells depends on how similar the corresponding underlying regressions functions are. To accomplish this, we introduce data-driven weights that measure the similarity (distance) of regression functions across different individuals. Using observations from other individuals, on the one hand, introduces bias, but on the other hand, adding additional observations in the estimation process might reduce variance. By constructing the data-driven weights, our method leads to a better variance-bias trade-off and hence faster convergence rates in theory.
Generally speaking, the proposed methodology is applicable to almost any statistical procedure which allows weighted estimation, including kernel estimation, series / sieves, and modern machine learning methods like Lasso, Boosting and many others methods. As the weights employed for the smoothing of the regression functions are data-driven, we call our method \textquotedblleft adaptive discrete smoothing\textquotedblleft, or ADS in the rest of the paper. Our ADS method applies to both discrete and continuous indices $\alpha$ without having any prior knowledge of $\alpha$. As the rise of digitization in many fields leads to complex, high-dimensional data sets, we focus in the this paper on Lasso. In this paper, we focus on panel data models. The methods can also be used for non-parametric regression with both continuous and discrete variables, as considered in LiRacine. Adaptive discrete smoothing for non-parametric regression is considered in an accompanying paper (CLS2020). For the theoretical properties of our estimator first results are available for some estimation methods upon request.\footnote{An updated version with theoretical results is work in progress.}
Outline In Section 2 we give a heuristic motivation for our procedure. In Section 3 the ADS procedure is formally introduced in a general setting and specified for relevant problems. Section 4 presents a set of simulation studies to demonstrate the power of our methods compared to other alternatives. In Section 5, we present an empirical application: a study on didichuxing.com's panel data for predicting gap between supply and demand. Finally, we conclude.
Since the use of panel data is popular in many disciplines such as social, medical, financial studies, the panel data model has always been an active research area (see, e.g., Baltagi1992,Wooldbridge:10,Hsiao:14). Due to the large volume of work in this area, we only provide a brief survey of closely related works.
Our paper is related to different lines of recent research on panel data model. The first line is high-dimensional (or diverging dimensional) panel data model (see, e.g., Kock:13,kock:16:panel,BCHK:FE,zhu2017). In addition to parameter estimation and inference, Li:16:break and QianSu:16 further study the estimation of common structural breaks in linear panel data model in the diverging dimensional case.
Moreover, our work extends also non-parametric panel methods and recent approaches employing clustering for estimation of panel data models. Nonlinear panel data have been a field of active research. A comprehensive survey is given in AB:2011. Most of the literature on non- and semi-parametric panel models is based on the assumption that the regression function is the same across individuals; see Henderson, Mammen and Qian:03 among many others. As argued above, this assumption might not be realistic realistic in many applications. To relax the assumption, recent research has focused on assuming a group structure and employing cluster methods for panel data models. This means that every individual belongs to a group and the group assignment is unknown and has to be estimated. In such a setting, the number of groups usually needs to be finite and predefined. VL:2017 considered a nonparametric regression model that the observed individuals can be grouped into a number of classes whose members all share the same regression function. This is a special case of the model in Example (ref) with discrete indices (see Section (ref)). They develop a statistical procedure to estimate the unknown group structure and then estimate each regression function by averaging the individual functional estimates within each group. Bonhomme:2015 introduce time-varying grouped patterns of heterogeneity in linear panel data models. The parameters are estimated using a grouped fixed effects estimator that minimizes a least squares criterion with respect to all possible groupings of the cross-sectional units based on $K$-means clustering. SLM2016 develop two-step and iterative panel data estimators based on a discretization of unobserved heterogeneity employing clustering. SSP:2016 provide a novel mechanism for identifying and estimating latent group structures in panel data using penalized techniques. They consider both linear and nonlinear models where the regression coefficients are heterogeneous across groups but homogeneous within a group and the group membership is unknown. SSP:2016 proposes a novel penalty term called classifier-Lasso ($C$-Lasso). More specially, assuming that the true regression coefficients for $N$ individuals $\{\beta_1^*, \ldots, \beta_N^*\}$ only take $K_0$ distinct vector values, the $C$-Lasso imposes the non-convex penalty $\sum_{i=1}^N \prod_{k=1}^{K_0} \|\beta_i-\alpha_k\|_2$, where both $\{\beta_i\}_{i=1}^N$ and $\{\alpha_k\}_{k=1}^{K_0}$ are the decision variables in the optimization. This mixed additive-multiplication form of penalty tries to shrink individual regression coefficients $\beta_i$ to an unknown group-level coefficient vector $\alpha_k$.
Despite the popularity of investigating the grouping effect, there are several limitations in the existing approaches. First, the computation of the estimators is usually quite demanding. For example, some estimators rely on the $k$-means algorithm to learn the group structure. However, finding the optimal solution to the $k$-means clustering problem is known to be NP-hard even for two clusters Aloise:09. Other estimators either depend on the exhaustive search of the group structure or involve solving some non-convex optimization problems for identifying the group structure. Second, the estimators with an explicit clustering step usually require the number of clusters is known a priori. Approaches have been developed to determine the unknown number of groups but this is again computational demanding and might need additional assumptions. Third, it is very reasonable that in many applications there are not only group effects, but also individual effects. There might not be a hard group structure among different individuals. For example, some individuals might have similar coefficients but do not share exactly the same model. Finally, except for VL:2017, most existing research mainly focuses on linear or parametric panel data models and does not deal with general nonparametric regression functions.
To address these challenges, we propose a unified approach for a wide range of panel data models that include both parametric and non-parametric models, continuous and discrete variables, fixed-dimensional and high-dimensional settings. Instead of enforcing the “hard clustering” that assigns each individual to exactly one group, our approach can be interpreted as a \textquotedblleft soft clustering\textquotedblright with weights determined by the similarities between groups. This soft clustering approach is not only computationally attractive as compared to those “hard clustering” methods but also avoids fixing the number of groups prior to that.
The methodology we introduce can be applied directly for nonparametric panel models, e.g. employing kernel or Sieves methods. But in the rest of the paper we will focus on linear and high-dimensional panel data models. First, modern data sets are often high-dimensional and of particular interest. Second, non-parametric models can often be represented as approximate sparse models as defined in the remark above.
To illustrate the core of our idea, we consider estimation of (nonparametric) panel functions / longitudinal data. Figure (ref) shows for illustration individual specific functions for four individual. A naive approach might be to estimate a non-parametric function for each individual separately. But if the number of observations per individual $T$ is small, the regression functions cannot be estimated precisely. It seems that the regression functions for individuals 1 and 2 and for individuals 3 and 4 are similar. Hence, for estimation of the regression function of individual $1$ it is reasonable to use (\textquotedblleft borrow\textquotedblright) observations from individual $2$ for estimation, maybe with some lower weight. This might introduce bias, but decrease variance, leading to an increased MSE. Our idea is now to use data-dependent weights $\lambda$ and the weights reflect the similarity between the curves of all individuals. If the curves for individuals, here for example 1 and 2, are similar the weights should be close to one, as the information in the observations for individual 2 is valuable for estimation the function of individual 1. If the curves are very different, like the functions for 1 and 3, the weights should be small or zero. In the adaptive discrete smoothing procedure we propose the weights are based on the similarity between the two curves. The similarity is measured by a distance measure / metric $\rho$ based on initial estimates, $\rho(\hat f_i(\cdot), \hat f_j(\cdot))$ where $\hat f_k, k=1,\ldots,N$ denote the estimated individual regression functions. The weight $\lambda$ is then given by the expression $\delta \exp(- \gamma \rho(\hat f_i(\cdot), \hat f_j(\cdot))$ with $\delta$ and $\gamma$ denoting constants. The weights for observation belonging to the individual itself are set to $1$. The weights are hence determined in data-driven / data-dependent way. Based on initial estimates for each individual, a weighting matrix is determined, measuring the similarity of functions. Finally, the individual functions are estimated using the weights. This procedure can be applied to all estimation methods which allow for weighted estimation. As for the initial estimates few observations might be available, but many potential covariates, we focus in this paper on Lasso estimation.
To make the idea more formal, we first introduce further notation. We consider panel data, in particular we observe $N$ individuals and for each individual we observe $T$ periods. The dependent variable is denoted as $y_i=(y_{i1}, \ldots, y_{iT})'$, $i=1,\ldots,n$ and the predictors are denoted as $x_i=(x_{i1},\ldots,x_{iT})^T$. The covariates $x_{it}$ are $p-$dimensional vectors $(x_{it,1}, \ldots, x_{it,p})'$.
Given any $i$, we assume that $f_i(x_{it}):=f(x_{it},\alpha_i)$ and $y_{it}=f(x_{it},\alpha_i) + \varepsilon_{it}$ for error terms $\varepsilon_{it}$. The $\alpha_i$ stands for the individual specific effect in the non-parametric panel data model. In general, the $\alpha_i$, serving as an low-dimensional index of $f_i$, but can be considered as multiple dimensional as well. In the panel data setting, $\alpha_i$ is usually consider as a scalar.
One naive way of predicting $y_{it}$ is through building individual-level models:
$\hat f_i:=\mathop{\mathrm{arg\,min}{}}_{f\in \mathcal{F}} \sum_{t=1}^T (y_{it}-f(x_{it}))^2$, where $\mathcal{F}$ is a pre-specified functional space.
However, the naive estimator suffers from several drawbacks. First of all, in most of the real applications of panel data, we face the large $N$ and small $T$ problem, and estimating functions independently from other individual's observations lead to limited amount of data - only a sample size of $T$ for estimating each $f_i$.
In the traditional linear panel data model, $f_i(x_{it})=\alpha_i+x_{it}\beta+\varepsilon_{it}$, with $\mathbb{E}[\varepsilon_{it}|x_{it},\alpha_i]=0$. In such a model, the estimation process utilizes $N\times T$ samples at the same time because by assumption, $\beta$ is a common parameter across all individuals $i$. On contrast, the random coefficient model $f_i(x_{it}) = \alpha_i + x_{it}\beta_i+\varepsilon_{it}$ is a model where all $\beta_i$ are randomly distributed and can only be estimated by using individual longitudinal data without additional assumptions.
Suppose there exists group structures amongst all individuals in the panel data. One extreme case is that the number of such groups is only finite, which means that there are hidden labels that we do not observe, which we call them $\alpha_i$, $i=1,2,...,N$. When $\alpha_i=\alpha_j$, we say that the two individuals belong to the same group, and $f_i(x)=f(x,\alpha_i) =f(x,\alpha_j)=f_j(x)$. The linear panel data model is an special case because the coefficients $\beta$ are common across all individual $i$. A more general case is that we assume the distribution of $\alpha_i$ is continuous, and we allow that all individuals are different but might share some similarities. As $N$ becomes larger and larger, for any individual $i$, there must exist other individuals whose individual effects $\alpha$ are close enough to $\alpha_i$. Our idea is to assign proper weights to each individual $j\neq i$, and such weights should reflect the similarity between $\alpha_i$ and $\alpha_j$.
In this section we introduce a generic algorithm for adaptive discrete smoothing of panel data. We consider the model $y_{it}=f(x_{it},\alpha_i) + \varepsilon_{it}, i = 1,\ldots,N$ and $t=1,\ldots T$. $x_{it}$ are $p-$dimensional vectors of regressors. As mentioned in the previous section the algorithm consists of three steps: First, providing initial estimates $\hat f_i(x_{it}), i=1,\ldots,N$. Second, constructing of the weighting matrix $W$. Third, for all $i=1,\ldots,N$: weighted estimation of $\tilde{f}_i(x_{it})$. The algorithm is very flexible and can be combined with any estimation methods that allows weighted estimation, including ordinary least squares, kernel regression, series regression, maximum likelihood estimation and modern machine learning methods like Lasso, Boosting, Neural Networks. The generic algorithm can be described as follows:
In the following we will specify the generic algorithm for linear panel data models and estimate them with ols in the low-dimensional setting and with lasso in the high-dimensional setting.
To begin with, let us first consider the linear panel data model from Example 2 (ref) in a low-dimensional setting, i.e. where the dimensionality $p$ of each predictor $x_{it}$ is fixed and independent of the sample size. We illustrate our ADS estimator for this case based on the OLS estimation.
When the sample size $N$ and $T$ are large, the choice of $\delta$ in (ref) does not matter for the asymptotic performance of the ADS procedure. In practice, we could simply choose a $\delta\in (0,1)$. For example, in our study, the $\delta$ is set to be $0.5$. We further make a few important comments about the algorithm.
One interesting special case of the Algorithm (ref) is the high-dimensional linear regression case which is the core contribution of the paper. As high-dimensional data are more and more available for researchers, we would like to work out a special version of Algorithm (ref) for Lasso estimation of panel data in this subsection.
For high-dimensional linear regression, many estimators have been proposed in literature, such as Lasso Tibshirani:96, Dantzig selector CandesETao2007, square-root Lasso BCW-SqLasso (or scaled Lasso SunZhang:12), $L_2$ Boosting LS:2016, and post-Lasso BCPostLASSO. In principle, our ADS algorithm can be combined with any estimator for high-dimensional linear regression. For the ease of illustration, we will adopt the most widely used Lasso estimator and present the high-dimensional ADS algorithm next.
Consider the following high-dimensional model: for $1\leq i \leq N$ and $1 \leq t \leq T$,
where each $\varepsilon_{it}$ are i.i.d. errors draw from a normal distribution with mean zero and variance $\sigma^2$, $N(0, \sigma^2)$.
For each $\beta_i^* \in \mathbb{R}^p$ we allow $p \gg T$, and $p$ has to be of order $O(\exp(T^\alpha))$ for some $\alpha\in (0,1)$. Each $\beta_i^*$ depends on the unobserved variable $\alpha_i$, i.e. $\beta_i^*=\beta(\alpha_i)$ where $\beta: \mathbb{R} \rightarrow \mathbb{R}^p$ and $\alpha_i$ is a scalar. This can be interpreted in different ways as pointed out earlier.
For the high dimensional data, in Algorithm (ref), we can replace Lasso with other statistical or machine learning procedures that work well in high-dimensions, e.g., $L_2$ Boosting (LS:2016), Post-Lasso (BCPostLASSO). For brevity, we shall not repeat these procedures in this paper and restrict to the Lasso case.
In this section we present results from a simulation study, in particular we compare our methods to the \textquotedblleft frequentist\textquotedblright approach, i.e. estimating individual functions with only observations from this individual. We present results for OLS and Lasso regression.
We consider the following model
with $\varepsilon_{it} \sim N(0, \sigma^2)$ iid and $x_i=(x_{i1}, \ldots, x_{iT})'$ ($T \times (p+1)$ matrix) with $x_{it}$ a $(p+1)-$dimensional vector containing a constant of $1$. For $(x_{it2}, \ldots, x_{it(p+1)})$ we consider two settings:
We consider two data generating processes, one specifies the $(p+1)-$dimensional coefficient vector $\beta_i$ as a function of an random, unobserved quantity $\alpha_i$ and one assumes that the $\beta_i$ are drawn from a normal distribution with constant correlation $\rho$ between the same component of different observations $i$ but independent components.
The individual coefficients $\beta_i$ ($p+1$-dimensional) are simulated in such a way -- based on the normal distribution -- that the components are uncorrelated for individual $i$, but the (same) components are correlated between individuals $i$ and $j$ with correlation $\rho$. Namely, $cor(\beta_{ik}, \beta_{il})=0$ for $k \neq l$ and $cor(\beta_{ik}, \beta_{jk})=\rho$ for $i\neq j$. So, $(\beta_{1l}, \ldots, \beta_{Nl}) \sim \mathcal{N}(0,\Sigma)$. The $N-$dimensional correlation matrix has entries $\rho$ off the diagonal. The vectors are iid for different components $l$ $(l=1,\ldots,p+1)$.
This DGP is similar to 1, but $\beta_i$ is generated differently, namely: \[ \beta_i'=(1,\ldots,1)' + (\alpha_i, \alpha_i^2, -\alpha_i, - \alpha_i^2,\alpha_i, \alpha_i/2, \alpha_i/3, \ldots). \] The $\alpha_i$ are drawn iid from a uniform distribution, $\alpha_i \sim \mathcal{U}(0,1)$.
Here we estimate a model comparable to (ref), but we use Lasso and Lasso with discrete smoothing for estimation. Lasso usually assumes a sparse setting, i.e. it is assumed that there is a set of $p+1$ potential variables, but only a small subset of $s+1$ regressors has non-zero coefficients. As data generating processes we use variants of DGP 1 and DGP 2 which are described above: the first $s+1$ components are simulated in exactly the same way as in DGP1 and DGP2 and the remaining $p-s$ variables are set equal to zero. In the simulations we vary $n,p,s$ and $T$. We decompose the vector $\beta_i=(\beta_i^1, \beta_i^2)$ in two parts: $\beta_i^1=(\beta_{i1}, \ldots, \beta_{i(s+1)})$ and $\beta_i^2=(0,\ldots,0).$ Hence we can formulate the DGPs in the following way
In this section we present the simulation results. We generate the data according to the DGPs described above and estimate it with OLS (DGP 1 and 2) or Lasso (DGP 3 and 4) and the smoothing version introduced in this paper. We vary different parameters. For estimation we use data set of size $n,T$. The forecasts are evaluated out-of-sample (same sample size as used for estimation) according to the mean squared error (MSE). We set the number of repetitions to $R=500$. We use a setting where the design matrix $X$ is iid (“X iid”) and setting where $X$ is correlated (“X corr”), i.e. has Toeplitz structure. In the linear case we set the number of parameters $N,T,p$. In the Lasso case we use a exact sparsity design, where $p$ denotes the number of parameters and $s$ the number of non-zero coefficients.
The results can be summarized as follows: when $T$ is small and $N$ is large our methods performs very favorable. When $N$ is small and $T$ is large, discrete smoothing does not add much benefit. This is exactly what one would expect. Overall, the simulation results highlight the usefulness of our approach for empirical applications.
\textcolor{white}{.}
Didi is a major ride-sharing company in China with more than $400$ million users across $400$ cities. As the services are based on smartphone applications the company can collect a huge amount of data on a daily basis concerning detailed information on asked and provided rides including detailed geographical information and time stamps. We use the raw data on single rides to estimate the gap in demand which is defined as the difference of requested rides, i.e. the number of calls in a certain location, within a period of 10 minutes and the number of answers in this location, within this period. The gap is an important variable for the company as good predictions of this variable help to hold available drivers and it serves as input for dynamic pricing strategies. The data set contains the following variables:
We build a model to predict the gaps on location level, i.e. for each district we estimate a model for gap. It seems plausible that the demand depends on the location (e.g. city center vs. outskirt). This leads to $66$ individual location units in the city we consider for our analysis. The total sample size of observations for all units is $n=135,674$. We split the data set in a training sample ($n=105,000$) and a testing sample ($n=30,674$) under consideration of the location structure. So to estimate each model appr. 1,500 observations are available.\footnote{For each unit the last days of the period of record are removed.} We model variables like day of month and day of week as categorical leading to a large set of potential covariates for each unit, namely $46$ which is challenging. This situation which is common for many real word applications, in particular data sets collected on the internet fits well for the methods we proposed as additional information from other observations is very valuable in this setting.
We estimate for each unit linear regression models with OLS and Lasso and compare the results with the adaptive discrete smoothing method for each estimator introduced in this paper. Although the OLS estimator is not well defined for units with $p>n$, it can be used for prediction.
We apply the ADS procedure to didichuxing.com's data. This data involves 66 districts. In our study, we found that our ADS procedure outperforms all other methods, measured by both in-sample and out-of-sample MSE. Already in the simulation study we found clear evidence that the ADS procedure does better in MSE compared to traditional methods, in many different settings.
In this paper, we propose a novel adaptive discrete smoothing procedure (ADS). Such a procedure is applicable to both non-linear and high-dimensional panel data.\footnote{The core idea of the procedure can also applied to many other statistical problems. An application to non-parametric estimation with discrete and continuuos regressors is given in CLS2020.} The procedure is especially useful in the large cross-sectional size and small longitude scenario, where building individual models based on individual data is highly imprecise, while heterogeneity across cross-sectional individuals prevents good performance of a uniform modeling process for all individuals. The results of our simulation strongly support our procedure compared to other existing econometrics or statistics methodologies. Our procedure can also be interpreted as a clustering method. In future research we would like to derive the theoretical properties (work in progress) and explore more formally the connection to cluster methods.