Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
113,729 characters · 29 sections · 67 citation commands
Cross Validation Based Model Selection via Generalized Method of Moments
Structural estimation of economic models is one of the most widely used methodologies in empirical economics nowadays in variety of fields. Structural estimation enables researchers to interpret latent variable, as well as it allows researchers to perform a counterfactual simulation. Arguably, however, one of the largest shortcoming in the structural estimation procedure lies in the selection of a proper model. That is, the specification of estimation models is usually chosen by researchers and rarely empirically tested. The structural estimation itself does not directly address on it from the data, because the estimation is performed by assuming the model reflects the true data generating process (angrist2010). On a paper it is a common practice for economists to verbally argue and defend their model specification in a descriptive way. However, since the validity of the counterfactual simulation crucially depends on the goodness of the model, verifying and choosing a proper model empirically is of particular importance. Especially, we often simplify a model for the ease of tractability: Such simplifications is preferred to be subject to some assessment.
When a structural model is estimated in economics, researchers often use generalized method of moments (GMM) as well as maximum likelihood. As to selecting a true model, smith1992 and riversvuong2002 offer a model selection procedure for GMM based on the difference of empirical moments. Their core idea is a simple use of the GMM minimand as a fitness of the model with the observed data: That is, to select the model of the smallest GMM minimand when it is estimated\footnote{The theory provided in riversvuong2002 applies to broader range of model selection criteria. However, it is often implemented as GMM minimand comparison. See bonnet2010inference or berto2007vertical for example.}. Although such a procedure is asymptotically consistent in choosing a true or "better" model, the performance of model selection with limited sample size is still uncertain. In some applications, economists have to make an inference from a relatively small number of observations. Given a limited size of the sample, their procedures may be subject to "over-fitting": excessively complicated models can fit tighter to the observations in hand with better "goodness-of-fit" criterion, and thus is selected as a better model even if the model is not very true.
To avoid over-fitting problem, some model selection criteria such as AIC-GMM or BIC-GMM "penalize" the number of parameters in a model (andrews1999). However, the complexity of economic models is not simply measured by the number of parameters. Structural model may include non-parametric components in specification (e.g., gautier2013nonparametric), where we cannot apply a penalization based on number of parameters. Additionally, estimation procedure sometimes involves nonparametric approximation only for certain models. For example, estimation of dynamic demand model in gowrisankaran2012dynamics includes a nonparametric approximation of a value function, which may make their model more flexible than static demand model. To date, it is not well understood how these factors contribute to the over-fitting issue nor how to penalize its flexibility.
In this paper, we offer a novel approach to this problem that helps researchers to identify the best model specification from the data. Our idea is to apply the cross-validation (CV) method, which is commonly used in other areas such as machine learning, in evaluating the predictive power of the model. The main idea behind cross-validation is to split the data into several portions so that test of a model fit is implemented on a different data from the one used for estimating parameters. As a result, the estimated moment suffers a smaller over-fitting than in-sample model selection.
The largest advantage of sample splitting lies in its wide range of potential applications. On applying CV, one does not need to take the number of model parameters explicitly. As a result, it can select the true model among parametric, non-parametric and even semi-parametric models. Moreover, CV can be applied not only in selecting models, but also selecting hyper-parameters of estimation and even estimation method itself. For example, estimation of dynamic model often includes approximation of value function on a discrete grid space, where the coarseness of the grid space has not been paid adequate attention though it heavily influences the performance of estimation. As to the example of estimation method, random coefficient demand system can be estimated in various specifications, such as parametric or non-parametric, through various methodologies such as nested fixed point algorithm or constrained optimization approach (MPEC, su2012constrained) and they may yield different results especially in limited sample size.
Economists typically evaluate estimation techniques and model specification by checking how the true parameters are recovered in a Monte-Carlo simulation. However, the best specification or methodology may vary across different data or the "true" data generating process that researchers do not observe. Thus, it is preferable to make an assessment in real-world data as well, and CV offers a practical approach to that end. Taking a wide range of applications into consideration, conducting CV in selecting models deserves a significant portion of attention.
Although CV is commonly used in data science fields such as machine learning and data mining, its applicability to economic models is not obvious. In machine learning and data mining, the primal concern lies in how accurate the prediction of a regressor or classifier is. Meanwhile, in empirical economics, identifying the model reflecting the reality closer and estimating its model parameters are of primal concern, and machine learning literature does not provide a sufficient guarantee in identification of a model. This gap remains to be closed in applying data science methods in econometrics. Taking this into consideration, we propose an identifiable CV method for GMM.
We first prove the consistency of cross-validation algorithm: That is, the algorithm identifies a correctly specified model from misspecified models with the probability approaching to $1$ as the number of data increases. When a model is estimated through likelihood maximization, yang2007consistency proved the consistency of the cross-validation in non-parametric regression model selection. We prove an analogous result for GMM version of CV algorithm.
After giving the consistency, we test the performance of our cross-validation algorithm with a limited number of samples by Monte-Carlo simulation. Firstly, we examine a simple instrumental variable regression. We observe our algorithm selects a correctly specified mode over a misspecified model with high probability even when data size is limited. Importantly, our algorithm finds the correctly specified model even when the alternative model has higher flexibility (i.e., more parameters) than the true model, suggesting that it is robust to over-fitting. Furthermore, we compare the performance of our algorithm with Rivers-Vuong type GMM minimand comparison approach and also approaches based on GMM-AIC and GMM-BIC criteria that andrews1999 suggested. The result implies that the comparison of GMM minimand suffers over-fitting, and as a result it often selects a misspecified model of higher complexities. Though GMM-AIC and GMM-BIC based approaches attempt to solve the over-fitting problem by penalizing the flexibility of model, their performance turns out to be extremely sensitive to the model specification, and as a result, they often fail to find the correctly specified model.
Secondly, we conduct another experiment in more complex nonlinear models. We use a collusive pricing model similar to the ones of bresnahan1987 and hu2014collusion, where their objective of model selection is to detect a potential tacit collusion from the sales and price data. We simulate the price and quantity data from perfectly competitive setting and partially collusive setting, and test if our algorithm discovers the true conduct or not. We show that our cross-validation procedure generally perform well to identify the true pricing structure from a limited amount of data. We show how CV outperforms the simple GMM fitting comparison without data split.
In addition, we propose a method to apply cross-validation algorithm when estimation is based on Mathematical Programming of Equilibrium Constraint (MPEC) approach. MPEC is proposed by su2012constrained and is one of the state-of-the-art estimation methodologies. MPEC achieves high computational efficiency by avoiding the nested fixed point algorithm, and its convenience is earning significant attention especially in the industrial organization research community. Though application of CV to MPEC is not straightforward, we provide a modified algorithm of CV applicable to MPEC estimation.
Finally, we perform our algorithm on a cutting-edge structural model with real-world data. The model we adopt is dynamic demand and dynamic pricing model of conlon2012dynamic. The dynamic models are considered to be the recent frontier of the industrial organization community and used in many applications (such as lee2013vertical). However, the superiority of the dynamic models compared with static models on its explainability of the consumer behavior is not sufficiently supported. Likewise, the dynamic pricing model is a frontier research topic in the industrial organization (nair2007intertemporal,luo2015operating), but its empirical support against static model is only descriptive. We apply our CV algorithm to the market data of an online retailer based in the UK to test dynamic models against static models. We show that the results are mixed across different products, even though they are sold by the same retailer.
The paper proceeds as follows. In Section 2, we formally introduce cross-validation in GMM and discuss its econometric property. In particular, we prove its asymptotic consistency. In Section 3, we demonstrate a Monte-Carlo experiment of model selection in IV regression. In Section 4, we perform a further experiment in an oligopolistic pricing model as a nonlinear example. Section 5 explains how we can modify the algorithm when it is applied to MPEC approach. Section 6 presents the setup and results of the real-world application of the dynamic pricing model using online-retailer data. Section 7 concludes the paper.
Let $\mathbf{v}=\{v_t\}$ be a random vector of observed data in $\mathbf{V} \subset R^d$. Let $\mathcal{M}_i$ for $i=1,2$ be the two candidate models to explain the observed data. Each model, if correctly specified, is characterized by a set of moment conditions $f^{(i)}:\mathbf{V}\times\Theta^{(i)}\rightarrow \mathcal{R}^{q_i}$ such that
where $\theta^{(i)}\in \Theta^{(i)}$ denotes the parameters of a model $i$ to be estimated. Let $p_{i}$ be the dimension of $\theta^{(i)}$ Given the observation $\{v_t\}_{t=1,,,T}$, the parameters of each model are estimated via GMM;
where
Let $\underset{T\rightarrow\infty}{\text{plim}}W_T^{i}=W^{i}$, and the population analogue of the moment conditions be
Assume that $\underset{T\to\infty}{\text{plim}}\theta_T^{(i)}=\theta^{(i)}_0$ exists. The null hypothesis is that $\mathcal{M}_1$ and $\mathcal{M}_2$ are asymptotically equivalent;
Two alternative hypotheses are that $\mathcal{M}_1$ is asymptotically better than $\mathcal{M}_2$ or the other way around;
Cross-validation is a model selection procedure in which the data is split into two subsets called training set and validation set. The set of parameters of each model is trained in the trained set, and its goodness is evaluated with the validation set. Let $r \ge 2$, $k<r$ be integers. In leave-$k$-out $r$-fold cross-validation (($k,r$)-CV), we first split $T$ datapoints into $r$ disjoint subsets. At each round of CV, We use $r-k$ of them as the training data, and the other $k$ as the validation data. Multiple number of rounds among possible splits are performed to reduce variability. Namely, let
be the indices of the $j$-th split. Let $\{S \subset \{1,2,\dots,r\}: |S| = r-k\}$ and
be subset of datapoints consisted of folds in $S$. The moment on this datapoints is denoted as
and the model trained to minimize the moment is denoted as
Once the model is trained, it is validated by the rest of datapoints as:
where $\mathcal{N}_{\setminus S} = \{1,\dots,T\} \setminus \mathcal{N}_{S}$. In ($k,r$)-CV, the averaged validation score of each model
is compared, and the model of smaller averaged validation score is selected. The procedure is summarized in Algorithm (ref).
\if0
たぶんここは今日中には完成しないけど書いてる
Let
In this section, we prove the asymptotic normality of the random variable $N_T$ when the number of split $r$ is large. The crux is that $Q_{S,\mathrm{valid}}^{(i)}(\theta_{S}^{(i)})$ for each split $S$ is exchangeable random variables.
The following theorem states that, Under the assumption that the correlation of two different splits are sufficiently weak, $N_T$ is asymptotically normal.
\fi
In this section, we derive the consistency of CV in GMM model selection. Let one of the models is misspecified. Without loss of generality, we assume the first model is the true model\footnote{Of course, the model selection method should not exploit this fact.}. The true model satisfies the following moment condition:
The latter model is assumed to be misspecified: that is, for any $\theta^{(2)}$ the following holds:
The misspecification is divided into two local and non-local ones hall2005.
Alternatively, we can make a weaker assumption that the sample moment of the misspecified model converges to zero slower than that of the true model. This assumption covers cases where the misspecified model is more general (or too general) than the true model. This is the case, for example, the utility function in the true model is a linear function of price but the misspecified model incorporates higher order polynomials.
Note that, in either definition of misspecification, the researcher does not know which model is true, and our interest lies in consistently choosing the true model over a misspecified model based on the dataset.
In the previous literature, Smith (1992) offers a pairwise comparison process for consistent model selection. However, it has some practical disadvantages when applied to empirical research: (i) A pairwise comparison could be extremely demanding if the space of candidate models is large, and (ii) it may be subject to over-fitting problem. To avoid those issues, the most common practice in the field of machine learning is to apply cross-validation (CV) algorithm. In the literature in statistics, Yang (2006,2007) have shown that even the simplest CV procedure can find a true model consistently when the data structure is regression form, i.e. $y_i=f(x_i)+\epsilon_i$. Likewise to the literature, we define a consistent model selection as below:
To derive the consistency of CV, we define the following assumptions.
In the following we prove the following theorem.
We first states lemmas that are proven in hall2005, and by using them we prove the theorem.
This section proposes a statistical hypothesis testing on our CV-based model selection. Let \[ R_{\mathrm{CV}} = \frac{|\mathcal{N}_{S}|^{1/2}(Q_{\mathrm{valid}}^{(1)}-Q_{\mathrm{valid}}^{(2)})}{\hat{\sigma}^2} \] be the test statistics that indicates either the first or the second hypothesis is better than the other. Here, $\hat{\sigma}^2$ is the estimator of the limiting variance $\sigma_0^2$ of $R_{\mathrm{CV}}$. The null hypothesis of the test is \[ \mathrm{H}_0: Q_0^{(1)}(\theta^{(1)}_0) = Q_0^{(2)}(\theta^{(2)}_0). \] These are two alternative hypotheses of interest: The first one indicates $\mathcal{M}_{1}$ is better than $\mathcal{M}_{2}$. That is, \[ \mathrm{H}_1^{(a)}: Q_0^{(1)}(\theta^{(1)}_0) < Q_0^{(2)}(\theta^{(2)}_0) \] and the second one indicates $\mathcal{M}_{2}$ is better than $\mathcal{M}_{2}$: \[ \mathrm{H}_1^{(b)}: Q_0^{(1)}(\theta^{(1)}_0) > Q_0^{(2)}(\theta^{(2)}_0). \] Following riversvuong2002,hallpelletier2011, we discuss conditions where the statistics $R_{\mathrm{CV}}$ is asymptotically normal. We first consider the testing statistics in the general case in Section (ref). Moreover, we show in the case the dependency among splits are sufficiently small in Section (ref), where the statistics is represented in a much computationally efficient way.
We pose the following assumption on the structure of the weight matrix that is essentially the same as hallpelletier2011:
To discuss statistical testing, we need to have asymptotic normality property. The following assumption guarantees that the moment is “well-behaved” around the optimal value $\theta_0^{(i)}$.
For the ease of discussion, we further add the following notation. Let $F_{S}^{(i)} = |\mathcal{N}_{\setminus S}|^{-1/2} \sum_{t \in \mathcal{N}_{\setminus S}} \{f^{(i)}(v_t,\theta^{(i)})\}$. Let $G_0^{(i)} = \mathbb{E}[\partial f^{(i)}(v_t,\theta^{(i)})/\partial \theta^{(i)}]$, and its empirical counterpart be $G_S^{(i)} = |\mathcal{N}_{S}|^{-1} \sum_{t \in \mathcal{N}_{\setminus S}} (\partial f^{(i)}(v_t,\theta^{(i)})/\partial \theta^{(i)})$. Let $\{S_1,\dots,S_{{}_r C _k}\} = \{S \subset \{1,2,\dots,r\}: |S| = r-k\}$ be the set of all splits. We also denote $\theta = (\theta^{(1)}, \theta^{(2)})$, and $\theta_0$ and $\hat{\theta}_{S}$ are defined in the same way.
\if0 The true variance $\sigma_0^2$ for large sample is decomposed as \[ \sigma_0^2 = R_*' V_* R_*. \] \fi
Then, $V_*$ is the \[ V_* = \left(
\right) \] where $c_{j, j'}$ is a submatrix such that
Moreover,
where $B_i$ is the $q_{i}^2 \times q_{i}(q_{i}+1)/2$ matrix such that $\mathrm{vec}(W^{(i)}) = B_i \mathrm{vech}(W^{(i)})$.
Calculating the asymptotic variance of Theorem (ref) requires a calculation of a matrix with its size proportional to the number of splits, which in some cases is computationally prohibitive. This section consider the case where the dependency between the validation data is sufficiently small. In such a case, we can circumvent the computation of a large matrix.
In particular, the leave-one-out CV (special case of our CV with $k=1$) when each datapoint is identically and independently distributed (i.i.d), the following assumption holds:
The proof of Theorem (ref) directly follows by following the same steps as Theorem (ref) with additional fact that Assumption (ref) implies the block-diagonal property of $V_*$ as $c_{i,j} \rightarrow 0$ for $i \ne j$ and the identity of each block.
In this section we present a simple simulation of instrumental variables (IV) regression models to illustrate the consistency of our cross-validation algorithm of model selection. This example also highlights how GMM-minimand-based model comparison and cross-validation can exhibit different results. The setting is similar to the one on hallpelletier2007. Suppose the true data generating process is
where $\boldsymbol{y}$ is a $T\times 1$ vector and $X_{1}$ and $X_{2}$ are $T\times p_1$ and $T\times p_2$ matrix respectively. $X_1$ and $X_2$ are generated from instrumental variables as
where $Z_1$ and $Z_2$ are $T\times c_1$ and $T\times c_2$ matrix respectively.
We consider a case where we have two candidate models to compare. The first model exploits the explanatory variables $X_1$ and instrumental variables $Z_1$.
whereas the second model employs $X_2$ and $Z_2$;
Each model has different explanatory variables as well as the set of instrumental variables so that two models are non-nested. In addition, there are two important differences between the two candidates. First, the second model can be "misspecified" when $\alpha\neq 0$, because the instrumental variables $Z_2$ influences $\boldsymbol y$ directly and thus IVs are not independent from $\boldsymbol \epsilon^2$. When $\alpha>0$ and does not decrease with the number of observations, i.e. $\alpha=10$, it is globally misspecified, which results in inconsistent estimates of the parameters.
The second difference is that the number of the variables. In the following, we assume that $p^1\leq p^2$, meaning that the second model has a larger number of explanatory variables. As discussed earlier, this may cause "over-fitting" issue to the estimation even if the model is falsely specified. In such a case, previous literature proposes the ways to penalize the model by the number of parameters (andrews1999). We compare the performance of the proposed method with the ones of those existing methods in the later section.
Though this example may seem to be somewhat arbitrary, similar problems arise in many situations when econometric models are compared. Specifically, one model can be flexible (or even "over flexible") but misspecified, while the other is simpler but accurate. Some researchers may not value the simplicity, but they would prefer a "correctly specified" model than misspecified models. For example, think of a case where economists try to explain wage from education and other variables, where education is endogenous and has to be proxied by IVs. The misspecified model includes incorrect IVs that gives bias to the estimate of the coefficient. Even if one model exhibits a good fit to the data, if the coefficient of interest is not properly estimated, such a model does not serve well for labor economists. In those occasions, our algorithm serves to help researchers to find the most "correct" model. Our method is general enough so that any specification can be compared.
First we consider the case where over-fitting is a concern as the misspecified model has more parameters therefore could exhibit better fit to the data. We compare our methodology in this case to the model selection procedures proposed by andrews1999 as well as simple GMM comparison as in the previous section. andrews1999 defines GMM-AIC and GMM-BIC criterion as
for $i=1,2$. The procedure chooses the model that exhibits smaller value of the criterion.
Figure (ref) shows the empirical probability of choosing the correctly specified model by cross-validation. One can see that, even when the model 2 has larger number of variables, it chooses the model 1 with very high chance even when the data is limited. When the bias parameter of the model 2 $\alpha$ is as large as $12.$, it selects the first model with probability $91.2\%$ even when the data size is only $100$ and the second model has $9$ variables compared to $3$ of the first model.
On the other hand, GMM based model selection performs extremely poorly when the misspecified model has much more variables than the first model. When $p^2=9$, even with data size $1600$ the accuracy is as bad as $59.1\%$, only slightly above chance level of $50\%$ (when $\alpha=12.$). With data size $200$, it chooses the second model only for $15.7\%$, clearly indicating it is subject to over-fitting.
Note that in our setting, GMM-AIC and GMM-BIC exhibit exactly same choice of models as simple GMM based selection. This is due to the unbalance of two terms in the criterion. In our case, the first term is typically on order of more than $10^5$, while the second term is no greater than $10^2$. Many factors influence the magnitude of the first term, such as the choice of weighting matrix or number of moment conditions. Our result suggests that while cross validation robustly performs in many situations, performance of GMM based model selection is sensitive to those settings.
We turn to the case where the two models have the same number of parameters, while the second model is misspecified. As the number of parameters is the same across two models, note that GMM, GMM-BIC, and GMM-AIC simply choose the model with smaller GMM minimand. Figure (ref) compares the performance of cross-validation algorithm and the GMM minimand based model selection when the second model is globally misspecified. The $y$-axis shows the probability that the correctly specified model is chosen by each algorithm. The result indicates that when overfitting is not a concern, GMM based model selection performs slightly better than cross validation, especially when the data is smaller.
In this section, we demonstrate another Monte-Carlo study to show how our algorithm works in a structural estimation incorporating nonlinear and non-nested models. Specifically, we simulate and estimate a variant of a price collusion model suggested by Bresnahan (1987). The goal of our model selection procedure is to detect whether the firms are colluded, or determining the price competitively using the share and price data. The underlying idea is that the prices of the products of colluded firms are determined to maximize the joint profit, while the competitive price should maximizes the profit of individual firms. Therefore, given the same (true) parameters in demand and cost function, the pricing pattern varies according to the collusive structure. A methodology to study whether collusive behavior exists within a certain industry is by itself an important research topic because ignoring the possibility of collusive pricing may lead to a biased inference of cost estimation, which could be a critical problem for policy implication in applications such as merger analysis.
In the same way as the previous section, we compare the performance of CV-based algorithm to GMM-minimand-based algorithm based on the theory of riversvuong2002. Note that since the number of parameters in a model does not vary across collusive structure, AIC or BIC adjustment does not influence the model selection criteria. We show that in a realistic sample size, CV performs better than in-sample comparison in many cases.
The shares and prices are simulated from a standard logit demand system and static pricing. We simulate data assuming a certain collusive structure. Then we test if and how often CV algorithm can discover the assumed collusive structure. The estimation process is similar to hu2014collusion.
Assume each firm produces a single product and denote them as $j=1,...,J$. The markets are denoted as $t=1,...,T$. The demand is assumed to be a simple logit demand specification: the utility of a consumer $i$ purchasing a product $j$ in a market $t$ is expressed as
where $X_{jt}$ is the observed characteristics that influence the demand and $\xi_{jt}$ is the unobserved utility shock . Assuming $\epsilon_{ijt}$ follows i.i.d type-I extreme value distribution, the share function is
where $\mathbf{p}_t=\{p_{jt}\}_{j=1,...,J}$ is the vectorized prices and $M_t$ is the market size which is known to the researcher. For simplicity, we do not allow random-coefficients (berry1995automobile) as typically done in applications.
Firms' marginal cost is expressed as
,where $Y_{jt}$ is the observed characteristics that affect the marginal cost, and $\lambda_{jt}$ is the i.i.d cost shocks. The profit of each product is
We assume that colluded firms jointly maximize their net profit, sum of $\pi_{jt}$ over $j$ in a group. Define $\boldsymbol{\Delta}$ as a $J\times J$ matrix of price elasticity of colluded products where the $(j,r)$th element is
By solving the first order conditions, the equilibrium prices are determined to satisfy
where $\mathbf{D}_t$ and $\mathbf{MC}_t$ are a vectorized representation of $D_{jt}(\mathbf{P}_t)$ and $\{ MC_{jt} \}_{j=1,..,J}$ respectively.
The parameter estimation under each model follows a standard GMM procedure with instrumental variables. Let $Z$ be instrumental variables that influence the price but are not correlated with the unobserved shocks $\xi$ and $\lambda$. Given a model, the parameters are chosen to minimize the GMM objective defined from the moment condition
The instrumental variables $Z$ include (i) own characteristics, (ii) square of own characteristics, (iii) mean of characteristics in a market, and (iv) square of mean characteristics in a market. The weighting matrix is set to be $W=(Z'Z)^{-1}$.
The candidate models are represented as partitions of firms into price-colluded groups. For instance, if the number of firms is two ($j=1,2$), the possible models are either competitive ($\{\{1\},\{2\}\}$) or collusive ($\{\{1,2\}\}$). If three firms (j=1,2,3), possible models are $\{ \{1\},\{2\},\{3\} \}$ (all competitive), $\{ \{1\}, \{2,3\} \}$, $\{ \{1,2\}, \{3\} \}$, $\{ \{1, 3\}, \{2\} \}$, and $\{ \{1,2,3\} \}$ (all colluded).
We consider different number of observed markets, $T=\{25,50,75,100\}$, realistic numbers for real world application\footnote{For instance, nevo2001measuring observes $94$ independent markets.}. We also vary the true value of price coefficient to test the performance with different difficulty of model selection. Along with the data size, the difficulty of model selection depends on how different the observed data would be across different models. In this particular example, the key difference between models is generated from cross price elasticity. When the cross price elasticity is low, competitive price and colluded price do not differ as much, which makes it harder to find the true model. In logit-demand, the cross price elasticity is calculated by multiplying the share of the two products. Thus, lower price coefficient generally makes model selection easer as it increases the realized share, and the cross price elasticity as a result. For each setting, we generate 100 synthetic dataset and perform the model selection in each.
Table (ref) reports the mean and standard deviation of CV score across true models and candidate models with the price coefficient equals to $-.1$ and $-.3$. The second column represents the true partition of firms, and the third to seventh are the results corresponding to each candidate model. The CV score of the true model is on average smaller than the mis-specified models in any specification. Also, the standard deviation of the score is smaller for the true model. Both mean and standard deviation of the true model decline in the number of observations.
We report the probability that each candidate model is chosen by our algorithm in table (ref). In each setting, the probability to find the true model increases in the number of markets, which corresponds to our theoretical finding in section 2. For comparison, Table (ref) presents the same for GMM-minimiand comparison.
Figure (ref) compares the performance of our model selection to a simple in-sample GMM fit comparison under different price coefficient. It shows that our CV algorithm generally performs better than in-sample comparison. The difference is particularly large when the true model is partially colluded (second column). As seen in Table (ref), GMM comparison tends to select all-competitive model in such a case.
In this section, we propose a method to apply cross-validation algorithm when estimation is based on Mathematical Programming of Equilibrium Constraint (MPEC) approach proposed by su2012constrained. MPEC approach formulates the estimation as an optimization problem with constraints: The variables of the optimization consists of structural parameters as well as endogenous latent economic variables, and the constraints among the variables represent the equilibrium condition that the economic model requires.
The application of the cross validation procedure to MPEC estimation is not straightforward: If parameters estimated from training data is substituted in a MPEC model with test data directly, the constraints would be not satisfied in general. In such a case, we cannot directly compare GMM objective on test data across models since we also have to consider the violation of constraints as indication of model misfit.
Taking the above discussion into consideration, we propose a modified cross validation procedure. We differentiate the choice variables for the optimization problem into two categories: model variables and observation-specific variables. Model variables are specific to the model, therefore shared across training and test data. Observation-specific variables are latent variables defined on each observation. For instance, in BLP demand estimation example on dube2012improving, the price elasticity is a parameter assumed to be constant across observations, thus treated as a model variable. Meanwhile, the unobserved utility shock ($\xi_{jt}$ in their notation) is defined for each datapoint, thus regarded as observation specific.
Our modification is simple. In training data, we jointly choose the model variables and observation-specific variables to optimize the GMM objective function with equilibrium constraints. In test data, we still solve a constrained optimization problem, but only with respect to observation-specific variables while the model variables are set to the estimates from training data. The algorithm is described in detail below and summarized in Algorithm (ref).
We first outline the MPEC formulation of parameter estimation. Here we follow the notation of su2012constrained except that we allow some endogenous variables to be observation-specific. Suppose an econometric model $\mathcal{M}_i$ is expressed with the parameter vector $\theta$, a vector of endogenous variables $\sigma$, and endogenous variables that are observation-specific $ \eta$, and the equilibrium constraint $h(\theta,\sigma,\eta)=0$. In MPEC formulation, each model is characterized by a set of moment conditions with equilibrium constraints:
Given the observation $\{v_t\}_{t=1,,,T}$, the parameters of each model are estimated via MPEC:
where
Let $\theta_T^{(i), \mathrm{GMM-MPEC}}$ be the parameters that are solution of Eq. (ref), and let $\theta_T^{(i), \mathrm{GMM}}$ be the solution of standard GMM (i.e., Eq. (ref)). Moreover, let
and $V_T^{(i), \mathrm{GMM}}(\theta) = Q_T^{(i)}(\theta)$. The equivalence of GMM and GMM-MPEC implies
We split the observations in the same way as section 2. The moment on the datapoints $S$ is
We train the model to minimize the moment under equilibrium constraint. The trained model is denoted as
Once the model is trained, it is validated by the rest of datapoints. Instead of simply evaluating the GMM objective in the validation data at the trained model parameters, observation-specific endogenous variables need to be chosen so that the equilibrium constraints are satisfied. We do so by minimizing the GMM objective subject to equilibrium constraints with respect to $\eta$ only, while model parameters are fixed at trained value. Formally,
The averaged validation score of each model
is compared and the model of smaller averaged validation score is selected.
In this section, we perform our model selection procedure in a structural model with a real-world dataset. The models we compare are dynamic and static demand and pricing model that are taken from conlon2012dynamic. In particular, we first apply our cross-validation algorithm to test either the state-of-the-art dynamic demand model (gowrisankaran2012dynamics) or the traditional static demand model (berry1995automobile) has stronger explanatory power in the consumer behavior. To this aim, we use monthly sales and price data of an online-retail shop. Furthermore, we consider supply side dynamics of pricing that takes the seasonality and consumer skimming into consideration such as nair2007intertemporal: We investigate whether or not such a model explains the observed pricing pattern better than traditional static profit maximization model that is based on the consumer model selected in the previous step.
Structural estimation of a dynamic model has been an important frontier in industrial organization, both on demand side and supply side. On demand side, dynamic model of consumer behavior has been widely applied by researchers recently (gowrisankaran2012dynamics). The underlying idea in the dynamic demand model is that consumers are forward-looking regarding the changes in the market such as price and make a dynamic decision by considering the future market state. Such a model is justified by the fact that important parameters such as price elasticity could be severely mis-estimated by ignoring the forward-looking behavior of consumers. Meanwhile, similar mis-estimation would occur if a researcher applies a dynamic model in the case the consumers are in fact myopic. From a market level data, it is not directly visible if consumers are forward-looking or myopic.
Contrary to the demand side, dynamic pricing in supply side has a long history of theoretical studies dating back to coase1972durability. Nevertheless, little empirical attention is paid until recent years (nair2007intertemporal). Under certain conditions, firms have the incentive to determine current price by taking its effect on the future profit into consideration. For example, when consumers are heterogeneous in an evaluation of a product, firms are motivated to "skim" high-evaluation consumers in earlier periods by setting a high price and later lower it. With myopic consumers (as in luo2015operating), the pricing decision boils down to a dynamic programming of a firm in the case of monopoly or a dynamic game between firms in the case of oligopoly. If the consumers are also forward-looking, the pricing boils down to a dynamic game between consumers and firms as studied in nair2007intertemporal. In this case, the observed price and demand are interpreted as a result of dynamic equilibrium.
It is not straightforward to infer if the pricing is dynamic or not from the market level data. A declining tendency on the price does not always indicate that firms are making pricing decision dynamically: If the consumers are heterogeneous in either product evaluation or price sensitivity and leave market after purchase, a myopic optimal price may be decreasing in periods since the remaining consumers are more price elastic.
Applying dynamic pricing model to data generated from myopic pricing would cause a significant bias in the estimates of supply-side parameters such as marginal cost. For instance, a dynamic pricing model may interpret an observed high price in a certain period as a firm sparing some demand for the future, while it is a result of high marginal cost in truth. Therefore, estimation of supply-side model parameters such as marginal cost requires researchers to know if firms are myopic or forward-looking.
As it is important to correctly specify the dynamic feature of the agent's decision making both on demand and supply side, researchers are encouraged to verify whether the decision making is static or dynamic from the data rather than appealing to intuition, desirably based on real-world datasets. Regarding this aspect, we demonstrate our cross-validation algorithm to compare two by two alternative models; dynamic or myopic consumers, and dynamic or myopic firms. The models are estimated via GMM-MPEC. We take a simple dynamic model from conlon2012dynamic.
We perform estimation and model selection on a dataset of price and sales of an online-retailer based in UK. The data is taken from the University of California Irvine (UCI) Machine Learning Repository (henceforth, UCI). UCI repository consists of more than 300 datasets. The data used in this study is available here at \url{https://archive.ics.uci.edu/ml/datasets/Online+Retail} free of charge. We consider the use of such a publicly available dataset increases a reproducibility of a research process. In machine learning field, researchers are encouraged to compare the performance of a newly proposed model or algorithm to old ones with a publicly available dataset, and the UCI repository is widely used in this aim.
We consider models of 2 by 2 design: static or dynamic demand, static or dynamic pricing. We denote each model as $m\in\{1,2,3,4\}$, where $m=1,2$ assume static demand, $m=3,4$ assume dynamic demand, $m=1,3$ assume static pricing, and $m=2,4$ assume dynamic pricing. For simplicity, we assume that the firm and consumers make their purchase decision independently across products. It is entirely possible to test if this assumption is valid or not using our CV algorithm, but we omit it as the main purpose of this section is an illustration of model selection procedure. The consumers are heterogeneous in price sensitivity and the constant term of utility as in random coefficients model. We assume that consumers make a purchase at most once for each product within the considered period. This assumption is justified by the transaction level data. Among all the transactions used in the data, 75.8% of them are made by consumers who purchased the same product only once in the considered period. An alternative approach is to model repeated purchase and inventory behavior explicitly as in hendel2006measuring, but we do not take this path for tractability.
In each period, consumers in the market decide whether to purchase a product or not to maximize their objective function. If the demand is assumed to be static, the objective function is simply the utility function defined below. If the demand is dynamic, the objective function is the infinite-period sum of discounted utility.
Denote products as $j=1,...,J$ and period as $t=1,...,T$. Consumer $i$'s utility of purchasing a product $j$ at period $t$ is
where $p_{jt}$ is the price of a product $j$ in period $t$, $\mathbf{X}_{jt}$ is the observable characteristics, and $\xi_{jt}$ is the i.i.d preference shock, which enters the moment conditions. $\epsilon_{ijt}$ is the logit error term that follows type-I extreme value distribution and i.i.d across periods and products. The utility of not purchasing is $u_{i0t}=\epsilon_{i0t}$ as the non-random component is normalized to be zero. The random coefficients follow a normal distribution.
,where $(\alpha^p,\alpha^0)$ are the population mean of the utility coefficients, $\nu^p_{i}$ and $\nu^0_{i}$ are draws from a standard normal distribution, and $(\rho^p,\rho^0)$ are the standard deviation of the distribution of the random coefficients.
In the static demand model, the consumers simply compare the utility of purchase to non-purchase in each period. Thus the purchase probability is
In the dynamic demand model, the consumers make purchase decision by comparing the instant utility to the value of waiting until next period. Let $\Omega_{ijt}^d$ be a state space for a consumer $i$ on product $j$ at period $t$ and $W_{ij}(\Omega_{ijt}^c)$ be a value function associated to the state. The Bellman equation is expressed as
The purchase probability of product $j$ of a consumer $i$ at period $t$ is
Following conlon2012dynamic, we make an assumption that consumers have perfect foresight over a transition of state $\Omega^{d}_{ijt}$. Formally,
where
for all $i$, $j$, and $t$. The second line is a direct consequence of the first line following the argument of rust1987optimal. An alternative and more popular specification is to assume that consumers form an expectation of the future state by certain functional form, typically an AR(1) regression. Compared to functional assumption perfect foresight reduces the computational burden significantly as it avoids integration over a distribution for calculating expectation (See conlon2012dynamic for further discussion.) Also, note that by our CV algorithm we can even investigate which of perfect foresight and AR(1) assumption makes the model more accurate, which we believe is an interesting future work.
Finally, for both static and dynamic model let $M_{ijt}$ be the market size of consumers for a product $j$ at period $t$. Given the consumers purchase the same product at most once, the market size transition for any model $m\in\{1,2,3,4\}$ follows
We express the marginal cost of product $j$ at period $t$ for the retailer as $MC_{jt}$ where
$X^{cost}_{jt}$ is the observable characteristics of the product, and $\lambda_{jt}$ is the cost shock i.i.d across time and products.
Denote the states of a product $j$ for the retailer at period $t$ as $\boldsymbol{\Omega}^s_{jt}$. $\boldsymbol{\Omega}^s_t$ includes the market size of each consumer segment $\{M_{ijt}\}_{i}$ and the draw of unobserved utility shock, $\{\xi_{ijt}\}_{i}$ and $\lambda_{jt}$. Given the demand system described above, the demand function is written as
The instant profit function of a product $j$ at period $t$ is therefore
In static pricing model, $m=1,3$, the retailer simply chooses the price to maximize the myopic profit:
In dynamic pricing model ($m=2,4$), the retailer maximizes the net profit over time with discounting. The discounting factor $\beta$ is assumed to be same with consumers. The retailer determines the price after observing the realization of the shocks, $\{\xi_{ijt}\}_{i}$ and $\lambda_{jt}$. The value function of a product $j$ is expressed as
where the expectation is over the unobserved cost shock in the next period, $\lambda_{jt+1}$. The optimal price is determined as
Similar to the demand side, we assume that the retailer has a perfect information on the transition of the error draw.
This section describe the equilibrium condition for each model. When consumers and firms are both static ($m=1$), the equilibrium price and demand are the standard one as in many models such as berry1995automobile. When consumers are static but firms are dynamic ($m=2$), pricing can be seen as a single agent dynamic optimization problem with continuous choice variable $p_{jt}$. Similarly, when consumers are dynamic but firms are static ($m=3$), consumers solve a single agent dynamic optimization problem. The consumers problem is an optimal stopping problem as the choice is the timing of purchase. When both consumers and the retailer are both dynamic ($m=4$), we assume their behavior is at Markov Perfect Nash Equilibrium (MPNE) where consumers' and retailer's prediction of the value function matches to the realization.
We obtain our data from UCI machine Learning Repository. The UCI Machine Learning Repository maintains more than three hundreds datasets that are intensely used by machine learning community for empirical investigation and comparison of algorithms. When researchers propose a new model or algorithm in machine learning field, a common practice is to test its performance on the dataset in this repository. Such a culture gives a thorough idea on the practical performance of existing models and algorithms. Moreover, it helps a new researcher replicate the results on the existing papers.
The dataset we utilize in this study is the online retail data created by chen2012data, posted on UCI Machine Learning Repository in November 2015. The data is publicly available at \url{https://archive.ics.uci.edu/ml/datasets/Online+Retail}. The information about the data source is provided by the authors as follows: "The online retailer under consideration is a UK-based and registered non-store business with some 80 members of staff. The company was established in 1981 mainly selling unique all-occasion gifts. For years in the past, the merchant relied heavily on direct mailing catalogs, and orders taken over phone calls. It was only 2 years ago that the company launched its own web site and shifted completely to the web. Since then the company has maintained a steady and healthy number of customers. The company also uses Amazon.co.uk to market and sell its products."
The data include all the transactions occurred on this retailer from December 2010 to December 2011. Each transaction information includes quantity, unit price, consumer ID, and country. We dropped any sales to outside UK. The majority of the sales is inside UK and non-UK sales has only limited amount (approximately 20%.) Since our purpose is to demonstrate application of CV model selection to static and dynamic models, we aggregate the data into a monthly sales of each product so that the data format follows typical market level data and we can apply commonly used economic models. The monthly sales is simply a sum of the quantity sold in a particular month. The monthly price is calculated as the average of the price of transaction occurred in each month weighted by the quantity. We omitted the products that have any zero sales in the considered months from the data.
On the top of price and sales data, the author hand-coded product category and subcategory based on the description of products. The categories include {\it Children}, {\it Decoration}, or {\it Kitchen}. The number of products as well as basic statistics are summarized in table (ref). Figure (ref) shows the average of monthly price and quantity sold in each category. It shows that the dynamics is heterogeneous across categories. For instance, the price of products in {\it Gift} and {\it Decoration} show tendency to decline over periods, while {\it Home and Garden} or {\it Candle} show more fluctuation.
We implement model selection for the demand side and supply side sequentially. First we test if the demand is static or dynamic. Subsequently, we test if the pricing is static or dynamic, assuming the demand model chosen in the previous step. The endogenous variables such as the market size $M_{ijt}$ and the share $s_{ijt}$ are estimated in the demand side, and imported over to the supply side estimation. Importantly, we do not have to specify the pricing model on estimation of demand side by virtue of perfect foresight assumption. We treat the data in each category independently.
We adapt 3-fold cross validation, $(k,r)=(1,3)$. Because the data has a panel structure of products and periods, either the product-wise or period-wise split is possible. We adopt split based on products. That is, we split the products into three groups, and use two of them to estimate a model and use the last one for validation.
To estimate each model by GMM-MPEC, we formulate the estimation as a minimization problem of GMM objective with equilibrium constraints based on the model described above. Under the assumptions we impose, the equilibrium constraints are convex and mostly either linear or quadratic. This fact ensures that we are able to find an optimal solution of the estimation problem.
First we describe the MPEC formulation of demand models. For the static demand model ($m=1,2$), the set of constraints are
for all $(i,j,t)$.
For dynamic demand model, the constraints are similar except the consumers compare the purchase utility to the value of waiting until next period.
for all $(i,j,t)$.
The model parameters to estimate are $\theta^d = (\alpha^p,\alpha^0,\rho^p,\rho^0)$. The data to input are the realized demand $D_{jt}$, the observed price $p_{jt}$, and the random draws $\nu^p_i$ and $\nu^0_i$. The predicted share $s_{ijt}$, the market size of each consumer type $M_{ijt}$, and the error draw $\xi_{jt}$ are the endogenous variables. In the dynamic demand model, the value function $w_{ijt}$ is also observation-specific endogenous variable to choose for the optimization.
We define the supply side estimation problem by the first order condition and the Bellman equation. By abusing notation, let $D_{jt}^m(p)$ as a demand function with respect to price in model $m$. The supply side equilibrium constraints of static pricing model is that the observed prices are chosen to maximize the instant profit:
for all $(i,j,t)$.
Instead of the third line above, the dynamic pricing model includes Bellman equation:
The model parameters to estimate is $\theta^s=\gamma$. $MC_{jt}$, $\lambda_{jt}$, and the value function are observation-specific endogenous variables. $M_{ijt}$ and $s_{ijt}$ are estimated in the demand side as endogenous variables.
In both static and dynamic model, the constraint includes the retailer's optimization problem. We convert it to the first order condition when solving for the estimation. The details are in the Appendix.
The GMM objective is a function defined by moment conditions
where $Z$ is the instrumental variables. It includes category and subcategory dummies, period dummy, and the market size of consumer segments $\{M_{it}\}_{i}$. The market size information is correlated with price because it relates to the price elasticity. Since we assume that the unobserved shocks are not serially correlated, the market size at period $t$ is not correlated with the shocks in the same period. Further detail of the setting for estimation is described in the Appendix.
Table (ref) presents the cross validation score of each model. The second from the last column shows the demand model selected by CV. The last column exhibits the selected pricing model. One can see that the selected model varies across categories. On demand side, the data on {\it Children} {\it Decoration}, and {\it Kitchen} are explained better by the static model, while the dynamic model is preferred on other categories. On supply side, static pricing explained the data of {\it Crafts}, {\it Decoration}, and {\it Personal Item} better.
The result of model selection is difficult to interpret. One could try to provide some intuition: For instance, the products that fits static demand model better may be the ones that consumers cannot make a consumption plan. On products where the retailer engages in static pricing, it may be due to certain circumstance that researchers do not observe, such as a contract with wholesaler or limitation of inventory. However, prior to observing the result of cross validation, it is hard to make an reliable and scientific argument and justification for any model to be realistic.
The difficulty of interpretation in turn suggests that it is impractical for researchers to assume a certain model beforehand. Selecting a structural model based on intuition may severely bias the inference. To see the problem, (ref) shows the estimated price coefficient in each category in different specification. While in some cases two models exhibit fairly similar result, in some cases such as {\it Candle} or {\it Party} the result is largely different. Therefore, we recommend that researchers cross validate their models whenever possible, unless they have a strong reason to believe in certain model.
In this paper, we have proposed a cross-validation approach to model selection when models are estimated via GMM criterion. Cross-validation procedure can be readily implemented in any existing economic models without much extra work for researchers. We have proved its asymptotic consistency, and Monte-Carlo experiments in both linear and non-linear model confirm that cross-validation outperforms in-sample comparison that economists traditionally practice.
We also proposed a way to apply cross-validation when models are estimated through MPEC. As its real-world application, we adapt our CV based model selection to test dynamic demand model and dynamic pricing model in an online-retailer data. We find a quite diverse result across product categories. Unexpectedly, even on the same retailer it is not consistent whether a dynamic model is preferred or not. As the implication of structural estimation largely depends on the assumed model, this result suggests that economists should cross-validate their structural models rather than appealing to for reliability of their inference.