Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
108,263 characters · 13 sections · 73 citation commands
Moment Inequalities in the Context of Simulated and Predicted Variables
Keywords: Simulated moments, Moment inequalities, Smoothable convex functions
\onehalfspacing
Recently, estimating with moment inequalities rather than traditional equalities has proven increasingly popular Tamer2010. Even in contexts when a model is difficult or even impossible to compute exactly, economic theory may provide inequalities that are amenable to estimation. Thus, many of the examples in which moment inequalities are attractive are also examples in which computing the model is complex. This model complexity also leads the author to use simulation techniques, often to address the calculation of integrals over complex objects. However, the role of simulation in moment inequality estimation has been largely unexplored.
In this paper, we argue that using simulation or predicted values in the context of moment inequalities has important implications both for determining identified sets and for inference. We show that simulated moment inequality estimators suffer from small sample bias, which is a result of the irregular nature of the estimator. This irregularity is introduced by the following mechanism. Rather than finding optimal parameters through some kind of extremum estimator, moment inequality estimators typically build the estimated identified set or confidence region through level set computations, such as in ChernozhukovHanTamer2007 and AndrewsJia2012. The level set is defined by the intersection of moments, and at these intersections, the objective function may depend on the underlying moments in an irregular way. That is, at such intersections, the distribution of the objective function can be particularly sensitive to perturbations to the underlying moments. Since these intersections often define the maximum or minimum of the confidence interval in a particular direction, they determine the range of parameters in the confidence interval, and thus the intersections are often of greatest interest. Whereas approximation error (for instance, due to simulation) in moment equality estimators for small samples is of second order importance, we show that irregular objective functions promote approximation error in small samples to first order importance.
In this paper, we first describe formally the phenomena that we are interested in. A starting place is the well-known result for moment equalities that estimation that requires simulation is consistent, even for a fixed number of draws McFadden:1989kx,PakesPollard1989. Moment inequality estimators are similar to moment equalities because they take means over simulated or predicted variables before transforming them into an objective function. Simulated moment functions are consistent (and asymptotically unbiased) for their population counterparts $E[m(X,\cdot)]$ for a finite number of draws. This ensures that level-set estimators of identified sets ChernozhukovHanTamer2007 are also consistent in the Hausdorff distance.
We show that the similarity breaks down when it comes to inference in finite samples, due to the irregular feature of the objective function. We explore this phenomenon in two Monte Carlo experiments. The first one focuses on estimation of the point of intersection of multiple simulated moment inequalities, and is meant to maximize the scope of the problem we discuss. The second is a more realistic treatment: We study an entry game in the spirit of CilibertoTamer2009. Moment inequalities are generated by the lower and upper bounds of entry probabilities conditional on covariates that are classified into a finite number of bins. In our Monte Carlo experiments, we find that the coverage probabilities of the confidence regions are distorted severely when there are bins with a limited number of observations and a small number of simulation draws is used. The presence of such bins is common in empirical applications. Therefore, inference methods that properly account for the effects of simulation are needed.
We propose a new solution to the problems associated with the irregularity of the objective function, including those due to simulated moments. The key is that the commonly used objective functions involve smoothable convex functions Beck:2012xy. Our method “regularizes" or smoothes the objective function, and we show that this method leads to a straightforward bias correction method. While the idea of regularization appears in some other contexts, such as HaileTamer2003, Chernozhukov:2015aa and Masten:2017aa, we formally show that this approach has uniform validity in the context of inference with simulated variables. Our regularization method is based on the class of $\mu$-smooth approximations studied in the non-smooth optimization literature Nesterov:2005qv,Beck:2012xy. We provide conditions on the choice of approximating functions and regularization parameters that ensure the uniform validity of an inference procedure that combines the proposed regularization scheme with a straightforward bootstrap resampling. In addition to being attractive theoretically, we show that this approach performs well in practice, even relative to techniques that implement bias correction via the adjustment of the critical value such as Andrews:2010jk and Chernozhukov:2013fj. Specifically, our Monte Carlo experiments show that the proposed method controls the size well and is often less conservative than the existing methods.
Before moving on, we want to stress that simulation is common in many well-known applications of moment inequalities. For instance, HaileTamer2003 simulate bids in order to place bounds on the value of participants in an auction. CilibertoTamer2009 simulate an entry game to determine upper and lower bounds for the probability of a firm entering under different equilibrium selection mechanisms. Ho2009 uses inequalities to study network formation between hospitals and insurers, and uses predicted profit from a network as an explanatory variable for the firm's choices. The profit function is based on a random coefficient logit demand function as in BerryLevPakes1995, which involves simulation. Eizenberg2011 studies firms choosing product characteristics, and also relies on simulated demand predictions drawn from a BerryLevPakes1995 demand system. KawaiWatanabe2013 simulates market effects in a model of strategic voting that uses moment inequalities to address unobserved beliefs about other voters. Although not strictly moment inequalities, the method of BajariBenkardLevin uses simulation in the context of a minimum distance estimator based on inequalities to study dynamic oligopoly games. Well-known applications are Ryan2012 and FowlieReguantRyan2015.
While our paper focuses on simulation as a source of small-sample bias, the same problem is introduced by using predicted values from some prior estimation stage. For example, Holmes2011 studies the diffusion of Walmart using moment inequalities, and HoudeNewberrySeim2017 take a similar approach to study the locations of Amazon's fulfillment centers. Both papers use profits or revenues as explanatory variables in their moment inequalities, where revenues and profits are constructed from estimated models. Although neither of these papers use simulation in any stage of their estimation, the fact that there is estimation error associated with these variables brings up similar issues to the approximation error introduced by simulation.
Our paper follows in a long line of research on problems with using simulation and prediction in estimation procedures. For instance, Hausman:1983aa terms using predicted or simulated values in non-linear estimation procedures the “Forbidden Regression,” and GourierouxMontfort1996 shows that simulated Maximum Likelihood is inconsistent for any fixed number of samples due to the non-linearity of maximum likelihood estimator. Our result is in fact not due to non-linearity, as moment inequality estimators are not inherently non-linear. Indeed, we show consistency even for a fixed number of samples. Rather, our result emphasizes the irregularity of the level set estimator in a moment inequalities context, which makes the confidence interval irregular at important points. Interestingly, BajariBenkardLevin show in Table 5 that their inequalities estimator may be inferior to an estimator based on moment equalities PakesOstrovskyBerry2007 if one can be implemented. They ascribe this to the non-linearity of the second-stage estimator.\footnote{For example, on page 1362, BajariBenkardLevin write: “The results above suggest that the inequality estimator may exhibit bias in small samples. This bias arises because the second-stage objective function is nonlinear in the first-stage estimates."} Our paper provides an alternative explanation, which is the irregularity of the level-set estimator in the context of first-stage simulation.
Let $X_i\in\mathcal X\subset \mathbb R^{d_X}$ be a random vector, $\theta\in\Theta\subset\mathbb R^{d_\theta}$ be a structural parameter and $m:\mathcal X\times\Theta\to\mathbb R^{J}$ be a function known up to the parameter. Consider the (unconditional) moment inequality restrictions:
We call the set of parameter values satisfying these restrictions an identified set and denote it by $\Theta_I$. In models where $m_j$ is difficult to evaluate analytically, simulation methods are often employed to obtain its approximation. Throughout, we consider the setting where $m_j$ can be written
for some known function $M_j$ and a conditional distribution $P(\cdot|x)$, which may also depend on the parameter. This allows one to draw simulated samples $u_r,r=1,\cdots,R$ from the conditional distribution and approximate $m_j$ by a simulation counterpart $R^{-1}\sum_{r=1}^RM_j(x,u_r,\theta)$. We use the subscript $R$ to denote statistics constructed from simulated samples. This setting parallels the classical method of simulated moments (MSM) McFadden:1989kx,PakesPollard1989 except that the moment conditions in (ref) involve inequality restrictions.
For making inference for the structural parameter $\theta$ or its identified set (the set of $\theta$s satisfying (ref)), level-sets of criterion functions are commonly used ChernozhukovHanTamer2007,Andrews:2010jk. Following the literature, we consider set estimators and confidence regions of the form:
where $T_{n,R}$ is a test statistic (properly scaled sample criterion function), and $c_{n,R}$ is a possibly data-dependent critical value. The variable $n$ is the number of observations in our sample, and the subscript {\em n} denotes statistics constructed from that sample. Throughout, we consider criterion functions that can be written as
for some index function $S:\mathbb R^J\times\mathbb R^{J\cdot J}\to\mathbb R$, which aggregates the vector of sample moments $\bar m_{n,R}(\theta)\equiv (nR)^{-1}\sum_{i=1}^n\sum_{r=1}^RM_j(x,u_r,\theta)$ normalized by an estimator $\hat\Sigma_{n,R}$ of the asymptotic covariance matrix. Examples include $S(m,\Sigma)=\max_{j=1}^J\Sigma^{-1/2}_{jj}m_j$ and $S(m,\Sigma)=\sum_{j=1}^J(\Sigma^{-1/2}_{jj}m_j)_+^2.$
A key observation is that the level sets commonly used in the literature may depend on the underlying moments and hence simulation errors in an irregular manner. We illustrate this point using simplifications of well-known examples in the literature.
The next example is an entry game based on BresnahanReiss1991,Berry1992,Tamer2003aRES,CilibertoTamer2009.
Whereas CilibertoTamer2009 places inequalities on the probability of equilibrium outcomes, our next example follows the approach of PPHI2008 to generate inequalities directly from agent utility functions and revealed preference. This approach has been utilized to study strategic environments such as product introductions Eizenberg2011 and network formation Ho2009. In both of these binary choice examples, the researchers estimate variable profits in a pre-stage and use the moment inequalities to estimate the fixed cost associated with a positive choice.\footnote{Similar examples are Nosko:2014aa, Wollmann:2014aa, CrawfordYurukoglu2012, and Gowrisankaran:2014aa.} To the extent that variable profits are estimated with some error, this approach introduces analogous problems to the ones we highlight in the context of simulation. The problem is particularly clear if the variable profits are based on simulation estimators. That is the case for Eizenberg2011 and Ho2009, which utilize a simulated demand system BerryLevPakes1995 in the pre-stage. We provide an example here, based on Eizenberg2011:
We start with an observation that, similar to moment equalities, level-set estimators that use simulation are consistent even for a fixed number of draws. This similarity arises because estimators of the moments take means over simulated variables before transforming them into an objective function and level-set estimators depend on the estimated moments in a continuous way. For each $i$, let $W_i=(X_i,u_{i,1},\dots,u_{i,R})'$ and let $\hat m_{j,R}(W_i,\theta)=R^{-1}\sum_{r=1}^RM_j(X_i,u_{i,r},\theta)$. Then, it holds under mild regularity conditions (Assumption (ref) in Appendix A) that
uniformly in $\theta$ as $n\to\infty$ for any fixed $R$.\footnote{Moreover, under the assumption of Proposition (ref), the estimated moments are asymptotically unbiased in the sense that the empirical process $\frac{1}{\sqrt n}\sum_{i=1}^n(\hat m_{j,R}(W_i,\cdot)-E_P[m(X_i,\cdot)])$ converge weakly to a Gaussian process with zero mean.}
We state the Hausdorff consistency of level-set estimators as a proposition under a set of assumptions similar to those in ChernozhukovHanTamer2007. For this, let $d_H(A,B)\equiv \max\{\sup_{a\in A}\inf_{b\in B}\|a-b\|,\sup_{b\in B}\inf_{a\in A}\|a-b\|\}$ denote the Hausdorff distance between two sets $A,B$.
While simulation based level-set estimators are consistent, the similarity to moment equalities breaks down when it comes to inference. In models characterized by moment equalities, finite simulation draws affect a confidence region primarily through the asymptotic variance of a point estimator. However, with moment inequalities, this is no longer the case. A noteworthy feature of the level sets based on moment inequalities is that its boundary may depend on the sample moments in a non-standard manner. To see this, in Example (ref), write the boundary of the level set as
where, for each $j$, $m_{j,n,R}=(nR)^{-1}\sum_{i=1}^n\sum_{r=1}^R1\{u_{j,i,r}<X_{j,i}\}$. Note that $\phi(m)=\min\{m_1+c/\sqrt n,m_2+c/\sqrt n\}$ is a nonlinear function that is not differentiable at points such that $m_1=m_2$. While this function is still directionally differentiable, the (directional) derivative of $\phi$ at $E_P[m(X_i,\theta)]$ can be shown to depend on the underlying data generating process in a discontinuous manner. This has important consequences on inference. Namely, even if the simulators for the moments are consistent (with a fixed simulation size), they may introduce finite sample biases to the boundary of the level-set, which in turn may affect the performance of inference in non-trivial ways. Below, we show numerically that this can sometimes result in severe size distortions in empirically relevant settings.
The non-standard nature of inference in moment inequality models and its finite-sample properties have been extensively studied in the recent literature Andrews:2009aa,Andrews:2010jk,Hirano:2012qv,Chernozhukov:2013fj,Fang:2014eu. However, to our knowledge, its consequence in relation to simulation-based inference has not been explored. One of our goals here is to quantify the effects of simulation in the context of moment inequalities and provide a practical guidance for empirical studies.
We start with a simple numerical experiment. Slightly generalizing Example (ref), consider $J$ moment inequality restrictions on a scalar parameter $\theta$:
The goal of the experiment is to compare the performance of two types of confidence intervals for $\theta$: one that computes the moment above analytically and the other that approximates the moment by simulation. Let $X_i\equiv(X_{i,1},\cdots,X_{i,J})'$ be generated as an i.i.d. random vector following a $J$-dimensional standard normal distribution. For each $i$ and $r$, let $u_{i,r}\equiv(u_{i,1,r},\cdots,u_{i,J,r})'$ be generated as a $J$-dimensional standard normal vector independent of $X_i$. For the simulation-based confidence region, we draw, for each $i$, a random sample $\{u_{i,r}\}_{r=1}^R$ of size $R$.
For each $c\ge 0,$ define
where $\Phi(\cdot)$ is the cumulative distribution function of a standard normal distribution. The first confidence region $\mathcal C_n^{\text{Ana}}$ computes the moments analytically, while the second confidence region $\mathcal C_{n,R}^{\text{Sim}}$ computes them using simulation. To investigate the effect of simulation on the test statistic only, we use a common critical value $c$ for both confidence intervals. This critical value $c$ is calculated as the $1-\alpha$ quantile of the maximum of $J$ independent normal random variables with mean $0$ and variance $Var(\Phi(X_i))$, which corresponds to the limiting distribution of $T_n(\theta)=\sqrt n\max_{j=1,\cdots,J}\{\theta-n^{-1}\sum_{i}\Phi(X_{j,i})\}$ under the least favorable configuration (i.e. $\theta=E[1\{u_j<X_j\}]=0$ for all $j$). Note that this critical value does not account for the fact that a finite number of draws is used in (ref).
Below, we report the probabilities of the confidence intervals covering the upper bound $\theta^U$ of the identified set.\footnote{ For the one-sided confidence intervals in (ref)-(ref), covering the upper bound $\theta^U$ of the identified set is the least favorable event for covering the identified set or covering each point in the identified set.} Table (ref) shows the coverage probabilities of the confidence intervals for a nominal level of $1-\alpha=0.95$. We report simulation results based on sample size $n\in\{100,250, 1000 \}$, the number of simulation draws $R\in\{1,5,10,20\}$, and the number of moment inequalities $J \in \{2,5,10,30\}$. For each setting, we generate $1000$ Monte Carlo replications. Here, the experiments are designed to investigate the performance of the confidence intervals when relatively small numbers of draws are used. However, note that the number of simulation draws in this range is used in practice CilibertoTamer2009.
The coverage probabilities of the confidence intervals depend on the number of simulation draws $R$ and the number of inequalities $J$ in non-trivial ways. For any $n$ and $J$, reducing the number of draws $R$ lowers the coverage probability below the nominal level, resulting in a size distortion. This distortion is particularly severe when the number $J$ of inequality restrictions is large. For example, consider the case with $J=30$ inequalities. This setting is relevant for empirical examples that involve moderate to many inequalities. In this case, even for $n=1000$, the simulation based confidence intervals have coverage probabilities significantly below the nominal level: 0.235 ($R$=1), 0.810 ($R$=5), 0.881 ($R$=10), and 0.912 ($R$=20) respectively. The size distortion is particularly severe when only one simulation draw is used for each $X_i$. The size distortions are not as severe as this case when the number of inequalities is relatively low (e.g. $J$=2 and 5). However, the coverage probabilities are still below the nominal level in all cases.\footnote{We note that the analytical confidence interval is also undersized when $J$ is large. This is due to the fact that the critical value is also calculated by a simulation based approximation. However, the magnitude of the distortion is limited (at most 2%).}
This experiment shows that simulation errors can have nontrivial impacts on the finite sample performance of the confidence intervals. In particular, size distortions can be severe in models with moderate to many moment inequalities. Heuristically, the size distortion arises because the boundary of the confidence interval is an irregular transformation of the underlying moment functions. When the analytical moment $\Phi(X_{j,i})$ is replaced with the simulation counterpart $\frac{1}{R}\sum_{r=1}^{R}1\{u_{j,i,r}<X_{j,i}\}$, an approximation error (of order $O_p(R^{-1/2})$) remains. To see the effect of this, write the right end point of $\mathcal C_{n,R}^{\text{Sim}}(c)$ as
where $r_{j,i}\equiv\frac{1}{R}\sum_{r=1}^{R}1\{u_{j,i,r}<X_{j,i}\}-\Phi(X_{j,i})$. Taking the minimum introduces a downward bias to the estimated boundary. In other words, the right end point of the confidence interval gets pushed inward, reducing the coverage probability. This bias tends to be more severe when there are many binding moment inequalities with non-negligible approximation errors. The naive critical value does not take this into account. In finite samples, where the variation of $\frac{1}{n}\sum_{i=1}^n r_{j,i}$ is not negligible, ignoring the effect of the simulation error may therefore result in misleading inference.
Predicted variables have similar effects on inference. Slightly modifying Example (ref), consider the restrictions
where $F_j$ is an unknown function, which can be estimated separately. As discussed earlier, it is common in empirical practice to estimate some functions (such as the profit function in Example (ref)) before conducting inference based on the moment inequalities. Let $N_1\in \mathbb N$ denote the number of observations used to estimate $F$ in the first stage and let $\hat F_{j,N_1}$ be the first-stage estimator of $F_j$. If $F_j$ is known up to a finite-dimensional parameter $\gamma\in \mathbb R^{d_\gamma}$, it can be estimated by a parametric first-stage estimator $\hat F_{j,N_1}(\cdot)=F_j(\cdot;\hat\gamma_{N_1})$. It can also be estimated nonparametrically or with simulation. Replacing $F_j(X_{j,i})$ with its prediction $\hat F_{j,N_1}(X_{j,i})$ introduces an approximation error $r_{j,i}=\hat F_{j,N_1}(X_{j,i})-F_j(X_{j,i})$, which often satisfies $r_{j,i}=O_p(N_1^{-\eta})$ for some $0<\eta\le 1/2$.\footnote{The rate depends on the estimator and assumptions imposed on $F$. For parametric problems, it is common to have $\eta=1/2$, while $\eta<1/2$ is common for nonparametric problems.} Therefore, ignoring the variation of the first-stage error $r_{j,i}$ can have a consequence similar to the one discussed above.
The magnitude of the first-stage error is of order $N_1^{-\eta}$, which must be evaluated in context. For instance, the total number of observations in the first stage may be large, but if the first stage uses location fixed effects, the relevant $N_1$ is the number of observations in each location, which may be quite small in some cases. Note that in recognition that first-stage estimation error may be an issues, Holmes2011 and HoudeNewberrySeim2017 implement a procedure similar to the first procedure we discuss (in Section (ref)).
We note that the irregularity mentioned above does not arise in classical moment equality models. For comparison purposes, we briefly discuss this point. Suppose that $\theta=\theta_0$ is the unique solution to the moment equality restrictions:
for $J\ge d_\theta$. A method of simulated moments (MSM) estimator $\hat\theta_{n,R}$ is defined as
where $\hat m_R(X_i,\theta)=(\hat m_{1,R}(X_i,\theta),\cdots,\hat m_{J,R}(X_i,\theta))'$. Confidence regions can be constructed around $\hat\theta_n$. Under regularity conditions that ensure asymptotic normality, the MSM estimator depends on the sample moments in a regular manner.
Let $D_0=\nabla_\theta E_P[m(X_i,\theta_0)]$ and $\Omega_0=E_P[m(X_i,\theta_0)m(X_i,\theta_0)']$. It is well-known that, under regularity conditions, the MSM estimator is asymptotically linear in the sense that
where the influence function $l(x,u,\theta_0)$ has zero mean and measures the (first-order) effect of each observation $(x,u)=(X_i,u_{i,r})$ on the variation of the estimator and hence determines the asymptotic variance of $\hat\theta_{n,R}$ Newey:1994aa,GourierouxMontfort1996. In sum, the first-order effect of simulation enters only the asymptotic variance of the estimator.
To see the effect of simulation on confidence intervals, consider a slight modification of Example (ref) where $\theta_0$ solves
It is straightforward to show that an MSM estimator with equal weights on the moment conditions is given by $\hat\theta_{n,R}=(\bar{m}_{1,n,R}+\bar{m}_{2,n,R})/2$. A one-sided confidence interval on $\theta_0$ can be constructed as
where $c$ is the $1-\alpha$ quantile of the asymptotic (normal) distribution of the MSM estimator. The form of the confidence interval shows that (i) the boundary of the confidence interval depends on the sample moments in a smooth manner; and hence (ii) simulation errors in the MSM estimator affects $\mathcal C^{\text{MSM}}_{n,R}$, but it can easily be accounted for by adjusting $c.$ That is, the increased variance of the MSM estimator can be accommodated using a suitable estimator of the asymptotic variance that accounts for $R$ being finite. For moment inequalities, however, it turns out that this type of variance correction is not enough. As we show in Section (ref), one also needs to account for a potential bias in the estimated boundary.
Before proceeding further, we illustrate a common feature of empirical examples that can potentially cause serious size distortions if simulation is used naively. The distortion can be particularly severe when the number of draws is small. To highlight this, we design a data generating process based on the entry game in Example (ref).
Let $Z_{i}=(Z_{1,i},Z_{2,i})$ collect the observable characteristics of the two firms. We let $Z_i$ be generated as a discrete random vector supported on a finite set $\mathcal Z=\{z_k,k=1,\cdots,K\}$. Table (ref) gives the distribution and support of $Z_i$. In empirical studies, it is a common practice to classify continuous state variables into a finite number of bins. When such discretization is used, some bins may contain a limited number of observations. The distribution in Table (ref) emulates this feature by assigning low probabilities to some bins.
We generate unobservable characteristics $u_i=(u_{1,i},u_{2,i})$ as a bivariate standard normal vector independent of $Z_{i}$. For simplicity, we assume symmetry between the firms $(\beta=\beta_j,\Delta=\Delta_j,j=1,2)$ and set $\theta\equiv(\beta,\Delta)=(0.9,-0.5)$. For some values of $u_i$, the model predicts both $Y_i=(1,0)$ and $(0,1)$ as multiple equilibria. If this is the case, we select the equilibrium $Y_i=(1,0)$ with probability $0.7$ (independent of $(Z_i,u_i)$). The knowledge on the selection mechanism is not used for inference, and hence the agnosticism about the equilibrium selection rule leads to partial identification of parameters. Figure (ref) shows the identified set $\Theta_I$ for $\theta$ based only on the moment inequalities (ref)-(ref).\footnote{To focus on the non-standard effects through the moment inequalities, we drop the moment equality restrictions in this exercise. Due to the presence of covariates with 15 support points, there are a total of $30$ unconditional moment inequalities.} In what follows, we report coverage probabilities on one of the extreme points $\theta^{U}=(0.8880,-0.4015)$ of $\Theta_I$, which gives the upper bound on the competitive effect parameter $\Delta$.
Using the specification above and instrument functions $1\{Z_i=z_k\},k=1,\cdots,K$, we transform the conditional moment inequalities in (ref)-(ref) into the following unconditional moment inequalities:
where the entry probabilities
are calculated using either an analytical expression or a frequency simulator. For example, using the parametric specification, $H_{1}(Z_i,\theta)$ may be computed analytically as $\Phi(-Z_{1,i}'\beta)\times \Phi(Z_{2,i}'\beta)$. Alternatively, using a simulator one may compute the same object as $ R^{-1}\sum_{r=1}^R1\{u_{1,i,r}\le -Z_{1,i}\beta,u_{2,i,r}>-Z_{2,i}\beta\}$, where $(u_{1,i,r},u_{2,i,r}),r=1,\cdots,R$ are drawn from the bivariate standard normal distribution.
Our benchmark inference procedure is implemented as follows. The confidence region takes the form: $CS_{n}=\{\theta\in\Theta: T_{n,R}(\theta)\le c_{n}(\theta)\}$, where $T_{n,R}$ is the statistic proposed by Rosen:2008aa and further refined by AndrewsJia2012:
where $\tilde \Sigma_{n,R}$ is a suitable estimator of the asymptotic variance of the moments. The critical value $c_{n}(\theta)$ is computed using a bootstrap procedure combined with the generalized moment selection (GMS) procedure Andrews:2010jk,AndrewsJia2012.\footnote{We use $\tilde{\Sigma}_{n}(\theta)=\hat{\Sigma}_{n}(\theta)+\max\{0.012-\text{det}(\hat{\Omega}_{n}(\theta)),0\}\hat{D}_{n}(\theta)$, where $\hat{D}_{n}(\theta)=Diag(\hat{\Sigma}_{n}(\theta))$ and $\hat{\Omega}_{n}(\theta)=\hat{D}_{n}^{-1/2}(\theta)\hat{\Sigma}_{n}(\theta)\hat{D}_{n}^{-1/2}(\theta)$. We set $\kappa_{n}=n^{1/16}$ for the GMS parameter. } For details on the GMS procedure, we refer to the references above, but we briefly describe its mechanism to highlight the potential effects of simulation on this procedure. The key idea of the GMS is to compute the critical value by selecting the moments that are relevant to the asymptotic null distribution of the test. For example, in an implementation of the GMS, the $j$-th moment inequality is “selected” and used to calculate $c_n$ if the studentized moment is smaller than a tuning parameter $\kappa_{n}$, i.e. $\frac{\sqrt n\bar{m}_{j,n,R}(\theta)}{\hat{\sigma}_{j,n,R}(\theta)}\le \kappa_{n}$, where $\hat{\sigma}_{j,n,R}^{2}(\theta)$ is the $j$-th diagonal element of $\hat{\Sigma}_{n,R}(\theta)$. The critical value is then computed as the $1-\alpha$ quantile of the bootstrapped statistic where the sample moments are replaced with the bootstrap analog of the selected moments. We note here that the simulated moments could also potentially affect the GMS step, but its consequence is not immediately clear.
Table (ref) reports simulation results based on sample size $n\in\{250,500, 1000,2000\}$ and the number of simulation draws $R=\{1,5,10,20\}$. We simulate $1000$ datasets for each setting. The coverage probabilities of the confidence regions with nominal level $95\%$ are evaluated for the true upper bound $\theta^{U}$.
The main finding is that, for small sample size and simulation size, the coverage probabilities of the confidence region are distorted in a significant way. In particular, with $n=250$, the coverage probabilities of the simulation-based confidence regions vary from 39.1% to 45.3%, significantly below the nominal level. Even for a moderate sample size, these confidence regions exhibit size distortions when the number of draws $R$ is small. For example, for $n=2000$, the coverage probabilities of the simulated-based confidence region is 75.6% when only a single draw is used. However, this gets improved (to 93.5%) with $R=5$ draws. Table (ref) suggests that, with $R=20$, the coverage probabilities of simulation based confidence regions are close to those of of the analytical ones in relatively large samples ($n=1000,2000$).
This experiment suggests that the test statistic may not be able to benefit from averaging of the simulation errors when the number of observations in each bin is small. Combined with the irregular nature of the test statistic, this can result in a severe size distortion. The presence of bins with small numbers of observations is common in empirical settings. In such cases, care must be taken.\footnote{We conjecture that the same comment applies to conditional moment inequality models if the essential sample size is small (i.e. small $n$ and bandwidth $h^{d_Z}$).} One possibility may be to increase the number of simulation draws for such bins. However, the choice of the number of draws becomes an arbitrary component of inference. Another possibility is to modify the procedure to explicitly account for the effects of the simulation. In the next section, we explore several possibilities along this line.
As a final remark, we note that the simulation not only affects the value of the test statistic but also the general moment selection procedure. Table (ref) reports the number of times (out of 1000) the set of the selected moments differ between the simulated and analytical methods. This difference arises because the simulation adds perturbation to the studentized moment. With a limited number of draws, this effect on the GMS may not be negligible. As a result, some of the “selected” inequalities (i.e. $\sqrt{n}\bar{m}_{j,n}(\theta)/\hat{\sigma}_{j,n}(\theta)\ge -\kappa_{n}$) according to the GMS based on the analytical moment may be discarded (i.e. $\sqrt{n}\bar{m}_{j,n,R}(\theta)/\hat{\sigma}_{j,n,R}(\theta)<- \kappa_{n}$) under the GMS based on the simulated moment, and vice versa.
We consider two methods to account for the simulation errors. One is to correct the critical value. The other is to correct or “regularize” the test statistic. The former method closely follows the recent development on the moment inequality literature, and hence we keep its discussion minimal. The second method is a novel approach and has several attractive features for simulation based methods. It combines a simple bias correction method with a bootstrap critical value based on a regular (and differentiable) functional of a random vector that is asymptotically normal. To our knowledge, the uniform validity of such a method in the context of simulation-based inference is new to the literature.
Commonly used inference methods provide approximations to the distribution of the test statistic in (ref). Many of these methods are based on resampling techniques. As we saw in the previous section, ignoring the variation due to simulation can result in poor inference. Recall that, in the numerical experiment in Section (ref), the size distortion occurred because the naive critical value did not take into account the bias that was due to the extra variation from the approximation error $\frac{1}{n}\sum_{i=1}^n r_{j,i}$ (in (ref)). A straightforward way to correct this effect is to adjust the critical value. One can achieve this by a bootstrap critical value that resamples the simulated variables $u_{i,r}$ along with the original sample $X_i$ across bootstrap replications. Specifically, let $\{X^*_i,i=1,\dots,n\}$ be a bootstrap sample drawn with replacement from the empirical distribution. For each $i$, let $\{u^*_{i,r},r=1,\dots,R\}$ be drawn from $P(\cdot|X^*_i)$.
We then let $\hat m^*_{R}(W^*_i,\theta)=R^{-1}\sum_{r=1}^RM(X^*_i,u^*_{i,r},\theta)$ be the simulation approximation to the (conditional) moment function in the $i$-th bootstrap sample. Define
This resampled moment is constructed so that it mimics the behavior of $\bar m_{n,R}$ including the variation due to the simulated variables. The final step is to use the resampled moment above when one applies an existing method for computing the critical value.\footnote{Due to the non-standard nature of $S$, consistently approximating the limiting distribution of $T_{n,R}$ uniformly over a large class of DGPs is not possible in general. Hence, the existing methods resort to conservative distortions.} For example, consider the generalized moment selection (GMS) in Andrews:2010jk. Let $\hat D_{n,R}(\theta)=\text{diag}(\hat\Sigma_{n,R}(\theta))$ and let $\hat \Omega_{n,R}(\theta)$ be an estimator of the correlation matrix, i.e. $\hat \Omega_{n,R}(\theta)=\hat D_{n,R}(\theta)^{-1/2}\hat\Sigma_{n,R}(\theta)\hat D_{n,R}(\theta)^{-1/2}.$ This procedure uses the $1-\alpha$ quantile $c_{\kappa,n,R}(\theta)$ of the following statistic
where $Z^*_{n,R}(\theta)=\sqrt n(\bar m^*_{n,R}(\theta)-\bar m_{n,R}(\theta))$ is the bootstrapped empirical process, and $\xi_{n,R}$ is a $J\times 1$ vector whose components are
The generalized moment selection (GMS) function $\varphi$ then selects the inequalities that are relevant for the inference for $\theta$ based on $\xi_{n,R}$ (see Andrews:2010jk for details). The resulting confidence region is
In Example (ref), applying this method with a $t$-test based GMS function ($\varphi(\xi_j)=0$ if $\xi_j\ge -1$ and $=-\infty$ otherwise) yields the following confidence interval
where the critical value $c_{\kappa,n,R}$ is the $1-\alpha$ quantile of the bootstrapped statistic\footnote{The critical value and the standard deviations in this example are not indexed by $\theta$ because the re-centered moment functions $\bar m_{j,n,R}(\theta)-E[m_j(X_i,\theta)],j=1,\dots,J$ in the test statistic do not depend on $\theta$.}
The confidence interval in this example is the intersection of the sample (upper) bounds that are suitably expanded. The amount of the expansion $c_{\kappa,n,R}\frac{\hat\sigma_{j,n,R}}{\sqrt n}$ accounts for the variation of each simulated moment. This type of confidence interval is also considered in the context of conditional moment restrictions in Chernozhukov:2013fj.
In addition to the modification of the existing methods, we propose a novel and computationally simple inference procedure that accounts for simulation. The key idea is as follows. Recall that commonly used test statistics take the form: $T_{n,R}(\theta)=S(\sqrt n\bar m_{n,R}(\theta),\hat\Sigma_{n,R}(\theta))$ whose irregular behavior arises due to the non-smoothness of $S$. Our procedure replaces $S$ by a smooth approximation $S_\mu$, which satisfies certain regularity conditions. This approach has several attractive features. First, the proposed method has a uniform approximation property. That is, for any $(m,\Sigma)$, $|S_\mu(m,\Sigma)- S(m,\Sigma)|\le \beta\mu$ for a known uniform constant $\beta$ and the degree of smoothness $\mu>0$, which is chosen by the researcher. Accounting for the approximation error is then straightforward because $\beta$ is known. Second, the approximated test statistic $\tilde T_{n,R}(\theta)\equiv S_\mu(\sqrt n\bar m_{n,R}(\theta),\hat\Sigma_{n,R}(\theta))$ obeys standard limit theorems uniformly over a large class of DGPs and over a range of values for $\mu$. This in turn allows one to employ a standard resampling method such as bootstrap to calculate the critical value. Finally, smooth approximations to a wide class of non-smooth convex functions are available thanks to the recent developments in the non-smooth convex optimization literature Nesterov:2005qv,Beck:2012xy. Using these results, we provide functional forms of smooth approximations to some of the commonly used test statistics. The idea of regularizing test statistics (or estimated bounds) also appears in related contexts HaileTamer2003,Chernozhukov:2015aa,Kaido:2016aa,Masten:2017aa. Our contribution here is to show its uniform validity in the context of inference with simulated variables. A potential price for this computationally simple method is the possibility of inference becoming conservative for some choice of the smoothing parameter. We will examine this point numerically in Section (ref).
Below, we illustrate our approach using Example (ref). \setcounter{example}{0}
In general, we define our confidence set by
where the test statistic and the critical value are calculated as follows
where $c_{n,R,1-\alpha}$ is an estimate of the $1-\alpha$ quantile of
Here, $S_\mu$ is an approximation to $S$, which has smoothness properties that are useful for analyzing and correcting the behavior of the test statistic in the presence of simulated variables. We introduce the following notion of approximation based on Beck:2012xy.\footnote{The third condition in Definition (ref) is not required in Beck:2012xy but is satisfied by all $\mu$-smooth approximations we use in this paper and is useful for establishing asymptotic results.}
Definition (ref) (i) requires that $\phi_\mu$ has a uniform approximation property, which is the key condition for bias correction. Definition (ref) (ii) requires that $\phi_\mu$'s derivative is Lipschitz continuous, which plays a role in making the limiting distribution of $Z_{\mu,n,R}$ depend continuously on the underlying DGP. Given this definition, we require $S$ and $S_\mu$ to satisfy the following condition. For this, let $\mathbb P^J$ be the set of $J\times J$ symmetric positive semi-definite matrices.
In what follows, we also call $S_\mu$ the $\mu$-smooth approximation of $S$. Table (ref) gives the index functions we consider and their $\mu$-smooth approximations with associated parameters. Details are provided in Appendix A. These functions satisfy Assumption (ref).\footnote{It is also possible to consider the index function $S(m,\Sigma)=\inf_{t\in\mathbb R^J_{-}}(m-t)'\Sigma^{-1}(m-t)$ considered in Rosen:2008aa whose $\mu$-smooth approximation is given by $S_\mu(m,\Sigma)=\max_{u\in \mathbb R^J_+}m'u-\frac{1+2\mu}{4}u'\Sigma u$ with $\chi=2$. However, the statistic based on this approximation requires a second-order $\delta$-method to obtain a valid limiting distribution, which makes the analysis more complicated. As such, we leave this possibility for future research.}
In summary, we propose the following procedure to construct confidence regions.
Algorithm 1:
Similar to the consistent estimation of the asymptotic covariance matrix in the MSM literature, we use a simulation sample of larger size in the algorithm GourierouxMontfort1996. This is to re-center the bootstrapped root at an object that tends to the population counterpart and mimic the behavior of the root in (ref) by the bootstrap sample. While $R_2$ needs to be large for the asymptotic approximation to be valid, this simulation needs to be done only once.
Below, we establish the asymptotic validity of the inference procedure described above. For each $j$, we let $\sigma^2_{P,j}(\theta)\equiv Var_P(\hat m_{j,R}(X_i,u_i,\theta))$ and $\Omega_P(\theta)\equiv Corr_P(\hat m_{j,R}(X_i,u_i,\theta))$, where $\hat m_{j,R}(X_i,u_i,\theta)=R^{-1}\sum_{r=1}^R M_j(X_i,u_{i,r},\theta)$. We then let $D S_\mu$ denote the gradient of the map $m\mapsto S_\mu(m,\Sigma).$ Let $\Psi$ be the set of $J$-by-$J$ correlation matrices $\Omega$ with $\det(\Omega)>\epsilon$ for some $\epsilon>0$. We make the following assumption on the model.
Assumption (ref) (i)-(vi) are based on the conditions in Andrews:2009aa,Andrews:2010jk. In (iii), we assume an i.i.d. sample. With this assumption, the following estimator of the asymptotic covariance can be used:\footnote{While we establish validity of inference for i.i.d. samples, one could potentially relax this assumption and allow for strictly stationary and strongly mixing data by adopting an alternative estimator of the asymptotic covariance and modifying the bootstrap procedure properly. See Andrews:2009aa for the inference framework that allows such data.}
We then let $\hat V_{n,R}(\theta)=\text{diag}(\hat\Sigma_{n,R}(\theta))$. For the purpose of stating an assumption, let us also define an infeasible estimator of $\Sigma_P$ as follows:
This estimator can be computed only if simulation is not required. We then let $\hat V_{n}(\theta)=\text{diag}(\hat\Sigma_{n}(\theta))$.
In Assumption (ref) (v), $\Psi$ contains all $J\times J$ correlation matrices whose determinant is bounded from below by $\epsilon>0$. This condition is required for one of the index functions used in Andrews:2010jk. In our setting, we use this condition to ensure that the limiting distribution of $Z_{\mu,n,R}$ is continuously distributed. With additional notation, this assumption can be relaxed so that the lower bound on the determinant is required only for the correlation matrix of a suitable subset of the moment functions.\footnote{See Kaido:2017aa for generalization of the condition along this line.} Or it can be dropped entirely at the price of an additional tuning parameter to handle a potentially discontinuos limiting distribution.
Assumption (ref) (vii) is an additional condition, which we add to the standard set of assumptions. It requires that the gradient of the smoothed index function $S_\mu$ does not vanish. This allows us to use the (first-order) $\delta$-method. For the functions in Table (ref), the condition is satisfied when $\theta\mapsto E_P[m_j(X_i,\theta)]$ is continuous and $\sigma_{P,j}(\theta),j=1,\cdots,J$ are uniformly bounded away from 0.
The following theorem ensures that the proposed confidence region controls the asymptotic confidence size uniformly over the parameter space $\mathcal F$.
In this section, we show results of main Monte Carlo simulations to examine the performance of the methods that account for the simulation error.
Following our example on the intersection bounds in Section (ref), we first use the method that corrects the critical value described in Section (ref). We construct confidence intervals for $\theta $ with level $1-\alpha$ using:
where $\bar{m}_{j,n,R}=(nR)^{-1}\sum_{i=1}^{n}\sum_{r=1}^{R}1\{u_{j,i,r}<X_{j,i}\},$ $\hat{\sigma}_{j,n,R}$ is the estimated standard deviation of the $j$-th simulated moment, and $c_{\kappa,n,R}$ is a critical value computed as the $1-\alpha$ quantile of
where $(\bar{m}_{j,n,R}^{*},\hat{\sigma}_{j,n,R}^{*})$ is the bootstrap quantities corresponding to $(\bar{m}_{j,n,R},\hat{\sigma}_{j,n,R})$, and $\xi_{j,n,R}$ is as defined in (ref) with $\kappa_n=\sqrt{\ln n}$. When $T^*_{n,R}$ is computed, we redraw simulation draws $\{u_{i,r},r=1,...R\}$ to account for simulation variations when simulating the bootstrap samples.
We also consider our inference procedure based on the regularized statistic. Specifically, we use $\phi_{\mu}(\bar{m}_{j,n,R})=-\mu \ln\sum_{j=1}^{J}e^{-\bar{m}_{j,n,R}/\mu}$ to approximate $\phi(\bar{m}_{j,n,R})=\min_{j=1,..J}\bar{m}_{j,n,R}$ used in computing the confidence region in equation (ref). We then construct confidence intervals for $\theta$ with level $1-\alpha$ using:
where $\tilde{c}_{n,R}$ is an estimated critical value in (ref). We also redraw simulation draws $\{u_{i,r},r=1,...R\}$ when simulating the bootstrap samples. In the experiments, we use $R_2=100$ to compute $(\bar m_{n,R_2},\hat\Sigma_{n,R_2})$ in the bootstrap step (Step 2 in Algorithm 1). For both confidence intervals, $S=1000$ Monte Carlo replications are generated.
Table (ref) reports the probabilities of the confidence intervals covering the upper bound $\theta^U$ of the identified set using the critical value correction. The coverage probabilities of the confidence intervals are all above the nominal level after correcting the critical value. This can be contrasted with the coverage probabilities of the confidence intervals without any correction. The critical value correction method tends to make the confidence interval somewhat conservative when $R$ is small. In some cases, the coverage probability is very close to 1 and the corrected confidence interval is substantially longer than the one based on the regularization method as we discuss below.
Table (ref) reports the coverage probabilities of the confidence intervals based on the $\mu$-smooth approximation. We set the smoothing parameter to $\mu=0.02$ and $0.04$ in our Monte Carlo experiments. The coverage probabilities of $\tilde{\mathcal C}_{n,R}$ are all above those of the confidence intervals without correction and close to the nominal level in many cases. These results indicate that the inference procedure with the $\mu$-smooth approximation is an effective method for correcting the size distortion caused by the finite number of draws.
Tables (ref) and (ref) report the median length of the confidence intervals. The median length of a one-sided confidence interval is computed as the difference between the median of the upper bound of the confidence interval and the right end point of the identified set. Table (ref) shows that the confidence interval with the critical value correction is often much longer than the confidence interval without any correction, which is consistent with the robust size control property. The difference between the two shrinks as $R$ gets large. Comparing the two tables, one can see that the regularization based confidence intervals $\tilde{\mathcal C}_{n,R}$ are often significantly shorter than $\mathcal C_{n,R}^{\text{CV}}$. A close inspection of the simulation results showed that this was because the regularized statistic had a smaller variance that that of the non-regularized statistic, which led to shorter confidence intervals even after the bias correction.\footnote{This may be a generic feature of the $\mu$-smooth approximation method. We leave its general analysis for future work.}
The overall pattern remains the same even in the presence of locally slack constraints. Tables (ref)-(ref) report the coverage probabilities and excess length of the confidence intervals when some of the constraints are slack. The slack constraints are introduced by shifting $X_j$'s mean by $1/\sqrt n$ for the first $J/5$ constraints. One can see that both correction methods achieve valid coverage across all values of simulation draws. The regularization based confidence interval becomes slightly more conservative in terms of coverage probabilities compared to the case without the slack constraints. However, its length is still shorter than that of the critical value correction confidence interval across all cases.
In sum, the simulation results show that the regularization based confidence interval $\tilde{\mathcal C}_{n,R}$ works well both in terms of size and length. Its size is controlled reasonably well even under some DGPs that make simulated confidence intervals without correction severely undersized. The critical value correction method also achieves robust size control. However, it tends to be overly conservative when $R$ is small.
This paper explores the effects of simulated moments on the performance of inference methods based on moment inequalities. Due to the irregularity of the boundary of the confidence regions, simulation errors can affect the performance of inference in non-standard ways. This can result in a severe distortion especially when the number of inequality restrictions is large and the essential sample size is small. To account for the effect of the simulation error, we propose a novel way to construct confidence regions using regularized statistics and establish an asymptotic size control result. The simulation results confirm the robust size control property of the proposed method. An interesting avenue for future research is on the choice of the smoothing parameter that accounts for the trades-off between the amount of bias correction and variance of the regularized statistic.